Guilty reading
Product management depends on the ability to read the tea leaves and foresee a distinct future that is both appealing and achievable. We should embrace its uncertainty.
“To read fiction means to play a game by which we give sense to the immensity of things that happened, are happening, or will happen in the actual world. By reading narrative, we escape the anxiety that attacks us when we try to say something true about the world.”
― Umberto Eco
Product managers engage with narratives daily — those from our users, our customers, our co-workers, the companies we work for — in order to create new narratives that help drive alignment and action between our teams and stakeholders, and ultimately towards delivering change. But for us to understand what happened, is happening, or will happen (in the near future) in the worlds of our users and consumers, and then turn those events into desirable, sustainable change, we should excel at interpretation and avoid the pitfalls of fossilizing processes.
Who tells a tale adds a tail
In her book Continuous Discovery Habits, Teresa Torres discusses how the brain will do everything it can to create a coherent story, even when it lacks the facts to support it. She says¹
… we tend to come up with coherent reasons to explain our behavior that are often not grounded in reality. To be clear, this behavior is a function of how our brains work and not the result of the interview participants trying to deceive us. In fact, many of these biases come into play because our interview subjects are trying to be helpful.
Product managers should keep this in mind when talking to users and customers and prompting them for information. As the human brain is better at answering questions than providing facts, people instinctively convey information in the ways they believe will reduce attrition with others. In any interaction, we create the stories of who we would like to be perceived as (ideal), not who we actually are (factual), making multiple real-time judgement calls that place us on a spectrum between being specific or vague, absolutely candid or completely dishonest, volunteering lots of information or barely just enough.
One tool we often use to strike a balance is generalization. It makes us sound more reasonable (more logical, less instinctive) and hopefully more compelling: saying I ate salad regularly in 2023 paints a more positive image than I ate salad once a month in 2023. Even if both sentences are strictly speaking true, the former speaks to the ideal and the latter, to the factual.
While not necessarily bad in and of themselves, generalized statements can become an obstacle to product managers. Part of the job is to generalize from specifics based on limited data, and we are constantly evaluated on our accuracy. Ivan told me he eats salad regularly, so if I assume he eats salad weekly and plan my hypothetical fresh produce delivery product around that metric, I’m likely going to fail.
Our ability to combine and interpret the ideal and the factual, the qualitative and the quantitative, is what enables us to estimate at which future point we can have the largest impact for our business based on when we expect our teams to be able to deliver the most meaningfully valuable change to a significant group of users and/or customers.² By the same token, understanding the context from which someone speaks — their place of speech³ — enables us to create a broader field of opportunities in and from which to work, thus increasing our chances of creating relevant change.
However, we can only see through generalizations or fabrications when we have clues to perceive them as such: shared social conventions enable us to understand when someone is being deliberately broad or specific, and both verbal and non-verbal clues given by the speaker serve as indicators of their confidence in what they are saying. Without these, we are left to guess what exactly was the intent of the message, which is one of the reasons why it is more difficult to gauge intent from a text than from spoken voice. If I understand Ivan’s context, I can ask him whether regularly means daily, weekly, or monthly.
In the absence of qualitative context — and increasingly, instead of it — we resort to hard data as a proxy to infer meaning or intent, losing valuable information in the process. Writing for Mind The Product, Matt LeMay says
When people talk about “data-driven product management,” there is often an unspoken assumption that this only means quantitative data — information that can be captured by numbers. This assumption is profoundly dangerous, because it often leaves us unable to speak to why our assumptions or hypotheses might be playing out in the way that they are.
When qualitative information is presented as a statement of fact, we must search for hard data to corroborate or disprove it. And this is not always so simple.
Loose lips may sink ships
Case in point: in April 2023, the Washington Post published a story about how ChatGPT fabricated information accusing a law professor of sexually harassing a student.
While conducting a research on the legal implications of AI-generated content, the AI was prompted about instances of sexual harassment by professors in US law schools and to include quotes from relevant articles. The response it provided cited articles from reputable publications in the US — including The Post itself and the LA Times — which were never published.
Understanding how and why AIs generate the results they do is difficult. Prompts will almost always generate a response, with a few exceptions — eg, if there’s no data to extrapolate from or if the request goes against explicit rules in its code. Similar to humans, AIs will strive to provide an answer, even if it is a complete fabrication. They work based on valid logical processes such as inference, induction, and generalization, but left unchecked these can also lead to a form of gossip and hearsay, as shown by The WP story:
[…] The Post re-created [Eugene] Volokh’s exact query in ChatGPT and Bing. The free version of ChatGPT declined to answer, saying that doing so “would violate AI’s content policy, which prohibits the dissemination of content that is offensive [or] harmful.” But Microsoft’s Bing, which is powered by GPT-4, repeated the false claim about [Jonathan] Turley — citing among its sources an op-ed by Turley published by USA Today… outlining his experience of being falsely accused by ChatGPT.
In other words, the media coverage of ChatGPT’s initial error about Turley appears to have led Bing to repeat the error — showing how misinformation can spread from one AI to another.
In trying to produce a positive response to the user’s prompt, tools like ChatGPT create compelling-sounding stories that are more concerned with answering the prompt/question than with being factual about it. It seems, then, our creations mirror our own image: the same behavior is also present in humans, as shown by Michael Gazzaniga’s left brain interpreter study (also discussed by Torres).⁴
The WP story illustrates two aspects of interpretational processes that are quite relevant for product managers. More broadly, it discusses the difficulties of accessing factual information — something difficult to do in any case, but perhaps even more so when involving AIs due to the black box they are placed in and the potential reach and real-world consequences of a statement (ie, any user running the same query could be exposed to it). And on a more specific level, it highlights how having more quantitative data does not necessarily make a problem easier to solve. Giving the trained model access to more data did not make its answers more reliable, because it lacks the ability to interpret. Doing so provided it with more information from which to infer and generalize, not a means to fix the problematic statements.⁵
Quantitative data then helps us answer how relevant something is (because of frequency or intensity). Qualitative data, however, enables us to understand why something is relevant. Quote LeMay:
We might be able to see, for example, that users are not clicking through on our new feature as much as we had hoped they would. But without qualitative data — information that can be captured by words and stories — we are often powerless to act upon quantitative trends.
One very good example of this is the New Coke. It was launched in the US in April 1985, at a time Coca-Cola’s market share lead over Pepsi had been slowly slipping for 15 consecutive years. The assumption was that consumers preferred Pepsi’s sweeter taste. As a response, the company designed a new formula that was approved in nearly 200.000 blind tests, indicating a clear preference for it over the competitor’s. Two months after launch, the company was getting almost 300% more complaint calls from customers. New Coke remained available for nearly three months before a recall was issued and a return to the old formula was announced in July that year.
What happened? What Coca-Cola failed to do was to qualify their planned change with their existing market. Coca-Cola put it this way,
What these [blind] tests didn’t show, of course, was the bond consumers felt with their Coca‑Cola — something they didn’t want anyone, including The Coca‑Cola Company, tampering with.
No such thing as an innocent reading
To me, product management is interpretative work at the nexus of both the quantitative and the qualitative. Ultimately, our most valuable asset is our capacity to frame a qualitative better world through the use of quantitative data that can challenge our conclusions and assumptions. Every finished product is the best it can be — and also the foundation of a new, even better product. As such, it exists permanently between two qualitative states that can only be measured by our goals in each venture:
- judged through the lens of v2, v1 of a product is always lacking
- conversely, v1 cannot be taken as being correct by default, otherwise we have no room in which to improve
- progress can only be measured by how far we’ve come towards the arbitrary qualities agreed upon for the future version
Marty Cagan wrote about being worried with the over-processification of the profession. I believe many people share those concerns — especially after some CEOs and organizations started re-framing PMs or doing without them entirely.
To my eyes, we product managers lose our purpose in the organizations we serve when we hide behind one or more of the countless flavors of processes, frameworks, and 10-steps-to-become-a-better-PM articles that year in, year out promise to turn it into a simple job that anyone can do. We lose value because process alone cannot tell us where we are, only how far we are from the qualitative goal we (team + stakeholders) agreed upon.
Purely quantitative data also cannot help. It can only show us the world as is, and at best describe a projected trend given existing references and models — the projection is always biased.⁶
Neither process nor quantitative data can set goals: objectives are qualitative by definition. No product can be created or changed without a qualitative decision.
And while I do not dispute that anyone could do product work, I do not believe interpretation as a skill to be something that can be crystallized as a framework or process; it relies too much on our individual reading of the world and everything in it to be successfully reduced to a set of fixed steps:
- a broad set of personal references;
- the ability to navigate and switch between different sets of distinct social, cultural, and economical contexts;
- the ability to apply logical processes to limited information sets, to generalize and filter; and most of all,
- it asks practitioners to question everything and to not adopt anything blindly.
It is not baking — rather, it’s more like cooking.
At the end of the day, the variables that influence our reading of both the quantitative and qualitative data are ever changing — even if only so slightly — and it is our job to decide what our teams should be working towards. We get to call out what we believe will generate the most value, what we believe will create the most change. And while we all need some form of workflow, process, or framework to work from, there’s no one-size-fits-all. Change teams within the same company, or change the individuals in the team you’re working in, and you’ll more likely than not need to change the way you work.
We read. We test. We fail. We learn. We adapt. We build.
We read again.
As Louis Althusser said, “there is no such thing as an innocent reading, we must ask what reading we are guilty of.” We should stop trying to find things to bail us out of this responsibility and own it instead.
Notes
¹ Teresa Torres, Continuous Discovery Habits: Discover Products That Create Customer Value and Business Value, pp. 75, May 19, 2021.
² Think about the first iPhone (2007): it was released about five years after the BlackBerry 5810 kickstarted the smartphone party, roughly a year after than the Blackberry 8800 popularized the mobile device market in the US, and six months after the LG Prada came out as the first touchscreen mobile phone — and yet the value users gained from it was exponentially bigger than would have been possible should Apple have focused on matching the immediate competition at any given point. Apple started developing the first iPhone in prototype mode around 2000, but only greenlit the project in 2006.
³ Concept created by Brazilian philosopher Djamila Ribeiro, place of speech (lugar de fala) affirms each and every person has a social locus in the world — a place of speech — that shapes both what they are able to say and how what they have to say is received by others. A brief description can be found here: https://www.uib.no/bnkf/143972/%E2%80%9C-place-speech%E2%80%9D-according-djamila-ribeiro
⁴ Continuous Discovery Habits, pp. 76–77. Torres summarizes Gazzaniga’s study, in which split-brain patients (patients whose connection between left and right sides of the brain has been severed) were shown an image visible only to their left eye. They were then asked to choose a related card with their left hand and to explain why they chose the card they did. The right side of the brain controls the left side of the body (and vice-versa), but language is processed and generated only in the left side. In the study, the left hemisphere had to justify something for which it had no information and was not aware of. Yet instead of being stumped, participants were able to fabricate an answer, even if it had no basis in reality. Later studies have shown this is not limited to split-brain patients.
⁵ To be clear, the same also happens when talking to users and customers who have some knowledge of the process or tool in question. As Stephen Hawking put it, “the greatest enemy of knowledge is not ignorance, it is the illusion of knowledge.”
⁶ For example: given the sequence 1–2–4–8–16, one could assume 32 to be the next value in the chain because of the trend and the model we inferred based on the data. We would be surprised if the next value were 5, and would have to reevaluate our model and projections.
