“When you develop your opinions on the basis of weak evidence, you will have difficulty interpreting subsequent information that contradicts these opinions, even if this new information is obviously more accurate.”
“Most of what people call “insight”, garnered from surveys, focus groups, contextual inquiry, usability tests, and quantitative data analysis, is complete horseshit.” – Me
How often, when debating around a conference room table about a particular product concept or feature, does a member of the team cite an observation, no matter how fleeting, from a usability test participant the previous day? How much credence was given to that observation? Why did it seem like it carried more weight in the team’s decision-making process, even if it was a single observation that may have contradicted observations from 20 previous usability tests? Why is that? What cognitive biases are at play here?
I am a tireless advocate for doing solid research and analysis to inform decision making about the products and features we choose to design, and I am stating quite emphatically that from my observations most user experience and product designers are doing it wrong. Between the cognitive biases we bring to contextual inquiry, focus groups and usability tests to our blatant ignorance of statistics we carry like an albatross when reviewing quantitative data – we’re doing it wrong and wasting a shit-load of money in the process. Worst of all, we think we’re doing it right and we’re proud that – hey – we’re doing solid research whilst all those other schmucks are either doing genius design or stakeholder-driven design such that we don’t even understand how bad our decisions-making is. It gets trickier, because in the previous sentence, I said we don’t even know how bad they are because of 2 more cognitive biases: post-purchase rationalization, that is – we create a false narrative of the benefit to the time and effort we just spent doing research, even when faced with the Semmelweis reflex, which is the tendency to reject new data which contradicts a previously held paradigm (that user research is valuable).
The reason we tend to give more credence to an observed behavior in yesterday’s usability test relative to the previous 20 tests is what is called the Recency Bias, and it’s one of many cognitive biases that plague product design. Simply put, Recency Bias is “a cognitive bias that results from disproportionate salience of recent stimuli or observations — the tendency to weigh recent events more than earlier events.” There are many more. I think there is enough juice here to write an article on cognitive biases in conducting user research and usability testing – but this isn’t that article.
Here is a different problem. How often, when a team needs to make a decision about an A/B test of two slightly different concepts that have been in the market for six weeks, did people make positive and persuasive arguments that one design “won” over another, even with no understanding of the significance of the sample size? If your company is good enough to be doing serious quantitative analysis to aid in decision making, why does it seem reasonable to essentially toss a dart at a board?
“Things always become obvious after the fact.” – Just about everyone.
This fall, I choose to read 4 books not at all about user experience design; or usability testing; or god-forbid – the new cult of “lean startups,” a problematic concept/methodology/religion (whatever), that enjoys the benefit of few peer-reviewed case studies and zero scientific basis as to it’s purported efficacy in creating innovative products or companies. Such is the beauty of faith - evidence need not be an incentive for the faithful to fall on their knees in awe. Karl Popper would roll over in his grave.
But back to the books – I choose to read 4 that had nothing to do with design per se, but everything to do with how we make decisions, what is the nature of our decision making process, and how ignorant are we to our own cognitive biases that both rule or daily lives, and yet are almost completely opaque to us. These books include two by a popular writer on neuroscience, the godfather of behavioral economics, and a proprietary stock trader turned philosopher of science. I recommend reading all four at the same time, or at least in quick succession because in many ways the build on each other.
Here are my four recommendations for your holiday reading. All are quite excellent, and will make you rethink how you make decisions about everything from usability testing to prioritizing feature designs to the simple cognitive biases that govern your everyday life. You will thank me for it. I have included book reviews from more eloquent reviewers than myself, also called the “appeal to authority fallacy,” since none of the reviewers are authorities on math, statistics, behavioral economics, or neuroscience.” 
Fooled By Randomness, Nassim Taleb
Trackbacks
One Trackback