You’re Wrong & Don’t Know It: Selection Biases

A few weeks ago, my parents offered to watch our two-and-a-half-year-old son one evening; and quick to take them up on their offer, my girlfriend and I decided to make dinner reservations. More accurately, she made reservations. 

“I know the perfect place to go,” she said, listing half a dozen of her nearest and dearest, all claiming it’s the best restaurant they’ve been to in years. Naturally, I was leery, but then again, she did list what seemed like a diverse set of people, and my logic was that laws of probability were in my favour. 

The night of our reservation, we arrive at the restaurant to see a lineup practically out the door. Another good sign, I think, especially as we had a reservation. 

Once seated, a young woman placed menus in front of us. Filling up our water, she mentioned that someone would be by shortly to tell us the specials before rushing diligently back to her outpost near the front door. It had been a long day. I skipped lunch, and I couldn’t deny that what was initially dread had turned into anticipation. Maybe it was the soft lighting, the laughter of surrounding tables, or a night out without children present, but I had an undeniable smile as I looked down and prepared to open the menu, feeling like I could eat a horse!

That smile soon left after opening the menu. 

That’s because 5 of those 6 friends who recommended the restaurant were vegetarian (a fact I knew but didn’t account for), and the menu I opened was that of a raw vegan restaurant. 

And that is just a time in recent memory where I’ve been negatively impacted by selection bias. I probably don’t even know it’s happening most of the time.

Selection Bias – It’s All In Who You Ask

“Everything in my own immediate experience supports my deep belief that I am the absolute center of the universe, the realest, most vivid and important person in existence.” – David Foster Wallace

Selection bias is caused when non-random data, individuals or groups are examined, resulting in a failure to represent the intended population to be analyzed adequately. 

In the above instance, I mistakenly assumed that the research group (her six friends) was, in fact, an accurate random representation of the general population’s tastes and preferences. Unfortunately, this caused me to come to the wrong conclusion about the restaurant and well, while the food was delicious, it wasn’t what I wanted that night. At least on a day when I skip lunch.

In the first article, I mention that there’s nothing you can do about biases aside from educating yourself in hopes of recognizing them and mitigating their impact. And that’s true. Unfortunately, when it comes to Selection Bias, or more accurately, Selection Biases, it’s more than one thing you need to be aware of. 

Whether you’re designing a product for an intended target audience or trying to select a place for dinner, having the right data is vital and a big step towards ensuring that you’re able to sidestep the following.

Types of Selection Bias

Anchoring Effect: Among the most common, this bias refers to the human tendency to become overly attached and reliant on the first piece of information offered (or the “anchor”) when making decisions. 

Availability Cascade (Aka, The Truth Effect): This bias stems from the idea that as a piece of information is shared more and more publicly, regardless of its truth or validity, the more likely that piece of information will be considered credible, causing a self-sustaining cycle of belief. 

Confirmation Bias: According to the American Psychological Association, a Confirmation Bias is the tendency to gather evidence that confirms preexisting expectations, typically by emphasizing or pursuing supporting evidence while dismissing or failing to seek contradictory evidence.

Conservatism Bias: When it comes to making decisions, Conservatism is a tendency to inadequately revise and reconsider one’s beliefs after being presented with new information. 

Pro-Innovation Bias: This bias is the tendency to magnify and overestimate the value of an innovative product’s qualities while at the same time downplaying its constraints. In other words, if it looks new and shiny and has a better camera, we assume it’s better. 

Publication Bias: Resulting from the selective publication of information based on the trajectory and perceived impact of said information. 

Recency: This cognitive bias is the tendency for people to favour recent events and the information they provide over historical ones. 

Salience (Vividness Bias): This interesting bias is the tendency to focus on information, data, and details that are most noteworthy and prominent over others that may be more significant but don’t capture our attention. 

Satisficing Bias: This is the bias that refers to a person’s preference to select a satisfactory outcome rather than the optimal one. 

Survivorship Bias: The focusing on people, data, and information that is present and favourable over information that is not. In other words, it’s the ignoring of all information that didn’t survive after evaluation.

Mitigation Mombo – Avoidance Strategies for Selection Bias

Not to sound cliché, but knowledge really is power when it comes to Selection Biases. The trick to overcoming them is to be ruthless in examining the usability, reliability and credibility of all the data and the sources it comes from.

Before relying too heavily on any data, it’s prudent to ask yourself these questions:

• Are your sources reliable? 
  • What are the qualifications and reputable expertise of the source?
  • Do they have a history of accuracy?
  • Is it objective and its motive free from bias or influence?
  • What’s the proximity or relationship to original source of information or event?
• Is your data usable?
  • How comparable is it to other available sources?
  • When was the data obtained? Is it sufficiently recent?
  • Is it representative of your intended population?
  • Does it correlate to your intended subject matter?
• How credible is your data?
  • Is it consistent with data from other sources?
  • How likely is this result, given the context?
  • What information was open for interpretation?
  • Are you confident in the references provided?
• What information is missing?
  • Were enough sources utilized?
  • Have you gotten a second opinion? 
  • If found, did you assess the resources required to fill gaps?
  • Alternatively, are you able to utilize historical references or lessons learned?

Selection Biases: Victim or Accomplice? 

In retrospect, the biggest injustice resulting from my dinner date wasn’t that I had to eat raw vegan food (which was delicious), but rather, I had to stop at a drive-thru on the way home and only have myself to blame. I unintentionally (as often the worse things are) relied on inaccurate data, a consensus from a group that, had I taken the time and even considered the above questions, would know are the furthest thing from my intended population. 

Now, imagine any of the above Selection Biases made their way into something far more impacting (and they do) than what I had for dinner. Because in that scenario, really only I was affected. But when you make decisions, designs, and build products for the masses, the impact of those biases is much, much greater. And whether knowingly or not; you can’t claim to be a victim in that scenario. Rather, an accomplice.

But suppose you stop and, with an honest assessment, ensure you’re asking the right questions before celebrating preferential outcomes. In that case, your data will not only be more valid, but the solutions it guides will have a far more significant and meaningful impact. 

Next in our series on Cognitive Biases, we’ll dive into Social Biases, what they are, what they look like and what you can do to mitigate them.