When Survivorship Bias meets Superstitious Learning

Why people's views of how to be a successful entrepreneur are so wrong, and why that can hurt them

Today, I am writing about survivorship bias, but I am not going to start with the plane-with-bullet-holes diagram.

Every post about survivorship bias starts with the plane-with-bullet-holes diagram, but I won’t, because the famous illustration of survivorship bias was not actually drawn by Abraham Wald (who developed the concept of survivorship bias in WWII). It was made for Wikipedia in 2016. And it features a Ventura bomber, which was mostly retired when Wald wrote his report.

Wald’s report did even not include the story of the armor and the bullet holes, in fact, he no illustrations, just math. And, despite my best efforts, this attempt to add bullet holes to his paper was less compelling than the apocryphal bomber:

With that out of the way. I wanted to write about how survivorship bias really messes with our minds, and is especially prevalent in the context of entrepreneurship.

This paper (by George Lifchits, Ashton Anderson, Daniel G. Goldstein, Jake M. Hofman, and Duncan J. Watts) is an excellent study of why this is true. They point out that the success stories that we read about founders are biased in two different ways:

1. Most stories in the press are about the habits and approaches of successful founders - this is selection on the dependent variable

2. But stories also seek to find common linkages among founders, and reduce their success to just one or two factors - this is selection on the independent variable

Together, this creates a really misleading impression about which founders win. It is worth spending a minute on the diagram in the paper to see why:

The study then showed people articles about startup founders, one set of which showed that unicorns were founded by dropouts at a higher rate (not true) and the other that they were founded by graduates at a higher rate. And then they told them that they selected the articles to illustrate this particular bias.

It didn’t help stop the bias. If you read about college dropouts becoming unicorn founders, you are willing to bet that they outperform graduates & create elaborate justifications for your decision. The effects were huge: “In our experiment,
we observed a strikingly large difference in participants’ decisions depending on which condition they were assigned to; while 87% of people shown examples of successful graduates bet that graduates are more likely to succeed, only 32% of people shown examplesof successful dropouts bet on graduates, a difference of 55 percentage points.”

This bias is bad enough, but it becomes even worse when you consider that people not only try to learn from these survivors, but they try and learn from the small subset of “good practices” that are the focus of popular reports and books. So people learn both from the wrong companies and the wrong lessons.

Learning the wrong lessons by focusing on symbols of success rather than harder-to-identify systems and processes is called superstitious learning. It is like trying to replicate the success of Amazon by having people write 6 page memos before every meeting, even though the memos are definitely not the main reason why Amazon has succeeded, or even why its meetings might work better. Instead, they are just easy-to-copy and heavily reported on rituals, not the deeper seeds of success.

And this can become a problem. This paper by Justin Miller shows that when markets are stable & lessons clear, less superstitious learning occurs. In changing markets, founders copy the wrong things, assuming they are the key to winning, and things start to go seriously wrong.

Even worse, in fields where luck is major factor in success, trying to learn anything at all from the “best” surviving companies is not only fruitless, it is destructive. In fact, the best learnings come from companies that do well, but not exceptionally well, since those companies are likely to have embraced less risk, and more likely to be skilled.

So, what to do about this? Even though it doesn’t help as much as we would like, be aware of these biases and actively think about survivorship bias and superstitious learning when you are reading any profile of a successful person or company. We also learn a lot from failure, so make sure to seek out failed cases along with successful ones.

Finally, as a business school professor, it is probably not surprising that I think part of the answer is to get people to read less popular accounts and more studies and analyses that take survivorship bias into account. This resarch can also help you to recalibrate your expectations. For example, many people are surprised to learn that:

• Founders are older than many people think. The mean age of a founder is 42. One getting VC? 42. And the founders most likely to have a hypergrowth startup are 45-59!

• Flat startups do worse than those with hierarchy. Flat companies tend to suffer as they grow, since the lack of structure holds them back. Far from being a sign of becoming a boring corporation, hiring good middle managers is a huge key to startup success.

• There are few simple links between personality & successful entrepreneurship. Anyone who says there is a reliable test for being a good founder is wrong. (We can predict who will try to become a founder, but not who will win). So you shouldn’t be discouraged if you don’t fit the standard founder type.

More on startup myths in future posts (I also have a book on the topic)…