During World War II, the United States created a classified program called The Statistical Research Group (SRG). Its purpose was to help the US army solve problems using equations and formulae. In other words, SRG was into wartime math.
One interesting problem military gave SRG was about making sense of bullet holes in American planes. When those fighter planes came back from the battle, they were usually covered with bullet holes inflicted by the enemy guns. The military wanted to figure out the most optimum way to put the armor on the plane.
Well, couldn’t they armor the whole plane?
[show_to accesslevel=’almanack-all’]
They could but the armour makes the plane heavier and hence more difficult to fly and maneuver and consume more fuel. So their problem was to figure out the strategic places where they should put the armour on the body of the aircraft so that its chances of getting shot down with enemy bullets is minimized.

Do you see a pattern in the bullet holes? You’d notice that there are more bullet holes in the fuselage and not so many on the engines.
When the ISG scientists asked for more information, military gave them the following data — the fuselage section had 1.73 bullet holes per square foot, and for the engine section it was 1.11 bullet holes/sq. feet.
How would you explain this skewed distribution of bullet holes on the surface of the plane? Maybe the engines were located at such an angle that it was hard for the German bullets to find their way to the engines. A second possibility could be that those firing the guns were skillfully avoiding the engines and targeting only the fuselage.
I found this example in Jordan Ellenberg’s excellent book How Not to Be Wrong.
The first and obvious conclusion that ISG statisticians arrived was that the plane needed most armour where it was getting hit the most. However, Abraham Wald, a brilliant mathematician in ISG saw something else which everyone was missing until then.
Here’s an interesting analogy. If you go to the recovery room of a make-shift hospital in a war-zone, and see that a lot many people have bullet wounds on their limbs as compared to people with bullet holes in their head, what would you conclude?
- People don’t get shot in the head much;
- People who get shot in the head, don’t make it to the recovery room, i.e., they die.
Both the hypothesis explain what you observed in the recovery room but, framed in this way, it’s easier to see that the second option makes far more sense. And that was Wald’s insight. Ellenberg writes in his book —
The armor, said Wald, doesn’t go where the bullet holes are. It goes where the bullet holes aren’t: on the engines. Wald’s insight was simply to ask: where are the missing holes? The ones that would have been all over the engine casing, if the damage has been spread equally all over the plane? The missing bullet holes were on the missing planes. The reason planes were coming back with fewer hits to the engine is that planes that got hit in the engine weren’t coming back. Whereas the large number of planes returning to base with a thoroughly Swiss-cheeses fuselage is pretty strong evidence that hits to the fuselage can (and therefore should) be tolerated.
The information that military gave to ISG had data only for those planes that had survived the mission, i.e., the data was polluted by survivorship bias.
Survivorship bias is a very important mental model. It’s a filter that can separate a lot of bullshit and save you from bad decisions. Once you understand how survivorship bias distorts our perception, you will start noticing it everywhere.
What’s common between these super successful tech billionaires — Bill Gates, Mark Zuckerberg, Steve Jobs, Larry Page, and Surgey Brin? They are all college dropouts. And we all know that it would be a stupidity to believe that dropping out of college to pursue your entrepreneurial dreams is necessarily a sound strategy. Any such belief is the simplest example of survivorship bias.
Have you ever participated in employee satisfaction surveys that your employers might have conducted in your office? Most such surveys are plagued by survivorship bias. Gabriel Weinberg in his new book Super Thinking, writes —
…let’s suppose your company wants to understand whether it has a problem with employee motivation. Like many companies, you might choose to study this potential problem via an employee engagement survey. Employees missing the survey due to a scheduled vacation would be random and not likely to introduce bias, but employees not filling it out due to apathy would be nonrandom and would likely bias the results. That’s because the latter group is made up of disengaged employees, and by not participating, their disengagement is not being captured.
Surveys like this also do not usually account for the opinions of former employees.Unhappy employees may have chosen to leave the company, but you cannot capture their opinions when you survey only current employees. Results are therefore biased based on measuring just the population that survived , in this case the employees remaining at the company.
Survivorship bias is closely related to the idea of Alternate Histories which I had written about a few years back as part of the Latticework series.
This reminds me of a conversation I had with a mutual fund salesman sometime back. He was claiming that his fund had given a stellar performance in last 10 years. And there was no question about the authenticity of performance numbers he was quoting. It was a reputed and probably one of the biggest Mutual Fund companies (it still is). That time, there was no SEBI regulation on fund categorization and AMCs routinely launched new funds with every year.
So when I asked this sales guy about how many funds have been launched since the beginning of that AMC and how many of them have actually survived, he didn’t have any answer.
Ellenberg argues —
Mutual funds don’t live forever. Some flourish, some die. The ones that die are, by and large, the ones that don’t make money. So judging a decade’s worth of mutual funds by the ones that still exist at the end of the ten years is like judging our pilots’ evasive maneuvers by counting the bullet holes in the planes that came back. What would it mean if we never found more than one bullet hole per plane? Not that our pilots are brilliant at dodging enemy fire, but that the planes that got hit twice went down in flames?
At this point you could ask — what about those planes that got hit on sections other than the engine and still crashed? Since we can’t see the bullet holes on the planes that were shot down, how do we know that the planes that didn’t come back were only those that got hit on the engine? Now, that’s a fair question.
I’ll leave that for you to puzzle over. Let me know if you discover any useful answers.
[/show_to] [hide_from accesslevel=’almanack-all’]
- Spotlight: Big ideas from Value Investing and why applying them in your investment decision making will be a great deal
- InvestorInsights: Interviews with experienced value investors, learners, and deep thinkers
- StockTalk: Thorough analysis of business models of companies (without any recommendations)
- Behaviouronomics: Deep analysis of human behaviour and how it impacts investment decision making
- BookWorm: Reviews of the best books on Value Investing and related subjects
- Free Course – Financial Statement Analysis for Smart People (otherwise priced at Rs 5,900)
- Archives: Instant access to our huge archive from the past three years