Essays · The trap

Survivorship bias in backtesting, explained

By Daniel Roache · 14 July 2026 · 7 min read

The most famous example isn't from finance. In the Second World War, analysts studied the bombers that returned from missions and mapped where they were riddled with bullet holes, planning to reinforce those spots. The statistician Abraham Wald pointed out the obvious flaw: they were only looking at the planes that came back. The armour belonged where the returning planes had no holes — because the planes hit there never returned to be counted.

Every backtest is at risk of making exactly Wald's mistake.

What survivorship bias is

Survivorship bias is the tendency to analyse only the successful or surviving cases while quietly excluding the ones that failed or disappeared. In backtesting it shows up in several forms at once, and each one pushes your numbers in the flattering direction.

Where it hides in a strategy test

• Delisted securities. If your universe is "stocks in the index today," you've excluded every company that went to zero, got acquired at a loss, or was kicked out for under-performing. Your backtest bought only the survivors — a luxury the past didn't offer.
• Dead funds and dead strategies. Studies of managed funds find survivorship inflates reported annual returns by around 2% simply by dropping the funds that closed. The same happens to your own research folder: the ideas you silently killed six months ago don't appear in the track record of the ideas you kept.
• Hidden risk. The effect is worse for drawdowns than for returns. One analysis found survivorship bias caused an average underestimation of hedge-fund drawdowns by 14 percentage points — making strategies look far less risky than they actually were. The blown-up funds are exactly the ones removed from the sample.
• Selection in your own process. If you test twenty strategies, deploy the two that looked best, and only ever talk about those two, you've built a survivorship-biased narrative about your own skill.

Unrealistically high returns that quietly beat every benchmark are the red flag. Often the benchmark isn't being beaten — the losers have just been deleted from the comparison.

How to strip it out

1. Use point-in-time, survivorship-free data. Your universe on any past date should contain the securities that existed on that date, including the ones later delisted. Good data vendors sell survivorship-bias-free datasets specifically for this.
2. Keep the graveyard. Record every strategy you test, including the ones you kill. Your honest hit-rate is survivors divided by everything you tried, not survivors divided by survivors. At The Refutation we publish the graveyard on purpose for exactly this reason.
3. Fix the test window before you look. Survivorship's cousin is choosing the evaluation period after seeing the result. Pick your held-out regimes in advance so you can't unconsciously select the window that flatters the strategy.
4. Judge by drawdown, not just return. Because the bias hides risk more than reward, always look at the worst stretch. A survivor-only sample will show you a suspiciously shallow one.

Survivorship bias and general overfitting are the two reasons a backtest can look wonderful and mean nothing. Both are defeated by the same instinct: count the failures, and test on data the strategy never had the luxury of choosing.

Keep reading