Survivorship Bias: Why Success Case Studies Fool Us

We have all read it, perhaps on LinkedIn, complete with an upward-sloping chart: “We rewrote the titles of 300 pages and in three months traffic grew by 38%”.
The case study is honest, the numbers are real, and the temptation to copy the recipe is strong. But there is a question the post does not ask, and it is worth asking on its behalf: how many other companies did the same thing, and how did it go for them?
Whoever rewrote their titles and lost 10% does not write a case study. Nor does whoever saw no change at all. Their silence is not a detail: it is half of the data.

In the article on the p-value we saw that even “significant” results can mislead us. Here the trap is even simpler, and for that very reason more insidious: we only look at those who made it. It is called survivorship bias.

In short: survivorship bias is the error we make when we draw conclusions by looking only at the cases that passed some selection, ignoring those that were discarded along the way. If the selection depends on the very outcome we are measuring (the case study gets published only if things went well, an article stays online only if it brings traffic) the average computed on the survivors is biased, sometimes heavily. We fight it by always asking who is missing from the dataset, and why, by defining the group to study before knowing the outcome, and by keeping the data on the cases that failed.

What we will cover:

• What is survivorship bias?
• An example: 1,000 sites and 55 case studies
• Those who do not come back do not tell
• A second example: content pruning
• Where survivorship bias hides in SEO
• How to protect ourselves?
• Frequently asked questions
• Try it yourself
• Further reading

---

What is survivorship bias?

The most famous story comes from the Second World War. In New York, at Columbia University, there was the Statistical Research Group, a team of mathematicians working for the American armed forces; one of them was Abraham Wald.
The problem, in the version usually told, was this: the planes returning from missions were riddled with bullet holes, but not uniformly. The fuselage and the wings had many, the engines very few. Where should armour be added, knowing it could not go everywhere without making the plane too heavy?

The instinctive answer is “where the holes are”. Wald proposed the exact opposite: the armour goes where the holes are missing.
The planes being examined were the ones that came back. If the engines of the survivors had few holes, it was not because engines were rarely hit: it was because planes hit in the engines did not come back. The missing holes were on the planes that had been shot down, which nobody could count.

(The anecdotal version simplifies a great deal: Wald’s work was a real mathematical method for estimating aircraft vulnerability from the damage on the survivors. But the heart of the reasoning is exactly that.)

So: survivorship bias is the error of drawing conclusions only from the cases that passed a selection, as if they represented everyone.
It is not a calculation error: the averages on the survivors are computed perfectly well. It is an error about who enters the calculation. And it becomes serious when the selection depends on the outcome we care about: planes come back if they were not hit in a vital spot, the case study gets published if the strategy worked.

An example: 1,000 sites and 55 case studies

Let us build the example by simulation, so that we know the truth. A thousand sites make the same change, for instance they rewrite the titles of their main pages, and the change does nothing at all: the traffic change over the following three months is pure noise, the normal fluctuation of a site from one quarter to the next (we simulate it with a standard deviation of about 15%).
Then we do what happens in reality: only those who saw at least +25% write a case study.

I generate the data in R:

set.seed(42)
n_sites <- 1000
change_q1 <- exp(rnorm(n_sites, 0, 0.15)) - 1   # traffic change over three months
publish   <- change_q1 >= 0.25                  # those who tell their success

(The noise is generated on the log scale and then turned into a percentage: this way a site can lose at most 100% but can also gain more than 100%, as real traffic does.)

Let us first compute the truth, that is the average over all thousand sites, and the share of those that improved:

round(100 * mean(change_q1), 1)       # average change, in %
# [1] 0.7
round(100 * mean(change_q1 > 0), 1)   # % of sites that improved
# [1] 48.5

As expected from a useless change, the average is practically zero (+0.7%) and more than half of the sites got worse. Now let us look at the world as we see it, that is through the published case studies:

sum(publish)                              # how many case studies
# [1] 55
round(100 * mean(change_q1[publish]), 1)  # average change reported
# [1] 34.9
round(100 * range(change_q1[publish]), 1)
# [1] 25.1 68.9

Out come 55 case studies, with an average growth of +34.9% and peaks up to +68.9%. 100% of the case studies report a success, while among all the sites fewer than half succeeded.
The reader who collects them all and averages them, scrupulously, concludes that rewriting titles is worth +35% traffic. Every single number is true, and the conclusion is completely false.

There is a simple way to expose the trick: ask the 55 winners how the following quarter went. If the recipe worked, they should keep growing. I simulate another quarter of noise:

change_q2 <- exp(rnorm(n_sites, 0, 0.15)) - 1   # the following quarter
round(100 * mean(change_q2[publish]), 1)        # the 55 winners, three months later
# [1] 2.2
sum(change_q2[publish] >= 0.25)                 # how many do it again
# [1] 4

The winners go back to everybody’s average (+2.2%), and only 4 out of 55 repeat their success, more or less as many as chance alone would predict. It is the regression to the mean we have already met: whoever was selected for an extreme result tends, at the next measurement, to fall back into line. Survivorship bias and regression to the mean are two faces of the same problem: we chose the cases by looking at the outcome.

Those who do not come back do not tell

The idea is ancient. Cicero tells that the philosopher Diagoras of Melos, on Samothrace, was shown the votive tablets painted by those who had survived a shipwreck after praying to the gods: was this not proof that prayers work? Diagoras replied by asking where the portraits were of those who had prayed and drowned.
Nassim Taleb calls this body of invisible data the silent graveyard: the cases that did not make it do not write, do not paint tablets, do not come back to base. And precisely because they are silent, we forget they exist.

The delicate point is that the silent graveyard is invisible by construction. It is not data missing through carelessness, which a little care can recover: it is the very mechanism that produces the data that excludes it. No case study will ever tell us how many others tried the same thing.

A second example: content pruning

Survivorship bias does not need LinkedIn: it can arise inside our own data. Take a blog that publishes 100 articles a year from 2020 to 2025. Every year it runs a content audit, what is known as content pruning: articles at least one year old that bring fewer than 30 visits a month are removed (or set to noindex) with probability 1/2, because someone judges them salvageable and someone else does not.
In the simulation, and this is the point, an article’s traffic does not depend in any way on the year it was written.

I build the blog in R:

set.seed(2025)
years <- 2020:2025
posts <- data.frame(year = rep(years, each = 100))
posts$visits <- round(rlnorm(nrow(posts), log(40), 1.1))   # visits per month
posts$online <- TRUE
for (audit in 2021:2025) {
  candidates <- posts$online & posts$year < audit & posts$visits < 30
  posts$online[candidates & runif(nrow(posts)) < 0.5] <- FALSE
}

Traffic follows a log-normal distribution, as it usually does: many articles with few visits, a few articles with a great many. That is exactly why we use the median as a summary and not the mean, which the few big hits would drag upwards.

A year after the last audit, someone wonders whether to invest in evergreen content. They export the online articles from the CMS and compute the median visits by year of publication:

online <- posts[posts$online, ]
tapply(online$visits, online$year, median)
#  2020 2021 2022 2023 2024 2025
#  78.0 78.5 73.0 54.0 54.0 41.5

The result looks clear-cut: the 2020 articles bring almost twice the visits of the 2025 ones. “Content ages well, traffic accumulates over time.” Now let us look at the same table over all the articles ever published, including those removed:

tapply(posts$visits, posts$year, median)
#  2020 2021 2022 2023 2024 2025
#  38.0 38.0 43.0 38.0 40.5 41.5

The growth disappears: the year of publication makes no difference, as we had established. The “age” effect was produced entirely by the audits. The 2020 articles went through five rounds of pruning and 62% of them are left, the strongest; the 2025 articles went through none and are all still there, weak ones included. By comparing groups that went through different selections, we measured the selection, not the content.

There is a subtle irony here: precisely those who work well, and clean up regularly, build themselves an archive that lies. And the lie is convenient, because it confirms that old content is valuable and that pruning worked.

Where survivorship bias hides in SEO

Once we learn to recognise it, we find it everywhere. A few examples from everyday practice:

• Ranking factor studies. Every so often a study measures the pages in first position: so many words on average, so many images, so many backlinks. But the study only looks at the top ten positions: how many pages with the same features sit on page four? Without them we cannot tell whether those features distinguish the winners or are simply common to everyone. (It is also a matter of correlation mistaken for causation, but the first flaw is in the sample.)
• “What the sites that survived the core update have in common”. Without looking at the sites with the same features that collapsed instead, the list of the survivors’ virtues proves nothing.
• Search Console. It shows the pages and queries that obtained impressions: pages Google never shows do not appear, and neither do queries anonymised for privacy. The interface tables, moreover, stop at the first thousand rows. Analysing “the site’s pages” starting from there means analysing the pages that already work.
• A/B testing best practices. Collections of “winning tests” tell us about the variants that won, not about the ten identical ones that lost elsewhere. And the winners of an A/B test stopped at the right moment, as we saw when discussing peeking, are survivors selected twice.
• Portfolios and testimonials. An agency shows the clients that grew, not the ones it lost; a course shows the students who found a job. It is not dishonesty: it is how a shop window works. But the shop window is not the warehouse.
• Tool datasets. Expired domains, closed sites and deleted pages silently drop out of backlink and visibility indexes. A historical trend computed on the sites active today is a trend of the survivors.

How to protect ourselves?

There is no formula that corrects survivorship bias after the fact: if the discarded cases are lost, no calculation brings them back. There are, however, habits that prevent it.

Define the group before knowing the outcome. This is the lesson of the second example. If we want to know how the 2020 articles perform, the right group is all the articles published in 2020, not those still online. Epidemiologists call this a cohort study: the whole group is followed from the start, the dead included. In the simulation, it is exactly the difference between the dashed line and the solid one.

Ask for the denominator. In front of a success, the question is always “out of how many attempts?”. Fifty-five enthusiastic case studies out of a thousand attempts tell a very different story from fifty-five out of sixty. If the denominator is unknown, the honest conclusion is that we do not know.

Keep the dead. In our work the silent graveyard can, in part, be preserved: old sitemaps, the list of removed or redirected URLs with their dates, full Search Console exports downloaded regularly, the results of all A/B tests, not only the winning ones. It costs little, and it turns an archive that lies into one we can question.

Compare with those who did nothing. The best way to know whether a change works is to compare the modified pages with similar pages left as they were, chosen before looking at the results: this is the idea behind SEO split testing. The control group is, in a sense, the graveyard we build on purpose.

Distrust large effects selected for success. When a result reaches us because it was exceptional, let us expect it to be less exceptional next time. It is not pessimism: it is what we saw happen to the 55 winners.

Frequently asked questions

What is the most famous example of survivorship bias?

The Second World War planes analysed by the mathematician Abraham Wald. The planes that came back had few hits on the engines, and the temptation was to armour the most damaged parts. Wald understood that hits on the engines were missing because the planes hit there did not come back: the armour belonged exactly where, on the survivors, there were no holes.

Are survivorship bias and selection bias the same thing?

Survivorship bias is a special case of selection bias. Selection bias occurs whenever the sample analysed does not represent the population of interest; in survivorship bias the sample is made of those who passed a selection, and the selection usually depends on the outcome itself, such as success or staying around over time.

Why are success case studies unreliable?

Not because they are false, but because they are selected: they get published when the strategy worked and stay in the drawer when it did not. A collection of case studies therefore shows only the lucky tail of the results, and the average of the successes reported can be very far from the real effect, which may even be zero.

How is survivorship bias different from regression to the mean?

They are closely related. Survivorship bias is about who we see: we only observe the cases that passed a selection based on the outcome. Regression to the mean is about what happens next: cases selected for an extreme result tend to come back towards the average at the following measurement. Selecting case studies by their success produces both at once.

Can survivorship bias be corrected after the data has been collected?

Only if we recover information on the excluded cases, or if we know precisely the mechanism that excluded them and can model it. If the excluded cases have simply vanished, no statistical technique brings them back: that is why it pays to prevent it, by defining the group to study before knowing the outcome and by keeping the data on the cases that failed.

Try it yourself

The code in this article lends itself to a few experiments.

• Lower the publication threshold from +25% to +10% (publish <- change_q1 >= 0.10): how many case studies come out now, and what is their average? The distortion shrinks, but does it disappear?
• Give the change a real but small effect, +3%, by adding log(1.03) to the mean of rnorm(). How many times over do the published case studies exaggerate the real effect?
• In the pruning simulation, raise the removal probability from 1/2 to 1 (runif(nrow(posts)) < 1): how does the table of online articles change? And if the threshold moves from 30 to 60 visits?

The last experiment is the most instructive: the more severe the pruning, the more the archive “proves” that old content is better, without anything having changed in the content itself.

---

We have seen that looking only at those who survived a selection distorts the averages, even when every data point is true. There is a related trap that awaits us whenever a piece of news seems much more likely than it really is: forgetting how rare, to begin with, the event we are looking for is. A test that is 99% accurate, if what it looks for is rare enough, can be wrong most of the times it raises the alarm. It is the base rate fallacy, and it will be the next stop on our journey through the traps hidden in data.

---

Further reading

On the role of chance in success and on the survivors we mistake for winners, I recommend Fooled by Randomness by Nassim Nicholas Taleb: a brilliant and combative book, born in the world of finance but perfect for anyone who has to judge strategies, case studies and consultants by their results.