In the article on the p-value we saw that even “significant” results can mislead us. Here the trap is even simpler, and for that very reason more insidious: we only look at those who made it. It is called survivorship bias.

In short: survivorship bias is the error we make when we draw conclusions by looking only at the cases that passed some selection, ignoring those that were discarded along the way. If the selection depends on the very outcome we are measuring (the case study gets published only if things went well, an article stays online only if it brings traffic) the average computed on the survivors is biased, sometimes heavily. We fight it by always asking who is missing from the dataset, and why, by defining the group to study before knowing the outcome, and by keeping the data on the cases that failed.

What we will cover:

• What is survivorship bias?
• An example: 1,000 sites and 55 case studies
• Those who do not come back do not tell
• A second example: content pruning
• Where survivorship bias hides in SEO
• How to protect ourselves?
• Frequently asked questions
• Try it yourself
• Further reading

What is survivorship bias?

The most famous story comes from the Second World War. In New York, at Columbia University, there was the Statistical Research Group, a team of mathematicians working for the American armed forces; one of them was Abraham Wald.
The problem, in the version usually told, was this: the planes returning from missions were riddled with bullet holes, but not uniformly. The fuselage and the wings had many, the engines very few. Where should armour be added, knowing it could not go everywhere without making the plane too heavy?

The instinctive answer is “where the holes are”. Wald proposed the exact opposite: the armour goes where the holes are missing.
The planes being examined were the ones that came back. If the engines of the survivors had few holes, it was not because engines were rarely hit: it was because planes hit in the engines did not come back. The missing holes were on the planes that had been shot down, which nobody could count.

(The anecdotal version simplifies a great deal: Wald’s work was a real mathematical method for estimating aircraft vulnerability from the damage on the survivors. But the heart of the reasoning is exactly that.)

So: survivorship bias is the error of drawing conclusions only from the cases that passed a selection, as if they represented everyone.
It is not a calculation error: the averages on the survivors are computed perfectly well. It is an error about who enters the calculation. And it becomes serious when the selection depends on the outcome we care about: planes come back if they were not hit in a vital spot, the case study gets published if the strategy worked.

An example: 1,000 sites and 55 case studies

Let us build the example by simulation, so that we know the truth. A thousand sites make the same change, for instance they rewrite the titles of their main pages, and the change does nothing at all: the traffic change over the following three months is pure noise, the normal fluctuation of a site from one quarter to the next (we simulate it with a standard deviation of about 15%).
Then we do what happens in reality: only those who saw at least +25% write a case study.

I generate the data in R:

set.seed(42)
n_sites <- 1000
change_q1 <- exp(rnorm(n_sites, 0, 0.15)) - 1   # traffic change over three months
publish   <- change_q1 >= 0.25                  # those who tell their success

(The noise is generated on the log scale and then turned into a percentage: this way a site can lose at most 100% but can also gain more than 100%, as real traffic does.)

Let us first compute the truth, that is the average over all thousand sites, and the share of those that improved:

round(100 * mean(change_q1), 1)       # average change, in %
# [1] 0.7
round(100 * mean(change_q1 > 0), 1)   # % of sites that improved
# [1] 48.5

As expected from a useless change, the average is practically zero (+0.7%) and more than half of the sites got worse. Now let us look at the world as we see it, that is through the published case studies:

sum(publish)                              # how many case studies
# [1] 55
round(100 * mean(change_q1[publish]), 1)  # average change reported
# [1] 34.9
round(100 * range(change_q1[publish]), 1)
# [1] 25.1 68.9

Out come 55 case studies, with an average growth of +34.9% and peaks up to +68.9%. 100% of the case studies report a success, while among all the sites fewer than half succeeded.
The reader who collects them all and averages them, scrupulously, concludes that rewriting titles is worth +35% traffic. Every single number is true, and the conclusion is completely false.

There is a simple way to expose the trick: ask the 55 winners how the following quarter went. If the recipe worked, they should keep growing. I simulate another quarter of noise:

change_q2 <- exp(rnorm(n_sites, 0, 0.15)) - 1   # the following quarter
round(100 * mean(change_q2[publish]), 1)        # the 55 winners, three months later
# [1] 2.2
sum(change_q2[publish] >= 0.25)                 # how many do it again
# [1] 4

The winners go back to everybody’s average (+2.2%), and only 4 out of 55 repeat their success, more or less as many as chance alone would predict. It is the regression to the mean we have already met: whoever was selected for an extreme result tends, at the next measurement, to fall back into line. Survivorship bias and regression to the mean are two faces of the same problem: we chose the cases by looking at the outcome.

Those who do not come back do not tell

The idea is ancient. Cicero tells that the philosopher Diagoras of Melos, on Samothrace, was shown the votive tablets painted by those who had survived a shipwreck after praying to the gods: was this not proof that prayers work? Diagoras replied by asking where the portraits were of those who had prayed and drowned.
Nassim Taleb calls this body of invisible data the silent graveyard: the cases that did not make it do not write, do not paint tablets, do not come back to base. And precisely because they are silent, we forget they exist.

The delicate point is that the silent graveyard is invisible by construction. It is not data missing through carelessness, which a little care can recover: it is the very mechanism that produces the data that excludes it. No case study will ever tell us how many others tried the same thing.

A second example: content pruning

Survivorship bias does not need LinkedIn: it can arise inside our own data. Take a blog that publishes 100 articles a year from 2020 to 2025. Every year it runs a content audit, what is known as content pruning: articles at least one year old that bring fewer than 30 visits a month are removed (or set to noindex) with probability 1/2, because someone judges them salvageable and someone else does not.
In the simulation, and this is the point, an article’s traffic does not depend in any way on the year it was written.

I build the blog in R:

set.seed(2025)
years <- 2020:2025
posts <- data.frame(year = rep(years, each = 100))
posts$visits <- round(rlnorm(nrow(posts), log(40), 1.1))   # visits per month
posts$online <- TRUE
for (audit in 2021:2025) {
  candidates <- posts$online & posts$year < audit & posts$visits < 30
  posts$online[candidates & runif(nrow(posts)) < 0.5] <- FALSE
}

Traffic follows a log-normal distribution, as it usually does: many articles with few visits, a few articles with a great many. That is exactly why we use the median as a summary and not the mean, which the few big hits would drag upwards.

A year after the last audit, someone wonders whether to invest in evergreen content. They export the online articles from the CMS and compute the median visits by year of publication:

online <- posts[posts$online, ]
tapply(online$visits, online$year, median)
#  2020 2021 2022 2023 2024 2025
#  78.0 78.5 73.0 54.0 54.0 41.5

The result looks clear-cut: the 2020 articles bring almost twice the visits of the 2025 ones. “Content ages well, traffic accumulates over time.” Now let us look at the same table over all the articles ever published, including those removed:

tapply(posts$visits, posts$year, median)
#  2020 2021 2022 2023 2024 2025
#  38.0 38.0 43.0 38.0 40.5 41.5

The growth disappears: the year of publication makes no difference, as we had established. The “age” effect was produced entirely by the audits. The 2020 articles went through five rounds of pruning and 62% of them are left, the strongest; the 2025 articles went through none and are all still there, weak ones included. By comparing groups that went through different selections, we measured the selection, not the content.

There is a subtle irony here: precisely those who work well, and clean up regularly, build themselves an archive that lies. And the lie is convenient, because it confirms that old content is valuable and that pruning worked.

Where survivorship bias hides in SEO

Once we learn to recognise it, we find it everywhere. A few examples from everyday practice:

• Ranking factor studies. Every so often a study measures the pages in first position: so many words on average, so many images, so many backlinks. But the study only looks at the top ten positions: how many pages with the same features sit on page four? Without them we cannot tell whether those features distinguish the winners or are simply common to everyone. (It is also a matter of correlation mistaken for causation, but the first flaw is in the sample.)
• “What the sites that survived the core update have in common”. Without looking at the sites with the same features that collapsed instead, the list of the survivors’ virtues proves nothing.
• Search Console. It shows the pages and queries that obtained impressions: pages Google never shows do not appear, and neither do queries anonymised for privacy. The interface tables, moreover, stop at the first thousand rows. Analysing “the site’s pages” starting from there means analysing the pages that already work.
• A/B testing best practices. Collections of “winning tests” tell us about the variants that won, not about the ten identical ones that lost elsewhere. And the winners of an A/B test stopped at the right moment, as we saw when discussing peeking, are survivors selected twice.
• Portfolios and testimonials. An agency shows the clients that grew, not the ones it lost; a course shows the students who found a job. It is not dishonesty: it is how a shop window works. But the shop window is not the warehouse.
• Tool datasets. Expired domains, closed sites and deleted pages silently drop out of backlink and visibility indexes. A historical trend computed on the sites active today is a trend of the survivors.

How to protect ourselves?

There is no formula that corrects survivorship bias after the fact: if the discarded cases are lost, no calculation brings them back. There are, however, habits that prevent it.

Define the group before knowing the outcome. This is the lesson of the second example. If we want to know how the 2020 articles perform, the right group is all the articles published in 2020, not those still online. Epidemiologists call this a cohort study: the whole group is followed from the start, the dead included. In the simulation, it is exactly the difference between the dashed line and the solid one.

Ask for the denominator. In front of a success, the question is always “out of how many attempts?”. Fifty-five enthusiastic case studies out of a thousand attempts tell a very different story from fifty-five out of sixty. If the denominator is unknown, the honest conclusion is that we do not know.

Keep the dead. In our work the silent graveyard can, in part, be preserved: old sitemaps, the list of removed or redirected URLs with their dates, full Search Console exports downloaded regularly, the results of all A/B tests, not only the winning ones. It costs little, and it turns an archive that lies into one we can question.

Compare with those who did nothing. The best way to know whether a change works is to compare the modified pages with similar pages left as they were, chosen before looking at the results: this is the idea behind SEO split testing. The control group is, in a sense, the graveyard we build on purpose.

Distrust large effects selected for success. When a result reaches us because it was exceptional, let us expect it to be less exceptional next time. It is not pessimism: it is what we saw happen to the 55 winners.

Frequently asked questions

What is the most famous example of survivorship bias?

The Second World War planes analysed by the mathematician Abraham Wald. The planes that came back had few hits on the engines, and the temptation was to armour the most damaged parts. Wald understood that hits on the engines were missing because the planes hit there did not come back: the armour belonged exactly where, on the survivors, there were no holes.

Are survivorship bias and selection bias the same thing?

Survivorship bias is a special case of selection bias. Selection bias occurs whenever the sample analysed does not represent the population of interest; in survivorship bias the sample is made of those who passed a selection, and the selection usually depends on the outcome itself, such as success or staying around over time.

Why are success case studies unreliable?

Not because they are false, but because they are selected: they get published when the strategy worked and stay in the drawer when it did not. A collection of case studies therefore shows only the lucky tail of the results, and the average of the successes reported can be very far from the real effect, which may even be zero.

How is survivorship bias different from regression to the mean?

They are closely related. Survivorship bias is about who we see: we only observe the cases that passed a selection based on the outcome. Regression to the mean is about what happens next: cases selected for an extreme result tend to come back towards the average at the following measurement. Selecting case studies by their success produces both at once.

Can survivorship bias be corrected after the data has been collected?

Only if we recover information on the excluded cases, or if we know precisely the mechanism that excluded them and can model it. If the excluded cases have simply vanished, no statistical technique brings them back: that is why it pays to prevent it, by defining the group to study before knowing the outcome and by keeping the data on the cases that failed.

Try it yourself

The code in this article lends itself to a few experiments.

• Lower the publication threshold from +25% to +10% (publish <- change_q1 >= 0.10): how many case studies come out now, and what is their average? The distortion shrinks, but does it disappear?
• Give the change a real but small effect, +3%, by adding log(1.03) to the mean of rnorm(). How many times over do the published case studies exaggerate the real effect?
• In the pruning simulation, raise the removal probability from 1/2 to 1 (runif(nrow(posts)) < 1): how does the table of online articles change? And if the threshold moves from 30 to 60 visits?

The last experiment is the most instructive: the more severe the pruning, the more the archive “proves” that old content is better, without anything having changed in the content itself.

We have seen that looking only at those who survived a selection distorts the averages, even when every data point is true. There is a related trap that awaits us whenever a piece of news seems much more likely than it really is: forgetting how rare, to begin with, the event we are looking for is. A test that is 99% accurate, if what it looks for is rare enough, can be wrong most of the times it raises the alarm. It is the base rate fallacy, and it will be the next stop on our journey through the traps hidden in data.

Further reading

On the role of chance in success and on the survivors we mistake for winners, I recommend Fooled by Randomness by Nassim Nicholas Taleb: a brilliant and combative book, born in the world of finance but perfect for anyone who has to judge strategies, case studies and consultants by their results.

On this blog the p-value appears in dozens of articles, from hypothesis testing to A/B testing, from regression to multiple comparisons. It is the most used number in applied statistics, and probably the most misunderstood. It is worth stopping once to look at what it says, and above all at what it does not say.

In short: the p-value is the probability of observing data at least as extreme as those obtained, assuming the null hypothesis is true (that is, that the effect does not exist). It is not the probability that the null hypothesis is true, nor the probability that the result is due to chance. A small p-value does not say the effect is large, and a p-value above 0.05 does not say there is no effect. The 0.05 threshold is a convention, not a natural boundary. To decide, we also need the size of the effect, the confidence interval and an estimate of how many of the ideas we test actually work.

What we will cover:

• What is the p-value, in one sentence
• Misconception 1: “there is a 97% probability that it works”
• Misconception 2: “a small p means a large effect”
• Misconception 3: “p above 0.05, so there is no effect”
• Misconception 4: “0.05 is the line between true and false”
• What should we report, then?
• Frequently asked questions
• Try it yourself
• Further reading

What is the p-value, in one sentence

Let’s start from an A/B test on two ad headlines: 4,000 impressions each, 200 clicks for headline A (CTR 5.0%) and 240 for headline B (CTR 6.0%). The test on the two proportions returns p = 0.0498.

To read that number we need a small exercise of imagination. Suppose the two headlines are identical, that the CTR difference does not exist and everything depends on the chance of clicks. That is the null hypothesis. In that imaginary world, how likely would we be to see a difference at least as large as the one observed, 40 clicks out of 4,000? The answer is the p-value: about 5%.

As a formula, with H₀ the null hypothesis:

\(p = P(\text{data at least this extreme} \mid H_0 \text{ true})\)

The vertical bar reads “assuming that”: the whole calculation happens inside the null hypothesis. The p-value measures how surprising the data would be if the effect did not exist. It says nothing, directly, about how likely it is that the effect exists. And it is precisely here that almost all the misconceptions are born.

One consequence that few people know, but which helps to understand everything else: when the null hypothesis is true, the p-value is a random number between 0 and 1, with a uniform distribution. A p of 0.03 is exactly as rare as a p of 0.73. We check it by simulating 10,000 A/B tests in which B changes nothing, and another 10,000 in which B takes the CTR from 5% to 6%:

one_test <- function(ctr_b, n = 4000, ctr_a = 0.05) {
  clicks_a <- rbinom(1, n, ctr_a)
  clicks_b <- rbinom(1, n, ctr_b)
  prop.test(c(clicks_a, clicks_b), c(n, n), correct = FALSE)$p.value
}

set.seed(1)
p_null <- replicate(10000, one_test(0.05))     # B changes nothing
p_effect <- replicate(10000, one_test(0.06))   # B really works (+20%)

mean(p_null < 0.05)
# [1] 0.0481
quantile(p_null, c(0.25, 0.50, 0.75))
#       25%       50%       75%
# 0.2483948 0.4970851 0.7524771
mean(p_effect < 0.05)
# [1] 0.501

With no effect, 4.8% of tests fall below 0.05 (the promised 5%, give or take chance) and the quartiles fall at 0.25, 0.50 and 0.75: the distribution is flat. With a real effect, the p-values pile up near zero, but only half fall below 0.05: with 4,000 impressions per variant, the test’s power is 50%.

Misconception 1: “there is a 97% probability that it works”

It is the most widespread misconception, and the most costly. A p of 0.03 does not mean there is a 3% probability that the null hypothesis is true, nor 97% that the effect exists. The p-value is the probability of the data assuming H₀, not the probability of H₀ given the data. It sounds like wordplay, but the two probabilities can be miles apart.

An example from outside statistics. The probability that a person is American, given that they are the President of the United States, is very high. The probability that they are the President, given that they are American, is about one in three hundred and thirty million. Inverting the “given that” changes everything.

To get from the p-value to the question we really care about (does this idea work?) one ingredient is missing: how many of the ideas we put to the test work in the first place. Let’s take a realistic example. We test 1,000 ideas (headlines, meta descriptions, calls to action) and only one in ten really works; the other nine are neutral. I simulate it in R with the same tests as before:

set.seed(5)
works <- seq_len(1000) <= 100        # the good ideas are 100 out of 1,000
p_ideas <- sapply(works, function(w) one_test(ifelse(w, 0.06, 0.05)))

table(significant = p_ideas < 0.05, works)

#            works
# significant FALSE TRUE
#       FALSE   853   50
#       TRUE     47   50

Out of 1,000 tests, 97 come out “significant”. Fifty are ideas that really work; 47 are neutral ideas that passed the threshold by chance. Among our “winners”, almost one in two is a false alarm, and yet each of them had a p-value below 0.05.

The sum can also be done without simulation. If π is the share of ideas that work, α the threshold and the power the probability of spotting a good idea, the share of false alarms among the significant results is:

\(\text{share of false alarms} = \frac{\alpha \, (1 – \pi)}{\alpha \, (1 – \pi) + \text{power} \cdot \pi}\)

The numerator is the neutral ideas that pass by chance (5% of the 900); the denominator adds the good ideas the test recognises (50% of the 100). With our numbers: 45 / (45 + 50) = 47%. The table shows how the share changes as the two unknowns vary:

The same 5% threshold produces 6% of false alarms or almost half, depending on how good the ideas we test are and how much data we have. It is the same mechanism as the base rate fallacy in medical tests: an accurate test for a rare disease still produces an avalanche of false positives.

Misconception 2: “a small p means a large effect”

The p-value mixes two things: how large the effect is and how much data we have. With abundant data, even a minuscule effect produces a minuscule p-value.

Search Console offers the opportunity every day. Two sections of the site, 10 million impressions each, CTR of 5.00% against 5.05%: a difference of five hundredths of a point, 1% in relative terms. I compute it in R:

n <- 1e7
prop.test(c(n * 0.05, n * 0.0505), c(n, n), correct = FALSE)$p.value
# [1] 3.091392e-07

A p-value of 0.0000003: “highly significant”. But the effect is worth five clicks every ten thousand impressions, and almost certainly does not justify any decision. Significant, in statistics, means “unlikely to be due to chance”, not “important”. For importance we need to look at the size of the effect, the subject of the article on effect size and power analysis.

Misconception 3: “p above 0.05, so there is no effect”

It is the mirror-image misconception, and just as dangerous. Let’s go back to the figure with the two distributions: when B really works, with 4,000 impressions per variant, half the tests stay above 0.05. A “non-significant” p-value means that the data are not enough to rule out chance, not that the effect has been ruled out.

An example. Headline A: 200 clicks out of 4,000. Headline B: 222 clicks out of 4,000, an 11% higher CTR. I check it in R, looking at the confidence interval too:

t <- prop.test(c(222, 200), c(4000, 4000), correct = FALSE)
t$p.value
# [1] 0.271176
t$conf.int     # CTR difference, B minus A
# [1] -0.004295887  0.015295887

The p-value is 0.27, well above the threshold. But the confidence interval of the difference runs from −0.43 to +1.53 CTR points: in relative terms, from an 8.6% drop to a 30.6% increase. The data are compatible with a slightly worse headline and with a clearly better one. Concluding “headline B is useless” would be like saying a match not yet played ended in a draw.

Absence of evidence is not evidence of absence. The test is telling us that more data are needed, not that the idea should be thrown away.

Misconception 4: “0.05 is the line between true and false”

The 0.05 threshold does not come from nature: it comes from Ronald Fisher, who in the 1920s proposed it as a convenient reference, one in twenty, to flag results that deserved a closer look. Over time it became a bureaucratic boundary, with a curious effect: a result with p = 0.049 gets published, communicated, applied; one with p = 0.051 ends up in the drawer.

How different, in the data, are those two results? In our A/B test with 200 clicks for headline A, if headline B gets 239 the p-value is 0.0555; if it gets 240 it is 0.0498. One click in eight thousand separates “not significant” from “significant”. No reasonable person would change their mind over one click, yet the threshold asks us to.

In 2016 the American Statistical Association published an official statement on the p-value, the first time in its 177-year history that it had taken a position on a single aspect of statistical practice. It contains six principles. Three say exactly what we have seen: the p-value does not measure the probability that the hypothesis is true; it does not measure the size of an effect; conclusions should not be based only on whether a threshold is passed.

There is then a fourth way of going wrong, the subtlest, because it concerns not the reading of the p-value but the way we arrive at it. The p-value holds for a test decided before looking at the data. If we try many metrics, many segments, many stopping points, and keep the result that passes the threshold, the 5% no longer holds. It is what we saw when discussing multiple comparisons and peeking, and it even has an unflattering name: p-hacking.

What should we report, then?

The p-value should not be thrown away: it answers its own question, the one about chance, honestly. We just need to stop making it answer questions that are not its business. In day-to-day practice, a well-reported result has three parts.

The size of the effect, in the unit that matters for the business: CTR points, clicks per month, euros. Not “significant”, but “+0.8 CTR points, about 1,200 more clicks a month”.

The confidence interval, which says how uncertain we are about that size: “between +0.1 and +1.5 points”. It contains everything that is in the p-value (if it excludes zero, the test is significant) and a great deal more.

The context of the test: was it the only planned test, or one of twenty? Was the power sufficient? How promising were the ideas to begin with?

Anyone who wants to answer directly the question “what is the probability that B is better than A?” has to change approach: it is the territory of the Bayesian A/B test, which does answer that question, provided we declare our starting assumptions.

Frequently asked questions

What does a p-value of 0.05 mean?

It means that, if the effect did not exist, we would observe data at least this extreme in about 5 cases out of 100. It does not mean there is a 95% probability that the effect exists, nor a 5% probability that the result is due to chance.

Is the p-value the probability that the null hypothesis is true?

No. It is the probability of the data assuming the null hypothesis is true, not the probability of the null hypothesis given the data. To get the latter we also need to know how plausible the hypothesis was before the test: with unpromising ideas, even among results with p below 0.05 false alarms can be almost half.

Does a very small p-value indicate an important effect?

No. The p-value depends both on the size of the effect and on the amount of data: with millions of impressions, even an irrelevant CTR difference produces a minuscule p-value. Importance is judged from the size of the effect and its confidence interval.

If the p-value is greater than 0.05, does the effect not exist?

Not necessarily. It means the data are not enough to rule out chance. If the test had little power, a real effect can easily produce a high p-value: the confidence interval shows which effects are still compatible with the data.

Why 0.05, of all numbers?

It is a convention proposed by Ronald Fisher in the 1920s as a convenient reference, a probability of one in twenty. It has no natural foundation: a result with p = 0.049 and one with p = 0.051 carry practically the same evidence.

Try it yourself

The article’s code lends itself to a few revealing experiments.

• Raise the share of good ideas from 100 to 500 out of 1,000 (works <- seq_len(1000) <= 500) and recompute the table: how many of the “winners” are false alarms now? The result should approach the table’s 9%.
• Increase the impressions per variant from 4,000 to 16,000 in the one_test() function: what happens to the power, and to the share of false alarms among the significant ideas?
• Repeat the p_null simulation and count the tests with p between 0.04 and 0.05, and those between 0.94 and 0.95. Why are they about the same number?

The last exercise is the shortest, and perhaps the most instructive: when there is no effect, an “almost significant” p-value is no closer to the truth than a very high one.

We have seen that, among significant results, the share of false alarms depends on how good the ideas we test are. But there is an even more insidious way of being fooled by successes: looking only at the ones that made it. The case studies we read tell of the strategies that worked, never of the hundred identical ones that failed. It is survivorship bias, and it is the next step of our journey through the traps of data.

Further reading

On the p-value, its uses and its abuses, I recommend The Art of Statistics by David Spiegelhalter: it explains what a significance test really says, why so many “significant” results do not replicate and how to reason about uncertainty without being dazzled by a threshold.

In our line of work the dead salmon has a familiar name: Search Console. We compare this month’s CTR with last month’s across 200 queries, find a dozen “significant” changes and write in the report which queries improved. But how many of those twelve really changed? This is the problem of multiple comparisons, and we have already seen its outline when talking about SEO split testing and peeking.

In short: when many statistical tests are run together, the 5% threshold no longer holds for the whole set: with 10 independent tests the probability of at least one false positive rises to 40%, with 100 to 99%. Multiple comparisons are corrected in two ways. The Bonferroni correction (or its improved version, the Holm method) controls the probability of making even a single false positive, by dividing the threshold by the number of tests: it is cautious, but loses many real effects. The Benjamini-Hochberg method instead controls the share of false positives among the discoveries (the false discovery rate): it is less strict and better suited to sifting through many pages or queries. In R we use p.adjust().

What we will cover:

• Why do more tests mean more false alarms?
• An example: 200 queries in Search Console
• The Bonferroni correction (and Holm’s)
• Family-wise error or share of false discoveries?
• The Benjamini-Hochberg method
• Which correction should we choose?
• Where multiple comparisons hide in SEO
• Frequently asked questions
• Try it yourself
• Further reading

Why do more tests mean more false alarms?

Let’s recall what the 5% threshold promises in a hypothesis test: if there is no effect, the test will give us a false alarm 5 times in 100. It is a promise that holds for one test. The trouble begins when we run many of them, and any one turning out significant is enough for us to raise our hand.

A quick example. Rolling a die once, the probability of a 6 is one in six; rolling it ten times, the probability of at least one 6 rises to 84%. Tests work the same way. If one test alone has a 95% probability of not giving a false alarm, two independent tests have 95% of 95%, and so on. The probability of at least one false positive across m independent tests, which statisticians call the family-wise error rate (FWER), is therefore:

\(FWER = 1 – (1 – \alpha)^m\)

where α is the threshold of the single test (0.05) and m the number of tests. With 10 tests it is 1 − 0.95¹⁰ = 40%; with 14 it passes 50%; with 100 it reaches 99.4%.

There is another way to see it, even more direct: across m tests with no real effect, the expected number of false positives is m × α. Across 200 queries that did not change at all, we should expect about ten “significant” changes (200 × 0.05 = 10). They are not an anomaly: they are exactly what the 5% threshold promises.

An example: 200 queries in Search Console

We build the example by simulation, so we know the truth. Take 200 queries, each with a few thousand impressions a month and a CTR between 1% and 6%. For 180 of them nothing changes from one month to the next; for the first 20 the CTR really rises, by 50%. For each query we compare the two months with a test on proportions, as we would in a report.

I generate the data in R:

set.seed(7)
n_queries <- 200
impressions <- round(300 + rlnorm(n_queries, log(5000), 0.8))  # monthly impressions
ctr <- runif(n_queries, 0.01, 0.06)                             # CTR between 1% and 6%
real <- seq_len(n_queries) <= 20                                # the 20 that really change
clicks_before <- rbinom(n_queries, impressions, ctr)
clicks_after  <- rbinom(n_queries, impressions, ifelse(real, ctr * 1.5, ctr))

# one test per query: last month's CTR against this month's
p <- mapply(function(a, b, n) prop.test(c(a, b), c(n, n))$p.value,
            clicks_before, clicks_after, impressions)

Now let’s do what is usually done: flag as “changed” the queries with a p-value below 0.05, and compare with the truth.

table(discovery = p < 0.05, real)

#          real
# discovery FALSE TRUE
#     FALSE   171    2
#     TRUE      9   18

The test flagged 27 queries. Eighteen really changed, but nine did not: they are false alarms, exactly 5% of the 180 unchanged queries. In plainer terms: a third of our report’s “discoveries” is noise, and nothing in the numbers tells us which third. If we devoted an analysis, an intervention or a line of commentary to the client to each of those nine queries, we would be working on a ghost.

The Bonferroni correction (and Holm’s)

The oldest solution, and the most intuitive, is to make every single test stricter. If we want the probability of at least one false positive over the whole family of tests to stay at 5%, it is enough to divide the threshold by the number of tests. This is the Bonferroni correction:

\(\alpha_{Bonferroni} = \frac{\alpha}{m} = \frac{0.05}{200} = 0.00025\)

A query has “changed” only if its p-value falls below 0.00025. In R there is no need to do the division by hand: p.adjust() returns adjusted p-values, to be compared with the usual threshold. I check it in R:

table(discovery = p.adjust(p, method = "bonferroni") < 0.05, real)

#          real
# discovery FALSE TRUE
#     FALSE   180    6
#     TRUE      0   14

No false alarms: the 14 queries flagged are all real. The price, though, shows in the top right-hand corner. Six queries that really changed have vanished from the radar, because a 50% rise on a few hundred clicks is not enough to fall below such a low threshold. Bonferroni buys the certainty of not being wrong with blindness to part of the real effects.

There is a refinement that costs nothing, the Holm method. Sort the p-values from smallest to largest; compare the first with α/m, the second with α/(m − 1), the third with α/(m − 2), and stop at the first one that does not pass its threshold. It guarantees the same control as Bonferroni and is always at least as powerful, often more so. In our example it gives the same result (14 discoveries, none false); with p.adjust(p, method = "holm") there is no reason to prefer Bonferroni over it.

Family-wise error or share of false discoveries?

Before going further, it is worth pausing on a question that sounds philosophical and is in fact practical: what do we want to control?

Bonferroni and Holm control the probability of making even one false positive. That is the right requirement when every error is costly: we decide to redo the template of every product page on the strength of a test, and a false positive means weeks of wasted work.

But in many SEO analyses the situation is different. We sift through 200 queries or 500 pages to understand which deserve a closer look. Here a false positive is not a catastrophe: it is one more glance at a page that did not need it. What we do not want is for the discoveries to be mostly false alarms. The quantity to control then becomes the share of false positives among the discoveries, which statisticians call the false discovery rate (FDR). An FDR of 5% means: of the queries we flag, on average 5% will be noise.

They are two different promises, and the second is far less demanding than the first: that is why, with the same data, it lets through more real effects.

The Benjamini-Hochberg method

The method that controls the FDR, proposed by Yoav Benjamini and Yosef Hochberg in 1995, is surprisingly simple. In words: we sort the p-values from smallest to largest and, instead of a fixed threshold, we use a threshold that grows with the position in the ranking. The p-value in first place must be below 1/m × 0.05, the one in second place below 2/m × 0.05, and so on:

\(p_{(i)} \le \frac{i}{m} \cdot q\)

where p₍ᵢ₎ is the p-value in position i, m the number of tests and q the FDR we accept (here 0.05). We look for the lowest position in the ranking (the largest p-value) that satisfies the inequality: that query and all those before it are the discoveries.

Let’s look at the p-values of our example from position 14 to 20, each with its threshold:

The last position below its threshold is 17: the first 17 queries are the discoveries. Bonferroni, with its fixed threshold of 0.00025, would have stopped at 14. I check it in R:

table(discovery = p.adjust(p, method = "BH") < 0.05, real)

#          real
# discovery FALSE TRUE
#     FALSE   179    4
#     TRUE      1   16

Seventeen discoveries, sixteen real and one false: one false alarm in seventeen, 5.9%, in line with the promised 5%. Compared with Bonferroni we have recovered two queries that really changed, at the price of a single error. And compared with the uncorrected report we have gone from nine false alarms in 27 to one in 17.

The chart shows the logic well. At the top of the ranking are the large effects, with minuscule p-values: any method finds them. The problem is the middle zone, where real but modest effects mix with noise. Bonferroni discards it wholesale; the flat threshold accepts it wholesale; Benjamini-Hochberg crosses it with a threshold that widens just enough.

(n.b. the method guarantees an average FDR of 5%: in a single report the actual share can be higher or lower. Repeating the simulation 300 times, Benjamini-Hochberg produces on average 0.6 false discoveries per report, a share of 3.7%: below the promised 5%.)

Which correction should we choose?

There is no correction that is right in absolute terms: there is the one that fits the question. In short:

The most common case in day-to-day practice, however, is another one: we had not decided anything beforehand. We opened Search Console, filtered, segmented, compared, and stopped at the numbers that looked interesting. In this scenario no correction is possible, because we do not even know how many tests we really ran.

Where multiple comparisons hide in SEO

The month-on-month query report is the most obvious case, but multiplicity comes in many disguises.

Segments. “CTR has not changed, but on mobile in Lombardy it has”: slice the data enough ways by device, country, category and page type, and some significant slice always turns up. It is the same trap as Simpson’s paradox, seen from the opposite side.

Metrics. In an SEO split test, looking at clicks, impressions, CTR, position and conversions means running five tests. That is why the primary metric is chosen in advance.

Variants. An A/B test with five variants against the control is five comparisons; if we also compare the variants with each other they become fifteen. For comparisons between all pairs of means, ANOVA has a dedicated tool of its own, Tukey’s test.

Time. Looking at a test every day while reserving the right to stop it is a multiplicity hidden in the calendar: it is the peeking problem.

Correlations. A correlation matrix among 20 metrics contains 190 pairs: about ten will turn out “significant” even if the metrics were all independent.

(n.b. in reality things are even worse than in our simulation. A query’s CTR does not fluctuate only because of the binomial chance of clicks: it changes with position, with SERP features, with the device mix. prop.test() ignores these sources of variability and underestimates the uncertainty, so on real data false alarms exceed 5%. Multiple comparisons corrections do not fix a wrong model: they assume it is right.)

Frequently asked questions

What are multiple comparisons in statistics?

They are situations in which many statistical tests are run on the same data or the same question: comparing many queries, many segments, many metrics or many variants. Each test has its own probability of a false positive, and together the probabilities add up: with enough tests, some “significant” result arrives by pure chance.

How is the Bonferroni correction applied?

Divide the significance threshold by the number of tests: with 20 tests and a 5% threshold, each test needs a p-value below 0.05 / 20 = 0.0025. Equivalently, multiply each p-value by the number of tests and compare it with 0.05. In R, p.adjust(p, method = "bonferroni") does it.

What is the difference between FWER and FDR?

The FWER (family-wise error rate) is the probability of making at least one false positive across the set of tests; the FDR (false discovery rate) is the average share of false positives among the results declared significant. Controlling the FWER is stricter and suited to a few costly decisions; controlling the FDR suits the exploration of many hypotheses.

When should we use Benjamini-Hochberg instead of Bonferroni?

When the tests are many and the aim is to select what to investigate, accepting that a small share of the discoveries is false: analyses of hundreds of queries or pages, screening of segments, anomaly searches. In these cases Bonferroni is so strict that it loses a good part of the real effects.

Is the correction needed even with few tests?

With two or three tests the overall risk grows only a little (from 5% to about 10-14%), but it grows. If the comparisons were decided in advance and are few, the Holm method costs very little in power and keeps us safe. If instead a single primary comparison was decided in advance, no correction is needed.

Try it yourself

The example code is complete, and lends itself to a few experiments.

• Set the number of queries that really change to 0 (real <- rep(FALSE, n_queries)) and count the discoveries with and without correction. Without correction there should be about ten; with Bonferroni and with Benjamini-Hochberg almost always zero. This is the dead salmon case.
• Bring the CTR increase down from 1.5 to 1.2: a smaller effect, more realistic for a rewritten title. How many of the 20 real queries does each method find?
• Repeat the simulation with 2,000 queries instead of 200, still with 20 real ones. What happens to the Bonferroni threshold, and how many real queries survive?

The last exercise, in particular, shows why large sites need the FDR: the more tests we run, the blinder Bonferroni makes us.

We have talked about p-values throughout the article: correcting them, ranking them, comparing them with different thresholds. But what exactly does a p-value of 0.03 say? That there is a 97% probability the effect is real? That the null hypothesis has a 3% probability of being true? Neither, and it is perhaps the most misunderstood number in all of applied statistics. It will be the subject of the next step: the p-value and its misconceptions.

Further reading

To understand why so many «significant» results do not stand up to the facts, and how much the number of questions we have put to the data matters, I recommend The Art of Statistics by David Spiegelhalter: a book that explains uncertainty without needless formulas and without shortcuts. For multiple comparisons in online experiments, from metrics to variants, the reference remains Trustworthy Online Controlled Experiments by Ron Kohavi, Diane Tang and Ya Xu.

In marketing, the classic answer to this problem is the A/B test: two versions, users split at random, results compared. With SEO, though, there is an insurmountable obstacle, because Google sees only one version of the page. We cannot split the users: we can split the pages. That is the idea behind SEO split testing, and the statistical tool that makes it reliable has a name that sounds like a tongue-twister: difference-in-differences.

In short: an SEO split test measures the effect of a change (titles, templates, structured data) by randomly dividing a set of similar pages into test pages, which receive the change, and control pages, which are left untouched. A before/after comparison on its own is misleading, because seasonality and core updates hit every page; difference-in-differences (DiD) subtracts the change in the control pages from the change in the test pages, and what remains is the effect of the edit. A t-test on the page-by-page changes tells us whether that effect can be distinguished from chance. The pages must be numerous and homogeneous: detecting a +8% effect typically takes a few hundred of them.

What we will cover:

• Why the classic A/B test does not work for SEO
• Why before/after comparisons mislead
• How do we set up an SEO split test?
• An example: 200 product pages and a core update
• What is difference-in-differences?
• How do we check that the control is a good control?
• How many pages do we need, and for how long?
• The pitfalls of split testing
• What if we cannot randomise?
• Frequently asked questions
• Try it yourself
• Further reading

Why the classic A/B test does not work for SEO

In an A/B test on a landing page’s conversion rate, each visitor sees version A or version B, assigned at random. Since randomisation makes the two groups of users alike in everything else, a difference in results can be attributed to the version. Chance acts as the guarantor.

With rankings the mechanism jams straight away. In SEO, the user who counts is only one: Googlebot. If we show it two versions of the same page depending on the visit, at best it cannot tell which one to index, at worst it is called cloaking. And the outcome we want to measure (rankings, impressions, organic clicks) depends on how Google evaluates the page, not on how an individual visitor experiences it.

The way out is to move the randomisation one level up. Instead of users, we draw pages: one group gets the change, the other gets nothing. It works when the site has many pages built on the same template, answering the same kind of search: the product pages of an e-commerce site, the listings of a classifieds site, category pages, the city pages of a portal. A site with ten pages that are all different, it should be said at once, cannot run a split test: it does not have enough units to randomise.

Why before/after comparisons mislead

Without a control group, the only possible comparison is with the past. The trouble is that the past differs from the present in a thousand ways that have nothing to do with our change:

• seasonality: in December people search for gifts, in August far less for everything else;
• algorithm updates, which move the traffic of entire sectors within a few days;
• competition, which in the meantime publishes, improves, disappears;
• regression to the mean: if we chose to work precisely on the pages that had done worst, part of the improvement would have come anyway.

Each of these factors can be larger than the effect we are looking for. A better title might be worth 5-10% more clicks; a core update can take away 20% in a week. Comparing only before with after, we measure the sum of everything that changed, and our edit is just one piece of it, often the smallest.

The control group solves the problem at the root: control pages live through the same season, the same core update, the same competition as the test pages. The only thing that sets them apart is the change.

How do we set up an SEO split test?

The rules are few, but all of them must be respected.

Homogeneous pages. Test and control must be pages of the same type, with the same template and the same search intent. Mixing product pages and blog posts means comparing apples with oranges.

Random assignment. We do not choose which pages to change: we draw them. If we picked “the most promising ones”, we would introduce exactly the initial difference the test is meant to avoid. A useful refinement is paired randomisation: sort the pages by traffic, form pairs of neighbouring pages and, within each pair, toss a coin to decide which one goes into the test. The two groups are then born balanced on traffic too, which is the bulkiest factor.

A long enough “before” period. We need it to know how the pages behave when nobody touches them, and to check that test and control move together (we will come back to this). Six to eight weeks is a good starting point.

Decide in advance when to look. Test duration and metric are fixed beforehand. Looking at the data every morning and stopping at the first favourable signal is the peeking problem, and it inflates false positives exactly as it does in an A/B test.

An example: 200 product pages and a core update

As always, an example helps to clarify things. We build it by simulation, so we know the truth and can see which methods recover it.

Let’s imagine an e-commerce site with 200 product pages on the same template. We decide to rewrite the titles of half of them, drawn in pairs. In our simulated world the change works: it brings 8% more clicks. But a week after it goes live, a core update takes 15% of clicks away from every page on the site, test and control alike. And every page, as always happens, also fluctuates on its own, going up or down a few positions.

I build the pages in R, with paired randomisation:

set.seed(2020)
n_pages <- 200
base <- rlnorm(n_pages, meanlog = log(30), sdlog = 1)   # average clicks per day

# paired randomisation: sort by traffic and draw within each pair
ord <- order(base)
group <- character(n_pages)
for (k in seq(1, n_pages, by = 2)) {
  group[ord[k:(k + 1)]] <- sample(c("test", "control"))
}
pages <- data.frame(page = 1:n_pages, base = base, group = group,
                    drift = rlnorm(n_pages, -0.25^2 / 2, 0.25))

tapply(pages$base, pages$group, median)
#  control     test
#    32.36    31.96

Paired randomisation has done its job: the “typical” page makes about 32 clicks a day in both groups.

Then I generate 14 weeks of daily clicks: 8 before the change and 6 after. On day 57 the new titles arrive (on the test pages only), on day 64 the core update (on all of them); at weekends traffic drops by 30%.

simulate_clicks <- function(pages, effect = 1.08, change = 57, core_update = 64) {
  d <- expand.grid(page = pages$page, day = 1:98)
  d$group <- pages$group[d$page]
  lambda <- pages$base[d$page] *
    ifelse(d$day %% 7 %in% c(6, 0), 0.7, 1) *                   # weekend
    ifelse(d$day >= core_update, 0.85, 1) *                     # core update
    ifelse(d$day >= change, pages$drift[d$page], 1) *           # drift
    ifelse(d$group == "test" & d$day >= change, effect, 1) *    # the change
    rlnorm(nrow(d), 0, 0.15)                                    # noise
  d$clicks <- rpois(nrow(d), lambda)
  d$period <- ifelse(d$day >= change, "after", "before")
  d
}
clicks <- simulate_clicks(pages)

(n.b. the drift is each page’s own fluctuation after the change: new competitors, links gained or lost, small shifts in position. It averages 1, but an individual page can gain or lose 30-40%.)

Now let’s put ourselves in the shoes of someone who made the change and has no control group. For each page I compute the average daily clicks before and after, and their change:

changes <- function(d) {
  m <- aggregate(clicks ~ page + group + period, data = d, FUN = mean)
  w <- reshape(m, idvar = c("page", "group"), timevar = "period",
               direction = "wide")
  w$delta <- log(w$clicks.after / w$clicks.before)
  w
}
w <- changes(clicks)

# average change of the test pages, as a percentage
exp(mean(w$delta[w$group == "test"])) - 1
# [1] -0.101

The pages with the new titles lost, on average, 10% of their clicks. Without a control, the conclusion would be immediate: the change did damage, let’s roll it back. And it would be wrong, because we know (we decided it ourselves) that the change works.

(n.b. we work on the logarithm of the after/before ratio for two reasons. Pages have wildly different volumes, from one click a day to a few hundred, and an effect such as “+8%” is proportional, not a fixed number of clicks: on the log scale, percentage changes become differences, which can be added and averaged. And the mean of the logarithms, brought back with exp(), is the geometric mean: it does not get dragged around by a page going from 2 to 9 clicks.)

What is difference-in-differences?

Now let’s look at the control pages as well:

tapply(w$delta, w$group, function(x) exp(mean(x)) - 1)
# control    test
#  -0.166  -0.101

The control pages, which nobody touched, lost 16.6%. That is the core update, plus the fluctuations of the period: it is what would have happened to the test pages too, had we not changed them. Economists call it the counterfactual, the story that was never staged.

The whole idea of difference-in-differences is right here. First we compute the change in each group (the first difference), then the difference between the two changes (the second difference). In words: the effect of the change is how much better, or worse, the test pages did compared with the control pages. As a formula, writing Δ for the average change on the log scale:

\(DiD = \bar{\Delta}_{test} – \bar{\Delta}_{control} = (\log T_{after} – \log T_{before}) – (\log C_{after} – \log C_{before})\)

where T and C are the clicks of the test and control pages. Everything that hit both groups in the same way (seasonality, the core update, weekends) appears in both brackets and cancels out in the subtraction. Only what distinguishes the groups is left: the change.

The chart tells the story better than any table. In the first week after the change the test pages really do rise, and anyone looking only at those days would cry victory; then the core update arrives and everything plunges. But the control plunges further. The gap between the two lines, which holds until the end, is our effect.

It remains to be seen whether that gap is large enough not to be a fluke. The pages are few and noisy: perhaps the draw put into the test group, through sheer bad luck for the control, a few pages that would have risen anyway. That is a question for a statistical test, and since we have one change per page and two groups to compare, the tool is the familiar two-sample t-test. I compute it in R:

t.test(delta ~ group, data = w)

# t = -2.1086, df = 197.34, p-value = 0.03624
# 95 percent confidence interval:
#  -0.144123939 -0.004821942
# mean in group control    mean in group test
#            -0.1809326            -0.1064596

The difference between the two means is 0.0745 on the log scale, that is exp(0.0745) − 1 = +7.7% clicks: very close to the true 8%. The p-value of 0.036 tells us that a difference this large, had the change been useless, would come up by chance fewer than 4 times in 100.

The confidence interval, however, is the most honest piece of information. R reports it for the difference control − test; flipped the way round we care about and turned into a percentage, it runs from +0.5% to +15.5%. In plainer terms: the change almost certainly worked, but the data are compatible both with a negligible effect and with an effect twice as large as the estimate.

This second figure explains why nobody should judge a change by looking at one page, or even ten. The fluctuations of a single page are much larger than the effect we are looking for: dozens of pages with the new titles lost more than untouched control pages. Only the average over many pages, compared with the average of the control, brings the signal to the surface.

(n.b. the same result comes out of a regression, lm(delta ~ group, data = w): the grouptest coefficient is 0.0745, with the same p-value. It is the form to use when we want to add other variables, such as page type, and the reason why textbooks almost always present DiD as a regression model.)

How do we check that the control is a good control?

The whole argument rests on one assumption: that, without the change, test and control would have moved in the same way. It is called the parallel trends assumption, and it is the point where a split test stands or falls.

We cannot check it in the “after” period, because there we do not see the counterfactual. But we can check it in the “before” period, when nobody had touched anything. Two tools, both extremely useful.

The first is the chart we have already seen: before the change the lines must overlap. If in the “before” period the test pages were already growing faster than the control, the draw went wrong, or the pages are not really homogeneous, and any later difference is suspect.

The second is a placebo test: we pretend the change happened halfway through the “before” period and repeat the analysis. Since nothing happened at that moment, the DiD must come out as zero. I check it in R, moving the fake change to day 29:

pre <- subset(clicks, day < 57)
pre$period <- ifelse(pre$day >= 29, "after", "before")
t.test(delta ~ group, data = changes(pre))

# t = 0.071079, df = 197.57, p-value = 0.9434

A difference of practically zero (−0.09%), p-value 0.94: the control behaved like the test when nothing told them apart. That is exactly what we want to see. A “significant” placebo test, on the contrary, would be an alarm bell not to be ignored.

How many pages do we need, and for how long?

In our example it went well: p-value 0.036, effect recovered. But how well did it go? We find out with power analysis: the probability that the test detects an effect that is really there.

To compute it we need three ingredients: the effect we want to detect (+8%, that is log(1.08) = 0.077 on the log scale), the number of pages and the variability of the page-by-page changes within each group. We estimate the latter from the test data: it is about 0.25. Computing it in R:

s <- sqrt(mean(tapply(w$delta, w$group, var)))   # within-group standard deviation
s
# [1] 0.2497

power.t.test(n = 100, delta = log(1.08), sd = s)$power
# [1] 0.583

power.t.test(power = 0.8, delta = log(1.08), sd = s)$n
# [1] 166.3

With 100 pages per group, power is 58%: little better than a coin toss. If we repeated the very same experiment many times, in four cases out of ten we would conclude that the change is useless, while knowing that it works. To reach 80%, the usual threshold, we need about 166 pages per group, more than 330 in total.

The figure gives the measure of the problem. The typical effects of SEO changes are small, and small effects require many pages. A +15% shows up even on a hundred product pages; a +5%, which on a large e-commerce site is still worth a lot of money, requires more than 800.

Before starting, then, it pays to do the sum backwards: how many pages do we have? With that number, what minimum effect can we hope to see? The variability is estimated from the same pages, comparing two past periods as long as the test. If the minimum detectable effect is larger than what we can reasonably expect, the test is underpowered. In that case we either widen it to more pages, or accept that it will only say “I don’t know”.

As for time, two considerations. Google must first crawl and reprocess the changed pages, and on large sites that can take more than a week: the first days after the change are often best left out of the comparison. Then, a longer “after” period reduces day-to-day noise, but not page-to-page noise, which is the bulk of it. Lengthening the test helps less than widening it: four to six weeks is usually a good compromise.

The pitfalls of split testing

The method is sound, but in day-to-day practice there is no shortage of ways to be fooled.

Contamination. Test and control pages do not live in isolation: they link to each other, share menus and category pages. If the change alters internal links, for instance by adding a related-products block, the effect spills over onto the control too, and DiD underestimates it.

Pages with very few clicks. A page going from 2 clicks a day to 0 has an infinite change on the log scale. It is better to exclude from the start the pages below a minimum traffic threshold, or to work on impressions, which are far more numerous.

The metric chosen after the fact. Clicks, impressions, CTR, average position, conversions: if we look at all of them, one will turn out “significant” by chance. The primary metric is decided beforehand, and the others remain background information. This is the multiple comparisons problem, and it deserves a discussion of its own.

Groups that are too different. If the site has pages with very different volumes, it pays to randomise in pairs or in blocks (by category, by traffic band), as we did. A simple draw over 40 pages can produce, through sheer bad luck, a test group holding all the seasonal pages.

What if we cannot randomise?

So far we have taken for granted that we can choose which pages to change. But some of the most important, and riskiest, decisions cannot be split in two: a domain or platform migration, a new template for the whole site, the switch to HTTPS. In these cases there is no internal control group, because all the pages change together. And the before/after comparison is as unreliable as ever.

The idea of difference-in-differences, however, survives. If we cannot observe the counterfactual inside the site, we can rebuild it from outside: from series that live through the same seasons and the same algorithm updates, but that the migration does not touch. A sister site from the same group, a section of the site left out of the migration, the Google Trends index for the main topics. In the “before” period we estimate how our traffic moves together with these series; in the “after” period we use the same relationship to predict how much traffic we would have had without the migration. The difference between observed and predicted is the effect.

Let’s take an example. I simulate 180 days of clicks for a site that migrates on day 121, losing 12% of its traffic; ten days later a core update takes 25% away from the whole sector. Alongside there are two control series: a sister site that did not migrate, and Google Trends.

set.seed(3)
days <- 1:180; migration <- 121                           # 4 months before, 2 after
sector <- exp(cumsum(rnorm(180, 0, 0.02))) *              # sector trend
          ifelse(days %% 7 %in% c(6, 0), 0.75, 1) *        # weekend
          (1 - 0.25 * (days > 130))                        # sector-wide core update
noise <- function(sd) exp(as.numeric(arima.sim(list(ar = 0.6), 180, sd = sd)))
site   <- 800 * sector * noise(0.05) * ifelse(days >= migration, 0.88, 1)
sister <- 300 * sector * noise(0.03)                       # sister site, not migrated
trends <- 60  * sector * noise(0.05)                       # Google Trends for the topic
before <- days < migration

mean(site[!before]) / mean(site[before]) - 1
# [1] -0.273

The before/after says −27%: more than twice the true effect, because it charges the core update to the migration too. So I build the counterfactual with a multiple regression on the “before” period, on the log scale, and project it onto the “after” period:

m <- lm(log(site) ~ log(sister) + log(trends), subset = before)
summary(m)$r.squared
# [1] 0.792

predicted <- predict(m, newdata = data.frame(sister, trends))
exp(mean(log(site[!before]) - predicted[!before])) - 1
# [1] -0.125

sum(exp(predicted[!before])) - sum(site[!before])   # clicks lost in 60 days
# [1] 4532.7

The two control series explain 79% of the site’s daily fluctuations before the migration. The comparison with the counterfactual estimates an effect of −12.5%, a step away from the true −12%: about 4,500 clicks lost in two months, out of 36,400 expected.

How far can we trust that −12.5%? The days of a time series are not independent of each other (a good day tends to be followed by another good day), and an interval computed as if they were would be too optimistic. Here too the placebo comes to our aid, this time in time: we pretend the migration happened at various moments of the “before” period, estimate the model on the preceding days and measure the fake “effect” over the next 30 days.

placebo <- sapply(seq(40, 90, by = 5), function(k) {
  mk <- lm(log(site) ~ log(sister) + log(trends), subset = days < k)
  pk <- predict(mk, newdata = data.frame(sister, trends))
  exp(mean(log(site[k:(k + 29)]) - pk[k:(k + 29)])) - 1
})
range(placebo)
# [1] -0.018  0.028

When nothing happened, the method “finds” effects between −1.8% and +2.8%: that is its noise threshold. The migration’s −12.5% lies well outside that band, and we can regard it as real.

The method we have just built by hand is the heart of CausalImpact, the R package developed at Google for exactly this problem. Instead of a plain regression it uses a Bayesian time-series model, which accounts for trend, seasonality and the dependence between days, and returns the effect directly with its credible interval. The logic is the same: a counterfactual built from series the intervention did not touch.

Frequently asked questions

What is an SEO split test?

It is an experiment that measures the effect of a change on organic traffic by randomly dividing a set of similar pages into two groups: the test pages receive the change, the control pages stay as they are. Comparing how the two groups change separates the effect of the edit from seasonality, algorithm updates and competition.

What is the difference between an SEO split test and an A/B test?

An A/B test splits users: each one sees one of two versions of the same page. An SEO split test splits pages, because Google must see a single version of each. That is why it needs a site with many pages of the same type.

How long should an SEO split test run?

Usually six to eight weeks of data before the change and four to six weeks after, leaving out the first days while Google is still crawling the changed pages. Lengthening the period helps less than increasing the number of pages.

Can we run a split test with only a few pages?

With a few dozen pages only large effects, 15-20% or more, can be detected. For typical effects of 5-10% a few hundred pages are usually needed: it is worth computing the test’s power before starting.

How do we measure the impact of an SEO migration?

By comparing post-migration traffic with a counterfactual: a prediction of how much traffic there would have been without the migration, built from series the migration does not touch (a sister site, a section that was not migrated, Google Trends). The R package CausalImpact automates this approach.

What can we test with an SEO split test?

Any change that can be applied to a group of pages on the same template: titles and meta descriptions, heading structure, structured data, content blocks, internal links, page speed. Changes that touch the whole site at once, such as a migration, are not suitable: there is no internal control group, and a counterfactual built from outside is needed.

Try it yourself

The example code is complete: just copy it into R and change one parameter at a time.

• Set effect to 1 (the change does nothing) and repeat the analysis with a few different set.seed() values. How often does the t-test wrongly declare a significant effect? (It should happen about 5 times in 100.)
• Set effect back to 1.08 and double the pages, n_pages <- 400: how much does the confidence interval narrow?
• “Break” the randomisation, putting the 100 pages with the most traffic into the test group instead of drawing them, and give the big pages a more favourable drift. What does the placebo test say?

For the last exercise, a hint: it is the quickest way to understand why randomisation is not a bureaucratic detail.

There is a question we have brushed against without tackling it. In a split test we usually look at one primary metric, but the temptation to look at ten is very strong: clicks, impressions, CTR, position, and then every category of pages on its own. Every extra look is one more chance to find a “significant” effect that does not exist. How quickly these false alarms pile up, and how to correct for them, is the subject of multiple comparisons: the next step of our journey through the traps of data.

Further reading

On online experimentation, the reference is Trustworthy Online Controlled Experiments by Ron Kohavi, Diane Tang and Ya Xu: from choosing metrics to the pitfalls of control groups, it is written by the people who ran experimentation at Microsoft, Google and LinkedIn.

We have already tackled a close problem, classifying the intent of a query with Naive Bayes — but there we had an ingredient we lack today: a set of already labelled examples to learn from. Here nobody has handed us the labels. This is the territory of clustering, one of the most used tools of machine learning, and in this article we build it in R with its two classic algorithms: K-means and hierarchical clustering.

What we will cover:

• Grouping without labels: the idea of clustering
• K-means: centroids and the problem of choosing k
• Reading the clusters: who are these groups?
• Hierarchical clustering: the dendrogram
• Which method, and the traps
• Try it yourself
• Further reading

Grouping without labels: the idea of clustering

The difference with Naive Bayes is one of principle, not of detail. There we did supervised learning: we had queries already marked as informational, navigational or transactional, and we taught the algorithm to recognise new ones. Here we do unsupervised learning: nobody told us which and how many groups exist. Clustering does not verify a label we already know: it looks for a structure we did not know was there.

Grouping requires two things. The first is describing each keyword with numbers: in our example we will use search volume, cost per click (cpc), average position and word count.
The second is a notion of distance: two keywords are “close” if their numbers are alike. The most common distance is the Euclidean one, the same we would use on a map, only computed in a four-dimensional space (one per metric).

There is, however, a trap to defuse straight away. Volume is measured in tens of thousands, cpc in cents of a euro: left as they are, the distances would be dominated by volume, and cpc would count for almost nothing.
Before computing any distance we must put all the variables on the same scale, standardising them — in R with the scale() function, which subtracts the mean from each column and divides by the standard deviation. Only then does one euro of difference in cpc and ten thousand searches of difference in volume “weigh” the same.

K-means: centroids and the problem of choosing k

The idea of K-means is almost naive in its simplicity. We decide into how many groups (k) we want to split the data; the algorithm places k representative points, the centroids, and then repeats two steps until things settle: it assigns each keyword to the nearest centroid, then moves each centroid to the centre of the keywords assigned to it. At each pass the groups grow a little more cohesive, until they stop moving.

What the algorithm tries to minimise, in words, is the internal spread of the groups: the sum of the (squared) distances of each point from the centroid of its own cluster. In a formula:

where \( C_k \) is the k-th cluster, \( \mu_k \) its centroid and the double sum runs over all points of all groups. The lower the WCSS (within-cluster sum of squares), the more compact the groups.

The sore point remains: we have to decide k ourselves, before starting. A help comes from the elbow method: we try several values of k and watch how the WCSS falls. At first adding a cluster helps a lot, then the improvements become marginal; the “elbow” of the curve — the point where the descent flattens — suggests a reasonable k. I build the keyword table and compute the WCSS from 1 to 6 groups in R:

kw <- data.frame(
  keyword = c("running shoes","running shoes men","nike pegasus","nike pegasus 40",
              "best trail running shoes 2026","how to choose running shoes",
              "running shoes deal","buy trail shoes online",
              "trail vs road running shoes","running shoes overpronation",
              "asics gel nimbus","saucony endorphin","trail running shoes review",
              "discount running shoes","running shoe store london"),
  volume   = c(40000,18000,12000,8000,2400,1900,3200,880,1300,2100,9000,4000,1500,2600,720),
  cpc      = c(0.45,0.55,0.30,0.35,0.40,0.10,0.95,1.10,0.08,0.30,0.28,0.33,0.15,0.90,0.85),
  position = c(3.1,4.2,2.0,5.5,8.1,11.2,6.0,9.4,14.0,7.3,2.5,6.8,12.1,5.9,4.7),
  n_words  = c(2,3,2,3,5,5,3,4,6,3,3,2,3,3,4)
)

# standardise the four metrics (different scales -> same weight)
X <- scale(kw[, c("volume","cpc","position","n_words")])

# elbow method: WCSS for k from 1 to 6
set.seed(1)
wss <- sapply(1:6, function(k) kmeans(X, centers = k, nstart = 10)$tot.withinss)
round(wss, 1)
# [1] 56.0 34.4 20.7 12.4  8.7  6.9

The drop is steep up to three groups (56 → 34 → 21) and then slows down markedly (21 → 12 → 9 → 7). The elbow is never a sharp line — it is a reading, not a theorem — but here it reasonably points to k = 3. So I run K-means with three centroids:

set.seed(1)
km <- kmeans(X, centers = 3, nstart = 25)
kw$cluster <- km$cluster
table(km$cluster)
# 1 2 3
# 4 7 4

n.b. the argument nstart = 25 restarts the algorithm 25 times from different initial centroids, keeping the best solution: K-means can in fact get stuck in a local minimum depending on where it starts, and restarting several times is the standard defence. The set.seed(1) only serves to make the example reproducible (including the cluster numbering, which is itself arbitrary).

Reading the clusters: who are these groups?

Having three groups is useless until we understand what they represent. The most direct way is to look at the average metrics of each cluster.
I compute them on the original scale (not the standardised one, which is unreadable):

aggregate(kw[, c("volume","cpc","position","n_words")],
          by = list(cluster = kw$cluster), FUN = mean)
#   cluster volume  cpc position n_words
# 1       1   1775 0.18    11.35    4.75
# 2       2  13300 0.37     4.49    2.57
# 3       3   1850 0.95     6.50    3.50

Now the groups speak.
Cluster 2 collects the very high-volume queries (13,300 searches on average), short (two-three words), well positioned and with modest cpc: they are the generic and brand heads — “running shoes”, “nike pegasus”, “asics gel nimbus”. Cluster 1 has low volumes, long queries (almost five words), low positions and minimal cpc: it is the informational long-tail — “how to choose running shoes”, “trail vs road running shoes”, “trail running shoes review”. Cluster 3 stands out for a very high cpc (€0.95) and contains the clearly commercial queries — “running shoes deal”, “buy trail shoes online”, “discount running shoes”, “running shoe store london”.

It is worth pausing a moment on what just happened. Without giving the algorithm any label, the three groups that emerge closely echo a split by intent — informational research, generic and brand heads, commercial queries — close to the one that with Naive Bayes we had instead to teach it through examples.
The alignment, mind you, is not magic: it emerges because the metrics we chose (cpc, length, position) indirectly track intent, not because clustering knows it — of the queries’ meaning, here, it has not seen a single word. The reading is still immediately actionable: the informational cluster calls for articles and guides, the commercial one for product and offer pages, the high-volume one for robust pillar pages.

Hierarchical clustering: the dendrogram

K-means forced us to choose k in advance. Hierarchical clustering flips the approach: it decides nothing a priori, and builds instead a complete tree of groupings. It starts with each keyword as a group of its own, then progressively merges the two closest, then the two closest among those remaining, and so on up to a single big group. The result is a dendrogram: a tree showing at what “height” (that is, at what distance) each merge happens.

I build it in R by first computing the distance matrix, then the tree:

d  <- dist(X)                       # euclidean distances between standardised keywords
hc <- hclust(d, method = "ward.D2") # hierarchical tree (Ward's criterion)
plot(hc, labels = kw$keyword)       # the dendrogram

The beauty of the dendrogram is that we make the decision on how many groups to keep after seeing it, simply by “cutting” it at a chosen height: a low cut leaves many small groups, a high cut a few large ones. I cut at three groups and compare with K-means:

kw$cluster_hc <- cutree(hc, k = 3)
table(kw$cluster_hc)
# 1 2 3
# 8 3 4

The partition is not identical to the K-means one (here the groups have 8, 3 and 4 keywords), and that is normal: the two methods optimise different criteria and on little data the differences show.
But the substance of the groupings — the high-cpc transactional block, the high-volume heads, the informational tail — remains recognisable in both. When two different methods converge on the same story, we can trust that story a little more.

Which method, and the traps

The choice between the two, in practice, follows a few simple rules. K-means is fast and efficient even on tens of thousands of keywords, but it wants to be told k and tends to build “spherical” groups of similar size. The hierarchical one does not ask for k in advance and gives the dendrogram — precious for seeing how the groups nest inside one another — but becomes heavy when the keywords are too many. The most common practice: explore with hierarchical on a sample to get a sense of the number of groups, then apply K-means to the whole set with the k thus identified.

A word of caution, the most important of all: clustering always finds groups, even when there are none. Even pure noise comes back dutifully split into k tidy clusters. The number of groups, the metrics chosen to describe the keywords, the standardisation: they are all our decisions, and each one changes the result. A grouping is not a truth discovered in the data, it is a working hypothesis that only makes sense if it survives the test of business common sense. If we cannot explain a cluster in words, it probably does not really exist.

There is then a question of dimensions. Here we used four metrics, but in real operations the variables describing a keyword can be many more, and with many dimensions distances lose meaning (everything tends to look equally far). It is exactly the problem that principal component analysis knows how to ease, compressing many metrics into a few components before passing the baton to clustering.

Try it yourself

The best way to understand clustering is to watch it change its answer as the choices change. Building on the code above:

1. Skip the standardisation: run K-means directly on kw[, c("volume","cpc","position","n_words")] without scale(). Do the groups all collapse onto volume? It is the practical demonstration of why we standardise.
2. Change k: try four or five groups and re-read the averages. Does the informational cluster split into sensible sub-themes or are you just cutting noise?
3. Change the merging criterion of the hierarchical method: method = "complete" or "average" instead of "ward.D2". Does the dendrogram change shape? And the groups cut at k=3?

A hint: always keep an eye on the per-cluster averages with aggregate(). It is there, and not in the code, that you decide whether a grouping is useful or just an elegant partition of nothing.

We grouped the keywords by how they behave — volume, cost, position.
But what matters most to an SEO is left out: their meaning. Two queries can have different metrics and mean the same thing, or similar metrics and opposite intents. Grouping by sense, and not only by numbers, means turning the very text of the queries into something measurable: it is the job of text mining, where words become vectors and similarity is computed on language. And that is where we will pick up next.

Further reading

If you want to go deeper into clustering — K-means, hierarchical methods, the choice of the number of groups and the pitfalls of interpretation — An Introduction to Statistical Learning by James, Witten, Hastie and Tibshirani devotes a lucid chapter to unsupervised learning, with R labs that retrace exactly the steps we saw here. It is the reference I recommend to anyone who wants to move from the toy example to clustering on real data.

We have already seen, talking about correlation, that position and CTR move together along a steep curve; and that the next step — using one variable to predict another — is the job of linear regression.
Here the two threads tie together: we turn that curve into an expected CTR and measure, page by page, how far each one deviates from it. It is the way to stop reading CTRs as absolute numbers and start reading them for what they really are: deviations from a norm.

What we will cover:

• Why a CTR, on its own, means nothing
• Modelling the CTR curve: three roads
• An example with Search Console data
• The other two roads: loess and nls at work
• Residuals: who earns less than they should
• Reading the deviations without fooling ourselves
• Try it yourself
• Further reading

Why a CTR, on its own, means nothing

There are industry tables telling us what the CTR of each position “should” be: the first around 25-30%, the second below half, and so on going down. They are useful as a general horizon, but to judge our pages they lead us astray: CTR depends on the type of query (a heavily clicked brand term or a cold informational search), on the sector, on how crowded the SERP is with ads and rich snippets.
The average CTR of “position 3” on an American e-commerce benchmark has almost nothing to say to our technical blog in Italian.

The way out is to stop comparing ourselves with an external table and build the reference curve on our own data.
We take all the pages, their average positions and their CTRs, and we trace the curve that describes the typical behaviour of CTR as position changes for the way our own site works. That curve becomes the yardstick: the expected CTR of a page is the value the curve assigns it, given its position.
The gap between the actual CTR and that expected value is the information we were after.

A CTR only makes sense next to the position that produced it: it is the deviation from the curve, not the absolute number, that tells us whether a page is working well or leaving clicks on the table.

Modelling the CTR curve: three roads

Building the curve means estimating a function that, given the position, returns the expected CTR. We already know the shape of that curve by eye: it starts high, plummets across the first positions and then flattens towards zero. A straight line does not describe it; we need something that curves.

There is, however, a detail that changes the whole way of reasoning, and it is the kind of deviation we care about.
We do not care that a page gets “two CTR points less” than expected: at the top of the SERP two points are crumbs, at the bottom they are a doubling. We care about the multiplicative deviation — “it earns half of what it should”, “it earns double”.
And a multiplicative deviation is best handled on a logarithmic scale, where a ratio becomes a difference.

The most natural shape for a curve of this kind is the power law, that is the idea that CTR is proportional to position raised to a negative exponent:

where \( a \) sets the general level and \( b \) (negative) governs how fast the descent is. The beauty arrives when we take the logarithm of both sides, which turns that curve into a straight line:

In other words: the logarithm of CTR is a linear function of the logarithm of position. And estimating a straight line is exactly what we know how to do with regression. From here, three roads to build the curve.

The first, and the one I recommend as the workhorse, is a linear regression on the logarithms — lm(log(ctr) ~ log(position)). It is in base R, it is interpretable (the slope \( b \) is the elasticity of CTR to position: by what percentage CTR drops for each percentage point of extra position), and its residuals are already on a logarithmic scale, hence multiplicative, exactly as we need.
It also extrapolates to rarely observed positions, and it can be weighted by impressions (weights = impression), so that pages with a handful of clicks do not skew the curve as much as those with tens of thousands of views. It is a pragmatic choice rather than the theoretically optimal weight (for a proportion the variance also depends on the CTR itself), but in practice it works very well.

The second is non-linear regression with nls, which estimates \( a \) and \( b \) directly on the natural scale of CTR without going through logarithms. It is an elegant refinement, but it must be primed with sensible starting values (which we fish out precisely from the log-log regression) and on messy data it may fail to converge. I keep it for when I need a clean parameter to put in a report, not as a starting point.

The third is local smoothing with loess, which imposes no shape on the curve and lets the data “draw it”. It is perfect for seeing the trend at a glance, but it wobbles on the tails (few pages in first position) and above all it does not extrapolate: outside the observed range it has nothing to say. It is an exploratory tool, not the model on which to base judgements.

So: we start from the log-log regression weighted by impressions as the working model, we compare it by eye with a loess to check we are not forcing the wrong shape, and we move to nls only if we need the explicit exponent. Let us see it at work.

An example with Search Console data

We start from an extract like the one anyone can download from Search Console: one row per page, with impressions, clicks and average position. In reality CTR is the ratio of clicks to impressions; here, since these are example data, we go the other way round — we set a plausible CTR and reconstruct the clicks.
I build the table in R with twelve example pages (with, on purpose, a couple of anomalous cases):

gsc <- data.frame(
  page       = c("/technical-seo-guide","/seo-audit-checklist",
                 "/keyword-research-guide","/campaign-roi-calculator",
                 "/google-analytics-tutorial","/statistics-glossary",
                 "/seo-tool-review","/link-building-guide",
                 "/competitor-analysis","/attribution-model",
                 "/ranking-report","/meta-tag-optimization"),
  impression = c(  9800, 5400, 12500, 2100, 7600, 1500,
                   8300, 4200,  6100,  900, 3300, 1800),
  position   = c(  1.3,  2.1,   3.4,  4.0,  4.6,  5.2,
                   2.8,  6.1,   7.0,  8.3,  9.1, 10.2)
)
# observed CTR (usually computed as click / impression)
gsc$ctr   <- c(0.232, 0.150, 0.034, 0.071, 0.066, 0.060,
               0.171, 0.048, 0.041, 0.012, 0.031, 0.028)
gsc$click <- round(gsc$impression * gsc$ctr)

I now estimate the expected-CTR curve with the regression on logarithms, weighting each page by its impressions:

fit <- lm(log(ctr) ~ log(position), data = gsc, weights = impression)
round(coef(fit), 3)
# (Intercept)  log(position)
#      -1.240          -1.088

The slope is −1.088: a value close to −1 describes an almost inversely proportional curve, where doubling the position (going, say, from 3 to 6) cuts the CTR roughly in half.
It is the same steep drop we had glimpsed when measuring correlation, but now written in a formula we can query: given a position number, it returns the typical CTR that position implies on our site.

The other two roads: loess and nls at work

We picked the log-log regression as our working model, but we had promised all three roads. It is worth seeing the other two genuinely at work on the same twelve data points, to understand what they add — and above all where they stumble.

Non-linear regression estimates \( a \) and \( b \) directly on the CTR scale, without going through logarithms. It must be primed with sensible starting values, which we fish out precisely from the log-log we just estimated: the intercept brought back to the natural scale with exp is our \( a \), the slope is our \( b \). I set it running in R:

start <- list(a = exp(coef(fit)[1]), b = coef(fit)[2])  # primed from the log-log
fit_nls <- nls(ctr ~ a * position^b, data = gsc,
               weights = impression, start = start)
round(coef(fit_nls), 3)
#      a       b
#  0.315  -1.073

It converges, and returns an exponent of −1.073, practically the same as the log-log, but with a bonus: now \( a \) and \( b \) are numbers you can read on the true CTR scale. An \( a \) of 0.315 says, in plain terms, that at the top of the SERP the typical CTR is around 31% — exactly the kind of clean parameter to put in a report.
The price we pay is fragility: without those starting values, or on noisier data, nls may fail to converge at all and hand us back only an error.

Local smoothing with loess does the opposite: it imposes no shape, it lets the data draw the curve. I estimate it and ask it for the expected CTR at a few positions, including the first:

fit_lo <- loess(ctr ~ position, data = gsc, span = 0.9)
round(predict(fit_lo, data.frame(position = c(1, 2, 3, 5))), 3)
# [1]    NA 0.173 0.111 0.029

And here is the limit in a single output: at position 1, loess returns NA. The minimum our data observe is 1.3, and outside that range loess refuses to commit — it does not extrapolate.
For a CTR curve this is a serious flaw: the first position, the one we care about most, is often the least populated, and that is exactly where the local method leaves us empty-handed.

Put on the same chart, the three roads tell the whole story at a glance:

In the middle, where pages abound, the three curves almost coincide: any method works when the data speak clearly. It is on the tails that they part ways — loess wobbles behind the few pages it finds and stops dead at the edge of the data, while lm and nls continue smoothly even where observations are scarce.

So: the weighted log-log stays the workhorse — interpretable, extrapolable, with residuals already on a multiplicative scale. nls refines it when we need a clean exponent to write down; loess is the critical eye that, before we trust the model, tells us whether we are forcing the wrong shape. Three different tools for a single craft: turning a position into an expected CTR.

Residuals: who earns less than they should

Having the curve means being able to compute, for each page, its expected CTR and compare it with the actual one. The comparison, as we said, must be made in terms of a ratio and not a difference: ratio = actual_ctr / expected_ctr. A value around 1 says the page earns as predicted; well below 1 that it is leaving clicks on the table; well above 1 that it captures more than its share.
I compute the expected CTR, the ratio, and flag the cases that truly deviate — but only if they have enough impressions to make their CTR reliable:

gsc$ctr_exp <- exp(predict(fit))           # back from the log scale to the natural one
gsc$ratio   <- gsc$ctr / gsc$ctr_exp

gsc$flag <- ifelse(gsc$ratio < 0.6 & gsc$impression >= 1000, "UNDER",
             ifelse(gsc$ratio > 1.4 & gsc$impression >= 1000, "OVER", "ok"))

gsc[order(gsc$ratio),
    c("page","position","impression","ctr","ctr_exp","ratio","flag")]

n.b. predict gives us the logarithm of the expected CTR, because that is the scale on which we estimated the model: exp brings it back to an actual CTR. Strictly speaking exp returns the median of the expected CTR, not the arithmetic mean (under log-normal errors the true mean is a touch higher), but for the relative ratios we care about the distinction is immaterial.
The output, sorted from the lowest ratio to the highest:

The case that jumps out is /keyword-research-guide: it sits in third position, where the curve would expect a CTR of 7.6%, and instead it gathers a meagre 3.4% — less than half of what it should, on twelve thousand five hundred impressions that make the figure rock solid.
It is a strong, immediately actionable hypothesis: in all likelihood the title and the meta description are not doing their job, and a rewrite could unlock clicks the position had already earned.

At the opposite end there is /seo-tool-review, which in second-to-third position earns almost double the expected. It is not a problem, it is a lesson: something in that snippet works beautifully — a magnetic title, a rich card, a perfect match with intent — and it is worth understanding what, to try to replicate it elsewhere. Residuals are not only there to find the sick ones: over-performances are the case studies from which to learn what, on our site, makes people click.

Reading the deviations without fooling ourselves

There is a detail in the table that is the heart of the whole matter, and that is easy to miss. /attribution-model has a ratio of 0.41 — even lower than /keyword-research-guide — yet we did not flag it. The reason is in the impressions column: nine hundred.
A CTR computed on so little data is almost pure noise, and next month it will drift back towards its mean regardless of anything we do. Flagging it as a “page to optimise” would send us chasing a ghost. Two pages with the same deviation, two opposite verdicts, and the only thing making the difference is the amount of data behind them.

A word of caution: the expected CTR is a conditional typical value — the median the curve associates with a position — not a law of nature. A page can “under-perform” for reasons that have nothing to do with the title: a brand query inflating competitors’ CTR, a featured snippet or a block of ads eating the clicks before the first organic result, a purely informational intent already satisfied by reading the snippet. And a CTR built on few impressions measures almost nothing: it will regress towards its mean on its own, as we saw talking about regression to the mean. A negative residual is a hypothesis to verify — “maybe the title earns little here” — not a verdict to execute.

It is also worth remembering that we estimated the curve on our own data, and that data influences it: a handful of very anomalous pages can tilt it just enough to shift the judgements on the others.
We can already see it in our table: ten pages out of twelve have a ratio above 1, but it is not that the site “over-performs” almost everywhere. It is that the single large negative deviation, /keyword-research-guide, weighs a great deal (twelve thousand five hundred impressions) and pulls the curve downwards, raising the ratio of all the others as a side effect. The “centre” of the cloud, in short, is not exactly 1, and it is better to read the ratios in relative terms — who sits well below and who well above the bulk of the group — rather than against the hard threshold of one.
This is why the impression weighting is precious, and why it pays to recompute the curve whenever the picture changes, instead of treating it as a constant carved in stone.

Try it yourself

The best way to internalise the mechanism is to get your hands on it. Building on the code above, there are three interesting directions to explore:

1. Aggregate by query instead of by page: the same page can appear on dozens of searches with different positions and CTRs, and often it is there — on the single query — that the missed opportunity hides. The curve and the residuals are built in the exact same way.
2. Drop the impression weighting — lm(log(ctr) ~ log(position)) without weights — and compare the exponent and the verdicts with the weighted version: how much do the conclusions shift once the pages with more data no longer have a louder voice? It is the way to feel how much the weighting matters, instead of taking it on faith.
3. Change the impression threshold below which you do not trust the CTR: going from 1000 to 3000, which pages leave the radar? The right number does not exist in the abstract, it depends on how much traffic your site moves.

A hint: the structure never changes — you estimate the curve, you predict the expected value, you look at the ratio. It is by playing with the threshold and the level of aggregation that you really understand how much of what we call an “under-performing page” is signal and how much is, simply, noise.

Spotting a page that earns less than expected for its position is a close cousin of another problem every analyst knows: spotting a day that earns less than expected over time, a drop or a spike in traffic that does not square with the usual trend.
It is the same reasoning on residuals — observed value against expected value — moved from the space of positions to the axis of time, where the expected value is provided by the historical trend of the series. From there springs anomaly detection: telling signal from noise when the numbers move over time, and the next step of our path.

Further reading

If you want to go deeper into regression, logarithmic transformations and the reading of residuals — the very backbone of the model we built here — and then push beyond the power law towards local methods like loess, An Introduction to Statistical Learning by James, Witten, Hastie and Tibshirani is the book I recommend: it covers both the “why” of the logarithms and the “how” of interpreting coefficients, with hands-on labs in R, always starting from applied problems.

The trouble is that queries are many and always new, and classifying them by hand does not scale. We need a method that learns from a handful of labelled examples and then copes on its own with queries it has never seen. The algorithm that does this with almost disarming elegance is Naive Bayes, and — as the name hints — it starts once again from the Bayes’ theorem that has accompanied us throughout this path.

What we will cover:

• From the theorem to the classifier
• Training on labelled queries
• Classifying new queries
• The limits of “naive”
• Try it yourself
• Further reading

From the theorem to the classifier

Let us pick up the thread. Bayes’ theorem tells us how to update the probability of a hypothesis in the light of the observed data. Here the hypothesis is “this query belongs to the informational class” (or transactional), and the data are the words that make up the query. We want, in other words, the probability of a class given the text: in symbols, P(class | words).

Computing it directly would be a nightmare, because the possible combinations of words are boundless. Bayes lets us flip the problem around: instead of asking how probable the class is given the words, we ask how probable those words are given the class — a question the training data can answer. The rule, first in words and then in a formula, is that the probability of a class given a query is proportional to the prior probability of the class times the probability of observing those words if the class were that one:

Let us unpack the symbols. P(class) is the prior: how frequent that class is to begin with, before looking at the text (if half of the example queries are informational, the prior is 0.5). P(word | class) is how often that word appears in the queries of that class. The symbol ∏ is simply the product: we multiply the contributions of all the words in the query. The sign ∝ (“proportional to”) reminds us that we are dropping a constant denominator, identical across classes: since in the end we only care which class wins, we can ignore it with no harm.

And here it is, the spot where “naive” hides. Multiplying the probabilities of the individual words as if they were independent of one another amounts to assuming that, given the class, the presence of one word says nothing about the presence of the others. It is a plainly false assumption about real language — “credit” and “card” certainly do not appear at random independently of each other — and it is precisely this naivety that gives the algorithm its name. The surprising thing, as we shall see, is that despite starting from so crude an assumption the method works beautifully in practice.

Training on labelled queries

Training a Naive Bayes means one thing only: counting. For each class we count how many times each word appears in the example queries, and from those counts we derive the P(word | class). No optimisation, no iterations: we scan the data once and fill a table of frequencies.

Let us start from a small set of hand-labelled queries, five informational and five transactional. I train the classifier in R like this:

train <- list(
  info  = c("how to bake a cake", "what is statistics", "seo guide for beginners",
            "free r tutorial", "how bayesian works"),
  trans = c("buy shoes online", "iphone price deal", "best cheap hosting",
            "purchase seo course", "gym membership discount")
)
tok <- function(s) unlist(strsplit(tolower(s), "\\s+"))
vocab <- unique(unlist(lapply(unlist(train), tok)))
counts <- lapply(train, function(docs) {
  w <- table(factor(unlist(lapply(docs, tok)), levels = vocab)); w + 1
})
tots <- sapply(counts, sum); V <- length(vocab)
prior <- sapply(train, length) / length(unlist(train))   # 0.5 / 0.5

Let us see what happens line by line. The tok function splits each query into lowercase words (a spartan tokenisation, but enough). vocab is the vocabulary, the list of all distinct words seen in training. For each class, table(factor(...)) counts the occurrences of each vocabulary word; tots is the total of the counts per class, and prior here is 0.5 and 0.5 because the two classes have the same number of examples.

There is a detail in that w + 1 worth pausing on, because it is the trick that holds the whole edifice up. If a word never appears in the queries of a class, its count is zero, and with it the entire product of probabilities would collapse to zero: a single unknown word would be enough to drive the class probability to zero, wiping out the contribution of all the others. It is the classic case where “multiplying by zero” ruins the party. The fix is called Laplace smoothing: we add 1 to the count of every word, in every class, before computing the proportions. No word any longer has exactly zero probability, only a very small one.

The price of smoothing is that the per-class totals gain the number of vocabulary words (the units added one per word): that is why tots is not the raw token counts of the two classes but the smoothed totals. With these numbers, for example, the word “buy” (present once among the transactional queries, never among the informational ones) ends up clearly more probable under trans than under info — and it is exactly this imbalance that pushes a query toward the right intent. A neutral word like “seo”, appearing once on each side, stays roughly balanced between the two classes and does not move the needle.

Classifying new queries

With the frequency table ready, classifying a new query is genuinely child’s play: we tokenise it, sum the logarithms of the probabilities of each word (summing logarithms instead of multiplying probabilities avoids the product of many tiny numbers going into numerical underflow, but the result is the same), add the logarithm of the prior, and pick the class with the highest score. I compute it in R:

classify <- function(query) {
  w <- tok(query)
  logp <- log(prior)
  for (cl in names(train)) {
    p <- counts[[cl]][w]; p[is.na(p)] <- 1            # out-of-vocab word: neutral weight
    logp[cl] <- logp[cl] + sum(log(as.numeric(p) / tots[cl]))
  }
  names(which.max(logp))
}
cat("'buy seo course discount' ->", classify("buy seo course discount"), "\n")
cat("'how to learn statistics'   ->", classify("how to learn statistics"), "\n")

The output is clear-cut:

'buy seo course discount' -> trans
'how to learn statistics'   -> info

As we can see, the classifier assigns the first query to the transactional intent and the second to the informational one, exactly as a human would have. It is worth noting how it gets there. In the first query “buy” and “discount” pull decisively toward trans, “seo” stays neutral, and not even “course” — which in training appeared in the transactional “purchase seo course” — rows against it: the verdict is solid. In the second, “statistics” is a markedly informational word, and “learn”, though out of vocabulary, does no harm thanks to that p[is.na(p)] <- 1 which assigns never-seen words a neutral weight, identical for both classes: having nothing to say, it simply does not vote.

A handful of example queries per side is very little, and yet the mechanism is already all here. In a real case it is enough to replace the handfuls of labelled queries with a few hundred or thousand queries pulled from Search Console and annotated for intent, and the same code — unchanged in structure — becomes an intent classifier you can actually use.

The limits of “naive”

Before diving in, though, it is worth knowing where the method shows its seams, because its simplicity has a flip side.

The most obvious limit is precisely the independence assumption it takes its name from. By treating each word as detached from the others, Naive Bayes ignores order and context entirely: to it “cheap running shoes” and “the economics of running shoes” are the same bag of words. In intent classification this matters little, but in subtler tasks it can mislead. Then there is the question of out-of-vocabulary words: a query made only of terms never seen in training would be decided by the prior alone, that is by a pure coin toss — the sign that the example dataset is too thin and needs widening.

A note of caution that holds for any classifier, and all the more for one trained on little data: the model learns exactly what we show it, biases included. If the transactional example queries all contain the word “buy”, the classifier will associate purchase intent with that term and will struggle on an equally transactional but lexically different “add to cart”. The quality and representativeness of the labelled data matter more than the sophistication of the algorithm: a Naive Bayes fed with varied, balanced examples beats a sophisticated model trained badly.

That said, Naive Bayes remains a genuinely valuable tool to have in the box: it is extremely fast to train, needs little data to get going, is interpretable (we can always go and look at which words drove the decision) and in text classification it holds its own against far more complex models. It is often the baseline to beat before bringing in heavier artillery — and this is where the door opens onto machine learning proper, where classification is done with trees, regressions and networks that drop the independence assumption in exchange for more power (and less transparency).

Try it yourself

The code above is a perfect playground for building intuition. Search queries are not only informational or transactional: at least a third family is missing, the navigational one (someone searching “facebook login” or “gironi blog” just wants to reach a specific site). Here are a few changes to try:

1. Add a nav class to train with four or five navigational queries (“facebook login”, “youtube”, “amazon sign in”, “gmail inbox”), then retrain: the prior will no longer be 0.5 but roughly a third per class. How do the classifications of the earlier queries change?
2. Feed the classifier an ambiguous query like “iphone review” (informational? transactional?) and see which way it leans. Did the verdict make sense, given the words in training?
3. Remove the smoothing (replace w + 1 with w) and try classifying a query with a rare word: what happens to the score when a count is zero? It is the quickest way to see with your own eyes why Laplace is needed.

The nice part is that the structure of the code never changes: adding a class just means lengthening the train list, and everything else — vocabulary, counts, prior, classification rule — adapts on its own.

With this we close the Bayesian thread we have unspooled article after article: from estimating a conversion rate to comparing variants, from the adaptive allocation of traffic to this last leap, from deciding to classifying. The same theorem, reworn each time in a different guise, has proved a surprisingly sturdy common thread. From here the road forks toward machine learning in the full sense — decision trees, logistic regression, neural networks — where the methods give up the most comfortable assumptions in exchange for power, and where Bayes stays in the background as the grammar by which, in the end, we always learn from data.

Further reading

If you want to move from Naive Bayes to machine learning proper while keeping your feet on the ground (and R at hand), An Introduction to Statistical Learning by James, Witten, Hastie and Tibshirani is the book I recommend. It is the most accessible doorway into applied machine learning: it explains classification, trees and regression with the right rigour but without intimidating mathematics, and every chapter has R labs you can redo step by step. The second edition devotes space to Naive Bayes itself, so the jump from this article to the rest of the path is a natural one.

This article is part of the “The Bayesian Approach” path, a guided route through the articles on Bayesian statistics and inference for SEO.

There is a way to cut this bill, and it is called a multi-armed bandit. The idea is to shift traffic adaptively toward the variant that is winning while the test is still running, instead of waiting for the final verdict. In this article we build one with one of the most elegant and practical algorithms, Thompson sampling, which is the natural continuation of the Bayesian reasoning we have followed so far.

What we will cover:

• The exploration versus exploitation dilemma
• Thompson sampling: letting the posterior decide
• How much we really gain: the regret avoided
• When a bandit makes sense, and when it does not
• Try it yourself
• Further reading

Exploration versus exploitation

The name comes from slot machines: a one-armed bandit is the casino machine, and let us imagine we have several of them in front of us, each with an unknown win probability different from the others. We have a limited number of tokens. At each play we must choose which arm to pull. What is the strategy that maximises the total winnings?

The translation for anyone doing SEO or marketing is immediate: the arms are the variants (three versions of a title tag, of a call to action, of a landing page), the tokens are the visitors, the winnings are the conversions. Each visitor must be assigned to a variant, and we want to maximise total conversions across the whole experiment.

Here arises the underlying tension, the one that makes the problem interesting. On one hand we would like to exploit the variant that so far seems the best, to bank as many conversions as possible. On the other we must keep exploring the others too, because “so far it seems the best” rests on little data and might be a wrong impression.
It is a delicate balance: too much exploration and we waste traffic on mediocre variants; too much exploitation and we risk crowning the wrong winner on the basis of an early stroke of luck. The exploration-exploitation dilemma is the heart of every multi-armed bandit problem: each choice is at once an opportunity for gain and an opportunity for learning, and the two pull in opposite directions.

Thompson sampling: letting the posterior decide

The classic A/B test solves the dilemma in the crudest possible way: it explores and nothing more, in equal parts, until the end. Half the traffic to A, half to B, no adaptation. Thompson sampling solves it in a far cleverer, and almost surprisingly simple, way.

Let us pick up the Bayesian thread. For each variant we keep a posterior on its conversion rate: as we saw when estimating the conversion rate, starting from a Beta prior and observing binary outcomes (conversion yes/no), the posterior of each arm is again a Beta distribution, updated with every visitor. At the start, when we know nothing, each arm begins from a non-informative Beta(1, 1) prior.

Thompson’s rule, in words before formulas, is this: instead of asking “which is the variant with the highest mean so far?”, at each visitor we sample a rate at random from the posterior of each arm, and play the arm that produced the highest sample. It is a way of choosing “in proportion to the probability of being the best”: a variant we are very uncertain about can still win the draw now and then (and that is how it keeps being explored), but as the data accumulate its samples concentrate and, if it really is worse, it almost stops being chosen on its own.

The elegance lies precisely here: there is no exploration parameter to tune by hand. The uncertainty of the posterior is the engine of exploration. The more uncertain an arm, the more its samples are spread out, the more often it happens to win the draw and get tried; the more certain it becomes, the more its samples tighten around the true value and the arm gets played (or avoided) decisively.

I simulate in R the whole process on three variants with true rates of 5%, 7% and 9% (which the algorithm of course does not know), over 5000 visitors:

set.seed(7)
true_rates <- c(0.05, 0.07, 0.09)   # 3 variants, the third is the best
K <- length(true_rates); N <- 5000
alpha <- rep(1, K); beta_ <- rep(1, K)   # Beta(1,1) prior per arm
pulls <- rep(0, K); rewards <- rep(0, K)
for (t in 1:N) {
  theta <- rbeta(K, alpha, beta_)        # sample a rate per arm
  arm <- which.max(theta)                 # play the best sampled arm
  r <- rbinom(1, 1, true_rates[arm])      # outcome (conv yes/no)
  alpha[arm] <- alpha[arm] + r; beta_[arm] <- beta_[arm] + (1 - r)
  pulls[arm] <- pulls[arm] + 1; rewards[arm] <- rewards[arm] + r
}

The loop is all there is. At each iteration we sample a theta per arm (rbeta), choose the maximum (which.max), simulate the outcome (rbinom) and update the parameters of the arm played: a conversion increases its alpha by one, a non-conversion its beta_. No other logic, no threshold to calibrate.

How much we gain: the regret avoided

Let us see where the simulation took us. The first thing to look at is how the 5000 visitors were distributed across the three arms:

cat("pulls per arm:", pulls, "\n")
cat("bandit total conv:", sum(rewards), "\n")

Output: pulls per arm = 359, 278, 4363; bandit total conv = 424.

Of the 5000 visitors, a full 4363 ended up on the best variant, and only a little over 600 in total on the two worse ones. The algorithm, without our having told it anything about the true rates, worked out on its own which arm to reward and steered the vast majority of the traffic there. It is exactly the exploitation we wanted, reached through just enough early exploration to tell the arms apart.

Now the comparison that reveals the size of the advantage. A classic A/B test would have split the 5000 visitors equally across the three variants, roughly 1667 each, for the whole duration. The expected conversions in that scenario are simply the mean of the three rates times the number of visitors:

exp_ab <- sum(true_rates) / K * N
cat("expected conv equal-split A/B:", round(exp_ab), "\n")
cat("regret avoided (approx):", round(sum(rewards) - exp_ab), "conv\n")

Output: expected conv equal-split A/B = 350; regret avoided (approx) = 74 conv.

An equal-split A/B would have taken home about 350 conversions; the bandit collected 424. The difference, about 74 more conversions on the very same traffic, is what is technically called the regret avoided: the regret, that is the conversions lost by playing the wrong arms, that the adaptive allocation saved us. In plainer terms, for the same number of visitors the bandit converted over 20% more, simply by not insisting on the weak variants.
Note that we did not have to sacrifice anything to get it: at the end of the experiment we still know the winner (indeed we know it with great confidence, given the 4363 data points gathered on it), and in the meantime we converted more. This is the structural advantage of the bandit over the fixed-sample test.

When it makes sense (and when it does not)

That said, a bandit is not the right answer to every question, and it is important to understand where it shines and where instead a traditional A/B test remains preferable.

The bandit is at its best when traffic is continuous and long-running (an always-on page, a campaign running for months) and when the variants to compare are many: there the adaptive allocation pays off, because there is time and volume to shift the traffic and plenty of weak variants to drop quickly. It is ideal for ongoing optimisations where the goal is to maximise conversions along the whole journey, not to take a clean statistical snapshot at a given moment.

When instead the goal is precisely that snapshot — a precise, unbiased estimate of how much a variant is better, perhaps to defend in front of a client or to use for a strategic decision — the fixed-sample test remains more suitable: precisely because it explores in equal parts, it gathers balanced data on all variants and produces sharper estimates of the effect. The bandit, by concentrating early on the winner, gathers little data on the losers and therefore estimates by how much they are worse less well.

A note of caution: the bandit assumes that the conversion rates stay stable over time. But the real world is often non-stationary — seasonality, shifts in audience, a promotion that kicks in. If the best variant changes after the algorithm has already concentrated on another, a naive Thompson sampling like the one shown here struggles to notice, because it has stopped exploring the alternatives. In changing contexts we need variants that “forget” old data (for example by gradually discounting past observations), otherwise we risk staying anchored to a winner that no longer is one.

Try it yourself

Simulation is a perfect playground for building intuition. Taking the code above, try changing the starting numbers and observe how the pulls redistribute:

1. Bring the true rates closer, for example true_rates <- c(0.07, 0.08, 0.09): with more similar variants, how much more traffic is needed before the best arm breaks away? Does the concentration of pulls stay this sharp?
2. Cut the traffic drastically (N <- 500): with few visitors does the bandit still have time to spot the winner, or does exploration eat up the whole budget?
3. Add variants: move to four or five arms. With more alternatives to discard, does the advantage in regret avoided over the equal-split A/B grow or shrink?

Hint: the structure of the loop never changes, you just modify true_rates and N. It is precisely by watching how the pulls vector tips (or fails to tip) as a function of the distance between the rates that you truly grasp what Thompson sampling does under the hood.

So far we have used Bayes to decide between variants: estimate a rate, compare two, allocate traffic adaptively. But the same machinery — a prior, some data, a posterior — also serves a different and very common task: classifying, that is assigning to each new observation the most probable label given its features. It is the leap from deciding to classifying, and the algorithm that does it with disarming elegance, once again starting from Bayes’ theorem, is Naive Bayes: the subject of the next article.

Further reading

If you want to explore bandits applied to website optimisation, Bandit Algorithms for Website Optimization by John Myles White is the book I recommend. It is a slim, avowedly practical volume that compares A/B tests and bandit algorithms (epsilon-greedy, softmax, UCB) precisely from the standpoint of someone optimising pages and conversions, with the code at hand. It is the ideal starting point for anyone wanting to move from the simulation we have seen to a bandit that actually runs on their own site.

This article is part of the “The Bayesian Approach” path, a guided route through the articles on Bayesian statistics and inference for SEO.

The Bayesian approach answers both questions natively. In this article we apply it to the comparison of two variants, picking up the thread we left hanging when we estimated the conversion rate of a single variant.

It is the question behind every conversion test with an SEO angle: two versions of the same landing page competing for the same organic traffic, or two title-and-meta wordings judged by their actual versus expected CTR in the SERP. Wherever there are successes over trials — clicks over impressions, sign-ups over visits, conversions over sessions — the reasoning that follows is the same.

What we will cover:

• Two posteriors instead of one: a distribution per variant
• What is the probability that B really wins
• By how much it is better: the distribution of the difference
• When to stop: expected loss and the peeking problem
• Try it yourself
• Further reading

Two posteriors instead of one

When we estimated the conversion rate of a single variant, we saw that — starting from a Beta prior and observing binomial data — the posterior is again a Beta distribution. The updating rule was simple arithmetic: if the prior is Beta(α, β) and we observe \( k \) conversions out of \( n \) sessions, the posterior is:

Put in words: we add the observed conversions to the first parameter and the non-conversions to the second, nothing more.

In an A/B test we do not have a single proportion, we have two: one for the control (A) and one for the treatment (B). The mechanism, however, is identical: we build one posterior per variant, independently, applying the same rule twice.

Here is a quick example. We tested two versions of a landing page that receives organic traffic, assigning visitors at random:

• Variant A (control): 90 conversions out of 1000 sessions → raw rate 9.0%
• Variant B (treatment): 120 conversions out of 1000 sessions → raw rate 12.0%

We start from a non-informative Beta(1, 1) prior for both — “we know nothing, before the data every rate is equally plausible”. Applying the rule, the posterior of A is Beta(91, 911) and that of B is Beta(121, 881).

I build the two posteriors in R, sampling them by simulation:

set.seed(42)
# Variant A: 90 conv / 1000 ; Variant B: 120 conv / 1000
cA <- 90; nA <- 1000; cB <- 120; nB <- 1000

# Posteriors with uniform Beta(1,1) prior
postA <- rbeta(1e5, 1 + cA, 1 + nA - cA)   # Beta(91, 911)
postB <- rbeta(1e5, 1 + cB, 1 + nB - cB)   # Beta(121, 881)

Now we hold two distributions, not two numbers. And this is precisely the point: instead of comparing 9.0% against 12.0% as if they were fixed values, we compare the whole uncertainty surrounding them. The operational questions become operations on these distributions.

It helps to keep the picture in mind: two ridges of probability side by side along the conversion-rate axis. The further B’s ridge sits to the right of A’s, and the less the two overlap, the more likely it becomes that B is genuinely the better variant.

What is the probability that B wins?

The first question — the one the p-value never answers directly — is the probability that B is genuinely better than A.
With the posteriors in hand, the calculation is almost trivial: we compare B’s samples with A’s, pair by pair, and count in what fraction of cases B exceeds A. That fraction is the probability we are after.

I compute in R the probability that B beats A:

cat("P(B>A) =", round(mean(postB > postA), 3), "\n")

Output: P(B>A) = 0.985.

There is a 98.5% probability that variant B converts better than variant A.
Notice the change of register compared to the frequentist version. We are not saying “the observed difference is unlikely under the null hypothesis”: we are saying, directly, that given the evidence collected it is almost certain that B is the better variant. This is exactly the statement we would want to base a decision on — and the Bayesian approach hands it over without circumlocutions.

By how much it is better: the distribution of the difference

Knowing that B wins with 98.5% probability is not enough to decide. There is almost surely an improvement, but if it were two tenths of a percentage point, perhaps it would not be worth shipping the new page. The next question is therefore: by how much is it better?

So far we have kept A and B separate, one posterior each. But the object we actually care about is neither of them on its own: it is the distance between them. The answer therefore lives in the distribution of the difference between the two posteriors. We subtract, sample by sample, A’s rate from B’s: we obtain a new distribution, that of the uplift. From it we read both the typical value (the mean) and a credible interval that quantifies its uncertainty.

I compute in R the difference and its 95% interval:

diff <- postB - postA
cat("mean uplift (pct points) =", round(mean(diff)*100, 2), "\n")
cat("95% CI of difference =", round(quantile(diff, c(.025,.975))*100, 2), "\n")

Output: mean uplift = 3.00 pct points, 95% CI = [0.31, 5.68].

The expected gain is about 3 percentage points of conversion, with a 95% credible interval running from 0.31 to 5.68 points.
Here too the meaning is direct, not an abstract property of the procedure: there is a 95% probability that the true improvement of B over A lies between 0.3 and 5.7 percentage points. The interval does not touch zero, which confirms — consistently with the earlier 98.5% — that B is almost certainly superior. But the valuable figure is the width: the improvement could be modest (half a point) or robust (over five points), and this spread is information the operational decision must keep in mind.

When to stop: expected loss and the peeking problem

In the article on classic A/B testing we devoted space to one of the most insidious errors: peeking, that is, glancing at the interim data and stopping as soon as the difference looks significant. In the frequentist framework this inflates the false-positive rate, because each glance is effectively a new test on the same null hypothesis, and repeated tests multiply the chances of being wrong.

The Bayesian approach changes the nature of the problem. Here we are not repeating a test on a null hypothesis: we are updating a belief. Today’s posterior becomes tomorrow’s prior, and looking at the data as it arrives does not “consume” an error budget in the same way. This does not mean we can stop on a whim: we still need a stopping rule declared in advance. And the natural Bayesian rule is not “stop when P(B>A) is high”, but is based on expected loss.

The idea is this: if we choose B but A were in fact the better variant, we are wrong, and the size of the error is how much A beats B in those cases. The expected loss of choosing B is the average of this “regret” over all the residual uncertainty. In plain words: by how much, on average, we would regret having chosen B if we were wrong.

I compute in R the expected loss of choosing B:

# Expected loss of choosing B: average loss if A were actually better
loss_B <- mean(pmax(postA - postB, 0))
cat("expected loss choosing B =", round(loss_B*100, 3), "pct points\n")

Output: expected loss choosing B = 0.007 pct points.

The expected loss of choosing B is a mere 0.007 percentage points: negligible. In plainer terms, even in the unlucky scenario where we were wrong, the average damage would be tiny. We then set a tolerance threshold before starting — for example “I stop when the expected loss drops below 0.01 points” — and let the test run until we reach it.

And here is where we see why the expected loss, and not “P(B>A) is high”, is the right criterion. If we recompute at every step of the test, as traffic comes in, the two quantities do not trip at the same moment: the probability that B wins crosses 95% well before the expected loss becomes truly negligible.

Try it yourself

A lead generation website tests two variants of its contact form. On variant A we observe 45 conversions out of 600 sessions; on B, 52 conversions out of 600 sessions.

1. Build the two posteriors with a non-informative Beta(1, 1) prior: postA <- rbeta(1e5, 1 + 45, 1 + 555) and the analogue for B.
2. Compute P(B>A): is B better with a probability high enough to convince you?
3. Compute the mean uplift and 95% interval of the difference: does the interval touch zero?
4. Compute the expected loss of choosing B. With these numbers (closer to each other than in the case above), how does it change compared to the article’s example?

Hint: the structure of the code is identical to the one we used. Only the starting counts change — and the result, far less clear-cut, is precisely why the interval and the expected loss matter more than a plain “B won”.

So far we have compared two variants at a fixed sample size: we collect the data, compute, decide. And yet we have just watched that probability recompute itself at every step, as traffic comes in — and if one variant is turning out clearly better, why keep sending half the visitors to the worse one? We can do better: use that real-time probability to allocate traffic adaptively, shifting it toward the winning variant while the test is still running. It is the leap from the test to the bandit, the subject of the next article.

Further reading

If you want to explore Bayesian A/B testing with a practical, code-oriented angle, Bayesian Methods for Hackers by Cameron Davidson-Pilon is the book I recommend. It tackles Bayesian reasoning starting from programming rather than formal mathematics, and devotes an explicit chapter to the Bayesian comparison of variants — probability that B wins, distribution of the difference, expected loss. It is written for those who learn better by reading code than proofs.

This article is part of the “The Bayesian Approach” path, a guided route through the articles on Bayesian statistics and inference for SEO.

The problem always starts the same way. A small e-commerce store has collected 23 conversions out of 412 sessions. The raw rate is 23/412 ≈ 5.6%. A seemingly precise number. But how much do we trust it? We could be looking at the true 3% or the true 9% — with that sample, we simply do not know. The point estimate “5.6%” says nothing about its own uncertainty.

What we will cover:

• The Beta-Binomial model: why Beta is the natural distribution for a conversion rate
• Non-informative prior: letting the data speak
• Informative prior: using historical data without cheating
• Today’s posterior is tomorrow’s prior
• Try it yourself
• Further reading

From single click to rate: the Beta-Binomial model

Each session is a binary event: the user converts or does not.
With \( n \) sessions and \( k \) conversions, the generative mechanism is binomial. The parameter we want to estimate — the true conversion rate \( \theta \) — is a proportion: a value between 0 and 1.

When the prior on \( \theta \) is a Beta distribution and the data are binomial, something very convenient happens: the posterior is also a Beta distribution. This is called a conjugate prior, and it means Bayesian updating reduces to a simple arithmetic operation on the parameters.

The updating rule is: if the prior is Beta(α, β), after observing \( k \) conversions out of \( n \) sessions the posterior is:

In plain words: we add the observed conversions to α and the observed failures to β. The prior Beta(α, β) encodes in α the “conversions already seen” (or an equivalent belief) and in β the “non-conversions”. Each new observation updates both counters.

Non-informative prior: letting the data speak

The most neutral prior possible for a proportion is the uniform distribution on [0, 1], which corresponds to Beta(1, 1): all values of the rate are considered equally plausible before seeing any data.

Our case: 23 conversions out of 412 sessions (389 non-conversions).

We calculate the posterior in R:

# Observed data
conv <- 23; sess <- 412; nonconv <- sess - conv

# NON-INFORMATIVE prior: uniform Beta(1,1)
a0 <- 1; b0 <- 1
a1 <- a0 + conv; b1 <- b0 + nonconv      # posterior Beta(24, 390)

cat("Non-informative: mean =", round(a1/(a1+b1), 4), "\n")
cat("  95% CI =", round(qbeta(c(.025,.975), a1, b1), 4), "\n")

Output: mean = 0.058, 95% CI = [0.0376, 0.0824].

The posterior Beta(24, 390) has mean 5.8% and a 95% credible interval between 3.8% and 8.2%.
The Bayesian credible interval is not an abstract statistical exercise: it means directly that there is a 95% probability that the true conversion rate lies between 3.8% and 8.2%. Not a frequency over infinite repetitions — a direct probability statement on the parameter.

With 412 sessions, the uncertainty is still appreciable: almost 5 percentage points of width. The point estimate 5.6% was misleading in its precision.

A note of caution: the Bayesian credible interval and the frequentist confidence interval have similar numbers but profoundly different meanings. The frequentist 95% is a property of the procedure (“repeating the experiment 100 times, 95 intervals would contain the true parameter”); the Bayesian 95% is a direct statement about the parameter in the specific case we are analysing.

Informative prior: using historical data without cheating

The non-informative prior is the honest starting point when we know nothing. But often we do know something: years of campaigns, sector history, category benchmarks.
Our e-commerce has four seasons of history with an average conversion rate around 4%. How do we translate this knowledge into a prior?

The Beta(8, 192) distribution has mean exactly 8/(8+192) ≈ 4% and — because α+β = 200 — a concentration equivalent to “trusting” our data as much as 200 fictitious historical sessions. It is not an arbitrary number: it is a declared and verifiable choice.

We calculate the informative posterior in R:

# INFORMATIVE prior from history: mean ~4% -> Beta(8, 192)
a0i <- 8; b0i <- 192
a1i <- a0i + conv; b1i <- b0i + nonconv  # posterior Beta(31, 581)

cat("Informative: mean =", round(a1i/(a1i+b1i), 4), "\n")
cat("  95% CI =", round(qbeta(c(.025,.975), a1i, b1i), 4), "\n")

Output: mean = 0.0507, 95% CI = [0.0347, 0.0694].

The informative posterior Beta(31, 581) gives mean 5.1% and 95% credible interval between 3.5% and 6.9%.
Two things to notice. First: the mean drops slightly from 5.8% to 5.1% — the prior “pulls” the estimate toward the historical 4%. Second: the interval narrows (from 4.4 to 3.4 percentage points) — the historical data acts as additional information, so uncertainty decreases.

With limited data, the informative prior helps: it adds information where data alone is insufficient. With many data points — thousands of sessions — the prior gets overwhelmed by the data and the difference between informative and non-informative priors becomes negligible. This is a fundamental feature of Bayesian inference: the prior matters when data is scarce; data always wins in the end.

Today’s posterior is tomorrow’s prior

The most practical elegance of the Bayesian approach is sequential updating. After one month, new data arrives: 15 conversions from 300 additional sessions. We do not need to start over — the posterior we just calculated becomes the new prior.

We update in R:

# New data: 15 conversions from 300 additional sessions
conv2 <- 15; sess2 <- 300
a2 <- a1i + conv2; b2 <- b1i + (sess2 - conv2)   # Beta(46, 866)

cat("After update: mean =", round(a2/(a2+b2), 4), "\n")
cat("  95% CI =", round(qbeta(c(.025,.975), a2, b2), 4), "\n")

Output: mean = 0.0504, 95% CI = [0.0372, 0.0655].

Three stages compared:

The credible interval has narrowed at each stage: 5.3 percentage points for the pure prior, 3.4 after the first month, 2.8 after the update. The mean remained stable: the new data confirms the previous estimate instead of shifting it, and uncertainty decreases as expected.
This is the real operational advantage: there is no need to wait for a “large enough” sample to accumulate before making any estimate. We start from an uncertain estimate and refine it progressively, with each new data point.

Try it yourself

A lead generation website has a historical conversion rate around 2%. After an optimisation campaign, 8 conversions are observed from 150 sessions.

1. Build an informative prior reflecting the historical 2%: try Beta(4, 196) — it has mean exactly 2%.
2. Calculate the posterior after 8 conversions from 150 sessions.
3. Calculate the 95% credible interval.
4. Now try a non-informative Beta(1, 1) prior: does the posterior change much? Why?

Hint: the formula is always the same — qbeta(c(.025, .975), a0 + conv, b0 + nonconv). The only thing that changes is the starting point.

What we have built so far is the estimation of a rate for a single variant. The next step is comparing two variants — a control page and a modified page — and calculating the Bayesian probability that one beats the other. That is exactly what we will do in the next article: Bayesian A/B testing.

Further reading

If you want to understand how Bayesian reasoning enters practical decisions — from market forecasting to uncertainty estimation in real data — The Signal and the Noise by Nate Silver is the book I recommend. Silver devotes explicit chapters to Bayesian updating, showing it in concrete contexts (weather forecasting, politics, sports) that make the idea of “updating beliefs with new data” immediately intuitive. Rigorous but written like a story, it is a rare kind of book that leaves you thinking differently about uncertainty.

This article is part of the “The Bayesian Approach” path, a guided route through the articles on Bayesian statistics and inference for SEO.