Multiple Comparisons: Why Out of 200 Queries Some Always «Improve» by Chance

In 2009 a group of neuroscientists put a salmon in a functional MRI scanner, showed it a series of photographs of people in social situations and “asked” it what emotions they were feeling. The analysis found a small area of the brain with statistically significant activity. The catch is that the salmon was dead.
The study was a very serious joke (it won an Ig Nobel in 2012), and its target was not MRI: it was a widespread methodological error. With enough tests, something “significant” turns up anyway. The scanner had compared tens of thousands of small brain volumes, one by one, each at the 5% threshold.

In our line of work the dead salmon has a familiar name: Search Console. We compare this month’s CTR with last month’s across 200 queries, find a dozen “significant” changes and write in the report which queries improved. But how many of those twelve really changed? This is the problem of multiple comparisons, and we have already seen its outline when talking about SEO split testing and peeking.

In short: when many statistical tests are run together, the 5% threshold no longer holds for the whole set: with 10 independent tests the probability of at least one false positive rises to 40%, with 100 to 99%. Multiple comparisons are corrected in two ways. The Bonferroni correction (or its improved version, the Holm method) controls the probability of making even a single false positive, by dividing the threshold by the number of tests: it is cautious, but loses many real effects. The Benjamini-Hochberg method instead controls the share of false positives among the discoveries (the false discovery rate): it is less strict and better suited to sifting through many pages or queries. In R we use p.adjust().

What we will cover:

• Why do more tests mean more false alarms?
• An example: 200 queries in Search Console
• The Bonferroni correction (and Holm’s)
• Family-wise error or share of false discoveries?
• The Benjamini-Hochberg method
• Which correction should we choose?
• Where multiple comparisons hide in SEO
• Frequently asked questions
• Try it yourself
• Further reading

---

Why do more tests mean more false alarms?

Let’s recall what the 5% threshold promises in a hypothesis test: if there is no effect, the test will give us a false alarm 5 times in 100. It is a promise that holds for one test. The trouble begins when we run many of them, and any one turning out significant is enough for us to raise our hand.

A quick example. Rolling a die once, the probability of a 6 is one in six; rolling it ten times, the probability of at least one 6 rises to 84%. Tests work the same way. If one test alone has a 95% probability of not giving a false alarm, two independent tests have 95% of 95%, and so on. The probability of at least one false positive across m independent tests, which statisticians call the family-wise error rate (FWER), is therefore:

\(FWER = 1 – (1 – \alpha)^m\)

where α is the threshold of the single test (0.05) and m the number of tests. With 10 tests it is 1 − 0.95¹⁰ = 40%; with 14 it passes 50%; with 100 it reaches 99.4%.

There is another way to see it, even more direct: across m tests with no real effect, the expected number of false positives is m × α. Across 200 queries that did not change at all, we should expect about ten “significant” changes (200 × 0.05 = 10). They are not an anomaly: they are exactly what the 5% threshold promises.

An example: 200 queries in Search Console

We build the example by simulation, so we know the truth. Take 200 queries, each with a few thousand impressions a month and a CTR between 1% and 6%. For 180 of them nothing changes from one month to the next; for the first 20 the CTR really rises, by 50%. For each query we compare the two months with a test on proportions, as we would in a report.

I generate the data in R:

set.seed(7)
n_queries <- 200
impressions <- round(300 + rlnorm(n_queries, log(5000), 0.8))  # monthly impressions
ctr <- runif(n_queries, 0.01, 0.06)                             # CTR between 1% and 6%
real <- seq_len(n_queries) <= 20                                # the 20 that really change
clicks_before <- rbinom(n_queries, impressions, ctr)
clicks_after  <- rbinom(n_queries, impressions, ifelse(real, ctr * 1.5, ctr))

# one test per query: last month's CTR against this month's
p <- mapply(function(a, b, n) prop.test(c(a, b), c(n, n))$p.value,
            clicks_before, clicks_after, impressions)

Now let’s do what is usually done: flag as “changed” the queries with a p-value below 0.05, and compare with the truth.

table(discovery = p < 0.05, real)

#          real
# discovery FALSE TRUE
#     FALSE   171    2
#     TRUE      9   18

The test flagged 27 queries. Eighteen really changed, but nine did not: they are false alarms, exactly 5% of the 180 unchanged queries. In plainer terms: a third of our report’s “discoveries” is noise, and nothing in the numbers tells us which third. If we devoted an analysis, an intervention or a line of commentary to the client to each of those nine queries, we would be working on a ghost.

The Bonferroni correction (and Holm’s)

The oldest solution, and the most intuitive, is to make every single test stricter. If we want the probability of at least one false positive over the whole family of tests to stay at 5%, it is enough to divide the threshold by the number of tests. This is the Bonferroni correction:

\(\alpha_{Bonferroni} = \frac{\alpha}{m} = \frac{0.05}{200} = 0.00025\)

A query has “changed” only if its p-value falls below 0.00025. In R there is no need to do the division by hand: p.adjust() returns adjusted p-values, to be compared with the usual threshold. I check it in R:

table(discovery = p.adjust(p, method = "bonferroni") < 0.05, real)

#          real
# discovery FALSE TRUE
#     FALSE   180    6
#     TRUE      0   14

No false alarms: the 14 queries flagged are all real. The price, though, shows in the top right-hand corner. Six queries that really changed have vanished from the radar, because a 50% rise on a few hundred clicks is not enough to fall below such a low threshold. Bonferroni buys the certainty of not being wrong with blindness to part of the real effects.

There is a refinement that costs nothing, the Holm method. Sort the p-values from smallest to largest; compare the first with α/m, the second with α/(m − 1), the third with α/(m − 2), and stop at the first one that does not pass its threshold. It guarantees the same control as Bonferroni and is always at least as powerful, often more so. In our example it gives the same result (14 discoveries, none false); with p.adjust(p, method = "holm") there is no reason to prefer Bonferroni over it.

Family-wise error or share of false discoveries?

Before going further, it is worth pausing on a question that sounds philosophical and is in fact practical: what do we want to control?

Bonferroni and Holm control the probability of making even one false positive. That is the right requirement when every error is costly: we decide to redo the template of every product page on the strength of a test, and a false positive means weeks of wasted work.

But in many SEO analyses the situation is different. We sift through 200 queries or 500 pages to understand which deserve a closer look. Here a false positive is not a catastrophe: it is one more glance at a page that did not need it. What we do not want is for the discoveries to be mostly false alarms. The quantity to control then becomes the share of false positives among the discoveries, which statisticians call the false discovery rate (FDR). An FDR of 5% means: of the queries we flag, on average 5% will be noise.

They are two different promises, and the second is far less demanding than the first: that is why, with the same data, it lets through more real effects.

The Benjamini-Hochberg method

The method that controls the FDR, proposed by Yoav Benjamini and Yosef Hochberg in 1995, is surprisingly simple. In words: we sort the p-values from smallest to largest and, instead of a fixed threshold, we use a threshold that grows with the position in the ranking. The p-value in first place must be below 1/m × 0.05, the one in second place below 2/m × 0.05, and so on:

\(p_{(i)} \le \frac{i}{m} \cdot q\)

where p₍ᵢ₎ is the p-value in position i, m the number of tests and q the FDR we accept (here 0.05). We look for the lowest position in the ranking (the largest p-value) that satisfies the inequality: that query and all those before it are the discoveries.

Let’s look at the p-values of our example from position 14 to 20, each with its threshold:

Rank	p-value	BH threshold (rank/200 × 0.05)	CTR really changed?
14	0.000018	0.00350	yes
15	0.00070	0.00375	yes
16	0.00124	0.00400	yes
17	0.00356	0.00425	no
18	0.00531	0.00450	yes
19	0.00683	0.00475	no
20	0.00720	0.00500	yes

The last position below its threshold is 17: the first 17 queries are the discoveries. Bonferroni, with its fixed threshold of 0.00025, would have stopped at 14. I check it in R:

table(discovery = p.adjust(p, method = "BH") < 0.05, real)

#          real
# discovery FALSE TRUE
#     FALSE   179    4
#     TRUE      1   16

Seventeen discoveries, sixteen real and one false: one false alarm in seventeen, 5.9%, in line with the promised 5%. Compared with Bonferroni we have recovered two queries that really changed, at the price of a single error. And compared with the uncorrected report we have gone from nine false alarms in 27 to one in 17.

The chart shows the logic well. At the top of the ranking are the large effects, with minuscule p-values: any method finds them. The problem is the middle zone, where real but modest effects mix with noise. Bonferroni discards it wholesale; the flat threshold accepts it wholesale; Benjamini-Hochberg crosses it with a threshold that widens just enough.

(n.b. the method guarantees an average FDR of 5%: in a single report the actual share can be higher or lower. Repeating the simulation 300 times, Benjamini-Hochberg produces on average 0.6 false discoveries per report, a share of 3.7%: below the promised 5%.)

Which correction should we choose?

There is no correction that is right in absolute terms: there is the one that fits the question. In short:

Situation	Correction	Why
A single test, decided before looking at the data	none	there is no multiplicity
A few costly decisions (a template, a migration, the winning variant of an A/B/n test)	Holm	even one error is expensive
Many queries, pages or segments from which to choose what to investigate	Benjamini-Hochberg	what matters is the share of false discoveries, not the single error

The most common case in day-to-day practice, however, is another one: we had not decided anything beforehand. We opened Search Console, filtered, segmented, compared, and stopped at the numbers that looked interesting. In this scenario no correction is possible, because we do not even know how many tests we really ran.

Where multiple comparisons hide in SEO

The month-on-month query report is the most obvious case, but multiplicity comes in many disguises.

Segments. “CTR has not changed, but on mobile in Lombardy it has”: slice the data enough ways by device, country, category and page type, and some significant slice always turns up. It is the same trap as Simpson’s paradox, seen from the opposite side.

Metrics. In an SEO split test, looking at clicks, impressions, CTR, position and conversions means running five tests. That is why the primary metric is chosen in advance.

Variants. An A/B test with five variants against the control is five comparisons; if we also compare the variants with each other they become fifteen. For comparisons between all pairs of means, ANOVA has a dedicated tool of its own, Tukey’s test.

Time. Looking at a test every day while reserving the right to stop it is a multiplicity hidden in the calendar: it is the peeking problem.

Correlations. A correlation matrix among 20 metrics contains 190 pairs: about ten will turn out “significant” even if the metrics were all independent.

(n.b. in reality things are even worse than in our simulation. A query’s CTR does not fluctuate only because of the binomial chance of clicks: it changes with position, with SERP features, with the device mix. prop.test() ignores these sources of variability and underestimates the uncertainty, so on real data false alarms exceed 5%. Multiple comparisons corrections do not fix a wrong model: they assume it is right.)

Frequently asked questions

What are multiple comparisons in statistics?

They are situations in which many statistical tests are run on the same data or the same question: comparing many queries, many segments, many metrics or many variants. Each test has its own probability of a false positive, and together the probabilities add up: with enough tests, some “significant” result arrives by pure chance.

How is the Bonferroni correction applied?

Divide the significance threshold by the number of tests: with 20 tests and a 5% threshold, each test needs a p-value below 0.05 / 20 = 0.0025. Equivalently, multiply each p-value by the number of tests and compare it with 0.05. In R, p.adjust(p, method = "bonferroni") does it.

What is the difference between FWER and FDR?

The FWER (family-wise error rate) is the probability of making at least one false positive across the set of tests; the FDR (false discovery rate) is the average share of false positives among the results declared significant. Controlling the FWER is stricter and suited to a few costly decisions; controlling the FDR suits the exploration of many hypotheses.

When should we use Benjamini-Hochberg instead of Bonferroni?

When the tests are many and the aim is to select what to investigate, accepting that a small share of the discoveries is false: analyses of hundreds of queries or pages, screening of segments, anomaly searches. In these cases Bonferroni is so strict that it loses a good part of the real effects.

Is the correction needed even with few tests?

With two or three tests the overall risk grows only a little (from 5% to about 10-14%), but it grows. If the comparisons were decided in advance and are few, the Holm method costs very little in power and keeps us safe. If instead a single primary comparison was decided in advance, no correction is needed.

Try it yourself

The example code is complete, and lends itself to a few experiments.

• Set the number of queries that really change to 0 (real <- rep(FALSE, n_queries)) and count the discoveries with and without correction. Without correction there should be about ten; with Bonferroni and with Benjamini-Hochberg almost always zero. This is the dead salmon case.
• Bring the CTR increase down from 1.5 to 1.2: a smaller effect, more realistic for a rewritten title. How many of the 20 real queries does each method find?
• Repeat the simulation with 2,000 queries instead of 200, still with 20 real ones. What happens to the Bonferroni threshold, and how many real queries survive?

The last exercise, in particular, shows why large sites need the FDR: the more tests we run, the blinder Bonferroni makes us.

---

We have talked about p-values throughout the article: correcting them, ranking them, comparing them with different thresholds. But what exactly does a p-value of 0.03 say? That there is a 97% probability the effect is real? That the null hypothesis has a 3% probability of being true? Neither, and it is perhaps the most misunderstood number in all of applied statistics. It will be the subject of the next step: the p-value and its misconceptions.

---

Further reading

To understand why so many «significant» results do not stand up to the facts, and how much the number of questions we have put to the data matters, I recommend The Art of Statistics by David Spiegelhalter: a book that explains uncertainty without needless formulas and without shortcuts. For multiple comparisons in online experiments, from metrics to variants, the reference remains Trustworthy Online Controlled Experiments by Ron Kohavi, Diane Tang and Ya Xu.