{"id":3847,"date":"2026-06-23T10:40:54","date_gmt":"2026-06-23T09:40:54","guid":{"rendered":"https:\/\/www.gironi.it\/blog\/?p=3847"},"modified":"2026-06-23T10:40:55","modified_gmt":"2026-06-23T09:40:55","slug":"peeking-problem-ab-testing","status":"publish","type":"post","link":"https:\/\/www.gironi.it\/blog\/en\/peeking-problem-ab-testing\/","title":{"rendered":"The peeking problem: why sneaking a look at an A\/B test inflates false positives"},"content":{"rendered":"\n

On 21 January 2015 Optimizely \u2014 one of the most widely used A\/B testing platforms in the world \u2014 switched on a completely new statistical engine for all of its customers, the New Stats Engine<\/em>.
It wasn’t a technical whim: the old engine, built around a classic fixed-horizon t-test (Fixed Horizon<\/em>) and developed with statisticians from Stanford, had a flaw that affected anyone who looked at a test’s results before the end. And we look at a test’s results always<\/em>, before the end.<\/p>\n\n\n\n

Optimizely had measured the problem themselves, simulating A\/A tests \u2014 two identical variants, where by construction neither is better than the other, so any declared “winner” is a false alarm.
According to the figures published by Optimizely, on tests of 5,000 visitors anyone checking the numbers after every<\/em> visitor saw 57% of A\/A tests declare a false winner at least once<\/strong>; checking every 500 visitors the figure dropped to 26%, every 1,000 to 20%. Chilling numbers for a tool that is supposed to help us decide with rigour. The rewrite \u2014 sequential inference plus false discovery rate control, what they call always-valid \u2014 was meant precisely to bring the error, as they put it, “from over 30% to 5%”.<\/p>\n\n\n\n

It’s the same deception we ran into closing the article on regression to the mean<\/a>: there we selected the worst-performing pages \u2014 an extreme instant in the space<\/em> of the data \u2014 and let ourselves be fooled by their rebound. Here we select an extreme instant in time<\/em>: we stop the moment the test proves us right. The mechanism is a cousin, the risk identical.<\/p>\n\n\n\n\n\n\n\n

What peeking is<\/h2>\n\n\n\n

Anyone who runs an A\/B test<\/a> knows it well: the test is running, the data come in day after day, and the temptation to sneak a look at the dashboard is irresistible.
Peeking<\/em> isn’t the mere act of looking: it’s looking while reserving the right to stop the test<\/em> the moment the result becomes significant. It’s that “great, variant B has crossed the threshold, let’s wrap up here and declare the winner” said halfway through data collection.<\/p>\n\n\n\n

The delicate point is that every look accompanied by the possibility of stopping is one more statistical test<\/strong>.
A single test with a 5% threshold accepts, by definition, a 5% chance of crying “winner” when in fact there’s no difference at all. But if we repeat that same test twenty times over the course of collection, and all we need is for just one<\/em> of those twenty times to cross the threshold in order to stop and declare victory, then the chance of stumbling into a false positive is no longer 5%: it accumulates<\/strong> with every look.<\/p>\n\n\n\n

This isn’t the usual multiplicity of someone comparing ten variants at once. Here the multiplicity is hidden in time<\/em>: a single variant, looked at many times. It’s the same logic by which a single coin toss rarely gives a strange result, but if one is allowed to look after every toss and stop at the first favourable moment, sooner or later that moment arrives \u2014 and it gets mistaken for a signal.<\/p>\n\n\n\n

What peeking costs: a simulation<\/h2>\n\n\n\n

Words convince us up to a point; numbers convince us far more. I simulate in R an A\/A test, that is two variants with exactly the same<\/strong> true conversion rate (10%): any difference that emerges is noise, and any declared “victory” is a false positive by construction.
I set the stage by fixing the random number generator’s seed (so the numbers are reproducible), the function that computes the p-value of the comparison between two proportions, and the function that simulates a single experiment and reports whether at some point it declared a (false) winner:<\/p>\n\n\n\n

set.seed(2025)\n\np_vero  <- 0.10    # same conversion rate for A and B (H0 true)\nn_arm   <- 2000    # visitors per variant at the end of the test\nn_sim   <- 4000    # number of simulated experiments\nalpha   <- 0.05\nsguardi <- 20      # how many times we \"peek\" during collection\nlook_at <- round(seq(n_arm \/ sguardi, n_arm, length.out = sguardi))\n\n# p-value of a two-proportion, two-sided z-test\npval_ab <- function(xa, na, xb, nb) {\n  pp <- (xa + xb) \/ (na + nb)\n  se <- sqrt(pp * (1 - pp) * (1 \/ na + 1 \/ nb))\n  2 * pnorm(-abs((xa \/ na - xb \/ nb) \/ se))\n}\n\n# one A\/A experiment: TRUE if it declares a (false) winner\nesperimento <- function(soglia, guarda) {\n  a <- cumsum(rbinom(n_arm, 1, p_vero))\n  b <- cumsum(rbinom(n_arm, 1, p_vero))\n  for (k in guarda) {\n    p <- pval_ab(a[k], k, b[k], k)\n    if (!is.na(p) && p < soglia) return(TRUE)\n  }\n  FALSE\n}<\/code><\/pre>\n\n\n\n
Let’s start from the correct behaviour: a single test, at the end, on the 2,000 visitors per variant. I run it 4,000 times and count how many declare a winner:<\/p>\n\n\n\n
# fixed horizon: a single test, at the end\nfisso <- mean(replicate(n_sim, esperimento(alpha, n_arm)))\ncat(sprintf(\"Fixed horizon: %.1f%% false positives\\n\", 100 * fisso))\n# Fixed horizon: 5.0% false positives<\/code><\/pre>\n\n\n\n
Out comes 5.0%<\/strong>: exactly the level we declared with the 5% threshold. The test, used as it should be, keeps its promise. 
Now I change one thing only: instead of looking once at the end, I look twenty times during collection and stop at the first moment the p-value drops below 0.05. I add the intermediate looks and run again:<\/p>\n\n\n\n
# peeking: a test at every look, stop at the first significant one\npeek <- mean(replicate(n_sim, esperimento(alpha, look_at)))\ncat(sprintf(\"Peeking (%d looks): %.1f%% false positives\\n\", sguardi, 100 * peek))\n# Peeking (20 looks): 24.3% false positives<\/code><\/pre>\n\n\n\n
From 5.0%<\/strong> to 24.3%<\/strong>. 
The same data, the same test, the same threshold: the only thing that changed is when we decided to look, and the false positive rate has nearly quintupled.<\/strong> Almost one A\/A test in four, in which the two variants are identical by construction, convinces us we’ve found a winner. The 24.3% from our simulation and the 30% reported by Optimizely tell the same story with different data: peeking isn’t a venial sin, it’s the most effective way to fool ourselves.<\/p>\n\n\n\n
Solution 1: the fixed horizon<\/h2>\n\n\n\n
The simplest cure is also the most annoying one: decide beforehand<\/em> how much data to collect, and then have the discipline to wait until the end without stopping early, whatever the dashboard says in the meantime. 
It’s what the simulation has just shown us: with a single test at the end, the false positive stays nailed to the promised 5%. No magic, just the elimination of opportunistic looks.<\/p>\n\n\n\n
“How much data” isn’t a number plucked from thin air: it depends on how small a difference we want to be able to detect and on how much certainty we demand. It’s the sample size calculation, which is done before launching the test with our significance calculator<\/a> and which rests on the concepts of effect size and power analysis<\/a>. 
Once that number is fixed, the fixed horizon is the safest road: no statistical correction to apply, no threshold to tweak. The price, though, is paid in patience \u2014 resisting the curiosity for days or weeks \u2014 and this, in operational reality, is exactly what almost no one manages to do.<\/p>\n\n\n\n
Solution 2: looking without cheating<\/h2>\n\n\n\n
And if monitoring on the fly really were necessary \u2014 because a test that’s going terribly needs stopping, because the stakeholders want updates? 
Then the way is not to look in secret with the usual threshold, but to look openly<\/em> with a stricter one. The idea is simple: if at every look we raise the bar, making it harder to cry “winner” on each occasion, we can arrange for the overall<\/strong> error \u2014 summed across all the looks \u2014 to stay at the 5% we wanted. I calibrate in R the per-look threshold, trying ever more stringent values on the same twenty looks as before:<\/p>\n\n\n\n
# stricter per-look threshold that brings the overall error back to ~5%\nfor (sg in c(0.05, 0.02, 0.01, 0.005)) {\n  fp <- mean(replicate(n_sim, esperimento(sg, look_at)))\n  cat(sprintf(\"  threshold %.3f -> %.1f%% overall\\n\", sg, 100 * fp))\n}\n#   threshold 0.050 -> 25.1% overall\n#   threshold 0.020 -> 11.7% overall\n#   threshold 0.010 ->  6.6% overall\n#   threshold 0.005 ->  3.3% overall<\/code><\/pre>\n\n\n\n
As we can see, the usual 0.05 threshold produces a 25.1% overall error (the peeking disaster again), but as we make it stricter the error comes back down: around 0.01 \u2014 a threshold five times more stringent than the standard one \u2014 the overall error returns close to the nominal 5%.<\/strong> It’s the price to be paid for the right to peek: at every single look much more evidence is required, in exchange for the freedom to look often.<\/p>\n\n\n\n
What we’ve just shown is a homemade, constant-threshold version of the idea. The “textbook” boundaries \u2014 more refined, with thresholds that change over the course of the test, like those of Pocock or O’Brien-Fleming \u2014 are obtained in R with the gsDesign<\/code> package, and commercial tools like Optimizely use an always-valid<\/em> variant (the so-called mSPRT) of the same underlying idea. 
The fine mathematics changes, not the principle: to look often without cheating one must demand, at every look, more evidence than a single test would ask for.<\/p>\n\n\n\n
A word of caution: a result seen during the test, on its own, proves nothing: what counts is when the decision to look was made.<\/strong> 
The same p-value below 0.05 means different things depending on whether it’s the only fixed-horizon test or the first of the twenty at which one reserved the right to stop. Without declaring in advance how and when the data will be examined, any “winner” that emerges on the fly is suspect.<\/p>\n\n\n\n
Try it yourself<\/h2>\n\n\n\n
To feel the mechanism up close, let’s start from the script and change a single parameter: the number of sguardi<\/code> (looks). 
Let’s go from weekly monitoring (few looks) to daily monitoring (many looks) and re-run the peeking simulation. What to expect: the more frequently one peeks, the higher the false positive rate climbs \u2014 the frequency of looks is the fuel of the problem. Then let’s redo the threshold calibration with that new number of looks and check that, by choosing a strict enough threshold, the overall error comes back under control all the same. It’s the proof, first-hand, that peeking isn’t a curse: it’s just a bill that has to be paid.<\/p>\n\n\n\n
\n\n\n\n
There’s one last trap in this family, perhaps the most insidious of all, because it doesn’t hide in our own data but in the data others tell us about. When we read an agency’s case study \u2014 “we increased conversions by 300% with this tactic” \u2014 we’re looking at a survivor: the thousand identical attempts that failed are something nobody mentions. It’s survivorship bias<\/em>, the reason case studies lie even when they tell the truth, and it’s the next step on our journey through the pitfalls of marketing data.<\/p>\n\n\n\n
\n\n\n\n
Further Reading<\/h3>\n\n\n\n
On peeking, early stopping and sequential testing the reference \u2014 in English \u2014 remains Trustworthy Online Controlled Experiments<\/em><\/a> by Ron Kohavi, Diane Tang and Ya Xu: written by people who led the experimentation platforms at Microsoft, Google and LinkedIn, it devotes explicit pages to all the ways a running A\/B test can fool us, and to how to defend against them. It’s the book anyone who has to take online experiments seriously pulls out of the drawer.<\/p>\n","protected":false},"excerpt":{"rendered":"
On 21 January 2015 Optimizely \u2014 one of the most widely used A\/B testing platforms in the world \u2014 switched on a completely new statistical engine for all of its customers, the New Stats Engine. It wasn’t a technical whim: the old engine, built around a classic fixed-horizon t-test (Fixed Horizon) and developed with statisticians … Leggi tutto “The peeking problem: why sneaking a look at an A\/B test inflates false positives”<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_uag_custom_page_level_css":"","footnotes":""},"categories":[161],"tags":[],"class_list":["post-3847","post","type-post","status-publish","format-standard","hentry","category-statistics"],"lang":"en","translations":{"en":3847,"it":3846},"uagb_featured_image_src":{"full":false,"thumbnail":false,"medium":false,"medium_large":false,"large":false,"1536x1536":false,"2048x2048":false,"post-thumbnail":false},"uagb_author_info":{"display_name":"Paolo Gironi","author_link":"https:\/\/www.gironi.it\/blog\/author\/autore-articoli\/"},"uagb_comment_info":0,"uagb_excerpt":"On 21 January 2015 Optimizely \u2014 one of the most widely used A\/B testing platforms in the world \u2014 switched on a completely new statistical engine for all of its customers, the New Stats Engine. It wasn’t a technical whim: the old engine, built around a classic fixed-horizon t-test (Fixed Horizon) and developed with statisticians…","_links":{"self":[{"href":"https:\/\/www.gironi.it\/blog\/wp-json\/wp\/v2\/posts\/3847","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.gironi.it\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.gironi.it\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.gironi.it\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.gironi.it\/blog\/wp-json\/wp\/v2\/comments?post=3847"}],"version-history":[{"count":1,"href":"https:\/\/www.gironi.it\/blog\/wp-json\/wp\/v2\/posts\/3847\/revisions"}],"predecessor-version":[{"id":3854,"href":"https:\/\/www.gironi.it\/blog\/wp-json\/wp\/v2\/posts\/3847\/revisions\/3854"}],"wp:attachment":[{"href":"https:\/\/www.gironi.it\/blog\/wp-json\/wp\/v2\/media?parent=3847"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.gironi.it\/blog\/wp-json\/wp\/v2\/categories?post=3847"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.gironi.it\/blog\/wp-json\/wp\/v2\/tags?post=3847"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}