Paolo Gironi - notes on data analysis, SEO, statistics and retro computing

The peeking problem: why sneaking a look at an A/B test inflates false positives

On 21 January 2015 Optimizely — one of the most widely used A/B testing platforms in the world — switched on a completely new statistical engine for all of its customers, the New Stats Engine.
It wasn’t a technical whim: the old engine, built around a classic fixed-horizon t-test (Fixed Horizon) and developed with statisticians from Stanford, had a flaw that affected anyone who looked at a test’s results before the end. And we look at a test’s results always, before the end.

Optimizely had measured the problem themselves, simulating A/A tests — two identical variants, where by construction neither is better than the other, so any declared “winner” is a false alarm.
According to the figures published by Optimizely, on tests of 5,000 visitors anyone checking the numbers after every visitor saw 57% of A/A tests declare a false winner at least once; checking every 500 visitors the figure dropped to 26%, every 1,000 to 20%. Chilling numbers for a tool that is supposed to help us decide with rigour. The rewrite — sequential inference plus false discovery rate control, what they call always-valid — was meant precisely to bring the error, as they put it, “from over 30% to 5%”.

It’s the same deception we ran into closing the article on regression to the mean: there we selected the worst-performing pages — an extreme instant in the space of the data — and let ourselves be fooled by their rebound. Here we select an extreme instant in time: we stop the moment the test proves us right. The mechanism is a cousin, the risk identical.

Regression to the Mean: the SEO Fix That Worked… by Accident

In the Israeli Air Force, Daniel Kahneman recounts, the flight instructors were sure of one thing: praising a cadet after an excellent manoeuvre made him worse, scolding him after a terrible one made him better.
They had seen it happen a thousand times in the field, so it had to be true: with pilots, severity works and compliments backfire.
Except it wasn’t true. An exceptional manoeuvre — in either direction — is part skill and part luck; and luck, on the next attempt, doesn’t repeat. After a brilliant flight you tend to drift back toward your own average (and it looks as if the praise hurt), after a disastrous one you drift back toward the average (and it looks as if the scolding helped). The instructors were crediting themselves with an effect that was just regression to the mean.

The very same illusion waits for us every time we look at a site’s data and decide whether one of our changes “worked”.

A/B Testing: How to Run Statistically Valid Experiments (and the Mistakes to Avoid)

Over the previous articles we have looked at how hypothesis testing works and how the two-sample t-test lets us compare two groups rigorously. We have also built confidence intervals, learned to quantify the uncertainty of our estimates, and seen with the Central Limit Theorem why all this works even when the data are not normal.

But there is one question that, in the day-to-day reality of anyone doing SEO and marketing, comes up almost daily: which variant performs better? Which ad headline brings more clicks? Which landing page converts more? Which meta description draws attention? It is not an academic question: it is the question that separates data-driven decisions from opinions disguised as strategies.

The good news is that we already have all the tools to answer it. A/B testing is nothing more than the direct application of the statistical inference concepts we have built step by step: hypothesis testing, comparison between groups, significance. In this article we put it all together.

PCA (Principal Component Analysis): reduce SEO data complexity without losing insights

SEO and web marketing analysis almost always presents us with the same problem: we have too many metrics and we don’t know which ones really matter. Search volume, CPC, competition, CTR, bounce rate, time on page, conversions by source, average position — the list grows with every new tool we add to our stack.

The problem is not a lack of data: it’s that these metrics are often correlated with each other (more impressions lead to more clicks, which lead to more conversions, which increase costs). When variables move together, most of the information they contain is redundant. Principal Component Analysis (PCA) is the tool we have to cut through this redundancy: it reduces a set of many correlated variables to a few uncorrelated components that capture most of the original information.

In clearer terms: PCA finds the “directions” in which our data vary the most, and lets us project everything else onto those few dimensions. It is like looking at a three-dimensional object from its most informative angle instead of from a random viewpoint.

Correlation: Pearson, Spearman and Kendall (and Why It Isn’t Causation)

Anyone who looks at a website’s data does it constantly, often without noticing: they spot that two things seem to move together. Pages that sit higher in the SERP get more clicks; the ones where users linger longer convert more; longer articles appear to rank better. These are valuable hunches, but they stay vague until we answer a precise question: how much do these pairs of numbers move together? And in what sense? We need an index that turns the impression “they go hand in hand” into a comparable measure. That index is correlation, and it is one of the most used — and most misunderstood — tools in all of applied statistics.

paologironi blog