Category: statistics

Anyone who looks at a website’s data does it constantly, often without noticing: they spot that two things seem to move together. Pages that sit higher in the SERP get more clicks; the ones where users linger longer convert more; longer articles appear to rank better. These are valuable hunches, but they stay vague until we answer a precise question: how much do these pairs of numbers move together? And in what sense? We need an index that turns the impression “they go hand in hand” into a comparable measure. That index is correlation, and it is one of the most used — and most misunderstood — tools in all of applied statistics.

We closed the article on the A/B test significance calculator with a promise. We said that the p-value answers a single question — does the effect exist? — and that, on its own, it adds nothing else. It does not tell us how large the effect is, nor whether it is worth the effort of shipping it. It is time to keep that promise, because the two questions the p-value leaves hanging are exactly what separates reading data with method from stopping at the first threshold that glitters.

The two questions have precise names. The first — how big is it? — is the effect size. The second — with the data I have, could I even have seen an effect like this? — is the power of the test, and the reasoning that gets us to an answer is called power analysis. We examine them one at a time, as always with an example at hand.

Our A/B test has run its course: variant B shows a higher conversion rate than variant A. The temptation to declare a winner and ship the change is strong. But first there is a question to answer, the same one that runs through this whole series: is the difference we observe a real signal, or just statistical noise?

This calculator is the natural complement of the sample size calculator: that one works before the test and tells us how many users we need; this one works after and tells us whether the result we obtained is statistically significant. If you have read the article on hypothesis testing, you will recognise the machinery at once: behind the scenes sits a z-test for comparing two proportions.

Continue reading “A/B Test Significance Calculator”