A/B Testing: How to Run Statistically Valid Experiments

Over the previous articles we have looked at how hypothesis testing works and how the two-sample t-test lets us compare two groups rigorously. We have also built confidence intervals, learned to quantify the uncertainty of our estimates, and seen with the Central Limit Theorem why all this works even when the data are not normal.

But there is one question that, in the day-to-day reality of anyone doing SEO and marketing, comes up almost daily: which variant performs better? Which title tag brings more clicks? Which landing page converts more? Which meta description draws attention? It is not an academic question: it is the question that separates data-driven decisions from opinions disguised as strategies.

The good news is that we already have all the tools to answer it. A/B testing is nothing more than the direct application of the statistical inference concepts we have built step by step: hypothesis testing, comparison between groups, significance. In this article we put it all together.

What an A/B Test Is

An A/B test is, in essence, a controlled experiment: we take two variants of something (a page, a headline, a call-to-action), randomly assign users to one of the two variants, and measure which one produces better results.

Variant A is the control (the current version, the one we are already using). Variant B is the treatment (the new version we want to test). The logic is the same as a scientific experiment: we change one variable at a time, keep everything else constant, and observe whether the change produces a measurable effect.

Three elements make an A/B test reliable. Randomisation: users are assigned to A or B at random. This is essential, because if we showed A in the morning and B in the afternoon, any observed difference might depend on the time of day, not on the variant. The control group: without A as a reference, we wouldn’t know whether B’s results are good or bad. And finally a success metric defined in advance: CTR, conversion rate, time on page. The metric must be chosen before collecting the data, not after (we will come back to this point shortly).

The diagram summarises the path we will follow: from the initial question to the final decision, passing through hypotheses, randomisation, data collection and the statistical test. Every step has its pitfalls.

But why do we need statistics? Because data are noisy. If variant A has a CTR of 5.0% and variant B of 5.3%, is that difference real or just random fluctuation?
The naked eye cannot tell: we need a formal test. And it is precisely the two-sample test we have already seen — applied to proportions rather than means.

Setting Up an A/B Test Correctly

Before collecting data, we have to set up the test rigorously. Let’s see how.

Choosing the metric. The metric must be clear, measurable and directly linked to the goal. For a title tag, the natural metric is the CTR (Click-Through Rate). For a landing page, the conversion rate. For a blog article, perhaps the average time on page.
Always keep this in mind: a vague metric (“people like the page more”) is not a metric.

Defining the hypotheses. As in every statistical test, we start from a null hypothesis and an alternative hypothesis:

\( H_0 \): the two variants have the same effect (no difference between A and B)
\( H_1 \): the two variants have a different effect (a difference exists)

The test assumes the observations are independent and the assignment to variants is truly random. In practice, this means: no double-counting sessions from the same user, no time-of-day alternation (A in the morning, B in the afternoon). These conditions seem obvious, but they are violated more often than you might think.

The statistical test. When we compare two proportions (such as two CTRs or two conversion rates), the appropriate test is the two-proportion z-test. The logic is the same as the two-sample t-test, but adapted to binary data (click/no-click, conversion/no-conversion).

The test statistic is computed as follows. First, we compute the pooled proportion, which is our best estimate of the common proportion under the null hypothesis:

\( \hat{p} = \frac{x_1 + x_2}{n_1 + n_2} \\ \)

where \( x_1 \) and \( x_2 \) are the successes (clicks, conversions) in the two groups, and \( n_1 \) and \( n_2 \) the sample sizes.

Then we compute the z statistic:

\( z = \frac{\hat{p}_1 – \hat{p}_2}{\sqrt{\hat{p}(1-\hat{p})\left(\frac{1}{n_1} + \frac{1}{n_2}\right)}} \\ \)

The numerator is the observed difference between the two proportions; the denominator is the standard error under the null hypothesis.
The ratio tells us how many “standard-error units” separate the two proportions: the higher it is, the harder the difference is to attribute to chance.

Example: CTR of Two Title Tags

Let’s take a concrete example. We tested two title tag variants for an important page on the site:

Title A (control): 1500 impressions, 75 clicks → CTR = 5.0%
Title B (treatment): 1500 impressions, 105 clicks → CTR = 7.0%

Title B looks better, but is the difference statistically significant? Let’s compute it step by step.

Step 1: the pooled proportion:

\( \hat{p} = \frac{75 + 105}{1500 + 1500} = \frac{180}{3000} = 0.06 \\ \)

Step 2: the standard error:

\( SE = \sqrt{0.06 \times 0.94 \times \left(\frac{1}{1500} + \frac{1}{1500}\right)} = \sqrt{0.0564 \times 0.00133} \approx 0.00867 \\ \)

Step 3: the z statistic:

\( z = \frac{0.07 – 0.05}{0.00867} \approx 2.31 \\ \)

Step 4: the p-value. For a two-tailed test, \( p \approx 0.021 \).

So: the p-value is below 0.05. We can reject the null hypothesis and conclude that the difference between the two title tags is statistically significant. Title B has a significantly higher CTR — from a statistical standpoint.

Two-proportion z-test: sampling distribution under H₀ with the observed value

The figure shows what “statistically significant” means with our data. The curve is the distribution of the standardised difference under H₀: if there were no real difference between A and B, the z-value would almost always fall between −1.96 and +1.96 (the red lines). Our z = 2.31 falls beyond the threshold, in the rejection region. The area shaded in orange is the p-value: the probability of observing a difference this large (or larger) by pure chance is only 2.1% — small enough to rule out chance.

Let’s run the same test in R:

n1 <- 1500; x1 <- 75    # Title A
n2 <- 1500; x2 <- 105   # Title B
p1 <- x1 / n1  # 0.05
p2 <- x2 / n2  # 0.07

p_pool <- (x1 + x2) / (n1 + n2)
se <- sqrt(p_pool * (1 - p_pool) * (1/n1 + 1/n2))
z <- (p2 - p1) / se
p_value <- 2 * (1 - pnorm(abs(z)))

cat("z =", round(z, 3), "\n")
cat("p-value =", round(p_value, 4), "\n")

Result: z = 2.306, p-value = 0.0211.

Worked Example: Conversion Rate of Two Landing Pages

Let’s move on to a more elaborate example. An e-commerce store is testing two variants of its landing page:

Page A (current design): 1000 visitors, 35 conversions → conversion rate = 3.5%
Page B (new design): 1000 visitors, 58 conversions → conversion rate = 5.8%

The difference looks substantial (2.3 percentage points), but with these numbers is it enough to rule out chance?

Let’s check in R with prop.test(), which runs the two-proportion test:

result <- prop.test(x = c(35, 58), n = c(1000, 1000))
print(result)

The function returns the p-value of the test and, very usefully, the confidence interval of the difference between the two proportions. In this case the p-value is about 0.019 — below 0.05, so the difference is statistically significant.

But it is the confidence interval of the difference that gives us the most valuable information: not only whether B is better than A, but by how much, and with what margin of uncertainty. If the CI of the difference runs from about 0.4 to 4.2 percentage points, we know that B is almost certainly better, and the improvement lies within that range. That is far richer information than a simple “yes, it’s significant”.

The p-value tells us whether we can doubt the null hypothesis; the confidence interval tells us how large the observed effect might be. They answer two different questions, and the CI answers the one we ultimately care about most: not just “is there a difference?”, but “how big is it?”.

n.b.: prop.test() applies a continuity correction (Yates’s correction) that makes the test slightly more conservative. For large samples the difference is negligible; for small samples, it is a welcome caution.

The Most Common Mistakes

A/B testing is a powerful tool, but a treacherous one. The ease with which a test can be set up hides serious methodological pitfalls. Let’s look at the most frequent ones.

Stopping the Test Too Early

It is the strongest temptation: after a few days, B looks clearly better than A. Why wait any longer?
Because those preliminary results are noise, not signal.

The problem has a technical name: peeking. Every time we look at the interim data and decide whether to stop, we increase the probability of a false positive. It’s like tossing a coin: if we stop every time we get three heads in a row, we’ll conclude the coin is rigged. But it isn’t — we simply haven’t given it enough tosses.

How to avoid it: define the required sample size beforehand and wait until you reach that number before drawing conclusions. In the meantime, you can use our sample size calculator to determine how many users you need before launching the test.

Testing Too Many Variants Without Correction

Another frequent mistake: testing three, four, five variants at the same time (A/B/C/D…) and then comparing them all pairwise. The problem is that of multiple comparisons: the more comparisons we make, the more likely we are to find at least one significant result by pure chance.

With 5 variants and 10 pairwise comparisons, the probability of finding at least one false positive rises from 5% to almost 40%. This is not a detail: it is an error that invalidates the entire test.

How to avoid it: if multiple comparisons are needed, apply a Bonferroni correction (divide the α threshold by the number of comparisons) or, better still, limit yourself to testing one variant at a time.

Ignoring the Power of the Test

We know the risk of a false positive well (type I error, α). But there is a mirror risk that is often ignored: the false negative (type II error, β). It happens when B really is better than A, but our test fails to detect it.

The most common cause? A sample that is too small. If we have only 100 visitors per variant, the test does not have enough “power” to detect small but real differences. We will conclude “no significant difference” not because the difference doesn’t exist, but because we didn’t have enough data to see it.

How to avoid it: compute the required sample size before launching the test, based on the minimum effect we want to detect. This is the subject of power analysis: use the sample size calculator to check whether your test has enough power.

Confusing Statistical Significance with Practical Significance

A low p-value does not automatically mean the result is important. With very large samples, even microscopic differences become statistically significant. If we test two variants on 500,000 visitors, a CTR difference of 0.01% (from 5.00% to 5.01%) might come out significant. Operationally, though, it is irrelevant. The p-value answers the question “is the difference real?”, not the question “is the difference big enough to matter to us?”. For the latter we need a different measure — the effect size — which we cover in a dedicated article.

Frequentist vs Bayesian Approach

Everything we have seen so far follows the frequentist approach: we compute a test statistic, compare it with a reference distribution, obtain a p-value and make a binary decision (reject or fail to reject \( H_0 \)).

It works, and works well. But it has limits that you feel in everyday practice. The p-value does not tell us “by how much B is better than A”. It does not tell us “what the probability is that B is genuinely superior”. And if we collect new data, we cannot simply update the result: we have to recompute everything from scratch.

There is an alternative approach that answers directly the question we care about most: what is the probability that B is better than A?
It is the Bayesian approach, to which we have devoted a dedicated article: Bayesian A/B Testing, where we see how to build a Beta posterior for each variant, compute P(B > A), read the distribution of the difference (by how much B is better, not just whether) and decide when to stop the test using expected loss.

Practical SEO Example: Meta Description A/B Test

Let’s look at one last scenario, very common in everyday practice. We have two meta description variants for a key page on the site. Alternating the two versions (two weeks each, to minimise seasonal effects) and consulting the Search Console data, we get:

Meta A: 3200 impressions, 128 clicks → CTR = 4.0%
Meta B: 3100 impressions, 155 clicks → CTR = 5.0%

Let’s check in R:

prop.test(c(128, 155), c(3200, 3100))

The p-value is about 0.064 — above the 0.05 threshold, so we cannot reject the null hypothesis. The confidence interval of the difference also includes zero, confirming the non-significance. A borderline result, which tells us: with these data we don’t have enough evidence to conclude that Meta B is genuinely better.

Which approach should we use? For a simple test like this, the frequentist approach with prop.test() is more than sufficient: we have large samples, the question is clear. The Bayesian approach becomes more valuable when the samples are small, when we want to update the result as new data arrive, or when we have prior knowledge to incorporate (for example, we know that for that type of page the CTR is typically between 3% and 7%).

But the operational decision must not rest on the p-value alone. We have to ask: is the difference (one percentage point more of CTR) big enough to justify the change? With 3000-plus impressions a month, one percentage point more means about 30 additional clicks. Is that significant for our business?
This is a question statistics cannot resolve on its own — it is a judgement that falls to us.

FAQ

What is the difference between a one-tailed and a two-tailed test?
A two-tailed test checks whether B is different from A (better or worse). A one-tailed test checks whether B is better than A (or worse, depending on direction). When in doubt, always use the two-tailed test — it is more conservative.

Can I run A/B tests with more than two variants?
Yes, but each additional variant increases the risk of false positives. If you test 5 variants, you need to correct the significance threshold (e.g., Bonferroni). Better to test one variant at a time, unless you use a specific multivariate test design.

Does the p-value tell me how much better B is than A?
No. The p-value only tells you whether the observed difference is compatible with the null hypothesis. To know how much better B is, you need the confidence interval of the difference or an effect size measure.

When should I stop an A/B test?
When you have reached the planned sample size — not before. Stopping “because B is winning” is the most common trap (peeking). If the test runs longer than expected without reaching significance, it might mean the effect is too small to detect with the available data.

Try It Yourself

An e-commerce store is testing two call-to-action variants on a product page:

Variant A (“Add to cart”): 450 visits, 23 conversions
Variant B (“Buy it now”): 430 visits, 31 conversions

Compute the conversion rate of each variant
Run the test with prop.test(c(23, 31), c(450, 430)) and interpret the p-value
Does the confidence interval of the difference include zero?
At the 5% significance level, is the difference statistically significant?

Hint: if the p-value is above 0.05, we cannot conclude that one variant is better than the other — but this does not mean they are equal. It might simply mean we don’t have enough data. It is exactly the problem of the power of the test that we discussed.

A/B testing gives us a rigorous framework for making decisions based on data, not intuition. But as we have seen, a well-run test tells us whether there is a significant difference — it does not tell us how large that effect is, nor how much data we need to detect it with confidence. Those are the questions of effect size and power analysis, the next tools in our path. For the sample size, the interactive calculator lets you get the exact number in real time.

A/B Testing: How to Run Statistically Valid Experiments (and the Mistakes to Avoid)

In this article: