statistics

Which Statistical Test Should We Use for an A/B Test? A Practical Guide with R Examples

The test is over. We have two columns of numbers, one for variant A and one for B, and a question that seems simple: is the difference we see real, or is it chance?
Then we open a manual, or ask around, and the question multiplies: z-test or t-test? Student or Welch? And the chi-square, and Fisher, and that Mann-Whitney someone always recommends? Choosing a tool seems to require already knowing the answer.

The good news is that the choice is not a matter of taste. We have already seen, when talking about A/B testing, how to set up an experiment; here we take the next step. The right test depends on two or three questions about our data, always the same ones, and once we understand those the choice more or less makes itself. We line them up in a diagram, and then we see each test at work on a concrete case, with the R code to do it again.

In short: three things matter when choosing the test of an A/B test. What we measure: proportions (conversions, clicks) or continuous values (time, order value). How many variants we compare: two, or three and more. The characteristics of the data: small counts, very skewed distributions, extreme values. For two conversion rates we use the two-proportion z-test (in R prop.test()), for three or more variants the chi-square, with small counts Fisher’s exact test. For two means, Welch’s t-test, which unlike Student’s t-test does not assume equal variances; if instead the question is not about the mean but about which group tends to have higher values, the Mann-Whitney test; for three or more variants, ANOVA. All of them assume independent groups. And the rule “z-test above 30 cases, t-test below” is a common misconception to forget.

What we will cover:


The three questions that choose the test

The first question is the most important, and it is also the easiest: what are we measuring?
If for each visitor the answer is “yes or no” (did they convert, did they click, did they leave their contact), we are counting successes out of attempts, and the quantity we compare is a proportion. If instead for each visitor we have a number that can take many values (seconds on the page, euros spent, pages viewed), we are comparing continuous values, and usually their means. The two families have different tests, because chance behaves differently in the two worlds.

The second question is how many variants we are comparing. With two there are tests designed for the purpose; with three or more we need tests that look at all of them together, because comparing them pair by pair multiplies the risk of finding differences that do not exist (it is the problem of multiple comparisons).

The third question changes with the family. For proportions: are the expected counts in each cell large enough (the traditional threshold is 5)? For continuous values: is the mean what we care about, or is it enough to know whether one variant tends to have higher values than the other? With very skewed metrics, like revenue, the two questions can have different answers, and the first has to be settled before choosing the test.

The whole reasoning rests on an assumption that is worth declaring: the groups are independent, and each observation corresponds to the unit we assigned at random to the variants (usually the user). If the same user appears several times, or sees both variants, the sessions are not independent observations, and procedures different from the ones in this guide are needed.

The diagram for choosing the test of an A/B test. We start from what we measure: with proportions, if the expected counts are at least 5 in every cell we use the two-proportion z-test for two variants and the chi-square for three or more, otherwise Fisher’s exact test; with continuous values, ANOVA for three or more variants and, for two, Welch’s t-test if we care about the mean, the Mann-Whitney test if we care about which variant tends to have higher values.

N.B. the diagram leaves out a question that many manuals put first: “does the sample exceed 30 cases?”. The rule “z-test above 30, t-test below” is a common misconception to forget. For means, when we do not know the population variance (that is, always, in practice), we use the t-test, and sample size alone does not decide between z and t: with a lot of data the two give practically the same result. The reliability of the t-test depends rather on its assumptions and on the shape of the data: with little, very skewed data, or with extreme values, even the t-test can be wrong. For proportions, the condition that matters is not the number of visitors but the number of expected successes and failures: a thousand visits with two conversions are “few”, even though a thousand is more than thirty.

Let us now see each test at work.


The two-proportion z-test

It is the most used test in A/B tests, because the most used metric is a conversion rate. Let us take an example: two versions of a landing page, 4,000 visits each. A converts 200 times (5.0%), B 252 times (6.3%).

The idea of the test is to compare the observed difference between the two rates with the difference that chance alone would produce if the two variants were in reality identical. In words: we compute the common conversion rate (\( \hat{p} \), all conversions divided by all visits), from there the typical fluctuation we expect between two samples of those sizes (the standard error), and we express the observed difference in standard-error units: that is z.

\( \displaystyle z = \frac{\hat{p}_B – \hat{p}_A}{\sqrt{\hat{p}\,(1 – \hat{p})\left(\frac{1}{n_A} + \frac{1}{n_B}\right)}} \)

where \( \hat{p}_A \) and \( \hat{p}_B \) are the rates of the two variants, \( n_A \) and \( n_B \) the visits, and \( \hat{p} \) the common rate.

I do the calculation in R, first by hand and then with the ready-made function:

conversions   <- c(A = 200, B = 252)
visits        <- c(A = 4000, B = 4000)

p      <- conversions / visits                 # 0.050 and 0.063
p_pool <- sum(conversions) / sum(visits)       # 0.0565
se     <- sqrt(p_pool * (1 - p_pool) * (1 / visits[1] + 1 / visits[2]))
z      <- (p[2] - p[1]) / se                   # 2.52
2 * pnorm(-abs(z))                             # p-value: 0.0118

prop.test(conversions, visits, correct = FALSE)
# X-squared = 6.341, df = 1, p-value = 0.0118

The z is 2.52 and the p-value 0.012: with a 5% threshold the difference is statistically significant. prop.test() with correct = FALSE returns the same p-value, even though it prints a statistic called X-squared: it is the square of z (2.52² = 6.34), because with two groups the z-test and the uncorrected chi-square are the same test written in two ways. (Without correct = FALSE, prop.test() applies Yates’s continuity correction, which is more cautious, and the p-value grows a little.)

The confidence interval of the difference runs from 0.3 to 2.3 percentage points (prop.test prints it as A − B, that is with the minus sign). It is more informative than the p-value alone: it tells us that B converts better, but that the real gain could be as modest as three tenths of a point.

A warning: the variance of a proportion is not a number to choose. Every visit, whether it converts or not, has variance p(1 − p); the conversion rate calculated on n visits has variance p(1 − p)/n. A mistake one sees from time to time is comparing a rate with the historical one using a “known variance” taken from who knows where: for example, checking whether going from 5% to 6% on 500 users is an improvement, with a variance of 0.0025, gives z = 4.47, a result that looks rock solid. With the right variance, 0.05 × 0.95 / 500, z is 1.03 and the p-value (one-sided, because the question is whether there is an improvement) is 0.15: no evidence of improvement. The same data, two opposite conclusions, and what makes the difference is a formula.


The chi-square: when there are more than two variants

With three variants of a page the z-test is no longer enough, and the temptation is to run three pairwise tests. The chi-square test answers the right question with a single test: do the three variants all convert in the same way?

Let us take an example: three variants of a call to action, 3,000 visits each, with 150, 165 and 201 conversions (5.0%, 5.5% and 6.7%). I build the table in R and run the test:

tab3 <- rbind(conversions = c(A = 150, B = 165, C = 201),
              no          = c(A = 2850, B = 2835, C = 2799))
chisq.test(tab3)
# X-squared = 8.4742, df = 2, p-value = 0.01445

The chi-square compares the observed frequencies with the ones we would expect if the variants were identical (here 172 conversions each, that is one third of the total). With a p-value of 0.014 we conclude that at least one variant converts differently from the others. Which one, the test does not say: to find out we need pairwise comparisons corrected for their multiplicity, for example with pairwise.prop.test(), which applies Holm’s correction automatically.


Fisher’s exact test: when the numbers are small

The z-test and the chi-square work well when the counts are large enough: the practical rule asks for at least 5 expected cases in each cell of the table. On B2B sites, where one counts contacts and not clicks, this condition often fails.

Let us take an example: a landing page for requesting a quote, 90 visits per variant. A collects 1 contact (1.1%), B 8 (8.9%). If the variants were identical we would expect 4.5 contacts each: below the threshold. I run Fisher’s exact test in R, and for comparison the chi-square:

lead <- rbind(A = c(lead = 1, no = 89),
              B = c(lead = 8, no = 82))
fisher.test(lead)
# p-value = 0.03444

chisq.test(lead)
# X-squared = 4.2105, df = 1, p-value = 0.04017
# Warning in chisq.test(lead): Chi-squared approximation may be incorrect

Fisher’s test uses no approximations: it computes the exact probability of observing a table like this one (or more extreme) if the variants were equal, and gives a p-value of 0.034. The chi-square arrives at a similar number, but R itself warns us that with counts this small its approximation is not reliable. With few successes, Fisher’s test is the safe choice; with large counts, the chi-square and Fisher generally give very similar results.

N.B. 9 contacts in total are very few. The test says that so clear a difference is hardly just chance, but the confidence interval of the odds ratio (from 0.003 to 0.90) is extremely wide: we still know very little about the size of the effect.


Student’s t-test

Now let us move to continuous values. The two-sample t-test compares the means of two groups, and its classic version, Student’s, assumes that the two groups have the same variance.

Let us take an example: the time on the page (in seconds) of 25 sessions for each of two variants of an article. I simulate the data in R and run the test, with var.equal = TRUE to ask for exactly Student’s version:

set.seed(2026)
time_A <- round(rnorm(25, 95, 18))
time_B <- round(rnorm(25, 108, 18))

t.test(time_B, time_A, var.equal = TRUE)
# t = 5.1168, df = 48, p-value = 5.409e-06
# 95 percent confidence interval: 14.71499 33.76501
# mean of x mean of y: 113.32 89.08

Variant B keeps readers for an average of 113 seconds against 89, and the difference (24 seconds, with an interval from 15 to 34) is clearly significant. Here the assumption of equal variances is reasonable: we simulated the data with the same standard deviation, and those of the two samples (17.2 and 16.3) are close. With real data, two similar standard deviations do not prove that the variances are equal: it is one more reason to start from Welch.

N.B. without var.equal = TRUE, R does not run Student’s test but Welch’s: it is a deliberate choice, and the next section explains why.


Welch’s t-test: the starting choice

Welch’s t-test is a variant of Student’s t-test that does not assume equal variances. It corrects the standard error and the degrees of freedom according to the variances of the two groups, and when the variances really are similar it gives practically the same result as Student’s. For this reason R uses it by default, and it is the most sensible starting choice.

The difference becomes important when the groups have different sample sizes and different variances. Let us take an example: the average order value in a test on a new cart page, with 40 orders for variant A and only 22 for B, which however are much more variable (a standard deviation of about 20 euros against 9). I simulate the data and compare the two tests:

set.seed(7)
order_A <- round(rnorm(40, 62, 9))
order_B <- round(rnorm(22, 71, 24))

t.test(order_B, order_A)                     # Welch, the default
# t = 2.0461, df = 26.441, p-value = 0.05081

t.test(order_B, order_A, var.equal = TRUE)   # Student
# t = 2.4672, df = 60, p-value = 0.01649

The means are 73.5 and 64.4 euros. Student’s test gives a p-value of 0.016 and seems to close the case; Welch’s gives 0.051. Student is too sure of itself, because it treats the small and very variable group as if it were as stable as the large one. More than the p-value, what matters is the interval Welch returns for the difference: from −0.03 to 18.3 euros. B could be worth almost nothing more than A, or almost twenty euros an order; Student, with its interval from 1.7 to 16.5 euros, would have made us believe we know more.


The Mann-Whitney test: when the mean deceives

Some web metrics have a shape very far from the bell. Revenue per session is the typical example: many sessions with a few euros, a few with many, and now and then an enormous order that on its own moves the mean.

Let us take an example: the revenue of 30 sessions with a purchase for each variant, with strongly skewed data and, in group A, an order of 640 euros. I simulate the data in R:

set.seed(11)
revenue_A <- round(rlnorm(30, log(20), 1.0), 1)
revenue_B <- round(rlnorm(30, log(32), 1.0), 1)
revenue_A[1] <- 640                          # an enormous order in group A

c(mean(revenue_A), mean(revenue_B))          # 40.0 and 43.4
c(median(revenue_A), median(revenue_B))      # 15.8 and 33.4

t.test(revenue_B, revenue_A)                 # Welch: p-value = 0.8784
wilcox.test(revenue_B, revenue_A)            # Mann-Whitney: W = 645.5, p-value = 0.003938

The means are almost equal, 40 and 43 euros, and Welch’s t-test sees no difference (p = 0.88). But the medians tell another story: 16 euros against 33. The 640-euro order on its own inflates the mean of group A and hides the fact that, session by session, B tends to take in more.

The Mann-Whitney test (which R calls wilcox.test, because it is the same thing as the Wilcoxon test for two independent samples) does not work on the values but on their ranks, that is, on the position of each value in the ranking of all of them: the 640-euro order counts as “the largest”, not as 640. With a p-value of 0.004 it says that the values of B tend to be higher than those of A.

Careful, though, about what it measures. The Mann-Whitney test does not compare means, but the tendency of one group to have higher values than the other. If for the business what counts is total revenue, that is the mean, that 640-euro order is real and must not be ignored: in that case we need more data, or methods designed for the means of skewed data such as the bootstrap. We discuss it in the article on parametric and non-parametric tests.


ANOVA: three variants or more

With three or more variants and a continuous metric, the equivalent of the chi-square is analysis of variance, ANOVA. It compares the variability between the variants with the variability within the variants, and answers with a single test the question: are the means all equal? Like Student’s t-test, in its classic version it assumes independent groups with similar variances and data not too far from normality; when the variances differ there is Welch’s ANOVA (oneway.test()), followed by pairwise comparisons that do not assume equal variances (Games-Howell, for example) instead of Tukey.

Let us take an example: three variants of a page, 20 sessions each, and the time on the page. I simulate the data and run the ANOVA, followed by Tukey’s test to understand which variants differ:

set.seed(99)
time3   <- c(round(rnorm(20, 95, 20)), round(rnorm(20, 104, 20)), round(rnorm(20, 116, 20)))
variant <- factor(rep(c("A", "B", "C"), each = 20))

tapply(time3, variant, mean)    # A 87.95  B 104.80  C 105.75
summary(aov(time3 ~ variant))
#             Df Sum Sq Mean Sq F value  Pr(>F)
# variant      2   4011  2005.6   5.473 0.00669
# Residuals   57  20886   366.4

TukeyHSD(aov(time3 ~ variant))
#      diff       lwr      upr     p adj
# B-A 16.85  2.283331 31.41667 0.0196284
# C-A 17.80  3.233331 32.36667 0.0129265
# C-B  0.95 -13.616669 15.51667 0.9865163

ANOVA says that the means are not all equal (F = 5.47, p = 0.007), and Tukey specifies where: B and C keep readers longer than A (about 17-18 seconds more), while between B and C no significant difference emerges. Which does not mean they are equivalent: the interval for their difference runs from −13.6 to +15.5 seconds, too wide to rule out a difference that matters. The share of variability explained by the variant (η²) is 0.16: a large effect by the conventions, but with 20 sessions per group the intervals stay wide.


And regression?

Student’s t-test and classic ANOVA are special cases of a single tool, the linear model. I verify it in R on the data of Student’s test, writing the comparison as a regression with one variable that indicates the variant:

d <- data.frame(time    = c(time_A, time_B),
                variant = factor(rep(c("A", "B"), each = 25)))
summary(lm(time ~ variant, data = d))$coefficients
#             Estimate Std. Error   t value     Pr(>|t|)
# (Intercept)    89.08   3.349786 26.592743 2.235587e-30
# variantB       24.24   4.737313  5.116825 5.409094e-06

The coefficient of variant B is 24.24 seconds, that is the difference between the means, and its t (5.12) and its p-value are identical to those of Student’s test. (With Welch the equivalence is not so direct: we need standard errors that take the different variances into account.) The advantage of regression is that we can add other variables: the device, the traffic source, the day of the week. In a randomised test they serve to reduce noise and make the estimate more precise; when the variants were not assigned at random they help, but on their own they are not enough to remove the hidden differences between the groups. The next step is multiple regression.


An overall look

What we measureVariantsDataTestIn R
proportions2expected counts ≥ 5two-proportion z-testprop.test(..., correct = FALSE)
proportions3 or moreexpected counts ≥ 5chi-squarechisq.test()
proportions2 or moreexpected counts < 5Fisher’s exact testfisher.test()
continuous values2similar variancesStudent’s t-testt.test(..., var.equal = TRUE)
continuous values2the general case, the mean is what we care aboutWelch’s t-testt.test()
continuous values2which group tends to have higher values is what we care aboutMann-Whitneywilcox.test()
continuous values3 or moresimilar variancesANOVA (+ Tukey)aov(), TukeyHSD()
continuous values3 or moredifferent variancesWelch’s ANOVAoneway.test()

Whichever test we choose, the same rules of the game always apply: the test is decided before looking at the data (choosing it afterwards, trying several until one gives the answer we hope for, is a form of p-hacking), the duration of the test is fixed in advance instead of stopping at the first favourable result (the peeking problem), and a small p-value says that a difference is hardly chance, not that it is large. For quick calculations on two conversion rates there is also the significance calculator.


Frequently asked questions

Which test should we use to compare two conversion rates?
The two-proportion z-test, which in R we run with prop.test(..., correct = FALSE). If at least one cell of the table has an expected count below 5 (conversions or non-conversions), Fisher’s exact test (fisher.test()) is better.

What is the difference between Student’s t-test and Welch’s t-test?
Student’s t-test assumes that the two groups have the same variance; Welch’s does not. When the variances are similar they give the same result; when they differ, especially with groups of different sizes, Student’s can be too optimistic. That is why Welch is the starting choice, and it is the default in R.

When should we use the Mann-Whitney test instead of the t-test?
When the question is whether one variant tends to have higher values than the other, more than how much the means differ: this happens often with very skewed data or with extreme values. If instead it is the mean that matters (total revenue, for example), skewness alone is not enough to justify Mann-Whitney: we need more data or methods such as the bootstrap.

Do we need a sample of at least 30 to use the z-test?
No, it is a common misconception: sample size alone does not decide between z and t. For means with unknown variance we use the t-test, whose reliability depends on its assumptions and on the shape of the data more than on a threshold. For proportions the condition that matters is having enough expected successes and failures (at least 5 per cell), not enough visitors.

Which test should we use with three or more variants?
For proportions the chi-square (or Fisher’s exact test, if some expected count is below 5); for means ANOVA, or Welch’s ANOVA (oneway.test()) if the variances differ. Then pairwise comparisons corrected for multiplicity (for example pairwise.prop.test(), Tukey’s test or, with unequal variances, Games-Howell) to understand which variants differ.


Try it yourself

The code of every section can be run again and modified. Three exercises to put the choice of test to the test:

  1. Let us make the z-test small. We revisit the example of the two landing pages with 400 visits per variant instead of 4,000, and nearly the same rates (20 and 25 conversions). Is the result still significant? What does the confidence interval say?
  2. Let us remove the 640-euro order. In the Mann-Whitney example we delete the first value of group A and redo both Welch’s t-test and the Mann-Whitney test: how much do the two p-values change? Which of the two tests was more sensitive to that order?
  3. Let us make the variances equal. In the Welch example we simulate group B with standard deviation 9, like group A, and compare Student and Welch again: how close do the two p-values come?

A hint: in all three cases the question is the same. Not “which test gives the best result”, but “which test makes the right assumptions for these data”.


Choosing the right test solves half of the problem: it tells us whether a difference is credible. The other half comes before, and concerns design: how many visitors do we need for the test to have a reasonable chance of noticing a difference we really care about? That is the subject of effect size and power analysis, the natural next step to continue from.


Further reading

For anyone who wants to get serious with online experiments, Trustworthy Online Controlled Experiments by Kohavi, Tang and Xu is the reference book on A/B testing, written by those who led experimentation at Microsoft and Google: it provides valuable discussion of precisely the skewed metrics such as revenue per user. For the statistical foundations of the tests, The Art of Statistics by David Spiegelhalter is the clearest way to learn what a test really assumes and what it does not tell us, with no formulas getting in the way.

paolo

Recent Posts

The month the form stopped counting: finding and reconstructing tracking gaps

The September report is almost ready, and there is one number that does not fit:…

4 days ago

How many pages should we protect in a site migration? The Lorenz curve and traffic thresholds

The redesign is approved, the launch date is set, and sooner or later the question…

4 days ago

When the yardstick changes: structural breaks in SEO metrics

The annual report lands on the desk with good news on the front page: the…

5 days ago

Are Google’s AI answers taking our clicks? Measuring it with expected CTR and a control group

A site's technical guides climb in the search results, show up more often, and yet…

6 days ago

When clicks vary more than expected: overdispersion

Clicks and sessions vary more than Poisson allows: how to detect it in R and…

1 week ago

Two years of data and a seasonality to remove: doing year-over-year right

An annual report often ends with a single line: clicks from Google fell 11.8% compared…

1 week ago