Hypothesis testing: how to tell if a result is due to chance

We are running an A/B test on two title tags for a product page. Variant A gets 127 clicks out of 4,000 impressions (CTR 3.17%), variant B gets 152 clicks out of 4,000 impressions (CTR 3.80%). A difference of 25 clicks. Is this a real lift, or just random noise?

Questions like this — is this difference due to chance? — arise constantly in any data-driven activity. Hypothesis testing is the statistical procedure that allows us to answer with a methodical approach, distinguishing results that reflect a real phenomenon from those that are merely sampling noise.

What we’ll cover:

• What is a hypothesis test
• Null and alternative hypotheses
• Type I and Type II errors
• One-tailed vs two-tailed tests
• The five-step procedure
• An SEO example: A/B test on CTR
• The p-value
• Statistical power
• Sample size
• Try it yourself
• Further reading

---

What is a hypothesis test

In everyday life we often have to make decisions based on incomplete information. A new title tag generates a few more clicks: is the title actually better, or was it simply a lucky week?

Hypothesis testing is a statistical procedure that allows us to pose a question based on sample information, to reach a data-driven decision.

The difference between probability and inference is the starting point:

• If we know the population parameters and want to know the probability of obtaining a particular result, we are in the realm of probability.
• If from a sample we try to infer the population values, we are in the territory of inference. Hypothesis testing is the main tool of statistical inference.

Null and alternative hypotheses

In hypothesis testing we always weigh two hypotheses. The status quo is called the null hypothesis, denoted H₀. It is the hypothesis that there is no effect, no difference — that everything is due to chance.

What we do is test the null hypothesis against an alternative hypothesis, denoted Hₐ — which is, in general, the hypothesis we believe in or want to prove.

We then choose a significance level or alpha level, denoted α. The common standard is α = 0.05, which corresponds to a 95% confidence level. Based on the alpha level we determine one or more critical regions: the areas of the sampling distribution where, if our test statistic falls, we reject the null hypothesis.

If the value we obtain falls in a critical region, we have sufficient statistical evidence to reject H₀ in favour of H₁.

Type I and Type II errors

The test result, of course, does not constitute a certainty. Two types of error are possible:

• Type I error: rejecting the null hypothesis when it is actually true. The probability of this error is exactly α, the significance level we set.
• Type II error: failing to reject the null hypothesis when it is actually false. The probability of this error is denoted β, and it depends on several factors (sample size, true effect size, α level).

The clearest way to visualise the two types of error is this table:

Calculating the probability of a Type II error is not as straightforward as for a Type I error. We will tackle it shortly, when we discuss statistical power.

One-tailed vs two-tailed tests

The test can be one-tailed, when the alternative hypothesis is directional (e.g. “the mean is greater than” or “the mean is less than”), or two-tailed, when the alternative hypothesis is simply “different from”:

In a one-tailed test, the entire critical region is concentrated on one side of the distribution. In a two-tailed test, the critical region is split into two symmetric tails, each with area α/2.

The choice between one-tailed and two-tailed must be made before collecting data, based on the research question. If we only want to know whether B is different from A, we use two tails. If we want to know whether B is better than A, we use a one-tailed (right) test.

The five-step procedure

The operational sequence of hypothesis testing can be summarised in five steps:

1. State the null hypothesis H₀ and the alternative hypothesis Hₐ.
2. Set the significance level α (usually 0.05).
3. Choose the appropriate distribution: normal (z) or Student’s t.
4. Collect the data and compute the test statistic.
5. Draw conclusions: reject or fail to reject H₀.

The choice between normal and t distribution depends on:

• Do we know the population standard deviation σ? If yes, use the normal.
• Is the sample large enough (n > 30)? If yes, we can use the normal even without σ (thanks to the central limit theorem).
• If the answer to both is no, use the Student’s t distribution, which we discuss in the dedicated article.

An SEO example: A/B test on CTR

Let’s return to the opening example. We ran an A/B test on two title tags:

• Variant A: 127 clicks out of 4,000 impressions → CTR 3.17%
• Variant B: 152 clicks out of 4,000 impressions → CTR 3.80%

The difference is 25 clicks, or 0.63 percentage points. Is this a real effect or noise?

Let’s follow the five steps.

1. Hypotheses

• H₀: the CTR of B equals the CTR of A (difference = 0)
• Hₐ: the CTR of B is greater than the CTR of A (difference > 0, one-tailed test)

2. Significance level

We set α = 0.05 (the standard level).

3. Distribution

For proportions, the sampling distribution of the difference under H₀ is approximately normal (the CLT guarantees this with n = 4,000). We use the z-test for two proportions.

4. Computation

In R:

# Data
n <- 4000
clicks_A <- 127; clicks_B <- 152

# Pooled proportion under H0
p0 <- (clicks_A + clicks_B) / (2 * n)       # 0.034875

# Standard error of the difference
se <- sqrt(p0 * (1 - p0) * (1/n + 1/n))     # ~0.00410

# Observed z-score
z_obs <- (clicks_B/n - clicks_A/n) / se     # ~2.22
z_obs

Output:

[1] 2.224

The observed z value is 2.22 — the difference between the two CTRs lies more than two standard deviations away from the value expected under H₀ (zero). The figure shows the position of the observed value relative to the sampling distribution:

The red area to the right of the dashed blue line is the p-value: the probability of observing a difference equal to or greater than the one obtained, assuming H₀ is true.

5. Conclusion

The p-value is approximately 0.013, below the α = 0.05 threshold. We reject the null hypothesis: the difference is statistically significant. Variant B produces a higher CTR than variant A, and chance is not a plausible explanation.

# Compute the p-value (one-tailed right test)
p_val <- 1 - pnorm(z_obs)
p_val

Output:

[1] 0.0131

We can also verify the result with R’s built-in function:

prop.test(c(127, 152), c(4000, 4000), alternative = "greater", correct = FALSE)

The p-value

The p-value is the probability of obtaining a result equal to or more extreme than the one observed, assuming the null hypothesis is true. A small p-value indicates that the observed result is unlikely under H₀, leading us to reject it.

Critical z-values for common significance levels:

Level α	One-tailed	Two-tailed
0.05	±1.645	±1.960
0.01	±2.326	±2.576

Statistical power

In hypothesis testing, power is the probability of correctly rejecting the null hypothesis when it is false. In formula: power = 1 − β, where β is the probability of a Type II error.

Power depends on four factors:

• Effect size: the larger the real effect, the easier to detect
• Sample size: more data means higher power
• Significance level α: lowering α reduces power
• Data variability: less variability means higher power

Let’s return to our CTR example. Our test with n = 4,000 per variant detected a difference of about 0.63 percentage points. But what if the real effect were smaller — say, 0.3 percentage points? How much power would we have?

The curve shows power as the alternative proportion (μₐ) varies. With n = 4,000 and α = 0.05, to detect a difference corresponding to μₐ = 4.5% (a 1.5 percentage point lift), power is about 76%. For our observed effect (3.80%, about 0.8 points above 3%), power is around 50% — roughly a coin flip.

To increase power, we need a larger sample:

# Power analysis for different sample sizes
library(pwr)
pwr.2p.test(h = ES.h(0.038, 0.0317),   # effect size for proportions
            n = 4000, sig.level = 0.05, alternative = "greater")

Output:

     Difference of proportion power calculation

              h = 0.0497
              n = 4000
      sig.level = 0.05
          power = 0.5346
    alternative = greater

Our test’s power is about 53%. If the real effect is this size, we have nearly a 1-in-2 chance of missing it. To reach 80% power (the commonly accepted standard), we would need to almost triple the sample.

See the full article on effect size and power analysis for a comprehensive treatment.

Sample size

Before launching a test, we can determine the required sample size by specifying:

1. The expected value under H₀
2. The alternative value we consider important
3. The significance level α
4. The desired power (usually 80%)
5. Data variability

In R, using pwr:

# How many impressions to detect a lift from 3.2% to 3.8% at 80% power?
pwr.2p.test(h = ES.h(0.038, 0.032),
            power = 0.80, sig.level = 0.05,
            alternative = "greater")

Output:

     Difference of proportion power calculation

              h = 0.0336
              n = 13749.56
      sig.level = 0.05
          power = 0.80
    alternative = greater

We need about 13,750 impressions per variant to have 80% power to detect a 0.6 percentage point CTR difference. This is exactly the kind of analysis that should be done before launching a test, not after seeing the results. You can also use our interactive sample size calculator for these evaluations.

Try it yourself

An A/B test on a landing page collected the following data:

• Version A: 84 conversions out of 3,200 sessions (rate 2.62%)
• Version B: 106 conversions out of 3,200 sessions (rate 3.31%)

1. Formulate the null and alternative hypotheses for a one-tailed (right) test.
2. Compute the z-score and p-value in R (hint: prop.test).
3. Is the difference statistically significant at α = 0.05?
4. With 3,200 sessions per variant, what power does the test have to detect a 0.7 percentage point effect? (hint: pwr.2p.test)
5. How many sessions would be needed for 80% power?

Further reading

For a rigorous and comprehensive treatment of hypothesis testing, with exercises and all formal steps, the standard reference is Statistics for Business and Economics by Newbold, Carlson and Thorne. For a hands-on, application-focused approach directly relevant to online experiments, see Trustworthy Online Controlled Experiments by Kohavi, Tang and Xu.

This article is part of the «Statistical Inference» path, the curated guide to articles on tests, confidence intervals, and data-driven decision making.