T-Test with R examples for SEO: comparing CTR, conversions and metrics

In the article on hypothesis testing we saw how to check whether a sample mean is compatible with a hypothesis about the population. In the examples we proposed, however, we knew the population standard deviation, sigma — which allowed us to use the normal distribution and the Z-score.

In practice this is a rather rare case. If we do not know the population sigma, or if we are working with small samples, we need a different distribution: the t distribution, or Student’s distribution.

A Brief Historical Digression

In the early 1900s, the chemist and statistician William Sealy Gosset, employed at the Guinness brewery (and a collaborator of Karl Pearson), discovered that when working with very small samples, the distributions of the mean differed significantly from the normal. As the sample size changed, the shape of the distribution changed too — and as the sample grew, it gradually approximated the normal.

Unable to reveal his identity so as not to benefit competitors, he published under the pseudonym “Student”. This is why distributions for small samples are now known as Student’s t distributions.

---

The t distribution is symmetric about zero, but it is “flatter” than the standardised normal: a greater portion of its area lies in the tails.

A larger sample causes the t distribution to approximate the normal ever more closely. The differences are greatest when we have few degrees of freedom.

What do we mean by degrees of freedom? The number of sample values that have the “freedom” to vary without changing the sample mean. In practice, for a one-sample test:

\( df = n – 1 \\ \)

where n is the sample size.

The procedure for hypothesis testing with the t distribution largely mirrors what we saw with the normal. We state the null hypothesis H₀ and the alternative hypothesis Hₐ, then compute the test statistic:

\( t = \frac{\bar{x} – \mu}{\frac{s}{\sqrt{n}}} \\ \)

where \( \frac{s}{\sqrt{n}} \) is the estimated standard error (also denoted SE).

An Example Is Worth a Thousand Explanations

A light bulb manufacturer claims that its product has a mean lifespan of at least 4200 hours. We take a sample of n = 10 bulbs and find a sample mean of 4000 hours, with a sample standard deviation of 200 hours.

\( n = 10 \quad \bar{x} = 4000 \quad s = 200 \\ \)

We set up the test:

\( H_0 : \mu \ge 4200 \qquad H_a : \mu < 4200 \)

We choose a significance level α = 0.05 (corresponding to a 95% confidence level). In the t distribution critical value table, we look up the value for 9 degrees of freedom and α = 0.05 (one-tailed). The critical value is 1.833. We will reject H₀ if the computed t is less than −1.833.

The standard error:

\[ SE = \frac{s}{\sqrt{n}} = \frac{200}{\sqrt{10}} = 63.3 \]

And the t value:

\[ t = \frac{\bar{x} – \mu}{SE} = \frac{4000 – 4200}{63.3} = -3.16 \]

The t value falls in the critical region: we reject H₀. There is sufficient statistical evidence to conclude that the mean lifespan is less than the claimed 4200 hours.

The p-Value

We can also evaluate the hypothesis by asking: what is the probability of obtaining a test statistic value equal to or more extreme than the one observed, assuming H₀ is true? This probability is the p-value.

Using R or a calculator we obtain t = −3.16 and p = 0.00578. The commands for the calculators:

• TI-83: 2nd → DISTR → tcdf(−1E99, −3.16, 9)
• Casio FX (STAT mode): DIST → t → tcd with Lower −1E99, Upper −3.16, df 9

There is just a 0.578% probability of observing a result like ours if H₀ were true. p < 0.05: we reject H₀ in favour of the alternative hypothesis.

Confidence Interval

When we reject a hypothesis, it is useful to estimate the true mean. In our example, we rejected the manufacturer’s claim. But how long do the bulbs actually last?

The confidence interval needs three things: the sample mean, the standard error, and the critical value.

\( \text{Margin of Error} = t_{\text{critical}} \times SE \\ \)

In our case: ME = 1.833 × 63.3 ≈ 116. The 95% confidence interval is:

\( 4000 \pm 116 = [3884,\ 4116] \\ \)

The value of 4200 hours falls outside the interval, confirming the test result.

The t-Test in R

R lets us perform the test simply and completely. We prepare a vector of 10 measurements with a mean of 4000 and use t.test():

bulb_life <- c(4100, 3900, 3800, 4200, 4000,
               4100, 3900, 3800, 4200, 4000)
t.test(bulb_life, mu = 4200, alternative = "less")

 One Sample t-test

data:  bulb_life
t = -3.1623, df = 9, p-value = 0.005788
alternative hypothesis: true mean is less than 4200
95 percent confidence interval:
     -Inf 4115.791
sample estimates:
mean of x
     4000

A Practical Case: Do Rewritten Title Tags Perform Better?

Let us bring the tool onto the ground we actually care about. Suppose we have rewritten the title tags of ten pages and we want to know whether their CTR has shifted from the 3.0% that was the group’s historical average. Ten pages are few, and we do not know the population variance: this is exactly where the t distribution is at its best.

Here are the measured CTRs, as percentages:

ctr <- c(3.8, 4.2, 3.1, 4.9, 3.5, 4.1, 2.9, 4.6, 3.7, 4.4)
t.test(ctr, mu = 3.0)

 One Sample t-test

data:  ctr
t = 4.5276, df = 9, p-value = 0.001431
95 percent confidence interval:
 3.460337 4.379663
sample estimates:
mean of x
     3.92

Let us read the output. The sample mean is 3.92%, almost a full point above the reference. The t value is 4.53 with 9 degrees of freedom: a measure of how many estimated standard deviations the observed mean lies from the hypothesised 3.0%. The p-value is 0.0014, well below the 0.05 threshold: a gap like this, if the true mean were really 3.0%, we would see by pure chance a little more than once in a thousand times. The rewrite produced an effect that is hard to dismiss as noise.

The 95% confidence interval runs from 3.46% to 4.38% and does not contain 3.0%. It is the same conclusion as the test, seen from another angle — not just “the effect exists” but “here is the plausible range.”

A word of caution: with just ten pages that range is still almost a full percentage point wide; more data would narrow it.

Effect Size: Cohen’s d

The p-value tells us whether the difference is statistically significant. To understand how large it is, we compute Cohen’s d for a one-sample test:

\( d = \frac{\bar{x} – \mu}{s} \\ \)

d <- (3.92 - 3.0) / sd(ctr)
d

[1] 1.432

Rule of thumb: d ≈ 0.2 is small, 0.5 medium, 0.8 large. Our d = 1.43 is a large effect — the difference is not only statistically significant but also substantial.

Try It Yourself

We have seven landing pages and want to know whether their conversion rate departs from the 2.5% target. Here are the values, as percentages:

cr <- c(2.8, 3.1, 2.4, 3.5, 2.9, 3.3, 2.7)
t.test(cr, mu = 2.5)

Run the test, read t, df, p-value and confidence interval. The mean is 2.96%, t is 3.24 on 6 df, the p-value is 0.018, and the 95% CI runs from 2.61% to 3.30%. Since it does not include 2.5%, the seven pages beat the target in a way that is hard to attribute to chance.

FAQ

When should I use the t distribution instead of the normal?
Whenever the population standard deviation is unknown and we estimate it from the sample. In practice: almost always. The normal with known sigma is more of a textbook case than a real situation.

Does the t distribution work with very small samples?
Yes, provided the data are not strongly skewed. For n ≥ 30, the t and the normal are practically indistinguishable. For n < 15, check normality with shapiro.test().

What are degrees of freedom?
In a one-sample test, df = n − 1. It is the number of values that can vary freely given a fixed mean. The more df we have, the closer the t approximates the normal.

---

Student’s t distribution gave us a way to reason when the data are few and the population variance is unknown — the situation, as we have seen, we almost always find ourselves in. So far we have compared a single group against a reference value. The question that arises naturally is: what if there were two groups to compare — version A and version B of a page, the before and after of an intervention? That is the next step: the two-sample t-test.

---

Further Reading

For a comprehensive treatment of the t distribution, hypothesis testing, and the full machinery of statistical inference, Statistica by Newbold, Carlson and Thorne provides a rigorous yet accessible walkthrough of the theory and practice, from small-sample tests to confidence interval construction. Amazon