The Two-Sample t-Test: How to Compare Two Groups (and Get It Right)

In the article on hypothesis testing we saw how to check whether a sample mean is compatible with a hypothesis about the population. But in practice the most common case is comparing two groups.

Does a new drug lower blood pressure more than a placebo? Does a training course improve test scores? Does a new landing page produce a higher CTR? In each case we have two groups (treatment vs control, before vs after, variant A vs B) and we want to know whether their means differ. The right tool is the two-sample t-test.

To get our bearings right away:

Scenario	Test
Two different groups (e.g. men vs women)	Independent-samples t-test
Before and after on the same subject	Paired t-test
Two page variants (A/B test)	Independent-samples t-test
Blood pressure before and after therapy	Paired t-test

The choice between these two tests is not technical: it is conceptual. If the data are paired and we use an independent test, we lose power because we throw away the pairing information. If we use a paired test on independent data, we violate the independence assumption and risk finding significance where there is none.

---

When to Use the Independent-Samples t-Test?

Assumptions

For the test to be valid, several conditions must be met:

• A random sample from each population
• Observations within each sample are independent
• The two samples are independent of each other
• The observations come from approximately normal populations, or the sample size is large enough

The t-test is fairly robust to moderate deviations from normality, especially with large samples (n ≥ 30 per group). It becomes problematic with very small samples and strongly skewed distributions.

Hypotheses:

\( H_0 : \mu_1 = \mu_2 \\ \) \( H_a : \mu_1 \neq \mu_2 \\ \)

In other words: $\( H_0 : \mu_1 – \mu_2 = 0 \)\( , \)\( H_a : \mu_1 – \mu_2 \neq 0 \)$

The Test Statistic

It all starts with the difference between the observed means. We compute t as:

\( t = \frac{(\bar{x}_1 – \bar{x}_2) – (\mu_1 – \mu_2)}{SE_{(\bar{x}_1 – \bar{x}_2)}} \\ \)

Where:

• $\( \bar{x}_1 – \bar{x}_2 \)$ is the observed difference between sample means
• $\( \mu_1 – \mu_2 \)$ is the hypothesised difference under H₀ (zero)
• $\( SE_{(\bar{x}_1 – \bar{x}_2)} \)$ is the standard error of the difference

The standard error is calculated as follows — and this is the first important point:

\( SE_{(\bar{x}_1 – \bar{x}_2)} = \sqrt{\frac{s^2_1}{n_1} + \frac{s^2_2}{n_2}} \\ \)

N.B.: this is the Welch formula — it does not assume equal variances between the two groups, and it is more robust when sample sizes or variabilities differ. By default, t.test() in R uses Welch’s t-test, not Student’s classic t-test. It is the safest choice in practice: unless there is a strong reason to assume equal variances, Welch is preferable.

If we know for certain the variances are similar (balanced samples with close variances), we can use var.equal = TRUE, which corresponds to the classic pooled-variance t-test. For a more systematic comparison of the two approaches, see the Guide to Statistical Tests.

Degrees of freedom? I will let R calculate them (it uses the Welch-Satterthwaite approximation, which is a bit cumbersome by hand). Alternatively, a conservative approach: df = min(n₁, n₂) − 1.

Checking Assumptions in R

Before running the test, we check normality:

# Shapiro-Wilk test for each group
shapiro.test(varA)
shapiro.test(varB)

# Q-Q plot for visual inspection
par(mfrow = c(1, 2))
qqnorm(varA, main = "Group A")
qqline(varA)
qqnorm(varB, main = "Group B")
qqline(varB)

If the Shapiro-Wilk p-value is greater than 0.05, we have no evidence to reject the normality hypothesis.

A Worked Example

Here is a practical scenario. Suppose we have two landing page variants for an SEO campaign: version A (the current one, the control) and version B (with a new headline and CTA). We want to know whether the new version yields a higher average CTR. We record 10 days of traffic for each variant and measure the CTR percentage:

varA <- c(2.8, 3.1, 3.5, 2.9, 3.3, 3.0, 2.7, 3.2, 3.4, 3.1)
varB <- c(3.2, 3.8, 3.5, 4.1, 3.6, 3.9, 3.3, 3.7, 4.0, 3.4)

t.test(varA, varB)

 Welch Two Sample t-test

data:  varA and varB
t = -3.092, df = 17.6, p-value = 0.006297
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.831 -0.157
sample estimates:
mean of x mean of y
     3.10      3.65

Let us interpret the output. The mean difference is 0.55 percentage points (3.65 − 3.10), with a p-value of 0.0063 — well below the 0.05 threshold. The 95% confidence interval goes from −0.83 to −0.16: it does not contain zero. We have sufficient statistical evidence to reject H₀ in favour of the alternative hypothesis: the mean CTR of variant B differs from that of variant A.

Effect Size: Cohen’s d

Cohen’s d standardises the difference by dividing by the pooled standard deviation:

\( d = \frac{\bar{x}_1 – \bar{x}_2}{s_{pooled}} \\ \) \( s_{pooled} = \sqrt{\frac{(n_1 – 1)s^2_1 + (n_2 – 1)s^2_2}{n_1 + n_2 – 2}} \\ \)

Let us calculate in R:

n1 <- length(varA); n2 <- length(varB)
pooled_sd <- sqrt(((n1-1)*sd(varA)^2 + (n2-1)*sd(varB)^2) / (n1+n2-2))
d <- (mean(varB) - mean(varA)) / pooled_sd
d

[1] 1.38

Rule of thumb: d ≈ 0.2 is small, 0.5 medium, 0.8 large. Our d = 1.38 is a large effect — the difference is not only statistically significant but also substantial.

Statistical Power

If we do not find a significant difference, is it because there is none, or because the sample was too small? Power (1 − β) is the probability of detecting an effect of a given size, if it exists.

Let us calculate in R the sample size needed to detect a large effect with 80% power:

library(pwr)
pwr.t.test(d = 0.8, power = 0.80, sig.level = 0.05,
           type = "two.sample", alternative = "two.sided")

     Two-sample t test power calculation

              n = 25.52
              d = 0.8
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

We need about 26 observations per group to have an 80% chance of detecting a large effect. With our 10 observations per group, power is much lower — the fact that we still found significance is a sign that the effect is genuinely strong.

When to Use the Paired (Dependent) t-Test?

When the same subjects are measured before and after an intervention, we have paired data. The test does not compare the means of two groups, but rather the mean of the differences between before and after.

Assumptions

• The sample of differences is random
• The paired differences are independent of one another
• The differences come from an approximately normal population (or n is large enough)

Hypotheses:

\( H_0 : \delta = 0 \\ \) \( H_a : \delta \neq 0 \\ \)

where δ is the mean difference in the population.

The formula is analogous to the one-sample test, applied to the differences:

\( t = \frac{\bar{d} – \delta}{SE_{\bar{d}}} \\ \)

where $\( \bar{d} \)\( is the mean of the sample differences and \)\( SE_{\bar{d}} = \frac{s_d}{\sqrt{n}} \)$ is the standard error.

The standard deviation of the differences:

\( s_d = \sqrt{\frac{\Sigma(d – \bar{d})^2}{n – 1}} \\ \)

Checking Assumptions in R

diff <- after - before
shapiro.test(diff)

# Visual inspection
qqnorm(diff)
qqline(diff)

A Worked Example

Imagine this scenario: an SEO instructor wants to assess the effectiveness of an intensive course. She gives participants a 100-point test on the first day (pre) and the same test at the end of the course (post). The same students — not two different groups — are measured twice. Here are the scores:

pre  <- c(68, 75, 82, 71, 79, 65, 73, 77, 70, 74)
post <- c(72, 78, 85, 76, 82, 70, 75, 80, 74, 79)

diff <- post - pre
diff

[1] 4 3 3 5 3 5 2 3 4 5

Let us check normality of the differences:

shapiro.test(diff)

 Shapiro-Wilk normality test

data:  diff
W = 0.905, p-value = 0.248

p > 0.05 → we do not reject the normality hypothesis. Let us proceed:

t.test(pre, post, paired = TRUE)

 Paired t-test

data:  pre and post
t = -7.746, df = 9, p-value = 2.86e-05
alternative hypothesis: true mean difference is not equal to 0
95 percent confidence interval:
 -4.171 -2.029
sample estimates:
mean difference
         -3.1

The mean improvement is 3.1 points, with p < 0.001. The 95% confidence interval goes from −4.17 to −2.03 points — it does not contain zero. We have statistical evidence of a significant improvement after the course.

Effect Size for Paired Data

d_paired <- mean(diff) / sd(diff)
d_paired

[1] 2.45

Decision Flowchart

A text version of the reasoning:

1. Same subjects measured twice? → Paired t-test
2. Two independent groups? → Welch t-test (default)
3. Non-normal data with small samples? → Non-parametric test (Mann-Whitney or Wilcoxon)

When NOT to Use the t-Test

The t-test is powerful, but not universal:

• Strongly skewed distributions with small samples (n < 15 per group): the t-test loses reliability. Use the Mann-Whitney test (wilcox.test in R) for independent samples, or the Wilcoxon signed-rank test for paired data.
• More than two groups: you need analysis of variance (ANOVA), which we will cover in a future article.
• Ordinal data: the t-test works on means, which assume at least interval-scale data. For rankings or Likert scales, non-parametric tests are better.

FAQ

What is the difference between paired and independent t-tests?
In the paired t-test we compare the mean of the differences within each subject (before/after). In the independent version we compare the means of two separate groups. The choice depends on the experimental design, not our preference.

When should I use Welch instead of the classic t-test?
Always, by default. Welch does not assume equal variances and is more robust. Use var.equal = TRUE only if you have a solid reason to believe the variances are similar (balanced samples with close variances).

Does the t-test work with small samples?
Yes, provided the data are approximately normal. With n < 15 per group, always check normality with shapiro.test().

What if my data are not normal?
Use wilcox.test() in R: for independent samples (Mann-Whitney) or with paired = TRUE for paired data (Wilcoxon signed-rank).

Try It Yourself

Take the independent-sample example data and calculate:

1. A one-tailed test: does variant B have a higher CTR than A? (alternative = “greater”)
2. What sample size would be needed to detect a medium effect (d = 0.5) with 80% power?

---

We have covered two fundamental tools for comparing groups: the independent-samples t-test, the workhorse of A/B testing, and the paired t-test, indispensable for pre/post designs. In both cases the p-value alone is not enough: effect size, power, and confidence intervals complete the picture. In the next article we will tackle the case of three or more groups — and that is where analysis of variance (ANOVA) comes in.

---

Further Reading

For a detailed and systematic treatment of two-sample testing — pooled vs Welch, paired designs, and the assumptions that underpin them — Statistica by Newbold, Carlson and Thorne provides the full theoretical and practical framework needed to apply these tests correctly in real-world settings. Amazon