This is exactly the problem that Analysis of Variance (ANOVA) solves: determining whether at least one group differs genuinely from the others, without falling into the trap of running dozens of separate comparisons.

The term “analysis of variance” sounds counterintuitive: we are comparing means, not variances. The name comes from the way the test uses variances to determine whether the means differ. ANOVA compares the variance of the means between groups with the variance within each group. If the former is sufficiently larger than the latter, we have evidence that the groups are not equivalent.

The Intuition in a Metaphor

Imagine three school classes. In each class the students have fairly similar grades (low within-group variance), but the class averages are very different (high between-group variance). In this case it is easy to conclude that the classes are genuinely different.

If instead the grades within each class vary wildly — brilliant students next to struggling ones — distinguishing one class from another becomes much harder, even if the averages look different. The within-group variability “drowns out” the differences between groups.

This is what ANOVA is all about: a ratio between the variability between groups and the variability within groups. When the former clearly dominates the latter, the F value grows and we have evidence against the null hypothesis.

When to Use ANOVA

ANOVA is a parametric test. It requires a few conditions:

• Normality: data in each group should approximate a normal distribution — the F statistic assumes that residuals follow a normal distribution
• Homogeneity of variances: groups should have approximately equal variances (homoscedasticity), otherwise the pooled variance estimate becomes biased
• Independence: observations are independent of each other — no observation should influence another
• Continuous dependent variable and categorical independent variable (the factor, with two or more levels)

Why ANOVA Instead of Multiple t-Tests?

A natural question arises: why use ANOVA when we could simply compare every pair of groups with a t-test?

The problem is not the inconvenience of running many tests. The problem is that each test has a probability α of producing a false positive (Type I error). When we run many independent tests, these probabilities accumulate.

For k groups, the number of pairwise comparisons is:

With 4 groups that would be 6 t-tests. The probability of committing at least one Type I error becomes:

With α = 0.05 and n = 6:

That is 26.5%: more than one in four. Unacceptable.

When comparing three or more groups, ANOVA is preferable because it runs a single test with a single α, controlling the overall error rate.

The Simplest Case: One-Way ANOVA

One-way ANOVA is the simplest form. The data are organised into groups based on a single categorical variable (the factor). The question is:

> Are the observed differences between the means too large to be attributed to chance?

Let us formalise the hypotheses:

How to Interpret the F Value

The core of ANOVA is the F ratio:

If the null hypothesis is true, both \( MS_b \) and \( MS_e \) estimate the same variance, and F is close to 1. If F is significantly greater than 1, we have evidence that the means are not all equal.

A Step-by-Step Example

Let us take a concrete case. We want to know whether the type of device (desktop, mobile, tablet) affects goal completions on the site. We track monthly data for 6 months:

Group means:

And the grand mean (mean of all observations): \( \bar{x} = 47 \).

To understand where the F statistic comes from, we need to decompose the total variability into two parts: the variability between groups and the variability within groups. Without going through every manual calculation — nobody does them by hand anymore, as we will see shortly — from these we derive the respective variances (Mean Squares) and finally the F ratio.

F = 4.86 exceeds the critical value (3.68 for df = 2,15 and α = 0.05). With a p-value of 0.0236, we reject the null hypothesis: at least one device has a different mean.

How to Interpret the p-Value

The p-value in ANOVA is read like any hypothesis test: it is the probability of observing an F value equal to or more extreme than the one obtained, assuming H₀ is true. A small p-value (typically < 0.05) tells us that the observed differences are unlikely to be due to chance.

But note: statistical significance does not mean practical importance. A very small p-value can hide a negligible effect if the sample is large. That is why we need effect size.

Effect Size: η² (Eta-Squared)

The p-value tells us whether a difference exists. Eta-squared tells us how large it is.

Rules of thumb for η²:

• 0.01: small effect
• 0.06: medium effect
• 0.14: large effect

Our η² = 0.39 is a very large effect: the type of device explains 39% of the variability in goal completions.

What to Do After ANOVA: Post-Hoc Tests

ANOVA tells us that a difference exists, but not which groups differ. For that we need post-hoc tests.

The most common is Tukey’s HSD (Honestly Significant Difference), which compares all pairs of means while correcting for multiple comparisons.

In our example, Tukey HSD would tell us that the significant difference is between tablet and desktop, while the other pairs do not reach significance.

A Practical SEO Case: Does Time on Page Depend on Channel?

Let us put ANOVA to work on a question anyone looking at site data eventually asks: do users coming from different channels behave differently?

We have time on page (in seconds) for 8 sessions from each of four channels:

organic <- c(98, 112, 85, 105, 92, 120, 88, 101)
direct  <- c(90, 102, 78, 95, 110, 84, 99, 88)
paid    <- c(72, 65, 80, 58, 70, 75, 62, 68)
social  <- c(55, 48, 62, 50, 44, 58, 52, 46)

time   <- c(organic, direct, paid, social)
channel <- factor(rep(c("organic","direct","paid","social"), each = 8))

Compute in R:

summary(aov(time ~ channel))

             Df Sum Sq Mean Sq F value   Pr(>F)
channel       3  11913    3971   46.82 4.89e-11 ***
Residuals    28   2375      85

F = 46.82 and the p-value is on the order of 10⁻¹¹: well below any reasonable threshold. We reject the null hypothesis: at least one channel behaves differently.

Let us look at the means:

tapply(time, channel, mean)

  direct  organic     paid   social
    93.2    100.1     68.8     51.9

Organic is the most engaged channel (100 seconds), followed by direct (93). Social stops at 52 seconds — traffic from feeds tends to bounce quickly.

Let us calculate η² to understand the effect size:

11913 / (11913 + 2375)  # ≈ 0.83

0.83: the channel explains 83% of the variability in time on page. A huge effect. In real-world settings such high values are quite rare: this example is deliberately constructed to make the method’s workings evident.

TukeyHSD(aov(time ~ channel))

                      diff       lwr        upr     p adj
direct-organic      -6.875  -24.423    10.6730 0.7165876
paid-organic       -31.250  -48.798   -13.7020 0.0003217
social-organic     -48.250  -65.798   -30.7020 0.0000001
paid-direct        -24.375  -41.923    -6.8270 0.0041674
social-direct      -41.375  -58.923   -23.8270 0.0000073
social-paid        -17.000  -34.548     0.5480 0.0598936

The post-hoc test confirms that all differences involving paid and social versus organic and direct are significant. The difference between organic and direct, however, is not statistically significant.

Checking ANOVA Assumptions

Before trusting the results, it is wise to verify the assumptions:

Normality of residuals:

model <- aov(time ~ channel)
shapiro.test(residuals(model))

Homogeneity of variances (homoscedasticity):

bartlett.test(time ~ channel)

If these assumptions are violated, non-parametric alternatives such as the Kruskal-Wallis test can be used.

In Everyday Practice

Nobody computes an ANOVA table by hand anymore. Programs like R, Python (scipy.stats), SPSS, or Jamovi perform the calculation in seconds. Understanding what the different components represent — the decomposition of variance, the F ratio, the degrees of freedom — remains essential for correctly interpreting the result.

Try It Yourself

An e-commerce store ran three campaigns (A, B, C) and wants to know whether the average order value differs significantly.

camp_A <- c(42, 38, 45, 50, 41, 47, 39, 44)
camp_B <- c(48, 52, 46, 55, 50, 53, 49, 51)
camp_C <- c(44, 40, 49, 43, 46, 41, 45, 47)

value    <- c(camp_A, camp_B, camp_C)
campaign <- factor(rep(c("A","B","C"), each = 8))
summary(aov(value ~ campaign))

Run the test, read F and p-value, calculate η².

F = 10.77, p = 0.0006, η² = 0.44. Campaign B (mean €50.5) performs better than A (€43.2) and C (€44.4), and the difference is statistically significant. Tukey HSD would confirm this by pointing at B.

FAQ

Does ANOVA work with only two groups?
Yes, but in that case it is equivalent to a two-tailed t-test (and F = t²).

What does “one-way” and “two-way” mean?
One-way: a single factor (e.g., channel). Two-way: two factors simultaneously (e.g., channel × device), which also allows studying their interaction.

Can I use ANOVA with small samples?
Yes, but check normality and homoscedasticity. With unequal group sizes, ANOVA is still robust if the deviations are not extreme.

ANOVA is one of those ideas that, once understood, you see everywhere: whenever we compare a continuous metric across multiple groups — channels, devices, variants, time periods — it is the right tool for not confusing noise with signal. And it is not an island: under the hood, analysis of variance and linear regression are the same model, seen from two different angles. It is from that model — the linear model — that the most powerful part of applied statistics flows, and that is where our journey continues.

Further Reading

Analysis of variance in all its forms — one-way, two-way, repeated measures — is covered in Statistics by Newbold, Carlson and Thorne, for those who want to go beyond the introduction in this article. Amazon