Analysis of Variance: ANOVA Explained Simply

Three Google Ads campaigns. Or four landing pages, or five versions of a newsletter. The averages seem different: but are these real differences or just statistical noise?

This is exactly the problem that Analysis of Variance (ANOVA) solves: determining whether at least one group differs genuinely from the others, without falling into the trap of running dozens of separate comparisons.

In short: ANOVA (analysis of variance) tests whether the means of three or more groups are all equal or whether at least one differs from the others. It does so by comparing the variability between the groups with the variability within them: their ratio is the F statistic. If the means are equal, the two variances estimate the same quantity; a large enough F (p-value below the chosen threshold) is evidence against the hypothesis that the means are equal. ANOVA tells us whether there is a difference, not where: to find out which groups differ we need post-hoc tests such as Tukey’s HSD; to find out how much it matters, η². In R: summary(aov(y ~ group)).

The term “analysis of variance” sounds counterintuitive: we are comparing means, not variances. The name comes from the way the test uses variances to determine whether the means differ. ANOVA compares the variance of the means between groups with the variance within each group. If the former is sufficiently larger than the latter, we have evidence that the groups are not equivalent.

The Intuition in a Metaphor

Let us imagine three school classes. In each class the students have fairly similar grades (low within-group variance), but the class averages are very different (high between-group variance). In this case it is easy to conclude that the classes are genuinely different.

If instead the grades within each class vary wildly — brilliant students next to struggling ones — distinguishing one class from another becomes much harder, even if the averages look different. The within-group variability “drowns out” the differences between groups.

This is what ANOVA is all about: a ratio between the variability between groups and the variability within groups. When the former clearly dominates the latter, the F value grows and we have evidence against the null hypothesis.

When to Use ANOVA

ANOVA is a parametric test. It requires a few conditions:

  • Normality: data in each group should approximate a normal distribution — the F statistic assumes that residuals follow a normal distribution
  • Homogeneity of variances: groups should have approximately equal variances (homoscedasticity), otherwise the pooled variance estimate becomes biased
  • Independence: observations are independent of each other — no observation should influence another
  • Continuous dependent variable and categorical independent variable (the factor, with two or more levels)

Why ANOVA Instead of Multiple t-Tests?

A natural question arises: why use ANOVA when we could simply compare every pair of groups with a t-test?

The problem is not the inconvenience of running many tests. The problem is that each test has a probability α of producing a false positive (Type I error). When we run many independent tests, these probabilities accumulate.

For k groups, the number of pairwise comparisons is:

\( \displaystyle n = \frac{k(k-1)}{2} \)

With 4 groups that would be 6 t-tests. The probability of committing at least one Type I error becomes:

\( \displaystyle 1 – (1 – \alpha)^n \)

With α = 0.05 and n = 6:

\( \displaystyle 1 – (1 – 0.05)^6 = 1 – 0.735 = 0.265 \)

That is 26.5%: more than one in four. Unacceptable.

When comparing three or more groups, ANOVA is preferable because it runs a single test with a single α, controlling the overall error rate.

The Simplest Case: One-Way ANOVA

One-way ANOVA is the simplest form. The data are organised into groups based on a single categorical variable (the factor). The question is:

> Are the observed differences between the means too large to be attributed to chance?

Let us formalise the hypotheses:

\( \displaystyle H_0: \mu_1 = \mu_2 = \dots = \mu_k \)

\( \displaystyle H_a: \text{at least one mean differs from the others} \)

Important: ANOVA tells us THAT a difference exists, not WHERE it is. To identify which groups differ, we need a post-hoc test, which we will see shortly.

How to Interpret the F Value

The core of ANOVA is the F ratio:

\( \displaystyle F = \frac{\text{variance between groups}}{\text{variance within groups}} = \frac{MS_b}{MS_e} \)

If the null hypothesis is true, both \( MS_b \) and \( MS_e \) estimate the same variance, and F is close to 1. If F is significantly greater than 1, we have evidence that the means are not all equal.

ANOVA was developed by Ronald Fisher in the 1920s. The F statistic it relies on is named in his honour as the Fisher-Snedecor F distribution.

F distribution and critical region
F distribution and critical region

A Step-by-Step Example

Let us take a concrete case. We want to know whether the type of device (desktop, mobile, tablet) affects goal completions on the site. For each device we take the goal completions on six days; for this example, we assume three independent samples of six days each:

DesktopMobileTablet
394530
675445
786422
595239
424638
513541

Group means:

\( \displaystyle \bar{x}_{\text{desktop}} = 56 \quad \bar{x}_{\text{mobile}} = 49.3 \quad \bar{x}_{\text{tablet}} = 35.8 \)

And the grand mean (mean of all observations): \( \bar{x} = 47 \).

To understand where the F statistic comes from, we need to decompose the total variability into two parts: the variability between groups and the variability within groups. Without going through every manual calculation — nobody does them by hand anymore, as we will see shortly — from these we derive the respective variances (Mean Squares) and finally the F ratio.

SourceSSdfMSFp
Device (between)1266.82633.44.860.0236
Residual (within)1954.215130.3
Total3220.917

F = 4.86 exceeds the critical value (3.68 for df = 2,15 and α = 0.05). With a p-value of 0.0236, we reject the null hypothesis: at least one device has a different mean.

How to Interpret the p-Value

The p-value in ANOVA is read like any hypothesis test: it is the probability of observing an F value equal to or more extreme than the one obtained, assuming H₀ is true. A small p-value (typically < 0.05) tells us that the observed differences are unlikely to be due to chance.

But note: statistical significance does not mean practical importance. A very small p-value can hide a negligible effect if the sample is large. That is why we need effect size.

Effect Size: η² (Eta-Squared)

The p-value measures the evidence against the hypothesis that the means are equal. Eta-squared tells us how much the difference weighs: it is the share of the observed variability associated with the groups, a measure of effect size.

\( \displaystyle \eta^2 = \frac{SS_b}{SS_{\text{total}}} = \frac{1266.8}{3220.9} \approx 0.39 \)

A very small p-value tells us that a difference probably exists. But it does not tell us whether that difference is large or negligible. That is why it is good practice to always accompany ANOVA with a measure of effect size.

The conventional (Cohen’s) thresholds for η²:

  • 0.01: small effect
  • 0.06: medium effect
  • 0.14: large effect

Our η² = 0.39 is a very large effect: the type of device explains 39% of the variability in goal completions.

What to Do After ANOVA: Post-Hoc Tests

ANOVA tells us that a difference exists, but not which groups differ. For that we need post-hoc tests.

The most common is Tukey’s HSD (Honestly Significant Difference), which compares all pairs of means while correcting for multiple comparisons.

In our example, Tukey HSD would tell us that the significant difference is between tablet and desktop, while the other pairs do not reach significance.

A Practical SEO Case: Does Time on Page Depend on Channel?

Let us put ANOVA to work on a question anyone looking at site data eventually asks: do users coming from different channels behave differently?

We have time on page (in seconds) for 8 sessions from each of four channels:

organic <- c(98, 112, 85, 105, 92, 120, 88, 101)
direct  <- c(90, 102, 78, 95, 110, 84, 99, 88)
paid    <- c(72, 65, 80, 58, 70, 75, 62, 68)
social  <- c(55, 48, 62, 50, 44, 58, 52, 46)

time   <- c(organic, direct, paid, social)
channel <- factor(rep(c("organic","direct","paid","social"), each = 8))

Compute in R:

summary(aov(time ~ channel))
             Df Sum Sq Mean Sq F value   Pr(>F)
channel       3  11913    3971   46.82 4.89e-11 ***
Residuals    28   2375      85

F = 46.82 and the p-value is on the order of 10⁻¹¹: well below any reasonable threshold. We reject the null hypothesis: at least one channel behaves differently.

Let us look at the means:

tapply(time, channel, mean)
 direct organic    paid  social 
 93.250 100.125  68.750  51.875 

Organic is the most engaged channel (100 seconds), followed by direct (93). Social stops at 52 seconds — traffic from feeds tends to bounce quickly.

Let us calculate η² to understand the effect size:

11913 / (11913 + 2375)  # ≈ 0.83

0.83: the channel explains 83% of the variability in time on page. A huge effect. In real-world settings such high values are quite rare: this example is deliberately constructed to make the method’s workings evident.

TukeyHSD(aov(time ~ channel))
                  diff        lwr        upr     p adj
organic-direct   6.875  -5.697225  19.447225 0.4549371
paid-direct    -24.500 -37.072225 -11.927775 0.0000652
social-direct  -41.375 -53.947225 -28.802775 0.0000000
paid-organic   -31.375 -43.947225 -18.802775 0.0000012
social-organic -48.250 -60.822225 -35.677775 0.0000000
social-paid    -16.875 -29.447225  -4.302775 0.0053389

The post-hoc test confirms that all differences involving paid and social versus organic and direct are significant, and so is the one between social and paid (17 seconds, p = 0.005). The difference between organic and direct, however, is not statistically significant.

Checking ANOVA Assumptions

Before trusting the results, it is wise to verify the assumptions:

Normality of residuals:

model <- aov(time ~ channel)
shapiro.test(residuals(model))

Homogeneity of variances (homoscedasticity):

bartlett.test(time ~ channel)

If these assumptions are violated, non-parametric alternatives such as the Kruskal-Wallis test can be used (although it compares distributions rather than means) or, if the only problem is unequal variances, Welch’s ANOVA (oneway.test()).

In Everyday Practice

Nobody computes an ANOVA table by hand anymore. Programs like R, Python (scipy.stats), SPSS, or Jamovi perform the calculation in seconds. Understanding what the different components represent — the decomposition of variance, the F ratio, the degrees of freedom — remains essential for correctly interpreting the result.

Try It Yourself

An e-commerce store ran three campaigns (A, B, C) and wants to know whether the average order value differs significantly.

camp_A <- c(42, 38, 45, 50, 41, 47, 39, 44)
camp_B <- c(48, 52, 46, 55, 50, 53, 49, 51)
camp_C <- c(44, 40, 49, 43, 46, 41, 45, 47)

value    <- c(camp_A, camp_B, camp_C)
campaign <- factor(rep(c("A","B","C"), each = 8))
summary(aov(value ~ campaign))

Let us run the test, read F and the p-value, and compute η².

F = 10.77, p = 0.0006, η² = 0.51. Campaign B (mean €50.5) performs better than A (€43.2) and C (€44.4), and the difference is statistically significant. Tukey HSD would confirm this by pointing at B.

Frequently asked questions

What is ANOVA in statistics?
It is a test that compares the means of three or more groups in one go. It splits the variability of the data into a part due to differences between the groups and a part due to differences within them; divided by their degrees of freedom they become two variances (the mean squares), and their ratio is the F statistic.

Does ANOVA work with only two groups?
Yes, but in that case it is equivalent to a two-tailed t-test (and F = t²).

What do “one-way” and “two-way” mean?
One-way: a single factor (e.g., channel). Two-way: two factors simultaneously (e.g., channel × device), which also allows studying their interaction.

Can ANOVA be used with small samples?
Yes, but normality and homoscedasticity should be checked, bearing in mind that with little data these checks have little power. ANOVA’s robustness depends on how balanced the groups are: with unequal group sizes and unequal variances it is better to switch to Welch’s ANOVA.

What if the assumptions do not hold?
If the variances differ between groups, Welch’s ANOVA (oneway.test() in R) does not assume them equal; if the data are far from normal, the non-parametric Kruskal-Wallis test (kruskal.test()), which however tests a different hypothesis: it compares the distributions, and reads as a difference in location only if the groups have distributions of similar shape.


ANOVA is one of those ideas that, once understood, we see everywhere: whenever we compare a continuous metric across multiple groups — channels, devices, variants, time periods — it is the right tool for not confusing noise with signal. And it is not an island: under the hood, analysis of variance and linear regression are the same model, seen from two different angles. It is from that model — the linear model — that the most powerful part of applied statistics flows, and that is where our journey continues.


Further Reading

Analysis of variance in all its forms — one-way, two-way, repeated measures — is covered in Statistics by Newbold, Carlson and Thorne, for those who want to go beyond the introduction in this article. Amazon

Leave a Reply

Your email address will not be published. Required fields are marked *