Over the past few weeks I have reworked some of the most-read pages of this blog from top to bottom: new openings, rewritten titles, FAQ sections. In a month I will open Search Console, and the temptation will be the usual one: compare the clicks before with the clicks after.
If they have gone up, will it be thanks to me? And if they have gone down, will I have got everything wrong? A before/after comparison cannot answer either question: in the meantime the season changes, competitors move, the algorithm shifts. One core update in the wrong week and the best work in the world looks like a disaster.
In marketing, the classic answer to this problem is the A/B test: two versions, users split at random, results compared. With SEO, though, there is an insurmountable obstacle, because Google sees only one version of the page. We cannot split the users: we can split the pages. That is the idea behind SEO split testing, and the statistical tool that makes it reliable has a name that sounds like a tongue-twister: difference-in-differences.
In short: an SEO split test measures the effect of a change (titles, templates, structured data) by randomly dividing a set of similar pages into test pages, which receive the change, and control pages, which are left untouched. A before/after comparison on its own is misleading, because seasonality and core updates hit every page; difference-in-differences (DiD) subtracts the change in the control pages from the change in the test pages, and what remains is the effect of the edit. A t-test on the page-by-page changes tells us whether that effect can be distinguished from chance. The pages must be numerous and homogeneous: detecting a +8% effect typically takes a few hundred of them.
What we will cover:
In an A/B test on a landing page’s conversion rate, each visitor sees version A or version B, assigned at random. Since randomisation makes the two groups of users alike in everything else, a difference in results can be attributed to the version. Chance acts as the guarantor.
With rankings the mechanism jams straight away. In SEO, the user who counts is only one: Googlebot. If we show it two versions of the same page depending on the visit, at best it cannot tell which one to index, at worst it is called cloaking. And the outcome we want to measure (rankings, impressions, organic clicks) depends on how Google evaluates the page, not on how an individual visitor experiences it.
The way out is to move the randomisation one level up. Instead of users, we draw pages: one group gets the change, the other gets nothing. It works when the site has many pages built on the same template, answering the same kind of search: the product pages of an e-commerce site, the listings of a classifieds site, category pages, the city pages of a portal. A site with ten pages that are all different, it should be said at once, cannot run a split test: it does not have enough units to randomise.
Without a control group, the only possible comparison is with the past. The trouble is that the past differs from the present in a thousand ways that have nothing to do with our change:
Each of these factors can be larger than the effect we are looking for. A better title might be worth 5-10% more clicks; a core update can take away 20% in a week. Comparing only before with after, we measure the sum of everything that changed, and our edit is just one piece of it, often the smallest.
The control group solves the problem at the root: control pages live through the same season, the same core update, the same competition as the test pages. The only thing that sets them apart is the change.
The rules are few, but all of them must be respected.
Homogeneous pages. Test and control must be pages of the same type, with the same template and the same search intent. Mixing product pages and blog posts means comparing apples with oranges.
Random assignment. We do not choose which pages to change: we draw them. If we picked “the most promising ones”, we would introduce exactly the initial difference the test is meant to avoid. A useful refinement is paired randomisation: sort the pages by traffic, form pairs of neighbouring pages and, within each pair, toss a coin to decide which one goes into the test. The two groups are then born balanced on traffic too, which is the bulkiest factor.
A long enough “before” period. We need it to know how the pages behave when nobody touches them, and to check that test and control move together (we will come back to this). Six to eight weeks is a good starting point.
Decide in advance when to look. Test duration and metric are fixed beforehand. Looking at the data every morning and stopping at the first favourable signal is the peeking problem, and it inflates false positives exactly as it does in an A/B test.
As always, an example helps to clarify things. We build it by simulation, so we know the truth and can see which methods recover it.
Let’s imagine an e-commerce site with 200 product pages on the same template. We decide to rewrite the titles of half of them, drawn in pairs. In our simulated world the change works: it brings 8% more clicks. But a week after it goes live, a core update takes 15% of clicks away from every page on the site, test and control alike. And every page, as always happens, also fluctuates on its own, going up or down a few positions.
I build the pages in R, with paired randomisation:
set.seed(2020)
n_pages <- 200
base <- rlnorm(n_pages, meanlog = log(30), sdlog = 1) # average clicks per day
# paired randomisation: sort by traffic and draw within each pair
ord <- order(base)
group <- character(n_pages)
for (k in seq(1, n_pages, by = 2)) {
group[ord[k:(k + 1)]] <- sample(c("test", "control"))
}
pages <- data.frame(page = 1:n_pages, base = base, group = group,
drift = rlnorm(n_pages, -0.25^2 / 2, 0.25))
tapply(pages$base, pages$group, median)
# control test
# 32.36 31.96Paired randomisation has done its job: the “typical” page makes about 32 clicks a day in both groups.
Then I generate 14 weeks of daily clicks: 8 before the change and 6 after. On day 57 the new titles arrive (on the test pages only), on day 64 the core update (on all of them); at weekends traffic drops by 30%.
simulate_clicks <- function(pages, effect = 1.08, change = 57, core_update = 64) {
d <- expand.grid(page = pages$page, day = 1:98)
d$group <- pages$group[d$page]
lambda <- pages$base[d$page] *
ifelse(d$day %% 7 %in% c(6, 0), 0.7, 1) * # weekend
ifelse(d$day >= core_update, 0.85, 1) * # core update
ifelse(d$day >= change, pages$drift[d$page], 1) * # drift
ifelse(d$group == "test" & d$day >= change, effect, 1) * # the change
rlnorm(nrow(d), 0, 0.15) # noise
d$clicks <- rpois(nrow(d), lambda)
d$period <- ifelse(d$day >= change, "after", "before")
d
}
clicks <- simulate_clicks(pages)(n.b. the drift is each page’s own fluctuation after the change: new competitors, links gained or lost, small shifts in position. It averages 1, but an individual page can gain or lose 30-40%.)
Now let’s put ourselves in the shoes of someone who made the change and has no control group. For each page I compute the average daily clicks before and after, and their change:
changes <- function(d) {
m <- aggregate(clicks ~ page + group + period, data = d, FUN = mean)
w <- reshape(m, idvar = c("page", "group"), timevar = "period",
direction = "wide")
w$delta <- log(w$clicks.after / w$clicks.before)
w
}
w <- changes(clicks)
# average change of the test pages, as a percentage
exp(mean(w$delta[w$group == "test"])) - 1
# [1] -0.101The pages with the new titles lost, on average, 10% of their clicks. Without a control, the conclusion would be immediate: the change did damage, let’s roll it back. And it would be wrong, because we know (we decided it ourselves) that the change works.
(n.b. we work on the logarithm of the after/before ratio for two reasons. Pages have wildly different volumes, from one click a day to a few hundred, and an effect such as “+8%” is proportional, not a fixed number of clicks: on the log scale, percentage changes become differences, which can be added and averaged. And the mean of the logarithms, brought back with exp(), is the geometric mean: it does not get dragged around by a page going from 2 to 9 clicks.)
Now let’s look at the control pages as well:
tapply(w$delta, w$group, function(x) exp(mean(x)) - 1)
# control test
# -0.166 -0.101The control pages, which nobody touched, lost 16.6%. That is the core update, plus the fluctuations of the period: it is what would have happened to the test pages too, had we not changed them. Economists call it the counterfactual, the story that was never staged.
The whole idea of difference-in-differences is right here. First we compute the change in each group (the first difference), then the difference between the two changes (the second difference). In words: the effect of the change is how much better, or worse, the test pages did compared with the control pages. As a formula, writing Δ for the average change on the log scale:
\(DiD = \bar{\Delta}_{test} – \bar{\Delta}_{control} = (\log T_{after} – \log T_{before}) – (\log C_{after} – \log C_{before})\)
where T and C are the clicks of the test and control pages. Everything that hit both groups in the same way (seasonality, the core update, weekends) appears in both brackets and cancels out in the subtraction. Only what distinguishes the groups is left: the change.
The chart tells the story better than any table. In the first week after the change the test pages really do rise, and anyone looking only at those days would cry victory; then the core update arrives and everything plunges. But the control plunges further. The gap between the two lines, which holds until the end, is our effect.
It remains to be seen whether that gap is large enough not to be a fluke. The pages are few and noisy: perhaps the draw put into the test group, through sheer bad luck for the control, a few pages that would have risen anyway. That is a question for a statistical test, and since we have one change per page and two groups to compare, the tool is the familiar two-sample t-test. I compute it in R:
t.test(delta ~ group, data = w)
# t = -2.1086, df = 197.34, p-value = 0.03624
# 95 percent confidence interval:
# -0.144123939 -0.004821942
# mean in group control mean in group test
# -0.1809326 -0.1064596The difference between the two means is 0.0745 on the log scale, that is exp(0.0745) − 1 = +7.7% clicks: very close to the true 8%. The p-value of 0.036 tells us that a difference this large, had the change been useless, would come up by chance fewer than 4 times in 100.
The confidence interval, however, is the most honest piece of information. R reports it for the difference control − test; flipped the way round we care about and turned into a percentage, it runs from +0.5% to +15.5%. In plainer terms: the change almost certainly worked, but the data are compatible both with a negligible effect and with an effect twice as large as the estimate.
This second figure explains why nobody should judge a change by looking at one page, or even ten. The fluctuations of a single page are much larger than the effect we are looking for: dozens of pages with the new titles lost more than untouched control pages. Only the average over many pages, compared with the average of the control, brings the signal to the surface.
(n.b. the same result comes out of a regression, lm(delta ~ group, data = w): the grouptest coefficient is 0.0745, with the same p-value. It is the form to use when we want to add other variables, such as page type, and the reason why textbooks almost always present DiD as a regression model.)
The whole argument rests on one assumption: that, without the change, test and control would have moved in the same way. It is called the parallel trends assumption, and it is the point where a split test stands or falls.
We cannot check it in the “after” period, because there we do not see the counterfactual. But we can check it in the “before” period, when nobody had touched anything. Two tools, both extremely useful.
The first is the chart we have already seen: before the change the lines must overlap. If in the “before” period the test pages were already growing faster than the control, the draw went wrong, or the pages are not really homogeneous, and any later difference is suspect.
The second is a placebo test: we pretend the change happened halfway through the “before” period and repeat the analysis. Since nothing happened at that moment, the DiD must come out as zero. I check it in R, moving the fake change to day 29:
pre <- subset(clicks, day < 57)
pre$period <- ifelse(pre$day >= 29, "after", "before")
t.test(delta ~ group, data = changes(pre))
# t = 0.071079, df = 197.57, p-value = 0.9434A difference of practically zero (−0.09%), p-value 0.94: the control behaved like the test when nothing told them apart. That is exactly what we want to see. A “significant” placebo test, on the contrary, would be an alarm bell not to be ignored.
A warning: difference-in-differences removes everything that hits test and control in the same way, but nothing that hits them differently.
If the core update penalises precisely the kind of title we introduced, or if in the meantime a competitor attacks only some of our categories, DiD will attribute that effect to the change as well. Randomisation makes these coincidences unlikely, not impossible: that is why checking parallel trends and running a placebo test are not optional.
In our example it went well: p-value 0.036, effect recovered. But how well did it go? We find out with power analysis: the probability that the test detects an effect that is really there.
To compute it we need three ingredients: the effect we want to detect (+8%, that is log(1.08) = 0.077 on the log scale), the number of pages and the variability of the page-by-page changes within each group. We estimate the latter from the test data: it is about 0.25. Computing it in R:
s <- sqrt(mean(tapply(w$delta, w$group, var))) # within-group standard deviation
s
# [1] 0.2497
power.t.test(n = 100, delta = log(1.08), sd = s)$power
# [1] 0.583
power.t.test(power = 0.8, delta = log(1.08), sd = s)$n
# [1] 166.3With 100 pages per group, power is 58%: little better than a coin toss. If we repeated the very same experiment many times, in four cases out of ten we would conclude that the change is useless, while knowing that it works. To reach 80%, the usual threshold, we need about 166 pages per group, more than 330 in total.
The figure gives the measure of the problem. The typical effects of SEO changes are small, and small effects require many pages. A +15% shows up even on a hundred product pages; a +5%, which on a large e-commerce site is still worth a lot of money, requires more than 800.
Before starting, then, it pays to do the sum backwards: how many pages do we have? With that number, what minimum effect can we hope to see? The variability is estimated from the same pages, comparing two past periods as long as the test. If the minimum detectable effect is larger than what we can reasonably expect, the test is underpowered. In that case we either widen it to more pages, or accept that it will only say “I don’t know”.
As for time, two considerations. Google must first crawl and reprocess the changed pages, and on large sites that can take more than a week: the first days after the change are often best left out of the comparison. Then, a longer “after” period reduces day-to-day noise, but not page-to-page noise, which is the bulk of it. Lengthening the test helps less than widening it: four to six weeks is usually a good compromise.
The method is sound, but in day-to-day practice there is no shortage of ways to be fooled.
Contamination. Test and control pages do not live in isolation: they link to each other, share menus and category pages. If the change alters internal links, for instance by adding a related-products block, the effect spills over onto the control too, and DiD underestimates it.
Pages with very few clicks. A page going from 2 clicks a day to 0 has an infinite change on the log scale. It is better to exclude from the start the pages below a minimum traffic threshold, or to work on impressions, which are far more numerous.
The metric chosen after the fact. Clicks, impressions, CTR, average position, conversions: if we look at all of them, one will turn out “significant” by chance. The primary metric is decided beforehand, and the others remain background information. This is the multiple comparisons problem, and it deserves a discussion of its own.
Groups that are too different. If the site has pages with very different volumes, it pays to randomise in pairs or in blocks (by category, by traffic band), as we did. A simple draw over 40 pages can produce, through sheer bad luck, a test group holding all the seasonal pages.
So far we have taken for granted that we can choose which pages to change. But some of the most important, and riskiest, decisions cannot be split in two: a domain or platform migration, a new template for the whole site, the switch to HTTPS. In these cases there is no internal control group, because all the pages change together. And the before/after comparison is as unreliable as ever.
The idea of difference-in-differences, however, survives. If we cannot observe the counterfactual inside the site, we can rebuild it from outside: from series that live through the same seasons and the same algorithm updates, but that the migration does not touch. A sister site from the same group, a section of the site left out of the migration, the Google Trends index for the main topics. In the “before” period we estimate how our traffic moves together with these series; in the “after” period we use the same relationship to predict how much traffic we would have had without the migration. The difference between observed and predicted is the effect.
Let’s take an example. I simulate 180 days of clicks for a site that migrates on day 121, losing 12% of its traffic; ten days later a core update takes 25% away from the whole sector. Alongside there are two control series: a sister site that did not migrate, and Google Trends.
set.seed(3)
days <- 1:180; migration <- 121 # 4 months before, 2 after
sector <- exp(cumsum(rnorm(180, 0, 0.02))) * # sector trend
ifelse(days %% 7 %in% c(6, 0), 0.75, 1) * # weekend
(1 - 0.25 * (days > 130)) # sector-wide core update
noise <- function(sd) exp(as.numeric(arima.sim(list(ar = 0.6), 180, sd = sd)))
site <- 800 * sector * noise(0.05) * ifelse(days >= migration, 0.88, 1)
sister <- 300 * sector * noise(0.03) # sister site, not migrated
trends <- 60 * sector * noise(0.05) # Google Trends for the topic
before <- days < migration
mean(site[!before]) / mean(site[before]) - 1
# [1] -0.273The before/after says −27%: more than twice the true effect, because it charges the core update to the migration too. So I build the counterfactual with a multiple regression on the “before” period, on the log scale, and project it onto the “after” period:
m <- lm(log(site) ~ log(sister) + log(trends), subset = before)
summary(m)$r.squared
# [1] 0.792
predicted <- predict(m, newdata = data.frame(sister, trends))
exp(mean(log(site[!before]) - predicted[!before])) - 1
# [1] -0.125
sum(exp(predicted[!before])) - sum(site[!before]) # clicks lost in 60 days
# [1] 4532.7The two control series explain 79% of the site’s daily fluctuations before the migration. The comparison with the counterfactual estimates an effect of −12.5%, a step away from the true −12%: about 4,500 clicks lost in two months, out of 36,400 expected.
How far can we trust that −12.5%? The days of a time series are not independent of each other (a good day tends to be followed by another good day), and an interval computed as if they were would be too optimistic. Here too the placebo comes to our aid, this time in time: we pretend the migration happened at various moments of the “before” period, estimate the model on the preceding days and measure the fake “effect” over the next 30 days.
placebo <- sapply(seq(40, 90, by = 5), function(k) {
mk <- lm(log(site) ~ log(sister) + log(trends), subset = days < k)
pk <- predict(mk, newdata = data.frame(sister, trends))
exp(mean(log(site[k:(k + 29)]) - pk[k:(k + 29)])) - 1
})
range(placebo)
# [1] -0.018 0.028When nothing happened, the method “finds” effects between −1.8% and +2.8%: that is its noise threshold. The migration’s −12.5% lies well outside that band, and we can regard it as real.
The method we have just built by hand is the heart of CausalImpact, the R package developed at Google for exactly this problem. Instead of a plain regression it uses a Bayesian time-series model, which accounts for trend, seasonality and the dependence between days, and returns the effect directly with its credible interval. The logic is the same: a counterfactual built from series the intervention did not touch.
A warning: the counterfactual is only as credible as the control series.
They must move with the site before the intervention (checked with the R² and with the placebo), and above all they must not be touched by the intervention itself. If the migration also changes the sister site, or if we use Google Trends as a control for a campaign that also boosted searches, the counterfactual moves together with the site and the effect vanishes from the books.
It is an experiment that measures the effect of a change on organic traffic by randomly dividing a set of similar pages into two groups: the test pages receive the change, the control pages stay as they are. Comparing how the two groups change separates the effect of the edit from seasonality, algorithm updates and competition.
An A/B test splits users: each one sees one of two versions of the same page. An SEO split test splits pages, because Google must see a single version of each. That is why it needs a site with many pages of the same type.
Usually six to eight weeks of data before the change and four to six weeks after, leaving out the first days while Google is still crawling the changed pages. Lengthening the period helps less than increasing the number of pages.
With a few dozen pages only large effects, 15-20% or more, can be detected. For typical effects of 5-10% a few hundred pages are usually needed: it is worth computing the test’s power before starting.
By comparing post-migration traffic with a counterfactual: a prediction of how much traffic there would have been without the migration, built from series the migration does not touch (a sister site, a section that was not migrated, Google Trends). The R package CausalImpact automates this approach.
Any change that can be applied to a group of pages on the same template: titles and meta descriptions, heading structure, structured data, content blocks, internal links, page speed. Changes that touch the whole site at once, such as a migration, are not suitable: there is no internal control group, and a counterfactual built from outside is needed.
The example code is complete: just copy it into R and change one parameter at a time.
effect to 1 (the change does nothing) and repeat the analysis with a few different set.seed() values. How often does the t-test wrongly declare a significant effect? (It should happen about 5 times in 100.)effect back to 1.08 and double the pages, n_pages <- 400: how much does the confidence interval narrow?For the last exercise, a hint: it is the quickest way to understand why randomisation is not a bureaucratic detail.
There is a question we have brushed against without tackling it. In a split test we usually look at one primary metric, but the temptation to look at ten is very strong: clicks, impressions, CTR, position, and then every category of pages on its own. Every extra look is one more chance to find a “significant” effect that does not exist. How quickly these false alarms pile up, and how to correct for them, is the subject of multiple comparisons: the next step of our journey through the traps of data.
On online experimentation, the reference is Trustworthy Online Controlled Experiments by Ron Kohavi, Diane Tang and Ya Xu: from choosing metrics to the pitfalls of control groups, it is written by the people who ran experimentation at Microsoft, Google and LinkedIn.
"The new title has a p-value of 0.03: there is a 97% probability that it…
In 2009 a group of neuroscientists put a salmon in a functional MRI scanner, showed…
It happens with every reasonably serious project: you export the keyword list from Search Console…
Anyone who spends their days inside Search Console knows that little nagging feeling: a page…
In the article on the multi-armed bandit we used Bayes to decide between variants: shifting…
In the article on Bayesian A/B testing we compared two variants at a fixed sample…