Are Google’s AI answers taking our clicks? Measuring it with expected CTR and a control group

A site’s technical guides climb in the search results, show up more often, and yet get fewer clicks. The CTR of the guides drops from 8.6% to 5.8% in six months.
The first explanation that comes to mind is also the most fashionable one: the AI-generated answers at the top of the SERP (the AI Overviews, and then AI Mode) answer the question before anyone feels the need to click. But is that really what happened? Maybe the guides got worse. Maybe the whole site dropped. Maybe the season changed.

We have already built, when talking about expected vs actual CTR, the right yardstick for judging a CTR: the curve linking position to the typical CTR of our own site. That yardstick, however, takes a snapshot of a single moment. Here we set it in motion: we use the same curve for two periods, before and after an event, and we pair it with a control group, that is, pages the event should not touch.
The reasoning that comes out of it has a name, difference-in-differences, and it is useful well beyond AI answers: an algorithm update, a new SERP format, a partial migration.

In short: comparing CTR before and after an event is misleading for two reasons: positions change (and CTR with them), and everything else changes in the same period too. The first problem is solved by expected CTR, estimated on the before period only and applied to both: the ratio of actual to expected clicks becomes comparable over time. The second is solved by a control group: the effect of the event is the difference between the change in the exposed pages and the change in the control pages. In our simulated example the guides go from 1.035 to 0.592 times their expected clicks, the product pages from 1.085 to 1.014: the estimated effect is a 39% drop, very close to the 40% we built into the data. Everything rests on an assumption that has to be checked, the parallel trends assumption.

What we will cover:


The naive comparison and its two flaws

Let us start from data built on purpose, reproducing the shape of a real case without using its numbers. Sixty pages of a site: thirty guides, answering informational questions (exactly the ones AI answers focus on), and thirty product pages, catching transactional searches. For each page and for twelve months we have average position, impressions and clicks, as in a Search Console export.
From month 7 something happens: the CTR of the guides alone falls by 40%. At the same moment, to make our life harder, the guides climb a little in the results and appear more often, and the whole site loses 5% of CTR for reasons that have nothing to do with the event (the season, a change in the SERP, whatever happens in real life).

I build the data in R:

set.seed(2026)

n     <- 30
pages <- data.frame(
  page    = c(sprintf("guide-%02d", 1:n), sprintf("product-%02d", 1:n)),
  group   = rep(c("guides", "product"), each = n),
  pos0    = round(runif(2 * n, 1.5, 12), 1),        # starting position
  impr0   = round(rlnorm(2 * n, log(3000), 0.6)),   # starting monthly impressions
  quality = rlnorm(2 * n, 0, 0.25)                  # snippet quality, fixed per page
)

data <- merge(pages, data.frame(month = 1:12))
data <- data[order(data$page, data$month), ]
data$after <- data$month >= 7
exposed    <- data$group == "guides"

# after the event the guides climb a little and appear more often
data$position    <- round(data$pos0 * exp(rnorm(nrow(data), 0, 0.06)) *
                          ifelse(exposed & data$after, 0.85, 1), 1)
data$impressions <- round(data$impr0 * exp(rnorm(nrow(data), 0, 0.10)) *
                          ifelse(exposed & data$after, 1.25, 1))

true_ctr <- 0.30 * data$position^-1 * data$quality *
  ifelse(data$after, 0.95, 1) *                # shock common to the whole site (-5%)
  ifelse(exposed & data$after, 0.60, 1) *      # the event: -40% CTR on the guides only
  exp(rnorm(nrow(data), 0, 0.10))              # month-to-month noise
data$clicks <- rbinom(nrow(data), data$impressions, pmin(true_ctr, 0.9))
data$ctr    <- data$clicks / data$impressions
data$period <- ifelse(data$after, "after", "before")

The true CTR follows a power law with exponent −1, the same shape we found in the article on expected CTR; each page then has its own snippet “quality”, fixed over time, and a bit of month-to-month noise.

I compute the comparison we would all make by instinct: overall CTR and average position by group, before and after.

raw <- aggregate(cbind(clicks, impressions) ~ group + period, data = data, FUN = sum)
raw$ctr <- raw$clicks / raw$impressions
avg_pos <- aggregate(position ~ group + period, data = data, FUN = mean)
merge(raw, avg_pos)
groupperiodclicksimpressionsCTRaverage position
guidesbefore47,414550,7948.61%5.11
guidesafter40,454694,2055.83%4.38
productbefore42,137684,8666.15%6.97
productafter38,637682,0655.67%6.98

The CTR of the guides falls by 32% (5.83 divided by 8.61 gives 0.677). It looks like a measure of the effect, and instead it is wrong in two ways.

The first flaw: the guides went up, from position 5.1 to 4.4. All else being equal, going up means more CTR, not less. Raw CTR compares pages sitting in different places of the SERP, and so it underestimates the damage: the real fall is the one relative to the CTR the guides should have had in their new position.
The second flaw: the whole site dropped in the same period. The product pages too, which the event does not touch, lose 8% of CTR. Blaming the event for the whole drop of the guides would mean loading onto it what would have happened anyway.

Expected CTR answers the first flaw, the control group the second. Let us look at them one at a time.


One curve for two periods

Expected CTR is the value that our site’s position-CTR curve assigns to a page, given its position. The ratio of actual to expected clicks tells us whether a page performs better or worse than its position promises, and it has a precious property: it takes position into account. If a guide climbs from 5 to 4, its expected clicks grow with it, and the ratio remains a fair comparison. (With one caveat: the correction holds to the extent that the curve describes the link between position and CTR well. A wrong curve leaves some residual dependence on position in the ratio.)

There is, however, a choice to make, and it is the one that makes everything else possible: the curve must be estimated on the period before the event only, and then applied to both periods.
The reason is that the yardstick must not depend on what it measures. If we estimated it on all twelve months, the clicks lost by the guides would enter the curve too, and in the after period the guides sit in different positions than before: the curve would change shape because of the event itself, and with it the expected clicks on which we build the whole comparison. (A simple uniform lowering of the curve, on the other hand, would do no harm: it would touch both groups in the same proportion and cancel out in the final comparison.) Estimated on the before period, the curve is a yardstick the event could not touch.

I estimate the curve with the regression on logarithms weighted by impressions, as in the article on expected CTR, but only on months 1 to 6:

before <- subset(data, !after & clicks > 0)
curve  <- lm(log(ctr) ~ log(position), data = before, weights = impressions)
round(coef(curve), 3)
#   (Intercept) log(position)
#        -1.135        -1.055

The slope of −1.055 is close to the −1 we put into the simulation. Now I compute the expected clicks for every page and every month, and I add up actual and expected clicks by group and period:

data$expected_ctr    <- exp(predict(curve, newdata = data))
data$expected_clicks <- data$impressions * data$expected_ctr

tab <- aggregate(cbind(clicks, expected_clicks) ~ group + period, data = data, FUN = sum)
tab$ratio <- tab$clicks / tab$expected_clicks
tab
groupperiodactual clicksexpected clicksactual / expected
guidesbefore47,41445,7951.035
guidesafter40,45468,3270.592
productbefore42,13738,8281.085
productafter38,63738,1141.014

Look at the row of the guides in the after period: with the new positions and the new impressions, the curve expected more than 68 thousand clicks, and just over 40 thousand arrived. The ratio goes from 1.035 to 0.592, a 43% fall. Taking position into account has brought out a greater loss than raw CTR suggested.

N.B. the ratios of the before period are not exactly 1: there is a single curve for the whole site, and guides and products each deviate from it a little in their own way. This is not a problem, because we will never compare the ratios with 1, but each group with itself.


The control group

Expected CTR has accounted for differences in position, but it says nothing about the second flaw: the drop that would have hit the guides even without the event. To estimate it we need a control group, that is, a set of pages with two features:

  • the event does not touch it, or touches it much less: for AI answers, product pages and transactional searches, where people want to buy or get in touch and a summary at the top of the SERP is not enough;
  • it undergoes everything else just like the exposed pages: the season, algorithm updates, changes in the way Search Console measures.

It is the same logic as SEO split testing, with one important difference: there we choose the control group ourselves, at random, before intervening; here the “intervention” was decided by someone else, and the two groups are given, not drawn. It is what is called a quasi-experiment. Without random assignment, the similarity of the two groups is not guaranteed: it has to be argued and, where possible, checked in the data. We will come back to this shortly.

In our case the product pages, in the after period, go from 1.085 to 1.014 times their expected clicks: a 6.6% drop the event did not cause, because the event does not concern them. It is our estimate of what would have happened to the guides if the AI answers had not been there.


The difference of the differences

We have two changes. That of the guides, which mixes the effect of the event with everything else; and that of the product pages, which contains only everything else. The effect of the event is what is left when we remove the second from the first: the difference of the differences.

Since we work with ratios (a 43% fall, a 6.6% fall), the “difference” becomes a ratio of ratios. In words: we divide the change of the guides by the change of the control pages.

\( \displaystyle \text{effect} = \frac{R_{\text{guides, after}} \;/\; R_{\text{guides, before}}}{R_{\text{product, after}} \;/\; R_{\text{product, before}}} \)

where each \( R \) is the ratio of actual to expected clicks for a group in a period. The numerator holds the change of the guides, the denominator that of the control. If the event had done nothing, the two changes would be equal and the effect would be 1.

I compute it in R:

r <- function(g, p) tab$ratio[tab$group == g & tab$period == p]
change_guides  <- r("guides", "after") / r("guides", "before")    # 0.572
change_product <- r("product", "after") / r("product", "before")  # 0.934
change_guides / change_product                                     # 0.612

The guides fall to 0.572 times their previous level, the product pages to 0.934. The ratio between the two is 0.612: the event took from the guides about 39% of the clicks they would otherwise have received. In the simulation the true effect was 40%, and the common shock 5%: the method has separated them almost perfectly.

It is worth lining up the three estimates we have met:

comparisonestimated effect
Raw CTR of the guides, before and after−32%
Difference-in-differences on raw CTR−26%
Difference-in-differences on expected clicks−39%
True effect, built into the simulation−40%

The second row is instructive: the control group alone is not enough (0.677 divided by 0.921 gives 0.735). Without expected CTR, the guides’ climb in the rankings keeps masking part of the damage. The two corrections work on different flaws, and we need both.


The same estimate with its interval

A single number, however well built, does not tell us how far we can trust it. To get a confidence interval we write the difference-in-differences as a model for counts, of the family we met when talking about overdispersion.

The idea is this. We model the clicks of each page in each month starting from its expected clicks, which enter the model as an offset: a term with coefficient fixed at 1, which tells the model “start from here”. On top of the expected clicks the model estimates three things: a level of its own for each page (its quality, fixed over time), a shift common to all pages in the after period, and an extra shift for the guides only in the after period. This last one is the effect of the event.
The level of each page means each page is compared with itself: it is the “one page at a time” version of the table above.

I estimate the model with the quasi-Poisson:

data$event <- as.integer(exposed & data$after)   # 1 only for the guides in the after period
mod <- glm(clicks ~ factor(page) + after + event + offset(log(expected_clicks)),
           family = quasipoisson, data = data)

ci <- function(m, k) {
  b <- coef(m)[k]; se <- sqrt(vcov(m)[k, k])
  round(exp(c(estimate = b, low = b - 1.96 * se, high = b + 1.96 * se)), 3)
}
ci(mod, "event")      # 0.607  from 0.587 to 0.627
ci(mod, "afterTRUE")  # 0.939  from 0.917 to 0.962
summary(mod)$dispersion   # 2.98

The estimated effect is 0.607, with a 95% interval from 0.587 to 0.627: a drop between 37% and 41%. The common shock is 0.939, that is, 6% fewer clicks across the whole site, and this too is close to the 5% of the simulation.

The dispersion factor, 2.98, tells us that clicks vary three times more than a Poisson would allow: this is why we use the quasi-Poisson, which widens the standard errors accordingly.
It is worth trying to remove factor(page) from the model (and put a term for the group in its place): the point estimate barely changes (0.612), but the dispersion rises to 20.9 and the interval widens, from 0.561 to 0.668. Without each page’s level, the difference in quality between one page and another ends up in the noise. Comparing each page with itself removes a large share of variability that has nothing to do with the event.

N.B. this interval should be taken for what it is, because it rests on two simplifications. The first: it treats the expected CTR curve as if it were known, while it is an estimate too. Since the same curve holds for both groups and both periods, much of that uncertainty cancels out in the comparison; what remains weighs more when the positions of the two groups change in very different ways.
The second: it treats the months of the same page as independent observations, once its level is accounted for. In real data they almost never are (a good month tends to be followed by another good month), and the quasi-Poisson corrects the dispersion, not this dependence. The most direct remedy is the bootstrap over pages: we resample whole pages, with all their months, and at each round we re-estimate curve and model. It does not solve everything (a shock that hits many pages of the same group together remains a problem), but it gives a more honest interval.


Everything we have done rests on an assumption we have so far left implicit: that without the event, the guides and the product pages would have moved together. It is what allows us to use the drop of the product pages as an estimate of the “natural” drop of the guides. It is called the parallel trends assumption, and it is the point on which difference-in-differences stands or falls.
Since we work with ratios, “moving together” must be understood in a proportional sense: without the event, the ratio of actual to expected clicks of the guides would have changed by the same percentage as that of the product pages (not by the same number of points). In more technical terms, the trends must be parallel on the logarithmic scale.

A warning: difference-in-differences measures a causal effect only if, in the absence of the event, the two groups would have moved in parallel. If the guides were already losing ground before the AI answers (because they are ageing, because competitors published something better), the method attributes to the event a drop that is not its own. The assumption cannot be proven, because it concerns a world we did not observe; it can only be made credible by looking at the months before the event.

The simplest check is to look at the ratio of actual to expected clicks month by month, for the two groups. I compute it in R:

monthly <- aggregate(cbind(clicks, expected_clicks) ~ group + month, data = data, FUN = sum)
monthly$ratio <- monthly$clicks / monthly$expected_clicks
The ratio of actual to expected clicks, month by month, for the guides (exposed to the event) and the product pages (control group). In the six months before the event the two lines move up and down together, without one drifting away from the other; from month 7 the guides collapse to around 0.6, while the product pages stay just above 1.
The ratio of actual to expected clicks, month by month, for the guides (exposed to the event) and the product pages (control group). In the six months before the event the two lines move up and down together, without one drifting away from the other; from month 7 the guides collapse to around 0.6, while the product pages stay just above 1.

In months 1 to 6 the two lines fluctuate, but neither takes a path of its own: the guides stay between 0.97 and 1.08, the product pages between 1.00 and 1.16. From month 7 the guides drop suddenly to around 0.6 and stay there. It is the picture we want to see: a clean jump at the moment of the event, no drift before it.

The eye, though, is easily persuaded. Two formal checks, both on the months before the event only.

The first looks for a different trend for the guides: we add to the model a term that grows month by month for the guides only. If the guides were already sliding, its coefficient would be negative and far from zero.

I estimate the trend in R:

pre <- subset(data, !after)
pre$guides_month <- (pre$group == "guides") * pre$month
trend <- glm(clicks ~ factor(page) + factor(month) + guides_month + offset(log(expected_clicks)),
             family = quasipoisson, data = pre)
summary(trend)$coefficients["guides_month", ]
# estimate 0.0060   standard error 0.0069   p = 0.38

A trend difference of 0.6% a month, with a standard error larger than the estimate: no evidence of a trend specific to the guides. Careful, though, not to read it as a proof of absence: the 95% interval runs roughly from −0.8% to +2% a month, and with only six months of data a small drift could escape us.

The second is a placebo: we pretend the event happened in month 4, when in reality nothing happened, and we redo the estimate. If we found an “effect” there too, it would be a sign that something is wrong: groups already moving differently, or a model that does not describe the data well.

I compute the placebo in R:

pre$fake <- as.integer(pre$group == "guides" & pre$month >= 4)
placebo <- glm(clicks ~ factor(page) + I(month >= 4) + fake + offset(log(expected_clicks)),
               family = quasipoisson, data = pre)
ci(placebo, "fake")
# 1.027  from 0.981 to 1.076  (p = 0.25)

The placebo effect is 1.027, with an interval that comfortably includes 1: where nothing happened, we find no evidence of an effect. Here too, the interval (from −1.9% to +7.6%) reminds us that “no evidence” does not mean “no difference”.

Let us keep the limit of these checks well in mind: at most they say that before the event the groups were not moving in a visibly different way, not that they would have kept doing so after it. If in the same month as the event something else had happened that touches only the guides (a rewrite of their titles, a technical problem in their section of the site), no check on the data before could notice it. There we need knowledge of the site, not statistics.


Robustness: varying the analysis choices

An estimate that holds only with one particular choice of time window, or only with certain pages included, is not an estimate: it is a lucky break. Before trusting it, we vary those choices and see whether the result still stands.

The first test is the window: instead of all six months after, we take only the last one. If the effect were a flash in the pan, or if it grew over time, we would see it.

last <- subset(data, !after | month == 12)
ci(update(mod, data = last), "event")
# 0.591  from 0.553 to 0.630

The effect of month 12 alone is 0.591, with an interval that largely overlaps that of the six months: the result of the last month is consistent with the overall one. It is not enough to say that the effect is constant over time (the two estimates share the same before months, and a slow evolution could escape us), but it rules out the possibility that the entire drop was confined to the first few months.

With real data, there are also two exclusions that are always worth trying.

  • The homepage: it behaves in its own way (brand searches, navigational traffic), and a single page with a great many impressions can move a whole group on its own.
  • Brand queries: people searching for the site’s name click anyway, AI answers or not. Mixing them with the others dilutes the effect, and if they weigh differently in the two groups they distort it.

If removing one or the other changes the estimate little, we have a robust result; if it changes a lot, we have discovered that the effect sits entirely in a few pages, and that is even more useful information.

Finally, there is a trap specific to Search Console, which deserves an article of its own: sometimes what changes is not user behaviour but the way we measure it. In September 2025, for example, Google stopped supporting the &num=100 parameter (the one that made it possible to see a hundred results per page), and many sites saw impressions fall and average positions suddenly improve, without anything changing in real traffic. If a change like that falls close to our event, the control group protects us only if it hits both groups in the same way.


Significant does not mean large (nor small)

Let us go back for a moment to the simplest comparison of all: the CTR of the guides before and after, with a test on proportions. We use it only as an illustration: the test treats each impression as a trial independent of the others, and ignores the differences between pages and between months.

I run the test in R:

g <- subset(raw, group == "guides")
prop.test(g$clicks[c(2, 1)], g$impressions[c(2, 1)])$p.value   # before, after
# 0: too small for R to tell it apart from zero

With more than a million impressions the p-value is so small that R prints it as zero. But we know: the p-value is the probability of observing a difference at least this extreme if there were no true difference (and if the assumptions of the test held). The p-value measures surprise, not size. With these volumes, even a difference of a tenth of a point would be “significant”.

For size there is the effect size, and for two proportions the classic measure is Cohen’s h, the difference between the two proportions after a transformation that makes them comparable (the arcsine of the square root).

I compute h in R:

h <- function(p1, p2) abs(2 * asin(sqrt(p1)) - 2 * asin(sqrt(p2)))
h(0.0861, 0.0583)   # 0.108

A value of 0.108 sits below the 0.2 threshold Cohen called a “small effect”. Should we conclude the effect is negligible? Not at all. h measures the distance between two proportions in a general sense, designed to compare different studies; the 0.2 threshold knows nothing about what a click is worth to our site. What matters for a site is something else: how many clicks the event took away.

The calculation is direct, and we do it on the aggregate table. We take the expected clicks of the guides in the after period, bring them back to the level the guides had before and correct them for the common shock measured on the control: it is the estimate of how many clicks the guides would have received without the event, with the same impressions and positions. We subtract those actually received:

clicks_without_event <- sum(data$expected_clicks[exposed & data$after]) *
  r("guides", "before") * change_product
clicks_without_event - sum(data$clicks[exposed & data$after])
# 25,626 clicks lost, out of 40,454 received

In six months the guides lost about 25,600 clicks to the lower CTR (with the page-level model the count is similar, about 26,200). N.B. the calculation holds impressions and positions fixed at the values observed after the event: if the event had changed those too, that part would not be included. In other words: for every 100 clicks that arrived, there are 63 that would have arrived without the event and did not. A “small” effect according to Cohen’s h, and a huge one for whoever has to make the site’s traffic add up. The labels “small”, “medium”, “large” are conventions born in other fields; the right measure of size is the one that translates into the decision to be made.


From the click to the citation

Once the drop is measured, the temptation is to close the matter with “AI answers take 39% of the clicks on the guides”. The number is right, but the story might not be over.
If a guide is cited as a source in the AI answer, part of its value has moved, not disappeared: the brand shows up in front of the searcher, even if that person does not click. The click, in short, is a measure of the value of the guides, not the value itself.

Measuring that presence is a statistical problem of its own: answers change from one search to another, the tools that monitor them sample a limited number of questions, and the same question can give different answers an hour apart. Counting citations is easy; estimating with honest uncertainty how often we appear is a different matter altogether. We will tackle it in a dedicated article.


Frequently asked questions

What is difference-in-differences, in one sentence?
It is the comparison between the change in a group touched by an event and the change in a group not touched in the same period: the difference between the two changes is the estimate of the event’s effect.

Why is comparing CTR before and after not enough?
Because CTR changes with position and with everything that happens to the site in the same period. Expected CTR removes the effect of position, the control group removes that of everything else.

How do we choose the control group?
It must be little or not at all exposed to the event and, for the rest, subject to the same influences as the exposed pages. For AI answers, transactional pages are a good candidate; the choice must be argued and then checked against the trends of the previous months.

What if the pre-event trends are not parallel?
Then the difference of the differences does not measure the effect of the event. We can look for a better control group, model the trend explicitly or, at the very least, state that the estimate is not reliable.


Try it yourself

The code is all above, and the simulation is designed to be modified. Three directions to put the method to the test:

  1. Let us break the parallel trends. We multiply the true_ctr of the guides by exp(-0.03 * (data$month - 1)), that is, we make the guides lose 3% a month from the start, before the event. What is the difference of the differences now? Does the trend test on the before period notice it?
  2. Let us estimate the curve on all twelve months instead of the first six only, and redo the table of ratios: how much do the coefficients of the curve change, and how much the estimated effect? Why does the effect change much less than the curve?
  3. Let us choose a wrong control. We apply the event to half of the product pages too (with a smaller effect, say −20%) and redo the estimate: what happens to the effect when the control group is “contaminated”?

A hint: all three tests answer the same question. The method is not a formula that produces the truth from any data, but a line of reasoning that works as long as its assumptions hold: it is worth seeing with our own eyes how wrong it goes when they do not.


Throughout the article we have taken for granted that Search Console clicks and impressions always measure the same thing. They do not: sometimes a number changes all at once not because people’s behaviour changed, but because the way of counting it did, as in the case of the &num=100 parameter. Telling a change in the phenomenon apart from a change in the yardstick is the subject of the next article, on structural breaks in metrics.


Further reading

For anyone working with a site’s data, Trustworthy Online Controlled Experiments by Kohavi, Tang and Xu is the reference on online experiments, and devotes a chapter to observational studies: exactly what can (and cannot) be concluded when we did not draw the control group ourselves. For the broader question of how to tell what causes what from data we did not design, The Art of Statistics by David Spiegelhalter is a clear and honest guide, with no formulas getting in the way.

Leave a Reply

Your email address will not be published. Required fields are marked *