Expected vs Actual CTR: Find Underperforming Pages in R

Anyone who spends their days inside Search Console knows that little nagging feeling: a page sits steadily in third position, yet the clicks are few, a CTR that looks like it belongs at the bottom of the page.
The question we usually ask is the wrong one: not “how many clicks does it get?”, but the more uncomfortable one — “how many clicks should it get, sitting where it sits?”. Without a benchmark, a 3% CTR tells us nothing: for position 8 it would be excellent, for position 2 a small disaster.
What we are missing, in order to judge, is an expected CTR: the value to compare the actual one against.

We have already seen, talking about correlation, that position and CTR move together along a steep curve; and that the next step — using one variable to predict another — is the job of linear regression.
Here the two threads tie together: we turn that curve into an expected CTR and measure, page by page, how far each one deviates from it. It is the way to stop reading CTRs as absolute numbers and start reading them for what they really are: deviations from a norm.

What we will cover:

Why a CTR, on its own, means nothing
Modelling the CTR curve: three roads
An example with Search Console data
Residuals: who earns less than they should
Reading the deviations without fooling ourselves
Try it yourself
Further reading

Why a CTR, on its own, means nothing

There are industry tables telling us what the CTR of each position “should” be: the first around 25-30%, the second below half, and so on going down. They are useful as a general horizon, but to judge our pages they lead us astray: CTR depends on the type of query (a heavily clicked brand term or a cold informational search), on the sector, on how crowded the SERP is with ads and rich snippets.
The average CTR of “position 3” on an American e-commerce benchmark has almost nothing to say to our technical blog in Italian.

The way out is to stop comparing ourselves with an external table and build the reference curve on our own data.
We take all the pages, their average positions and their CTRs, and we trace the curve that describes the typical behaviour of CTR as position changes for the way our own site works. That curve becomes the yardstick: the expected CTR of a page is the value the curve assigns it, given its position.
The gap between the actual CTR and that expected value is the information we were after.

A CTR only makes sense next to the position that produced it: it is the deviation from the curve, not the absolute number, that tells us whether a page is working well or leaving clicks on the table.

Modelling the CTR curve: three roads

Building the curve means estimating a function that, given the position, returns the expected CTR. We already know the shape of that curve by eye: it starts high, plummets across the first positions and then flattens towards zero. A straight line does not describe it; we need something that curves.

There is, however, a detail that changes the whole way of reasoning, and it is the kind of deviation we care about.
We do not care that a page gets “two CTR points less” than expected: at the top of the SERP two points are crumbs, at the bottom they are a doubling. We care about the multiplicative deviation — “it earns half of what it should”, “it earns double”.
And a multiplicative deviation is best handled on a logarithmic scale, where a ratio becomes a difference.

The most natural shape for a curve of this kind is the power law, that is the idea that CTR is proportional to position raised to a negative exponent:

\( \text{CTR} = a \cdot position^{b} \\ \)

where \( a \) sets the general level and \( b \) (negative) governs how fast the descent is. The beauty arrives when we take the logarithm of both sides, which turns that curve into a straight line:

\( \log(\text{CTR}) = \log(a) + b \cdot \log(position) \\ \)

In other words: the logarithm of CTR is a linear function of the logarithm of position. And estimating a straight line is exactly what we know how to do with regression. From here, three roads to build the curve.

The first, and the one I recommend as the workhorse, is a linear regression on the logarithms — lm(log(ctr) ~ log(position)). It is in base R, it is interpretable (the slope \( b \) is the elasticity of CTR to position: by what percentage CTR drops for each percentage point of extra position), and its residuals are already on a logarithmic scale, hence multiplicative, exactly as we need.
It also extrapolates to rarely observed positions, and it can be weighted by impressions (weights = impression), so that pages with a handful of clicks do not skew the curve as much as those with tens of thousands of views. It is a pragmatic choice rather than the theoretically optimal weight (for a proportion the variance also depends on the CTR itself), but in practice it works very well.

The second is non-linear regression with nls, which estimates \( a \) and \( b \) directly on the natural scale of CTR without going through logarithms. It is an elegant refinement, but it must be primed with sensible starting values (which we fish out precisely from the log-log regression) and on messy data it may fail to converge. I keep it for when I need a clean parameter to put in a report, not as a starting point.

The third is local smoothing with loess, which imposes no shape on the curve and lets the data “draw it”. It is perfect for seeing the trend at a glance, but it wobbles on the tails (few pages in first position) and above all it does not extrapolate: outside the observed range it has nothing to say. It is an exploratory tool, not the model on which to base judgements.

So: we start from the log-log regression weighted by impressions as the working model, we compare it by eye with a loess to check we are not forcing the wrong shape, and we move to nls only if we need the explicit exponent. Let us see it at work.

An example with Search Console data

We start from an extract like the one anyone can download from Search Console: one row per page, with impressions, clicks and average position. In reality CTR is the ratio of clicks to impressions; here, since these are example data, we go the other way round — we set a plausible CTR and reconstruct the clicks.
I build the table in R with twelve example pages (with, on purpose, a couple of anomalous cases):

gsc <- data.frame(
  page       = c("/technical-seo-guide","/seo-audit-checklist",
                 "/keyword-research-guide","/campaign-roi-calculator",
                 "/google-analytics-tutorial","/statistics-glossary",
                 "/seo-tool-review","/link-building-guide",
                 "/competitor-analysis","/attribution-model",
                 "/ranking-report","/meta-tag-optimization"),
  impression = c(  9800, 5400, 12500, 2100, 7600, 1500,
                   8300, 4200,  6100,  900, 3300, 1800),
  position   = c(  1.3,  2.1,   3.4,  4.0,  4.6,  5.2,
                   2.8,  6.1,   7.0,  8.3,  9.1, 10.2)
)
# observed CTR (usually computed as click / impression)
gsc$ctr   <- c(0.232, 0.150, 0.034, 0.071, 0.066, 0.060,
               0.171, 0.048, 0.041, 0.012, 0.031, 0.028)
gsc$click <- round(gsc$impression * gsc$ctr)

I now estimate the expected-CTR curve with the regression on logarithms, weighting each page by its impressions:

fit <- lm(log(ctr) ~ log(position), data = gsc, weights = impression)
round(coef(fit), 3)
# (Intercept)  log(position)
#      -1.240          -1.088

The slope is −1.088: a value close to −1 describes an almost inversely proportional curve, where doubling the position (going, say, from 3 to 6) cuts the CTR roughly in half.
It is the same steep drop we had glimpsed when measuring correlation, but now written in a formula we can query: given a position number, it returns the typical CTR that position implies on our site.

Residuals: who earns less than they should

Having the curve means being able to compute, for each page, its expected CTR and compare it with the actual one. The comparison, as we said, must be made in terms of a ratio and not a difference: ratio = actual_ctr / expected_ctr. A value around 1 says the page earns as predicted; well below 1 that it is leaving clicks on the table; well above 1 that it captures more than its share.
I compute the expected CTR, the ratio, and flag the cases that truly deviate — but only if they have enough impressions to make their CTR reliable:

gsc$ctr_exp <- exp(predict(fit))           # back from the log scale to the natural one
gsc$ratio   <- gsc$ctr / gsc$ctr_exp

gsc$flag <- ifelse(gsc$ratio < 0.6 & gsc$impression >= 1000, "UNDER",
             ifelse(gsc$ratio > 1.4 & gsc$impression >= 1000, "OVER", "ok"))

gsc[order(gsc$ratio),
    c("page","position","impression","ctr","ctr_exp","ratio","flag")]

n.b. predict gives us the logarithm of the expected CTR, because that is the scale on which we estimated the model: exp brings it back to an actual CTR. Strictly speaking exp returns the median of the expected CTR, not the arithmetic mean (under log-normal errors the true mean is a touch higher), but for the relative ratios we care about the distinction is immaterial.
The output, sorted from the lowest ratio to the highest:

page	position	impression	ctr	ctr_exp	ratio	flag
/attribution-model	8.3	900	0.012	0.029	0.41	ok
/keyword-research-guide	3.4	12500	0.034	0.076	0.44	UNDER
/technical-seo-guide	1.3	9800	0.232	0.218	1.07	ok
/campaign-roi-calculator	4.0	2100	0.071	0.064	1.11	ok
/seo-audit-checklist	2.1	5400	0.150	0.129	1.16	ok
/competitor-analysis	7.0	6100	0.041	0.035	1.18	ok
/ranking-report	9.1	3300	0.031	0.026	1.18	ok
/link-building-guide	6.1	4200	0.048	0.040	1.19	ok
/google-analytics-tutorial	4.6	7600	0.066	0.055	1.20	ok
/meta-tag-optimization	10.2	1800	0.028	0.023	1.21	ok
/statistics-glossary	5.2	1500	0.060	0.048	1.25	ok
/seo-tool-review	2.8	8300	0.171	0.094	1.81	OVER

The case that jumps out is /keyword-research-guide: it sits in third position, where the curve would expect a CTR of 7.6%, and instead it gathers a meagre 3.4% — less than half of what it should, on twelve thousand five hundred impressions that make the figure rock solid.
It is a strong, immediately actionable hypothesis: in all likelihood the title and the meta description are not doing their job, and a rewrite could unlock clicks the position had already earned.

At the opposite end there is /seo-tool-review, which in second-to-third position earns almost double the expected. It is not a problem, it is a lesson: something in that snippet works beautifully — a magnetic title, a rich card, a perfect match with intent — and it is worth understanding what, to try to replicate it elsewhere. Residuals are not only there to find the sick ones: over-performances are the case studies from which to learn what, on our site, makes people click.

Reading the deviations without fooling ourselves

There is a detail in the table that is the heart of the whole matter, and that is easy to miss. /attribution-model has a ratio of 0.41 — even lower than /keyword-research-guide — yet we did not flag it. The reason is in the impressions column: nine hundred.
A CTR computed on so little data is almost pure noise, and next month it will drift back towards its mean regardless of anything we do. Flagging it as a “page to optimise” would send us chasing a ghost. Two pages with the same deviation, two opposite verdicts, and the only thing making the difference is the amount of data behind them.

A word of caution: the expected CTR is a conditional typical value — the median the curve associates with a position — not a law of nature. A page can “under-perform” for reasons that have nothing to do with the title: a brand query inflating competitors’ CTR, a featured snippet or a block of ads eating the clicks before the first organic result, a purely informational intent already satisfied by reading the snippet. And a CTR built on few impressions measures almost nothing: it will regress towards its mean on its own, as we saw talking about regression to the mean. A negative residual is a hypothesis to verify — “maybe the title earns little here” — not a verdict to execute.

It is also worth remembering that we estimated the curve on our own data, and that data influences it: a handful of very anomalous pages can tilt it just enough to shift the judgements on the others.
We can already see it in our table: ten pages out of twelve have a ratio above 1, but it is not that the site “over-performs” almost everywhere. It is that the single large negative deviation, /keyword-research-guide, weighs a great deal (twelve thousand five hundred impressions) and pulls the curve downwards, raising the ratio of all the others as a side effect. The “centre” of the cloud, in short, is not exactly 1, and it is better to read the ratios in relative terms — who sits well below and who well above the bulk of the group — rather than against the hard threshold of one.
This is why the impression weighting is precious, and why it pays to recompute the curve whenever the picture changes, instead of treating it as a constant carved in stone.

Try it yourself

The best way to internalise the mechanism is to get your hands on it. Building on the code above, there are three interesting directions to explore:

Aggregate by query instead of by page: the same page can appear on dozens of searches with different positions and CTRs, and often it is there — on the single query — that the missed opportunity hides. The curve and the residuals are built in the exact same way.
Replace the log-log regression with a loess(ctr ~ position) and compare the two expected CTRs: where do they match and where do they diverge? On the tails (first positions, few pages) which of the two seems more prudent to you?
Change the impression threshold below which you do not trust the CTR: going from 1000 to 3000, which pages leave the radar? The right number does not exist in the abstract, it depends on how much traffic your site moves.

A hint: the structure never changes — you estimate the curve, you predict the expected value, you look at the ratio. It is by playing with the threshold and the level of aggregation that you really understand how much of what we call an “under-performing page” is signal and how much is, simply, noise.

Spotting a page that earns less than expected for its position is a close cousin of another problem every analyst knows: spotting a day that earns less than expected over time, a drop or a spike in traffic that does not square with the usual trend.
It is the same reasoning on residuals — observed value against expected value — moved from the space of positions to the axis of time, where the expected value is provided by the historical trend of the series. From there springs anomaly detection: telling signal from noise when the numbers move over time, and the next step of our path.

Expected vs Actual CTR: finding the pages that earn fewer clicks than their position deserves