We have already tackled a close problem, classifying the intent of a query with Naive Bayes — but there we had an ingredient we lack today: a set of already labelled examples to learn from. Here nobody has handed us the labels. This is the territory of clustering, one of the most used tools of machine learning, and in this article we build it in R with its two classic algorithms: K-means and hierarchical clustering.

What we will cover:

• Grouping without labels: the idea of clustering
• K-means: centroids and the problem of choosing k
• Reading the clusters: who are these groups?
• Hierarchical clustering: the dendrogram
• Which method, and the traps
• Try it yourself
• Further reading

Grouping without labels: the idea of clustering

The difference with Naive Bayes is one of principle, not of detail. There we did supervised learning: we had queries already marked as informational, navigational or transactional, and we taught the algorithm to recognise new ones. Here we do unsupervised learning: nobody told us which and how many groups exist. Clustering does not verify a label we already know: it looks for a structure we did not know was there.

Grouping requires two things. The first is describing each keyword with numbers: in our example we will use search volume, cost per click (cpc), average position and word count.
The second is a notion of distance: two keywords are “close” if their numbers are alike. The most common distance is the Euclidean one, the same we would use on a map, only computed in a four-dimensional space (one per metric).

There is, however, a trap to defuse straight away. Volume is measured in tens of thousands, cpc in cents of a euro: left as they are, the distances would be dominated by volume, and cpc would count for almost nothing.
Before computing any distance we must put all the variables on the same scale, standardising them — in R with the scale() function, which subtracts the mean from each column and divides by the standard deviation. Only then does one euro of difference in cpc and ten thousand searches of difference in volume “weigh” the same.

K-means: centroids and the problem of choosing k

The idea of K-means is almost naive in its simplicity. We decide into how many groups (k) we want to split the data; the algorithm places k representative points, the centroids, and then repeats two steps until things settle: it assigns each keyword to the nearest centroid, then moves each centroid to the centre of the keywords assigned to it. At each pass the groups grow a little more cohesive, until they stop moving.

What the algorithm tries to minimise, in words, is the internal spread of the groups: the sum of the (squared) distances of each point from the centroid of its own cluster. In a formula:

where \( C_k \) is the k-th cluster, \( \mu_k \) its centroid and the double sum runs over all points of all groups. The lower the WCSS (within-cluster sum of squares), the more compact the groups.

The sore point remains: we have to decide k ourselves, before starting. A help comes from the elbow method: we try several values of k and watch how the WCSS falls. At first adding a cluster helps a lot, then the improvements become marginal; the “elbow” of the curve — the point where the descent flattens — suggests a reasonable k. I build the keyword table and compute the WCSS from 1 to 6 groups in R:

kw <- data.frame(
  keyword = c("running shoes","running shoes men","nike pegasus","nike pegasus 40",
              "best trail running shoes 2026","how to choose running shoes",
              "running shoes deal","buy trail shoes online",
              "trail vs road running shoes","running shoes overpronation",
              "asics gel nimbus","saucony endorphin","trail running shoes review",
              "discount running shoes","running shoe store london"),
  volume   = c(40000,18000,12000,8000,2400,1900,3200,880,1300,2100,9000,4000,1500,2600,720),
  cpc      = c(0.45,0.55,0.30,0.35,0.40,0.10,0.95,1.10,0.08,0.30,0.28,0.33,0.15,0.90,0.85),
  position = c(3.1,4.2,2.0,5.5,8.1,11.2,6.0,9.4,14.0,7.3,2.5,6.8,12.1,5.9,4.7),
  n_words  = c(2,3,2,3,5,5,3,4,6,3,3,2,3,3,4)
)

# standardise the four metrics (different scales -> same weight)
X <- scale(kw[, c("volume","cpc","position","n_words")])

# elbow method: WCSS for k from 1 to 6
set.seed(1)
wss <- sapply(1:6, function(k) kmeans(X, centers = k, nstart = 10)$tot.withinss)
round(wss, 1)
# [1] 56.0 34.4 20.7 12.4  8.7  6.9

The drop is steep up to three groups (56 → 34 → 21) and then slows down markedly (21 → 12 → 9 → 7). The elbow is never a sharp line — it is a reading, not a theorem — but here it reasonably points to k = 3. So I run K-means with three centroids:

set.seed(1)
km <- kmeans(X, centers = 3, nstart = 25)
kw$cluster <- km$cluster
table(km$cluster)
# 1 2 3
# 4 7 4

n.b. the argument nstart = 25 restarts the algorithm 25 times from different initial centroids, keeping the best solution: K-means can in fact get stuck in a local minimum depending on where it starts, and restarting several times is the standard defence. The set.seed(1) only serves to make the example reproducible (including the cluster numbering, which is itself arbitrary).

Reading the clusters: who are these groups?

Having three groups is useless until we understand what they represent. The most direct way is to look at the average metrics of each cluster.
I compute them on the original scale (not the standardised one, which is unreadable):

aggregate(kw[, c("volume","cpc","position","n_words")],
          by = list(cluster = kw$cluster), FUN = mean)
#   cluster volume  cpc position n_words
# 1       1   1775 0.18    11.35    4.75
# 2       2  13300 0.37     4.49    2.57
# 3       3   1850 0.95     6.50    3.50

Now the groups speak.
Cluster 2 collects the very high-volume queries (13,300 searches on average), short (two-three words), well positioned and with modest cpc: they are the generic and brand heads — “running shoes”, “nike pegasus”, “asics gel nimbus”. Cluster 1 has low volumes, long queries (almost five words), low positions and minimal cpc: it is the informational long-tail — “how to choose running shoes”, “trail vs road running shoes”, “trail running shoes review”. Cluster 3 stands out for a very high cpc (€0.95) and contains the clearly commercial queries — “running shoes deal”, “buy trail shoes online”, “discount running shoes”, “running shoe store london”.

It is worth pausing a moment on what just happened. Without giving the algorithm any label, the three groups that emerge closely echo a split by intent — informational research, generic and brand heads, commercial queries — close to the one that with Naive Bayes we had instead to teach it through examples.
The alignment, mind you, is not magic: it emerges because the metrics we chose (cpc, length, position) indirectly track intent, not because clustering knows it — of the queries’ meaning, here, it has not seen a single word. The reading is still immediately actionable: the informational cluster calls for articles and guides, the commercial one for product and offer pages, the high-volume one for robust pillar pages.

Hierarchical clustering: the dendrogram

K-means forced us to choose k in advance. Hierarchical clustering flips the approach: it decides nothing a priori, and builds instead a complete tree of groupings. It starts with each keyword as a group of its own, then progressively merges the two closest, then the two closest among those remaining, and so on up to a single big group. The result is a dendrogram: a tree showing at what “height” (that is, at what distance) each merge happens.

I build it in R by first computing the distance matrix, then the tree:

d  <- dist(X)                       # euclidean distances between standardised keywords
hc <- hclust(d, method = "ward.D2") # hierarchical tree (Ward's criterion)
plot(hc, labels = kw$keyword)       # the dendrogram

The beauty of the dendrogram is that we make the decision on how many groups to keep after seeing it, simply by “cutting” it at a chosen height: a low cut leaves many small groups, a high cut a few large ones. I cut at three groups and compare with K-means:

kw$cluster_hc <- cutree(hc, k = 3)
table(kw$cluster_hc)
# 1 2 3
# 8 3 4

The partition is not identical to the K-means one (here the groups have 8, 3 and 4 keywords), and that is normal: the two methods optimise different criteria and on little data the differences show.
But the substance of the groupings — the high-cpc transactional block, the high-volume heads, the informational tail — remains recognisable in both. When two different methods converge on the same story, we can trust that story a little more.

Which method, and the traps

The choice between the two, in practice, follows a few simple rules. K-means is fast and efficient even on tens of thousands of keywords, but it wants to be told k and tends to build “spherical” groups of similar size. The hierarchical one does not ask for k in advance and gives the dendrogram — precious for seeing how the groups nest inside one another — but becomes heavy when the keywords are too many. The most common practice: explore with hierarchical on a sample to get a sense of the number of groups, then apply K-means to the whole set with the k thus identified.

A word of caution, the most important of all: clustering always finds groups, even when there are none. Even pure noise comes back dutifully split into k tidy clusters. The number of groups, the metrics chosen to describe the keywords, the standardisation: they are all our decisions, and each one changes the result. A grouping is not a truth discovered in the data, it is a working hypothesis that only makes sense if it survives the test of business common sense. If we cannot explain a cluster in words, it probably does not really exist.

There is then a question of dimensions. Here we used four metrics, but in real operations the variables describing a keyword can be many more, and with many dimensions distances lose meaning (everything tends to look equally far). It is exactly the problem that principal component analysis knows how to ease, compressing many metrics into a few components before passing the baton to clustering.

Try it yourself

The best way to understand clustering is to watch it change its answer as the choices change. Building on the code above:

1. Skip the standardisation: run K-means directly on kw[, c("volume","cpc","position","n_words")] without scale(). Do the groups all collapse onto volume? It is the practical demonstration of why we standardise.
2. Change k: try four or five groups and re-read the averages. Does the informational cluster split into sensible sub-themes or are you just cutting noise?
3. Change the merging criterion of the hierarchical method: method = "complete" or "average" instead of "ward.D2". Does the dendrogram change shape? And the groups cut at k=3?

A hint: always keep an eye on the per-cluster averages with aggregate(). It is there, and not in the code, that you decide whether a grouping is useful or just an elegant partition of nothing.

We grouped the keywords by how they behave — volume, cost, position.
But what matters most to an SEO is left out: their meaning. Two queries can have different metrics and mean the same thing, or similar metrics and opposite intents. Grouping by sense, and not only by numbers, means turning the very text of the queries into something measurable: it is the job of text mining, where words become vectors and similarity is computed on language. And that is where we will pick up next.

Further reading

If you want to go deeper into clustering — K-means, hierarchical methods, the choice of the number of groups and the pitfalls of interpretation — An Introduction to Statistical Learning by James, Witten, Hastie and Tibshirani devotes a lucid chapter to unsupervised learning, with R labs that retrace exactly the steps we saw here. It is the reference I recommend to anyone who wants to move from the toy example to clustering on real data.

We have already seen, talking about correlation, that position and CTR move together along a steep curve; and that the next step — using one variable to predict another — is the job of linear regression.
Here the two threads tie together: we turn that curve into an expected CTR and measure, page by page, how far each one deviates from it. It is the way to stop reading CTRs as absolute numbers and start reading them for what they really are: deviations from a norm.

What we will cover:

• Why a CTR, on its own, means nothing
• Modelling the CTR curve: three roads
• An example with Search Console data
• The other two roads: loess and nls at work
• Residuals: who earns less than they should
• Reading the deviations without fooling ourselves
• Try it yourself
• Further reading

Why a CTR, on its own, means nothing

There are industry tables telling us what the CTR of each position “should” be: the first around 25-30%, the second below half, and so on going down. They are useful as a general horizon, but to judge our pages they lead us astray: CTR depends on the type of query (a heavily clicked brand term or a cold informational search), on the sector, on how crowded the SERP is with ads and rich snippets.
The average CTR of “position 3” on an American e-commerce benchmark has almost nothing to say to our technical blog in Italian.

The way out is to stop comparing ourselves with an external table and build the reference curve on our own data.
We take all the pages, their average positions and their CTRs, and we trace the curve that describes the typical behaviour of CTR as position changes for the way our own site works. That curve becomes the yardstick: the expected CTR of a page is the value the curve assigns it, given its position.
The gap between the actual CTR and that expected value is the information we were after.

A CTR only makes sense next to the position that produced it: it is the deviation from the curve, not the absolute number, that tells us whether a page is working well or leaving clicks on the table.

Modelling the CTR curve: three roads

Building the curve means estimating a function that, given the position, returns the expected CTR. We already know the shape of that curve by eye: it starts high, plummets across the first positions and then flattens towards zero. A straight line does not describe it; we need something that curves.

There is, however, a detail that changes the whole way of reasoning, and it is the kind of deviation we care about.
We do not care that a page gets “two CTR points less” than expected: at the top of the SERP two points are crumbs, at the bottom they are a doubling. We care about the multiplicative deviation — “it earns half of what it should”, “it earns double”.
And a multiplicative deviation is best handled on a logarithmic scale, where a ratio becomes a difference.

The most natural shape for a curve of this kind is the power law, that is the idea that CTR is proportional to position raised to a negative exponent:

where \( a \) sets the general level and \( b \) (negative) governs how fast the descent is. The beauty arrives when we take the logarithm of both sides, which turns that curve into a straight line:

In other words: the logarithm of CTR is a linear function of the logarithm of position. And estimating a straight line is exactly what we know how to do with regression. From here, three roads to build the curve.

The first, and the one I recommend as the workhorse, is a linear regression on the logarithms — lm(log(ctr) ~ log(position)). It is in base R, it is interpretable (the slope \( b \) is the elasticity of CTR to position: by what percentage CTR drops for each percentage point of extra position), and its residuals are already on a logarithmic scale, hence multiplicative, exactly as we need.
It also extrapolates to rarely observed positions, and it can be weighted by impressions (weights = impression), so that pages with a handful of clicks do not skew the curve as much as those with tens of thousands of views. It is a pragmatic choice rather than the theoretically optimal weight (for a proportion the variance also depends on the CTR itself), but in practice it works very well.

The second is non-linear regression with nls, which estimates \( a \) and \( b \) directly on the natural scale of CTR without going through logarithms. It is an elegant refinement, but it must be primed with sensible starting values (which we fish out precisely from the log-log regression) and on messy data it may fail to converge. I keep it for when I need a clean parameter to put in a report, not as a starting point.

The third is local smoothing with loess, which imposes no shape on the curve and lets the data “draw it”. It is perfect for seeing the trend at a glance, but it wobbles on the tails (few pages in first position) and above all it does not extrapolate: outside the observed range it has nothing to say. It is an exploratory tool, not the model on which to base judgements.

So: we start from the log-log regression weighted by impressions as the working model, we compare it by eye with a loess to check we are not forcing the wrong shape, and we move to nls only if we need the explicit exponent. Let us see it at work.

An example with Search Console data

We start from an extract like the one anyone can download from Search Console: one row per page, with impressions, clicks and average position. In reality CTR is the ratio of clicks to impressions; here, since these are example data, we go the other way round — we set a plausible CTR and reconstruct the clicks.
I build the table in R with twelve example pages (with, on purpose, a couple of anomalous cases):

gsc <- data.frame(
  page       = c("/technical-seo-guide","/seo-audit-checklist",
                 "/keyword-research-guide","/campaign-roi-calculator",
                 "/google-analytics-tutorial","/statistics-glossary",
                 "/seo-tool-review","/link-building-guide",
                 "/competitor-analysis","/attribution-model",
                 "/ranking-report","/meta-tag-optimization"),
  impression = c(  9800, 5400, 12500, 2100, 7600, 1500,
                   8300, 4200,  6100,  900, 3300, 1800),
  position   = c(  1.3,  2.1,   3.4,  4.0,  4.6,  5.2,
                   2.8,  6.1,   7.0,  8.3,  9.1, 10.2)
)
# observed CTR (usually computed as click / impression)
gsc$ctr   <- c(0.232, 0.150, 0.034, 0.071, 0.066, 0.060,
               0.171, 0.048, 0.041, 0.012, 0.031, 0.028)
gsc$click <- round(gsc$impression * gsc$ctr)

I now estimate the expected-CTR curve with the regression on logarithms, weighting each page by its impressions:

fit <- lm(log(ctr) ~ log(position), data = gsc, weights = impression)
round(coef(fit), 3)
# (Intercept)  log(position)
#      -1.240          -1.088

The slope is −1.088: a value close to −1 describes an almost inversely proportional curve, where doubling the position (going, say, from 3 to 6) cuts the CTR roughly in half.
It is the same steep drop we had glimpsed when measuring correlation, but now written in a formula we can query: given a position number, it returns the typical CTR that position implies on our site.

The other two roads: loess and nls at work

We picked the log-log regression as our working model, but we had promised all three roads. It is worth seeing the other two genuinely at work on the same twelve data points, to understand what they add — and above all where they stumble.

Non-linear regression estimates \( a \) and \( b \) directly on the CTR scale, without going through logarithms. It must be primed with sensible starting values, which we fish out precisely from the log-log we just estimated: the intercept brought back to the natural scale with exp is our \( a \), the slope is our \( b \). I set it running in R:

start <- list(a = exp(coef(fit)[1]), b = coef(fit)[2])  # primed from the log-log
fit_nls <- nls(ctr ~ a * position^b, data = gsc,
               weights = impression, start = start)
round(coef(fit_nls), 3)
#      a       b
#  0.315  -1.073

It converges, and returns an exponent of −1.073, practically the same as the log-log, but with a bonus: now \( a \) and \( b \) are numbers you can read on the true CTR scale. An \( a \) of 0.315 says, in plain terms, that at the top of the SERP the typical CTR is around 31% — exactly the kind of clean parameter to put in a report.
The price we pay is fragility: without those starting values, or on noisier data, nls may fail to converge at all and hand us back only an error.

Local smoothing with loess does the opposite: it imposes no shape, it lets the data draw the curve. I estimate it and ask it for the expected CTR at a few positions, including the first:

fit_lo <- loess(ctr ~ position, data = gsc, span = 0.9)
round(predict(fit_lo, data.frame(position = c(1, 2, 3, 5))), 3)
# [1]    NA 0.173 0.111 0.029

And here is the limit in a single output: at position 1, loess returns NA. The minimum our data observe is 1.3, and outside that range loess refuses to commit — it does not extrapolate.
For a CTR curve this is a serious flaw: the first position, the one we care about most, is often the least populated, and that is exactly where the local method leaves us empty-handed.

Put on the same chart, the three roads tell the whole story at a glance:

In the middle, where pages abound, the three curves almost coincide: any method works when the data speak clearly. It is on the tails that they part ways — loess wobbles behind the few pages it finds and stops dead at the edge of the data, while lm and nls continue smoothly even where observations are scarce.

So: the weighted log-log stays the workhorse — interpretable, extrapolable, with residuals already on a multiplicative scale. nls refines it when we need a clean exponent to write down; loess is the critical eye that, before we trust the model, tells us whether we are forcing the wrong shape. Three different tools for a single craft: turning a position into an expected CTR.

Residuals: who earns less than they should

Having the curve means being able to compute, for each page, its expected CTR and compare it with the actual one. The comparison, as we said, must be made in terms of a ratio and not a difference: ratio = actual_ctr / expected_ctr. A value around 1 says the page earns as predicted; well below 1 that it is leaving clicks on the table; well above 1 that it captures more than its share.
I compute the expected CTR, the ratio, and flag the cases that truly deviate — but only if they have enough impressions to make their CTR reliable:

gsc$ctr_exp <- exp(predict(fit))           # back from the log scale to the natural one
gsc$ratio   <- gsc$ctr / gsc$ctr_exp

gsc$flag <- ifelse(gsc$ratio < 0.6 & gsc$impression >= 1000, "UNDER",
             ifelse(gsc$ratio > 1.4 & gsc$impression >= 1000, "OVER", "ok"))

gsc[order(gsc$ratio),
    c("page","position","impression","ctr","ctr_exp","ratio","flag")]

n.b. predict gives us the logarithm of the expected CTR, because that is the scale on which we estimated the model: exp brings it back to an actual CTR. Strictly speaking exp returns the median of the expected CTR, not the arithmetic mean (under log-normal errors the true mean is a touch higher), but for the relative ratios we care about the distinction is immaterial.
The output, sorted from the lowest ratio to the highest:

The case that jumps out is /keyword-research-guide: it sits in third position, where the curve would expect a CTR of 7.6%, and instead it gathers a meagre 3.4% — less than half of what it should, on twelve thousand five hundred impressions that make the figure rock solid.
It is a strong, immediately actionable hypothesis: in all likelihood the title and the meta description are not doing their job, and a rewrite could unlock clicks the position had already earned.

At the opposite end there is /seo-tool-review, which in second-to-third position earns almost double the expected. It is not a problem, it is a lesson: something in that snippet works beautifully — a magnetic title, a rich card, a perfect match with intent — and it is worth understanding what, to try to replicate it elsewhere. Residuals are not only there to find the sick ones: over-performances are the case studies from which to learn what, on our site, makes people click.

Reading the deviations without fooling ourselves

There is a detail in the table that is the heart of the whole matter, and that is easy to miss. /attribution-model has a ratio of 0.41 — even lower than /keyword-research-guide — yet we did not flag it. The reason is in the impressions column: nine hundred.
A CTR computed on so little data is almost pure noise, and next month it will drift back towards its mean regardless of anything we do. Flagging it as a “page to optimise” would send us chasing a ghost. Two pages with the same deviation, two opposite verdicts, and the only thing making the difference is the amount of data behind them.

A word of caution: the expected CTR is a conditional typical value — the median the curve associates with a position — not a law of nature. A page can “under-perform” for reasons that have nothing to do with the title: a brand query inflating competitors’ CTR, a featured snippet or a block of ads eating the clicks before the first organic result, a purely informational intent already satisfied by reading the snippet. And a CTR built on few impressions measures almost nothing: it will regress towards its mean on its own, as we saw talking about regression to the mean. A negative residual is a hypothesis to verify — “maybe the title earns little here” — not a verdict to execute.

It is also worth remembering that we estimated the curve on our own data, and that data influences it: a handful of very anomalous pages can tilt it just enough to shift the judgements on the others.
We can already see it in our table: ten pages out of twelve have a ratio above 1, but it is not that the site “over-performs” almost everywhere. It is that the single large negative deviation, /keyword-research-guide, weighs a great deal (twelve thousand five hundred impressions) and pulls the curve downwards, raising the ratio of all the others as a side effect. The “centre” of the cloud, in short, is not exactly 1, and it is better to read the ratios in relative terms — who sits well below and who well above the bulk of the group — rather than against the hard threshold of one.
This is why the impression weighting is precious, and why it pays to recompute the curve whenever the picture changes, instead of treating it as a constant carved in stone.

Try it yourself

The best way to internalise the mechanism is to get your hands on it. Building on the code above, there are three interesting directions to explore:

1. Aggregate by query instead of by page: the same page can appear on dozens of searches with different positions and CTRs, and often it is there — on the single query — that the missed opportunity hides. The curve and the residuals are built in the exact same way.
2. Drop the impression weighting — lm(log(ctr) ~ log(position)) without weights — and compare the exponent and the verdicts with the weighted version: how much do the conclusions shift once the pages with more data no longer have a louder voice? It is the way to feel how much the weighting matters, instead of taking it on faith.
3. Change the impression threshold below which you do not trust the CTR: going from 1000 to 3000, which pages leave the radar? The right number does not exist in the abstract, it depends on how much traffic your site moves.

A hint: the structure never changes — you estimate the curve, you predict the expected value, you look at the ratio. It is by playing with the threshold and the level of aggregation that you really understand how much of what we call an “under-performing page” is signal and how much is, simply, noise.

Spotting a page that earns less than expected for its position is a close cousin of another problem every analyst knows: spotting a day that earns less than expected over time, a drop or a spike in traffic that does not square with the usual trend.
It is the same reasoning on residuals — observed value against expected value — moved from the space of positions to the axis of time, where the expected value is provided by the historical trend of the series. From there springs anomaly detection: telling signal from noise when the numbers move over time, and the next step of our path.

Further reading

If you want to go deeper into regression, logarithmic transformations and the reading of residuals — the very backbone of the model we built here — and then push beyond the power law towards local methods like loess, An Introduction to Statistical Learning by James, Witten, Hastie and Tibshirani is the book I recommend: it covers both the “why” of the logarithms and the “how” of interpreting coefficients, with hands-on labs in R, always starting from applied problems.

The trouble is that queries are many and always new, and classifying them by hand does not scale. We need a method that learns from a handful of labelled examples and then copes on its own with queries it has never seen. The algorithm that does this with almost disarming elegance is Naive Bayes, and — as the name hints — it starts once again from the Bayes’ theorem that has accompanied us throughout this path.

What we will cover:

• From the theorem to the classifier
• Training on labelled queries
• Classifying new queries
• The limits of “naive”
• Try it yourself
• Further reading

From the theorem to the classifier

Let us pick up the thread. Bayes’ theorem tells us how to update the probability of a hypothesis in the light of the observed data. Here the hypothesis is “this query belongs to the informational class” (or transactional), and the data are the words that make up the query. We want, in other words, the probability of a class given the text: in symbols, P(class | words).

Computing it directly would be a nightmare, because the possible combinations of words are boundless. Bayes lets us flip the problem around: instead of asking how probable the class is given the words, we ask how probable those words are given the class — a question the training data can answer. The rule, first in words and then in a formula, is that the probability of a class given a query is proportional to the prior probability of the class times the probability of observing those words if the class were that one:

Let us unpack the symbols. P(class) is the prior: how frequent that class is to begin with, before looking at the text (if half of the example queries are informational, the prior is 0.5). P(word | class) is how often that word appears in the queries of that class. The symbol ∏ is simply the product: we multiply the contributions of all the words in the query. The sign ∝ (“proportional to”) reminds us that we are dropping a constant denominator, identical across classes: since in the end we only care which class wins, we can ignore it with no harm.

And here it is, the spot where “naive” hides. Multiplying the probabilities of the individual words as if they were independent of one another amounts to assuming that, given the class, the presence of one word says nothing about the presence of the others. It is a plainly false assumption about real language — “credit” and “card” certainly do not appear at random independently of each other — and it is precisely this naivety that gives the algorithm its name. The surprising thing, as we shall see, is that despite starting from so crude an assumption the method works beautifully in practice.

Training on labelled queries

Training a Naive Bayes means one thing only: counting. For each class we count how many times each word appears in the example queries, and from those counts we derive the P(word | class). No optimisation, no iterations: we scan the data once and fill a table of frequencies.

Let us start from a small set of hand-labelled queries, five informational and five transactional. I train the classifier in R like this:

train <- list(
  info  = c("how to bake a cake", "what is statistics", "seo guide for beginners",
            "free r tutorial", "how bayesian works"),
  trans = c("buy shoes online", "iphone price deal", "best cheap hosting",
            "purchase seo course", "gym membership discount")
)
tok <- function(s) unlist(strsplit(tolower(s), "\\s+"))
vocab <- unique(unlist(lapply(unlist(train), tok)))
counts <- lapply(train, function(docs) {
  w <- table(factor(unlist(lapply(docs, tok)), levels = vocab)); w + 1
})
tots <- sapply(counts, sum); V <- length(vocab)
prior <- sapply(train, length) / length(unlist(train))   # 0.5 / 0.5

Let us see what happens line by line. The tok function splits each query into lowercase words (a spartan tokenisation, but enough). vocab is the vocabulary, the list of all distinct words seen in training. For each class, table(factor(...)) counts the occurrences of each vocabulary word; tots is the total of the counts per class, and prior here is 0.5 and 0.5 because the two classes have the same number of examples.

There is a detail in that w + 1 worth pausing on, because it is the trick that holds the whole edifice up. If a word never appears in the queries of a class, its count is zero, and with it the entire product of probabilities would collapse to zero: a single unknown word would be enough to drive the class probability to zero, wiping out the contribution of all the others. It is the classic case where “multiplying by zero” ruins the party. The fix is called Laplace smoothing: we add 1 to the count of every word, in every class, before computing the proportions. No word any longer has exactly zero probability, only a very small one.

The price of smoothing is that the per-class totals gain the number of vocabulary words (the units added one per word): that is why tots is not the raw token counts of the two classes but the smoothed totals. With these numbers, for example, the word “buy” (present once among the transactional queries, never among the informational ones) ends up clearly more probable under trans than under info — and it is exactly this imbalance that pushes a query toward the right intent. A neutral word like “seo”, appearing once on each side, stays roughly balanced between the two classes and does not move the needle.

Classifying new queries

With the frequency table ready, classifying a new query is genuinely child’s play: we tokenise it, sum the logarithms of the probabilities of each word (summing logarithms instead of multiplying probabilities avoids the product of many tiny numbers going into numerical underflow, but the result is the same), add the logarithm of the prior, and pick the class with the highest score. I compute it in R:

classify <- function(query) {
  w <- tok(query)
  logp <- log(prior)
  for (cl in names(train)) {
    p <- counts[[cl]][w]; p[is.na(p)] <- 1            # out-of-vocab word: neutral weight
    logp[cl] <- logp[cl] + sum(log(as.numeric(p) / tots[cl]))
  }
  names(which.max(logp))
}
cat("'buy seo course discount' ->", classify("buy seo course discount"), "\n")
cat("'how to learn statistics'   ->", classify("how to learn statistics"), "\n")

The output is clear-cut:

'buy seo course discount' -> trans
'how to learn statistics'   -> info

As we can see, the classifier assigns the first query to the transactional intent and the second to the informational one, exactly as a human would have. It is worth noting how it gets there. In the first query “buy” and “discount” pull decisively toward trans, “seo” stays neutral, and not even “course” — which in training appeared in the transactional “purchase seo course” — rows against it: the verdict is solid. In the second, “statistics” is a markedly informational word, and “learn”, though out of vocabulary, does no harm thanks to that p[is.na(p)] <- 1 which assigns never-seen words a neutral weight, identical for both classes: having nothing to say, it simply does not vote.

A handful of example queries per side is very little, and yet the mechanism is already all here. In a real case it is enough to replace the handfuls of labelled queries with a few hundred or thousand queries pulled from Search Console and annotated for intent, and the same code — unchanged in structure — becomes an intent classifier you can actually use.

The limits of “naive”

Before diving in, though, it is worth knowing where the method shows its seams, because its simplicity has a flip side.

The most obvious limit is precisely the independence assumption it takes its name from. By treating each word as detached from the others, Naive Bayes ignores order and context entirely: to it “cheap running shoes” and “the economics of running shoes” are the same bag of words. In intent classification this matters little, but in subtler tasks it can mislead. Then there is the question of out-of-vocabulary words: a query made only of terms never seen in training would be decided by the prior alone, that is by a pure coin toss — the sign that the example dataset is too thin and needs widening.

A note of caution that holds for any classifier, and all the more for one trained on little data: the model learns exactly what we show it, biases included. If the transactional example queries all contain the word “buy”, the classifier will associate purchase intent with that term and will struggle on an equally transactional but lexically different “add to cart”. The quality and representativeness of the labelled data matter more than the sophistication of the algorithm: a Naive Bayes fed with varied, balanced examples beats a sophisticated model trained badly.

That said, Naive Bayes remains a genuinely valuable tool to have in the box: it is extremely fast to train, needs little data to get going, is interpretable (we can always go and look at which words drove the decision) and in text classification it holds its own against far more complex models. It is often the baseline to beat before bringing in heavier artillery — and this is where the door opens onto machine learning proper, where classification is done with trees, regressions and networks that drop the independence assumption in exchange for more power (and less transparency).

Try it yourself

The code above is a perfect playground for building intuition. Search queries are not only informational or transactional: at least a third family is missing, the navigational one (someone searching “facebook login” or “gironi blog” just wants to reach a specific site). Here are a few changes to try:

1. Add a nav class to train with four or five navigational queries (“facebook login”, “youtube”, “amazon sign in”, “gmail inbox”), then retrain: the prior will no longer be 0.5 but roughly a third per class. How do the classifications of the earlier queries change?
2. Feed the classifier an ambiguous query like “iphone review” (informational? transactional?) and see which way it leans. Did the verdict make sense, given the words in training?
3. Remove the smoothing (replace w + 1 with w) and try classifying a query with a rare word: what happens to the score when a count is zero? It is the quickest way to see with your own eyes why Laplace is needed.

The nice part is that the structure of the code never changes: adding a class just means lengthening the train list, and everything else — vocabulary, counts, prior, classification rule — adapts on its own.

With this we close the Bayesian thread we have unspooled article after article: from estimating a conversion rate to comparing variants, from the adaptive allocation of traffic to this last leap, from deciding to classifying. The same theorem, reworn each time in a different guise, has proved a surprisingly sturdy common thread. From here the road forks toward machine learning in the full sense — decision trees, logistic regression, neural networks — where the methods give up the most comfortable assumptions in exchange for power, and where Bayes stays in the background as the grammar by which, in the end, we always learn from data.

Further reading

If you want to move from Naive Bayes to machine learning proper while keeping your feet on the ground (and R at hand), An Introduction to Statistical Learning by James, Witten, Hastie and Tibshirani is the book I recommend. It is the most accessible doorway into applied machine learning: it explains classification, trees and regression with the right rigour but without intimidating mathematics, and every chapter has R labs you can redo step by step. The second edition devotes space to Naive Bayes itself, so the jump from this article to the rest of the path is a natural one.

This article is part of the “The Bayesian Approach” path, a guided route through the articles on Bayesian statistics and inference for SEO.

There is a way to cut this bill, and it is called a multi-armed bandit. The idea is to shift traffic adaptively toward the variant that is winning while the test is still running, instead of waiting for the final verdict. In this article we build one with one of the most elegant and practical algorithms, Thompson sampling, which is the natural continuation of the Bayesian reasoning we have followed so far.

What we will cover:

• The exploration versus exploitation dilemma
• Thompson sampling: letting the posterior decide
• How much we really gain: the regret avoided
• When a bandit makes sense, and when it does not
• Try it yourself
• Further reading

Exploration versus exploitation

The name comes from slot machines: a one-armed bandit is the casino machine, and let us imagine we have several of them in front of us, each with an unknown win probability different from the others. We have a limited number of tokens. At each play we must choose which arm to pull. What is the strategy that maximises the total winnings?

The translation for anyone doing SEO or marketing is immediate: the arms are the variants (three versions of a title tag, of a call to action, of a landing page), the tokens are the visitors, the winnings are the conversions. Each visitor must be assigned to a variant, and we want to maximise total conversions across the whole experiment.

Here arises the underlying tension, the one that makes the problem interesting. On one hand we would like to exploit the variant that so far seems the best, to bank as many conversions as possible. On the other we must keep exploring the others too, because “so far it seems the best” rests on little data and might be a wrong impression.
It is a delicate balance: too much exploration and we waste traffic on mediocre variants; too much exploitation and we risk crowning the wrong winner on the basis of an early stroke of luck. The exploration-exploitation dilemma is the heart of every multi-armed bandit problem: each choice is at once an opportunity for gain and an opportunity for learning, and the two pull in opposite directions.

Thompson sampling: letting the posterior decide

The classic A/B test solves the dilemma in the crudest possible way: it explores and nothing more, in equal parts, until the end. Half the traffic to A, half to B, no adaptation. Thompson sampling solves it in a far cleverer, and almost surprisingly simple, way.

Let us pick up the Bayesian thread. For each variant we keep a posterior on its conversion rate: as we saw when estimating the conversion rate, starting from a Beta prior and observing binary outcomes (conversion yes/no), the posterior of each arm is again a Beta distribution, updated with every visitor. At the start, when we know nothing, each arm begins from a non-informative Beta(1, 1) prior.

Thompson’s rule, in words before formulas, is this: instead of asking “which is the variant with the highest mean so far?”, at each visitor we sample a rate at random from the posterior of each arm, and play the arm that produced the highest sample. It is a way of choosing “in proportion to the probability of being the best”: a variant we are very uncertain about can still win the draw now and then (and that is how it keeps being explored), but as the data accumulate its samples concentrate and, if it really is worse, it almost stops being chosen on its own.

The elegance lies precisely here: there is no exploration parameter to tune by hand. The uncertainty of the posterior is the engine of exploration. The more uncertain an arm, the more its samples are spread out, the more often it happens to win the draw and get tried; the more certain it becomes, the more its samples tighten around the true value and the arm gets played (or avoided) decisively.

I simulate in R the whole process on three variants with true rates of 5%, 7% and 9% (which the algorithm of course does not know), over 5000 visitors:

set.seed(7)
true_rates <- c(0.05, 0.07, 0.09)   # 3 variants, the third is the best
K <- length(true_rates); N <- 5000
alpha <- rep(1, K); beta_ <- rep(1, K)   # Beta(1,1) prior per arm
pulls <- rep(0, K); rewards <- rep(0, K)
for (t in 1:N) {
  theta <- rbeta(K, alpha, beta_)        # sample a rate per arm
  arm <- which.max(theta)                 # play the best sampled arm
  r <- rbinom(1, 1, true_rates[arm])      # outcome (conv yes/no)
  alpha[arm] <- alpha[arm] + r; beta_[arm] <- beta_[arm] + (1 - r)
  pulls[arm] <- pulls[arm] + 1; rewards[arm] <- rewards[arm] + r
}

The loop is all there is. At each iteration we sample a theta per arm (rbeta), choose the maximum (which.max), simulate the outcome (rbinom) and update the parameters of the arm played: a conversion increases its alpha by one, a non-conversion its beta_. No other logic, no threshold to calibrate.

How much we gain: the regret avoided

Let us see where the simulation took us. The first thing to look at is how the 5000 visitors were distributed across the three arms:

cat("pulls per arm:", pulls, "\n")
cat("bandit total conv:", sum(rewards), "\n")

Output: pulls per arm = 359, 278, 4363; bandit total conv = 424.

Of the 5000 visitors, a full 4363 ended up on the best variant, and only a little over 600 in total on the two worse ones. The algorithm, without our having told it anything about the true rates, worked out on its own which arm to reward and steered the vast majority of the traffic there. It is exactly the exploitation we wanted, reached through just enough early exploration to tell the arms apart.

Now the comparison that reveals the size of the advantage. A classic A/B test would have split the 5000 visitors equally across the three variants, roughly 1667 each, for the whole duration. The expected conversions in that scenario are simply the mean of the three rates times the number of visitors:

exp_ab <- sum(true_rates) / K * N
cat("expected conv equal-split A/B:", round(exp_ab), "\n")
cat("regret avoided (approx):", round(sum(rewards) - exp_ab), "conv\n")

Output: expected conv equal-split A/B = 350; regret avoided (approx) = 74 conv.

An equal-split A/B would have taken home about 350 conversions; the bandit collected 424. The difference, about 74 more conversions on the very same traffic, is what is technically called the regret avoided: the regret, that is the conversions lost by playing the wrong arms, that the adaptive allocation saved us. In plainer terms, for the same number of visitors the bandit converted over 20% more, simply by not insisting on the weak variants.
Note that we did not have to sacrifice anything to get it: at the end of the experiment we still know the winner (indeed we know it with great confidence, given the 4363 data points gathered on it), and in the meantime we converted more. This is the structural advantage of the bandit over the fixed-sample test.

When it makes sense (and when it does not)

That said, a bandit is not the right answer to every question, and it is important to understand where it shines and where instead a traditional A/B test remains preferable.

The bandit is at its best when traffic is continuous and long-running (an always-on page, a campaign running for months) and when the variants to compare are many: there the adaptive allocation pays off, because there is time and volume to shift the traffic and plenty of weak variants to drop quickly. It is ideal for ongoing optimisations where the goal is to maximise conversions along the whole journey, not to take a clean statistical snapshot at a given moment.

When instead the goal is precisely that snapshot — a precise, unbiased estimate of how much a variant is better, perhaps to defend in front of a client or to use for a strategic decision — the fixed-sample test remains more suitable: precisely because it explores in equal parts, it gathers balanced data on all variants and produces sharper estimates of the effect. The bandit, by concentrating early on the winner, gathers little data on the losers and therefore estimates by how much they are worse less well.

A note of caution: the bandit assumes that the conversion rates stay stable over time. But the real world is often non-stationary — seasonality, shifts in audience, a promotion that kicks in. If the best variant changes after the algorithm has already concentrated on another, a naive Thompson sampling like the one shown here struggles to notice, because it has stopped exploring the alternatives. In changing contexts we need variants that “forget” old data (for example by gradually discounting past observations), otherwise we risk staying anchored to a winner that no longer is one.

Try it yourself

Simulation is a perfect playground for building intuition. Taking the code above, try changing the starting numbers and observe how the pulls redistribute:

1. Bring the true rates closer, for example true_rates <- c(0.07, 0.08, 0.09): with more similar variants, how much more traffic is needed before the best arm breaks away? Does the concentration of pulls stay this sharp?
2. Cut the traffic drastically (N <- 500): with few visitors does the bandit still have time to spot the winner, or does exploration eat up the whole budget?
3. Add variants: move to four or five arms. With more alternatives to discard, does the advantage in regret avoided over the equal-split A/B grow or shrink?

Hint: the structure of the loop never changes, you just modify true_rates and N. It is precisely by watching how the pulls vector tips (or fails to tip) as a function of the distance between the rates that you truly grasp what Thompson sampling does under the hood.

So far we have used Bayes to decide between variants: estimate a rate, compare two, allocate traffic adaptively. But the same machinery — a prior, some data, a posterior — also serves a different and very common task: classifying, that is assigning to each new observation the most probable label given its features. It is the leap from deciding to classifying, and the algorithm that does it with disarming elegance, once again starting from Bayes’ theorem, is Naive Bayes: the subject of the next article.

Further reading

If you want to explore bandits applied to website optimisation, Bandit Algorithms for Website Optimization by John Myles White is the book I recommend. It is a slim, avowedly practical volume that compares A/B tests and bandit algorithms (epsilon-greedy, softmax, UCB) precisely from the standpoint of someone optimising pages and conversions, with the code at hand. It is the ideal starting point for anyone wanting to move from the simulation we have seen to a bandit that actually runs on their own site.

This article is part of the “The Bayesian Approach” path, a guided route through the articles on Bayesian statistics and inference for SEO.

The Bayesian approach answers both questions natively. In this article we apply it to the comparison of two variants, picking up the thread we left hanging when we estimated the conversion rate of a single variant.

It is the question behind every conversion test with an SEO angle: two versions of the same landing page competing for the same organic traffic, or two title-and-meta wordings judged by their actual versus expected CTR in the SERP. Wherever there are successes over trials — clicks over impressions, sign-ups over visits, conversions over sessions — the reasoning that follows is the same.

What we will cover:

• Two posteriors instead of one: a distribution per variant
• What is the probability that B really wins
• By how much it is better: the distribution of the difference
• When to stop: expected loss and the peeking problem
• Try it yourself
• Further reading

Two posteriors instead of one

When we estimated the conversion rate of a single variant, we saw that — starting from a Beta prior and observing binomial data — the posterior is again a Beta distribution. The updating rule was simple arithmetic: if the prior is Beta(α, β) and we observe \( k \) conversions out of \( n \) sessions, the posterior is:

Put in words: we add the observed conversions to the first parameter and the non-conversions to the second, nothing more.

In an A/B test we do not have a single proportion, we have two: one for the control (A) and one for the treatment (B). The mechanism, however, is identical: we build one posterior per variant, independently, applying the same rule twice.

Here is a quick example. We tested two versions of a landing page that receives organic traffic, assigning visitors at random:

• Variant A (control): 90 conversions out of 1000 sessions → raw rate 9.0%
• Variant B (treatment): 120 conversions out of 1000 sessions → raw rate 12.0%

We start from a non-informative Beta(1, 1) prior for both — “we know nothing, before the data every rate is equally plausible”. Applying the rule, the posterior of A is Beta(91, 911) and that of B is Beta(121, 881).

I build the two posteriors in R, sampling them by simulation:

set.seed(42)
# Variant A: 90 conv / 1000 ; Variant B: 120 conv / 1000
cA <- 90; nA <- 1000; cB <- 120; nB <- 1000

# Posteriors with uniform Beta(1,1) prior
postA <- rbeta(1e5, 1 + cA, 1 + nA - cA)   # Beta(91, 911)
postB <- rbeta(1e5, 1 + cB, 1 + nB - cB)   # Beta(121, 881)

Now we hold two distributions, not two numbers. And this is precisely the point: instead of comparing 9.0% against 12.0% as if they were fixed values, we compare the whole uncertainty surrounding them. The operational questions become operations on these distributions.

It helps to keep the picture in mind: two ridges of probability side by side along the conversion-rate axis. The further B’s ridge sits to the right of A’s, and the less the two overlap, the more likely it becomes that B is genuinely the better variant.

What is the probability that B wins?

The first question — the one the p-value never answers directly — is the probability that B is genuinely better than A.
With the posteriors in hand, the calculation is almost trivial: we compare B’s samples with A’s, pair by pair, and count in what fraction of cases B exceeds A. That fraction is the probability we are after.

I compute in R the probability that B beats A:

cat("P(B>A) =", round(mean(postB > postA), 3), "\n")

Output: P(B>A) = 0.985.

There is a 98.5% probability that variant B converts better than variant A.
Notice the change of register compared to the frequentist version. We are not saying “the observed difference is unlikely under the null hypothesis”: we are saying, directly, that given the evidence collected it is almost certain that B is the better variant. This is exactly the statement we would want to base a decision on — and the Bayesian approach hands it over without circumlocutions.

By how much it is better: the distribution of the difference

Knowing that B wins with 98.5% probability is not enough to decide. There is almost surely an improvement, but if it were two tenths of a percentage point, perhaps it would not be worth shipping the new page. The next question is therefore: by how much is it better?

So far we have kept A and B separate, one posterior each. But the object we actually care about is neither of them on its own: it is the distance between them. The answer therefore lives in the distribution of the difference between the two posteriors. We subtract, sample by sample, A’s rate from B’s: we obtain a new distribution, that of the uplift. From it we read both the typical value (the mean) and a credible interval that quantifies its uncertainty.

I compute in R the difference and its 95% interval:

diff <- postB - postA
cat("mean uplift (pct points) =", round(mean(diff)*100, 2), "\n")
cat("95% CI of difference =", round(quantile(diff, c(.025,.975))*100, 2), "\n")

Output: mean uplift = 3.00 pct points, 95% CI = [0.31, 5.68].

The expected gain is about 3 percentage points of conversion, with a 95% credible interval running from 0.31 to 5.68 points.
Here too the meaning is direct, not an abstract property of the procedure: there is a 95% probability that the true improvement of B over A lies between 0.3 and 5.7 percentage points. The interval does not touch zero, which confirms — consistently with the earlier 98.5% — that B is almost certainly superior. But the valuable figure is the width: the improvement could be modest (half a point) or robust (over five points), and this spread is information the operational decision must keep in mind.

When to stop: expected loss and the peeking problem

In the article on classic A/B testing we devoted space to one of the most insidious errors: peeking, that is, glancing at the interim data and stopping as soon as the difference looks significant. In the frequentist framework this inflates the false-positive rate, because each glance is effectively a new test on the same null hypothesis, and repeated tests multiply the chances of being wrong.

The Bayesian approach changes the nature of the problem. Here we are not repeating a test on a null hypothesis: we are updating a belief. Today’s posterior becomes tomorrow’s prior, and looking at the data as it arrives does not “consume” an error budget in the same way. This does not mean we can stop on a whim: we still need a stopping rule declared in advance. And the natural Bayesian rule is not “stop when P(B>A) is high”, but is based on expected loss.

The idea is this: if we choose B but A were in fact the better variant, we are wrong, and the size of the error is how much A beats B in those cases. The expected loss of choosing B is the average of this “regret” over all the residual uncertainty. In plain words: by how much, on average, we would regret having chosen B if we were wrong.

I compute in R the expected loss of choosing B:

# Expected loss of choosing B: average loss if A were actually better
loss_B <- mean(pmax(postA - postB, 0))
cat("expected loss choosing B =", round(loss_B*100, 3), "pct points\n")

Output: expected loss choosing B = 0.007 pct points.

The expected loss of choosing B is a mere 0.007 percentage points: negligible. In plainer terms, even in the unlucky scenario where we were wrong, the average damage would be tiny. We then set a tolerance threshold before starting — for example “I stop when the expected loss drops below 0.01 points” — and let the test run until we reach it.

And here is where we see why the expected loss, and not “P(B>A) is high”, is the right criterion. If we recompute at every step of the test, as traffic comes in, the two quantities do not trip at the same moment: the probability that B wins crosses 95% well before the expected loss becomes truly negligible.

Try it yourself

A lead generation website tests two variants of its contact form. On variant A we observe 45 conversions out of 600 sessions; on B, 52 conversions out of 600 sessions.

1. Build the two posteriors with a non-informative Beta(1, 1) prior: postA <- rbeta(1e5, 1 + 45, 1 + 555) and the analogue for B.
2. Compute P(B>A): is B better with a probability high enough to convince you?
3. Compute the mean uplift and 95% interval of the difference: does the interval touch zero?
4. Compute the expected loss of choosing B. With these numbers (closer to each other than in the case above), how does it change compared to the article’s example?

Hint: the structure of the code is identical to the one we used. Only the starting counts change — and the result, far less clear-cut, is precisely why the interval and the expected loss matter more than a plain “B won”.

So far we have compared two variants at a fixed sample size: we collect the data, compute, decide. And yet we have just watched that probability recompute itself at every step, as traffic comes in — and if one variant is turning out clearly better, why keep sending half the visitors to the worse one? We can do better: use that real-time probability to allocate traffic adaptively, shifting it toward the winning variant while the test is still running. It is the leap from the test to the bandit, the subject of the next article.

Further reading

If you want to explore Bayesian A/B testing with a practical, code-oriented angle, Bayesian Methods for Hackers by Cameron Davidson-Pilon is the book I recommend. It tackles Bayesian reasoning starting from programming rather than formal mathematics, and devotes an explicit chapter to the Bayesian comparison of variants — probability that B wins, distribution of the difference, expected loss. It is written for those who learn better by reading code than proofs.

This article is part of the “The Bayesian Approach” path, a guided route through the articles on Bayesian statistics and inference for SEO.

The problem always starts the same way. A small e-commerce store has collected 23 conversions out of 412 sessions. The raw rate is 23/412 ≈ 5.6%. A seemingly precise number. But how much do we trust it? We could be looking at the true 3% or the true 9% — with that sample, we simply do not know. The point estimate “5.6%” says nothing about its own uncertainty.

What we will cover:

• The Beta-Binomial model: why Beta is the natural distribution for a conversion rate
• Non-informative prior: letting the data speak
• Informative prior: using historical data without cheating
• Today’s posterior is tomorrow’s prior
• Try it yourself
• Further reading

From single click to rate: the Beta-Binomial model

Each session is a binary event: the user converts or does not.
With \( n \) sessions and \( k \) conversions, the generative mechanism is binomial. The parameter we want to estimate — the true conversion rate \( \theta \) — is a proportion: a value between 0 and 1.

When the prior on \( \theta \) is a Beta distribution and the data are binomial, something very convenient happens: the posterior is also a Beta distribution. This is called a conjugate prior, and it means Bayesian updating reduces to a simple arithmetic operation on the parameters.

The updating rule is: if the prior is Beta(α, β), after observing \( k \) conversions out of \( n \) sessions the posterior is:

In plain words: we add the observed conversions to α and the observed failures to β. The prior Beta(α, β) encodes in α the “conversions already seen” (or an equivalent belief) and in β the “non-conversions”. Each new observation updates both counters.

Non-informative prior: letting the data speak

The most neutral prior possible for a proportion is the uniform distribution on [0, 1], which corresponds to Beta(1, 1): all values of the rate are considered equally plausible before seeing any data.

Our case: 23 conversions out of 412 sessions (389 non-conversions).

We calculate the posterior in R:

# Observed data
conv <- 23; sess <- 412; nonconv <- sess - conv

# NON-INFORMATIVE prior: uniform Beta(1,1)
a0 <- 1; b0 <- 1
a1 <- a0 + conv; b1 <- b0 + nonconv      # posterior Beta(24, 390)

cat("Non-informative: mean =", round(a1/(a1+b1), 4), "\n")
cat("  95% CI =", round(qbeta(c(.025,.975), a1, b1), 4), "\n")

Output: mean = 0.058, 95% CI = [0.0376, 0.0824].

The posterior Beta(24, 390) has mean 5.8% and a 95% credible interval between 3.8% and 8.2%.
The Bayesian credible interval is not an abstract statistical exercise: it means directly that there is a 95% probability that the true conversion rate lies between 3.8% and 8.2%. Not a frequency over infinite repetitions — a direct probability statement on the parameter.

With 412 sessions, the uncertainty is still appreciable: almost 5 percentage points of width. The point estimate 5.6% was misleading in its precision.

A note of caution: the Bayesian credible interval and the frequentist confidence interval have similar numbers but profoundly different meanings. The frequentist 95% is a property of the procedure (“repeating the experiment 100 times, 95 intervals would contain the true parameter”); the Bayesian 95% is a direct statement about the parameter in the specific case we are analysing.

Informative prior: using historical data without cheating

The non-informative prior is the honest starting point when we know nothing. But often we do know something: years of campaigns, sector history, category benchmarks.
Our e-commerce has four seasons of history with an average conversion rate around 4%. How do we translate this knowledge into a prior?

The Beta(8, 192) distribution has mean exactly 8/(8+192) ≈ 4% and — because α+β = 200 — a concentration equivalent to “trusting” our data as much as 200 fictitious historical sessions. It is not an arbitrary number: it is a declared and verifiable choice.

We calculate the informative posterior in R:

# INFORMATIVE prior from history: mean ~4% -> Beta(8, 192)
a0i <- 8; b0i <- 192
a1i <- a0i + conv; b1i <- b0i + nonconv  # posterior Beta(31, 581)

cat("Informative: mean =", round(a1i/(a1i+b1i), 4), "\n")
cat("  95% CI =", round(qbeta(c(.025,.975), a1i, b1i), 4), "\n")

Output: mean = 0.0507, 95% CI = [0.0347, 0.0694].

The informative posterior Beta(31, 581) gives mean 5.1% and 95% credible interval between 3.5% and 6.9%.
Two things to notice. First: the mean drops slightly from 5.8% to 5.1% — the prior “pulls” the estimate toward the historical 4%. Second: the interval narrows (from 4.4 to 3.4 percentage points) — the historical data acts as additional information, so uncertainty decreases.

With limited data, the informative prior helps: it adds information where data alone is insufficient. With many data points — thousands of sessions — the prior gets overwhelmed by the data and the difference between informative and non-informative priors becomes negligible. This is a fundamental feature of Bayesian inference: the prior matters when data is scarce; data always wins in the end.

Today’s posterior is tomorrow’s prior

The most practical elegance of the Bayesian approach is sequential updating. After one month, new data arrives: 15 conversions from 300 additional sessions. We do not need to start over — the posterior we just calculated becomes the new prior.

We update in R:

# New data: 15 conversions from 300 additional sessions
conv2 <- 15; sess2 <- 300
a2 <- a1i + conv2; b2 <- b1i + (sess2 - conv2)   # Beta(46, 866)

cat("After update: mean =", round(a2/(a2+b2), 4), "\n")
cat("  95% CI =", round(qbeta(c(.025,.975), a2, b2), 4), "\n")

Output: mean = 0.0504, 95% CI = [0.0372, 0.0655].

Three stages compared:

The credible interval has narrowed at each stage: 5.3 percentage points for the pure prior, 3.4 after the first month, 2.8 after the update. The mean remained stable: the new data confirms the previous estimate instead of shifting it, and uncertainty decreases as expected.
This is the real operational advantage: there is no need to wait for a “large enough” sample to accumulate before making any estimate. We start from an uncertain estimate and refine it progressively, with each new data point.

Try it yourself

A lead generation website has a historical conversion rate around 2%. After an optimisation campaign, 8 conversions are observed from 150 sessions.

1. Build an informative prior reflecting the historical 2%: try Beta(4, 196) — it has mean exactly 2%.
2. Calculate the posterior after 8 conversions from 150 sessions.
3. Calculate the 95% credible interval.
4. Now try a non-informative Beta(1, 1) prior: does the posterior change much? Why?

Hint: the formula is always the same — qbeta(c(.025, .975), a0 + conv, b0 + nonconv). The only thing that changes is the starting point.

What we have built so far is the estimation of a rate for a single variant. The next step is comparing two variants — a control page and a modified page — and calculating the Bayesian probability that one beats the other. That is exactly what we will do in the next article: Bayesian A/B testing.

Further reading

If you want to understand how Bayesian reasoning enters practical decisions — from market forecasting to uncertainty estimation in real data — The Signal and the Noise by Nate Silver is the book I recommend. Silver devotes explicit chapters to Bayesian updating, showing it in concrete contexts (weather forecasting, politics, sports) that make the idea of “updating beliefs with new data” immediately intuitive. Rigorous but written like a story, it is a rare kind of book that leaves you thinking differently about uncertainty.

This article is part of the “The Bayesian Approach” path, a guided route through the articles on Bayesian statistics and inference for SEO.

Optimizely had measured the problem themselves, simulating A/A tests — two identical variants, where by construction neither is better than the other, so any declared “winner” is a false alarm.
According to the figures published by Optimizely, on tests of 5,000 visitors anyone checking the numbers after every visitor saw 57% of A/A tests declare a false winner at least once; checking every 500 visitors the figure dropped to 26%, every 1,000 to 20%. Chilling numbers for a tool that is supposed to help us decide with rigour. The rewrite — sequential inference plus false discovery rate control, what they call always-valid — was meant precisely to bring the error, as they put it, “from over 30% to 5%”.

It’s the same deception we ran into closing the article on regression to the mean: there we selected the worst-performing pages — an extreme instant in the space of the data — and let ourselves be fooled by their rebound. Here we select an extreme instant in time: we stop the moment the test proves us right. The mechanism is a cousin, the risk identical.

What peeking is

Anyone who runs an A/B test knows it well: the test is running, the data come in day after day, and the temptation to sneak a look at the dashboard is irresistible.
Peeking isn’t the mere act of looking: it’s looking while reserving the right to stop the test the moment the result becomes significant. It’s that “great, variant B has crossed the threshold, let’s wrap up here and declare the winner” said halfway through data collection.

The delicate point is that every look accompanied by the possibility of stopping is one more statistical test.
A single test with a 5% threshold accepts, by definition, a 5% chance of crying “winner” when in fact there’s no difference at all. But if we repeat that same test twenty times over the course of collection, and all we need is for just one of those twenty times to cross the threshold in order to stop and declare victory, then the chance of stumbling into a false positive is no longer 5%: it accumulates with every look.

This isn’t the usual multiplicity of someone comparing ten variants at once. Here the multiplicity is hidden in time: a single variant, looked at many times. It’s the same logic by which a single coin toss rarely gives a strange result, but if one is allowed to look after every toss and stop at the first favourable moment, sooner or later that moment arrives — and it gets mistaken for a signal.

What peeking costs: a simulation

Words convince us up to a point; numbers convince us far more. I simulate in R an A/A test, that is two variants with exactly the same true conversion rate (10%): any difference that emerges is noise, and any declared “victory” is a false positive by construction.
I set the stage by fixing the random number generator’s seed (so the numbers are reproducible), the function that computes the p-value of the comparison between two proportions, and the function that simulates a single experiment and reports whether at some point it declared a (false) winner:

set.seed(2025)

p_vero  <- 0.10    # same conversion rate for A and B (H0 true)
n_arm   <- 2000    # visitors per variant at the end of the test
n_sim   <- 4000    # number of simulated experiments
alpha   <- 0.05
sguardi <- 20      # how many times we "peek" during collection
look_at <- round(seq(n_arm / sguardi, n_arm, length.out = sguardi))

# p-value of a two-proportion, two-sided z-test
pval_ab <- function(xa, na, xb, nb) {
  pp <- (xa + xb) / (na + nb)
  se <- sqrt(pp * (1 - pp) * (1 / na + 1 / nb))
  2 * pnorm(-abs((xa / na - xb / nb) / se))
}

# one A/A experiment: TRUE if it declares a (false) winner
esperimento <- function(soglia, guarda) {
  a <- cumsum(rbinom(n_arm, 1, p_vero))
  b <- cumsum(rbinom(n_arm, 1, p_vero))
  for (k in guarda) {
    p <- pval_ab(a[k], k, b[k], k)
    if (!is.na(p) && p < soglia) return(TRUE)
  }
  FALSE
}

Let’s start from the correct behaviour: a single test, at the end, on the 2,000 visitors per variant. I run it 4,000 times and count how many declare a winner:

# fixed horizon: a single test, at the end
fisso <- mean(replicate(n_sim, esperimento(alpha, n_arm)))
cat(sprintf("Fixed horizon: %.1f%% false positives\n", 100 * fisso))
# Fixed horizon: 5.0% false positives

Out comes 5.0%: exactly the level we declared with the 5% threshold. The test, used as it should be, keeps its promise.
Now I change one thing only: instead of looking once at the end, I look twenty times during collection and stop at the first moment the p-value drops below 0.05. I add the intermediate looks and run again:

# peeking: a test at every look, stop at the first significant one
peek <- mean(replicate(n_sim, esperimento(alpha, look_at)))
cat(sprintf("Peeking (%d looks): %.1f%% false positives\n", sguardi, 100 * peek))
# Peeking (20 looks): 24.3% false positives

From 5.0% to 24.3%.
The same data, the same test, the same threshold: the only thing that changed is when we decided to look, and the false positive rate has nearly quintupled. Almost one A/A test in four, in which the two variants are identical by construction, convinces us we’ve found a winner. The 24.3% from our simulation and the 30% reported by Optimizely tell the same story with different data: peeking isn’t a venial sin, it’s the most effective way to fool ourselves.

Solution 1: the fixed horizon

The simplest cure is also the most annoying one: decide beforehand how much data to collect, and then have the discipline to wait until the end without stopping early, whatever the dashboard says in the meantime.
It’s what the simulation has just shown us: with a single test at the end, the false positive stays nailed to the promised 5%. No magic, just the elimination of opportunistic looks.

“How much data” isn’t a number plucked from thin air: it depends on how small a difference we want to be able to detect and on how much certainty we demand. It’s the sample size calculation, which is done before launching the test with our significance calculator and which rests on the concepts of effect size and power analysis.
Once that number is fixed, the fixed horizon is the safest road: no statistical correction to apply, no threshold to tweak. The price, though, is paid in patience — resisting the curiosity for days or weeks — and this, in operational reality, is exactly what almost no one manages to do.

Solution 2: looking without cheating

And if monitoring on the fly really were necessary — because a test that’s going terribly needs stopping, because the stakeholders want updates?
Then the way is not to look in secret with the usual threshold, but to look openly with a stricter one. The idea is simple: if at every look we raise the bar, making it harder to cry “winner” on each occasion, we can arrange for the overall error — summed across all the looks — to stay at the 5% we wanted. I calibrate in R the per-look threshold, trying ever more stringent values on the same twenty looks as before:

# stricter per-look threshold that brings the overall error back to ~5%
for (sg in c(0.05, 0.02, 0.01, 0.005)) {
  fp <- mean(replicate(n_sim, esperimento(sg, look_at)))
  cat(sprintf("  threshold %.3f -> %.1f%% overall\n", sg, 100 * fp))
}
#   threshold 0.050 -> 25.1% overall
#   threshold 0.020 -> 11.7% overall
#   threshold 0.010 ->  6.6% overall
#   threshold 0.005 ->  3.3% overall

As we can see, the usual 0.05 threshold produces a 25.1% overall error (the peeking disaster again), but as we make it stricter the error comes back down: around 0.01 — a threshold five times more stringent than the standard one — the overall error returns close to the nominal 5%. It’s the price to be paid for the right to peek: at every single look much more evidence is required, in exchange for the freedom to look often.

What we’ve just shown is a homemade, constant-threshold version of the idea. The “textbook” boundaries — more refined, with thresholds that change over the course of the test, like those of Pocock or O’Brien-Fleming — are obtained in R with the gsDesign package, and commercial tools like Optimizely use an always-valid variant (the so-called mSPRT) of the same underlying idea.
The fine mathematics changes, not the principle: to look often without cheating one must demand, at every look, more evidence than a single test would ask for.

A word of caution: a result seen during the test, on its own, proves nothing: what counts is when the decision to look was made.
The same p-value below 0.05 means different things depending on whether it’s the only fixed-horizon test or the first of the twenty at which one reserved the right to stop. Without declaring in advance how and when the data will be examined, any “winner” that emerges on the fly is suspect.

Try it yourself

To feel the mechanism up close, let’s start from the script and change a single parameter: the number of sguardi (looks).
Let’s go from weekly monitoring (few looks) to daily monitoring (many looks) and re-run the peeking simulation. What to expect: the more frequently one peeks, the higher the false positive rate climbs — the frequency of looks is the fuel of the problem. Then let’s redo the threshold calibration with that new number of looks and check that, by choosing a strict enough threshold, the overall error comes back under control all the same. It’s the proof, first-hand, that peeking isn’t a curse: it’s just a bill that has to be paid.

There’s one last trap in this family, perhaps the most insidious of all, because it doesn’t hide in our own data but in the data others tell us about. When we read an agency’s case study — “we increased conversions by 300% with this tactic” — we’re looking at a survivor: the thousand identical attempts that failed are something nobody mentions. It’s survivorship bias, the reason case studies lie even when they tell the truth, and it’s the next step on our journey through the pitfalls of marketing data.

Further Reading

On peeking, early stopping and sequential testing the reference — in English — remains Trustworthy Online Controlled Experiments by Ron Kohavi, Diane Tang and Ya Xu: written by people who led the experimentation platforms at Microsoft, Google and LinkedIn, it devotes explicit pages to all the ways a running A/B test can fool us, and to how to defend against them. It’s the book anyone who has to take online experiments seriously pulls out of the drawer.

The very same illusion waits for us every time we look at a site’s data and decide whether one of our changes “worked”.

It’s worth clearing up the name straight away, because it misleads: regression to the mean has nothing to do with linear regression, the model that fits a line between two variables. Here “regression” means going back, reverting: extreme values sliding back toward their average. They are two different things that happen to share a word.

What regression to the mean is

The mechanism is simple, and once you see it you can’t unsee it.
Almost every number we measure is the sum of two parts: a “true”, stable value and a dose of noise — random fluctuation of the moment. A page’s average SERP position in a given month depends on its real relevance, but also on chance: the algorithm wobbling, a competitor who pushed hard that month, the query’s seasonality, a handful of clicks more or fewer.

Now, when we single out the extreme cases — the worst-performing pages, the worst month — we are almost always picking situations where the noise pushed everything in the same unfavourable direction.
At the next measurement that noise won’t repeat identically, and the value will tend to climb back toward its true mean — without anyone having done anything. It’s a purely statistical fact, not an SEO phenomenon: the more extreme a measurement, the more likely the next one is less extreme.

The optimization that worked by accident

Let’s see what this mechanism does in everyday work. Suppose we track the average SERP position of 300 pages over two consecutive months. I simulate the scenario in R, giving each page a stable “true” position and adding a random fluctuation each month:

set.seed(48)

# 300 pages, each with its "true" SERP position (stable over time)
pos_vera <- runif(300, 3, 40)

# two consecutive months: same true position, different random noise
mese1 <- pos_vera + rnorm(300, 0, 8)
mese2 <- pos_vera + rnorm(300, 0, 8)

# the 60 worst pages in month 1 (in the SERP "worse" = higher number)
peggiori <- order(mese1, decreasing = TRUE)[1:60]

round(mean(mese1[peggiori]), 1)   # starting average position
# [1] 39.1

Our 60 worst pages start from an average position of 39.1: bottom-of-the-third-page territory. We decide to act — rewrite the titles, update the content, fix internal links — and we check again a month later.
Here’s the result, with no algorithm change in the meantime and, above all, with no real intervention at all in the simulation:

round(mean(mese2[peggiori]), 1)   # one month later
# [1] 33

From 39.1 to 33: about six positions gained. A jump that would look great in a report, and that anyone would be tempted to credit to the optimization just carried out.
Too bad there is no optimization in the code: the pages improved on their own, because they had been chosen precisely for being extreme and the noise that had sunk them in month 1 didn’t repeat. For scale: the average position of all 300 pages is about 22, and it’s toward that value that the worst ones are reverting.

How not to be fooled: the control group

If the improvement comes anyway, how can we tell whether our optimization had any real effect?
The answer is the same one we’d use for a drug: we need a control group. We split the 60 worst pages into two random halves: one we “optimize”, the other we deliberately leave alone. Then we compare how much they improve:

# split the 60 worst pages into two random groups
gruppo <- sample(rep(c("optimized", "control"), each = 30))

# average improvement (month1 - month2) in the two groups
round(tapply(mese1[peggiori] - mese2[peggiori], gruppo, mean), 1)
# control optimized
#     6.0       6.1

The “optimized” group gains 6.1 positions, the control group — left untouched — gains 6.0.
Practically the very same improvement. Our optimization, in the simulation, added nothing: all of the gain was regression to the mean, and the control exposes it by showing it would have happened anyway.

The lesson is this: an improvement, on its own, proves nothing.
When you act precisely on the pages (or campaigns) that were doing worst, part of their rebound is guaranteed regardless of you. Without a comparison against what you did not touch, you can’t know how much of the result is your doing and how much is just reversion toward the mean.

It’s the same reasoning, it should be said, behind a properly run A/B test: you compare the variant against a concurrent control, not against the “before”. And it’s a close cousin of the trap we met discussing correlation and causation: here too we mistake a sequence in time (“I acted, then it improved”) for a causal link.

Try it yourself

To lock in the mechanism, try rebuilding the scenario changing a single detail: instead of the 60 worst pages, select the 60 best of month 1 (order(mese1)[1:60], without decreasing) and watch how they behave in month 2.

What to expect: the champions revert toward the mean too, but by getting worse — top positions contain luck that doesn’t repeat. It’s the mirror image of the same phenomenon, and it explains why “that golden month” or “that page that was flying” so often can’t be reproduced: you hadn’t lost your magic touch, you were just drifting back toward your average.

There’s an even more insidious variant of this trap, because it hides inside the very tools we use to decide with rigour. If we look at a test while it’s still running and stop as soon as the numbers please us, we are picking an extreme instant exactly as we picked the worst pages — and we’ll be fooled in the same way. It’s the peeking problem, and it’s the next stop on our tour of marketing-data pitfalls.

Further Reading

The flight-instructor story we opened with comes from Thinking, Fast and Slow by Daniel Kahneman, Nobel laureate in economics: it’s the book that made regression to the mean — and dozens of other biases in our reasoning — widely known, told through examples that stick. If you keep one thing from this article, keep this one: it’s a vaccine against the illusion of having understood why the numbers move.

In this article:

• What an A/B test is
• Setting up an A/B test correctly
• Worked example: conversion rate of two landing pages
• The most common mistakes
• Frequentist vs Bayesian approach
• Practical SEO example: meta description A/B test
• FAQ
• Try it yourself

Over the previous articles we have looked at how hypothesis testing works and how the two-sample t-test lets us compare two groups rigorously. We have also built confidence intervals, learned to quantify the uncertainty of our estimates, and seen with the Central Limit Theorem why all this works even when the data are not normal.

But there is one question that, in the day-to-day reality of anyone doing SEO and marketing, comes up almost daily: which variant performs better? Which title tag brings more clicks? Which landing page converts more? Which meta description draws attention? It is not an academic question: it is the question that separates data-driven decisions from opinions disguised as strategies.

The good news is that we already have all the tools to answer it. A/B testing is nothing more than the direct application of the statistical inference concepts we have built step by step: hypothesis testing, comparison between groups, significance. In this article we put it all together.

What an A/B Test Is

An A/B test is, in essence, a controlled experiment: we take two variants of something (a page, a headline, a call-to-action), randomly assign users to one of the two variants, and measure which one produces better results.

Variant A is the control (the current version, the one we are already using). Variant B is the treatment (the new version we want to test). The logic is the same as a scientific experiment: we change one variable at a time, keep everything else constant, and observe whether the change produces a measurable effect.

Three elements make an A/B test reliable. Randomisation: users are assigned to A or B at random. This is essential, because if we showed A in the morning and B in the afternoon, any observed difference might depend on the time of day, not on the variant. The control group: without A as a reference, we wouldn’t know whether B’s results are good or bad. And finally a success metric defined in advance: CTR, conversion rate, time on page. The metric must be chosen before collecting the data, not after (we will come back to this point shortly).

The diagram summarises the path we will follow: from the initial question to the final decision, passing through hypotheses, randomisation, data collection and the statistical test. Every step has its pitfalls.

But why do we need statistics? Because data are noisy. If variant A has a CTR of 5.0% and variant B of 5.3%, is that difference real or just random fluctuation?
The naked eye cannot tell: we need a formal test. And it is precisely the two-sample test we have already seen — applied to proportions rather than means.

Setting Up an A/B Test Correctly

Before collecting data, we have to set up the test rigorously. Let’s see how.

Choosing the metric. The metric must be clear, measurable and directly linked to the goal. For a title tag, the natural metric is the CTR (Click-Through Rate). For a landing page, the conversion rate. For a blog article, perhaps the average time on page.
Always keep this in mind: a vague metric (“people like the page more”) is not a metric.

Defining the hypotheses. As in every statistical test, we start from a null hypothesis and an alternative hypothesis:

• \( H_0 \): the two variants have the same effect (no difference between A and B)
• \( H_1 \): the two variants have a different effect (a difference exists)

The test assumes the observations are independent and the assignment to variants is truly random. In practice, this means: no double-counting sessions from the same user, no time-of-day alternation (A in the morning, B in the afternoon). These conditions seem obvious, but they are violated more often than you might think.

The statistical test. When we compare two proportions (such as two CTRs or two conversion rates), the appropriate test is the two-proportion z-test. The logic is the same as the two-sample t-test, but adapted to binary data (click/no-click, conversion/no-conversion).

The test statistic is computed as follows. First, we compute the pooled proportion, which is our best estimate of the common proportion under the null hypothesis:

where \( x_1 \) and \( x_2 \) are the successes (clicks, conversions) in the two groups, and \( n_1 \) and \( n_2 \) the sample sizes.

Then we compute the z statistic:

The numerator is the observed difference between the two proportions; the denominator is the standard error under the null hypothesis.
The ratio tells us how many “standard-error units” separate the two proportions: the higher it is, the harder the difference is to attribute to chance.

Example: CTR of Two Title Tags

Let’s take a concrete example. We tested two title tag variants for an important page on the site:

• Title A (control): 1500 impressions, 75 clicks → CTR = 5.0%
• Title B (treatment): 1500 impressions, 105 clicks → CTR = 7.0%

Title B looks better, but is the difference statistically significant? Let’s compute it step by step.

Step 1: the pooled proportion:

Step 2: the standard error:

Step 3: the z statistic:

Step 4: the p-value. For a two-tailed test, \( p \approx 0.021 \).

So: the p-value is below 0.05. We can reject the null hypothesis and conclude that the difference between the two title tags is statistically significant. Title B has a significantly higher CTR — from a statistical standpoint.

The figure shows what “statistically significant” means with our data. The curve is the distribution of the standardised difference under H₀: if there were no real difference between A and B, the z-value would almost always fall between −1.96 and +1.96 (the red lines). Our z = 2.31 falls beyond the threshold, in the rejection region. The area shaded in orange is the p-value: the probability of observing a difference this large (or larger) by pure chance is only 2.1% — small enough to rule out chance.

Let’s run the same test in R:

n1 <- 1500; x1 <- 75    # Title A
n2 <- 1500; x2 <- 105   # Title B
p1 <- x1 / n1  # 0.05
p2 <- x2 / n2  # 0.07

p_pool <- (x1 + x2) / (n1 + n2)
se <- sqrt(p_pool * (1 - p_pool) * (1/n1 + 1/n2))
z <- (p2 - p1) / se
p_value <- 2 * (1 - pnorm(abs(z)))

cat("z =", round(z, 3), "\n")
cat("p-value =", round(p_value, 4), "\n")

Result: z = 2.306, p-value = 0.0211.

Worked Example: Conversion Rate of Two Landing Pages

Let’s move on to a more elaborate example. An e-commerce store is testing two variants of its landing page:

• Page A (current design): 1000 visitors, 35 conversions → conversion rate = 3.5%
• Page B (new design): 1000 visitors, 58 conversions → conversion rate = 5.8%

The difference looks substantial (2.3 percentage points), but with these numbers is it enough to rule out chance?

Let’s check in R with prop.test(), which runs the two-proportion test:

result <- prop.test(x = c(35, 58), n = c(1000, 1000))
print(result)

The function returns the p-value of the test and, very usefully, the confidence interval of the difference between the two proportions. In this case the p-value is about 0.019 — below 0.05, so the difference is statistically significant.

But it is the confidence interval of the difference that gives us the most valuable information: not only whether B is better than A, but by how much, and with what margin of uncertainty. If the CI of the difference runs from about 0.4 to 4.2 percentage points, we know that B is almost certainly better, and the improvement lies within that range. That is far richer information than a simple “yes, it’s significant”.

The p-value tells us whether we can doubt the null hypothesis; the confidence interval tells us how large the observed effect might be. They answer two different questions, and the CI answers the one we ultimately care about most: not just “is there a difference?”, but “how big is it?”.

n.b.: prop.test() applies a continuity correction (Yates’s correction) that makes the test slightly more conservative. For large samples the difference is negligible; for small samples, it is a welcome caution.

The Most Common Mistakes

A/B testing is a powerful tool, but a treacherous one. The ease with which a test can be set up hides serious methodological pitfalls. Let’s look at the most frequent ones.

Stopping the Test Too Early

It is the strongest temptation: after a few days, B looks clearly better than A. Why wait any longer?
Because those preliminary results are noise, not signal.

The problem has a technical name: peeking. Every time we look at the interim data and decide whether to stop, we increase the probability of a false positive. It’s like tossing a coin: if we stop every time we get three heads in a row, we’ll conclude the coin is rigged. But it isn’t — we simply haven’t given it enough tosses.

How to avoid it: define the required sample size beforehand and wait until you reach that number before drawing conclusions. In the meantime, you can use our sample size calculator to determine how many users you need before launching the test.

Testing Too Many Variants Without Correction

Another frequent mistake: testing three, four, five variants at the same time (A/B/C/D…) and then comparing them all pairwise. The problem is that of multiple comparisons: the more comparisons we make, the more likely we are to find at least one significant result by pure chance.

With 5 variants and 10 pairwise comparisons, the probability of finding at least one false positive rises from 5% to almost 40%. This is not a detail: it is an error that invalidates the entire test.

How to avoid it: if multiple comparisons are needed, apply a Bonferroni correction (divide the α threshold by the number of comparisons) or, better still, limit yourself to testing one variant at a time.

Ignoring the Power of the Test

We know the risk of a false positive well (type I error, α). But there is a mirror risk that is often ignored: the false negative (type II error, β). It happens when B really is better than A, but our test fails to detect it.

The most common cause? A sample that is too small. If we have only 100 visitors per variant, the test does not have enough “power” to detect small but real differences. We will conclude “no significant difference” not because the difference doesn’t exist, but because we didn’t have enough data to see it.

How to avoid it: compute the required sample size before launching the test, based on the minimum effect we want to detect. This is the subject of power analysis: use the sample size calculator to check whether your test has enough power.

Confusing Statistical Significance with Practical Significance

Frequentist vs Bayesian Approach

Everything we have seen so far follows the frequentist approach: we compute a test statistic, compare it with a reference distribution, obtain a p-value and make a binary decision (reject or fail to reject \( H_0 \)).

It works, and works well. But it has limits that you feel in everyday practice. The p-value does not tell us “by how much B is better than A”. It does not tell us “what the probability is that B is genuinely superior”. And if we collect new data, we cannot simply update the result: we have to recompute everything from scratch.

There is an alternative approach that answers directly the question we care about most: what is the probability that B is better than A?
It is the Bayesian approach, to which we have devoted a dedicated article: Bayesian A/B Testing, where we see how to build a Beta posterior for each variant, compute P(B > A), read the distribution of the difference (by how much B is better, not just whether) and decide when to stop the test using expected loss.

Practical SEO Example: Meta Description A/B Test

Let’s look at one last scenario, very common in everyday practice. We have two meta description variants for a key page on the site. Alternating the two versions (two weeks each, to minimise seasonal effects) and consulting the Search Console data, we get:

• Meta A: 3200 impressions, 128 clicks → CTR = 4.0%
• Meta B: 3100 impressions, 155 clicks → CTR = 5.0%

Let’s check in R:

prop.test(c(128, 155), c(3200, 3100))

The p-value is about 0.064 — above the 0.05 threshold, so we cannot reject the null hypothesis. The confidence interval of the difference also includes zero, confirming the non-significance. A borderline result, which tells us: with these data we don’t have enough evidence to conclude that Meta B is genuinely better.

Which approach should we use? For a simple test like this, the frequentist approach with prop.test() is more than sufficient: we have large samples, the question is clear. The Bayesian approach becomes more valuable when the samples are small, when we want to update the result as new data arrive, or when we have prior knowledge to incorporate (for example, we know that for that type of page the CTR is typically between 3% and 7%).

But the operational decision must not rest on the p-value alone. We have to ask: is the difference (one percentage point more of CTR) big enough to justify the change? With 3000-plus impressions a month, one percentage point more means about 30 additional clicks. Is that significant for our business?
This is a question statistics cannot resolve on its own — it is a judgement that falls to us.

FAQ

What is the difference between a one-tailed and a two-tailed test?
A two-tailed test checks whether B is different from A (better or worse). A one-tailed test checks whether B is better than A (or worse, depending on direction). When in doubt, always use the two-tailed test — it is more conservative.

Can I run A/B tests with more than two variants?
Yes, but each additional variant increases the risk of false positives. If you test 5 variants, you need to correct the significance threshold (e.g., Bonferroni). Better to test one variant at a time, unless you use a specific multivariate test design.

Does the p-value tell me how much better B is than A?
No. The p-value only tells you whether the observed difference is compatible with the null hypothesis. To know how much better B is, you need the confidence interval of the difference or an effect size measure.

When should I stop an A/B test?
When you have reached the planned sample size — not before. Stopping “because B is winning” is the most common trap (peeking). If the test runs longer than expected without reaching significance, it might mean the effect is too small to detect with the available data.

Try It Yourself

An e-commerce store is testing two call-to-action variants on a product page:

• Variant A (“Add to cart”): 450 visits, 23 conversions
• Variant B (“Buy it now”): 430 visits, 31 conversions

1. Compute the conversion rate of each variant
2. Run the test with prop.test(c(23, 31), c(450, 430)) and interpret the p-value
3. Does the confidence interval of the difference include zero?
4. At the 5% significance level, is the difference statistically significant?

Hint: if the p-value is above 0.05, we cannot conclude that one variant is better than the other — but this does not mean they are equal. It might simply mean we don’t have enough data. It is exactly the problem of the power of the test that we discussed.

A/B testing gives us a rigorous framework for making decisions based on data, not intuition. But as we have seen, a well-run test tells us whether there is a significant difference — it does not tell us how large that effect is, nor how much data we need to detect it with confidence. Those are the questions of effect size and power analysis, the next tools in our path. For the sample size, the interactive calculator lets you get the exact number in real time.

Further Reading

If you want to dig deeper into the methodology of online experiments, Trustworthy Online Controlled Experiments by Ron Kohavi, Diane Tang and Ya Xu is the world reference on A/B testing. The authors led the experimentation platforms at Microsoft, Amazon and LinkedIn — and the book covers everything, from test design to the pitfalls we saw in this article, all the way to the organisational aspects that make the difference between a well-run test and a sterile exercise.

For those who want to explore the Bayesian approach to A/B testing (which we have just introduced), Bayesian Statistics the Fun Way by Will Kurt is an accessible and surprisingly entertaining introduction. It explains priors, posteriors and Bayesian updating with examples that don’t require a maths degree — and it uses R for the computational part.

This article is part of the «The Bayesian Approach» path, a guided route through the articles on Bayesian statistics and inference for SEO.

In this article:

• What PCA is
• The mathematical foundations
• Practical applications
• PCA in web marketing
• Implementing PCA in R
• Verification and interpretation
• FAQ

SEO and web marketing analysis almost always presents us with the same problem: we have too many metrics and we don’t know which ones really matter. Search volume, CPC, competition, CTR, bounce rate, time on page, conversions by source, average position — the list grows with every new tool we add to our stack.

The problem is not a lack of data: it’s that these metrics are often correlated with each other (more impressions lead to more clicks, which lead to more conversions, which increase costs). When variables move together, most of the information they contain is redundant. Principal Component Analysis (PCA) is the tool we have to cut through this redundancy: it reduces a set of many correlated variables to a few uncorrelated components that capture most of the original information.

In clearer terms: PCA finds the “directions” in which our data vary the most, and lets us project everything else onto those few dimensions. It is like looking at a three-dimensional object from its most informative angle instead of from a random viewpoint.

What PCA Is

To grasp the idea of dimensionality reduction, picture a city with many interconnected streets. PCA works like an urban-planning system that identifies the main traffic arteries. By focusing on these “main roads”, we get a clear view of the city’s structure without having to analyse every single side street.

In the context of web marketing and data analysis, PCA is a powerful tool for several reasons. It is effective for visualising and exploring high-dimensional datasets, making it easy to spot trends, patterns or outliers. It is also commonly used in the data pre-processing stage for machine learning, extracting the most informative features from large datasets while preserving relevant information. Another advantage is its ability to *minimise or eliminate multicollinearity and overfitting***, frequent problems in web marketing datasets with many potentially correlated variables.

The Mathematical Foundations

To understand how PCA works we need to familiarise ourselves with a few concepts. Nothing scary — we take them one at a time.

Variance and covariance. Variance measures how spread out a single variable is around its mean. Covariance measures how two variables move together: positive means they tend to increase or decrease together, negative means they move in opposite directions. PCA looks for directions (components) that maximise variance — because more variance means more information.

The covariance matrix summarises the covariances between all pairs of variables. It has the variances on the diagonal and the covariances off the diagonal. It is the starting point of PCA.

Eigenvalues and eigenvectors. This is the mathematical core. The principal components are simply the eigenvectors of the covariance matrix. An eigenvector identifies a direction in the data space; the associated eigenvalue tells us how much variance that direction captures.

The formula is straightforward:

where \( \Sigma \) is the covariance matrix, \( \mathbf{v} \) is the eigenvector (the direction of the component), and \( \lambda \) is the eigenvalue (the variance along that direction). Finding the principal components means solving this equation for \( \Sigma \).

The figure makes the idea immediate. The grey dots are two correlated variables: as X increases, Y tends to increase too. The red arrow (PC1) is the direction in which the data vary the most — it is the principal axis. The green arrow (PC2) is orthogonal and captures the remaining variance. This is the geometric idea of PCA: finding the axes that “explain” most of the variability in the data.

Explained variance. This is the metric that tells us how valuable each component is. The proportion of variance explained by a component is:

where \( \lambda_k \) is the eigenvalue of the \( k \)-th component and \( p \) is the total number of components. The cumulative explained variance is the sum of the first \( k \) proportions: it tells us how much original information we keep by retaining only \( k \) components.

As a side note: criteria like the Kaiser rule (keep only components with eigenvalue > 1) and the scree plot (the ordered eigenvalues graph, with the “elbow” as the cut-off point) help choose the number of components to retain.

Practical Applications

PCA is a versatile technique with a wide range of applications. In image processing, it is used for compression. In genomics, it helps identify the most critical genes. In finance, for risk analysis and portfolio optimisation. In healthcare, for medical image analysis. In security, for biometric systems. In climatology, for analysing large environmental datasets.

For data analysis and marketing specifically, PCA makes it possible to simplify complex datasets, reduce noise, extract the most significant features, and improve the performance of predictive models. Its ability to visualise high-dimensional data in two or three dimensions makes it easier to spot patterns, trends and outliers.

An important clarification: PCA does not eliminate some variables while keeping others. Instead, it builds new variables — the principal components — obtained as linear combinations of the original ones. It is not feature selection: it is feature extraction. The difference is subtle but crucial.

PCA in Web Marketing

Let’s see how PCA applies to concrete problems in our daily work.

Keyword analysis. A keyword dataset has numerous metrics: search volume, competition, CPC, average position on Google and Bing. By applying PCA, we can condense these dimensions into a few components that capture the underlying themes. For example, we might find that one component captures the “potential value” of the keyword (high volume + high CPC) and another captures “competitiveness” (high competition + low ranking).

Traffic metric analysis. Sessions, bounce rate, time on page, conversions by source — PCA can reveal latent variables driving performance. One component might capture user engagement and another the effectiveness of different traffic sources.

User segmentation. By analysing behavioural data with many variables, PCA identifies natural groupings of users, enabling more defined segments.

Campaign performance analysis. Impressions, clicks, conversions, cost, CTR, CPA — PCA reveals the key factors that determine campaign success.

Implementing PCA in R

Let’s now run PCA on real data, with two examples that mirror the scenarios we just described.

First, let’s set up the keyword positioning data:

set.seed(123)
n_keywords <- 100
keywords <- paste0("keyword_", 1:n_keywords)
search_volume <- round(runif(n_keywords, min = 100, max = 10000))
competition <- runif(n_keywords, min = 0.1, max = 0.9)
cpc <- round(rnorm(n_keywords, mean = 2.5, sd = 1), 2)
ranking_google <- round(rnorm(n_keywords, mean = 15, sd = 10), 0)
ranking_bing <- round(rnorm(n_keywords, mean = 12, sd = 8), 0)

keyword_data <- data.frame(
  Keyword = keywords,
  Search_Volume = search_volume,
  Competition = competition,
  CPC = cpc,
  Ranking_Google = ranking_google,
  Ranking_Bing = ranking_bing
)

head(keyword_data)

Result:

    Keyword Search_Volume Competition  CPC Ranking_Google Ranking_Bing
1 keyword_1          2947   0.5799912 1.79             37            6
2 keyword_2          7904   0.3662588 2.76             28            6
3 keyword_3          4149   0.4908904 2.25             12            4
4 keyword_4          8842   0.8635791 2.15             20            4
5 keyword_5          9411   0.4863219 1.55             11            9
6 keyword_6           551   0.8122802 2.45             10           15

Now the campaign performance data:

set.seed(456)
n_campaigns <- 50
campaign_ids <- paste0("campaign_", 1:n_campaigns)
impressions <- round(runif(n_campaigns, min = 1000, max = 100000))
clicks <- round(impressions * runif(n_campaigns, min = 0.01, max = 0.1))
conversions <- round(clicks * runif(n_campaigns, min = 0.005, max = 0.05))
cost <- round(clicks * runif(n_campaigns, min = 0.1, max = 2), 2)
ctr <- round((clicks / impressions) * 100, 2)
cpa <- round(cost / conversions, 2)
cpa[is.nan(cpa)] <- 0

campaign_data <- data.frame(
  Campaign_ID = campaign_ids,
  Impressions = impressions,
  Clicks = clicks,
  Conversions = conversions,
  Cost = cost,
  CTR = ctr,
  CPA = cpa
)

head(campaign_data)

Result:

  Campaign_ID Impressions Clicks Conversions    Cost  CTR    CPA
1  campaign_1        9866    873          14 1093.32 8.85  78.09
2  campaign_2       21841   1788          20 3360.17 8.19 168.01
3  campaign_3       73563   2866          66 2764.48 3.90  41.89
4  campaign_4       85361   4121          73 1422.12 4.83  19.48
5  campaign_5       79051   3432         133 1623.28 4.34  12.21
6  campaign_6       33864   3064         126 6047.70 9.05  48.00

Now run PCA with prcomp(). It is essential to scale the data (scale. = TRUE) before applying PCA: otherwise variables with larger scales (thousands of impressions vs fractions of CPC) would dominate the analysis.

pca_keywords <- prcomp(keyword_data[, 2:6], scale. = TRUE)
summary(pca_keywords)

pca_campaigns <- prcomp(campaign_data[, 2:7], scale. = TRUE)
summary(pca_campaigns)

Results:

# summary(pca_keywords) — keywords (5 variables, 5 components)
Importance of components:
                          PC1    PC2    PC3    PC4    PC5
Standard deviation     1.1381 1.0298 0.9894 0.9305 0.8941
Proportion of Variance 0.2591 0.2121 0.1958 0.1732 0.1599
Cumulative Proportion  0.2591 0.4712 0.6670 0.8401 1.0000

# summary(pca_campaigns) — campaigns (6 variables, 6 components)
Importance of components:
                          PC1    PC2    PC3     PC4    PC5     PC6
Standard deviation     1.7837 1.2229 0.9303 0.49392 0.4250 0.18138
Proportion of Variance 0.5303 0.2492 0.1442 0.04066 0.0301 0.00548
Cumulative Proportion  0.5303 0.7795 0.9238 0.96442 0.9945 1.00000

The figure tells the story better than the numbers. For keywords (left) the variance is fairly evenly distributed: the metrics are largely uncorrelated and PCA cannot compress them much without losing information. For campaigns (right), the first two components explain almost 80% of the variance — the metrics are strongly correlated and two dimensions suffice to describe nearly everything.

The enlarged plot on the campaign data confirms: PC1+PC2 exceed 80% cumulative variance. The dashed line at 80% is a common threshold — below it lie components that contribute little.

The loadings (pca_keywords\( rotation) show the correlation between original variables and components, helping interpret the meaning of each component. The scores (pca_keywords \)x) represent the projection of the original data onto the new space.

For further visualisation, you can use the scree plot (plot(pca_keywords)) and the biplot (biplot(pca_keywords)), which displays both scores and loadings in the plane of the first two components.

Verification and Interpretation

Interpreting principal components requires domain knowledge. If in the keyword data PCA the first component has high positive loadings for search volume and CPC, it might represent “high-potential keywords”. If the second component is dominated by ranking, it might represent “actual visibility”. Interpretation is always context-dependent.

Keep in mind that principal components do not have a “natural” meaning. They are mathematical constructs that we must interpret by looking at the loadings of the original variables. There is no predefined label for PC1: we have to build it, based on the data and the context.

FAQ

When does it make sense to use PCA?
When we have many correlated variables and want to reduce them to a few interpretable dimensions. It is ideal for exploration, visualisation, and pre-processing for machine learning. It does not make sense when variables are already few and independent.

How many components should I keep?
It depends on the cumulative explained variance. A rule of thumb: stop when the scree plot curve flattens (the “elbow”), or when cumulative variance reaches 70-80%. The Kaiser rule (eigenvalue > 1) is another criterion, but should be used flexibly.

Why do I need to scale the data before PCA?
Because PCA maximises variance. If one variable is measured in thousands (impressions) and another in hundredths (CTR), the first would artificially dominate the analysis. Scaling (mean 0, standard deviation 1) puts all variables on an equal footing.

Does PCA work with non-linear data?
No, standard PCA assumes linear relationships. For non-linear structures there are variants like Kernel PCA, or non-linear techniques such as t-SNE and UMAP.

Conclusion

Principal Component Analysis gives us an elegant way to untangle the complexity of our data. It reduces dimensionality, reveals hidden patterns, improves predictive models, and makes visualisable what would otherwise be a jumble of numbers.

Its power, however, also lies in its limitations: it works well when relationships are linear, requires scaling, and the interpretation of the components is our responsibility, not the algorithm’s. Used with awareness, it is one of the most versatile tools in the web marketing data analysis toolkit.

Further Reading

Principal Component Analysis is covered with exemplary clarity in An Introduction to Statistical Learning by James, Witten, Hastie and Tibshirani, alongside other unsupervised learning techniques.