The redesign is approved, the launch date is set, and sooner or later the question nobody wants to face comes up: which pages do we check one by one?
There are usually two answers, and both are wrong. “All of them” is impossible when the URLs showing up in Search Console number in the thousands. “The main ones” means nothing, because everyone has their own in mind: the homepage, the service pages, that article that once did well.
There is a third answer, and it can be measured in half an hour: rank the pages by traffic, count how many are needed to reach half, 80%, 90% of clicks, and check how stable those lists are. We have already met the tool when discussing the Gini index and the Lorenz curve. Here we put it to work on a concrete decision.
In short: on many sites organic traffic is highly concentrated, and a few dozen URLs often bring in half of the clicks. How concentrated it is on a given site, though, has to be measured. Ranking URLs by clicks and accumulating their shares gives the operational thresholds (how many URLs for 50, 80, 90% of clicks), which tell us where to focus the manual work of a migration: a redirect map checked by hand, content checks, monitoring after launch. The lists should be checked for stability, by shifting the time window and resampling the days (bootstrap), and combined with pages that matter for other reasons: conversions, inbound links, documents. The Gini index sums up concentration in a single number, but it changes a lot depending on whether it is computed over all URLs with impressions or only over those with at least one click: the set it refers to must always be stated.
What we will cover:
Let us clear one thing up straight away, to avoid a dangerous misunderstanding. In a migration every URL that changes address and has an equivalent on the new site must get its permanent redirect (301), whether it has a lot of traffic or very little: that is not what is being discussed here. URLs that stay the same need nothing, and removed content with no relevant replacement is better off answering with a 404 or a 410 than ending up on the homepage. Most redirects, however, are written as rules (“everything that was under /news/ goes under /blog/“), and rules fail silently: a page that lands on the category instead of the matching article, content that no longer exists on the new site and gets sent to the homepage, a parameter that gets lost.
What cannot be done for thousands of URLs is the manual work: checking that each redirect leads to the right page, that the destination page has the same content, the same title, the same structured data, and then following it in the weeks after launch. That work takes time, so we should focus our effort where it pays off. So the right question is not “which pages do we save” but “on which pages would a mistake be costly”. The first approximation of the answer lies in the clicks.
Let us build an example. The data are entirely simulated, but they have the shape that real sites often show: 5,000 URLs receiving impressions in Search Console, and traffic that follows a power law, that is, a few pages with a great many clicks and a long tail of pages with very few. We simulate fourteen months of daily clicks, URL by URL, with a rate that also varies from one month to the next, as happens to real pages.
I build the data in R:
set.seed(2026)
N <- 5000
day <- seq.Date(as.Date("2025-01-01"), as.Date("2026-02-28"), by = "day")
month <- format(day, "%Y-%m")
# average daily clicks per URL: a power law with a slightly flat head
weight <- (1:N + 10)^-1.7
rate <- 50 * weight / sum(weight)
# every URL has better and worse months
months <- unique(month)
mfactor <- matrix(exp(rnorm(N * length(months), 0, 0.3)), N, length(months))
colnames(mfactor) <- months
# daily clicks, one row per URL and one column per day
daily_clicks <- sapply(seq_along(day), function(d) rpois(N, rate * mfactor[, month[d]]))
url <- sprintf("/page-%04d/", 1:N)
rownames(daily_clicks) <- url For the analysis we take the last twelve months, from March 2025 to February 2026. Twelve months, not three, for a precise reason: a page that only works in December, or only in spring, would look dead in a short window.
I add up the clicks of each URL over the window:
win_a <- day >= as.Date("2025-03-01")
clicks <- rowSums(daily_clicks[, win_a])
sum(clicks > 0); sum(clicks) Of the 5,000 URLs, 1,253 received at least one click in twelve months, for a total of 18,858 clicks. The other 3,747 were seen in search results, but nobody clicked on them.
The next step is the Lorenz curve turned upside down: instead of ranking from the poorest page to the richest, we rank from the richest to the poorest, because the question is “how many pages do I need, starting from the top”. For each position k in the ranking we compute the cumulative share, that is, the sum of the clicks of the first k pages divided by the total clicks:
\(S(k) = \frac{x_{(1)} + x_{(2)} + \dots + x_{(k)}}{\sum_{i=1}^{n} x_i}\)
where x₍₁₎ is the page with the most clicks, x₍₂₎ the second, and so on. S(k) starts from the share of the first page and reaches 1 when we have taken all the pages.
The logarithmic scale on the horizontal axis is not an affectation: on a normal scale, the interesting part (the first hundred pages) would take up a thin strip on the left, and the rest of the chart would be a long flat line. On a logarithmic scale each tick multiplies the number of pages by ten, and we can clearly see how much every extra percentage point costs.
From the curve we read the thresholds: the first position at which the cumulative share reaches 50%, 80%, 90%, 95%.
I compute them in R:
thresholds <- function(clicks, p = c(0.5, 0.8, 0.9, 0.95)) {
o <- sort(clicks, decreasing = TRUE)
cum <- cumsum(o) / sum(o)
setNames(sapply(p, function(x) which(cum >= x)[1]), paste0(p * 100, "%"))
}
thresholds(clicks) | share of clicks | URLs needed | of URLs with clicks | of all URLs with impressions |
|---|---|---|---|
| 50% | 18 | 1.4% | 0.4% |
| 80% | 89 | 7.1% | 1.8% |
| 90% | 220 | 17.6% | 4.4% |
| 95% | 458 | 36.6% | 9.2% |
Eighteen pages bring in half of the traffic. For 80%, 89 are enough, that is 7% of the pages receiving clicks: so much for 80/20. Taking 20% of the pages with clicks, in our example, gets us to 91% of traffic. The famous Pareto rule, here, underestimates concentration.
But let us also look at the other side of the thresholds, which is the one that really decides. The last page to enter the 80% list had 30 clicks in a year, a little over two a month. The last one to enter the 90% list had 7. Going from 80 to 90% means more than doubling the work (from 89 to 220 pages) to recover ten points of traffic, spread over pages that are individually worth very little. From 90 to 95% the pages double again.
A reasonable division of labour, then, is this:
The exact threshold is not sacred. 80% is a reasonable starting point, but if the budget allows reaching 90%, so much the better; what matters is that the choice is made by looking at the curve, not by gut feeling.
The curve says it all, but sometimes we need a single number, for instance to compare two sites, or the same site in two different years. That is the job of the Gini index, which is 0 when all pages have the same traffic and approaches 1 when a single page has everything. We compute it with a short function (alternatively there is Gini() from the ineq package):
gini <- function(x) {
x <- sort(x); n <- length(x)
sum((2 * seq_len(n) - n - 1) * x) / (n * sum(x))
}
gini(clicks) # all 5,000 URLs with impressions
gini(clicks[clicks > 0]) # only the 1,253 URLs with at least one click The first is 0.966, the second 0.864. Same site, same clicks, two rather different numbers. The difference lies entirely in the denominator: the 3,747 pages with zero clicks, if we include them, are 3,747 more “poor” pages, and inequality grows.
Which of the two is right? Both, but they answer different questions. The Gini over all URLs with impressions measures how concentrated clicks are among all the pages Google shows, including those that receive none; the one over URLs with clicks only measures how concentrated traffic is among the pages that receive some. (Neither tells us how well the site turns impressions into clicks: impressions do not enter the calculation, and for that we need CTR.) A Gini index without the set it was computed on cannot be compared with anything, and comparing one site’s Gini computed one way with another’s computed the other way leads to meaningless conclusions.
We computed the thresholds on one year of data. But had we taken a slightly different year, would the list be the same? It is a practical question: if the 80% list changed from top to bottom by shifting the window by two months, it would not be a solid basis for deciding where to invest the work.
The first check is the simplest: we redo the calculation on a twelve-month window shifted back by two months, from January to December 2025, and compare the lists.
I compare the two windows in R:
win_b <- day <= as.Date("2025-12-31")
clicks_b <- rowSums(daily_clicks[, win_b])
thresholds(clicks_b)
top_list <- function(clicks, p) names(sort(clicks, decreasing = TRUE))[seq_len(thresholds(clicks, p))]
la <- top_list(clicks, 0.8); lb <- top_list(clicks_b, 0.8)
length(intersect(la, lb))
all(lb %in% top_list(clicks, 0.9)) The thresholds barely move: 18, 90, 226 and 470 URLs, against 18, 89, 220 and 458. Of the 89 URLs in the first 80% list, 85 are also in the second. The four that drop out have between 30 and 32 clicks, that is, they sit right on the edge. And all 90 URLs of the second list are included in the 90% list of the first window: a good argument for using the 90% threshold as a safety margin.
The second check is more systematic: the bootstrap over days. We resample with replacement the 365 days of the window, recompute clicks per URL and the thresholds, and repeat five hundred times. It is a way of asking how different the lists would be if the year had gone a little differently, day by day.
I resample the days in R:
set.seed(1)
idx_a <- which(win_a)
boot <- replicate(500, {
d <- sample(idx_a, replace = TRUE)
cb <- rowSums(daily_clicks[, d])
c(thresholds(cb, 0.8), thresholds(cb, 0.9))
})
apply(boot, 1, quantile, c(0.025, 0.975)) The 95% interval (between the 2.5th and the 97.5th percentile of the repetitions) runs from 84 to 91 URLs for the 80% threshold, and from 202 to 220 for the 90% one. We can then count how many times each URL enters the 80% list across the five hundred repetitions.
I count the appearances in R:
set.seed(1)
appearances <- rowSums(replicate(500, {
d <- sample(idx_a, replace = TRUE)
cb <- rowSums(daily_clicks[, d])
url %in% top_list(cb, 0.8)
}))
sum(appearances >= 475) # present in at least 95% of cases
sum(appearances > 25 & appearances < 475) # on the edge 71 URLs are almost always there, in at least 95% of the repetitions: they are the core, the ones beyond discussion. Another 36 sit on the edge, sometimes in and sometimes out. For the migration the conclusion is simple: edge URLs are treated like those in the list, because putting them in costs little and leaving them out because of a whim of the sample would cost more.
The bootstrap should be taken for what it is: an exploratory analysis, telling us how much random variation matters within the year we observed. By resampling days one by one we treat them as independent and interchangeable, and we lose the links between neighbouring days and seasonality. With real data, where a good week tends to be followed by another good week, it is better to resample blocks of consecutive days (whole weeks, for example: this is the block bootstrap), which preserves at least part of the dependence between neighbouring days. The block length should be chosen by looking at the data, and on its own it does not deal with seasonality and trends.
And there is a limit no resampling can overcome: in a simulation, stability depends on how we built it. Our pages change rate from month to month, but the underlying ranking stays the same; on a real site there may be pages growing fast or declining, and then the lists can differ more when we shift the window. That is exactly why the check must be run on one’s own data before using the list.
A warning: the thresholds are percentages of the clicks we manage to extract, and those are not necessarily all of them. Exports from the Search Console web interface stop at a thousand rows, and the API does not guarantee to return every row either, especially on large sites; moreover, queries anonymised for privacy do not appear when data are requested by query. It is best to download data by page (without the query dimension), through the API or the BigQuery export, and to compare, for the same period and filters, the total of the extracted clicks with the one in the performance chart. It is not a proof of completeness (totals by page and by property are aggregated differently and may not match anyway), but a large difference is a warning sign. High thresholds (95%, 99%) are the most exposed to these gaps; the 80 and 90% thresholds, which depend on the pages with the most traffic, much less so.
Organic clicks are a good starting criterion, but not the only one. Some pages with few clicks need the same protection in a migration as the top of the ranking:
The simplest way to account for them is to build the final list as the union of several lists, each with its own criterion. With real data converting_pages would come from Google Analytics and linked_pages from a backlink analysis tool; here we simulate them with a few URLs picked at random from the tail.
I merge the lists in R:
set.seed(3)
converting_pages <- sample(url[1001:N], 15)
linked_pages <- sample(url[501:N], 25)
final_list <- union(top_list(clicks, 0.9),
union(converting_pages, linked_pages))
length(final_list) The result is 260 URLs: the 220 from the click list, plus 40 pages that would never have made it on traffic alone. Each list has its own threshold, and each page in the final list carries the reason it is there: knowing that a page was included for its links and not for its clicks changes the way it is checked after launch.
Finally, the same list is also the starting point for measuring the outcome of the migration. Comparing the clicks of the pages in the list before and after launch is far more informative than looking at the site total, where minor pages dilute everything. And the comparison must be made carefully: a migration is the textbook case of a structural break, a day after which the site is no longer the same.
How many pages should be checked by hand in a migration?
It depends on how concentrated the traffic is, and it can be measured: rank URLs by organic clicks over the last twelve months and count how many are needed to reach 80 or 90% of clicks. On many sites this means a few dozen or a few hundred, even when there are thousands of URLs. The other URLs that change address and have an equivalent still get a redirect, written as rules and spot-checked; those that do not change stay where they are, and those removed with no relevant replacement answer with a 404 or a 410.
Does the 80/20 rule apply to SEO traffic?
It is a heuristic, not a law: concentration varies from site to site. In this article’s example 7% of the pages with clicks bring 80% of the traffic, and 20% bring 91%, so the rule underestimates concentration; on a site with many pages of similar traffic it might overestimate it. Rather than applying the rule, it is better to compute the curve on one’s own data.
Should the Gini index be computed over all URLs or only those with clicks?
Both are possible, but the results differ (0.966 against 0.864 in our example) and answer different questions. What matters is always stating the set it was computed on, and comparing only indices computed the same way.
How can we check that the list of important pages is reliable?
With two tests: recomputing the list on a time window shifted by a few months, and with a bootstrap over days (better in blocks of weeks), which shows which pages almost always enter the list and which sit on the edge. Edge pages are worth including.
The simulation lends itself to a few experiments:
Hint: the third test shows clearly why the twelve-month window is not pedantry. With three months of data the tail pages almost all have zero or one click, and their place in the ranking becomes a lottery.
We built the list with Search Console clicks, which Google measures on its own side and which the site’s code cannot interrupt. The visits and conversions we measure on the site, on the other hand, the ones needed for the list of converting pages and for judging the outcome of the migration, are far more fragile, and a migration is the moment when tracking breaks most often: a tag that is not copied into the new theme, a misconfigured consent banner, a property that is not updated. Recognising those gaps in the data, and reconstructing with judgement what we did not measure, is the subject of the next article, on tracking gaps.
For the underlying idea, that a highly skewed distribution is told badly by an average and well by a curve, The Art of Statistics by David Spiegelhalter is the clearest read I know: it teaches how to summarise skewed data without being fooled by the mean, and how to say honestly how sure we are of what we claim. To put the analysis into practice in R, from loading Search Console data to charts, R for Data Science by Wickham, Çetinkaya-Rundel and Grolemund is the reference handbook (also freely available online).
The September report is almost ready, and there is one number that does not fit:…
The annual report lands on the desk with good news on the front page: the…
A site's technical guides climb in the search results, show up more often, and yet…
Clicks and sessions vary more than Poisson allows: how to detect it in R and…
An annual report often ends with a single line: clicks from Google fell 11.8% compared…
A site receives 612 requests this year and received 640 the year before. The arithmetic…