What is the Monte Carlo method

The story of the Monte Carlo method begins in the most unlikely way: with a mathematician in bed playing cards. In 1946, Stanisław Ulam, a Polish mathematician recovering from surgery, found himself playing solitaire to pass the time. Being a mathematician, he wondered: what are the chances of winning a game?

The problem was theoretically solvable: just enumerate every possible combination of cards and count the favorable ones. In practice, however, the number of combinations was so enormous that an exact calculation was completely impractical. Ulam then had an insight as simple as it was powerful: instead of computing the exact probability, why not simulate hundreds of games and count how many times you win?

The idea is disarmingly simple. If we play 1,000 games and win 230 of them, we can estimate the probability of winning at about 23%. The more games we simulate, the closer our estimate gets to the true value. This is, in essence, the Monte Carlo method: using random simulation to solve problems that would be too complex to tackle analytically.

Ulam shared the idea with his colleague John von Neumann, arguably the most brilliant mathematician of the 20th century, who immediately saw its potential. Von Neumann realized that ENIAC — one of the very first electronic computers, which filled an entire room — could run thousands of simulations in reasonable time. Together, they developed the method for a problem far more serious than solitaire: the diffusion of neutrons in atomic weapons, as part of the Manhattan Project at Los Alamos.

The name “Monte Carlo” was chosen as a code name, a reference to the famous Monte Carlo Casino in Monaco. Legend has it that the inspiration came from Ulam’s uncle, a notorious gambler. After all, the heart of the method is chance itself: generating random numbers to explore spaces of possibility too vast to traverse systematically.

From those early nuclear experiments of the 1940s, the Monte Carlo method has spread to every field of science and engineering. Today it is one of the most widely used computational tools in the world, from particle physics to finance, from cinematic rendering to drug discovery. Let’s see how it works.

Fundamental concepts

The Monte Carlo method rests on a statistical principle we’ve encountered before: the law of large numbers. In simple terms, this law tells us that the average of a random sample approaches the population average as the sample grows. Translated into Monte Carlo language: the more simulations we run, the more accurate our result will be.

To run a Monte Carlo simulation, we need random numbers. In practice, computers don’t generate truly random numbers: they use deterministic algorithms that produce sequences of pseudo-random numbers with statistical properties indistinguishable from real randomness. In R, for example, the runif() function generates uniformly distributed numbers between 0 and 1.

A crucial aspect is the rate of convergence. The Monte Carlo estimation error decreases as 1/√n, where n is the number of simulations. This means that to halve the error, we need to quadruple our simulations; to gain one more decimal digit of precision, we need 100 times more iterations. It’s not particularly efficient, but the beauty of the method lies in the fact that it works regardless of the problem’s complexity: whether the problem has 2 or 2,000 variables, the convergence rate remains the same.

In practice, we must always balance desired precision with available computational resources. Increasing the number of simulations comes at a cost in computation time. Fortunately, modern computers make this trade-off much more favorable than in the days of ENIAC.

The Monte Carlo method in action

Let’s see concretely how the Monte Carlo method is applied. The procedure follows four fundamental steps:

1. Define the model. First, we identify the problem’s variables and the probability distributions that govern them. For instance, if we want to simulate an investment’s return, the model will include the expected return (mean) and volatility (standard deviation), typically assuming normally distributed returns.

2. Generate random scenarios. Using a pseudo-random number generator, we produce thousands of possible scenarios. Each scenario represents an “alternative history”: one way things could play out.

3. Compute the result for each scenario. For each scenario, we apply the model and obtain a result. If we’re simulating an investment, the result is the final portfolio value.

4. Aggregate the results. Finally, we analyze the set of results: we compute the mean, the median, the percentiles. This gives us not just an estimate of the expected outcome, but an entire distribution of possibilities. And this is where Monte Carlo truly shines: it tells us not only “how much we’re likely to earn” but also “how much we could lose in the worst case.”

Let’s use a quick example to illustrate convergence. Imagine flipping a coin and trying to estimate the probability of heads. After 10 flips, we might get 7 heads (70%), an estimate far from the true 50%. After 100 flips, we’ll be closer, perhaps 53%. After 10,000 flips, our estimate will be very close to 50%. This is Monte Carlo in action: replacing a theoretical calculation with an experiment repeated thousands of times.

The power of the method lies in its flexibility. While analytical methods require closed-form solutions (which often don’t exist for complex problems), Monte Carlo only requires the ability to simulate the process. If we can write a program that generates one scenario, Monte Carlo gives us the distribution of outcomes.

Practical examples: estimating π and portfolio returns

Example 1: estimating the value of π

The most classic and pedagogically effective example of the Monte Carlo method is estimating the number π. The idea is elegant: consider a square of side 2 with a circle of radius 1 inscribed inside it. The area of the square is 4, the area of the circle is π. If we generate random points inside the square, the proportion falling inside the circle will be approximately π/4.

We compute this in R with 100,000 points:

set.seed(123)
n <- 100000
x <- runif(n, -1, 1)
y <- runif(n, -1, 1)
inside <- (x^2 + y^2) <= 1
pi_estimate <- 4 * sum(inside) / n
pi_estimate
# [1] 3.13956

The same in Python:

import random
random.seed(123)
n = 100000
inside = sum(1 for _ in range(n)
             if random.uniform(-1, 1)**2 + random.uniform(-1, 1)**2 <= 1)
pi_estimate = 4 * inside / n
print(pi_estimate)
# 3.14268

With 100,000 points we already get a reasonable estimate, though not extremely precise: we’re accurate to about two decimal places. As we mentioned, gaining another digit of precision would require roughly 100 times more points. The computer does all the heavy lifting.

Example 2: estimating portfolio returns

Let’s move to an example closer to real-world applications. Suppose we have a portfolio of three stocks with the following characteristics:

We want to estimate the probability that the portfolio return exceeds 10%. We simulate in R with 10,000 scenarios:

set.seed(42)
sim_A <- rnorm(10000, mean = 0.08, sd = 0.12)
sim_B <- rnorm(10000, mean = 0.10, sd = 0.15)
sim_C <- rnorm(10000, mean = 0.12, sd = 0.18)
sim_portfolio <- 0.4 * sim_A + 0.3 * sim_B + 0.3 * sim_C
prob_result <- mean(sim_portfolio >= 0.10)
prob_result
# [1] 0.4504

The same in Python:

import random
random.seed(42)
n = 10000
count = 0
for _ in range(n):
    a = random.gauss(0.08, 0.12)
    b = random.gauss(0.10, 0.15)
    c = random.gauss(0.12, 0.18)
    ptf = 0.4 * a + 0.3 * b + 0.3 * c
    if ptf >= 0.10:
        count += 1
print(count / n)
# 0.4479

The result tells us there’s roughly a 45% chance of exceeding 10% return. Notice how Monte Carlo gives us not a single number, but an entire distribution: we could easily compute the median return, the worst-case 5th percentile, the probability of loss, and so on.

Monte Carlo Simulator

To make the concept even more tangible, we’ve built an interactive simulator that applies the Monte Carlo method to predict the future value of an investment. The underlying model is the Geometric Brownian Motion (GBM), the same model used in the famous Black-Scholes framework for options pricing.

Intuitively, an asset’s future price is computed as the current price multiplied by a random growth factor. The formula is:

S(t+1) = S(t) × exp((μ − σ²/2) + σ × Z)

where μ is the expected annual return (the “average growth”), σ is the volatility (how much the price fluctuates — our measure of uncertainty), and Z is a random number drawn from a normal distribution. Each simulation generates a different path: some scenarios see the portfolio grow substantially, others see it decline. The histogram shows the distribution of all possible outcomes.

Modern applications of the Monte Carlo method

From the nuclear physics of the 1940s, the Monte Carlo method has spread to domains that Ulam and von Neumann could never have imagined. Let’s look at some of the most fascinating applications.

3D rendering and cinema. Every time we watch a Pixar film or a blockbuster with visual effects, we’re admiring Monte Carlo at work. The technique is called path tracing: to compute the color of each pixel, the software simulates millions of light rays bouncing between surfaces in the scene. Each ray follows a random path, and the average of thousands of paths produces the photorealistic image we see on screen.

Finance and risk management. In the financial world, Monte Carlo is ubiquitous. Banks use it to calculate Value at Risk (VaR) — the maximum probable loss of a portfolio over a given time horizon. It’s the same principle as our simulator, applied to portfolios with hundreds of assets and complex correlations. Pricing exotic options that lack closed-form solutions also relies on Monte Carlo simulations.

Drug discovery. In pharmaceutical research, Monte Carlo is used to simulate molecular docking: how a candidate molecule binds to a target protein. By simulating millions of possible spatial configurations, researchers identify the most promising compounds before synthesizing them in the lab, saving years of experimentation.

Climate models. Models predicting climate change are inherently uncertain: they depend on emission scenarios, atmospheric feedback, ocean dynamics. Monte Carlo allows exploration of thousands of parameter combinations and generates the uncertainty bands we see in IPCC reports. Not a single prediction, but a distribution of possible futures.

Artificial intelligence. In machine learning, a technique called Monte Carlo dropout uses simulation to estimate the uncertainty of a neural network’s predictions. And the famous AlphaGo by DeepMind, which in 2016 defeated the world Go champion, used Monte Carlo Tree Search (MCTS) to explore possible moves in a game with more configurations than atoms in the universe.

Advantages and limitations of the Monte Carlo method

Like any statistical tool, the Monte Carlo method has its strengths and limitations. Let’s examine them honestly.

Flexibility. The greatest advantage is versatility: Monte Carlo applies to complex problems of any size and in any field, from finance to engineering, physics to biology. It doesn’t require closed-form solutions, only the ability to simulate the process.

Accuracy. With a sufficient number of simulations, the estimate can be made arbitrarily precise. The more we run the method, the closer the result converges to the true value.

Scalability. Unlike grid-based methods, which suffer from the “curse of dimensionality” (cost explodes with the number of variables), Monte Carlo maintains the same convergence rate regardless of the number of dimensions. This makes it the only practical tool for high-dimensional problems.

However, the method also presents significant limitations:

Slow convergence. The 1/√n rate means that gaining one digit of precision requires 100 times more simulations. For problems demanding very high precision, this can be prohibitive.

Computational cost. For complex problems (many variables, heavy models), each individual simulation may require significant time. Multiplied by thousands or millions of iterations, the cost becomes considerable.

To mitigate these limitations, variance reduction techniques have been developed over the years, enabling more precise results with fewer simulations:

• Importance sampling: sampling from an alternative distribution that “concentrates” simulations in the most informative regions.
• Control variates: using a correlated variable with known expected value to reduce the estimate’s variance.
• Stratified sampling: dividing the space into homogeneous subgroups and sampling from each.
• Antithetic variates: exploiting pairs of negatively correlated random numbers to reduce variance.

The Monte Carlo method represents one of the most powerful tools in computational statistics. In future articles, we’ll explore how some of these techniques — particularly the bootstrap, a close relative of Monte Carlo — apply to concrete problems in statistical inference.

Further reading

For a deeper dive into the Monte Carlo method and its applications in finance, Monte Carlo Methods in Financial Engineering by Paul Glasserman is the most comprehensive reference: it covers theory and practice with detailed examples in derivative pricing and risk management.

Calculating the sample size in advance is essential to avoid two classic mistakes: stopping the test too early and declaring a winner that does not exist, or letting it run too long, wasting traffic and time. In other words, it is about finding the right balance between resources and rigour.

If you have read the article on A/B Testing, you will recall that power analysis is the statistical method that lets us determine this threshold. And if you have studied confidence intervals, you already know that significance level and test power are not abstract concepts but operational levers that directly affect sample size.

The calculator below automates this process: simply enter your test parameters to instantly get the number of observations needed per variant and, if you know your daily traffic, an estimate of the test duration in days.

The calculator

Enter the parameters of your A/B test and the calculator will instantly return the required sample size.

The formula: how the calculation works

The calculator uses the standard formula for comparing two proportions with a two-tailed z-test. Let us walk through it step by step.

We start with the parameters we enter:

• p₁: the baseline conversion rate (control), expressed as a proportion. If our CR is 5%, then p₁ = 0.05.
• p₂: the expected conversion rate for the variant. If the minimum detectable effect (MDE) is 20% relative, then p₂ = p₁ × (1 + MDE/100) = 0.05 × 1.20 = 0.06.
• α: the significance level, i.e. the probability of declaring an effect when there is none (Type I error). With α = 0.05 we work at 95% confidence.
• 1 − β: the power of the test, i.e. the probability of detecting an effect when it actually exists. With power 0.80, we have an 80% chance of catching the effect.

The formula is:

\( n = \frac{\left[z_{\alpha/2} + z_{\beta}\right]^2 \cdot \left[p_1(1-p_1) + p_2(1-p_2)\right]}{(p_1 – p_2)^2} \)

Where z_α/2 and z_β are the quantiles of the standard normal distribution. For the most common values:

• α = 0.05 → z_α/2 = 1.96
• α = 0.01 → z_α/2 = 2.576
• β = 0.20 (power 0.80) → z_β = 0.842
• β = 0.10 (power 0.90) → z_β = 1.282

Worked example. Suppose we have a baseline conversion rate of 3% and we want to detect a 20% relative increase (i.e. going from 3% to 3.6%), with α = 0.05 and power = 0.80:

• p₁ = 0.03, p₂ = 0.036
• z_α/2 = 1.96, z_β = 0.842
• Numerator: (1.96 + 0.842)² × [0.03 × 0.97 + 0.036 × 0.964] = 7.849 × 0.0638 = 0.5008
• Denominator: (0.03 − 0.036)² = 0.000036
• n = 0.5008 / 0.000036 ≈ 13,911 per variant

So to detect a 20% relative effect on a 3% CR, we need roughly 13,900 observations per variant (nearly 28,000 in total). These numbers are worth reflecting on: if our site gets 500 visitors a day, the test will take about 56 days. This is one of the reasons why, in practice, most A/B tests on medium-traffic sites take weeks, not days.

How to use the calculator

How to choose the MDE. The minimum detectable effect is the trickiest parameter. Rather than asking “how much would we like the metric to improve”, we should ask: what is the smallest improvement that would justify the effort of implementing the change? An MDE of 5% relative requires enormous samples; an MDE of 50% is easy to detect but rarely realistic. The 10–30% range is a good starting point for most conversion rate tests.

An important detail: the MDE in the calculator is relative, not absolute. An MDE of 20% on a baseline CR of 5% means we are looking to detect a shift from 5% to 6% (one absolute percentage point, but 20% of the starting value).

How to estimate daily traffic. The traffic to enter is that of the pages involved in the test, not the total site traffic. If the test is on the checkout page and it receives 300 visits per day, the correct value is 300. You can get this figure from your analytics tool (GA4, Matomo, or similar) by averaging the last 30 days to smooth out daily fluctuations.

You might also like

• A/B Testing: A Guide to Statistical Tests for A/B Analysis
• Confidence Intervals
• Hypothesis Testing

Further reading

The most comprehensive reference on the rigorous design of online experiments is: Trustworthy Online Controlled Experiments by Ron Kohavi, Diane Tang and Ya Xu. It covers sample size, power analysis and much more, drawing on decades of practical experience at Microsoft and Google.

Machine Learning is changing the way we see the world around us. From weather prediction to medical diagnosis, from content recommendations on streaming platforms to financial fraud detection, Machine Learning is increasingly present in our daily lives.

But what exactly is it, and how does it work? In this post, we will explore the fundamental concepts of Machine Learning and see how it can be used to solve real-world problems. We will also look at how to get started with Machine Learning, what resources are available, and how to use this technology to improve both work and everyday life.

Caveat: This article is a simple introduction to a vast subject. It is written for anyone who wants to understand the basic concepts of Machine Learning, without requiring advanced technical or mathematical knowledge. At the end of the post, we provide a set of useful resources for anyone who wishes to explore the topic further and continue what is a truly fascinating journey.

What Is Machine Learning

Machine Learning, or automated learning, is a technology that allows machines to “learn” from data and improve their performance without being explicitly programmed. In other words, Machine Learning enables machines to “learn” from experience, just as humans do.

There are two main types of Machine Learning: supervised Machine Learning and unsupervised Machine Learning.

In supervised Machine Learning, the model is “trained” on a dataset that includes examples of both inputs and desired outputs. The model then uses these examples to make predictions on new data. In unsupervised Machine Learning, the model must “discover” on its own the structures and relationships within the data, without being guided by pre-defined examples.

Machine Learning is used across a wide range of applications, from weather prediction to medical diagnosis, from content recommendations to financial fraud detection. In general, the goal of Machine Learning is to automate decisions and predictions based on data, improving the efficiency and accuracy of the process.

Types of Machine Learning: Supervised and Unsupervised

As we have already seen, Machine Learning can be divided into two main categories: supervised Machine Learning and unsupervised Machine Learning.

Supervised Machine Learning is the most common type of automated learning and is based on a set of already labeled data. In other words, the model is “trained” on a dataset that includes examples of both inputs and desired outputs. The model then uses these examples to learn to make inferences on new data. For example, a spam classifier could be trained on a set of emails labeled as “spam” or “not spam,” and then used to classify new incoming emails.

Unsupervised Machine Learning, on the other hand, is based on a set of unlabeled data. In other words, the model must “learn” on its own to discover structures and relationships within the data. A typical example of this type of learning is clustering, where data is divided into groups (clusters) based on their similarities.

In general, we can say that supervised Machine Learning uses labeled data to make predictions or classifications, while unsupervised Machine Learning uses unlabeled data to make discoveries or identify relationships within the data.

Main Supervised Learning Algorithms

The main supervised Machine Learning algorithms are:

• Linear Regression: used for quantitative predictions on a continuous variable. For example, predicting the price of a house based on its square footage.

  We have written dedicated posts on this topic, which may be very helpful for a proper understanding:
  Correlation and Regression Analysis
  Multiple Regression Analysis

• Logistic Regression: used for categorical variable predictions, i.e., when the output is one class among two or more possibilities. For example, predicting whether a patient has a certain disease or not.
• Decision Trees: used for both classification and regression. They consist of a decision graph where each node represents a decision and each branch represents an outcome.
• Random Forest: a variant of decision trees where multiple trees are used to make predictions, and then the average of the trees’ predictions is used.
• Gradient Boosting: an algorithm that uses a series of decision trees in succession to improve predictions.
• Support Vector Machine (SVM): used for classification when the data is linearly separable.
• k-Nearest Neighbors (k-NN): used for classification based on the similarity of data points relative to a reference point.
• Naive Bayes: used for probability-based classification.

Main Unsupervised Learning Algorithms

• Clustering: used to divide data into groups or clusters based on their similarities. The most common clustering algorithm is k-means.
• Principal Component Analysis (PCA): used to reduce the dimensionality of data, that is, to transform a set of correlated variables into a set of uncorrelated variables.
• Density-Based Spatial Clustering (DBSCAN): used to find clusters based on the density of data points.
• Association Rule Mining (Apriori, FP-Growth): used to find association rules between variables.
• Anomaly Detection Algorithms (One-class SVM, Isolation Forest): used to detect elements that deviate from the norm.
• Self-Organizing Maps (SOM): used to visualize hidden structures in the data.
• Structure Detection Algorithms (Spectral Clustering, Hierarchical Clustering): used to find hierarchical relationships in the data.

These are some of the main unsupervised Machine Learning algorithms, but there are many others. As with supervised Machine Learning, the choice of algorithm depends on the characteristics of the specific problem and the nature of the data.

In practice, choosing the right algorithm for a specific solution is a critical decision that can determine the success or complete failure of a data analysis effort.

The Main Phases of the Machine Learning Process

1. Data Collection: The first phase consists of gathering the data needed for the problem to be solved. This data must be cleaned, formatted, and prepared for processing.
2. Data Analysis: Once the data has been collected, it is important to explore it in order to better understand the problem and identify any interesting relationships or characteristics.
3. Model Selection: The next phase consists of choosing the most appropriate Machine Learning model for the problem at hand. There are many available algorithms, including decision trees, neural networks, and support vector machines (SVM).
4. Model Training: Once the model has been selected, it must be “trained” using the training data. This process allows the model to “learn” from the data and become capable of making predictions on new data.
5. Model Evaluation: Once trained, the model must be evaluated on a test dataset to verify its accuracy.
6. Model Deployment: If the model has shown good performance, it can be used to solve the problem in question and deployed to a production environment.
7. Monitoring and Maintenance: The model must be monitored to ensure it continues to function correctly and, if necessary, updated or replaced if performance declines.

Getting Started with Machine Learning: Tutorials and Resources

Machine Learning is a rapidly evolving field, and there are many resources available for those who want to get started. Any list is necessarily incomplete and subject to personal preferences, but here are some good starting points:

Tutorials: There are numerous tutorials available online that cover the basics of Machine Learning. For example, the scikit-learn data science website has a tutorial section that explains how to use the library to build some of the most common models.
https://scikit-learn.org/stable/tutorial/index.html

Books: There are many books on the subject, but some of the classics in the field include:
“Introduction to Machine Learning” by Alpaydin: https://www.amazon.com/Introduction-Machine-Learning-Adaptive-Computation/dp/0262028182
“Python Machine Learning” by Raschka and Mirjalili: https://www.packtpub.com/data/python-machine-learning-third-edition

Online Courses: There are many online courses that cover the basics of Machine Learning, such as the excellent course by Andrew Ng on Coursera:
https://www.coursera.org/learn/machine-learning
or the Machine Learning course by fast.ai:
https://www.fast.ai/

Tools: There are many tools and libraries that can be used to explore data and build models. Some of the most popular include:

scikit-learn: a Machine Learning library for Python
https://scikit-learn.org/stable/
TensorFlow: a Machine Learning library developed by Google
https://www.tensorflow.org/
Keras: a high-level interface for building neural networks in TensorFlow
https://keras.io/
PyTorch: an open-source Machine Learning library developed by Facebook
https://pytorch.org/

In general, we recommend starting with tutorials and online courses to become familiar with the basic concepts, and then continuing with books and tools to deepen understanding and develop practical skills. To become a good data scientist, it is also important to work with real data and not just tutorials or exercises. Seeking out Machine Learning projects or competitions can help build concrete experience.

Experimenting with Code: Jupyter Lab and Google Colab

Jupyter Lab and Google Colab are both free and powerful tools for data exploration, learning, and testing Machine Learning code.

How can we use both tools to create development environments and share our work with others?

Jupyter Lab is the new interface for Jupyter Notebook that provides an integrated development environment for working with notebooks. It is an interactive development environment that allows us to write, run, and document Python and R code within a web browser. It is particularly useful for data analysis and for learning Machine Learning.

To get started, Jupyter Lab needs to be installed on a local machine. This can be done easily using Anaconda, a Python distribution that includes Jupyter Lab and many other data science libraries. Once installed, Jupyter Lab can be launched from the command line and a new notebook opened to write and run code. Jupyter Lab is available at: https://jupyter.org/

It is also possible to test the environment directly in the browser with JupyterLite:

Google Colab is a cloud-based development environment that allows us to write and run Python and R code within a web browser without any installation. It is a very convenient option, because Colab can be accessed from any device with an internet connection and work can be shared with others simply by providing a link. It also allows the use of a GPU or TPU to make computations more powerful. Google Colab is available at: https://colab.research.google.com/

Both tools allow us to create a sequence of cells containing code and text. Code can be executed within cells and the results displayed directly in the notebook. This makes Jupyter Lab and Google Colab ideal for data exploration, Machine Learning learning, sharing, and documenting work.

You might also like

• Correlation and Regression Analysis
• Logistic Regression
• How to Use Decision Trees

Further Reading

For a solid introduction to the statistical foundations of machine learning—including regression, model selection, and prediction—Introduzione all’econometria by Stock and Watson provides the quantitative framework that underpins many ML techniques. For a hands-on guide to online experimentation and A/B testing—essential skills for deploying ML models in production—Trustworthy Online Controlled Experiments by Kohavi, Tang and Xu is the definitive reference.

These few words alone are enough to grasp the extraordinary importance of this index for economic and political studies, and why it is worth getting to know it a little more closely.

The index is one of the greatest achievements of Corrado Gini, one of the foremost Italian statisticians (who was unfortunately personally connected to the fascist regime. He inspired Mussolini’s famous “Ascension Day” speech of 1927 on the issues of birth rates and eugenics).

It was in 1912 that Gini published his article “Variabilità e mutabilità” (Variability and Mutability), in which he expanded on the work of Max Otto Lorenz, who as early as 1905 had introduced the famous curves (now known as “Lorenz curves”) describing the percentages of wealth held by increasing percentages of the population.

The Lorenz Curve

Lorenz introduced a highly effective graphical representation, placing on the horizontal axis the points P_i (that is, the cumulative fraction of the first i income earners: P_i = i / n) and on the vertical axis the corresponding values Q_i (the cumulative fraction of income held by the first i income earners). Connecting these points produces the concentration curve, known as the Lorenz curve.

The difference between P_i and Q_i measures, in proportion, the share of total income that the first i individuals lack in order to reach a state of equal distribution.
The larger this difference, the more the remaining n − i individuals concentrate a significant portion of the total amount on themselves.

The measure of income inequality is the arithmetic mean of the normalised differences (that is, of the quantities P_i − Q_i / P_i, for i = 1, 2, 3, …, n − 1).

Gini thus managed to develop, in his 1912 work and then in 1914, “his” coefficient, which measures the percentage of the area between the given curve and the 45-degree line, relative to the area between the latter and the flat curve.

In practice, it indicates how much the corresponding Lorenz curve deviates from complete equality in the distribution of wealth.

In one sentence: the ratio of the area of concentration to its maximum (which is 0.5) coincides exactly with R.

An Example

Let us build the Lorenz curve: the vertical axis shows the income percentages of households, while the horizontal axis shows the percentages of households.
If 30% of households earned 30% of the income, 40% of households earned 40% of the income, and so on, we would have a perfectly equal distribution — that is, a straight line at 45 degrees.

The Lorenz curve instead represents the actual distribution of income: the deviation of the Lorenz curve from the line of perfect equality (that is, from the 45-degree line) constitutes the measure of inequality in income distribution.

The ratio of the area between the line of perfect equality and the Lorenz curve (that is, the shaded area in the figure) to the area of triangle 0AB is the Gini coefficient.

The Definition of the Concentration Index R

R can be defined independently of the Lorenz curve: it equals the normalised simple mean difference divided by its maximum, that is:

\(
R = \frac{\text{mean absolute difference}}{2 \times \text{mean of values}} \\
\)

R is therefore an index expressed as a number between the theoretical values 0 and 1 — theoretical because they correspond, respectively, to the case of perfect equity in wealth distribution (everyone has the same income) and the case of maximum inequality (a single unit holds all the income). It is a “pure” value that allows comparison between different countries or territorial areas, proving extraordinarily useful in the field of socio-economic analysis.

Computing R… in R!

Countless R libraries contain a function for calculating the Gini index (the most widely used package is probably “ineq“, easily found with a search on CRAN), which is not included among R’s base functions.

However, since the calculation itself is not particularly complex, we find it useful to present a version of the function below.

1 – We start by computing the mean absolute difference

Delta <- function(variable) {
  n <- length(variable)
  avg <- mean(variable)
  sorted_variable <- sort(variable)
  (4 * sum((1:n) * sorted_variable) / n - 2 * avg * (n + 1)) / (n - 1)
}

2 – Now obtaining the Gini concentration ratio is just one line!

gini <- Delta(variable) / (2 * mean(variable))

What If I Don’t Use R?

Fair point. R is a fantastic tool, but not everyone uses it. An index as important as Gini can be useful to many people who do not deal with statistics every day and are not familiar with R. The most universal and widespread programming language, even among non-programmers, is Python. Naturally, as with R, there are many possible implementations of the Gini coefficient, but in this case too, doing it ourselves is simple and instructive.

The solution we liked best comes from a post on planspace.org — here is the function, 8 lines in all:

def gini(list_of_values):
    sorted_list = sorted(list_of_values)
    height, area = 0, 0
    for value in sorted_list:
        height += value
        area += height - value / 2.
    fair_area = height * len(list_of_values) / 2.
    return (fair_area - area) / fair_area

First, the function sorts the list of values in ascending order. Then, it uses a for loop to compute the height and area of the Lorenz curve.

The height is calculated as the cumulative sum of the values in the list, while the area is computed as the area of the trapezoid between the current value and the previous value in the list. The total area of the Lorenz curve is then calculated as half the total height of the curve multiplied by the length of the list.

Finally, the Gini index is computed as the difference between the “fair area” (half the total area of the Lorenz curve if there were no inequality) and the actual area of the Lorenz curve, divided by the fair area.

Gini Index Values Around the World

• For a general overview, we can visit the website of the Organisation for Economic Co-operation and Development (OECD).
• A comparison of values across European countries is provided by Eurostat.
• On the ISTAT website it is possible to compare Gini index data across the various Italian regions.

You might also like

• Descriptive Statistics: Measures of Position
• Descriptive Statistics: Measures of Variability
• The Data: The 4 Scales of Measurement

Further Reading

For a brilliant, accessible exploration of statistical thinking—including how inequality measures like the Gini coefficient help us understand the world—The Art of Statistics by David Spiegelhalter offers a masterful blend of rigour and clarity.

Searching for relationships between two categorical variables is a very common goal for researchers. Think, for example, of the classic question that marketers ask: who is more likely to buy certain product categories, young or old people, men or women…

Two-Way Tables and Marginal Distributions

A two-way table is a table with rows and columns that helps organize data from categorical variables:

• Rows represent the possible categories for one qualitative variable, for example males and females.
• Columns represent the possible categories for a second qualitative variable, for example whether someone likes pizza or not…

A marginal distribution shows how many total responses there are for each category of the variable. The marginal distribution of a variable can be determined by looking at the “Total” column (or row).

Let’s look at an example.

Note: I couldn’t think of anything particularly clever, so I created a table (with fictitious data, of course) of rare silliness, imagining that the two categorical variables concern education level and favorite sci-fi series…

We build the table in R:

scifi_fans <- matrix(c(44, 38, 26, 53, 35, 30, 58, 22, 29), ncol = 3, byrow = TRUE)
rownames(scifi_fans) <- c("degree", "diploma", "lower education")
colnames(scifi_fans) <- c("star trek", "star wars", "doctor who")
scifi_fans <- as.table(scifi_fans)
scifi_fans

and we get something like this:

                 star trek   star wars   doctor who
degree               44          38          26
diploma              53          35          30
lower education      58          22          29

Remember? A marginal distribution shows how many total responses there are for each category of the variable (at the margins, precisely, where the Total column or row is...).

We can compute row totals in R with:

margin.table(scifi_fans, 1)

and column totals with:

margin.table(scifi_fans, 2)

We can also find the "grand total" with:

margin.table(scifi_fans)

Here is the table with totals:

              star trek   star wars   doctor who   TOTAL
degree            44          38          26        108
diploma           53          35          30        118
lower ed.         58          22          29        109
TOTAL            155          95          85        335

So the marginal totals by education level are 108 for degree holders, 118 for diploma holders, 109 for lower education.

Likewise, the marginal totals by sci-fi series type are 155 for Star Trek, 95 for Star Wars, 85 for Doctor Who.

The grand total must be the same in both directions, in this case 335.

We could also have displayed a complete table with totals using just a few lines of R code:

scifi_fans <- matrix(c(44, 38, 26, 53, 35, 30, 58, 22, 29), ncol = 3, byrow = TRUE)

row_names <- c("degree", "diploma", "lower education")
col_names <- c("star trek", "star wars", "doctor who")
dimnames(scifi_fans) <- list(row_names, col_names)

# Compute column totals using apply
col_totals <- apply(scifi_fans, 2, sum)
# Add row with column totals using rbind
scifi_fans2 <- rbind(scifi_fans, col_totals)
# Compute row totals
row_totals <- apply(scifi_fans2, 1, sum)
# Add column with row totals
cont_table <- cbind(scifi_fans2, row_totals)

# Print the table
cont_table

We can then ask ourselves (and answer): what percentage of degree holders has a soft spot for Doctor Who?
Elementary, Watson (oh wait, that was a different series...):

26/108 = 0.24 = 24% of degree holders prefer Doctor Who

And how many Star Wars fans hold a diploma?

35/95 = 0.37 = 37% of Star Wars fans are diploma holders

In R, we can directly obtain row proportions with the function:

prop.table(scifi_fans, 1)

and the result will be:

                 star trek    star wars    doctor who
degree           0.4074074    0.3518519    0.2407407
diploma          0.4491525    0.2966102    0.2542373
lower ed.        0.5321101    0.2018349    0.2660550

(as we can see, the row totals add up to 1, or 100%)

or column proportions with:

prop.table(scifi_fans, 2)

and the result will be:

                 star trek    star wars    doctor who
degree           0.2838710    0.4000000    0.3058824
diploma          0.3419355    0.3684211    0.3529412
lower ed.        0.3741935    0.2315789    0.3411765

(as we can see, the column totals add up to 1, or 100%)

As always, there is more than one way to get the result. We can also install the "gmodels" package and use the CrossTable function (we'll leave it to R's built-in help to show all the command options...):

install.packages("gmodels")
library(gmodels)
scifi_fans <- matrix(c(44, 38, 26, 53, 35, 30, 58, 22, 29), ncol = 3, byrow = TRUE)
rownames(scifi_fans) <- c("degree", "diploma", "lower education")
colnames(scifi_fans) <- c("star trek", "star wars", "doctor who")

CrossTable(scifi_fans, prop.r = "false", prop.c = "false", prop.t = "false", prop.chisq = "false")

So what is all this good for? The answer is: for example, to compute conditional probability.

Conditional Probability

Before we see what it is and why it is an extremely useful concept in everyday life, we need a few preliminary definitions about probability.

An event is something that occurs with one or more possible outcomes.
An experiment is the process of measuring or making an observation.

Key definition: the probability of an event is the ratio of the number of favorable cases to the number of possible cases

\( P(A) = \frac {\text{number of favorable cases}}{\text{number of possible cases}}\\ \)

Let us also recall that:

• The probability that two events both occur can never be greater than the probability that each event occurs separately.
• If two possible events, A and B, are independent, then the probability that both occur is the product of their individual probabilities.
• If an event can have a certain number of different and distinct possible outcomes (A, B, C, etc.), then the probability that A or B occurs equals the sum of the individual probabilities of A and B, and the sum of the probabilities of all possible outcomes (A, B, C, etc.) equals 1, i.e. 100%.

The conditional probability of an event A with respect to an event B is the probability that A occurs, given that B has occurred.

The formula is:

\( P(A|B) = \frac {P(A \text{ and } B)}{P(B)}\\ \)

If a probability is based on one variable it is a marginal probability; if on two or more variables it is called a joint probability.

• The probability of an event P(A) is: \( \frac {\text{marginal probability of A}}{\text{Total}}\\ \)
• The joint probability of two events is: \( \frac {P(A \text{ and } B)}{\text{Total}}\\ \)
• The conditional probability of outcome A given the occurrence of condition B is: \( \frac {P(A \text{ and } B)}{P(B)}\\ \)

In other words:

A joint probability is the probability that someone selected from the entire group has two particular characteristics at the same time. That is, both characteristics occur jointly. We find a joint probability by taking the value of the cell at the intersection of A and B and dividing by the grand total.

To find a conditional probability, we take the value of the cell at the intersection of A and B and divide it by the marginal total of B, i.e. the variable expressing the event that has occurred.

It's time for a second example. We take the data from:
Ellis GJ and Stone LH. 1979. Marijuana Use in College: An Evaluation of a Modeling Explanation. Youth and Society 10:323-334.

The study asks whether a college student is more likely to smoke marijuana if their parents had used drugs in the past. Here is the table:

                    parents    parents     Total
                      use      no use
student uses          125        94         219
student does not use   85       141         226
Total                 210       235         445

Let's apply our knowledge to answer these questions:

1. If the parents used soft drugs in the past, what is the probability that their child does the same in college?

This is a case of conditional probability.
We recall \( P(A|B) = \frac {P(A \text{ and } B)}{P(B)}\\ \), therefore

P(student uses given that parents used) = 125 / 210 = 0.59 = 59%

2. A student is selected at random and does not use marijuana. What is the probability that their parents used it?

Here again we face a question that asks for a conditional probability. Therefore:

P(parents used given that student does not use) = 85 / 226 = 0.376 = 37.6%

3. What is the probability of selecting a student who does not use marijuana and whose parents used it in the past?

In this case we need to find a joint probability, so:

\( \frac {P(A \text{ and } B)}{\text{Total}}\\ \), therefore \( \frac {85}{445} = 0.19\\ \).

The probability is approximately 19%.

Dependence and Independence

If the outcomes of A and B influence each other, we say that the two variables are in a relationship of dependence.
Conversely, we say the two variables are independent.

More rigorously: we can state that event B is independent of event A if:

P(B|A) = P(B)

or

P(A|B) = P(A)

If this is not the case, the events are dependent on each other.

Therefore:

• P(A and B) = P(A) P(B) if and only if A and B are independent events.
• P(A | B) = P(A) and P(B | A) = P(B) if and only if A and B are independent events.

Let's examine the independence of categorical variables...

Let's explain this better with an example.

Let A be the event that people enjoy cycling.
B expresses whether they enjoy roast lamb. (Makes perfect sense, right?)

We build our contingency table:

                  Likes cycling   Doesn't like cycling   Total
Likes roast lamb       95                36               131
No roast lamb          15                19                34
---------------------------------------------------------------
Total                 110                55               165

Let's remember what it means for two events to be independent. It means this:
P(A | B) = P(A)

But in our case we see that
P(A) = 66.7%
because 110/165 = 0.67

P(A | B) = 72.5%
because 95/131 = 0.725

We recall that \( P(A|B) = \frac {P(A \text{ and } B)}{P(B)}\\ \), therefore \( \frac {95}{131} = 0.725\\ \).

From the result it is clear that \( P(A) \neq P(A|B) \) -- the two events are NOT independent (therefore they are dependent).

After all, everyone knows that there is a clear dependence between loving cycling and loving roast lamb!

You might also like

• First Steps into the World of Probability
• The Chi-Square Test
• Bayesian Statistics

Further Reading

For a comprehensive treatment of contingency tables, conditional probability, and the full machinery of categorical data analysis, Statistica by Newbold, Carlson and Thorne provides a rigorous yet accessible framework for applying these concepts in real-world settings.

The Poisson distribution is useful for measuring how many events can occur within a given time horizon, such as the number of customers entering a shop in the next hour, or the number of pageviews on a website in the next minute, and so on.

Lambda: The Average Rate of Events

An important element: each time interval is assumed to be independent of all others.

We need to know the average number of events or the rate at which they occur within the time interval. We represent this value with the Greek letter lambda:

\( X \sim Po(\lambda) \\ \\ \)

To calculate the probability that there are r occurrences in a specific interval:

\( P (X=r) = \frac{e^{-\lambda} \lambda^{r}}{r!} \\ \\ \)

For example, if:

\( X \sim Po(2) \\ \\ r=3 \)

we get:

\( P (X=3) = \frac{e^{-2} \cdot 2^{3}}{3!} =\frac{e^{-2} \cdot 8}{6} = e^{2} \cdot 1.333 = 0.180 \\ \\ \)

That is, 18%.

Poisson and Binomial: A Side Note

If

\( X \sim Po(\lambda x) \\ Y \sim Po(\lambda y) \\ \\ \)

then

\( X + Y \sim Po(\lambda x + \lambda y) \\ \\ \)

If

\( X \sim Bin(n,p) \\ \\ \)

and n is large and p is small, then we can approximate the binomial with the Poisson:

\( X \sim Po(n \cdot p) \\ \\ \)

Differences Between the Poisson and Binomial Distributions

The Poisson and binomial distributions are both discrete probability distributions used to model rare events. The main difference between the two concerns the number of trials and successes.

The binomial distribution is biparametric, meaning it is characterised by two parameters n and p, where n represents the number of trials and p the probability of success in each trial.

In contrast, the Poisson distribution is uniparametric, meaning it is characterised by a single parameter λ representing the average number of events per interval.

Furthermore, the binomial distribution is used when the number of trials is finite and the number of successes cannot exceed n, whereas the Poisson distribution is used when the number of trials is essentially infinite.

A Practical Example

A vending machine malfunctions on average 3.4 times per week. What is the probability that the machine will not break down next week?

\( P (X=0) = \frac{e^{- \lambda} \cdot \lambda ^{r}}{r!} \\ \\ = \frac{e^{-3.4} \cdot 3.4 ^{0}}{0!} = \\ \frac{e^{-3.4} \cdot 1}{1} = 0.033 \\ \)

We notice that the probability is very low indeed — just 3.3%.

Note: X=0 because we are looking at the probability that the machine does not break down.

In R we would use the command:

dpois(0, 3.4)

Now let us calculate the probability that the vending machine breaks down exactly 3 times during the next week.

\( P (X=3) = \frac{e^{-3.4} \cdot 3.4 ^{3}}{3!} = \frac{e^{-3.4} \cdot 39.304}{6} = 0.216 \\ \)

The probability is 21.6%.

Moving on to a third question: what are the expected value and the variance of the vending machine malfunctions?

\( E(X) = \lambda = 3.4 \\ Var(X) = \lambda = 3.4 \\ \)

As we can see, within the Poisson distribution lambda represents not only the mean but also the variance. This is known as the mean-variance equality property of the Poisson distribution.

Therefore, if lambda is large, the Poisson distribution will be more concentrated around its mean and its variance will also be large; if lambda is small, the distribution will be less concentrated around its mean and its variance will also be small.

The Poisson Distribution Applied to SEO

There are several aspects that make the Poisson distribution potentially interesting for website traffic analysis. It is a simple and well-understood statistical model that can be readily applied to website traffic data — for example, to estimate the average rate of requests or visits per unit of time and to predict the probability of observing a certain number of requests or visits in the future.

However, we should keep in mind that there are also many limitations to using the Poisson distribution for SEO-oriented web traffic analysis.

First, the Poisson distribution assumes that events occur independently and at a constant rate, which may not always hold for website traffic. For example, website traffic might exhibit peak frequencies or internal symmetry that the Poisson distribution cannot capture.

Second, the Poisson distribution is a memoryless process, meaning it does not account for any history of past events. This can be a limitation when analysing website traffic data that display trends or seasonality.

Third, the Poisson distribution assumes that events are discrete and countable, which may not always be appropriate for modelling continuous variables such as response time or page load time. Finally, the Poisson distribution is a simple model that may not capture all the complexities of real-world website traffic.

There are several alternative models for website traffic analysis that can be used when the Poisson distribution is not appropriate.

Alternative Models for Web Traffic Analysis

One alternative is the Negative Binomial distribution, which can handle overdispersion and capture peak frequencies or internal symmetry in website traffic data.

Another alternative is the Lognormal distribution, which can be used to model continuous variables such as response time or page load time.

The Exponential distribution can also be used to model the time intervals between requests or visits to a website.

Using Poisson for Website Click Estimates

Suppose we have a website that receives on average 10 clicks per hour and we want to estimate the probability of getting a certain number of clicks in one hour using the Poisson distribution. We can use R to carry out the following steps:

1. We start by loading the ggplot2 library and defining the average number of clicks per hour (our lambda):

library(ggplot2)

# Average number of clicks per hour
lam <- 10

2. We now compute the probability mass function (PMF) of the Poisson distribution for each possible number of clicks using the dpois() function. For example, to calculate the probability of getting exactly 15 clicks:

clicks <- 15
prob <- dpois(clicks, lam)
cat(paste("The probability of getting", clicks, "clicks per hour is", prob, "\n"))

The output is:

The probability of getting 15 clicks per hour is 0.0347180696306841

3. We compute the PMF of the Poisson distribution for a range of possible click counts using dpois() and display the results in a chart. For example, to calculate the probability of getting from 0 to 30 clicks:

x <- 0:30
pmf <- dpois(x, lam)

4. We now plot the probability for each possible number of clicks:

ggplot(data.frame(x=x, pmf=pmf), aes(x, pmf)) +
  geom_bar(stat="identity") +
  xlab("Number of clicks") +
  ylab("Probability") +
  ggtitle(paste("PMF of the Poisson distribution with lambda =", lam))

5. We compute the CDF of the Poisson distribution and plot it:

# Compute the CDF of the Poisson distribution
cdf <- ppois(x, lam)

# Plot the CDF
ggplot(data.frame(x=x, cdf=cdf), aes(x, cdf)) +
  geom_step() +
  xlab("Number of clicks") +
  ylab("Cumulative probability") +
  ggtitle(paste("CDF of the Poisson distribution with lambda =", lam))

6. We calculate the number of clicks corresponding to a 90% probability:

q <- qpois(0.9, lam)
cat(paste("For a 90% probability, the number of clicks must be up to", q, "\n"))

The output is:

For a 90% probability, the number of clicks must be up to 14

For convenience, here is the equivalent Python script:

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import poisson

# Define the average number of clicks per hour
lam = 10

clicks = 15
prob = poisson.pmf(clicks, lam)
print(f"The probability of getting {clicks} clicks per hour is {prob}")

x = np.arange(0, 31)
pmf = poisson.pmf(x, lam)

plt.bar(x, pmf)
plt.xlabel('Number of clicks')
plt.ylabel('Probability')
plt.title(f'PMF of the Poisson distribution with lambda = {lam}')
plt.show()

cdf = poisson.cdf(x, lam)

plt.step(x, cdf)
plt.xlabel('Number of clicks')
plt.ylabel('Cumulative probability')
plt.title(f'CDF of the Poisson distribution with lambda = {lam}')
plt.show()

q = poisson.ppf(0.9, lam)
print(f"For a 90% probability, the number of clicks must be up to {q}")

You might also like

• Probability Distributions: Discrete Distributions and the Binomial
• Anomaly Detection: How to Identify Outliers in Data
• The Negative Binomial Distribution

Further Reading

For an accessible yet thorough introduction to probability distributions—including the Poisson—Finalmente ho capito la statistica by Maurizio De Pra covers these topics in a clear and approachable style, ideal for building solid intuition before moving on to more advanced topics.

How Many Trials Until the First Success?

We use the geometric distribution when we perform independent trials, each of which can result in either success or failure, and we want to know how many trials are needed to obtain the first success.

In symbols:

\( X \sim Geo(p) \\ \\ \)

• \(X\) is the number of trials needed to obtain the first success.
• \(r\) is the number of trials.
• \(P\) is the probability of success on each trial.
• We also define, as is natural: q = 1 – p

Here is where it gets interesting. We have:

\( \\ P(X=r) = p \times q ^ {r-1} \\ \)

P therefore denotes the probability that the first success occurs on trial number r.
Let us continue our reasoning:

\( P(X > r) = q ^ {r} \)

This allows us to calculate the probability that more than r trials are needed before the first success, as well as:

\( P(X \leq r) = 1 – q ^ {r} \\ \)

which helps us find the probability that r trials or fewer are needed to achieve the first success. The expected value is:

\( E(X) = \frac{1}{P} \\ \)

The variance is:

\( Var(X) = \frac{q}{P^{2}} \)

Worked Examples

We know that the probability of an ice skater completing a course without incident is 0.4. Therefore:

\( X \sim Geo(0.4) \\ \)

X is the number of attempts our skater must make in order to complete a course without any incident.

We are now ready to apply our new knowledge.

Let us calculate the expected number of attempts before achieving a success:

\( E(X) = \frac{1}{P} \\ \)
therefore
\( \frac{1}{0.4} = 2.5 \)

The variance in the number of attempts is quickly calculated:

\( Var(X) = \frac{q}{p^{2}} \\ \)
that is
\( \frac{0.6}{0.4^{2}} = \frac{0.6}{0.16} = 3.75 \\ \)

The probability of succeeding on the second attempt, after having failed the first:

\( P(X=2) = P \times q = 0.4 \times 0.6 = 0.24 \\ \)
that is, 24%

The probability of succeeding in 4 attempts or fewer? Easy!

\( P(X \leq 4) = 1-q^{4} = 1 – 0.6^{4} = 1 – 0.1296 \\ \)

That is 0.8704, or 87%.

The probability of needing more than 4 attempts? A simple calculation:

\( P(X > 4) = q^{4} = 0.6^{4} \\ \)

That is 0.1296, or about 13%.

Computing in R

Now that we have the formulas well in mind, we can let our laziness take over and use R to do the heavy lifting.

With P(X=2) and P=0.4:

dgeom(1, 0.4)

where 1 is the number of failures before the first success.

P(X<=4) and P=0.4:

pgeom(3, 0.4)

Simple, quick, and fun!

You might also like

• Probability Distributions: Discrete Distributions and the Binomial
• The Negative Binomial Distribution
• The Normal Distribution

Further Reading

For an accessible yet thorough introduction to discrete probability distributions—including the geometric—Finalmente ho capito la statistica by Maurizio De Pra covers these topics in a clear and approachable style, ideal for building solid intuition.

The tools available to the SEO professional are countless, and yet it is undeniable that a sense of disappointment lingers within us. Too often we deal with data of apparent strategic importance that turn out, when put to the test, to be fallacious or imprecise—mere red herrings.

Raise your hand if you have never felt a sense of frustration using any—truly any—competitive analysis tool, to give a concrete example.

Wittgenstein stated that whereof one cannot speak, thereof one must be silent.

Too often we draw conclusions based on utterly unreliable data. The trap represented by eye-catching graphs, conclusions just a click away, special effects—it is always with us. The tool that serves up the ready-made analysis we need, ready to use, simply does not exist.

Too often SEO is treated by its practitioners not so much as an art (that would be useful and is even necessary, though not sufficient) but as a sort of bag of tricks for apprentice storytellers, or as a stage for conjuring acts based on evanescent numbers. A sum of a few SEO tools and report templates good for every occasion. Or sometimes shamanistic practices combined into miracle recipes.

Back to statistics. SEO is obviously not just statistics, but it rests on numbers. Basic statistics, to begin with. The ABCs of descriptive statistics.

Let us examine what the only solid data we have tell us—the data that come from our own sites. Not sampled data, not data derived from presumed traffic based on average positions in search engine indices, without clinging to fanciful correlations presented as certainties on the basis of r values close to 0…

Our data. Let us look at what our data tell us.

Sometimes it seems that reasoning in “simple” terms—median and five-number summary—boxplots and histograms, hypothesis testing and moving averages—is reductive. Other times I meet professionals who consider even a basic grounding in statistics superfluous, ignoring the fact that statistics has a fundamental, irreplaceable virtue: it makes us sceptical, and exceedingly doubtful.

We know little, in truth. Little can be understood by looking at numbers generated mostly by what an algorithm—unknown in composition and in how its variables are weighted—compresses into positions in an index. That little, however, must be reasonably supported by data and by common sense.

The rest is art, precisely. Intuition, experience, and more. Only by tracing the boundaries between what is reasonably (statistically) expressed by the data in our possession and what our experience suggests can the insight, the decisive experiment, emerge.

Only in this way, I believe, can we provide a service of genuine value.

SEO is an intrinsically treacherous subject. Approaching it requires experience, a certain dose of courage, and a necessary sense of one’s own limits.

You might also like

• The Data: The 4 Scales of Measurement
• Descriptive Statistics: Measures of Position
• Guide to Statistical Tests for A/B Analysis

Further Reading

For a brilliant, accessible exploration of why statistical thinking matters—and how it protects us from misleading data—The Art of Statistics by David Spiegelhalter shows how to reason clearly with numbers in a world full of uncertainty.

In other words, measures of variability allow us to assess how data are spread around a central value, which may be represented, for example, by the mean or the median. They provide valuable information about the distribution of data, enabling a better understanding of the phenomenon under observation.

The techniques for measuring the variability of datasets are numerous. Among them, the most widely known (and most commonly used) are:

• the range
• the mean deviation and the variance
• the standard deviation
• the coefficient of variation

We will also visualise the concepts of central tendency and dispersion by revisiting skewness and introducing the concept of kurtosis.

The Range

The range is the difference between the maximum and the minimum value of ungrouped data in a frequency distribution.

It is a very quick calculation, which in R can be computed as follows:

max(var) - min(var)

The maximum and minimum can also be displayed with:

range(var)

and they appear as the first and last terms in:

fivenum(var)

For grouped data, the range is defined as the difference between the upper boundary of the highest class and the lower boundary of the lowest class.

A trimmed range is a range from which a certain percentage of extreme values has been removed from both ends of the distribution (for instance, the middle 80 per cent).

The Mean Deviation

The mean deviation is a measure of variability based on the difference between each data point and the mean. If we calculated the average by summing the positive and negative differences between each value and the arithmetic mean, the result would always be zero. For this reason, we sum the absolute values of the differences:

\(
SM = \frac{\Sigma|X – \mu|}{N} \\
\)

Those “absolute values” pose some computational efficiency issues, which is why the mean deviation is not widely used. There is another way to eliminate negative values, and so we introduce the important concept of…

Variance

Variance is analogous to the mean deviation, since it is based on the differences between each data point and the mean, but these differences are squared before being summed. Variance is denoted by the lowercase sigma squared symbol, and the formula is:

\(
\sigma^{2}=\frac{\Sigma(X – \mu)^{2}}{N} \\ \\
\)

R has the var() function for computing variance, but it uses (n-1) in the denominator. To obtain the variance with N in the denominator, we can define a custom function:

pop_var <- function(x) { var(x) * (1 - 1/length(x)) }

In general, interpreting the value of a variance is difficult because the units it is expressed in are not the same as those of the original observations.

For this reason, the standard deviation was introduced.

The Standard Deviation: The Most Widely Used Measure of Variability

The standard deviation is simply the square root of the variance:

\(
\sigma = \sqrt{\frac{\Sigma(X - \mu)^{2}}{N}} \\ \\
\)

The standard deviation is of fundamental utility in statistics, particularly (as we shall see) in conjunction with the normal distribution.

For grouped data, we assume that the midpoint of each class represents all the measurements within that class. The variance formula then becomes:

\(
\sigma^{2}=\frac{\Sigma f(X - \mu)^{2}}{N} \\ \\
\)

and the standard deviation:

\(
\sigma = \sqrt{\frac{\Sigma f(X - \mu)^{2}}{N}} \\ \\
\)

In R, the function for computing the standard deviation is sd(). However, R uses (n-1) in the denominator. So, if we want the population standard deviation (with n in the denominator), we can define a dedicated function:

pop_sd <- function(x) { sqrt(sum((x - mean(x))^2) / length(x)) }

The Coefficient of Variation

The coefficient of variation indicates the relative magnitude of the standard deviation with respect to the distribution mean. It is extremely useful for comparing phenomena expressed in different units of measurement, since the CV is a "pure" number, independent of the unit of measurement:

\(
CV = \frac{\sigma}{\mu} \\
\)

As is often the case in R, there is a ready-made function: we can use cv(), defined in an external library, labstatR. Its usage is straightforward:

library(labstatR)
data <- c(24, 17, 21, 23, 15, 30, 24, 21, 24, 19,
          25, 28, 22, 20, 14, 19, 26, 29, 23, 25,
          24, 18, 27, 21)
cv(data)
# [1] 0.1817708

We can also calculate the value quite simply without resorting to external libraries:

data <- c(24, 17, 21, 23, 15, 30, 24, 21, 24, 19,
          25, 28, 22, 20, 14, 19, 26, 29, 23, 25,
          24, 18, 27, 21)
pop_sd <- function(x) { sqrt(sum((x - mean(x))^2) / length(x)) }
cv_data <- pop_sd(data) / mean(data)
cv_data
# [1] 0.1817708

The Shape of a Distribution

Frequency distributions can take on the most varied shapes. Among all of them, the one by far most important in statistics is the normal distribution, also known as the bell curve or Gaussian distribution.

In a normal distribution, data are arranged symmetrically around the mean. In a very straightforward way, to describe the shape of the distribution we simply compare the mean with the median: if they are equal, the distribution is symmetric. If the mean is greater than the median, we have positive skewness (with a longer "tail" on the right); if the mean is less than the median, the skewness is negative (with the longer "tail" on the left).

The best-known formula for calculating the skewness of a distribution is Pearson's coefficient of skewness:

\(
Skewness = \frac{3(\mu - med)}{\sigma} \\ \\
\)

A perfectly symmetric distribution has a skewness value of 0. A right-skewed (positive) distribution has a positive value, while a left-skewed distribution has a negative value.

Skewness values typically fall between -3 and 3, and the fact that the standard deviation appears in the denominator makes the value independent of the unit of measurement.

How do we calculate the skewness index in R? The simplest way is to use a library that provides the functions we need "ready to go":

library(moments)
data <- c(24, 17, 21, 23, 15, 30, 24, 21, 24, 19,
          25, 28, 22, 20, 14, 19, 26, 29, 23, 25,
          24, 18, 27, 21)
skewness(data)
# [1] -0.1918578

The moments library serves us well. Let us see, however, how to calculate the index without relying on a library. It is very simple. The first step is to remember that R uses n-1 in the denominator of the variance. We are reasoning about a population, however, with n in the denominator. So, let us define a function that gives us the value we need:

pop_var <- function(x) { var(x) * (1 - 1/length(x)) }

At this point we can calculate the skewness index:

data <- c(24, 17, 21, 23, 15, 30, 24, 21, 24, 19,
          25, 28, 22, 20, 14, 19, 26, 29, 23, 25,
          24, 18, 27, 21)
pop_var <- function(x) { var(x) * (1 - 1/length(x)) }
z <- (data - mean(data)) / sqrt(pop_var(data))
skew <- mean(z^3)
skew
# [1] -0.1918578

Kurtosis

Kurtosis is the degree of peakedness of a distribution curve, relative to the normal distribution.

We have three cases:

1. a tall curve, called leptokurtic, which is highly concentrated around its mean
2. a normal curve, called mesokurtic
3. a low and flat curve, called platykurtic, with little concentration around its mean

Kurtosis can be measured by dividing the fourth moment by the standard deviation raised to the fourth power. Sounds difficult? It is easier done than said. Here is the formula:

\(
Kurtosis = \frac{\Sigma f(X -\mu)^{4}}{\sigma^{4}} \\ \\
\)

The kurtosis of a mesokurtic curve has a value of 3. Naturally, a kurtosis coefficient less than 3 indicates a platykurtic curve, while a value greater than 3 indicates a leptokurtic curve.

As with the skewness index, the moments library provides a convenient ready-made function:

library(moments)
data <- c(24, 17, 21, 23, 15, 30, 24, 21, 24, 19,
          25, 28, 22, 20, 14, 19, 26, 29, 23, 25,
          24, 18, 27, 21)
kurtosis(data)
# [1] 2.480035

But we are not averse to computing it ourselves:

data <- c(24, 17, 21, 23, 15, 30, 24, 21, 24, 19,
          25, 28, 22, 20, 14, 19, 26, 29, 23, 25,
          24, 18, 27, 21)
pop_var <- function(x) { var(x) * (1 - 1/length(x)) }
z <- (data - mean(data)) / sqrt(pop_var(data))
kurt <- mean(z^4)
kurt
# [1] 2.480035

We have seen that measures of central tendency alone are not enough: two datasets can have the same mean yet behave in profoundly different ways. Variability tells us how much the data fluctuate, and the shape of the distribution tells us how they are arranged. In the next article, we will take the first steps into the world of probability—the essential bridge between descriptive statistics and the inferential tools that let us draw conclusions from data.

You might also like

• Descriptive Statistics: Measures of Position and Central Tendency
• The Normal Distribution
• Anomaly Detection: How to Identify Outliers in Your Data

Further Reading

If you want a gentle yet rigorous introduction to the concepts behind descriptive statistics—including variability, distributions, and the art of making sense of data—The Art of Statistics by David Spiegelhalter is an excellent starting point. Spiegelhalter has the rare ability to explain statistical concepts without dumbing them down, and his treatment of uncertainty and dispersion is particularly illuminating.

When probability values are assigned to all the possible numerical values of a random variable x, the result is a probability distribution.

In even simpler terms: a random variable is a variable whose values are each associated with a probability of being observed. The set of all possible values of a random variable and their associated probabilities is called a probability distribution. The sum of all probabilities is 1.

Discrete and Continuous Variables

There are two main types of random variables: discrete and continuous.

• A discrete r.v. can take on a discrete (finite or countable) set of real numbers. That is, we could list all possible values in a table together with their respective probabilities. An example is the outcome of rolling a die: there are 6 possible outcomes, each with a probability of 1/6 (and the sum of all probabilities, of course, equals 1).
• A continuous r.v. can take on all values within a real interval—that is, an infinite number of values within any given interval. The probability that X falls within a given interval is represented by the area under the probability distribution. In the case of a continuous random variable, probabilities are represented by means of a probability density function. The total area under the curve (i.e. the total probability) equals 1.

Depending on the case, we deal with various types of distributions. These are the most common:

Event Yes or Event No? The Bernoulli Random Variable

Consider a trial in which we are only interested in verifying whether a certain event has occurred or not. The random variable generated by such a trial will take the value 1 if the event has occurred, 0 otherwise. This r.v. is called a Bernoulli random variable.

Any dichotomous trial can be represented by a Bernoulli random variable.

A bit of notation. We denote a Bernoulli r.v. as follows:

\( x \sim Bernoulli(\pi) \\ \)

Its mean is:

\( E(x)=\pi \\ \)

And its variance is:

\( V(x)=\pi(1-\pi) \\ \)

All trials that produce only 2 possible outcomes generate Bernoulli random variables (for example, tossing a coin). Starting from this simple assumption, it is a very short step to the Binomial Distribution.

The Binomial Distribution

Rather than dwelling on the conceptual aspects—important as they are, and for which I refer to specialised texts—what I want to do here is show in practice, and as clearly as possible, what we are talking about. Let us start with a definition and then look at the characteristics and a few practical examples.

The Binomial random variable can be understood as a sum of Bernoulli random variables.

What does this mean? Simply that if we repeat the success–failure dichotomy of the Bernoulli random variable n times under the same conditions, the result will be a sequence of n independent sub-trials, each of which can be associated with a Bernoulli random variable.

What are the characteristics of the binomial distribution?

• There is a fixed number of trials (n).
• Each trial has two possible outcomes: success or failure.
• The probability of success (p) is the same for every trial.
• The outcome of one trial does not affect any other (the trials are independent).

If even one of these characteristics is absent, the binomial distribution does not apply.

From a practical standpoint, the binomial distribution allows us to calculate the probability of obtaining r successes in n independent trials.

The probability of a certain number of successes, r, depends on r itself, on the number of trials n, and on the individual probability, which we denote by p.

The probability of r successes in n trials is given by:

\( \frac{n!}{r!(n-r)!} \times p^r (1-p)^{n-r} \\ \)

Looks difficult? It really is not (and in practice it turns out to be useful and even fun!).

First, let us recall that the symbol ! in mathematics denotes the factorial. As you will certainly remember, the factorial of 3, i.e. 3!, is: 3 × 2 × 1 = 6; the factorial of 4, i.e. 4!, is: 4 × 3 × 2 × 1 = 24; and so on (it will not escape notice that the factorial grows very, very quickly as the number increases…).

The factorial of a natural number is the product of that number by all its predecessors.

With that said, let us first see how to find the mean—the centre of our distribution—and the variance. This way, we will have everything we need for a few practical examples.

Mean, Expected Value, and Variance of a Binomial Distribution

Let us call x the expected value. We can write our problem as follows:

\( x \sim Binomial(n, p) \\ \)

The mean is:

\( E(x) = n \times p \\ \)

The variance is:

\( Var(x) = n \times p \times (1 – p) \\ \)

At this point, an example is in order. Let us calculate the variance of a distribution with size n = 10 and individual probability p = 0.5 (i.e. 50%). For instance, this could represent ten coin tosses.

\( x \sim Binomial(10, 0.5) \\ \)

So the variance will be:

\( Var(x) = 10 \times 0.5 \times (1 – 0.5) = 2.5 \\ \)

And the mean, naturally, will be:

\( E(x) = 10 \times 0.5 = 5 \\ \)

Side note: it is intuitive that if p = 1 – p = 0.5, the probability distribution will be symmetric. If p < 0.5, it will be right-skewed, and if p > 0.5, it will be left-skewed.

An Example: Computing the Probability Density

Let us now introduce the concept of probability density, which is what we will use most often in real-world applications. This is when, for example, we want to know the probability that exactly two out of ten coin tosses come up heads.

To explain this more clearly, let us take a problem from a textbook:

If I cross a black mouse with a white one, there is a 3/4 probability that the offspring will be black and 1/4 that it will be white. What is the probability that out of 7 offspring, exactly 3 are white?

Let us write down the data straight away:

• n = 7
• r = 3
• p = 1/4, i.e. 0.25

And now? Shall we do the calculations by hand? Why not:

\( \frac{n!}{r!(n-r)!} \times p^r (1-p)^{n-r} \\ \\ \)

therefore:

\( \frac{7!}{3!4!} \times 0.25^{3} \times 0.75^{4} = 35 \times 0.0049439 = 0.173 \\ \)

That is, 17.3%.

Doing calculations by hand is fun, but we are lazy and have R at our disposal. In R, the probability density is computed by a simple function:

dbinom()

The problem is therefore solved with the simple instruction:

dbinom(3, 7, 0.25)
# [1] 0.173

which gives us 0.173, so the answer is 17.3%.

Other Discrete Distributions

There are equally interesting questions that call upon other discrete distributions:

• How many trials should we expect before obtaining a success? This is where the geometric distribution enters the scene.
• How many times can we expect an event to occur (or not) in a given time interval? That calls for the Poisson distribution.
• Are we sampling from a population without replacement? Then we use the hypergeometric distribution.

As we can see, this is a vast and fascinating topic, which we will explore (lightly) across several articles. In the next one, we will look at another important distribution: the beta distribution, which plays a central role in Bayesian statistics.

You might also like

• First Steps into the World of Probability
• The Hypergeometric Distribution
• The Negative Binomial Distribution

Further Reading

For an accessible yet thorough introduction to probability distributions and the reasoning behind them, Finalmente ho capito la statistica by Maurizio De Pra covers discrete distributions—including the binomial—in a clear and approachable style, ideal for building solid intuition before moving on to more advanced topics.