Imagine standing on a mountainous terrain, completely blindfolded. Your goal: reach the lowest point in the valley. You can’t see anything, but you can feel the slope of the ground beneath your feet. What do you do? You move in the direction where the ground goes down, one step at a time. If it slopes more steeply to the left, you go left. If it drops more to the right, you go right. With each step, you feel the slope again and redirect yourself.

This strategy, so simple and natural, is exactly what neural networks use to learn. Every time an AI model improves — learning to recognize a face, translate a sentence, or generate text — it does so by descending through a mathematical landscape, one step at a time, following the slope.

It’s called gradient descent, and it’s arguably the most important algorithm in modern machine learning.

From Cauchy to Neural Networks: A Brief History

The idea of following the slope to find a minimum has surprisingly ancient roots. In 1847, the French mathematician Augustin-Louis Cauchy published a method for solving systems of equations that, in essence, is already gradient descent: compute the direction of steepest ascent of a function and move in the opposite direction. Cauchy was not thinking about neural networks — their appearance was more than a century away — but he had formalized the principle that still powers artificial intelligence today.

For over a century the idea remained confined to pure mathematics. The breakthrough came in 1951, when Herbert Robbins and Sutton Monro proposed a stochastic version of the method: instead of computing the gradient on the entire problem, it is estimated on a random sample. This is the seed of what we now call Stochastic Gradient Descent (SGD) — we will discuss it in the final section. A few years later, in 1958, Frank Rosenblatt built the perceptron, the first machine learning model that uses gradient descent to learn from data. The excitement was enormous, but short-lived: in 1969, Minsky and Papert demonstrated the perceptron’s limitations, and interest in neural networks collapsed. This is the so-called AI winter.

The revival came in 1986, when David Rumelhart, Geoffrey Hinton, and Ronald Williams published the backpropagation algorithm: an efficient way to compute the gradient in neural networks with many layers. It was the key that unlocked deep learning. From that moment on, gradient descent was no longer a theoretical exercise but the practical tool used to train neural networks. The final piece of this story was set by Diederik Kingma and Jimmy Ba in 2014, with the Adam optimizer — which we will encounter again in the final section of this article. Nearly two centuries separate Cauchy from Adam, yet the principle has remained the same: feel the slope, take a step in the opposite direction.

The Math, Explained Geometrically

The Cost Function: Measuring How Wrong the Model Is

Before we descend, we need to know what we are minimizing. The blindfolded person is looking for the lowest point in the valley — but in machine learning, what exactly is that valley?

Let’s take a concrete example. Suppose we want to predict the price of a house knowing only its area. We have five houses for which we know the area and the actual price, and our model is the simplest possible: a line through the origin, price = m × area, where m is the only parameter to adjust.

For each value of m, the model makes a prediction. If m is too low, the predictions underestimate the actual prices; if it is too high, they overestimate them. We need a number that tells us how much the model is getting wrong: this is the cost function (or loss function).

The most widely used cost function is the Mean Squared Error (MSE): for each house, we compute the difference between the predicted price and the actual price, square it, and take the average of all these errors. In formula:

MSE(m) = (1/n) ∑_i (price_i − m × area_i)²

If we plot MSE(m) as m varies, we get a parabola-shaped curve: the same parabola we will use shortly as an example of gradient descent. This is no coincidence — the parabola is the cost function, and its lowest point is the value of m that makes the model as good as possible.

The examples that follow are available in both R and Python — choose whichever language you’re more comfortable with.

Let’s compute the cost function in R for our house example:

# Cost function: predicting the price of a house
area <- c(50, 70, 90, 120, 150)
price <- c(150, 200, 260, 340, 400)  # thousands of euros

# Model: price = m * area
# MSE cost function as m varies
m_values <- seq(1, 4, by = 0.01)
mse <- sapply(m_values, function(m) mean((price - m * area)^2))

plot(m_values, mse, type = "l", lwd = 2, col = "steelblue",
     xlab = "m (slope)", ylab = "MSE",
     main = "Cost function as m varies")
m_best <- m_values[which.min(mse)]
abline(v = m_best, col = "red", lty = 2)
cat("The value of m that minimizes the error:", round(m_best, 2), "\n")

Let’s verify in Python:

import numpy as np
import matplotlib.pyplot as plt

# Cost function: predicting the price of a house
area = np.array([50, 70, 90, 120, 150])
price = np.array([150, 200, 260, 340, 400])  # thousands of euros

# Model: price = m * area
# MSE cost function as m varies
m_values = np.linspace(1, 4, 301)  # equivalent to seq(1, 4, by=0.01) in R
mse = np.array([np.mean((price - m * area)**2) for m in m_values])

plt.plot(m_values, mse, lw=2, color="steelblue")
plt.xlabel("m (slope)")
plt.ylabel("MSE")
plt.title("Cost function as m varies")
m_best = m_values[np.argmin(mse)]
plt.axvline(m_best, color="red", linestyle="--")
plt.show()
print(f"The value of m that minimizes the error: {m_best:.2f}")

Now we know what to minimize: the cost function. The gradient tells us how.

Derivative: the slope beneath your feet

When you have a function of a single variable — think of it as a trail that goes up and down — the derivative at a point tells you how steep the trail is at that point. If the derivative is positive, you’re going uphill. If it’s negative, you’re going downhill. If it’s zero, you’re on flat ground: perhaps a peak, perhaps a dip.

Concrete example: the function f(x) = x² describes a parabola. Its derivative is f'(x) = 2x. If you’re at x = 3, the derivative is 6: you’re climbing steeply. At x = -1, the derivative is -2: you’re going downhill. At x = 0, the derivative is zero: you’re at the lowest point.

The gradient: a multidimensional compass

In practice, the functions we care about don’t depend on just one variable. A machine learning model can have hundreds, thousands, or billions of parameters. The landscape is no longer a trail but a surface in a high-dimensional space — impossible to visualize, but perfectly manageable with mathematics.

The gradient is the vector that collects all partial derivatives of the function with respect to each parameter. If the function depends on two variables (x, y), the gradient is:

∇f = (∂f/∂x, ∂f/∂y)

If it depends on a thousand variables, the gradient is a vector with a thousand components. In every case, the gradient points in the direction of steepest ascent. To find the minimum, just go in the opposite direction: minus the gradient.

Here’s the fundamental formula of gradient descent:

θ_new = θ_old − α · ∇f(θ)

Let’s break it down piece by piece:

• θ represents the model’s parameters — the “knobs” the algorithm adjusts to improve
• ∇f(θ) is the gradient: it indicates the direction of steepest ascent at the current point
• α (alpha) is the learning rate: the size of the step we take at each iteration
• The minus sign makes us go in the opposite direction of the gradient, i.e. downhill

That’s all there is to it. Compute where you’re going uphill, take a step in the opposite direction, repeat.

A numerical example in R

Let’s see gradient descent in action on the function f(x) = x². We know the minimum is at x = 0. Can the algorithm find it starting from a random point?

# Gradient descent on f(x) = x^2
# The derivative is f'(x) = 2x

f <- function(x) x^2         # objective function
grad_f <- function(x) 2 * x  # derivative (gradient in 1D)

x <- 10              # starting point
alpha <- 0.1          # learning rate
n_iter <- 50          # number of iterations
path <- numeric(n_iter)

for (i in 1:n_iter) {
  path[i] <- x
  x <- x - alpha * grad_f(x)  # the fundamental rule
}

cat("Starting point: 10\n")
cat("After 50 iterations: x =", round(x, 8), "\n")
cat("Function value:", round(f(x), 10), "\n")

# Visualization
curve(x^2, from = -11, to = 11, lwd = 2, col = "steelblue",
      main = "Gradient descent on f(x) = x^2",
      xlab = "x", ylab = "f(x)")
points(path, path^2, col = "red", pch = 19, cex = 0.7)
lines(path, path^2, col = "red", lty = 2)

Let’s verify in Python:

import numpy as np
import matplotlib.pyplot as plt

# Gradient descent on f(x) = x^2
f = lambda x: x**2           # objective function
grad_f = lambda x: 2 * x     # derivative (gradient in 1D)

x = 10.0            # starting point
alpha = 0.1          # learning rate
n_iter = 50          # number of iterations
path = np.zeros(n_iter)

for i in range(n_iter):
    path[i] = x
    x = x - alpha * grad_f(x)  # the fundamental rule

print(f"Starting point: 10")
print(f"After 50 iterations: x = {x:.8f}")
print(f"Function value: {f(x):.10f}")

# Visualization
xs = np.linspace(-11, 11, 200)
plt.plot(xs, xs**2, lw=2, color="steelblue")
plt.plot(path, path**2, "ro--", markersize=4)
plt.title("Gradient descent on f(x) = x\u00b2")
plt.xlabel("x")
plt.ylabel("f(x)")
plt.show()

Running this code, you can see the algorithm start from x = 10 and quickly converge toward x = 0. The first steps are large (the slope is steep), then they shrink as we approach the bottom of the parabola. After 50 iterations, x is essentially zero.

The Learning Rate: Big Steps or Small Steps?

Back to our blindfolded person. How big is the step they take at each iteration? This is exactly the question of the learning rate (α), and the answer is less obvious than it seems.

Steps too small (α very low): the person moves with extreme caution, shifting their foot just a few centimeters at a time. They’ll eventually reach the bottom of the valley, but it could take forever. In machine learning, this means extremely long training times and high computational costs.

Steps too large (α too high): the person takes enormous leaps. Instead of gently descending into the valley, they overshoot it, end up on the other side, bounce back, and keep oscillating without ever settling. In extreme cases, the jumps get bigger and bigger and the person ends up higher than where they started. In machine learning, this is called divergence: the model gets worse instead of better.

Steps just right: a good learning rate allows you to descend quickly without oscillating. In practice, finding the right value requires experimentation. It’s one of the most artisanal aspects of machine learning.

Convergence: knowing when to stop

How does the blindfolded person know they’ve arrived? They feel the ground is flat in all directions: the gradient is (nearly) zero. In practice, the algorithm stops when the improvement between iterations becomes negligible, or when it has reached a maximum number of iterations.

The most common stopping criteria are:

• The gradient norm drops below a minimum threshold (the terrain is nearly flat)
• The difference in f(θ) between two consecutive iterations is smaller than a fixed tolerance
• The maximum number of iterations has been reached (computational budget exhausted)

The effect of the learning rate: a visual comparison

This R code shows the effect of three different learning rate values on the same function:

# Comparing three learning rates on f(x) = x^2
gradient_descent <- function(x0, alpha, n_iter = 30) {
  x <- x0
  path <- numeric(n_iter)
  for (i in 1:n_iter) {
    path[i] <- x
    x <- x - alpha * 2 * x  # theta_new = theta_old - alpha * grad
  }
  return(path)
}

x0 <- 8  # same starting point for all

# Three different learning rates
slow   <- gradient_descent(x0, alpha = 0.01)   # too small
right  <- gradient_descent(x0, alpha = 0.1)    # good compromise
fast   <- gradient_descent(x0, alpha = 0.9)    # nearly unstable

# Visualization
par(mfrow = c(1, 3))

# alpha = 0.01 (too slow)
curve(x^2, from = -10, to = 10, lwd = 2, col = "steelblue",
      main = expression(paste(alpha, " = 0.01 (too slow)")))
points(slow, slow^2, col = "red", pch = 19, cex = 0.6)
lines(slow, slow^2, col = "red", lty = 2)

# alpha = 0.1 (just right)
curve(x^2, from = -10, to = 10, lwd = 2, col = "steelblue",
      main = expression(paste(alpha, " = 0.1 (good compromise)")))
points(right, right^2, col = "darkgreen", pch = 19, cex = 0.6)
lines(right, right^2, col = "darkgreen", lty = 2)

# alpha = 0.9 (nearly unstable)
curve(x^2, from = -10, to = 10, lwd = 2, col = "steelblue",
      main = expression(paste(alpha, " = 0.9 (nearly unstable)")))
points(fast, fast^2, col = "orange", pch = 19, cex = 0.6)
lines(fast, fast^2, col = "orange", lty = 2)

par(mfrow = c(1, 1))

Let’s compare in Python:

import numpy as np
import matplotlib.pyplot as plt

def gradient_descent(x0, alpha, n_iter=30):
    x = x0
    path = np.zeros(n_iter)
    for i in range(n_iter):
        path[i] = x
        x = x - alpha * 2 * x  # theta_new = theta_old - alpha * grad
    return path

x0 = 8.0  # same starting point for all
slow  = gradient_descent(x0, alpha=0.01)   # too small
right = gradient_descent(x0, alpha=0.1)    # good compromise
fast  = gradient_descent(x0, alpha=0.9)    # nearly unstable

fig, axes = plt.subplots(1, 3, figsize=(14, 4))
xs = np.linspace(-10, 10, 200)
for ax, data, color, title in zip(axes,
        [slow, right, fast],
        ["red", "darkgreen", "orange"],
        ["\u03b1 = 0.01 (too slow)", "\u03b1 = 0.1 (good compromise)",
         "\u03b1 = 0.9 (nearly unstable)"]):
    ax.plot(xs, xs**2, lw=2, color="steelblue")
    ax.plot(data, data**2, "o--", color=color, markersize=4)
    ax.set_title(title)
plt.tight_layout()
plt.show()

With α = 0.01, the red dots move sluggishly: after 30 iterations we’re still far from the minimum. With α = 0.1, convergence is fast and clean. With α = 0.9, the algorithm oscillates visibly at each step, bouncing from one side of the parabola to the other before settling — a slightly higher learning rate would diverge entirely.

What Can Go Wrong: Geometric Intuition

The mathematical landscape of a real model is not a nice symmetric parabola. It’s wild terrain, with secondary valleys, ridges, plateaus, and shapes that defy imagination. Here are the classic problems, explained through our landscape analogy.

Local minima: secondary valleys

Imagine terrain with multiple dips: one deep valley (the global minimum) and several shallower hollows (local minima). The blindfolded person has no way of knowing whether the valley they’re in is the deepest one. They feel flat ground beneath their feet and stop, convinced they’ve arrived. But they might be in a shallow hollow while the true minimum lies somewhere else entirely.

In practice, this is a less severe problem than once thought. Modern neural networks have so many parameters that local minima tend to have objective function values similar to the global minimum. It’s like terrain with many valleys, but all at roughly the same altitude: ending up in any one of them works fine.

Saddle points: the horse saddle

A more insidious problem are saddle points. Imagine sitting on a horse saddle: if you move forward or backward, you descend; if you move left or right, you ascend. At that point the gradient is zero — the terrain feels flat — but you’re not at a minimum. You’re at a point that is a minimum in some directions and a maximum in others.

In high-dimensional spaces, saddle points are far more common than local minima. Fortunately, modern variants of gradient descent (with some noise or momentum, as we’ll see) generally manage to escape saddle points.

Narrow valleys: the zigzag

Imagine a very narrow, elongated valley, like a canyon. The gradient points almost perpendicularly to the canyon walls, not along the canyon toward the bottom. The blindfolded person ends up bouncing from one wall to the other, making an inefficient zigzag instead of walking straight toward the bottom.

This happens when the problem’s variables have very different scales: some change rapidly, others slowly. It’s a common issue in practice, and one of the main motivations for advanced optimizers like Adam, which we’ll see in the final section.

From a Parabola to ChatGPT

So far we’ve seen gradient descent on a parabola: a problem with a single variable. It’s the simplest possible case. But the beauty of this algorithm is that it works in exactly the same way at any scale.

The scale of parameters

Here’s how the number of parameters grows as models become more complex:

• Simple linear regression: 2 parameters (slope and intercept). The landscape is a 3D surface, easy to visualize.
• Neural network for handwritten digit recognition: ~100,000 parameters. The landscape has 100,000 dimensions.
• ResNet-50 (image classification, 2015): ~25 million parameters.
• GPT-3 (ChatGPT’s predecessor): 175 billion parameters.
• GPT-4 and frontier models (2023-2025): estimated over one trillion parameters.

The principle is identical: compute the gradient, take a step in the opposite direction, repeat. What changes is the scale of the computation. GPT-4’s gradient is a vector with over a trillion components, computed on billions of text fragments, using thousands of processors in parallel. But the formula is the same one we saw on the parabola.

Where you see it in action (without knowing it)

Every time you interact with an AI system, gradient descent has been working behind the scenes:

• Netflix and Spotify recommending what to watch or listen to: recommendation models are trained with gradient descent on billions of user interactions
• Google Translate and automatic translators: neural networks with hundreds of millions of parameters, optimized with gradient descent on massive parallel text corpora
• Voice assistants (Siri, Alexa): speech recognition uses deep neural networks, trained with the same algorithm
• Self-driving cars: the networks that recognize pedestrians, traffic lights, and lane markings are trained with gradient descent variants
• ChatGPT, Claude, Gemini: Large Language Models are the most extreme case — gradient descent applied to billions of parameters on trillions of text tokens

The key message is this: the power of modern AI lies not in the complexity of the optimization algorithm, but in scale. Gradient descent is conceptually simple. What made the AI revolution possible is the ability to apply it to enormous models on enormous amounts of data, thanks to ever more powerful hardware.

The Evolutions: Better Shoes for Our Explorer

Vanilla gradient descent — what we’ve seen so far — works, but it has the limitations we described: it can be slow, it can oscillate, it can get stuck. Over the years, researchers have developed variants that address these problems. Without diving into the formulas, here are the key ideas.

Stochastic Gradient Descent (SGD)

Instead of computing the gradient on the entire dataset at each step (computationally very expensive), SGD computes it on a small random sample (mini-batch). It’s as if the blindfolded person, instead of feeling the entire terrain around them, only sampled a few random points. The slope estimate is noisy but correct on average, and the computation speed is enormously greater. The noise, paradoxically, is also useful: it helps escape local minima and saddle points.

Momentum

Imagine a ball rolling down the hill instead of a person walking. The ball builds up velocity: if the slope continues in the same direction, it accelerates. If the slope changes direction, the ball slows down before reversing. This is momentum: the algorithm “remembers” the direction it was moving and adds the current gradient to it. The result is that it crosses flat zones faster and oscillates less in narrow valleys.

Adam: the Swiss army knife

Adam (Adaptive Moment Estimation) combines the momentum idea with a learning rate that adapts automatically for each parameter. Parameters that change little get larger steps; those that change a lot get smaller steps. It’s as if the blindfolded person had smart shoes that adjust the step length based on the terrain under each foot.

Adam has become the de facto standard for training most modern neural networks. It’s robust, requires little manual tuning, and works well across a wide range of problems. Nearly all the models you use every day — from Spotify to ChatGPT — were trained with Adam or one of its variants.

Back to the Mountain

The blindfolded person we started with now has better shoes. They have a ball that builds velocity instead of legs that take rigid steps. They have soles that automatically adapt to the terrain. And above all, they’re not walking on a mountain with two or three dimensions: they’re walking through a landscape with billions of dimensions.

But the principle is exactly the same. Feel the slope. Take a step where it goes down. Repeat.

Gradient descent is not a spectacular algorithm. It lacks the elegant complexity of a genetic algorithm or the narrative charm of adversarial networks. It’s a mechanical procedure, almost trivial. But it’s the mechanical procedure upon which the entire artificial intelligence revolution rests. From Netflix suggestions to image-generating models, from self-driving cars to simultaneous translation, it all comes down to this: a function to minimize, a gradient to compute, a step to take.

The next time a voice assistant understands your question, or an automatic translator nails a nuance, remember: behind the scenes, a very sophisticated version of our blindfolded person has walked billions of steps through a landscape with billions of dimensions. And found a valley deep enough to be useful.

Further Reading

To place gradient descent in the broader context of statistical learning, An Introduction to Statistical Learning by James, Witten, Hastie and Tibshirani is the read we recommend: rigorous, yet always practice-oriented.

With linear regression we learned to predict a number. Here, though, the outcome is a switch, on or off, and a straight line is not the right tool: we’ll see it even ends up predicting negative probabilities. The right tool is logistic regression, one of the most widely used models of all, from marketing to medicine. Let’s build it together, starting from an example with conversions.

In short: logistic regression is the statistical model that estimates the probability of a binary outcome (converts or not, clicks or not) from one or more variables. Instead of a straight line it uses the logistic function, an S-shaped curve that always stays between 0 and 1. The coefficients are expressed in log-odds: exponentiated, they become odds ratios, the factor by which the odds of the event are multiplied when a variable increases by one unit. In R it is fitted with glm(..., family = binomial) and predicted probabilities come from predict(..., type = "response"). To classify, we then need a threshold, which with rare events has to be chosen with care: 0.5 is almost never the right one.

What we’ll cover:

• Why isn’t a straight line enough?
• From probability to odds to logit: the logistic equation
• How do we fit a logistic regression in R?
• How do we read the coefficients? Odds ratios
• From probabilities to decisions: where to put the threshold?
• The conditions for using it
• Frequently asked questions
• Try it yourself
• Further reading

Why isn’t a straight line enough?

Let’s take 5,000 organic sessions on an e-commerce site. For each one we know three things: how many pages it viewed, whether it came from a smartphone or a desktop, and whether the starting query contained the brand name. And we know whether the session ended with a purchase: 271 converted, 5.4%. The data are simulated, so we know the truth and can check whether the model recovers it.

The most immediate idea is to treat conversion as a number, 1 if it happened and 0 if not, and run a least-squares line through it. It even has a name, the linear probability model, and in some cases it works acceptably. But let’s see what happens with our data.

The line says every extra page is always worth 1.9 percentage points of conversion, whether we go from 1 to 2 pages or from 8 to 9. The data tell another story: sessions with few pages convert very little and almost regardless, then the probability speeds up. And when we also add device and brand, things get worse: the linear model assigns 654 sessions out of 5,000 a negative conversion probability, down to −2.2%. A probability below zero has no meaning at all. At the other end, it gives the best profile (12 pages, desktop, brand query) 28.8%, where logistic regression, as we’ll see, estimates 72.9%.

The problem is one of shape. A probability lives between 0 and 1, and near the edges it has to “brake”: it can’t go above 100% or below zero. A straight line, instead, goes on forever. We need a curve with built-in brakes.

From probability to odds to logit: the logistic equation

Logistic regression solves the problem with a change of scale in two steps. The first step is the odds, a word anyone familiar with betting knows well: instead of the probability that an event happens, we look at the ratio between the probability that it happens and the probability that it doesn’t.

\(\text{odds} = \frac{p}{1-p}\)

With a probability of 20%, the odds are 0.2 / 0.8 = 0.25: “one to four”, one success for every four failures. At 50% the odds are 1, and at 80% they rise to 4. The odds have no ceiling at 1: they run from zero to infinity.

The second step is to take their logarithm. The logarithm of the odds, called the logit, also removes the floor at zero and can take any value, from minus infinity to plus infinity. And it is precisely on this unconstrained scale that logistic regression places its linear equation, identical in form to that of multiple regression:

\(\ln\left(\frac{p}{1-p}\right) = b_0 + b_1 x_1 + b_2 x_2 + \dots + b_k x_k\)

where p is the probability of the event, the x’s are the variables (the predictors) and the b’s are the coefficients to estimate. To go back to the probability we just reverse the path, and we get the logistic function, also called the sigmoid because of its S shape:

\(p = \frac{1}{1 + e^{-(b_0 + b_1 x_1 + \dots + b_k x_k)}}\)

Whatever value the expression in brackets takes, the result always stays between 0 and 1. These are the built-in brakes: logistic regression is a straight line on the log-odds scale, which becomes an S-shaped curve on the probability scale.

The coefficients are not found by least squares but by maximum likelihood: among all possible curves, we pick the one that makes the data we actually observed most probable. The calculation is iterative, and we gladly leave it to the software.

How do we fit a logistic regression in R?

Let’s simulate the 5,000 sessions. In real life the data would come from Google Analytics or the CRM; here we build them ourselves, with known effects, so we can check the result:

set.seed(7)
n <- 5000
sessions <- data.frame(
  pages  = 1 + rpois(n, 2.5),    # pages viewed in the session
  mobile = rbinom(n, 1, 0.6),    # 1 = smartphone, 0 = desktop
  brand  = rbinom(n, 1, 0.25)    # 1 = arrived from a query with the brand name
)
true_logit <- -4.4 + 0.35 * sessions$pages - 0.5 * sessions$mobile + 1.2 * sessions$brand
sessions$converts <- rbinom(n, 1, plogis(true_logit))

mean(sessions$converts)
# [1] 0.0542

The plogis() function is the logistic function itself: it turns the true logit into a probability, and rbinom() tosses a loaded coin with that probability for each session. That’s why family = binomial appears in the model: each session is a Bernoulli trial, the building block of the binomial distribution.

Let’s fit the model:

model <- glm(converts ~ pages + mobile + brand, data = sessions, family = binomial)
summary(model)
# Coefficients:
#             Estimate Std. Error z value Pr(>|z|)
# (Intercept) -4.37566    0.18872 -23.187  < 2e-16 ***
# pages        0.34742    0.03584   9.693  < 2e-16 ***
# mobile      -0.50057    0.12834  -3.900 9.61e-05 ***
# brand        1.19691    0.12868   9.302  < 2e-16 ***
#
#     Null deviance: 2107.0  on 4999  degrees of freedom
# Residual deviance: 1923.8  on 4996  degrees of freedom
# AIC: 1931.8

The model recovers almost perfectly the values we used to build the data: 0.347 against 0.35 for pages, −0.501 against −0.5 for mobile, 1.197 against 1.2 for brand. The columns read as in linear regression: estimate, standard error, the z statistic (here instead of t) and the p-value. The deviance is the logistic counterpart of the residual sum of squares: it drops from 2107 (the model with no predictors) to 1924, a sign that the three variables explain something.

The sign of the coefficients already tells us a lot: more pages and brand queries push the conversion probability up, smartphones push it down. But by how much? Here logistic regression asks for one more step.

How do we read the coefficients? Odds ratios

The coefficients are on the log-odds scale, which nobody is used to thinking in. The trick is to exponentiate them: e raised to the coefficient is the odds ratio, the number by which the odds of conversion are multiplied when the variable increases by one unit, holding the others fixed. Let’s compute them together with their confidence intervals:

round(exp(cbind(OR = coef(model), confint(model))), 3)
#                OR 2.5 % 97.5 %
# (Intercept) 0.013 0.009  0.018
# pages       1.415 1.319  1.519
# mobile      0.606 0.471  0.780
# brand       3.310 2.572  4.261

Let’s translate row by row:

• pages: each extra page viewed multiplies the odds of conversion by 1.415, which means they grow by about 41.5% (with an interval between +32% and +52%);
• mobile: on a smartphone the odds are 60.6% of those on desktop, about 40% lower;
• brand: a session arriving from a query with the brand name has odds 3.3 times those of a non-brand session.

None of the intervals contains 1, the value that would mean “no effect”.

This last point deserves a chart. Let’s compute the predicted probabilities for four session profiles, with type = "response", which returns probabilities instead of log-odds:

profiles <- data.frame(pages  = c(3, 3, 8, 8),
                       mobile = c(1, 0, 1, 0),
                       brand  = c(0, 1, 0, 1))
round(predict(model, newdata = profiles, type = "response"), 3)
#     1     2     3     4
# 0.021 0.106 0.109 0.401

A smartphone session, non-brand, with 3 pages viewed converts with a 2.1% probability. The same number of pages on desktop and from a brand query is worth 10.6%. With 8 pages, desktop and brand, we reach 40.1%.

Let’s check the first value by hand, to see there’s no magic. The logit is −4.376 + 0.347 × 3 − 0.501 × 1 + 1.197 × 0 = −3.834. The probability is 1 / (1 + e^3.834) = 1 / (1 + 46.2) = 0.021.

The four curves share the same odds ratio per page, 1.415. And yet in percentage points the effect of one extra page changes enormously: for a mobile non-brand session, going from 2 to 3 pages adds 0.6 points (from 1.5% to 2.1%); for a mobile session from a brand query, going from 8 to 9 adds 7.6 (from 28.9% to 36.5%). That’s the S-curve at work, and it’s why in reports it pays to show both the odds ratio and a few predicted probabilities for concrete profiles.

From probabilities to decisions: where to put the threshold?

Often we want a decision from the model: which sessions to show an offer to, which users to put into a remarketing campaign. To turn a probability into a yes or a no we need a threshold, and the instinctive choice is 0.5: if the probability is above 50%, we predict a conversion.

To judge the model honestly, though, we can’t evaluate it on the same data we fitted it on. Let’s split the sessions: we fit on 3,500 and measure the predictions on the other 1,500, which the model has never seen.

set.seed(11)
rows <- sample(n, 3500)
train <- sessions[rows, ]
test <- sessions[-rows, ]
model_train <- glm(converts ~ pages + mobile + brand, data = train, family = binomial)
p_test <- predict(model_train, newdata = test, type = "response")

table(predicted = p_test > 0.5, actual = test$converts)
#          actual
# predicted    0    1
#     FALSE 1410   89
#     TRUE     1    0
mean((p_test > 0.5) == test$converts)
# [1] 0.94

94% accuracy: it looks like a great result. But let’s look at the table: of the 89 sessions that actually converted, the model identified none. It simply answered “no” almost every time. And with a 5.9% conversion rate, anyone who always answers “no”, with no model at all, gets 94.1% accuracy, better than us.

With rare events, accuracy is a misleading metric. The model isn’t useless: it’s the threshold that’s wrong. In our data almost no session goes above a 50% probability, so a 0.5 threshold amounts to never flagging anything. Let’s lower it to 0.1:

table(predicted = p_test > 0.1, actual = test$converts)
#          actual
# predicted    0    1
#     FALSE 1292   62
#     TRUE   119   27

Now the model flags 146 sessions and catches 27 that really convert: 30% of conversions (the sensitivity, or recall). Among the flagged sessions, 18.5% convert (the precision), more than three times the starting 5.9%. Accuracy drops to 88%, but the model has become useful.

The chart shows the trade-off. Converting sessions sit further to the right on average, but the two distributions overlap a lot: every threshold is a swap between conversions caught and false alarms. The right threshold isn’t decided by statistics, it’s decided by the cost of the errors: if showing an offer to someone who won’t buy costs little, a low threshold pays off.

To judge the model independently of the threshold, the AUC is often used, the area under the ROC curve: it is the probability that, taking at random one session that converts and one that doesn’t, the model gives the first a higher probability. It’s 0.5 for a model that guesses and 1 for a perfect one. Let’s compute it with ranks, the same idea behind the Wilcoxon test:

r <- rank(p_test)
n1 <- sum(test$converts); n0 <- sum(1 - test$converts)
round((sum(r[test$converts == 1]) - n1 * (n1 + 1) / 2) / (n1 * n0), 3)
# [1] 0.749

An AUC of 0.75 is a fair result: with only three variables, the model orders three pairs out of four correctly.

The conditions for using it

Logistic regression asks for less than linear regression (no normality of residuals, no constant variance), but a few conditions remain:

• the outcome is binary: only two categories (for three or more, see the frequently asked questions below);
• the observations are independent: ten sessions from the same user are not ten independent pieces of information;
• the relationship is linear on the logit scale: if the effect of pages grows up to a point and then flattens out, it has to be modelled (for example with bands of pages viewed);
• the predictors are not too correlated with each other, the multicollinearity problem we’ll discuss in the next article;
• there are enough events: a widespread rule of thumb asks for at least ten or so events (in our case, conversions) per predictor. We have 271 for 3 variables, well above that.

Finally there is a less known condition, and it’s not unusual to read the exact opposite: that logistic regression is used “when the data are linearly separable”. The opposite is true: if a variable perfectly separates the yeses from the noes, logistic regression breaks down. Maximum likelihood pushes the coefficient towards infinity to turn the curve into a perfect step, and the standard error explodes. It happens more often than we’d think, even with famous datasets.

Frequently asked questions

What is the difference between linear and logistic regression?

Linear regression predicts a continuous numeric variable, such as traffic or revenue. Logistic regression predicts the probability of a binary outcome, such as converts or doesn’t convert. To do so it models the logarithm of the odds linearly and returns probabilities that always lie between 0 and 1.

What is the odds ratio in logistic regression?

It is the exponential of a coefficient: it tells by how much the odds of the event are multiplied when the variable increases by one unit, holding the others fixed. An odds ratio of 1.4 means odds 40% higher, one of 0.6 odds 40% lower, one equal to 1 no effect.

How do we get predicted probabilities from a logistic regression?

By applying the logistic function to the linear combination of the coefficients: p = 1 / (1 + e^−(b0 + b1x1 + … + bkxk)). In R, predict(model, newdata, type = "response") returns the probabilities directly; without type = "response" it returns log-odds.

What if there are more than two categories?

If the categories have no order (for example desktop, mobile and tablet), we use multinomial logistic regression, in R with multinom() from the nnet package. If they have a natural order (low, medium, high), we use ordinal logistic regression, in R with polr() from the MASS package.

What is the right threshold for classifying?

It depends on the cost of the two kinds of error and on how frequent the event is. With rare events, such as conversions, a 0.5 threshold often leads to never flagging anything. It’s worth looking at the prediction table for several thresholds, or at the AUC to judge the model independently of the threshold.

How much data does a logistic regression need?

What matters most is the number of events, the cases in the less frequent category. A widespread rule of thumb asks for at least ten or so per predictor: with 50 conversions, better not to go beyond four or five variables.

Try it yourself

The article’s code lends itself to a few experiments.

• Try a 0.2 threshold instead of 0.1: how many conversions does the model catch, and with how many false alarms? Compare the precision with the one at 0.1.
• Remove brand from the model (converts ~ pages + mobile) and look at the AUC on the test set: how much is the information about the starting query worth on its own?
• Raise the true coefficient for pages from 0.35 to 0.7 in the simulation: how does the shape of the curve change, and what happens to the 0.5 threshold?

The last exercise shows well the link between the strength of the predictors and the difficulty of classification: a model with strong effects pushes the probabilities towards the edges, and there one threshold is as good as another.

In our example the three variables are independent by construction: pages viewed don’t depend on the device, brand doesn’t depend on pages. In real life that’s almost never the case. People arriving from a search for the brand name also tend to browse more pages, and when predictors overlap the coefficients become unstable and hard to tell apart. That’s multicollinearity, and together with two other names that sound scarier than they deserve, heteroscedasticity and autocorrelation, it’s the next stop on our journey through regression.

Further reading

To go further into logistic regression and classification, I recommend An Introduction to Statistical Learning by James, Witten, Hastie and Tibshirani: its chapter on classification starts from exactly the question of why linear regression doesn’t fit a yes/no outcome, then walks through logistic regression, confusion matrices and thresholds, with R labs that let us reproduce every step.