![\"Infographic:](\"https:\/\/www.gironi.it\/blog\/wp-content\/uploads\/2026\/03\/slide-esploratore-bendato.jpg\")
<\/figure>\n\n\n\n\n\n\n\nFrom Cauchy to Neural Networks: A Brief History<\/h2>\n\n\n\n
The idea of following the slope to find a minimum has surprisingly ancient roots. In 1847<\/strong>, the French mathematician Augustin-Louis Cauchy published a method for solving systems of equations that, in essence, is already gradient descent: compute the direction of steepest ascent of a function and move in the opposite direction. Cauchy was not thinking about neural networks — their appearance was more than a century away — but he had formalized the principle that still powers artificial intelligence today.<\/p>\n\n\n\n For over a century the idea remained confined to pure mathematics. The breakthrough came in 1951<\/strong>, when Herbert Robbins and Sutton Monro proposed a stochastic<\/strong> version of the method: instead of computing the gradient on the entire problem, it is estimated on a random sample. This is the seed of what we now call Stochastic Gradient Descent<\/em> (SGD) — we will discuss it in the final section. A few years later, in 1958<\/strong>, Frank Rosenblatt built the perceptron<\/strong>, the first machine learning model that uses gradient descent to learn from data. The excitement was enormous, but short-lived: in 1969, Minsky and Papert demonstrated the perceptron’s limitations, and interest in neural networks collapsed. This is the so-called AI winter<\/em>.<\/p>\n\n\n\n The revival came in 1986<\/strong>, when David Rumelhart, Geoffrey Hinton, and Ronald Williams published the backpropagation<\/strong> algorithm: an efficient way to compute the gradient in neural networks with many layers. It was the key that unlocked deep learning. From that moment on, gradient descent was no longer a theoretical exercise but the practical tool used to train neural networks. The final piece of this story was set by Diederik Kingma and Jimmy Ba in 2014<\/strong>, with the Adam<\/strong> optimizer — which we will encounter again in the final section of this article. Nearly two centuries separate Cauchy from Adam, yet the principle has remained the same: feel the slope, take a step in the opposite direction.<\/p>\n\n\n\n Before we descend, we need to know what<\/strong> we are minimizing. The blindfolded person is looking for the lowest point in the valley — but in machine learning, what exactly is that valley?<\/p>\n\n\n\n Let’s take a concrete example. Suppose we want to predict the price of a house knowing only its area. We have five houses for which we know the area and the actual price, and our model is the simplest possible: a line through the origin, price = m × area<\/strong>, where m<\/em> is the only parameter to adjust.<\/p>\n\n\n\n For each value of m<\/em>, the model makes a prediction. If m<\/em> is too low, the predictions underestimate the actual prices; if it is too high, they overestimate them. We need a number that tells us how much<\/em> the model is getting wrong: this is the cost function<\/strong> (or loss function<\/em>).<\/p>\n\n\n\n The most widely used cost function is the Mean Squared Error<\/strong> (MSE): for each house, we compute the difference between the predicted price and the actual price, square it, and take the average of all these errors. In formula:<\/p>\n\n\n\n MSE(m) = (1\/n) ∑i<\/sub> (pricei<\/sub> − m × areai<\/sub>)²<\/strong><\/p>\n\n\n\n If we plot MSE(m) as m<\/em> varies, we get a parabola-shaped curve: the same parabola we will use shortly as an example of gradient descent. This is no coincidence — the parabola is the cost function<\/strong>, and its lowest point is the value of m<\/em> that makes the model as good as possible.<\/p>\n\n\n\n The examples that follow are available in both R and Python — choose whichever language you’re more comfortable with.<\/p>\n\n\n\n Let’s compute the cost function in R for our house example:<\/p>\n\n\n\n Let’s verify in Python:<\/p>\n\n\n\n Now we know what to minimize: the cost function. The gradient tells us how<\/em>.<\/p>\n\n\n\n When you have a function of a single variable — think of it as a trail that goes up and down — the derivative<\/strong> at a point tells you how steep the trail is at that point. If the derivative is positive, you’re going uphill. If it’s negative, you’re going downhill. If it’s zero, you’re on flat ground: perhaps a peak, perhaps a dip.<\/p>\n\n\n\n Concrete example: the function f(x) = x² describes a parabola. Its derivative is f'(x) = 2x. If you’re at x = 3, the derivative is 6: you’re climbing steeply. At x = -1, the derivative is -2: you’re going downhill. At x = 0, the derivative is zero: you’re at the lowest point.<\/p>\n\n\n\n In practice, the functions we care about don’t depend on just one variable. A machine learning model can have hundreds, thousands, or billions of parameters. The landscape is no longer a trail but a surface in a high-dimensional space — impossible to visualize, but perfectly manageable with mathematics.<\/p>\n\n\n\n The gradient<\/strong> is the vector that collects all partial derivatives of the function with respect to each parameter. If the function depends on two variables (x, y), the gradient is:<\/p>\n\n\n\n ∇f = (∂f\/∂x, ∂f\/∂y)<\/strong><\/p>\n\n\n\n If it depends on a thousand variables, the gradient is a vector with a thousand components. In every case, the gradient points in the direction of steepest ascent. To find the minimum, just go in the opposite direction: minus the gradient<\/strong>.<\/p>\n\n\n\n Here’s the fundamental formula of gradient descent:<\/p>\n\n\n\n θnew<\/sub> = θold<\/sub> − α · ∇f(θ)<\/strong><\/p>\n\n\n\n Let’s break it down piece by piece:<\/p>\n\n\n\n That’s all there is to it. Compute where you’re going uphill, take a step in the opposite direction, repeat.<\/p>\n\n\n\n Let’s see gradient descent in action on the function f(x) = x². We know the minimum is at x = 0. Can the algorithm find it starting from a random point?<\/p>\n\n\n\n Let’s verify in Python:<\/p>\n\n\n\n Running this code, you can see the algorithm start from x = 10 and quickly converge toward x = 0. The first steps are large (the slope is steep), then they shrink as we approach the bottom of the parabola. After 50 iterations, x is essentially zero.<\/p>\n\n\n\n Back to our blindfolded person. How big is the step they take at each iteration? This is exactly the question of the learning rate<\/strong> (α), and the answer is less obvious than it seems.<\/p>\n\n\n\n Steps too small<\/strong> (α very low): the person moves with extreme caution, shifting their foot just a few centimeters at a time. They’ll eventually reach the bottom of the valley, but it could take forever. In machine learning, this means extremely long training times and high computational costs.<\/p>\n\n\n\n Steps too large<\/strong> (α too high): the person takes enormous leaps. Instead of gently descending into the valley, they overshoot it, end up on the other side, bounce back, and keep oscillating without ever settling. In extreme cases, the jumps get bigger and bigger and the person ends up higher than where they started. In machine learning, this is called divergence<\/em>: the model gets worse instead of better.<\/p>\n\n\n\n Steps just right<\/strong>: a good learning rate allows you to descend quickly without oscillating. In practice, finding the right value requires experimentation. It’s one of the most artisanal aspects of machine learning.<\/p>\n\n\n\n How does the blindfolded person know they’ve arrived? They feel the ground is flat in all directions: the gradient is (nearly) zero. In practice, the algorithm stops when the improvement between iterations becomes negligible, or when it has reached a maximum number of iterations.<\/p>\n\n\n\n The most common stopping criteria are:<\/p>\n\n\n\n This R code shows the effect of three different learning rate values on the same function:<\/p>\n\n\n\n Let’s compare in Python:<\/p>\n\n\n\n With α = 0.01, the red dots move sluggishly: after 30 iterations we’re still far from the minimum. With α = 0.1, convergence is fast and clean. With α = 0.9, the algorithm oscillates visibly at each step, bouncing from one side of the parabola to the other before settling — a slightly higher learning rate would diverge entirely.<\/p>\n\n\n\n The mathematical landscape of a real model is not a nice symmetric parabola. It’s wild terrain, with secondary valleys, ridges, plateaus, and shapes that defy imagination. Here are the classic problems, explained through our landscape analogy.<\/p>\n\n\n\n Imagine terrain with multiple dips: one deep valley (the global minimum) and several shallower hollows (local minima). The blindfolded person has no way of knowing whether the valley they’re in is the deepest one. They feel flat ground beneath their feet and stop, convinced they’ve arrived. But they might be in a shallow hollow while the true minimum lies somewhere else entirely.<\/p>\n\n\n\n In practice, this is a less severe problem than once thought. Modern neural networks have so many parameters that local minima tend to have objective function values similar to the global minimum. It’s like terrain with many valleys, but all at roughly the same altitude: ending up in any one of them works fine.<\/p>\n\n\n\n A more insidious problem are saddle points<\/strong>. Imagine sitting on a horse saddle: if you move forward or backward, you descend; if you move left or right, you ascend. At that point the gradient is zero — the terrain feels flat — but you’re not at a minimum. You’re at a point that is a minimum in some directions and a maximum in others.<\/p>\n\n\n\n In high-dimensional spaces, saddle points are far more common than local minima. Fortunately, modern variants of gradient descent (with some noise or momentum, as we’ll see) generally manage to escape saddle points.<\/p>\n\n\n\n Imagine a very narrow, elongated valley, like a canyon. The gradient points almost perpendicularly to the canyon walls, not along the canyon toward the bottom. The blindfolded person ends up bouncing from one wall to the other, making an inefficient zigzag instead of walking straight toward the bottom.<\/p>\n\n\n\n This happens when the problem’s variables have very different scales: some change rapidly, others slowly. It’s a common issue in practice, and one of the main motivations for advanced optimizers like Adam, which we’ll see in the final section.<\/p>\n\n\n\n So far we’ve seen gradient descent on a parabola: a problem with a single variable. It’s the simplest possible case. But the beauty of this algorithm is that it works in exactly the same way at any scale.<\/p>\n\n\n\n Here’s how the number of parameters grows as models become more complex:<\/p>\n\n\n\n The principle is identical: compute the gradient, take a step in the opposite direction, repeat. What changes is the scale of the computation. GPT-4’s gradient is a vector with over a trillion components, computed on billions of text fragments, using thousands of processors in parallel. But the formula is the same one we saw on the parabola.<\/p>\n\n\n\n Every time you interact with an AI system, gradient descent has been working behind the scenes:<\/p>\n\n\n\n The key message is this: the power of modern AI lies not in the complexity of the optimization algorithm, but in scale<\/strong>. Gradient descent is conceptually simple. What made the AI revolution possible is the ability to apply it to enormous models on enormous amounts of data, thanks to ever more powerful hardware.<\/p>\n\n\n\n Vanilla gradient descent — what we’ve seen so far — works, but it has the limitations we described: it can be slow, it can oscillate, it can get stuck. Over the years, researchers have developed variants that address these problems. Without diving into the formulas, here are the key ideas.<\/p>\n\n\n\n Instead of computing the gradient on the entire dataset at each step (computationally very expensive), SGD computes it on a small random sample (mini-batch<\/em>). It’s as if the blindfolded person, instead of feeling the entire terrain around them, only sampled a few random points. The slope estimate is noisy but correct on average, and the computation speed is enormously greater. The noise, paradoxically, is also useful: it helps escape local minima and saddle points.<\/p>\n\n\n\n Imagine a ball rolling down the hill instead of a person walking. The ball builds up velocity: if the slope continues in the same direction, it accelerates. If the slope changes direction, the ball slows down before reversing. This is momentum: the algorithm “remembers” the direction it was moving and adds the current gradient to it. The result is that it crosses flat zones faster and oscillates less in narrow valleys.<\/p>\n\n\n\n Adam<\/strong> (Adaptive Moment Estimation<\/em>) combines the momentum idea with a learning rate that adapts automatically for each parameter. Parameters that change little get larger steps; those that change a lot get smaller steps. It’s as if the blindfolded person had smart shoes that adjust the step length based on the terrain under each foot.<\/p>\n\n\n\n Adam has become the de facto standard for training most modern neural networks. It’s robust, requires little manual tuning, and works well across a wide range of problems. Nearly all the models you use every day — from Spotify to ChatGPT — were trained with Adam or one of its variants.<\/p>\n\n\n\n The blindfolded person we started with now has better shoes. They have a ball that builds velocity instead of legs that take rigid steps. They have soles that automatically adapt to the terrain. And above all, they’re not walking on a mountain with two or three dimensions: they’re walking through a landscape with billions of dimensions.<\/p>\n\n\n\n But the principle is exactly the same. Feel the slope. Take a step where it goes down. Repeat.<\/p>\n\n\n\n Gradient descent is not a spectacular algorithm. It lacks the elegant complexity of a genetic algorithm or the narrative charm of adversarial networks. It’s a mechanical procedure, almost trivial. But it’s the mechanical procedure upon which the entire artificial intelligence revolution rests. From Netflix suggestions to image-generating models, from self-driving cars to simultaneous translation, it all comes down to this: a function to minimize, a gradient to compute, a step to take.<\/p>\n\n\n\n The next time a voice assistant understands your question, or an automatic translator nails a nuance, remember: behind the scenes, a very sophisticated version of our blindfolded person has walked billions of steps through a landscape with billions of dimensions. And found a valley deep enough to be useful.<\/p>\n","protected":false},"excerpt":{"rendered":"What We’ll Cover<\/h3>\n
\n
The Math, Explained Geometrically<\/h2>\n\n\n\n
The Cost Function: Measuring How Wrong the Model Is<\/h3>\n\n\n\n
# Cost function: predicting the price of a house\narea <- c(50, 70, 90, 120, 150)\nprice <- c(150, 200, 260, 340, 400) # thousands of euros\n\n# Model: price = m * area\n# MSE cost function as m varies\nm_values <- seq(1, 4, by = 0.01)\nmse <- sapply(m_values, function(m) mean((price - m * area)^2))\n\nplot(m_values, mse, type = \"l\", lwd = 2, col = \"steelblue\",\n xlab = \"m (slope)\", ylab = \"MSE\",\n main = \"Cost function as m varies\")\nm_best <- m_values[which.min(mse)]\nabline(v = m_best, col = \"red\", lty = 2)\ncat(\"The value of m that minimizes the error:\", round(m_best, 2), \"\\n\")<\/code><\/pre>\n\n\n\nimport numpy as np\nimport matplotlib.pyplot as plt\n\n# Cost function: predicting the price of a house\narea = np.array([50, 70, 90, 120, 150])\nprice = np.array([150, 200, 260, 340, 400]) # thousands of euros\n\n# Model: price = m * area\n# MSE cost function as m varies\nm_values = np.linspace(1, 4, 301) # equivalent to seq(1, 4, by=0.01) in R\nmse = np.array([np.mean((price - m * area)**2) for m in m_values])\n\nplt.plot(m_values, mse, lw=2, color=\"steelblue\")\nplt.xlabel(\"m (slope)\")\nplt.ylabel(\"MSE\")\nplt.title(\"Cost function as m varies\")\nm_best = m_values[np.argmin(mse)]\nplt.axvline(m_best, color=\"red\", linestyle=\"--\")\nplt.show()\nprint(f\"The value of m that minimizes the error: {m_best:.2f}\")<\/code><\/pre>\n\n\n\nDerivative: the slope beneath your feet<\/h3>\n\n\n\n
![\"Graph](\"https:\/\/www.gironi.it\/blog\/wp-content\/uploads\/2026\/03\/slide-derivata-pendenza.jpg\")
<\/figure>\n\n\n\nThe gradient: a multidimensional compass<\/h3>\n\n\n\n
\n
![\"Visual](\"https:\/\/www.gironi.it\/blog\/wp-content\/uploads\/2026\/03\/slide-anatomia-passo.jpg\")
<\/figure>\n\n\n\nA numerical example in R<\/h3>\n\n\n\n
# Gradient descent on f(x) = x^2\n# The derivative is f'(x) = 2x\n\nf <- function(x) x^2 # objective function\ngrad_f <- function(x) 2 * x # derivative (gradient in 1D)\n\nx <- 10 # starting point\nalpha <- 0.1 # learning rate\nn_iter <- 50 # number of iterations\npath <- numeric(n_iter)\n\nfor (i in 1:n_iter) {\n path[i] <- x\n x <- x - alpha * grad_f(x) # the fundamental rule\n}\n\ncat(\"Starting point: 10\\n\")\ncat(\"After 50 iterations: x =\", round(x, 8), \"\\n\")\ncat(\"Function value:\", round(f(x), 10), \"\\n\")\n\n# Visualization\ncurve(x^2, from = -11, to = 11, lwd = 2, col = \"steelblue\",\n main = \"Gradient descent on f(x) = x^2\",\n xlab = \"x\", ylab = \"f(x)\")\npoints(path, path^2, col = \"red\", pch = 19, cex = 0.7)\nlines(path, path^2, col = \"red\", lty = 2)<\/code><\/pre>\n\n\n\nimport numpy as np\nimport matplotlib.pyplot as plt\n\n# Gradient descent on f(x) = x^2\nf = lambda x: x**2 # objective function\ngrad_f = lambda x: 2 * x # derivative (gradient in 1D)\n\nx = 10.0 # starting point\nalpha = 0.1 # learning rate\nn_iter = 50 # number of iterations\npath = np.zeros(n_iter)\n\nfor i in range(n_iter):\n path[i] = x\n x = x - alpha * grad_f(x) # the fundamental rule\n\nprint(f\"Starting point: 10\")\nprint(f\"After 50 iterations: x = {x:.8f}\")\nprint(f\"Function value: {f(x):.10f}\")\n\n# Visualization\nxs = np.linspace(-11, 11, 200)\nplt.plot(xs, xs**2, lw=2, color=\"steelblue\")\nplt.plot(path, path**2, \"ro--\", markersize=4)\nplt.title(\"Gradient descent on f(x) = x\\u00b2\")\nplt.xlabel(\"x\")\nplt.ylabel(\"f(x)\")\nplt.show()<\/code><\/pre>\n\n\n\nThe Learning Rate: Big Steps or Small Steps?<\/h2>\n\n\n\n
![\"Visual](\"https:\/\/www.gironi.it\/blog\/wp-content\/uploads\/2026\/03\/slide-learning-rate.jpg\")
<\/figure>\n\n\n\nConvergence: knowing when to stop<\/h3>\n\n\n\n
\n
The effect of the learning rate: a visual comparison<\/h3>\n\n\n\n
# Comparing three learning rates on f(x) = x^2\ngradient_descent <- function(x0, alpha, n_iter = 30) {\n x <- x0\n path <- numeric(n_iter)\n for (i in 1:n_iter) {\n path[i] <- x\n x <- x - alpha * 2 * x # theta_new = theta_old - alpha * grad\n }\n return(path)\n}\n\nx0 <- 8 # same starting point for all\n\n# Three different learning rates\nslow <- gradient_descent(x0, alpha = 0.01) # too small\nright <- gradient_descent(x0, alpha = 0.1) # good compromise\nfast <- gradient_descent(x0, alpha = 0.9) # nearly unstable\n\n# Visualization\npar(mfrow = c(1, 3))\n\n# alpha = 0.01 (too slow)\ncurve(x^2, from = -10, to = 10, lwd = 2, col = \"steelblue\",\n main = expression(paste(alpha, \" = 0.01 (too slow)\")))\npoints(slow, slow^2, col = \"red\", pch = 19, cex = 0.6)\nlines(slow, slow^2, col = \"red\", lty = 2)\n\n# alpha = 0.1 (just right)\ncurve(x^2, from = -10, to = 10, lwd = 2, col = \"steelblue\",\n main = expression(paste(alpha, \" = 0.1 (good compromise)\")))\npoints(right, right^2, col = \"darkgreen\", pch = 19, cex = 0.6)\nlines(right, right^2, col = \"darkgreen\", lty = 2)\n\n# alpha = 0.9 (nearly unstable)\ncurve(x^2, from = -10, to = 10, lwd = 2, col = \"steelblue\",\n main = expression(paste(alpha, \" = 0.9 (nearly unstable)\")))\npoints(fast, fast^2, col = \"orange\", pch = 19, cex = 0.6)\nlines(fast, fast^2, col = \"orange\", lty = 2)\n\npar(mfrow = c(1, 1))<\/code><\/pre>\n\n\n\nimport numpy as np\nimport matplotlib.pyplot as plt\n\ndef gradient_descent(x0, alpha, n_iter=30):\n x = x0\n path = np.zeros(n_iter)\n for i in range(n_iter):\n path[i] = x\n x = x - alpha * 2 * x # theta_new = theta_old - alpha * grad\n return path\n\nx0 = 8.0 # same starting point for all\nslow = gradient_descent(x0, alpha=0.01) # too small\nright = gradient_descent(x0, alpha=0.1) # good compromise\nfast = gradient_descent(x0, alpha=0.9) # nearly unstable\n\nfig, axes = plt.subplots(1, 3, figsize=(14, 4))\nxs = np.linspace(-10, 10, 200)\nfor ax, data, color, title in zip(axes,\n [slow, right, fast],\n [\"red\", \"darkgreen\", \"orange\"],\n [\"\\u03b1 = 0.01 (too slow)\", \"\\u03b1 = 0.1 (good compromise)\",\n \"\\u03b1 = 0.9 (nearly unstable)\"]):\n ax.plot(xs, xs**2, lw=2, color=\"steelblue\")\n ax.plot(data, data**2, \"o--\", color=color, markersize=4)\n ax.set_title(title)\nplt.tight_layout()\nplt.show()<\/code><\/pre>\n\n\n\nWhat Can Go Wrong: Geometric Intuition<\/h2>\n\n\n\n
![\"Three](\"https:\/\/www.gironi.it\/blog\/wp-content\/uploads\/2026\/03\/slide-insidie-topografiche.jpg\")
<\/figure>\n\n\n\nLocal minima: secondary valleys<\/h3>\n\n\n\n
Saddle points: the horse saddle<\/h3>\n\n\n\n
Narrow valleys: the zigzag<\/h3>\n\n\n\n
From a Parabola to ChatGPT<\/h2>\n\n\n\n
The scale of parameters<\/h3>\n\n\n\n
\n
Where you see it in action (without knowing it)<\/h3>\n\n\n\n
\n
The Evolutions: Better Shoes for Our Explorer<\/h2>\n\n\n\n
Stochastic Gradient Descent (SGD)<\/h3>\n\n\n\n
Momentum<\/h3>\n\n\n\n
Adam: the Swiss army knife<\/h3>\n\n\n\n
Back to the Mountain<\/h2>\n\n\n\n