How momentum differs — you step by the velocity instead of the gradient
Momentum is a variant of gradient descent that steps by a velocity built up from past gradients instead of by the current gradient. It is also called the momentum optimizer or SGD with momentum, and when training a neural network in PyTorch you switch it on with the momentum argument of torch.optim.SGD. Gradient descent got slower the more the curvature differed between directions (the larger the condition number). Momentum lengthens the steps in the less curved direction and reduces that slowness.
The velocity takes the place of the gradient in Eq. (1). The velocity is times the previous velocity plus the current gradient.
| Symbol | Name | Meaning |
|---|---|---|
| momentum coefficient | How much of the previous velocity to keep. . The protagonist of this page. With , Eq. (2) becomes Eq. (1). Depending on the text it is written (Sutskever et al., PyTorch docs) or (Goodfellow et al.) | |
| velocity | The direction you actually step in at step . Before the first step, |
The other symbols (, , , , , , , the curvature ) follow the symbol table on the gradient descent page.
The name comes from the momentum of a heavy ball rolling down a slope. It is the tendency to keep going the way you were going, so it is also described as inertia. Because of friction, at every step the ball loses of its velocity, keeps of it, and picks up the push of the slope. measures that velocity toward uphill (toward ), so Eq. (2) subtracts from .
Unroll Eq. (2) backward one line at a time, and the velocity is the sum of past gradients, each shrunk by a factor of for every step further back.
If the gradient is the same every time, Eq. (3) approaches times that gradient, that is, times, the sum of the geometric series. One step grows to as much as 2 times gradient with , and 10 times with .11 This page computes with the full gradient, not a mini-batch.
Why momentum descends faster than gradient descent when β is right
Because when is right, in the less curved direction the velocity builds up while the sign of the gradient stays the same, and the steps get longer. On the same bowl, from the same start, with , the loss after 20 steps is 1.24 for gradient descent () and 0.106 for (Figure 2). In the long run 'right' means , which the next section derives.
The bowl is from the gradient descent page, its gradient is , and the start is . Figure 1 shows three paths with fixed at 0.05 and only changed.
figures/momentum_paths.pyLet's follow two steps by hand with . In the first step , so you land on , the same point as gradient descent. The gradient for the second step is , and the velocity is . So .
Compared with gradient descent's , has moved forward by 0.539 instead of 0.429. With , throughout the 20 steps, so the component of the gradient ( itself) never changes sign either. So from the second to the twentieth step ( to ), the component of the velocity is larger in absolute value than the component of the gradient.
In exchange, overshoots the bottom because of the leftover velocity. At , , but the component of is , so . That is why the path in Figure 1 goes back and forth in .
What's the difference between SGD and momentum?
The difference is the quantity used for one step. SGD without momentum steps by the current gradient; momentum steps by a velocity built from past gradients, each shrunk by a factor of (Eq. (2)). On the same bowl with , the loss after 20 steps is 1.24 for SGD and 0.106 for momentum with . Whether the gradient comes from a mini-batch (stochastic) is the same for both; in PyTorch the only difference is the momentum argument of torch.optim.SGD.
Why too large a β slows it down again
Because the velocity lingers too long, so even after passing the bottom it keeps going the same way for a while. With , goes down to a loss of 0.606 at step 6 and then climbs back to 5.34 at step 8 (Figure 2, left).
From step 11 on, has a larger loss than at every step, and its 1.18 after 20 steps is close to gradient descent's 1.24.22 That is one point on a curve that goes up and down: one step earlier, at step 19, it is 0.490, smaller than gradient descent's 1.378. So speed is compared with the asymptotic factor below. The right panel of Figure 2 measures this value for each .
figures/momentum_loss.pyOn a quadratic bowl with and fixed, the factor that multiplies a coordinate at each step cannot, averaged over a long run, go below (when the component along that direction is not 0). This long-run average is called the asymptotic factor. The motion along one direction is a mix of two factors whose product is , so the slower one cannot be smaller than (derived below).
With , . Averaged over a long run, the size of a coordinate cannot shrink faster than about 5.1% per step, which is in loss. A single step is not bound by this value. The path goes up and down as in the left panel of Figure 2: from step 5 to step 6 the loss falls from 4.75 to 0.606, a drop of 87%, and over the next two steps it rises again.
This 0.949 is almost the same as the factor of gradient descent with , and larger than the slower factor 0.885 of (derived below). So in the long run it falls behind , and with it becomes slower than even gradient descent.
Why √β — solving Eq. (2) along one direction
is the curvature of that direction (the symbol from the gradient descent page), is the coordinate along that direction ( or on this bowl), and is a root of the quadratic equation below.
From the right-hand equation of Eq. (2), (at , , so set ). Substitute this into , and disappears.
It is Eq. (1) plus times the previous step. In a direction with curvature (gradient ), .
Put in and you get . The product of the two roots is , so the larger one has absolute value at least . The roots are complex when , and then both have absolute value exactly .
For , , the direction () gives , so the larger root is . That is smaller than gradient descent's 0.95. The direction () has the complex roots of , so its factor is .
With , the direction has complex roots when , that is, when is above about 0.603. The direction already has complex roots at smaller , so from there on the factors of both directions are . means .
Both roots have absolute value below 1 when ( in the symbol table) and , that is, .
If you also choose , the slower factor is smallest at the on the boundary where the roots of the direction of smallest curvature and of the direction of largest curvature both turn from real to complex. Writing the condition number as , that is , , and the factor there is . For this bowl () that gives , , and a factor of about 0.52. With , , and reaches 0.9 at .
Raising the momentum coefficient makes steps in the same direction longer, but raising it too far makes the path overshoot the bottom and spend steps coming back, so it gets slower instead.
How to choose β and the learning rate
The right depends on the learning rate, so you choose the two together. By the loss after 20 steps on this bowl, is best when , but with the value 0.9 is worse than gradient descent. PyTorch's default is (no momentum), and 0.9 is commonly used. Goodfellow et al. give 0.5, 0.9, and 0.99 as examples and note that it is sometimes started small and raised later.
Lowering the learning rate does not push the asymptotic factor below . So on this bowl is slower than the best of gradient descent, , whatever you choose.
Why 0.9 is common all the same can be explained by the condition number. If you choose as well, the best for this bowl, whose condition number is 10, is and . The asymptotic factor there is 0.52, smaller than the 0.818 that is the best of gradient descent (the end of the folded block 'Why √β'). The larger the condition number, the larger the best : about 0.67 at 100, and 0.9 at about 1440. 0.9 is too large for a condition number of 10, and it is the right value for problems whose condition number is above a thousand.
Does momentum reduce the zigzag (oscillation) of gradient descent?
With the right , momentum makes the zigzag (oscillation) die out faster. In gradient descent with , flips sign at every step and its size is multiplied by 0.8. Switching on at the same lowers this factor, averaged over a long run, to , so after 20 steps is 0.0006 instead of 0.017. With the factor rises to 0.949, and the path keeps oscillating for longer instead.
When is momentum a bad idea?
If you keep the learning rate as it is and switch on a large , it can be slower than gradient descent. At , which is already fast, the loss after 20 steps is 0.005 for , about 0.00001 for , and 3.02 for . Measuring from 0.72 to 0.95 at intervals of 0.001, every value gives a larger loss than gradient descent. The asymptotic factor is larger than gradient descent's .
Do I need to change the learning rate when I use momentum?
Turning on changes the step size and the divergence limit by different factors, so choose the learning rate again together with . A step in a direction where the gradient is constant grows by a factor of (10 times for 0.9). But the divergence limit widens only by a factor of , so the limit 0.2 of the direction becomes just 0.38 at . So if you raise only at the same learning rate, the path may not diverge but easily overshoots the bottom.
Common misconceptions about momentum
- Turning on momentum without touching the learning rate leaves the stride the same — If the gradient is constant, the velocity in Eq. (3) grows to times the gradient, so with
momentum=0.9one step gets up to 10 times longer even at the samelr. - The larger β, the faster — With the loss after 20 steps is smallest near , and for it is larger than even gradient descent's 1.24 (Figure 2). The lower bound of the asymptotic factor also grows with .
Where momentum shows up in AI
In the training loop of a neural network, momentum changes only one stage, the update optimizer.step(). Clearing the gradients, computing the loss, and getting the gradient by backpropagation are the same as in gradient descent. step() keeps a velocity for each parameter and updates it with Eq. (2).
| This page | PyTorch |
|---|---|
momentum in torch.optim.SGD(params, lr=0.05, momentum=0.5). Default 0 (no momentum, Eq. (1)); a commonly used value is 0.9. The docs write it as | |
optimizer.state[p]["momentum_buffer"], which the optimizer keeps for each parameter. At the first step() it starts as the gradient itself (with dampening=0 this is the same as Eq. (2) with ) | |
lr (default 1e-3) | |
| Eq. (2) | optimizer.step() when momentum > 0, dampening and weight_decay are 0, and nesterov and maximize are False (everything other than momentum at its default) |
dampening | From the second step() on, the in Eq. (2) is multiplied by (1 − dampening). At the first step() the buffer is the gradient itself. Default 0 |
nesterov=False | The default. True gives Nesterov momentum (planned) and requires momentum > 0 and dampening=0. Its convergence condition differs from that of Eq. (2) (below the table) |
With Nesterov momentum, the learning-rate limit in a direction with curvature is , not . In the direction of this bowl with , that is 0.15, not 0.3.
Sutskever et al. and Keras put the learning rate inside the velocity and write , .33 With a fixed this gives the same path as Eq. (2) (the of this form is the of Eq. (2) multiplied by ). If the learning rate changes from step to step, the velocity in this form keeps past learning rates, and the two forms differ.
The next page, Adam, divides this velocity, coordinate by coordinate, by the size of past gradients. Adam's corresponds to this page's , and that page writes the velocity as because it uses for a different quantity.
What does momentum=0.9 mean in PyTorch?
momentum=0.9 means that at every step you keep 90% of the previous velocity and add the current gradient (the of Eq. (2)). The default is 0, and 0.9 is a commonly used value. On this bowl with , the loss after 20 steps is 1.18, larger than for (0.106). Why 0.9 is common all the same is the condition-number story in the section 'How to choose β and the learning rate'.
Check it in code
Two update lines change from the code on the gradient descent page. The code keeps a velocity velocity and steps by the velocity instead of the gradient. Change it to beta = 0 and you get the result of gradient descent with .
import numpy as np
def L(p):
x, y = p
return 0.5 * (x**2 + 10 * y**2)
def grad(p):
x, y = p
return np.array([x, 10 * y]) # ∇L = (x, 10y)
theta = np.array([-4.4, 1.5]) # θ₀
eta = 0.05 # η
beta = 0.5 # β (0 이면 경사하강법)
velocity = np.zeros(2) # v₋₁ = 0
for t in range(21):
if t % 5 == 0: print(f"t={t:2d} L={L(theta):.3f}")
velocity = beta * velocity + grad(theta) # 식 (2) 왼쪽
theta = theta - eta * velocity # 식 (2) 오른쪽$ python momentum.py
t= 0 L=20.930
t= 5 L=4.161
t=10 L=1.214
t=15 L=0.360
t=20 L=0.106Try it yourself
- With and , where is after two steps? How far has moved from the start, and how does that compare with gradient descent and ?
Answer
Up to and , everything is the same whatever is. , so . has moved forward by 0.627, farther than gradient descent (0.429) and (0.539), and has already passed the bottom, at .
Further reading
- Some methods of speeding up the convergence of iteration methods — B. T. Polyak, 1964. USSR Computational Mathematics and Mathematical Physics 4(5). The original source of the 'heavy ball' method, which adds β times the previous step. It analyzes the rate of convergence on quadratic functions. The folded derivation in the section 'Why too large a β slows it down again' relies on it.
- On the importance of initialization and momentum in deep learning — Sutskever, Martens, Dahl, Hinton, 2013. ICML. It writes classical momentum and Nesterov momentum side by side, and shows by experiment that the momentum coefficient and the initialization matter in training deep networks. The μ notation in the symbol table and the form that puts the learning rate inside the velocity in the section 'Where momentum shows up in AI' rely on it.
- Deep Learning, Chapter 8 Optimization for Training Deep Models — Goodfellow, Bengio, Courville, 2016. Section 8.3.2 explains momentum as an exponentially decaying moving average of past gradients and shows that with a constant gradient the maximum speed is 1/(1 − α) times. The paragraph below Eq. (3), the common values in the section 'How to choose β and the learning rate', and the section 'Common misconceptions about momentum' rely on it.
- torch.optim.SGD — PyTorch official documentation, checked against version 2.15. Argument defaults (
momentum=0,dampening=0,nesterov=False), the update pseudocode, and a note that it is implemented in a form different from Sutskever et al. The mapping table in the section 'Where momentum shows up in AI' relies on it.
Video notes
- 00:00 Opening question — “What's the difference between SGD and momentum?”
- 00:40 Write Eq. (1) and Eq. (2) side by side on the board — the only change is , the momentum of a heavy ball
- 01:50 Eq. (3) — the velocity is a sum of past gradients, times the gradient if it is constant
- 03:00 Figure 1 — two steps by hand,
- 04:30 Figure 2 — where turns back, the low point near , the asymptotic factor
- 05:40 PyTorch
momentum=0.9and the learning rate · wrap-up
This is the filming plan. After upload, the timestamps will be replaced with the real ones.