How Adam differs — the average gradient is divided, coordinate by coordinate, by the size of past gradients
Adam (adaptive moment estimation) is an optimizer that steps by the average of the gradients divided, coordinate by coordinate, by the size of past gradients. It is the 'Adam optimizer' used to train neural networks. It combines the average of momentum with the per-coordinate division of RMSProp (see the question below).
In formulas, it uses the average of the gradients in place of the velocity of momentum (Eq. (1)), and divides it, coordinate by coordinate, by the square root of , the average of past squared gradients. A correction that fixes the bias of the first few steps is added too.
Eq. (1) is Eq. (2) of the momentum page.11 That page writes the velocity as . On this page is Adam's second moment, so the velocity is written . Adam replaces it with four lines.
An exponential moving average is an average that keeps mixing the average so far, weighted by , with the new value, weighted by (Eqs. (2) and (3)). A moment is the statistical name for the average of a value (first) and the average of its square (second); it is a different word from momentum.
| Symbol | Name | Meaning |
|---|---|---|
| gradient | The of the current step. and the square root and division in Eq. (5) are taken component by component | |
| velocity | The velocity of Eq. (1). The sum of past gradients, each shrunk by a factor of (the of the momentum page; different from the in that page's derivation) | |
| first moment | The exponential moving average of the gradients. . For the same sequence of gradients with , | |
| second moment | The exponential moving average of the squared gradients. The protagonist of this page. | |
| exponential decay rates | How much of the previous value and keep. | |
| bias-corrected moments | The averages, which start at 0 and are biased low, after correction. The in the exponent is the number of gradients that have gone into the average | |
| epsilon | A small number added to the denominator so as not to divide by 0 | |
| effective learning rate | Eq. (6). The factor that multiplies for each coordinate |
The other symbols (, , , , , , , , ) follow the symbol tables of momentum and gradient descent. Unless stated otherwise, Adam on this page uses the PyTorch defaults , , .
Why bias correction — without it the first step is 3.16 times as long
An exponential moving average gives the gradient from steps ago a weight of . Starting from 0, the weights so far add up to only , so the average is biased low. Eq. (4) divides by that sum so that the weights add up to 1.
At the first step and , so without correction each coordinate's step is times . With correction, and . approaches 1 slowly (0.0198 at the 20th step), so even at the 20th step, without correction, the step from the same , is times as long.
Even with correction, a step can exceed . If a coordinate's gradient arrives for the first time after 1000 steps in a row with gradient 0, then and , so that step is times , and after 5000 steps about (the sparse case that Kingma and Ba mention in Section 2.1).
How is Adam different from momentum?
Momentum steps by a velocity built up from past gradients; Adam steps by their average divided, coordinate by coordinate, by the size of past gradients (Eq. (5)). So with a steady gradient, the step of momentum () grows up to 10 times gradient, while Adam's step is in each coordinate whatever the size of the gradient. On this bowl with , the first step moves 0.22 in and 0.75 in for momentum, and 0.05 in both for Adam.
How is Adam different from RMSProp?
RMSProp steps by the gradient divided, coordinate by coordinate, by the square root of the exponential moving average of past squared gradients (the of Eq. (3)). Adam adds two things to that. It uses the average in place of the gradient (Eq. (2)), and it corrects the bias of both averages (Eq. (4)). RMSProp has no correction, so with the PyTorch default alpha=0.99 its first step is times lr.
What changes when each coordinate gets its own learning rate
The size of one step in each coordinate is held to roughly the learning rate or less, whatever the scale of the gradient. Written out one coordinate at a time, Eq. (5) is momentum with a different learning rate attached to each coordinate. Methods that fit the learning rate to each coordinate like this are called adaptive learning rate methods.
and are the components of and (the subscript is dropped), and has the same form.
At the start of the bowl , . In Eq. (3) , and Eq. (4) divides it by , so the corrected . With , and . In the same way , so both coordinates move by exactly 0.05, landing on .
Gradient descent with the same moves 0.22 in and 0.75 in . In Adam the two coordinates move by almost the same amount throughout the 20 steps, so in the left panel of Figure 1 only Adam's path is almost a straight line.
figures/adam_paths.pyThe learning rate of is smaller than that of because the gradient of is larger. Their ratio is 3.41 at and 1.94 at , different from the curvature ratio 10 (Figure 2, right). That is because what Adam divides by is not the curvature but the size of the gradients seen so far, which on this bowl is curvature × distance to the bottom. It means Adam does not simply erase the condition number (the curvature ratio) of 10 that made gradient descent slow.
Adam divides the gradient in each coordinate by the size of past gradients , which holds each coordinate's step to roughly the learning rate or less, whatever the scale of the gradient.
Adam's learning rate is not a multiplier on the gradient but a rough cap on each coordinate's step
The of gradient descent is a factor that multiplies the gradient, while the of Adam is a rough cap on the size of one step in each coordinate. So even at the same , the losses after 20 steps differ: 7.40 for Adam, 1.24 for gradient descent, and 1.18 for momentum () (Figure 2, left).
On this bowl, during the first 600 steps taken with and 0.16, Adam's step does not exceed in any coordinate.22 The largest is the of the first step, and near the bottom it is much smaller: at step with , moves only . Measured as a vector, the length of the first step is . Walk much longer and steps larger than do appear. With , at step moves (see the section 'What to watch out for when using Adam').
The size of the loss does not change the step either. Multiply the loss by 10 and the curvature becomes 100, so gradient descent with has , past the limit 2, and diverges; but the path of Adam () stays the same to within for all 120 steps.
So Adam's learning rate, too, is chosen for each problem. On this bowl, the Adam learning rate that is fastest to bring the loss below and keep it there through step 600 is 0.16, which takes 104 steps.33 On this page ‘keep below’ means the loss does not rise back above within 600 steps. Learning rates were measured from 0.001 to 2 at intervals of 0.001. Gradient descent with takes 30 steps (Figure 2, left).
figures/adam_loss.pyWhy is Adam's default lr 1e-3?
Adam's lr (learning rate) is not a factor that multiplies the gradient but close to a cap on each coordinate's step size, so a single default can be set regardless of the size of the loss. 1e-3 (0.001) is the value Kingma and Ba report as good on the problems they tested. It means 'move each parameter by at most about 0.001 per step', which is small for this bowl, where the path has to travel 4.4 to reach the bottom. On this bowl with lr = 1e-3, after 20 steps you are at , and it takes 7616 steps for the loss to fall below .
What to watch out for when using Adam
Lowering can make the path wobble near the bottom (the question below), and there are problems on which even Adam does not converge.
Why are Adam's β₁ and β₂ 0.9 and 0.999?
They are the defaults Kingma and Ba report as good on the problems they tested, and what they mean is the length over which the average is taken. The most recent steps carry 63–65% of the weight (, ), so averages about 10 steps and about 1000 steps. When remembers past large gradients for a long time, the steps also get smaller as the gradients get smaller. On this bowl with , looking at steps 500 to 2000, keeps the loss below throughout, while goes up and down between and .
Even the default can bounce back if you keep walking much longer at a fixed learning rate. On this bowl the loss rises back above after about 6900 steps with and after about 10260 steps with , reaching 0.10 and 0.012 respectively (checked for these two learning rates, up to 20000 steps).
Reddi et al. give a simple convex problem on which Adam does not converge, and proposed AMSGrad, which uses the maximum of so far instead of in the denominator of Eq. (5) (PyTorch amsgrad=True).
How is Adam different from AdamW?
Adam and AdamW differ in where the weight decay (weight_decay) is applied. Adam adds weight_decay × to the gradient, so this term is also divided by . AdamW (torch.optim.AdamW) keeps this term out of the gradient and multiplies directly by 1 − × weight_decay. The PyTorch defaults differ too: weight_decay=0 for Adam and 1e-2 for AdamW.
Common misconceptions about Adam
- Adam's lr means the same as SGD's lr, so you can carry over the value you were using — The step of SGD is lr × gradient, while Adam's step is roughly lr or less in each coordinate, so on this bowl with , the losses after 20 steps are 1.24 versus 7.40.
- Adam always converges faster than SGD or momentum — On this bowl, whatever learning rate from 0.001 to 2 you choose, Adam takes 104 steps or more to keep the loss below , while gradient descent with takes 30.
- ε only prevents division by 0 and does not affect the result — When it is large enough to compare with , it cuts the learning rate. On this bowl with , shrinks the first step's from to , and the 205 steps it takes to keep the loss below become 315.
Where Adam shows up in AI
In the training loop (zero_grad() → compute the loss → backpropagation backward() → step()), Adam, too, changes only the last step(). step() keeps two buffers, and , for each parameter and computes Eqs. (2)–(5), so the optimizer state is twice the number of parameters.
| This page | PyTorch torch.optim.Adam |
|---|---|
lr (default 1e-3). The docs write it as | |
betas=(0.9, 0.999) (default) | |
eps=1e-8 (default). Added outside the bias-corrected (the same as Eq. (5)) | |
optimizer.state[p]["exp_avg"], ["exp_avg_sq"] | |
| in Eq. (4) | optimizer.state[p]["step"] (1 at the first step(); the docs write the exponent as ) |
| Eq. (5) | optimizer.step() when the other arguments are at their defaults |
weight_decay=0 | The default. If greater than 0, weight_decay × is added to and goes into Eqs. (2) and (3). How AdamW (planned) differs is in the question in the section 'What to watch out for when using Adam' |
amsgrad=False | The default. With True, is made from the maximum of so far (the AMSGrad of Reddi et al.) |
Check it in code
Compared with the code on the momentum page, there are now two buffers, and . The in the output moves by 0.24 to 0.25 every five steps, that is, by close to per step.
import numpy as np
def L(p):
x, y = p
return 0.5 * (x**2 + 10 * y**2)
def grad(p):
x, y = p
return np.array([x, 10 * y]) # ∇L = (x, 10y)
theta = np.array([-4.4, 1.5]) # θ₀
eta, beta1, beta2, eps = 0.05, 0.9, 0.999, 1e-8 # η, β₁, β₂, ε
m, v = np.zeros(2), np.zeros(2) # m₋₁ = v₋₁ = 0
for t in range(21):
if t % 5 == 0: print(f"t={t:2d} L={L(theta):.3f} x={theta[0]:.3f} y={theta[1]:.3f}")
g = grad(theta)
m = beta1 * m + (1 - beta1) * g # 식 (2)
v = beta2 * v + (1 - beta2) * g**2 # 식 (3)
m_hat, v_hat = m / (1 - beta1**(t + 1)), v / (1 - beta2**(t + 1)) # 식 (4)
theta = theta - eta * m_hat / (np.sqrt(v_hat) + eps) # 식 (5)$ python adam.py
t= 0 L=20.930 x=-4.400 y=1.500
t= 5 L=16.436 x=-4.150 y=1.251
t=10 L=12.684 x=-3.902 y=1.007
t=15 L=9.685 x=-3.657 y=0.774
t=20 L=7.399 x=-3.415 y=0.560Try it yourself
- Follow the second step by hand with . From , what are , , and , and what percentage of is the second step?
Answer
. Since , , and dividing by gives . Since , , and dividing by and taking the square root gives . The ratios of the two components are 0.9997 and 0.9990, so , and the second step is also 99.97% and 99.90% of . The gradient has barely changed, so the average and the size are almost the same.
Further reading
- Adam: A Method for Stochastic Optimization — Kingma, Ba, 2015. ICLR. Algorithm 1 is Eqs. (2)–(5) (it writes the learning rate as and counts steps from 1), Section 3 explains the bias correction, and Section 2.1 explains that the step size is roughly bounded by the learning rate and does not depend on the scale of the gradient. Section 5 describes the relation to RMSProp and AdaGrad. The defaults , , , also come from here. The section 'How Adam differs' and its RMSProp question, the section 'Adam's learning rate is not a multiplier on the gradient but a rough cap on each coordinate's step', the folded explanation of bias correction, and the two question blocks on the default lr and β rely on it.
- On the Convergence of Adam and Beyond — Reddi, Kale, Kumar, 2018. ICLR. It gives a simple convex problem on which Adam does not converge, and proposes AMSGrad, which uses the maximum of past in the denominator. The section 'What to watch out for when using Adam', the
amsgradrow of the mapping table, and the section 'Common misconceptions about Adam' rely on it. - Deep Learning, Chapter 8 Optimization for Training Deep Models — Goodfellow, Bengio, Courville, 2016. Section 8.5 explains in turn AdaGrad, RMSProp, and Adam, which change the learning rate for each parameter, and Section 8.5.3 is Adam (it writes the learning rate as , this page's as , and as and , and and as and ). The formulas in the section 'How Adam differs' rely on it.
- torch.optim.Adam — PyTorch official documentation, checked against version 2.15. Argument defaults (
lr=1e-3,betas=(0.9, 0.999),eps=1e-8,weight_decay=0,amsgrad=False,decoupled_weight_decay=False) and the update pseudocode. It notes thatdecoupled_weight_decay=Trueis the same as AdamW. It describesepsas a term added to the denominator for numerical stability, which, read only that way, easily leads to the third common misconception. The mapping table in the section 'Where Adam shows up in AI' relies on it. - torch.optim.RMSprop — PyTorch official documentation, checked against version 2.15. The default
alpha=0.99and the update pseudocode, which has no bias correction. The question 'How is Adam different from RMSProp?' relies on it. - torch.optim.AdamW — PyTorch official documentation, checked against version 2.15. The default
weight_decay=1e-2and the pseudocode that subtracts the decay from directly instead of adding it to the gradient. The question 'How is Adam different from AdamW?' relies on it.
Video notes
- 00:00 Opening question — “How is Adam different from momentum?”
- 00:40 Write Eq. (1) and Eqs. (2)–(5) side by side on the board — the average , the size , the correction
- 02:00 The first step by hand — , 0.05 in both coordinates
- 03:10 Figure 1, left — Adam walking in an almost straight line; and in the right panel of Figure 2
- 04:20 Figure 2, left — slower at the same ; lr is a rough cap on each coordinate's step
- 05:30 PyTorch
Adam(lr=1e-3, betas=(0.9, 0.999))· wrap-up
This is the filming plan. After upload, the timestamps will be replaced with the real ones.