OptimizationAdam#003

Adam

Adam is a deep-learning optimizer that divides the average gradient by its past size per coordinate. At lr 0.05 its first step moves x and y by 0.05; gradient descent moves 0.22 and 0.75. How it differs from momentum and RMSProp.

Key symbol
v
Reading time
11 min
Prerequisites
MomentumExponential Moving AverageMini-batch SGD

Translated by an agent and not reviewed yet. Original: 한국어

How Adam differs — the average gradient is divided, coordinate by coordinate, by the size of past gradients

Adam (adaptive moment estimation) is an optimizer that steps by the average of the gradients divided, coordinate by coordinate, by the size of past gradients. It is the 'Adam optimizer' used to train neural networks. It combines the average of momentum with the per-coordinate division of RMSProp (see the question below).

In formulas, it uses the average of the gradients mm in place of the velocity uu of momentum (Eq. (1)), and divides it, coordinate by coordinate, by the square root of vv, the average of past squared gradients. A correction that fixes the bias of the first few steps is added too.

ut=β ut−1+∇L(θt),θt+1=θt−η ut(1)u_t = \beta \, u_{t-1} + \nabla L(\theta_t), \qquad \theta_{t+1} = \theta_t - \eta \, u_t \tag{1}

Eq. (1) is Eq. (2) of the momentum page.11 That page writes the velocity as vv. On this page vv is Adam's second moment, so the velocity is written uu. Adam replaces it with four lines.

An exponential moving average is an average that keeps mixing the average so far, weighted by β\beta, with the new value, weighted by 1−β1 - \beta (Eqs. (2) and (3)). A moment is the statistical name for the average of a value (first) and the average of its square (second); it is a different word from momentum.

mt=β1 mt−1+(1−β1) gt(2)m_t = \beta_1 \, m_{t-1} + (1 - \beta_1) \, g_t \tag{2} vt=β2 vt−1+(1−β2) gt2(3)v_t = \beta_2 \, v_{t-1} + (1 - \beta_2) \, g_t^2 \tag{3} m^t=mt1−β1 t+1,v^t=vt1−β2 t+1(4)\hat m_t = \frac{m_t}{1 - \beta_1^{\,t+1}}, \qquad \hat v_t = \frac{v_t}{1 - \beta_2^{\,t+1}} \tag{4} θt+1=θt−η m^tv^t+ϵ(5)\theta_{t+1} = \theta_t - \eta \, \frac{\hat m_t}{\sqrt{\hat v_t} + \epsilon} \tag{5}
Definition
SymbolNameMeaning
gtg_tgradientThe ∇L(θt)\nabla L(\theta_t) of the current step. gt2g_t^2 and the square root and division in Eq. (5) are taken component by component
utu_tvelocityThe velocity of Eq. (1). The sum of past gradients, each shrunk by a factor of β\beta (the vtv_t of the momentum page; different from the utu_t in that page's derivation)
mtm_tfirst momentThe exponential moving average of the gradients. m−1=0m_{-1} = 0. For the same sequence of gradients with β1=β\beta_1 = \beta, mt=(1−β1) utm_t = (1 - \beta_1)\,u_t
vtv_tsecond momentThe exponential moving average of the squared gradients. The protagonist of this page. v−1=0v_{-1} = 0
β1,β2\beta_1, \beta_2exponential decay ratesHow much of the previous value mm and vv keep. 0≤β1,β2<10 \le \beta_1, \beta_2 < 1
m^t,v^t\hat m_t, \hat v_tbias-corrected momentsThe averages, which start at 0 and are biased low, after correction. The t+1t+1 in the exponent is the number of gradients that have gone into the average
ϵ\epsilonepsilonA small number added to the denominator so as not to divide by 0
ηx,ηy\eta_x, \eta_yeffective learning rateEq. (6). The factor that multiplies m^\hat m for each coordinate

The other symbols (θ\theta, η\eta, ∇L\nabla L, β\beta, LL, tt, xx, yy, kk) follow the symbol tables of momentum and gradient descent. Unless stated otherwise, Adam on this page uses the PyTorch defaults β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, ϵ=10−8\epsilon = 10^{-8}.

Why bias correction — without it the first step is 3.16 times as long

An exponential moving average gives the gradient from ss steps ago a weight of (1−β)βs(1 - \beta)\beta^s. Starting from 0, the t+1t + 1 weights so far add up to only 1−βt+11 - \beta^{t+1}, so the average is biased low. Eq. (4) divides by that sum so that the weights add up to 1.

At the first step m0=0.1 g0m_0 = 0.1\,g_0 and v0=0.001 g02v_0 = 0.001\,g_0^2, so without correction each coordinate's step is 0.1/0.001≈3.160.1/\sqrt{0.001} \approx 3.16 times η\eta. With correction, m^0=g0\hat m_0 = g_0 and v^0=g02\hat v_0 = g_0^2. 1−β2 t+11 - \beta_2^{\,t+1} approaches 1 slowly (0.0198 at the 20th step), so even at the 20th step, without correction, the step from the same mm, vv is (1−0.920)/1−0.99920≈6.2(1 - 0.9^{20})/\sqrt{1 - 0.999^{20}} \approx 6.2 times as long.

Even with correction, a step can exceed η\eta. If a coordinate's gradient arrives for the first time after 1000 steps in a row with gradient 0, then mt=0.1 gtm_t = 0.1\,g_t and vt=0.001 gt2v_t = 0.001\,g_t^2, so that step is (0.1/0.001)⋅1−0.9991001/(1−0.91001)≈2.52(0.1/\sqrt{0.001}) \cdot \sqrt{1 - 0.999^{1001}}/(1 - 0.9^{1001}) \approx 2.52 times η\eta, and after 5000 steps about 3.15η3.15\eta (the sparse case that Kingma and Ba mention in Section 2.1).

How is Adam different from momentum?

Momentum steps by a velocity built up from past gradients; Adam steps by their average m^\hat m divided, coordinate by coordinate, by the size of past gradients v^\sqrt{\hat v} (Eq. (5)). So with a steady gradient, the step of momentum (β=0.9\beta = 0.9) grows up to 10 times η×\eta \times gradient, while Adam's step is η\eta in each coordinate whatever the size of the gradient. On this bowl with η=0.05\eta = 0.05, the first step moves 0.22 in xx and 0.75 in yy for momentum, and 0.05 in both for Adam.

How is Adam different from RMSProp?

RMSProp steps by the gradient divided, coordinate by coordinate, by the square root of the exponential moving average of past squared gradients (the vv of Eq. (3)). Adam adds two things to that. It uses the average mm in place of the gradient (Eq. (2)), and it corrects the bias of both averages (Eq. (4)). RMSProp has no correction, so with the PyTorch default alpha=0.99 its first step is 1/1−0.99=101/\sqrt{1 - 0.99} = 10 times lr.

What changes when each coordinate gets its own learning rate

The size of one step in each coordinate is held to roughly the learning rate or less, whatever the scale of the gradient. Written out one coordinate at a time, Eq. (5) is momentum with a different learning rate attached to each coordinate. Methods that fit the learning rate to each coordinate like this are called adaptive learning rate methods.

xt+1=xt−ηx m^x,ηx=ηv^x+ϵ(6)x_{t+1} = x_t - \eta_x \, \hat m_x, \qquad \eta_x = \frac{\eta}{\sqrt{\hat v_x} + \epsilon} \tag{6}

m^x\hat m_x and v^x\hat v_x are the xx components of m^t\hat m_t and v^t\hat v_t (the subscript tt is dropped), and yy has the same form.

At the start (−4.4, 1.5)(-4.4,\ 1.5) of the bowl L=12(x2+10y2)L = \tfrac{1}{2}(x^2 + 10y^2), g0=(−4.4, 15)g_0 = (-4.4,\ 15). In Eq. (3) v0=0.001 g02v_0 = 0.001\,g_0^2, and Eq. (4) divides it by 1−0.999=0.0011 - 0.999 = 0.001, so the corrected v^0=g02=(19.36, 225)\hat v_0 = g_0^2 = (19.36,\ 225). With η=0.05\eta = 0.05, ηx=η/4.4≈0.0114\eta_x = \eta/4.4 \approx 0.0114 and ηy=η/15≈0.0033\eta_y = \eta/15 \approx 0.0033. In the same way m^0=g0\hat m_0 = g_0, so both coordinates move by exactly 0.05, landing on θ1=(−4.35, 1.45)\theta_1 = (-4.35,\ 1.45).

Gradient descent with the same η\eta moves 0.22 in xx and 0.75 in yy. In Adam the two coordinates move by almost the same amount throughout the 20 steps, so in the left panel of Figure 1 only Adam's path is almost a straight line.

Paths from the same start (-4.4, 1.5) over the contours of L. Left: 20 steps of gradient descent (β = 0), momentum (β = 0.9), and Adam at the same learning rate η = 0.05; only Adam's is almost a straight line. Right: 120 steps of gradient descent with η = 0.18 and Adam with η = 0.16, each learning rate chosen separately.
Figure 1 Paths from the same start (−4.4, 1.5)(-4.4,\ 1.5) over the contours of LL. Left: 20 steps of gradient descent (β=0\beta = 0), momentum (β=0.9\beta = 0.9), and Adam at the same learning rate η=0.05\eta = 0.05; only Adam's is almost a straight line. Right: 120 steps of gradient descent with η=0.18\eta = 0.18 and Adam with η=0.16\eta = 0.16, each learning rate chosen separately. figures/adam_paths.py

The learning rate of yy is smaller than that of xx because the gradient of yy is larger. Their ratio ηx/ηy\eta_x/\eta_y is 3.41 at t=0t = 0 and 1.94 at t=119t = 119, different from the curvature ratio 10 (Figure 2, right). That is because what Adam divides by is not the curvature but the size of the gradients seen so far, which on this bowl is curvature × distance to the bottom. It means Adam does not simply erase the condition number (the curvature ratio) of 10 that made gradient descent slow.

Key idea

Adam divides the gradient in each coordinate by the size of past gradients v^\sqrt{\hat v}, which holds each coordinate's step to roughly the learning rate η\eta or less, whatever the scale of the gradient.

Adam's learning rate is not a multiplier on the gradient but a rough cap on each coordinate's step

The η\eta of gradient descent is a factor that multiplies the gradient, while the η\eta of Adam is a rough cap on the size of one step in each coordinate. So even at the same η=0.05\eta = 0.05, the losses after 20 steps differ: 7.40 for Adam, 1.24 for gradient descent, and 1.18 for momentum (β=0.9\beta = 0.9) (Figure 2, left).

On this bowl, during the first 600 steps taken with η=0.05\eta = 0.05 and 0.16, Adam's step does not exceed η\eta in any coordinate.22 The largest is the η\eta of the first step, and near the bottom it is much smaller: at step t=50t = 50 with η=0.05\eta = 0.05, yy moves only 0.044η0.044\eta. Measured as a vector, the length of the first step is 2 η\sqrt{2}\,\eta. Walk much longer and steps larger than η\eta do appear. With η=0.16\eta = 0.16, at step t=6898t = 6898 yy moves 1.46η1.46\eta (see the section 'What to watch out for when using Adam').

The size of the loss does not change the step either. Multiply the loss by 10 and the yy curvature becomes 100, so gradient descent with η=0.05\eta = 0.05 has ηk=5\eta k = 5, past the limit 2, and diverges; but the path of Adam (η=0.05\eta = 0.05) stays the same to within 10−810^{-8} for all 120 steps.

So Adam's learning rate, too, is chosen for each problem. On this bowl, the Adam learning rate that is fastest to bring the loss below 10−410^{-4} and keep it there through step 600 is 0.16, which takes 104 steps.33 On this page ‘keep below’ means the loss does not rise back above 10−410^{-4} within 600 steps. Learning rates were measured from 0.001 to 2 at intervals of 0.001. Gradient descent with η=0.18\eta = 0.18 takes 30 steps (Figure 2, left).

Left: the loss L against the step t (log scale); the horizontal line is 10^-4, and each dot marks the step from which the loss stays below it (looking up to step 600). Right: the effective learning rates η_x, η_y (Eq. (6)) of Adam with η = 0.05; both are below 0.05, and the ratio η_x/η_y is 3.41 at t = 0 and 1.94 at t = 119.
Figure 2 Left: the loss LL against the step tt (log scale); the horizontal line is 10−410^{-4}, and each dot marks the step from which the loss stays below it (looking up to step 600). Right: the effective learning rates ηx\eta_x, ηy\eta_y (Eq. (6)) of Adam with η=0.05\eta = 0.05; both are below 0.05, and the ratio ηx/ηy\eta_x/\eta_y is 3.41 at t=0t = 0 and 1.94 at t=119t = 119. figures/adam_loss.py

Why is Adam's default lr 1e-3?

Adam's lr (learning rate) is not a factor that multiplies the gradient but close to a cap on each coordinate's step size, so a single default can be set regardless of the size of the loss. 1e-3 (0.001) is the value Kingma and Ba report as good on the problems they tested. It means 'move each parameter by at most about 0.001 per step', which is small for this bowl, where the path has to travel 4.4 to reach the bottom. On this bowl with lr = 1e-3, after 20 steps you are at (−4.380, 1.480)(-4.380,\ 1.480), and it takes 7616 steps for the loss to fall below 10−410^{-4}.

What to watch out for when using Adam

Lowering β2\beta_2 can make the path wobble near the bottom (the question below), and there are problems on which even Adam does not converge.

Why are Adam's β₁ and β₂ 0.9 and 0.999?

They are the defaults Kingma and Ba report as good on the problems they tested, and what they mean is the length over which the average is taken. The most recent 1/(1−β)1/(1 - \beta) steps carry 63–65% of the weight (1−0.910=65%1 - 0.9^{10} = 65\%, 1−0.9991000=63%1 - 0.999^{1000} = 63\%), so mm averages about 10 steps and vv about 1000 steps. When vv remembers past large gradients for a long time, the steps also get smaller as the gradients get smaller. On this bowl with η=0.05\eta = 0.05, looking at steps 500 to 2000, β2=0.999\beta_2 = 0.999 keeps the loss below 10−1810^{-18} throughout, while β2=0.9\beta_2 = 0.9 goes up and down between 10−810^{-8} and 2.5×10−52.5 \times 10^{-5}.

Even the default β2=0.999\beta_2 = 0.999 can bounce back if you keep walking much longer at a fixed learning rate. On this bowl the loss rises back above 10−410^{-4} after about 6900 steps with η=0.16\eta = 0.16 and after about 10260 steps with η=0.05\eta = 0.05, reaching 0.10 and 0.012 respectively (checked for these two learning rates, up to 20000 steps).

Reddi et al. give a simple convex problem on which Adam does not converge, and proposed AMSGrad, which uses the maximum of vv so far instead of vv in the denominator of Eq. (5) (PyTorch amsgrad=True).

How is Adam different from AdamW?

Adam and AdamW differ in where the weight decay (weight_decay) is applied. Adam adds weight_decay × θ\theta to the gradient, so this term is also divided by v^\sqrt{\hat v}. AdamW (torch.optim.AdamW) keeps this term out of the gradient and multiplies θ\theta directly by 1 − η\eta × weight_decay. The PyTorch defaults differ too: weight_decay=0 for Adam and 1e-2 for AdamW.

Common misconceptions about Adam

  • Adam's lr means the same as SGD's lr, so you can carry over the value you were using — The step of SGD is lr × gradient, while Adam's step is roughly lr or less in each coordinate, so on this bowl with η=0.05\eta = 0.05, the losses after 20 steps are 1.24 versus 7.40.
  • Adam always converges faster than SGD or momentum — On this bowl, whatever learning rate from 0.001 to 2 you choose, Adam takes 104 steps or more to keep the loss below 10−410^{-4}, while gradient descent with η=0.18\eta = 0.18 takes 30.
  • ε only prevents division by 0 and does not affect the result — When it is large enough to compare with v^\sqrt{\hat v}, it cuts the learning rate. On this bowl with η=0.05\eta = 0.05, ϵ=1\epsilon = 1 shrinks the first step's ηx\eta_x from η/4.4\eta/4.4 to η/5.4\eta/5.4, and the 205 steps it takes to keep the loss below 10−410^{-4} become 315.

Where Adam shows up in AI

In the training loop (zero_grad() → compute the loss → backpropagation backward() → step()), Adam, too, changes only the last step(). step() keeps two buffers, mm and vv, for each parameter and computes Eqs. (2)–(5), so the optimizer state is twice the number of parameters.

This pagePyTorch torch.optim.Adam
η\etalr (default 1e-3). The docs write it as γ\gamma
β1,β2\beta_1, \beta_2betas=(0.9, 0.999) (default)
ϵ\epsiloneps=1e-8 (default). Added outside the bias-corrected v^t\sqrt{\hat v_t} (the same as Eq. (5))
mt,vtm_t, v_toptimizer.state[p]["exp_avg"], ["exp_avg_sq"]
t+1t + 1 in Eq. (4)optimizer.state[p]["step"] (1 at the first step(); the docs write the exponent as tt)
Eq. (5)optimizer.step() when the other arguments are at their defaults
weight_decay=0The default. If greater than 0, weight_decay × θt\theta_t is added to gtg_t and goes into Eqs. (2) and (3). How AdamW (planned) differs is in the question in the section 'What to watch out for when using Adam'
amsgrad=FalseThe default. With True, v^t\hat v_t is made from the maximum of vtv_t so far (the AMSGrad of Reddi et al.)

Check it in code

Compared with the code on the momentum page, there are now two buffers, mm and vv. The xx in the output moves by 0.24 to 0.25 every five steps, that is, by close to η\eta per step.

Math
mt=β1 mt−1+(1−β1) gt(2)m_t = \sym{beta1}{\beta_1} \, m_{t-1} + (1 - \sym{beta1}{\beta_1}) \, g_t \tag{2}vt=β2 vt−1+(1−β2) gt2(3)v_t = \sym{beta2}{\beta_2} \, v_{t-1} + (1 - \sym{beta2}{\beta_2}) \, g_t^2 \tag{3}θt+1=θt−η m^tv^t+ϵ(5)\sym{theta}{\theta}_{t+1} = \sym{theta}{\theta}_t - \sym{eta}{\eta} \, \frac{\sym{m_hat}{\hat m}_t}{\sqrt{\sym{v_hat}{\hat v}_t} + \sym{eps}{\epsilon}} \tag{5}
Code figures/adam.py
import numpy as np

def L(p):
    x, y = p
    return 0.5 * (x**2 + 10 * y**2)

def grad(p):
    x, y = p
    return np.array([x, 10 * y])    # ∇L = (x, 10y)

theta = np.array([-4.4, 1.5])       # θ₀
eta, beta1, beta2, eps = 0.05, 0.9, 0.999, 1e-8   # η, β₁, β₂, ε
m, v = np.zeros(2), np.zeros(2)     # m₋₁ = v₋₁ = 0
for t in range(21):
    if t % 5 == 0: print(f"t={t:2d}  L={L(theta):.3f}  x={theta[0]:.3f}  y={theta[1]:.3f}")
    g = grad(theta)
    m = beta1 * m + (1 - beta1) * g                          # 식 (2)
    v = beta2 * v + (1 - beta2) * g**2                       # 식 (3)
    m_hat, v_hat = m / (1 - beta1**(t + 1)), v / (1 - beta2**(t + 1))   # 식 (4)
    theta = theta - eta * m_hat / (np.sqrt(v_hat) + eps)     # 식 (5)
$ python adam.py
t= 0  L=20.930  x=-4.400  y=1.500
t= 5  L=16.436  x=-4.150  y=1.251
t=10  L=12.684  x=-3.902  y=1.007
t=15  L=9.685  x=-3.657  y=0.774
t=20  L=7.399  x=-3.415  y=0.560

Try it yourself

  1. Follow the second step by hand with η=0.05\eta = 0.05. From θ1=(−4.35, 1.45)\theta_1 = (-4.35,\ 1.45), what are m^1\hat m_1, v^1\sqrt{\hat v_1}, and θ2\theta_2, and what percentage of η\eta is the second step?
Answer

g1=(−4.35, 14.5)g_1 = (-4.35,\ 14.5). Since m0=0.1 g0=(−0.44, 1.5)m_0 = 0.1\,g_0 = (-0.44,\ 1.5), m1=0.9 m0+0.1 g1=(−0.831, 2.8)m_1 = 0.9\,m_0 + 0.1\,g_1 = (-0.831,\ 2.8), and dividing by 1−0.92=0.191 - 0.9^2 = 0.19 gives m^1≈(−4.3737, 14.7368)\hat m_1 \approx (-4.3737,\ 14.7368). Since v0=0.001 g02=(0.01936, 0.225)v_0 = 0.001\,g_0^2 = (0.01936,\ 0.225), v1=0.999 v0+0.001 g12≈(0.038263, 0.435025)v_1 = 0.999\,v_0 + 0.001\,g_1^2 \approx (0.038263,\ 0.435025), and dividing by 1−0.9992=0.0019991 - 0.999^2 = 0.001999 and taking the square root gives v^1≈(4.3751, 14.7520)\sqrt{\hat v_1} \approx (4.3751,\ 14.7520). The ratios ∣m^1∣/v^1\lvert \hat m_1 \rvert / \sqrt{\hat v_1} of the two components are 0.9997 and 0.9990, so θ2≈(−4.300, 1.400)\theta_2 \approx (-4.300,\ 1.400), and the second step is also 99.97% and 99.90% of η\eta. The gradient has barely changed, so the average m^\hat m and the size v^\sqrt{\hat v} are almost the same.

Further reading

  • Adam: A Method for Stochastic Optimization — Kingma, Ba, 2015. ICLR. Algorithm 1 is Eqs. (2)–(5) (it writes the learning rate as α\alpha and counts steps from 1), Section 3 explains the bias correction, and Section 2.1 explains that the step size is roughly bounded by the learning rate and does not depend on the scale of the gradient. Section 5 describes the relation to RMSProp and AdaGrad. The defaults α=0.001\alpha = 0.001, β1=0.9\beta_1 = 0.9, β2=0.999\beta_2 = 0.999, ϵ=10−8\epsilon = 10^{-8} also come from here. The section 'How Adam differs' and its RMSProp question, the section 'Adam's learning rate is not a multiplier on the gradient but a rough cap on each coordinate's step', the folded explanation of bias correction, and the two question blocks on the default lr and β rely on it.
  • On the Convergence of Adam and Beyond — Reddi, Kale, Kumar, 2018. ICLR. It gives a simple convex problem on which Adam does not converge, and proposes AMSGrad, which uses the maximum of past vv in the denominator. The section 'What to watch out for when using Adam', the amsgrad row of the mapping table, and the section 'Common misconceptions about Adam' rely on it.
  • Deep Learning, Chapter 8 Optimization for Training Deep Models — Goodfellow, Bengio, Courville, 2016. Section 8.5 explains in turn AdaGrad, RMSProp, and Adam, which change the learning rate for each parameter, and Section 8.5.3 is Adam (it writes the learning rate as ε\varepsilon, this page's ϵ\epsilon as δ\delta, mm and vv as ss and rr, and β1\beta_1 and β2\beta_2 as ρ1\rho_1 and ρ2\rho_2). The formulas in the section 'How Adam differs' rely on it.
  • torch.optim.Adam — PyTorch official documentation, checked against version 2.15. Argument defaults (lr=1e-3, betas=(0.9, 0.999), eps=1e-8, weight_decay=0, amsgrad=False, decoupled_weight_decay=False) and the update pseudocode. It notes that decoupled_weight_decay=True is the same as AdamW. It describes eps as a term added to the denominator for numerical stability, which, read only that way, easily leads to the third common misconception. The mapping table in the section 'Where Adam shows up in AI' relies on it.
  • torch.optim.RMSprop — PyTorch official documentation, checked against version 2.15. The default alpha=0.99 and the update pseudocode, which has no bias correction. The question 'How is Adam different from RMSProp?' relies on it.
  • torch.optim.AdamW — PyTorch official documentation, checked against version 2.15. The default weight_decay=1e-2 and the pseudocode that subtracts the decay from θ\theta directly instead of adding it to the gradient. The question 'How is Adam different from AdamW?' relies on it.

Video notes

Video notes
  • 00:00 Opening question — “How is Adam different from momentum?”
  • 00:40 Write Eq. (1) and Eqs. (2)–(5) side by side on the board — the average mm, the size vv, the correction
  • 02:00 The first step by hand — v^0=(19.36, 225)\hat v_0 = (19.36,\ 225), 0.05 in both coordinates
  • 03:10 Figure 1, left — Adam walking in an almost straight line; ηx\eta_x and ηy\eta_y in the right panel of Figure 2
  • 04:20 Figure 2, left — slower at the same η\eta; lr is a rough cap on each coordinate's step
  • 05:30 PyTorch Adam(lr=1e-3, betas=(0.9, 0.999)) · wrap-up

This is the filming plan. After upload, the timestamps will be replaced with the real ones.

Where this sits on the map

Gradient DescentOptimizationSeriesCalculusSigma NotationCalculusBackpropagationAIMomentumMethod · OptimizationExponential Moving AveragePlannedMini-batch SGDOptimizationAdamYou are hereAdamWMethod · Planned
Prerequisites
This page
Next
DonePlannedMethodThe path to this pageSee the full map →