OptimizationMomentum#002

Momentum

모멘텀

Momentum is an optimizer that steps by a velocity, a β-decayed sum of past gradients. At learning rate 0.05 the 20-step loss is 1.24 without it and 0.106 at β = 0.5. See why 0.9 is slow here yet common.

Key symbol
β
Reading time
12 min
Prerequisites
Gradient DescentSeriesMini-batch SGD

Translated by an agent and not reviewed yet. Original: 한국어

How momentum differs — you step by the velocity instead of the gradient

Momentum is a variant of gradient descent that steps by a velocity built up from past gradients instead of by the current gradient. It is also called the momentum optimizer or SGD with momentum, and when training a neural network in PyTorch you switch it on with the momentum argument of torch.optim.SGD. Gradient descent got slower the more the curvature differed between directions (the larger the condition number). Momentum lengthens the steps in the less curved direction and reduces that slowness.

The velocity vv takes the place of the gradient ∇L\nabla L in Eq. (1). The velocity is β\beta times the previous velocity plus the current gradient.

θt+1=θt−η ∇L(θt)(1)\theta_{t+1} = \theta_t - \eta \, \nabla L(\theta_t) \tag{1} vt=β vt−1+∇L(θt),θt+1=θt−η vt(2)v_t = \beta \, v_{t-1} + \nabla L(\theta_t), \qquad \theta_{t+1} = \theta_t - \eta \, v_t \tag{2}
Definition
SymbolNameMeaning
β\betamomentum coefficientHow much of the previous velocity to keep. 0≤β<10 \le \beta < 1. The protagonist of this page. With β=0\beta = 0, Eq. (2) becomes Eq. (1). Depending on the text it is written μ\mu (Sutskever et al., PyTorch docs) or α\alpha (Goodfellow et al.)
vtv_tvelocityThe direction you actually step in at step tt. Before the first step, v−1=0v_{-1} = 0

The other symbols (θ\theta, η\eta, ∇L\nabla L, LL, tt, xx, yy, the curvature kk) follow the symbol table on the gradient descent page.

The name comes from the momentum of a heavy ball rolling down a slope. It is the tendency to keep going the way you were going, so it is also described as inertia. Because of friction, at every step the ball loses 1−β1 - \beta of its velocity, keeps β\beta of it, and picks up the push of the slope. vv measures that velocity toward uphill (toward ∇L\nabla L), so Eq. (2) subtracts ηv\eta v from θ\theta.

Unroll Eq. (2) backward one line at a time, and the velocity is the sum of past gradients, each shrunk by a factor of β\beta for every step further back.

vt=∇L(θt)+β ∇L(θt−1)+β2 ∇L(θt−2)+⋯+βt ∇L(θ0)(3)v_t = \nabla L(\theta_t) + \beta \, \nabla L(\theta_{t-1}) + \beta^2 \, \nabla L(\theta_{t-2}) + \cdots + \beta^t \, \nabla L(\theta_0) \tag{3}

If the gradient is the same every time, Eq. (3) approaches 1+β+β2+⋯1 + \beta + \beta^2 + \cdots times that gradient, that is, 1/(1−β)1/(1-\beta) times, the sum of the geometric series. One step grows to as much as 2 times η×\eta \times gradient with β=0.5\beta = 0.5, and 10 times with β=0.9\beta = 0.9.11 This page computes with the full gradient, not a mini-batch.

Why momentum descends faster than gradient descent when β is right

Because when β\beta is right, in the less curved xx direction the velocity builds up while the sign of the gradient stays the same, and the steps get longer. On the same bowl, from the same start, with η=0.05\eta = 0.05, the loss after 20 steps is 1.24 for gradient descent (β=0\beta = 0) and 0.106 for β=0.5\beta = 0.5 (Figure 2). In the long run 'right' means 0<β<0.90250 < \beta < 0.9025, which the next section derives.

The bowl is L=12(x2+10y2)L = \tfrac{1}{2}(x^2 + 10y^2) from the gradient descent page, its gradient is ∇L=(x, 10y)\nabla L = (x,\ 10y), and the start is (−4.4, 1.5)(-4.4,\ 1.5). Figure 1 shows three paths with η\eta fixed at 0.05 and only β\beta changed.

Contours of L with paths of 20 steps from the same start (-4.4, 1.5) at the same learning rate η = 0.05, changing only β. β = 0 is the same path as gradient descent. 0.5 is ahead of gradient descent in the x direction at every step from the second step on, and overshoots the bottom back and forth in the y direction. 0.9 overshoots into x > 0 (as far as x = 2.10) and comes back.
Figure 1 Contours of LL with paths of 20 steps from the same start (−4.4, 1.5)(-4.4,\ 1.5) at the same learning rate η=0.05\eta = 0.05, changing only β\beta. β=0\beta = 0 is the same path as gradient descent. 0.50.5 is ahead of gradient descent in the xx direction at every step from the second step on, and overshoots the bottom back and forth in the yy direction. 0.90.9 overshoots into x>0x > 0 (as far as x=2.10x = 2.10) and comes back. figures/momentum_paths.py

Let's follow two steps by hand with β=0.5\beta = 0.5. In the first step v0=∇L(θ0)=(−4.4, 15)v_0 = \nabla L(\theta_0) = (-4.4,\ 15), so you land on θ1=(−4.18, 0.75)\theta_1 = (-4.18,\ 0.75), the same point as gradient descent. The gradient for the second step is ∇L(θ1)=(−4.18, 7.5)\nabla L(\theta_1) = (-4.18,\ 7.5), and the velocity is v1=0.5×(−4.4, 15)+(−4.18, 7.5)=(−6.38, 15)v_1 = 0.5 \times (-4.4,\ 15) + (-4.18,\ 7.5) = (-6.38,\ 15). So θ2=θ1−0.05 v1=(−3.861, 0)\theta_2 = \theta_1 - 0.05\, v_1 = (-3.861,\ 0).

Compared with gradient descent's θ2=(−3.971, 0.375)\theta_2 = (-3.971,\ 0.375), xx has moved forward by 0.539 instead of 0.429. With β=0.5\beta = 0.5, x<0x < 0 throughout the 20 steps, so the xx component of the gradient (xx itself) never changes sign either. So from the second to the twentieth step (v1v_1 to v19v_{19}), the xx component of the velocity is larger in absolute value than the xx component of the gradient.

In exchange, yy overshoots the bottom because of the leftover velocity. At θ2\theta_2, y=0y = 0, but the yy component of v2v_2 is 0.5×15+0=7.50.5 \times 15 + 0 = 7.5, so y3=0−0.05×7.5=−0.375y_3 = 0 - 0.05 \times 7.5 = -0.375. That is why the β=0.5\beta = 0.5 path in Figure 1 goes back and forth in yy.

What's the difference between SGD and momentum?

The difference is the quantity used for one step. SGD without momentum steps by the current gradient; momentum steps by a velocity built from past gradients, each shrunk by a factor of β\beta (Eq. (2)). On the same bowl with η=0.05\eta = 0.05, the loss after 20 steps is 1.24 for SGD and 0.106 for momentum with β=0.5\beta = 0.5. Whether the gradient comes from a mini-batch (stochastic) is the same for both; in PyTorch the only difference is the momentum argument of torch.optim.SGD.

Why too large a β slows it down again

Because the velocity lingers too long, so even after passing the bottom it keeps going the same way for a while. With η=0.05\eta = 0.05, β=0.9\beta = 0.9 goes down to a loss of 0.606 at step 6 and then climbs back to 5.34 at step 8 (Figure 2, left).

From step 11 on, β=0.9\beta = 0.9 has a larger loss than β=0.5\beta = 0.5 at every step, and its 1.18 after 20 steps is close to gradient descent's 1.24.22 That is one point on a curve that goes up and down: one step earlier, at step 19, it is 0.490, smaller than gradient descent's 1.378. So speed is compared with the asymptotic factor below. The right panel of Figure 2 measures this value for each β\beta.

At the same learning rate η = 0.05, the loss L against the step t (left), and the loss after 20 steps L_20 against β (right; both on a log scale). On the left, β = 0.5 lowers the loss at every step while 0.9 goes up and down. The dashed line on the right is the 1.24 of gradient descent (β = 0). For 0 < β ≤ 0.9 the loss is below it and for 0.91 ≤ β ≤ 0.95 above it, and the lowest point is about 0.002 at β ≈ 0.65.
Figure 2 At the same learning rate η=0.05\eta = 0.05, the loss LL against the step tt (left), and the loss after 20 steps L20L_{20} against β\beta (right; both on a log scale). On the left, β=0.5\beta = 0.5 lowers the loss at every step while 0.90.9 goes up and down. The dashed line on the right is the 1.24 of gradient descent (β=0\beta = 0). For 0<β≤0.90 < \beta \le 0.9 the loss is below it and for 0.91≤β≤0.950.91 \le \beta \le 0.95 above it, and the lowest point is about 0.002 at β≈0.65\beta \approx 0.65. figures/momentum_loss.py

On a quadratic bowl with η\eta and β\beta fixed, the factor that multiplies a coordinate at each step cannot, averaged over a long run, go below β\sqrt{\beta} (when the component along that direction is not 0). This long-run average is called the asymptotic factor. The motion along one direction is a mix of two factors whose product is β\beta, so the slower one cannot be smaller than β\sqrt{\beta} (derived below).

With β=0.9\beta = 0.9, 0.9≈0.949\sqrt{0.9} \approx 0.949. Averaged over a long run, the size of a coordinate cannot shrink faster than about 5.1% per step, which is 1−0.9=10%1 - 0.9 = 10\% in loss. A single step is not bound by this value. The path goes up and down as in the left panel of Figure 2: from step 5 to step 6 the loss falls from 4.75 to 0.606, a drop of 87%, and over the next two steps it rises again.

This 0.949 is almost the same as the xx factor 1−0.05=0.951 - 0.05 = 0.95 of gradient descent with η=0.05\eta = 0.05, and larger than the slower factor 0.885 of β=0.5\beta = 0.5 (derived below). So in the long run it falls behind β=0.5\beta = 0.5, and with β>0.952=0.9025\beta > 0.95^2 = 0.9025 it becomes slower than even gradient descent.

Why √β — solving Eq. (2) along one direction

kk is the curvature of that direction (the symbol from the gradient descent page), utu_t is the coordinate along that direction (xx or yy on this bowl), and rr is a root of the quadratic equation below.

From the right-hand equation of Eq. (2), η vt−1=θt−1−θt\eta\, v_{t-1} = \theta_{t-1} - \theta_t (at t=0t = 0, v−1=0v_{-1} = 0, so set θ−1=θ0\theta_{-1} = \theta_0). Substitute this into θt+1=θt−η ∇L(θt)−β η vt−1\theta_{t+1} = \theta_t - \eta\,\nabla L(\theta_t) - \beta\,\eta\, v_{t-1}, and vv disappears.

θt+1=θt−η ∇L(θt)+β (θt−θt−1)\theta_{t+1} = \theta_t - \eta\,\nabla L(\theta_t) + \beta\,(\theta_t - \theta_{t-1})

It is Eq. (1) plus β\beta times the previous step. In a direction with curvature kk (gradient kuk u), ut+1=(1+β−ηk) ut−β ut−1u_{t+1} = (1 + \beta - \eta k)\,u_t - \beta\,u_{t-1}.

Put in ut=rtu_t = r^t and you get r2−(1+β−ηk) r+β=0r^2 - (1 + \beta - \eta k)\,r + \beta = 0. The product of the two roots is β\beta, so the larger one has absolute value at least β\sqrt{\beta}. The roots are complex when (1+β−ηk)2<4β(1 + \beta - \eta k)^2 < 4\beta, and then both have absolute value exactly β\sqrt{\beta}.

For η=0.05\eta = 0.05, β=0.5\beta = 0.5, the xx direction (k=1k = 1) gives r2−1.45 r+0.5=0r^2 - 1.45\,r + 0.5 = 0, so the larger root is (1.45+1.452−2)/2≈0.885(1.45 + \sqrt{1.45^2 - 2})/2 \approx 0.885. That is smaller than gradient descent's 0.95. The yy direction (k=10k = 10) has the complex roots of r2−r+0.5=0r^2 - r + 0.5 = 0, so its factor is 0.5≈0.707\sqrt{0.5} \approx 0.707.

With η=0.05\eta = 0.05, the xx direction has complex roots when (0.95+β)2<4β(0.95 + \beta)^2 < 4\beta, that is, when β\beta is above about 0.603. The yy direction already has complex roots at smaller β\beta, so from there on the factors of both directions are β\sqrt{\beta}. β>0.95\sqrt{\beta} > 0.95 means β>0.9025\beta > 0.9025.

Both roots have absolute value below 1 when ∣β∣<1|\beta| < 1 (0≤β<10 \le \beta < 1 in the symbol table) and ∣1+β−ηk∣<1+β|1 + \beta - \eta k| < 1 + \beta, that is, 0<ηk<2(1+β)0 < \eta k < 2(1 + \beta).

If you also choose η\eta, the slower factor is smallest at the β\beta on the boundary where the roots of the direction of smallest curvature kmin⁡k_{\min} and of the direction of largest curvature kmax⁡k_{\max} both turn from real to complex. Writing the condition number as κ=kmax⁡/kmin⁡\kappa = k_{\max}/k_{\min}, that is β=(κ−1κ+1)2\beta = \left(\frac{\sqrt{\kappa} - 1}{\sqrt{\kappa} + 1}\right)^2, η=4(kmax⁡+kmin⁡)2\eta = \frac{4}{(\sqrt{k_{\max}} + \sqrt{k_{\min}})^2}, and the factor there is β\sqrt{\beta}. For this bowl (κ=10\kappa = 10) that gives β≈0.27\beta \approx 0.27, η≈0.231\eta \approx 0.231, and a factor of about 0.52. With κ=100\kappa = 100, β≈0.67\beta \approx 0.67, and β\beta reaches 0.9 at κ≈1442\kappa \approx 1442.

Key idea

Raising the momentum coefficient β\beta makes steps in the same direction longer, but raising it too far makes the path overshoot the bottom and spend steps coming back, so it gets slower instead.

How to choose β and the learning rate

The right β\beta depends on the learning rate, so you choose the two together. By the loss after 20 steps on this bowl, β≈0.65\beta \approx 0.65 is best when η=0.05\eta = 0.05, but with η=0.18\eta = 0.18 the value 0.9 is worse than gradient descent. PyTorch's default is β=0\beta = 0 (no momentum), and 0.9 is commonly used. Goodfellow et al. give 0.5, 0.9, and 0.99 as examples and note that it is sometimes started small and raised later.

Lowering the learning rate does not push the asymptotic factor below β\sqrt{\beta}. So on this bowl β=0.9\beta = 0.9 is slower than the best of gradient descent, 9/11≈0.8189/11 \approx 0.818, whatever η\eta you choose.

Why 0.9 is common all the same can be explained by the condition number. If you choose η\eta as well, the best for this bowl, whose condition number is 10, is β≈0.27\beta \approx 0.27 and η≈0.23\eta \approx 0.23. The asymptotic factor there is 0.52, smaller than the 0.818 that is the best of gradient descent (the end of the folded block 'Why √β'). The larger the condition number, the larger the best β\beta: about 0.67 at 100, and 0.9 at about 1440. 0.9 is too large for a condition number of 10, and it is the right value for problems whose condition number is above a thousand.

Does momentum reduce the zigzag (oscillation) of gradient descent?

With the right β\beta, momentum makes the zigzag (oscillation) die out faster. In gradient descent with η=0.18\eta = 0.18, yy flips sign at every step and its size is multiplied by 0.8. Switching on β=0.5\beta = 0.5 at the same η\eta lowers this factor, averaged over a long run, to 0.5≈0.707\sqrt{0.5} \approx 0.707, so after 20 steps ∣y∣|y| is 0.0006 instead of 0.017. With β=0.9\beta = 0.9 the factor rises to 0.949, and the path keeps oscillating for longer instead.

When is momentum a bad idea?

If you keep the learning rate as it is and switch on a large β\beta, it can be slower than gradient descent. At η=0.18\eta = 0.18, which is already fast, the loss after 20 steps is 0.005 for β=0\beta = 0, about 0.00001 for 0.50.5, and 3.02 for 0.90.9. Measuring β\beta from 0.72 to 0.95 at intervals of 0.001, every value gives a larger loss than gradient descent. The asymptotic factor 0.9≈0.949\sqrt{0.9} \approx 0.949 is larger than gradient descent's 1−0.18=0.821 - 0.18 = 0.82.

Do I need to change the learning rate when I use momentum?

Turning on β\beta changes the step size and the divergence limit by different factors, so choose the learning rate again together with β\beta. A step in a direction where the gradient is constant grows by a factor of 1/(1−β)1/(1-\beta) (10 times for 0.9). But the divergence limit 0<ηk<2(1+β)0 < \eta k < 2(1+\beta) widens only by a factor of (1+β)(1+\beta), so the limit 0.2 of the yy direction becomes just 0.38 at β=0.9\beta = 0.9. So if you raise only β\beta at the same learning rate, the path may not diverge but easily overshoots the bottom.

Common misconceptions about momentum

  • Turning on momentum without touching the learning rate leaves the stride the same — If the gradient is constant, the velocity in Eq. (3) grows to 1/(1−β)1/(1-\beta) times the gradient, so with momentum=0.9 one step gets up to 10 times longer even at the same lr.
  • The larger β, the faster — With η=0.05\eta = 0.05 the loss after 20 steps is smallest near β≈0.65\beta \approx 0.65, and for β≥0.91\beta \ge 0.91 it is larger than even gradient descent's 1.24 (Figure 2). The lower bound β\sqrt{\beta} of the asymptotic factor also grows with β\beta.

Where momentum shows up in AI

In the training loop of a neural network, momentum changes only one stage, the update optimizer.step(). Clearing the gradients, computing the loss, and getting the gradient by backpropagation are the same as in gradient descent. step() keeps a velocity vv for each parameter and updates it with Eq. (2).

This pagePyTorch
β\betamomentum in torch.optim.SGD(params, lr=0.05, momentum=0.5). Default 0 (no momentum, Eq. (1)); a commonly used value is 0.9. The docs write it as μ\mu
vtv_toptimizer.state[p]["momentum_buffer"], which the optimizer keeps for each parameter. At the first step() it starts as the gradient itself (with dampening=0 this is the same as Eq. (2) with v−1=0v_{-1} = 0)
η\etalr (default 1e-3)
Eq. (2)optimizer.step() when momentum > 0, dampening and weight_decay are 0, and nesterov and maximize are False (everything other than momentum at its default)
dampeningFrom the second step() on, the ∇L\nabla L in Eq. (2) is multiplied by (1 − dampening). At the first step() the buffer is the gradient itself. Default 0
nesterov=FalseThe default. True gives Nesterov momentum (planned) and requires momentum > 0 and dampening=0. Its convergence condition differs from that of Eq. (2) (below the table)

With Nesterov momentum, the learning-rate limit in a direction with curvature kk is 2(1+β)/((1+2β)k)2(1+\beta)/((1+2\beta)k), not 2(1+β)/k2(1+\beta)/k. In the yy direction of this bowl with β=0.5\beta = 0.5, that is 0.15, not 0.3.

Sutskever et al. and Keras put the learning rate inside the velocity and write vt=βvt−1−η ∇L(θt)v_t = \beta v_{t-1} - \eta\,\nabla L(\theta_t), θt+1=θt+vt\theta_{t+1} = \theta_t + v_t.33 With a fixed η\eta this gives the same path as Eq. (2) (the vtv_t of this form is the vtv_t of Eq. (2) multiplied by −η-\eta). If the learning rate changes from step to step, the velocity in this form keeps past learning rates, and the two forms differ.

The next page, Adam, divides this velocity, coordinate by coordinate, by the size of past gradients. Adam's β1\beta_1 corresponds to this page's β\beta, and that page writes the velocity as uu because it uses vv for a different quantity.

What does momentum=0.9 mean in PyTorch?

momentum=0.9 means that at every step you keep 90% of the previous velocity and add the current gradient (the β=0.9\beta = 0.9 of Eq. (2)). The default is 0, and 0.9 is a commonly used value. On this bowl with η=0.05\eta = 0.05, the loss after 20 steps is 1.18, larger than for β=0.5\beta = 0.5 (0.106). Why 0.9 is common all the same is the condition-number story in the section 'How to choose β and the learning rate'.

Check it in code

Two update lines change from the code on the gradient descent page. The code keeps a velocity velocity and steps by the velocity instead of the gradient. Change it to beta = 0 and you get the result of gradient descent with η=0.05\eta = 0.05.

Math
vt=β vt−1+∇L(θt),θt+1=θt−η vt(2)\sym{velocity}{v}_t = \sym{beta}{\beta} \, \sym{velocity}{v}_{t-1} + \sym{grad}{\nabla L}(\sym{theta}{\theta}_t), \qquad \sym{theta}{\theta}_{t+1} = \sym{theta}{\theta}_t - \sym{eta}{\eta} \, \sym{velocity}{v}_t \tag{2}
Code figures/momentum.py
import numpy as np

def L(p):
    x, y = p
    return 0.5 * (x**2 + 10 * y**2)

def grad(p):
    x, y = p
    return np.array([x, 10 * y])    # ∇L = (x, 10y)

theta    = np.array([-4.4, 1.5])    # θ₀
eta      = 0.05                      # η
beta     = 0.5                       # β  (0 이면 경사하강법)
velocity = np.zeros(2)               # v₋₁ = 0
for t in range(21):
    if t % 5 == 0: print(f"t={t:2d}  L={L(theta):.3f}")
    velocity = beta * velocity + grad(theta)   # 식 (2) 왼쪽
    theta = theta - eta * velocity             # 식 (2) 오른쪽
$ python momentum.py
t= 0  L=20.930
t= 5  L=4.161
t=10  L=1.214
t=15  L=0.360
t=20  L=0.106

Try it yourself

  1. With η=0.05\eta = 0.05 and β=0.9\beta = 0.9, where is θ2\theta_2 after two steps? How far has xx moved from the start, and how does that compare with gradient descent and β=0.5\beta = 0.5?
Answer

Up to v0=(−4.4, 15)v_0 = (-4.4,\ 15) and θ1=(−4.18, 0.75)\theta_1 = (-4.18,\ 0.75), everything is the same whatever β\beta is. v1=0.9×(−4.4, 15)+(−4.18, 7.5)=(−8.14, 21)v_1 = 0.9 \times (-4.4,\ 15) + (-4.18,\ 7.5) = (-8.14,\ 21), so θ2=(−4.18+0.407, 0.75−1.05)=(−3.773, −0.3)\theta_2 = (-4.18 + 0.407,\ 0.75 - 1.05) = (-3.773,\ -0.3). xx has moved forward by 0.627, farther than gradient descent (0.429) and β=0.5\beta = 0.5 (0.539), and yy has already passed the bottom, at −0.3-0.3.

Further reading

  • Some methods of speeding up the convergence of iteration methods — B. T. Polyak, 1964. USSR Computational Mathematics and Mathematical Physics 4(5). The original source of the 'heavy ball' method, which adds β times the previous step. It analyzes the rate of convergence on quadratic functions. The folded derivation in the section 'Why too large a β slows it down again' relies on it.
  • On the importance of initialization and momentum in deep learning — Sutskever, Martens, Dahl, Hinton, 2013. ICML. It writes classical momentum and Nesterov momentum side by side, and shows by experiment that the momentum coefficient and the initialization matter in training deep networks. The μ notation in the symbol table and the form that puts the learning rate inside the velocity in the section 'Where momentum shows up in AI' rely on it.
  • Deep Learning, Chapter 8 Optimization for Training Deep Models — Goodfellow, Bengio, Courville, 2016. Section 8.3.2 explains momentum as an exponentially decaying moving average of past gradients and shows that with a constant gradient the maximum speed is 1/(1 − α) times. The paragraph below Eq. (3), the common values in the section 'How to choose β and the learning rate', and the section 'Common misconceptions about momentum' rely on it.
  • torch.optim.SGD — PyTorch official documentation, checked against version 2.15. Argument defaults (momentum=0, dampening=0, nesterov=False), the update pseudocode, and a note that it is implemented in a form different from Sutskever et al. The mapping table in the section 'Where momentum shows up in AI' relies on it.

Video notes

Video notes
  • 00:00 Opening question — “What's the difference between SGD and momentum?”
  • 00:40 Write Eq. (1) and Eq. (2) side by side on the board — the only change is vv, the momentum of a heavy ball
  • 01:50 Eq. (3) — the velocity is a sum of past gradients, 1/(1−β)1/(1-\beta) times the gradient if it is constant
  • 03:00 Figure 1 — two steps by hand, θ2=(−3.861, 0)\theta_2 = (-3.861,\ 0)
  • 04:30 Figure 2 — where β=0.9\beta = 0.9 turns back, the low point near β≈0.65\beta \approx 0.65, the asymptotic factor β\sqrt{\beta}
  • 05:40 PyTorch momentum=0.9 and the learning rate · wrap-up

This is the filming plan. After upload, the timestamps will be replaced with the real ones.

Where this sits on the map