Reading order
Gradient Descent
Gradient descent lowers a loss by stepping against the gradient by the learning rate; it is how neural networks learn. After 20 steps a learning rate of 0.18 reaches loss 0.005, while 0.205 diverges to 79. See in figures and code how the most curved direction sets the limit 0.2.
Mini-batch SGD
Momentum
Momentum is an optimizer that steps by a velocity, a β-decayed sum of past gradients. At learning rate 0.05 the 20-step loss is 1.24 without it and 0.106 at β = 0.5. See why 0.9 is slow here yet common.
Exponential Moving Average
Adam
Adam is a deep-learning optimizer that divides the average gradient by its past size per coordinate. At lr 0.05 its first step moves x and y by 0.05; gradient descent moves 0.22 and 0.75. How it differs from momentum and RMSProp.
Nesterov Momentum
AdamW
Max and Argmax
Condition Number
Convexity and Local Minima
RMSProp
Learning Rate Schedule
Batch Size