수학곳곳OptimizationSubject

Optimization

최적화

Neural-network optimizers in reading order: gradient descent and its learning-rate limit, mini-batch SGD, momentum and Adam, all on the same loss ½(x² + 10y²).

Done
5
Draft
0
Planned
8
Map
See the full map →

Reading order

  1. Gradient Descent

    #001Done

    Gradient descent lowers a loss by stepping against the gradient by the learning rate; it is how neural networks learn. After 20 steps a learning rate of 0.18 reaches loss 0.005, while 0.205 diverges to 79. See in figures and code how the most curved direction sets the limit 0.2.

  2. Mini-batch SGD

    #020Done한국어

  3. Momentum

    #002Done

    Momentum is an optimizer that steps by a velocity, a β-decayed sum of past gradients. At learning rate 0.05 the 20-step loss is 1.24 without it and 0.106 at β = 0.5. See why 0.9 is slow here yet common.

  4. Exponential Moving Average

    Planned

  5. Adam

    #003Done

    Adam is a deep-learning optimizer that divides the average gradient by its past size per coordinate. At lr 0.05 its first step moves x and y by 0.05; gradient descent moves 0.22 and 0.75. How it differs from momentum and RMSProp.

  6. Nesterov Momentum

    Planned

  7. AdamW

    Planned

  8. Max and Argmax

    Planned

  9. Condition Number

    Planned

  10. Convexity and Local Minima

    Planned

  11. RMSProp

    Planned

  12. Learning Rate Schedule

    #536Done한국어

  13. Batch Size

    Planned