AI Models & Assistants / Knowledge library
Stochastic Gradient Descent (SGD)
Estimates a training objective's gradient from sampled examples or mini-batches
Why learn Stochastic Gradient Descent (SGD)?
It makes iterative learning practical on datasets too large for a full gradient at every update.
The problem it solves
It lowers the cost of each training update while retaining a useful descent direction on average.
Where it came from
SGD replaces an exact full-dataset gradient with cheaper noisy estimates. Mini-batch size, learning-rate schedules, momentum, and data sampling affect its speed and stability.
Build your understanding
Work through these exercises in order. Try the task before opening a reference.
- Define Stochastic Gradient Descent (SGD) and distinguish it from a nearby concept.
- Choose a realistic task where Stochastic Gradient Descent (SGD) is appropriate.
- Identify one failure mode or misuse of Stochastic Gradient Descent (SGD).
- Interpret a small result produced with Stochastic Gradient Descent (SGD).
- Explain one limitation or tradeoff of Stochastic Gradient Descent (SGD).
Documentation and sources
- Stochastic Gradient Descent Tricks (doi.org)