Gradient descent, step by step

Pick a function, a step size and a starting point. Shading shows f (darker = higher); the pink path is gradient descent.

Gradient descent: x ← x − γ·∇f(x) (+ momentum)

step size γ momentum α steps

Click or tap the picture to choose where to start.

Try thisOn the long valley, raise γ until the path zig-zags, then until it blows up. Now add momentum 0.8 with a smaller γ: it rolls down the valley floor much faster.