Gradient Descent: Learning Rate & Convergence Explained

Status: ready

Script

{
  "hook": "Converge Faster!",
  "tags": [
    "gradient descent",
    "learning rate",
    "machine learning",
    "optimization",
    "convergence",
    "deep learning",
    "calculus",
    "AI"
  ],
  "title": "Gradient Descent: Learning Rate & Convergence Explained",
  "sections": [
    {
      "formula": "theta = theta - alpha * grad(J)",
      "heading": "What Is Gradient Descent?",
      "narration": "Imagine you're standing on a foggy mountain and want to reach the valley below. Gradient descent is the algorithm that does exactly that for machine learning models. It starts with a random guess for the parameters, then repeatedly checks the slope of the loss function. It takes a small step downhill in the steepest direction, updates the weights, and repeats. Each step should lower the error a little bit. Over many iterations, you hopefully arrive at the minimum where the model performs best. The size of each step is controlled by one critical number called the learning rate. Get it right, and you descend smoothly. Get it wrong, and you might never reach the bottom."
    },
    {
      "formula": "small alpha = slow, large alpha = diverge",
      "heading": "Learning Rate Too Small or Large",
      "narration": "Think of the learning rate alpha as your step size. If alpha is tiny, like one millimeter, you will move downhill but it will take forever. Training becomes painfully slow, costs lots of compute, and you might get stuck in a flat region or local noise thinking you converged. On the other hand, if alpha is huge, you take giant leaps. You might jump over the valley, bounce from side to side, and your loss goes up and down wildly. Make it even larger and you diverge completely, with loss exploding to infinity. The ideal learning rate is in the middle, large enough to make fast progress but small enough to settle into the minimum without overshooting."
    },
    {
      "formula": "0 < alpha < 2 / L",
      "heading": "Convergence Math and Conditions",
      "narration": "So when does gradient descent actually guarantee convergence? For smooth convex functions, theory gives us a clear rule. If your loss has Lipschitz gradient with constant L, meaning slopes don't change too abruptly, then you converge if alpha is less than two over L. In practice for a simple quadratic, the optimal alpha is one over curvature, landing you at the bottom in one step. We monitor convergence by watching the loss curve flatten and the gradient norm approach zero. If loss decreases steadily, you're good. If it plateaus early, alpha may be too small. If it oscillates or spikes, alpha is too big. Mathematically, the error shrinks each iteration by a factor that depends on alpha, until you reach a tolerance."
    },
    {
      "formula": "alpha_t = alpha_0 / (1 + decay * t)",
      "heading": "Tuning Tricks That Actually Work",
      "narration": "You rarely guess the perfect learning rate on the first try, so use smart tuning. Start with a learning rate finder, try values like zero point one, zero point zero one, zero point zero zero one, and plot the loss. Pick the largest rate that still decreases smoothly. Better yet, use schedules. Start large for speed, then decay over time with step decay, exponential decay, or cosine annealing. Adaptive optimizers like Adam adjust alpha per parameter automatically, which is why they are so popular. Always plot loss versus iteration. A smooth descending curve means healthy convergence. Add momentum to roll past small bumps. Remember, tuning the learning rate is the single highest leverage skill for training neural networks faster and better."
    }
  ],
  "description": "Learn how learning rate controls gradient descent convergence. Discover why too large diverges, too small stalls, and how to find the sweet spot."
}

Costs

[
  {
    "id": 2,
    "video_id": 2,
    "llm_input_tokens": 135,
    "llm_output_tokens": 4345,
    "llm_reasoning_tokens": 3474,
    "llm_cost_usd": "0.000882",
    "tts_chars": 2802,
    "tts_cost_usd": "0.140100",
    "created_at": "2026-09-07T09:36:24.450Z"
  }
]