08. Optimization Algorithms: Gradient Descent, SGD with Momentum, RMSprop & Adam Optimizer Math • Mathematics, Statistics & Machine Learning (Big Tech Prep) • Rishabh Ki Kaksha

Mathematics, Statistics & Machine Learning (Big Tech Prep) › 08. Optimization Algorithms: Gradient Descent, SGD with Momentum, RMSprop & Adam Optimizer Math

50 mins

← Prev Next →

Course Syllabus

Module 1: Linear Algebra for Machine Learning & Deep Learning

01. Vectors, Vector Spaces, Dot Products & Norms (L1, L2, L-Infinity) in High Dimensions 02. Matrices, Matrix Multiplication & Linear Transformations (Rotation, Scaling, Shear) 03. Eigenvalues & Eigenvectors: Geometric Intuition, Diagonalization & Spectral Theorem 04. Principal Component Analysis (PCA) & Singular Value Decomposition (SVD) Math

Module 2: Multivariable Calculus, Gradients & Optimization Algorithms

05. Partial Derivatives, Directional Derivatives & The Gradient Vector (Del Operator) 06. The Jacobian Matrix & Hessian Matrix (Second Derivative Curvature Analysis) 07. Multivariate Chain Rule & The Mathematical Backbone of Neural Backpropagation 08. Optimization Algorithms: Gradient Descent, SGD with Momentum, RMSprop & Adam Optimizer Math

Module 3: Probability Theory & Random Variables

09. Probability Foundations: Axioms of Probability, Conditional Probability & Independence 10. Bayes' Theorem & Bayesian Inference: Prior, Likelihood, Posterior & Evidence 11. Probability Distributions: Gaussian (Normal), Bernoulli, Binomial, Poisson & Exponential Distributions

Module 4: Inferential Statistics & Hypothesis Testing

12. Expectation, Variance, Covariance & Pearson Correlation Coefficient 13. Law of Large Numbers & Central Limit Theorem (CLT) with Monte Carlo Simulations 14. Hypothesis Testing: Null Hypothesis, p-Values, Z-Test, Student's t-Test & Chi-Square Test 15. Maximum Likelihood Estimation (MLE) vs Maximum A Posteriori (MAP) Estimation

Module 5: Classical Machine Learning Algorithms from Scratch (NumPy)

16. Linear Regression from Scratch: Ordinary Least Squares (OLS) Closed Form vs Gradient Descent 17. Logistic Regression & Binary Cross-Entropy Loss from Scratch with NumPy 18. Regularization Mathematics: L1 Lasso (Sparsity) vs L2 Ridge (Weight Decay) Penalties

Interactive Lesson Studio

# 08. Optimization Algorithms: Gradient Descent, SGD with Momentum, RMSprop & Adam Optimizer Math

> Mathematics & ML Foundations Notes by Rishabh - Topic 08

---

### 1. The Optimization Evolutionary Tree 🌳

$$\text{SGD} \longrightarrow \text{Momentum (Velocity)} \longrightarrow \text{AdaGrad / RMSprop (Adaptive LR)} \longrightarrow \text{Adam (First + Second Moments)}$$

---

### 2. The Complete Adam (Adaptive Moment Estimation) Algorithm ⚡

For each iteration $t$, compute gradient $g_t = \nabla_\theta L(\theta_t)$:

1. **First Moment Vector (Moving Average of Gradients - Momentum):**
   $$m_t = \beta_1 m_{t-1} + (1 - \beta_1) g_t \quad (\beta_1 = 0.9)$$
2. **Second Moment Vector (Moving Average of Squared Gradients - Variance):**
   $$v_t = \beta_2 v_{t-1} + (1 - \beta_2) g_t^2 \quad (\beta_2 = 0.999)$$
3. **Bias Corrections (Warmup compensation for zero initialization):**
   $$\hat{m}_t = \frac{m_t}{1 - \beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1 - \beta_2^t}$$
4. **Parameter Update:**
   $$\theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{\hat{v}_t} + \epsilon} \hat{m}_t$$

---

### 3. Master Comparison Matrix

| Optimizer | Tracks Momentum? | Adaptive Per-Param LR? | Handles Sparse Gradients? |
| :--- | :--- | :--- | :--- |
| **Standard SGD** | ❌ No | ❌ No | ❌ Poor |
| **SGD with Momentum**| ✅ Yes ($\beta = 0.9$) | ❌ No | Moderate |
| **RMSprop** | ❌ No | ✅ Yes ($1/\sqrt{v_t}$) | ✅ Good |
| **Adam / AdamW** | 👑 **Yes ($m_t$)** | 👑 **Yes ($v_t$)** | 👑 **Industry Standard for LLMs** |

---

### 4. Interview Ke 3 Sawal Is Topic Se

1. **AdamW Adam se LLM training (Llama, GPT) me kyu behtar hai?** -> Adam me L2 regularization adaptive learning rate ke sath mix hokar decay rate distort kar deti hai; AdamW weight decay ko gradient updates se completely decouple karke directly weights par apply karta hai.
2. **Saddle Points (Zero gradient flat regions) se Momentum kaise bahar nikalta hai?** -> Accumulated velocity vector $m_t$ flat plateau regions me momentum carry forward karke optimization ko accelerate karta hai.
3. **Learning Rate Warmup schedules kyu zaroori hote hain?** -> Training ke start me gradients high variance hote hain; low learning rate se start karke moments ko stabilize karna divergence rokkta hai.

---
**Next Topic:** 09. Probability Foundations: Axioms of Probability, Conditional Probability & Independence

Loading lesson content...

Python 3.12

Full IDE

Live Execution Terminal

Click "Execute Live" to run code.
                

Lesson Mastery AI Quiz

AI Doubt Solver

CERTIFICATE OF COMPLETION

Rishabh Gehlot

PHP 8+ & Laravel 11 Master Track