16. Linear Regression from Scratch: Ordinary Least Squares (OLS) Closed Form vs Gradient Descent • Mathematics, Statistics & Machine Learning (Big Tech Prep) • Rishabh Ki Kaksha

Mathematics, Statistics & Machine Learning (Big Tech Prep) › 16. Linear Regression from Scratch: Ordinary Least Squares (OLS) Closed Form vs Gradient Descent

50 mins

← Prev Next →

Course Syllabus

Module 1: Linear Algebra for Machine Learning & Deep Learning

01. Vectors, Vector Spaces, Dot Products & Norms (L1, L2, L-Infinity) in High Dimensions 02. Matrices, Matrix Multiplication & Linear Transformations (Rotation, Scaling, Shear) 03. Eigenvalues & Eigenvectors: Geometric Intuition, Diagonalization & Spectral Theorem 04. Principal Component Analysis (PCA) & Singular Value Decomposition (SVD) Math

Module 2: Multivariable Calculus, Gradients & Optimization Algorithms

05. Partial Derivatives, Directional Derivatives & The Gradient Vector (Del Operator) 06. The Jacobian Matrix & Hessian Matrix (Second Derivative Curvature Analysis) 07. Multivariate Chain Rule & The Mathematical Backbone of Neural Backpropagation 08. Optimization Algorithms: Gradient Descent, SGD with Momentum, RMSprop & Adam Optimizer Math

Module 3: Probability Theory & Random Variables

09. Probability Foundations: Axioms of Probability, Conditional Probability & Independence 10. Bayes' Theorem & Bayesian Inference: Prior, Likelihood, Posterior & Evidence 11. Probability Distributions: Gaussian (Normal), Bernoulli, Binomial, Poisson & Exponential Distributions

Module 4: Inferential Statistics & Hypothesis Testing

12. Expectation, Variance, Covariance & Pearson Correlation Coefficient 13. Law of Large Numbers & Central Limit Theorem (CLT) with Monte Carlo Simulations 14. Hypothesis Testing: Null Hypothesis, p-Values, Z-Test, Student's t-Test & Chi-Square Test 15. Maximum Likelihood Estimation (MLE) vs Maximum A Posteriori (MAP) Estimation

Module 5: Classical Machine Learning Algorithms from Scratch (NumPy)

16. Linear Regression from Scratch: Ordinary Least Squares (OLS) Closed Form vs Gradient Descent 17. Logistic Regression & Binary Cross-Entropy Loss from Scratch with NumPy 18. Regularization Mathematics: L1 Lasso (Sparsity) vs L2 Ridge (Weight Decay) Penalties

Interactive Lesson Studio

# 16. Linear Regression from Scratch: Ordinary Least Squares (OLS) Closed Form vs Gradient Descent

> Mathematics & ML Foundations Notes by Rishabh - Topic 16

---

### 1. The Mathematical Model & OLS Normal Equation 📐

Given feature matrix $X \in \mathbb{R}^{N \times D}$ and labels $y \in \mathbb{R}^N$:
$$\text{Cost Function: } J(w) = \frac{1}{2N} \|Xw - y\|_2^2$$

Setting gradient $\nabla_w J(w) = 0$ gives the **Normal Equation Closed-Form Solution**:
$$\hat{w} = (X^T X)^{-1} X^T y$$

---

### 2. NumPy Implementation from Scratch

```python
import numpy as np

class LinearRegressionScratch:
    def __init__(self, lr=0.01, epochs=1000):
        self.lr = lr
        self.epochs = epochs
        self.w = None
        self.b = 0.0

def fit(self, X: np.ndarray, y: np.ndarray):
        n_samples, n_features = X.shape
        self.w = np.zeros(n_features)

for _ in range(self.epochs):
            y_pred = np.dot(X, self.w) + self.b
            # Compute analytical gradients:
            dw = (1 / n_samples) * np.dot(X.T, (y_pred - y))
            db = (1 / n_samples) * np.sum(y_pred - y)

# Gradient Descent step:
            self.w -= self.lr * dw
            self.b -= self.lr * db

def predict(self, X: np.ndarray) -> np.ndarray:
        return np.dot(X, self.w) + self.b
```

---

### 3. Master Comparison Matrix

| Method | Time Complexity | Needs Learning Rate Tuning? | Scales to Millions of Features? |
| :--- | :--- | :--- | :--- |
| **OLS Normal Equation** | $O(D^3)$ (Matrix Inversion) | ❌ No | ❌ Fails when $D > 10,000$ |
| **Gradient Descent** | $O(\text{Epochs} \cdot N \cdot D)$ | ✅ Yes ($\eta$) | 👑 **Scales to Billions of parameters** |

---

### 4. Interview Ke 3 Sawal Is Topic Se

1. **$(X^T X)$ matrix kab Non-Invertible (Singular) hoti hai?** -> Jab features me Multi-collinearity ho (redundant correlated features) ya number of features $D$ samples $N$ se zyada hon ($D > N$). Pseudo-inverse ($X^+$) ya L2 Ridge regularization ise fix karti hai.
2. **Mean Squared Error (MSE) loss convex function kyu hota hai?** -> Kyunki iska Hessian matrix $H = \frac{1}{N} X^T X$ Positive Semi-Definite (PSD) hota hai, jisse guarantee milti hai ki global minimum unique hai.
3. **Feature Scaling (Standardization / Min-Max) Gradient Descent convergence ko kaise speed-up karti hai?** -> Loss contours ko elongated narrow valleys ke badle circular bowl me convert karti hai jisse oscillations eliminate hoti hain.

---
**Next Topic:** 17. Logistic Regression & Binary Cross-Entropy Loss from Scratch with NumPy

Loading lesson content...

Python 3.12

Full IDE

Live Execution Terminal

Click "Execute Live" to run code.
                

Lesson Mastery AI Quiz

AI Doubt Solver

CERTIFICATE OF COMPLETION

Rishabh Gehlot

PHP 8+ & Laravel 11 Master Track