17. Logistic Regression & Binary Cross-Entropy Loss from Scratch with NumPy • Mathematics, Statistics & Machine Learning (Big Tech Prep) • Rishabh Ki Kaksha

Mathematics, Statistics & Machine Learning (Big Tech Prep) › 17. Logistic Regression & Binary Cross-Entropy Loss from Scratch with NumPy

50 mins

← Prev Next →

Course Syllabus

Module 1: Linear Algebra for Machine Learning & Deep Learning

01. Vectors, Vector Spaces, Dot Products & Norms (L1, L2, L-Infinity) in High Dimensions 02. Matrices, Matrix Multiplication & Linear Transformations (Rotation, Scaling, Shear) 03. Eigenvalues & Eigenvectors: Geometric Intuition, Diagonalization & Spectral Theorem 04. Principal Component Analysis (PCA) & Singular Value Decomposition (SVD) Math

Module 2: Multivariable Calculus, Gradients & Optimization Algorithms

05. Partial Derivatives, Directional Derivatives & The Gradient Vector (Del Operator) 06. The Jacobian Matrix & Hessian Matrix (Second Derivative Curvature Analysis) 07. Multivariate Chain Rule & The Mathematical Backbone of Neural Backpropagation 08. Optimization Algorithms: Gradient Descent, SGD with Momentum, RMSprop & Adam Optimizer Math

Module 3: Probability Theory & Random Variables

09. Probability Foundations: Axioms of Probability, Conditional Probability & Independence 10. Bayes' Theorem & Bayesian Inference: Prior, Likelihood, Posterior & Evidence 11. Probability Distributions: Gaussian (Normal), Bernoulli, Binomial, Poisson & Exponential Distributions

Module 4: Inferential Statistics & Hypothesis Testing

12. Expectation, Variance, Covariance & Pearson Correlation Coefficient 13. Law of Large Numbers & Central Limit Theorem (CLT) with Monte Carlo Simulations 14. Hypothesis Testing: Null Hypothesis, p-Values, Z-Test, Student's t-Test & Chi-Square Test 15. Maximum Likelihood Estimation (MLE) vs Maximum A Posteriori (MAP) Estimation

Module 5: Classical Machine Learning Algorithms from Scratch (NumPy)

16. Linear Regression from Scratch: Ordinary Least Squares (OLS) Closed Form vs Gradient Descent 17. Logistic Regression & Binary Cross-Entropy Loss from Scratch with NumPy 18. Regularization Mathematics: L1 Lasso (Sparsity) vs L2 Ridge (Weight Decay) Penalties

Interactive Lesson Studio

# 17. Logistic Regression & Binary Cross-Entropy Loss from Scratch with NumPy

> Mathematics & ML Foundations Notes by Rishabh - Topic 17

---

### 1. The Sigmoid Activation Function 📈

$$\sigma(z) = \frac{1}{1 + e^{-z}} \in (0, 1) \quad \text{where } z = w^T x + b$$
$$\text{Derivative: } \sigma'(z) = \sigma(z)(1 - \sigma(z))$$

---

### 2. Binary Cross-Entropy (BCE / Log-Loss) Loss Function

$$L(w) = -\frac{1}{N} \sum_{i=1}^N \left[ y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i) \right]$$

*The miraculous gradient formula is identical to linear regression:*
$$\frac{\partial L}{\partial w} = \frac{1}{N} X^T (\hat{y} - y)$$

---

### 3. Scratch Implementation in Pure NumPy

```python
import numpy as np

class LogisticRegressionScratch:
    def __init__(self, lr=0.1, epochs=1000):
        self.lr = lr
        self.epochs = epochs
        self.w = None
        self.b = 0.0

def _sigmoid(self, z):
        return 1.0 / (1.0 + np.exp(-np.clip(z, -250, 250)))

def fit(self, X: np.ndarray, y: np.ndarray):
        n_samples, n_features = X.shape
        self.w = np.zeros(n_features)

for _ in range(self.epochs):
            linear_model = np.dot(X, self.w) + self.b
            y_pred = self._sigmoid(linear_model)

dw = (1 / n_samples) * np.dot(X.T, (y_pred - y))
            db = (1 / n_samples) * np.sum(y_pred - y)

self.w -= self.lr * dw
            self.b -= self.lr * db

def predict_proba(self, X: np.ndarray) -> np.ndarray:
        return self._sigmoid(np.dot(X, self.w) + self.b)

def predict(self, X: np.ndarray, threshold=0.5) -> np.ndarray:
        return (self.predict_proba(X) >= threshold).astype(int)
```

---

### 4. Master Comparison Matrix

| Classification Metric | Formula | Best For |
| :--- | :--- | :--- |
| **Precision** | $\frac{TP}{TP + FP}$ | Spam detection (Minimizing False Positives) |
| **Recall (Sensitivity)** | $\frac{TP}{TP + FN}$ | Cancer / Fraud detection (Minimizing False Negatives) |
| **ROC-AUC Score** | Area under TPR vs FPR curve | Model comparison across all thresholds |

---

### 5. Interview Ke 3 Sawal Is Topic Se

1. **Logistic Regression me MSE loss kyu nahi use karte?** -> Sigmoid function ke sath MSE non-convex ban jata hai jisme multiple local minima hote hain; Cross-Entropy loss convex rehta hai with a guaranteed global minimum.
2. **Decision Boundary linear kyu hoti hai?** -> Kyunki $\sigma(w^T x + b) = 0.5 \iff w^T x + b = 0$, jo $D$-dimensional space me ek flat hyperplane represent karta hai.
3. **Softmax Regression (Multinomial Logistic Regression) multi-class me kaise generalize hota hai?** -> $P(y = k \mid x) = \frac{e^{z_k}}{\sum_{j=1}^K e^{z_j}}$ categorical cross-entropy loss ke sath.

---
**Next Topic:** 18. Regularization Mathematics: L1 Lasso (Sparsity) vs L2 Ridge (Weight Decay) Penalties

Loading lesson content...

Python 3.12

Full IDE

Live Execution Terminal

Click "Execute Live" to run code.
                

Lesson Mastery AI Quiz

AI Doubt Solver

CERTIFICATE OF COMPLETION

Rishabh Gehlot

PHP 8+ & Laravel 11 Master Track