10. DeepSpeed & PyTorch FSDP (Fully Sharded Data Parallel): Zero Redundancy Optimizer (ZeRO-1, ZeRO-2, ZeRO-3) • AI Infrastructure, GPU Serving & MLOps (CUDA, Triton, TGI) • Rishabh Ki Kaksha

AI Infrastructure, GPU Serving & MLOps (CUDA, Triton, TGI) › 10. DeepSpeed & PyTorch FSDP (Fully Sharded Data Parallel): Zero Redundancy Optimizer (ZeRO-1, ZeRO-2, ZeRO-3)

55 mins

← Prev Next →

Course Syllabus

Module 1: GPU Architecture, CUDA Internals & Hardware Hierarchy

01. NVIDIA GPU Architecture: Streaming Multiprocessors (SMs), Tensor Cores, CUDA Cores & HBM3 Memory Bandwidth 02. GPU Memory Hierarchy: Registers, Shared Memory (SRAM), L1/L2 Caches & High-Bandwidth VRAM 03. CUDA Programming Basics: Writing Custom GPU Kernels in C++/CUDA & Triton Python DSL

Part 1: GPU Acceleration, Serving & Production Guardrails

Module 2: High-Performance LLM Serving Engines (vLLM, TGI, Triton)

04. The KV Cache Bottleneck: Memory Footprint Calculations & PagedAttention Virtual Memory Architecture 05. Dynamic & Continuous Batching (Iteration-Level Scheduling) vs Static Batching in LLM Serving 06. Production Benchmark: vLLM vs HuggingFace TGI (Text Generation Inference) vs NVIDIA Triton Inference Server 07. Speculative Decoding & Medusa Multi-Head Decoding for 2-3x Faster Token Generation

Module 3: Multi-GPU & Distributed Inference: Tensor & Pipeline Parallelism

08. Distributed AI Parallelism Paradigms: Data Parallelism (DP), Tensor Parallelism (TP), Pipeline Parallelism (PP) 09. Tensor Parallelism in Action: Megatron-LM Matrix Splitting across NVLink & NVSwitch Networks 10. DeepSpeed & PyTorch FSDP (Fully Sharded Data Parallel): Zero Redundancy Optimizer (ZeRO-1, ZeRO-2, ZeRO-3)

Module 4: MLOps Pipelines: Model Registry, Experiment Tracking (MLflow/W&B)

11. MLOps Foundations: Model Versioning, Artifact Registries & Lineage Tracking with MLflow and Weights & Biases 12. Automated Continuous Training Pipelines with Kubeflow Pipelines & GitHub Actions CI/CD for AI Models 13. Feature Stores in Production (Feast): Real-Time Online Feature Retrieval vs Offline Batch Processing

Module 5: Kubernetes GPU Orchestration, Auto-Scaling & Continuous Observability

14. Kubernetes for AI Workloads: NVIDIA GPU Operator, Device Plugins & Time-Slicing vs Multi-Instance GPU (MIG) 15. Horizontal Pod Auto-Scaling (HPA) for LLM Services based on TTFT (Time to First Token) & Queue Depth 16. Serverless GPU Architecture: Ray Serve, Modal & RunPod for Auto-Scaling AI APIs with Zero Cold Starts 17. Observability for GenAI: OpenLLMetry, Arize Phoenix, Langfuse & Prometheus GPU Utilization Metrics 18. Enterprise Cloud Cost Optimization: Spot GPU Instances, Multi-Cloud Orchestration & Slicing VRAM Costs

Interactive Lesson Studio

# 10. DeepSpeed & PyTorch FSDP (Fully Sharded Data Parallel): Zero Redundancy Optimizer (ZeRO-1, ZeRO-2, ZeRO-3)

> AI Infrastructure & MLOps Notes by Rishabh - Topic 10

---

### 1. The ZeRO (Zero Redundancy Optimizer) Stages (Microsoft DeepSpeed 🚀)

Standard Data Parallelism me har GPU par poora model, gradients aur optimizer states redundant copy hoti hain.  
**ZeRO eliminates all redundancy without changing communication volume:**

1. **ZeRO-Stage 1 (Optimizer State Partitioning - $4\times$ memory reduction):** Adam optimizer states ($m_t, v_t$) ko saare GPUs ke beech shard karta hai.
2. **ZeRO-Stage 2 (Gradient Partitioning - $8\times$ memory reduction):** Gradients ko bhi shard karta hai (Only owner GPU retains gradient for its parameters).
3. **ZeRO-Stage 3 / PyTorch FSDP (Full Parameter Partitioning - $N\times$ memory reduction 👑):** Model parameters ko bhi shard karta hai! Forward pass ke waqt dynamic `All-Gather` se parameters fetch hote hain aur execute hote hi memory se delete ho jaate hain!

---

### 2. Master Comparison Matrix

---

### 3. Interview Ke 3 Sawal Is Topic Se

1. **PyTorch FSDP (Fully Sharded Data Parallel) DeepSpeed ZeRO-3 ka native alternative kyu hai?** -> PyTorch me zero external dependencies ke sath native C++ runtime me integrated hai with automatic backward pre-fetching.
2. **ZeRO-Offload kya karta hai?** -> Optimizer states aur computation ko GPU VRAM se host CPU RAM aur NVMe SSD par offload karke single GPU par 20B model train karne deta hai.
3. **FSDP me `backward_prefetch = BackwardPrefetch.BACKWARD_PRE` flag kya optimize karta hai?** -> Current layer ka backward pass chalte waqt next layer ke parameters ko background CUDA stream me pre-fetch kar leta hai.

---
**Next Topic:** 11. MLOps Foundations: Model Versioning, Artifact Registries & Lineage Tracking with MLflow and Weights & Biases

Loading lesson content...

Python 3.12

Full IDE

Live Execution Terminal

Click "Execute Live" to run code.
                

Lesson Mastery AI Quiz

AI Doubt Solver

CERTIFICATE OF COMPLETION

Rishabh Gehlot

PHP 8+ & Laravel 11 Master Track