11. Direct Preference Optimization (DPO): Eliminating Reward Models via Implicit Policy Optimization Math • Model Fine-Tuning, Quantization & RLHF (PEFT, LoRA, vLLM) • Rishabh Ki Kaksha

Model Fine-Tuning, Quantization & RLHF (PEFT, LoRA, vLLM) › 11. Direct Preference Optimization (DPO): Eliminating Reward Models via Implicit Policy Optimization Math

50 mins

← Prev Next →

Course Syllabus

Module 1: Pre-training vs Fine-Tuning & Dataset Engineering

01. LLM Adaptation Paradigms: Pretraining, Continued Pretraining, Full Fine-Tuning vs PEFT 02. Fine-Tuning Dataset Engineering: Instruction Datasets (Alpaca/ShareGPT formats), Quality Filtering & Deduplication 03. Tokenization Traps: Prompt Formatting, Chat Templates (ChatML / Llama-3 format) & Loss Masking on Prompts

Module 2: Parameter-Efficient Fine-Tuning (PEFT) & LoRA / QLoRA Internals

04. Low-Rank Adaptation (LoRA) Mathematics: Intrinsic Rank Hypothesis, Matrix Factorization (W + B*A) & Alpha Scaling 05. QLoRA Deep Dive: 4-Bit NormalFloat (NF4), Double Quantization & Paged Optimizers for VRAM Efficiency 06. Target Modules Selection (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj) & Rank Tradeoffs

Module 3: Supervised Fine-Tuning (SFT) with HuggingFace TRL & Unsloth

07. Hugging Face TRL (Transformer Reinforcement Learning) & SFTTrainer Architecture 08. Ultra-Fast Fine-Tuning with Unsloth (Manual Triton Kernels & 5x Speedup on Single GPUs) 09. Monitoring Loss Curves: Overfitting Detection, Validation Perplexity & Catastrophic Forgetting Mitigation

Module 4: Alignment Algorithms: DPO (Direct Preference Optimization) & RLHF

10. RLHF (Reinforcement Learning from Human Feedback) Architecture: Reward Models & PPO (Proximal Policy Optimization) 11. Direct Preference Optimization (DPO): Eliminating Reward Models via Implicit Policy Optimization Math 12. ORPO (Odds Ratio Preference Optimization) & KTO (Kahneman-Tversky Optimization) Alignment Techniques

Module 5: Model Quantization (GGUF, AWQ, GPTQ) & High-Throughput vLLM Serving

13. Post-Training Quantization (PTQ): 4-Bit & 8-Bit Quantization Mathematics (Zero Point & Scale Factors) 14. Quantization Formats: GGUF (llama.cpp on CPU/Mac), AWQ (Activation-aware Weight Quantization) & GPTQ 15. Merging LoRA Adapters back into Base Model Weights for Zero-Inference Latency Overhead 16. High-Throughput Enterprise Model Serving with vLLM (PagedAttention, Continuous Batching & Tensor Parallelism)

Interactive Lesson Studio

# 11. Direct Preference Optimization (DPO): Eliminating Reward Models via Implicit Policy Optimization Math

> Model Fine-Tuning & Quantization Notes by Rishabh - Topic 11

---

### 1. The DPO Revolution (Rafailov et al. - Stanford 🚀)

PPO me alag Reward Model aur unstable Reinforcement Learning training karni padti thi.  
**DPO Mathematical Insight:** Bradley-Terry reward function ko direct closed-form me substitute karke **Reward Model ko completely eliminate kar diya!**

$$\mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = -\mathbb{E}_{(x, y_w, y_l)}\left[ \log \sigma\left( \beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)} \right) \right]$$

---

### 2. DPO Training with TRL `DPOTrainer`

```python
from trl import DPOTrainer, DPOConfig

dpo_trainer = DPOTrainer(
    model=model,
    ref_model=None, # PEFT LoRA automatically disables adapters for ref_model!
    beta=0.1, # KL penalty coefficient
    train_dataset=preference_dataset, # {"prompt", "chosen", "rejected"}
    tokenizer=tokenizer,
    max_length=1024,
    args=DPOConfig(
        output_dir="./dpo_results",
        learning_rate=5e-7, # DPO requires very small learning rates!
        bf16=True,
    ),
)
dpo_trainer.train()
```

---

### 3. Master Comparison Matrix

---

### 4. Interview Ke 3 Sawal Is Topic Se

1. **DPO me $\beta$ parameter kya control karta hai?** -> Reference model $\pi_{\text{ref}}$ ke anchor constraint ki strength ($\beta = 0.1$ standard; lower $\beta$ gives more freedom to align to preferences).
2. **PEFT LoRA DPO me VRAM kaise bachata hai?** -> Reference model alag se load karne ki zaroorat nahi padti; LoRA adapter disable karke reference policy probabilities compute hoti hain.
3. **DPO me Chosen vs Rejected generation probabilities ka log-ratio kya express karta hai?** -> Implicit reward difference $\hat{r}(x, y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}$.

---
**Next Topic:** 12. ORPO (Odds Ratio Preference Optimization) & KTO (Kahneman-Tversky Optimization) Alignment Techniques

Loading lesson content...

Python 3.12

Full IDE

Live Execution Terminal

Click "Execute Live" to run code.
                

Lesson Mastery AI Quiz

AI Doubt Solver

CERTIFICATE OF COMPLETION

Rishabh Gehlot

PHP 8+ & Laravel 11 Master Track