Model Fine-Tuning, Quantization & RLHF (PEFT, LoRA, vLLM) › 12. ORPO (Odds Ratio Preference Optimization) & KTO (Kahneman-Tversky Optimization) Alignment Techniques

Course Syllabus

01. LLM Adaptation Paradigms: Pretraining, Continued Pretraining, Full Fine-Tuning vs PEFT 02. Fine-Tuning Dataset Engineering: Instruction Datasets (Alpaca/ShareGPT formats), Quality Filtering & Deduplication 03. Tokenization Traps: Prompt Formatting, Chat Templates (ChatML / Llama-3 format) & Loss Masking on Prompts

04. Low-Rank Adaptation (LoRA) Mathematics: Intrinsic Rank Hypothesis, Matrix Factorization (W + B*A) & Alpha Scaling 05. QLoRA Deep Dive: 4-Bit NormalFloat (NF4), Double Quantization & Paged Optimizers for VRAM Efficiency 06. Target Modules Selection (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj) & Rank Tradeoffs

07. Hugging Face TRL (Transformer Reinforcement Learning) & SFTTrainer Architecture 08. Ultra-Fast Fine-Tuning with Unsloth (Manual Triton Kernels & 5x Speedup on Single GPUs) 09. Monitoring Loss Curves: Overfitting Detection, Validation Perplexity & Catastrophic Forgetting Mitigation

10. RLHF (Reinforcement Learning from Human Feedback) Architecture: Reward Models & PPO (Proximal Policy Optimization) 11. Direct Preference Optimization (DPO): Eliminating Reward Models via Implicit Policy Optimization Math 12. ORPO (Odds Ratio Preference Optimization) & KTO (Kahneman-Tversky Optimization) Alignment Techniques

13. Post-Training Quantization (PTQ): 4-Bit & 8-Bit Quantization Mathematics (Zero Point & Scale Factors) 14. Quantization Formats: GGUF (llama.cpp on CPU/Mac), AWQ (Activation-aware Weight Quantization) & GPTQ 15. Merging LoRA Adapters back into Base Model Weights for Zero-Inference Latency Overhead 16. High-Throughput Enterprise Model Serving with vLLM (PagedAttention, Continuous Batching & Tensor Parallelism)