15. Exactly-Once Processing Semantics (EOS), Idempotency Keys & Deduplication Architecture • High-Level System Design (HLD): Distributed Systems at Scale • Rishabh Ki Kaksha

High-Level System Design (HLD): Distributed Systems at Scale › 15. Exactly-Once Processing Semantics (EOS), Idempotency Keys & Deduplication Architecture

45 mins

← Prev Next →

Course Syllabus

Module 1: Distributed Systems Core & Scalability Fundamentals

01. Scalability Dimensions: Vertical Scaling vs Horizontal Scaling & Latency vs Throughput 02. CAP Theorem vs PACELC Theorem: Consistency, Availability & Partition Tolerance Tradeoffs 03. Consistency Models: Strong Consistency, Eventual Consistency, Read-Your-Writes & Monotonic Reads

Module 2: Networking, Load Balancing & Reverse Proxies

04. Layer 4 vs Layer 7 Load Balancing: Round Robin, Weighted, Least Connections & IP Hash 05. High-Availability Load Balancers: Keepalived, VRRP, Anycast DNS & Failover Clustering 06. API Gateway Pattern: Authentication, Rate Limiting, Request Routing & Circuit Breakers

Module 3: Database Scaling: Sharding, Replication & Distributed Transactions

07. Database Replication: Master-Slave vs Multi-Master & Synchronous vs Asynchronous Replication 08. Database Sharding: Horizontal Partitioning, Directory-Based Sharding & Rebalancing Hotspots 09. Distributed Transactions: Two-Phase Commit (2PC) vs Saga Pattern (Choreography vs Orchestration)

Module 4: Caching Architectures, CDN & Consistent Hashing

10. Caching Strategies: Cache-Aside, Read-Through, Write-Through, Write-Around & Write-Back 11. Cache Invalidation Nightmares: Cache Avalanche, Cache Stampede, Thundering Herd & Cache Penetration 12. Consistent Hashing with Virtual Nodes (Solving Cache Clustering & Dynamic Scaling) 13. Content Delivery Networks (CDN): Edge Caching, Dynamic Site Acceleration & Origin Shielding

Module 5: Message Brokers, Event-Driven Architecture & Stream Processing

14. Message Queues vs Event Streams: RabbitMQ vs Apache Kafka (Partitioning, Consumer Groups) 15. Exactly-Once Processing Semantics (EOS), Idempotency Keys & Deduplication Architecture 16. Distributed Unique ID Generators: Twitter Snowflake, UUIDv7 & Database Ticket Servers

Module 6: FAANG Classic HLD Architectural Case Studies

17. System Design Case Study 1: URL Shortener (TinyURL / Bitly) at 100M Daily Requests 18. System Design Case Study 2: Distributed Rate Limiter & Web Application Firewall 19. System Design Case Study 3: Real-Time Chat & Notification System (WhatsApp / Slack) 20. System Design Case Study 4: Social Media News Feed (Twitter/Instagram) with Fan-out on Write/Read 21. System Design Case Study 5: Video Streaming Platform (YouTube / Netflix) Transcoding Architecture 22. System Design Case Study 6: Ride Hailing Geospatial Dispatch Architecture (Uber / Ola)

Interactive Lesson Studio

# 15. Exactly-Once Processing Semantics (EOS), Idempotency Keys & Deduplication Architecture

> High-Level Design (HLD) Notes by Rishabh - Topic 15

---

### 1. The 3 Messaging Delivery Guarantees 📦

1. **At-Most-Once:** Messages leak/lose ho sakte hain, par duplicate kabhi nahi honge.
2. **At-Least-Once (Industry Default):** Zero message loss guarantee, lekin network retry hone par duplicate delivery ho sakti hai.
3. **Exactly-Once Semantics (EOS):** Har message **strictly aur exactly 1 baar** process hota hai!

---

### 2. Idempotency Key Pattern (Stripe Payment Safety Standard 💳)

> **Idempotency Definition:**
> Ek operation ko chahe 1 baar execute karo ya 100 baar, **final state of the system hamesha exact same rehta hai!**

```
Client ➔ POST /charge (Idempotency-Key: "req_xyz123")
   ⬇
[ Redis / DB Check ] ➔ Key exists?
   ├── YES ➔ Return cached successful charge response instantly! (No double deduction!)
   └── NO  ➔ Lock key ➔ Execute bank charge ➔ Save result in Redis ➔ Return
```

---

### 3. Master Comparison Matrix

---

### 4. Interview Ke 3 Sawal Is Topic Se

1. **Distributed Database me Idempotency Keys kitni der tak retain karni chahiye?** -> Transaction lifespan ke anusaar standard 24 hours se 7 days tak Redis TTL ke sath.
2. **Database Insert operations me natural idempotency kaise achieve hoti hai?** -> Unique constraints (`UNIQUE INDEX (payment_reference_id)`) aur `ON DUPLICATE KEY UPDATE` / `INSERT ... ON CONFLICT DO NOTHING` statements use karke.
3. **Kafka Exactly-Once Semantics (EOS) internally kaise kaam karta hai?** -> Producer Idempotence (Sequence numbers per partition) + Transactional Coordinator 2PC markers in downstream consumer topics.

---
**Next Topic:** 16. Distributed Unique ID Generators: Twitter Snowflake, UUIDv7 & Database Ticket Servers

Loading lesson content...

Python 3.12

Full IDE

Live Execution Terminal

Click "Execute Live" to run code.
                

Lesson Mastery AI Quiz

AI Doubt Solver

CERTIFICATE OF COMPLETION

Rishabh Gehlot

PHP 8+ & Laravel 11 Master Track