17. Asynchronous Task Queues with Redis & BullMQ (Retries, Backoff, Dead Letters) • Node.js Backend Architecture: Libuv to High-Scale Microservices • Rishabh Ki Kaksha

Agar registration endpoint par:
1. DB me user save karo (20ms).
2. Welcome Email bhejo (2000ms).
3. PDF Invoice generate karo (3000ms).
4. Video avatar thumbnail resize karo (5000ms).
User ko response aane me **10+ seconds lagenge** aur agar SMTP server down hua toh registration fail ho jayegi!

**The Asynchronous Task Queue Pattern:**
> HTTP Request turant `201 Created` return karti hai (20ms), aur heavy tasks **Redis Queue (BullMQ)** me push ho jaate hain jise background worker processes consume karte hain!

---

### 2. BullMQ Producer-Consumer Architecture

```typescript
import { Queue, Worker } from "bullmq";
import Redis from "ioredis";

const connection = new Redis(process.env.REDIS_URL, { maxRetriesPerRequest: null });

// 1. PRODUCER (API Server):
export const emailQueue = new Queue("emailQueue", { connection });

export async function queueWelcomeEmail(user: { id: string; email: string }) {
    await emailQueue.add(
        "sendWelcome",
        { email: user.email, userId: user.id },
        {
            attempts: 5,                  // Retry up to 5 times
            backoff: { type: "exponential", delay: 5000 }, // 5s, 10s, 20s, 40s...
            removeOnComplete: true
        }
    );
}

// 2. CONSUMER WORKER (Dedicated Background Worker Process):
const emailWorker = new Worker("emailQueue", async (job) => {
    console.log(`Processing job ${job.id} for ${job.data.email}...`);
    await mailer.send(job.data.email, "Welcome to Platform!");
}, { connection, concurrency: 10 }); // Handles 10 parallel jobs!

emailWorker.on("failed", (job, err) => {
    console.error(`❌ Job ${job?.id} failed after all retries:`, err);
    // Automatically sent to Dead Letter Queue (DLQ)!
});
```

---

### 3. Delayed Jobs & Recurring Cron Tasks

```typescript
// Send a reminder email after 24 hours:
await emailQueue.add("sendReminder", { userId: "101" }, { delay: 24 * 60 * 60 * 1000 });

// Recurring Cron Job (Every Sunday at midnight):
await emailQueue.add("weeklyDigest", {}, { repeat: { pattern: "0 0 * * 0" } });
```

---

### 4. Master Comparison Matrix

---

### 5. Interview Ke 3 Sawal Is Topic Se

1. **Dead Letter Queue (DLQ) kya hota hai?** -> Wo queue jisme wo jobs transfer ho jaate hain jo saare retry attempts fail hone ke baad bhi successfully process nahi ho sake, taaki engineers unhe debug karke manually re-run kar sakein.
2. **BullMQ background me Redis ke kaunse data structures use karta hai?** -> Redis Streams, Sorted Sets (`ZSET` for delayed/scheduled jobs) aur Hashes (`HSET` for job payloads).
3. **Producer aur Worker ko alag-alag repositories/servers me kyu deploy kiya jata hai?** -> Taaki background heavy processing API server ke CPU/RAM ko degrade na kare aur dono ko independently scale kiya ja sake.

---
**Next Topic:** 18. Microservices Architecture: gRPC Protocols with Protocol Buffers in Node.js

Loading lesson content...

JavaScript

Full IDE

Live Execution Terminal

Click "Execute Live" to run code.
                

Lesson Mastery AI Quiz

AI Doubt Solver

CERTIFICATE OF COMPLETION

Rishabh Gehlot

PHP 8+ & Laravel 11 Master Track