Task Queues & Background Jobs

Why some work doesn't belong in the request cycle. Retries, DLQs, idempotency.

Why Background Jobs?

Some work is too slow for a request-response cycle. Sending an email might take 500ms. Generating a PDF might take 5 seconds. Resizing 20 images might take 30 seconds.

Making the user wait → bad UX.
Making the server block → resource waste.

Solution: Enqueue the work. Return 202 Accepted immediately. Process in the background asynchronously.

Also useful for:
• Retry logic (if email sending fails, retry 3x with backoff)
• Rate limiting outbound API calls
• Fan-out operations (send notification to 10,000 users)
• Periodic tasks (cleanup old data, send daily digests)
• Decoupling services (service A doesn't call service B directly)

Message Queue Fundamentals

A message queue is a buffer between producers (who enqueue work) and consumers (workers who do the work).

Producer → [Queue] → Worker

Properties:
• FIFO ordering (usually) — first in, first out
• At-least-once delivery — message delivered at least once (may be duplicated)
• Acknowledgement — worker signals "done" so queue removes the message
• Dead Letter Queue (DLQ) — failed messages go here after max retries

Popular tools:
• Redis (BullMQ, Sidekiq) — simple, great for most cases
• RabbitMQ — advanced routing, exchange patterns
• Apache Kafka — event streaming, massive throughput, message replay
• AWS SQS / GCP Pub-Sub — managed, serverless

Job Retry & Dead Letter Queues

Jobs fail. Networks drop. External APIs timeout. Design for failure.

Retry strategy:
1. Job fails → move to retry queue
2. Wait with exponential backoff: 5s → 25s → 125s → ...
3. After max retries (e.g., 5), move to Dead Letter Queue
4. DLQ items can be inspected, reprocessed manually, or alerted on

// BullMQ example
const queue = new Queue("emails");
queue.add("send-welcome", { userId: "123" }, {
  attempts: 5,
  backoff: { type: "exponential", delay: 5000 }
});

Idempotency: Design jobs to be safe to run twice. If a job runs twice due to at-least-once delivery, the second run should not cause double-charging, double-emails, etc.

Use idempotency keys: "I already processed this job ID, skip."

Scheduled Jobs (Cron)

Cron-like jobs run on a schedule. Daily reports, cleanup tasks, cache warming.

Cron expression syntax: * * * * *
                         │ │ │ │ └── day of week (0-7, Sun=0/7)
                         │ │ │ └──── month (1-12)
                         │ │ └────── day of month (1-31)
                         │ └──────── hour (0-23)
                         └────────── minute (0-59)

"0 9 * * 1" → 9 AM every Monday
"0 0 1 * *" → Midnight on 1st of every month
"*/15 * * * *" → Every 15 minutes

Pitfalls:
• Multiple instances: if you have 3 servers, the cron fires 3 times. Use distributed locking or a dedicated scheduler (Kubernetes CronJob, Celery Beat).
• Timezone awareness: use UTC for server crons, convert to user timezones in logic.
• Overlapping: if job takes 2 hours and runs hourly, overlap happens. Detect and skip.