Building a Simple Job Queue with Redis

Sending a welcome email, resizing an uploaded image, generating a PDF report — none of these should happen inside the HTTP request that triggered them. A job queue lets you hand the work off and respond to the user immediately, and Redis has just enough primitives built in to build a reasonably solid one without pulling in a heavier message broker.

Why not just do the work inline

The naive version does the slow work directly in the request handler:

app.post('/signup', async (req, res) => {
  const user = await createUser(req.body);
  await sendWelcomeEmail(user); // blocks the response on an SMTP round trip
  res.json(user);
});

If the email provider is slow or briefly down, every signup hangs or fails, even though creating the user succeeded. A queue decouples "the user was created" from "the email got sent" — the request returns fast, and a separate worker process handles delivery, with retries, independently.

The core primitive: a blocking list pop

Redis lists support LPUSH (add to the front) and BRPOP (block and pop from the back, waiting if the list is empty). That pair is a queue:

import { createClient } from 'redis';

const redis = createClient();
await redis.connect();

// Producer: enqueue a job
async function enqueue(queueName, job) {
  await redis.lPush(queueName, JSON.stringify(job));
}

// Consumer: block until a job is available, then process it
async function dequeue(queueName) {
  const result = await redis.brPop(queueName, 0); // 0 = block indefinitely
  return JSON.parse(result.element);
}

Usage:

await enqueue('emails', { type: 'welcome', userId: user.id });

// worker.js — runs as a separate process
while (true) {
  const job = await dequeue('emails');
  await sendWelcomeEmail(job.userId);
}

This works, but it has a serious gap: if the worker crashes after BRPOP removes the job from the list but before it finishes processing, that job is gone. BRPOP is destructive the instant it returns — there's no record that the job was ever picked up.

Making it reliable: the reliable-queue pattern

Redis's own documentation describes a pattern for this: atomically move the job from the main queue to a per-worker "processing" list, so it's never in a state where it exists nowhere.

async function dequeueReliable(queueName, processingKey) {
  const raw = await redis.brPopLPush(queueName, processingKey, 0);
  return raw ? JSON.parse(raw) : null;
}

async function ack(processingKey, rawJob) {
  await redis.lRem(processingKey, 1, rawJob);
}

while (true) {
  const raw = await redis.brPopLPush('emails', 'emails:processing', 0);
  try {
    const job = JSON.parse(raw);
    await sendWelcomeEmail(job.userId);
    await redis.lRem('emails:processing', 1, raw);
  } catch (err) {
    console.error('job failed, leaving in processing list for recovery', err);
  }
}

If the worker dies mid-job, the job stays sitting in emails:processing instead of vanishing. A separate recovery process — run periodically, or on worker startup — can scan that list for jobs older than some threshold and push them back onto the main queue for another attempt.

Adding retry with backoff

A job that fails shouldn't just come back around instantly and fail again in a tight loop. Track an attempt count in the job payload and use a delayed re-enqueue:

async function handleJob(raw, processingKey) {
  const job = JSON.parse(raw);
  try {
    await sendWelcomeEmail(job.userId);
    await redis.lRem(processingKey, 1, raw);
  } catch (err) {
    await redis.lRem(processingKey, 1, raw);
    job.attempts = (job.attempts ?? 0) + 1;

    if (job.attempts >= 5) {
      await redis.lPush('emails:dead-letter', JSON.stringify(job));
      return;
    }

    const delayMs = 1000 * 2 ** job.attempts;
    setTimeout(() => enqueue('emails', job), delayMs);
  }
}

Note the dead-letter queue: after enough failed attempts, stop retrying automatically and move the job somewhere a human can inspect it. Without this, a permanently broken job (bad data, a downstream service that's actually gone for good) retries forever and drowns out real work in the logs.

Scheduled / delayed jobs with a sorted set

LPUSH/BRPOP only handles "run as soon as possible." For "run this at a specific future time" — a reminder email, a subscription renewal check — a sorted set works better, using the scheduled timestamp as the score:

async function scheduleAt(queueName, job, runAtMs) {
  await redis.zAdd(`${queueName}:scheduled`, {
    score: runAtMs,
    value: JSON.stringify(job),
  });
}

A poller periodically moves due jobs into the regular queue:

async function promoteDueJobs(queueName) {
  const now = Date.now();
  const due = await redis.zRangeByScore(`${queueName}:scheduled`, 0, now);

  for (const raw of due) {
    await redis.multi()
      .zRem(`${queueName}:scheduled`, raw)
      .lPush(queueName, raw)
      .exec();
  }
}

setInterval(() => promoteDueJobs('emails'), 1000);

The multi() transaction matters here — it ensures a job is never removed from the scheduled set without also landing in the active queue, even if the process is interrupted between the two operations.

When to stop building this yourself

This gets you a genuinely useful queue, but production job systems also need things that take real effort to get right: job priorities, concurrency limits per queue, a dashboard for inspecting stuck jobs, exactly-once-ish delivery guarantees under Redis failover. At that point, reach for a maintained library — BullMQ is the standard choice in the Node ecosystem and implements all of the patterns above (and more) on top of Redis. Building a minimal version yourself first is worth doing once, though — it's the fastest way to understand what a job queue is actually responsible for, so you can evaluate whether a library's defaults fit your failure modes instead of trusting them blindly.