Dead letter queues in Kafka: a consumer-side, Kafka-only approach

Introduction

A dead letter queue (DLQ) is a topic where messages that cannot be processed are parked instead of blocking the consumer or being dropped. The consumer keeps moving, and you keep a durable copy of the failure to inspect and replay later.

Kafka Connect and Kafka Streams ship built-in dead letter handling. Connect routes failed records to a configured DLQ topic, and Streams lets you handle deserialization and production exceptions and route them yourself. Standard Kafka consumers have no such thing. And unlike SQS, JMS, RabbitMQ, or similar systems where the broker handles dead lettering for you, in Kafka it is an application-level pattern. This fits Kafka’s design: a dumb broker and a smart consumer. The broker stores and serves messages. What to do with a bad message is the consumer’s responsibility.

The DLQ is a common and well-established pattern. Confluent, Uber, and others document their own versions of it, and most Kafka client frameworks ship some form of it out of the box. What follows is the version I use and recommend, and the opinions I hold about where it applies and where it does not.

In the examples, I use a topic called payments, and two consumer services called notifier and reconciler.

Scope

This article is about one specific shape of the pattern. Two things are out of scope, and two more are deliberate constraints on the approach.

Queues for Kafka. I am deliberately skipping the recent release of Queues for Kafka. I’ll discuss this in a future article, and perhaps call out where you should consider them below.

Kafka only. Ideally, retry and DLQ topics are plain Kafka topics. I do not reach for a database, SQS, or any other store to hold failed messages. The reason is fewer moving parts. A Kafka-only design uses the tooling, monitoring, retention, and ordering semantics you already run. Some guides recommend evolving toward a durable external retry store. For true failures that buys you very little. An unparsable message does not become parseable because it is sitting in Postgres. You pay for a whole new piece of infrastructure, management overhead, and tooling to support it, but get nothing back.

Consumer-side only. I do not use producer-side DLQs. A publish to a healthy broker should succeed. If it does not, that is a broker availability problem, not a dead letter case. A producer should not decide what a downstream consumer can or cannot process; those systems are decoupled on purpose. A DLQ handles messages that arrive at a consumer and cannot be processed. It does not handle messages that never arrived. The producer’s job is to produce valid, well formed data.

The named pattern for produce-path failure is store and forward: the producer stages messages elsewhere and publishes once the broker returns. That is a real pattern, but it solves a different problem, and it is not a DLQ.

No cascading retry topics. Some designs chain a message through several retry topics with increasing delays before it reaches the DLQ. I do not use them. So as not to distract from the main article, the full reasoning is in the addendum at the end. If you are able to justify their use, then who am I to say no? I would be hard to convince however that the added complexity and risk is worth it.

DLQs are exceptional

First and foremost. While a handy, and often necessary tool, DLQs should be considered exceptional. If you are managing DLQs every day, you should spend time investigating why messages end up on the DLQ, and fix those underlying issues instead. That said, DLQs exist, and I want to make their use as efficient as possible. That is the point of this article. The best DLQ is one that is never used.

When not to use a DLQ

DLQs break ordering. Replaying a message processes it well after its original position in the log, so ordering on arrival is lost. If order is part of the contract, a DLQ violates that contract every time it replays. This is not controversial; even articles that advocate heavily for retry and DLQ pipelines concede it. If you need strict ordering, do not replay from a DLQ. Rely on head-of-line blocking, or redesign.

You can still use a DLQ to stop an unparsable message from blocking the partition, but you give up replaying it. In that case the DLQ is informational only: a durable record that a message could not be processed.

How many extra topics do you need?

In a large system, a lot. But it is affected by the semantics you choose. Using my approach the answer is two topics per consumer group per topic it consumes: one retry topic and one DLQ. The main topic is shared across groups, so the multiplier is the number of consumer groups, not topics. Two things drive the total: whether you use a retry topic for replay, and how many consumer groups you run.

Replay through a retry topic, not the main topic. You do not strictly need a retry topic. You could republish a failed message straight back to the main topic. But the main topic fans out to every consumer group subscribed to it, so every consumer reprocesses the message. That is duplicate processing, and it causes incidents and additional workload. A single poorly behaving consuming application will have real world effects on the rest of your applications.

The retry topic exists for one reason: it is a replay channel that only the originally failed consumer reads. It lets you re-drive a message through one consumer without touching the others.

Scope per consumer, not per topic. Apply the same logic to the DLQ and the retry topic themselves: each consumer group gets its own, named for the consumer, not just the topic: payments.notifier.dlq and payments.reconciler.dlq not payments.dlq.

There are two reasons. The first is ownership. A shared DLQ cannot tell you which consumer failed, so you cannot alert or triage cleanly. The second is the same fan-out problem one level down. A message lands in the DLQ because one consumer couldn’t process it. The others most likely processed it fine. If the DLQ and its replay path are shared across consumers, replaying that message pushes it back at consumers that already handled it. Same duplicate processing, same incidents.

Some guides say “use per-topic DLQs.” They are arguing against a single global DLQ for the whole system. I am making the finer distinction one level down: per consumer group, not per topic. The replay fan-out problem is why I go that extra level.

There is a cost to this. Every topic and every partition in Kafka carries a base cost in broker resources, metadata, and replication. A retry topic and a DLQ per consumer group means more topics and more partitions than running with a main topic alone, and that increases rapidly as you add consumers. The isolation and the simpler per-consumer reasoning are what you buy with it. That is the trade: more partitions to run against cleaner ownership and safer replay.

A real-world implementation

Here is a setup that follows directly from those principles. It resembles the Spring retry-topic pattern, and similar designs show up across the industry.

Every topic-consumer gets its own retry topic and its own DLQ. Each consumer reads from both the main topic and its retry topic. If your organisation already ships a Kafka client library wrapper, the second consumer can be added automatically, so application code does not have to think about it. It can also be done more simply by subscribing to both topics on a single consumer:

consumer.subscribe(List.of("payments", "payments.notifier.retry"));

The alternative is to create two consumers, which call the same dispatch method.

mainConsumer.subscribe(List.of("payments"));
retryConsumer.subscribe(List.of("payments.notifier.retry"));
// both dispatch to the same handler

Which of the two you choose is a tradeoff you need to make. A single consumer subscribing to both topics is simpler and cheaper to run, but it couples the two streams under one poll loop, one max.poll.interval.ms budget, and one rebalance fate. There is no priority between topics, so slow or repeatedly failing retry processing delays the main topic, and a single slow batch can blow the poll interval and trigger a rebalance that drops the main topic’s partitions too. You also tune both streams the same way.

Two consumers cost more to run in terms of infrastructure and code complexity, but break that coupling: independent poll loops, failure isolation, independent tuning of concurrency and backoff, and a rebalance on one topic does not disturb the other. Low volume with fast, cheap processing favours the single consumer. Once retry processing can be slow or bursty, split them. In my opinion, retries are infrequent and low volume compared to the main topic, so a single consumer should not be a problem. If 50% of your messages are retries, you have much bigger issues to diagnose first.

A message that cannot be parsed or processed is sent to the DLQ. If it is unparsable, the client library should dead letter it automatically. An engineer can also dead letter a message explicitly by calling a deadletter() (or similar) method on the wrapper, for whatever reason they decide.

To replay, the message is copied from the DLQ to the retry topic. It is never republished to the main topic. This is the whole point of the retry topic: other consumers of the main topic do not see the replayed message and do not reprocess it.

A note on idempotency. The event-driven purist (which I aim to be when practical) will say consumers should process idempotently and handle duplicates, so republishing to the main topic should be safe. If a service has already processed a message, it should either ignore the duplicate, or process it, but leave the system with exactly the same state. In practice that is often not true. Republishing the same message to a topic that multiple services consume can cause significant incidents if your systems are not idempotent. I have seen this cause significant system damage, and in some cases monetary and reputational damage to organisations. The per-consumer scoping and the retry topic both exist because of that reality, not in spite of it.

Retry topics for transient failures are considered an anti-pattern

A DLQ and a retry topic should only handle true failures: messages that cannot be parsed or processed. They should not be used to ride out a temporary downstream outage. In this case, I am referring to a downstream system as one which I am dependent upon to process the message in real time. This generally means a system I call via HTTP, gRPC, or similar, though it could also be a DynamoDB table or other datastore embedded in your system. If you are exceeding DynamoDB (or similar) quotas, you should not use a retry queue. You should increase your DDB quota, or throttle your Kafka consumption. Using retries to handle a downstream that is briefly unavailable is a common recommendation, and it is wrong.

A message can either be processed or it cannot. If it cannot be processed because a downstream system (invoked by real-time rpc) is down, the state of the message has not changed. Only the state of the downstream has. Retrying does not fix that, and storing the retry for later makes it worse.

Consider the two failure modes.

If the downstream is fully down, you cannot process anything until it returns. Head-of-line blocking is fine here. It is backpressure, and it is a decoupled system working as intended. You lose nothing by waiting, and you keep your ordering intact.

If the downstream is partially down, usually because it is under load, then saving up retries and re-driving them later piles more work onto a system that is already struggling. You kick it while it is down, and likely make the issue worse. The correct response is to throttle or back off and keep processing in order, not to shovel failures into a retry topic to replay in a burst later.

If downstream failures are a real and recurring concern, the better fix is structural: put Kafka in front of that integration too, rather than calling the downstream synchronously. Publish to a topic the downstream consumes at its own pace. A downstream outage then shows up as consumer lag, not as processing failures that need retry and DLQ handling. The downstream catches up when it returns, and order is preserved the whole time. This is one of the main reasons to use Kafka in the first place. It decouples your systems and absorbs outages instead of propagating them.

In both cases the answer is the same. Slow down and preserve order. Do not dead letter a healthy message because something downstream is temporarily unavailable.

This section deserves an article of its own, and that will follow shortly.

Managing and re-driving messages

Once you run DLQs, the daily question is operational: how do you see what is in them, and how do you replay safely?

The baseline answer is unpleasant. Inspecting a DLQ means CLI scripts against the topic. Replaying means reading records off the DLQ and producing them onto the retry topic, in order, while tracking what you have already re-driven so you do not send the same records twice. Done by hand it is slow and error-prone, and it is easy to replay too much or double-process. Before you replay anything, confirm the underlying fault is actually fixed.

You will need to build tooling for this, and you can lean on some Kafka features to help. Stand up a separate consumer for the DLQ. It can replay a single message, or as many as you like, from the DLQ onto the retry topic. Commit the DLQ consumer’s offset to track which messages have been replayed. The last catch is the retry count: you have to track it yourself. I put it in headers, and will cover that in an upcoming CloudEvents article. While you are at it, capture a failure reason if you can. Put that in the header too.

Replay tooling is tricky to get right. You can build it yourself, but Factor House is building it into Kpow as a first-class tool. Today you can start with the Clone to Topic feature, and monitoring, alerting, and replay management are coming.

Closing

A DLQ is a small, deliberate tool. Consumer-side, inside Kafka, scoped to a consumer group, and reserved for messages that will never process. Everything else is throttling and backoff against consumer lag, not a queue of retries. Keep it that boring and it stays an asset instead of a liability.

Addendum: cascading retry topics

A cascading retry setup chains a message through a series of retry topics with increasing delays. A failed message goes to a short-delay topic, then a longer one, then a longer one again, and only lands in the DLQ once every tier is exhausted. The Spring and Uber designs are the well-known examples.

I do not use them, for three reasons.

First, they exist to solve transient downstream failures. That is the problem I argued against earlier. If retrying a healthy message because a downstream is briefly down is the wrong move, then a multi-tier machine built to do exactly that, on a schedule, is more wrong, not less. The right response to a struggling downstream is to throttle or back off and keep order, not to spray graduated retries at it.

Second, the complexity and cost are real and multiply quickly. Each tier is another topic, another consumer, more partitions, more monitoring, more network traffic, more disk, and another place a message can sit. Because everything is scoped per consumer group, every tier you add is paid for once per consumer. The operational surface grows quickly for a mechanism that should rarely fire.

Third, it blurs what the DLQ is for. A retry topic and a DLQ should be for true dead letters: messages that cannot be parsed or processed. Cascading retries turn the retry path into a backoff and scheduling mechanism for downstream availability, which is a different job. Keep downstream availability in the domain of backpressure and throttling. Keep the retry and DLQ path for bad messages. Again, this section could probably do with its own article, and that is on my backlog.

Further reading

• Kai Waehner, “Error Handling via Dead Letter Queue in Apache Kafka”. Broad overview of DLQ options, and makes the throttling-over-retry point for downstream timeouts.
• Uber Engineering, “Building Reliable Reprocessing and Dead Letter Queues with Apache Kafka”. The canonical tiered retry-topic design.
• Baeldung, “Dead Letter Queue for Kafka With Spring”. Practical Spring implementation walkthrough.
• kafkabr, “Como implementar Dead Letter Topic com Spring Kafka”. Spring DLT implementation (Portuguese).
• Confluent, “Kafka Connect Deep Dive: Error Handling and Dead Letter Queues”. The Connect-level DLQ, and the fail-fast versus route-and-continue decision.
• Confluent, “Apache Kafka Dead Letter Queue: A Comprehensive Guide”. Monitoring, bounded retries, and schema-registry prevention.
• Spring for Apache Kafka, “Non-Blocking Retries: How the Pattern Works”. The framework reference for the retry-topic pattern this setup resembles.
• AxonOps, “Dead Letter Queues”. Frames Kafka DLQs as a message-quality concern under the dumb-broker, smart-consumer model.
• Mansi Bhadani, “Building a Dead-Letter Queue for Your Kafka Pipeline the Right Way”. Separates messages that arrived and failed from messages that never arrived.
• Michael Diggin, “Side Stepping Head of Line Blocking in Kafka Consumers”. Consumer-library wrapper approach and ordering-preserving throughput.
• Chayan Mann, “Building a Reliable Retry Mechanism in Kafka”. Argues for dedicated retry topics, and concedes the ordering and idempotency costs.

Related reading

• Clone to topic for DLQs in Apache Kafka