Kafka poison pill tools work at two layers: frameworks inside the consumer that isolate a bad record automatically (Spring Kafka’s error handlers, Kafka Streams exception handlers, Kafka Connect’s error tolerance, share group delivery limits), and operator tools that find and skip one after it stalls a partition (the Kafka CLI, kcat, Kpow, AKHQ, Kafbat UI, Conduktor Console).
Most production teams need one from each layer, because the framework only catches the failures it was written for. Kpow is Factor House’s product, and I work at Factor House as a Solutions Architect, so it is scored on the same rubric and the same sources as every other option. It sits in the second layer. A partition that is stuck right now is better served by how to find and skip a poison pill in Kafka, which has the commands in order. The complete Kafka guide covers the rest of the tooling picture.
What to score a poison pill tool on
A poison pill is a record a consumer can never process, so it fails on the same offset every poll and the partition stops. Five criteria separate the tools, and each comes from how these incidents actually unfold.
1. Does it isolate the record before the consumer loops?
The cheapest poison pill is one the consumer routes aside on its own. Deserialization failures are the hard case, because they happen inside poll(), before your code sees the record. Incremental cooperative rebalancing reduces the blast radius when a consumer fails, but a poison pill will still get a partition stuck, so isolation has to be built into the consumer or its framework.
2. Can it find the record, and say why it failed?
Kafka’s metrics do not tell you a record is malformed. They tell you one partition’s lag is climbing. A tool scores well here if it shows the specific offset, the deserialization error and the schema it was expected to match, without you writing a one-off consumer to read the bytes. This matters beyond poison pills. As I said in my talk on Kafka operational issues, data inspection tooling is essential for handling lost-message complaints, since the large majority of the time the message was on Kafka all along. When the failure is a decode error, how to diagnose a Kafka deserialization error covers finding its cause.
3. Can it skip the record safely?
Skipping means moving one partition’s committed offset forward by one, for one consumer group. The safe version previews the change, is scoped to a single partition, and only runs when the group has no active members. That last condition comes from Kafka, not from any tool. As Derek Troy-West, our co-founder and CEO, puts it, Kafka itself requires a consumer group to be stopped before any changes can be applied, and every tool works within what Kafka allows.
4. Does the record survive?
Skipping drops the record for that group. A tool scores well if it keeps a copy somewhere you can repair and replay it from. My view on the destination is that the steady-state goal for a dead letter queue is effectively zero messages, because every entry is a signal that something failed and you now pay an operational tax to unwind it. Where a topic has several consumers, a pattern that works well is a retry topic per topic and consumer pair, so a replay only reaches the consumer that failed. Services should be idempotent so duplicates are safe, but I have seen several multi-million dollar incidents caused by services that were not.
5. Who is allowed to do it, and is it recorded?
A change to a consumer group’s committed offset can silently drop or reprocess data if it is scoped wrong. Tom Crowley, our founding engineer, described the usual production path in 2021, a problem now covered in break-glass access with temporary policies: incrementing one offset means a VPN, a jumpbox, and “the right combination of bash commands against the Kafka cluster”, from a jumpbox that “generally has full access to the Kafka cluster” with “no audit log recording the actions being committed”. Tools that answer this well gate the skip behind a role or an approval and log who did it. The same question across every risky operation is covered in the best tools to control destructive Kafka operations.
Isolating poison pills inside the consumer
These frameworks act without a human, which makes them the first line of defence. Each only covers the application type it belongs to.
Spring Kafka error-handling deserializer and dead letter publishing. Spring’s documentation is direct about the gap: when a deserializer fails, “Spring has no way to handle the problem, because it occurs before the poll() returns.” The ErrorHandlingDeserializer, described in Spring Kafka’s serialization documentation, wraps the real deserializer and, on failure, returns a null value with the exception and raw bytes in a header. The DefaultErrorHandler, per Spring Kafka’s error handling documentation, then treats DeserializationException as fatal, skipping retries, and a DeadLetterPublishingRecoverer publishes the record to <originalTopic>-dlt on the same partition by default. For processing errors the handler retries and then logs the record after ten failures by default, unless you configure a recoverer.
Spring Kafka non-blocking retries. @RetryableTopic, covered in Spring Kafka’s non-blocking retries documentation, moves a failing record to back-off retry topics and finally a dead letter topic, so the main partition keeps flowing. The trade is ordering, because later records overtake the one being retried.
Kafka Streams exception handlers. The deserialization.exception.handler setting in the Kafka Streams configuration reference chooses LogAndFailExceptionHandler or LogAndContinueExceptionHandler, and processing.exception.handler does the same for errors in your topology. A dead letter topic means a custom handler, and the Apache documentation notes that those writes are side effects outside Streams’ processing guarantees.
Kafka Connect error tolerance. For sink connectors, errors.tolerance=all skips problematic records and errors.deadletterqueue.topic.name sends them to a DLQ topic, per the Kafka Connect configuration reference. errors.deadletterqueue.context.headers.enable defaults to false, and it is worth turning on so each DLQ record carries its error context.
Share groups. Queues for Kafka count delivery attempts per record. Once a record reaches group.share.delivery.count.limit (default 5, per the broker configuration reference), KIP-932 moves it to the Archived state instead of delivering it again. That is a broker-side answer, but only for consumers that use share groups.
Any of these that writes to a retry or DLQ topic needs a way out of its own loop. The failure to design against is an ETL that reads from and writes back to the same DLQ or retry topic, so every failure re-enters the loop and the backlog grows without end. A retry count header, checked before each re-publish, breaks it.
Poison pill tools compared
The table scores every option on the five criteria in order. Consumer-side frameworks come first, operator tools second.
| Tool | Isolates automatically | Finds the record and the error | Skips it safely | Keeps the record | Who can do it, and the record of it | Source |
|---|---|---|---|---|---|---|
Spring Kafka ErrorHandlingDeserializer + DeadLetterPublishingRecoverer |
Yes, for Spring listeners, including deserialization failures | The exception and raw bytes travel in headers | The container commits past the record once it is recovered | Yes, to <topic>-dlt |
Whatever deploys the application | Spring Kafka docs |
Spring Kafka @RetryableTopic |
Yes, for processing failures, through retry topics | Retry and DLT topics hold the record | The main partition keeps moving | Yes, in the final DLT | Whatever deploys the application | Spring Kafka docs |
| Kafka Streams exception handlers | Yes, log-and-continue or a custom handler | Logs, or a quarantine topic you write to | Log-and-continue skips inside the app | Only with a custom handler | Whatever deploys the application | Apache Kafka docs |
Kafka Connect errors.tolerance + DLQ |
Yes, for converter and transform errors in sink connectors | DLQ records carry error headers when enabled | Tolerated records are skipped by the task | Yes, on the DLQ topic | Whoever edits the connector config | Apache Kafka docs |
| Share groups (KIP-932) | Yes, after the delivery count limit | Records move to the Archived state | The broker stops delivering it | Archived, not copied to a topic | Broker and group configuration | Apache Kafka docs |
| kafka-console-consumer.sh + kafka-consumer-groups.sh | No | Yes, read one offset from one partition, then work out the error yourself | Yes, --shift-by 1 on topic:partition, dry run by default, group must be inactive |
Only if you copy the bytes yourself | Whoever holds CLI credentials, with no audit | Apache Kafka docs |
| kcat | No | Reads one offset and prints raw bytes, good for a hex dump | No offset reset | Can re-produce what you captured | Whoever holds credentials, with no audit | GitHub: edenhill/kcat |
| Kpow | No, it is an operator tool | Yes, Data Inspect’s Poison only mode lists records that failed to deserialize, with partition and offset | Yes, a partition-level Skip offset action, scheduled until the group is EMPTY | Yes, Clone to topic copies records byte for byte | RBAC, staged mutations for approval, and an audit log (Enterprise) | Kpow docs |
| AKHQ | No | Browses topic data in the UI | Per-partition offset update, gated by the UPDATE_OFFSET role |
Can produce records back to a topic | Roles, plus opt-in audit events to a Kafka topic | akhq.io |
| Kafbat UI | No | Browses messages in the UI | Offset reset, gated by the RESET_OFFSETS permission |
Can produce messages back to a topic | RBAC, plus an audit log to a topic or the console | Kafbat UI docs |
| Conduktor Console | No | Browses and consumes topic data in the UI | Offset reset scoped to topic-partitions, with shift-by and a preview, group must be inactive | Can produce records back to a topic | RBAC with a consumer group Reset permission, and an audit log | Conduktor’s documentation, Consumer groups, Topics, RBAC and Audit logs pages |
How the options score
On isolating the record automatically, Spring Kafka is the most complete framework, because the ErrorHandlingDeserializer closes the deserialization gap and the recoverer keeps the record. Kafka Connect is the only option that needs no code, and share groups are the only broker-side answer. None of the operator tools, Kpow included, stop a pill from stalling a partition in the first place.
On finding the record and the error, Kpow leads. Its Poison only mode returns nothing but the records that failed against the deserializer you chose, and Retain mode keeps them in normal results flagged as “Deserialization exception”. AKHQ, Kafbat UI and Conduktor show you a record once you know where to look. The CLI and kcat get you the bytes, and the diagnosis is yours.
On skipping safely, the CLI, Kpow, AKHQ, Kafbat UI and Conduktor can all move one partition’s offset. The CLI previews by default and Conduktor has a preview step. Kpow schedules the skip and applies it once the group is EMPTY, so it cannot fire against running consumers.
On keeping the record, the consumer-side frameworks win, because they write the DLQ record at the moment of failure with the error attached. Among operator tools, Kpow’s Clone to topic and the produce features in AKHQ, Kafbat UI and Conduktor all let you put a copy somewhere before skipping. What to do with the copies afterwards is compared in the best tools to manage a dead letter queue.
On governance, Kpow gives the most control: role-based permissions per action, staged mutations that need an administrator’s approval, temporary policies that grant GROUP_EDIT for a fixed window, and an audit log of every mutation. AKHQ and Kafbat UI both have roles and an opt-in audit trail you read from a Kafka topic. The CLI has none of this.
Each option in detail
Rank 1 Kpow
36 out of 50 Total
Listed first because it is our product. Scores are unadjusted.
- Type
- Kafka UI
- Find
- Data Inspect, Poison only mode
- Enterprise
- Staged mutations, audit log, bulk clone
- Isolates before the loop
- 0 out of 10
- Finds record and error
- 10 out of 10
- Skips it safely
- 9 out of 10
- Keeps the record
- 7 out of 10
- Who can do it, and the record
- 10 out of 10
What it does. Data Inspect finds malformed records across topics, Skip offset moves one partition past a bad record, Clone to topic keeps a byte-level copy, and every action runs through Kpow’s RBAC and audit log.
Where it wins. Finding the bad record and governing the skip. The Poison only mode and schema metadata turn “which record is it” into a query, and staged mutations turn “who is allowed” into a policy.
Where it falls short. It does not isolate anything automatically, so it complements a consumer-side framework rather than replacing one. Staged mutations, the audit log and bulk clone are Enterprise features.
Compare Kpow vs AKHQKpow vs Kafbat UI
Rank 2 Spring Kafka error handling
34 out of 50 Total
- Type
- Consumer-side framework
- Covers
- Spring listeners, deserialization included
- Dead letter topic
- <originalTopic>-dlt
- Isolates before the loop
- 10 out of 10
- Finds record and error
- 6 out of 10
- Skips it safely
- 7 out of 10
- Keeps the record
- 9 out of 10
- Who can do it, and the record
- 2 out of 10
What it does. ErrorHandlingDeserializer catches deserialization failures, DefaultErrorHandler retries or recovers, and DeadLetterPublishingRecoverer writes the failed record to a dead letter topic.
Where it wins. It is the most complete automatic path for JVM consumers, and it handles the deserialization case that plain consumers cannot.
Where it falls short. It only protects Spring applications, and the default -dlt resolver needs the dead letter topic to have at least as many partitions as the original.
Rank 3 Kafka Connect error tolerance
28 out of 50 Total
- Type
- Consumer-side framework
- Setup
- errors.tolerance=all, configuration only
- DLQ
- Sink connectors only
- Isolates before the loop
- 7 out of 10
- Finds record and error
- 5 out of 10
- Skips it safely
- 6 out of 10
- Keeps the record
- 8 out of 10
- Who can do it, and the record
- 2 out of 10
What it does. Skips records that fail conversion or transformation and, for sink connectors, writes them to a DLQ topic.
Where it wins. Configuration only, no code.
Where it falls short. It covers converter and transform errors, the DLQ is sink-only, and the context headers are off unless you enable them.
Rank 4 Conduktor Console
conduktor.io
26 out of 50 Total
- Type
- Commercial console
- Skip
- Topic-partition scope, shift-by, preview
- Isolates before the loop
- 0 out of 10
- Finds record and error
- 5 out of 10
- Skips it safely
- 8 out of 10
- Keeps the record
- 5 out of 10
- Who can do it, and the record
- 8 out of 10
What it does. A commercial console whose offset reset takes a topic-partition scope, supports shift-by, and previews the new offsets, and which requires the group to be inactive, per Conduktor’s documentation.
Where it wins. The reset flow’s scope and preview are as careful as the CLI’s, in a UI.
Where it falls short. Like every operator tool, it acts after a partition has stalled, so it needs a consumer-side framework alongside it.
AKHQ and Kafbat UI
akhq.io, github.com/kafbat/kafka-ui
23 out of 50 Total
- Type
- Open-source Kafka UIs
- Skip permission
- UPDATE_OFFSET (AKHQ), RESET_OFFSETS (Kafbat UI)
- Isolates before the loop
- 0 out of 10
- Finds record and error
- 5 out of 10
- Skips it safely
- 6 out of 10
- Keeps the record
- 5 out of 10
- Who can do it, and the record
- 7 out of 10
What they do. Open-source UIs that browse topic data, change consumer group offsets and produce records, each with role-based permissions and an opt-in audit trail written to Kafka.
Where they win. Free, and enough to find a record by offset and skip it without a jumpbox.
Where they fall short. Finding the bad record on a busy topic starts from its offset, which you get from the consumer’s error or the CLI. The audit trail is a Kafka topic you have to read and retain yourself.
Compare Kpow vs AKHQKpow vs Kafbat UIAKHQ vs Kafbat UIAKHQ reviewKafbat UI review
Rank 6 Kafka Streams exception handlers
21 out of 50 Total
- Type
- Consumer-side framework
- Handlers
- LogAndFail or LogAndContinue
- Dead letter topic
- Custom handler you write
- Isolates before the loop
- 7 out of 10
- Finds record and error
- 3 out of 10
- Skips it safely
- 6 out of 10
- Keeps the record
- 3 out of 10
- Who can do it, and the record
- 2 out of 10
What it does. Chooses whether a record that fails to deserialize, process or produce stops the application or is logged and skipped.
Where it wins. It is built in, and log-and-continue keeps a topology running through bad data.
Where it falls short. Log-and-continue discards the record. Keeping it means a custom handler writing to a quarantine topic, outside Streams’ processing guarantees.
Rank 7 Share groups
18 out of 50 Total
- Type
- Broker-side delivery limit
- Setting
- group.share.delivery.count.limit, default 5
- Scope
- Share group consumers only
- Isolates before the loop
- 6 out of 10
- Finds record and error
- 1 out of 10
- Skips it safely
- 7 out of 10
- Keeps the record
- 2 out of 10
- Who can do it, and the record
- 2 out of 10
What it does. Counts deliveries per record and archives a record that reaches the limit.
Where it wins. The broker enforces it, so a poison pill cannot loop forever.
Where it falls short. It only applies to share group consumers, and an archived record is not copied anywhere you can repair it from.
Kafka CLI and kcat
kafka.apache.org, github.com/edenhill/kcat
16 out of 50 Total
- Type
- Command-line tools
- Skip
- --reset-offsets --shift-by 1
- Safety net
- Dry run by default
- Isolates before the loop
- 0 out of 10
- Finds record and error
- 4 out of 10
- Skips it safely
- 7 out of 10
- Keeps the record
- 4 out of 10
- Who can do it, and the record
- 1 out of 10
What they do. kafka-console-consumer.sh or kcat reads the record at an offset, and kafka-consumer-groups.sh --reset-offsets --shift-by 1 skips it.
Where they win. Always available, and the dry-run default on offset resets is a real safety net.
Where they fall short. No record of who ran what, and the error diagnosis is manual. The --topic <topic> form without a partition shifts every partition, a mistake that is easy to make under pressure.
How Factor House approaches poison pills
Kpow’s Data Inspect came out of this problem. Tom built its three deserialization options around a prospect whose topics mixed several schema types in one topic, including records that conformed to no schema. Drop record is the default and matches a plain consumer. Retain record keeps failed records in the results, flagged as “Deserialization exception”, so you can see which partition and offset failed. Poison only shows nothing else. When only certain schema IDs belong on a topic, a kJQ filter that asserts those IDs surfaces every record outside them. The Data Inspect documentation covers the modes and Clone to topic, and Clone to topic for DLQs explains the replay side.
The skip itself is the Skip offset action, which “increments the current offset by 1” at the partition level and waits for the group to reach EMPTY. On a governed cluster, staged mutations put an administrator’s approval in front of it, and the audit log records the request and the policies it was checked against. Kpow Community Edition includes consumer offset management and Data Inspect, so skipping a poison pill does not require an Enterprise licence. In the live Kpow demo you can try the finding half of the job: run Data Inspect against a topic, switch the deserialization option to Retain or Poison only, and open a consumer group to see the offset actions on each partition.
Kpow live demo
Test poison pill handling on a live cluster
Open the Kpow demo to inspect records that fail deserialization and see how a group offset change is scoped and audited.
Built for platform and data engineers running Kafka in production.
Try the Kpow demoFAQ
What is the best tool for handling poison pills in Kafka?
Use a consumer-side framework to isolate them automatically, such as Spring Kafka’s ErrorHandlingDeserializer with a dead letter recoverer or Kafka Connect’s error tolerance. Add an operator tool for the ones that get through. Kpow is strongest at finding the malformed record and governing the skip, and the Kafka CLI is enough if one engineer owns the cluster and no audit is required.
Can Kafka skip a bad message automatically?
A plain consumer cannot. Spring Kafka, Kafka Streams and Kafka Connect can be configured to skip or dead-letter a failing record, and share groups archive a record after its delivery count limit. Otherwise someone has to move the committed offset with kafka-consumer-groups.sh or a UI.
Does Kafka have a built-in dead letter queue?
Only in Kafka Connect, for sink connectors. Everywhere else a DLQ is a pattern in your consumer code or framework. Dead letter queues in Kafka: patterns and pitfalls compares the implementation paths.
Do I need to stop my consumers to skip a poison pill?
Yes, for any offset change. Kafka only accepts new committed offsets for a group with no active members, whichever tool you use. Kpow schedules the change and applies it once the group is empty.