The alert says consumer lag is climbing, but only on one partition. The group’s committed offset for that partition has not moved in twenty minutes, every other partition is near zero, and the consumer logs repeat the same line on every poll:
org.apache.kafka.common.errors.RecordDeserializationException: Error deserializing VALUE
for partition orders-3 at offset 18442. If needed, please seek past the record to continue consumption.A poison pill is a record the consumer can never process, so it fails, retries the same offset, and fails again, and every record behind it on that partition waits. The fix is to confirm that is what you are looking at, capture the record, and move the group’s committed offset past it. Each step below ends with the command that proves it. The rest of the troubleshooting material is in the complete Kafka guide.
Skip the poison pill and get the partition moving
Step 1: confirm one partition is stuck on one offset
Describe the group twice, a minute or two apart:
kafka-consumer-groups.sh --bootstrap-server <broker:9092> \
--describe --group <group>A poison pill looks like this: one partition’s CURRENT-OFFSET is identical in both runs while its LAG grows, and the other partitions keep advancing. Two lookalikes are worth ruling out before you skip anything.
A slow consumer is still making progress. If CURRENT-OFFSET moved at all between the two runs, it is not a poison pill, and skipping a record will not help. The consumer needs more capacity or a faster processing path.
A rebalance loop shows up in the group’s state rather than in one offset. Check it:
kafka-consumer-groups.sh --bootstrap-server <broker:9092> \
--describe --group <group> --stateIf the state keeps flipping to rebalancing and the member list churns, a record may still be the trigger, but the mechanism is different. I have seen a team set max.poll.interval.ms to 20 minutes so their consumer could process large pre-built batches. It ran for twelve months until one batch took longer than the window. The coordinator declared the consumer dead, a rebalance handed the partition to another consumer, that consumer tried the same batch, timed out, and the cycle repeated at around 2am. That batch was effectively a poison pill, and skipping it breaks the loop exactly as it would for a malformed record.
Step 2: read the partition and offset from the error
The Java consumer’s RecordDeserializationException names the partition and offset in its message, as in the log line above: partition 3 of orders, offset 18442. Application exceptions thrown from your own processing code usually do not, so log record.partition() and record.offset() in your error handler before you need them. If you have neither, the CURRENT-OFFSET from Step 1 is the committed position the group will resume from, which is the record it keeps failing on.
Step 3: capture the record before you skip it
Skipping is permanent for that consumer group. Save the record’s bytes first, using the commands in the next section, so you can fix the producer or replay it later. If there is a dead letter topic in your design, copy the record there.
Step 4: stop the group, dry-run the reset, then execute
Kafka only accepts an offset change for a group with no active members. The Apache Kafka operations documentation says to “first make sure that the consumer instances are inactive”, and the Admin API’s alterConsumerGroupOffsets states that “the group must be empty”. Scale the consumers to zero, then plan the reset scoped to the one partition. Without --execute, the command only prints the plan:
kafka-consumer-groups.sh --bootstrap-server <broker:9092> \
--group <group> --topic orders:3 \
--reset-offsets --shift-by 1 --dry-runThe NEW-OFFSET column should read 18443. If it does, apply it:
kafka-consumer-groups.sh --bootstrap-server <broker:9092> \
--group <group> --topic orders:3 \
--reset-offsets --shift-by 1 --execute--to-offset 18443 does the same thing when you know the exact target. The topic:partition form matters: --topic orders on its own shifts every partition of the topic by one, which skips a good record on every healthy partition. If you only know roughly when the bad records started, --to-datetime resets to the first offset at or after a timestamp, but check the dry run carefully because it can move you backwards and replay data.
Scoping, previewing and auditing resets across tools is compared in the best tools to reset consumer group offsets. Restart the consumers and describe the group again. The proof is CURRENT-OFFSET on partition 3 moving past 18443 and its LAG falling.
Find and inspect the bad record safely
You want the record’s bytes without committing anything on behalf of your application’s group. The console consumer reads one partition from one offset, and without --group it runs under its own generated group, so your application’s offsets are untouched:
kafka-console-consumer.sh --bootstrap-server <broker:9092> \
--topic orders --partition 3 --offset 18442 --max-messages 1 \
--property print.offset=true --property print.key=true \
--property print.headers=trueFrom Kafka 4.2 the formatter options are passed as --formatter-property instead of --property. If the payload is binary, kcat is easier to pipe into a hex dump:
kcat -b <broker:9092> -C -t orders -p 3 -o 18442 -c 1 -f '%s' | xxd | headThe first bytes usually tell you which kind of poison pill it is. How to diagnose a Kafka deserialization error goes deeper on reading them. A record written by a Schema Registry serializer starts with a zero byte followed by a four-byte schema ID. A record that starts with anything else was not written by that serializer, which points to a producer using the wrong serializer or a different application writing to the topic. A valid header with a schema ID your consumer does not expect points to a schema change that was never rolled out to the consumer.
A UI shortens this when the topic is busy. Kpow’s Data Inspect has a Partition mode that starts from an offset, and three deserialization options: Drop record (the default, matching a plain consumer), Retain record, which keeps failed records in the results flagged as “Deserialization exception” alongside their partition and offset, and Poison only, which shows nothing but the records that failed to deserialize. Tom Crowley, our founding engineer, built those options around a prospect whose topics mixed several schema types, including records that matched no schema at all. His approach for that case also works on your topics: when only certain schema IDs should appear on a topic, write a kJQ filter that asserts those IDs, and every record outside them surfaces on its own.
AKHQ, Kafbat UI and Kafdrop all browse topic data in the browser as well, which is enough to eyeball a record. What they add over the console consumer is speed. What only some of them add is a record of who looked.
Product demo · 8 min
Apache Kafka data inspection & search: Kpow demo
Chad Harris walks through data inspection in Kpow: filtering topic data with kJQ, running high-volume streaming searches with no scan limit, narrowing scans by partition or key, and downloading, cloning, or producing result sets to other topics.
Stop the next poison pill from stalling a partition
Once the partition is moving, decide where the next bad record should go instead of blocking the partition. Dead letter queues in Kafka: patterns and pitfalls compares the three implementation paths, and a consumer-side, Kafka-only approach keeps the DLQ scoped per consumer group. Once records land there, the best tools to manage a dead letter queue compares how to triage and replay them. In outline:
- Spring Kafka. Deserialization fails before
poll()returns, so Spring’s listener container cannot catch it. TheErrorHandlingDeserializer, described in Spring Kafka’s serialization documentation, wraps the real deserializer and passes the failure to the error handler as a header carrying the exception and the raw bytes. From there theDefaultErrorHandlerandDeadLetterPublishingRecoverer, covered in Spring Kafka’s error handling documentation, publish it to<topic>-dlton the same partition. - Kafka Streams. The
deserialization.exception.handlersetting in the Kafka Streams configuration reference chooses betweenLogAndFailExceptionHandlerandLogAndContinueExceptionHandler, or a custom handler that writes the record to a quarantine topic. - Kafka Connect. For sink connectors,
errors.tolerance=allwitherrors.deadletterqueue.topic.nameset routes converter and transform failures to a DLQ topic, as described in the Kafka Connect configuration reference. Seterrors.deadletterqueue.context.headers.enable=truetoo, because it defaults to false and the headers tell you why each record failed.
Two settings reduce the blast radius when a pill still gets through. Keep max.poll.interval.ms at its five-minute default so a stuck consumer is detected quickly. Incremental cooperative rebalancing helps, but a poison pill will still get a partition stuck. And put a Schema Registry serializer on every producer to a topic, so records are written against a registered schema and incompatible changes are rejected at registration rather than discovered at consume time.
When a record must be removed from the topic itself rather than skipped by one group, that is a different operation, covered in deleting records in Kafka.
max.poll.interval.ms is a failure detection timeout, not a processing budget.
Chad Harris, Solutions Architect at Factor House
How Kpow finds and skips a poison pill
Choosing between tooling for this job, rather than running the commands above, is covered in the best tools to manage poison pills in Kafka. Kpow’s own path is on the Kpow Data Inspect page: find the record with the Poison only option, keep a copy with Clone to topic, which copies records byte for byte to a topic you choose from a result’s Actions menu, then use the Skip offset action on the stuck partition in the consumer group view. Skip offset “increments the current offset by 1” and can only be applied at the partition level, per the consumer group documentation. Like every group offset action in Kpow it is scheduled, and runs once the group reaches the EMPTY state, for up to 15 minutes by default. In the live Kpow demo you can run Data Inspect in Partition mode from a chosen offset and switch the deserialization option between Drop, Retain and Poison only, the views that locate a bad record.
On a governed cluster the skip can be a staged mutation that an administrator approves, and the audit log records who skipped which offset. Tom wrote up the problem that approach solves in 2021, and it is now covered in break-glass access with temporary policies: incrementing one offset in production usually means a VPN, a jumpbox and “the right combination of bash commands against the Kafka cluster”, from a jumpbox that “generally has full access to the Kafka cluster” with “no audit log recording the actions being committed”. Triage, repair and replay covers the same workflow end to end, including repairing a record and producing it back.
Kpow live demo
Find the bad record in the demo
Open the Kpow demo and search a topic at a specific offset to see how a record that fails deserialization shows up, before you touch a consumer group.
Built for platform and data engineers running Kafka in production.
Try the Kpow demoFAQ
What is a poison pill in Kafka?
A record a consumer cannot process no matter how many times it retries, usually because it cannot be deserialized or breaks an assumption in the processing code. The consumer fails on the same offset every poll, so that partition stops advancing while the others carry on.
How do I skip a single message in Kafka?
Stop every consumer in the group, then run kafka-consumer-groups.sh --reset-offsets --shift-by 1 scoped to --topic <topic>:<partition>, first with --dry-run and then with --execute. Restart the consumers and confirm the partition’s committed offset has moved past the bad record.
Can I skip a poison pill without stopping the consumer group?
Not with an offset reset, because Kafka rejects offset changes for a group with active members. The alternatives live in the application: catch the failure, write the record to a dead letter topic, and let the consumer commit past it.
How is a poison pill different from a slow consumer?
A slow consumer’s committed offset still moves, just not fast enough, so it needs capacity. A poison pill’s offset does not move at all, so adding consumers changes nothing. Describe the group twice a minute apart and compare CURRENT-OFFSET to tell them apart.