How to diagnose a Kafka deserialization error

A Kafka deserialization error shows up as a consumer that stops making progress on one partition while the rest of the group carries on. The log repeats the same SerializationException (in the Java client, a RecordDeserializationException that names the partition and offset), consumer lag on that one partition climbs, and if the service restarts it fails on exactly the same record. A deserialization error means the consumer received bytes it could not turn back into an object with the deserializer it was configured with.

The exception text usually tells you which of five causes you are looking at, before you open a single payload:

What the log says	Most likely cause	Where to go on this page
Unknown magic byte!	The record was not written in the Confluent wire format, often raw JSON or a plain string on a topic the consumer expects to be registry-encoded	Schema registry and evolution failures
Error retrieving Avro value schema for id 1234 with Schema 1234 not found; error code: 40403	The schema ID in the record does not exist in the registry this consumer is pointed at	Schema registry and evolution failures
Error deserializing Avro message for id 1234	The schema was found but the bytes could not be read with it, usually a compatibility break or a reader class that cannot resolve the writer’s schema	Schema registry and evolution failures
A JSON, Protobuf or String parser error with no schema ID in it	The wrong deserializer for what is on the topic, or bytes that are genuinely corrupt	Root cause: read the raw bytes
A NoSuchMethodError, VerifyError or “Bad type on operand stack” inside a deserializer class	The deserializer code changed underneath you, not the data	Root cause: read the raw bytes

Those strings come from Confluent’s serializer library and the Apache Kafka client, which is what most JVM teams run. Other clients word them differently but fail on the same five causes.

Unblock the pipeline first

A consumer that cannot deserialize a record cannot commit past it either, so every restart replays the same failure and the partition sits still. In my Kafka operational issues webinar I put it this way: “incremental cooperative rebalancing does help reduce the blast radius of this, but a poison pill will still cause individual partitions to get stuck.” There are three ways to get the partition moving, and one thing to do before any of them. A record that fails every time it is read is a poison pill, and the dedicated guide to finding and skipping a poison pill in Kafka covers the skip itself step by step, with a dry run first.

Before you skip anything, find out why the record failed and save a copy of it. The same goes for killing or restarting the stuck consumer: you need to understand why a consumer is not advancing before you kill it. The replacement reads the same offset and fails the same way, and each restart costs the group a rebalance, which across a large group can turn one stuck partition into a rebalance storm.

Route it to a dead letter topic. A dead letter queue (DLQ) moves the unreadable record to a separate topic with enough context to replay it later, so the consumer keeps going. Kafka has no DLQ primitive of its own outside Kafka Connect, so in a plain consumer or a Streams app this is code you write or a framework feature you turn on. Our guide to dead letter queues in Kafka covers the patterns, and the comparison of Kafka DLQ tools covers inspecting and replaying what lands there. One warning from the same webinar: if the job that reads the DLQ can write failures back to the same DLQ, a record that fails again re-enters the loop forever. Put a retry count in a header and stop once it passes a limit.

Log it and skip it in code. Catch the exception, record the partition, offset and raw bytes, then seek one past the failed offset. This is the right answer when the record is known garbage, and the wrong answer when it is the first of thousands produced by a broken deployment.

Move the committed offset by hand. When the consumer is down and you need it past one record now, reset the group’s offset for that one partition to one past the failing offset. If the bad record is at 48213, the new committed offset is 48214. The group must have no active members while you do this, so stop the consumers first:

bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --group orders-service --topic orders:3 \
  --reset-offsets --to-offset 48214 --execute

Leave off --execute for a dry run that prints what would change. The flags are documented under managing consumer groups in the Apache Kafka operations guide, and Kafka offset management tools compares the tools that do this with a dry run and an audit trail. One thing not to do while you are in here: switch auto.offset.reset to earliest to “make sure nothing is lost”. If the group ID ever changes, that setting reprocesses the whole topic. I recommend latest in production with offset resets as a deliberate, manual operation.

Framework-specific handling

Each client stack gives you a different hook for catching the failure, and the default behaviour of most of them is to stop.

Plain Java consumer. poll() throws RecordDeserializationException when a record cannot be deserialized. Since Kafka 3.9 it carries the raw record as well as its position, exposed as: topicPartition(), offset(), keyBuffer() and valueBuffer() (Apache Kafka javadoc). That makes the log-and-skip pattern a few lines:

try {
    ConsumerRecords<String, Order> records = consumer.poll(Duration.ofMillis(500));
    process(records);
} catch (RecordDeserializationException e) {
    deadLetter.send(e.topicPartition(), e.offset(), e.keyBuffer(), e.valueBuffer(), e);
    consumer.seek(e.topicPartition(), e.offset() + 1);
}

Spring Kafka. Wrap your real deserializer in ErrorHandlingDeserializer. Spring’s reference explains why this exists: the failure “occurs before the poll() returns”, so the listener never sees it. The wrapper instead returns a null value with a DeserializationException header that carries the cause and the raw bytes, and the container’s error handler decides what happens next (Spring for Apache Kafka reference). Our Kafka with Spring Boot guide covers the surrounding configuration.

Kafka Streams. The deserialization exception handler decides whether a bad record stops the application or is skipped. The default is LogAndFailExceptionHandler, which shuts the stream thread down. Switching to the log-and-skip handler is set by:

deserialization.exception.handler=org.apache.kafka.streams.errors.LogAndContinueExceptionHandler

On Kafka 4.0 and later the property is deserialization.exception.handler, with default.deserialization.exception.handler kept as a deprecated alias for older applications (Kafka Streams configuration reference). The same page suggests a custom handler that forwards corrupt records to a quarantine topic, which I prefer to silent skipping, for the reasons in the section above.

Python. Confluent’s Python client raises ValueDeserializationError (or KeyDeserializationError) from the deserializing consumer’s poll(), and the exception keeps a reference to the original message. Catch it inside the poll loop, write the message out, and continue, rather than letting it end the process.

Schema registry and evolution failures

On Avro, Protobuf or JSON Schema topics, a deserialization failure is more often a disagreement between what a producer wrote and what the consumer’s registry and deserializer expected than corrupt data. Three causes cover the registry side.

Magic byte mismatch

A registry-aware serializer writes every record in the same envelope: one magic byte with the value 0, then a 4-byte schema ID, then the encoded payload. The matching deserializer reads the first byte, and if it is anything other than 0 it throws Unknown magic byte! without looking further. The usual cause is a producer that wrote plain JSON or a string to a topic the consumer expects to be Avro: a test script, a new service that skipped the registry serializer, or a replay tool that re-encoded records on the way back in.

The proof is the first byte of the failing record. A registry-encoded record starts with 00. A JSON record starts with 7b, the byte for {:

kcat -b localhost:9092 -C -t orders -p 3 -o 48213 -c 1 -f '%s' | xxd | head -2

The fix is on the producer side. Change the consumer’s deserializer only if the topic is meant to carry plain JSON, and if it is meant to carry both, that topic has a bigger problem than this incident.

Schema ID not found in the registry

The record carries a valid envelope, but the ID in bytes 1 to 4 does not exist in the registry the consumer asked. The consumer logs Error retrieving Avro value schema for id 1234 and the registry’s response underneath it reads Schema 1234 not found; error code: 40403. The usual causes are a consumer pointed at a different registry than the producer (staging registry, production cluster), data mirrored between clusters whose registries were never kept in step, or a schema that was hard-deleted while records written with it were still inside the topic’s retention.

The proof is to decode the ID from the record and ask the registry the consumer uses for it directly:

kcat -b localhost:9092 -C -t orders -p 3 -o 48213 -c 1 -f '%s' | head -c 5 | xxd
# 00000000: 0000 0004 d2   magic byte 00, then schema ID 0x000004d2, which is 1234
curl -s http://schema-registry:8081/schemas/ids/1234

A 404 from the consumer’s registry, and a 200 from the producer’s, settles it. On deletion, Confluent’s registry soft-deletes by default, and a soft delete keeps the schema ID available for lookup. A permanent delete does not, which is why a permanent delete should only follow the expiry of every record that references the schema.

Compatibility break after a producer schema change

The schema is found, but the consumer cannot read the bytes with the schema it wants to read them as. The log says Error deserializing Avro message for id 1234, and the cause underneath names the field. A generic consumer reads with the writer’s schema, fetched by ID, so it rarely fails at this layer. The failure appears when the consumer deserializes into a generated class, for example Avro with specific.avro.reader=true or a compiled Protobuf type, and that class cannot resolve the schema the producer used. A field removed without a default, a type change, or a renamed enum symbol will do it.

A registry only prevents this if the subject’s compatibility mode matches the order in which you deploy. The mode defines which direction of reading is guaranteed:

• BACKWARD: a consumer on the new schema can read data written with the previous one. Upgrade consumers first. This is the default in Confluent’s registry.
• FORWARD: a consumer on the previous schema can read data written with the new one. Upgrade producers first.
• FULL: both directions hold, so the deploy order stops mattering.
• NONE: nothing is checked.

Under BACKWARD, a producer that ships first can register a schema the registry accepts and still break every consumer still running the old generated class. Moving the subject to FULL (or FULL_TRANSITIVE, which checks against every earlier version rather than only the latest) is the setting that holds up when producer and consumer teams deploy independently. Setting a subject to NONE to push a schema through is how most of these breaks start.

The proof is the registry’s own compatibility check, run with the schema the consumer was compiled against:

curl -s http://schema-registry:8081/config/orders-value
curl -s -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data @consumer-schema.json \
  http://schema-registry:8081/compatibility/subjects/orders-value/versions/latest
# {"is_compatible":false}

The fix is either to roll the producer back, or to register a new version that restores compatibility, usually by adding the missing field back with a default. Choosing a registry tool that runs this check before a schema is registered, rather than after consumers have failed, is the subject of the schema registry tools comparison.

Root cause: read the raw bytes

When the exception does not mention a schema ID, stop guessing and print the failing record’s bytes. Four answers are possible, and each one points somewhere different: the bytes are empty or null, they are valid in a different format, they are valid in the expected format but your deserializer rejects them, or they are genuinely corrupt.

kcat -b localhost:9092 -C -t orders -p 3 -o 48213 -c 1 \
  -f 'key: %K bytes, value: %S bytes\n%s\n' | xxd | head -20

The format string is kcat’s own, documented in its README: %S prints the value size, %s the value, %K the key size.

Empty or null. A value size of 0 or -1 is a tombstone or an empty record rather than corrupt data, so the failure is a deserializer that does not handle null, which is a code fix, and on a compacted topic the tombstone is expected.

A different format. Readable JSON, a Protobuf message with no envelope, or a string where a number was expected means the wrong deserializer is configured for this topic, or someone wrote the wrong format to it, and the first bytes tell you which. Plain Protobuf without a registry needs a custom deserializer, and the registry-encoded kind needs the registry one.

Valid bytes, failing deserializer. When the bytes decode fine with a known-good tool but your consumer still throws, the problem is in your deserializer code or its dependencies. We had exactly this at Factor House. When Kpow moved its Confluent serdes dependencies to 8.0.1, the bundled protobuf library went from 3.25.5 to 4.31.1. Custom serdes that contained protobuf classes generated by the older compiler then failed, at compile time with a missing makeExtensionsImmutable() and at runtime with a “Bad type on operand stack” verification error, on data that had not changed. Derek, our co-founder and CEO, wrote up the Custom Serdes and protobuf 4.31.1 change. The fix was regenerating the classes with protoc v31.1. If an error appears straight after a dependency bump, check the library versions before you check the topic.

Genuinely corrupt. Truncated payloads, bytes that decode as nothing, and headers written wrong by a client that implements the protocol itself do happen, just less often than people assume. I have seen a third-party client write three bytes into a header field that should have been four, which looked harmless until an official client started rebalancing every time it met one of those records. That is the case for sticking to the official Apache Kafka clients, or ones that wrap librdkafka. Corrupt records go to the DLQ, and the producer that wrote them gets fixed.

A last check before you close the incident: count how many records fail, not just the first one. One bad record is a skip. Ten thousand is a producer bug, and skipping them one at a time will not end the incident.

How Factor House approaches it

The fastest part of this diagnosis is the part most teams do by hand: finding the records that fail and seeing what is in them. Kpow does that in its Data Inspect view. Data Inspect shows each record’s schema ID and deserializer, and it has three deserialization options for failed records (Kpow docs, data inspect): Drop record (the default), Retain record, which keeps failed records in the results flagged as “Deserialization exception”, and Poison only, which returns nothing but the records that failed. Poison only over a time window answers the “one record or ten thousand” question in a single query, across one topic or several, with a per-partition count of deserialization errors in the results metadata.

When the partition has to move, Kpow’s consumer group actions include a partition-level Skip offset, which increments the committed offset by one, alongside offset reset by value, timestamp or datetime (Kpow docs, groups). It reads offsets for EMPTY groups straight from the AdminClient, which the docs call out as the case “when a poison message causes an entire consumer group to go offline”. With RBAC enabled, who can skip or reset is controlled by the GROUP_EDIT action, and Kpow Enterprise records each of those actions in its audit log.

For registry-backed topics, Kpow connects to Confluent-compatible registries including Confluent Schema Registry, Apicurio Registry and Karapace, plus AWS Glue, Google, Redpanda and Buf registries (Kpow docs, schema registry), and from the schema UI you can update a subject’s compatibility and compare schema versions. Setting it up against more than one registry is covered in integrating Confluent-compatible registries in Kpow. Custom serdes, the case in the protobuf story above, are covered in the Kpow serdes documentation.

The test worth running in a Kpow demo is the one this page keeps coming back to: pick a topic, set the deserialization option to Poison only over the last few hours, and see how quickly you get from “a consumer is stuck” to the failing records, their schema IDs and a per-partition count.

Kpow live demo

Decode a failing record in the demo

Open the Kpow demo and inspect a topic with automatic Avro, Protobuf and JSON deserialization to see what a consumer should have read.

Built for platform and data engineers running Kafka in production.

Try the Kpow demo→

FAQ

What causes “Unknown magic byte!” in Kafka?

The consumer’s registry deserializer expected a record starting with the Confluent wire-format envelope, a 0 byte followed by a 4-byte schema ID, and the first byte was something else. Almost always a producer wrote plain JSON or a string to a topic the consumer expects to be registry-encoded. Print the first bytes of the record to confirm, then fix the producer.

How do I skip a message that fails to deserialize?

In code, catch RecordDeserializationException, save the raw bytes it carries, and seek to the next offset. Without code, stop the consumers and move the group’s offset for that partition one past the bad record with kafka-consumer-groups.sh --reset-offsets --to-offset. Save a copy of the record first, because once you skip it the only other copy is the one that ages out of the topic.

Why does my consumer keep failing on the same offset after a restart?

The committed offset has not moved, so the restarted consumer fetches the same record and fails the same way. Restarting, scaling out or killing the consumer only adds rebalances. The partition moves when the record is handled in code, routed to a DLQ, or the offset is moved past it.

Is BACKWARD or FULL compatibility better for schema registry?

BACKWARD protects you when consumers deploy before producers. FULL protects you whichever side deploys first, which is the realistic case when separate teams own producers and consumers. If you cannot guarantee deploy order, use FULL or FULL_TRANSITIVE.

For the rest of the operational picture, start with the complete Kafka guide.

Related reading

• Kafka: the complete guide
• Kafka consumers in production
• Dead letter queues in Kafka: patterns and pitfalls
• What is Kafka rebalancing?
• Kafka operational issues: how to survive them
• How to find and skip a poison pill in Kafka