Retry visibility for Lambda functions triggered by MSK

I lead several Kafka-based projects in insurance, on event-driven systems. Retry handling is one of the things you have to settle early in that kind of architecture: what happens to a record that fails, how many times it is retried, and whether you can see any of it while it happens. Like a lot of enterprises, we go serverless where it makes sense, so I wanted to work through that question with MSK and Lambda specifically.

Every path for handling a failed Kafka record (Spring Kafka’s error handler, a Connect connector’s dead letter queue config, a Streams topology’s exception handler) assumes a consumer process you write and control. Lambda removes that process. There is no consumer loop where you can catch an exception or attach a handler, just an event source mapping (the poller AWS runs on your behalf) invoking a function per batch.

When one record in that batch fails, Lambda’s default is to retry the whole batch, not just the failing record. Nothing about that retry is visible: no attempt count, nothing distinguishing the first try from the tenth.

AWS does offer native ways to narrow how much of the batch a failure affects, covered further down. This piece is about a different gap. Nothing tells you which retry attempt you are on. To see that default behavior directly, I built a small lab comparing two versions of the same handler against the same batch: one relying on Lambda’s own retries (the implicit path), one handling failures itself (the explicit path).

The default path, and its blind spot

A minimal handler for an MSK-triggered Lambda looks unremarkable in Java: implement RequestHandler<KafkaEvent, Void>, iterate over the records in the batch, process each one. KafkaEvent hands over the batch as a map of topic-partition to a list of records, each with a base64-encoded key and value. Nothing in this data structure indicates what happens once processing fails.

In a local unit test, I left this handler with no error handling at all. A stand-in downstream call, built to fail on command, threw an exception. Locally, that only proves the exception propagates out of the function unhandled; nothing about batch-level retries can be observed from a unit test. On AWS, that same unhandled exception makes Lambda mark the entire invocation as failed. A failed invocation is what triggers the event source mapping to retry the whole batch.

AWS documents the default here as no configured attempt limit. Retries continue until the record ages out of the topic, unless a limit is set explicitly. The mapping here was left at that default. Nothing in the handler’s own code exposes this retry behavior. Without an on-failure destination configured, nothing else does either. There is no attempt counter, no distinction between “this is the first invocation” and “this is the fourth retry of the same batch.”

Confirming it against a real cluster

I wired this up against a real MSK Serverless cluster, with three Lambda functions behind it: the two handlers being compared and a third that published records seeded to fail on demand. CloudWatch showed the exact same request ID on every retry, with an identical stack trace and only a few milliseconds of difference in duration. I expected a new ID each time; every invocation is supposed to get its own. I checked the log twice before I believed it. Nothing in this log tells you which attempt you are looking at. You would have to count identical blocks yourself to know how many times the batch has already been retried.

CloudWatch logs showing the same request ID retrying repeatedly with an identical stack trace

Building a dead letter topic yourself

To close that gap, the handler takes responsibility for its own failures. The handler catches every failure itself, so a plain Void return with no exception is exactly what a normal, fully successful invocation looks like to the event source mapping. There is no partial failure to report, and that does not depend on which polling mode the event source mapping runs in. The handler republishes the original record, unchanged, to a dedicated Kafka topic (orders-events-dlt in this lab). The original topic, partition, offset, and failure reason travel as headers alongside the untouched key and value:

@Override
public void send(DltRecord record) {
    ProducerRecord<String, String> producerRecord =
            new ProducerRecord<>(dltTopic, record.key(), record.value());
    producerRecord.headers()
            .add(new RecordHeader("dlt-original-topic",
                    record.originalTopic().getBytes(StandardCharsets.UTF_8)))
            .add(new RecordHeader("dlt-original-partition",
                    String.valueOf(record.originalPartition()).getBytes(StandardCharsets.UTF_8)))
            .add(new RecordHeader("dlt-original-offset",
                    String.valueOf(record.originalOffset()).getBytes(StandardCharsets.UTF_8)))
            .add(new RecordHeader("dlt-failure-reason",
                    record.failureReason().getBytes(StandardCharsets.UTF_8)))
            .add(new RecordHeader("dlt-attempts",
                    String.valueOf(record.attempts()).getBytes(StandardCharsets.UTF_8)));
    producer.send(producerRecord);
    producer.flush();
}

The producer that sends these records is created once per execution environment. The IAM handshake is not cheap, so it is not worth repeating on every warm call. The producer flushes synchronously before the handler returns. The execution environment freezes immediately afterward, and AWS documents that unfinished background work is not guaranteed to complete: anything still buffered at that point could sit unsent indefinitely. If the publish to the dead letter topic itself fails, that exception propagates like any other and the batch goes back through the same retry path described above. This design narrows that edge case. It does not remove it.

This design shares the core idea behind Spring Kafka’s DeadLetterPublishingRecoverer, republishing the original record with failure context in headers. The header names and retry mechanics around the republish are custom to this lab. The handler decides for itself what counts as unrecoverable, at runtime, before the event source mapping’s retry configuration gets a say.

The difference, side by side

I ran the same three-record batch through both paths, with one record engineered to fail every time. The implicit path treated the whole batch as one unit: the bad record blocked the two good ones, and the batch kept retrying with no visibility into how many times.

Two panels comparing the implicit retry path, where a failing record blocks the whole batch, against the explicit path, where the failing record is isolated and routed to a dead letter topic inspected via Kpow

Left at the infinite default with no destination, the implicit path was stopped manually once the pattern was confirmed, before retention could close it out on its own.

The explicit path processed the batch record by record. It retried the failing record with a short in-process backoff (counted separately from anything the event source mapping itself tracks) and logged each attempt with a number and a reason. That backoff ran inside the handler’s own billed time and held the partition while it waited. Once it gave up, it moved only that record to the dead letter topic:

attempt=1 recordId=orders-events-0-0 result=failure reason=forced failure
attempt=2 recordId=orders-events-0-0 result=failure reason=forced failure
attempt=3 recordId=orders-events-0-0 result=failure reason=forced failure
recordId=orders-events-0-0 sent to DLT after 3 attempts
attempt=1 recordId=orders-events-0-1 result=success
attempt=1 recordId=orders-events-0-2 result=success

CloudWatch logs from the explicit path showing three numbered attempts, the record sent to the DLT, then two other records succeeding on their first try

For orders-events-0-0 (topic, partition, and offset in one string), the DLT record carried the original key and value untouched, with headers reading dlt-original-topic=orders-events, dlt-original-partition=0, dlt-original-offset=0, dlt-failure-reason=forced failure, dlt-attempts=3.

What the native options deliver

AWS also offers native alternatives to a custom dead letter topic. One option is Lambda’s built-in on-failure destinations. Point a Kafka event source mapping at one, and Lambda ships a failure record there once retries are exhausted or a record exceeds the mapping’s own maximum age setting. That setting is separate from the topic’s own retention on the broker. Because it was left at its infinite default here, the topic’s retention was the real bound. Once a destination is configured, AWS also caps retries at 10 after the initial invocation, giving exhaustion a fixed point.

The resulting record carries failure metadata rather than the message itself. That metadata includes a request ID, the function’s ARN, a condition such as RetryAttemptsExhausted, an approximate invoke count paired with that request ID, and Kafka batch details such as the source topic and per-partition offsets. That invoke count only exists after the fact, in the destination record. It says nothing while a batch is still mid-retry, which is the exact gap shown above.

AWS also offers ways to narrow the batch itself, addressing a different problem than the one just shown. ReportBatchItemFailures lets the handler report exactly which records in a batch failed so only those get retried. Split batch on error divides a failing batch in two and retries each half separately. Both reduce how many records a single failure affects. Neither adds an attempt count or a failure reason, the two things the explicit path just demonstrated. Nor would either have changed anything about that path. It already catches every failure itself and always returns successfully. There was never a partial failure for either mechanism to act on.

AWS documentation also describes a Kafka-topic on-failure destination, added in November 2025, that carries the actual key and value alongside the failure metadata, producing what is effectively a native dead letter topic. It applies only to event source mappings running in provisioned poller mode. That is a separate setting from the MSK Serverless cluster mode used here. Provisioned poller mode is the simpler path when it is already in use. Without provisioned poller mode, a destination provides only metadata, and building a dead letter topic at the application level stays the more portable choice.

Seeing the dead letter topic in Kpow

A dead letter topic built this way carries its most useful information in headers. Filtering by those headers matters more than plain viewing: which reason fired, how many attempts it took. I connected Kpow to the same MSK Serverless cluster over IAM auth, pointing it at the bootstrap brokers with the SASL/IAM configuration described in Factor House’s setup guide for MSK. Data Inspect showed partition, offset, timestamp, and the decoded key and value, with the headers in the same view. None of it had to be assembled by hand from a CLI consumer.

Kpow’s Data Inspect view on the dead letter topic, showing partition, offset, timestamp, and the decoded record

Kpow 96.2 added a “Clone to topic” action that replays a record straight from Data Inspect back into a topic, gated behind role-based access control (RBAC). This is the natural complement to a header-heavy dead letter topic: filter down to the records actually worth retrying, then send them back.

Where the visibility gap comes from

An on-failure destination adds an invoke count and a failure condition, but only once retries are already over. Its Kafka-topic variant goes further and adds the record itself, though only in provisioned poller mode. A handler that catches its own failures skips both limits: it can record every attempt, in full, while retries are still happening. Visibility is not a byproduct of any of these. It has to be built.

This lab is public, built in Java 25 behind a plain RequestHandler entry point, with the Terraform for the AWS side included: github.com/cmoslem/msk-retry-lab. Run mvn test to reproduce the comparison locally, no AWS account needed.