Clone to topic for Dead Letter Queues in Apache Kafka

Kpow 96.2 adds Clone to topic, a byte-level clone of records to a retry topic directly from Data Inspect search results. Cloning utilises the same Role-Based Access Control (RBAC) and audit log controls as the rest of Kpow, so replaying Dead Letter Queue (DLQ) records becomes a governed UI operation instead of bespoke CLI tooling.

DLQ workflow within Kpow

DLQs are a common approach when using Kafka to park records that cannot be processed, rather than blocking the consumer or dropping records. The problem comes later when listing those records for investigation and replaying them once the underlying issue is resolved, which can be a manual and error-prone process, even when teams build custom CLI tooling for it.

Listing

Kpow's Data Inspect searches records across multiple topics in a single query. For listing DLQ records, Kpow's long-lived search result tabs really shine: They maintain cursors to continue consuming records on demand. Some of our customers keep browser sessions open for multiple weeks to do just that!

Replaying

Kpow 96.2 introduces Clone to topic for both individual records and a batch of Data Inspect search results, performing a byte-level clone to a retry topic in a few clicks.

When you pair this with Kpow's RBAC policies, you can restrict the source and destination topics for clones. For example, on a payments topic you can enforce that only records from per-consumer DLQ topics may be cloned into per-consumer retry topics:

policies:
  - actions:
      - "TOPIC_CLONE_SOURCE"
    effect: Allow
    resource: 
      - "cluster"
      - "*"
      - "topic"
      - "payments.*.dlq"
    roles: [ "kafka-users" ]
  - actions:
      - "TOPIC_CLONE_SINK"
    effect: Allow
    resource:
      - "cluster"
      - "*"
      - "topic"
      - "payments.*.retry"
    roles: [ "kafka-users" ]

Some records need correcting before replay rather than a straight clone. For those, you can send Data Inspect results to Data Produce, preview and amend the records, then produce them to a retry topic. Malformed records are best fixed at the root cause, but this gives you a path for the cases that slip through.

Available now

Clone to topic turns DLQ replay from a manual, ungoverned task into a few clicks, with RBAC and audit logging applied to every clone. It is available now in Kpow 96.2. See the Data Inspect documentation for setup and governance details.