Factor House blog

Clone to topic for Dead Letter Queues in Apache Kafka

Nicolas Venegas — Sat, 27 Jun 2026 00:00:00 GMT

Kpow 96.2 adds Clone to topic, a byte-level clone of records to a retry topic directly from Data Inspect search results. Cloning utilises the same Role-Based Access Control (RBAC) and audit log controls as the rest of Kpow, so replaying Dead Letter Queue (DLQ) records becomes a governed UI operation instead of bespoke CLI tooling.

DLQ workflow within Kpow

DLQs are a common approach when using Kafka to park records that cannot be processed, rather than blocking the consumer or dropping records. The problem comes later when listing those records for investigation and replaying them once the underlying issue is resolved, which can be a manual and error-prone process, even when teams build custom CLI tooling for it.

Listing

Kpow’s Data Inspect searches records across multiple topics in a single query. For listing DLQ records, Kpow’s long-lived search result tabs really shine: They maintain cursors to continue consuming records on demand. Some of our customers keep browser sessions open for multiple weeks to do just that!

Replaying

Kpow 96.2 introduces Clone to topic for both individual records and a batch of Data Inspect search results, via our Bulk actions, performing a byte-level clone to a retry topic in a few clicks.

When you pair this with Kpow’s RBAC policies, you can restrict the source and destination topics for clones. For example, on a payments topic you can enforce that only records from per-consumer DLQ topics may be cloned into per-consumer retry topics:

Some records need correcting before replay rather than a straight clone. For those, you can send Data Inspect results to Data Produce, preview and amend the records, then produce them to a retry topic. Malformed records are best fixed at the root cause, but this gives you a path for the cases that slip through.

Available now

Clone to topic turns DLQ replay from a manual, ungoverned task into a few clicks, with RBAC and audit logging applied to every clone. It is available now in Kpow 96.2. See the Data Inspect documentation for setup and governance details.

Kafdrop: Review, pricing, and best alternatives in 2026

Factor House — Sat, 27 Jun 2026 00:00:00 GMT

Key takeaways

Kafdrop is a free, open-source Kafka UI built on Spring Boot, used most often for local development and simple single-cluster setups.
It has no built-in authentication or access control; any internet-facing deployment requires an external auth proxy or gateway.
KRaft mode (the default Kafka cluster mode since Kafka 3.3) is not supported, and the maintainers closed the request as “not planned.”
Performance degrades sharply at scale: one reported case took over 30 minutes to load topic and partition data for a cluster with roughly 1,000 topics.
If you need RBAC, reliable performance at scale, or a supported product with a maintained roadmap, Kpow from Factor House is a commercial alternative worth evaluating.

What is Kafdrop?

Kafdrop is an open-source web UI for Apache Kafka, built on the Spring Boot framework. It provides a browser-based interface for browsing topics, viewing partition state, inspecting messages, and performing basic topic administration such as creating and deleting topics.

The project is hosted at obsidiandynamics/kafdrop on GitHub and maintained by the Obsidian Dynamics team. It is widely cited as one of the easiest Kafka UIs to deploy in a local Docker Compose environment, which explains its prevalence in tutorials and development workflows.

Kafdrop runs as a stateless Java process and connects to a Kafka cluster over standard broker protocols. It does not require a separate backend datastore, which keeps the local setup minimal.

Kafdrop review

Functionalities

Kafdrop covers the fundamentals: topic browsing, partition inspection, and message viewing. Developer cnatsis, in a monitoring tools evaluation thread on r/apachekafka, classifies it as a “Topic Administration tool” used specifically to “create/edit/delete topics & view messages.”

The feature set beyond those basics is limited. Kafdrop does not support searching or filtering messages within a topic by key or value content. Developer felheartx, in a comparative analysis on r/devops, describes this gap directly: “Kafdrop doesn’t offer the ability to search for specific messages within a topic (e.g. filter by a provided javascript expression, or find a message where the key equals a specific string).”

Message encoding is not auto-detected either. Users must manually configure the deserialization format per topic; newer tools handle this automatically, whereas Kafdrop “requires the user to actively configure that,” as felheartx notes in the same thread.

Performance is a documented constraint at scale. GitHub issue #270 (obsidiandynamics/kafdrop) records a case where Kafdrop took over 30 minutes to load topic and partition information when connected to an SSL-enabled Kafka broker with approximately 1,010 topics and 2,000 partitions.

In published comparisons, Kafdrop sits below richer alternatives. Duc Quoc, writing for Towards Data Science, characterises Kafdrop as “a pretty average tool” with an interface that “is not spectacular,” and recommends Kafka-UI for teams with more complex operational needs.

Deployment and operations

Kafdrop is consistently praised for being lightweight and fast to deploy. On r/dotnet, developer DotDeveloper describes it as “super lightweight and easy to spin up if you just want to see what’s flowing through your topics.” On r/apachekafka, johannz adds: “It’s not a full management tool but it’s quick and easy to troubleshoot issues with.”

It runs comfortably alongside a standard local Kafka stack with minimal resource overhead. One engineer on r/dataengineering reported running four Docker containers (orchestrator, Kafka, Zookeeper, and Kafdrop) on an Apple M2 MacBook Air with 24 GB RAM alongside active editors and remote desktops without resource issues.

One friction point for Kubernetes teams: Kafdrop’s Helm chart source code is available in the repository, but it is not hosted in any official Helm repository. Teams cannot install it via a standard helm repo add / helm install workflow. The Pi Cluster open-source project documents this gap explicitly and deploys Kafdrop using Kustomize as a workaround.

Access control and security

Kafdrop has no built-in authentication or access control. Developer felheartx, in the r/devops comparative analysis, states that Kafdrop “doesn’t have an authentication & authorization concept (user is in charge of protecting the webapp using third party tools…).”

A Help Net Security article from December 2021 describes the consequence plainly: Kafdrop provides “a UI to make it easy to review live Kafka clusters, without authentication.” An internet-facing instance exposes full cluster visibility with no login barrier.

The Pi Cluster open-source project, which deploys Kafdrop in a production-adjacent Kubernetes environment, confirms that this remains true: their implementation relies on an external gateway (Envoy Gateway) to enforce security policy, since Kafdrop provides no user management natively.

Connecting Kafdrop to a SASL-secured broker is also a friction point. On r/apachekafka, user Youth-Character describes the experience: “i was using kafdrop with kafka and it was working fine but when i added sasl authentication i can’t seem to find any docs on how to integrate kafdrop.” The workaround requires manually mapping external .properties files inside the Docker container.

Native RBAC and SSO support are not available.

User interface

The UI is functional within its intended scope. For a quick look at a single cluster, it loads and navigates without friction.

Practitioners who need more describe it as minimal rather than polished. Duc Quoc’s Towards Data Science review positions Kafdrop as suitable for teams wanting “a quick visual view of a simple cluster, not for teams that need rich operational tooling or daily-driver UI features.” No strong positive characterisation of the UI surface surfaced in the sources reviewed; neutral-to-adequate is the dominant register.

Kafdrop’s visualisation can also surface symptoms without providing context for root cause analysis. One developer on r/dataengineering described observing all messages routing to a single partition in Kafdrop, where the tool had no mechanism to indicate that the root cause was a producer-side key-hashing configuration rather than a broker or UI failure.

Ecosystem

Kafdrop does not natively support AWS IAM MSK authentication. A community workaround exists via environment variables (KAFKA_IAM_ENABLED, KAFKA_SASL_MECHANISM=AWS_MSK_IAM) for connecting to IAM-authenticated MSK clusters on EKS using IRSA. The underlying pull request (PR #287, obsidiandynamics/kafdrop) remained open with unresolved merge conflicts for over a year. A collaborator stated willingness to merge the contribution but the conflicts were never resolved, indicating slow responsiveness to cloud-integration work from the community.

KRaft mode (ZooKeeper-free Kafka, the default from Kafka 3.3 onward) is not supported. The topic view becomes unresponsive when connecting to a KRaft cluster. GitHub issue #670 (obsidiandynamics/kafdrop) was closed as “not planned,” meaning there is no committed roadmap to address this incompatibility.

Customer support

Kafdrop is an open-source project with no enterprise support tier, no dedicated community forum, and no documented support channel beyond GitHub issues.

Maintainer responsiveness appears slow based on the available evidence. PR #287 (AWS MSK IAM support) sat open with unresolved conflicts for more than a year without a resolution. Issue #670 (KRaft support) was closed as “not planned” rather than addressed. Both cases suggest the project is not actively expanding its compatibility surface or cloud integration story.

For a tool used primarily in local development environments, this may be acceptable. For teams that need timely fixes or cloud-integration support from a maintained codebase, the absence of any supported tier is a genuine constraint.

Best for

Kafdrop suits individual engineers and small teams who need a zero-cost, fast-to-deploy visual layer over a single Kafka cluster in a local or sandbox environment. It fits naturally into Docker Compose development stacks, where it can be added as a container without meaningful resource overhead and removed just as easily.

It is a poor fit for:

Teams running KRaft-mode Kafka clusters (Kafka 3.3+ default): the incompatibility is known and closed as not planned.
AWS MSK deployments using IAM authentication: native support does not exist; the community workaround is unofficially maintained.
Any team needing authentication, RBAC, or SSO: none of these are available natively, and layering them in requires platform engineering overhead.
Large clusters with 1,000+ topics and SSL: severe performance degradation is documented.
Multi-cluster management: not supported.

Kafdrop pricing

Kafdrop is free and open-source, released under the Apache License 2.0. There are no licensing fees, subscription plans, or commercial editions.

Pricing tiers

There is a single tier: free. The project has no commercial offering of any kind.

Free trial

There is no trial concept because Kafdrop has no paid tier to trial. You can run it immediately from the public Docker image or build it from source at no cost.

Kafdrop competitors and alternatives

Teams that outgrow Kafdrop typically move toward tools with built-in authentication, richer message search, or multi-cluster support. The market spans free open-source options, community-edition commercial tools, and fully commercial platforms with enterprise support.

Tool	Best for	Type	Key functionalities	Deployment & ops	Access control	User interface	Pricing
AKHQ	Enterprise teams needing native identity provider integration	OSS	Topic admin, consumer group management, native Azure AD/OIDC	Web-based; higher resource footprint than Kafdrop	Native OIDC/OAuth2	Full-featured; more complex UI than Kafdrop	Free
Redpanda Console (formerly Kowl)	Teams needing JS-based message filtering and Protobuf support	OSS / Commercial	JS expression search, dynamic Protobuf schema compilation, auto-decoding	Go / React SPA; lightweight	Available in commercial tiers	Modern, polished	Free OSS tier; commercial tiers via Redpanda
Kafka-UI	Teams with multi-cluster environments and moderate monitoring needs	OSS	Multi-cluster management, topic admin, consumer group monitoring	Web-based; Docker and Kubernetes	Basic auth available	Clean, functional	Free
Lenses	Teams requiring SQL querying over Kafka topics and connector monitoring	Commercial	SQL Studio, topology maps, automated connector restarts, DLQ management	Web UI	RBAC, SSO	Rich, SQL-driven	Tiered, with seat limits
Kpow by Factor House	Teams needing enterprise RBAC, high performance at scale, and dedicated support	Commercial, Community Edition available	Topic admin, consumer group management, schema registry, advanced RBAC, audit logging	Stateless; Docker, Kubernetes, ECS	Advanced RBAC; WCAG-compliant UI	Performant, WCAG-compliant	Per-cluster; no per-seat penalty as team grows

For a broader comparison of Kafka UI tools, see the Kafka UI comparison guide.

Frequently asked questions about Kafdrop

How much does Kafdrop cost, and is there a free tier?

Kafdrop is free. It is open-source software released under the Apache License 2.0. There are no paid plans, commercial editions, or usage limits. You download and run it at no cost.

When is Kafdrop a better choice than the alternatives?

Kafdrop suits teams that need a zero-cost, minimal-setup Kafka UI for local or sandbox use. If you are running a single ZooKeeper-based cluster in Docker Compose and need quick topic visibility, Kafdrop installs in minutes and adds negligible resource overhead.

When are the alternatives a better choice than Kafdrop?

Alternatives suit you better when you need authentication or access control built in, KRaft cluster support, message search and filtering, AWS MSK with IAM auth, multi-cluster management, or reliable performance at scale. Kafdrop covers none of these natively.

Does Kafdrop support KRaft mode?

No. Kafdrop fails to display topic information when connected to a KRaft cluster, and GitHub issue #670 requesting support was closed as “not planned.” Teams running Kafka 3.3 or later in KRaft mode should evaluate alternative tools before committing to Kafdrop.

Is Kafdrop safe to deploy outside a local environment?

With careful configuration. Kafdrop has no built-in authentication, so an unprotected instance exposes full cluster visibility to anyone who can reach it. Teams running it outside a local network must place it behind an auth proxy. It also exposes write operations, so accidental topic deletion is possible.

Kafka dashboard: the features that matter in production

Factor House — Sat, 27 Jun 2026 00:00:00 GMT

Running Apache Kafka in production means managing distributed state across brokers, topics, partitions, and consumer groups at the same time. A Kafka dashboard gives you a single interface to observe that state in real time, without assembling metrics by hand or reaching for the CLI every time something looks wrong. This article covers what a production-grade Kafka dashboard should do, what to look for when evaluating one, and how Kpow by Factor House approaches the problem.

What is a Kafka dashboard?

A Kafka dashboard is a web interface that aggregates and visualises the operational state of an Apache Kafka cluster. It typically displays broker health, consumer group lag, partition assignments, topic throughput, and schema registry state, giving platform and data engineering teams a shared view of a cluster without running CLI commands manually.

Most CLI tools surface a snapshot of a single metric at a time. A Kafka dashboard aggregates across brokers and topics, renders trends over time, and makes cluster state readable to anyone on the team, not just whoever knows the right kafka-consumer-groups.sh flags. At the partition level, that difference matters: a group-level lag summary can look acceptable while a single partition is falling badly behind.

A Kafka dashboard typically visualises:

Broker health and JVM metrics
Consumer group lag, by topic and by partition
Partition assignments and ISR (in-sync replica) state
Topic throughput in messages and bytes per second
Schema registry contents and schema evolution state

An example Kafka dashboard - Kpow by Factor House

Why Kafka is hard to observe

Kafka’s architecture makes operational visibility harder than it initially appears. A healthy cluster requires every broker to agree on partition leadership, maintain replication, and serve consumers at acceptable latency simultaneously. When something is off, the symptom and the cause are often in different parts of the cluster.

Consumer lag is a particular challenge. It is a lagging indicator by nature: by the time lag accumulates to a level that looks alarming at the group level, the relevant consumer may already be hours behind on specific partitions. Without partition-level visibility, you are watching an average that conceals the problem.

Schema evolution adds another layer. When a producer changes a schema and a downstream consumer has not been updated, you may not see an error until records hit the consumer’s deserialiser. A dashboard with schema registry integration makes this state visible before it causes failures.

At scale, manual inspection becomes impractical. A cluster with hundreds of topics, multiple consumer groups, and several environments cannot be monitored with periodic CLI snapshots. Production operations require continuous, aggregated visibility.

What to look for in a Kafka dashboard

Not all Kafka dashboards are built for production use. Some are read-only tools useful in development. Others surface cluster-level metrics but lose detail at the partition level, which is exactly where most operational problems hide. When evaluating options, the following criteria are worth assessing carefully.

Real-time lag visibility at the partition level. Group-level lag summaries are a useful starting point, but they are not sufficient for production. A partition within a topic can be stalled while others process normally, and a group-level average will not surface it. A production-grade dashboard shows lag per partition, updates continuously, and lets you drill from the group view to individual partitions without switching tools.

Consumer group inspection and offset management. Beyond observing lag, your team needs to act on it. That means being able to inspect individual consumer group members, see their partition assignments, and when necessary, reset or adjust offsets directly from the UI. Without this, offset management defaults back to the CLI, with all the access-control and audit implications that involves.

Broker and partition health metrics. Broker health goes beyond whether a broker is reachable. Under-replicated partitions, ISR changes, and leader imbalances are the early signals of instability. A dashboard should surface these alongside the topics and partitions affected, not just a raw count.

Multi-cluster support. Platform teams routinely manage separate clusters for development, staging, and production. A dashboard that requires a separate deployment per cluster, or cannot aggregate views across environments, creates avoidable operational overhead.

Schema Registry integration. For teams using Avro, Protobuf, or JSON Schema, the registry is a critical part of the pipeline. A dashboard that integrates schema registry state lets you see which schemas are in use, view historical versions, and spot compatibility issues before they reach consumers.

RBAC and access control. Not everyone who needs to view the Kafka dashboard should be able to reset offsets or modify topic configuration. Role-based access control lets you assign permissions at the operation level, per team. Without it, you are typically choosing between open access and restricted access, with nothing in between. In regulated environments, audit logging matters equally: a record of who did what, and when, is often a compliance requirement rather than a convenience.

Operational simplicity. A monitoring tool that is itself hard to maintain creates a different kind of overhead. Deployment model, configuration surface area, and the number of external dependencies all affect how much engineering time the dashboard consumes. The simpler the deployment, the less cognitive load it adds.

Kpow is built to meet these criteria without requiring significant configuration effort or ongoing maintenance.

Kpow: a Kafka dashboard for production environments

Kpow is a Kafka dashboard built by Factor House. It deploys inside your own infrastructure as a single stateless JVM container, connects directly to your Kafka cluster, and requires no external data transmission. Cluster data stays within your environment.

The deployment model is deliberate. For teams in regulated industries or with strict data residency requirements, sending cluster telemetry to an external SaaS platform is not always acceptable. Kpow runs on-premises, in a Docker container, in a Kubernetes pod, or on any JVM-compatible host, wherever your Kafka cluster lives.

It is designed for teams running Kafka seriously in production: platform engineers managing multiple clusters, data engineering teams that need reliable observability without assembling it from individual components, and organisations that require audit logging and access control as part of their compliance posture.

Monitoring Topics in Kpow

Core features of Kpow’s Kafka dashboard

Real-time consumer lag visibility. Kpow surfaces lag at the partition level, not just the consumer group level. From the consumer groups view, you can drill from a group summary to individual partitions in a few clicks, seeing current offset, log end offset, and lag count per partition as they update. For teams managing high-throughput pipelines, the difference between group-level and partition-level visibility is often the difference between catching a stalled consumer early and discovering it after records have accumulated for hours.

Broker and cluster health. The broker view surfaces JVM metrics alongside Kafka-specific health indicators: under-replicated partition counts, ISR state, and leader distribution across the cluster. Leader imbalance is a common source of uneven load in large clusters and is difficult to detect without partition-level visibility. Kpow aggregates this state without requiring a separate JMX exporter or Prometheus pipeline.

Topic and partition management. From the Kpow UI you can inspect topic configuration, view partition assignments, create new topics, and modify configuration without leaving the dashboard. Partition-level inspection shows which brokers hold which replicas and the current replication state, which is useful when investigating ISR issues or planning rebalancing operations.

Consumer group inspection and offset management. Kpow lets you inspect the current state of any consumer group: member assignments, partition ownership, current and committed offsets, and group coordinator. When you need to reset offsets, the operation is performed through the UI and recorded in the audit log, keeping offset management accessible to authorised users without requiring CLI access to the cluster.

Schema Registry integration. Kpow integrates with Confluent Schema Registry, exposing schema subjects, versions, and compatibility settings directly in the dashboard. Teams using Avro, Protobuf, or JSON Schema can browse schemas and view historical versions without switching to a separate tool. Schema Registry visibility is useful when debugging serialisation errors or planning schema evolution.

Multi-cluster support. A single Kpow deployment can connect to multiple Kafka clusters, including self-managed brokers and managed services such as Amazon MSK, Confluent Cloud, Redpanda, Aiven, StreamNative, Google Cloud Managed Kafka, NetApp Instaclustr, and Oracle Cloud. Each cluster appears under its own namespace in the UI with independent access control configuration. For platform teams managing several environments, this eliminates separate dashboard deployments per cluster and simplifies access management.

Kafka Streams and Kafka Connect. Kpow extends beyond core Kafka to cover the wider ecosystem. For teams using Kafka Streams, it surfaces stream topology state and task assignments. For teams using Kafka Connect, it shows connector state, task status, and configuration. Both are surfaces that are typically invisible in basic monitoring setups and that matter when debugging data pipeline failures.

Viewing Kafka Streams in Kpow

Kpow vs open-source alternatives

The most common alternatives to Kpow are AKHQ, CMAK, and a DIY stack built on Grafana, Prometheus, and JMX exporters. Each has a legitimate use case, but the trade-offs become significant as production requirements grow.

Capability	Kpow	AKHQ	Kafdrop	Grafana + Prometheus
Consumer lag (partition-level)	Yes	Yes	Yes	With JMX exporter
Topic and partition management	Yes	Yes	Read-only	No
Offset management	Yes	Limited	No	No
RBAC	Yes	Partial	No	Limited
Audit logging	Yes	No	No	No
Schema Registry integration	Yes	Yes	No	No
Kafka Streams visibility	Yes	No	No	No
Kafka Connect visibility	Yes	No	No	No
Multi-cluster (single deployment)	Yes	Yes	No	Yes (multiple scrape targets)
Commercial support	Yes	No	No	No

‍

AKHQ (v0.27.0, March 2025) is a capable open-source Kafka dashboard with topic browsing, consumer group inspection, and schema registry integration. It includes role-based access control with LDAP and OIDC integration, but the implementation has notable limits in production: there is no audit logging, no native data masking, and fine-grained multi-cluster RBAC requires additional configuration effort. For teams with compliance requirements around PII, HIPAA, PCI-DSS, or GDPR, these gaps are significant. AKHQ also has no commercial support, which means issue resolution depends on the open-source community and project maintainers.

Kafdrop (v4.2.0, July 2025) is a lightweight Kafka UI suited to development environments and ad-hoc inspection. It displays topic metadata, partition assignments, and message content, but it does not support consumer group offset reset, has no access control, and provides no audit trail. The project itself describes Kafdrop as a developer tool rather than team infrastructure. Most teams use it for local development convenience and look for something more capable in production.

Grafana with Prometheus and JMX exporters can surface Kafka broker metrics effectively, but it is a metrics pipeline, not a Kafka management tool. Setting it up requires configuring the JMX exporter per broker, writing or importing dashboard JSON, and managing the Prometheus scrape configuration. It provides no topic management, no offset manipulation, and no consumer group inspection. Keeping the JMX dashboards accurate as the cluster changes is also ongoing work. The result is a metrics view rather than a Kafka dashboard in the operational sense.

The core trade-off with open-source options is engineering time. The tooling is free, but configuring it, maintaining it, and covering the gaps around access control and audit logging is not. Kpow is a commercial product that runs out of the box and includes the capabilities that open-source alternatives require significant effort to approximate or cannot provide at all.

Who uses Kpow and when

Platform and infrastructure teams managing multiple environments. When a team is responsible for several Kafka clusters across development, staging, and production, operational overhead compounds quickly. Kpow’s multi-cluster support and centralised access control reduce the surface area they need to manage.

Engineering teams in regulated industries. Finance, healthcare, and government organisations often need audit logging and RBAC as part of their compliance posture, not as optional extras. Kpow provides both out of the box, deployed entirely within the organisation’s own infrastructure, with no data leaving the environment.

Development teams who need faster feedback. Even outside production, a team building against Kafka benefits from being able to inspect consumer groups, view messages, and confirm schema compatibility without reaching for the CLI. Kpow is fast to deploy in a development environment and straightforward enough that its value is immediate.

Getting started with Kpow

Kpow connects to any Kafka cluster via standard bootstrap properties. It deploys as a Docker container, a JAR, or via Helm chart, and can be running against your cluster in a few minutes. There is no agent to install on your brokers, and no cluster data leaves your environment.

You can try Kpow with a free 30-day trial. If you want to see it against your own cluster before committing, a guided demo is also available.

FAQ

What is a Kafka dashboard?

A Kafka dashboard is a web interface that aggregates and visualises the operational state of an Apache Kafka cluster. It typically displays broker health, consumer group lag, partition assignments, topic throughput, and schema registry state, giving platform and data engineering teams a single view of a cluster without running CLI commands manually.

What metrics should a Kafka dashboard show?

A production-grade Kafka dashboard should display consumer group lag by topic and partition, broker health and JVM metrics, under-replicated partition count, ISR state, topic throughput in messages and bytes per second, and consumer group member state. Partition-level detail is important: group-level summaries alone can mask significant lag in specific partitions.

Can I use Grafana as a Kafka dashboard?

Grafana combined with JMX exporters and Prometheus can surface Kafka broker metrics, but it requires significant setup: you need to configure the JMX exporter per broker, write or import dashboard JSON, and manage the Prometheus scrape pipeline. It also provides no topic management, offset manipulation, or consumer group inspection. For a team that needs both metrics and management, Grafana covers one part of the picture rather than acting as a complete Kafka dashboard.

Is there a free Kafka dashboard?

Several open-source Kafka dashboards are available at no cost. AKHQ and Kafdrop are the most widely used. Both offer topic browsing and consumer group inspection. AKHQ includes partial RBAC via LDAP and OIDC integration; Kafdrop has no access control at all. Neither provides audit logging, which is a common requirement in regulated environments.

How do I monitor Kafka consumer lag?

Consumer lag in Kafka is the difference between the latest offset produced to a partition and the last offset committed by a consumer group. You can check it with kafka-consumer-groups.sh --describe for a point-in-time snapshot, or use a dashboard that polls __consumer_offsets and surfaces lag per partition continuously. Partition-level visibility is what distinguishes a production-grade dashboard from basic tooling: group-level averages can look acceptable while individual partitions fall significantly behind.

What is the difference between a Kafka dashboard and Kafka monitoring?

Kafka monitoring is the broader practice of collecting, alerting on, and analysing Kafka metrics, often using Prometheus, Grafana, or a commercial observability platform. A Kafka dashboard adds interactive management: browsing topics, inspecting consumer groups, managing offsets, and viewing schemas. Most Kafka dashboards include monitoring views; most monitoring tools do not include management capabilities.

How does Kpow compare to AKHQ?

Kpow is a commercial Kafka dashboard with RBAC, audit logging, multi-cluster support, and Kafka Streams and Connect visibility built in. AKHQ is an open-source alternative with strong topic and consumer group browsing and some RBAC via LDAP and OIDC, but no audit logging and limited fine-grained access control in multi-cluster environments. The main trade-off is operational overhead: AKHQ requires more configuration and ongoing maintenance; Kpow is designed to run in production without significant setup.

Does Kpow support Confluent Cloud and Amazon MSK?

Yes. Kpow supports a range of managed Kafka services including Amazon MSK, Confluent Cloud, Redpanda, Aiven, StreamNative, Google Cloud Managed Kafka, NetApp Instaclustr, and Oracle Cloud. Connection is configured via standard Kafka bootstrap properties. Check the Kpow documentation for any provider-specific configuration notes.

What is the best Kafka monitoring tool?

The right tool depends on your requirements. For teams that need metrics only, Prometheus with JMX exporters and Grafana is a common open-source approach. For teams that need management capabilities, RBAC, audit logging, and cross-cluster visibility in a single product, a dedicated Kafka dashboard like Kpow is a better fit. Open-source tools like AKHQ or Kafdrop suit development environments or smaller teams with lower compliance requirements.

How do I visualise Kafka consumer groups?

Consumer group state can be viewed with kafka-consumer-groups.sh --describe, which shows lag, current offset, log end offset, and consumer ID per partition. A Kafka dashboard goes further: it shows group state (stable, rebalancing, empty), member assignments per partition, and real-time lag trends, making it easier to spot rebalancing storms or stalled consumers without scripting or periodic manual checks.

Is Kpow a Kafka dashboard?

Yes. Kpow is a Kafka dashboard built by Factor House. It provides real-time visibility into broker health, consumer lag, topic and partition state, schema registry, Kafka Streams, and Kafka Connect, all from a single UI deployed within your own infrastructure.

Kafka management console: what to look for in a tool

Factor House — Sat, 27 Jun 2026 00:00:00 GMT

Operating Kafka through the CLI works at small scale, but it creates problems as teams and environments grow. A Kafka management console solves that by centralising administrative operations behind a web interface that any team member can use without needing direct broker access. This article defines what a capable console should cover, maps the current tool landscape, and explains what separates a production-ready console from a basic monitoring UI.

What is a Kafka management console?

A Kafka management console is a web-based interface for operating and administering an Apache Kafka environment. It provides controls for topics, consumer groups, schemas, connectors, and access management, replacing CLI-based operations with a UI that can be safely delegated to team members without direct broker access.

The distinction worth drawing is between a console and a read-only monitoring tool. A monitoring tool shows you what is happening: metrics, consumer lag, throughput, broker health. A management console lets you act on it. You can create or reconfigure a topic, reset a consumer group offset, deploy a connector, or revoke a user’s access, all from the same interface, without touching the CLI or exposing broker credentials.

The Kafka CLI covers the same administrative operations, but it requires shell access, direct cluster connectivity, and enough familiarity with the tooling to avoid mistakes under pressure. A console delegates those operations through a controlled interface. That separation matters when multiple teams need access to a shared cluster, when your compliance requirements demand an audit trail, or when you want to avoid the kind of offset reset that takes the wrong consumer group offline.

The rest of this article defines what a production-grade console should cover and how to evaluate the options available.

An example Kafka management console - Kpow by Factor House

Who needs a Kafka management console

A command-line workflow is reasonable for a single engineer managing one cluster. It becomes a liability at most other scales.

Platform and infrastructure teams responsible for multiple environments face a coordination problem. Running separate CLI tooling per cluster, with per-environment credentials, is slow and error-prone. A multi-cluster console reduces that to a single interface.

Engineering teams in regulated industries (financial services, healthcare, payments) often have non-negotiable requirements around access control and audit trails. The Kafka CLI provides neither. A console that implements RBAC and immutable audit logging meets those requirements in a way that CLI workflows cannot.

Organisations scaling beyond a single cluster hit a point where ad-hoc CLI operations create coordination risk. When multiple teams are creating topics, managing consumer groups, or deploying connectors on the same cluster, a console with RBAC prevents one team’s operations from affecting another’s.

Teams running Kafka Connect or Kafka Streams at scale find CLI-only connector management particularly fragile under time pressure. Pausing a connector, checking task health, or restarting a failed task through direct REST API calls is workable in isolation but introduces risk in shared environments.

Teams that need to delegate Kafka access safely, giving topic owners limited visibility and control without sharing admin credentials, need RBAC at the console level, not just at the Kafka ACL level.

Core management capabilities

The table below is a reference checklist for evaluating any Kafka management console. Not every tool covers all of these; some focus on monitoring only, some cover a subset of management operations, and a smaller number support the full stack including Connect, Streams, RBAC, and audit logging.

Capability	What it enables	Why it matters in production
Topic management	Create, configure, inspect, and delete topics from the UI	Eliminates direct CLI access for routine operations; reduces misconfiguration risk
Consumer group control	View member state, partition assignment, and reset offsets	Essential for recovery workflows; dangerous to perform ad hoc from the CLI under load
Partition reassignment	Rebalance partitions across brokers	Required after scaling or rack changes; complex to do manually
Schema Registry integration	Manage Avro, Protobuf, and JSON schemas with compatibility checks	Prevents schema breaks in production pipelines
Kafka Connect management	Deploy, configure, pause, and restart connectors	Removes CLI dependency for connector operations
Kafka Streams inspection	View running topologies and task health	Surfaces lag and failure states that do not appear in standard broker metrics
Data inspect	Browse, filter, and deserialise messages in the console	Critical for debugging; saves building bespoke consumer scripts for one-off investigations
RBAC	Role-based access control scoped to cluster, topic, or consumer group	Enables safe delegation without sharing admin credentials
Audit logging	Immutable record of all console actions	Required for regulated industries; invaluable for incident investigation
Multi-cluster support	Operate multiple environments from one interface	Reduces context-switching and avoids separate tooling per environment

Coverage varies considerably across tools. The sections below map the current landscape before covering how Kpow addresses the full capability stack.

Open-source vs commercial Kafka management consoles

Several open-source tools cover the basics well. Commercial tools add security features, enterprise support, and multi-cluster capabilities that become relevant as requirements grow.

The main open-source options

AKHQ (formerly KafkaHQ) is the most widely deployed open-source Kafka management UI. It covers topic management, consumer group inspection, Schema Registry, and Kafka Connect. The project is actively maintained under the Apache 2.0 licence, with v0.26.0 released in March 2026. AKHQ has no paid tier and no commercial support offering. RBAC is limited compared to commercial tools.

Kafbat is the community-maintained continuation of Provectus kafka-ui, which Provectus paused in September 2023. Kafbat is actively developed: version 1.5.0 (released in 2025) added live consumer lag updates, MessagePack serialisation support, and CSV export. RBAC is available through YAML-based configuration, with OAuth2 (Google, GitHub, Azure AD), LDAP, and Active Directory support. Notable gaps include no per-role data masking, no approval workflows, and no policy enforcement layer.

Redpanda Console has an open-source edition that covers topic management, consumer group inspection, and basic data browsing. In the open-source tier, RBAC and SSO are not available; those are enterprise-only features. The open-source edition is a reasonable choice for development environments or teams using Redpanda itself.

CMAK (Cluster Manager for Apache Kafka, originally Kafka Manager from Yahoo) is no longer actively maintained. The last release was 3.0.0.6 in April 2022. More critically, CMAK requires a ZooKeeper endpoint and has no KRaft support, making it incompatible with Kafka 4.0 and later. It is not a viable choice for new deployments.

AKHQ, a popular open-source Kafka management console

What open-source tools do well and where they fall short

Open-source tools handle the core daily operations (topic creation, consumer group inspection, basic message browsing) without significant gaps. For a single-cluster development environment or a small team with no compliance requirements, AKHQ or Kafbat is a defensible choice.

The gaps appear at the edges that matter in production: RBAC is either absent, limited, or requires significant configuration work; audit logging is typically not provided; multi-cluster support is limited or absent; and commercial support is not available. If access control, audit trails, and multi-cluster management are requirements rather than nice-to-haves, open-source tools require substantial additional work to meet them.

What commercial tools add

Commercial tools such as Kpow, Conduktor, and Lenses.io typically include RBAC as a first-class feature, immutable audit logging, multi-cluster management from a single interface, and commercial support contracts. The trade-off is licence cost and the added consideration of running a closed-source tool in your infrastructure.

For regulated environments, or for platform teams managing more than one or two clusters with multiple teams sharing access, commercial tooling generally reduces the operational burden significantly compared to assembling equivalent capabilities from open-source tools.

How Kpow covers the full capability stack

Kpow is a commercial Kafka management console built by Factor House. It covers the full capability table above: topic management, consumer group control, Schema Registry, Kafka Connect, Kafka Streams, data inspect, RBAC, audit logging, and multi-cluster support, all from a single interface.

Data inspection in Kpow

Deployment model. Kpow deploys inside your own infrastructure. It runs as a Docker container, a Helm chart on Kubernetes, a JAR, via AWS CloudFormation on ECS, or from the AWS Marketplace. No data leaves your environment: agents connect directly to your brokers, Schema Registry, and Connect clusters. There are no external dependencies and no inbound connections required.

Multi-cluster management. A single Kpow instance can manage multiple Kafka clusters, Schema Registries, and Connect clusters. This is useful for teams operating separate environments (production, staging, development) or multiple independent clusters, where switching between separate tools per cluster creates overhead.

RBAC and audit logging. RBAC in Kpow is scoped at the cluster, topic, or consumer group level: you can give a team read access to one topic and write access to another without granting broader permissions. Audit logging records every action taken through the console with a timestamp and user identity. Both are first-class features, not optional add-ons.

Data inspect. Kpow’s data inspect supports Avro, Protobuf, JSON, and custom deserializers. Operators can browse, filter, and deserialise messages without exposing raw broker access, which is important in environments where message contents are sensitive.

Kafka Streams support. Kpow surfaces running topologies and task health for Kafka Streams applications, including information that does not appear in standard broker metrics and is difficult to access without dedicated tooling.

Frequently asked questions

What is a Kafka management console?

What’s the difference between a Kafka management console and a Kafka monitoring tool?

A monitoring tool shows you what is happening: metrics, lag, throughput. A management console lets you act on it: create or modify topics, reset consumer group offsets, manage connectors, and control access. Many tools do both, but the distinction matters when evaluating whether a tool is read-only or operational.

Can I manage Kafka without using the CLI?

Yes. A Kafka management console exposes the same administrative operations as the Kafka CLI (topic creation, consumer group management, offset resets, partition reassignment) through a web UI. Most consoles also add RBAC so operators can perform these actions without needing direct shell or broker access.

What are the best open-source Kafka management consoles?

The most widely used open-source options are AKHQ and Kafbat (a maintained fork of Provectus kafka-ui). Redpanda Console has an open-source edition suited to Redpanda deployments. CMAK is no longer actively maintained and is incompatible with Kafka 4.0 due to its ZooKeeper dependency. AKHQ and Kafbat are the most defensible choices for standard Apache Kafka deployments.

What is Kafka RBAC and does a management console support it?

Kafka RBAC (role-based access control) lets you define what each user or team can do: view topics, reset offsets, manage connectors, scoped to specific clusters or resources. Not all consoles implement RBAC; those that do typically layer it on top of Kafka’s native ACLs. Kpow implements RBAC at the console level, scoped to cluster, topic, or consumer group.

How do I reset consumer group offsets without the CLI?

A management console with consumer group control lets you select a group, choose a topic and partition, and reset the offset to a specific position: earliest, latest, a timestamp, or a specific offset value. This avoids running kafka-consumer-groups.sh directly and reduces the risk of resetting the wrong group or partition under pressure.

Can I manage multiple Kafka clusters from one interface?

Yes, if the console supports multi-cluster management. Kpow, Conduktor, and Lenses.io all support multiple clusters from a single interface. Most open-source tools are single-cluster by default, though some support switching between configured environments.

What is Kafka audit logging and why does it matter?

Audit logging in a Kafka management console records every action taken, including who created or deleted a topic, who reset a consumer group, and who changed a connector, along with a timestamp and user identity. This is required in regulated industries for compliance and is invaluable for diagnosing production incidents caused by administrative changes.

How do I manage Kafka Connect from a console?

A console with Connect support lets you deploy, configure, pause, restart, and delete connectors through the UI, and view connector status and task health. This replaces direct calls to the Connect REST API and is safer in shared environments where multiple teams manage connectors on the same cluster.

What’s the difference between Kpow, Conduktor, and AKHQ?

Kpow and Conduktor are commercial tools with enterprise features: RBAC, audit logging, multi-cluster support, and commercial support contracts. Compared to Conduktor, Kpow does not require a proxy in the data path. AKHQ is open-source and widely used for topic and consumer group management but has more limited RBAC and no audit logging capabilities. The right choice depends on team size, compliance requirements, and whether commercial support matters.

Is there a free Kafka management console?

Yes. AKHQ and Kafbat are free and open-source. Redpanda Console has a free open-source edition. Kpow has a Community Edition and an Enterprise trial. For development and small team use, open-source tools are generally sufficient; commercial options are more common in regulated or large-scale production environments where RBAC and audit logging are requirements.

How do I inspect Kafka messages without writing a consumer?

Most management consoles include a data inspect or message browser feature. You select a topic, optionally filter by key, header, or value, choose a time range or offset, and the console deserialises and displays messages, supporting Avro, Protobuf, JSON, and sometimes custom deserializers. This replaces ad-hoc consumer scripts for debugging.

Getting started with Kpow

To get started with Kpow you need a running Kafka cluster (self-managed, Amazon MSK, Confluent Cloud, Redpanda, or another Kafka-compatible platform) and a container runtime or JVM environment.

Kpow deploys as a single container or JAR. Configuration is environment-variable based: you point Kpow at your bootstrap servers, and optionally at your Schema Registry and Connect clusters, and it connects directly without any intermediate services or agents. A typical install takes under 30 minutes from first pull to a running console.

Deployment options include Docker, Docker Compose for local development, Helm for Kubernetes, AWS CloudFormation for ECS, and the AWS Marketplace. The same artefact covers all environments.

Kpow is maintained by Factor House with commercial support available for Enterprise customers.

Give Kpow a try with a free 30-day Enterprise trial. You can connect it to any Kafka cluster in minutes and deploy via Docker, Helm, or JAR. No credit card is required for the Community Edition.

Kafka message key best practices

Factor House — Sat, 27 Jun 2026 00:00:00 GMT

What is a Kafka message key?

A Kafka message key is an optional byte array attached to every record produced to a topic. The producer sets it independently from the message value, and the broker never inspects its contents. In the Java client, the key is serialized via key.serializer, a separate configuration from value.serializer.

The key serves exactly two purposes in Kafka:

Routing. When no explicit partition is specified, the producer’s Partitioner decides which partition a record goes to. With the default Java partitioner, the rule is:

partition = Math.abs(Utils.murmur2(keyBytes)) % numPartitions

This is deterministic: the same key always maps to the same partition, for as long as numPartitions stays constant.

Compaction identity. For topics with cleanup.policy=compact, the key is the primary key against which older values are garbage-collected. Kafka retains only the latest value per key.

The key plays no role in authentication, deduplication, or consumer-side indexing. Consumers can ignore the key entirely if their use case doesn’t require it.

Kafka message structure

A Kafka record consists of four producer-supplied fields and three broker-assigned fields:

Field	Set by	Description
Key	Producer	Arbitrary bytes. Drives partition routing and compaction.
Value	Producer	Arbitrary bytes. The payload.
Headers	Producer	Optional key-value metadata. Useful for routing metadata, correlation IDs, schema version hints.
Timestamp	Producer / Broker	Either CreateTime (producer clock) or LogAppendTime (broker clock), depending on message.timestamp.type.
Topic	Broker	The topic the record was written to.
Partition	Broker	The partition within the topic.
Offset	Broker	The monotonically increasing position within the partition.

Kafka’s only ordering guarantee is per-partition. Records with the same key land on the same partition and are therefore ordered relative to each other. Records with different keys may land on different partitions, and Kafka makes no ordering promises across partitions.

For common key serializer choices: StringSerializer, LongSerializer, and UUIDSerializer cover the vast majority of production use cases and produce keys that are readable in CLI tooling such as kafka-console-consumer --print.key. Schema Registry serializers (KafkaAvroSerializer, KafkaProtobufSerializer) work for keys, but most teams use them only on the value and keep keys as primitive types for human-readable inspection.

Kafka message key best practices

1. Use a key when you need per-entity ordering

Kafka guarantees message order within a partition. If your consumers need events for a specific entity to arrive in the order they were produced, that entity’s identifier must be the key.

Common examples: accountId for banking transactions (credits and debits must apply in sequence), orderId for e-commerce state transitions, driverId or deliveryId for ride-hailing status updates, deviceId for IoT telemetry. In each case, the ordering boundary is the entity, and the key must reflect that boundary precisely. Choosing a coarser key, such as region or status, groups unrelated entities onto the same partition without providing the ordering guarantee your consumers actually need.

2. Use null keys deliberately when ordering does not matter

For log ingestion, metrics pipelines, telemetry, and other append-only workloads where no entity-level ordering is required, producing with a null key is the correct choice.

With null keys, the producer’s partitioner handles distribution. Since Kafka 3.3, the default adaptive partitioner (KIP-794) batches records to a partition based on bytes produced rather than time, and actively routes away from slow brokers. Confluent’s own benchmarks for the KIP-794 partitioner reported p99 latency dropping from roughly 11 seconds to 154 milliseconds under an abnormal-broker scenario.

If you are still on Kafka 2.4-3.2, the sticky partitioner (KIP-480) handles null keys by batching to one partition until linger.ms elapses or the batch fills, then rotating. DoorDash reported tuning linger.ms to 50-100 ms reduced broker CPU by 30-40% on their high-volume null-keyed event ingestion pipeline.

The risky case is accidentally null keys on a topic that requires ordering. A producer that fails to set a key on every record will exhibit intermittent ordering failures that are difficult to reproduce. Add a producer interceptor or unit test that rejects null keys for any topic where per-entity ordering is a requirement.

3. Choose high-cardinality, evenly distributed keys

The murmur2 hash distributes keys across partitions, but only as evenly as the key’s own distribution allows. A low-cardinality key will leave most partitions idle and concentrate traffic on a few.

Good key candidates: userId, orderId, deviceId, sessionId, accountId, transactionId. These typically have many distinct values and no single dominant value.

Poor key candidates: country, region, status, eventType, boolean flags. With status ∈ {SUCCESS, FAILED}, only two partitions ever receive traffic regardless of how many partitions the topic has.

tenantId is a common intermediate case. It may appear high-cardinality, but most B2B systems have a power-law distribution of tenant size. One large tenant can produce the majority of your traffic. If you need tenant-level grouping, combine tenantId with a finer-grained field using a composite key.

4. Always key by an immutable identifier

A key that changes over time breaks ordering and creates orphan records in compacted topics. The most common mistake is using a mutable field like username, email, or status as the key.

If a user changes their username, every subsequent event for that user lands on a different partition from all historical events. Any downstream consumer joining on the key now sees two different histories for the same logical entity. In a compacted topic, the compactor treats the old and new keys as separate records, keeping both as the “current state” indefinitely.

Always use a surrogate, immutable identifier: userId rather than username, accountNumber rather than email, deviceId rather than hostname. If the natural identifier in your domain is mutable, generate an internal UUID at entity creation time and use that as the key.

5. Keys are required for log compaction

cleanup.policy=compact instructs Kafka to retain only the latest record for each key. This makes a compacted topic behave like a distributed key-value store of current state. It is the foundation of Kafka Streams’ changelog topics, CDC streams, and any event-sourced projection that needs to rebuild state from scratch.

A null-keyed record on a compacted topic cannot be compacted. It sits in the log indefinitely, causing unbounded disk growth. Confluent’s documentation is explicit on this point: compacted topics require records with keys.

A null value paired with a non-null key is a tombstone. It signals the compactor to delete that key once delete.retention.ms elapses. If you need to “delete” a record from a compacted topic, produce a tombstone with the same key.

Add a producer-side interceptor that rejects null keys for any topic with cleanup.policy=compact. Catching this in the producer is far cheaper than diagnosing unbounded disk growth in production.

6. Align keys for stateful stream processing

Kafka Streams, ksqlDB, and Flink all require co-partitioning for joins and aggregations. Co-partitioning means: the same number of partitions on both sides, the same partitioning strategy, and the same key type.

Kafka Streams enforces this at startup and will throw TopologyException: Topics not co-partitioned if partition counts differ across joined topics. ksqlDB enforces it at query creation time.

If you cannot co-partition the source topics, the options are: insert an explicit KStream#repartition() step (Kafka 2.6+ via KIP-221) to repartition on the fly, or use GlobalKTable for the smaller side of a join (which broadcasts the entire topic to every instance).

When multiple producer applications write to the same topic, they must all use the same partitioner. The Java default client uses murmur2. The librdkafka client (used by Python, Go, .NET, and C# producers) defaults to CRC32 with random assignment for null keys. Two producers using different hash functions and writing the same logical key will route to different partitions, silently breaking co-partitioning and any downstream join. Set partitioner=Murmur2Random on all librdkafka clients that share a topic with a Java producer.

7. Standardize key serializers across producer teams

A producer using IntegerSerializer and a consumer using StringDeserializer will deserialize silently incorrect data, potentially without throwing an exception. The failure typically surfaces as incorrect behavior in a downstream state store or join.

Define the key serializer in a shared schema or data contract alongside the value schema. For topics consumed by Kafka Streams or ksqlDB, include the key type in the topic metadata. Some teams track this in Confluent Schema Registry using RecordNameStrategy for the key subject; others maintain a topic catalog alongside their schema definitions. Whichever mechanism you use, the key type should be as explicit and controlled as the value schema.

8. Plan for hot keys before you need to

Every production Kafka system at sufficient scale encounters key skew. New Relic’s events pipeline team found that 1.5% of their query keys produced 90% of the events processed for aggregation. Slack observed per-broker hot-spotting persistent enough to mandate that every topic’s partition count be a multiple of the broker count.

The diagnostic approach: use kafka-consumer-groups.sh --describe and look for a single partition with 5-10x the consumer lag of its neighbours. Even disk usage across brokers but uneven consumer lag typically points to a processing bottleneck rather than key skew. Confirm by sampling the top keys by message rate for an hour. If the top 1% of keys produce more than 20% of traffic, you have a skew problem.

Mitigations, in order of preference:

Composite keys combine a high-cardinality field with a coarser one for locality: tenantId|userId or region|customerId. This preserves logical grouping while spreading load across more hash buckets.

Salting appends a deterministic random suffix to known hot keys: userId#0 through userId#K-1. Per-user ordering is lost, but per-(user, salt) ordering is preserved. Consumers that need a final per-user aggregation re-key downstream on userId only. This pattern is used by Pinterest’s MemQ system for storage-layer hot key mitigation.

Time-window bucketing appends a time bucket: userId|2025-10-21T10:00. This works for hotness that is bursty rather than sustained.

Do not respond to consumer lag by immediately adding partitions. That breaks the hash mapping and the ordering guarantee for keyed topics, as described in point 9 below.

9. Over-provision partitions at topic creation

Adding partitions to an existing topic changes the murmur2 modulus. Records for customer-42 produced before the change and records produced after may land on different partitions. Per-key ordering is broken from that point forward, with no automated warning to downstream consumers.

Confluent’s operations documentation is explicit: “If the number of partitions changes, this delivery guarantee may no longer hold.”

Choose your initial partition count based on the consumer parallelism you expect 12-18 months out, then over-provision beyond that by a factor of 2-3. For small clusters, Slack’s rule is useful: make partition counts a multiple of the broker count to distribute partition leadership evenly. For large clusters, anything under 100 partitions per topic starts to produce uneven load distribution.

If you must add partitions, do it during a low-traffic window, document that ordering is broken at the cutover point, and validate that downstream Kafka Streams topologies do not assume positional ordering across the partition count boundary.

10. Rekey via a new topic, never in-place

If the key on an existing topic needs to change, the safe pattern is to create a new topic with the desired key and partition count, run a rekeying pipeline to migrate data, and then cut consumers over once they have caught up.

The steps:

Create the new topic (orders.v2) with the target partition count and key scheme.
Run a rekeying consumer: sourceStream.selectKey(...).to("orders.v2") in Kafka Streams, or an equivalent Flink or custom consumer.
Dual-write from producers to both topics during the migration window.
Switch consumers to the new topic once consumer lag on orders.v2 reaches zero.
Drain and delete the old topic.

Rewriting keys into the same topic is not a safe operation. It interleaves records under two different key schemes in the same partition, making it impossible for any consumer to maintain a consistent ordering guarantee.

11. Set idempotent producer configuration

Producer configuration affects ordering guarantees independently of the key. Without an idempotent producer, setting retries > 0 and max.in.flight.requests.per.connection > 1 allows retries to reorder records within a partition even when the key is correct.

The recommended production configuration:

enable.idempotence=true acks=all max.in.flight.requests.per.connection=5 retries=2147483647

enable.idempotence=true has been the default since Kafka 3.0. The broker tracks a producer ID and sequence number per partition and deduplicates retries automatically. If you are running a producer configuration inherited from before Kafka 3.0, audit it before assuming ordering is safe.

12. Document key semantics alongside the schema

The key type, cardinality expectations, and ordering guarantees are not inferable from the schema alone. A topic named user-events with a userId key does not communicate whether records are ordered per-user, whether the topic is compacted, or whether multiple producer applications write to it.

At minimum, document: the key type and serializer, what the key represents, whether per-key ordering is guaranteed, and whether the topic is compacted. Teams using Confluent Platform have Topic Tags and data contract features for this. If you are managing Kafka independently, a README alongside your schema definitions or an entry in your internal data catalog is sufficient.

How Kpow helps with Kafka message key management

Kpow provides observability and management capabilities that are useful at several points in the key lifecycle described above.

Inspecting key content and distribution. Kpow’s Data Inspect feature allows you to search topic records by key directly. The Key mode queries records from one or more topics, starting from a configured point in time, and matches on an exact key value. This is useful for confirming that a key is routing to the expected partition, checking that a compacted topic is not accumulating null-keyed records, or debugging serialization mismatches. Auto SerDes mode will attempt to infer the key and value deserialization format automatically, which reduces the setup overhead when you are inspecting an unfamiliar topic for the first time.

kJQ filters allow server-side filtering across message keys and values at high throughput, which is practical for profiling key distribution across large topics.

Monitoring consumer lag per partition. Kpow’s consumer group monitoring shows lag at the partition level, which is the first place a hot key problem becomes visible. A single partition running significantly behind its peers is the diagnostic signal described in best practice 8. Kpow exposes this breakdown in the consumer group topology view and in the detailed consumer table.

Managing offsets during rekeying operations. When executing the new-topic rekeying pattern described in best practice 10, Kpow’s offset management tools let you reset or adjust consumer group offsets at the group, topic, or partition level, which can be necessary when validating that a cutover consumer has fully caught up before switching production traffic.

Kpow connects to any Kafka cluster and deploys via Docker, Helm, or JAR. You can try it with a free 30-day trial.

For guidance on managing message payload size alongside key strategy, see our Kafka message size best practice guide.

Kafka security architecture: best practices for production

Factor House — Sat, 27 Jun 2026 00:00:00 GMT

Key takeaways

Kafka ships with authentication, encryption, and authorization disabled by default. Every security control must be explicitly configured.
There are four distinct traffic flows in a Kafka deployment, each requiring its own security controls: client-to-broker, broker-to-broker, broker-to-controller, and admin clients.
A layered security model covering network isolation, TLS, authentication, authorization, and audit logging is more reliable than any single control in isolation.
ZooKeeper is deprecated as of Kafka 4.0. Clusters still running ZooKeeper mode will lose security patches in November 2025.
Native ACLs become difficult to manage at scale. External authorizers (OPA, Ranger, or custom implementations) address the operational limits.
The gap between “audit logging is enabled” and “audit logging is actionable” is wider than most teams expect.

Security challenges unique to Kafka

The most important thing to understand about Kafka security is the starting position: Kafka ships insecure by default. Authentication, encryption, and authorization are all off out of the box. An unconfigured Kafka cluster allows any application that can reach the network to produce, consume, and administer the cluster without restriction.

This is documented behavior, not a bug. Kafka was designed for ease of use in development environments. The problem is that teams stand up a cluster for testing, skip hardening, and then find themselves with a production deployment built against open listeners.

Beyond the defaults, Kafka’s attack surface is more complex than a conventional service with a single authentication boundary. There are four distinct traffic flows, each of which requires its own security controls:

Client to broker: Producers and consumers connecting to send or receive data
Broker to broker: Inter-broker replication traffic, which is the most commonly missed flow
Broker to controller: Control plane traffic managing cluster metadata, ACLs, and topic configurations (ZooKeeper in older deployments, KRaft controllers in current ones)
Admin clients: CLI tools, management UIs, Kafka Connect workers, Schema Registry, all operating as principals against the cluster

You can configure TLS and authentication perfectly for client connections and remain exposed through an unauthenticated ZooKeeper port or an admin UI with no access controls. Each flow needs to be accounted for independently.

Kafka is also typically multi-tenant in practice. Multiple teams, services, and sometimes external partners share the same infrastructure. Every new producer-consumer relationship expands the authorization surface that needs to be maintained.

The four pillars of Kafka security

Pillar	Mechanism	Threat it addresses
Encryption	TLS	Data intercepted in transit
Authentication	SASL / mTLS	Unauthorized clients connecting
Authorization	ACLs / RBAC	Legitimate clients overreaching
Audit logging	kafka.authorizer.logger + external tooling	Undetected breaches or misuse

‍

Each pillar addresses a distinct threat. Implementing three of four still leaves a meaningful gap: a cluster with strong authentication and fine-grained ACLs but no audit logging gives you no visibility into what happened when something goes wrong.

Encryption: securing data in transit

The three flows you need to encrypt

Most teams focus on client-to-broker TLS and stop there. Two other flows are commonly overlooked.

Inter-broker replication is the most frequently missed. Replication traffic carries all data transiting the cluster. Teams that configure TLS for client connections often leave inter-broker traffic on PLAINTEXT because it is “internal.” The relevant config property, security.inter.broker.protocol, defaults to PLAINTEXT. You need to explicitly set it to SSL or SASL_SSL. Anyone with network access between broker nodes can intercept replication traffic if this is not configured.

Broker-to-controller traffic (either ZooKeeper or KRaft controller) is similarly often left unencrypted. Control plane traffic includes ACL definitions, topic configurations, and cluster membership data.

A minimal broker configuration with inter-broker encryption enabled:

listeners=SASL_SSL://public:9093,SSL://internal:9094 advertised.listeners=SASL_SSL://public.example.com:9093,SSL://internal.example.com:9094 listener.security.protocol.map=SASL_SSL:SASL_SSL,SSL:SSL sasl.enabled.mechanisms=SCRAM-SHA-256 sasl.mechanism.inter.broker.protocol=SCRAM-SHA-256 ssl.keystore.location=/var/private/ssl/kafka.server.keystore.jks ssl.keystore.password=[keystore-secret] ssl.truststore.location=/var/private/ssl/kafka.server.truststore.jks ssl.truststore.password=[truststore-secret]

TLS version

TLS 1.3 should be enforced for all production traffic. Older TLS versions have known vulnerabilities and should be explicitly disabled via ssl.enabled.protocols.

Certificate management

Certificate rotation is where TLS implementations tend to break down in practice. Teams configure TLS, confirm it is working, and then do not rotate certificates until something expires in production. Automation is the correct approach: certificate rotation should be handled by your PKI infrastructure on a schedule, not manually triggered when something breaks. Uber’s internal PKI (uPKI) pushes new X.509 key-cert pairs to brokers and clients before TTLs expire; that pattern is worth replicating regardless of the tooling you use.

Encryption at rest

Kafka does not provide built-in encryption at rest. Data stored on broker disks is protected only at the infrastructure layer: volume encryption (AWS EBS, GCP Persistent Disk) or OS-level encryption. In environments requiring stronger guarantees, some teams encrypt message payloads before they reach the broker, so that the broker itself cannot read the data. This approach adds key management and downstream decryption complexity, but it is the appropriate model for zero-trust environments handling sensitive data.

Authentication: proving identity

Kafka supports four SASL mechanisms and mTLS as an alternative authentication model. The right choice depends on your environment and existing identity infrastructure.

SASL mechanisms compared

SASL/PLAIN sends credentials in cleartext. It is only acceptable over an encrypted connection (SASL_SSL, not SASL_PLAINTEXT). A common misconfiguration is using PLAIN over an unencrypted listener, which exposes usernames and passwords on the wire. Restrict PLAIN to development environments.

SASL/SCRAM (SHA-256 or SHA-512) uses a challenge-response protocol that does not transmit passwords in cleartext. Credentials are stored in ZooKeeper (ZK mode) or cluster metadata (KRaft). SCRAM-SHA-256 is the minimum for production password-based authentication; SCRAM-SHA-512 is stronger.

SASL/GSSAPI (Kerberos) integrates with enterprise identity systems via Active Directory or LDAP. It provides SSO across the organization and strong authentication, at the cost of operational complexity: Kerberos infrastructure, ticket renewal management, and keytab files. The right choice for large on-premises enterprises already running Kerberos. Less practical for cloud-native environments or teams without existing Kerberos investment.

SASL/OAUTHBEARER uses OAuth 2.0 / OpenID Connect tokens. As of 2025 this is the standard authentication mechanism for cloud-native Kafka deployments. It integrates with Okta, Keycloak, Entra ID, and similar identity providers. Short-lived tokens reduce the credential exposure window compared to long-lived passwords, and centralized identity management simplifies principal lifecycle.

mTLS (mutual TLS) requires both the client and the broker to present certificates, providing two-way authentication. The broker validates the client by certificate rather than by username and password. It works well for service-to-service communication where you can automate the certificate lifecycle. It becomes operationally burdensome for human users and dynamic client environments where certificate distribution is difficult to manage.

Choosing a mechanism

Scenario	Recommended mechanism
Cloud-native, modern IdP	SASL/OAUTHBEARER
Enterprise on-prem with Kerberos	SASL/GSSAPI
Service-to-service, automatable certs	mTLS
Simpler production environment, no Kerberos	SASL/SCRAM-SHA-256
Development only	SASL/PLAIN (with TLS)

Authorization: locking down topics, groups, and the cluster

How Kafka ACLs work

A Kafka ACL is a rule with the structure: Principal + Resource + Operation + Permission (Allow/Deny).

Principal: A user identity, for example User:consumer-app or User:kafka-broker
Resource types: Topics, Consumer Groups, Cluster, Transactional IDs, Delegation tokens
Operations: Read, Write, Create, Delete, Alter, Describe, DescribeConfigs, AlterConfigs, ClusterAction, IdempotentWrite, All
Permission: Allow or Deny

When allow.everyone.if.no.acl.found=false (the setting you should always use in production), access is denied by default if no ACL matches. The consequence: the first thing you need to do when enabling authorization is create ACLs for your brokers themselves, so they can replicate.

Role patterns

A read-only consumer needs READ on specific topics, READ on its consumer groups, and DESCRIBE on topics. Missing the DESCRIBE permission causes cryptic “unknown topic” errors even when the topic exists, which is a common troubleshooting dead end.

A write-only producer needs WRITE on specific topics and DESCRIBE on the cluster for metadata requests.

`# Minimal producer ACL
kafka-acls –bootstrap-server kafka-broker:9093 \
–command-config admin.properties \
–add \
–allow-principal User:producer-app \
–operation WRITE \
–operation DESCRIBE \
–topic orders

Minimal consumer ACL

kafka-acls –bootstrap-server kafka-broker:9093 \
–command-config admin.properties \
–add \
–allow-principal User:consumer-app \
–operation READ \
–operation DESCRIBE \
–topic orders

kafka-acls –bootstrap-server kafka-broker:9093 \
–command-config admin.properties \
–add \
–allow-principal User:consumer-app \
–operation READ \
–group consumer-group-1`

The practical failure mode in authorization is permission creep. ACLs accumulate over time: teams grant broad permissions to reduce access request friction, services get decommissioned without their principals being revoked, and you end up with producers that can also consume and consumer groups with write access they should not have. Auditing ACLs regularly is as important as setting them correctly in the first place.

The limits of native ACLs

Native Kafka ACLs become operationally difficult at scale for several reasons:

Every user-topic-operation combination requires an explicit entry
There is no concept of time-limited access; access granted during an incident stays granted until someone manually revokes it
No group-based assignment; you manage individual principals
Maintenance overhead grows with team and topic count

External authorizers address these limitations. Open Policy Agent (OPA) evaluates authorization requests against policy-as-code and can also detect security policy violations (overly permissive ACLs, brokers not configured for TLS) before they reach production. Apache Ranger provides centralized policy management with a UI, commonly used in Hadoop and enterprise data environments. At the extreme end of scale, Uber implemented a custom KafkaAuthorizer delegating to their internal IAM framework, using an attribute-based model where a single generic policy replaces thousands of individual ACL entries. Topic ownership changes propagate automatically without any policy update. Most teams will not build this from scratch, but it illustrates what breaks down with native ACLs at scale.

Audit logging: what Kafka logs and what it doesn’t

Native logging capabilities

Kafka does not provide audit logging out of the box, but it does provide the components to build one. The primary mechanism is kafka.authorizer.logger:

Logs denied operations at INFO level
Logs allowed operations at DEBUG level
Each entry includes the principal, client host, attempted operation, and resource

logger.authorizer.name=kafka.authorizer.logger logger.authorizer.level=DEBUG logger.authorizer.appenderRef.authorizer.ref=authorizerAppender logger.authorizer.additivity=false

The practical problem: on a busy cluster processing 10,000 requests per second, DEBUG-level authorizer logs generate gigabytes per hour. Teams enable DEBUG for compliance purposes and find it unmanageable in practice. The result is often logs that capture denials but not allows, meaning you know who was refused access, but not who successfully read your payments topic.

What a useful audit trail covers

A complete audit trail should capture:

Authentication events: successful logins, failed attempts, mechanism used
Authorization decisions: both allows and denies
ACL changes: who created, modified, or deleted ACLs
Topic access: producer writes and consumer reads per principal
Administrative operations: topic creation and deletion, configuration changes

Shipping logs to a SIEM

Audit logs sitting in broker-local rolling files are not useful for compliance or incident response. The standard approach:

Configure kafka.authorizer.logger with a dedicated rolling file appender per broker
Use a log shipper (Filebeat, Fluentd, Logstash) to forward to Elasticsearch, Splunk, or your SIEM of choice
Set retention appropriate to your compliance requirements. SOC 2 Type II and HIPAA commonly require 90 days or more; some regulations require 7 years
Build alert rules: failed authentication spikes, ACL changes outside change windows, unexpected principals accessing sensitive topics

When an auditor asks who accessed a specific dataset in the last 90 days, you need 90 days of aggregated, queryable authorizer logs. Teams that have not configured centralized log shipping frequently discover they have 7 days of rolling files on broker disks. That gap typically costs months of remediation work.

Network-level security: the layer beneath Kafka

Kafka was designed with a somewhat trusted network assumption, originating from LinkedIn’s internal infrastructure. That assumption does not hold in multi-cloud, cloud-native, or multi-tenant deployments. Network security is not a substitute for authentication and authorization, but it is an additional layer. Even with authentication configured, a broker port reachable from the public internet is an unnecessary attack surface.

Listener configuration

Kafka brokers can expose multiple listeners with different security protocols. Separating internal, external, and controller traffic onto distinct listeners provides isolation:

listeners=INTERNAL://0.0.0.0:9093,EXTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:9094 listener.security.protocol.map=INTERNAL:SASL_SSL,EXTERNAL:SASL_SSL,CONTROLLER:SASL_SSL

The internal listener handles broker-to-broker and trusted service traffic. The external listener handles client applications. The controller listener (KRaft) handles controller-to-broker communication in isolation.

Advertised listeners

advertised.listeners tells clients what address to reconnect to after the initial bootstrap connection. A misconfigured advertised listener can expose an internal hostname to external clients, or direct traffic to a PLAINTEXT listener regardless of how the initial connection was configured. The client connects to the bootstrap address, receives metadata, and then reconnects to the advertised listener. If that address resolves to an unencrypted port, subsequent connections are unencrypted. This is a particularly common misconfiguration in containerized environments.

Network isolation

Kafka brokers should not have public IP addresses unless explicitly required. Use VPC peering, private endpoints, or PrivateLink for cross-account or cross-region access. Firewall rules and security groups should restrict Kafka ports (9092, 9093, and 2181 for ZooKeeper) to known IP ranges: application subnets and management bastion hosts. Port 2181 is frequently the most exposed, and ZooKeeper with unauthenticated access on a reachable port is a critical risk.

ZooKeeper and KRaft: securing the control plane

ZooKeeper: the overlooked attack surface

ZooKeeper stores cluster membership, topic configurations, partition assignments, ACL definitions, and in older Kafka versions, consumer group offsets. An attacker with access to an unauthenticated ZooKeeper can read all of this, modify ACLs, delete topics, and inject bogus broker registrations.

The common failure mode is that teams focus on securing the brokers while leaving ZooKeeper accessible on a “trusted” internal network. Any host compromised on that network then has access to the cluster’s entire control plane.

ZooKeeper is deprecated. Kafka 3.9, released November 2024, is the last version to support ZooKeeper mode. ZooKeeper-based clusters lose official security patches in November 2025. Kafka 4.0, released March 2025, removed ZooKeeper entirely. If your cluster is still running ZooKeeper mode, migration to KRaft is a security requirement at this point, not an optimization.

KRaft: what changes

KRaft eliminates ZooKeeper entirely. Cluster metadata moves into an internal Kafka topic (__cluster_metadata), managed by a quorum of controller nodes using the Raft consensus protocol.

From a security standpoint, the main improvements are:

Single security model. Previously you needed separate SASL and ACL configuration for Kafka and ZooKeeper independently. Teams that secured Kafka correctly but not ZooKeeper remained exposed. KRaft removes that second attack surface.

Simplified audit surface. One system’s logs to monitor instead of two.

Faster failover. Controller failover drops from minutes to seconds, reducing the window where the cluster is in an uncertain state.

What does not change: ACLs, authentication, and TLS still require proper configuration. The __cluster_metadata topic is internal but subject to the cluster’s overall security posture. Sensitive data in KRaft records is not encrypted by Kafka itself; disk-level encryption remains necessary for at-rest protection.

For production KRaft deployments: a minimum of three controller nodes for quorum (five for high availability), deployed on dedicated nodes with fast SSD storage across separate availability zones, with the controller listener isolated from client listeners.

Putting it together: a layered security model

The correct mental model for Kafka security is defense in depth. Each layer provides protection when another is compromised.

Network layer VPC isolation, security groups, private listeners | Encryption layer TLS 1.3 for all traffic: client-broker, broker-broker, control plane | Authentication layer SASL (SCRAM / OAUTHBEARER / GSSAPI) or mTLS (no anonymous connections) | Authorization layer ACLs or external authorizer (OPA / Ranger / custom), least privilege | Audit layer Centralized, retained, queryable logs shipped to SIEM | Operational layer Regular ACL review, certificate rotation, patching, misconfiguration scanning

If an attacker compromises a credential and gets through the authentication layer, the authorization layer limits what they can access. If they get through authorization, the audit layer creates a detectable signal. No single layer is sufficient on its own.

Common security misconfigurations

These are the configurations that most frequently go wrong in real deployments.

Unauthenticated listeners left open. ALLOW_PLAINTEXT_LISTENER=yes is a real environment variable in the Bitnami Docker image, included for development convenience. In clusters that started as “a quick test,” this configuration persists into production. Any host that can reach port 9092 becomes a producer, consumer, or admin client.

Wildcard ACLs in production. Granting User:* ALLOW ALL on a topic makes the authorization layer functionally meaningless. You have authenticated clients, but all of them can do everything.

Super users beyond the initial admin. The super.users property bypasses all ACL checks. In many environments this list grows over time as engineers add themselves for debugging access that never gets revoked. It should be a short, actively managed list.

Inter-broker encryption disabled. Client connections use TLS, replication does not. security.inter.broker.protocol defaults to PLAINTEXT and must be explicitly overridden.

ZooKeeper exposed without authentication. Port 2181 open on a security group, no SASL configured, no ACLs on ZooKeeper nodes. With ZooKeeper deprecated, this is increasingly a migration-blocking technical debt issue as well as a security risk.

SASL/PLAIN over unencrypted connections. Using SASL_PLAINTEXT with the PLAIN mechanism transmits credentials in cleartext. Credentials are visible to anyone capturing traffic on the network segment.

Schema Registry and Kafka Connect left unsecured. The security posture applied to brokers should extend to adjacent components. Connect workers running as Kafka clients need authentication and authorization. Schema Registry exposes an HTTP API that needs its own TLS and authentication. A Connect worker with admin-level cluster permissions and no authentication is a privileged path into your cluster.

Audit log retention too short. Seven days of rolling files on broker-local disk fails SOC 2, HIPAA, and PCI-DSS audits that require 90-day or longer access history. Configure centralized shipping with appropriate retention from the start.

Permission creep over time. Deprecated services still have active principals with write access. Team membership changes but ACLs do not. Treat ACL review as a recurring operational task, not a one-time setup.

How Kpow helps with Kafka security management

One area where Kafka’s native tooling is limited is visibility into what is actually happening across your cluster, particularly around access control and user activity.

Kpow is a management and monitoring tool for Kafka that includes capabilities relevant to the authorization and audit sections of the architecture described above.

Authorization management. Kpow supports both simple access control via environment variable configuration and full role-based access control that integrates with your identity provider. RBAC in Kpow maps user roles to specific actions across clusters, topics, consumer groups, schemas, and connectors. Users are denied all actions by default; permissions are explicitly granted. This means every operator using Kpow to administer your cluster is subject to the same least-privilege model you are applying to your Kafka clients.

Audit logging. Kpow captures all user actions in an audit log, stored in an internal topic (__oprtr_audit_log) and queryable from the UI. The audit log records the contents of each request, the identity of the user who made it, and the authorization result including which policies were evaluated. Admins can view the last seven days of activity from within the application.

For integration with external systems, Kpow supports webhook delivery of audit log events to Slack, Microsoft Teams, or any custom endpoint. You can configure verbosity to capture mutations only, queries only, or all events, and the structured JSON payload is suitable for routing into a SIEM or triggering downstream workflows.

This addresses a specific gap in the native Kafka audit story: while kafka.authorizer.logger records broker-level authorization decisions, it does not capture actions taken by humans using a management UI. Kpow’s audit log covers that layer.

You can try Kpow with a free 30-day trial, and connect it to any Kafka cluster in minutes by deploying via Docker, Helm, or JAR.

Summary

Building a production-ready Kafka security architecture requires deliberate configuration across multiple layers. The starting point is acknowledging that Kafka ships with all security controls disabled. From there, the work is methodical: TLS across all three traffic flows, authentication appropriate to your environment and identity infrastructure, authorization scoped to the principle of least privilege, and audit logging that is centralized, retained, and queryable, not just enabled.

The operational work is as important as the initial configuration. Certificate rotation, ACL review, and misconfiguration scanning need to be recurring processes. Security configurations that are correct at deployment drift over time as teams, services, and infrastructure change. The clusters that end up with the most significant exposures are not the ones that were never secured but the ones that were secured once and then not maintained.

Kafka UI: The Ultimate Guide

Chad Harris — Sat, 27 Jun 2026 00:00:00 GMT

A Kafka UI is a web-based interface that sits on top of an Apache Kafka cluster and turns the broker, topic, partition, and consumer-group APIs into something you can read and operate from a browser. The CLI tools shipped with Kafka are sufficient when you run a handful of topics on a single cluster. They become a friction point as soon as several teams share the platform, when operators need a consistent view across environments, or when an incident requires you to inspect partition state in seconds rather than minutes.

To get a sense of the operational surface area a Kafka UI has to cover at the upper end, JPMorgan runs 102 clusters with around 510 nodes and 13,000 topics, ingesting roughly 400 billion events per day. At that scale, the difference between a UI that surfaces under-replicated partitions clearly and one that forces an operator back to kafka-topics.sh is the difference between a five-minute fix and a thirty-minute incident.

This guide is written for platform and data engineers evaluating which Kafka UI to standardise on. It covers what a Kafka UI does, how the market is segmented, the criteria worth weighing during evaluation, and how the major tools compare in practice.

What does a Kafka UI actually do?

Most Kafka UIs converge on the same set of functional categories. The depth of coverage varies, but the categories are stable.

Topic management. Create, delete, and reconfigure topics, inspect partition layout and replica placement, and read messages from a partition with offset, key, header, and value filtering. Topic management is the day-to-day surface for both operators and developers, covering everything from partition count decisions to per-topic retention and compaction settings.

Consumer group monitoring and offset management. View consumer group membership, per-partition lag, current offset against log-end offset, and reset or seek consumer offsets when a consumer needs to replay or skip a window of data.

Broker and cluster health. Broker liveness, controller identity, partition distribution, under-replicated and offline partitions, and basic throughput metrics at the broker level.

Schema registry integration. Browse subjects, view schema versions and compatibility settings, and decode Avro, Protobuf, or JSON Schema payloads in the message viewer so message content is human-readable.

Access control. Surface and manage ACLs or RBAC role assignments depending on the underlying authorisation model, and capture an audit trail of who changed what.

Connector management. List Kafka Connect connectors, view status and task health, pause and restart connectors, and inspect configuration.

Producer tooling. Send test messages with arbitrary keys, headers, and serialisers from inside the UI, exercising the same producer path that application code uses.

Each of these categories maps to a different operational job, and a tool’s coverage of any one of them is rarely binary. Most UIs do the easy parts well and differ on the harder parts: schema-aware filtering, RBAC granularity, Connect task observability, and offset-reset workflows.

The Kafka UI landscape

The market splits into three rough categories.

Open-source and self-hosted. Kafbat UI, Redpanda Console, and AKHQ sit here. They are free to download, ship as Docker images or self-contained binaries, and cover the core functional categories well. Governance features such as fine-grained RBAC, audit trails, and data masking are typically thinner or behind a paid tier. A comparison of the most widely-used options is available in best free kafka ui tools.

Commercial and self-hosted or managed. Kpow, Conduktor, Lenses.io, and Kadeck sit here. They charge a per-cluster, per-user, or per-seat fee in exchange for deeper RBAC, audit logging, governance workflows, and vendor support. Deployment is usually still under your control. For a broader view across both open-source and commercial options, best kafka management tools covers the full landscape.

Vendor-bundled. Confluent Control Center is the canonical example. It ships as part of Confluent Platform and integrates tightly with Confluent’s Schema Registry, ksqlDB, Cluster Linking, and MDS-based RBAC. The trade-off is that it is effectively locked to Confluent deployments.

One landscape shift worth flagging: Kafka 4.0 fully removed ZooKeeper in March 2025, and KRaft is now the only supported metadata mode. Any UI that still relies on ZooKeeper-based metadata APIs has a gap to close. The well-maintained tools have already made the transition. If you are evaluating something less actively developed, confirm KRaft support before committing.

How to evaluate a Kafka UI: key criteria

The criteria below are the ones I weight most heavily during a real evaluation. Different teams will rank them differently, but skipping any of them tends to produce regret a year in.

Monitoring and observability depth

The baseline question is whether the UI surfaces the metrics that actually move during an incident: consumer lag at partition granularity, under-replicated partitions, offline partitions, broker request latency, and partition skew across brokers. The easy metrics (broker count, topic count, total messages) are uniform across tools. The harder ones are not.

Datadog provides a useful illustration of why visibility matters. The team reduced a host-subscriptions topic from 500,000 messages per second to 5,000 messages per second, a 100x reduction, after gaining clearer visibility into their pipeline. The optimisation freed over 600 CPU cores and 1 TB of memory. The lesson is that observability is what makes optimisation possible: you cannot improve what you cannot see.

A Kafka UI handles the real-time slice of this: point-in-time consumer monitoring, cluster health, and producer visibility. For time-series storage, alerting, and capacity planning, the UI should complement a proper monitoring stack rather than replace it. The kafka monitoring guide covers that boundary in full, including guidance on kafka dashboards, best kafka monitoring tools, and integrations with tools like Grafana.

Access control and multi-tenancy

When a shared cluster supports more than two teams, ACLs alone become difficult to manage. Granular RBAC, audit logging, and an identity-provider integration (SAML, OIDC, LDAP) are the features that matter. Look for role-based authorisation at the topic, consumer-group, and connector level, with the ability to scope roles to a single environment. For multi-tenant Kafka deployments, the UI is where access policies become visible and auditable across teams.

Uber illustrates the scaling problem well. Their custom KafkaAuthorizer uses a single attribute-based policy to replace what would otherwise be thousands of individual ACL entries. A UI that surfaces and manages permissions weakly would make that model unworkable in operation, even if the authentication, authorisation, and encryption at the broker level are sound.

For a deeper treatment of the security model underneath the UI, see the kafka security architecture article.

Schema registry support

Schema-aware message viewing is the difference between a useful inspection workflow and a Base64-decoded mess. The UI should resolve subjects automatically, decode Avro, Protobuf, and JSON Schema payloads, surface compatibility rules, and let you view historical schema versions. Schema evolution visibility is particularly useful during incidents involving consumer deserialisation failures.

Kafka Connect management

For teams using Kafka Connect for CDC, sink-to-warehouse, or system integration, the UI should surface connector lifecycle (paused, running, failed), per-task status, and configuration. Restart and pause controls from the UI shorten the loop during connector incidents. Use cases range from ServiceNow integrations to Jira event pipelines, and in each case connector observability is what separates a quick diagnosis from a long one.

Notion replaced custom connectors with Confluent’s pre-built Connect integrations after concluding that the old approach was, in their words, “too expensive and difficult to maintain at scale.” The change saved over $1 million in 2022 and tripled engineering productivity. A UI that surfaces connector health directly reduces the operational cost of running Connect at scale.

Stream processing and ksqlDB visibility

Kafka Streams applications and ksqlDB queries are first-class citizens in many production estates, but few UIs treat them as such. The ones that do let you inspect topology, internal state stores, and per-query status. If you run Streams applications in production, the ability to navigate from a consumer group back to the topology that owns it, and forward to the state-store partitions that back it, removes a category of debugging that would otherwise need bespoke tooling. For ksqlDB users, the equivalent is a UI that lists queries, their source and sink topics, and their current status.

Deployment model and operational overhead

Self-hosted versus SaaS, single container versus Helm chart, and how the UI authenticates to the cluster (direct broker access, a JMX scrape, or an agent proxy). Security teams will care about the data-plane access model: a UI that proxies all client traffic to brokers has a very different blast radius from one that only reads metadata.

Installation footprint matters during evaluation. Most open-source tools ship a single container that can be running against a test cluster within minutes. Commercial tools tend to involve a Helm chart, an admin database, and an initial RBAC bootstrap, which is closer to an hour of work. SaaS options remove the deployment step entirely but introduce data-residency and connectivity questions.

Multi-cluster and multi-environment management

Almost every team running Kafka in production runs at least three clusters: development, staging, and production. Larger estates add regional clusters, isolation clusters for regulated workloads, and dedicated clusters for high-throughput pipelines. A UI that handles only one cluster at a time forces operators to tab between browser windows, breaks audit trails at cluster boundaries, and makes cross-environment comparison (does the staging topic config actually match production?) tedious.

When evaluating multi-cluster support, look for: a cluster switcher that preserves context, unified search across clusters, environment-aware RBAC so that production write permissions do not leak to development users, and a way to compare topic configuration across clusters without leaving the UI. Some tools also offer global catalogs that list every topic across every cluster in one view, which is useful in estates where the same logical topic name is used across environments.

Producer and consumer tooling

The ability to produce a test message and consume from an arbitrary offset is a small feature with large operational impact. During incidents, the question “is the producer actually sending, and is the broker actually accepting?” comes up often. A UI that lets you produce a known-good message from a known-good identity narrows the search space quickly. Related tools for managing offsets and records from the CLI include the kafka offset tool, the ability to delete records from a topic, and the kafka admin tool.

Pricing model and total cost of ownership

Free is not the same as low cost. Open-source tools have no licence fee, but the engineering time spent deploying them, upgrading them when the upstream project releases a new version, patching them when a CVE lands, and carrying them on the on-call rotation is real. For a small team running a single cluster, that overhead is trivial. For a platform team running ten clusters across three regions, it adds up.

Commercial tools split into two pricing shapes. Per-cluster pricing (often used for self-hosted commercial tools) scales with infrastructure and is predictable as headcount changes. Per-seat or per-user pricing scales with adoption: the more useful the tool is, the more it costs. For estates where the goal is broad self-service, per-cluster pricing tends to age better. For estates where access is restricted to a small platform team, per-seat pricing can be cheaper.

A complete TCO model should include: licence fees, the engineering cost of deployment and maintenance (often half a person, sometimes more), support tier costs, and the opportunity cost of operator time spent in CLI workflows that the UI would have automated. The last item is hardest to quantify but often the largest.

Performance and footprint at scale

How does the UI itself behave when it is pointed at a cluster with thousands of topics and hundreds of consumer groups? Page load times, search responsiveness, message-viewer latency, and the memory footprint of the UI server become operational concerns at the kind of scale referenced in the JPMorgan and Walmart case studies. Some UIs paginate well and use server-side filtering; others pull metadata in bulk and degrade noticeably past a few thousand topics. If your estate is at that scale, ask vendors for a benchmark against a comparable topic count, and if you are evaluating an open-source tool, test it against a representative cluster before committing.

Disaster recovery and cluster linking visibility

For multi-region deployments, the UI’s view into replication health matters. MirrorMaker 2 topology, replication lag between source and target clusters, and the configuration of replicated topics are all things you want to inspect from one place rather than from a mix of CLI and broker logs. Confluent Cluster Linking introduces a different model that some UIs surface natively and others ignore. Barclays runs Confluent Kafka on Amazon EKS with multi-region active-active and active-passive configurations, and Walmart operates Kafka across public and private clouds. At that footprint, replication visibility is a security and compliance concern as much as an operational one.

Kafka UI and security

Security is non-negotiable when a UI has read and write access to a production cluster. The kafka security architecture has several layers, and the UI touches most of them. The relevant dimensions are:

Authentication. SSO via SAML or OIDC, LDAP integration for on-prem identity providers, and local users as a fallback. Avoid tools that only support local users in production deployments.

Authorisation granularity. Cluster-level, topic-level, consumer-group-level, and connector-level role assignments. The right granularity depends on how many teams share the cluster: a single-team deployment can live with cluster-level roles, a multi-team deployment cannot.

Audit logging. Every write action (offset reset, topic creation or deletion, connector restart, ACL change) should produce a tamper-evident audit record. For regulated industries, the audit log needs to be exportable to a SIEM.

Network exposure. A UI server with broker credentials is a sensitive surface. Bind it to an internal network, put it behind your existing identity proxy, and avoid exposing it to the public internet even with authentication enabled.

Data-plane access model. Some UIs proxy all client traffic to brokers, which means the UI host needs to be sized for the throughput it serves. Others only read metadata and produce or consume on demand, which keeps the UI host small but moves the data path back to the client. Both are valid; the right choice depends on your network model.

Barclays is a useful reference here: Confluent Kafka on Amazon EKS, multi-region active-active and active-passive, Kafka-centric systems owned by a central platform team under SRE principles. At that scale and in a regulated industry, an unsecured kafka security surface is a material risk, and the platform team’s choice of UI is part of the compliance story.

Kafka UI and monitoring: where the overlap ends

Most Kafka UIs ship some level of built-in metrics. They are not substitutes for a proper monitoring stack. The boundary between the two is worth being explicit about.

A Kafka UI is for real-time operational interaction: point-in-time inspection, debugging, ad-hoc reads of message content, manual offset management, and connector control. Its job is to answer “what is happening right now, and what action do I want to take?”

A monitoring stack (Prometheus and Grafana, Datadog, New Relic, Dynatrace) is for time-series storage, alerting on thresholds, SLO tracking, anomaly detection, and historical analysis. Its job is to answer “what has happened over the last hour, day, or quarter, and should someone be paged?”

A useful checklist when evaluating where to draw the line:

UI responsibilities: consumer-group lag at the moment of inspection, partition state, broker liveness, message content inspection, schema lookup, ad-hoc producer test, connector status and restart.
Monitoring stack responsibilities: lag trended over time, alerting when lag breaches a threshold, broker JVM metrics, request-latency percentiles, retention and tiered-storage usage trends, capacity planning.

In practice, both layers consume similar broker JMX metrics. The standard convention is to scrape with kafka-exporter and the JMX exporter, store in Prometheus, and visualise in Grafana. A UI that exposes its own Prometheus scrape endpoint can plug into that pipeline without duplication. Read best practices for kafka data observability for guidance on metric collection patterns and the overlap between tooling layers in depth.

Netflix handles between 700 billion and 2 trillion events per day across its Keystone pipeline and Data Mesh platform. At that volume, a UI alone cannot provide the alerting and anomaly detection that production operations require, and the monitoring stack is what catches the slow-moving regressions a real-time UI is never going to flag. The two layers are complementary, not interchangeable.

Tool-by-tool breakdown

Before the per-tool detail, the comparison table below summarises the dimensions most teams care about during a first-pass evaluation.

Tool	Deployment	Licence	RBAC	Multi-cluster	Schema Registry	Connect	Pricing model
Kpow	Self-hosted container	Commercial (Community Edition available)	Full (SAML, OIDC, LDAP)	Yes (up to 12 per instance)	Yes	Yes	Per cluster
Kafbat UI	Self-hosted container	Open-source (Apache 2.0)	Basic (YAML)	Yes	Yes	Yes	Free
Redpanda Console	Self-hosted Go binary	Open-core (BSL / RCL)	Behind enterprise licence	Limited without enterprise	Yes	No	Free + enterprise
AKHQ	Self-hosted Micronaut app	Open-source (Apache 2.0)	Basic	Yes	Yes	Yes	Free
Conduktor	Self-hosted web platform	Commercial (open-core Community tier)	Full	Yes	Yes	Yes	Per seat
Lenses.io	Kubernetes-required	Commercial	Full	Yes	Yes	Yes	Per cluster + per user
Kadeck	Desktop and server	Commercial	Limited (Desktop)	Yes (Server tier)	Yes	Yes	Per user
Confluent Control Center	Bundled with Confluent Platform	Commercial (Confluent Enterprise)	Full (MDS-based)	Yes	Yes	Yes	Bundled

Kpow (Factor House)

Commercial, self-hosted. Stateless JVM container that runs against any Apache Kafka cluster, with deep RBAC backed by SAML, OIDC, LDAP, or Keycloak SSO, full support for Schema Registry, Kafka Connect, ksqlDB, ACL management, and Kafka Streams topology visualisation. Up to 12 clusters per instance. Designed for SRE and platform teams running multi-cluster Kafka in regulated environments. Pricing is per cluster, which keeps cost predictable as headcount changes.

Kafbat UI

Open-source under Apache 2.0, self-hosted as a container. Wide adoption, an active maintainer community, and good general-purpose coverage including multi-cluster support, Avro, Protobuf, and JSON deserialisation, Schema Registry integration, and CEL-based filtering. RBAC is YAML-based and minimal. Best fit for small to mid-size teams with modest governance needs. Worth noting that Kafbat UI is the active fork of what used to be the “kafka-ui” project on GitHub, which is the source of the recurring naming confusion in the community.

Redpanda Console

Open-core, Go binary. Fast message viewer and a clean operator experience. The community edition is BSL-licensed; multi-cluster support, SSO, RBAC, and data masking sit behind the Redpanda enterprise licence. Connect management is not included. Best fit for existing Redpanda customers with enterprise contracts. If you run Apache Kafka rather than Redpanda, the value is narrower.

AKHQ

Open-source under Apache 2.0, deployed as a Micronaut application. GitOps-friendly configuration, OIDC, OAuth2, LDAP, and GitHub SSO, full Connect and Schema Registry coverage, and ksqlDB support. Basic RBAC since version 0.25. No native data masking, and UI performance is known to degrade on very large clusters. Best fit for teams wanting a free, GitOps-native tool without server-side data masking requirements.

Conduktor

Commercial, with an open-core Community tier. Self-hosted web platform with an optional Gateway proxy for ownership tracking, topic catalogs, self-service workflows. Be aware the proxy can introduce risk due to being in the data path. Best fit for large organisations where multiple teams share infrastructure and need explicit ownership and self-service. Per-seat pricing scales with adoption: published estimates for 100 users across 3 clusters land in the $80,000-$150,000 per year range.

Lenses.io

Commercial, Kubernetes-required (SQL Processors run as Kubernetes pods). The differentiator is SQL-driven stream processing alongside observability, including topology visualisation, a Kafka-to-Kafka replicator, a global multi-cluster catalog, and data policies. Best fit for teams wanting SQL-driven stream processing inside the same tool that handles operational interaction. The Kubernetes dependency adds operational overhead, and post-acquisition product direction is worth confirming with the vendor during evaluation.

Kadeck

Commercial. Desktop-first product with a server tier for team deployments. Good for individual developer use and quick local inspection of clusters; less suited to team-based operations where shared audit trails and centralised RBAC are required.

Confluent Control Center

Bundled with Confluent Platform Enterprise. Native integration with the Confluent ecosystem: Schema Registry, ksqlDB, Cluster Linking, and MDS-based RBAC. Effectively locked to Confluent deployments, and the advanced features depend on MDS being present. Typical Confluent Platform pricing ranges from roughly $50,000 to $500,000+ per year, depending on cluster footprint and support tier.

Kafka UI in practice: three operational scenarios

The criteria above are easier to evaluate against concrete jobs. The three scenarios below cover the most common ones I see during platform evaluations.

Debugging consumer lag during an incident

A consumer group has fallen behind. The on-call engineer opens the UI, navigates to the consumer group, and looks at the per-partition lag. One partition is responsible for the bulk of the lag; the others are caught up. The UI shows the partition leader, the consumer instance currently assigned to it, and the timestamp of the last committed offset.

From there, the engineer can inspect the messages near the current offset, look at the message size distribution, and check whether the consumer is failing to process a specific message type. Reviewing the consumer configuration and consumer properties can surface whether the issue is a processing timeout, a deserialisation error, or a throughput mismatch. If the partition has fallen too far behind to catch up within the SLA window, an offset reset to a later position becomes a deliberate decision rather than a panicked one. The UI captures the reset in its audit log, and kafka logs at the broker level can confirm what the consumer actually received.

Managing topics and partitions at scale

DoorDash operates five Kafka clusters managed by a central Real-Time Streaming Platform team. At that footprint, topic governance covers replication factor, partition count, retention policy, compaction settings, and per-topic ACLs. Good partition key selection and message key design matter here too, since the UI makes partition skew visible but the root cause is usually upstream. Message size is another configuration dimension the UI surfaces clearly. Doing this through CLI scripts is workable; doing it through a UI that lets the platform team review every new-topic request, apply a template, and audit the result is faster and less error-prone.

The UI matters most when teams outside the platform team need to inspect or request changes. A self-service workflow that proposes the change, routes it to a platform reviewer, and applies it on approval is the difference between a healthy multi-tenant cluster and one where every team has direct admin access “just to get things done.” For the broader operational discipline around kafka cluster management, cluster best practices, and kafka scaling best practices, those articles cover the decisions that sit behind the UI.

Onboarding teams to a shared cluster

Walmart’s Kafka infrastructure serves over 25,000 consumers across public and private clouds as of June 2024, processing trillions of messages per day. A self-serve UI model is what makes that footprint tractable. New teams use the UI to inspect topics they have read access to, check consumer group health, produce test messages against a staging cluster, and request access to additional topics through a workflow the platform team owns.

The alternative (a central team handling every read request, every consumer-group inspection, and every offset query) does not scale past a few dozen tenant teams. The UI is the lever that moves the work from the platform team to the tenants without giving them broker-admin access.

What’s next for Kafka UI tooling

Three forward-looking themes are worth flagging for anyone making a multi-year decision.

KIP-932 and queues. KIP-932 introduces queue semantics to Kafka through a new “share group” consumer model. Once shipped broadly, this changes the consumer model in ways that will require UI tooling to expose a new kind of entity alongside topics and traditional consumer groups. UIs that don’t adapt will show an incomplete picture of cluster state.

KRaft as the steady state. With ZooKeeper fully removed in Kafka 4.0, the metadata model is simpler and more consistent. UIs that previously read ZooKeeper for cluster state now interact exclusively with the Kafka metadata quorum. This opens the door to richer, faster metadata exposure in tooling, and it eliminates a class of stale-metadata bugs that used to surface in ZooKeeper-era UIs. The kafka architecture article covers the KRaft metadata model in detail.

AI assistants and natural-language operators. Several commercial roadmaps now include some form of natural-language interaction: ask the UI to find consumer groups that have been lagging for more than an hour, or to summarise the recent topic configuration changes across the production estate. Whether this becomes a meaningful operational interface or stays a demo feature is still open, but it is worth tracking. The credible version of this is one where the UI translates a question into an API call against the same metadata it already surfaces, rather than a chatbot that hallucinates broker IDs.

Tiered storage visibility. As more vendors ship tiered storage (Confluent, Aiven, Redpanda, and the upstream KIP-405 work), the UI needs to surface what data lives on the broker, what has been tiered to object storage, and what the cost implications are. This is a relatively new area, and tools differ significantly in how clearly they present it.

Choosing the right Kafka UI

A concise decision framework, summarising the categories above:

Small team or personal use: Kpow Community Edition or CMAK. Free, easy to deploy, sufficient for one or two clusters with light governance needs.
GitOps-native preference: AKHQ. Configuration in version control, OIDC out of the box.
Compliance, audit, or regulated industry: Kpow or Conduktor. Full RBAC, audit logging, and identity-provider integration.
Data engineering or SQL-style exploration: Lenses.io. SQL Processors and a global catalog suit data engineering workflows.
Already on Confluent Platform: Confluent Control Center. Suits if you are completely committed to the Confluent stack.
Enterprise, multi-cluster, multi-tenant: Kpow or Conduktor, with the pricing model as the deciding factor. Per-cluster pricing favours broad adoption; per-seat pricing becomes expensive for larger teams.

“Free” is not the same as “low cost.” Self-hosted open-source tools carry engineering overhead for deployment, version upgrades, security patching, and on-call. For a single-cluster team, that overhead is trivial. For a platform team running a regulated estate across multiple regions, the all-in cost of an open-source tool often exceeds the licence cost of a commercial one.

Conclusion

The right Kafka UI reduces operational overhead, makes security enforceable across teams, and gives operators, developers, and data engineers a meaningful window into cluster behaviour. The wrong choice, or no choice at all, is a form of operational debt that compounds as the cluster grows: every offset reset done by hand, every ACL change applied via CLI, and every consumer-group lag investigation that takes an hour instead of five minutes adds up. The UI is the operational layer that sits above architecture, monitoring, and security, and the right one is the one that fits your team’s scale, governance needs, and pricing tolerance.

At Factor House, we built Kpow for platform and SRE teams running Kafka in regulated, multi-cluster environments. If your evaluation maps to that profile, you can try Kpow for free for 30 days against any Kafka cluster, deployed via Docker, Helm, or JAR.

FAQ

What is a Kafka UI?

A Kafka UI is a web-based interface for managing and inspecting an Apache Kafka cluster. It exposes topics, partitions, consumer groups, brokers, schemas, and connectors through a browser, removing the need to use the Kafka CLI for routine operational tasks.

What is the difference between a Kafka UI and a Kafka monitoring tool?

A Kafka UI is for real-time operational interaction: inspecting cluster state, reading messages, and performing actions like offset resets or connector restarts. A Kafka monitoring tool such as Prometheus and Grafana or Datadog is for time-series storage, alerting, SLO tracking, and historical trend analysis. Most production estates use both.

Which Kafka UI tools are free and open source?

The widely-used open-source options are Kafbat UI (Apache 2.0), AKHQ (Apache 2.0), and Redpanda Console (BSL community edition, with enterprise features behind a paid licence). Each is self-hosted and free to download. While not open source, Kpow Community Edition is also a viable option.

Is Confluent Control Center free to use?

No. Confluent Control Center is bundled with Confluent Platform Enterprise and is not available as a standalone free product. Pricing is included in the broader Confluent Platform licence, which typically ranges from around $50,000 to $500,000+ per year depending on cluster footprint and support tier.

How do I manage consumer group offsets from a Kafka UI?

Most Kafka UIs let you view the current offset and log-end offset for each partition in a consumer group, and reset the offset to the earliest, latest, a specific timestamp, or a specific offset value. The action is typically gated by an RBAC permission and recorded in an audit log. The exact workflow varies by tool but the underlying API is the standard Kafka admin client.

Can a Kafka UI manage Kafka Connect connectors?

Yes, most Kafka UIs integrate with Kafka Connect to list connectors, surface task status, and provide pause, resume, and restart controls. Configuration can usually be inspected and edited from the UI. Redpanda Console is a notable exception: it does not include Connect management.

How do I secure a Kafka UI in a production environment?

Authenticate users through your identity provider (SAML, OIDC, or LDAP) rather than local users. Enforce RBAC at the topic, consumer-group, and connector level. Bind the UI to an internal network behind your existing identity proxy. Enable audit logging and ship the audit log to a SIEM. Ensure the UI uses a service account with the minimum broker permissions it actually needs.

Which Kafka UI tools support RBAC and SSO?

Full RBAC with SSO is offered by Kpow, Conduktor, Lenses.io, and Confluent Control Center. AKHQ supports SSO and basic RBAC since version 0.25. Kafbat UI has basic YAML-based RBAC. Redpanda Console requires the enterprise licence for SSO and RBAC.

What is the best Kafka UI for a small team versus an enterprise deployment?

For a small team running one or two clusters, an open-source tool such as Kpow Community Edition, Kafbat UI or AKHQ usually covers the operational needs at zero licence cost. For an enterprise deployment with multiple clusters, regulated workloads, and many tenant teams, a commercial tool with full RBAC, audit logging, and vendor support (Kpow, Conduktor, Lenses.io, or Confluent Control Center) tends to be a better fit.

Can a Kafka UI manage multiple clusters from one interface?

Yes. Kpow, Conduktor, Lenses.io, AKHQ, Kafbat UI, and Confluent Control Center all support multi-cluster management from a single instance. Cluster switching, unified search, and environment-aware RBAC vary in maturity, and Redpanda Console’s multi-cluster support requires the enterprise licence.

Does Kafka UI work with KRaft mode (Kafka 4.x)?

The actively maintained Kafka UIs (Kpow, Kafbat UI, AKHQ, Conduktor, Lenses.io, Redpanda Console, Confluent Control Center) all support KRaft mode. With Kafka 4.0 having fully removed ZooKeeper as of March 2025, KRaft is the only supported metadata mode going forward, and any UI that still depends on ZooKeeper APIs has a gap to close.

What is the difference between Kafka UI and Kafbat UI?

“Kafka UI” was the original project name for an open-source Kafka management interface formerly maintained on GitHub by Provectus. The project was forked and is now actively maintained as Kafbat UI by the same core community of maintainers. If you see references to “kafka-ui” in older documentation, they almost always refer to what is now Kafbat UI.

How does a Kafka UI handle schema registry and Avro or Protobuf messages?

A Kafka UI with Schema Registry integration resolves the schema for a message by reading the schema ID from the message header, fetching the schema definition from the registry, and using it to deserialise the payload into a human-readable form. Avro, Protobuf, and JSON Schema are supported by the major tools. The UI typically also exposes subject management, version history, and compatibility-rule inspection.

Dead letter queues in Kafka: patterns and pitfalls

Factor House — Fri, 26 Jun 2026 00:00:00 GMT

A single malformed record can block an entire Kafka partition. When your consumer encounters a message it cannot deserialize - a corrupted Avro payload, a JSON field with an unexpected type, a schema version that was deprecated months ago - it throws an exception, retries, and throws again. Consumer group lag climbs. Every healthy message behind that one record waits. This is a poison pill, and the standard solution is a dead letter queue.

Kafka has no native dead letter queue primitive outside of Kafka Connect. The broker has no concept of a failed message; failure is entirely a consumer-side concern. As a result, the DLQ is a pattern you implement in your consumer application, your Connect connector configuration, or your Kafka Streams topology. This article covers all three paths with working examples, explains the retry-topic pattern that should precede any DLQ, and walks through the production failure modes that are frequently omitted from shorter treatments of the topic.

By the end, you should be able to implement a DLQ in your own stack and know which failure modes to test for before shipping to production.

Key takeaways

Kafka has no native dead letter queue outside of Kafka Connect. In all other cases, the DLQ pattern is implemented at the consumer application, Kafka Connect connector, or Kafka Streams topology layer.
The production-grade shape is retry-topic plus DLQ: transient failures should cycle through a bounded retry pass before a message is parked, keeping the DLQ as a terminal state rather than another retry tier.
Spring Kafka, Kafka Connect, and Kafka Streams each handle the DLQ differently. Connect is the only path that requires no custom code; Kafka Streams requires the most.
The failure modes that most often break a DLQ in production are ordering loss on replay, schema mismatches in the DLQ topic itself, infinite retry loops between the source and retry topics, and side effects being re-applied during blind replay.
Monitoring the age of the oldest unprocessed DLQ record is more actionable than monitoring topic depth alone, because it is independent of traffic volume.
Kpow, the Kafka UI, includes bulk actions for managing dead letter queues, letting you clone and replay records from the DLQ to a retry or source topic with RBAC controls and a full audit trail.

What is a dead letter queue, and why doesn’t Kafka have one?

In message brokers like RabbitMQ and AWS SQS, a dead letter queue is a broker-level feature. When a message exceeds its delivery attempts or cannot be routed, the broker moves it to a designated DLQ. Application code does not need to handle the routing.

Kafka’s architecture works differently. The broker stores ordered byte sequences in partitioned logs and advances consumer offsets when the consumer commits them. Whether a message is valid or processable is a question the consumer answers, not the broker. If a consumer determines a message cannot be processed, it is the consumer’s responsibility to publish that record elsewhere and commit the source offset so the partition can advance.

This means the DLQ pattern in Kafka is implemented at one of three layers:

In the consumer application, using a framework like Spring Kafka with DefaultErrorHandler and DeadLetterPublishingRecoverer
In a Kafka Connect connector, using the native errors.deadletterqueue.* configuration
In a Kafka Streams topology, using custom DeserializationExceptionHandler and ProductionExceptionHandler implementations

Each path covers a different failure surface and has different operational trade-offs.

When you actually need a DLQ

Not every processing failure belongs in a DLQ. The pattern is appropriate in three situations.

Deserialization failures. A message arrives with an incompatible Avro schema, a field with the wrong type, or corrupted bytes. The consumer cannot read the record at all. No amount of retrying will fix a malformed message, so it needs to be parked for inspection.

Business-rule rejections. The message is well-formed and deserializes correctly, but the application logic cannot handle it: a referenced entity no longer exists, a required downstream system returned a permanent error, or a validation rule fails unconditionally. These are not transient failures. The application has determined the message cannot be processed as received.

Retry budget exhaustion. A message fails due to a transient cause - a downstream API timeout, a temporary database outage - but the consumer has retried it enough times that continuing to retry would be more disruptive than parking it. Once the retry budget is exhausted, the message should move to the DLQ rather than blocking the consumer.

Transient failures that are likely to resolve on their own should go through a retry topic first, not directly to the DLQ. The DLQ should be the terminal state: messages there require human or automated intervention to assess and replay.

The retry-topic plus DLQ pattern

The production-grade shape for Kafka consumer error handling combines a retry-topic tier with a terminal DLQ, rather than routing directly to the DLQ on first failure.

The flow works like this:

A message fails processing on the main consumer.
If the failure is transient, the consumer publishes the message to a retry topic (preserving the original key) and commits the source offset. The original consumer continues to the next record.
A retry-topic consumer picks the message up after a delay. If it fails again, it either cascades to a second retry topic with a longer delay, or moves to the DLQ once the retry budget is exhausted.
If the failure is immediately classified as non-transient (deserialization failure, schema mismatch), the consumer skips the retry tier entirely and publishes directly to the DLQ.

Uber’s Insurance Engineering team documented a tiered approach of this kind in their engineering post on reliable reprocessing by Ning Xia and Phani Marupaka. Their implementation uses one Kafka topic per retry level (payments.retry-1, payments.retry-2, and so on), each with a consumer that applies a processing delay before re-attempting. Messages cascade through these levels before reaching the final DLQ topic. Their framing is worth adopting: consumer success means reaching a conclusive outcome for every message, either processed or parked, never silently dropped.

Two distinctions are worth being precise about.

A retry topic is for transient failures: the message may succeed on a second or third attempt once the transient condition resolves. A DLQ is the terminal park: the message has exhausted its remediation options and requires intervention before it can be processed. Treating the DLQ as another retry tier by routing messages back to the source automatically after parking them creates infinite loops, which is covered in the production failure modes section below.

Blocking versus non-blocking retry is a related choice. Blocking retry stalls the consumer on the failed message until it succeeds or a timeout elapses. This preserves strict per-partition ordering but prevents progress on the partition while the retry is running. Non-blocking retry publishes the message to a retry topic immediately and continues processing, trading per-key ordering for throughput. For most production workloads, non-blocking retry is the better default, with the exception of cases where strict per-key ordering is a hard requirement of the downstream system.

Implementing a DLQ with Spring Kafka

Spring Kafka’s DefaultErrorHandler combined with DeadLetterPublishingRecoverer is the most common implementation path for JVM-based consumers. Here is a working configuration:

@Configuration
public class KafkaErrorHandlingConfig {

    @Bean
    public DefaultErrorHandler errorHandler(KafkaTemplate<?, ?> kafkaTemplate) {
        // Routes failed records to <source-topic>.DLT, preserving the source partition.
        DeadLetterPublishingRecoverer recoverer =
            new DeadLetterPublishingRecoverer(kafkaTemplate);

        // Retry up to 3 times at 1-second intervals before invoking the recoverer.
        FixedBackOff backOff = new FixedBackOff(1000L, 3L);

        DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, backOff);

        // These exception types bypass retries entirely and go straight to the DLT.
        handler.addNotRetryableExceptions(
            DeserializationException.class,
            ValidationException.class
        );

        return handler;
    }
}

By default, DeadLetterPublishingRecoverer routes to a topic named <source-topic>.DLT and preserves the source partition number. Your DLT topic must have at least as many partitions as the source topic, and the original message key is retained. Both conditions are necessary for replay ordering to work correctly.

One implementation detail worth understanding: the recoverer publishes the failed record to the DLT before the consumer commits the offset on the source topic. This is the at-least-once guarantee in practice. If the consumer crashes after publishing to the DLT but before committing, the source message will be reprocessed on restart, potentially resulting in a duplicate DLT entry. If you are writing to a DLT from a transactional consumer, configure enable.idempotence=true on the DLT producer to guard against this.

Spring Kafka adds a set of diagnostic headers to every DLT record:

HeaderContentskafka_dlt-original-topicSource topic namekafka_dlt-original-partitionPartition the record came fromkafka_dlt-original-offsetOffset within that partitionkafka_dlt-exception-messageException message as a stringkafka_dlt-exception-fqcnFully qualified exception class name

These headers are what allow you to locate the original record, understand what failed, and replay it correctly. Without them, a DLT is a topic full of byte arrays with no diagnostic context.

DLQ in Kafka Connect

Kafka Connect has first-class DLQ support through three connector properties:

errors.tolerance=all
errors.deadletterqueue.topic.name=my-connector.dlq
errors.deadletterqueue.context.headers.enable=true
errors.deadletterqueue.topic.replication.factor=3

Setting errors.tolerance=all without specifying a deadletterqueue.topic.name causes Connect to silently skip failed records. Always pair errors.tolerance=all with an explicit DLQ topic name to ensure failed records are routed rather than dropped.

With errors.deadletterqueue.context.headers.enable=true, Connect adds headers to each DLQ record describing the failure: the exception class, the stack trace, the source connector name, and the source topic, partition, and offset. Robin Moffatt’s deep-dive on Kafka Connect error handling covers a useful recovery pattern: chain a second sink connector that reads from the DLQ topic and attempts an alternative deserialization strategy, for example falling back to raw JSON if the primary Avro conversion fails.

There is a significant limitation that is frequently misunderstood: Connect’s DLQ catches converter and transform errors only, not task-level errors. When a converter throws during deserialization or a Single Message Transform fails, the record goes to the DLQ. When a Kafka Connect task itself fails due to a misconfigured sink, an unreachable endpoint, or an unhandled exception in the task code, the task halts entirely and no records are routed to the DLQ. Task-level failures require a different remediation path: investigate the task logs, fix the root cause, and restart the connector task.

DLQ in Kafka Streams

Kafka Streams routes failed records through two exception handler interfaces that address different failure surfaces.

DeserializationExceptionHandler fires when a record cannot be deserialized in the source topology. The built-in LogAndContinueExceptionHandler logs a warning and drops the record. This is not a DLQ. The record is gone. If your topology is using LogAndContinueExceptionHandler without a custom handler alongside it, you may be losing records without knowing it. A custom handler that routes to a DLQ topic looks like this:

public class DlqDeserializationHandler implements DeserializationExceptionHandler {

    private static final String DLQ_TOPIC = "my-app.deserialization-errors";
    private final Producer<byte[], byte[]> dlqProducer;

    @Override
    public DeserializationHandlerResponse handle(
            ProcessorContext context,
            ConsumerRecord<byte[], byte[]> record,
            Exception exception) {

        Headers headers = new RecordHeaders(record.headers());
        headers.add("dlq-source-topic",
            record.topic().getBytes(UTF_8));
        headers.add("dlq-source-partition",
            ByteBuffer.allocate(4).putInt(record.partition()).array());
        headers.add("dlq-source-offset",
            ByteBuffer.allocate(8).putLong(record.offset()).array());
        headers.add("dlq-exception-class",
            exception.getClass().getName().getBytes(UTF_8));

        dlqProducer.send(new ProducerRecord<>(
            DLQ_TOPIC, record.partition(),
            record.key(), record.value(), headers));

        return DeserializationHandlerResponse.CONTINUE;
    }
}

props.put(
    StreamsConfig.DEFAULT_DESERIALIZATION_EXCEPTION_HANDLER_CLASS_CONFIG,
    DlqDeserializationHandler.class
);

ProductionExceptionHandler covers exceptions that occur when Kafka Streams attempts to write output records: serialization failures on the output side, or produce errors to destination topics. A custom ProductionExceptionHandler can route these to a separate DLQ topic using the same pattern.

Comparing the three implementation paths

Capability	Spring Kafka	Kafka Connect	Kafka Streams
Catches deserialization errors	Yes	Yes	Yes (custom handler required)
Catches business-logic errors	Yes	No	Yes (ProductionExceptionHandler)
Catches task/topology-level errors	No	No	No
Built-in retry before DLQ	Yes (BackOff config)	No	No
Error context headers	Yes (built-in)	Yes (built-in)	Custom implementation
Requires custom code	Minimal	None	Yes
Preserves original message key	Yes (default)	Yes	Yes (if implemented)

The mistakes that break a DLQ in production

Losing ordering on replay

When you publish to a DLQ topic, the key you use determines which partition the record lands on. If you change the key or publish with no key, Kafka assigns the record to a partition based on the new (or null) key. When you later replay from the DLQ back to the source topic, the record arrives on a different partition than the original, which breaks per-key ordering guarantees for any downstream consumer that depends on them.

The correct approach is to preserve the original message key when writing to the DLQ. Spring Kafka’s DeadLetterPublishingRecoverer does this by default. In custom implementations, explicitly copy the original key. For replay to maintain ordering correctly, your DLQ topic should have the same partition count as the source topic, or you should have a deliberate strategy for re-keying on replay and an understanding of the ordering consequences.

Schema mismatch on the DLQ itself

One common reason a message ends up in a DLQ is a schema mismatch: the consumer cannot deserialize the record against the expected Schema Registry subject. If your DLQ topic is configured to use the same Schema Registry subject as the source topic, you have a circular problem: you still cannot deserialize the record in order to write it to the DLQ, because the deserialization failure is precisely what you are trying to park.

The solution is to write raw bytes to the DLQ rather than using Schema Registry-managed serialization for the DLQ value. Store failure context as record headers: the original schema ID from the Confluent wire format prefix, the source topic and offset, and the exception type. When a reader later inspects the DLQ, the headers tell it what schema was expected. As the source topic evolves through new schema versions, a raw-bytes DLQ remains readable regardless of schema state.

Infinite retry loops between source and retry topic

If your retry-topic consumer routes exhausted messages back to the source topic, and the source consumer routes failed messages back to the retry topic, you have a loop. Both hops may seem individually reasonable, but together they create a circuit that can spin indefinitely on a message that will never succeed.

The mitigation is a retry-count header. Before publishing a message to a retry topic, read the current retry count from the headers, increment it, and write it back. When the count reaches the retry budget, route to the DLQ instead of the retry topic. Spring Kafka’s DefaultErrorHandler manages this internally. In custom implementations, you need to track it explicitly and test that the budget is actually enforced before deploying to production.

Idempotency on replay

Replaying a record from the DLQ re-runs your consumer logic. If your consumer executed side effects before the failure occurred - inserting a database row, sending an external notification, initiating a payment - replaying the record will execute those side effects again. Blind replay is safe only for consumers whose operations are idempotent.

Before building replay tooling, audit your consumer for side effects that cannot be safely re-applied. Non-idempotent operations need either a deduplication check keyed on the original offset and topic (available in the DLQ record headers) or a transactional pattern that ties the side effect to the Kafka offset commit.

Silent failure when the DLQ producer itself fails

If the DLQ producer cannot connect, times out, or the DLQ topic does not exist, your consumer needs an explicit policy. The options are: pause and alert without committing the offset, commit the offset and accept the loss, or crash the consumer process.

Committing the offset when the DLQ write has failed means the record is gone permanently. For most applications, this is the wrong trade-off. The better default is to not commit, allowing the consumer to restart and reprocess from the last committed offset, combined with an alert that fires when DLQ write failures begin occurring. This requires that your consumer is idempotent for records it may reprocess, which is another reason to audit for idempotency before the first production incident.

Monitoring and replaying the DLQ

Two metrics provide meaningful signal for a DLQ in production.

Consumer lag on the DLQ topic (topic depth) tells you how many records have been parked and not yet consumed by a replay consumer. An increasing depth signals upstream failures. A depth of zero means the DLQ is clear.

Age of the oldest unprocessed record is more operationally useful than depth alone. A DLQ with 200 records that arrived in the last 10 minutes is a different situation from 200 records sitting there for 8 hours. Most monitoring systems expose the timestamp of the earliest unread record in a topic. An alert that fires when any record has been in the DLQ for more than a configured time threshold is more actionable than a pure depth threshold, because it is independent of traffic volume. It stays relevant at both low and high throughput, and it fires whether lag accumulated slowly or all at once.

For replay, the most common approaches are:

kcat (formerly kafkacat): consume from the DLQ topic and produce to the source or retry topic via stdin/stdout pipe. Sufficient for small-volume one-off replays.
Kafka MirrorMaker 2: can mirror a DLQ topic to another cluster or reroute it to a different topic, with filtering via record header predicates. Higher operational overhead for straightforward cases.
Custom replay service: a dedicated consumer that reads from the DLQ and produces to the source, optionally with a transform step to correct invalid records before re-submission. Most production teams build some version of this eventually.

Off-the-shelf replay tooling for Kafka is thinner than what exists in RabbitMQ or SQS ecosystems, and most teams end up scripting or building their own.

A related architectural choice is whether to maintain one DLQ topic per source topic or a single shared DLQ. One-per-source makes depth monitoring unambiguous and simplifies replay routing: you always know which source the records came from, and a depth alert on payments.DLT points directly at the payments consumer. A shared DLQ is simpler to operate at small scale but becomes harder to attribute as the number of upstream consumers grows, since topic depth is no longer associated with a single system. The replay consumer for a shared DLQ also needs to consult the source topic header on each record to determine where to route it.

Managing Kafka DLQ in Kpow

Kafka has no native DLQ outside of Kafka Connect. In all other cases, the pattern is implemented at the application or framework level, and its quality reflects the care taken in the implementation.

The production-grade shape is retry-topic plus DLQ: transient failures should have a bounded retry pass before a message is parked, and the DLQ itself should be the terminal state rather than another retry tier. The operational surface - monitoring for record age, alerting promptly, and replaying records with confidence about idempotency - is what separates a DLQ that exists from a DLQ that is useful during an incident.

If you want a UI for inspecting DLQ topics, monitoring record age, and replaying records back to a source or retry topic with RBAC controls and a full audit trail, take a look at Kpow.

‍

The Clone to Topic feature introduced in Kpow 96.2 performs a byte-level clone of DLQ records to any permitted topic directly from the Data Inspect interface, with per-topic RBAC policies restricting which topics can serve as clone sources and destinations, and a complete audit log of every replay operation. You can try it free for 30 days against any Kafka cluster.

Best Kafka management tools for 2026

Factor House — Wed, 17 Jun 2026 00:00:00 GMT

Apache Kafka ships with a CLI and nothing else. Every web UI, observability layer, and governance workflow your team relies on is a third-party project built on top of it. That makes the choice of management tool consequential: it shapes how quickly engineers can debug a stuck consumer group, how cleanly you can handle a PII audit, and whether your platform team spends its time on actual infrastructure rather than running kafka-consumer-groups.sh on behalf of every application team.

This article covers the tools that appear most consistently on real-world Kafka shortlists in 2026, with honest assessments of where each one fits and where it falls short. For a focused comparison of web interfaces specifically, see our Best Kafka UI roundup.

10 best Kafka management tools

The table below summarises the tools covered in this article. Scores reflect the current state of each product as of mid-2026.

Tool	Type	Multi-cluster	RBAC/Security	Active?	Pricing
Kpow	Commercial	Yes	Full (SAML/OIDC/LDAP)	Yes	$3,950/cluster/yr
Confluent Control Center	Commercial (bundled)	Yes	Full (MDS)	Yes	Bundled with Confluent Platform
AKHQ	Open-source	Yes	Basic	Yes	Free
Conduktor	Commercial / open-core	Yes	Full	Yes	From ~$80k/yr (100 seats, 3 clusters)
Kafbat UI	Open-source	Yes	Basic	Yes	Free
Redpanda Console	Open-core	Limited	Enterprise licence only	Yes	Free (community); contact sales
Lenses HQ	Commercial	Yes	Full	Yes	From $4,000/yr (1 cluster, 15 users)
Kafdrop	Open-source	No	None	Minimal	Free
Offset Explorer	Commercial (desktop)	Yes	None	Yes	$99/user/yr
CMAK	Open-source	Yes	None	No	Free

1. Kpow by Factor House

Type: Commercial (proprietary, self-hosted container). Community Edition available for non-production use.

Kpow is a single stateless JVM container that connects to any Kafka cluster, whether that’s self-managed, MSK, Confluent Cloud, Redpanda, Aiven, or Instaclustr, and presents a unified UI and REST API covering topics, consumer groups, Kafka Connect, Schema Registry, ksqlDB, ACLs, and broker configuration. One instance manages up to 12 clusters. Telemetry is stored in internal Kafka topics on the monitored cluster itself, so there are no external databases, sidecars, or additional infrastructure to manage. The current release is v95.5.

Best for

SRE and platform teams running multi-cluster Kafka in regulated environments where audit logging, server-side data masking, and air-gapped deployment are required. Kpow is the primary recommendation for fintech, healthcare, and public sector teams.

Strengths

Deployment simplicity. A single stateless container, configurable entirely via environment variables. No Postgres, no RocksDB, no persistent volumes to worry about.

Security coverage. RBAC, multi-tenancy, SAML/OIDC/LDAP/Keycloak SSO, and server-side data masking with a streamed audit log that ships to any SIEM. This is where most open-source alternatives stop short.

kJQ search. A JQ-like predicate language for filtering messages across millions of records directly in the UI, without exporting data.

Kafka Streams topology visualisation. Native, rendered in-product. Most other tools in this list do not offer this.

Pricing transparency. Kpow pricing is publicly available, and scales based on the number of clusters, not per user. Licenses can be purchased from Factor House, or through the AWS Marketplace on an annual or pay-as-you-go basis.

Accessibility. WCAG 2.1 AA compliant, per Factor House’s documentation.

Limitations

No managed SaaS option. You are responsible for running the container.

Setup and maintenance

Kpow runs as a Docker container, deployable via Docker Compose, Helm, CloudFormation, or AWS Marketplace. Configuration is handled through environment variables. Because there is no external state store, upgrades are a container swap.

Pricing

Pricing

Console Community: free (up to 50 users, 3 clusters). Console Scale: per-seat, 50-seat minimum, pricing via Conduktor sales.

5. Kafbat UI (formerly Provectus kafka-ui)

Type: Open-source (Apache 2.0).

When Provectus paused development on provectus/kafka-ui in September 2023, the original maintainers forked it as kafbat/kafka-ui. The Kafbat fork is now the active line. If your environment is still pulling provectuslabs/kafka-ui Docker images, switch immediately: the original repo carried CVE-2023-52251, an RCE that took roughly 4.5 months to patch after initial disclosure.

Kafbat UI is the most modern-looking open-source Kafka UI currently available, with a clean interface and active release cadence. Latest release: v1.3.0 (July 2025).

Best for

Small to mid-size teams that want a clean, modern interface and can tolerate basic RBAC and client-side-only data masking.

Strengths

Multi-cluster support, Avro/Protobuf/JSON deserialization, Schema Registry and Connect integration, CEL-based message filtering (replacing the Groovy filters that caused the RCE), YAML-based RBAC, MCP support added in v1.3.0.

Limitations

Volunteer-maintained with no commercial backing or SLA. RBAC is basic, with no team or namespace ownership model. Data masking is regex-based and client-side, which means it can be bypassed. Not suitable for environments with server-side masking requirements.

Pricing

Free.

6. Redpanda Console

Type: Open-core. Community edition under BSL; enterprise features under the Redpanda Community License (RCL).

Originally Kowl by CloudHut, acquired by Redpanda. The Go binary is fast and lightweight, and the message viewer is the best in class for UX. The catch is the licensing model: every feature required for multi-team production operations (SSO, RBAC, data masking, audit logging) is behind a paid Redpanda Enterprise licence.

Best for

Teams already running Redpanda who have an enterprise contract. For vanilla Apache Kafka shops, the community edition is a topic viewer, not a management tool.

Limitations

If you are not a Redpanda customer, the enterprise features that make it a serious production tool are inaccessible without a sales engagement. Per the Redpanda documentation, if an enterprise licence expires while features are enabled, the Console shuts those features down. Enterprise pricing is not publicly listed.

Pricing

Free for community features. Enterprise pricing tied to your Redpanda cluster contract; contact Redpanda sales.

7. Lenses HQ

Type: Commercial.

Lenses HQ is a multi-cluster Kafka management and stream processing platform, now owned by Celonis following a 2021 acquisition. Its main differentiation is SQL Processors: stream processing jobs written as SQL that execute as Kubernetes pods. No other tool in this list offers an equivalent. Lenses also provides a topology visualisation layer, a Kafka-to-Kafka replicator (K2K), and a global multi-cluster catalogue with data policies for masking and security groups.

No deployment overhead. Works from any network location with cluster access. Handles awkward auth configurations reliably. Plugin SDK for custom deserialisers. Current version: 3.0.4.

Limitations

Single-user by design. Per-user commercial licensing scales linearly at $99/user/year, which makes it expensive for teams. No server-side access control, no audit trail, no team-level governance.

Pricing

$99 per user/year (commercial). Free for verified personal use.

10. CMAK (Cluster Manager for Apache Kafka)

Type: Open-source (Apache 2.0). Formerly Yahoo Kafka Manager.

CMAK was the dominant Kafka admin UI before 2020. Its last release was v3.0.0.6 on 29 April 2022. Since then, the project has not shipped a release, and its ZooKeeper-centric design is a poor fit for KRaft-mode Kafka 4.x. Open issues include bugs filed in 2016 and 2017 that remain unresolved.

Best for

Legacy installations that already run it and cannot be migrated in the short term. Do not deploy CMAK for new projects.

Limitations

No releases since April 2022. ZooKeeper dependency makes it incompatible with Kafka 4.x KRaft mode. No Schema Registry integration, no Connect management, no message browsing.

Pricing

Free.

Best free Kafka management tools

First recommendation: Kpow Community Edition. The Community Edition is a free Docker image intended for non-production environments. It gives your team access to Kpow’s interface and feature set for local development, staging, and ephemeral clusters, so if you later move to Enterprise, there is no learning curve. It is the most capable free option if your goal is evaluating production-grade tooling.

Second recommendation: AKHQ. For teams that need a free tool in production and have no server-side masking requirements, AKHQ is the most defensible choice. It has a mature Helm story, GitOps-native configuration, and a broader enterprise adoption record than any other free option.

Third recommendation: Kafbat UI. For greenfield environments where governance requirements are light, Kafbat UI is the most actively developed open-source option and the most approachable for teams new to Kafka management tooling.

Best open-source Kafka management tool

AKHQ. It is the most production-tested free open-source option, with a GitOps configuration model, multi-cluster support, SSO via OIDC/OAuth2/LDAP, and a track record of deployment at organisations including Adobe and BlaBlaCar. The one firm caveat: it has no native data masking. If that is a requirement, you need a commercial tool.

Kafbat UI is a close second for teams starting from scratch, but it carries the inherent risk of being volunteer-maintained with no commercial backing. Check the GitHub release cadence before committing.

Choosing the right tool

Cluster count and scale. Per-cluster pricing (Kpow) becomes significantly more cost-effective than per-seat pricing (Conduktor) once your team grows past around 20-30 users per cluster. Run the numbers for your specific headcount and cluster count before committing to either model.

Compliance and data governance. RBAC and SSO are table stakes for any team beyond a handful of developers. Server-side data masking with an auditable log narrows the viable list considerably: Kpow, Conduktor, and Lenses HQ all provide it properly. AKHQ and Kafbat UI do not.

Deployment model. All tools covered here are self-hosted. None offers a managed SaaS option, so you are absorbing the operational overhead regardless. Factor that into TCO calculations, especially for open-source tools: 0.2 FTE/year to maintain AKHQ at a burdened engineering cost of $200,000-$300,000/year is $40,000-$60,000/year before you account for security patching, upgrades, or on-call coverage.

Kafka provider compatibility. If you are running MSK, Confluent Cloud, Redpanda, or Aiven rather than self-managed Kafka, verify provider-specific compatibility before trialling anything. Kpow is explicitly tested against all major managed Kafka providers. Confluent Control Center is only practical for Confluent Platform deployments.

Developer tooling vs. team tooling. Kafdrop and Offset Explorer are personal developer tools, not team infrastructure. Both are useful for ad-hoc inspection and local development, but neither replaces a server-side management layer for production operations.

Maintenance risk. CMAK is effectively abandoned and should not be used for new deployments. The original Provectus kafka-ui repo is no longer maintained; use Kafbat UI if you want that codebase. AKHQ, Kafbat, and Kafdrop are all volunteer-led, which is a risk to quantify before depending on them in regulated environments.

FAQ

What is the difference between a Kafka UI and a Kafka management tool? A Kafka UI typically refers to a web interface for browsing topics and messages. A Kafka management tool covers a broader scope: consumer group management, offset resets, ACL administration, Schema Registry, Connect, and audit logging. Several tools in this list provide both.

Which Kafka management tool works with Amazon MSK? Kpow, AKHQ, Kafbat UI, and Conduktor all support MSK. Kpow documents MSK as an explicit provider with dedicated configuration guidance. Confluent Control Center is not practical for MSK deployments. Kpow is available on the AWS Marketplace.

Do any Kafka management tools offer a free trial? Kpow offers a 30-day free Enterprise trial. AKHQ, Kafbat UI, and CMAK are free to run without a trial period.

Which tool is best for teams with HIPAA or PCI-DSS requirements? Requirements for data masking and auditable access logs narrow the list to Kpow, Conduktor, and Lenses HQ. Of these, Kpow has the lowest deployment complexity and the only publicly listed price. AKHQ and Kafbat UI do not provide masking and should not be used as the primary tool in regulated data environments.

Is CMAK still maintained? No. The last release was April 2022. It also depends on ZooKeeper, which is removed in Kafka 4.x. CMAK should be treated as deprecated for any new deployment.

How does Kpow pricing compare to Conduktor for large teams? Kpow is priced per cluster, not per seat. For 3 clusters and 100 users, Kpow costs $13,500/year at AWS Marketplace list price. Conduktor’s per-seat model at a similar scale typically comes in at $80,000-$150,000/year at list price. The gap widens as team size grows.

Best Kafka monitoring tools for 2026

Factor House — Wed, 17 Jun 2026 00:00:00 GMT

Apache Kafka monitoring is harder than it looks. By the time Kafka actually matters to your organisation, you’re running multiple clusters across cloud providers, Schema Registry is drifting, and the monitoring tool you stood up eighteen months ago now has its own Postgres database, its own backup regime, and a runbook for the rebalancing loop it enters every time a broker restarts.

A serious monitoring layer needs to handle at least six jobs simultaneously: broker and cluster health, consumer lag at the partition level, schema correctness, topology operations (Connect, Streams, Flink), message inspection during incidents, and, for regulated teams, governance including RBAC, SSO, audit logging, and PII masking. Most tools do one or two of those jobs well. Few handle all six without operational overhead that rivals the cluster itself.

This guide covers the twelve tools that come up most in production engineering conversations in 2026. If you’re evaluating management UIs more broadly, see our Best Kafka UI guide.

12 best Kafka monitoring tools

Tool	Type	Best for	Pricing
Kpow	Full-stack enterprise UI	Multi-cluster, regulated environments	From $4,500/yr per cluster
Confluent Control Center	Enterprise UI	All-in Confluent Platform deployments	Bundled with Confluent Platform
Conduktor	Commercial UI + proxy	Large-scale governance and multi-tenancy	Per seat + per cluster (Gateway)
AKHQ	Open-source UI	Non-regulated teams, GitOps environments	Free (Apache 2.0)
Redpanda Console	Open-source UI	Redpanda-first teams	Free core; SSO/RBAC require Enterprise
Kafka UI (Provectus)	Open-source UI	Lightweight single-cluster visibility	Free (Apache 2.0)
Kafdrop	Lightweight UI	Local and dev use	Free (Apache 2.0)
Prometheus + Grafana	Metrics stack	Time-series alerting substrate	Free (OSS)
Kminion	Metrics exporter	Prometheus-native consumer lag	Free (MIT)
Burrow	Lag evaluator	Threshold-free lag alerting	Free (Apache 2.0)
Datadog	SaaS observability	Teams standardised on Datadog	Per host + custom metrics
Lenses.io	SQL-driven platform	SQL-over-Kafka and cross-cluster replication	Enterprise sales

1. Kpow by Factor House

Kpow is a commercial enterprise UI and API for Apache Kafka and the surrounding ecosystem (Schema Registry, Kafka Connect, ksqlDB), covering monitoring, operations, and governance from a single stateless Docker container with zero external dependencies. It stores telemetry in internal Kafka topics on the cluster it monitors, so its resilience is tied directly to the cluster’s: if your cluster is up, Kpow is up.

Best for

Platform and SRE teams managing multiple clusters across MSK, Confluent Cloud, Redpanda, and self-managed brokers with enterprise security and compliance requirements. The default choice for financial services, healthcare, and fintech.

Strengths

Stateless single container with no Postgres, RocksDB, or sidecars to operate or back up
Full enterprise governance: RBAC, SSO (Okta, OIDC, SAML, LDAP, Keycloak), server-side PII masking, and a complete user-action audit log shippable to any SIEM
kJQ server-side message filtering across JSON, Avro, Protobuf, and Transit, with no throwaway consumer code required during incidents
First-class MSK IAM authentication and native AWS Glue Schema Registry support (rare among competing tools)
Multi-cluster from a single install; Prometheus endpoints per cluster and topic feed your existing Grafana stack
Native KRaft support

For teams that need a free tool with no production restrictions, AKHQ is the most mature open-source option, with broad feature coverage and a genuine auth story via OIDC and LDAP. As a third pick, Kafka UI by Provectus is the easier starting point if you need quick topic visibility and don’t require auth or multi-cluster support.

Best open-source Kafka monitoring tools

AKHQ is the most mature and complete free open-source option. Its YAML-based configuration, multi-cluster support, and auth backends (LDAP, OIDC, JWT) go meaningfully further than most free alternatives, making it a genuine fit for GitOps-oriented teams in non-regulated environments.

For metrics specifically, Kminion is the best open-source Prometheus exporter for consumer group lag detail, and pairs well with Grafana for teams building a metrics substrate from scratch. Use both together if your priority is a fully free, instrumented, alerting-capable stack, then layer a UI on top as your requirements grow.

Key considerations when choosing

Deployment complexity

The monitoring tool should not require more infrastructure than the Kafka cluster it monitors. Count every hard dependency (Postgres, Redis, Cortex, RocksDB, per-broker agent) as a separate operational liability with its own upgrade path, backup regime, and failure mode.

Enterprise readiness

For regulated environments, the requirements are specific: RBAC, SSO (OIDC/SAML/LDAP), a user-action audit log you can ship to a SIEM, and server-side data masking that prevents PII from reaching the browser. Client-side masking does not satisfy compliance requirements. Anything less shifts risk onto your team.

Pricing model

Per-seat pricing penalises team growth. Per-host or per-broker pricing penalises horizontal scaling. Per-cluster pricing is the most predictable model for most platform teams. Factor engineering maintenance hours into the total cost of open-source options. They are not always cheaper.

Managed service compatibility

MSK with IAM authentication, Confluent Cloud, Aiven, Redpanda, and Strimzi all have different auth and metadata behaviour. Verify support against your actual production topology. Specifically test MSK IAM auth and AWS Glue Schema Registry if your stack includes them. Many tools claim compatibility but treat both as secondary integrations.

Monitoring scope

Broker metrics are necessary but not sufficient. Ensure the tool covers per-partition consumer lag, Schema Registry visibility, Kafka Connect management, and message inspection with native deserialization. If you are running Kafka Streams or Flink jobs, confirm topology visibility before committing.

FAQ

What is the best Kafka monitoring tool for production use?

Kpow by Factor House is the strongest all-round choice: stateless deployment, enterprise RBAC and SSO, server-side data masking, multi-cluster support, and per-cluster pricing. For open-source-only environments, AKHQ is the most complete free option.

What is the best free Kafka monitoring tool?

Kpow Community Edition offers the most functionality without a license fee: same codebase as the commercial product, limited to non-production clusters. For production-free options, AKHQ is the most mature open-source pick.

Can I monitor Kafka consumer lag without a UI?

Yes. Kpow continuously collects Kafka telemetry, including per-consumer-group and per-partition lag, and emits it as a data stream you can route to any observability system. Whether you’re running Prometheus, Datadog, Grafana, or a custom alerting stack, you can consume Kpow’s metrics directly without needing a separate lag-monitoring tool or a dedicated UI.

Does Confluent Control Center work with MSK or self-managed Kafka?

No, not meaningfully. Control Center is built for Confluent Platform and loses most of its value outside that context. For mixed-broker or multi-cloud Kafka fleets, you need a vendor-agnostic tool.

How do I monitor Kafka in a regulated environment?

You need server-side data masking, RBAC, SSO, and a user-action audit log shippable to your SIEM. Kpow covers all of these from a single stateless container, with no secondary database required. Verify that any tool you evaluate masks data server-side, not in the browser.

What Kafka monitoring tool works best with AWS MSK?

Kpow provides first-class MSK IAM authentication and native AWS Glue Schema Registry support. Most tools treat MSK as a secondary integration or lack IAM auth support entirely. Test both explicitly during any PoC against an MSK cluster.

Kafka broker monitoring

Factor House — Thu, 04 Jun 2026 00:00:00 GMT

Brokers are the operational core of a Kafka cluster. They receive produce requests, serve fetch requests, manage partition replicas, and coordinate leader election. When a broker degrades, every producer and consumer connected to it is affected, often before your monitoring stack surfaces anything useful.

This article covers broker-level monitoring specifically: the JMX metrics to watch, thresholds to alert on, process monitoring scripts, and how to diagnose the most common failure modes. For cluster-wide concerns, consumer lag, and producer delivery guarantees, refer to the separate guides on Kafka cluster monitoring, Kafka consumer monitoring, and Kafka producer monitoring. For a full overview of Kafka observability, see the Kafka monitoring guide.

Key takeaways

Kafka brokers expose metrics via JMX by default; Prometheus users need the JMX Exporter agent to scrape them.
UnderReplicatedPartitions and ActiveControllerCount are the two most critical metrics for cluster health.
Broker monitoring covers three layers: the Kafka process, the JVM, and the host OS. All three require attention.
A lightweight process monitoring script can detect broker failures faster than most monitoring stacks.
Most broker incidents trace to a small set of root causes: disk saturation, leader imbalance, network thread exhaustion, or ISR instability.

What is Kafka broker monitoring?

A Kafka broker handles several concurrent responsibilities: writing incoming messages to disk, serving read requests from consumers, replicating partition data to follower brokers, and participating in leader election. Each of these activities has distinct failure modes.

Broker health is not the same as cluster health. A single degraded broker can produce replica lag, leader imbalance, and consumer fetch failures without the cluster appearing unavailable. Topics remain accessible on unaffected brokers, but the partitions whose leaders or followers are on the degraded broker will silently degrade. Identifying which broker is under pressure, and why, is the purpose of broker-level monitoring.

Monitoring spans three layers:

Kafka process metrics (exposed via JMX): replication state, request throughput, controller status, failure rates.
JVM metrics: heap usage, garbage collection pause duration, open file descriptors.
Host OS metrics: disk capacity, disk I/O throughput, network utilization.

Consumer lag, producer delivery guarantees, and cluster-wide partition distribution are covered in separate articles. This article focuses on the broker process and what runs immediately around it.

How Kafka brokers expose metrics

JMX (default)

Kafka exposes metrics as JMX MBeans by default. Server-side broker metrics use Yammer Metrics internally; native Java clients use Kafka’s own metrics registry. Both project their measurements onto MBeans hosted in the JVM’s MBean server.

Remote JMX access is disabled by default. To enable it, set the JMX_PORT environment variable when starting the broker:

export JMX_PORT=9999

In production, raw JMX access is a security risk. Because JMX supports remote method invocation (RMI), an unauthenticated client can invoke operations on MBeans, including modifying runtime configurations and triggering JVM shutdowns. Use KAFKA_JMX_OPTS to enforce authentication and encryption:

export KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote \
  -Dcom.sun.management.jmxremote.authenticate=true \
  -Dcom.sun.management.jmxremote.ssl=true \
  -Dcom.sun.management.jmxremote.port=9999 \
  -Dcom.sun.management.jmxremote.rmi.port=9999 \
  -Dcom.sun.management.jmxremote.password.file=/etc/kafka/jmxremote.password \
  -Dcom.sun.management.jmxremote.access.file=/etc/kafka/jmxremote.access"

Tools such as jconsole, jmxterm, and kafka-run-class.sh kafka.tools.JmxTool can query JMX directly once access is enabled.

Prometheus via JMX Exporter

The standard approach for Prometheus-based stacks is to run the Prometheus JMX Exporter as a Java agent alongside the broker JVM.

Running the exporter in local agent mode (rather than as a standalone HTTP server) is preferred. It avoids the serialization overhead of remote RMI polling and captures additional host-level process metrics including JVM CPU and memory utilization.

The agent translates JMX MBeans to Prometheus metrics and serves them on an HTTP endpoint, typically on port 7071. It requires a YAML configuration file that maps MBean paths to Prometheus metric names. A community-maintained configuration for Kafka is available in the JMX Exporter GitHub repository. Keep this configuration file in version control alongside your broker configuration and review it when upgrading Kafka versions.

Kafka metrics reporters (pluggable)

Kafka supports custom MetricsReporter implementations via the metric.reporters setting in server.properties. This allows metrics to be pushed directly to external systems without going through JMX or Prometheus. Confluent Platform ships a reporter that sends metrics to Confluent Control Center. Vendors such as Datadog and New Relic provide their own reporter implementations as well.

KRaft mode: In KRaft mode (GA since Kafka 3.3), some ZooKeeper-related MBeans are removed and several KRaft-specific MBeans appear in their place, including FencedBrokerCount and LastAppliedRecordLagMs. If you are migrating from ZooKeeper mode, or if you have inherited dashboards built against an older cluster, audit your MBean paths before deploying them against a KRaft cluster.

Key metrics to monitor

The sections below cover metrics across all three layers. JMX MBean paths are given for direct JMX access; if you are using Prometheus, metric names follow the pattern produced by the JMX Exporter configuration.

Replication health metrics

These are the highest-priority metrics on any broker. A non-zero value in the first three rows means data availability is at risk.

Metric name	JMX MBean path	Description	Why it matters
`UnderReplicatedPartitions`	`kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions`	Partitions where the ISR count is below the replication factor	A sustained non-zero value means the cluster has fewer replicas than configured to fall back on if a leader fails
`UnderMinIsrPartitionCount`	`kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount`	Partitions with fewer in-sync replicas than `min.insync.replicas`	Producers configured with `acks=all` receive `NotEnoughReplicasException` when this is non-zero
`AtMinIsrPartitionCount`	`kafka.server:type=ReplicaManager,name=AtMinIsrPartitionCount`	Partitions where ISR count equals `min.insync.replicas` exactly	One more broker going offline pushes these into the under-min state; treat as a warning signal
`IsrShrinksPerSec`	`kafka.server:type=ReplicaManager,name=IsrShrinksPerSec`	Rate at which replicas are leaving the ISR	Sustained shrinks indicate follower replicas are falling behind, typically from disk pressure or GC pauses
`IsrExpandsPerSec`	`kafka.server:type=ReplicaManager,name=IsrExpandsPerSec`	Rate at which replicas are rejoining the ISR	High expand rate alongside shrinks indicates ISR churn rather than a clean recovery

Controller metrics

There is always exactly one active controller in a Kafka cluster. These metrics confirm that invariant holds and that the controller is performing well.

Metric name	JMX MBean path	Description	Why it matters
`ActiveControllerCount`	`kafka.controller:type=KafkaController,name=ActiveControllerCount`	Whether this broker is currently the active controller (0 or 1)	The cluster-wide sum must be exactly 1. Zero means no metadata management; more than 1 indicates a split-brain condition
`OfflinePartitionsCount`	`kafka.controller:type=KafkaController,name=OfflinePartitionsCount`	Partitions with no available leader	Non-zero means those partitions are completely unavailable to producers and consumers
`LeaderElectionRateAndTimeMs`	`kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs`	Frequency and duration of leader elections	Frequent elections trigger consumer rebalances and producer retries; elevated duration suggests controller pressure
`FencedBrokerCount`	`kafka.controller:type=KafkaController,name=FencedBrokerCount`	Brokers the controller has marked as unreachable (KRaft only)	Any fenced broker represents lost partition leadership and requires immediate investigation
`LastAppliedRecordLagMs`	`kafka.controller:type=KafkaController,name=LastAppliedRecordLagMs`	Metadata replication lag relative to the active controller (KRaft only)	Should be 0 on the active controller; values above 1,000ms on standby nodes indicate metadata replication delay

Request handling metrics

These metrics indicate whether the broker is keeping up with its request load.

Metric name	JMX MBean path	Description	Why it matters
`RequestHandlerAvgIdlePercent`	`kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent`	Average idle percentage of the request handler thread pool	Values below 0.20 indicate saturation on request processing
`NetworkProcessorAvgIdlePercent`	`kafka.network:type=SocketServer,name=NetworkProcessorAvgIdlePercent`	Average idle percentage of the network processor threads	Target above 0.30; below 0.20 indicates network thread exhaustion, often caused by slow consumers
`TotalTimeMs` (Produce)	`kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce`	End-to-end latency for produce requests	P99 target is under 100ms; sustained spikes indicate broker-side pressure or slow disk
`TotalTimeMs` (FetchConsumer)	`kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchConsumer`	End-to-end latency for consumer fetch requests	Sustained elevation traces to replica lag or disk read pressure
`RequestQueueSize`	`kafka.network:type=RequestChannel,name=RequestQueueSize`	Requests waiting for a handler thread	Should remain below 10; sustained growth indicates handler thread saturation
`FailedProduceRequestsPerSec`	`kafka.server:type=BrokerTopicMetrics,name=FailedProduceRequestsPerSec`	Rate of failed produce requests	Any non-zero value requires investigation
`FailedFetchRequestsPerSec`	`kafka.server:type=BrokerTopicMetrics,name=FailedFetchRequestsPerSec`	Rate of failed fetch requests	Any non-zero value requires investigation

Latency sub-phases. TotalTimeMs is the sum of five phases: time waiting in the request queue (RequestQueueTimeMs), local processing time on the partition leader (LocalTimeMs), replication wait time for acks=all producers (RemoteTimeMs), time waiting in the response queue (ResponseQueueTimeMs), and time to transmit the response to the client (ResponseSendTimeMs). When P99 TotalTimeMs is elevated, checking each sub-phase narrows the root cause. High RemoteTimeMs points to replication pressure. High LocalTimeMs typically indicates disk write saturation or message format conversion overhead.

Throughput and I/O metrics

These metrics give a baseline view of broker load and are useful for capacity planning.

Metric name	JMX MBean path	Description	Why it matters
`BytesInPerSec`	`kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec`	Bytes received per second	Sudden spikes can saturate disk write bandwidth or network interfaces
`BytesOutPerSec`	`kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec`	Bytes sent per second	Includes replication traffic; track separately from consumer traffic where possible
`MessagesInPerSec`	`kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec`	Messages received per second	Useful for understanding message rate independently of message size

Log and disk metrics

Metric name	JMX MBean path	Description	Why it matters
`OfflineLogDirectoryCount`	`kafka.log:type=LogManager,name=OfflineLogDirectoryCount`	Log directories that are offline	Any non-zero value means the broker cannot write to some partitions
`DeadThreadCount`	`kafka.log:type=LogCleaner,name=DeadThreadCount`	Log cleaner threads that have failed silently	Non-zero means compaction has stopped; `uncleanable-bytes` will grow until disk fills
`uncleanable-bytes`	`kafka.log:type=LogCleaner,name=uncleanable-bytes`	Data volume waiting to be compacted	A persistent upward trend on compacted topics indicates log cleaner failure
Disk space (OS)	n/a – monitor via node exporter or equivalent	Percentage of disk used on the Kafka log directory	Kafka stops accepting produce requests when disk fills completely

JVM metrics

JVM pressure is a common root cause of ISR instability and request latency spikes. If a broker’s JVM pauses long enough during garbage collection, the broker fails to send heartbeats to the controller. This triggers a timeout and forces partition leadership reassignment, which looks from the outside like an ISR shrink or elevated election rate.

Metric name	Source	Description	Why it matters
JVM heap used	`java.lang:type=Memory`	Heap in use as a percentage of max	Sustained above 75% increases GC pressure and pause frequency
GC pause duration	`java.lang:type=GarbageCollector`	Duration of stop-the-world GC events	Pauses above approximately 1 second cause ISR shrinks and heartbeat timeouts; the G1GC target is 20ms
GC frequency	`java.lang:type=GarbageCollector`	Number of GC events per minute	High frequency of short pauses often precedes a longer pause event
Open file descriptors	`java.lang:type=OperatingSystem`	Number of open file handles	Kafka opens many handles for log segments; hitting the OS limit causes broker errors

Heap sizing. Kafka uses a hybrid memory model: a small JVM heap handles partition metadata, indexes, and producer state, while the OS page cache holds hot log segment data. Allocating too much to the JVM heap reduces the page cache, which forces consumer reads to disk. On a 64 GB host, a typical configuration is 6-12 GB for the JVM heap and the remainder for the OS page cache. A rough rule of thumb is 1-2 MB of heap per active partition replica hosted on the broker.

Broker process monitoring script

The JMX metrics above require the broker to be running and reachable. A process-level check catches failures that happen before or outside the metrics stack: a broker crash, a port that is not listening, a startup failure, or a JVM that is hung during initialization. The scripts below complement JMX monitoring rather than replacing it.

Check the broker process is running (jps / ps)

jps (JVM process status, included in the JDK) lists running JVM processes by main class. For Apache Kafka, the main class is kafka.Kafka.

#!/bin/bash
BROKER_CLASS="kafka.Kafka"

if jps -l | grep -q "${BROKER_CLASS}"; then
  echo "Broker process is running"
  exit 0
else
  echo "ERROR: Kafka broker process not found"
  exit 1
fi

If jps is not available, fall back to ps:

#!/bin/bash
if ps aux | grep -q '[k]afka.Kafka'; then
  echo "Broker process is running"
  exit 0
else
  echo "ERROR: Kafka broker process not found"
  exit 1
fi

Confluent Platform: The main class for Confluent Server is io.confluent.kafka.server.ConfluentServer. Adjust the grep pattern accordingly.

A running process does not mean the broker is healthy. It may be in an error state or hung waiting on disk or network. Use this check as a first-line detector, not a health indicator.

Check the broker port is accepting connections

A process check confirms the JVM is alive, but not that it is accepting Kafka connections. Check that the listener port (default 9092) is open and accepting connections:

#!/bin/bash
BROKER_HOST="localhost"
BROKER_PORT=9092
TIMEOUT=5

if nc -z -w "${TIMEOUT}" "${BROKER_HOST}" "${BROKER_PORT}" 2>/dev/null; then
  echo "Broker port ${BROKER_PORT} is accepting connections"
  exit 0
else
  echo "ERROR: Broker not accepting connections on ${BROKER_HOST}:${BROKER_PORT}"
  exit 1
fi

In environments without netcat, use the /dev/tcp bash built-in:

#!/bin/bash
BROKER_HOST="localhost"
BROKER_PORT=9092

if (echo > /dev/tcp/${BROKER_HOST}/${BROKER_PORT}) 2>/dev/null; then
  echo "Broker port ${BROKER_PORT} is accepting connections"
  exit 0
else
  echo "ERROR: Broker not accepting connections on ${BROKER_HOST}:${BROKER_PORT}"
  exit 1
fi

JMX-based health check script

A port check validates TCP connectivity but not broker health. Use kafka-run-class.sh with kafka.tools.JmxTool to query UnderReplicatedPartitions directly. This works without additional tooling if the JDK and Kafka binaries are available.

#!/bin/bash
JMX_HOST="localhost"
JMX_PORT=9999
MBEAN="kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions"

result=$(kafka-run-class.sh kafka.tools.JmxTool \
  --jmx-url "service:jmx:rmi:///jndi/rmi://${JMX_HOST}:${JMX_PORT}/jmxrmi" \
  --object-name "${MBEAN}" \
  --attributes Value \
  --one-time true 2>/dev/null | tail -n 1 | awk -F',' '{print $NF}')

if [ -z "${result}" ]; then
  echo "ERROR: Could not retrieve JMX metric"
  exit 2
fi

if [ "${result}" -gt 0 ]; then
  echo "WARNING: UnderReplicatedPartitions = ${result}"
  exit 1
else
  echo "OK: UnderReplicatedPartitions = 0"
  exit 0
fi

This gives a more meaningful health signal than a port check. A broker can accept TCP connections while being in a degraded replication state.

Kafka Admin API check

kafka-broker-api-versions.sh is a lightweight liveness check that validates the Kafka protocol layer, not just TCP connectivity:

kafka-broker-api-versions.sh --bootstrap-server localhost:9092

If the broker is healthy, this returns the list of supported API versions. If the protocol handshake fails or the broker is not responsive, it exits with an error. Use this as a quick manual check or wrap it in a script for automated monitoring. It is a useful final check after a rolling restart to confirm each broker has rejoined the cluster before proceeding to the next.

Broker monitoring tools

Prometheus and Grafana

The most common self-hosted stack. The Prometheus JMX Exporter scrapes broker JMX metrics, Prometheus stores them as time-series data, and Grafana renders dashboards. Community dashboards for Kafka are available on Grafana Labs, though quality and metric coverage vary. Maintaining a complete and accurate Kafka dashboard requires ongoing attention as cluster topology and Kafka versions change.

Managed monitoring platforms (Datadog, New Relic)

Agent-based collection where vendor agents handle JMX scraping and forwarding. Both Datadog and New Relic provide out-of-the-box Kafka dashboards and alert policies, which reduces setup time compared to the self-hosted Prometheus stack. Cost scales with host count and metrics volume. Default alert thresholds may need adjustment for high-throughput deployments.

Confluent Control Center

Available in Confluent Platform, not Apache Kafka. Provides deep integration with Confluent-specific metrics and a built-in interface for broker health, topic management, and consumer lag. Less useful if you are running vanilla Apache Kafka.

Kpow by Factor House

Kpow is purpose-built for Kafka observability. It surfaces broker health, partition state, ISR status, and throughput metrics without requiring a separate Prometheus stack or Grafana instance, and it runs inside your own network. Try it free for 30 days.

Alerting strategy for broker monitoring

The goal is to page on-call when there is an active data risk and route everything else to a dashboard or asynchronous channel. Static thresholds on percentage-based metrics can generate false alarms during quiet periods: a single failed request in a low-traffic window can push an error rate to 50% and trigger a high-severity alert despite having no real impact. Organize alerts by severity:

Priority	Area	Metrics	Check interval	Route
P0	Immediate cluster health	`OfflinePartitionsCount`, `ActiveControllerCount`	15 seconds	On-call pager
P1	Data safety	`UnderReplicatedPartitions`, `IsrShrinksPerSec`, `FencedBrokerCount`	30 seconds	High-priority chat or pager
P2	Performance signals	`TotalTimeMs` P99, broker throughput, thread idle percentages	1 minute	Slack channel
P3	Capacity planning	CPU, memory pools, disk usage	5 minutes	Email or ticket
P4	Deep diagnostics	Per-partition metrics, thread dumps	As needed	Dashboard only

Recommended thresholds based on the Apache Kafka documentation and operational guidance:

Metric	Warning threshold	Critical threshold	Notes
`UnderReplicatedPartitions`	> 0 for > 2 minutes	> 0 for > 5 minutes	Short-lived spikes during rolling restarts are expected
`ActiveControllerCount` (cluster sum)	!= 1	–	Any deviation is immediately critical; no grace period
`OfflinePartitionsCount`	> 0	> 0	Alert immediately; affected partitions are unavailable
`RequestHandlerAvgIdlePercent`	< 0.30	< 0.20	Evaluate on a sustained basis, not momentary spikes
`NetworkProcessorAvgIdlePercent`	< 0.30	< 0.20	Same as above
Disk usage	> 70%	> 85%	Kafka stops accepting produce requests when the log directory fills completely
GC pause duration	> 500ms	> 1 second	Stop-the-world pauses above 1 second cause ISR shrinks and heartbeat timeouts

Link every production alert to a runbook. When an alert fires, the on-call engineer should be able to identify the root cause and begin remediation without first looking up what the metric means. A useful runbook includes the metric’s JMX MBean source, related broker log entries to check, and step-by-step remediation for the most common causes of that alert.

Common broker issues and how to diagnose them

Symptom	Relevant metrics	Likely root cause	Diagnosis and remediation
`UnderReplicatedPartitions` rising	`UnderReplicatedPartitions`, `IsrShrinksPerSec`	Follower broker is slow or unreachable	Check GC pause times and network connectivity on the lagging follower; check disk I/O on the follower
Produce latency spikes	`TotalTimeMs` (Produce), `LocalTimeMs`, disk I/O	Disk write saturation	Decompose `TotalTimeMs` into sub-phases; check disk throughput; consider partition reassignment to less-loaded brokers
`OfflinePartitions` suddenly non-zero	`OfflinePartitionsCount`, `ActiveControllerCount`	Leader broker crashed with no available replicas	Check broker startup logs; if replication factor was 1, data may be lost
Network thread exhaustion	`NetworkProcessorAvgIdlePercent`	Too many slow consumers or large fetch sizes	Increase `num.network.threads`; investigate slow consumers; consider reducing fetch size on affected consumers
Request handler exhaustion	`RequestHandlerAvgIdlePercent`	Too many concurrent requests or heavy compression	Increase `num.io.threads`; check for compression-related CPU spikes
Broker JVM out of memory	JVM heap, GC events	Heap too small for partition count	Increase broker heap (`-Xmx`); check for large message batches; review partition replica count on that broker
Broker failing to start	`OfflineLogDirectoryCount`	Corrupt log directory or disk failure	Check broker startup logs; run `kafka-log-dirs.sh` to identify the offline directory
ISR churning (shrinks and expands)	`IsrShrinksPerSec`, `IsrExpandsPerSec`	GC pauses or network jitter causing replicas to briefly fall behind	Check GC logs; consider tuning `replica.lag.time.max.ms`; verify network stability between brokers
Disk space growing unexpectedly	`uncleanable-bytes`, `DeadThreadCount`	Log cleaner thread has failed silently	Check whether `DeadThreadCount` is greater than 0; if log cleaner threads have died, a broker restart is required

Kafka broker monitoring best practices

Monitor all three layers: Kafka process metrics, JVM metrics, and host OS metrics. A problem at any layer will eventually surface in the others.
Alert on UnderReplicatedPartitions and ActiveControllerCount before everything else. These are the leading indicators of cluster health degradation.
Set disk usage alerts at 70% and 85%. Kafka stops accepting produce requests when the log directory fills completely and does not degrade gracefully before that threshold.
Use a lightweight process check script as a first-line detector. It can catch broker failures before your metrics pipeline does, particularly in the window between a crash and the first missed metrics scrape.
Keep your JMX Exporter configuration in version control and treat it as part of your Kafka infrastructure. Changes to it affect what your dashboards and alerts can observe.
Do not rely on consumer lag alone as a broker health signal. Consumer lag is a symptom; broker metrics tell you the cause.
In KRaft mode, audit your dashboard configuration during and after the migration. ZooKeeper-related MBeans are removed; KRaft-specific MBeans such as FencedBrokerCount and LastAppliedRecordLagMs appear in their place.
Link every production alert to a runbook that includes the metric’s JMX MBean source, related log entries to check, and remediation steps for the most common causes.

Monitor Kafka brokers with Factor House

Kpow surfaces broker health, partition state, ISR status, and throughput metrics from inside your own network. You get visibility into the metrics covered in this article without running a separate Prometheus stack or maintaining Grafana dashboards. Give it a try with a free 30-day trial.

Kafka cluster monitoring

Factor House — Thu, 04 Jun 2026 00:00:00 GMT

Kafka clusters are typically monitored from the broker up: process health checks, JVM metrics, disk alerts, per-broker request latencies. That instrumentation is necessary, but it has a blind spot. A broker can report normal metrics in isolation while the cluster as a whole is in a degraded state. Partition leadership can be unbalanced. Replication can be falling behind across the fleet. The controller can fail silently. None of these conditions are visible from a single broker’s perspective.

This article covers cluster-level monitoring: the aggregated, cross-broker view that reveals how the cluster is functioning as a unit. It covers the metrics that matter at this layer, how to collect them across a broker fleet, how to alert on them effectively, and how to diagnose the most common cluster-level failures.

Per-broker internals — request thread pools, JVM heap, disk I/O — are covered in the Kafka broker monitoring article. Consumer-side lag monitoring is covered in the Kafka consumer monitoring article. See the Kafka monitoring guide for the full picture.

Key takeaways

Cluster monitoring is different from broker monitoring: it gives you an aggregated view of replication health, partition distribution, and control plane state across all brokers.
ActiveControllerCount, OfflinePartitionsCount, and UnderReplicatedPartitions are the three cluster-level metrics that most directly signal data availability risk.
In KRaft mode (Kafka 3.3+ GA, default from 4.0), the control plane metrics change — ZooKeeper-era metrics are replaced by KRaft-native equivalents.
A cluster health check script that verifies five or six key metrics is usually enough to catch the most serious problems; the value is in running it consistently and alerting on deviations.
Capacity planning at the cluster level is about trend monitoring, not single-point thresholds — watch bytes-in per broker, disk utilisation rate, and partition count per broker over time.

What is Kafka cluster monitoring?

Cluster monitoring is the practice of observing the aggregated state of all brokers rather than the internals of any single one. The distinction matters in practice.

Broker monitoring tells you whether an individual broker is healthy: its request thread utilisation, JVM heap pressure, disk throughput, and local ISR state. Cluster monitoring tells you whether the cluster as a whole is functioning: whether replication is consistent across the fleet, whether partition leadership is balanced, whether the controller is in a valid state, and whether the cluster has capacity headroom.

Both layers are necessary. Cluster health checks can miss per-broker pathologies that are not yet visible at the aggregate level. A JVM GC pause on one broker, for example, may not immediately register as a replication deficit on the cluster health dashboard. Conversely, per-broker monitoring alone will not surface a partition imbalance, a controller failure, or a sustained ISR shrink rate that only becomes meaningful when measured across the fleet.

The same JMX interface exposes data at both levels. The difference is in how you query and aggregate it.

Key cluster-level metrics

Cluster-level metrics fall into two categories: metrics that are properties of the cluster as a whole (controller state, partition distribution), and metrics that are only meaningful when aggregated across brokers (replication health, throughput). The four groups below cover both.

Control plane health

The controller is the broker responsible for partition leader elections and cluster metadata management. There is always exactly one active controller in a functioning cluster. Deviations are critical.

Metric name	JMX MBean path	Description	Target / alert threshold
`ActiveControllerCount`	`kafka.controller:type=KafkaController,name=ActiveControllerCount`	Number of active controllers on this broker (0 or 1). Must sum to exactly 1 across all brokers.	Alert (critical) if cluster-wide sum is not exactly 1
`OfflinePartitionsCount`	`kafka.controller:type=KafkaController,name=OfflinePartitionsCount`	Number of partitions with no available leader. Reported by the active controller.	Alert (critical) if > 0
`LeaderElectionRateAndTimeMs`	`kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs`	Rate and duration of partition leader elections. Frequent or slow elections indicate controller instability or broker churn.	Alert (warning) if rate is sustained above baseline; alert (critical) if P99 duration is elevated
`EventQueueSize`	`kafka.controller:type=ControllerEventManager,name=EventQueueSize`	Pending administrative events queued on the controller.	Alert (warning) if > 100
`AvgIdleRatio`	`kafka.controller:type=ControllerEventManager,name=AvgIdleRatio`	Fraction of time the controller event thread is idle. A value approaching 0 indicates a saturated controller.	Alert (warning) if approaching 0

KRaft mode: In KRaft mode (Kafka 3.3+, default from 4.0), the following metrics replace ZooKeeper-era control plane telemetry. Verify MBean paths against the Kafka version in use.

Metric name	JMX MBean path	Description	Target / alert threshold
`LastAppliedRecordLagMs`	`kafka.controller:type=KafkaController,name=LastAppliedRecordLagMs`	Delay between metadata commits on the active controller and local application on standby controllers. Elevated values mean metadata changes are propagating slowly.	Alert (warning) if elevated and sustained
`MetadataErrorCount`	`kafka.controller:type=KafkaController,name=MetadataErrorCount`	Count of failed metadata log operations.	Alert (critical) if > 0
`current-state` (quorum state)	`kafka.server:type=raft-metrics,name=current-state`	State of the KRaft quorum member on this broker (Leader, Follower, Candidate).	Alert if controller broker is not Leader
`commit-latency-avg`	`kafka.server:type=raft-metrics,name=commit-latency-avg`	Average latency for KRaft metadata log commits.	Alert (warning) if elevated

A KRaft quorum of size n (typically an odd number: 3 or 5) requires a strict majority of floor(n/2) + 1 active nodes to elect a leader and accept metadata commits. If the active quorum drops below this threshold, the control plane enters a read-only mode. Brokers continue to process produce and consume requests for existing partitions, but topic creation, partition reassignment, and ISR state modifications are blocked. If a partition leader fails while the quorum is unavailable, client requests to that partition will time out.

Replication fleet health

These are the most operationally significant cluster-level metrics. They reflect whether data is being replicated as configured across all brokers, not just on one.

Metric name	JMX MBean path	Description	Target / alert threshold
`UnderReplicatedPartitions`	`kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions`	Partitions where the current ISR count is less than the replication factor. Sum across all brokers for the cluster total.	Alert (critical) if sum > 0 for more than 5 minutes
`UnderMinIsrPartitionCount`	`kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount`	Partitions with fewer ISRs than `min.insync.replicas`. Producers configured with `acks=all` will receive errors.	Alert (critical) if > 0
`UncleanLeaderElectionsPerSec`	`kafka.controller:type=ControllerStats,name=UncleanLeaderElectionsPerSec`	Rate of elections where a non-ISR replica was promoted to leader. Indicates data loss has occurred or is imminent.	Alert (critical) if > 0
`IsrShrinksPerSec`	`kafka.server:type=ReplicaManager,name=IsrShrinksPerSec`	Rate at which replicas are leaving the ISR. Track per broker and as a cluster sum.	Alert (warning) if sustained above zero
`IsrExpandsPerSec`	`kafka.server:type=ReplicaManager,name=IsrExpandsPerSec`	Rate at which replicas are rejoining the ISR.	Monitor alongside `IsrShrinksPerSec` — sustained shrinks paired with expands indicate ISR instability

Note that UnderReplicatedPartitions is technically a per-broker metric, but it only becomes useful as a cluster-level aggregate. A URP count of zero on one broker says nothing about the remaining brokers. Sum it across the fleet.

Partition distribution

These metrics indicate whether work is balanced evenly across the broker fleet. Imbalances affect both throughput and fault tolerance.

Metric name	JMX MBean path	Description	Target / alert threshold
`PartitionCount`	`kafka.server:type=ReplicaManager,name=PartitionCount`	Total partition replicas on this broker. Compare across brokers.	Alert (warning) if the spread between the highest and lowest broker exceeds 20%
`LeaderCount`	`kafka.server:type=ReplicaManager,name=LeaderCount`	Number of partition leaders on this broker. Leaders handle all produce and fetch traffic for their partitions.	Alert (warning) if leader distribution is significantly uneven across brokers

Uneven partition or leader counts after a broker restart or rolling upgrade are common and typically self-correct via the preferred leader election mechanism. Alert only on sustained imbalance.

Throughput and capacity signals

These metrics inform both capacity planning and short-term incident response.

Metric name	JMX MBean path	Description	Target / alert threshold
`BytesInPerSec`	`kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec`	Bytes received per second per broker. Track per broker and as a cluster total.	Alert (warning) when a broker’s throughput exceeds 70-80% of its network or disk write capacity
`BytesOutPerSec`	`kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec`	Bytes sent per second per broker. Includes replication traffic.	Monitor alongside `BytesInPerSec` — replication multiplies egress by the replication factor
`MessagesInPerSec`	`kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec`	Messages received per second. Useful when message size varies across topics.	Cross-check against `BytesInPerSec`

Cluster vs broker monitoring: where the line sits

The cluster and broker layers are not always cleanly separable. Several metrics appear in per-broker JMX output but only carry meaning when interpreted at the cluster level.

UnderReplicatedPartitions is the clearest example: it is exposed per broker, but aggregating it across the fleet gives you the total replication deficit. A single broker reporting zero URPs is not informative if another broker is lagging. ActiveControllerCount is only interpretable as a cluster-wide sum — a value of 1 on one broker is normal; a cluster-wide sum of 0 or 2 is critical. Individual broker throughput metrics (BytesInPerSec per broker) are per-broker values that feed cluster-level capacity planning by revealing which brokers are carrying disproportionate load.

For replication and controller metrics, always aggregate. For throughput, collect per-broker and compare across the fleet. The Kafka broker monitoring article covers the internals — request thread saturation, JVM heap, disk flush latency — that complement the cluster view covered here.

Multi-broker observability setup

The collection challenge

JMX is a per-process interface. To get a cluster-wide picture, you need to scrape every broker’s JMX endpoint and aggregate the results centrally. In a three-broker cluster this is manageable manually; in a fleet of dozens, service discovery and automated aggregation become essential.

The most common production approach is Prometheus with the JMX Exporter, because it handles service discovery natively and the aggregation logic lives in PromQL.

Prometheus and JMX Exporter

Run the JMX Exporter as an in-process Java agent on each broker. Running it as an agent rather than a remote poller avoids authentication overhead and reduces JVM thread overhead compared to the remote polling approach:

KAFKA_OPTS="-javaagent:/opt/prometheus/jmx_prometheus_javaagent-0.16.1.jar=7071:/etc/kafka/kafka-jmx-config.yml"

The agent exposes Kafka’s JMX MBeans as Prometheus-formatted metrics on an HTTP endpoint (typically port 7071). Configure Prometheus with a scrape job that discovers all broker endpoints. In Kubernetes, a ServiceMonitor resource handles this automatically; in bare-metal or VM deployments, use static targets in prometheus.yml.

Aggregation across brokers happens in PromQL. To get the cluster-wide under-replicated partition count:

sum(kafka_server_replicamanager_underreplicatedpartitions)

Kafka-specific JMX Exporter YAML configuration files are maintained by the community. The Bitnami and confluentinc examples on GitHub are widely used starting points and include pre-built allowlists for the metrics covered in this article.

Other collection approaches

Datadog: Uses JMXFetch via the Agent daemon. Autodiscovery maps JMX ports to integrations automatically. The default cap of 350 metrics per instance means you will need to configure which metrics are collected carefully for larger clusters.

Kpow (Factor House): Uses the native Kafka Admin Client and Consumer APIs rather than JMX — no sidecar agent or broker-side configuration changes required.

If you’re evaluating monitoring tools, the Kafka observability tools comparison article covers the trade-offs in more detail.

Cluster health check script

A short Python script that checks six cluster-level conditions gives you a pass/fail signal suitable for cron jobs, CI pipelines, or on-call runbooks. The value is in running it consistently on a schedule and routing its output to your alerting channel.

What the script checks

ActiveControllerCount sum equals exactly 1
OfflinePartitionsCount equals 0
UnderReplicatedPartitions sum equals 0
UnderMinIsrPartitionCount equals 0
ISR shrink rate is below a configurable threshold
Leader count distribution — no single broker holds more than a configurable percentage of all leaders

Implementation

The Kafka Admin Client (kafka-python or confluent-kafka) exposes cluster metadata directly without requiring JMX access. Use AdminClient.describe_cluster() to retrieve broker and controller state, and list_topics() with topic metadata to enumerate partition and leader assignments.

UnderReplicatedPartitions and UnderMinIsrPartitionCount are not exposed via the Admin API; for those, query the JMX Exporter HTTP endpoint if Prometheus is already deployed, or fall back to jmxterm.

from kafka import KafkaAdminClient
import requests

BROKERS = ["broker1:9092", "broker2:9092", "broker3:9092"]
JMX_EXPORTER_HOSTS = ["broker1:7071", "broker2:7071", "broker3:7071"]
LEADER_SKEW_THRESHOLD = 0.20


def check_active_controller(admin):
    cluster_meta = admin.describe_cluster()
    controller = cluster_meta.controller
    if controller is None:
        return False, "No active controller"
    return True, f"Controller: broker {controller.id}"


def check_urp_from_jmx(hosts):
    total_urp = 0
    for host in hosts:
        try:
            r = requests.get(f"http://{host}/metrics", timeout=5)
            for line in r.text.splitlines():
                if (
                    "kafka_server_replicamanager_underreplicatedpartitions" in line
                    and not line.startswith("#")
                ):
                    total_urp += float(line.split()[-1])
        except Exception as e:
            print(f"  Warning: could not reach {host}: {e}")
    return total_urp


def check_leader_skew(admin):
    topics = admin.list_topics()
    leader_counts = {}
    for topic_metadata in topics.topics.values():
        for partition in topic_metadata.partitions.values():
            leader = partition.leader
            leader_counts[leader] = leader_counts.get(leader, 0) + 1
    if not leader_counts:
        return True, "No partitions"
    counts = list(leader_counts.values())
    skew = (max(counts) - min(counts)) / max(counts)
    ok = skew <= LEADER_SKEW_THRESHOLD
    return ok, f"Leader skew: {skew:.1%} (threshold {LEADER_SKEW_THRESHOLD:.0%})"


def run_health_check():
    admin = KafkaAdminClient(bootstrap_servers=BROKERS)
    results = []

    ok, msg = check_active_controller(admin)
    results.append(("ActiveControllerCount", ok, msg))

    urp = check_urp_from_jmx(JMX_EXPORTER_HOSTS)
    results.append(("UnderReplicatedPartitions", urp == 0, f"URP count: {int(urp)}"))

    ok, msg = check_leader_skew(admin)
    results.append(("LeaderSkew", ok, msg))

    admin.close()

    print("Kafka cluster health check")
    all_ok = True
    for check, passed, detail in results:
        status = "PASS" if passed else "FAIL"
        print(f"  [{status}] {check}: {detail}")
        if not passed:
            all_ok = False

    return all_ok


if __name__ == "__main__":
    import sys
    sys.exit(0 if run_health_check() else 1)

Limitations

The script reflects point-in-time state. A transient ISR shrink during a rolling restart will register as a failure unless you add a wait-and-recheck delay for replication metrics.
It does not replace continuous time-series monitoring — it is a runbook tool, not a substitute for alerting.
JVM and OS metrics are not covered here; those are addressed in the broker monitoring article.

The consumer monitoring article contains a consumer lag script that complements this one. Together they cover the three main health dimensions: cluster-wide replication and control plane state (this article), per-broker internals (broker monitoring), and consumer group lag (consumer monitoring).

Capacity planning using cluster metrics

Capacity planning at the cluster level is about detecting trends early enough to act before they become incidents. The signals below are leading indicators — most warrant investigation and planning rather than immediate paging.

When to add brokers

The primary signals that a new broker is needed:

BytesInPerSec per broker is consistently above 70% of the broker’s network or disk write capacity. A useful rule of thumb is to size for 3x peak traffic to allow for replication overhead and burst headroom. Replication multiplies egress by the replication factor: a cluster with a replication factor of 3 and 1 Gbps of ingest generates roughly 3 Gbps of outbound replication traffic.
RequestHandlerAvgIdlePercent is below 20% on multiple brokers over a sustained period, indicating that the request handler thread pool is saturated. This is typically caused by slow disk flushes, high JVM GC pauses, or high concurrent replication load from lagging followers.
Disk utilisation is growing at a rate that will exhaust available space within your retention window, with no further compression or retention policy levers available.
PartitionCount per broker is becoming significantly uneven after broker additions or failures — partition reassignment may be needed before adding a new broker.

Partition count planning

Partitions are the unit of parallelism in Kafka. Too few limits throughput; too many increases controller overhead and replication cost.

Track total partition count and partition count per broker. The controller’s EventQueueSize rising during periods of high topic creation activity is a signal that the cluster is approaching the limits of what the controller can manage comfortably.

For older Kafka versions (pre-2.6), a commonly cited practical limit is approximately 4,000 partitions per broker, though this varies considerably with hardware and workload. With KRaft, the practical limit is significantly higher — the consolidated metadata management architecture removes the ZooKeeper bottleneck that was the primary constraint in earlier versions. Consult the release notes and benchmark reports for the specific version you are running.

Replication factor auditing

Topics created with replication.factor=1 in a multi-broker cluster represent a silent durability risk. A single broker failure takes those partitions offline with no replication fallback.

Audit replication factors on a schedule using kafka-topics.sh --describe or the Admin API:

from kafka import KafkaAdminClient

admin = KafkaAdminClient(bootstrap_servers=["broker1:9092"])
topic_names = list(admin.list_topics())
topics = admin.describe_topics(topic_names)

for t in topics:
    for partition in t.partitions.values():
        if len(partition.replicas) < 3:
            print(f"{t.topic} partition {partition.partition}: "
                  f"replication factor {len(partition.replicas)}")

admin.close()

Retention and storage forecasting

Monitor disk utilisation per broker and extrapolate the growth rate from the past 7 and 30 days. BytesInPerSec is the primary driver of storage growth once you account for the replication factor and compression codec.

If storage is growing faster than expected, the first lever to check is log retention settings — both time-based (retention.ms) and size-based (retention.bytes). Reducing retention decreases storage pressure but can break consumers that have fallen behind. Document the trade-off before changing retention on production topics.

Alerting strategy for cluster-level monitoring

Not all metrics warrant the same response. Structuring alerts into two tiers avoids alert fatigue and ensures the right response time for each condition.

Critical alerts (page on-call immediately)

Metric	Condition	Reason
`ActiveControllerCount` (sum)	Not exactly 1	No controller or split-brain — metadata operations are halted
`OfflinePartitionsCount`	> 0	Partitions are completely unavailable to producers and consumers
`UnderMinIsrPartitionCount`	> 0	Producers with `acks=all` are receiving errors
`UnderReplicatedPartitions`	> 0 for > 5 minutes	Sustained replication deficit — data durability is degraded
`UncleanLeaderElectionsPerSec`	> 0	A non-ISR replica was promoted to leader, indicating data loss risk

Working Prometheus alert rules for the two most critical conditions:

groups:
  - name: kafka_cluster_alerts
    rules:
      - alert: KafkaOfflinePartitions
        expr: kafka_controller_offline_partitions_count > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Kafka offline partitions detected"
          description: "{{ $value }} offline partitions on {{ $labels.instance }}. Read and write requests are blocked."
          runbook_url: "https://wiki.internal/runbooks/kafka-offline-partitions"

      - alert: KafkaUnderReplicatedPartitions
        expr: sum(kafka_server_replicamanager_underreplicatedpartitions) > 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Kafka under-replicated partitions"
          description: "{{ $value }} under-replicated partitions across the cluster for more than 5 minutes."
          runbook_url: "https://wiki.internal/runbooks/kafka-under-replicated"

Warning alerts (alert team channel, investigate within the hour)

Metric	Condition	Reason
`IsrShrinksPerSec`	Sustained above zero	Replicas are falling behind; investigate before it becomes a URP
`LeaderCount` skew	> 20% spread across brokers	Producer and consumer traffic is unevenly distributed
`BytesInPerSec` per broker	> 70% of capacity	Approaching saturation on a burst
`EventQueueSize`	> 100	Controller is backlogged; administrative operations will be slow

Alert inhibition

When a broker fails, it typically triggers a cascade: a critical broker-down alert alongside secondary consumer lag and replication warnings. Alertmanager’s inhibition rules let you suppress those secondary warnings while the root-cause alert is active, keeping the on-call view clean:

inhibit_rules:
  - source_match:
      alertname: 'KafkaOfflinePartitions'
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['cluster', 'instance']

Alert fatigue note: ISR shrinks are expected during rolling restarts and broker upgrades. Consider suppressing IsrShrinksPerSec warnings during maintenance windows, or requiring a minimum duration of 10 minutes before the alert fires.

Common Kafka cluster-level issues and how to resolve them

Symptom	Likely metrics	Root cause	Remediation
Producers receiving `NotEnoughReplicasException`	`UnderMinIsrPartitionCount > 0`, `IsrShrinksPerSec` elevated	One or more brokers are offline or severely lagging, shrinking the ISR below `min.insync.replicas`	Investigate the lagging broker: check disk, network, and JVM GC metrics. Restore the broker, or reduce `min.insync.replicas` temporarily as a last resort
Cluster metadata operations timing out	`EventQueueSize > 100`, `LeaderElectionRateAndTimeMs` elevated	Controller is overloaded — too many concurrent topic operations, or a large total partition count	Reduce concurrent topic creation or deletion operations; review total partition count; consider a controller bounce if the queue does not drain
No active controller	`ActiveControllerCount` sum = 0	All brokers have lost controller state — typically a ZooKeeper quorum failure (ZK mode) or KRaft quorum loss	In ZK mode: investigate ZooKeeper ensemble health. In KRaft mode: check the metadata log and quorum state on controller-eligible brokers
Uneven consumer lag across partitions	`LeaderCount` skew, `BytesInPerSec` per broker uneven	Partition leader imbalance — one broker holds a disproportionate share of leaders	Run preferred leader election (`kafka-leader-election.sh --type preferred`) or reassign partitions
Disk filling faster than expected	`BytesInPerSec` elevated, disk utilisation trending up	High produce rate, long retention, or insufficient compression	Review retention settings, check compression codec, evaluate whether a broker addition is needed
ISR instability (repeated shrinks and expands)	`IsrShrinksPerSec` and `IsrExpandsPerSec` both elevated	Network instability between brokers, or JVM GC pauses causing follower replicas to miss heartbeats	Check network error rates between brokers; review GC pause durations on the lagging broker; consider increasing `replica.lag.time.max.ms` if GC pauses are a one-time event

Best practices for Kafka cluster monitoring

Monitor ActiveControllerCount as a cluster-wide sum, not per broker. A per-broker reading of 1 is normal; only the sum tells you whether the cluster has exactly one controller.
Treat UnderReplicatedPartitions as a cluster-level aggregate. Sum it across all brokers. A URP count of zero on one broker says nothing if another broker is lagging.
Separate replication traffic from consumer traffic in your throughput metrics. If BytesOutPerSec includes both, you cannot tell whether a spike is from a consumer or from replication catch-up.
Set alert durations on replication metrics, not just thresholds. An ISR shrink during a rolling restart is expected; one that persists for five minutes is not.
Audit replication factors on a schedule. Topics created with non-standard replication factors are common after high-velocity development cycles and represent silent durability risk.
In KRaft mode, add LastAppliedRecordLagMs and MetadataErrorCount to your standard cluster health dashboard. These have no ZooKeeper equivalents and are easy to overlook when migrating an existing monitoring setup.
Run a cluster health check script on a cron schedule — every five minutes works well — and route its output to your alerting channel. It catches conditions that continuous metric alerting can miss during gaps in scrape coverage.

Monitor Kafka clusters with Kpow

Kpow connects to any Kafka cluster using the native Admin Client and Consumer APIs — no JMX Exporter, no sidecar agent, and no broker-side configuration changes required. It surfaces the cluster-level metrics covered in this article — under-replicated partitions, controller state, partition distribution, throughput per broker — on a single dashboard, with replication health and partition distribution views that update in real time.

You can give Kpow a try with a free 30-day trial. Connect it to any Kafka cluster in minutes and deploy via Docker, Helm, or JAR.

Kafka consumer monitoring and performance tuning

Factor House — Thu, 04 Jun 2026 00:00:00 GMT

Key takeaways

Consumer lag is the primary indicator of pipeline health, but it tells you only that a consumer has fallen behind, not why
Poll idle ratio and rebalance rate reveal problems that lag alone misses
Effective performance tuning requires matching the right metric to the right configuration parameter
JMX gives you per-client visibility; tools that query broker metadata directly fill the gaps that client-side instrumentation leaves

What is Kafka consumer monitoring?

Kafka consumer monitoring is the practice of collecting and interpreting metrics that describe how consumer applications are reading from Kafka topics. It sits within Kafka monitoring as a broader discipline, which also covers broker health, topic throughput, and replication state.

Consumer monitoring is distinct from broker monitoring in one important way: the metrics you care about are generated on the client side. Kafka consumers expose instrumentation through Java Management Extensions (JMX), which means visibility depends on what the consumer JVM process exposes at runtime. Brokers can be fully operational while consumers fall behind, and broker metrics alone will not surface that.

The most important consumer monitoring metric

Consumer lag

Consumer lag is the difference between the latest offset written to a partition (the log end offset) and the last offset the consumer group has committed. For each partition:

lag = log_end_offset - committed_offset

Lag tells you how many records a consumer group has yet to process. It is the primary indicator of pipeline health because it reflects the gap between what producers are writing and what consumers have actually acknowledged.

The JMX metric records-lag-max (MBean: kafka.consumer:type=consumer-fetch-manager-metrics) reports the maximum lag across all partitions assigned to a single consumer instance. This is a per-instance metric. For a full picture of group-level lag across every partition of a topic, you need tooling that queries the broker directly rather than aggregating what individual clients report about themselves.

One thing worth understanding early: absolute lag values are context-dependent. A lag of 50,000 on a high-throughput analytics topic may represent a few seconds of processing backlog. The same figure on a payment processing topic could represent hours of critical delay. What matters more than the absolute number is whether lag is growing, stable, or draining. More on alerting thresholds below.

For a detailed treatment of consumer lag specifically, including how to instrument it and respond to it, see the article on Kafka consumer lag monitoring.

Other key metrics to monitor

Throughput

records-consumed-rate and bytes-consumed-rate (MBean: kafka.consumer:type=consumer-fetch-manager-metrics) measure how fast the consumer is reading from the broker. These are your baseline throughput indicators.

A consumer may have low lag while still running at a fraction of its expected throughput, which can point to producers writing slowly or to fetch configuration that unnecessarily limits batch size. Tracking throughput alongside lag helps distinguish between a consumer that is healthy and one that is barely keeping up.

Poll idle ratio

poll-idle-ratio-avg (MBean: kafka.consumer:type=consumer-metrics) measures the fraction of time the consumer’s poll loop is idle, waiting for the broker to return records. A value close to 1.0 means the consumer is spending most of its time waiting; a value close to 0 means it is spending almost all of its time processing records.

When poll idle ratio drops consistently toward 0, the consumer’s processing logic is the bottleneck, not the fetch pipeline. In this state, tuning fetch configuration has little effect. The correct response is to reduce per-record processing time, add consumer instances, or increase the topic’s partition count to allow more parallelism.

Error rate

fetch-error-rate (MBean: kafka.consumer:type=consumer-fetch-manager-metrics) counts failed fetch requests per second. A non-zero error rate points to connectivity issues between the consumer and the broker, authentication failures, or broker-side quota violations. On its own, occasional fetch errors may not cause visible lag if the consumer retries successfully. A sustained error rate typically will. Monitoring it alongside fetch latency helps you determine whether errors are causing delays or being absorbed by the retry logic.

Rebalance rate

A consumer group rebalance occurs whenever a consumer joins or leaves the group, or when partitions are reassigned. During a rebalance, all consumption for the affected group pauses. This is usually brief, but rebalances that happen frequently, or that take a long time to complete, cause visible lag spikes.

A rebalance storm occurs when repeated rebalances prevent the group from making meaningful progress between them. Common causes include:

Slow processing loops where the consumer exceeds max.poll.interval.ms before calling poll() again
Overly short session.timeout.ms values that cause the broker to consider a consumer dead during garbage collection pauses
Ungraceful shutdowns during rolling deployments

join-time-avg and sync-time-avg (MBean: kafka.consumer:type=consumer-coordinator-metrics) measure the time taken to complete the join and sync phases of each rebalance. If these values are consistently high, rebalances are expensive when they do occur, which amplifies any instability in group membership.

Commit rate and commit latency

commit-rate and commit-latency-avg (MBean: kafka.consumer:type=consumer-coordinator-metrics) describe how frequently and how quickly offset commits are completing.

Commit latency matters because elevated values signal broker responsiveness issues or network degradation. It also has a direct consequence for correctness: if a consumer crashes before its most recent offsets are committed, it reprocesses messages from the last successfully committed point. Higher commit latency widens that reprocessing window.

Fetch latency

fetch-latency-avg (MBean: kafka.consumer:type=consumer-fetch-manager-metrics) measures the round-trip time for fetch requests to the broker. This metric is useful as a bridge between consumer-side and broker-side observability. When fetch latency is high, the cause is usually upstream: high broker disk I/O, network saturation, or broker-side throttling. Consumer monitoring surfaces the symptom first; broker monitoring tells you the cause.

Partition offsets

Beyond lag, it is worth tracking how your consumer group manages offsets over time. The auto.offset.reset configuration determines where a consumer starts reading when no committed offset exists for a partition. The latest setting means it starts from the newest message, potentially skipping records written before the consumer joined. The earliest setting means it reads from the beginning of the partition’s retained log. Knowing which policy is in effect is important context for interpreting gaps in consumption, particularly after a new deployment or a consumer group reset.

Commit frequency also matters operationally. Committing too infrequently widens the reprocessing window after a crash; committing too frequently increases metadata load on the group coordinator.

Broker and infrastructure metrics

High broker disk I/O, network saturation, or replica lag will surface in consumer monitoring as elevated fetch latency or increased error rates. The relationship is worth understanding: consumers do not operate independently of the brokers they read from. If consumer fetch latency is climbing but all consumer-side metrics look normal, the issue is upstream. For a full treatment of broker-side monitoring, refer to the upcoming Kafka broker monitoring article.

Alerting thresholds

The main risk with consumer lag alerting is relying on static absolute thresholds. A threshold of 10,000 messages means something very different on a topic receiving one million messages per second versus one receiving one hundred. Absolute thresholds in the former case would fire constantly; in the latter, a genuine problem could sit below the threshold for hours.

A more reliable approach is to alert on the rate of change in lag rather than its absolute value. If lag is stable at 50,000 messages, the consumer is processing at roughly the same rate the producer is writing, and the situation may be acceptable. If lag is growing at 500 messages per second and has been doing so for five minutes, the consumer is falling behind and action is warranted. The Prometheus deriv() function is useful for this:

deriv(kafka_consumergroup_lag[10m]) > 500

Complement this with a stall detection alert for cases where the consumer has stopped committing entirely despite having an active backlog:

sum(delta(kafka_consumergroup_current_offset[5m])) by (consumergroup, topic) == 0
and sum(kafka_consumergroup_lag) by (consumergroup, topic) > 1000

A stalled consumer with a growing backlog typically indicates a deadlocked thread, a poison-pill message blocking the processing loop, or a crashed consumer group that has not been restarted.

For partition-level alerting, records-lag-max from JMX lets you alert on the worst-performing partition assigned to a given consumer instance. Group-level tooling that aggregates across all partitions of a topic provides a broader view than any single client can offer, and is particularly useful for identifying partition lag skew, where one partition falls significantly behind the average.

Tooling for collecting and monitoring metrics

JMX and the Prometheus JMX Exporter

By default, Kafka consumer JMX metrics are not remotely accessible. To enable scraping, you configure the JMX Prometheus Java Agent at JVM startup:

export KAFKA_OPTS="-javaagent:/path/to/jmx_prometheus_javaagent.jar=9404:/path/to/kafka-jmx-config.yml"

This exposes the JMX MBeans on a Prometheus-compatible endpoint that you can scrape and visualise in Grafana. The metrics listed in this article all become available through this approach.

JMX-based monitoring has a meaningful limitation: the metrics are produced by the client process itself. If the consumer crashes, the metric stream stops. A dead consumer group looks identical to one that has never reported metrics. You do not automatically get a “consumer is down” signal; you get an absence of data.

For a look at building more effective Kafka dashboards on top of JMX data, see Beyond JMX: supercharging Grafana dashboards with high-fidelity metrics.

External lag exporters

To address the liveness gap in JMX monitoring, external lag exporters query broker metadata directly rather than relying on what each consumer client reports. These tools read the __consumer_offsets internal topic to track committed offsets and evaluate group health without depending on a live consumer JVM. This means the monitoring plane continues to report accurate lag even when a consumer group is offline.

Burrow, the open-source lag monitoring service built at LinkedIn, takes this approach. It evaluates consumer group health over a sliding window of recent commits, classifying groups as OK, WARN, ERR, STALL, or STOP based on commit activity and lag trend, without requiring you to set static thresholds. This distinction matters in practice: a group that is STALL (committing the same offsets repeatedly) is a different problem from one that is STOP (no commits at all), and treating them the same way with a simple lag threshold would miss the difference.

OpenTelemetry

OpenTelemetry support for Kafka consumer metrics is available through the OpenTelemetry Java Agent and community instrumentation libraries. Integration depth varies by Kafka client version, and Prometheus-based JMX scraping remains the more established approach for Kafka-specific observability. If your organisation already operates an OpenTelemetry pipeline, it is worth evaluating whether Kafka consumer metrics can be routed through it consistently, though broker-side visibility will still likely require a dedicated exporter.

Kafka consumer performance tuning

Reducing consumer lag

The first step when lag is growing is to check poll idle ratio. If the value is well above 0, the consumer has capacity to process more records per fetch cycle, and the bottleneck is likely in how data is being fetched rather than how it is being processed.

To increase fetch efficiency:

Raise fetch.min.bytes (default: 1 byte) to instruct the broker to wait until it has a meaningful batch of data before responding. This reduces fetch request frequency and improves throughput at the cost of slightly higher latency per fetch.
Raise fetch.max.wait.ms (default: 500ms) to control how long the broker waits to accumulate fetch.min.bytes worth of data. Raising this allows the broker to return larger batches in each response.
Increase max.poll.records to allow the consumer to process more records per poll loop iteration. If you do this, confirm that your batch processing time stays within max.poll.interval.ms; exceeding it triggers a rebalance.

If poll idle ratio is near 0, the bottleneck is in processing logic, not in fetching. Tuning fetch parameters will have little effect. The options are to reduce per-record processing time (for example, by batching downstream writes or switching to asynchronous I/O), to add consumer instances up to the partition count, or to increase the topic’s partition count to raise the parallelism ceiling.

Reducing rebalance frequency

The session.timeout.ms and heartbeat.interval.ms configuration pair is worth understanding precisely. The session timeout is the window within which the consumer must send at least one heartbeat before the group coordinator declares it dead. If the poll loop takes too long due to slow message processing or a GC pause, the heartbeat thread may not run within that window, and the consumer will be evicted from the group.

A configuration that provides more margin for transient pauses:

session.timeout.ms=45000 gives the consumer 45 seconds to heartbeat before being considered dead
heartbeat.interval.ms=15000 sends heartbeats every 15 seconds, one-third of the session timeout

max.poll.interval.ms is separate from the session timeout. It defines the maximum allowed time between successive poll() calls. If processing a batch takes longer than this value, the consumer will be removed from the group regardless of whether its heartbeats are current. If your processing is legitimately slow, increase this value to match expected batch processing time rather than masking the issue by reducing batch size.

For deployments on Kafka 2.4 and later, switching to CooperativeStickyAssignor (partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor) means that during a rebalance, only the partitions being migrated are paused. The rest of the group continues consuming. This significantly reduces the impact of rebalances caused by rolling restarts or transient consumer joins.

Improving throughput

Consumer parallelism is bounded by partition count. You cannot have more active consumer instances in a group than there are partitions; additional consumers will sit idle. If the throughput ceiling for a consumer group is a concern, increasing partition count on the topic is the only way to raise the maximum parallelism available to you.

For bulk throughput, fetch.max.bytes (default: 50MB) and max.partition.fetch.bytes (default: 1MB) control how much data the consumer requests per fetch. Increasing max.partition.fetch.bytes is relevant when topics carry large messages, since the default can limit how many records come back in each fetch response. Be aware that larger fetch sizes increase memory pressure on the consumer, as fetched records are buffered before processing begins.

Monitor more effectively with Kpow

Kpow provides group-level lag visibility across all partitions of a topic, partition-level drill-down for isolating slow consumers, and configurable alerting without building and maintaining a custom Prometheus pipeline. You can connect it to any Kafka cluster and start monitoring consumer groups in minutes.

Give it a try with a free 30-day trial. You can deploy via Docker, Helm, or JAR.

Kafka monitoring: a complete guide for platform engineers

Factor House — Thu, 04 Jun 2026 00:00:00 GMT

Apache Kafka is designed to be fast and fault-tolerant, but those properties only hold when you can see what is happening inside the cluster. Without instrumentation, a broker disk filling up, a follower falling behind, or a consumer group stalling can each go unnoticed until the damage is done.

This guide covers every layer of the Kafka monitoring surface: brokers, replication, consumers, producers, JVM, storage, and tooling. It is written for platform engineers and data engineers who are responsible for running Kafka in production and who need concrete, actionable guidance rather than a catalogue of metric names.

What is Kafka monitoring?

Kafka monitoring is the practice of collecting, observing, and alerting on the operational signals that describe the health and performance of a running Kafka cluster. The monitoring surface spans three layers:

JMX metrics exposed by the broker and client JVMs through Java Management Extensions, covering everything from replication state to request handler utilisation
OS and infrastructure metrics that describe the host resources Kafka depends on: disk I/O, network throughput, file descriptors, and page cache pressure
Application-level metrics that measure the end-to-end impact of Kafka on the business: consumer lag as a time delay, producer error rates, and pipeline latency

Metrics are exposed via two separate registries. Server-side internals use the Yammer Metrics library; Java-based clients use the Kafka Metrics registry. Both expose their telemetry through JMX, making performance data available to external systems such as Prometheus exporters, Datadog, and Dynatrace.

By default, remote JMX access is disabled to prevent unauthenticated access to the JVM. To enable it, set the JMX_PORT environment variable in your broker startup script, or configure the com.sun.management.jmxremote.* system properties explicitly. You can verify metric exposure with Kafka’s built-in kafka-run-class.sh kafka.tools.JmxTool.

KRaft and ZooKeeper

ZooKeeper was deprecated in Kafka 3.x and removed entirely in Kafka 4.0. Modern clusters run in KRaft mode, where metadata is managed via a Raft-based consensus protocol inside the broker JVM itself. This architectural shift changes what you monitor: ZooKeeper session health, ZkClient/SessionExpiredPerSec, and the ControlPlaneNetworkProcessorAvgIdlePercent metric are no longer present. In their place, KRaft introduces consensus metrics under kafka.server:type=raft-metrics and a new set of metadata loader and controller queue metrics. The relevant KRaft-specific sections are called out throughout this guide.

Why Kafka monitoring matters

Kafka is designed to be resilient, but specific failure modes are silent until they cause data loss or downstream outages.

Replication lag becomes data loss. When a follower falls out of the In-Sync Replicas (ISR) set, the cluster’s effective replication factor drops. If a second broker fails before the first recovers, you can lose data permanently. This happens in minutes; without an alert on UnderReplicatedPartitions, you will not know until a producer starts receiving NotEnoughReplicasException.

Consumer lag is an invisible SLA breach. A consumer group can be running, committing offsets, and processing records while accumulating a lag of millions of messages behind the producer. The application looks healthy. Downstream dashboards, notifications, and fraud checks are operating on data that is hours old.

Broker failure cascades. A single slow broker can trigger ISR shrinks across every partition it leads, which can then cause leader elections and controller re-elections if the GC pause lasts long enough. What starts as a noisy garbage collection cycle on one node can propagate into a cluster-wide availability event within a minute.

The business cost is concrete. A post-incident analysis of a production Kubernetes-hosted Kafka failure documented $240,000 in SLA penalties over 62 minutes, driven by a cascading broker crash that pushed consumer lag to 14 hours across 10,000 topics.

The 10 most critical Kafka metrics

The table below is the minimum monitoring footprint for any production Kafka cluster. Every metric here has caused production incidents when left unmonitored.

Metric	JMX MBean path	What it measures	Healthy range	Alert threshold
UnderReplicatedPartitions	kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions	Partitions whose current ISR count is smaller than the replication factor	0	Page if > 0 sustained for 5 minutes
OfflinePartitionsCount	kafka.controller:type=KafkaController,name=OfflinePartitionsCount	Partitions with no active leader — reads and writes fail immediately	0	Page immediately if > 0
ActiveControllerCount	kafka.controller:type=KafkaController,name=ActiveControllerCount	Whether this broker is the active controller; sum across cluster must equal exactly 1	1 on leader, 0 on all others	Page if cluster sum is not 1 for 5 minutes
records-lag-max	kafka.consumer:type=consumer-fetch-manager-metrics,client-id=(…),name=records-lag-max	Maximum offset lag across all partitions assigned to this consumer	Topic-dependent	Alert on rate of growth, not absolute value — see consumer section
RequestHandlerAvgIdlePercent	kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent	Fraction of time I/O request handler threads are idle	> 0.6 (60%)	Warning at < 0.3; critical at < 0.2
NetworkProcessorAvgIdlePercent	kafka.network:type=SocketServer,name=NetworkProcessorAvgIdlePercent	Fraction of time network processor threads are idle	> 0.3 (30%)	Warning if sustained < 0.3
BytesInPerSec / BytesOutPerSec	kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec / BytesOutPerSec	Total inbound and outbound byte throughput	Varies by cluster	Alert if approaching NIC saturation; track trend not absolute
GC pause duration	java.lang:type=GarbageCollector,name=G1 Young Generation (or ZGC equivalent)	Stop-the-world pause time in milliseconds	< 200ms (G1GC), < 5ms (ZGC)	Warning at > 200ms P99; critical at > 500ms P99
Disk usage %	OS metric (or kafka.log:type=Log,name=Size,topic=(…),partition=(…))	Disk capacity consumed relative to total; Kafka fills disks faster than expected due to replication	< 70%	Warning at > 70%; critical at > 85%
IsrShrinksPerSec	kafka.server:type=ReplicaManager,name=IsrShrinksPerSec	Rate at which replicas are being dropped from the ISR	0	Warn on any sustained positive rate; page if growing rapidly

Kafka monitoring best practices

1. Instrument with JMX exporter and Prometheus from day one

Retrofitting observability onto a running Kafka cluster is painful. The Prometheus JMX Exporter runs as a Java agent inside the broker JVM, translating nested MBean attributes into flat, label-enriched Prometheus series. Configuring it requires a static YAML rule file and a JVM flag at startup:

export KAFKA_OPTS="$KAFKA_OPTS -javaagent:./jmx_prometheus_javaagent.jar=8080:./kafka-metrics.yml"

Consumer group lag is not exposed through JMX directly; it requires a separate kafka-exporter that polls the AdminClient API and calculates the delta between log end offsets and committed consumer offsets. Deploy both from the start.

2. Alert on `UnderReplicatedPartitions > 0` without delay

A non-zero under-replicated partition count means your replication factor is not being honoured. The cluster is operating with reduced durability. If one more broker fails before the ISR recovers, you can permanently lose data. There is no steady-state condition that produces a non-zero reading in a healthy cluster, so any positive value should trigger an immediate alert and investigation.

Pay equal attention to the ISR shrink rate (IsrShrinksPerSec). Many teams watch only for offline partitions, which means silent degradation: the ISR can shrink from 3 to 2 to 1 over hours without triggering an alert, only surfacing as a critical incident when the last replica fails.

3. Track consumer lag as a rate of change, not just an absolute value

A lag of 50,000 messages on a topic that processes 100,000 messages per second represents 500ms of delay. On a topic that processes 10 messages per second, it represents more than 80 minutes. An absolute offset threshold is meaningless without context.

Implement time-based lag thresholds wherever possible. A practical formula for topics where time-based monitoring is not directly available:

Critical threshold (in messages) = target_SLO_seconds × consumption_rate_per_second × (1 + safety_margin)

For a payment processor consuming 100 messages per second with a 120-second SLO and a 20% safety margin: 120 × 100 × 1.2 = 14,400 messages.

Alert on a growing lag trend before the lag itself becomes critical. LinkedIn’s open-source Burrow evaluates lag over a sliding window of offset commits rather than at an absolute threshold, categorising consumer group health as OK, WARNING, ERROR, or STALL, catching stalled consumers even when their lag is temporarily low.

4. Set JVM heap and GC budgets before tuning anything else

Kafka brokers run on the JVM. A garbage collection pause on a follower broker suspends its ReplicaFetcherThread. If that pause exceeds replica.lag.time.max.ms (default 30 seconds), the partition leader evicts the follower from the ISR. The resulting ISR shrink can cascade into write failures for producers using acks=all if the ISR drops below min.insync.replicas.

For brokers with heap sizes under 16GB, G1GC with -XX:MaxGCPauseMillis=20 and -XX:InitiatingHeapOccupancyPercent=35 is a sensible baseline. Set -XX:G1HeapRegionSize=16M to prevent large message batches from being classified as humongous objects. For large heaps (32GB+), Generational ZGC (-XX:+UseZGC -XX:+ZGenerational) delivers sub-millisecond pause times at the cost of roughly 10-20% more CPU.

The key monitoring signal: a GC pause exceeding 200ms P99 should trigger a warning. A pause exceeding 500ms P99 should trigger a page.

5. Distinguish broker-side lag from consumer-side lag before taking action

When consumer lag is growing, the cause is either upstream (the producer is writing faster than the consumer can keep up) or downstream (the consumer is processing too slowly). These have different remediation paths.

Start by comparing BytesInPerSec (producer throughput) against consumer polling rate. If throughput has spiked, add partitions and consumer instances. If throughput is flat but lag is growing, look at consumer processing time: check records-consumed-rate versus records-lag-max, and look for slow synchronous I/O inside the poll loop, such as database writes, external HTTP calls, or CPU-bound transformations. These are the most common source of consumer lag in practice.

Also check for JVM GC pauses on the consumer side. A consumer GC pause exceeding session.timeout.ms (default 45 seconds) causes the broker to evict the consumer and trigger a rebalance, which itself pauses consumption and accelerates lag accumulation.

6. Use per-topic and per-partition views, not just cluster-level aggregates

Cluster-level aggregates mask partition-level problems. A single hot partition receiving 80% of traffic, or a single topic with a replication.factor of 1 in a three-broker cluster, will not show up in aggregate health dashboards.

Monitor LeaderCount and PartitionCount per broker to detect partition imbalance. An uneven leader distribution means some brokers are handling a disproportionate share of read and write operations. Use kafka-leader-election.sh for preferred leader elections when imbalance is detected.

For consumer lag, alert at the consumer group + topic + partition level, not just at the group level. Partition hot-spotting, where a single partition receives a disproportionate share of traffic due to key skew, concentrates lag on one consumer thread while others are idle.

7. Test your alerting pipeline regularly

An alert that fires but never reaches the on-call engineer is operationally equivalent to having no alert at all. Run periodic drills where you deliberately trigger alert conditions in a non-production environment and verify end-to-end delivery: metric fires, alert routes, notification delivers, runbook is accessible.

For KRaft clusters, also verify that your monitoring tooling handles the new metric namespaces. Several tools still default to ZooKeeper-era MBean paths. Metrics like ControlPlaneNetworkProcessorAvgIdlePercent no longer exist in Kafka 4.0 clusters; dashboards using them will show no data without erroring, creating silent gaps.

Cluster and broker health monitoring

Broker health metrics

A healthy broker has:

ActiveControllerCount equal to 1 (on exactly one broker in the cluster)
OfflinePartitionsCount equal to 0
UnderReplicatedPartitions equal to 0
Request handler and network processor idle percentages above 30%

The request processing model uses two thread pools. Network processor threads handle socket connections and read incoming data frames. Request handler (I/O) threads dequeue those frames and execute the actual disk reads and writes. The two idle percentage metrics reflect the load on each pool.

When NetworkProcessorAvgIdlePercent falls below 0.3, network threads are saturated, likely from a connection storm or a sudden spike in short-lived connections. When RequestHandlerAvgIdlePercent falls below 0.2, the I/O subsystem is the bottleneck, which typically means disk write pressure or GC-induced slowdowns.

Total request latency breaks down into:

RequestQueueTimeMs — time waiting in queue for a handler thread
LocalTimeMs — active processing on the partition leader
RemoteTimeMs — waiting for follower acknowledgment (when acks=all)
ResponseSendTimeMs — serialising and writing the response to the socket

A spike in RemoteTimeMs points to follower lag or network issues between brokers. A spike in LocalTimeMs points to disk pressure on the leader.

Other key broker metrics:

Metric	JMX MBean path	Alert note
LeaderElectionRateAndTimeMs	kafka.controller:type=ControllerStats,name=LeaderElectionRateAndTimeMs	Any election is notable; frequent elections indicate instability
UncleanLeaderElectionsPerSec	kafka.controller:type=ControllerStats,name=UncleanLeaderElectionsPerSec	Any non-zero value means data loss has occurred — page immediately
RequestQueueSize	kafka.network:type=RequestChannel,name=RequestQueueSize	Alert if sustained > 10
PurgatorySize (Produce)	kafka.server:type=DelayedOperationPurgatory,name=PurgatorySize,delayedOperation=Produce	Growing size indicates slow followers or disk saturation

JVM and GC monitoring

Kafka brokers are JVM processes. GC pause behaviour is not a background concern: it directly causes ISR shrinks and consumer lag spikes.

The mechanics are straightforward. When a follower broker experiences a stop-the-world GC pause, its ReplicaFetcherThread is suspended. If the pause exceeds replica.lag.time.max.ms (default 30 seconds), the partition leader evaluates the replica as a “slow follower” using:

val laggingReplicas = candidateReplicas.filter(r =>
  (time.milliseconds - r.lastCaughtUpTimeMs) > maxLagMs)

The leader then evicts the paused follower from the ISR, raising IsrShrinksPerSec and driving UnderReplicatedPartitions above zero. If min.insync.replicas is not met after the eviction, producers with acks=all receive NotEnoughReplicasException.

The same dynamic affects consumers. A consumer JVM GC pause that exceeds session.timeout.ms (default 45 seconds) causes the Group Coordinator to evict the consumer and trigger a rebalance. All consumers in the group pause during the rebalance, accumulating lag.

Key JVM metrics to monitor:

Metric	What to look for
GC pause duration (P99)	Warning at > 200ms; critical at > 500ms
GC frequency	Warning if major GC is occurring more than once every 30 seconds
Heap utilisation post-GC	Warning at > 70% of max heap; critical at > 85%

G1GC vs ZGC at a glance: G1GC is the sensible default for heaps under 16GB. Its main risk is humongous object allocation: any object over 50% of the G1 region size is allocated directly into the Old Generation, bypassing Young Generation copy collection and causing heap fragmentation. Setting -XX:G1HeapRegionSize=16M raises this threshold to 8MB, which covers most Kafka message batches. Generational ZGC (available in JDK 21+) delivers sub-millisecond pause times by running all collection phases concurrently with the application, but requires at least 8 vCPUs and 32GB of heap to run stably, and consumes 10-20% more CPU than G1GC. For containerised deployments with cgroup CPU limits below 4 vCPUs, G1GC is the safer choice.

KRaft metadata monitoring

Kafka 4.0 removes ZooKeeper entirely. In KRaft mode, a subset of brokers act as controllers, maintaining the cluster metadata log (__cluster_metadata) via the Raft consensus protocol.

What ZooKeeper monitoring you can remove:

zookeeper.connect, zookeeper.session.timeout.ms, ZkClient/SessionExpiredPerSec
ControlPlaneNetworkProcessorAvgIdlePercent (replaced by the unified NetworkProcessorAvgIdlePercent)
ZooKeeper ruok port-2181 health checks

What to add for KRaft:

Metric	JMX MBean path	What it means
current-state	kafka.server:type=raft-metrics,name=current-state	Node’s role: leader, follower, candidate, or observer. Sustained “candidate” indicates election instability
high-watermark	kafka.server:type=raft-metrics,name=high-watermark	Highest committed offset in the metadata log. Primary reference for consensus progress
log-end-offset	kafka.server:type=raft-metrics,name=log-end-offset	Latest offset written to local disk. Lag vs. high-watermark reveals replication delay
commit-latency-avg	kafka.server:type=raft-metrics,name=commit-latency-avg	Average time to commit a metadata operation. Spikes indicate disk write bottlenecks on the controller
MetadataLoaderIdleRatio	kafka.server:type=MetadataLoader,name=MetadataLoaderIdleRatio	Fraction of time the metadata loader thread is waiting. Near 0.0 means thread saturation
metadata-apply-error-count	kafka.server:type=broker-metadata-metrics,name=metadata-apply-error-count	Failed metadata operations — must always be 0
EventQueueSize	kafka.controller:type=ControllerEventManager,name=EventQueueSize	Pending administrative tasks. Alert if > 100
TimedOutBrokerHeartbeatCount	kafka.controller:type=KafkaController,name=TimedOutBrokerHeartbeatCount	Broker heartbeat timeouts. A growing rate indicates network saturation or broker GC issues

The broker registration state (kafka.controller:type=KafkaController,name=BrokerRegistrationState,broker=X) has three states: Fenced (not participating in leadership), Controlled Shutdown (graceful exit), and Active (fully operational). Fenced brokers do not serve client traffic.

For manual diagnostics, use kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status to inspect active leader, current epoch, high watermark, and voter node IDs. This replaces zookeeper-shell.sh for quorum health checks.

See Kafka cluster monitoring and Kafka broker monitoring for more.

Topic and partition health monitoring

Topic-level metrics reveal problems that broker-level aggregates hide.

Key metrics:

Metric	JMX MBean path	Notes
MessagesInPerSec	kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec,topic=(…)	Per-topic message ingestion rate
BytesInPerSec / BytesOutPerSec	kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec,topic=(…)	Per-topic byte throughput
UnderReplicatedPartitions per topic	kafka.cluster:type=Partition,topic=(…),name=UnderReplicated,partition=(…)	Partition-level under-replication
LogEndOffset	kafka.log:type=Log,name=LogEndOffset,topic=(…),partition=(…)	Latest written offset per partition

Partition imbalance is detected by comparing LeaderCount across brokers. In a healthy cluster, partition leadership is distributed proportionally. If one broker holds 60% of leaders while others hold 20% each, read and write traffic is concentrated on one node, driving up its request handler utilisation while others sit idle.

A spike in MessagesInPerSec that is not accompanied by a proportional increase in BytesInPerSec can indicate a change in average message size, which can affect whether messages are classified as humongous objects in the G1GC allocator.

Consumer lag monitoring

Consumer lag is the most operationally significant Kafka metric for most teams. It is the direct measurement of how far behind downstream processing is from real-time.

The formula is simple:

Consumer lag = Log End Offset (partition) - Consumer Committed Offset (partition)

The complexity lies in what that number means. A consumer group may be processing 9,000 messages per second while an upstream producer pushes 10,000. The consumer is running. Throughput dashboards show it as healthy. But it is accumulating 1,000 messages of lag every second.

Consumer lag patterns

Lag does not accumulate uniformly. Different accumulation patterns indicate different root causes:

Pattern	Behaviour	Typical cause
Gradual drift	Slow linear climb over hours	Minor throughput mismatch; application logic slower than required
Sudden spike	Near-vertical jump	Consumer crash, network partition, upstream traffic burst
Flat high lag	Large stable gap, neither growing nor shrinking	Consumer at capacity, capped by partition count or resource limits
Sawtooth	Cyclical rise and fall	Batch processing, JVM GC pauses, database commit throttling

What causes consumer lag in practice

Slow processing inside the poll loop. Synchronous database writes, blocking HTTP calls, and CPU-bound transformations inside the record processing loop are the most frequent cause. Each blocked thread cannot call poll(), causing the broker to eventually view the consumer as dead and trigger a rebalance.

JVM GC pauses. A consumer GC pause exceeding session.timeout.ms (default 45 seconds) causes the Group Coordinator to evict the consumer and trigger a full group rebalance. All consumers pause during rebalance. The recovered consumer then faces a large backlog, which can exceed max.poll.interval.ms if processing is slow, triggering another rebalance.

Poison pill records. A single malformed record that a consumer cannot process without throwing an exception will cause that consumer to crash-restart in an infinite loop against the same offset. Lag accumulates linearly on that partition while the consumer cycles. Every record queued behind the bad offset is blocked.

Partition hot-spotting. When a business key is highly concentrated (for example, a major tenant ID), one partition receives a disproportionate share of traffic. One consumer thread is at 100% CPU while the rest are idle. Lag accumulates on the hot partition while aggregate lag metrics look manageable.

Transactional producer stalls. Consumers in read_committed isolation mode will not receive messages from open, uncommitted transactions. If a transactional producer crashes with an open transaction, downstream consumers stall until the transaction coordinator times out the transaction, which defaults to transaction.timeout.ms = 60000 (60 seconds).

Consumer lag monitoring approach

Use time-based thresholds where possible, with different SLOs per topic type:

Fraud detection:       Warning at 10s lag  |  Critical at 30s
Payment processor:     Warning at 30s lag  |  Critical at 2m
Analytics ETL:         Warning at 10m lag  |  Critical at 30m

For offset-based thresholds where time-based metrics are unavailable:

Critical threshold = target_SLO_seconds × consumption_rate_per_second × (1 + safety_margin)

For a payment processor consuming 100 messages/second with a 120-second SLO and a 20% safety margin: 120 × 100 × 1.2 = 14,400 messages.

Alert on the rate of change of lag, not just its current value. A lag that is 5,000 messages and growing at 1,000 messages per minute needs attention before it crosses any absolute threshold.

See Kafka consumer monitoring for more.

Producer monitoring

Kafka producers expose client-side metrics under kafka.producer:type=producer-metrics,client-id=(...).

Key producer metrics:

Metric	JMX attribute	What to watch
record-send-rate	record-send-rate	Average records sent per second; a drop indicates backpressure
record-error-rate	record-error-rate	Any sustained non-zero value requires investigation
request-latency-avg	request-latency-avg	Warning at > 100ms avg; critical at > 500ms P99
buffer-available-bytes	buffer-available-bytes	Alert when approaching zero — producer is blocked
waiting-threads	waiting-threads	Number of application threads blocked on buffer exhaustion
produce-throttle-time-max	produce-throttle-time-max	Non-zero means the broker is rate-limiting this producer

The failure cascade for a saturated producer works as follows: the broker enforces a quota and delays its response to the producer. The producer retains batches longer, buffer-available-bytes falls. Once the buffer is exhausted, application threads calling send() block, waiting-threads spikes, and the application is effectively paused.

A sudden rise in record-error-rate can indicate network problems, broker-side disk issues, ACL misconfigurations, or the producer hitting max.message.bytes. Check the corresponding broker metric BytesRejectedPerSec (kafka.server:type=BrokerTopicMetrics,name=BytesRejectedPerSec) to determine if the broker is actively rejecting oversized batches.

See Kafka producer monitoring for more.

Replication and data durability monitoring

Replication metrics are the most critical class of Kafka metrics. They describe whether your data is actually protected.

The key metrics:

Metric	JMX MBean path	What a non-zero value means
UnderReplicatedPartitions	kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions	ISR is smaller than replication factor — durability is degraded
UnderMinIsrPartitionCount	kafka.server:type=ReplicaManager,name=UnderMinIsrPartitionCount	ISR has dropped below min.insync.replicas — acks=all writes are failing
IsrShrinksPerSec	kafka.server:type=ReplicaManager,name=IsrShrinksPerSec	Replicas are being dropped from ISR; investigate immediately
IsrExpandsPerSec	kafka.server:type=ReplicaManager,name=IsrExpandsPerSec	Replicas are recovering; should follow a shrink event
UncleanLeaderElectionsPerSec	kafka.controller:type=ControllerStats,name=UncleanLeaderElectionsPerSec	An out-of-sync replica was elected leader — data has been lost

UncleanLeaderElectionsPerSec > 0 indicates a data loss event. An out-of-sync replica became the partition leader, which means any records that were committed to the previous leader’s log but not yet replicated to this replica are permanently gone. This is not a warning state; it is an incident that requires forensic investigation.

The relationship between replication.factor, min.insync.replicas, and ISR count determines your durability envelope. With a 3-replica topic and min.insync.replicas=2:

ISR = 3: full durability, normal operation
ISR = 2: degraded redundancy, writes succeed, one more failure loses data
ISR = 1: acks=all writes fail; the cluster is protecting you from data loss by refusing writes

Monitor IsrShrinksPerSec as a leading indicator before checking UnderReplicatedPartitions. The shrink rate tells you the cluster is in the process of degrading; the partition count tells you how far it has degraded.

Performance and throughput monitoring

Throughput metrics reveal whether the cluster is approaching a resource ceiling, and which resource is the binding constraint.

Throughput metrics:

Metric	JMX MBean path
BytesInPerSec	kafka.server:type=BrokerTopicMetrics,name=BytesInPerSec
BytesOutPerSec	kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec
MessagesInPerSec	kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec
TotalTimeMs (Produce P99)	kafka.network:type=RequestMetrics,name=TotalTimeMs,request=Produce
TotalTimeMs (FetchConsumer P99)	kafka.network:type=RequestMetrics,name=TotalTimeMs,request=FetchConsumer

To identify whether a throughput ceiling is a network, disk, or CPU constraint:

Network saturation: BytesInPerSec + BytesOutPerSec approaching NIC bandwidth, combined with a drop in NetworkProcessorAvgIdlePercent
Disk saturation: RequestHandlerAvgIdlePercent dropping while LocalTimeMs increases; elevated LogFlushRateAndTimeMs
CPU saturation: Host CPU above 70% sustained, often correlated with message format conversions — check MessageConversionsTimeMs

Note that BytesOutPerSec is typically two to three times BytesInPerSec in a cluster with a replication factor of 3, because follower fetch traffic is counted as outbound. A sudden rise in BytesOutPerSec without a corresponding rise in BytesInPerSec can indicate a catch-up read: a consumer group that fell behind is now reading cold log segments from disk, which also degrades page cache for all other consumers on that broker.

Storage and retention monitoring

Kafka fills disks faster than most teams expect. The calculation is: data_rate × replication_factor × retention_period. A 1 GB/s ingest rate with replication factor 3 and 7-day retention requires 1.8 TB of raw disk per broker at minimum, before accounting for segment overhead.

Storage metrics to track:

Metric	Notes
Disk bytes used (OS metric)	Alert at > 70% capacity; critical at > 85%
kafka.log:type=Log,name=Size	Log size per topic-partition
kafka.log:type=LogFlushStats,name=LogFlushRateAndTimeMs	Log flush P99 latency — warning at > 500ms
kafka.log:type=Log,name=LogEndOffset	Track for unexpected jumps or stalls

Log compaction can fall behind write rate in high-throughput compacted topics, causing the log to grow beyond what compaction alone would maintain. The leading indicator is a growing LogEndOffset on compacted topics without corresponding LogStartOffset advancement. Compaction threads are configured via log.cleaner.threads; if the disk is I/O bound, adding cleaner threads will not help — you need faster storage.

Retention policy misconfiguration is a common source of disk exhaustion. Time-based retention (log.retention.hours) and size-based retention (log.retention.bytes) interact: Kafka enforces whichever threshold is hit first. A topic configured for 7-day retention with no size limit on a high-throughput cluster can consume terabytes before the time window triggers.

Infrastructure and OS metrics monitoring

Kafka is sensitive to the underlying host in ways that most applications are not.

Disk I/O. Kafka is a disk-bound workload. It relies on the OS page cache for low-latency reads and sequential writes. Monitor iowait and disk read/write throughput at the OS level. Elevated iowait correlates directly with increased LocalTimeMs in broker request latency.

Network. With a replication factor of 3, every byte written to a broker is retransmitted twice to followers. Monitor both inbound and outbound interface utilisation. On cloud instances, also watch for traffic-shaping events (AWS CloudWatch TrafficShaping metric): cloud providers will silently throttle network traffic when a burstable instance hits its baseline, causing latency spikes that are difficult to attribute without this metric.

Page cache. The OS page cache is Kafka’s primary read acceleration mechanism. Consumer groups that are actively tailing the log read from page cache, not from disk. A consumer group that falls significantly behind forces the broker into cold disk reads, which evict hot data from the page cache and degrade all other consumers on that broker. Monitor OS-level memory pressure: if the page cache is under pressure, iowait and request latency will climb together.

File descriptors. Kafka opens one file descriptor per log segment. In a cluster with many topics and partitions, file descriptor exhaustion is a real failure mode. Set nofile limits to at least 100,000 per process and monitor current FD usage.

Memory swappiness. Set vm.swappiness = 1 to keep JVM heap pages in physical RAM. Swapping JVM pages to disk during garbage collection cycles introduces severe disk I/O latency, compounding pause times.

Kafka Connect monitoring

Kafka Connect is a separate JVM process with its own monitoring surface.

Connector and task state. The primary health signal for Connect is the connector and task status. A connector can be in RUNNING, PAUSED, or FAILED state. Tasks have the same states, plus UNASSIGNED. A FAILED task that does not recover automatically requires investigation of the Connect worker log.

Key Connect metrics:

Metric	JMX MBean path	Notes
connector-status	kafka.connect:type=connector-metrics,connector=(…)	Check for FAILED state
task-status	kafka.connect:type=connector-task-metrics,connector=(…),task=(…)	Track FAILED and UNASSIGNED tasks
connector-total-record-errors	kafka.connect:type=connector-metrics,connector=(…)	Error count per connector
source-record-write-rate	kafka.connect:type=source-task-metrics,connector=(…),task=(…)	Write rate to Kafka topics
sink-record-send-rate	kafka.connect:type=sink-task-metrics,connector=(…),task=(…)	Delivery rate to downstream system
put-batch-max-time-ms	kafka.connect:type=sink-task-metrics,connector=(…),task=(…)	P99 write latency to the sink

For source connectors, monitor the lag between the source system and the Kafka topic. For database CDC connectors, this is typically the replication slot lag in the source database.

For sink connectors, monitor the offset commit latency (offset-commit-success-percentage) and any increase in task restart counts. A task that restarts repeatedly is often encountering a poison-pill record in the topic, a downstream write failure that is not being handled by the dead-letter queue configuration, or a connection timeout to the downstream system.

Error records delivered to dead-letter queues should also be monitored. A growing DLQ depth indicates that the connector is encountering records it cannot process, which may signal a schema change, a data quality issue, or a configuration mismatch.

Kafka Streams monitoring

Kafka Streams applications expose metrics under kafka.streams MBeans.

Thread and task state:

Metric	JMX MBean path	What to watch
thread-state	kafka.streams:type=stream-thread-metrics,thread-id=(…)	Alert if any thread is in DEAD or PARTITIONS_REVOKED state
process-rate	kafka.streams:type=stream-thread-metrics,thread-id=(…)	Processing rate; a drop indicates a downstream bottleneck
commit-latency-avg	kafka.streams:type=stream-thread-metrics,thread-id=(…)	Latency to commit processed records; a spike often indicates state store pressure

For stateful topologies with RocksDB state stores:

Metric	Notes
rocksdb-total-block-cache-usage	Track memory usage; exhaustion causes state store reads to hit disk
rocksdb-estimate-num-keys	Useful for capacity planning and detecting key accumulation
rocksdb-compaction-pending	Pending compaction indicates write pressure on the state store

The most common operational issue in Kafka Streams is excessive state store size causing the application to lag behind its input topic. Monitor the processing rate (process-rate) against the input topic’s MessagesInPerSec. If the Streams application is consistently slower than its input, the topology needs more parallelism, or the state store operations need to be profiled.

Application and business-level monitoring

Infrastructure metrics tell you that the cluster is running. Business-level metrics tell you whether it is doing what you need it to do.

For most teams, the most useful business-level signal derived from Kafka metrics is end-to-end latency: the time between a record being written by a producer and that record being processed by a consumer. This is consumer lag expressed as wall-clock time rather than offset distance.

How to build this:

Produce a timestamp in the record metadata (or use the Kafka producer timestamp if reliable)
At the consumer, calculate now - record.timestamp for each processed record
Track this as a histogram; alert on P95 and P99, not just the mean

Concrete examples of business-level Kafka observability:

An order processing pipeline: lag in the order-confirmed topic expressed as seconds of latency between order placement and confirmation email dispatch
A fraud detection pipeline: lag in the transactions topic as a direct multiplier on fraud window size — a 5-minute lag means 5 additional minutes of unchecked transactions
A real-time dashboard: lag in the events topic as the staleness of displayed data

Building a “Kafka health to business outcome” dashboard that shows consumer lag alongside the downstream metric it affects gives on-call engineers the context to treat lag alerts with appropriate urgency. When it is clear that 2 minutes of lag on the payment-events topic means 2 minutes of delayed fraud detection, the alert becomes easier to prioritise correctly.

Alerting and incident detection

The following table covers the minimum required alert set for a production Kafka cluster. Use this as a starting point and adjust thresholds based on your SLOs.

Metric	Condition	Severity	Rationale
OfflinePartitionsCount	> 0	Page	Partitions are completely unreachable; client reads and writes fail immediately
ActiveControllerCount (cluster sum)	Not equal to 1 for 5 minutes	Page	Split-brain or total controller loss; metadata operations halted
UnderReplicatedPartitions	> 0 sustained for 10+ minutes	Page	Durability degraded; one more failure risks data loss
UncleanLeaderElectionsPerSec	> 0	Page	Data loss has occurred; investigation required
UnderMinIsrPartitionCount	> 0	Page	Producers with acks=all are failing; client-visible write errors
IsrShrinksPerSec	Sustained positive rate	Warning	Leading indicator of under-replication; investigate cause
RequestHandlerAvgIdlePercent	< 0.3 (30%) sustained	Warning	Request queue building up; latency degradation imminent
RequestHandlerAvgIdlePercent	< 0.2 (20%)	Page	Severe I/O saturation; client timeouts likely
NetworkProcessorAvgIdlePercent	< 0.3 (30%)	Warning	Network thread saturation; connection drops possible
GC pause P99	> 200ms	Warning	Risk of ISR shrink on affected broker
GC pause P99	> 500ms	Page	ISR shrink likely; acks=all writes at risk
JVM heap post-GC	> 70% of max heap	Warning	Increasing GC pressure; Full GC risk
JVM heap post-GC	> 85% of max heap	Page	Imminent OutOfMemoryError or stop-the-world Full GC
Disk capacity	> 70%	Warning	Review retention configuration; add capacity
Disk capacity	> 85%	Page	Disk exhaustion will crash the broker
Consumer lag rate of change	Growing trend for 5+ minutes	Warning	Consumer falling behind; investigate root cause
Consumer time-lag	Exceeds per-topic SLO threshold	Page	Downstream SLA breach in progress
metadata-apply-error-count (KRaft)	> 0	Page	Metadata corruption on broker; restart required
EventQueueSize (KRaft)	> 100	Warning	Controller overloaded with administrative tasks

Alert philosophy notes:

Alert on OfflinePartitions and UncleanLeaderElections unconditionally — there is no benign explanation for these conditions in a healthy cluster.
Alert on UnderReplicatedPartitions only after a sustained window (5-10 minutes) to avoid false positives during normal broker restarts and rolling updates.
Avoid alerting on consumer lag using a raw offset count. Express it relative to throughput (time-based) or use a trend-based approach like Burrow.
IsrShrinksPerSec is a leading indicator; UnderReplicatedPartitions is a lagging one. Alert on both, at different severities.

Capacity planning and forecasting

Capacity planning for Kafka requires tracking how quickly the cluster is consuming its three primary resources: network, disk, and broker count.

Disk capacity projection. The formula for disk required per broker is:

(ingestion_rate × replication_factor × retention_period) / broker_count

At 500 MB/s ingestion with replication factor 3 and 7-day retention across 6 brokers: 500 MB/s × 3 × 604,800s / 6 = ~150 TB raw storage per broker.

Track the rate of disk growth via BytesInPerSec and project forward at current growth rate. Alert when you have less than 30 days of headroom at current trajectory.

Partition count and broker density. In KRaft mode, the historical ZooKeeper-era limit of roughly 4,000 partitions per broker and 200,000 partitions per cluster no longer applies. KRaft’s Raft-based metadata management supports millions of partitions. However, each partition still consumes file descriptors, open segment handles, and memory for replication tracking. A practical upper bound of 10,000-20,000 partitions per broker remains sensible for most workloads.

Rebalancing cost. When adding brokers to expand capacity, Kafka does not automatically redistribute partitions. Use Cruise Control or kafka-reassign-partitions.sh to generate and execute reassignment plans. Always throttle reassignment throughput using --throttle <bytes_per_second> to prevent replication traffic from saturating broker network interfaces during business hours.

Kafka monitoring tools

The choice of monitoring tool affects both what you get out of the box and how much engineering is required to reach full coverage.

Tool	Ingestion method	Consumer lag	Dashboard setup	Broker footprint
Prometheus + Grafana	Pull via JMX Exporter sidecar agent	Requires separate kafka-exporter deployment	Manual JSON dashboard import	Low
Datadog	Pull via JMXFetch daemon (350 metric limit)	Native via Data Streams Monitoring	Pre-packaged SaaS dashboards	Moderate
Dynatrace	Process-bound OneAgent injection	Native via OneAgent introspection	Out-of-the-box Kafka Overview dashboard	Low
Confluent Control Center	Stream-based; requires Metrics Reporter JAR on every broker	Native in Normal mode	Proprietary; no external integrations	Very high (8-16 GB heap)
Kpow	Native AdminClient API polling — no JMX required	Native via AdminClient offset polling	Pre-packaged native UI	Near-zero (no broker-side agent)

What requires custom configuration in each tool:

Prometheus/Grafana: Consumer lag (separate kafka-exporter), custom MBean patterns in the kafka-2_0_0.yml YAML rule file, Grafana dashboard variable mapping. JMX histogram metrics are converted to static gauges, losing cumulative sum data needed for accurate rate calculations.
Datadog: JMX-enabled agent image (-jmx tag), pod annotations for Autodiscovery, custom metrics.yaml filter to stay within the 350-metric-per-instance limit. AWS MSK clusters require a separate CloudWatch integration — the standard Kafka agent check cannot query MSK directly.
Dynatrace: Topic name filters to prevent metric cardinality explosion (and the associated Davis Data Unit billing overage), and a custom plugin.json schema for non-standard MBeans. The Davis AI anomaly detection engine baselines normal behaviour automatically, which is particularly valuable for consumer lag patterns that vary with traffic shape.
Confluent Control Center: Proprietary Metrics Reporter JAR on every broker classpath, rolling cluster restart to activate. Legacy versions (CP 7.x) use client-side monitoring interceptors that create roughly 50 internal topics and significant metadata overhead. CP 8.0 migrates to Prometheus-based pull collection, requiring a migration if upgrading from the interceptor model. Not compatible with non-Confluent distributions (OSS Kafka, MSK, Aiven).
Kpow: No JMX or broker-side agent. Kpow queries the AdminClient API directly, eliminating JMX serialisation overhead on the broker. Streams topology visualisation requires embedding the kpow-streams-agent dependency in the application code. Prometheus egress requires enabling PROMETHEUS_EGRESS=true and configuring basic authentication.

For organisations that need to minimise broker resource overhead while maintaining full observability, running Prometheus paired with Kpow is a practical architecture: Kpow’s metadata engine eliminates JMX sidecars on the brokers, and its Prometheus egress endpoint feeds clean telemetry into a centralised Prometheus TSDB. You can try Kpow free for 30 days — it connects to any Kafka cluster in minutes and can be deployed via Docker, Helm, or JAR.

Summary

Kafka monitoring is not a single dashboard. It is a set of overlapping layers: broker internals, replication state, JVM health, consumer lag, producer behaviour, storage capacity, and OS resources. Each layer has distinct failure modes, and a blind spot in any one of them can become a production incident.

The minimum viable monitoring set for any production cluster:

Alert immediately on OfflinePartitionsCount > 0 and UncleanLeaderElectionsPerSec > 0
Alert after a sustained window on UnderReplicatedPartitions > 0
Track IsrShrinksPerSec as a leading indicator of durability degradation
Use time-based consumer lag thresholds, not raw offset counts
Monitor JVM GC pause duration as a direct predictor of ISR instability
Watch RequestHandlerAvgIdlePercent and NetworkProcessorAvgIdlePercent for broker saturation
For KRaft clusters, add quorum health metrics from kafka.server:type=raft-metrics

The rest — per-topic metrics, Connect and Streams coverage, business-level lag thresholds — is built on top of this foundation.

A detailed guide to Kafka producer monitoring

Factor House — Thu, 04 Jun 2026 00:00:00 GMT

Key takeaways

The Kafka Java producer client exposes metrics through JMX under the kafka.producer:type=producer-metrics MBean, covering throughput, latency, buffering, error rates, and connection state.
record-error-rate and bufferpool-wait-time are the two most important metrics to alert on. The first signals delivery failures; the second signals back-pressure that can stall application threads.
Producer metrics should be correlated with broker-side metrics. Elevated request-latency-avg on the client often originates from broker disk saturation, replication lag, or quota throttling rather than from the producer itself.
Configuration choices — acks, linger.ms, batch.size, max.in.flight.requests.per.connection, and compression.type — directly affect observable metrics and should inform how you interpret them.
This article covers the Apache Kafka Java producer client, targeting Kafka 3.x (with notes on 2.x where relevant). Clients based on librdkafka, used by Python, Go, and C producers, expose a different metric surface and are not covered here.

What is Kafka producer monitoring?

Kafka producer monitoring is the practice of observing the runtime state of a Kafka producer client: how many records it is sending, how quickly, whether any are failing, and whether back-pressure from the broker is affecting the application.

The Apache Kafka Java producer client uses an asynchronous two-thread architecture. When your application calls producer.send(), records are serialized and written to the RecordAccumulator, a memory buffer that groups records into batches by partition. A background Sender thread then drains those batches over persistent TCP connections to the appropriate broker leaders.

This design decouples your application thread from network I/O, but it also means failures and bottlenecks can be indirect. A broker that is slow to acknowledge writes does not immediately cause send() to fail. Instead, batches accumulate in the buffer until buffer.memory is exhausted, at which point your application thread blocks. Understanding this cascade is central to interpreting producer metrics correctly.

Key producer metrics to monitor

All Java producer metrics are exposed through JMX. Producer-level metrics use the MBean pattern:

kafka.producer:type=producer-metrics,client-id={client-id}

Topic-scoped metrics are available separately under:

kafka.producer:type=producer-topic-metrics,client-id={client-id},topic={topic}

The metric names below apply to Kafka 3.x. Most were introduced in Kafka 2.x and are consistent across both versions.

Throughput metrics

Metric	JMX attribute	Description	Why it matters
`record-send-rate`	`record-send-rate`	Records sent per second to a specific topic	Baseline for producer health; a drop to zero on a running producer indicates a stall
`byte-rate`	`byte-rate`	Bytes sent per second to a specific topic	Measures network load; correlate with broker `BytesInPerSec` to confirm delivery

Both of these are topic-scoped metrics available under the producer-topic-metrics MBean and require a topic label in your query.

Latency metrics

Metric	JMX attribute	Description	Why it matters
`request-latency-avg`	`request-latency-avg`	Average round-trip time in ms for produce requests	Rising values indicate broker degradation, replication lag, or network issues
`request-latency-max`	`request-latency-max`	Maximum observed produce request latency in ms	Useful for detecting outlier latency, particularly under `acks=all`
`record-queue-time-avg`	`record-queue-time-avg`	Average time in ms that a batch spends in the RecordAccumulator before being sent	Indicates whether the Sender thread is keeping up with the application; target below 50ms under normal conditions

Error and retry metrics

Metric	JMX attribute	Description	Why it matters
`record-error-rate`	`record-error-rate`	Failed record sends per second	Should be 0.0 at all times; any sustained non-zero value means records are being dropped
`record-retry-rate`	`record-retry-rate`	Record send retries per second	Brief spikes are expected during leader elections; sustained elevation suggests persistent instability

Batching efficiency metrics

Metric	JMX attribute	Description	Why it matters
`batch-size-avg`	`batch-size-avg`	Average batch size in bytes at the time of send	Compare to your `batch.size` config; significantly lower values suggest the producer is flushing batches too early
`records-per-request-avg`	`records-per-request-avg`	Average number of records per produce request	High values indicate efficient batching; low values under heavy load suggest `linger.ms` is too short
`compression-rate-avg`	`compression-rate-avg`	Average compression ratio across sent batches	Lower values mean better compression; useful for comparing before and after a `compression.type` change

Buffer and back-pressure metrics

Metric	JMX attribute	Description	Why it matters
`buffer-available-bytes`	`buffer-available-bytes`	Remaining send buffer capacity in bytes	Should stay close to the configured `buffer.memory`; dropping toward zero means the producer is accumulating faster than the Sender thread can drain
`bufferpool-wait-time`	`bufferpool-wait-time`	Fraction of time the appender thread is waiting for buffer space	Target: 0.0; any sustained non-zero value indicates back-pressure; values above 0.10 warrant investigation
`waiting-threads`	`waiting-threads`	Number of application threads currently blocked in `send()`	Target: 0; any non-zero value means application threads are stalling, which can cascade into upstream service timeouts

Connection metrics

Metric	JMX attribute	Description	Why it matters
`connection-count`	`connection-count`	Number of active TCP connections to brokers	Should equal the number of brokers the producer is actively writing to
`io-wait-time-ns-avg`	`io-wait-time-ns-avg`	Average nanoseconds the Sender thread spent waiting for ready sockets	A very high value may indicate low throughput; a near-zero value means the thread is continuously busy
`connection-creation-rate`	`connection-creation-rate`	New TCP connections established per second	Elevated rates suggest connections are being dropped and re-established, typically due to network instability
`connection-close-rate`	`connection-close-rate`	Sockets closed per second	Elevated alongside `connection-creation-rate` indicates connection flapping

Producer configuration effects on metrics

The configuration you pass to the producer at startup has a direct and measurable effect on the metrics you observe at runtime. Understanding these relationships helps you interpret metric values correctly and tune configuration to meet your performance objectives.

acks setting

The acks parameter controls how many broker acknowledgments the producer waits for before considering a record successfully delivered.

acks=0: the producer does not wait for any acknowledgment. request-latency-avg will be minimal, but record-error-rate will not reflect broker-side failures — records can be lost silently without the client being aware.
acks=1: the leader broker acknowledges after writing to its local log. This produces moderate request-latency-avg values and captures leader-side failures.
acks=all (or acks=-1): the leader waits for all in-sync replicas to acknowledge before responding. This produces the highest request-latency-avg values — typically two to five times higher than acks=1 in a healthy cluster — but provides the strongest durability guarantee.

Under acks=all, any replication lag on follower replicas will appear directly in request-latency-avg. Correlate elevated latency values with the broker-side PurgatorySize metric to determine whether the delay originates from the replication layer.

Idempotence and transactions

Setting enable.idempotence=true provides exactly-once delivery semantics per partition. The broker assigns each producer a producer ID and tracks sequence numbers per partition, discarding any duplicates that result from retries.

Idempotent producers tend to show higher baseline record-retry-rate values than non-idempotent producers, because the client retries aggressively on transient failures. This is expected behavior; the broker deduplicates on its side.

When enable.idempotence=true is configured, max.in.flight.requests.per.connection must be 5 or lower. Exceeding this limit prevents the broker from guaranteeing ordering of sequence numbers across concurrent in-flight batches, which can produce OutOfOrderSequenceException errors that surface as elevated record-error-rate.

For transactional producers, two additional metrics are worth monitoring: flush-time-ns-total, which records cumulative nanoseconds spent blocked inside .flush() during transactional boundary commits, and txn-send-offsets-time-ns-total, which tracks time spent publishing offset maps within transactions.

linger.ms and batch.size

linger.ms controls how long the Sender thread waits before dispatching an incomplete batch. batch.size sets the maximum batch size in bytes.

A higher linger.ms value — for example, 50 to 100ms — allows more records to accumulate per batch. You will observe higher batch-size-avg and records-per-request-avg, and lower network overhead per record. The trade-off is increased record-queue-time-avg, since records wait longer in the accumulator before being sent.

A lower linger.ms value — 0 to 5ms — minimises record-queue-time-avg, which is appropriate for latency-sensitive workloads, but typically produces smaller batches and lower throughput per network request.

If batch-size-avg is consistently close to your configured batch.size, the producer is filling batches before the linger timer fires. This is a sign of a high-throughput workload where linger.ms has less influence on batch efficiency.

max.in.flight.requests.per.connection

This setting controls how many unacknowledged produce requests the Sender thread can pipeline on a single broker connection before blocking.

Higher values increase throughput by allowing more requests in flight simultaneously, but they introduce ordering risks. If a broker returns an error for request N and the producer retries, request N+1 may arrive before the retry completes, resulting in out-of-order delivery on that partition.

For idempotent producers, the broker’s sequence number tracking handles this risk — but only up to 5 concurrent in-flight requests per connection. Beyond that, idempotence guarantees do not hold.

Monitor requests-in-flight alongside record-retry-rate. If retries are frequent and requests-in-flight is consistently at the configured maximum, reducing the in-flight limit may improve delivery stability at some cost to throughput.

compression.type

Compression is applied per batch before the batch is dispatched by the Sender thread. The available options are none, gzip, snappy, lz4, and zstd.

compression-rate-avg reflects the effectiveness of the chosen algorithm: a value of 0.4 means the compressed batch is 40% of its original size. Lower values indicate better compression, which reduces network bandwidth and broker disk usage.

The CPU cost varies significantly by algorithm. lz4 and snappy offer fast compression with moderate ratios, which suits latency-sensitive workloads. zstd and gzip achieve better compression ratios but are more CPU-intensive. If you observe higher request-latency-avg after enabling compression, check producer host CPU utilisation — the Sender thread may be CPU-bound.

Compression shifts CPU work from the broker to the producer at write time, and to the consumer at read time. Weigh the network and storage savings against the CPU cost on both sides.

buffer.memory and max.block.ms

buffer.memory sets the total size of the RecordAccumulator pool in bytes (default: 32 MB). max.block.ms sets how long an application thread will block in send() waiting for buffer space before throwing a TimeoutException.

When broker throughput drops — due to disk saturation, replication lag, or quota throttling — the Sender thread drains batches more slowly. Batches accumulate in the accumulator, buffer-available-bytes drops toward zero, and application threads begin to block. bufferpool-wait-time rises above zero. If a thread remains blocked for longer than max.block.ms, the call fails with a TimeoutException.

waiting-threads rising above zero is always a signal that the producer is experiencing back-pressure. Increasing buffer.memory gives more headroom before application threads block, but does not address the underlying cause of the back-pressure.

Collecting producer metrics

JMX collection from the producer process

The Kafka Java producer client registers JMX MBeans with the platform MBean server of its JVM automatically. No additional configuration is required for local JMX access.

To enable remote JMX access for external tooling, pass the following JVM flags when starting your producer application:

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9999
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false

For production environments, replace the authenticate=false and ssl=false flags with appropriate authentication and TLS configuration.

Once the JMX port is open, you can connect using tools like jconsole, jmxterm, or any JMX-compatible client to inspect MBeans under kafka.producer:type=producer-metrics,client-id={client-id}.

Prometheus JMX exporter

The most common production approach is to run the Prometheus JMX Exporter as a Java agent inside the producer JVM. It polls the local MBean server and exposes the metrics as a Prometheus-compatible HTTP endpoint, without requiring remote JMX access.

Add the agent to your JVM startup arguments:

export JAVA_OPTS="-javaagent:/usr/share/java/prometheus/jmx_prometheus_javaagent-0.20.0.jar=9404:/etc/prometheus/kafka-producer-jmx-rules.yml"

A rules configuration file that captures both producer-level and topic-level metrics:

lowercaseOutputName: true
lowercaseOutputLabelNames: true

rules:
  # Producer-level metrics (throughput, latency, buffer, errors)
  - pattern: 'kafka.producer<type=producer-metrics, client-id=(.+)><>([^:]+):'
    name: kafka_producer_$2
    type: GAUGE
    labels:
      client_id: "$1"

  # Topic-level metrics (per-topic record-send-rate, byte-rate)
  - pattern: 'kafka.producer<type=producer-topic-metrics, client-id=(.+), topic=(.+)><>([^:]+):'
    name: kafka_producer_topic_$3
    type: GAUGE
    labels:
      client_id: "$1"
      topic: "$2"

This configuration exposes metrics like kafka_producer_record_error_rate{client_id="orders-service"} and kafka_producer_topic_record_send_rate{client_id="orders-service",topic="orders"} at http://localhost:9404/metrics.

In Strimzi-managed Kubernetes deployments, you can keep the JMX port (typically 9999) open for tooling while dedicating port 9404 to Prometheus scraping.

Multi-producer aggregation

When running multiple producer instances — across replicated services or multiple application pods — you need a strategy for identifying individual producers in your metrics pipeline.

Each producer instance is identified by its client.id configuration. Set this to something that includes both the service name and the instance identifier, for example orders-service-pod-0. This value becomes the client_id label in Prometheus, allowing you to filter and aggregate metrics per producer.

If client.id is left at its default value of producer-1, all instances in a fleet will appear identical in your metrics store. When a single instance begins experiencing elevated record-error-rate or bufferpool-wait-time, you will have no way to isolate it.

Tag producer metrics with at least:

client_id: the producer instance identifier
topic: the target topic (available on topic-scoped metrics)
Any additional labels your platform uses for environment, region, or service ownership

Micrometer and OpenTelemetry integration

Micrometer

If you are using Spring Kafka or another framework that integrates with Micrometer, the KafkaClientMetrics binder forwards JMX metrics to any Micrometer-supported registry. Metric names follow the same naming convention as the JMX attributes but are prefixed with kafka.producer. and use dots rather than underscores — for example, kafka.producer.record-error-rate instead of kafka_producer_record_error_rate.

OpenTelemetry

The OpenTelemetry JMX Metric Gatherer supports Kafka producers natively through its kafka-producer target system. The gatherer runs as a sidecar or separate process, queries the producer JVM over JMX/RMI, and pushes metrics to an OpenTelemetry Collector:

receivers:
  jmx/producer:
    jar_path: /opt/opentelemetry-jmx-metrics.jar
    endpoint: localhost:9999
    target_system: kafka-producer
    collection_interval: 10s

exporters:
  otlp:
    endpoint: otel-collector.monitoring.svc.cluster.local:4317
    tls:
      insecure: true

service:
  pipelines:
    metrics:
      receivers: [jmx/producer]
      exporters: [otlp]

The OTel target system exposes metric names prefixed with kafka.producer. — for example, kafka.producer.record-retry-rate and kafka.producer.request-latency-avg.

Producer health check script

The script below provides a point-in-time diagnostic for a running Kafka producer. It queries a Prometheus JMX Exporter endpoint and evaluates the key indicators of producer health against fixed thresholds.

Companion scripts covering broker-side and consumer-side checks are described in the Kafka broker monitoring article and the Kafka consumer monitoring article. The three scripts share the same pattern: query the metrics endpoint, evaluate against thresholds, and exit with a status code compatible with standard monitoring integrations.

Prerequisites:

The Prometheus JMX Exporter is running on the producer JVM (default port: 9404)
curl and awk are available in the shell environment

Thresholds:

Metric	Severity	Threshold
`record-error-rate`	Critical	> 0.0 per second
`bufferpool-wait-time`	Critical	> 0.10 (10% of appender time)
`waiting-threads`	Critical	> 0
`record-queue-time-avg`	Warning	> 500ms
`request-latency-avg`	Warning	> 200ms
`record-retry-rate`	Warning	> 10 per second

#!/usr/bin/env bash
# kafka-producer-health-check.sh
#
# Checks the health of a running Kafka producer by querying its
# Prometheus JMX Exporter endpoint.
#
# Usage:
#   ./kafka-producer-health-check.sh [client_id] [metrics_port]
#
# Arguments:
#   client_id    (optional) Filter output to a specific producer client.id
#   metrics_port (optional) JMX Exporter HTTP port. Default: 9404
#
# Exit codes:
#   0 = OK
#   1 = WARNING
#   2 = CRITICAL
#   3 = UNKNOWN

CLIENT_ID="${1:-}"
METRICS_PORT="${2:-9404}"
METRICS_URL="http://localhost:${METRICS_PORT}/metrics"

# Thresholds
CRIT_RECORD_ERROR_RATE="0.0"
CRIT_BUFFERPOOL_WAIT="0.10"
CRIT_WAITING_THREADS="0"
WARN_QUEUE_TIME_MS="500"
WARN_REQUEST_LATENCY_MS="200"
WARN_RETRY_RATE="10.0"

EXIT_CODE=0

fetch_metric() {
  local name="$1"
  local label_filter="$2"
  if [ -n "$label_filter" ]; then
    curl -sf "$METRICS_URL" | grep "^${name}{" | grep "${label_filter}" | awk '{print $2}' | head -1
  else
    curl -sf "$METRICS_URL" | grep "^${name}" | awk '{print $2}' | head -1
  fi
}

check() {
  local label="$1"
  local value="$2"
  local threshold="$3"
  local op="$4"
  local severity="$5"

  if [ -z "$value" ]; then
    printf "%-10s %s (metric not found)\n" "UNKNOWN" "$label"
    [ "$EXIT_CODE" -lt 3 ] && EXIT_CODE=3
    return
  fi

  local triggered
  triggered=$(awk "BEGIN { print (${value} ${op} ${threshold}) ? 1 : 0 }")

  if [ "$triggered" -eq 1 ]; then
    printf "%-10s %s = %s  [threshold %s %s]\n" "$severity" "$label" "$value" "$op" "$threshold"
    [ "$severity" = "CRITICAL" ] && EXIT_CODE=2
    [ "$severity" = "WARNING"  ] && [ "$EXIT_CODE" -lt 2 ] && EXIT_CODE=1
  else
    printf "%-10s %s = %s\n" "OK" "$label" "$value"
  fi
}

LABEL_FILTER=""
[ -n "$CLIENT_ID" ] && LABEL_FILTER="client_id=\"${CLIENT_ID}\""

echo "Kafka producer health check"
echo "Endpoint:  ${METRICS_URL}"
echo "Client ID: ${CLIENT_ID:-all}"
echo "---"

check "record-error-rate" \
  "$(fetch_metric kafka_producer_record_error_rate    "$LABEL_FILTER")" \
  "$CRIT_RECORD_ERROR_RATE"   ">" "CRITICAL"

check "bufferpool-wait-time" \
  "$(fetch_metric kafka_producer_bufferpool_wait_time "$LABEL_FILTER")" \
  "$CRIT_BUFFERPOOL_WAIT"     ">" "CRITICAL"

check "waiting-threads" \
  "$(fetch_metric kafka_producer_waiting_threads      "$LABEL_FILTER")" \
  "$CRIT_WAITING_THREADS"     ">" "CRITICAL"

check "record-queue-time-avg (ms)" \
  "$(fetch_metric kafka_producer_record_queue_time_avg "$LABEL_FILTER")" \
  "$WARN_QUEUE_TIME_MS"       ">" "WARNING"

check "request-latency-avg (ms)" \
  "$(fetch_metric kafka_producer_request_latency_avg  "$LABEL_FILTER")" \
  "$WARN_REQUEST_LATENCY_MS"  ">" "WARNING"

check "record-retry-rate" \
  "$(fetch_metric kafka_producer_record_retry_rate    "$LABEL_FILTER")" \
  "$WARN_RETRY_RATE"          ">" "WARNING"

AVAIL=$(fetch_metric kafka_producer_buffer_available_bytes "$LABEL_FILTER")
TOTAL=$(fetch_metric kafka_producer_buffer_total_bytes     "$LABEL_FILTER")
if [ -n "$AVAIL" ] && [ -n "$TOTAL" ] && [ "$TOTAL" != "0" ]; then
  USED_PCT=$(awk "BEGIN { printf \"%.1f\", (1 - ${AVAIL} / ${TOTAL}) * 100 }")
  printf "%-10s %s = %s%%  (%s of %s bytes available)\n" \
    "INFO" "buffer-utilisation" "$USED_PCT" "$AVAIL" "$TOTAL"
fi

echo "---"
case "$EXIT_CODE" in
  0) echo "Result: OK"       ;;
  1) echo "Result: WARNING"  ;;
  2) echo "Result: CRITICAL" ;;
  3) echo "Result: UNKNOWN"  ;;
esac

exit "$EXIT_CODE"

Limitations: This script evaluates one point in time. It does not detect trends, multi-instance comparisons, or intermittent spikes. It also does not query broker-side metrics — a clean result here does not rule out a broker problem contributing to producer behaviour. Use it for incident investigation or pre-deployment checks, not as a substitute for continuous alerting.

Alerting strategy for Kafka producer monitoring

The table below describes a baseline alert set for Kafka producer monitoring. For each alert, the condition reflects sustained behaviour rather than instantaneous spikes, which reduces false positives during normal transient events like leader elections.

Alert	Metric	Condition	Likely cause	Suggested response
Delivery failures	`record-error-rate`	> 0 sustained for 2 minutes	Broker unavailable, serialization error, authorization failure	Check producer logs for exception type; verify broker health and topic ACLs
Producer back-pressure	`bufferpool-wait-time`	> 0.10 for 1 minute	Broker too slow to accept records; send buffer exhausted	Correlate with broker `TotalTimeMs` and `PurgatorySize`; consider temporarily increasing `buffer.memory` while diagnosing
Application threads blocked	`waiting-threads`	> 0 for 30 seconds	Buffer memory depleted under sustained back-pressure	Same as above; blocked application threads can cascade into upstream service timeouts
Rising request latency	`request-latency-avg`	> 200ms for 5 minutes	Broker disk degradation, replication lag, or quota throttling	Check broker `TotalTimeMs` component breakdown; inspect ISR count and `PurgatorySize`
Poor batching efficiency	`batch-size-avg`	Consistently below 25% of configured `batch.size`	Producer sending too frequently relative to `linger.ms`	Increase `linger.ms`; verify that the producer is handling meaningful load
Silent producer	`record-send-rate`	= 0 with producer process running	Application-level stall, topic ACL block, or misconfiguration	Check application logs; verify the topic exists and the producer has write permission

For use with Prometheus Alertmanager, the following rule definitions cover the most critical conditions:

groups:
  - name: kafka-producer-alerts
    rules:
      - alert: ProducerDeliveryFailures
        expr: rate(kafka_producer_record_error_total[5m]) > 0.1
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Producer {{ $labels.client_id }} is dropping records"
          description: >
            The record error rate on client {{ $labels.client_id }} has exceeded 0.1 errors/second
            for 2 minutes. Records are failing to deliver and may be lost.

      - alert: ProducerBackPressure
        expr: rate(kafka_producer_bufferpool_wait_time_total[2m]) > 0.15
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Producer {{ $labels.client_id }} buffer exhaustion"
          description: >
            Memory allocation wait rate for client {{ $labels.client_id }} has exceeded 15% over
            2 minutes. Application threads are blocking on .send() calls.

      - alert: ProducerQueueDelay
        expr: kafka_producer_record_queue_time_avg > 500
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Producer {{ $labels.client_id }} accumulator queue delay"
          description: >
            Average record queue wait time has exceeded 500ms for 5 minutes. This may indicate
            broker degradation, network bottlenecks, or quota throttling.

      - alert: ProducerHighRetryRate
        expr: rate(kafka_producer_record_retry_total[5m]) > 10.0
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "High retry rate on producer {{ $labels.client_id }}"
          description: >
            The record retry rate is exceeding 10 retries/second for 3 minutes. This suggests
            transient network instability or a cluster rebalancing event.

The thresholds above are starting points rather than universal standards. A request-latency-avg of 200ms may be acceptable for a batch analytics pipeline but not for a payment processing service. Adjust thresholds to match your SLOs and your cluster’s normal operating range.

Common Kafka producer issues and how to diagnose

Producer stalling due to full send buffer

Symptom: Application threads block on send(), eventually throwing TimeoutException after max.block.ms elapses.

Metric signals: bufferpool-wait-time approaching or above 0.10; buffer-available-bytes near zero; waiting-threads above 0; record-queue-time-avg elevated.

Diagnostic steps:

Confirm that buffer-available-bytes is depleted and bufferpool-wait-time is non-zero. This pattern confirms back-pressure rather than a serialization issue — serialization failures drop records before they reach the buffer, leaving buffer-available-bytes near its maximum.
Check broker TotalTimeMs and its component latencies. If the local log append component (t_local) is elevated, the broker disk may be saturated. If the replication wait component (t_remote) is elevated, there may be ISR lag.
Check broker RequestQueueSize. A growing queue indicates that broker network threads are saturated and cannot process incoming requests fast enough.
If the broker is healthy, review batch-size-avg relative to batch.size. An oversized batch.size can cause the buffer to fill faster than the Sender thread can drain it, particularly at low throughput where batches take longer to fill.

Retry storm from leader election

Symptom: A spike in record-retry-rate and request-latency-avg, typically coinciding with a broker restart or failure event.

Metric signals: record-retry-rate spikes; request-latency-avg increases; record-error-rate may briefly rise before returning to zero.

Cause: During a partition leader election, brokers return NotLeaderForPartitionException for produce requests targeting the old leader. The Kafka client retries these requests automatically once metadata is refreshed and a new leader is elected.

Brief retry spikes during failover are expected and do not indicate data loss, as long as record-error-rate returns to zero after the election completes and record-retry-rate subsides. If retries persist, check that retries and retry.backoff.ms are set appropriately for your cluster’s typical election duration, and verify that the affected topic’s ISR is healthy.

Idempotent producer failure under high in-flight requests

Symptom: The broker returns OutOfOrderSequenceException; the producer surfaces this as a delivery failure.

Metric signals: record-error-rate spikes; record-retry-rate may be elevated.

Cause: When enable.idempotence=true is set, the broker tracks sequence numbers per partition to detect and discard duplicates. If max.in.flight.requests.per.connection exceeds 5, the broker cannot reliably order sequence numbers across concurrent in-flight batches, which produces this exception.

To resolve this, set max.in.flight.requests.per.connection to 5 or lower when enable.idempotence=true is configured. From Kafka 3.0 onward, the client enforces this constraint automatically when idempotence is explicitly enabled; earlier versions do not.

Serialization errors surfacing as send failures

Symptom: record-error-rate spikes, but broker metrics and network metrics are unchanged.

Metric signals: record-error-rate elevated; request-latency-avg, record-queue-time-avg, and broker-side metrics all remain at normal levels; buffer-available-bytes stays near its maximum.

Cause: Serialization failures occur before a record is written to the RecordAccumulator. If the serializer throws an exception — for example, when a value does not conform to an Avro schema — the record is never queued and no network activity occurs. This is the key distinction from network or broker failures, where buffer utilisation also rises.

Check producer logs for SerializationException or schema registry errors. If you are using Confluent Schema Registry, a schema compatibility violation will produce this pattern: record-error-rate rises while all broker and buffer metrics remain stable.

Compression CPU overhead causing latency regression

Symptom: request-latency-avg increases after enabling or changing compression, even though broker health is unchanged.

Metric signals: request-latency-avg elevated; compression-rate-avg low (indicating good compression, but at a CPU cost); producer host CPU utilisation elevated.

Cause: Compression is performed by the Sender thread before each batch is dispatched. On a CPU-constrained host, heavyweight algorithms such as gzip or zstd can slow the Sender thread enough to increase observed request latency, even though the broker is responding quickly.

To diagnose, compare request-latency-avg before and after enabling compression, and monitor producer host CPU. If compression is causing a latency regression, consider switching to lz4, which typically offers a better throughput-to-CPU ratio than gzip or zstd for latency-sensitive workloads. Weigh the network bandwidth and broker disk savings against the producer CPU cost before deciding.

Best practices for Kafka producer monitoring

Monitor record-error-rate and bufferpool-wait-time as the minimum baseline for every producer. These two metrics surface the most operationally significant failure conditions — delivery failures and upstream stalling — and should be alerting targets in any environment.
Set acks=all and enable.idempotence=true for any workload where data loss is unacceptable, and account for the resulting increase in request-latency-avg when setting alert thresholds. A latency threshold calibrated for acks=1 will produce false positives under acks=all.
Use linger.ms greater than 0 in throughput-optimised workloads, and verify the effect by monitoring batch-size-avg and records-per-request-avg. If batch sizes do not increase after raising linger.ms, the producer may already be limited by batch.size or by low topic throughput.
Set client.id explicitly on every producer instance. In environments running multiple producer instances, undifferentiated client IDs make it impossible to isolate a misbehaving instance when record-error-rate or bufferpool-wait-time rises in your aggregated metrics.
Align alert thresholds with your service-level objectives rather than generic defaults. The threshold values in this article are reasonable starting points, but what constitutes a normal or degraded baseline varies by workload type and cluster configuration.
Correlate producer metrics with broker and consumer metrics before concluding that a producer is the source of a problem. Elevated request-latency-avg is frequently a symptom of broker disk saturation, replication lag, or quota throttling. For a complete picture, refer to the Kafka broker monitoring article and the Kafka consumer monitoring article.
For transactional producers, extend your monitoring to include flush-time-ns-total and txn-send-offsets-time-ns-total. Spikes in these values indicate bottlenecks in the transactional commit path that would not appear in the standard metric set but can cause application threads to stall.

Monitor Kafka producers with Factor House

Kpow provides real-time visibility into your Kafka producers without requiring a separate metrics pipeline. You can inspect producer metrics, configuration, and per-topic throughput directly from the UI, alongside broker and consumer monitoring in the same tool.

Give Kpow a try for yourself with a free 30-day trial. You can connect it to any Kafka cluster in minutes and deploy it via Docker, Helm, or JAR.

How Adidas uses Apache Kafka in production

Factor House — Tue, 02 Jun 2026 00:00:00 GMT

Adidas runs Apache Kafka at the centre of two distinct platform layers: a business activity monitoring pipeline introduced around 2015, and an internal observability platform called HOLMES that scaled to 100 billion messages per day during the shift to digital commerce in 2020. The engineering decisions behind those platforms — custom GoLang Kafka tooling to avoid JVM overhead, ksqlDB user-defined functions for field-level data masking, and an AsyncAPI-driven GitOps pipeline for self-service topic provisioning — are documented in detail across the company’s engineering blog and Confluent conference talks.

Company overview

Adidas is a global sportswear manufacturer and retailer operating e-commerce, retail, and supply chain systems across multiple regions. Its platform engineering team maintains the streaming infrastructure that underpins real-time business monitoring, observability, and data integration across those systems.

Kafka adoption at Adidas began around 2015 with the Business Activity Monitoring 2.0 project. By mid-2018 that platform was ingesting events from 29 source systems across 74 topics at 6 million messages per day. The more significant inflection came in 2020, when Adidas’s shift toward direct digital commerce drove the HOLMES observability platform to 100 billion messages per day — a roughly 16,000-fold increase in two years that required architectural decisions well beyond standard Kafka configurations.

The most recent publicly documented development, presented at Current London 2025, is a self-service Kafka platform that replaced manual central-team provisioning with a GitOps pipeline backed by a custom domain-specific language and AsyncAPI specifications, cutting provisioning time from days to seconds.

Key Kafka milestones:

~2015: Business Activity Monitoring 2.0 initiated; Apache Kafka selected as the streaming backbone
April 2018: Adidas attends Kafka Summit London alongside Apple, Audi, IBM, BBC, and ING
June 2018: Iñaki Alzorriz publishes “From Monitoring to Data Streaming” on adidoescode; platform at 74 topics, 29 sources, 6 million messages per day
2020: HOLMES observability platform scales to 100 billion messages per day during Adidas’s digital commerce expansion
May 2021: Jose Manuel Cristobal presents “Navigating the Observability Storm with Kafka” at Kafka Summit Europe 2021
July 2021: Adil Houmadi publishes article on extending ksqlDB with custom UDFs for regional data masking
February 2023: Gabriel Barreras documents Kafka Connect event sourcing pitfalls and solutions on adidoescode
2025: Jose Manuel Cristobal and Guillermo Lagunas present the self-service Kafka platform at Current London 2025

Adidas’s Kafka use cases

Business activity monitoring

Kafka serves as the event transport layer for Adidas’s Business Activity Monitoring 2.0 platform, carrying events from 29 source systems into a complex event processing engine for real-time analytics and reporting. The platform implements a fan-out pub-sub pattern for data extraction, stateful event processing, and data replication across internal teams. Iñaki Alzorriz described this architecture in a June 2018 post on the adidoescode Medium publication.

Observability and SRE (HOLMES)

HOLMES is Adidas’s internal observability system. Kafka is the streaming backbone that ingests all infrastructure logs and metrics from Kubernetes-based services, enabling problem detection, root cause analysis, and predictive alerting across e-commerce systems. At peak adoption in 2020, the platform processed 100 billion messages per day. Jose Manuel Cristobal covered the full architecture in a talk at Kafka Summit Europe 2021.

Event sourcing via Kafka Connect

Adidas uses Kafka Connect with a JDBC Source Connector to capture inserts and updates from Oracle 19c (deployed on AWS RDS) and transform them into Kafka events for downstream consumers. Gabriel Barreras documented the implementation and its pitfalls in a February 2023 adidoescode post.

Regional data streaming with ksqlDB

ksqlDB handles the splitting of a main Kafka topic into regional sub-topics, filtering data by region and masking sensitive fields using a custom user-defined function. Adil Houmadi described this pattern in a July 2021 adidoescode post.

Self-service Kafka platform

The current state of the platform, presented at Current London 2025, enables Kafka stakeholders to directly manage topics, permissions, schemas, and connectors without involving the central platform team. A GitOps pipeline backed by a custom DSL and AsyncAPI specifications handles all asset provisioning. Guillermo Lagunas and Jose Manuel Cristobal presented the details in “From Days to Seconds: Adidas’ Journey to Scalable Kafka Self-Service” at Current London 2025.

Scale and throughput

100 billion messages per day processed by HOLMES at peak adoption in 2020 (Jose Manuel Cristobal, Kafka Summit Europe 2021)
6 million messages per day, 74 topics, 29 source systems at the BAM 2.0 pilot stage in mid-2018 (Iñaki Alzorriz, 2018)
Single region, 3 availability zones for the HOLMES Kafka deployment, with a multitenant configuration serving 2 tenants at the time of the 2021 talk (Jose Manuel Cristobal, 2021)

Adidas’s Kafka architecture

HOLMES observability platform

HOLMES is structured around four layers, deployed on Kubernetes in a single region across 3 availability zones.

The ingestion layer collects Prometheus metrics per Kubernetes namespace using a Kubernetes Operator deployed with Helm Charts. A custom GoLang application called Prom2Kafka uses the Prometheus Remote Write protocol with protobuf data models to push those metrics into Kafka. Log collection is handled by Fluentd, which is auto-deployed to Kubernetes namespaces using annotations, requiring no manual configuration per service.

The streaming layer is Apache Kafka, deployed open-source with SSL and mutual TLS for all client connections. The cluster is multitenant, with ACLs and connection quotas enforcing isolation between tenants.

The storage layer uses two custom GoLang tools: KafkaToPromMetrics consumes from Kafka and writes metrics into Victoria Metrics tenants; Filebeat and Logstash handle log forwarding with suppression capabilities into OpenDistro (an Apache 2.0 build of Elasticsearch). Kafka Streams suppressors filter non-compliant logs and high-rate metrics before they reach storage, controlling ingest volume at 100 billion messages per day.

The consumption layer provides Grafana for metrics dashboarding and alerting, and Kibana for log analysis with multi-tenancy support.

Self-service Kafka platform

The current platform is described as vendor-agnostic and non-opinionated, meaning it does not depend on Confluent-specific managed services. Teams manage topics, permissions, schemas, and connectors through a GitOps pipeline using a custom DSL specification. AsyncAPI specifications serve as both the documentation standard and the authoritative source of truth for resource provisioning. The data catalogue is built on top of these AsyncAPI specs, allowing teams to discover available topics, message schemas, and ownership metadata without consulting the platform team.

Producer architecture

Metrics are ingested via Prom2Kafka using the Prometheus Remote Write protocol and protobuf serialisation. For application event pipelines, Avro is the documented serialisation format with Schema Registry enforcement. Spring Boot (Java) is used in Kafka consumer and producer services for application-layer event streaming.

Consumer architecture

For HOLMES, KafkaToPromMetrics is a GoLang Kafka consumer writing metrics from Kafka into Victoria Metrics. The GoLang client was chosen specifically to avoid the JVM overhead that would be significant at 100 billion messages per day. ksqlDB consumers handle regional topic splitting with a single persistent query.

Stream processing

Kafka Streams is used for stateful stream processing in HOLMES, specifically for suppressing high-rate metrics and filtering non-compliant log events before they reach the storage layer. ksqlDB handles SQL-based stream transformations, including region filtering and field-level hashing via custom UDFs.

Kafka Connect ecosystem

Kafka Connect with a JDBC Source Connector pulls inserts and updates from Oracle 19c on AWS RDS. The source connector polls the database on a configurable interval using timestamp-based watermarking.

Special techniques and engineering innovations

Custom GoLang Kafka tooling to avoid JVM overhead

At 100 billion messages per day, Adidas chose to build two bespoke GoLang applications rather than use standard JVM-based Kafka clients. Prom2Kafka handles ingestion from Prometheus into Kafka via the Remote Write protocol and protobuf, and KafkaToPromMetrics consumes from Kafka and writes into Victoria Metrics. Both were written specifically to avoid the memory footprint and latency characteristics of JVM-based clients at that throughput. Jose Manuel Cristobal documented this decision in the Kafka Summit Europe 2021 slide deck.

ksqlDB user-defined functions for field-level data masking

Rather than building a separate pipeline stage for PII handling, Adidas extended ksqlDB with a custom Java UDF that applies SHA-256 hashing to sensitive fields. A single persistent query filters events by region and applies the UDF to produce separate regional topics with masked data. Adil Houmadi documented the implementation, including the Java class structure and deployment steps, in the July 2021 adidoescode post.

AsyncAPI as infrastructure config, not just documentation

The self-service platform uses AsyncAPI specifications as the source of truth for provisioning Kafka resources including topics, schemas, connectors, and ACLs. This goes beyond the typical documentation use case for AsyncAPI: the specifications are consumed directly by a GitOps pipeline that provisions resources on commit. The result is that teams create and modify Kafka infrastructure through pull requests against AsyncAPI specs rather than through tickets to a central team. Guillermo Lagunas and Jose Manuel Cristobal presented this architecture at Current London 2025.

Kafka Streams suppressors for ingest cost control

HOLMES uses Kafka Streams suppressors to filter events before they reach Victoria Metrics and OpenDistro. Non-compliant logs and metrics producing at unusually high rates are suppressed at the streaming layer rather than at the storage layer, which limits the storage write volume at 100 billion messages per day without requiring changes to the producing services.

Operating Kafka at scale

Deployment model: Self-managed open-source Kafka on Kubernetes, deployed in a single region across 3 availability zones. The HOLMES cluster uses a multitenant configuration with SSL and mTLS for all client connections. Adidas does not document use of a managed Kafka service for these workloads.

Security and multi-tenancy: All Kafka clusters enforce authentication and encryption via SSL and mutual TLS. Quotas and ACLs provide isolation between tenants within multitenant clusters. The Adidas API Guidelines document TLS requirements and ACL patterns as platform standards for all Kafka topics.

GitOps-driven lifecycle management: Topics, permissions, schemas, and connectors are managed declaratively through a GitOps pipeline backed by AsyncAPI specifications. Provisioning changes are introduced as pull requests; the pipeline applies them automatically on merge. This eliminates the need for the central platform team to manually provision resources and makes the state of Kafka infrastructure version-controlled and auditable.

SLA tracking and adoption KPIs: After moving to the self-service model, Adidas tracks resolution time SLAs and adoption KPIs to measure operational improvement. The published outcome is a reduction from days to seconds for provisioning tasks, which the team uses as an ongoing measure of platform health.

Prometheus and Helm Chart deployment: The Prometheus collection layer in HOLMES is deployed via Kubernetes Operator with Helm Charts. Fluentd is deployed automatically using Kubernetes namespace annotations, keeping the observability ingest layer fully infrastructure-as-code without requiring per-service configuration.

Data catalogue via AsyncAPI: Adidas’s Platform and Engineering team maintains a data catalogue built on AsyncAPI specifications so teams can discover available topics, message schemas, field definitions, headers, and ownership metadata. This replaces informal knowledge of what topics exist and who owns them with a structured, searchable registry backed by the same specs that provision the infrastructure.

Challenges and how they solved them

Central provisioning became a bottleneck as adoption grew

As Kafka adoption scaled across Adidas, the central platform team was responsible for manually creating topics, assigning permissions, registering schemas, and configuring connectors for every team that needed access. The process took days per request and was a source of delays and configuration errors.

Adidas built a vendor-agnostic self-service platform using a custom DSL and AsyncAPI specifications as the provisioning layer, fed through a GitOps pipeline. Individual teams now manage their own Kafka resources through pull requests. Provisioning time dropped from days to seconds. Guillermo Lagunas and Jose Manuel Cristobal presented the before and after at Current London 2025.

Race condition in Kafka Connect JDBC Source Connector causing silent data loss

When concurrent database transactions overlapped with the JDBC Source Connector’s query window, records whose commit timestamps fell between the connector’s query executions were silently skipped. The connector’s watermark advanced past those records, and they were never re-ingested.

The immediate fix was tuning timestamp.delay.interval.ms to introduce a buffer period, ensuring that pending transactions complete before the next poll cycle advances the watermark. Gabriel Barreras documented the root cause and the configuration fix in the February 2023 adidoescode post, and also noted that for use cases requiring consistent low-latency change data capture, Debezium or a CDC-native database approach is preferable to JDBC Source Connectors.

Scaling observability ingest to 100 billion messages per day

The shift to digital commerce during 2020 required HOLMES to ingest an order of magnitude more traffic than it was originally designed for, without a proportional increase in infrastructure cost.

Adidas handled this through a combination of architectural choices: GoLang Kafka clients for the ingestion and storage layers to avoid JVM overhead, Kafka Streams suppressors to reduce the volume of events reaching storage, and Victoria Metrics as a cost-effective time-series backend. The result was a platform that reached 100 billion messages per day without requiring a full re-architecture.

Full tech stack

Category	Tools	Notes
Message broker	Apache Kafka	Open-source deployment, self-managed on Kubernetes
Stream processing	Kafka Streams	Stateful processing; used for log and metrics suppression in HOLMES
Stream processing	ksqlDB	SQL-based transformations; regional topic splitting and PII masking
Kafka client (custom)	Prom2Kafka (GoLang)	Prometheus Remote Write ingestion into Kafka using protobuf
Kafka client (custom)	KafkaToPromMetrics (GoLang)	Kafka consumer writing metrics from Kafka into Victoria Metrics
Connectors	Kafka Connect (JDBC Source)	Database event sourcing from Oracle 19c (AWS RDS)
Schema management	Schema Registry	Schema management and enforcement for Kafka topics
Serialisation	Apache Avro	Message serialisation format used with Schema Registry
Log collection	Fluentd	Auto-deployed in Kubernetes via namespace annotations
Metrics collection	Prometheus	Per-namespace metrics collection; deployed via Kubernetes Operator and Helm Charts
Metrics storage	Victoria Metrics	Time-series metrics storage backend; described as fast, cost-effective, scalable
Log forwarding	Filebeat + Logstash	Lightweight log forwarding with suppression into OpenDistro
Log storage	OpenDistro (Elasticsearch)	Apache 2.0 Elasticsearch build for log indexing and search
Dashboards and alerting	Grafana	Metrics dashboarding and alerting on top of Victoria Metrics
Log analysis	Kibana	Multi-tenant log analysis and visualisation
Orchestration	Kubernetes	Container orchestration for HOLMES and the data streaming platform
Infrastructure config	AsyncAPI	Documentation standard and GitOps provisioning DSL for all Kafka resources
Source database	Oracle 19c (AWS RDS)	Source for Kafka Connect JDBC event sourcing
Application framework	Spring Boot (Java)	Used in Kafka consumer and producer services

Key contributors

Iñaki Alzorriz (Director of Platform Engineering, Adidas): Authored the 2018 adidoescode post describing the Kafka-based data streaming initiative and Business Activity Monitoring 2.0. adidoescode, June 2018
Jose Manuel Cristobal (Senior Platform Engineer / Director Platform Engineering, Adidas): Presented “Navigating the Observability Storm with Kafka” at Kafka Summit Europe 2021; co-presented “From Days to Seconds” at Current London 2025. Kafka Summit Europe 2021; Current London 2025
Guillermo Lagunas (Platform Engineering, Adidas): Co-presented the self-service Kafka platform at Current London 2025. Current London 2025
Gabriel Barreras (Platform Engineering, Adidas): Documented Kafka Connect JDBC Source Connector race conditions and solutions. adidoescode, February 2023
Adil Houmadi (Platform Engineering, Adidas): Documented ksqlDB UDF extension for regional data masking. adidoescode, July 2021

Key takeaways for your own Kafka implementation

Choose your Kafka client language based on throughput requirements. At 100 billion messages per day, Adidas replaced JVM-based clients with GoLang to reduce memory footprint and latency. If you are running high-throughput observability or telemetry pipelines, the client runtime overhead is worth evaluating early.
ksqlDB user-defined functions let you extend SQL-based pipelines without a separate processing stage. Adidas used a custom Java UDF to apply SHA-256 hashing within a persistent ksqlDB query, handling both regional routing and PII masking in a single step. If your stream processing requirements push beyond what built-in ksqlDB functions cover, UDFs are worth considering before adding a separate processing tier.
AsyncAPI can serve as infrastructure config, not just documentation. Adidas uses AsyncAPI specifications as the source of truth for a GitOps pipeline that provisions topics, schemas, connectors, and ACLs directly. If you are managing Kafka resources manually or via tickets, treating API specifications as executable configuration is an approach that scales better as the number of topics and teams grows.
JDBC Source Connectors require careful watermark configuration for correctness. The timestamp-based polling model in the Kafka Connect JDBC Source Connector can miss records when concurrent transactions span query boundaries. Setting timestamp.delay.interval.ms to buffer the watermark advance is a necessary tuning step; for strict correctness requirements, a CDC-native connector such as Debezium is worth evaluating instead.
Suppression at the streaming layer is more cost-effective than suppression at storage. Adidas used Kafka Streams suppressors to filter high-rate metrics and non-compliant logs before they reached Victoria Metrics and Elasticsearch. This limits storage write volume without requiring changes to producing services and avoids paying ingestion costs for data you will filter out anyway.

Sources and further reading

Primary sources

Iñaki Alzorriz, “From Monitoring to Data Streaming — Data Streaming Initiative in Adidas” (June 2018)
Jose Manuel Cristobal, “Navigating the Observability Storm with Kafka” — Kafka Summit Europe 2021
Jose Manuel Cristobal, “Navigating the Observability Storm with Kafka” — slide deck
Guillermo Lagunas and Jose Manuel Cristobal, “From Days to Seconds: Adidas’ Journey to Scalable Kafka Self-Service” — Current London 2025
Gabriel Barreras, “Event Sourcing with Kafka Connect: Inconsistency, Pitfalls & Solutions” (February 2023)
Adil Houmadi, “Extending ksqlDB Built-in Capability” (July 2021)
Adidas Platform & Engineering, Adidas API Guidelines — Kafka Asynchronous Guidelines

Try Kpow with your Kafka cluster

If you are monitoring a Kafka cluster at any scale, you can try Kpow free for 30 days. It connects to any Kafka cluster in minutes and deploys via Docker, Helm, or JAR.

How Afterpay uses Apache Kafka in production

Factor House — Tue, 02 Jun 2026 00:00:00 GMT

Afterpay runs Apache Kafka as the event stream for its Global Payments Platform, operating it under PCI DSS constraints that required a custom network architecture and bespoke Kafka client libraries. Payment events leave the PCI zone via a two-hop AWS PrivateLink design, serialised with Avro and signed with AWS KMS-backed HMAC signatures before reaching the shared Kafka cluster. A separate challenge emerged after Block’s 2022 acquisition: Kafka-archived data from Afterpay’s Sydney AWS region was feeding Delta merge operations running in US West, generating more than USD 1,500 per day in S3 cross-region egress. The solution, Project Teleport, restructured the pipeline to keep compute co-located with data at each stage and delivered annual savings of approximately USD 540,000.

Company overview

Afterpay is a buy-now-pay-later platform founded in Sydney, Australia in 2014 and acquired by Block Inc. in 2022. It operates in Australia, New Zealand, the United States, the United Kingdom, Canada, and Europe, processing instalment payment agreements for millions of consumers at retail and online merchants. Afterpay’s infrastructure runs on AWS, with its primary data engineering operations historically anchored in the Sydney region.

The acquisition by Block brought an integration challenge: Afterpay’s Kafka-sourced data needed to flow into Block’s consolidated data platform, which operated in US West. Reconciling two distinct AWS regions — with different data ownership, egress cost profiles, and pipeline architectures — is the defining data engineering problem that shaped Afterpay’s most recent Kafka work.

Key Kafka milestones:

Pre-2022: Afterpay runs Kafka for PCI-compliant payment event streaming from its Global Payments Platform.
2022 (acquisition): Block acquires Afterpay. Afterpay’s Kafka data in Sydney (APSE2) needs to integrate with Block’s US West (USW2) data platform.
September 2022: Jing Li publishes “Implementing Kafka in the Payments PCI World,” documenting the two-hop PrivateLink architecture, custom Kafka client libraries, and PCI compliance controls deployed across all Afterpay payment environments.
November 2024: Bulk migration of Afterpay Kafka topics to Project Teleport commences.
March 2025: Project Teleport migration completes. Unni Krishnan publishes the full case study, reporting annual savings of approximately USD 540,000 and zero-downtime migration of approximately 120 Kafka topics.

Afterpay’s Kafka use cases

PCI-compliant payment event streaming

Afterpay’s primary Kafka use case is streaming payment events out of the PCI DSS zone to downstream consumers. Every payment processed by Afterpay’s Global Payments Platform generates events that need to reach risk, analytics, and reporting systems without those systems having direct access to the PCI network. Kafka provides the durable, decoupled transport layer; the custom client libraries and network architecture enforce the compliance boundary.

Risk decisioning

Kafka-sourced payment event data feeds Afterpay’s Risk Decisioning systems downstream. These are among the approximately 200 datasets delivered from Afterpay’s Kafka pipelines via Project Teleport, providing risk teams with timely event data for fraud and credit decisions.

Business intelligence and financial reporting

Afterpay’s business intelligence and financial reporting teams are major consumers of the Delta Lake pipeline fed by Kafka-archived data. Project Teleport consolidates approximately 200 downstream datasets from the Sydney Kafka archive into the Block data platform in US West, making them available to BI and finance tooling.

Machine learning pipelines

Databricks Spark Declarative Pipelines read streaming data from Afterpay’s Kafka topics to feed machine learning workloads. The data lake uses a medallion architecture (bronze, silver, gold) on Delta Lake, with Unity Catalog and AWS Glue providing governance and cataloguing.

Scale and throughput

Daily pipeline volume: 9 TB of Kafka-archived data processed per day by Afterpay’s data team (Unni Krishnan, March 2025).
Topics migrated: Approximately 120 Kafka topics migrated under Project Teleport (Unni Krishnan, March 2025).
Downstream datasets: Approximately 200 datasets delivered to risk, BI, and financial reporting from Kafka pipelines (Unni Krishnan, March 2025).
Payment events published: “Millions of messages” published since the PCI Kafka implementation was deployed to all payments environments (Jing Li, September 2022).
Compression ratio: Approximately 50% reduction in data volume from Avro-to-Parquet conversion in Project Teleport Stage 1.
Egress saving: Approximately USD 540,000 per year from restructuring the cross-region pipeline (Unni Krishnan, March 2025).

Afterpay’s Kafka architecture

Deployment

Afterpay’s Kafka infrastructure runs on AWS, with the primary archival cluster anchored in Sydney (APSE2). The payments Kafka cluster is operated by Block’s Platform Engineering team. Cross-region data processing targets AWS US West (USW2) via Project Teleport.

PCI zone network design

PCI DSS compliance prohibits mixing cardholder data environments with general infrastructure. For Kafka, this means the network path from the PCI zone to the Kafka broker must be isolated from shared transit networks.

AWS Transit Gateway was evaluated but rejected on two grounds: it lacked support for overlapping VPC CIDR ranges across Block’s network topology, and it would have created a direct routing path between the PCI zone and other Block networks. Afterpay’s solution uses dedicated AWS PrivateLinks with a two-hop architecture:

Hop 1: PCI zone connects to Platform Engineering’s network via a dedicated PrivateLink.
Hop 2: Platform Engineering’s network connects to the Kafka cluster via a second PrivateLink.

Non-PCI teams use AWS Transit Gateway with a single PrivateLink hop. The Payments Platform uses two hops, ensuring the PCI environment has no direct visibility into the shared transit network.

DNS resolution for Kafka brokers across the PrivateLink connections uses AWS Route53 private hosted zones. Schema Registry access from the PCI zone is gated through a Squid Proxy that explicitly whitelists Schema Registry DNS names as an egress control.

Custom Kafka client libraries

A standard Kafka client provides no protection against cardholder data leaving the PCI zone in message payloads, no mechanism for asynchronous publication, and no message-level integrity verification. Afterpay’s Payments Platform team built custom producer and consumer libraries to address all three (Jing Li, September 2022):

Card detection and obfuscation: The producer library applies RegEx-based cardholder data detection during Avro serialisation. Any field matching a card number pattern is obfuscated before the bytes are published to the Kafka topic. Luhn algorithm validation was planned as a second pass to reduce false positives.

Non-blocking publication: A dedicated thread pool dispatches Kafka publishing asynchronously. The payment processing thread is released immediately after serialisation; Kafka I/O proceeds independently. A Kafka broker outage or elevated publish latency cannot propagate into payment response times.

Message integrity verification: Producers sign each message using HMAC with an AWS KMS Customer Managed Key. Consumers within the PCI zone verify the signature with the same KMS key before processing. This provides a tamper-evidence guarantee independent of TLS, which is relevant for audit and compliance scenarios where message-level provenance needs to be demonstrable.

Kafka data archival (Confluent Sink Connectors)

Afterpay archives Kafka topics to S3 using Confluent Sink Connectors, which land hourly records as Avro files in the Sydney APSE2 region. These archived files are the input to the Project Teleport cross-region pipeline.

Project Teleport: cross-region data processing

Before Project Teleport, Afterpay’s data team ran Delta merge operations with compute in APSE2 writing to Delta tables in USW2. S3 cross-region egress costs exceeded USD 1,500 per day. The core problem was that every merge operation transferred data between regions at the point of processing.

Project Teleport restructures the pipeline into two stages, each co-located with its data:

Stage 1 (APSE2): DeltaSync Spark jobs use Databricks Autoloader for incremental Avro file discovery. Files are converted from Avro to compressed Parquet and loaded into streaming interface Delta tables. Compression reduces the volume transferred to USW2 by approximately 50%.

Stage 2 (USW2): Delta Live Tables (DLT) jobs apply transformations and run merge operations locally using the apply_changes API. The apply_changes API provides built-in deduplication, handling the duplicate events that arise from Kafka’s at-least-once delivery semantics without requiring downstream logic.

Apache Airflow orchestrates both stages. A checkpoint transfer mechanism ensures historical data is not reprocessed during migration. All approximately 120 topics were migrated between November 2024 and March 2025 with no reported downtime.

Producer architecture

Producers in the PCI pipeline use Afterpay’s custom producer library. The card detection, thread pool, and KMS signing layers run within the library; the underlying Kafka client handles batching and delivery. Avro is the serialisation format throughout.

Consumer architecture

Custom consumer libraries verify the KMS-based HMAC signature on each message before processing. The Schema Registry, accessed via Squid Proxy, provides schema resolution for Avro deserialisation.

Special techniques and engineering innovations

Two-hop PrivateLink for PCI network isolation

The two-hop PrivateLink design is notable for its explicit trade-off: it adds a network hop and DNS configuration overhead in exchange for a provable compliance boundary. The PCI zone has no visibility into the shared transit network; it sees only the Platform Engineering PrivateLink endpoint. This makes the network isolation auditable, which matters for PCI DSS assessments.

KMS-signed Kafka messages for message-level integrity

Transport-layer security (TLS) protects messages in transit but does not provide message-level provenance. If a message is intercepted and modified between the producer and broker, or replayed by an adversary with network access, TLS does not prevent this. HMAC signing with a KMS CMK addresses both scenarios: a consumer can verify that a message was produced by an entity that held the signing key at the time of production, and that the message has not been modified since. This is a meaningful addition in regulated environments where audit evidence at the message level is valuable.

Deduplication via Delta Live Tables `apply_changes`

Rather than writing custom deduplication logic for Kafka’s at-least-once delivery guarantees, Project Teleport delegates this to Delta Live Tables’ apply_changes API. The API applies upsert semantics as records land in the target table, treating retried events as updates rather than inserts. This keeps the pipeline logic simple and the deduplication behaviour well-defined.

Co-locating compute with data to eliminate cross-region egress

The USD 1,500/day egress problem arose because compute was separated from the data it was processing. Project Teleport’s solution is straightforward in principle: run each stage’s compute in the same region as its input data. The Avro-to-Parquet conversion in Stage 1 compounds the saving by reducing the volume transferred between regions before Stage 2 begins.

Operating Kafka at scale

Deployment model: Afterpay’s Kafka cluster runs on AWS, operated by Block’s Platform Engineering team. Confluent Sink Connectors handle the archival layer from Kafka to S3. Block does not publicly describe using a fully managed Kafka service such as Amazon MSK or Confluent Cloud for the primary cluster.

Schema governance: Avro is used throughout the PCI and archival pipelines. Schema Registry access from the PCI zone is controlled via a Squid Proxy, which enforces explicit DNS-level egress whitelisting. Schema updates must be compatible with existing consumers before being deployed.

Orchestration: Apache Airflow manages cross-stage orchestration in Project Teleport, coordinating the DeltaSync jobs in APSE2 and the DLT jobs in USW2.

Migration strategy: Project Teleport’s checkpoint transfer mechanism allows Kafka topic migration without reprocessing historical data. Bulk migration of approximately 120 topics ran over four months (November 2024 to March 2025) with zero reported downtime. Transient streaming interface Delta tables in Stage 1 use sliding window logic to avoid race conditions during migration.

Data lake governance: The Delta Lake medallion architecture (bronze, silver, gold) on S3 provides tiered data quality guarantees. Unity Catalog and AWS Glue handle metadata and governance across the platform.

Challenges and how they solved them

PCI network isolation for Kafka connectivity

Standard Kafka client connectivity over shared transit infrastructure is not viable under PCI DSS without additional isolation controls. AWS Transit Gateway was evaluated and rejected because it lacked support for overlapping VPC CIDR ranges across Block’s network topology, and because it would have exposed the PCI zone to other Block networks. Afterpay designed a two-hop PrivateLink architecture: PCI zone to Platform Engineering, then Platform Engineering to the Kafka cluster. DNS is resolved via Route53 private hosted zones; Schema Registry egress is explicitly controlled via Squid Proxy. Since rollout, millions of payment events have been published through this architecture.

Cardholder data leaking into Kafka message payloads

A standard Kafka producer serialises all fields without inspecting their content. In a payments context, this creates a risk that card numbers embedded in event payloads would be published to Kafka topics accessible outside the PCI zone. Afterpay’s custom producer library applies RegEx-based card number detection during Avro serialisation and obfuscates matching fields before publishing. The check runs inline in the serialisation path, with no external calls or added dependencies. Luhn algorithm validation was planned as a follow-up to reduce false positives.

Kafka I/O blocking payment processing

Synchronous Kafka publication on the main payment processing thread means that any Kafka broker latency — whether from a broker restart, network fluctuation, or backpressure — propagates into payment response times. Afterpay’s custom producer library introduces a dedicated thread pool that takes Kafka publishing off the main thread as soon as serialisation completes. The payment flow’s latency is entirely independent of Kafka’s availability. This is a straightforward pattern, but the explicit documentation of it as a PCI-adjacent concern is useful for other teams operating in regulated environments.

USD 1,500/day in cross-region S3 egress from Delta merge operations

After Block acquired Afterpay, data processing consolidated in US West. Afterpay’s Kafka archive was in Sydney (APSE2), and the Delta merge jobs ran compute in APSE2 against Delta tables in USW2. Every merge operation transferred data cross-region, generating more than USD 1,500/day in S3 egress costs. Project Teleport split the pipeline at the regional boundary: Stage 1 runs in APSE2 and converts Avro to Parquet locally, reducing transfer volume by ~50%; Stage 2 runs merge operations locally in USW2. The annual saving is approximately USD 540,000.

Duplicate events from Kafka at-least-once delivery

Kafka’s at-least-once delivery semantics mean that events may be published more than once, particularly during producer retries or consumer rebalances. Without explicit handling, these duplicates propagate into downstream Delta Lake tables as duplicate rows. Project Teleport uses Delta Live Tables’ apply_changes API to apply upsert semantics as records land in USW2, treating retried events as updates. The deduplication logic is built into the DLT framework rather than maintained as custom code.

Full tech stack

Category	Tools	Notes
Message broker	Apache Kafka	AWS-hosted; operated by Block’s Platform Engineering team
Serialisation format	Apache Avro	Used throughout PCI and archival pipelines
Schema registry	Confluent Schema Registry	Accessed via Squid Proxy from PCI zone
Connectors	Confluent Sink Connectors	Archive Kafka topics to S3 as hourly Avro files (APSE2)
Custom client libraries	Proprietary (Afterpay Payments Platform)	Card obfuscation, async thread pool publishing, KMS HMAC signing
Key management	AWS KMS (Customer Managed Keys)	HMAC signing for message integrity verification
Networking	AWS PrivateLink (two-hop), AWS Route53 private hosted zones, Squid Proxy	PCI zone Kafka connectivity and Schema Registry egress control
Stream processing	Databricks Spark Declarative Pipelines, Delta Live Tables	Kafka-to-Delta Lake ingestion; Project Teleport pipeline
File discovery	Databricks Autoloader	Incremental Avro file ingestion in Project Teleport Stage 1
Orchestration	Apache Airflow	Cross-stage pipeline orchestration in Project Teleport
Data lake storage	Delta Lake on S3	Medallion architecture; Avro-to-Parquet via Project Teleport
Compute platform	Databricks	DeltaSync jobs and DLT workloads
Metadata / catalogue	Unity Catalog, AWS Glue	Data lake governance post-acquisition
Analytics sink	Snowflake	Platform-agnostic BI access on Delta Lake

Key contributors

Jing Li — authored “Implementing Kafka in the Payments PCI World” (September 2022), detailing the PCI Kafka architecture including the two-hop PrivateLink design and custom client libraries. Implementing Kafka in the Payments PCI World
Unni Krishnan — authored “Project Teleport: Cost-Effective and Scalable Kafka Data Processing at Block” (March 2025), detailing the cross-region Kafka data processing pipeline and its cost outcomes. Project Teleport
Unnee Udayakumar — Senior Manager of Data Engineering, Cash App ML and Data Science organisation; primary contact named in the Databricks case study on Delta Lake and Kafka streaming integration. Databricks case study

Key takeaways for your own Kafka implementation

PCI DSS compliance does not require a separate Kafka cluster inside the PCI zone. Afterpay uses standard Kafka infrastructure outside the PCI zone and enforces the boundary at the network layer (two-hop PrivateLink) and the client layer (card detection in the serialiser, KMS signing, async publication). The compliance work is in the access controls and client libraries, not in the broker topology.
Message-level signing with KMS is a viable pattern for tamper-evidence in regulated pipelines. HMAC verification at the consumer provides provenance guarantees that TLS alone does not. The operational requirement is shared KMS key access between producers and authorised consumers, which fits naturally into IAM-based access control on AWS.
Keeping compute co-located with data at each pipeline stage eliminates cross-region egress costs. The pattern in Project Teleport is reusable: if you are running a multi-stage pipeline across regions, identify the point where data crosses the regional boundary and ensure compression or format conversion happens before that crossing, not after.
Delta Live Tables’ apply_changes API handles Kafka at-least-once deduplication without custom logic. If you are landing Kafka events into Delta Lake and your producer may retry, apply_changes provides the deduplication semantics you need at the framework level.
Cross-region pipeline migration is achievable without downtime using checkpoint transfer. Afterpay migrated approximately 120 Kafka topics across a four-month window with no reported downtime by using checkpoint transfer to avoid historical data reprocessing. If you are rearchitecting a Kafka-to-lakehouse pipeline, the migration approach matters as much as the target architecture.

Sources and further reading

Primary sources

Jing Li, “Implementing Kafka in the Payments PCI World” (September 2022)
Unni Krishnan, “Project Teleport: Cost-Effective and Scalable Kafka Data Processing at Block” (March 2025)
Unnee Udayakumar (attributed), “Consistently providing accurate financial insights” (Databricks case study)

Try Kpow with your Kafka cluster

If you are monitoring a Kafka cluster at any scale, you can try Kpow free for 30 days. It connects to any Kafka cluster in minutes and deploys via Docker, Helm, or JAR.

Apache Kafka 4.3.0: A guide for platform engineers

Factor House — Tue, 02 Jun 2026 00:00:00 GMT

Kafka 4.3.0 was released on 22 May 2026 with 25 KIPs and 600+ commits since 4.2.0. If you manage Kafka in production, the release notes cover what changed in full. This article focuses on which of those changes are likely to affect your upgrade window, your monitoring setup, and the operational issues that have been quietly causing friction.

This is primarily an operator’s release. The headline features are broker cordoning, partition size metrics, share group tuning, classic protocol deprecation, and a set of tiered storage improvements. These changes address the operational experience of running Kafka at scale rather than developer-facing functionality — the kind of work that accumulates in Jira tickets like “decommission broker without manual coordination” and “surface retention headroom per partition in monitoring.”

This article walks through the operationally significant changes, calls out the deprecations that deserve a calendar entry, and closes with a checklist for teams preparing an upgrade.

Broker and log directory cordoning — KIP-1066

The new cordoned.log.dirs configuration marks a log directory as off-limits for new partition replica placement. The broker stays up; existing replicas continue serving reads and writes. New partition assignments route around the cordoned directory. Kafka’s 4.3 operations documentation covers the full decommissioning workflow.

Before 4.3, draining a broker or a specific disk required careful manual coordination: partition reassignment scripts, watching ISR (in-sync replica) lag, and accepting more operational risk than most teams wanted because the tooling made it genuinely awkward. Cordoning does not solve every decommissioning problem, but it addresses the most friction-heavy part: stopping the cluster from making the situation worse while you work.

The workflow now looks like:

Cordon the log directory (or broker, via cordoned.log.dirs)
Let the cluster route new placements elsewhere; no new replicas land on that target
Drain existing replicas at your own pace using partition reassignment
Decommission once drained

This is particularly useful in cloud-native environments where disk replacement or node cycling is a routine event rather than an incident. Teams running Kubernetes-based Kafka deployments will find this materially reduces the operational surface area of a node drain.

Test this in staging first. Understand the interaction with your replica placement strategy and rack-awareness configuration before rolling it to production.

Partition size percentage metrics — KIP-1257

Retention headroom has always been one of the more tedious Kafka metrics to derive. The raw data exists (log size, retention bytes), but assembling per-partition headroom from JMX required either custom tooling or cobbled-together dashboards that most teams quietly abandoned.

KIP-1257 fixes this directly. New JMX metrics expose what percentage of maximum retention each topic-partition currently occupies. This gives you a first-class signal for:

Identifying partitions approaching retention thresholds before they cause data loss or unexpected compaction behaviour
Setting meaningful alerts on retention pressure without writing custom collectors
Spotting topics where retention config is badly misaligned with actual throughput

This metric belongs in every Kafka operations dashboard and is a straightforward addition once you upgrade.

For teams who want these metrics surfaced alongside consumer lag, broker health, and throughput in a single view, Kpow by Factor House ingests JMX and Kafka-native metrics and will pick up these new partition size indicators without additional configuration.

Share groups (Kafka’s queue-semantics consumer model, introduced in 4.0) arrive in 4.3 with the operational tuning options that early adopters have been waiting for.

KIP-1240 adds new broker-level and group-level configurations to control share group behaviour. The specifics depend on your use case, but the significance is structural: share groups now have the configuration surface area you would expect from a production-grade feature. Teams that adopted early on 4.0 or 4.1 and found themselves with insufficient levers should review the new configs in the 4.3 documentation.

KIP-1263 is quieter but arguably more impactful for clusters with high consumer group churn. The group coordinator’s assignment logic is improved to avoid recomputing assignments when not necessary. If you have environments with frequent consumer restarts, scaling events, or high group cardinality, this directly reduces coordinator overhead. It is not the kind of change that appears in a benchmark, but it smooths out the coordinator’s CPU profile in a busy cluster.

Where share groups stand: 4.3 is a meaningful maturation milestone. If your team is on 4.2 and has not started with share groups yet, this release is a good prompt to re-evaluate the roadmap. If you are on 4.0 or 4.1 and already running share groups, upgrade and take the new configs seriously. If you were waiting for the project to stabilise before committing, the pace of development across recent releases gives a reasonable basis for planning.

Classic rebalance protocol deprecation — KIP-1251 + KIP-1274

If your consumers are still using the classic rebalance protocol, 4.3 introduces Phase 1 of the deprecation process: warning only, with no removal yet. The direction is unambiguous.

When a consumer starts with the classic rebalance protocol, 4.3 logs a recommendation to migrate. This is the precursor to a formal deprecation signalled for a future release. The group.coordinator.rebalance.protocols broker configuration is also deprecated in this release, marked for removal in Kafka 5.0.

KIP-1251 is the complementary improvement: member epoch validation logic is tightened to reduce unnecessary fencing of group members. Concretely, this means fewer spurious rebalances triggered by epoch mismatches during consumer group stabilisation. If you have been seeing rebalance noise in stable groups, this is worth tracking after upgrading.

Action for operators: Audit your consumer configurations now. Identify any consumer groups still using the classic protocol and build a migration timeline. The cooperative protocol has been the correct choice for new consumers since 2.4. Teams that defer this work until Kafka 5.0 will have less time to address it comfortably.

Tiered storage: safer and smarter — KIP-1023 + KIP-1208 + KIP-1235

Three tiered storage changes in 4.3: two operational and one correctness fix that deserves immediate attention.

KIP-1235 — check this first if you run tiered storage. The default min.insync.replicas (ISR) for the internal __remote_log_metadata topic was incorrect and did not align with what you would expect for a durable internal topic. 4.3 introduces a dedicated remote.log.metadata.topic.min.isr configuration to set this explicitly. If you are running tiered storage today, check your current __remote_log_metadata topic ISR configuration and align it with remote.log.metadata.topic.min.isr. This is a correctness issue rather than a tuning recommendation.

KIP-1023 introduces follower.fetch.last.tiered.offset.enable (default: false). When enabled, bootstrapping a new follower starts from the last tiered offset rather than the beginning of the local log. The practical effect: new brokers joining a cluster with tiered storage sync faster, and leader amplification during follower bootstrap is reduced. This is directly relevant to teams scaling clusters with large tiered storage configurations.

KIP-1208 adds the remote.log.metadata.admin. prefix for configuring the admin client used by tiered storage’s RemoteLogMetadataManager. This is a configuration hygiene improvement that makes the namespace cleaner and the scoping more explicit.

Taken together, tiered storage in 4.3 is safer to run (KIP-1235), faster to scale (KIP-1023), and better configured (KIP-1208).

KRaft fetch size controls — KIP-1219

New configurations: controller.quorum.fetch.max.bytes and controller.quorum.fetch.snapshot.max.bytes.

These cap the amount of data retrieved by KRaft Fetch and FetchSnapshot requests respectively. For most teams, these will never need to be changed from default. If you run large clusters where the KRaft controller shares resources with broker workloads, or where snapshot fetch operations have caused controller memory spikes, these controls give you a lever that did not previously exist.

This is a niche but real pain point for teams running high-metadata-throughput clusters. If you have seen controller memory pressure during snapshot fetches, the relevance will be apparent.

OAuth client assertions — KIP-1258

Support for client assertion authentication in the client_credentials OAuth grant type.

This is a security posture improvement for teams using OAuth with enterprise identity providers (Okta, Azure Active Directory, and similar) that prefer or require signed client assertions over shared secrets. If your Kafka security implementation already uses client_credentials and you have been working around the absence of assertion support, 4.3 resolves it natively. If this does not describe your environment, it is unlikely to affect your upgrade decision.

Kafka Streams: what changed for stream processors

Not a Streams-heavy release, but several targeted improvements are worth cataloguing for teams running Streams in production.

KIP-1259: Automatic local state cleanup on startup. The new state.cleanup.dir.max.age.ms configuration tells Streams to delete state directories that have not been modified within the specified duration on startup. This addresses a genuine hygiene problem in containerised Streams deployments where stale state accumulates across restarts and pod cycling. Previously, cleaning this required external intervention or custom initialisation logic. If you run Streams in Kubernetes or any environment with ephemeral containers, add this to your config review.

KIP-1271 + KIP-1285: Headers in state stores. Record headers can now be stored in and retrieved from state stores, exposed through both the Processor API (KIP-1271) and the DSL (KIP-1285). Teams using headers to carry routing metadata, tracing context, or correlation IDs will find this closes a meaningful gap — previously those headers had to be stripped or handled separately before writing to state.

KIP-1270: ProcessingExceptionHandler for GlobalKTable threads. Exception handling now extends to GlobalKTable threads via the processing.exception.handler.global.enabled configuration. Before 4.3, exceptions in the global thread could cause opaque failures that were difficult to surface cleanly. This is a reliability improvement for any topology that uses global state.

KIP-1035: StateStore changelog offset management. Adds methods to the StateStore API for changelog offset management. This is an internal change primarily relevant to custom StateStore implementations. If you maintain a custom state store, review the updated API.

KIP-1247: Bytes utility class promoted to public API. The Bytes class is now part of the public API and will appear in the Javadoc. Minor, but useful for teams that have been importing it from internal packages.

KIP-1250: In-memory state store size metrics. New metrics tracking the number of keys in in-memory state stores. If your Streams topologies use in-memory stores and you have limited visibility into store growth, this is a monitoring addition worth wiring up.

KIP-1244: streams-scala module deprecated. If your team uses the Scala DSL for Kafka Streams, 4.3 is the start of the deprecation clock. Marked for removal in Kafka 5.0. Plan migration now rather than at the 5.0 deadline.

Kafka Connect

KIP-1239: RemoteClusterUtils.translateOffsets() now accepts multiple consumer groups in a single call. For teams running MirrorMaker 2 across clusters with many groups, this is a meaningful quality-of-life improvement that reduces the number of round trips required for offset translation workflows.

KIP-1273: A new ConnectPlugin interface standardises methods across all Kafka Connect plugin types. Teams building or maintaining custom connectors should review this — it establishes a common contract that all plugin types now implement.

KIP-1280: MirrorMaker metric names are being updated. The new metric.names.formats configuration on MirrorSourceConnector and MirrorCheckpointConnector lets you opt into the new metric names. Existing metric names are deprecated and will be removed in Kafka 5.0. If you have dashboards or alerts built on MirrorMaker metrics, schedule the migration before 5.0.

Deprecation roundup

Everything deprecated in 4.3.0 in one place. Set calendar reminders. Kafka 5.0 removals tend to arrive faster than teams expect.

What	KIP	Removal target
streams-scala module	KIP-1244	Kafka 5.0
group.coordinator.rebalance.protocols broker config	KIP-1237	Kafka 5.0
Existing MirrorMaker metric names	KIP-1280	Kafka 5.0
Classic consumer rebalance protocol	KIP-1274 (Phase 1 — warning only)	Future release

‍

The classic rebalance protocol deprecation is Phase 1 only in 4.3: no removal yet, just logging. The 5.0 items are the ones that need concrete action.

Upgrade readiness checklist

A practical checklist for teams preparing a 4.3.0 upgrade. Work through this before scheduling your maintenance window.

The operator signal

Kafka 4.3.0 continues a pattern visible across the 4.x series. The feature set is not about new consumer semantics or Streams DSL additions — it is about the operational experience of running Kafka at scale. Broker cordoning, retention headroom metrics, coordinator performance, tiered storage correctness, KRaft fetch controls: these are the kinds of improvements that accumulate into a platform that operations engineers can rely on.

Teams evaluating Kafka adoption or upgrade timing should note that 4.x has been consistently improving the operational story with each release. 4.3.0 continues that pattern.

Keeping up with a release cadence like this is easier with good cluster visibility. If you are upgrading to 4.3 and want to make the most of the new metrics and operational improvements, Kpow by Factor House gives you an observability layer that covers partition size headroom, consumer group health, tiered storage state, and broker-level metrics in a single interface. You can try it for yourself with a free 30-day trial.

Resources

How Apple uses Apache Kafka in production

Factor House — Tue, 02 Jun 2026 00:00:00 GMT

Apple runs Apache Kafka as a shared internal platform serving multiple engineering teams across the company. Since at least 2018, Apple’s engineers have presented at Kafka Summit on the operational realities of running Kafka as a managed, multi-tenant service at the scale you would expect from one of the world’s largest technology companies. Their publicly documented work spans partition management at scale, a custom zero-data-movement balancing algorithm, Kubernetes-native tiered storage, and an ongoing migration from password-based authentication to mTLS.

Company overview

Apple designs hardware, software, and services including the iPhone, iPad, Mac, Apple Watch, iCloud, Apple Music, Apple TV+, and the App Store. Those services operate at a scale that requires low-latency, fault-tolerant data infrastructure: iCloud alone serves hundreds of millions of users, while the App Store processes purchases across more than 175 countries.

Kafka sits within the data infrastructure layer that keeps these services fed with real-time event data. The iCloud Data organisation, for example, uses stream-processing technologies that include Kafka for building scalable data pipelines. Apple began operating Kafka as a multi-tenant internal service no later than 2018, when they first presented on the architecture at Kafka Summit London.

Key Kafka milestones:

2018: Apple presents “Kafka as a Service: A Tale of Security and Multi Tenancy” at Kafka Summit London, confirming that Kafka is operated as a managed internal platform with multi-tenancy and authentication controls.
2019: Noa Resare presents operational lessons from running Kafka at large scale at Kafka Summit New York, covering partition reassignment challenges at scale.
2023: Haochen Li and Yaodong Yang present a zero-data-movement partition placement algorithm at Kafka Summit London and deploy it to production as a Topic Operator.
February 2024: Tiered storage on Kafka 3.6 reaches production, following a year-long integration effort using Strimzi on AWS EKS.
2024: Apple’s tiered storage contribution is merged into Strimzi 0.40.0, making native tiered storage support available to the wider community.
2025: Gaurav Narula presents a hybrid mTLS migration strategy at Confluent Current, describing how Apple is moving from SASL password authentication to short-lived X.509 certificates.

Apple’s Kafka use cases

Apple’s Kafka deployment centres on internal data infrastructure rather than a single named product use case. Their engineering talks describe Kafka as a platform shared across teams, which is consistent with how large technology companies typically use Kafka at enterprise scale.

The iCloud Data organisation cites Kafka as part of the technology set used to build scalable, timely data pipelines for internal teams that ship product features. Apple’s job postings for iCloud data engineering roles reference Kafka alongside Apache Flink, Kafka Streams, and Apache Spark Streaming, suggesting that Kafka is used for real-time data movement between services as well as stream processing.

The Kafka as a Service model Apple described in 2018 implies a broad adoption pattern: rather than individual teams standing up their own clusters, a central platform team manages shared clusters, and product teams onboard as tenants.

Scale and throughput

Apple does not publish aggregate cluster statistics publicly. The figures available come directly from conference presentations:

Clusters: Multiple Kafka clusters operated as a shared internal service; exact count not publicly disclosed.
Brokers per cluster: 6 brokers confirmed in the 2024 tiered storage production environment.
Per-broker throughput: Measured in the Strimzi/tiered storage test environment: 270 MB/s with a single consumer; 250 MB/s with three consumers; 90 MB/s with five consumers across 400 topics.
Partitions: Described as reaching a volume where large-scale reassignment operations are a regular operational concern; exact count not publicly disclosed.
Storage: EBS-backed local storage per broker, supplemented by S3 tiered storage for older data following the 2024 production rollout.

Apple’s Kafka architecture

Apple’s Kafka clusters run on Kubernetes using the Strimzi operator on AWS EKS. Broker nodes use Amazon EBS volumes for local storage. Helm is used to manage Strimzi deployments across environments. This Kubernetes-native approach means cluster lifecycle operations, scaling, and upgrades are managed through Kubernetes custom resources rather than direct broker administration.

Since February 2024, Apple has operated Kafka with tiered storage (KIP-405) in production. Older log segments are offloaded to Amazon S3, separating data retention from local broker capacity. The Remote Log Metadata Manager (RLMM) uses an internal Kafka client secured with SSL/TLS using a PKCS12 keystore and truststore, configured via the rsm.config prefix.

Security is layered. Apple has operated Kafka with SASL authentication and ACL-based authorisation since at least 2018. As of 2025, the platform team is actively migrating toward mTLS using short-lived X.509 certificates, driven by the security advantages of certificate-based identity over long-lived passwords in a multi-tenant environment.

Producer and consumer architecture

Apple’s partition placement algorithm (described below) is built on the assumption that data ingestion workloads assign events randomly to partitions with no ordering requirements, and that all partitions from a topic are consumed evenly. This is consistent with a high-throughput event ingestion pattern where producers distribute load across partitions and consumers process each partition independently.

Consumer lag monitoring is handled by Burrow, which is deployed alongside CMAK for cluster administration.

Operating model

The platform is provided to internal teams as a managed service. This means the central Kafka team is responsible for cluster provisioning, upgrades, monitoring, and operational tooling, while product teams interact with the platform as tenants. Multi-tenancy brings authentication, authorisation, quota management, and topic namespace concerns, all of which Apple has addressed in successive talks.

Special techniques and engineering innovations

Zero-data-movement cluster balancing

Apple engineers Haochen Li and Yaodong Yang developed a partition placement strategy that achieves optimal cluster load balancing without moving any data between brokers. Traditional rebalancing tools, including Cruise Control, rebalance clusters by reassigning partitions, which involves copying data from one broker to another. Apple’s approach avoids that cost entirely by placing partitions correctly at the point of creation.

The algorithm enforces two properties across every topic: the number of replicas per broker is equal, and the number of leader replicas per broker is equal. The formula applied is partition_count = scale_number * broker_count, where scale_number is the number of leader replicas per broker for a topic. This is applied at topic creation, when partition count changes, and when brokers are added to a cluster.

The strategy is implemented as a Topic Operator and has been deployed to production. The result, as described by Haochen Li and Yaodong Yang, was well-balanced clusters with measurable storage cost savings and no performance degradation. The team submitted a Kafka Improvement Proposal (KIP) to upstream the approach to the Apache Kafka project.

Tiered storage on Strimzi

Apple ran a structured integration effort to bring Kafka tiered storage (KIP-405) into production on their Strimzi-managed clusters. The work began in February 2023 with an evaluation using the KIP-405 author’s Kafka 2.8.x development branch and progressed through a series of phases:

April 2023: Prototype integration of Strimzi and S3 plugin.
September 2023: Kafka 2.8 development branch fully tested.
November 2023: Kafka 3.6.0 dogfooding phase.
February 2024: Production rollout on Kafka 3.6.

The integration required upgrading Strimzi from 0.27.x to 0.38.x and Java from 11 to 17. Apple contributed native tiered storage support back to the Strimzi project, which was merged in Strimzi 0.40.0. This means you can now configure a custom remote storage manager in Strimzi via standard Kafka custom resource definitions, a capability that originated from Apple’s production work.

Performance tuning for tiered storage required several configuration adjustments: max.fetch.wait.time was increased to tolerate the additional latency of remote object fetches; remote.log.reader.threads and thread pool sizes were tuned for throughput; log segment sizes were optimised to keep RLMM metadata overhead manageable; and AWS multipart uploads were configured to run concurrently alongside range fetch APIs for parallel I/O.

mTLS migration

Gaurav Narula from Apple described a hybrid migration strategy for moving from SASL password authentication to mTLS. The challenge with any mTLS migration in a large shared platform is that you cannot require all clients to switch simultaneously without breaking production workloads.

Apple’s approach adds mTLS support directly to the SASL listener, so both authentication methods are accepted on the same port. Existing clients continue to authenticate with passwords while new or migrated clients present X.509 certificates. KafkaPrincipal identities are preserved across both methods, which means ACLs and quota configurations do not need to change when a client migrates. For inter-broker communication, each broker is configured with distinct server and client certificates.

The use of short-lived X.509 certificates addresses two specific weaknesses of SASL passwords: credential leak impact is bounded by certificate lifetime, and certificate-based authentication is not vulnerable to brute-force attacks.

Operating Kafka at scale

Kubernetes-native operations: Strimzi handles cluster provisioning, rolling upgrades, and configuration changes through Kubernetes custom resources. This removes the need for manual broker-level administration and integrates Kafka lifecycle management with existing container platform tooling.

Monitoring and observability: Burrow provides consumer lag monitoring. CMAK provides a cluster administration interface. Both run alongside the Strimzi operator in the Kafka platform stack.

Cluster rebalancing: Cruise Control is deployed for automated partition load balancing, though Apple’s zero-data-movement algorithm addresses a significant portion of imbalance at topic-creation time rather than reactively.

Partition reassignment at scale: Noa Resare’s 2019 talk described the operational challenge of managing large-scale partition reassignments: tracking progress, understanding the impact on producers and consumers during broker restarts, and applying debugging and mitigation strategies when reassignments do not proceed as expected. At large partition counts, these operations become significant engineering events rather than routine maintenance.

Upgrade strategy: The tiered storage work illustrates Apple’s approach to major upgrades: start with an evaluation phase on a development branch, build a prototype integration, run dogfooding before committing to production, and execute the production rollout as a distinct step. The Kafka 2.8 to 3.6 upgrade and the Strimzi 0.27 to 0.38 jump were managed as part of the tiered storage project rather than independently.

Challenges and how they solved them

Cluster imbalance without data movement

As Kafka clusters grow or traffic patterns shift, broker resource utilisation becomes uneven. CPU, disk, and leader distribution diverge, leading to over-provisioned capacity on some brokers and bottlenecks on others. The conventional remedy is partition reassignment, but at scale this involves copying large volumes of data between brokers, creating I/O pressure and operational risk.

Apple developed the zero-data-movement placement algorithm to solve this at the source. By enforcing balanced placement at topic-creation time, the problem of runtime imbalance is largely avoided. When it does occur, during cluster expansion or partition scaling, the same algorithm is applied to achieve balance without data movement.

Cost of EBS-backed data retention

Storing all log data on EBS volumes attached to broker nodes means storage costs scale directly with retention windows. For teams that need to retain large event histories for backfill, reprocessing, or compliance, this becomes a significant expense. Tiered storage decouples retention from broker capacity: older segments are offloaded to S3, which is substantially cheaper than EBS, while brokers remain sized for hot data and throughput.

The challenge Apple faced was that Strimzi did not natively support tiered storage configuration at the time. Solving this required building the integration, testing it across multiple Kafka versions, and contributing the result back to Strimzi before production deployment.

Password-based authentication in a multi-tenant service

In a shared Kafka platform with many producer and consumer clients, SASL password credentials create a persistent risk surface. Long-lived credentials can be leaked and remain valid indefinitely. They are also vulnerable to brute-force attacks in a way that certificate-based authentication is not.

The transition to mTLS eliminates these risks, but migrating a large multi-tenant platform is not straightforward: you cannot break existing clients while switching authentication mechanisms. Apple’s hybrid SASL-plus-mTLS approach on a shared listener resolves this by making the migration incremental and non-disruptive.

Large-scale partition reassignment

At the partition counts Apple operates, reassigning partitions, whether to recover from broker imbalance, decommission a broker, or respond to capacity changes, becomes a complex operational exercise. Tracking progress across many concurrent reassignments, managing the impact on producers and consumers, and debugging failures when they occur requires both tooling and operational discipline. Noa Resare’s 2019 talk shared the lessons Apple accumulated from running these operations repeatedly at scale.

Full tech stack

Category	Tools	Notes
Message broker	Apache Kafka 3.6+	Core streaming platform; tiered storage available from 3.6
Kafka operator	Strimzi	Kubernetes-native cluster lifecycle management on EKS
Container platform	AWS EKS	Kubernetes runtime for broker and operator workloads
Local broker storage	Amazon EBS	Block storage attached to broker nodes
Tiered (remote) storage	Amazon S3 (via KIP-405 plugin)	Cost-effective long-term event retention
Packaging	Helm	Strimzi deployment management
Cluster rebalancing	Cruise Control	Automated partition load balancing
Consumer lag monitoring	Burrow	Consumer group lag visibility
Cluster administration	CMAK	Kafka cluster management UI
Authentication	mTLS (X.509 / PKCS12), SASL	Client and inter-broker authentication; migrating toward mTLS
JVM	Java 17	Kafka broker and tooling runtime

Key contributors

Noa Resare — Apple engineer; spoke on operating Kafka at large scale, Kafka Summit NY 2019. Talk recording
Haochen Li — Apple software engineer; co-developed the zero-data-movement partition placement algorithm, Kafka Summit London 2023. Session
Yaodong Yang — Apple software engineer; co-developed the zero-data-movement partition placement algorithm, Kafka Summit London 2023.
Bo Gao — Apple engineer; led the Strimzi tiered storage integration project, Kafka Summit London 2024. Session
Lixin Yao — Apple engineer; co-led the Strimzi tiered storage integration project, Kafka Summit London 2024.
Gaurav Narula — Apple engineer; presented the hybrid mTLS migration approach, Confluent Current 2025. Session

Key takeaways for your own Kafka implementation

Address cluster imbalance at creation time, not at rebalance time. Apple’s zero-data-movement algorithm enforces balanced partition and leader placement when topics are created or scaled. This reduces or eliminates the need for reactive rebalancing operations, which are expensive in large clusters.
Tiered storage is a practical path to decoupling retention from broker capacity. Apple’s production journey from prototype (February 2023) to production (February 2024) demonstrates that tiered storage on Strimzi is achievable with deliberate testing phases, though it requires careful performance tuning, particularly around fetch latency and thread pool sizing.
Migrating authentication mechanisms in a shared platform requires a hybrid transition period. Adding mTLS to the existing SASL listener rather than replacing it allows clients to migrate at their own pace. Preserving KafkaPrincipal identities means existing ACLs remain valid throughout the transition.
If you operate Kafka as a managed service for internal teams, multi-tenancy concerns appear early. Security enforcement (authentication, authorisation, quotas), partition reassignment at scale, and cluster observability all become harder in a shared environment than in a single-team deployment. Apple’s public talks trace a consistent investment in each of these areas over several years.
Contributing upstream reduces long-term maintenance burden. Apple’s Strimzi tiered storage integration was contributed back to the project and merged in Strimzi 0.40.0. If you are building integrations on top of open source Kafka tooling, upstream contributions mean you are no longer carrying a fork.

Sources and further reading

Noa Resare, Apple — “Experiences Operating Apache Kafka® at Scale”, Kafka Summit NY 2019
Apple — “Kafka as a Service: A Tale of Security and Multi Tenancy”, Kafka Summit London 2018, video‍
Haochen Li and Yaodong Yang, Apple — “Balance Kafka Cluster with Zero Data Movement”, Kafka Summit London 2023
Bo Gao and Lixin Yao, Apple — “Leveraging Tiered Storage in Strimzi-Operated Kafka for Cost-Effective Streaming Applications”, Kafka Summit London 2024
Gaurav Narula, Apple — “Leave Your Passwords Behind: Embracing mTLS in Kafka”, Confluent Current 2025
dttung2905/kafka-in-production — Apple entries

If you work with a Kafka cluster and want visibility into consumer lag, broker health, and topic throughput without standing up additional infrastructure, try Kpow free for 30 days. You can connect it to any Kafka cluster in minutes and deploy via Docker, Helm, or JAR.

How Barclays uses Apache Kafka in production

Factor House — Tue, 02 Jun 2026 00:00:00 GMT

Most banks that adopt Apache Kafka keep it in the cloud or on commodity Linux hardware. Barclays does something less common: the bank runs separate Confluent Kafka deployments on Amazon EKS and on IBM Z-Linux, bringing the streaming layer into direct contact with mainframe workloads rather than bridging them through a connector from outside. That architectural choice sits alongside a Solace PubSub+ messaging layer that has been in place since 2009, creating a layered messaging estate that reflects the complexity of a global bank operating across front, middle, and back office.

Barclays is listed on the official Apache Kafka Powered By page with the description “Barclays utilizes Kafka for streaming and analytical information.” That is the bank’s only public characterisation of its Kafka usage. The rest of what is known comes from official job postings, vendor case studies, and industry award citations, all of which point to Confluent Kafka as the chosen distribution and to an engineering model built around SRE principles and Infrastructure as Code.

Company overview

Barclays is a British multinational bank headquartered in London, operating across retail banking, corporate banking, and investment banking. The bank serves tens of millions of customers and processes millions of financial transactions daily. Its investment banking arm, Barclays Investment Bank, operates across equities, fixed income, foreign exchange, and commodities.

In 2025, Barclays won the Databricks Data Intelligence Financial Services Industry Award for adopting “a scalable and unified data platform that supports global trade analytics, complex data warehousing use cases and real-time insights,” with the platform delivering “real-time streaming capabilities, native machine learning, generative AI capabilities and comprehensive governance.”

Key milestones in the bank’s messaging and streaming history:

2009: Barclays deploys Solace PubSub+ as its enterprise-wide high-speed messaging platform, integrating applications across front, middle, and back office.
2025: Barclays wins the Databricks Data Intelligence Financial Services Industry Award for its unified data platform with real-time streaming capabilities.
April 2026: Barclays posts simultaneous open roles for Confluent Kafka engineers across three distinct specialisms: cluster management, AWS deployment, and IBM Z-Linux deployment.

Barclays’ Kafka use cases

Barclays’ public description of its Kafka usage is deliberately broad: streaming and analytical information. The bank has not published engineering blog posts that map specific products or teams to Kafka pipelines, so the breakdown here is limited to what can be inferred from official sources.

The job postings describe engineering teams responsible for “event-driven architectures (EDA) using industry-standard messaging and streaming platforms,” with ownership of “Kafka-centric systems” framed as a platform engineering concern rather than a product-specific one. The phrasing suggests Kafka is positioned as shared infrastructure serving multiple consuming teams, with a central platform team responsible for cluster operations, capacity planning, and governance.

In Barclays’ post-trade technology estate, a microservices-based settlement platform handles cash settlement processing. A Camunda case study attributed to Shakir Ahmed (Director of Operations Technology and Strategy) and Larisa Kvetnoy (MD Post Trade Technology) describes this system as using a message bus for distributed microservices communication, targeting 500,000 or more daily settlement processes and handling T+1 regulatory settlement requirements. The case study does not explicitly name Kafka as the message bus; it is included here as context for the broader event-driven architecture at the bank.

Barclays’ Kafka architecture

The most distinctive aspect of Barclays’ Kafka setup is the split between two separate deployment environments, each with its own engineering specialism.

AWS cluster

Barclays runs Confluent Kafka on Amazon EKS. The cluster topology supports multi-region AWS deployments in both active-active and active-passive configurations. Engineers are required to design “multi-region AWS architectures, including active-active and active-passive deployments, with clear failover, disaster recovery, and data consistency strategies.” Kafka workloads run as containerised applications on Kubernetes, using StatefulSets, persistent volumes, services, and ingress controllers.

The active-active configuration is the more demanding of the two patterns: both regional clusters serve traffic simultaneously, which requires consistent offset management across regions, careful handling of replication lag, and a defined approach to data consistency for consumers that may be reading from either cluster. Barclays’ job requirements call for explicit expertise in this area. The specific replication mechanism (MirrorMaker 2, Confluent Replicator, or an alternative) is not named in available sources.

IBM Z-Linux cluster

A separate Confluent Kafka deployment runs on IBM Z-Linux. This is a less common pattern. The typical approach for mainframe-to-Kafka integration is to run Kafka on separate infrastructure and bridge the gap using Kafka Connect with an IBM MQ connector, IBM Data Gate for Confluent, or IBM’s Open Enterprise SDK for Apache Kafka. Barclays instead co-locates Confluent Kafka brokers on the IBM Z platform itself, running on the Linux partition of the mainframe.

The rationale for this approach is not documented in public sources, but the engineering case is straightforward: applications running natively on IBM Z can produce and consume Kafka events without traversing a network boundary to reach a separate Kafka cluster. For latency-sensitive workloads on mainframe systems, removing that hop is meaningful. The Z-Linux role requires understanding of “hybrid platform interoperability within regulated enterprise environments” and compliance with Barclays’ UK standards.

Coexisting messaging layer

Solace PubSub+ has been Barclays’ enterprise-wide high-speed messaging platform since 2009. The bank consolidated its messaging needs onto the Solace platform to integrate applications across front, middle, and back office, simplifying infrastructure and reducing licensing, datacenter, development, and support costs. The relationship between the Solace layer and the Confluent Kafka deployments is not described in any available source. Both appear to be active parts of Barclays’ messaging estate.

Producer and consumer architecture

Barclays’ Kafka engineering standards cover both sides of the producer-consumer boundary. On the producer side, engineers are expected to understand event ordering, partitioning strategy, replay capability, and exactly-once vs at-least-once semantics. On the consumer side, offset management and schema evolution are listed as required competencies. Confluent Schema Registry is part of the deployment, with schema evolution identified as an area requiring hands-on expertise. The schema encoding format (Avro, Protobuf, or JSON Schema) is not specified in available sources.

Kafka Connect ecosystem

Kafka Connect is in use across the Confluent Kafka deployments. The specific source and sink connectors in production are not named in any public source.

Special techniques and engineering innovations

Confluent Kafka on IBM Z-Linux. Running Kafka brokers natively on IBM Z-Linux is not a common deployment pattern. It requires a build of the JVM and Confluent runtime that targets the s390x architecture, and it places the streaming layer inside the mainframe’s security and compliance boundary rather than outside it. For a regulated institution like Barclays, keeping data within a known, auditable infrastructure footprint can simplify governance. The approach also avoids the latency introduced by external connectors.

KRaft mode. The IBM Z-Linux job posting references both ZooKeeper and KRaft mode, indicating Barclays is running or actively evaluating Confluent Kafka in KRaft (ZooKeeper-free) mode. KRaft became generally available in Kafka 3.3 and removes the external ZooKeeper dependency, simplifying cluster operations and reducing the number of processes to monitor and manage. Whether both the AWS and Z-Linux clusters are in KRaft mode is not stated in available sources.

Multi-region active-active on AWS. An active-active multi-region Kafka topology is more operationally complex than an active-passive warm-standby. Barclays’ requirements for this pattern specify explicit expertise in “failover, disaster recovery, and data consistency strategies,” suggesting the architecture is designed to stay up and serving traffic in the event of a regional AWS outage, not merely to recover to a secondary region after a failover event.

Operating Kafka at scale

Deployment model. The AWS cluster is deployed on Amazon EKS (self-managed Confluent Kafka on Kubernetes). The IBM Z-Linux cluster runs on IBM Z hardware, managed separately. Both deployments use Confluent Kafka rather than open-source Apache Kafka.

Infrastructure as Code. Cluster provisioning uses Terraform and AWS CloudFormation. The job posting describes this as enabling “repeatable, auditable, and scalable cloud provisioning.” Ansible handles configuration management and automation across the streaming platforms.

CI/CD and DevSecOps. GitLab is the source control and pipeline platform. Engineers apply DevSecOps practices, integrating security controls into the CI/CD pipeline rather than treating them as a separate gate.

SRE practices. Barclays applies “BUK Service First and SRE principles” to its Kafka platform. Engineers are accountable for platform stability, performance tuning, capacity planning, and incident response. DORA metrics, covering Deployment Frequency, Lead Time for Change, Change Failure Rate, and Mean Time to Recovery, are the stated mechanism for tracking delivery and operational health.

Testing. The engineering standard for Kafka services includes contract testing with PACT, unit testing with JUnit, integration and performance testing with JMeter, and mutation testing. Contract testing between producers and consumers is worth noting specifically: it catches schema or payload incompatibilities before deployment rather than at runtime, which reduces the risk of consumer failures caused by upstream producer changes.

Monitoring and alerting. The specific monitoring stack (metrics exporters, dashboards, alerting tools) is not named in any available source. The SRE accountability model implies alert-based on-call rotation, but the tooling is not described publicly.

Full tech stack

Category	Tools	Notes
Message broker	Confluent Kafka	Two separate deployments: AWS/EKS and IBM Z-Linux
Enterprise messaging	Solace PubSub+	Enterprise-wide high-speed messaging for front, middle, and back office; in place since 2009
Schema registry	Confluent Schema Registry	Schema evolution required; encoding format not specified in public sources
Connectors	Kafka Connect	Specific source and sink connectors not named publicly
Container orchestration	Amazon EKS (Kubernetes)	StatefulSets, persistent volumes, ingress controllers; used for AWS Kafka deployment
Infrastructure as Code	Terraform, AWS CloudFormation	Repeatable, auditable cloud provisioning
Configuration management	Ansible	Automation across cloud and streaming platforms
CI/CD	GitLab	Source control and DevSecOps pipelines
Containerisation	Docker	Local development environment
Workflow orchestration	Camunda Platform, Camunda 8	Post-trade settlement processing and T+1 regulatory compliance
Data platform	Databricks	Unified platform for trade analytics, warehousing, ML, and generative AI
Testing	PACT, JUnit, JMeter	Contract, unit, and performance testing for Kafka services; mutation testing also applied
Mainframe platform	IBM Z-Linux (zLinux)	Host environment for IBM Z-resident Confluent Kafka deployment

Key contributors

Name	Title	Contribution
Shakir Ahmed	Director of Operations Technology and Strategy	Named in the Camunda case study discussing Barclays’ microservices-based post-trade settlement platform and T+1 compliance programme
Larisa Kvetnoy	MD Post Trade Technology	Named alongside Shakir Ahmed in the Camunda case study; responsible for Barclays’ post-trade technology estate

Key takeaways for your own Kafka implementation

Running Kafka where your data lives reduces integration complexity. Barclays’ decision to deploy Confluent Kafka on IBM Z-Linux rather than bridging the mainframe through an external connector keeps the streaming layer inside the same infrastructure boundary as the workloads it serves. If you have latency-sensitive applications on a mainframe or another non-commodity platform, consider whether bringing the broker closer is viable before adding a connector hop.

Multi-region active-active is a design commitment, not just a configuration choice. Barclays’ AWS deployment explicitly covers active-active and active-passive patterns as separate engineering concerns. An active-active topology requires upfront decisions about consumer group management, replication, and data consistency that cannot be retrofitted easily. If you need this level of resilience, design for it from the start rather than treating it as a later scaling step.

Treat Kafka platform operations as an SRE discipline. Barclays applies formal SRE practices to its Kafka clusters, including DORA metrics, capacity planning accountabilities, and incident management runbooks. Kafka is not a fire-and-forget system at scale; defining the same operational standards you apply to application services is a practical way to maintain reliability as the platform grows.

Contract testing between producers and consumers is worth the overhead. Including PACT-based contract testing in the Kafka engineering standard catches schema incompatibilities before deployment, which matters more as the number of independent producer and consumer teams grows. The cost of running contract tests in CI is lower than the cost of debugging a consumer failure caused by an upstream schema change in production.

Multiple messaging layers can coexist if their roles are distinct. Barclays runs both Solace PubSub+ and Confluent Kafka as active parts of its messaging estate. That is not a transitional state; PubSub+ has been in place since 2009. If you are evaluating whether to consolidate onto a single messaging technology, Barclays’ architecture suggests that retaining a low-latency broker for specific front-office use cases alongside a general-purpose streaming platform may be a deliberate long-term choice rather than a migration that never completed.

Sources and further reading

Primary sources:

Apache Kafka Powered By — Barclays entry: https://kafka.apache.org/powered-by
Barclays Careers — Software Engineer, Confluent Kafka Management (JR-0000103144): https://search.jobs.barclays/job/bengaluru/software-engineer-confluent-kafka-management/13015/93793235776
Barclays Careers — Software Engineer, Confluent Kafka on IBM Z-Linux (JR-0000103136): https://barclays.wd3.myworkdayjobs.com/en-US/External_Career_Site_Barclays/job/Software-Engineer---Confluent-Kafka-on-IBM-Z-Linux_JR-0000103136
Solace customer case study — Barclays: https://solace.com/company/customers/barclays/
Camunda case study — Barclays post-trade settlement: https://camunda.com/case-study/barclays/
Databricks 2025 Data Intelligence Industry Awards: https://www.databricks.com/blog/announcing-winners-2025-data-intelligence-industry-awards

Try Kpow for your Kafka clusters:

If you manage Confluent Kafka clusters and want deeper visibility into consumer lag, topic throughput, and schema registry state, Kpow connects to any Kafka cluster in minutes. You can trial it free for 30 days and deploy via Docker, Helm, or JAR.

How Cash App uses Apache Kafka in production

Factor House — Tue, 02 Jun 2026 00:00:00 GMT

Cash App runs Apache Kafka as the primary eventing backbone for its distributed services, operated by an internal PubSub platform team that also provides SQS for job queues and NATS for real-time client updates. Two aspects of Cash App’s Kafka story are worth studying in detail: the evently-cloud bridge service, which absorbs Kafka client initialisation overhead at a proxy layer and reduced its pod count from 100 to 15 without SLA regression; and a data-centric encryption model that assigns distinct AWS KMS-backed keys per Kafka topic rather than per service, operating at more than 8 TB of encrypted data per day across Kafka and gRPC transport.

Company overview

Cash App is a consumer financial services product developed by Block Inc., offering peer-to-peer payments, banking, investing, and Bitcoin services to tens of millions of users primarily in the United States and the United Kingdom. Engineering at Cash App operates independently from Block’s other brands, with shared infrastructure provided by dedicated platform teams.

Kafka’s role at Cash App is as the central asynchronous message bus. The PubSub platform team owns the Kafka cluster and provides it alongside complementary messaging primitives as a managed internal offering: Kafka for durable event streaming, SQS for job queues, and NATS for low-latency real-time client updates. The team also operates Kpow as the Kafka management UI.

Key Kafka milestones:

Ongoing (pre-2022): Cash App runs Kafka for asynchronous event pub/sub; the evently-cloud Kotlin bridge service serves legacy services that cannot manage Kafka client connections directly.
December 2024: Yoav Amit, Gelareh Taban, and Matthew Miller publish “Encryption using data-specific keys,” documenting the shift from service-centric to data-centric encryption across Kafka and gRPC transport. Cash App is encrypting more than 8 TB/day at this point.
Post-2022 (Block acquisition of Afterpay): Cash App’s Delta Lake on Databricks adoption enables Spark Declarative Pipelines that read streaming data directly from Kafka for machine learning and data science workloads.

Cash App’s Kafka use cases

Asynchronous event pub/sub

The primary use of Kafka at Cash App is as the durable pub/sub backbone for distributed services. Engineering teams across Cash App produce and consume events through a managed platform offering rather than owning Kafka client connections themselves. This keeps operational overhead concentrated in the PubSub platform team while providing consistent eventing semantics to product teams.

Legacy service integration via evently-cloud

Not all Cash App services were built to manage long-lived Kafka client connections. The evently-cloud service fills this gap: it is a Kotlin service that exposes a REST API for fetching events from Kafka topics at specified offsets. Legacy services interact with Kafka through HTTP calls to evently-cloud, which handles client lifecycle, caching, and offset management internally. Alec Holmes documented this architecture on the Cash App Code Blog.

Machine learning and data science pipelines

Cash App uses Databricks Spark Declarative Pipelines to read streaming data from Kafka, feeding machine learning and data science workloads. The data lake uses a medallion architecture — bronze, silver, and gold tiers — on Delta Lake, governed by Unity Catalog and AWS Glue. This allows ML teams to work with Kafka-sourced event data without managing stream processing infrastructure directly.

Encrypted data transport

Cash App treats Kafka as part of its data transport infrastructure alongside gRPC. Both are subject to the same application-layer encryption model: encryption keys are scoped per Kafka topic rather than per service, and any producer or consumer must hold the relevant topic key via AWS IAM. This model was documented by Yoav Amit, Gelareh Taban, and Matthew Miller in December 2024.

Scale and throughput

Encryption volume: More than 8 TB of data per day encrypted at the application layer across Kafka and gRPC transport (Yoav Amit, Gelareh Taban, Matthew Miller — December 2024).
evently-cloud pod count: Reduced from 100 pods to 15 pods after request-affinity optimisation, with no SLA regression (Alec Holmes).
Total fleet size: Topic count, partition count, and cluster count are not publicly disclosed by Cash App.

Cash App’s Kafka architecture

Deployment

Cash App’s Kafka cluster runs on AWS. The evently-cloud service and its surrounding infrastructure run on Kubernetes using Istio as the service mesh. Before Istio, each service used a dedicated Elastic Load Balancer; after migration, load balancing is handled client-side via Envoy sidecars.

evently-cloud: Kafka bridge service

Kafka clients take several seconds to initialise and seek to specific topic offsets. In a service that creates a new client per request, this latency is incurred on every API call. evently-cloud avoids this by maintaining one long-lived cached Kafka client per topic consumer. When a request arrives, the service checks its cache; if a client already exists for that topic, the request is handled immediately without initialisation overhead.

The caching strategy only works if requests for the same topic consistently reach the same pod. Cash App solved this with Istio-based request affinity: Kafka topic identifiers are passed as HTTP headers, and Istio’s consistent-hashing load balancing routes requests for the same topic to the same pod. The affinity rule is topic-level, not session-level, which means the benefit applies across all consumers of a given topic regardless of which service made the request.

After deploying request affinity, the evently-cloud deployment was reduced from 100 pods to 15 pods. Cache hit rates improved, and the SLA was unaffected.

Per-topic encryption at application layer

Cash App’s encryption model shifted from a service-centric design — where services held keys tied to their own identity — to a data-centric design, where encryption keys are associated with the data itself. For Kafka, this means a distinct Tink keyset per topic. Any workload that produces or consumes from a topic must have explicit access to that topic’s key via AWS IAM policies and role chaining.

Keys are stored as Tink keysets in S3 buckets, backed by AWS KMS Customer Managed Keys. The practical effect is that a compromised service cannot decrypt data from topics outside its own access grants, even if it can reach the Kafka broker. The same model applies to gRPC transport, making it a consistent encryption boundary across Cash App’s data infrastructure. Block was encrypting more than 8 TB/day across both channels as of December 2024.

Producer and consumer architecture

Producers and consumers in Cash App’s primary Kafka workloads use Kafka clients managed either directly by services with the capability to do so, or indirectly through evently-cloud for services that cannot. In the encrypted pipeline, Tink keysets are loaded by producers before serialisation and by consumers before deserialisation, with the key lookup resolved via AWS KMS at access time.

Special techniques and engineering innovations

Topic-level client caching with Istio affinity

The combination of Kafka client caching in evently-cloud and Istio consistent-hashing affinity at the load balancer layer is an effective pattern for bridging Kafka’s stateful client model with a stateless HTTP API. The Kafka client’s initialisation cost is paid once per topic per pod lifecycle, not once per request. The Istio affinity rule ensures the cached client is reused rather than abandoned when new requests arrive. The 85% pod reduction (100 to 15) is a concrete outcome of applying this pattern.

Data-centric encryption keys scoped to Kafka topics

Most application-layer encryption in distributed systems is service-centric: a service holds a key and uses it for all its interactions. Cash App’s model is data-centric: the key belongs to the topic (or the gRPC endpoint), not the service. The implication for Kafka is that topic access control is enforced at the key level as well as at the broker ACL level. Rotating a key revokes access to historical data for any service that cannot access the new key. This is a meaningful security boundary in a financial services context where data sensitivity varies significantly by topic.

Operating Kafka at scale

Deployment model: Cash App’s Kafka cluster runs self-managed on AWS. The PubSub platform team operates the cluster alongside SQS and NATS as a unified internal messaging platform. There is no public statement from Cash App about using a managed Kafka service such as Amazon MSK or Confluent Cloud for its primary cluster.

Kafka management UI: The PubSub platform team operates Kpow as the Kafka management interface. This is the same team responsible for provisioning and maintaining the Kafka cluster.

Encryption key management: Tink keysets are stored in S3 and backed by AWS KMS Customer Managed Keys. AWS IAM policies govern which workloads can access which topic keys. The data safety levels framework, documented separately by Block’s security team, governs classification and retention policies for data that flows through Kafka.

Developer experience: The evently-cloud service reduces the barrier to Kafka adoption for engineering teams that are not equipped to manage persistent Kafka client connections. Teams that can manage clients directly do so; those that cannot use the REST API. The PubSub platform team maintains both paths.

Service mesh: Istio (Envoy sidecars) handles service-to-service communication for evently-cloud and the surrounding Kubernetes workloads. The migration from per-service ELBs to Istio client-side load balancing was what enabled the request-affinity optimisation that reduced evently-cloud’s pod footprint.

Challenges and how they solved them

Kafka client initialisation latency in evently-cloud

Kafka clients take seconds to initialise and seek to specific topic offsets. For a service that creates a new client per request, this overhead dominates response time. The first approach — running more pods so each one handled fewer requests — did not solve the root cause. The actual fix was to cache clients at the pod level and use Istio consistent-hashing to ensure requests for the same topic always route to the same pod. Once each pod’s client cache is warm, initialisation overhead disappears from the hot path. Pod count fell from 100 to 15 without SLA regression.

Service-centric encryption creating implicit cross-topic access

In a service-centric encryption model, a service that holds a key can decrypt any data encrypted with that key, regardless of which Kafka topic it came from. If a service’s key is compromised, or if the service is misconfigured to consume from a topic it was not intended to read, it can access data it should not. Cash App’s response was to move to per-topic keys under a data-centric model. Each Kafka topic has its own Tink keyset; access to that keyset is granted explicitly via AWS IAM. A service can only decrypt data from the topics it has been explicitly granted access to, regardless of what other topics it can reach at the network level.

Full tech stack

Category	Tools	Notes
Message broker	Apache Kafka	AWS-hosted; self-managed by the PubSub platform team
Kafka bridge service	evently-cloud (Kotlin)	REST API bridge for legacy services; topic-level client caching
Asynchronous processing	Amazon SQS, NATS	Alongside Kafka; SQS for job queues, NATS for real-time client updates
Encryption framework	Tink keysets	Application-layer, per-topic encryption
Key management	AWS KMS (Customer Managed Keys)	Topic-level CMKs; IAM role chaining for access control
Key storage	S3	Encrypted Tink keysets stored per topic
Service mesh	Istio (Envoy sidecars)	Request affinity for evently-cloud on Kubernetes
Infrastructure	AWS (Kubernetes, ELB)	Kafka cluster and evently-cloud deployment
Stream processing	Databricks Spark Declarative Pipelines	Read streaming data from Kafka for ML workloads
Data lake storage	Delta Lake on S3	Medallion architecture (bronze, silver, gold)
Compute platform	Databricks	ETL and ML workloads on Delta Lake
Metadata / catalogue	Unity Catalog, AWS Glue	Data lake governance
Kafka management UI	Kpow	Operated by the Cash App PubSub platform team

Key contributors

Alec Holmes (with Ryan Hall and Jan Zantinge) — authored “Request Affinity with Istio,” covering the evently-cloud architecture and the optimisation that reduced pod count from 100 to 15.
Yoav Amit, Gelareh Taban, Matthew Miller — authored “Encryption using data-specific keys” (December 2024), documenting the per-topic encryption model across Kafka and gRPC transport.

Key takeaways for your own Kafka implementation

Kafka client initialisation overhead can be absorbed at a bridge layer. If you have services that need Kafka access but cannot manage persistent client connections, a proxy service with topic-level client caching and request affinity is worth considering. The operational win at Cash App was an 85% reduction in pod count for the same workload.
Istio consistent-hashing is a practical way to implement topic affinity without a dedicated load balancer per service. Passing the Kafka topic name as an HTTP header and configuring a consistent-hash affinity rule gives you stateful routing in a stateless HTTP layer with minimal infrastructure overhead.
Per-topic encryption keys are more granular than per-service keys, and the operational overhead is manageable at scale. Cash App encrypts more than 8 TB/day under this model. The main requirement is an IAM structure that grants topic key access explicitly rather than implicitly through service identity.
Separating Kafka, job queues, and real-time push into distinct primitives simplifies service design. Cash App’s PubSub platform offers Kafka for durable streaming, SQS for queue-based job processing, and NATS for sub-second client updates as separate tools with distinct semantics. This avoids the anti-pattern of using Kafka for workloads where its durability guarantees are unnecessary overhead.

Sources and further reading

Primary sources

Alec Holmes, “Request Affinity with Istio” (Cash App Code Blog): https://code.cash.app/request-affinity-with-istio
Yoav Amit, Gelareh Taban, Matthew Miller, “Encryption using data-specific keys” (December 2024): https://code.cash.app/encryption-using-data-keys

Try Kpow with your Kafka cluster

If you are monitoring a Kafka cluster at any scale, you can try Kpow free for 30 days. It connects to any Kafka cluster in minutes and deploys via Docker, Helm, or JAR.

How Datadog uses Apache Kafka in production

Factor House — Tue, 02 Jun 2026 00:00:00 GMT

Datadog processes hundreds of trillions of observability events per day across logs, metrics, and traces, and Apache Kafka sits at the centre of that pipeline. The scale of the problem meant off-the-shelf Kafka tooling and clients eventually hit limits, and Datadog’s engineering teams have built a significant amount of infrastructure around Kafka to keep up: a custom Rust client, a control plane that routes traffic across hundreds of clusters without redeployment, and open-source SRE tooling that has been in production since 2018.

Company overview

Datadog is a cloud-based observability and monitoring platform used by engineering and operations teams to monitor infrastructure, applications, and logs. Its core product ingests telemetry data from customer environments, runs it through real-time processing pipelines, and makes it queryable and alertable at low latency.

The scale of that data ingestion is what drives most of Datadog’s Kafka decisions. As of February 2025, Datadog runs hundreds of Kafka clusters, thousands of topics, millions of partitions, and hundreds of consumer groups, handling terabytes of data per second.

Kafka has been documented in Datadog’s engineering blog since at least 2018, when the SRE team open-sourced kafka-kit to handle the operational overhead of 40+ clusters processing trillions of datapoints per day. The architecture has evolved substantially since then, with the most recent major development being the Streaming Platform: a custom abstraction layer that decouples Kafka clients from physical cluster topology.

Key Kafka milestones:

August 2018: kafka-kit open-sourced, automating partition rebalancing, broker replacements, and replication throttle management across 40+ clusters
June 2019: First detailed public documentation of Datadog’s Kafka operations, covering multi-region deployments and configuration practices
May 2019 (Kafka Summit London): Balthazar Rouberol presents on running Kafka clusters in Kubernetes
May 2022: Husky event store introduced, with Kafka as its primary ingestion front-end
February 2023: Exactly-once ingestion semantics and shard routing in Husky documented publicly
February 2025: Streaming Platform architecture published, revealing the full scope of Datadog’s Kafka infrastructure and the custom Rust client
2025 (multiple posts): CDC replication platform, configuration distribution, and live process metrics pipeline all documented, showing the breadth of Kafka’s role across teams

Datadog’s Kafka use cases

Observability event ingestion

The primary role of Kafka at Datadog is as the durable buffer for the Data Platform. All incoming observability events – logs, spans, and metrics – pass through Kafka before reaching downstream storage and processing systems. Guillaume Bort, a Staff Software Engineer at Datadog, described this as the foundation of the entire observability pipeline in a February 2025 engineering blog post.

Metrics storage ingestion (Husky)

Datadog’s third-generation event store, Husky, uses Kafka as its ingestion layer. Writers in Husky read from Kafka, buffer events briefly in memory, then upload them to blob storage (AWS S3). A Shard Router service reads from one Kafka cluster and writes to a separate output cluster with data organised into logical shards – groups of partitions that map to physical storage units. This design allows Kafka scaling and storage scaling to proceed independently.

Exactly-once ingestion routing

Husky’s second generation introduced a locality concept embedded in the Kafka pipeline: a given event’s tenant ID and timestamp deterministically route it to the same Kafka shard on every ingestion attempt. Combined with FoundationDB for metadata transactions, this eliminates duplicates without requiring stateful writers. Daniel Intskirveli and Cecilia Watt documented this in a February 2023 post.

CDC-based data replication

Kafka is also the backbone of Datadog’s internal change data capture (CDC) replication platform, which handles Postgres-to-Postgres replication, Postgres-to-Apache Iceberg pipelines for analytics, Cassandra replication, and cross-region Kafka replication. Sanketh Balakrishna and Andrew Zhang published the architecture in November 2025, documenting approximately 500 ms end-to-end replication lag for the initial use case.

Live process metrics pipeline

The live process metrics feature uses Kafka for host subscription signalling. The live data server publishes to a host subscriptions topic at 1-second intervals; the intake service consumes from that topic and maintains an in-memory cache of which hosts are currently being viewed, activating high-frequency data collection only for those hosts. Kai Zong Khor and William Yu described this architecture in an August 2025 post.

Configuration distribution

Kafka carries cache-invalidation notifications when tenant configuration changes are written to the database, propagating those changes to downstream services. Gabriel Reid documented this use case in a June 2025 post, noting that a newer v2 design complements Kafka with cloud object storage for this role.

Scale and throughput

As documented by Guillaume Bort in February 2025:

Clusters: Hundreds of Kafka clusters across Kubernetes and cloud environments
Topics: Thousands of topics
Partitions: Millions of partitions
Consumer groups: Hundreds of consumer groups
Daily event volume: Hundreds of trillions of observability events per day
Throughput: Terabytes of data per second

For context, the open-source kafka-kit page describes “double-digit gigabytes per second” bandwidth requiring petabytes of high-performance storage even with short retention windows.

In 2019, Emily Chang documented 40+ Kafka and ZooKeeper clusters processing trillions of datapoints daily across multiple data centres and regions, which gives a sense of how much the fleet has grown since then.

On a smaller scale, one indicative example is the live process metrics pipeline: before an architecture change in 2025, the host subscriptions topic was handling 500,000 messages per second. After redesigning when and what data was collected, that fell to 5,000 messages per second – a 100x reduction – while freeing over 600 CPU cores and 1 TB of memory.

Datadog’s Kafka architecture

The Streaming Platform

The most significant architectural layer above raw Kafka is the Streaming Platform, introduced publicly in February 2025. Rather than having clients connect to physical Kafka clusters, the Streaming Platform provides three logical abstractions:

Streams are multi-cluster logical topics. A Stream has a stable identifier that is independent of the physical cluster topology. Producers and consumers reference a Stream name; the control plane resolves that to one or more physical topics on one or more clusters. This decoupling means the underlying cluster can change without application redeployment.

Stream Lanes provide quality-of-service tiers within a Stream. A single Stream can have multiple lanes: a real-time lane for live traffic, a batch lane for lower-priority or late-arriving data, and a dead-letter queue lane for poison pills. When a malformed message blocks a consumer, the DLQ lane copies it out of the main partition, allowing the partition to make progress.

The Assigner is a custom multi-cluster coordinator that replaces Kafka’s default consumer group coordinator. It monitors cluster health and workload distribution across the entire fleet and can redirect traffic to a different cluster, decommission a cluster, or reallocate partitions – all without consumer redeployment.

libstreaming: the custom Kafka client

All client interaction with the Streaming Platform goes through libstreaming, a unified Kafka client library written in Rust with bindings for Java, Go, and Python. It initially wrapped rdkafka, but rdkafka’s handling of compressed batches (see Challenges below) led Datadog to replace it with a fully custom Kafka client in Rust. libstreaming manages cluster discovery, failover, and the advanced commit log (described below).

Topic organisation (kafka-kit model)

Rather than running one or a few large clusters, Datadog treats topics as first-class capacity units. Topics are grouped into shared pools for smaller workloads or promoted to dedicated pools when they outgrow shared resources. The kafka-kit tooling automates the placement logic, using either even partition distribution (“count” strategy) or storage-based bin-packing (“storage” strategy).

Multi-region redundancy

For critical pipelines, Datadog uses duplicate writes to primary and secondary Kafka clusters. If an unclean leader election occurs on one cluster, the data is recoverable from the other. Emily Chang documented this pattern in 2019 as a response to data loss events caused by out-of-sync replicas being elected leader before Kafka 0.11.0 made exactly-once semantics more broadly available.

Advanced commit log

The standard Kafka consumer offset model tracks a single pointer per partition, which means a consumer must choose between processing live traffic and reprocessing a backlog. Datadog’s Streaming Platform extends the commit metadata to track multiple offsets or offset ranges within a single partition simultaneously. This allows consumers to handle both live traffic and concurrent backlog reprocessing without head-of-line blocking.

Kubernetes deployment

All Kafka clusters run self-managed on Kubernetes, using StatefulSets and persistent volumes. Martin Dickson, a Senior Software Engineer on the Datadog Kafka team, covered this in a March 2024 talk, noting that Datadog operates dozens of self-managed Kubernetes clusters across a multi-cloud environment.

CDC pipeline architecture

The CDC replication platform uses Debezium as the change capture connector for Postgres and Cassandra sources. Debezium serialises captured changes into Avro format and publishes them to Kafka topics, along with schema updates, to a multi-tenant Kafka Schema Registry. Kafka Connect (with single message transforms for field manipulation, topic renaming, and column filtering) handles the fault-tolerant data movement between systems.

Producer architecture

Producers interact with the Streaming Platform through libstreaming, which manages batching and routing transparently. The system uses Zstandard (zstd) compression across pipelines. At the scale Datadog operates, even compression can introduce problems – an incoming compressed batch may expand to gigabytes on decompression, which the custom client addresses by enforcing limits on decompressed batch size rather than compressed size.

Consumer architecture

Consumer group management is handled by The Assigner rather than Kafka’s native group coordinator. Lag monitoring uses wall-clock time rather than pure message-count offsets: the advanced commit log enriches metadata with ingestion timestamps, and the Streaming Platform computes time lag by consuming __consumer_offsets topics across all clusters. This allows automated rebalancing to trigger based on actual data age.

For the live process metrics pipeline, each entry in the Kafka-backed cache includes a TTL, so hosts expire automatically when collection stops without requiring an explicit deletion event.

Special techniques and engineering innovations

Live traffic failover without redeployment

The Streaming Platform’s Assigner can redirect all traffic for a Stream from one cluster to another in seconds. The control plane creates a new topic on the target cluster, redirects producers and consumers, then drains the old topic. The same mechanism handles proactive load redistribution and cluster decommissioning without any application-side changes.

Relaxed ordering for parallelism

Datadog moved away from strict per-partition ordering for the main observability pipeline. Rather than relying on Kafka partition order for correctness, events are processed with at-least-once delivery and ordering is deferred to the downstream Husky storage layer. This unlocks substantially greater parallelism at petabyte scale.

Deterministic shard routing for deduplication

Husky’s shard routing assigns events to Kafka shards based on tenant ID and timestamp. Because the mapping is deterministic, the same event will always land in the same shard regardless of how many times it is re-ingested. Combined with FoundationDB for transactional metadata, this provides exactly-once semantics without stateful writers or distributed locking.

Storage-based partition rebalancing

kafka-kit’s topicmappr tool supports two placement strategies: count-based (even partition distribution across brokers) and storage-based (bin-packing partitions according to their current storage metrics). The storage strategy integrates with Datadog’s own metrics API to pull live partition size data, and includes rack-aware replica placement. This reduces the manual effort of broker replacements and rebalances at a fleet of hundreds of clusters.

TTL-based subscription expiry

In the live process metrics pipeline, host subscription state in the Kafka-backed cache includes a TTL. This handles ungraceful terminations: if a service crashes without sending a deletion event, its subscription entries expire naturally, preventing the intake service from continuing to collect data for a host no one is viewing.

Operating Kafka at scale

Deployment model: Entirely self-managed on Kubernetes, across dozens of Kubernetes clusters in a multi-cloud environment. Datadog does not use Confluent Cloud, MSK, or other managed Kafka services for its production clusters.

Monitoring: Datadog monitors its Kafka infrastructure with its own platform, tracking MBean metrics including MessagesInPerSec, BytesInPerSec, and ISR shrinks/expands. The kafka-kit autothrottle tool integrates directly with the Datadog metrics API to manage replication throttle rates dynamically during broker replacements.

Segment configuration tuning: For low-throughput topics, default Kafka segment settings can cause log retention to exceed the configured topic-level retention. Emily Chang documented reducing segment.ms to 43,200,000 ms (12 hours) and segment.bytes to approximately 100 MB to align segment lifecycle with topic-level retention. Open file handles are monitored to ensure frequent segment rollouts do not exhaust OS limits.

Offset retention: Consumer offset retention was extended from the default 1 day to 7 days, preventing data reprocessing on topics where consumer groups are temporarily offline.

Configuration change testing: Configuration changes are validated on replicated or mirrored clusters before being applied to production. ISR shrinks/expands and segment sizes are monitored continuously during rollouts.

Developer tooling: The kafka-kit suite (open-sourced August 2018, maintained through at least v4.2.1 in July 2023) handles topic mapping, partition rebalancing, broker replacements, and replication throttle management. It is available at github.com/DataDog/kafka-kit.

Schema governance: The CDC replication platform uses a multi-tenant Kafka Schema Registry with backward compatibility enforced. Schema changes are validated against the registry before application, preventing pipeline breakage from schema drift.

Challenges and how they solved them

Static cluster bindings preventing rapid failover

Applications were bound to physical Kafka cluster addresses at deploy time. Redirecting traffic to a different cluster required a full redeployment cycle. Datadog built the Streaming Platform abstraction so that producers and consumers reference logical Stream names, not cluster addresses. The Assigner resolves those names to physical clusters and can redirect traffic in seconds. The result is zero-downtime cluster failover and live traffic migration without touching application configuration.

Single-pointer offset model blocking concurrent backlog processing

Kafka’s standard offset model gives each consumer group a single pointer per partition. Reprocessing a backlog means abandoning live traffic on that partition, or running a separate consumer group with separate infrastructure. Datadog built a custom commit log that stores multiple offsets or offset ranges in a partition’s commit metadata simultaneously. Consumers can process live traffic and catch up on a backlog at the same time, on the same partition.

zstd compression bombs crashing consumers

The rdkafka client – which Datadog initially used as the foundation of libstreaming – limits batch sizes by compressed size. A message batch that is small when compressed can expand to gigabytes on decompression if the payload is highly repetitive. At Datadog’s ingestion rate, compressed batches large enough to crash consumer processes were not a theoretical problem. Datadog replaced rdkafka with a custom Rust Kafka client that enforces decompressed size limits. Consumer crashes from oversized decompressed batches were eliminated.

Offset-based lag insufficient for operational decisions

Standard Kafka consumer lag is expressed in message count: how many messages behind the latest offset the consumer group sits. That figure provides no information about how old the unprocessed data is. A consumer group that is 10 million messages behind on a high-throughput topic may be seconds behind in wall-clock time; on a low-throughput topic, those same 10 million messages might represent days of data. Datadog enriched commit metadata with ingestion timestamps and built wall-clock time lag computation by consuming __consumer_offsets topics across all clusters. Automated rebalancing now triggers on actual data age rather than message count.

Unclean leader elections causing data loss

In multi-region deployments running pre-0.11.0 Kafka (which enabled unclean leader elections by default), an out-of-sync replica could be elected leader when the in-sync leader became unavailable, discarding the unreplicated messages. Datadog’s response was duplicate writes: data written to both a primary and a secondary Kafka cluster. A data loss event on one cluster left the data intact on the other. The combination of Kafka 0.11.0’s changes to leader election defaults and duplicate-write redundancy eliminated permanent data loss from this failure mode.

Log segment retention exceeding topic-level retention on low-throughput topics

Kafka’s default segment.ms value is 7 days. On topics with a 36-hour retention setting and low message volume, segments stayed open for the full 7 days before rolling, meaning data was retained for longer than the configured policy. Reducing segment.ms to 12 hours and segment.bytes to approximately 100 MB brought segment lifecycle in line with topic-level retention.

Head-of-line blocking from poison pills and traffic spikes

A malformed message in a partition blocks all consumers on that partition until it is handled, and a traffic spike from one high-volume workload can starve other workloads sharing the same partitions. Stream Lanes within the Streaming Platform address both problems. Real-time, batch, and dead-letter workloads run on separate lanes within the same logical Stream. When the system detects a poison pill, it copies the message to the DLQ lane, allowing the main lane to advance. Different workload types can no longer block each other at the partition level.

Live process pipeline consuming 500,000 messages/second for inactive data

The intake service was receiving all process metrics for all hosts, even those that no user was actively viewing. The Kafka-consuming service maintained in-memory state for the entire host fleet. Kai Zong Khor and William Yu redesigned the pipeline so that the live data server only activates 2-second collection intervals for hosts currently being viewed, and signals this state to the intake service via Kafka at 1-second intervals. Peak throughput on the host subscriptions topic dropped from 500,000 to 5,000 messages per second, and the supporting infrastructure was scaled down by 98%.

Full tech stack

Category	Tools	Notes
Message broker	Apache Kafka	Self-managed; specific version not disclosed publicly
Kafka client	libstreaming (custom, Rust)	Replaces rdkafka; bindings for Java, Go, Python
Kafka client (legacy/external)	rdkafka / librdkafka	Still used in some services and Observability Pipelines
Kafka client (Go)	DataDog/confluent-kafka-go	Datadog fork of the Confluent Go client, wrapping librdkafka
Stream processing control plane	The Assigner (custom)	Replaces Kafka’s native group coordinator for the Streaming Platform
Schema registry	Kafka Schema Registry (multi-tenant)	Configured for backward compatibility; used in CDC pipelines
Serialisation format	Apache Avro	Used in CDC event pipelines
Change data capture	Debezium	Source connector for Postgres and Cassandra
Data movement	Kafka Connect	With single message transforms for CDC pipelines
Metadata store	FoundationDB	Transactional metadata for exactly-once ingestion in Husky
Blob storage	AWS S3	Target for Husky event store Writers
Analytics target	Apache Iceberg	Target for Postgres-to-analytics CDC pipelines
Compression	Zstandard (zstd)	Compression algorithm for Kafka messages
Kafka operations tooling	kafka-kit (open source, Go)	Covers topicmappr, autothrottle, registry, metricsfetcher
Orchestration	Kubernetes (StatefulSets)	Self-managed, multi-cloud, dozens of clusters
Monitoring	Datadog	Kafka clusters monitored with Datadog’s own platform

Key contributors

Guillaume Bort (Staff Software Engineer, Datadog): Led the Streaming Platform, libstreaming, and the custom Rust Kafka client. February 2025 engineering post
Jamie Alquiza (Staff Software Engineer, Datadog): Creator of kafka-kit; co-lead of early Kafka infrastructure scaling. kafka-kit introduction post, GitHub repo
Balthazar Rouberol (Data Reliability Engineer Team Lead, Datadog): Co-lead of Kafka and Cassandra infrastructure; presented “Running Production Kafka Clusters in Kubernetes” at Kafka Summit London 2019
Emily Chang: Authored the 2019 “Lessons learned from running Kafka at Datadog” post. Engineering blog
Richard Artoul and Cecilia Watt: Introduced the Husky event store and Kafka’s role as its ingestion layer. May 2022 post
Daniel Intskirveli and Cecilia Watt: Documented Husky’s exactly-once ingestion and shard routing. February 2023 post
Sanketh Balakrishna and Andrew Zhang: Designed and published the CDC replication platform. November 2025 post
Kai Zong Khor and William Yu: Redesigned the live process metrics pipeline, achieving a 100x throughput reduction. August 2025 post
Gabriel Reid: Documented configuration distribution via Kafka. June 2025 post
Martin Dickson (Senior Software Engineer, Kafka team): Covered self-managed Kafka on Kubernetes in a 2024 summit talk. Episode

Key takeaways for your own Kafka implementation

Decouple clients from physical topology early. Binding producers and consumers directly to cluster addresses makes failover expensive. Datadog built an abstraction layer so applications reference logical stream names. If you are running multiple clusters, even a lightweight routing layer pays off as the fleet grows.
Message-count lag is an incomplete signal. Whether 10 million lagging messages represents a problem depends entirely on how old they are. Enriching commit metadata with ingestion timestamps and computing wall-clock lag gives operations teams a signal they can actually act on.
Client-side compressed batch size limits are by compressed size, not decompressed size. At high throughput with zstd compression, this can cause decompressed batches to overwhelm consumer memory. If you are running high-compression pipelines at scale, verify how your client handles this and whether you need a custom or patched client.
Segment configuration affects retention on low-throughput topics. The default segment.ms of 7 days can cause data to be retained well beyond the topic-level retention setting on topics that receive infrequent writes. Tuning segment.ms and segment.bytes for low-throughput topics brings actual retention in line with policy.
QoS tiers within a topic reduce blast radius. Rather than isolating workloads entirely into separate topics or clusters, Datadog’s Stream Lanes provide lightweight isolation within a single logical stream. Separating real-time from batch traffic and routing poison pills to a dead-letter lane reduces the operational surface area without requiring full topic duplication.

Sources and further reading

Primary sources

Guillaume Bort, “Achieving relentless Kafka reliability at scale with the Streaming Platform” (February 2025)
Emily Chang, “Lessons learned from running Kafka at Datadog” (June 2019)
Jamie Alquiza, “Introducing Kafka-Kit: Tools for scaling Kafka” (August 2018)
Richard Artoul and Cecilia Watt, “Introducing Husky, Datadog’s third-generation event store” (May 2022)
Daniel Intskirveli and Cecilia Watt, “Husky: Exactly-once ingestion and multi-tenancy at scale” (February 2023)
Sanketh Balakrishna and Andrew Zhang, “Replication redefined: How we built a low-latency, multi-tenant data replication platform” (November 2025)
Kai Zong Khor and William Yu, “Scaling down to speed up: How we improved efficiency of live process metrics by 100x” (August 2025)
Gabriel Reid, “How we scaled fast, reliable configuration distribution to thousands of workload containers” (June 2025)
Balthazar Rouberol, “Running Production Kafka Clusters in Kubernetes” (Kafka Summit London 2019)
Jamie Alquiza and Balthazar Rouberol, “Datadog on Kafka” podcast (May 2020)
Martin Dickson, “Datadog on Stateful Workloads on Kubernetes” (March 2024)
kafka-kit GitHub repository

Try Kpow with your Kafka cluster

If you are monitoring a Kafka cluster at any scale, you can try Kpow free for 30 days. It connects to any Kafka cluster in minutes and deploys via Docker, Helm, or JAR.

Apache Kafka architecture: a complete guide to internals, components, and deployment

Factor House — Tue, 02 Jun 2026 00:00:00 GMT

How Kafka works: a mental model

Kafka is a distributed commit log. That framing matters: it is not a message queue, not a database, and not a pub-sub system, though it borrows ideas from all three. Understanding how it works starts with understanding what the log is and why that design choice dictates everything else.

At its core, Kafka stores records in topics. Each topic is divided into one or more partitions, and each partition is an ordered, immutable sequence of records written to disk. Producers append records to the end of a partition; consumers read from any position in the log by specifying an offset. Nothing is deleted when a consumer reads a record - the record stays in the log until a retention policy removes it. Multiple consumers reading the same partition at the same time each read independently, at their own position.

This design gives Kafka two properties that are difficult to achieve together in traditional message brokers: high throughput and replay. Because writes are sequential disk appends, they are fast. Because data persists, any consumer can re-read historical data, replay a stream from a specific point, or bootstrap a new consumer group without disturbing others.

Kafka separates producers from consumers entirely. A producer writes records to Kafka; it has no knowledge of which consumers exist or where they are reading. A consumer reads from Kafka; it has no interaction with producers. The log in between is the only coupling point. This decoupling is why Kafka scales: adding a new consumer does not affect producers or existing consumers, and a producer can continue writing whether consumers are online or not.

The cluster is a set of brokers. Each broker stores a subset of partitions. Some partitions are replicated across multiple brokers for durability. The cluster is managed by a controller, which tracks partition leadership and handles broker failures. As of Kafka 4.0, that controller runs on an internal consensus protocol called KRaft, with no external coordination dependency.

The Kafka log

A Kafka partition is a log. Physically, it is a directory on the broker’s filesystem. Inside that directory are one or more log segments, each of which is a pair of files: a .log file containing the raw record data, and an .index file that maps record offsets to byte positions in the .log file. A third file, the .timeindex, maps timestamps to offsets, enabling time-based lookups.

Segments are rolled when they exceed a size threshold (controlled by log.segment.bytes, defaulting to 1 GB) or when they age past a configured interval (log.roll.ms). Only the most recent segment in a partition is active and writable; older segments are sealed and can be cleaned or deleted by background threads.

Kafka uses two separate retention mechanisms:

Delete retention: Records older than a configured age (log.retention.ms) or beyond a size cap (log.retention.bytes) are deleted. The broker removes entire segments - it does not delete individual records from within a segment. This is the default for most topics.

Compacted retention: For topics with cleanup.policy=compact, Kafka retains only the most recent record for each key. A background thread called the log cleaner compares dirty segments (containing duplicate keys) against clean segments and writes a compacted output. A null-valued record (a tombstone) marks a key for deletion. The cleaner thread is governed by log.cleaner.min.dirty.ratio, which sets the minimum fraction of dirty log that must exist before compaction runs. Compacted topics are appropriate when you care about the latest state of a key rather than the full history: a user profile topic, for example, or a Kafka Streams changelog topic.

You can apply both policies to the same topic using cleanup.policy=compact,delete. Compaction removes redundant keys; deletion caps total retention by time or size.

The log is also the unit of replication. Followers replicate the partition leader’s log by issuing fetch requests. The high watermark marks the point up to which all in-sync replicas have confirmed they have written. Consumers only read up to the high watermark, so they never see records that might be lost if a leader failure occurred before full replication.

Core components of Kafka

Component	Role
Broker	Stores and serves partitions; handles producer and consumer connections
Topic	Logical category for records; divided into partitions
Partition	Ordered, append-only log; unit of parallelism and replication
Producer	Writes records to topics; controls partitioning and batching
Consumer	Reads records from partitions; tracks position via offsets
Consumer group	Set of consumers that share the work of reading a topic
KRaft controller	Manages cluster metadata and leader election (replaces ZooKeeper)
Schema Registry	Enforces schema contracts for record serialization
Kafka Connect	Integrates external systems via source and sink connectors
Kafka Streams	Client library for stateful stream processing inside the cluster
ksqlDB	SQL interface for stream processing
Kafka UI	Operational visibility layer

These components divide into three layers: the core data plane (brokers, topics, partitions, producers, consumers), the control plane (KRaft controller), and the ecosystem (Schema Registry, Connect, Streams, ksqlDB, and operational tooling). The diagram below shows how they relate.

Topics and partitions

A topic is a named stream of records. It is the unit of logical organization in Kafka: producers target a topic, consumers subscribe to one or more topics. Physically, a topic is implemented as one or more partitions spread across brokers.

Partition count and parallelism: The number of partitions in a topic determines the maximum read parallelism within a consumer group. A partition can only be assigned to one consumer in a group at a time, so a consumer group with more consumers than partitions will have idle consumers. Partition count is a durable decision: you can increase it after creation, but you cannot decrease it, and increasing it can break ordering guarantees for keyed records (because keys are hashed to partitions, and adding partitions changes the mapping).

A reasonable starting point is one partition per expected peak parallel consumer, with headroom for growth. For high-throughput topics, also account for per-partition memory overhead on brokers and per-partition file descriptor usage.

Partition assignment: When a producer sends a record without a key, Kafka assigns it to partitions using a sticky batching strategy: the producer fills a batch for one partition before moving to the next, which improves compression and throughput. When a record has a key, Kafka hashes the key to a partition using the DefaultPartitioner (murmur2 hash). Records with the same key always go to the same partition, preserving per-key ordering. You can supply a custom Partitioner implementation if you need different routing logic.

Compacted versus delete topics: Topics use either delete or compact cleanup policy, or both. Delete topics are appropriate for event streams where age determines relevance: logs, metrics, activity events. Compacted topics are appropriate for changelog semantics where the latest value per key is what matters: materialized state, lookup tables, Kafka Streams state stores.

Producers

The producer lifecycle for a single record moves through four stages: serialization, partitioning, batching, and network send.

Serialization and partitioning: The producer serializes the key and value using configured serializers (byte array, string, Avro, Protobuf, and so on) before anything else. After serialization, the partitioner assigns the record to a partition.

Batching: Records destined for the same partition and broker accumulate in a RecordBatch in the producer’s send buffer. Two configurations govern batch behavior:

batch.size: the maximum size in bytes of a single batch. The producer sends when the batch is full or when linger.ms elapses.
linger.ms: the maximum time the producer waits to fill a batch before sending. At linger.ms=0, the producer sends immediately. Increasing this trades latency for throughput by allowing larger batches.

Compression: Setting compression.type (gzip, snappy, lz4, zstd) compresses batches before sending. Compression reduces network I/O and broker storage at the cost of CPU. LZ4 and Zstandard offer the best throughput-to-compression-ratio tradeoff for most workloads.

Delivery guarantees and acks: The acks configuration controls when the leader considers a write complete:

acks=0: no acknowledgment; fire and forget.
acks=1: the leader acknowledges after writing to its local log, before followers replicate. A leader failure before replication means data loss.
acks=all (or -1): the leader waits for all in-sync replicas to acknowledge. Since Kafka 3.0 this is the default. Combined with min.insync.replicas=2 and replication.factor=3, this provides strong durability guarantees.

Idempotent producers and exactly-once semantics: Setting enable.idempotence=true assigns the producer a persistent producer ID and sequences each record batch. The broker deduplicates retried batches using the producer ID and sequence number, preventing duplicate records during transient network failures. Idempotent producers require acks=all and retries > 0; Kafka enforces these automatically when idempotence is enabled.

For end-to-end exactly-once semantics across multiple topics - including writes paired with consumer offset commits - use the transactional API: call initTransactions(), begin a transaction, produce and commit consumer offsets within the transaction, then commit or abort.

A common misconception is that exactly-once requires max.in.flight.requests.per.connection=1. This was true for early versions of idempotent producers. Since Kafka 0.11, idempotent producers safely handle up to five in-flight requests per partition while maintaining ordering and deduplication.

Brokers and the cluster

A Kafka cluster is a set of broker processes, each running on a separate machine or VM. Each broker is responsible for the partitions it hosts: writing records to local disk as producers send them, serving fetch requests from followers and consumers, and enforcing producer quotas, consumer quotas, and topic-level access controls.

Partition leadership: Every partition has a leader and zero or more followers. The leader handles all produce and fetch requests for that partition. Followers issue FetchRequests to the leader, replicating the log independently. When a broker fails, the controller elects a new leader from the in-sync replicas of each partition that broker led.

Controller responsibilities: In a KRaft cluster, a subset of nodes forms the controller quorum. The active controller is the single node responsible for metadata decisions: partition leader elections, ISR updates, broker registrations, topic creation, and ACL changes. The other nodes in the quorum maintain up-to-date replicas of the metadata log and can assume the active role in milliseconds if the active controller fails. The controller does not handle data reads or writes.

Rack awareness: Setting broker.rack on each broker causes the controller to spread partition replicas across distinct failure domains. In a three-AZ deployment with replication factor 3, each AZ gets one replica. A single AZ failure does not take the partition offline.

KRaft: Kafka without ZooKeeper

KRaft (Kafka Raft) is the internal consensus protocol that manages Kafka cluster metadata. It replaced Apache ZooKeeper as the coordination layer, removing a major operational dependency and resolving fundamental scaling limits in the original architecture.

Why ZooKeeper was removed: The ZooKeeper-based architecture stored cluster metadata externally. Whenever partition state changed, the active controller wrote to ZooKeeper, waited for ZooKeeper consensus, received a watch notification, updated its local memory, and then pushed update RPCs to every affected broker. This push-based propagation created two compounding problems. First, it imposed a hard ceiling on cluster scale: as partition counts grew past roughly 200,000, ZooKeeper’s metadata handling degraded, with coordination overhead growing substantially at scale. Second, controller failover was expensive: a newly elected controller had to pull and reconstruct its entire metadata cache from ZooKeeper before it could make any decisions, causing prolonged unavailability during failover on large clusters.

What KRaft does instead: KRaft treats cluster metadata as an append-only log stored in an internal topic called __cluster_metadata. The active KRaft controller is the leader of this log. Other controllers in the quorum act as followers, maintaining hot in-memory copies of the metadata. When a controller fails, a standby takes over in milliseconds because it already has the full metadata state in memory - no rebuild from an external store.

Rather than pushing metadata updates to brokers, KRaft uses a pull model. Brokers issue incremental FetchRequests to the active controller, pulling only the changes they have not yet seen. Because metadata changes are consumed from a single ordered log, every broker processes events in the same sequence, ensuring eventual consistency and preventing the metadata divergence that was possible with the legacy push model.

The controller quorum: The KRaft quorum uses a modified Raft consensus protocol (defined in KIP-595). To tolerate f controller failures, you need 2f+1 voters: a 3-node quorum tolerates 1 failure; a 5-node quorum tolerates 2. For production, run an odd number of dedicated controller nodes - not combined broker-controller nodes - to insulate the consensus layer from broker-side I/O pressure and garbage collection pauses. Co-locating controllers with data brokers exposes the control plane to the same resource contention it is trying to isolate.

Broker state machine: Each broker in a KRaft cluster transitions through four states:

State	Description
Offline	The broker process is stopped or is completing initial startup
Fenced	The broker is running but excluded from client RPCs - entered during startup or when it loses contact with the controller
Online	The broker is fully operational, registered with the active controller, and authorized to serve traffic
Stopping	The broker is in graceful shutdown; the controller migrates its partition leadership before taking it offline

If a broker loses contact with the controller and cannot send heartbeats, the controller fences it in the metadata log. Other brokers, consuming this event, redirect client traffic away from the fenced broker immediately.

Snapshotting (KIP-630): Because __cluster_metadata is an append-only log, it would grow indefinitely without compaction. KRaft uses a snapshotting mechanism: when enough records have accumulated since the last snapshot, the active controller serializes its in-memory metadata state to a checkpoint file. Followers download snapshots chunk-by-chunk via FetchSnapshot RPCs, validate the CRC, and load the new state. A newly joined or lagging broker can bootstrap quickly by fetching the latest snapshot and replaying only the log entries after the snapshot’s end offset.

Dynamic quorum membership (KIP-853): Previously, the controller quorum was statically configured via controller.quorum.voters. KIP-853 added support for adding or removing controller voters without a cluster restart, using AddVoter and RemoveVoter RPCs. Only one membership change is permitted at a time to prevent split-brain scenarios. This makes controller hardware replacement (after a disk failure, for example) an online operation.

Version timeline:

Kafka version	Date	Status
2.8	Early 2021	KRaft introduced as experimental
3.3	August 2022	KRaft marked production-ready (KIP-833); ZooKeeper deprecated
3.6	September 2023	ZK-to-KRaft live migration promoted to GA
3.9	May 2025	Final 3.x release; dynamic KRaft quorums (KIP-853) added
4.0	March 18, 2025	ZooKeeper support completely removed

ZooKeeper mode is not available in Kafka 4.0 and later. If you are running a ZooKeeper-based cluster on 3.x, migration to KRaft is required before upgrading to 4.x. The live migration process (KIP-866) runs in four phases - metadata copy, dual-write hybrid runtime, rolling broker reconfiguration, and finalization - and supports rollback through Phase 3.

Replication and fault tolerance

Kafka replication distributes partition data across multiple brokers to survive broker failures without data loss or service interruption.

In-sync replicas (ISR): Each partition maintains an ISR set: the replicas that are considered current with the leader. A follower stays in the ISR as long as it has fetched from the leader within replica.lag.time.max.ms (default: 30 seconds). If a follower stops fetching or falls behind, the leader removes it from the ISR by sending an AlterPartitionRequest to the controller. The controller commits the shrunk ISR to the metadata log. Crucially, only the persisted ISR matters for leader election: if a leader fails before its latest ISR shrink request reaches the controller, the controller elects from the last committed ISR state.

What happens when a broker fails: In KRaft mode, if a broker misses its heartbeat window with the active controller, the controller fences the broker by appending a FenceBrokerRecord to __cluster_metadata. Brokers consuming the metadata log discover the fence and redirect clients away. The controller then initiates leader elections for all partitions the failed broker led.

The controller selects a new leader from a priority hierarchy:

Priority	Candidate group	Behavior
1	In-sync replicas (ISR)	Selects the first active, unfenced ISR member. No data loss.
2	Eligible leader replicas (ELR)	If ISR is empty, selects from the ELR set (KIP-966).
3	Last known leader	If ELR is also empty, attempts the last known unfenced leader.
4	Out-of-sync replicas	If `unclean.leader.election.enable=true`, elects any surviving replica. Data loss occurs.
5	None	If unclean elections are disabled and no clean replica exists, the partition stays offline.

Unclean leader election: With unclean.leader.election.enable=false (the default), a partition stays offline rather than elect an out-of-sync replica. This protects data integrity at the cost of availability. With the setting enabled, the out-of-sync replica becomes leader, the ISR is reset to contain only that new leader, and any records acknowledged by the old leader but not replicated to the new leader are permanently lost. The controller marks the partition recovery state as RECOVERING. This setting should remain false for any topic where losing acknowledged writes is unacceptable. If you need to override it, do so at the topic level, not cluster-wide.

The default configuration durability gap: There is a trap worth naming directly. Since Kafka 3.0, acks defaults to all. However, min.insync.replicas still defaults to 1, and default.replication.factor defaults to 1. With min.insync.replicas=1, acks=all behaves identically to acks=1: the leader acknowledges a write as soon as its own log is written, because the ISR contains only itself. If two follower replicas are temporarily removed from the ISR (due to network issues), the leader accepts writes with acks=all and immediately acknowledges them. If that leader then crashes before the followers rejoin, those acknowledged records are gone.

For any production topic where data loss is unacceptable, set:

replication.factor=3
min.insync.replicas=2

With this configuration and acks=all, a write is only acknowledged after two brokers have it on disk. If two brokers fail simultaneously and the ISR drops to 1, producers receive a NotEnoughReplicasException rather than a silent acknowledgment that could be lost.

Leader epoch and truncation (KIP-101): Kafka uses a leader epoch - a monotonically increasing counter incremented on each leadership change - to prevent log divergence during follower recovery. When a follower restarts, it sends an OffsetForLeaderEpochRequest to the current leader, which responds with the log end offset for the follower’s epoch. The follower truncates only to that offset. Before KIP-101, followers truncated to their last local high watermark, which could cause data loss under a double failover scenario where a follower restarted and truncated before a second broker failure.

Consumers and consumer groups

Kafka consumers read records from partitions by subscribing to topics and issuing FetchRequests to the broker that leads each assigned partition.

Offsets and commits: A consumer’s position in a partition is its offset. Kafka stores committed offsets in the internal __consumer_offsets topic, keyed by (group.id, topic, partition). When a consumer restarts, it resumes from its last committed offset. Two configurations govern offset behavior:

auto.offset.reset: what to do when no committed offset exists. earliest reads from the beginning; latest reads from the current end; none throws an exception.
enable.auto.commit: whether the consumer automatically commits offsets on a schedule. Auto-commit is convenient but can commit before processing is complete, causing duplicate processing or data loss on restart. For reliable at-least-once guarantees, set enable.auto.commit=false and commit offsets manually after successful processing.

Consumer groups: A consumer group is a set of consumers identified by a shared group.id. Each partition in a subscribed topic is assigned to exactly one consumer in the group at a time. A group with 10 consumers and a topic with 10 partitions gets one partition per consumer. A group with 10 consumers and 4 partitions has 6 idle consumers: partition count is the ceiling on consumer-level parallelism within a group.

Rebalancing: When a consumer joins or leaves a group, or when partition count changes, Kafka reassigns partitions across group members. The protocol used for rebalancing determines how disruptive this is:

Eager rebalancing (legacy): all consumers stop consuming and release all their partitions. The group coordinator assigns partitions from scratch. This is a stop-the-world pause for the entire group.
Cooperative-sticky rebalancing (recommended): consumers retain as many existing partition assignments as possible. Only partitions that need to move change hands, and they do so incrementally. Consumers that keep their partitions continue processing throughout the rebalance.

To enable cooperative-sticky rebalancing, set partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor. This is not the default as of Kafka 3.x. Switching strategies in a running group requires a one-time migration: first add CooperativeStickyAssignor alongside the existing strategy, perform a rolling application restart, then remove the old strategy and restart again.

Consumer lag: Consumer lag is the difference between the log end offset of a partition and the consumer group’s last committed offset for that partition. Lag represents the amount of unprocessed data. A persistently growing lag means the consumer cannot keep up with the producer and is a leading indicator of downstream problems. Monitoring lag per consumer group and per partition is one of the most operationally important signals in any Kafka-backed system.

Cluster architecture and management

Adding brokers: Adding a broker to a running cluster is non-disruptive, but new brokers are not assigned existing partitions automatically. Run a partition reassignment using kafka-reassign-partitions.sh to generate and submit a reassignment plan. The plan is executed by the controller while the cluster continues serving traffic. Throttle the reassignment using kafka-configs.sh to limit the impact on produce and consume latency during the move.

Rolling upgrades: Kafka supports rolling upgrades without downtime. Upgrade one broker at a time, allowing it to re-join and reach full replication before moving to the next. Keep inter.broker.protocol.version (IBP) at the version of the oldest broker until all brokers are upgraded, then advance the IBP. In KRaft clusters, upgrade controllers before brokers.

Log retention and disk management: Retention is managed per-topic via log.retention.ms, log.retention.bytes, and cleanup.policy. Monitor per-broker disk usage; uneven partition distribution (common when some partitions are significantly larger than others) can cause a single broker to reach capacity before others. Consider tiered storage (see “What’s changing in Kafka architecture”) to offload older segments to object storage.

Key JMX metrics:

Metric	Why it matters
`UnderReplicatedPartitions`	Non-zero means at least one follower is lagging or offline
`OfflinePartitionsCount`	Non-zero means partitions with no available leader; immediate action required
`ActiveControllerCount`	Should be exactly 1 cluster-wide
`UncleanLeaderElectionsPerSec`	Should be 0; non-zero means data loss is occurring
`UnderMinIsrPartitionCount`	Non-zero means writes with `acks=all` are being rejected
Consumer lag	Per consumer group, per partition; growing lag indicates a consumer falling behind

Throughput versus latency tuning: The primary levers are on the producer side (linger.ms and batch.size) and the consumer side (fetch.min.bytes and fetch.max.wait.ms). Increasing linger.ms and batch.size improves throughput by sending larger batches at the cost of higher produce latency. Increasing fetch.min.bytes and fetch.max.wait.ms reduces the number of fetch requests at the cost of higher read latency. Neither set of parameters should be tuned in isolation from the other side of the pipeline.

Security

Kafka’s security model covers authentication, authorization, and encryption. All three require explicit configuration; Kafka ships with no authentication enabled by default.

Authentication (SASL): Kafka supports four SASL mechanisms:

PLAIN: Username and password transmitted in plaintext. Acceptable only when TLS is also enabled to encrypt the connection. Simple to configure; not suitable for environments where credentials might be intercepted at the network layer.
SCRAM-SHA-256 / SCRAM-SHA-512: Salted challenge-response. Credentials are stored as salted hashes and managed via the Kafka admin API (in KRaft clusters). Well-suited for most deployments without existing Kerberos infrastructure.
GSSAPI (Kerberos): Integrates with Active Directory or MIT Kerberos. Required in environments with existing Kerberos infrastructure. Operationally complex to configure correctly.
OAUTHBEARER: Delegates authentication to an OAuth 2.0 identity provider. Well-suited for cloud-native environments where OIDC is already in use. Requires a custom or third-party token validator implementation.

For new deployments without existing Kerberos infrastructure, SCRAM-SHA-512 over TLS is the most practical starting point. For cloud-native environments with identity provider integration, OAUTHBEARER is increasingly common.

Authorization (ACLs): Kafka ACLs grant or deny operations (Read, Write, Create, Delete, Describe, and others) on resources (topics, consumer groups, clusters) to principals (users, service accounts). ACLs are managed via kafka-acls.sh or the admin API. In KRaft clusters, use StandardAuthorizer rather than the legacy AclAuthorizer, which required ZooKeeper.

Encryption in transit: Configure TLS on broker listeners by setting a keystore and truststore. All inter-broker communication should also use TLS (security.inter.broker.protocol=SSL). mTLS (mutual TLS) is used in environments that want the broker to authenticate clients via client certificates, rather than relying on SASL.

Network isolation: Kafka brokers should not be exposed to the public internet. Place them in private subnets, restrict inbound access to known application IP ranges via security groups or firewall rules, and use separate listener configurations if you need to expose Kafka to both internal and external clients with different security settings (listeners and advertised.listeners).

Schemas and schema registry

Without schema management, a Kafka topic is a sequence of untyped bytes. Any producer can write any format; any consumer must be written to match. This implicit contract fails silently: a producer change that modifies a field type or removes a required field can break consumers without warning, and the break is discovered at deserialize time, not at produce time.

A schema registry solves this by storing schemas centrally and embedding a schema ID in each record. Before a producer serializes a record, it registers the schema (if not already registered) and encodes the schema ID as a prefix in the record value, using the Confluent wire format: a magic byte, a 4-byte schema ID, then the serialized payload. Consumers look up the schema by ID before deserializing. Incompatible schema changes are rejected at produce time.

Supported formats: All major schema registries support Avro, Protobuf, and JSON Schema. Avro is the most widely deployed in Kafka ecosystems; Protobuf is common in organizations already using gRPC; JSON Schema is convenient but produces larger payloads and is less strict about types.

Compatibility modes: The registry enforces schema evolution rules per-subject:

BACKWARD: New schema can read data written by the previous schema. Safe for consumers to upgrade before producers.
FORWARD: Previous schema can read data written by the new schema. Safe for producers to upgrade before consumers.
FULL: Both backward and forward compatible. The safest setting for most production topics.
NONE: No compatibility checking.

Registry implementations:

Confluent Schema Registry: The most widely deployed implementation. Defines the wire format that most Kafka serializers expect. Available as open source under a mix of Apache 2.0 and Confluent Community License terms, or as a managed service in Confluent Cloud.
Apicurio Registry: Fully Apache 2.0 licensed. Supports Avro, Protobuf, and JSON Schema. Compatible with the Confluent wire format, making it a drop-in alternative for clients using standard Confluent serializers. Maintained by Red Hat.
AWS Glue Schema Registry: Managed service integrated with MSK and other AWS services. Supports Avro and JSON Schema; Protobuf support was added more recently. Uses a different wire format from Confluent, requiring Glue-specific serializers.

For self-managed clusters or environments where license terms matter, Apicurio is the best-maintained open-source alternative to Confluent Schema Registry.

Kafka Connect

Kafka Connect is a framework for building and running connectors that move data between Kafka and external systems. Rather than writing custom producer or consumer applications for each system integration, you configure a connector with a set of properties, and Connect handles serialization, offset tracking, error handling, and parallelism.

Source connectors read from an external system and write to a Kafka topic. A PostgreSQL CDC source connector using Debezium, for example, tails the write-ahead log and produces records to Kafka as each committed row change occurs. Sink connectors read from a Kafka topic and write to an external system: an S3 sink connector writes records to S3, partitioned by date or hour.

Deployment modes:

Standalone mode: A single Connect worker process. No fault tolerance. Suitable for development or simple single-node pipelines.
Distributed mode: Multiple Connect workers form a group. Tasks are distributed across the group; if a worker fails, tasks are reassigned to surviving workers. This is the production deployment model.

Exactly-once semantics in Connect: Connect supports exactly-once delivery for source connectors in distributed mode (since Kafka 2.6), using the transactional producer API combined with offset storage in Kafka. The connector’s source offsets and the produced records are written in the same transaction, so a crash mid-write is recovered cleanly on restart. Not all connectors support this: the connector must manage offsets through the Connect framework rather than externally. Exactly-once for sink connectors depends on whether the target system can participate in transactions.

Connector ecosystem: Confluent Hub hosts several hundred connectors. Debezium provides production-grade CDC connectors for PostgreSQL, MySQL, SQL Server, MongoDB, and others, and is widely used for event sourcing and data replication patterns. Most major cloud data stores (S3, GCS, BigQuery, Redshift, Snowflake, Elasticsearch) have maintained sink connectors available.

A concrete example of a common Connect pipeline: a Debezium PostgreSQL source connector captures row changes from a transactional database and writes them to Kafka topics (one topic per table), from where an S3 sink connector archives them to object storage. Both ends are configured and managed in Connect; no custom producer or consumer code is written.

Kafka Streams

Kafka Streams is a client library for building stateful stream processing applications that read from and write to Kafka. Unlike Apache Flink or Spark Streaming, it runs inside your application process: there is no separate cluster to deploy, no cluster manager to operate, and no external state store required unless you choose to add one.

Core abstractions:

KStream: An unbounded stream of records. Each record is an independent event.
KTable: A changelog stream interpreted as a materialized table. Each record represents the latest value for a key. Supports joins and aggregations.
GlobalKTable: A KTable replicated in full to every application instance. Used for broadcast joins where one side is small relative to the other.

State stores: Aggregations and joins require state. Kafka Streams maintains state in local RocksDB stores, backed by changelog topics in Kafka. On restart, an application instance restores its state by replaying the changelog topic. State stores can also be queried externally via the interactive queries API, allowing the application to expose its materialized state as a queryable service.

When to use Kafka Streams versus alternatives:

Kafka Streams is the right choice when your processing logic is sophisticated and tied to Kafka topics, and when running a separate processing cluster adds more cost and complexity than it is worth. It handles stateful operations (joins, aggregations, windowing) well, and deploying it as a jar alongside your existing application is a significant operational simplicity advantage.
Apache Flink is preferable when you need to join Kafka streams with non-Kafka sources, when batch and streaming workloads need to share processing logic, or when the scale of state exceeds what RocksDB on a single JVM can comfortably handle.
ksqlDB provides a SQL interface over Kafka Streams. See the next section.

ksqlDB

ksqlDB is a SQL-based stream processing system built on top of Kafka Streams. It lets you define streaming transformations, aggregations, and joins using SQL syntax, without writing application code. It supports two query types: push queries, which return a continuous stream of results and are suited to streaming dashboards and alerting, and pull queries, which return point-in-time lookups against materialized state.

Relationship to Kafka Streams: ksqlDB compiles SQL statements into Kafka Streams topologies and runs them on ksqlDB server nodes. The state store model, changelog topics, and processing guarantees are the same as Kafka Streams; ksqlDB adds a query language and a server process.

Development activity: Contributor activity on the ksqlDB GitHub repository (github.com/confluentinc/ksql) declined materially from 2022 onward. Confluent acquired Immerok in early 2023 to accelerate a cloud-native Apache Flink offering, now marketed as Confluent Cloud for Apache Flink. Confluent’s product positioning has shifted decisively toward Flink; ksqlDB receives minimal prominence in current Confluent documentation and engineering content. If you are evaluating ksqlDB for a new deployment, the reduced development momentum is a concrete factor in the decision, not an editorial judgement.

Licensing: ksqlDB is distributed under the Confluent Community License, not Apache 2.0. The Community License prohibits using the software to build a competing SaaS product. For internal deployments this is rarely a practical concern, but it is worth understanding before committing.

Choosing a stream processing layer:

Option	What it is	When to choose it
Kafka Streams	Java/Scala client library embedded in your application	Sophisticated stateful processing on Kafka data; no separate cluster needed
ksqlDB	SQL interface over Kafka Streams; runs as a separate server	Lower barrier for SQL-fluent teams; simpler, lower-throughput use cases
Apache Flink	General-purpose stream and batch processor; separate cluster	Mixed Kafka and non-Kafka sources; large-scale state; batch and streaming in one system

Kafka architecture in practice

The following companies use Kafka as a core part of their production infrastructure.

Company	How they use Kafka
Apple	Runs Kafka across dozens of data centres to move billions of events per day through internal analytics and ML pipelines.
Afterpay	Uses Kafka as the central event bus for payment transaction processing and real-time fraud detection across its buy-now-pay-later platform.
Cash App	Streams financial transaction events through Kafka to power real-time payment processing, ledger updates, and fraud signals.
Adidas	Uses Kafka to unify e-commerce order events, inventory updates, and customer activity data across its global digital platform.
Notion	Propagates document change events through Kafka to keep real-time collaboration, search indexing, and audit logs in sync.
Bytedance	Processes trillions of user interaction events per day through Kafka to feed the recommendation engines behind TikTok and Douyin.
Datadog	Ingests metrics, logs, and distributed traces at massive scale through Kafka before routing them to purpose-built storage backends.
Walmart	Streams real-time inventory, order, and supply chain events through Kafka to coordinate fulfilment across thousands of stores and warehouses.
PayPal	Uses Kafka to process billions of financial events per day, connecting payment processing, fraud detection, and risk scoring pipelines.
Tencent	Runs one of the largest Kafka deployments in the world to handle messaging, activity data, and analytics across WeChat and its gaming platforms.
The New York Times	Uses Kafka to decouple its content publishing pipeline, streaming article events from the CMS through to personalisation, search, and reader analytics.
Grab	Moves real-time ride-hailing and food delivery events through Kafka to support driver-passenger matching, dynamic pricing, and operational dashboards.
Wix	Uses Kafka as the event backbone for its website-building platform, streaming user and site activity data into analytics and product features.
DoorDash	Streams order lifecycle events through Kafka to coordinate real-time delivery tracking, driver dispatch, and merchant notifications.
Goldman Sachs	Uses Kafka to distribute market data and trade events across trading systems, risk engines, and regulatory reporting pipelines.
JPMorgan	Processes financial transaction and market data events through Kafka to support real-time risk management and internal data platform services.
New Relic	Uses Kafka as the ingestion backbone for its observability platform, buffering and routing telemetry data before it reaches downstream storage.
PagerDuty	Routes alert and incident events through Kafka to decouple ingestion from routing logic and ensure reliable delivery at high ingest volumes.
Robinhood	Streams trading, market data, and account events through Kafka to power real-time order execution, position updates, and risk controls.
Salesforce	Uses Kafka to propagate CRM change events across its platform, feeding real-time automation, Einstein AI features, and customer-facing integrations.
Pinterest	Moves user activity and engagement events through Kafka to train recommendation models and update real-time content ranking signals.
Reddit	Streams user activity, voting, and moderation events through Kafka to power feed ranking, spam detection, and site-wide analytics.
Spotify	Processes hundreds of billions of user listening events per day through Kafka to feed personalisation, recommendations, and the Discover Weekly pipeline.
Shopify	Uses Kafka to stream order, merchant, and storefront events across its platform, decoupling the checkout flow from downstream fulfilment and analytics.
Netflix	Runs Kafka as the central nervous system for its data pipeline, handling viewing events, operational metrics, and chaos engineering signals at global scale.
Barclays	Uses Kafka to stream banking transaction events and market data across trading, risk, and regulatory reporting systems.
Airbnb	Streams booking, search, and host activity events through Kafka to support real-time pricing, fraud detection, and data warehouse ingestion.
LinkedIn	Created Kafka to solve its own activity stream and operational data pipeline problem; now uses it at trillion-message-per-day scale across all major platform services.
Cloudflare	Processes network security events and DNS query logs at internet scale through Kafka, routing data into threat detection and analytics pipelines.
Uber	Uses Kafka to stream trip, financial, and marketplace events across its global platform, supporting surge pricing, driver-partner payments, and real-time analytics.

Kafka architecture best practices

Set replication factor to 3 and min.insync.replicas to 2. The default replication.factor=1 is a single point of failure. For any production topic, replication.factor=3 with min.insync.replicas=2 ensures that a write is only acknowledged after it is on at least two brokers, and that the cluster can survive a single broker failure without data loss or partition unavailability.
Keep unclean.leader.election.enable=false. This is the default. For any topic where losing acknowledged writes is unacceptable, never override it cluster-wide. If you need to override it for a specific topic for availability reasons, do so at the topic level and document the trade-off explicitly.
Enable idempotent producers. Set enable.idempotence=true. For exactly-once delivery across multiple topics, use the transactional API. The advice to set max.in.flight.requests.per.connection=1 for idempotent producers is outdated: since Kafka 0.11, idempotent producers safely support up to five in-flight requests per partition.
Size partition count based on expected peak consumer parallelism, not just throughput. Consumer parallelism within a group is bounded by partition count. Under-partitioned topics create bottlenecks that cannot be resolved without increasing partition count (which risks breaking per-key ordering guarantees for keyed records).
Use cooperative-sticky rebalancing. Set partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor on all consumer groups. Eager rebalancing stops all consumers during reassignment; cooperative-sticky is incremental and significantly reduces the disruption of consumer group membership changes.
Commit consumer offsets after processing, not before. Set enable.auto.commit=false and commit offsets manually after records are fully processed. Auto-commit can commit before processing is complete, causing data loss on consumer restart.
Set rack-aware replica placement. Configure broker.rack on each broker. Without this, multiple replicas of the same partition can end up on brokers in the same availability zone or physical rack, negating the fault tolerance that replication provides.
Monitor the critical JMX metrics. Alert when OfflinePartitionsCount is above 0, UnderReplicatedPartitions is above 0 for more than a few minutes (allow for normal during rolling restarts), UncleanLeaderElectionsPerSec is above 0, and consumer lag grows unboundedly.
Enforce schema compatibility. Even if you control both producers and consumers today, schemas drift over time. Registering schemas and enforcing BACKWARD or FULL compatibility at the registry level catches breaking changes before they reach production consumers.
Use a Kafka UI for operational visibility. CLI tooling is sufficient for debugging in isolation, but for running Kafka in production you need a way to observe broker health, consumer lag, topic throughput, and cluster configuration without SSH access. A management interface is a standard part of a well-operated Kafka cluster.

Kafka management tools

If you are running Kafka in production, Kpow by Factor House is a Kafka UI built for the operational practices described in this article. It gives you real-time visibility into consumer lag, broker health, topic throughput, and cluster configuration in one place - the signals you need when something goes wrong.

Deployment options

How you run Kafka shapes your operational responsibilities, scaling limits, and cost profile.

Self-managed on VMs or bare metal: Full control over Kafka version, configuration, and hardware. The highest operational overhead: you are responsible for provisioning, upgrades, monitoring, security hardening, and failure recovery. The right choice for organizations with existing infrastructure teams, specific hardware requirements, or compliance constraints that prevent using managed services.

Kubernetes (Strimzi or Confluent Operator): Container-native deployment. Strimzi is open source (Apache 2.0) and provides Kubernetes Operators for Kafka clusters, Connect, MirrorMaker, and Bridge. Confluent Operator is a commercial product. Both abstract Kubernetes complexity for Kafka operations and are a reasonable choice for teams already running Kubernetes who want to standardize deployment tooling. The main challenge is that Kafka is a stateful system: getting persistent volume sizing, storage class selection, and pod anti-affinity right requires Kafka-specific knowledge on top of Kubernetes knowledge.

Confluent Cloud: Fully managed Kafka with proprietary add-on services (Confluent Schema Registry, hosted Connect connectors, Confluent Cloud for Apache Flink). Significantly reduces operational overhead. Higher cost per throughput unit than self-managed options. Proprietary extensions create vendor dependency. Available on AWS, Azure, and GCP.

Amazon MSK: AWS-native managed Kafka. Integrates with IAM for authentication and authorization, VPC for network isolation, and other AWS services. Kafka version support lags the Apache release schedule by several months; check the current MSK documentation for the latest supported version before planning an upgrade. MSK does not support all Kafka configurations and restricts some broker-level plugin use (relevant for some Connect connectors).

Other managed options:

Aiven for Apache Kafka: Multi-cloud managed service across AWS, GCP, Azure, and DigitalOcean. Tracks the Apache Kafka release schedule closely. A good option when cloud portability matters.
Redpanda Cloud: Redpanda is Kafka API-compatible but is a different broker implementation (written in C++, not Java). Compatible with most Kafka clients and tools. There are known API gaps relative to Apache Kafka; evaluate compatibility against your specific use case - particularly around transactions, exactly-once semantics, and any Kafka Streams or Connect usage - before committing.

Option	Control	Operational overhead	Cost	Vendor lock-in
Self-managed (VMs/bare metal)	Full	High	Low to medium	None
Kubernetes (Strimzi)	Full	Medium-high	Low to medium	None
Confluent Cloud	Limited	Low	High	High
Amazon MSK	Limited	Low	Medium	Medium (AWS)
Aiven	Limited	Low	Medium	Low
Redpanda Cloud	Limited	Low	Medium	Medium

What’s changing in Kafka architecture

ZooKeeper removal (complete): ZooKeeper support was removed in Kafka 4.0, released March 18, 2025. All clusters running Kafka 4.x operate on KRaft exclusively. If you are on Kafka 3.x with ZooKeeper, migration to KRaft is required before upgrading to 4.x.

Tiered storage (KIP-405): Tiered storage allows Kafka to offload older log segments to object storage (S3, GCS, Azure Blob Storage) while keeping recent segments on local broker disk. This decouples data retention from broker disk capacity: you can retain data for months without sizing broker storage to match. Tiered storage reached General Availability in Kafka 3.6 for self-managed clusters, implemented via a pluggable RemoteStorageManager interface. Confluent Cloud has its own implementation; MSK added tiered storage support separately. Check current documentation for feature flags and known limitations in your deployment environment.

Queue semantics (KIP-932): Kafka’s current partition assignment model gives each partition to exactly one consumer in a group, which preserves per-partition ordering but limits work distribution between consumers. KIP-932 proposes a “share group” model where multiple consumers can compete for records within the same partition, enabling queue-like behavior for workloads where ordering is not required and maximum consumer throughput matters. As of this writing, KIP-932 is available as an early access feature in some Kafka versions. Check the Apache Kafka KIP wiki for current status before building on it.

Eligible leader replicas (KIP-966): Previously, if the ISR for a partition became empty, the only options were waiting for a replica to recover or enabling unclean elections. KIP-966 introduced the Eligible Leader Replica (ELR) set: replicas not currently in the ISR but with a sufficient log offset to be safe candidates for election. This reduces partition unavailability in scenarios where the ISR shrinks to zero due to rolling restarts or multi-broker failure, without the data loss risk of unclean elections.

For a complete list of active proposals affecting Kafka’s core architecture, the Apache Kafka KIP wiki is the authoritative reference.

How Netflix uses Apache Kafka in production

Factor House — Tue, 02 Jun 2026 00:00:00 GMT

Netflix’s Kafka story is one of the more thoroughly documented in the industry. Starting with a single pipeline routing 30% of event traffic in early 2015, Netflix has grown to a system processing over 2 trillion events per day across thousands of brokers — with 20,000-plus Flink jobs connected by Kafka topics at the core of its real-time data infrastructure.

The engineering problem Kafka solves at Netflix is scale and decoupling across a massive, distributed microservice estate. Every interaction a member has — a play, a search, a scroll, a title impression — needs to reach multiple downstream systems within seconds. Kafka sits at the centre of that flow.

Company overview

Netflix is a streaming entertainment service operating in over 190 countries, with around 300 million subscribers as of 2025. At its scale, real-time data is not a feature; it drives personalisation, content decisions, studio operations, and service reliability. The platform generates enormous event volumes at all hours across all regions, which means the data infrastructure has to match both the throughput and the latency requirements of those use cases.

Netflix adopted Apache Kafka incrementally. Before Kafka, event data flowed through a system called Suro, a proprietary pipeline open-sourced in 2013. Kafka entered the picture in early 2015 as part of Keystone V1.5 — initially handling about 30% of event traffic alongside Chukwa for real-time consumers. By December 2015, Keystone V2 was in production with Kafka as the primary ingest layer, replacing Chukwa entirely.

Date	Milestone
2013	Suro open-sourced — Netflix’s pre-Kafka event pipeline
Feb 2015	Keystone V1.5 — Kafka introduced, handling ~30% of event traffic
Dec 2015	Keystone V2 live — Kafka becomes the primary fronted gateway
Apr 2016	36 clusters, 4,000+ brokers, 700 billion messages per day
Mar 2018	3,000+ brokers globally, 1 trillion messages per day, 99.99% availability
Sep 2019	Inca — sampled distributed tracing system on dedicated Kafka cluster published
2022	Data Mesh platform: 2 trillion+ events per day, 20,000+ Flink jobs
Nov 2023	Streaming SQL in Data Mesh: 1,200 SQL processors created in first year
Jul 2025	Tudum CQRS architecture replaced — Kafka retired for one read-heavy use case
Oct 2025	Real-Time Distributed Graph published: up to 1 million messages per second per topic
Dec 2025	Live streaming monitoring stack: 38 million events per second peak capacity

Netflix’s Kafka use cases

Netflix uses Kafka across several distinct domains. The use cases range from high-volume, low-latency event ingest to event-driven microservice coordination and real-time member personalisation.

Keystone pipeline — centralised event ingest and routing

The Keystone pipeline is Netflix’s primary data highway. Every interaction a member generates on any device flows through Keystone: plays, pauses, searches, row scrolls, title impressions. Kafka serves as the front door, ingesting these events from client applications and routing them to downstream sinks including Amazon S3, Elasticsearch, Apache Cassandra, and secondary Kafka topics.

The routing tier runs Flink and Samza jobs on EC2, processing events in near real time and updating member taste profiles in Cassandra within seconds of the originating interaction.

Stream processing as a service (Keystone SPaaS)

Keystone’s SPaaS tier makes Flink-over-Kafka available to internal teams as a managed service. Teams submit stream processing jobs without owning infrastructure. Use cases include generating real-time features for recommendation models (replacing 24-hour batch jobs), aggregating A/B test results, and building near-real-time personalisation pipelines.

Nitin Sharma, who led Content Finance infrastructure at Netflix, described the core value: “We’re able to replace batch workflows that ran once a day with streaming pipelines that update model inputs continuously.”

Data Mesh — inter-service data movement and processing

The Data Mesh platform, described in a 2022 Netflix TechBlog post, uses Kafka topics as the connective tissue between processing stages. Every upstream Flink Processor writes output to a Kafka topic; every downstream Processor reads from one. The platform manages topic provisioning, schema registration, and job lifecycle.

As of 2022, Netflix operates 20,000-plus Flink jobs connected by thousands of Kafka topics through this platform. A Streaming SQL layer added in 2023 wraps the Flink Table API behind standard SQL, allowing non-infrastructure teams to build pipelines without writing custom Processors. Within the first year of its launch, 1,200 SQL processors had been created by teams outside the data infrastructure organisation.

Content Finance — event-driven microservices

Netflix’s Content Finance engineering team (budgeting, talent payments, content accounting, cashflow) migrated from synchronous RPC calls to a Kafka-based event-driven architecture. Services now emit standardised events containing an entity ID, UUID, timestamp, operation type, and optional payload. Flink enriches and reorders these events before routing them to keyed Kafka topics consumed by Spring Boot services.

Before the migration, synchronous dependencies between services created cascading failures and split-brain state when an upstream service became unavailable. After the migration, at-least-once delivery with idempotent consumers eliminated the reconciliation overhead that had required manual intervention.

Monitoring and observability

Netflix uses Kafka at multiple layers of its observability stack.

The Title Health microservice ingests real-time title impression events via Kafka to validate content availability — artwork, recommendations, localisation — across thousands of monthly launches. Collector jobs process these events and surface problems before a title reaches significant audience.

The live streaming monitoring stack, introduced as Netflix expanded into live events, uses Kafka alongside Mantis, Atlas, and Druid to process up to 38 million events per second and deliver critical playback quality metrics within seconds. This stack was designed to detect degradation at the speed live events require.

Write-Ahead Log — database resilience

Netflix built a Write-Ahead Log (WAL) abstraction that uses Kafka (alongside Amazon SQS) as a pluggable message queue to decouple producers from consumers in database mutation flows. Each WAL namespace gets a dedicated Kafka topic and a dead-letter queue by default. The WAL supports delay queues, cross-region replication, and multi-table atomic mutations. The most common deployment pattern is as a delay queue for scheduled database writes.

Real-Time Distributed Graph — member taste profiles

Netflix’s Real-Time Distributed Graph (RDG) uses Kafka to carry member actions — views, ratings, interactions — into Flink jobs that populate a graph database representing member preferences. Kafka provides durable, replayable streams so downstream processors can consume events in real time or replay from an earlier offset if needed. Individual Kafka topics in the RDG system carry up to roughly 1 million messages per second.

Scale and throughput

Netflix’s published scale figures show consistent growth over a decade:

Year	Metric	Source
2016	36 clusters, 4,000+ broker instances, 700+ billion messages per day	Netflix TechBlog, “Kafka Inside Keystone Pipeline”
2017	700+ Kafka topics in production, 450 billion events per day	Shriya Arora, QCon New York 2017
2018	3,000+ brokers globally, 1 trillion messages per day, 99.99% availability	Allen Wang, QCon London 2018
2018	Up to 8 million events per second peak throughput	Netflix TechBlog, “Keystone Real-time Stream Processing Platform”
2019	2 billion trace messages per day through Inca (dedicated Kafka cluster)	Allen Wang, Netflix TechBlog
2022	2 trillion+ events per day, 20,000+ Flink jobs connected by Kafka topics	Netflix TechBlog, “Data Mesh”
2023	1,200 SQL processors created in the first year of Streaming SQL launch	Netflix TechBlog, “Streaming SQL in Data Mesh”
2025	Up to ~1 million messages per second per Kafka topic (RDG system)	Netflix TechBlog, “Real-Time Distributed Graph”
2025	38 million events per second peak capacity (live streaming monitoring)	InfoQ, December 2025

‍

Retention policies are tailored per topic based on throughput and record size. In the Keystone batch path, Kafka TTL was set to 4-6 hours with raw events backed up in HDFS and S3 for 1-2 days to support replay when Kafka retention had expired. More recent architectures use Apache Iceberg tables as a backfill source for pipelines that need data beyond Kafka’s retention window.

Netflix’s Kafka architecture

High-level topology

Netflix operates entirely on AWS and chose a multi-cluster topology specifically designed for the cloud’s characteristics. Rather than scaling a single large cluster, Netflix provisions many smaller, mostly immutable clusters. This limits blast radius, simplifies broker upgrades, and aligns with AWS’s preference for stateless, disposable infrastructure.

The Keystone pipeline uses a two-tier cluster layout:

Fronting Kafka clusters receive events from all producers. Their purpose is ingest.
Consumer Kafka clusters serve downstream consumers. Their purpose is read throughput.

Separating these tiers prevents consumer fan-out from affecting producer ingest performance. As the number of downstream consumer groups grew, serving them all from a single cluster became impractical.

Custom smart clients wrap the standard Kafka producer and consumer interfaces. Producers write to fronting clusters; consumers read from consumer clusters. The smart client handles routing and failover transparently, so application code interacts with a logical endpoint rather than a specific cluster.

Schema management

All Kafka events at Netflix use Apache Avro serialisation. Before any event can be written to a topic, its Avro schema must be registered in an internal Schema Registry. This enforces schema-at-producer: malformed or undeclared events cannot enter the pipeline. Avro provides roughly 3-5x size reduction versus JSON, which matters at multi-trillion-event-per-day volumes.

Producer architecture

The Keystone ingest layer accepts events via both a Java library embedded in producer applications and an HTTP proxy for services or devices that cannot embed the library. Producers target the fronting tier. Configuration details for acks, batching, and idempotency have not been published in detail in Netflix’s public posts, but the shift to rack-aware partition assignment (see Special Techniques below) indicates careful tuning of cross-AZ placement.

Consumer architecture

Consumer groups read from the consumer tier clusters. Offset management follows standard Kafka committed-offset patterns, with offset-based lag monitoring feeding the Atlas metrics platform via a custom Kafka MetricReporter. Netflix derives consumer lag by calculating log end offsets minus committed offsets per topic-partition.

A “stuck consumer” detection mechanism flags cases where committed consumer offsets stall within a given window, triggering First Responder, Netflix’s auto-remediation system, to relaunch affected stateless jobs.

Stream processing

Flink is the primary stream processing engine throughout Netflix’s Kafka infrastructure. In the Keystone SPaaS tier, Flink jobs run in Docker containers on EC2 and are managed by the platform’s reconciliation layer. The Data Mesh platform connects thousands of Flink Processors via Kafka topics, with job state stored in Amazon RDS as the authoritative source of truth.

A Streaming SQL layer (added in 2023) wraps the Flink Table API and translates SQL queries into Flink job graphs, eliminating the need for multiple intermediate Processors and Kafka topics for simple transformations.

Kafka Connect ecosystem

Netflix’s public posts do not describe Kafka Connect specifically. The routing layer in Keystone writes to S3 and Elasticsearch using managed Flink/Samza jobs rather than off-the-shelf connectors, suggesting custom integration rather than a standard Connect deployment.

Special techniques and engineering innovations

Rack-aware partition assignment

Allen Wang contributed Kafka’s rack-aware partition assignment feature upstream to Apache Kafka. The feature assigns partitions to brokers and consumers in the same AWS availability zone where possible, reducing cross-AZ data transfer costs and latency. At Netflix’s scale — thousands of brokers across multiple AZs — the economics of cross-AZ data transfer are material.

Immutable, small-cluster design

Netflix deliberately chose many small immutable clusters over one large cluster. In Allen Wang’s words from QCon London 2018, the decision reflects the reality that stateful services are harder to operate in the cloud than stateless ones. Smaller clusters limit the blast radius of failures and make rolling upgrades more predictable. The smart client layer abstracts this topology from application code.

External heartbeat monitoring for Kafka

Netflix’s monitoring service continuously sends heartbeat messages to each Kafka cluster and simultaneously consumes them from the outside. This verifies both the producer path and the consumer path from an external perspective rather than relying solely on broker-internal metrics. Wang presented this pattern at QCon SF 2019 as part of the broader monitoring and tracing infrastructure.

Inca — sampled distributed tracing on a dedicated Kafka cluster

After evaluating and rejecting Zipkin, Netflix built Inca, a message tracing and loss detection system. Trace signals propagate through Kafka record headers. All trace messages are stored on a dedicated Kafka cluster with replication factor 3, min ISR 2, and EBS-backed storage.

The stream processing challenge Inca had to solve was unpredictable trace arrival times: trace records for a single request can arrive out of order and at variable delays. Inca uses Flink’s GlobalWindow with custom triggers and treats committed consumer offsets as an external signal to determine when to close windows, rather than relying on event-time watermarks alone. The result is a false-positive loss-detection rate of 0.005%, processing 2 billion trace messages per day.

Iceberg as a Kafka backfill source

When Kafka retention expires, pipelines that need to replay historical data can fall back to Apache Iceberg data warehouse tables. Kafka topic records are persisted to Iceberg, providing a long-retention complement to Kafka’s short-retention window. This pattern is described in the Data Mesh blog post as part of the backfill story.

Dead-letter queues on every WAL namespace

Netflix’s WAL abstraction automatically provisions a DLQ alongside every Kafka topic it manages. Transient errors are retried with configurable delays; hard errors are parked for manual inspection. The WAL also supports cross-region replication and delay queues for scheduled database writes, without requiring application code to implement these patterns directly.

Operating Kafka at scale

Deployment model

Netflix runs Kafka entirely on AWS, with brokers on EC2 instances and EBS storage for the clusters that require durability (such as Inca’s dedicated tracing cluster). The multi-cluster topology is managed by a control-plane service that handles topic creation, partition metadata, and cluster assignment.

Monitoring and observability

Kafka operational metrics flow to Atlas, Netflix’s internal dimensional time-series metrics platform, via a custom Kafka MetricReporter. Consumer lag (log end offset minus committed offset per topic-partition) is the primary consumption health signal. SLAs are expressed as message consumption lag, transfer rates, and cross-region replication metrics.

For end-to-end pipeline health, Netflix uses watchdog monitors that track event propagation from producer to final sink, not just broker-level metrics.

Auto-remediation

When stuck-consumer alerts fire, First Responder automatically relaunches the affected stateless streaming jobs without human intervention. This reduces on-call toil and accelerates mean time to recovery for the most common class of streaming job failure.

Autoscaling

Capacity for Flink jobs connected to Kafka is calculated using workload predictions (quadratic or linear regression based on historical throughput) to maintain target processing rates during traffic fluctuations. The Keystone SPaaS reconciliation layer continuously compares desired job state (stored in RDS) against actual state (running Flink jobs) and self-heals divergences.

Schema governance

Schema registration is mandatory before any event can be written to a Kafka topic. The internal Schema Registry enforces this at the platform level. Teams cannot publish to Kafka without a registered Avro schema, which prevents malformed events from entering the pipeline and creates a centralised contract for every data stream.

Retention policies

Retention is set per topic based on throughput and record size. The general approach balances data availability for replay against storage cost. For use cases where data needs to outlast Kafka retention, Iceberg tables serve as the long-term store.

Developer experience

The Streaming SQL layer introduced in 2023 was explicitly designed to lower the bar for non-infrastructure teams. Engineers can write SQL queries to describe transformations; the platform translates them into Flink job graphs. Within a year, teams outside the data infrastructure organisation had created 1,200 SQL processors, indicating that the abstraction achieved meaningful adoption without requiring platform expertise.

Challenges and how they solved them

High-level consumer partition loss (2016)

In Keystone V1.5, Netflix used Kafka’s high-level consumer API and observed a known bug where the consumer could lose partition ownership and stop consuming some partitions after running stably for a period. The immediate symptom was data loss, not an error. The solution was moving to Keystone V2’s managed routing service with direct partition assignment and monitoring, replacing the high-level consumer with a more explicitly managed consumption model.

Problem: Partition ownership silently dropped under long-running high-level consumers.

‍Root cause: A known Kafka high-level consumer bug at the time. Solution: Direct partition assignment in the V2 routing layer.

‍Outcome: Partition ownership became explicit and monitorable.

Data loss from low-durability configurations

Allen Wang identified three categories of data loss in Netflix’s production Kafka environment, described at QCon SF 2019:

Replication factor of 2 combined with acks=1 producer configuration. When two brokers hold a partition and the leader fails before replication completes, writes are lost.
Partition leader clock drift causing unexpected log truncation in extreme conditions.
Deployment tooling assigning duplicate consumer group IDs, causing two independent consumer applications to share offsets.

Each was addressed through configuration changes and tooling improvements. The lesson Wang drew: data loss in Kafka rarely comes from Kafka’s own logic — it comes from the configuration choices made around it.

Problem: Unexplained data loss in production, hard to attribute to a single cause.

‍Root cause: Three distinct configuration and tooling mistakes rather than a single failure mode.

‍Solution: Higher replication factor, acks=all for critical topics, unique consumer group ID enforcement.

‍Outcome: Data loss incidents reduced; tracing infrastructure (Inca) built to detect any residual loss.

Consumer fan-out at scale

As the number of downstream consumer groups grew, the original single-cluster design could not efficiently serve hundreds of groups with different lag tolerances and throughput requirements. Producers and consumers competed for broker resources.

Problem: Consumer fan-out degraded ingest performance.

‍Root cause: No isolation between the write path and the read path.

‍Solution: Two-tier cluster topology: fronting clusters for producers, consumer clusters for consumers, with smart clients routing transparently.

‍Outcome: Independent scaling of ingest and consumption capacity.

Late-arriving events in stream processing (2017)

When Netflix migrated batch ETL to Flink and Kafka for recommendation model training, late-arriving events — those arriving after their intended event-time window — caused incorrect attribution in the model inputs. The Kafka TTL was 4-6 hours, which was shorter than the arrival delay for some events.

Problem: Late events attributed to the wrong time window, corrupting model training inputs.

‍Root cause: Event-time windows closed before all relevant events had arrived, and Kafka TTL was shorter than the late-arrival tail.

‍Solution: Time windowing with a post-processing pass to correctly attribute late events; raw events stored in HDFS for 1-2 days beyond Kafka TTL for replay.

‍Outcome: Recommendation model training inputs became accurate and replayable. Shriya Arora presented this at QCon New York 2017.

Microservice dependency brittleness (Content Finance)

Netflix Content Finance services experienced cascading failures and inconsistent state when upstream synchronous dependencies became unavailable. Split-brain state between services required manual reconciliation.

Problem: Synchronous RPC calls created hard dependencies between services.

‍Root cause: No buffering or retry layer between producers and consumers of data changes.

‍Solution: Kafka-based event-driven architecture with standardised event envelopes, at-least-once delivery, idempotent consumers, and reconciliation events for replay. Flink enriches and reorders events before routing to keyed topics consumed by Spring Boot services.

‍Outcome: Services became independently deployable and resilient to upstream failures. Nitin Sharma presented this at Kafka Summit SF 2019.

Tudum cache invalidation lag (2025)

The Tudum fan site used a CQRS pattern with Kafka to decouple CMS ingestion from the Cassandra read database. An ingestion service published read-optimised content to a Kafka topic; a page data service consumed it for storage.

In practice, the CQRS-with-Kafka architecture introduced cache refresh delays. Editors could not preview content updates for up to 60 seconds per key, which created operational friction for a content-heavy site where updates are frequent.

Problem: Cache invalidation lag of up to 60 seconds per key.

‍Root cause: CQRS overhead was not justified by the write volume — Tudum is a read-heavy, low-write-volume site.

‍Solution: Replaced Kafka and CQRS with RAW Hollow, Netflix’s in-memory object store, which handles the read-heavy pattern more directly.

‍Outcome: Simplified architecture with lower latency for content updates. This is a notable documented case where Netflix actively chose to move away from Kafka when a simpler alternative was more appropriate for the use case. Eugene Yemelyanau and Jake Grice documented this on the Netflix TechBlog in July 2025.

Full tech stack

Category	Tools	Notes
Message broker	Apache Kafka	Central message bus for event ingest, inter-service messaging, Data Mesh connective tissue, WAL queue, and monitoring event transport
Stream processing	Apache Flink	All stateless and stateful jobs in Keystone SPaaS and Data Mesh; window sizes from seconds to hours
Serialisation	Apache Avro	Mandatory serialisation format for all Kafka events; ~3-5x smaller than JSON
Schema registry	Internal Netflix Schema Registry	Schema registration required before any event can be written; enforces schema-at-producer
Storage sinks	Amazon S3, Apache Iceberg	S3 for cold storage and raw event backup; Iceberg for structured backfill when Kafka retention has expired
Database / profile store	Apache Cassandra	Member taste profile storage, fed by Flink jobs consuming Kafka streams
Search index	Elasticsearch	Search index sink from Keystone routing; also used in Netflix Studio Search
Real-time analytics	Apache Druid	Used alongside Kafka in the live streaming monitoring stack
Operational stream processing	Mantis	Netflix’s reactive stream processing platform; used in live streaming monitoring stack
Metrics platform	Atlas	Netflix’s internal dimensional time-series metrics platform; receives Kafka operational metrics via custom MetricReporter
Distributed tracing	Inca	Netflix-built message tracing and loss detection on a dedicated Kafka cluster; false-positive detection rate 0.005%
Job state / reconciliation	Amazon RDS	Single source of truth for Keystone SPaaS job desired state; drives reconciliation protocol
Metadata / coordination	Apache ZooKeeper	Kafka cluster metadata management in the Keystone era
Deployment	Spinnaker	Deployment orchestration for Flink jobs and Kafka-connected services
Alternative message queue	Amazon SQS	Used alongside Kafka in the WAL system for specific namespace configurations
Consumer application framework	Spring Boot, Spring Cloud Kafka	Consumer application framework used in Content Finance Engineering
In-memory object store	RAW Hollow	Replaced Kafka-based CQRS for Tudum — more suitable for read-heavy, low-write workloads

Key contributors

Name	Role	Contribution
Allen Wang	Architect, Real Time Data Infrastructure, Netflix	Architected multi-cluster Kafka infrastructure; contributed rack-aware partition assignment to Apache Kafka; spoke at Surge 2016, QCon London 2018, QCon SF 2019, Kafka Summit SF 2017; authored Inca blog post (2019)
Monal Daxini	Lead, Stream Processing, Real Time Data Infrastructure, Netflix	Led Keystone SPaaS; presented at Flink Forward 2016, Flink Forward 2017, AWS re:Invent 2017; co-authored Evolution of the Netflix Data Pipeline (2016)
Shriya Arora	Senior Data Engineer, Netflix	Presented “Migrating Batch ETL to Stream Processing” at QCon New York 2017; described late-arriving event handling and Kafka/Flink migration rationale
Nitin Sharma	Content Finance Infrastructure, Netflix	Authored “How Netflix Uses Kafka for Distributed Streaming” (Confluent blog, 2020); presented “Eventing Things — A Netflix Original!” at Kafka Summit SF 2019
Zhenzhong Xu	Founding engineer, Real Time Data Infrastructure; later led Stream Processing Engines, Netflix	Joined 2015; co-authored Evolution of the Netflix Data Pipeline (2016); published “The Four Innovation Phases of Netflix’s Trillions Scale Real-time Data Infrastructure” (Medium, 2022)
Prudhviraj Karumanchi and Vidhya Arvind	Staff Software Engineers, Data Platform, Netflix	Co-authored “Building a Resilient Data Platform with Write-Ahead Log at Netflix” (September 2025); presented at QCon SF 2024
Eugene Yemelyanau and Jake Grice	Technology Evangelist / Staff Engineer, Netflix	Co-authored “Netflix Tudum Architecture: from CQRS with Kafka to CQRS with RAW Hollow” (July 2025)

Key takeaways for your own Kafka implementation

Netflix’s Kafka architecture offers several decisions worth considering when you are planning or scaling your own implementation.

Separate your ingest and consumption clusters early. Netflix’s split between fronting clusters (producers) and consumer clusters (consumers) addressed a real problem: consumer fan-out degraded producer ingest. If your consumer group count is growing, this separation is worth evaluating before it becomes a performance issue rather than after.
Treat configuration choices as a first-class durability concern. Allen Wang’s account of data loss at Netflix is instructive because none of the three root causes were Kafka bugs. They were configuration and tooling decisions: low replication factor, weak ack settings, and duplicate consumer group IDs. Auditing these is lower-effort than building additional reliability infrastructure.
Enforce schema-at-producer, not schema-at-consumer. Requiring all events to use registered Avro schemas before they reach a topic prevents malformed data from entering the pipeline. Discovering schema issues downstream is significantly more expensive than preventing them at the source.
Build external health checks, not only internal ones. Netflix’s heartbeat monitoring, which writes and reads from every Kafka cluster via an external service, verifies both the producer and consumer paths independently of broker metrics. Broker health and pipeline health are not the same thing.
Revisit architectural choices when use case assumptions change. Netflix’s decision to replace Kafka-based CQRS on Tudum with an in-memory store is a useful reminder that Kafka is not always the right tool. The team documented that CQRS overhead was not justified by the actual write volume. Matching the architecture to the access pattern is more important than consistency for its own sake.

Sources and further reading

Primary sources

Allen Wang, Monal Daxini et al., “Evolution of the Netflix Data Pipeline” — https://netflixtechblog.com/evolution-of-the-netflix-data-pipeline-da246ca36905 (2016)
Netflix TechBlog, “Kafka Inside Keystone Pipeline” — https://netflixtechblog.com/kafka-inside-keystone-pipeline-dd5aeabaf6bb (2016)
Allen Wang, “Cloud-Native and Scalable Kafka Architecture” (QCon London 2018, InfoQ writeup) — https://www.infoq.com/presentations/cloud-native-kafka-netflix/
Allen Wang, “Monitoring and Tracing @Netflix Streaming Data Infrastructure” (QCon SF 2019, InfoQ writeup) — https://www.infoq.com/presentations/netflix-streaming-data-infrastructure/
Allen Wang, “Inca — Message Tracing and Loss Detection For Streaming Data @Netflix” — https://netflixtechblog.medium.com/inca-message-tracing-and-loss-detection-for-streaming-data-netflix-de4836fc38c9 (2019)
Shriya Arora, “Migrating Batch ETL to Stream Processing at Netflix” (QCon NY 2017, InfoQ) — https://www.infoq.com/articles/netflix-migrating-stream-processing/
Netflix TechBlog, “Keystone Real-time Stream Processing Platform” (InfoQ) — https://www.infoq.com/news/2018/09/Netflix-Keystone-Real-Time-Proc/
Nitin Sharma, “How Netflix Uses Kafka for Distributed Streaming” (Confluent blog) — https://www.confluent.io/blog/how-kafka-is-used-by-netflix/ (2020)
Nitin Sharma, “Eventing Things — A Netflix Original!” (Kafka Summit SF 2019) — https://www.slideshare.net/slideshow/eventing-things-a-netflix-original-nitin-sharma-netflix-kafka-summit-sf-2019-179806392/179806392
Bo Lei et al., “Data Mesh — A Data Movement and Processing Platform @ Netflix” — https://netflixtechblog.com/data-mesh-a-data-movement-and-processing-platform-netflix-1288bcab2873 (2022)
Guil Pires et al., “Streaming SQL in Data Mesh” — https://netflixtechblog.com/streaming-sql-in-data-mesh-0d83f5a00d08 (2023)
Eugene Yemelyanau and Jake Grice, “Netflix Tudum Architecture: from CQRS with Kafka to CQRS with RAW Hollow” (InfoQ) — https://www.infoq.com/news/2025/08/netflix-tudum-cqrs-raw-hollow/
Prudhviraj Karumanchi and Vidhya Arvind, “Building a Resilient Data Platform with Write-Ahead Log at Netflix” (InfoQ) — https://www.infoq.com/news/2025/10/netflix-wal-resilience/
Adrian Taruc and James Dalton, “How and Why Netflix Built a Real-Time Distributed Graph” — https://netflixtechblog.com/how-and-why-netflix-built-a-real-time-distributed-graph-part-1-ingesting-and-processing-data-80113e124acc (2025)
InfoQ, “From On-Demand to Live: How Netflix Built a Real-Time Monitoring Pipeline” — https://www.infoq.com/news/2025/12/netflix-live-streaming-pipeline/ (2025)

Kpow

If you are operating Kafka at scale and want full visibility into consumer lag, topic throughput, and broker health from a single interface, give Kpow a try. You can connect it to any Kafka cluster in minutes and get a free 30-day trial with full access — deploy via Docker, Helm, or JAR.

How New Relic uses Apache Kafka in production

Factor House — Tue, 02 Jun 2026 00:00:00 GMT

New Relic processes more than 15 million messages per second across an aggregate data rate approaching 1 Tbps, routing every telemetry data point through Apache Kafka from the moment it arrives to the moment it becomes queryable. Across more than 100 independent clusters, Kafka carries the full pipeline: ingestion, transformation, aggregation, and storage.

The engineering challenge New Relic solved was not just throughput. It was building a data pipeline that could absorb billions of data points per minute, isolate failures to a small portion of customers, and scale horizontally without rebuilding from scratch.

Company overview

New Relic is an observability platform used by engineers to monitor application performance, infrastructure, and distributed systems. The platform ingests data from agents running in customer environments and stores it in NRDB (New Relic Database), a proprietary time-series store capable of scanning trillions of events with a median query response time of around 60 milliseconds.

New Relic has been building on Kafka since at least 2018, when principal software engineer Amy Boyle published the company’s first detailed account of how Kafka connected the platform’s microservices and event processing pipelines. At that stage, the platform ran a single Kafka cluster in its own data centre, with thousands of broker nodes. By 2020, that cluster had reached the limits of horizontal scaling, which triggered a migration to Amazon Web Services and a shift to a cell-based architecture that now underpins the platform’s fault isolation model.

Date	Event
2018	Amy Boyle publishes the first public engineering account of New Relic’s Kafka architecture, covering event processing, the changelog pattern, and the durable cache pattern
2020	New Relic begins migrating from a single on-premises Kafka cluster (275+ brokers) to a cell-based architecture on AWS using Amazon MSK
2021	Migration completes; 95% of data ingestion moved to AWS, with more than 100 independent Kafka clusters now in operation
2021-07-29	Major incident: a Kafka broker failure, conflicting manual and automated recovery, and a retention change propagated across all cells causes a multi-hour data collection and alerting outage for US-region customers
2022-04	Anton Rodriguez presents the company’s eBPF-based Kafka monitoring approach at Kafka Summit London
2024-01	Tony Mancill’s Kafka best practices article updated, citing 15 million messages/sec and an aggregate data rate approaching 1 Tbps

New Relic’s Kafka use cases

Telemetry data bus for NRDB

Kafka serves as the primary data bus for New Relic’s Telemetry Data Platform. As described by Nic Benders, GVP and Chief Architect, Kafka “carries all customer data through each stage of our processing, evaluation, and storage pipeline.” Multiple downstream services aggregate, normalise, decorate, and transform telemetry data as it moves through Kafka before reaching NRDB. In normal operation, Kafka also absorbs backpressure: if a downstream stage slows down, Kafka buffers incoming data without dropping it.

Event data processing - Events Pipeline team

The Events Pipeline team handles fine-grained monitoring records: application errors, page views, and e-commerce transactions. Each record flows through a chain of containerised processing services connected via Kafka topics, passing through parsing, query matching, and aggregation stages in sequence. The team owns the partitioning strategy, consumer assignment configuration, and aggregation pipeline design for this data.

Microservice coordination across 40+ engineering teams

New Relic’s platform is built by more than 40 engineering teams. Kafka decouples these teams through asynchronous messaging, allowing services to produce and consume data independently without tight synchronous dependencies between producers and consumers.

“Time to glass” pipeline

Engineering teams on the Telemetry Data Platform track “time to glass”: how long it takes for a data point ingested by a customer agent to become queryable. Kafka connects each processing stage in this chain, and the pipeline is instrumented with OpenTelemetry distributed tracing to measure latency at each hop.

Scale & throughput

Metric	Figure	Source
Messages per second	More than 15 million	Tony Mancill, New Relic engineering blog
Aggregate data rate	Approaching 1 Tbps	Tony Mancill, New Relic engineering blog
Data points per minute ingested into NRDB	Over 3 billion	Nic Benders; Wendy Shepperd, New Relic engineering blog
Data volume per month	150 PB (at migration, 2021); 200 PB (later post)	Wendy Shepperd; Daniel Kim, New Relic engineering blog
Kafka clusters in operation	More than 100	Anton Rodriguez, Confluent podcast / Kafka Summit London 2022
Brokers in original single cluster	Over 275	Anton Rodriguez, Confluent podcast / Kafka Summit London 2022

Since the AWS migration, clusters are scoped by cell. Each cell contains one Kafka cluster alongside the ingestion service, pipeline services, and NRDB cluster it serves. Customer event data is partitioned by account to support efficient per-customer data locality within each cluster.

New Relic’s Kafka architecture

From monolith to cells

Before 2020, New Relic ran its Kafka workload in a single cluster in its own data centre. As described by Wendy Shepperd, GVP of Engineering, the company had “a massive single Kafka cluster in our data center with thousands of nodes processing data.” That cluster could not scale further horizontally.

The architectural response was to decompose the platform into independent cells, each a self-contained unit: one Kafka cluster, one ingestion service, one set of data pipeline services, and one NRDB cluster. With approximately 10 cells in the US region, a failure in one cell affects roughly 10% of customers rather than the full region. The 2021 incident (described below) revealed that infrastructure automation had not fully respected cell boundaries, resulting in a configuration change propagating across all cells simultaneously.

Amazon MSK

New Relic migrated to Amazon Managed Streaming for Apache Kafka (MSK) for its cell clusters as part of the AWS transition. Containerised consumer services run on Amazon EKS within each cell. The migration moved 95% of data ingestion to AWS in under one year.

Partition model

For permanent event storage, data is “partitioned by which customer account the data belongs to,” supporting efficient per-account retrieval. For the CPU-intensive match service - which must hold all registered queries in memory - random partitioning is used to distribute load evenly across consumer instances without per-key routing logic.

Consumer architecture

The Events Pipeline team manages partition assignment configuration to minimise the cost of rebalances on stateful consumers. StickyAssignor reduces state shuffling during rebalances. CooperativeStickyAssignor (available from Kafka 2.4) allows consumers to continue processing during a rebalance rather than stopping entirely. Static Membership (available from Kafka 2.3) avoids triggering rebalances when clients consistently identify themselves across restarts.

Special techniques & engineering innovations

Changelog pattern for stateful service startup

Services that maintain in-memory state consume their Kafka topic from the earliest offset on startup, replaying history to reconstruct state. The Queries topic uses a short TTL-based retention window of one hour to keep the replay time bounded. An early misconfiguration - segment.ms not aligned with retention.ms - caused the topic to take minutes to read on startup. Aligning the two values resolved the issue immediately.

“Durable cache” pattern

Replaying large volumes of ingest data on every service restart was not practical at New Relic’s scale. The solution was to introduce a “durable cache” pattern: stateful services periodically snapshot their in-memory state to a dedicated Kafka topic (a “snapshots” topic with the same partition count as the primary topic). On restart, a service reads the snapshot, then resumes consuming from the offset stored in the snapshot metadata, bypassing the need to replay the primary topic from the beginning.

LMAX Disruptor for ingest concurrency

New Relic uses the LMAX Disruptor - “an asynchronous blocking queue that distributes objects to worker threads” - in its ingest consumers to parallelise decompression, deserialisation, and business logic across threads. Each worker thread handles one partition, preserving single-threaded ordering guarantees per partition while maximising CPU utilisation across cores.

Two-stage aggregation for hotspot resolution

When the Events Pipeline team discovered that the top 1.5% of customer queries accounted for approximately 90% of processed events, it became clear that partitioning the aggregation topic by query ID was creating severe load imbalance. The solution was to split the aggregation service into two stages. Stage one uses random partitioning, distributing all events evenly across consumer instances for parallel partial aggregation. Stage two partitions by query ID, merging the partial results into final outputs. This arrangement significantly condenses stream volume before the final keyed step, eliminating the hotspot problem.

eBPF-based Kafka monitoring via Pixie

With more than 100 Kafka clusters and services written in multiple languages, deploying per-service Kafka instrumentation was impractical. Anton Rodriguez, Principal Software Engineer, presented New Relic’s solution at Kafka Summit London 2022: Pixie, a CNCF open-source project that uses eBPF (Extended Berkeley Packet Filter) to observe Kafka traffic at the Linux kernel network layer without any application-level code changes. Pixie measures consumer lag in milliseconds rather than offset counts, identifies rebalancing events through control message analysis, and works across consumer languages on Kubernetes. The approach gives New Relic uniform observability across its full cluster estate without per-service instrumentation overhead.

Operating Kafka at scale

Deployment

New Relic runs Kafka on Amazon MSK, with one MSK cluster per cell. Each cell is an isolated unit: its own MSK cluster, ingestion frontend, processing services, and NRDB cluster. Consumer services run as containerised workloads on Amazon EKS within the same cell.

Monitoring and observability

Consumer lag is monitored in milliseconds via Pixie, providing a time-based view of pipeline health rather than a raw offset count. The full Telemetry Data Platform pipeline is instrumented with OpenTelemetry distributed tracing to track data from agent ingestion through each Kafka-connected transformation stage to the point it becomes queryable in NRDB. This “time to glass” metric gives engineers a latency view across the entire processing chain.

Incident response

The July 2021 incident led to a public retrospective in which New Relic identified five compounding failure points in its incident response tooling and automation: no safety guardrails in emergency tools to prevent dangerous configuration changes; human error under high-pressure conditions; false confidence built up from past tool use; alert noise masking critical disk-space warnings; and automation scope that allowed configuration changes to propagate beyond the affected cell. Post-incident commitments included adding guardrails to tooling and restricting the blast radius of automation to individual cells.

Challenges & how they solved them

Horizontal scaling limit on the monolithic cluster

Problem: The original single on-premises Kafka cluster, with thousands of broker nodes, could not scale further horizontally.

Root cause: A monolithic cluster design with no workload isolation between data types, teams, or customers.

Solution: Migration to AWS using Amazon MSK, replacing the single cluster with more than 100 cell-level clusters.

Outcome: 95% of data ingestion moved to AWS in under one year. Each new cell adds capacity independently, and failures are isolated to approximately 10% of customers.

July 2021: broker failure cascades across cells

Problem: A Kafka broker in one US-region cell became unresponsive. Engineers manually initiated a restart while automated remediation triggered simultaneously, extending the degradation window. To preserve data during recovery, engineers extended Kafka retention settings using internal tooling. The retention change was silently applied to all cells by the infrastructure automation, exhausting disk space across the platform. A disk-space alert fired but was missed amid simultaneous unrelated alerts.

Root cause: Five compounding failures: no safety guardrails in emergency tooling; human error under stress; false confidence from prior tool use; alert noise masking a critical signal; and automation scope that bypassed cell isolation.

Outcome: Multi-hour data collection and alerting outage for US-region customers. New Relic published a detailed retrospective committing to tooling and automation improvements. Source: Nic Benders, newrelic.com/blog/best-practices/new-relic-reliability-incident-learning.

Partition hotspots in the aggregation pipeline

Problem: The top 1.5% of customer queries accounted for approximately 90% of processed events. Partitioning the aggregation topic by query ID concentrated this workload on a small number of partitions, causing load imbalance.

Root cause: Uneven distribution of query weight in the customer base.

Solution: Two-stage aggregation pipeline - random partitioning for partial aggregation, then query-ID partitioning for final merge.

Outcome: Hot partitions eliminated; stream volume significantly reduced before the final keyed aggregation stage.

Consumer lag measurement across 100+ clusters

Problem: Instrumenting consumer lag across more than 100 clusters, served by services in multiple languages, was not scalable.

Root cause: Scale and heterogeneity of the cluster estate and application stack.

Solution: eBPF via Pixie, observing Kafka traffic at the kernel network layer without application changes.

Outcome: Consumer lag reported in milliseconds across all clusters and languages; rebalancing events detected from control messages; no per-service instrumentation required.

Log segment misconfiguration causing slow topic consumption

Problem: Applications consuming the Queries topic took minutes to read it on startup, creating unacceptably long initialisation times.

Root cause: segment.ms was not aligned with retention.ms, causing Kafka to create large log segments that contained mostly expired data but still had to be scanned.

Solution: Aligned segment.ms to match retention.ms.

Outcome: Consumption time reduced from minutes to a manageable window.

Full tech stack

Category	Tools	Notes
Message broker	Apache Kafka	Core data bus for NRDB and all inter-service telemetry routing; one cluster per cell
Managed Kafka	Amazon MSK	Kafka hosting for all cell clusters on AWS, adopted during the 2020 migration
Container orchestration	Amazon EKS	Kubernetes platform for containerised Kafka consumers and data pipeline services within each cell
Kafka monitoring	Pixie (CNCF)	eBPF-based consumer lag and rebalance monitoring across 100+ clusters without per-service instrumentation
Distributed tracing	OpenTelemetry	Traces data from agent ingestion through all Kafka-connected processing stages to NRDB for “time to glass” measurement
Concurrency library	LMAX Disruptor	Asynchronous blocking queue used by ingest consumers to parallelise decompression, deserialisation, and business logic
Database	NRDB	Proprietary time-series database; downstream destination for all Kafka-processed telemetry; stores metrics, events, logs, and traces

Key contributors

Name	Title / Team	Contribution
Amy Boyle	Principal Software Engineer, New Relic	Authored the primary engineering posts on event processing architecture (changelog pattern, durable cache, LMAX Disruptor) and partitioning strategies (two-stage aggregation, hotspot resolution, assignor selection)
Tony Mancill	Lead Software Engineer, data ingest team, New Relic	Authored “20 best practices for Apache Kafka at scale,” drawing on production experience at 15 million messages/sec; primary source for throughput figures
Anton Rodriguez	Principal Software Engineer, New Relic	Presented “Monitoring Kafka Without Instrumentation Using eBPF” at Kafka Summit London 2022; primary source for cluster count (100+), broker count (275+), and the Pixie monitoring approach
Nic Benders	GVP and Chief Architect, New Relic	Authored the July 2021 incident retrospective; primary source for cell architecture design and Kafka’s role as the platform data bus
Wendy Shepperd	GVP of Engineering, New Relic	Primary source for the AWS migration: original monolithic cluster constraints, cell architecture shift, Amazon MSK adoption, and the 150 PB/month scale figure
Daniel Kim	Principal Developer Relations Engineer, New Relic	Authored the distributed tracing with OpenTelemetry post; source for the “time to glass” metric and the 200 PB/month data volume figure

Key takeaways for your own Kafka implementation

Cell architecture changes the failure model. New Relic’s shift from a single cluster to 100+ cell-scoped clusters reduced the blast radius of any single failure from 100% of customers to around 10%. If you are running a monolithic cluster and approaching scaling limits, decomposing by workload, region, or customer segment can give you both operational leverage and a clearer capacity model.
State management at ingest scale requires an explicit design. New Relic developed two patterns - the changelog pattern for small, bounded state and the durable cache pattern for large, unbounded state - because the naive approach of replaying the full topic backlog on restart became impractical. If your Kafka consumers maintain in-memory state, these are worth evaluating before you hit scale limits.
Partition hotspots are a distribution problem, not a Kafka problem. The Events Pipeline team’s discovery that 1.5% of queries drove 90% of events required a two-stage aggregation design rather than a configuration change. When you see uneven partition load, the partition key is usually the lever to adjust, and two-stage approaches (random partitioning for distribution, then keyed partitioning for correctness) are a repeatable solution.
Instrumentation at scale may require kernel-level approaches. Once New Relic crossed 100 clusters across heterogeneous service languages, per-service Kafka instrumentation was no longer a viable path. eBPF via Pixie gave uniform coverage without application changes. If you are managing Kafka at a similar scale, it is worth evaluating whether observability approaches that bypass the application layer are better suited to your environment.
Retention changes during incidents carry hidden risk. New Relic’s 2021 post-mortem is a useful reference for any team that uses emergency tooling to modify Kafka configuration under pressure. Extending retention to preserve data is a reasonable instinct, but the interaction between retention settings and available disk space, compounded by automation that did not respect cell boundaries, turned a contained failure into a region-wide outage. Guardrails in operational tooling and well-scoped automation blast radius are worth building before you need them.

Sources & further reading

#	Source	Author	Date
1	Using Apache Kafka for Real-Time Event Processing at New Relic	Amy Boyle, New Relic	2018
2	Effective strategies for Kafka topic partitioning	Amy Boyle, New Relic	Updated 2024
3	20 best practices for Apache Kafka at scale	Tony Mancill, New Relic	Updated January 2024
4	Monitoring extreme-scale Apache Kafka using eBPF at New Relic	Anton Rodriguez, New Relic	Kafka Summit London, April 2022
5	Our commitment to reliability and incident learning	Nic Benders, New Relic	August 2021
6	Transitioning to the cloud: New Relic’s journey to AWS	Wendy Shepperd, New Relic	2021
7	Distributed tracing for Kafka with OpenTelemetry	Daniel Kim, New Relic	2022

‍

If you are running Kafka in production, Kpow gives your team a single interface for monitoring consumer lag, inspecting topics, and managing cluster configuration. You can connect it to any Kafka cluster in minutes and try it free for 30 days.

How Notion uses Apache Kafka in production

Factor House — Tue, 02 Jun 2026 00:00:00 GMT

Notion is a productivity and knowledge management platform that lets individuals and teams build wikis, project trackers, databases, and documents in a single workspace. The core data model is block-based: every piece of content, from a paragraph to a table row to an inline image, is a block stored in Postgres.

Notion grew by millions of members in 2021, driven in part by pandemic-era adoption of remote collaboration tools. That growth exposed the limits of its existing data infrastructure: the legacy messaging architecture was not going to scale, and moving large Postgres datasets to analytics systems was taking more than a day end to end.

Notion began building a new data lake infrastructure in spring 2022, completed it by autumn 2022, and has continued extending it since. The same Kafka-based pipeline that solved the analytics latency problem in 2022 now also powers Notion AI search, retrieval-augmented generation (RAG), and real-time product integrations.

Key Kafka milestones:

Spring 2022: Notion begins building in-house data lake on Debezium, Kafka, Apache Hudi, Spark, and S3
Autumn 2022: Data lake pipeline completed; ingestion latency drops from more than a day to minutes for most tables
2022–2023: Migration to event-driven architecture on Confluent Cloud completed; engineering team reports tripling its productivity
2023–2024: Notion AI features (Autofill, AI search, RAG) launched on top of the Kafka-backed data pipeline
May 2024: Pipeline processing tens of MB/sec of Postgres row changes with minimal operational overhead, delivering net savings of over $1 million per year
April 2026: Multi-region Kafka architecture for data residency compliance documented publicly

Notion’s Kafka use cases

Change data capture for data lake ingestion

The primary use of Kafka at Notion is as the message bus in its CDC pipeline. Debezium captures row-level changes from each Postgres host and publishes them to Kafka. A Spark-based job (Apache Hudi Deltastreamer) consumes those messages and replicates the full state of each Postgres table into S3, where it is available for analytics, AI training, and search indexing.

Before this pipeline existed, batch exports moved data from Postgres to analytics systems in over a day. After the migration, most tables are available within minutes; the largest table (Notion’s block table) takes up to two hours. That latency range is acceptable for the analytics and AI workloads, which do not require sub-second freshness.

Event logging for analytics

The logging API endpoint writes analytics events to a regional Apache Kafka cluster when users take actions such as creating a block. Kafka then sinks that data to a region-specific Snowflake table, where it feeds usage analytics and business reporting.

AI and vector database synchronisation

Updates to workspace content are written to a regional Kafka cluster. Spark jobs consume those messages, generate updated embeddings, and write them back to the regional vector database. This keeps Notion AI’s search and generation capabilities current as users edit their workspaces.

Adam Hudson, Senior Software Engineer at Notion, described the goal: “Confluent’s platform allows us to stream changes as they happen, ensuring AI tools provide relevant information.”

Product integrations and real-time notifications

Notion uses Confluent Cloud’s managed Kafka to power real-time notifications and data synchronisation with third-party tools including Slack, Jira, and Google Drive. Pre-built Confluent connectors replaced a previous custom connector approach. Ekanth Sethuramalingam, Engineering Lead at Notion, described the old approach as “too expensive and difficult to maintain at scale.”

Scale and throughput

As documented by XZ Tie and co-authors (Notion Blog, July 2024) and Ekanth Sethuramalingam and Adam Hudson (Confluent case study):

Daily active users: 100+ million
Postgres shards feeding Kafka: 480
Throughput: Tens of MB/sec of Postgres row changes (as of May 2024)
Ingestion latency: Minutes for most tables; up to two hours for the block table
Cost impact: Net savings of over $1 million in 2022, with proportionally higher savings in 2023 and 2024
Productivity gain: Engineering team tripled its productivity after migrating to Confluent Cloud

Notion’s Kafka architecture

Deployment model

Notion runs Confluent Cloud (fully managed Apache Kafka) integrated with AWS. Ekanth Sethuramalingam summarised the decision: “We believe in solving problems unique to Notion. Having our team manage infrastructure was out of the question.” Confluent Cloud handles automatic cluster scaling, storage retention adjustments, and broker upgrades without manual intervention.

Debezium CDC connectors are the one component that Notion self-manages, running on AWS EKS (Elastic Kubernetes Service). The team configures one Debezium connector per Postgres host.

Topic design

Notion uses one Kafka topic per Postgres table. All Debezium connectors across all 480 Postgres shards write to the same topic for each table. The alternative, one topic per shard per table, would have created 480 times as many topics and significantly complicated downstream Hudi ingestion. The consolidated approach trades some message-level isolation for substantial reductions in topic count and operational complexity.

Regional Kafka clusters

Notion runs separate Kafka clusters per data-residency region, currently US and EU. Each region has its own Debezium, Kafka, Spark, and Snowflake stack. Customer data is written to its region’s Kafka cluster and processed entirely within that region. This design satisfies data residency compliance requirements without requiring message-level filtering or cross-region routing.

Schema management

Confluent Schema Registry is in use for data governance. Notion has also indicated plans to adopt Tableflow, Confluent’s feature for materialising Kafka topics into Apache Iceberg or Delta Lake tables.

Data pipeline flows

CDC data lake pipeline:

Postgres (480 shards)
  -> Debezium CDC (AWS EKS, one connector per Postgres host)
  -> Kafka (one topic per table, Confluent Cloud)
  -> Apache Hudi Deltastreamer (Spark, COPY_ON_WRITE + UPSERT)
  -> S3 (data lake)
  -> Snowflake (Snowpipe or Snowflake Sink Connector)

AI/embeddings pipeline:

Workspace content changes
  -> Regional Kafka cluster
  -> Spark jobs (embedding generation)
  -> Regional vector database

Event logging pipeline:

API endpoint (user action events)
  -> Regional Kafka cluster
  -> Region-specific Snowflake table

Kafka Connect ecosystem

Debezium (source connector) and Snowflake Sink Connector are the two primary connectors in production. S3 and PostgreSQL connectors are also listed as part of Notion’s Confluent Cloud setup.

Special techniques and engineering innovations

Consolidated fan-in from 480 shards into per-table topics

Rather than creating one topic per shard per table, Notion routes CDC events from all 480 Postgres shards into a single topic per table. This reduces the total topic count by a factor of 480 relative to a shard-per-topic design. Consumers, including the Hudi Deltastreamer job, see a unified stream of row changes regardless of which shard originated them.

Apache Hudi with COPY_ON_WRITE and UPSERT for update-heavy workloads

Notion’s data is highly update-driven. A collaborative document block can be edited continuously by multiple users. Notion evaluated several open table formats and chose Apache Hudi specifically because of its native integration with Debezium CDC message formats and better performance characteristics on workloads where updates dominate over appends. They use the COPY_ON_WRITE table type with UPSERT operations, which rewrites data files on each batch to maintain a fully queryable S3 table.

Full-stack regional replication for data residency

Notion’s approach to data residency is to avoid cross-region data flows entirely rather than filter or redact messages in transit. Each data-residency region runs its own complete Kafka-based pipeline stack. When a user in the EU edits a block, the change is captured locally, written to the EU Kafka cluster, processed by EU Spark jobs, and stored in the EU vector database and Snowflake instance. No portion of that data touches infrastructure outside the EU.

Operating Kafka at scale

Deployment model: Confluent Cloud for the Kafka broker layer; AWS EKS for Debezium connectors. The decision to use a fully managed broker service was explicit: Notion’s engineering team is focused on product problems, and managing Kafka broker infrastructure was considered a poor use of that capacity.

Connector stability: Because of the maturity of Debezium and EKS management tooling, Notion has upgraded the EKS and Kafka clusters only a few times over two years of production operation. The authors of the data lake post note this as one of the practical benefits of the chosen stack.

Data freshness monitoring: Notion tracks end-to-end ingestion latency per table. Small tables appear in Snowflake within minutes; the block table, which is the largest and most heavily updated, takes up to two hours. This per-table latency tracking allows the team to detect ingestion slowdowns before they affect downstream analytics or AI features.

Pre-built connector strategy: Rather than building custom connectors to third-party tools, Notion relies on Confluent Cloud’s pre-built connector library. This reduces maintenance overhead and allows the data platform team to focus on pipelines that are specific to Notion.

Challenges and how they solved them

Legacy messaging architecture couldn’t scale with rapid growth

Notion grew by millions of members in 2021. The existing messaging infrastructure was not built for that scale, and custom connectors to third-party tools required ongoing maintenance investment. Notion migrated to an event-driven architecture on Confluent Cloud. Within approximately one year, the migration was complete, the custom connector approach was retired, and the engineering team reported tripling its productivity.

Batch Postgres exports taking more than a day to reach analytics

Before the data lake migration, moving Postgres data to analytics systems relied on batch exports that took over a day end to end. That lag made it difficult to build analytics and AI features that required reasonably fresh data. Notion built the CDC pipeline with Debezium, Kafka, and Hudi Deltastreamer to replace batch exports with a continuous incremental stream. End-to-end ingestion time dropped to minutes for most tables.

Data residency compliance across regions

Expanding internationally required Notion to guarantee that EU customer data would never leave the EU. A centralised Kafka cluster would require complex per-message routing logic to satisfy this. Notion’s solution was to replicate the entire pipeline stack independently in each region. Cross-region data flows are not needed because each region is self-contained.

Full tech stack

Category	Tools	Notes
Message broker	Apache Kafka (Confluent Cloud)	Fully managed; one topic per Postgres table
CDC connector	Debezium	Deployed on AWS EKS; one connector per Postgres host
Stream processing	Apache Spark (Hudi Deltastreamer)	Consumes Kafka topics; COPY_ON_WRITE + UPSERT writes to S3
Schema registry	Confluent Schema Registry	Data governance; Tableflow adoption planned
Connectors	Debezium (source), Snowflake Sink Connector, S3 Connector, PostgreSQL Connector	Pre-built Confluent connectors used throughout
Storage	Amazon S3	Data lake; raw and processed data
Data warehouse	Snowflake	Analytics target; loaded via Snowpipe or Snowflake Sink Connector
Compute/orchestration	AWS EKS	Debezium connector deployment
Vector database	Regional vector DB (undisclosed)	Updated via Kafka and Spark embedding pipeline
Source database	PostgreSQL (480 shards)	CDC source

Key contributors

XZ Tie, Nathan Louie, Thomas Chow, Darin Im, Abhishek Modi, Wendy Jiao — Co-authors of the data lake architecture post, Notion Blog, 1 July 2024
Ekanth Sethuramalingam (Engineering Lead, Notion) — Quoted on the migration to Confluent Cloud and the rationale for managed infrastructure, Confluent case study
Adam Hudson (Senior Software Engineer, Notion) — Quoted on Confluent’s role in AI feature delivery; co-authored the multi-region data systems post, Confluent case study and Notion Blog, 9 April 2026
Justin Lee (Engineering, Notion) — Co-authored the multi-region data systems architecture post, Notion Blog, 9 April 2026

Key takeaways for your own Kafka implementation

Consolidating multi-shard CDC into per-table topics reduces operational complexity significantly. If you are running CDC from a sharded Postgres setup, routing all shards to a single topic per table rather than a topic per shard per table can reduce your total topic count by an order of magnitude and simplify consumer configuration.
Choose your table format based on your write pattern, not just ecosystem popularity. Notion evaluated open table formats and selected Apache Hudi specifically because of its update-heavy workload characteristics and native Debezium CDC integration. If your CDC workload is predominantly updates rather than inserts, Hudi’s COPY_ON_WRITE UPSERT semantics may suit it better than alternatives.
Fully managed Kafka shifts the cost from operations to product. Notion’s explicit reasoning for choosing Confluent Cloud was that managing Kafka broker infrastructure was not a good use of engineering capacity. Two years in, they had upgraded their clusters only a few times. If your team’s core competency is product rather than infrastructure, the operational savings from managed Kafka can be significant.
Data residency compliance is easier to design in than to bolt on. Notion handles data residency by running entirely independent Kafka stacks per region rather than implementing cross-region filtering. If you are building multi-region infrastructure, structuring each region as a self-contained pipeline from the start avoids the complexity of message-level routing and filtering later.
Track ingestion latency per table, not just aggregate throughput. Notion monitors end-to-end latency by table rather than relying on a single pipeline-wide metric. This lets the team quickly identify when a specific table is running behind without needing to diagnose the entire pipeline.

Sources and further reading

Primary sources

XZ Tie, Nathan Louie, Thomas Chow, Darin Im, Abhishek Modi, Wendy Jiao, “How Notion build and grew our data lake to keep up with rapid growth” (1 July 2024)
Ekanth Sethuramalingam, Adam Hudson (quoted), “How Notion Scales its AI and Lowers Operations Costs With Confluent” (Confluent, accessed May 2026)
Justin Lee, Adam Hudson, “Enabling Multi-Region Data Systems at Notion” (9 April 2026)

Try Kpow with your Kafka cluster

If you are monitoring a Kafka cluster at any scale, you can try Kpow free for 30 days. It connects to any Kafka cluster in minutes and deploys via Docker, Helm, or JAR.

How PagerDuty uses Apache Kafka in production

Factor House — Tue, 02 Jun 2026 00:00:00 GMT

PagerDuty operates one of the more instructive Kafka deployments in the incident management space: Apache Kafka underpins at least four distinct production systems, each with its own partitioning strategy, language runtime, and failure mode. The engineering blog documents these systems in unusual detail, including a candid post-incident report from August 2025 in which a single API quirk in the pekko-connectors-kafka library generated 4.2 million new Kafka producers per hour, 84 times the normal rate, exhausting JVM heap across the cluster and rejecting roughly 95% of incoming events for 38 minutes.

Kafka’s role at PagerDuty is to decouple a high-throughput alerting platform from the downstream systems responsible for processing events, scheduling notifications, and delivering them to on-call engineers, often under strict latency requirements.

Company overview

PagerDuty provides incident management and on-call scheduling software used by operations teams to detect, triage, and respond to production incidents. The platform handles hundreds of millions of API requests daily from customer monitoring systems, and downstream must reliably reach the right on-call engineer through the right contact method, at the right time.

PagerDuty’s adoption of Kafka was not a single architectural decision but a gradual replacement of earlier queuing approaches. The original notification system, called Artemis, used Apache Cassandra as a polling-based message queue. By 2017, PagerDuty engineers were publishing blog posts describing a Kafka-backed distributed task scheduler. By 2018, a senior engineer presented a Kafka Summit talk describing the migration of the core event ingestion pipeline from Cassandra-based queuing to Kafka. By January 2019, the Notification Scheduling Service (NSS) had replaced Artemis entirely and was live in production.

Date	Event
April 2017	David Van Geest publishes the first post in the “Distributed Task Scheduling with Akka, Kafka, Cassandra” series, describing the Scheduler architecture
August 2017	Part 2 of the Scheduler series published, covering dynamic load handling
October 2018	Chris Morris presents “From Propeller to Jet” at Kafka Summit San Francisco, describing the migration from Cassandra queuing to Kafka for event ingestion
December 2018	PagerDuty/scheduler open-sourced on GitHub
January 2019	Notification Scheduling Service (NSS), the Elixir/Kafka replacement for Artemis, goes live
August 2019	Jon Grieman presents the Elixir + CQRS incident timeline service at ElixirConf; notes it has been in production for three years
May 2020	Tanvir Pathan publishes “Writing Intelligent Health Checks for Kafka Services”
June 2020	Elora Burns publishes “Elixir at PagerDuty: Faster Processing with Stateful Services”, documenting NSS architecture and performance results
May 2021	Tamim Khan publishes post-mortem of the dentry cache investigation affecting Kafka hosts in staging
28 August 2025	Two Kafka outages (03:53 UTC and 16:38 UTC) disrupt event ingestion for US customers due to a pekko-connectors-kafka producer proliferation bug
4 September 2025	PagerDuty publishes a full post-incident report for the August 28 outages

PagerDuty’s Kafka use cases

PagerDuty uses Kafka across four production systems. Each serves a distinct function and was built by a different engineering team, which explains the variation in language runtimes, partitioning strategies, and architectural patterns across the deployment.

Event ingestion pipeline

Kafka is described by PagerDuty engineers as “the backbone of our decoupled, async architecture.” Customer monitoring systems publish alerts through PagerDuty’s Events API; those events are written to Kafka topics and consumed by downstream services that handle incident creation, escalation routing, and notification dispatch.

The Event Ingestion Admin (EIA) service sits in this pipeline. It consumes from multiple Kafka topics, including incoming_events and failed_events, and persists event state to ElastiCache for observability. The EIA service is written in Elixir and uses a GenServer-based consumer that polls partition offsets every 10 seconds to detect consumer stall.

Notification Scheduling Service

The Notification Scheduling Service (NSS) replaced the earlier Artemis system, which used Cassandra as a polling queue. Artemis required all processing to run in a single US West Coast datacenter to keep Cassandra query latency acceptable. NSS, built in Elixir on the BEAM runtime, uses Kafka to distribute notification traffic while decoupling message distribution from datacenter placement.

NSS publishes three categories of message across three parallel topic sets: updates to incident assignments and state, updates to user contact methods and notification rules, and feedback from downstream systems such as telephony confirmations. Users are consistently routed to the same partition set across all three topic categories, which allows a single Elixir process to hold all state for a given user in memory without cross-process synchronisation.

Elora Burns documented the results when NSS went live in January 2019: 10x throughput, half the lag time, and one-tenth the compute and storage footprint compared to Artemis.

Distributed task scheduler

The Scheduler library, open-sourced by PagerDuty in December 2018, uses Kafka as its queuing and partitioning layer for scheduled task execution. Services publish serialised task metadata to a Kafka topic; the Scheduler library consumes those messages and executes each task at its scheduled time. Use cases include emailing an engineer two days before an on-call shift and push-notifying a responder one minute after an incident opens.

Cassandra provides durable task persistence alongside Kafka: tasks are written to Cassandra at publication time and removed on completion. The Kafka message carries enough information to locate the persisted task record; Cassandra is the source of truth for durability.

Incident timeline CQRS service

An Elixir service powering PagerDuty’s incident timeline entries uses Kafka with a CQRS pattern. This service refactored a component of the Rails monolith into an Elixir umbrella application backed by Kafka for event streaming. Jon Grieman described it at ElixirConf 2019 as one of PagerDuty’s most frequently used services at that time, and noted it had already been in production for three years.

Scale and throughput

PagerDuty does not publish aggregate cluster-level metrics, but several concrete figures appear across the engineering blog posts and the August 2025 incident report.

Metric	Value	Source
Partitions per topic (EIA service)	64-100	Tanvir Pathan, May 2020
Kafka producer rate at incident peak	4.2 million new producers/hour (84x normal)	PagerDuty incident report, September 2025
Event rejection rate at peak	~95% of create event requests rejected	PagerDuty incident report, September 2025
Sustained error rate	18.87% of create event requests returning 5xx	PagerDuty incident report, September 2025
Cluster partition limit (noted as a constraint)	~50,000 partitions per cluster	Elora Burns, June 2020
Scheduler topic partition count	“Low hundreds” at provisioning time	PagerDuty/scheduler GitHub, design.md
NSS throughput improvement over Artemis	10x throughput, 0.5x lag, 0.1x compute and storage	Elora Burns, June 2020

‍

No absolute messages-per-second or bytes-per-second figures have been published. No total topic count or cluster count is disclosed publicly.

PagerDuty’s Kafka architecture

Deployment model

PagerDuty operates a multi-datacenter deployment. The Scheduler documentation describes runbooks for isolating a degraded datacenter by sequentially shutting down application nodes, Kafka brokers, and Cassandra nodes in the affected site, then allowing the healthy datacenter to recover. The NSS post notes that decoupling message distribution from datacenter placement was a primary motivation for replacing Artemis.

Multiple Kafka clusters are in use. The Scheduler documentation references a shared internal “bitpipe” Kafka cluster offered as a shared resource, separate from per-service clusters. Individual topics use partitioned replication with leader/follower broker configuration.

Broader data platform

Kafka acts as the async messaging layer between the Events API and downstream processing services, including the EIA service, NSS, and the incident timeline service. Cassandra provides durable persistence alongside Kafka in both the Scheduler and the earlier Artemis system. ElastiCache (Redis) stores ingested event state consumed from EIA Kafka topics. The JVM-side producer code uses Akka (now Pekko), while NSS and the EIA service run on Elixir/BEAM.

Producer architecture

The Events API service uses pekko-connectors-kafka for its Kafka producer abstraction. PagerDuty’s August 2025 incident traced directly to how this library handles producer instantiation: passing Kafka settings to the library creates a new producer instance silently, without a new keyword, rather than reusing an existing one. This API design was characterised in the incident report as a “visibility gap” that made the bug difficult to catch in code review.

The Transactional Outbox pattern is used to guarantee at-least-once delivery. Messages are written to a database table as part of the originating database transaction; a scraper then reads pending messages and publishes them to Kafka. During the August 2025 outage, this ensured no event data was lost even while the Kafka tier was degraded.

Consumer architecture

The Scheduler library implements a single-threaded Kafka consumer poll loop. All partition reassignment callbacks (onPartitionsRevoked, onPartitionsAssigned) execute within this poll thread, enabling synchronous blocking during rebalances. On partition revocation, the actor system shuts down, in-flight tasks complete, and database connections close before the partition is yielded. This design avoids concurrency issues during rebalancing at the cost of a simpler execution model.

The EIA service uses a GenServer-based consumer in Elixir. The health check polls Kafka partition offsets every 10 seconds and compares the result against a stored snapshot to detect consumer stall, independent of message volume. Consul pings the health endpoint on the same 10-second interval.

Stream processing

ksqlDB is referenced in the August 2025 incident report as part of the Kafka tier, but no further architectural detail is available in published sources.

Special techniques and engineering decisions

Consistent user-to-partition routing in NSS

The NSS design routes each user to a fixed set of Kafka partitions across all three parallel topic sets. The mapping is static: all traffic for a given user arrives on the same partitions regardless of which topic set carries the message. This allows a single Elixir Partition Owner process to hold the complete state for each user in memory, eliminating cross-process synchronisation. The approach accepts the constraint that repartitioning requires redeployment in exchange for simplicity in the stateful consumer design.

Partition-locked serial task execution in the Scheduler

The Scheduler library assigns tasks to partitions using a modulo-based Partitioner class keyed on orderingId. All tasks for the same orderingId route to the same partition and are processed serially by a single node. This guarantees ordering within a task group without any distributed locking. The partition count is fixed at provisioning time in the “low hundreds” because Kafka does not support changing partition count at runtime for an existing topic.

Transactional outbox for delivery guarantees

PagerDuty applies the Transactional Outbox pattern as a reliability layer between its application databases and Kafka. Messages are written to a database table within the same transaction as the business operation that generates them. A background scraper reads the outbox table and publishes messages to Kafka. This separates the durability guarantee (the database transaction) from the messaging guarantee (Kafka delivery), and proved effective during the August 2025 outage when Kafka was temporarily unavailable but no events were lost.

Single-threaded consumer poll loop for controlled rebalancing

Rather than handling partition assignment callbacks on a separate thread, the Scheduler’s consumer design runs all Kafka consumer interaction, including reassignment callbacks, within a single poll thread. This simplifies shutdown during rebalancing: the callback can block, wait for in-flight work to drain, and close resources in a predictable order before yielding the partition. The trade-off is that the poll thread cannot process new messages during a rebalance, which the design accepts as a reasonable cost for operational simplicity.

Operating Kafka at scale

Monitoring and observability

The Scheduler service uses Datadog dashboards with configurable tags to track instances across applications and environments. Published metrics include tasks enqueued to Kafka, task batch persistence to Cassandra, task execution counts, task queue depth in the SchedulerActor and ExecutorActor, and stale task counts via raw Cassandra queries.

The EIA service uses a GenServer-based health check that polls Kafka partition offsets every 10 seconds and compares them against a stored snapshot. Two earlier implementations were discarded: a time-based threshold approach produced false positives on low-throughput topics with natural traffic gaps, and an offset-tracking approach that updated state on every message caused the GenServer to crash under production load.

The August 2025 incident report identified gaps in monitoring that allowed the producer proliferation bug to go undetected until customer impact was severe: no alerting on Kafka producer count, no JVM heap alerting on the Kafka tier, no Kafka producer or consumer telemetry anomaly detection, and no automated service dependency map visibility.

Incident response and failover

For the Scheduler service, the documented runbook for datacenter degradation is: shut down all Scheduler application nodes, then shut down all Kafka brokers, then shut down all Cassandra nodes in the affected datacenter. This sequence allows the healthy datacenter to recover without split-brain state.

The August 2025 outage was mitigated in both instances by doubling JVM heap size and performing rolling restarts. Root cause identification required correlating JVM heap exhaustion with Kafka producer count metrics, a connection that was not immediately obvious from the monitoring available at the time.

Planned improvements post-August 2025

Following the incident, PagerDuty committed to: expanding JVM and Kafka-level monitoring, strengthening service dependency mapping, automating customer impact metrics into incident workflows, adding stricter change management guardrails with rollout readiness checklists, running monthly chaos engineering drills for incident procedures, and automating status page update workflows.

Challenges and how they solved them

Producer proliferation from a library API quirk (August 2025)

Problem: A new API usage-tracking feature caused 4.2 million new Kafka producers to be created per hour at 75% feature rollout, 84 times the normal rate, exhausting JVM heap across the cluster and rejecting roughly 95% of incoming events over a 38-minute window.

Root cause: The pekko-connectors-kafka library instantiates a new Kafka producer when Kafka settings are passed directly to it, rather than when a pre-created producer instance is passed. The API does not require a new keyword, making instantiation implicit and easy to miss in code review. The new feature triggered this path on every API request.

Solution: Mitigation in both incidents involved doubling JVM heap size and performing rolling restarts. After the second incident, engineers identified the root cause and rolled back the feature.

Outcome: No event data was lost due to the Transactional Outbox pattern. PagerDuty committed to a range of monitoring, change management, and chaos engineering improvements to prevent similar gaps. The incident affected US customers across two separate 38-minute windows separated by approximately 13 hours on August 28, 2025.

Dentry cache exhaustion causing Kafka host unresponsiveness (2020-2021)

Problem: Kafka hosts in PagerDuty’s staging environment became intermittently unresponsive for tens of seconds, causing client connectivity failures, under-replicated partitions, and leader elections. Newly provisioned machines showed identical symptoms after several days, ruling out hardware degradation.

Root cause: A third-party vendor application generated filesystem lookups for non-existent file paths on the root directory, creating massive negative dentry cache entries in the Linux kernel. perf top showed the kernel spending 55% of CPU time in __fsnotify_update_child_dentry_flags. The negative dentry entries accumulated over time until they triggered the lockup behaviour.

Solution: Dropping the dentry cache immediately resolved the lockup. The vendor issued a patched build that fixed the root cause.

Outcome: The investigation, documented by Tamim Khan, illustrates how Kafka host instability can originate entirely outside Kafka itself, and highlights the value of kernel-level profiling tools when broker behaviour is inexplicable at the application layer.

Health check false positives for low-throughput topics (2020)

Problem: The EIA service’s first health check approach flagged a consumer as unhealthy if no message had been processed in the previous 10 seconds. This produced false positives on topics with naturally sparse traffic.

Root cause: The threshold-based design assumed continuous message flow. A second approach that tracked offsets on every event crashed the GenServer under production message volumes.

Solution: The final design polls Kafka partition offsets every 10 seconds and compares the result against a stored snapshot. If offsets have not advanced across two consecutive polls on an active topic, the consumer is flagged as stalled. This approach is independent of message volume and stable under production load.

Scaling notification delivery beyond Cassandra polling (pre-2019)

Problem: The Artemis notification system used Cassandra polling as its queuing mechanism. The polling approach tied all processing to a single US West Coast datacenter to keep query latency acceptable, limiting geographic redundancy and imposing high infrastructure costs.

Solution: NSS replaced Artemis with an Elixir/Kafka design that routes users to fixed partition sets across topic categories. Kafka handles message distribution; Elixir processes on BEAM hold user state in memory without database polling.

Outcome: 10x throughput, half the lag, and one-tenth the compute and storage footprint at launch. The design also removed the datacenter placement constraint that had limited Artemis.

Full tech stack

Category	Tool	Role
Message broker	Apache Kafka	Core async messaging layer for event ingestion, notification delivery, task scheduling, and incident timeline event streaming
Stream processing	ksqlDB	Referenced as part of the Kafka tier in the August 2025 incident report; no further architectural detail published
Storage	Apache Cassandra	Durable task persistence in the Scheduler; primary datastore in the prior Artemis notification system
Storage	ElastiCache (Redis)	Stores ingested event state consumed from EIA Kafka topics
Runtime (JVM)	Scala with Akka / Pekko	Implementation language for the Scheduler library; Pekko (formerly Akka) provides the actor framework and the pekko-connectors-kafka Kafka client abstraction
Runtime (BEAM)	Elixir	Runtime for NSS, the EIA health check service, and the incident timeline CQRS service
Kafka client	pekko-connectors-kafka	Kafka producer and consumer abstraction used by the Events API JVM service; source of the August 2025 producer proliferation bug
Service discovery	Consul	Service discovery and health endpoint polling at 10-second intervals for the EIA service
Monitoring	Datadog	Dashboards and metrics for the Scheduler service, tracking task enqueue rates, execution counts, and queue depth
Architectural pattern	Transactional Outbox	Guarantees at-least-once Kafka message delivery by persisting messages to a database table before publishing
Architectural pattern	CQRS	Applied to the incident timeline Elixir/Kafka service to separate write and read models

Key contributors

Name	Title / team	Contribution
David Van Geest	Engineer, PagerDuty	Authored the “Distributed Task Scheduling with Akka, Kafka, Cassandra” blog series (April and August 2017); primary designer of the open-source Scheduler library
Chris Morris	Senior Software Engineer, PagerDuty	Presented “From Propeller to Jet: Upgrading Your Engines Mid-Flight” at Kafka Summit San Francisco 2018, describing the migration from Cassandra queuing to Kafka for event ingestion
Elora Burns	Engineer, PagerDuty	Authored “Elixir at PagerDuty: Faster Processing with Stateful Services” (June 2020), documenting the NSS Kafka partitioning architecture and performance results
Tanvir Pathan	Engineer, PagerDuty	Authored “Writing Intelligent Health Checks for Kafka Services” (May 2020), detailing the EIA Kafka consumer health check design and iteration
Tamim Khan	Engineer, PagerDuty	Authored “How Lookups for Non-Existent Files Can Lead to Host Unresponsiveness” (May 2021), diagnosing the dentry cache issue affecting Kafka hosts in staging
Jon Grieman	Senior Software Developer, PagerDuty	Presented “Elixir + CQRS: Architecting for Availability, Operability, and Maintainability at PagerDuty” at ElixirConf 2019, covering the Kafka-backed incident timeline service

Key takeaways for your own Kafka implementation

Consistent partition routing enables stateful in-memory consumers. NSS’s fixed user-to-partition mapping means each Elixir process owns a predictable slice of user state. If you’re building a stateful Kafka consumer, designing your partition key and count before you go live gives you a stable foundation that is difficult to change later.
Library API design can conceal instantiation behaviour. The pekko-connectors-kafka producer proliferation bug originated from an API where passing a settings object to a method created a new producer silently, without any indication to the caller. When adopting a Kafka client library, verify how it handles producer and consumer lifecycle, particularly whether convenience methods reuse or recreate underlying resources.
The Transactional Outbox pattern decouples durability from availability. PagerDuty lost no event data during a 38-minute outage because the business transaction and the Kafka publish were separated by an outbox table. If you cannot tolerate event loss during Kafka unavailability, the pattern is worth the additional complexity of the scraper component.
Kafka host instability can originate outside Kafka. The dentry cache exhaustion case shows that Kafka broker unresponsiveness was caused entirely by a third-party application generating filesystem lookups on the same host. Kernel-level profiling (perf top) was required to identify the root cause. If you experience intermittent broker lockups that don’t correlate with Kafka metrics, look at what else is running on the broker hosts.
Partition count is a provisioning-time decision. The Scheduler library documentation explicitly notes that partition count cannot be changed at runtime and recommends setting it in the “low hundreds” upfront. PagerDuty also notes the approximately 50,000-partition-per-cluster limit when designing NSS. Plan your partition count before your first production deployment, not as a response to scale pressure.

Sources and further reading

August 28 Kafka outages: what happened and how we’re improving — PagerDuty Engineering Blog, September 2025
Writing intelligent health checks for Kafka services — Tanvir Pathan, PagerDuty Engineering Blog, May 2020
Elixir at PagerDuty: faster processing with stateful services — Elora Burns, PagerDuty Engineering Blog, June 2020
Distributed task scheduling with Akka, Kafka, and Cassandra (Part 1) — David Van Geest, PagerDuty Engineering Blog, April 2017
Distributed task scheduling with Akka, Kafka, and Cassandra (Part 2) — David Van Geest, PagerDuty Engineering Blog, August 2017
How lookups for non-existent files can lead to host unresponsiveness — Tamim Khan, PagerDuty Engineering Blog, May 2021
From Propeller to Jet: Upgrading Your Engines Mid-Flight — Chris Morris, Kafka Summit San Francisco 2018
Elixir + CQRS: Architecting for Availability at PagerDuty — Jon Grieman, ElixirConf 2019
PagerDuty/scheduler — GitHub, PagerDuty open-source (design.md, operations.md, user.md)
PagerDuty Kafka Outage (InfoQ coverage) — InfoQ, September 2025

If you’re running Kafka in production, Kpow gives you a unified interface for monitoring consumer lag, inspecting topic data, and managing cluster configuration across multiple clusters. You can connect it to any Kafka cluster in minutes and try it free for 30 days.

How Pinterest uses Apache Kafka in production

Factor House — Tue, 02 Jun 2026 00:00:00 GMT

Pinterest runs one of the largest Kafka deployments in production: more than 3,000 brokers across 50+ clusters, handling 40 million messages per second at inbound peak and over 1.2 petabytes of data per day. What makes the story worth studying is not the scale alone, but the engineering that Pinterest built around it — a custom management platform, a byte-level replication enhancement, a broker-decoupled tiered storage sidecar, and eventually a second PubSub system that uses Kafka as its own internal notification queue.

Kafka sits at the center of Pinterest’s data infrastructure: it carries user activity events from hundreds of millions of users, drives real-time advertising budget enforcement, feeds machine learning signal pipelines over 300 billion ideas, and now underpins a next-generation CDC ingestion system that reduced data latency from 24 hours to under 15 minutes.

Company overview

Pinterest is a visual discovery platform with hundreds of millions of active users who save, search, and engage with ideas across categories ranging from home design to food to fashion. The platform’s core data challenge is connecting users to relevant content at scale, which requires near-real-time understanding of what users are doing across a corpus of 300 billion ideas.

Pinterest began building its Kafka-based logging infrastructure around 2014, when the engineering team evaluated open-source logging alternatives and started work on Singer, an internal logging agent that would eventually ship over one trillion messages per day to Kafka. The first public description of the Kafka pipeline appeared in February 2015, describing a chain from Singer through Kafka into Secor (a log persistence service) and then to S3 for offline analytics via Spark.

By November 2018, Pinterest was running 2,000+ brokers, handling 800 billion messages per day at a peak rate of 15 million messages per second. By December 2020, those figures had grown to 3,000+ brokers across 50+ clusters with a peak inbound throughput of 40 million messages per second.

Date	Event
2014	Pinterest begins evaluating open-source logging alternatives; Singer development starts
2015-02	First public description of Kafka-based real-time analytics pipeline (Singer → Kafka → Secor → S3 → Spark/MemSQL)
2017-08	DoctorKafka open-sourced (pinterest/DoctorK); managing 1,000+ brokers at the time
2018-11	Scale snapshot: 2,000+ brokers, 800 billion messages/day, 1.2 PB/day, 15 million messages/sec peak
2019-04	AZ-aware partitioning deployed, delivering 25% reduction in cross-AZ transfer costs
2019-06	Singer open-sourced (pinterest/singer)
2019-10	Real-time experiment analytics pipeline with Flink reaches production — described as Pinterest’s first Flink application in production
2019	Formal evaluation of PubSub scalability alternatives begins; leads to MemQ design
2020	Shallow Mirror deployed in production; major Kafka upgrade to 2.3.1 completed; broker fleet migrated from magnetic to SSD storage
2020-mid	MemQ enters production
2020-12	Scale snapshot: 50+ clusters, 3,000+ brokers, 3,000+ topics, ~500,000 partitions, 40 million+ messages/sec inbound peak
2021-11	MemQ announced publicly: up to 90% cheaper than equivalent Kafka deployment for ML data transport
2022-03	Unified PubSub Client (PSC) announced; Kafka and MemQ abstracted behind Topic URIs
2023-11	PSC in production at scale: over 90% Java app migration, 100% Flink migration, over 80% reduction in Flink restarts
2024-05	Broker-decoupled tiered storage enters production (20+ topics, ~200 TB/day offloaded to S3)
2026-02	Next-generation CDC ingestion pipeline published: Debezium/TiCDC → Kafka → Flink → Iceberg reduces data latency from 24+ hours to 15 minutes across thousands of pipelines

Pinterest’s Kafka use cases

Kafka at Pinterest spans data transport, real-time analytics, CDC, ML signal pipelines, and spam detection. Different engineering teams own different parts of the pipeline, which gives a clear picture of how deeply Kafka is embedded across the organization.

Event streaming and data warehouse transport

The foundational use case is shipping user activity events (impressions, clicks, close-ups, repins) from application servers into the data warehouse. Singer, Pinterest’s open-source logging agent, runs as a daemonset on tens of thousands of hosts and writes over one trillion messages per day to Kafka with at-least-once delivery and sub-5ms upload latency. From Kafka, Secor persists events to S3 for offline analytics and ML training pipelines. This is the base layer that most other pipelines build on.

Real-time advertising budget computation

The ads platform uses a Kafka-backed streaming pipeline to compute and enforce advertiser spend limits in real time. This requires low latency and strict ordering guarantees, making Kafka the natural fit rather than the cost-optimized MemQ system Pinterest later built for ML data.

Fresh content indexing and recommendation

Kafka events are consumed by Flink and Spark Streaming to keep the content index up to date with newly published Pins. This drives the recommendation system’s awareness of new content as it is created.

Spam detection (Guardian)

The Trust and Safety team taps user activity events from Kafka and passes them through Guardian, a custom real-time analytics and rules engine built on a columnar query model. A separate Flink-based pipeline also processes these events for spam filtering.

Visual signals pipeline (Kappa architecture)

The Content Acquisition and Media Platform team rebuilt their visual signals infrastructure around Kafka and Flink in a Kappa architecture. The pipeline ingests data across 50 ingestion pipelines covering 300 billion ideas. Apache Flink does stateful stream processing over image and video signals, replacing the previous Lambda architecture that maintained both batch and streaming code paths.

Real-time experiment analytics

Pinterest runs hundreds of concurrent product experiments. Previously, detecting a statistically significant drop in an experiment metric required 10+ hours of batch processing. A Kafka-based Flink pipeline — described as Pinterest’s first Flink application in production when it launched in October 2019 — brings detection time down to minutes. The pipeline reads from two Kafka topics (filtered_events and filtered_experiment_activations) and supports 200-300 concurrent experiment groups.

Change data capture (CDC)

Two generations of CDC pipelines use Kafka as the transport layer. An earlier system uses Maxwell to capture incremental database changes from MySQL and publish them to Kafka for downstream ad systems. A more recent system, published in February 2026, captures changes from MySQL, TiDB, and KVStore using Debezium and TiCDC, writes them to Kafka with under-one-second latency, and then processes them through Flink into Apache Iceberg tables on S3. Base tables refresh every 15 minutes to 1 hour, compared to 24+ hours with the previous full-table batch approach.

ML training data transport (MemQ)

For ML training data, where latency requirements are relaxed, Pinterest built MemQ: a cloud-native PubSub system that uses S3 micro-batching and achieves up to 90% cost savings compared to an equivalent Kafka deployment. MemQ handles the high-volume, latency-tolerant workloads that Kafka handles less efficiently. Notably, MemQ uses Kafka internally as its own notification queue.

Scale and throughput

As of the December 2020 snapshot published by the Logging Platform team:

Metric	Value
Production clusters	50+
Brokers	3,000+
Topics	3,000+
Partitions	~500,000
Inbound peak throughput	~25 GB/s (40 million+ messages/second)
Outbound peak throughput	~50 GB/s
Daily messages (from Singer alone)	1 trillion+
Max brokers per cluster	200
Replication factor	3 (across three availability zones)
AWS regions	3 (us-east-1 primary, us-east-2, eu-west-1)
Tiered storage (since May 2024)	~200 TB/day offloaded from broker disk to S3 across 20+ onboarded topics

Between November 2018 and December 2020, inbound peak throughput grew from 15 million to 40 million messages per second, driven by the addition of visual signal pipelines, the real-time experiment analytics system, and ongoing user growth.

Pinterest’s Kafka architecture

Deployment model

Pinterest runs Kafka entirely on Amazon Web Services, self-managed across three regions: us-east-1 (the primary region, carrying the majority of brokers), us-east-2, and eu-west-1. The cluster fleet runs on EC2 instances. As of the 2018 snapshot, the default instance type was d2.2xlarge with magnetic storage; high-fanout read clusters used d2.8xlarge. By 2020, the team had migrated to SSD-backed instances after identifying magnetic disk I/O wait as a bottleneck during broker replacement events.

Brokers in each cluster are spread across three availability zones, with topics assigned to specific clusters (the “brokerset” concept described below) to limit the blast radius of a cluster failure.

Brokerset: virtual clusters within physical clusters

Rather than mapping topics to individual brokers, Pinterest uses a “brokerset” concept: a static partition assignment group that functions as a virtual Kafka cluster within a physical cluster. Topics are assigned to a brokerset, and partition leaders are placed using a stride-based algorithm that ensures balanced distribution across availability zones. This means that a single physical cluster can host multiple isolated workloads, and a failure in one brokerset does not affect topics assigned to others.

The maximum cluster size is capped at 200 brokers. Clusters beyond a certain size become difficult to operate, so Pinterest provisions new clusters rather than growing existing ones past this limit.

Data pipeline flow

The canonical pipeline runs: application servers → Singer (logging agent, running as a daemonset on each host) → Kafka → downstream consumers (Apache Flink, Spark Streaming, Kafka Streams) and S3 via Secor. Schema formats include Apache Thrift and Protocol Buffers.

The CDC pipeline takes a different path: MySQL, TiDB, and KVStore databases → Debezium or TiCDC connectors → Kafka (sub-one-second write latency) → Apache Flink → Apache Iceberg tables on S3 → Apache Spark (upsert Merge Into operations every 15 minutes to 1 hour).

Cross-region replication

Pinterest uses MirrorMaker v1 to replicate data between the three AWS regions. They extended MirrorMaker with a technique they call Shallow Mirror, which skips decompression and recompression by operating at the RecordBatch level. This reduces CPU load on the MirrorMaker fleet by over 80% at peak. Pinterest proposed Shallow Mirror to the Apache Kafka community as KIP-712.

Management platform: Orion

From around 2021, Pinterest replaced its earlier tooling (DoctorKafka and the Kafka Manager CMAK) with Orion, an open-source cluster management and automation platform (available at pinterest/orion on GitHub). Orion handles automated broker replacement, rolling restarts, configuration updates, cluster upgrades, topic creation via a managed wizard with an approval workflow, and end-to-end data lineage tracking. It provides configurable sensor and operator pipelines that abstract operational tasks into composable automations.

PubSub abstraction: PSC

Pinterest introduced PSC (PubSub Client) in 2022 as a unified Java client library that abstracts Kafka and MemQ behind a Topic URI scheme in the format: protocol:/rn:service:environment:cloud_region:cluster:topic. Applications do not reference broker hostnames directly; service discovery is handled by PSC at runtime. By November 2023, over 90% of Java applications and 100% of Flink jobs had migrated to PSC. The platform team uses PSC to push standardized client configurations and to perform automated error remediation across the fleet.

Producer architecture

Producers are required to compress at source. Pinterest observed that using a single-partition partitioner (rather than round-robin) before compression increases compression ratios approximately 2x by improving data locality within each batch. The team standardized inter-broker protocol and log message format versions across all clusters to eliminate unnecessary format conversions.

Consumer architecture

Consumer group management is handled through Orion’s topic governance workflow. The real-time experiment analytics Flink job, as one example, runs at parallelism 256 across 8 master nodes and 50 worker nodes (EC2 c5d.9xlarge instances), checkpointing 100 GB of state to HDFS every 5 seconds. PSC handles consumer offset management across both Kafka and MemQ topics through the same interface.

Stream processing

Apache Flink is Pinterest’s primary stream processing engine for Kafka, used across visual signal pipelines, experiment analytics, CDC ingestion, and spam detection. Apache Spark Streaming also consumes from Kafka for certain workloads. Kafka Streams is used for monetisation and metrics use cases.

Kafka Connect ecosystem

CDC ingestion uses Debezium (for MySQL) and TiCDC (for TiDB) as Kafka source connectors. An earlier CDC system used Maxwell for MySQL changelog capture. Secor functions as a custom Kafka-to-S3 sink for log persistence.

Special techniques and engineering innovations

AZ-aware partitioning

Pinterest built a custom Kafka partitioner that directs producer messages and consumer reads to partition leaders located in the same AWS availability zone as the client. The partitioner uses the EC2 Metadata API to determine the client’s AZ, then cross-references Kafka’s rack awareness metadata to identify local partition leaders. If AZ metadata is unavailable, it falls back to selecting from all available workers. The result is a 25% reduction in cross-AZ data transfer costs. A companion custom S3 transporter partitioner applies the same logic for Secor-based log writes.

Shallow Mirror

Standard MirrorMaker v1 deserializes every RecordBatch and recompresses it before forwarding, which drives high CPU utilization and causes out-of-memory errors at Pinterest’s peak traffic. Shallow Mirror bypasses this by iterating over RecordBatches as byte buffer pointers rather than deserializing them. The implementation required solving three problems: correctly updating the BaseOffset field in each batch without full deserialization, handling the “first batch problem” where MirrorMaker crops the first batch in a fetch response, and maintaining throughput for small-batch high-frequency producers. Deployed in production in 2020, Shallow Mirror reduced MirrorMaker CPU load by over 80% at peak and was proposed to the Apache Kafka community as KIP-712.

Graph-algorithm-based leader rebalancing

When partition leadership becomes imbalanced across brokers, naively swapping leaders can still leave some brokers overloaded. Pinterest’s Logging Platform team published a two-part solution. Part 1 uses breadth-first search to find single-step leader swap sequences that improve balance. Part 2 models the rebalancing problem as a maximum-flow network to compute optimal swap sequences for more complex imbalances. Critically, both approaches move only partition leaders, not partition data, so the rebalancing operation has a very low cost compared to partition reassignment.

Broker-decoupled tiered storage

Rather than adding tiered storage inside the broker process (the approach taken by KIP-405, the native Kafka tiered storage implementation), Pinterest implemented a sidecar architecture. A Segment Uploader process runs alongside each broker, watching the broker’s log directories via filesystem watchers. When a segment is finalized, the Segment Uploader uploads it to S3 and tracks progress via offset.wm files. ZooKeeper is used for partition leadership detection to avoid duplicate uploads. S3 object keys use MD5 hash-based prefix entropy to distribute request load and avoid S3 hotspots.

Consumers can be configured in four modes: Remote Only, Kafka Only, Remote Preferred, and Kafka Preferred. The team extended log.segment.delete.delay.ms from the default 60 seconds to 5 minutes for low-traffic topics to avoid missed segment uploads during retention cleanup. Since May 2024, the system has processed approximately 200 TB of data per day across 20+ onboarded topics.

Static membership (KIP-345)

In cloud environments, Kafka Streams consumers restart frequently for reasons unrelated to actual consumer failure: host restarts, rolling deployments, spot instance preemptions. Each restart triggers a full group rebalance and RocksDB state rebuild. Pinterest engineer Liquan Pei collaborated with Confluent’s Boyang Chen on the static membership protocol, presented at Kafka Summit SF 2019 and merged into Kafka as KIP-345. Static membership lets consumers re-join a group without triggering a full rebalance by using a persistent instance ID.

Upstream Kafka contributions

Pinterest’s Logging Platform team has contributed several KIPs and patches to the Apache Kafka project: KIP-91 (the delivery.timeout.ms producer configuration), KIP-245 (passing Properties objects to the KafkaStreams constructor), KIP-276 and KIP-300 (precursors to KIP-345), KAFKA-6896 (producer and consumer metrics in Kafka Streams), and KAFKA-7023 / KAFKA-7103 (RocksDB bulk loading optimizations for Kafka Streams state stores). Vahid Hashemian from the Logging Platform team is an Apache Kafka Committer and PMC Member.

Operating Kafka at scale

Automated cluster healing: DoctorKafka (2017-2020)

The first automation Pinterest built was DoctorKafka (open-sourced in August 2017 at pinterest/DoctorK), which monitored broker metrics via a dedicated Kafka topic, detected broker failures, and automatically reassigned workload. Broker failures were handled almost daily at Pinterest’s scale; DoctorKafka reduced failure-related alerts by over 95%. To prevent cascading replica loss when multiple brokers failed simultaneously (given a replication factor of 3), DoctorKafka rate-limited broker replacements to one per time period. The system also analyzed 24-48 hours of historical broker statistics and treated network bandwidth as the primary resource metric for rebalancing decisions.

Orion (2020 onward)

Orion replaced DoctorKafka and CMAK as Pinterest’s primary cluster management interface. It provides automated broker replacement, rolling upgrades, configuration updates, and cluster provisioning. Topic creation goes through a managed wizard with an approval workflow inside Orion, and lineage is tracked end-to-end from producer to consumer. Orion is open-sourced at pinterest/orion.

Client configuration standardization via PSC

Before PSC, applications maintained direct Kafka client dependencies with hardcoded broker hostnames and SSL passwords scattered across configurations. This created outage exposure during maintenance events. PSC introduced URI-based service discovery, standardized client configurations delivered by the platform team, and automated error remediation for common remediable exceptions. The result was over an 80% reduction in Flink application restarts caused by remediable client exceptions, and approximately 275 fewer FTE hours per year in keep-the-lights-on work.

Kafka version management

As of 2018, Pinterest maintained a monthly cadence for tracking open-source Kafka release branches. By 2020, the fleet ran on version 2.3.1 with cherry-picked fixes. A key operational improvement was standardizing the inter-broker protocol version and log message format version across all clusters, which eliminated the overhead of unnecessary in-flight message format conversions during rolling upgrades.

Broker configuration

Broker heap size is set to 8 GB, increased from 4 GB after TLS was enabled (TLS adds approximately 122 KB of memory per SSL KafkaChannel at scale). Pinterest evaluated EBS st1 volumes as a storage alternative to local disks but found d2 local storage superior for their access patterns; later, they migrated from magnetic to SSD-backed instances after identifying I/O wait as the bottleneck during broker replacement.

Upgrade validation

Before rolling out a Kafka version upgrade, Pinterest constructs a parallel test pipeline that mirrors the topology of the target production pipeline. Double publishing sends data through both old and new infrastructure simultaneously, and the team compares throughput, latency, and error metrics before promoting the new version to production.

Challenges and how they solved them

Daily broker failures at cloud scale (2017)

At 1,000+ brokers, hardware failures were routine. Partition reassignment alone was not fast enough to handle degraded brokers, and manual intervention did not scale. The solution was DoctorKafka: automated detection of broker degradation via metrics published to a Kafka topic, followed by automatic workload reassignment. Rate-limiting was critical: without it, simultaneous failures of multiple brokers could exceed the replication factor, causing data loss. The result was over a 95% reduction in failure-related operational alerts.

Cross-AZ data transfer costs (2019)

Default Kafka producer and consumer behavior assigns clients to any available partition leader regardless of availability zone. At Pinterest’s scale, this generated substantial cross-AZ network transfer costs. The team built a custom partitioner using the EC2 Metadata API and Kafka rack awareness to route producers and consumers to partition leaders in their own AZ. The custom S3 transporter partitioner applied the same logic for Secor writes. Result: 25% reduction in cross-AZ transfer costs.

MirrorMaker CPU spikes during cross-region replication (2020)

Standard MirrorMaker v1 deserializes and recompresses every RecordBatch during cross-region replication. At Pinterest’s throughput, this caused sustained CPU spikes and out-of-memory errors during peak traffic. The Shallow Mirror implementation bypassed serialization by treating RecordBatches as opaque byte buffers, requiring custom handling for BaseOffset modification and the batch-cropping issue introduced by MirrorMaker’s fetch response handling. The result was over an 80% reduction in MirrorMaker CPU load.

Imbalanced partition leadership causing broker overload (2020)

As the cluster fleet grew, partition leadership distribution became uneven, overloading individual brokers. Naive leader swap approaches could move leaders without materially improving the overall balance. Pinterest modeled the rebalancing problem using graph algorithms: BFS for single-step swaps, and maximum-flow networks for general multi-step rebalancing. Both approaches operate only on leader assignments, not on partition data, keeping the operation lightweight.

Magnetic disk I/O bottleneck during broker replacement (pre-2020)

On d2.2xlarge instances with magnetic storage, brokers undergoing replacement showed 10-30% CPU I/O wait during the recovery process. Migrating to SSD-backed instances reduced I/O wait to under 0.1% at the p100 level.

Client misconfiguration and tight coupling to broker endpoints (pre-2022)

Before PSC, applications embedded Kafka broker hostnames, SSL passwords, and client configurations directly. Maintenance events that changed broker endpoints required coordinated client restarts. PSC addressed this by introducing URI-based topic addressing with server-side service discovery, platform-managed client configurations, and automated remediation for known failure modes.

Kafka unsuitable for ML training data transport (2018-2020)

Kafka’s strict ordering guarantees, partition rigidity, rebalancing overhead, and replication costs made it an expensive fit for high-volume ML training data pipelines, where sub-second latency is not required. The solution was MemQ: a cloud-native PubSub system using S3 micro-batching that achieves up to 90% cost reduction versus an equivalent Kafka deployment. MemQ handles latency-tolerant ML workloads while Kafka continues to handle all use cases that require strict ordering and low latency. MemQ uses Kafka as its internal notification queue.

Legacy batch database ingestion taking 24+ hours (pre-2026)

Full-table batch reprocessing reprocessed over 95% of unchanged data on every cycle, creating a 24-hour data latency and unnecessary compute cost. The next-generation CDC pipeline (Debezium/TiCDC → Kafka → Flink → Apache Iceberg) processes only the roughly 5% of records that change in a given day. CDC events land in Kafka within under one second. Base tables on S3 refresh every 15 minutes to 1 hour. Bucket partitioning by primary key hash, combined with bucket joins in downstream Spark queries, delivered a 40%+ reduction in compute cost.

Full tech stack

Category	Technology	Role
Message broker	Apache Kafka	Central message bus and event streaming platform for all latency-sensitive data transport
Message broker	MemQ (pinterest/memq)	Cloud-native PubSub system for latency-tolerant ML training data transport; uses Kafka internally as its notification queue; up to 90% cheaper than equivalent Kafka
Log shipping	Singer (pinterest/singer)	Open-source logging agent running on tens of thousands of hosts; ships 1 trillion+ messages/day to Kafka with at-least-once delivery and sub-5ms latency
Stream processing	Apache Flink	Stateful stream processing for real-time experiment analytics, visual signal pipelines (Kappa architecture), and CDC database ingestion
Stream processing	Apache Spark Streaming	Stream processing consumer of Kafka topics; also handles periodic Merge Into operations on Iceberg tables in the CDC pipeline
Stream processing	Kafka Streams	In-process stream processing for monetisation and metrics use cases
State backend	RocksDB	State backend for Kafka Streams jobs; Pinterest contributed bulk loading optimizations (KAFKA-7023, KAFKA-7103)
Cross-region replication	MirrorMaker v1 + Shallow Mirror	Cross-region Kafka replication; Pinterest’s Shallow Mirror extension bypasses decompression for 80%+ CPU reduction
Cluster management	Orion (pinterest/orion)	Open-source management and automation platform; broker replacement, rolling upgrades, topic governance, lineage tracking
Client abstraction	PSC (PubSub Client)	Unified Java client library abstracting Kafka and MemQ behind Topic URI-based service discovery; 100% Flink adoption, 90%+ Java app adoption
CDC connectors	Debezium	MySQL CDC source connector writing change events to Kafka with sub-one-second latency
CDC connectors	TiCDC	TiDB CDC source connector writing change events to Kafka
CDC connectors	Maxwell	Earlier MySQL CDC connector used for ad system database changelog capture
Table format	Apache Iceberg	Table format for CDC and base tables on S3 in the next-generation ingestion pipeline; uses Merge-on-Read strategy
Log persistence	Secor	Pinterest’s Kafka-to-S3 log persistence service
Object storage	Amazon S3	Remote storage for tiered storage segments, Iceberg tables, Flink checkpoints (CSV snapshots), and MemQ message storage
Checkpoint storage	HDFS	Flink checkpoint storage for experiment analytics jobs
Coordination	ZooKeeper	Kafka coordination; also used by the tiered storage sidecar for partition leadership detection
Compute	Amazon EC2	Kafka broker hosting (d2.2xlarge default, d2.8xlarge for high-fanout, later SSD instances); Flink worker nodes (c5d.9xlarge for experiment analytics)
Serialization	Apache Thrift / Protocol Buffers	Event serialization formats for Singer and Kafka producers
Client libraries	librdkafka, kafka-python, confluent-kafka-python	Kafka clients for non-Java (C++) and Python services
Source databases	MySQL, TiDB, KVStore	CDC source databases feeding the next-generation ingestion pipeline via Debezium and TiCDC

Key contributors

Name	Title / team	Contribution
Ambud Sharma	Tech Lead and Engineering Manager, Logging Platform	Led MemQ design and open-source release; co-authored AZ-aware partitioning and Singer open-source posts; overall Logging Platform leadership
Vahid Hashemian	Staff Software Engineer, Logging Platform (Apache Kafka Committer and PMC Member)	Co-authored “How Pinterest runs Kafka at scale” (2018) and Confluent blog post (2021); co-authored PSC articles (2022, 2023); upstream Kafka contributions including KIP-91 and KIP-245
Henry Cai	Software Engineer, Data Engineering	Co-authored “How Pinterest runs Kafka at scale” (2018); lead author of the Shallow Mirror article (2021); co-authored AZ-aware partitioning and Singer open-source posts
Jeff Xiang	Senior Software Engineer, Logging Platform	Lead author of the tiered storage article (2024); co-authored PSC articles (2022, 2023)
Ping-Min Lin	Software Engineer, Logging Platform	Authored graph algorithms for Kafka operations (Parts 1 and 2, 2020); MemQ performance optimization; co-authored AZ-aware partitioning post
Liquan Pei	Software Engineer	Co-authored “How Pinterest runs Kafka at scale” (2018); co-presented KIP-345 static membership at Kafka Summit SF 2019
Yu Yang	Data Engineering	Authored DoctorKafka open-source announcement (2017)
Ankit Patel	Software Engineer, Content Acquisition and Media Platform	Authored visual signals Lambda-to-Kappa architecture article (2020)
Parag Kesar, Ben Liu	Software Engineers, Data Engineering	Co-authored real-time experiment analytics with Flink article (2019)
Liang Mou, Yisheng Zhou, Elizabeth Nguyen, Owen Zhang	Logging Platform	Co-authored next-generation DB ingestion article (2026)

Key takeaways for your own Kafka implementation

Design for cluster isolation from the start. Pinterest’s brokerset concept — virtual clusters with stride-based partition assignment within a physical cluster — gives you topic-level failure isolation without the operational overhead of provisioning a separate physical cluster for every workload. The key is capping cluster size (Pinterest uses 200 brokers) and using the brokerset as the unit of risk management.

AZ-aware routing pays for itself at scale. Default Kafka clients are AZ-agnostic, and at a large enough fleet the cross-AZ data transfer costs are non-trivial. Pinterest built a custom partitioner using EC2 Metadata and Kafka rack awareness that routes producers and consumers to local partition leaders, achieving a 25% reduction in transfer costs. This is a relatively low-effort change for the savings it produces in cloud environments.

Replication overhead is often decompression, not network. Pinterest found that MirrorMaker’s CPU cost was dominated by decompression and recompression, not by network I/O. The Shallow Mirror approach — treating RecordBatches as opaque byte buffers during replication — reduced MirrorMaker CPU load by over 80%. Before investing in replication hardware, it is worth profiling whether the bottleneck is actually the wire or the processing.

Broker-side tiered storage is not the only option. Pinterest deliberately chose a sidecar architecture for tiered storage rather than waiting for KIP-405’s in-broker implementation. The sidecar Segment Uploader is operationally independent of the broker process, which simplifies upgrades and means a broker restart does not interrupt the upload pipeline. If tiered storage is a priority, evaluate whether broker-coupled or broker-decoupled implementations better fit your upgrade cadence and failure domains.

A unified client abstraction reduces operational debt significantly. PSC gave Pinterest’s platform team a single control point for client configurations, service discovery, and automated error remediation across both Kafka and MemQ. The result was over 80% fewer Flink restarts from client errors and an estimated 275 fewer FTE hours per year in KTLO work. A thin abstraction layer that separates application code from broker topology is worth the investment when you operate at more than a handful of clusters.

Sources and further reading

How Pinterest runs Kafka at scale - Henry Cai, Shawn Nguyen, Yi Yin, Liquan Pei et al., Pinterest Engineering Blog, 2018
Real-time analytics at Pinterest - Krishna Gade, Pinterest Engineering Blog, 2015
Running Kafka at scale at Pinterest - Eric Lopez, Heng Zhang, Henry Cai, Jeff Xiang et al., Confluent Engineering Blog, 2021
Pinterest tiered storage for Apache Kafka: a broker-decoupled approach - Jeff Xiang, Vahid Hashemian, Pinterest Engineering Blog, 2024
Open-sourcing DoctorKafka: Kafka cluster healing and workload balancing - Yu Yang, Pinterest Engineering Blog, 2017
Using graph algorithms to optimize Kafka operations, Part 1 - Ping-Min Lin, Pinterest Engineering Blog, 2020
Shallow Mirror - Henry Cai, Pinterest Engineering Blog, 2021
Fighting spam with Guardian: a real-time analytics and rules engine - Hongkai Pan, Pinterest Engineering Blog, 2021
Pinterest visual signals infrastructure: evolution from Lambda to Kappa architecture - Ankit Patel, Pinterest Engineering Blog, 2020
Real-time experiment analytics at Pinterest using Apache Flink - Parag Kesar, Ben Liu, Pinterest Engineering Blog, 2019
MemQ: an efficient, scalable cloud-native PubSub system - Ambud Sharma, Pinterest Engineering Blog, 2021
Unified PubSub Client at Pinterest - Vahid Hashemian, Jeff Xiang, Pinterest Engineering Blog, 2022
Open-sourcing Singer: Pinterest’s performant and reliable logging agent - Ambud Sharma, Indy Prentice, Henry Cai, Shawn Nguyen et al., Pinterest Engineering Blog, 2019
pinterest/orion on GitHub - Pinterest Engineering
Next-generation DB ingestion at Pinterest - Liang Mou, Yisheng Zhou, Elizabeth Nguyen, Owen Zhang, Pinterest Engineering Blog, 2026
Running Unified PubSub Client in production at Pinterest - Jeff Xiang, Vahid Hashemian, Jesus Zuniga, Pinterest Engineering Blog, 2023
Optimizing Kafka for the cloud - Eric Lopez, Henry Cai, Heng Zhang, Ping-Min Lin et al., Pinterest Engineering Blog, 2019
Static membership rebalance strategy designed for the cloud (Kafka Summit SF 2019) - Liquan Pei (Pinterest), Boyang Chen (Confluent), Kafka Summit SF 2019

If you are managing a Kafka deployment at scale, Kpow gives you a single interface for monitoring consumer lag, inspecting topics, and managing cluster operations across multiple clusters. A free 30-day trial is available and connects to any Kafka cluster in minutes via Docker, Helm, or JAR.

How Salesforce uses Apache Kafka in production

Factor House — Tue, 02 Jun 2026 00:00:00 GMT

Salesforce has published more detail about its Kafka operations than most companies its size, which makes its engineering blog an unusually complete record of what it takes to run Apache Kafka at genuine enterprise scale. By 2021, the company’s fleet had grown to 100+ clusters, 2,500+ brokers, 50,000+ topics, 300,000+ partitions, 40+ PB of storage, and throughput exceeding 3 trillion events per day — a figure that had been doubling annually.

The engineering problem Kafka solves at Salesforce is not a single use case but a platform-level one: how do you provide a unified, reliable event transport across a multi-tenant SaaS estate that spans hundreds of global data centres, hundreds of thousands of customer orgs, and applications ranging from operational metrics to real-time AI?

Company overview

Salesforce is a cloud-based CRM and enterprise software company founded in 1999 and headquartered in San Francisco. It serves more than 150,000 businesses globally across its Sales Cloud, Service Cloud, Marketing Cloud, and Agentforce AI product lines.

Salesforce adopted Kafka in production no later than May 2014, when it hosted an Apache Kafka edition of the SF Logging Meetup at its headquarters. By June 2014, Rajasekar Elango, a lead developer on the Monitoring and Management Team, was presenting the company’s early Kafka setup at a public Kafka Meetup — a 3-node ZooKeeper, 5-broker cluster with SSL/TLS mutual authentication and Avro serialisation.

The trigger for adoption was operational observability: the DVA (Diagnostics, Visibility, and Analytics) team needed a scalable transport layer for metrics and logs across a growing global data centre footprint.

Date	Event
May 2014	Apache Kafka in production use confirmed; SF Logging Meetup hosted at Salesforce HQ
June 2014	Rajasekar Elango presents SSL/TLS Kafka setup at public Kafka Meetup
Late 2015	Next-generation log pipeline (project Ajna / DeepSea) development begins
2016	Ajna fleet reaches 30+ clusters and hundreds of billions of events per day
December 2016	Log pipeline completeness measured at only ~25%; reliability programme begins
September 2017	Log pipeline reaches 7-nines completeness (99.99999%), sustained consistently
April 2018	Mirus (custom cross-cluster replication tool) fully replaces MirrorMaker in all production data centres
2021	Kafka Summit Americas: Lei Ye and Paul Davidson disclose fleet at 100+ clusters, 3+ trillion events/day
August 2024	Hyperforce perimeter telemetry pipeline published: Kafka + Spark Streaming + Druid, 60 TB/day
April 2025	Marketing Cloud 760-node zero-downtime migration into Ajna completed
July 2025	Agentforce AI agent audit pipeline published: 20 million model interactions/month over Kafka
May 2026	Conversational AI scaling post published: Kafka introduced for 30K+ concurrent conversations, targeting 100K

Salesforce’s Kafka use cases

Salesforce’s use cases cover the full breadth of what Kafka is typically used for in a large enterprise, from infrastructure telemetry to customer-facing products to AI pipelines.

Operational metrics and system health (project Ajna)

The DVA team built an internal Kafka-as-a-Service platform code-named Ajna to transport CPU metrics, machine reachability data, system logs, network flow data, application logs, JMX metrics, and custom database metrics across all global data centres to near-real-time dashboards. Ajna is the backbone from which most other Kafka use cases at Salesforce have grown.

Source: Nishant Gupta, Salesforce Engineering Blog

Log shipping pipeline

The Infrastructure group built a next-generation log pipeline on top of Ajna, routing logs from local per-data-centre Kafka clusters through cross-WAN replication to an aggregate cluster in the Secure Zone, and from there into DeepSea (an internal Hadoop/HDFS store) via a MapReduce consumer job called Kafka Camus. The target was 5-nines completeness; it eventually reached 7 nines.

Source: Sanjeev Sahu, Salesforce Engineering Blog

Platform Events and Change Data Capture

Salesforce’s Platform Events and Change Data Capture products expose a time-ordered, immutable event stream to hundreds of thousands of multi-tenant customer orgs. Kafka’s log-based storage model directly inspired the design: events are offset-based, durable, and replayable. Serialisation uses Apache Avro, with event definitions driven by Salesforce’s metadata layer.

Source: Alexey Syomichev, Salesforce Engineering Blog

Real-time ML insights for Einstein Activity Capture

The Activity Platform ingests 100+ million customer interactions per day through Kafka. A chain of Kafka Streams jobs processes the stream: a Gatekeeper spam filter passes clean events to a series of extractors (TensorFlow-based, regex/keyword, Spark ML, and Einstein Conversation Insights), which feed a transformer and then a persistor that writes sales activity insights to Cassandra.

Source: Rohit Deshpande, Salesforce Engineering Blog

Service Cloud chatbot audit and debug

Apache Kafka on Heroku backs the chatbot event pipeline for Service Cloud, targeting tens of millions of debug events daily with up to 7-day retention. The pipeline applies five fault-tolerance patterns on top of Kafka, described further in the special techniques section below.

Source: Mark Holton, Salesforce Engineering Blog

Hyperforce perimeter telemetry

Salesforce’s Hyperforce infrastructure generates 60 TB of raw perimeter telemetry daily from commercial CDNs. Spark Streaming normalises this data into protobuf format; Kafka then carries it to an Imply/Druid hypercube (18 dimensions, 13 measurements) serving real-time debugging dashboards for live-site incidents, latency analysis, and DDoS detection with sub-second query latency and 2-minute data freshness.

Source: Srinivas Ranganathan, Salesforce Engineering Blog

Conversational AI context storage for Agentforce

The Conversation Storage Service (CSS) uses Kafka for buffering, batching, and ordering conversational context that powers real-time AI systems — sentiment analysis, agent assist, and supervisor insights — across 50,000+ concurrent conversations at a peak ingestion rate of 30,000+ events per minute. The scaling target is 100,000 concurrent conversations.

Source: Ashima Kochar and Deepak Mali, Salesforce Engineering Blog

Agentforce AI agent audit trail

Kafka pub-sub ingestion handles the variable, spike-prone traffic from 20 million model interactions per month across 500+ enterprise customers, feeding a 30-day audit data retention pipeline backed by Salesforce Data Cloud and S3.

Source: Madhavi Kavathekar, Salesforce Engineering Blog

Cross-cluster replication

Salesforce replaced Apache MirrorMaker with its own open-source tool, Mirus, for cross-data-centre replication. Mirus is built on the Kafka Connect source connector API and has been in production since April 2018.

Source: Paul Davidson, Salesforce Engineering Blog

Scale and throughput

The fleet figures below are from the Kafka Summit Americas 2021 talk by Lei Ye and Paul Davidson, supplemented with per-system figures from individual engineering blog posts.

Metric	Value	Source
Total clusters	100+	Lei Ye and Paul Davidson, Kafka Summit Americas 2021
Total brokers	2,500+ (largest clusters: 250+ brokers each)	Lei Ye and Paul Davidson, Kafka Summit Americas 2021
Topics	50,000+	Lei Ye and Paul Davidson, Kafka Summit Americas 2021
Partitions	300,000+	Lei Ye and Paul Davidson, Kafka Summit Americas 2021
Storage	40+ PB	Lei Ye and Paul Davidson, Kafka Summit Americas 2021
Throughput	3+ trillion events/day (doubling annually)	Lei Ye and Paul Davidson, Kafka Summit Americas 2021
Fleet availability	Over 99.9%	Lei Ye and Paul Davidson, Kafka Summit Americas 2021
Marketing Cloud cluster (pre-migration 2025)	760+ nodes, 12 clusters, 1 million messages/second, 15 TB/day	Dheeraj Bansal and Ankit Jain, April 2025
Einstein Activity Capture	100+ million customer interactions/day	Rohit Deshpande
Hyperforce telemetry	60 TB raw perimeter data/day	Srinivas Ranganathan, August 2024
Agentforce AI	20 million model interactions/month, 2 billion predictions/month	Madhavi Kavathekar, July 2025
Conversational AI (CSS)	30,000+ events/minute peak; scaling target of 100K concurrent conversations	Ashima Kochar and Deepak Mali, May 2026
Chatbot pipeline	Tens of millions of events/day, 7-day retention	Mark Holton
Ajna fleet (2016 snapshot)	30+ clusters, ~300 Mbps aggregate throughput, hundreds of billions of events/day	Nishant Gupta, July 2016

Salesforce’s Kafka architecture

Ajna: Kafka-as-a-Service

Ajna is Salesforce’s internal Kafka platform. Each production data centre runs an “Ajna Local” cluster for ingest. Data replicates over WAN to an “Ajna Aggregate” cluster in a Secure Zone (DMZ), from which consumers — Argus (time series), DeepSea (Hadoop/HDFS), and stream-processing applications — pull downstream. Before April 2018, replication used Apache MirrorMaker; since then it has used Mirus.

The platform runs on a hybrid deployment model: on-premises clusters orchestrated with Choria (self-healing automation) and public cloud clusters deployed on Kubernetes. Continuous automated patching is applied across the fleet.

Cross-site connectivity

Rather than using Kafka’s advertised listeners over a VPN, Salesforce implemented a “Native Kafka Endpoint” using a standard load balancer fronting Envoy running in Layer 4 SNI-routing mode. This gives clients a stable single endpoint for cross-data-centre Kafka connections without VPN tunnels.

Source: Lei Ye and Paul Davidson, Kafka Summit Americas 2021

Topic design and schema management

Platform Events and Change Data Capture events use Apache Avro serialisation, with event definitions driven by Salesforce’s metadata layer. For the Conversation Storage Service, messages are keyed by conversation ID, ensuring all events for a given conversation land in the same partition and preserve ordering for downstream AI systems.

Producer architecture

The Agentforce audit pipeline uses Kafka’s pub-sub model to absorb bursty, business-hour traffic spikes before handing off to Data Cloud. Dynamic flow control mechanisms handle real-time traffic adjustment in that pipeline.

Consumer architecture

For the Conversation Storage Service, Salesforce applies curated consumer lag limits to maintain ordering guarantees for real-time AI workflows. Where consumer lag creates read-after-write consistency problems (at 50K concurrent conversations), an in-memory cache (VegaCache) bridges the gap rather than forcing changes to the Kafka consumer configuration.

Stream processing

The Einstein Activity Capture pipeline relies entirely on Kafka Streams for its ML processing chain. The application team chose Kafka Streams specifically because upgrades can be managed without coordinating with the infrastructure team — unlike the prior Storm-based pipeline.

Source: Rohit Deshpande, Salesforce Engineering Blog

Special techniques and engineering innovations

Mirus: custom cross-cluster replication

Mirus is Salesforce’s open-source replacement for Apache MirrorMaker, built on the Kafka Connect source connector API. Each MirusSourceTask runs an independent KafkaConsumer/KafkaProducer pair for a subset of partitions, enabling higher throughput over internet links than MirrorMaker’s single-producer model. A KafkaMonitor thread tracks partition changes dynamically, allowing configuration changes via the Connect REST API without a process restart. The tool includes custom JMX metrics for replication lag.

Source: Paul Davidson, engineering.salesforce.com and github.com/salesforce/mirus

Envoy as Layer 4 SNI-routing proxy

For cross-site Kafka connectivity, Salesforce runs Envoy in SNI-routing mode behind a standard load balancer, giving clients a stable single endpoint for cross-data-centre connections. This avoids the need to publish per-broker advertised listeners across WAN links.

Conversation-level Kafka partitioning

For the Conversation Storage Service, messages are keyed by conversation ID, ensuring all events for a given conversation land in the same partition and preserve ordering for downstream AI systems that need to reason over conversational context in sequence.

Fault-tolerant chatbot pipeline patterns

The Service Cloud chatbot pipeline layers five fault-tolerance patterns on top of Kafka: a per-endpoint circuit breaker (OPEN/HALF_OPEN/CLOSED states), a bulkhead (concurrent request limits per endpoint), timeout-plus-retry (maximum 6 attempts over 16 hours), UUID-per-event idempotency with upsert semantics, and Kafka partition replication across availability zones.

Source: Mark Holton, Salesforce Engineering Blog

SSL contribution to open-source Kafka

Salesforce’s DVA team contributed SSL support to the Kafka 0.8.2 series before it was part of the mainline project, and later advocated for per-topic throttling at the project level.

Source: Nishant Gupta, Salesforce Engineering Blog

4-byte embedded hash ID for deduplication

Because the log pipeline uses “at least once” delivery semantics, a 4-byte hashed unique ID is embedded in each log message so downstream consumers can detect and drop duplicates on replay.

Source: Sanjeev Sahu, Salesforce Engineering Blog

Operating Kafka at scale

Deployment model: Hybrid. On-premises clusters use Choria for automated patching and self-healing. Public cloud clusters run on Kubernetes. Continuous automated patching applies across the entire fleet.

Monitoring and observability stack:

Tool	Role
Cruise Control (LinkedIn open-source)	Automated cluster workload rebalancing
Xinfra Monitor (LinkedIn open-source)	Synthetic end-to-end availability and latency checks via synthetic workloads
Argus (Salesforce open-source)	OpenTSDB-based time series monitoring and alerting for Kafka and downstream systems
Splunk	Log analysis for Kafka operations
Wrangler (Salesforce internal)	Cross-cluster UI for topic management and fleet-wide visibility
Fidelity Assessment Service (internal)	Tracks log pipeline completeness end-to-end; results surfaced in Einstein Analytics dashboards

Source: Lei Ye and Paul Davidson, Kafka Summit Americas 2021

Automated recovery: Salesforce built a Kafka Auto-Fix Operations Tool that detects offline or under-replicated partitions, frozen brokers, and disk failures. It identifies the most advanced replica, then runs leader election and partition reassignment automatically. The target mean time to recovery is approximately 2 minutes with a zero data loss guarantee.

Fault injection testing: The infrastructure team uses a bespoke testing framework comprising a Test Coordinator (control plane web service), Load Generator (simulates producer/consumer traffic), Ops Tools (cluster configuration), a Fault Injection Framework built on Apache Trogdor and Kibosh, and a Result Analysis component (log and metric tracking). Test scenarios cover functional, performance, load, scale, upgrade/downgrade, and patching runs on deterministic infrastructure.

Migration monitoring: During the 2025 Marketing Cloud migration, the team ran 15 custom monitoring dashboards (cluster-level to node-level) with 24/7 automated alerting throughout.

Source: Dheeraj Bansal and Ankit Jain, Salesforce Engineering Blog

Challenges and how they solved them

MirrorMaker instability on WAN replication

Problem: Apache MirrorMaker 0.8.x was unstable and required a full process restart for any configuration change. Its single-producer model created throughput bottlenecks on internet-based WAN links between data centres.

Root cause: MirrorMaker’s architecture — one process per cluster pair, one producer per process — could not scale to Salesforce’s cross-data-centre replication volumes.

Solution: Salesforce built and open-sourced Mirus, a Kafka Connect-based tool with multiple independent consumer-producer pairs per worker process and dynamic REST API configuration.

Outcome: Mirus fully replaced MirrorMaker across all production data centres from April 2018.

Source: Paul Davidson, Salesforce Engineering Blog

Log pipeline completeness: from 25% to seven nines

Problem: Between December 2016 and September 2017, the log shipping pipeline could not reliably deliver logs to DeepSea. Completeness was as low as 25% at the start of the programme.

Root causes: Six distinct issues were identified: multi-threaded consumer logic couldn’t handle dynamic volume changes; MapReduce fault tolerance was set too low; Logstash had file-skipping and zombie-state bugs; MirrorMaker batch size (~1 MB) caused batch rejections; and “at least once” semantics produced duplicates.

Solutions: Multi-threaded consumers with dynamic volume adjustment; MapReduce fault tolerance raised to 50%; Logstash reconfigured to read from the beginning; a novel buffer replay for zombie states; MirrorMaker batch size cut from ~1 MB to 250 KB; a 4-byte embedded hash ID per log record for deduplication.

Outcome: Completeness reached 99.99999% (7 nines) by September 2017 and has been sustained consistently since.

Source: Sanjeev Sahu, Salesforce Engineering Blog

Marketing Cloud migration: 760 nodes, zero downtime

Problem: Marketing Cloud ran its own Kafka fleet — 760+ nodes, 12 clusters, 1 million messages/second — on CentOS 7 with a different Kafka version and authentication mechanism from the central Ajna platform (RHEL 9). Unifying it into Ajna required a zero-downtime migration under live production traffic.

Root cause: Accumulated divergence between Marketing Cloud’s self-managed Kafka and the centralised Ajna system over time.

Solution: An automated orchestration pipeline with pre-, in-flight, and post-validation checks; disk-level checksum validation before and after each step; rack-aware replica placement across separate physical racks; phased node rollout with health checks; 15 custom monitoring dashboards; and synthetic data validation post-migration with baseline signature matching.

Outcome: Zero-downtime migration completed; Marketing Cloud Kafka unified into Ajna.

Source: Dheeraj Bansal and Ankit Jain, Salesforce Engineering Blog

Read-after-write consistency at 50K concurrent conversations

Problem: At 50,000 concurrent conversations, Kafka consumers lagged behind producers, causing AI workflows to read stale conversational context immediately after a write.

Root cause: Kafka’s durability-vs-latency trade-off: freshly written messages were not yet visible to consumers by the time the AI system queried for the latest state.

Solution: VegaCache, an in-memory cache, was introduced to serve recent writes directly, bypassing the consumer lag for latency-sensitive reads while keeping Kafka as the durable ordered source of truth.

Outcome: Read-after-write consistency maintained at scale.

Source: Ashima Kochar and Deepak Mali, Salesforce Engineering Blog

Agentforce audit: bursty traffic and Data Cloud ingestion mismatch

Problem: AI agent traffic is highly bursty (business-hour spikes); Data Cloud’s ingestion API expected customer-owned S3 buckets, but Agentforce needed Salesforce-controlled S3 buckets — a file-size and ownership mismatch that blocked the integration.

Root cause: Architectural mismatch between Data Cloud’s design assumptions and Agentforce’s multi-tenant security model.

Solution: Kafka’s pub-sub model absorbs traffic spikes before handoff to Data Cloud; dynamic flow control mechanisms handle real-time traffic adjustment; iterative proof-of-concept work resolved the S3 ownership problem.

Outcome: The system supports 500+ enterprise customers with 20 million model interactions per month under a 30-day contractual audit retention window.

Source: Madhavi Kavathekar, Salesforce Engineering Blog

Full tech stack

Category	Tools	Notes
Message broker	Apache Kafka	Kafka 2.6 across main fleet (2021); early versions 0.8.2 and 0.9
Cross-cluster replication	Mirus (Salesforce open-source), Apache MirrorMaker (legacy)	Mirus replaced MirrorMaker in all data centres from April 2018; built on Kafka Connect source connector API
Schema registry / serialisation	Apache Avro	Used for Platform Events, Change Data Capture, and early monitoring pipeline
Stream processing	Kafka Streams, Apache Spark Streaming	Kafka Streams for Einstein Activity Capture ML chain; Spark Streaming for Hyperforce telemetry normalisation
Connectors	Kafka Connect (Mirus source connector), Kafka Camus	Camus is a MapReduce job consuming from Ajna Aggregate to DeepSea (HDFS)
Cluster coordination	Apache ZooKeeper	60+ ZooKeeper nodes in Marketing Cloud fleet alone before 2025 migration
Proxy / networking	Envoy	Layer 4 SNI-routing proxy for cross-site Native Kafka Endpoint
Monitoring	Cruise Control, Xinfra Monitor, Argus, Splunk, Wrangler, Datadog Vector	Cruise Control for workload rebalancing; Argus is OpenTSDB-based; Wrangler is Salesforce’s internal cross-cluster UI; Vector collects CDN telemetry
Deployment / infra	Choria (on-premises), Kubernetes (public cloud)	Choria provides self-healing and automated patching for on-premises clusters
ML inference	TensorFlow, Spark ML	Both used in Einstein Activity Capture extractor chain
Container orchestration (app layer)	Nomad (migrated from DC/OS)	Used by Einstein Activity Capture pipeline
Storage sinks	Apache Hadoop/HDFS (DeepSea), Cassandra, S3 (AWS)	DeepSea for log retention; Cassandra for Einstein insights; S3 for Agentforce audit and Mirus replication target
Real-time analytics	Imply / Apache Druid	18-dimension, 13-measurement hypercube for Hyperforce telemetry; sub-second query latency
Caching	VegaCache (Salesforce internal)	In-memory cache for read-after-write consistency in Conversation Storage Service
Log processing	Logstash	Runs on each host, publishing to local Ajna Kafka cluster
Fault injection	Trogdor, Kibosh	Foundation for Salesforce’s bespoke fault injection testing framework
Database	PostgreSQL	Configuration storage in Einstein pipeline; initial database for Conversation Storage Service before Kafka introduction

Key contributors

Name	Role	Contribution
Nishant Gupta	Sr. Director of Engineering, DVA team	Authored “Expanding Visibility with Apache Kafka”; presented Ajna platform publicly in July 2016
Rajasekar Elango	Lead Developer, Monitoring and Management Team	Presented Secure Kafka at Salesforce at Kafka Meetup, June 2014
Alexey Syomichev	Engineer, Salesforce	Authored “How Apache Kafka Inspired Our Platform Events Architecture”
Sanjeev Sahu	Engineer, Infrastructure	Authored “Our Journey to a Near Perfect Log Pipeline”
Rohit Deshpande	Engineer, Activity Platform / Einstein	Authored “Real-time Einstein Insights Using Kafka Streams”; contributed three Kafka Improvement Proposals (KIPs) to the Apache Kafka project
Paul Davidson	Software Architect, Kafka Platform	Original developer of Mirus; co-presented at Kafka Summit Americas 2021; authored “Open Sourcing Mirus”
Lei Ye	Principal Software Engineer, Infrastructure	Co-presented “Tales from the Four-Comma Club” at Kafka Summit Americas 2021
Mark Holton	Engineer, Service Cloud	Authored both chatbot pipeline blog posts
Dheeraj Bansal	Principal Member of Technical Staff	Led the Marketing Cloud 760-node Kafka migration (April 2025)
Ankit Jain	Senior Director of Software Engineering, Platform Architecture and Distributed Systems	Co-authored the Marketing Cloud migration post
Srinivas Ranganathan	Director of Software Engineering	Authored the Hyperforce perimeter telemetry pipeline post (August 2024)
Madhavi Kavathekar	Director of Software Engineering	Authored the Agentforce AI agent auditing systems post (July 2025)
Ashima Kochar	Lead Software Engineer, Service Cloud	Lead author on the conversational AI scaling post (May 2026)
Deepak Mali	Engineer, Service Cloud	Co-author on the conversational AI scaling post (May 2026)

Key takeaways for your own Kafka implementation

Build for replication from the start. Salesforce outgrew Apache MirrorMaker’s single-producer model once WAN replication volumes increased. If cross-datacenter or cross-region replication is in your roadmap, evaluate whether your replication tool can handle dynamic configuration changes without a process restart and whether its throughput model scales with your partition count.
Log pipeline completeness requires end-to-end measurement. Salesforce’s log pipeline sat at 25% completeness for months before a dedicated completeness-tracking service (the Fidelity Assessment Service) made the problem visible end-to-end. Instrumenting each stage independently is insufficient; you need a view that correlates events from ingest to sink.
“At least once” delivery requires an idempotency strategy. At Salesforce’s scale, duplicates from at-least-once delivery are not hypothetical edge cases. Embedding a hashed unique ID per message and using upsert semantics at the consumer is a concrete, low-overhead approach that avoids the complexity of exactly-once transaction semantics in many pipeline architectures.
Consumer lag is a first-class consistency concern for stateful workloads. The Conversation Storage Service’s read-after-write consistency problem is common when Kafka is used as a state transport for AI or event-sourcing workloads. An in-memory cache for recent writes, rather than tightening consumer lag SLOs on the Kafka cluster itself, can resolve the issue with less operational risk.
Invest in automated recovery tooling before you need it. Salesforce built its Kafka Auto-Fix Operations Tool to achieve a ~2-minute MTTR for partition and broker failures. At 2,500+ brokers, manual recovery runbooks cannot achieve this. Automating leader election and partition reassignment for predictable failure modes is worth the investment before the fleet reaches a size where manual intervention becomes the bottleneck.

Sources and further reading

#	Source	Author	Date
1	Expanding Visibility with Apache Kafka	Nishant Gupta	2016
2	How Apache Kafka Inspired Our Platform Events Architecture	Alexey Syomichev	2018
3	Real-time Einstein Insights Using Kafka Streams	Rohit Deshpande	2019
4	Our Journey to a Near Perfect Log Pipeline	Sanjeev Sahu	December 2017
5	Open Sourcing Mirus	Paul Davidson	October 2017
6	Building a Scalable Event Pipeline with Heroku and Salesforce	Mark Holton	2020
7	Building a Fault-Tolerant Data Pipeline for Chatbots	Mark Holton	2020
8	Inside Marketing Cloud’s Kafka Cluster Upgrade: Migrating 760 Nodes with 1,000,000 Messages/Second	Dheeraj Bansal, Ankit Jain	April 2025
9	Architecting AI Agent Auditing Systems in Agentforce	Madhavi Kavathekar	July 2025
10	Scaling AI-Driven Conversations from 10K to 100K While Maintaining Real-Time Consistency	Ashima Kochar, Deepak Mali	May 2026
11	Unlocking Real-Time Insights: Engineering the Hyperforce Experience	Srinivas Ranganathan	August 2024
12	Tales from the Four-Comma Club — Managing Kafka as a Service at Salesforce	Lei Ye, Paul Davidson	Kafka Summit Americas 2021
13	Enabling Real-Time Scenarios at Scale Using Kafka	Nishant Gupta	July 2016
14	Secure Kafka at Salesforce.com	Rajasekar Elango	June 2014
15	Mirus — GitHub	Paul Davidson (primary maintainer)	Last release June 2021

If you are running Kafka at scale and want visibility into consumer lag, partition health, and broker performance across your clusters, Kpow is a Kafka management and monitoring tool built for platform and data engineering teams. You can try it free for 30 days and connect it to any Kafka cluster in minutes via Docker, Helm, or JAR.

How Shopify uses Apache Kafka in production

Factor House — Tue, 02 Jun 2026 00:00:00 GMT

Shopify processes 1.75 trillion Apache Kafka messages every month across a fleet of self-managed clusters running on Google Kubernetes Engine. At the centre of that infrastructure is a change data capture pipeline that streams mutations from more than 100 MySQL database shards at a sustained 65,000 records per second, peaking at 100,000 records per second on Black Friday. Kafka is how Shopify moves data between its sharded monolith, its real-time processing layer, and its data warehouse — and has been since 2014, when the company also wrote and open-sourced its own Go Kafka client library because no suitable one existed.

Company overview

Shopify is a commerce platform that lets merchants of any size build and run online and physical retail operations. As of 2020, the platform had processed over $40 billion in total merchant sales across 175 countries, with a baseline of 2 million requests per minute scaling to 10 million at peak. The engineering organization is primarily Ruby on Rails and Go.

Shopify adopted Apache Kafka in 2014 as a central event bus for log aggregation and internal event collection. The initial deployment replaced a plain log-file batch system that introduced latency between event generation and availability in downstream dashboards and Hadoop storage. From 2014 to 2016, clusters were managed across data centre regions using Chef. In 2017, Shopify began migrating to Google Cloud Platform, completing the move to Kubernetes-managed Kafka by 2018. The change data capture system, which would become one of the most technically demanding parts of the Kafka estate, was rebuilt on Kafka Connect and Debezium in 2021.

Key milestones:

2013: Shopify open-sources Sarama, a Go client library for Apache Kafka 0.8, built internally to avoid a JVM dependency
2014: Kafka adopted as the internal data bus; Ruby on Rails producer pipeline built using SysV message queues and Go Sarama
2016: Multi-tenant Kafka clusters managed across data centre regions via Chef
2017: Cloud migration begins; flash sales Kafka architecture presented at Kafka Summit San Francisco
2018: Full migration to Kubernetes StatefulSets on GCP; Sam Obeid and Christopher Vollick present the work at Kafka Summit London
2020: Platform processes 1.75 trillion Kafka messages per month; CDC pipeline peaks at 100,000 records/sec during BFCM
2021: Log-based CDC pipeline published; 400TB of CDC data stored in Kafka
2022: HybridSource pattern for Flink and Kafka archival documented; BFCM live map pipeline using Flink, Kafka, and SSE published
2024: BFCM data push reaches 12TB per minute; Kafka partition scaling identified as critical for analytics data freshness

Shopify’s Kafka use cases

Event collection and log aggregation

The original use case from 2014. Kafka serves as the company-wide event bus for collecting events and aggregating log data from all internal systems and data centres. The data flows from producers through regional clusters, mirrors to an aggregate cluster, and lands in the data warehouse formatted as Apache Parquet. This pipeline replaced a batch log-file system that could not provide timely data for internal dashboards.

Change data capture

Shopify’s core application is a sharded monolith backed by more than 100 independent MySQL database shards. The CDC pipeline streams binary log mutations from every shard into Kafka, presenting downstream consumers with a single compacted topic per logical table rather than requiring them to subscribe to 100+ shard-specific topics.

Before 2021, this was handled by Longboat, a query-based extraction tool that polled tables periodically. The limitation was that hard deletes were invisible: once a row was removed, Longboat could not capture the deletion. The replacement system uses Kafka Connect and Debezium to read MySQL binary logs directly, capturing every insert, update, and delete with a P99 latency of under 10 seconds from database write to Kafka availability.

Real-time buyer signal pipeline (Shopify Inbox)

Shopify Inbox is the merchant-to-customer messaging product. A real-time pipeline combines two Kafka event types: Monorail events (Shopify’s internal structured event abstraction) and CDC events. These feed into Apache Beam jobs running on Google Cloud Dataflow that score customer intent signals and inform the Inbox response suggestion system. At non-peak hours, the pipeline processes tens of thousands of cart and checkout events per second.

BFCM live map

During Black Friday and Cyber Monday, Shopify runs a live map showing real-time global sales. Apache Flink reads from Kafka topics, computes per-region aggregations, and publishes results back to Kafka topics. A Golang SSE server subscribes to those output topics and pushes data to web clients. The 2021 BFCM event ingested 323 billion rows of data with end-to-end visualization latency of 21 seconds and 100% uptime throughout the event.

Elasticsearch multi-DC replication

Kafka is used to replicate Elasticsearch index updates across geographic regions, enabling a transition from active-passive to active-active multi-region search.

Microservice messaging

Ruby on Rails and Go services produce events through Monorail, Shopify’s internal event abstraction layer. Monorail enforces schemas, supports schema versioning, and defines explicit producer-consumer contracts on top of raw Kafka topics.

Scale and throughput

MetricValueSource / ContextMonthly Kafka messages1.75 trillionShopify Engineering Blog, December 2020Events per day (all clusters)BillionsShopify Engineering Blog, November 2018CDC average throughput65,000 records/secCDC architecture post, March 2021CDC peak throughput (BFCM 2020)100,000 records/secCDC architecture post, March 2021CDC data stored in Kafka400TB+CDC architecture post, March 2021Debezium connectors~150 across 12 Kubernetes podsCDC architecture post, March 2021Cart/checkout events per second (non-peak)Tens of thousandsShopify Inbox pipeline post, December 2021BFCM 2021 rows ingested323 billionSSE data streaming post, November 2022BFCM 2024 data push12TB per minuteBFCM readiness post, November 2025Producer pipeline throughput (2014)Thousands of events/sec; billions per weekRails producer pipeline post, July 2014

Shopify’s Kafka architecture

Cluster topology

Shopify runs multiple regional Kafka clusters, one per GCP region, and a single aggregate cluster. The aggregate cluster receives all data mirrored from the regional clusters via MirrorMaker and serves as the feed for the data warehouse. Clusters use 30-node configurations managed as Kubernetes StatefulSets with Persistent Volumes.

Kafka was previously deployed on VMs managed by Chef. The migration to Kubernetes, completed in 2018, was executed as a three-step process to avoid any downtime: deploy new cloud clusters, mirror the existing on-premises clusters to both aggregate clusters simultaneously, then migrate clients from the data centre to the cloud clusters.

Producer architecture

Rails and Go applications do not connect directly to Kafka brokers. Instead, they write events to a local SysV message queue. A Go-based producer process reads from that queue and produces to Kafka using the Sarama client library. When Shopify began containerising services with Docker, namespace isolation broke the SysV IPC mechanism, so a TCP-to-SysV proxy layer was added on the host to allow containerised services to continue writing to the queue.

This indirection fully decouples application code from Kafka availability: the SysV queue is sized to hold two hours of events, so Kafka cluster maintenance or restarts have zero impact on producers.

CDC pipeline architecture

The CDC pipeline for Shopify’s sharded monolith follows a four-stage pattern:

One Debezium MySQL connector per database shard reads from the MySQL binary log.
A RegexRouter transform consolidates events from all shards into intermediate Kafka topics.
A custom Kafka Streams application reads from the intermediate topics and demultiplexes records by table.
Output: one compacted Kafka topic per logical table, partitioned by primary key.

Consumers subscribe to the per-table output topics and see a unified view of each table regardless of how many shards the underlying data spans. Confluent Schema Registry provides data discovery and dependency tracking across CDC topics.

Stream processing

A large portion of Shopify’s Flink applications use Kafka as both source and sink. The BFCM live map is one example: Flink reads from Kafka, computes aggregations, and publishes results back to Kafka. The Streaming Capabilities team also manages HybridSource pipelines, described in the special techniques section below.

Consumer architecture

Kafka topics have per-topic retention policies. Expired data is archived to Google Cloud Storage rather than deleted, enabling Flink applications to backfill from historical data using the HybridSource connector before switching to live Kafka consumption.

Special techniques and engineering innovations

SysV message queue producer decoupling

Shopify’s producer pipeline places a SysV message queue between application code and the Kafka producer process. This pattern predates widespread use of producer buffering at the client level and gives Shopify a host-local buffer that survives Kafka cluster disruptions. The queue holds up to two hours of events, making cluster maintenance transparent to producers.

Compacted per-table CDC topics with cross-shard demultiplexing

Each logical database table is represented as a single compacted Kafka topic partitioned by primary key. Compaction keeps only the most recent record per key, giving consumers a current-state view without replaying full history. The complexity of 100+ underlying MySQL shards is absorbed entirely by the CDC pipeline: a RegexRouter transform consolidates shard-specific events into intermediate topics, and a custom Kafka Streams application repartitions records by table into the final output topics.

GCS Serde for large records

CDC events from large database rows can exceed Kafka’s 1MB per-message limit. Shopify implemented a custom Serde that detects oversized payloads, stores them in Google Cloud Storage, and writes a GCS pointer to the Kafka topic. Consumers use the same Serde to transparently retrieve the payload from GCS. Standard Kafka consumers remain compatible without modification.

HybridSource for Flink backfill

Kafka topics at Shopify have configured retention limits. When a topic expires data, that data transitions to GCS archives partitioned across thousands of splits. Flink applications that need to backfill historical data use the HybridSource connector to read from the GCS archive first, then seamlessly transition to the live Kafka topic. This removes the need for manual coordination between historical batch jobs and real-time stream processing.

Kubernetes rack-awareness and rolling restart safety

Kafka broker pods use inter-pod anti-affinity rules to prevent co-located replicas, and Kubernetes zone labels are mapped to Kafka rack configuration to ensure replicas are distributed across availability zones. Readiness probes gate rolling restarts, preventing more than one broker from going offline simultaneously during configuration changes or upgrades. Kafka’s sensitivity to concurrent broker restarts can trigger terabytes of partition rebalancing, and the pod anti-affinity and readiness probe combination is how Shopify avoids this.

Open-source contribution to Debezium

During initial CDC deployment, full table snapshots held MySQL read locks for hours at a time, blocking writes on production databases. A Shopify engineer implemented a lock-free snapshot mode for Debezium’s MySQL connector and contributed it upstream to the open-source project. This mode has since been available to all Debezium users.

Operating Kafka at scale

Deployment model: Self-managed on Google Kubernetes Engine. Brokers run as Kubernetes StatefulSets with dedicated Persistent Volumes. Kafka pods have resource-monitoring sidecars, node affinity rules to place brokers on dedicated nodes, and inter-pod anti-affinity to spread replicas.

Retention and archival: All Kafka topics have per-topic retention policies. Expired data is archived to Google Cloud Storage to support Flink HybridSource backfills rather than being discarded.

BFCM capacity planning (Game Days): Shopify runs annual chaos-engineering exercises called Game Days starting in spring, simulating 150% of the previous year’s BFCM peak load across three GCP regions. Game Days have identified Kafka-specific capacity gaps: the analytics infrastructure required partition count increases to maintain data freshness during traffic spikes, and these increases are now part of the pre-BFCM preparation checklist.

CDC connector management: Approximately 150 Debezium connectors are managed across 12 Kubernetes pods, with one connector per MySQL shard to isolate failure domains. Schema evolution for CDC consumers was an active area of work as of 2021, with the team implementing governance through Confluent Schema Registry for dependency tracking.

Cloud migration protocol: The migration from on-premises VMs to GCP Kubernetes was executed in three phases: deploy cloud clusters, run MirrorMaker to replicate from on-premises to both aggregate clusters simultaneously, then migrate clients. No downtime was required during the transition.

Challenges and how they solved them

JVM dependency for the Kafka client Kafka’s initial client libraries were Java and Scala only. Deploying a JVM instance on all of Shopify’s Go-native servers was considered impractical. In 2013, engineer Evan Huus built Sarama, a Go client library for Apache Kafka, and Shopify open-sourced it under the MIT license. Sarama is now maintained by IBM and remains a widely used Go Kafka client.

Docker namespace isolation breaking SysV IPC When Shopify containerised Ruby on Rails services, Docker’s process namespace isolation broke the SysV message queue mechanism used to buffer events for Kafka. The producer process ran on the host, but containerised services could no longer write to the host’s IPC namespace. The solution was a TCP-to-SysV proxy layer on the host that accepted messages over a TCP socket and wrote them into the SysV queue. Containerised services connect over TCP; the rest of the pipeline is unchanged.

CDC table snapshot locking Debezium’s initial snapshot mode for MySQL connector held a global read lock for the duration of a full table snapshot. For Shopify’s large tables, this meant hours of blocked writes on production databases. A Shopify engineer contributed a lock-free snapshot mode to the upstream Debezium project, reading table data without holding a lock. This resolved the blocking but introduced a different constraint: the lock-free mode cannot guarantee a consistent point-in-time snapshot while binlog events are also being consumed. Remaining limitations include tables too large to snapshot within practical timeframes.

Records exceeding the 1MB Kafka message size limit Certain database rows, particularly those with large text or blob fields, produced CDC events that exceeded Kafka’s default message size limit. Rather than raising the broker message size limit globally (which affects all topics), Shopify implemented a custom Serde that externalises oversized payloads to GCS and writes a pointer record to Kafka. Consumers using the same Serde retrieve the payload from GCS transparently.

100+ shard complexity for CDC consumers Shopify’s monolith spans more than 100 MySQL shards. A naive CDC design would require consumers to subscribe to shard-specific topics for every table they care about. Shopify’s solution abstracts the sharding entirely: a RegexRouter transform and a custom Kafka Streams demultiplexer consolidate all shard events into unified per-table output topics. Consumers interact only with the per-table topics.

Schema evolution in CDC topics Breaking changes to internal database schemas propagate to all consumers of the corresponding CDC topics. As of the March 2021 blog post, this was an active challenge with mitigation strategies under development. Confluent Schema Registry is used for dependency tracking.

Kafka partition count insufficient for BFCM analytics throughput Game Days exercises revealed that analytics Kafka topics needed higher partition counts to sustain data freshness at BFCM scale. Partition count increases are now a standard item in the pre-BFCM preparation checklist.

Full tech stack

Category	Technology	Role
Message broker	Apache Kafka	Central event bus and streaming platform for all internal pipelines
Kafka client	Sarama (originally Shopify/sarama, now IBM/sarama)	Go client library for all Kafka producers; built by Shopify in 2013
Deployment / infra	Kubernetes (GKE, GCP)	StatefulSet-based Kafka broker deployment and lifecycle management
CDC connector	Debezium	MySQL binary log CDC source connector; one instance per database shard
Connectors	Kafka Connect	Orchestrates Debezium connectors and RegexRouter transforms in the CDC pipeline
Schema registry	Confluent Schema Registry	Data discovery and dependency tracking for CDC topics
Stream processing	Kafka Streams	Custom application for CDC cross-shard demultiplexing by table
Stream processing	Apache Flink	Stateful stream processing; reads from and writes to Kafka for real-time analytics and BFCM live map
Stream / batch processing	Apache Beam / Google Cloud Dataflow	Unified batch and stream processing for the Shopify Inbox buyer signal pipeline
Batch processing	Apache Spark	Batch processing stage in the data platform pipeline
Storage sinks	Google Cloud Storage (GCS)	Archive for expired Kafka topic data; external payload store for oversized CDC records
Storage sinks	Google Cloud Bigtable	Delivery backend for the Reportify query service
Data format	Apache Parquet	On-disk format for data platform processed output
Replication	MirrorMaker	Replicates regional Kafka clusters to the aggregate cluster for data warehouse ingestion
Internal abstraction	Monorail	Internal event layer enforcing schemas and version contracts on top of raw Kafka topics
Streaming server	Go SSE server	Subscribes to Kafka topics and pushes processed data to web clients via Server-Sent Events (BFCM live map)

Key contributors

Name	Title / Team	Contribution
Evan Huus	Engineer, Shopify	Created Sarama, Shopify’s open-source Go Kafka client library (2013)
Simon Eskildsen	Engineer, Shopify	Designed the Ruby on Rails Kafka producer pipeline using SysV MQ and Go Sarama (2014)
Sam Obeid	Senior Production Engineer, Shopify	Led Kafka-on-Kubernetes migration; authored the 2018 engineering blog post; speaker at Kafka Summit London 2018
Christopher Vollick	Engineer, Shopify	Speaker at Kafka Summit London 2018, “Kafka in Containers in Docker in Kubernetes in the Cloud”
John Martin	Senior Production Engineer, Streaming Platform	Co-led log-based CDC architecture design and documentation (2021)
Adam Bellemare	Staff Engineer, Data Platform	Co-authored CDC architecture blog post (2021)
Ashay Pathak	Data Scientist, Messaging team	Co-authored Shopify Inbox real-time buyer signal pipeline post (2021)
Selina Li	Data Scientist, Messaging team	Co-authored Shopify Inbox real-time buyer signal pipeline post (2021)
Kevin Lam	Engineer, Streaming Capabilities team	Co-authored Flink optimisation post covering HybridSource and Kafka retention (2022)
Rafael Aguiar	Engineer, Streaming Capabilities team	Co-authored Flink optimisation post (2022)
Bao Nguyen	Senior Staff Data Engineer	Designed and documented the BFCM live map Kafka, Flink, and SSE pipeline (2022)
Arbab Ahmed	Reliability Engineering lead	Co-authored data platform scaling post covering 1.75 trillion monthly Kafka messages (2020)
Bruno Deszczynski	DPE Technical Program Manager	Co-authored data platform scaling post (2020)

Key takeaways for your own Kafka implementation

Decouple producers from brokers at the host level. Shopify’s SysV message queue layer means application code never holds a direct connection to Kafka. A two-hour local buffer absorbs cluster maintenance windows without producer-side changes. If your producers are tightly coupled to broker availability, consider an intermediary buffer before the producer client.
Build a CDC abstraction that hides database sharding from consumers. A one-connector-per-shard approach with a RegexRouter and a Kafka Streams demultiplexer lets you present unified per-table topics to downstream consumers regardless of how many shards exist. This makes the shard count a deployment detail rather than an API contract.
Compacted topics are a viable current-state store for CDC data. Shopify uses compacted per-table Kafka topics partitioned by primary key to give consumers the latest record per row without requiring them to replay full history. This is a practical alternative to maintaining a separate read database for certain consumer patterns.
Plan for large records before they cause incidents. If your data includes variable-length text or binary fields, the 1MB Kafka message limit will eventually be reached. Shopify’s GCS Serde externalises oversized payloads before they reach the broker, keeping the solution transparent to standard consumers and avoiding broker-level configuration changes that affect every topic.
Treat BFCM-scale capacity planning as a year-round process. Shopify’s Game Days exercises start in spring and run through autumn. The discovery that Kafka partition counts needed to increase for analytics freshness came from load testing, not from production incidents. If your traffic has a predictable peak season, model Kafka throughput requirements explicitly before the peak rather than scaling reactively.

Sources and further reading

Running Apache Kafka on Kubernetes at Shopify — Sam Obeid, Shopify Engineering Blog, November 2018
Kafka Producer Pipeline for Ruby on Rails — Simon Eskildsen, Shopify Engineering Blog, July 2014
Capturing Every Change From Shopify’s Sharded Monolith — John Martin and Adam Bellemare, Shopify Engineering Blog, March 2021
Building a Real-time Buyer Signal Data Pipeline for Shopify Inbox — Ashay Pathak and Selina Li, Shopify Engineering Blog, December 2021
3 (More) Tips for Optimizing Apache Flink Applications — Kevin Lam and Rafael Aguiar, Shopify Engineering Blog, December 2022
How to Reliably Scale Your Data Platform for High Volumes — Arbab Ahmed and Bruno Deszczynski, Shopify Engineering Blog, December 2020
How we prepare Shopify for BFCM (2025) — Kyle Petroski and Matthew Frail, Shopify Engineering Blog, November 2025
Using Server Sent Events to Simplify Real-time Streaming at Scale — Bao Nguyen, Shopify Engineering Blog, November 2022
Shopify open-sources Sarama, a client for Kafka 0.8 written in Go — Evan Huus, Shopify Technology Blog, August 2013
Shopify Flash-Sales and Apache Kafka — Kafka Summit San Francisco 2017
Kafka Summit London 2018 speaker announcement — Christopher Vollick and Sam Obeid, Confluent, 2018

If you want visibility into your own Kafka estate, including consumer lag, partition health, and topic throughput, give Kpow a try with a free 30-day trial. You can connect it to any Kafka cluster in minutes and deploy it via Docker, Helm, or JAR.

How Tencent uses Apache Kafka in production

Factor House — Tue, 02 Jun 2026 00:00:00 GMT

Tencent’s platform group processes 20 trillion messages per day across 700 Apache Kafka clusters, with individual applications generating burst traffic of up to 100 Gb/s. To operate at that volume, Tencent’s engineers built a federated cluster architecture that presents hundreds of physical Kafka clusters as a single logical deployment to producers and consumers, and contributed two Kafka improvement proposals back to the open-source community in the process.

The core engineering problem Kafka solves for Tencent is scale beyond what any single cluster can deliver, combined with the need to absorb overnight traffic spikes of up to 20x without requiring application-side changes.

Company overview

Tencent is a Chinese technology conglomerate operating social platforms (WeChat, QQ), digital media (Tencent Video, QQ Music), gaming, cloud infrastructure, and financial services. Its Platform and Content Group (PCG) alone spans internet, social, and content products serving hundreds of millions of daily active users.

The scale of data movement across these products is difficult to overstate. WeChat logs interactions across messaging, payments, and mini-programs. Tencent Video streams content and telemetry simultaneously. Real-time recommendation engines and trend detection systems depend on sub-second feature delivery across the same infrastructure.

Tencent began developing its own messaging infrastructure as early as 2013, when the Big Data team built TubeMQ, a Kafka-inspired internal system designed for high-throughput log collection. TubeMQ was open-sourced in 2019 and donated to the Apache Incubator, eventually becoming part of the Apache InLong project. For real-time data pipelines requiring broad ecosystem compatibility, Tencent PCG adopted Apache Kafka directly, and by 2020 had grown that deployment to 10 trillion messages per day. A year later, the figure had doubled.

Key milestones:

2013: Tencent Big Data team develops TubeMQ for internal messaging
2019: TubeMQ open-sourced and donated to the Apache Incubator
Pre-2020: Tencent PCG builds pproxy/cproxy proxy-layer federation for Kafka
2020-08: Kenway Chen, Kahn Chen, and George Shu publish a description of PCG’s federated architecture on the Confluent blog, disclosing 10 trillion messages/day across hundreds of clusters
2021-03 / 2021-06: George Shu submits KIP-694 (partition reduction) and KIP-693 (client-side circuit breaker) to the Apache Kafka community
2021-07: Kahn Chen and Liang Wang present at Kafka Summit APAC 2021, disclosing growth to 20 trillion messages/day and a move to a broker-layer master cluster federation
2023-12: Lu Shilin (CKafka kernel lead) publishes a detailed account of tiered storage at Tencent Cloud, covering local disk and COS hybrid retention

Tencent’s Kafka use cases

Tencent PCG uses Kafka across several distinct pipeline categories, each owned by different product teams.

Cross-region log ingestion: WeChat, QQ, QZone, Tencent Video, and news products route interaction and operational logs through Kafka for centralised ingestion and processing. The breadth of products means the platform must handle heterogeneous write patterns without per-team customisation.

Machine learning feature pipelines: Real-time recommendation engines and trend detection systems depend on Kafka for sub-second feature delivery. Latency requirements here are strict: stale features degrade recommendation quality directly.

Asynchronous microservice communication: Tencent PCG runs its internet, social, and content platforms as microservices communicating through Kafka topics. This decouples services that otherwise would require synchronous calls across product boundaries.

Real-time analytics: Business intelligence pipelines consume Kafka topics for real-time reporting across Tencent product lines, feeding dashboards and alerting systems.

Real-time data warehouse ingestion: Applications including Kandian and WeChat Video collect event data through Kafka into a downstream warehouse built on Hive, HBase, and HDFS, with Apache Flink handling stateful stream processing between the two layers.

Tiered cold/hot data storage (CKafka): Tencent Cloud’s managed Kafka service uses tiered storage to separate hot and cold data, offloading inactive log segments to Tencent Cloud Object Storage (COS) while keeping active data on local disk. This supports long-retention use cases without proportional disk cost growth.

Log collection and monitoring aggregation: CKafka’s product documentation describes compressed log collection and monitoring data aggregation as primary use cases for cloud customers, including integration with Tencent Cloud Log Service (CLS).

Scale and throughput

The scale figures below are drawn from public disclosures by named Tencent engineers at Kafka Summit APAC 2021 and the Confluent blog.

Messages per day: 20 trillion (July 2021)
Kafka clusters: 700 (July 2021)
Brokers (physical): ~500 (2020 figure, before master cluster design) (August 2020)
Maximum brokers per physical cluster: Up to 1,000 in the evolved federation design (July 2021)
Peak throughput (single product): 4 million messages/sec (64 GB/s) (August 2020)
Burst traffic (single application): 100 Gb/s (July 2021)
Maximum cluster bandwidth: 240 Gb/s at ~40% CPU utilisation (August 2020)
Target message loss rate: 0.01% end-to-end (August 2020)
Overnight spike magnitude: Up to 20x from product promotions and experiments (August 2020)

The jump from 10 trillion to 20 trillion messages per day between August 2020 and July 2021 gives a sense of the growth rate Tencent’s infrastructure team was tracking against. Static cluster sizing was not viable at this trajectory.

Tencent’s Kafka architecture

Tencent PCG’s Kafka architecture evolved through two distinct federation phases, both motivated by the need to exceed the scaling limits of individual clusters without requiring application changes.

Phase 1: proxy-layer federation (pproxy/cproxy)

The proxy-layer design presents multiple physical Kafka clusters as a single logical cluster to producers and consumers.

Producer proxy (pproxy) and consumer proxy (cproxy) each implement the Kafka broker protocol, so existing Kafka clients connect to them without modification. A lightweight name service maps client IDs to proxy broker collections. A custom controller service manages federation metadata: topic state across physical clusters, proxy node lifecycle, and partition composition for logical topics.

A logical topic with 8 partitions distributes those partitions across two physical clusters (4 partitions each). Clients interact only with logical topic metadata. When a physical cluster reaches capacity, a new cluster can be added to the federation in approximately 2 minutes. New clusters can be fully provisioned in 10 minutes. Metadata refresh latency runs at approximately 1 second.

The design supported up to 60 physical clusters per logical cluster, with 10 brokers per physical cluster.

Phase 2: broker-layer master cluster federation

The proxy layer imposed roughly 30% operational overhead. Every new Kafka API capability required interface modifications in both the pproxy and cproxy layers, creating code duplication and maintenance burden.

The replacement design moves federation logic into the broker layer. A master cluster aggregates metadata from multiple sub-clusters without running brokers of its own. Client SDKs fetch merged metadata directly from the master cluster, eliminating the proxy hop. Consumer groups and transactions are managed centrally in the master cluster. Redis stores configuration and offset data for disaster recovery.

This design used the KIP-500 role-splitting architecture (which removed ZooKeeper as a dependency) and reduced cluster failure recovery time from 30-60 minutes under standard Kafka to under 1 minute.

CKafka: Tencent Cloud’s managed Kafka service

CKafka is Tencent Cloud’s managed Kafka offering. Its control plane is isolated from the data plane: control operations go through the TencentCloud API SDK, while data operations use the standard Apache Kafka SDK. The service is 100% compatible with the Kafka API from version 0.9 through 2.8, supporting Kafka versions 2.4, 2.8, and 3.2.

CKafka deploys across availability zones with automated fault recovery and integrates with 15+ Tencent Cloud products, including TencentDB, Elasticsearch Service, COS, Tencent Kubernetes Engine (TKE), Oceanus (the managed Flink service), and CLS.

Producer architecture

Tencent’s pproxy layer handled producer-side batching transparently. In the master cluster design, producers connect directly to sub-cluster brokers using merged metadata from the master cluster. The 0.01% end-to-end message loss target shaped producer reliability configuration across the PCG platform.

Consumer architecture

Consumer group management moved to the master cluster in Phase 2, centralising offset tracking and group coordination. CKafka exposes offset reset and message querying by offset or time through the console, which covers the most common operational needs for cloud customers.

Stream processing

Apache Flink sits downstream of Kafka for stateful processing in Tencent’s real-time data warehouse pipelines. Kafka topics feed Flink jobs that produce output to Hive, HBase, and HDFS for downstream analytics.

Kafka Connect ecosystem

No public disclosure from Tencent covers their use of Kafka Connect connectors specifically. CKafka’s integration with TencentDB, CLS, and Oceanus is documented at the product level, but connector implementation details have not been published by named contributors.

Special techniques and engineering innovations

Logical-to-physical partition mapping

The pproxy/cproxy layer remaps logical partition numbers to physical cluster partitions transparently to clients. Capacity can be expanded without data rebalancing or application-side changes, and zero-downtime cluster decommission is supported: the target cluster is marked read-only, producer metadata is updated first to halt writes, and consumer metadata is updated after topics expire.

KIP-693: client-side circuit breaker for partition write errors

George Shu (General Manager, PCG Data Infrastructure and Platforms) submitted KIP-693 to the Apache Kafka community in 2021. The problem it addresses: disk failures and high disk utilisation in some partitions cause long write latency, which fills the shared producer buffer and degrades throughput for all other partitions on the same broker. KIP-693 proposes a configuration-driven circuit-breaking mechanism to mute failing partitions at the client level, isolating the degradation.

KIP-694: support for reducing topic partitions

KIP-694, also submitted by George Shu, addresses the inverse problem: Kafka supports adding partitions to a topic but not removing them. At Tencent’s scale, traffic spikes require rapidly expanding partition counts. After the spike, those partitions cannot be reclaimed, which steadily increases cluster metadata overhead. KIP-694 proposes lossless partition reduction as a native Kafka capability.

Tiered storage: local disk and COS hybrid (CKafka)

Lu Shilin (CKafka kernel lead) documented this architecture in December 2023. Hot data is retained on local cloud disks. Inactive log segments are uploaded asynchronously to Tencent COS (object storage). A “Local Retention” parameter controls how long uploaded segments persist locally; a separate “Retention” parameter governs the full data window.

To avoid I/O contention during upload and download, Tencent implemented parallel transfer with rate limiting. Preloading and prefetching load anticipated hot data into memory pools ahead of consumer reads. Memory management uses heap-off-heap ByteBuffer reuse to reduce GC pressure. Per-topic and per-cluster rollback capability is built in.

The motivation was straightforward: historical data access patterns were polluting the page cache, evicting hot data and degrading SLAs for active consumers.

Operating Kafka at scale

Deployment model: Tencent PCG runs a self-managed Kafka deployment across its private data centres. Tencent Cloud operates CKafka as a managed service for external cloud customers using a separate control plane.

Monitoring and observability: CKafka exposes multi-dimensional metrics at instance, topic, and consumer group granularity. A top-10 topic ranking by traffic is available in the console out of the box. One-click diagnostic inspection identifies cluster issues and risks. Tencent PCG built its own monitoring and automated management tooling internally to operate hundreds of clusters; the specific tooling stack has not been disclosed publicly by named contributors.

Cluster lifecycle management: Clusters can be initialised in 10 minutes and scaled by adding one physical cluster in 2 minutes (proxy-layer design). In the master cluster design, sub-clusters can be added without modifying the federation’s logical topology.

Partition rebalancing: CKafka performs automatic partition balancing during off-peak hours and supports automatic disk watermark adjustment to prevent business disruption during high-growth periods.

Security: ACL, SASL, and SSL authentication are console-configurable in CKafka. Integration with Tencent Cloud Access Management (CAM) and CloudAudit provides audit traceability for enterprise customers.

Developer experience: CKafka’s console supports offset reset and message querying by offset or time. Tiered storage provides granular rollback at topic or cluster level, which supports operational recovery scenarios without requiring full topic replay.

Challenges and how they solved them

20x overnight volume spikes

Product promotions and large-scale experiments routinely caused overnight traffic to spike 20x. Static cluster sizing was either wasteful during normal operation or insufficient during peaks.

Solution: the federated cluster design allows rapid provisioning of additional physical clusters (2 minutes to add one cluster) and decommissioning after peaks, without application-side changes. Clusters are treated as fungible capacity units rather than permanent infrastructure.

ZooKeeper-bounded partition count

Before Kafka 2.8, ZooKeeper imposed a practical ceiling well below one million partitions. For Tencent’s real-time data warehouse, which required a large number of topics with many partitions, this was a hard constraint.

Solution: the broker-layer federation design used the KIP-500 architecture (removing ZooKeeper) to scale beyond this ceiling. KIP-694 was submitted to the community to address the complementary problem of partition reduction after peaks.

5% annual disk failure rate causing 30-60 minute recovery

Standard Kafka’s disaster tolerance operates at the broker level. A 30-60 minute recovery window from cluster failure was incompatible with Tencent PCG’s SLA requirements.

Solution: the master cluster federation model reduced cluster failure recovery to under 1 minute. Recovery is handled at the federation coordination layer rather than requiring full cluster restart and leader election across hundreds of brokers.

30% operational overhead from the proxy layer

The pproxy/cproxy approach required continuous interface modifications to expose new Kafka API capabilities, creating code duplication across two proxy implementations.

Solution: the broker-layer master cluster model eliminated the proxy layer. SDK-level metadata merging replaced the proxy hop, and new Kafka API features became available without proxy modifications.

Keyed message ordering across federated clusters

Adding physical clusters to a federation could route messages with identical keys to different partitions, breaking ordering guarantees for key-partitioned topics.

Tencent’s approach here was pragmatic rather than architectural: they noted that keyed messages are used only occasionally in their pipelines, and they accepted the trade-off in favour of scalability. A complete technical solution for ordering across federated partitions was not described in their public disclosures.

Historical data access polluting page cache (CKafka)

Long-retention topics generated cold data reads that evicted hot data from cache, degrading SLAs for active consumers.

Solution: tiered storage offloads cold segments to COS asynchronously, keeping the page cache available for hot data. Prefetching loads anticipated consumer reads into memory pools before they are requested.

Full tech stack

Category	Tools	Notes
Message broker	Apache Kafka	Versions 2.4, 2.8, 3.2 supported in CKafka; PCG internal version not disclosed
Cluster federation (Phase 1)	pproxy / cproxy (custom)	Producer and consumer proxy implementing the Kafka broker protocol; routes to physical clusters
Cluster federation (Phase 2)	Master cluster (custom broker-layer design)	Aggregates sub-cluster metadata; central consumer group and transaction management
Configuration and offset storage	Redis	Used in master cluster federation for disaster recovery
Stream processing	Apache Flink	Stateful processing; consumes from Kafka for real-time data warehouse and lake ingestion
Remote storage (tiered)	Tencent Cloud Object Storage (COS)	Cold segment target in CKafka tiered storage architecture
Data warehouse	Hive / HBase / HDFS	Downstream storage receiving data via Kafka and Flink pipelines
Managed Kafka service	CKafka (TDMQ for CKafka)	Tencent Cloud’s managed offering; Kafka API-compatible (0.9-2.8)
Alternative messaging (non-Kafka)

Key contributors

Kenway Chen (Tech Lead and Engineering Manager, Tencent PCG): Co-author of the Confluent blog post describing PCG’s federated Kafka architecture (2020)
Kahn Chen (Software Architect, Tencent PCG): Co-author of the Confluent blog post; presented “Enhancing Apache Kafka for large-scale real-time data pipeline at Tencent” at Kafka Summit APAC 2021
George Shu (General Manager, PCG Data Infrastructure and Platforms): Co-author of the Confluent blog post; proposer of KIP-694 and KIP-693 to the Apache Kafka community
Liang Wang (Thirteen Wang) (Senior Engineer, Tencent): Co-presenter with Kahn Chen at Kafka Summit APAC 2021
Lu Shilin (鲁仕林) (Kernel Lead, Tencent Cloud CKafka): Author of “Kafka tiered storage practice and evolution at Tencent Cloud” (Juejin, December 2023)

Key takeaways for your own Kafka implementation

Treat individual clusters as bounded capacity units, not infinite resources. Tencent PCG found that growing individual clusters indefinitely was impractical. Their federation design treats each physical cluster as a bounded unit and routes logical topics across multiple clusters. If you are approaching partition or throughput ceilings, a federated or tiered architecture separates scaling concerns more cleanly than tuning a single cluster.
Separate the control plane from the data plane early. CKafka’s isolation of control operations (via the cloud API) from data operations (via the Kafka SDK) allowed Tencent Cloud to manage the managed service without exposing control-plane complexity to customers. For internal platforms, the same principle applies: provisioning, ACL management, and monitoring should not go through the same path as message production and consumption.
Model your traffic distribution before choosing partition counts. Tencent’s experience with keyed messages across federated partitions highlights that key-based ordering and federation do not compose cleanly. If your pipelines depend on key-partitioned ordering, design for that constraint at the federation boundary before scaling out clusters, rather than discovering the incompatibility under load.
Contribute upstream when you need features that standard Kafka does not provide. KIP-693 and KIP-694 both address real operational problems Tencent encountered at scale: partition write isolation and lossless partition reduction. Submitting KIPs rather than maintaining private forks keeps the internal codebase closer to upstream and benefits the broader community with solutions that have been tested against one of the world’s largest Kafka deployments.
Design tiered storage for access pattern isolation, not just cost reduction. Tencent Cloud’s CKafka tiered storage work shows that the primary motivation was cache isolation: cold data reads were evicting hot data and degrading SLAs. If you are evaluating Kafka tiered storage, measure its impact on page cache hit rates for active consumers as the primary success metric, not just storage cost savings.

Sources and further reading

Primary sources:

Kenway Chen, Kahn Chen, George Shu. “How Tencent PCG scales massive data pipelines with Apache Kafka.” Confluent Blog, 2020-08-03.
Kahn Chen, Liang Wang. “Enhancing Apache Kafka for large-scale real-time data pipeline at Tencent.” Kafka Summit APAC 2021, 2021-07-28.
Kahn Chen, Liang Wang. Kafka Summit APAC 2021 slide deck (SlideShare).
George Shu. KIP-694: Support Reducing Partitions for Topics. Apache Kafka KIP wiki, 2021-03.
George Shu. KIP-693: Client-side Circuit Breaker for Partition Write Errors. Apache Kafka KIP wiki, 2021-06.
Lu Shilin. “Kafka tiered storage practice and evolution at Tencent Cloud.” Juejin, 2023-12-19.
Tencent Cloud. “Comparison with Apache Kafka.” Tencent Cloud documentation.

If you want visibility into your own Kafka clusters, consumer lag, and topic throughput without building custom dashboards from scratch, Kpow gives you a 30-day free trial. You can connect it to any Kafka cluster in minutes and deploy via Docker, Helm, or JAR.

How Wix uses Apache Kafka in production

Factor House — Tue, 02 Jun 2026 00:00:00 GMT

Wix runs Apache Kafka as the event backbone for more than 2,200 microservices, processing 66 billion messages every day across 50,000 topics and 500,000 partitions. The scale is notable, but the more interesting engineering story is how Wix achieved it: by building Greyhound, an open-source Kafka SDK that abstracts the native client for every service in the fleet, and by migrating the entire platform to Confluent Cloud in 2021 without taking a single service offline.

Company overview

Wix is a cloud-based website creation and hosting platform used by more than 200 million registered users worldwide. At its core, the platform is a collection of loosely coupled product surfaces: site builder, e-commerce, bookings, blog, CRM, and several hundred third-party app integrations. Serving those surfaces requires Wix’s engineering organisation to coordinate over 2,200 microservices, deployed across 18 data centres and points of presence, spanning two cloud providers (Google Cloud and AWS) and four geographic regions.

Kafka became the communication layer for those services as the microservices architecture matured. By 2020, 1,500 services were exchanging events through Kafka. By the time Natan Silnitsky presented at Kafka Summit London in April 2022, that number had grown to 2,200+ services, 50,000 topics, and 500,000+ partitions, handling 15 billion business events per day. The 66 billion total daily message count includes the internal infrastructure traffic that flows through Kafka alongside business events.

Milestone	Date
Greyhound open-sourced on GitHub (v0.0.5)	June 2020
Engineering post: Kafka as event-driven system for 1,500 microservices	2020
Migration of 2,000 microservices to multi-cluster Confluent Cloud, zero downtime	2021
Kafka Summit Americas: “5 lessons learned migrating to Confluent Cloud”	2021
Kafka Summit London keynote + “Kafka-based global data mesh at Wix” talk	April 2022
Scale reaches 2,200+ microservices, 50K topics, 500K+ partitions, 15B daily business events	2022
Online ML feature store rebuilt on Kafka and Flink, supporting 3,000+ features	2023

Wix’s Kafka use cases

Microservices event-driven communication

The primary use of Kafka at Wix is replacing synchronous RPC between services with asynchronous events. Rather than service A calling service B directly, A publishes a domain event to a Kafka topic and any interested downstream service subscribes. This decoupling allows Wix to evolve services independently and absorb traffic spikes without cascading failures across the dependency graph.

Materialized views for query offloading

One concrete problem Kafka solved was query volume on Wix’s MetaSite service, which holds metadata for every site on the platform: owner, version, and installed applications. Before Kafka-based projections, this single service was receiving over 1 million requests per minute from other services that needed a slice of that data.

The solution was to stream all MetaSite change events to a Kafka topic and let each downstream consumer build its own purpose-built read model. Services that previously queried MetaSite directly now maintain a local projection that is updated via Kafka, reducing their dependency to an eventual-consistency relationship rather than a real-time RPC call.

Change data capture

Wix uses Debezium connectors to capture row-level database changes and publish them as Kafka events. Services downstream can consume the CDC topic and produce their own clean domain event contracts rather than exposing raw database schemas as the inter-service API.

Protobuf-backed API contracts

At Wix, a Kafka topic schema is an API contract. All Kafka messages are serialised with Protobuf, and the .proto file is the canonical definition shared across Kafka, gRPC, and external-facing APIs. This means a new domain can be onboarded and integrated with surrounding services within a few days.

Real-time ML feature store

Wix rebuilt its online feature store around Kafka and Apache Flink. The system processes billions of events daily and maintains over 3,000 ML features with near-real-time update latency, feeding personalisation and recommendation models across the platform.

Global data mesh

Wix’s platform spans four regions and two cloud providers. Rather than each service knowing where to find its upstreams, Kafka provides the transport layer for a global data mesh. Services publish to a logical topic namespace and Kafka handles the cross-region replication, meaning producers and consumers remain unaware of the underlying geographic topology.

Scale and throughput

Metric	Value
Total daily Kafka messages	66 billion
Daily business events (data mesh)	15 billion across 4 regions
Microservices using Kafka	2,200+
Topics	~50,000
Partitions	500,000+
Geographic regions	4
Data centres and POPs	18
Cloud providers	2 (Google Cloud, AWS)
Developers interacting with Kafka daily	900+

Wix’s Kafka architecture

The Greyhound SDK

No Wix service talks to Kafka through the native client directly. Every JVM service goes through Greyhound, the open-source Kafka SDK that Wix’s data streaming team built and maintains. Greyhound is written in Scala using the ZIO functional library and adds a higher-level interface on top of the native Kafka client, providing:

Concurrent message processing: configurable parallelism per consumer handler, not limited by partition count
Retry policies: first-class retry configuration on both consumers and producers, using dedicated retry topics with back-off rather than blocking partition processing
Batch processing: optional batch consumption for throughput-sensitive pipelines
Context propagation and metrics: distributed tracing context forwarded through events; Prometheus gauges emitted per topic-consumer-handler

For non-JVM services, Greyhound ships as a sidecar proxy. Services that are not on the JVM send and receive via the Greyhound proxy over gRPC, which then handles the Kafka interaction on their behalf. This is also what made the Confluent Cloud migration possible at scale: routing logic and cluster addressing only had to be updated in Greyhound, not in each of the 2,200 services.

The Greyhound repository is available at github.com/wix/greyhound.

Schema management

Wix does not use Confluent Schema Registry. When Wix evaluated it, the registry did not fit their requirement to unify Kafka event schemas, gRPC service contracts, and external API definitions in a single consistent system. Instead, Wix’s infrastructure and developer experience teams built a custom schema management platform backed by Protobuf. The platform provides automatic schema discovery, serialisation, validation, and compatibility enforcement across all three communication paradigms. The .proto file is the single source of truth for any data contract, regardless of whether the transport is Kafka, gRPC, or an HTTP API.

Deployment model

Wix runs on multi-cluster Confluent Cloud. Before 2021 the platform ran self-hosted Kafka clusters; the migration to managed Confluent is covered in the Challenges section below.

Stream processing

Apache Flink is used for stateful stream processing over Kafka topics. Wix runs Flink on Confluent Cloud in serverless mode, using FlinkSQL for complex aggregations and feature transformations for the ML feature store. Flink reads from and writes to Kafka topics directly.

Producer architecture

Kafka messages are Protobuf-serialised, with Greyhound handling serialisation through its default serialiser, which accepts Protobuf-derived Scala case classes. Producer configuration is managed centrally through Greyhound rather than in individual services.

Consumer architecture

Consumer group configuration, offset management, and retry topic routing are all managed through Greyhound. Lag monitoring is handled through the internal Kafka Control Plane, described in the Operations section. Greyhound emits a Prometheus gauge per topic-consumer-handler showing the currently longest-running handler in each pod, which feeds directly into lag dashboards.

Kafka Connect ecosystem

Wix uses Debezium as a Kafka Connect source connector for CDC, capturing database change events and publishing them as Kafka topics that downstream services can consume.

Special techniques and engineering innovations

gRPC fan-out proxy

High-volume topics consumed by many services were a meaningful driver of Confluent Cloud costs. Wix addressed this by building a push-based gRPC fan-out proxy. Instead of each subscribing service maintaining its own Kafka consumer for a shared topic, the proxy consumes each topic once and fans out delivery to N subscribers over gRPC streaming connections. This removed redundant consumer groups from Confluent and reduced Wix’s Kafka infrastructure bill by 30%.

Chunked message delivery

Kafka’s default maximum message size is 1 MB. Some Wix payloads, such as large site content objects, occasionally exceeded this limit. Rather than raising message.max.bytes cluster-wide, Wix’s engineer Amit Pe’er built a Chunks Producer/Consumer pattern. The producer splits an oversized message into fragments, persists each fragment to a local H2 database on disk, and sends only the chunk IDs over Kafka. The consumer retrieves chunks by ID, with retry logic that handles out-of-order arrival, and reassembles the original message before passing it to the handler. The Kafka topic carries only small identifier messages; the bulk data moves through the local disk store.

Zero-downtime migration to Confluent Cloud

The 2021 migration from self-hosted Kafka to Confluent Cloud moved 10,000 topics and 100,000+ partitions serving 2,000 microservices without any service downtime. The approach relied on three decisions:

Centralised routing in Greyhound: because all Kafka interaction goes through the SDK or proxy, cluster addressing only needed to change in one place
Replication bridge: a dedicated replication service mirrored messages from the self-hosted cluster to Confluent Cloud while consumers were being cut over, so no messages were lost during the transition window
Gradual per-topic cutover: topics were migrated in small batches rather than all at once, limiting blast radius and making per-topic rollback straightforward

Individual service owners made no code changes during the migration.

Operating Kafka at scale

Consumer lag tooling: TLLSR

Wix organises its Kafka operational tooling into five capabilities, described internally as TLLSR:

Trace: follow a specific message through the processing pipeline to diagnose where it was delayed or dropped
Lookup: retrieve a specific event by key from a topic, useful for investigating individual record issues
Longest-Running: identify the slowest currently active consumer handler across the fleet, surfaced via the Prometheus gauge Greyhound emits per pod
Skip: advance the consumer offset past a stuck message, used when a specific record is causing repeated processing failures and the data loss is acceptable
Redistribute: rebalance events from a lagging partition across all partitions, resolving single-partition lag caused by hot key distribution

These capabilities are available through Wix’s internal Kafka Control Plane UI, which includes a dedicated Consumers Lag View that developers use to investigate root causes across all 2,200 services from a single interface.

Monitoring

Greyhound emits Prometheus metrics for each topic-consumer-handler pair, including the longest-running handler currently active in each pod. These feed into the lag dashboards in the Control Plane. The combination of per-pod handler metrics and the TLLSR tooling means developers can diagnose most consumer lag issues without needing to inspect broker-side metrics directly.

Schema governance

Schema compatibility is enforced automatically through Wix’s custom schema platform. New schemas are reviewed before publishing, and breaking changes are caught at the tooling layer rather than at runtime. Because the same .proto files serve both Kafka and gRPC, schema governance covers the full inter-service communication surface in one process.

Challenges and how they solved them

Single-partition consumer lag

Problem: Events with poorly distributed keys concentrated all traffic on one partition, while adjacent partitions sat idle. The lagging partition accumulated a backlog that the consumer could not drain.

Root cause: Key design that did not distribute cardinality evenly across the partition space.

Solution: Wix built the Redistribute tool inside Greyhound, which takes events from the lagging partition and spreads them across all partitions. This restores the parallel processing that the consumer handler was configured for.

Outcome: Operators can resolve single-partition lag incidents through the Control Plane UI without code changes or consumer restarts.

Slow consumer handlers blocking a partition

Problem: One slow or stuck consumer handler blocked all subsequent messages in the same partition. Because Kafka delivers messages in order within a partition, a handler that does not complete its work holds up everything behind it.

Root cause: An unexpected downstream slowdown, typically a slow database call or a downstream service timeout, that caused handler processing time to spike.

Solution: The Longest-Running monitor in the Control Plane surfaces the offending handler and pod. For cases where the stuck message cannot be re-processed successfully, the Skip tool advances the offset past it. For higher-value flows where skipping is not acceptable, Greyhound’s retry policy routes the message to a retry topic, unblocking the main partition while the retry works in the background.

Migrating 2,000 services to Confluent Cloud without downtime

Problem: Wix’s self-hosted Kafka clusters had grown to 10,000 topics and 100,000+ partitions. Moving to managed Kafka without involving the owners of 2,000 individual services required a migration approach that was transparent to producers and consumers.

Root cause: Self-hosted operational overhead had grown to the point where the engineering investment in cluster management outweighed its benefits.

Solution: The Greyhound abstraction was the enabling factor. Because routing and cluster addressing were centralised in the SDK and proxy, the migration team updated one layer rather than 2,000 services. A replication bridge kept data flowing from old to new clusters during the cutover window, and per-topic gradual migration kept blast radius small.

Outcome: Zero downtime, no service code changes required.

Schema management across Kafka and gRPC

Problem: Wix needed to govern event schemas for Kafka topics, gRPC service contracts, and external APIs in a unified way. Confluent Schema Registry did not support this multi-paradigm requirement.

Root cause: Wix’s architecture uses Kafka, gRPC, and HTTP as first-class transport protocols, and each had historically been managed separately, creating drift and duplication.

Solution: A custom schema management platform, backed by Protobuf, that treats the .proto file as the single authoritative schema for all three transports. Automatic discovery, compatibility checking, and developer tooling are built on top of this layer.

Outcome: A new domain can be onboarded and fully integrated with surrounding services in a few days, down from weeks when schemas were managed separately per protocol.

Kafka bill from fan-out consumption

Problem: High-volume topics consumed by many services created redundant consumer groups on Confluent Cloud, each paying for the same data repeatedly.

Root cause: Each subscribing service maintained its own Kafka consumer, which is the natural pattern for Kafka but expensive when the same high-volume topic has many subscribers.

Solution: A push-based gRPC fan-out proxy that consumes each high-volume topic once and delivers to all subscribers over gRPC streaming, eliminating the redundant consumer groups.

Outcome: 30% reduction in Kafka infrastructure costs.

Full tech stack

Category	Tool	Role
Message broker	Apache Kafka (Confluent Cloud, multi-cluster)	Core event streaming platform; multi-region deployment across Google Cloud and AWS
Client SDK	Greyhound (Wix open source, Scala/ZIO)	High-level Kafka SDK for all JVM services; sidecar proxy for non-JVM services
Stream processing	Apache Flink (Confluent Cloud serverless)	Stateful stream processing for the online ML feature store; FlinkSQL for aggregations
Serialisation	Protobuf	Universal message format across Kafka, gRPC, and external APIs
Schema management	Custom internal platform	Unified Protobuf schema governance for Kafka, gRPC, and HTTP API contracts
CDC connector	Debezium	Captures database change events and publishes them to Kafka topics
Large-message storage	H2 (embedded)	On-disk chunk store for the Chunks Producer/Consumer pattern
Metrics	Prometheus	Consumer handler lag metrics emitted by Greyhound, per topic-consumer-handler per pod
Inter-service delivery	gRPC	Fan-out delivery from the Kafka proxy to downstream subscriber services
Infrastructure	Google Cloud + AWS	Multi-cloud hosting for Kafka clusters and compute across 4 regions

Key contributors

Name	Role	Contribution
Natan Silnitsky	Backend-Infra Engineer, Data Streaming Team	Primary architect of Wix’s Kafka infrastructure; creator and lead of Greyhound; presenter at Kafka Summit Americas 2021 and Kafka Summit London 2022. Author of most of the Wix Kafka engineering blog series.
Avi Perez	Engineering leader, Wix	Featured in the Kafka Summit London 2022 keynote presenting Wix’s approach to modern data flow and data pipelines.
Amit Pe’er	Engineer, Wix	Author of the Chunks Producer/Consumer pattern for handling oversized Kafka messages.

Key takeaways for your own Kafka implementation

Centralise your Kafka client layer early. Wix’s ability to migrate 2,000 services to Confluent Cloud without touching individual service code came directly from having a shared SDK that all services used. If each service had configured Kafka independently, the migration would have required coordinating changes across the entire engineering organisation. A shared client layer is worth the investment before you need to do something like this, not after.
Treat Kafka topic schemas as API contracts, not implementation details. Wix uses the same Protobuf .proto file as the contract for Kafka events, gRPC calls, and external APIs. This eliminates schema drift between communication protocols and makes onboarding new domains faster because the tooling and governance are the same regardless of transport.
Consumer lag tooling needs both visibility and remediation. Wix found that monitoring consumer lag is not enough on its own. Their TLLSR tooling pairs each monitoring capability (Trace, Lookup, Longest-Running) with a corresponding action (Skip, Redistribute). Without the remediation side, engineers can diagnose problems but still need to write ad-hoc scripts or perform manual offset management to resolve them.
Fan-out patterns can meaningfully reduce managed Kafka costs. If multiple services consume the same high-volume topic, maintaining separate consumer groups for each scales your Confluent Cloud costs linearly with the number of subscribers. A push-based fan-out proxy, consuming each topic once, can reduce that cost significantly as Wix found with their 30% bill reduction.
Migration to managed Kafka is feasible at large scale with the right abstraction. The routing of 10,000 topics and 100,000+ partitions from self-hosted clusters to Confluent Cloud, with zero downtime, was made tractable by having a single layer where cluster addressing lived. Any organisation considering a similar move should audit how many places in their codebase hold Kafka connection configuration before starting.

Sources and further reading

Natan Silnitsky — Using Apache Kafka as the event-driven system for 1,500 microservices at Wix (Wix Engineering / Medium, 2020)
Natan Silnitsky — Migrating to a multi-cluster managed Kafka with 0 downtime (Wix Engineering / Medium, 2021)
Natan Silnitsky — How Wix manages schemas for Kafka (and gRPC) used by 2,000 microservices (Wix Engineering / Medium, 2021)
Natan Silnitsky — 5 lessons learned for successful migration to Confluent Cloud (Kafka Summit Americas 2021)
Natan Silnitsky — Kafka-based global data mesh at Wix (Kafka Summit London 2022)
Natan Silnitsky — Troubleshooting Kafka for 2,000 microservices at Wix (Wix Engineering / Medium, 2022)
Natan Silnitsky — 6 event-driven architecture patterns, part 1 (Wix Engineering / Medium, 2021)
Natan Silnitsky — 6 event-driven architecture patterns, part 2 (Wix Engineering / Medium, 2021)
Amit Pe’er — Chunks producer/consumer: handling large messages within Kafka (Wix Engineering / Medium, 2021)
Greyhound — rich Kafka client library (Wix GitHub, 2020–present)

If you want visibility into your own Kafka consumers, lag, and message throughput, Kpow gives you a single interface for monitoring and managing Kafka clusters. You can connect it to any cluster, including Confluent Cloud, and try it free for 30 days.

Conduktor: Review, pricing, and best alternatives in 2026

Factor House — Mon, 01 Jun 2026 00:00:00 GMT

Key takeaways

Conduktor is a commercial Kafka governance platform built around RBAC, self-service topic workflows, and a proxy (Gateway) that enforces encryption and policy at the wire level without modifying producer or consumer code.
Its strongest use cases are multi-team enterprise environments where governance, compliance (PCI DSS, HIPAA, GDPR), and self-service are priorities.
Per-seat pricing ($1,200/seat/year on the Team Edition) becomes a significant line item for teams above 20-30 users; at 100 users across three clusters, estimated list price reaches $80,000-$150,000 per year.
The free Community tier has been incrementally restricted: version 1.43 reduced the number of allowed servers and users, and connecting to any SSL-enabled or multi-broker cluster requires a paid plan. Community sentiment on Reddit consistently describes the free tier as no longer viable for startup or staging environments. [r/apachekafka, multiple threads, 2023-2026]
Known gaps include the absence of distributed tracing, no integration with Azure’s native Schema Registry, and an operational overhead introduced by the Gateway proxy (2-10ms added latency, a required PostgreSQL 13+ dependency, and a single point of failure to manage for high availability).
Kpow is worth evaluating if per-cluster pricing, a stateless deployment model, proven stability on large partitioned clusters, or WCAG-compliant UI matters to your team.

What is Conduktor?

Conduktor is a commercial Kafka management and governance platform. It has two main components: Console, a React-based web UI for managing topics, schemas, connectors, consumer groups, and access controls across multiple clusters; and Gateway, a Kafka proxy that sits between clients and brokers to enforce encryption, data masking, quota policies, and multi-tenancy rules at the wire level without requiring application changes.

The product was originally distributed as a JavaFX desktop application. In 2023, Conduktor shifted its focus to Console (its centrally deployed web platform) and deprecated both the desktop application (end of life at end of 2025) and its Testing product. Conduktor achieved SOC2 Type II certification in 2023 and has grown its enterprise feature set — LDAP, RBAC, audit logging, and schema registry integrations — in response to requirements from regulated-industry customers.

Conduktor review

Functionalities

Conduktor’s core functionality covers the expected surface area for enterprise Kafka management: topic creation and management, consumer group inspection, schema registry integration (Confluent-compatible and AWS Glue), Kafka Connect management with a UI wizard for connector deployment, ksqlDB integration, and multi-cluster support.

The Gateway is the more differentiated component. It enables field-level encryption, data masking, and policy enforcement at the wire level. Key management integrates with AWS KMS, Azure Key Vault, GCP Cloud KMS, HashiCorp Vault, and Fortanix. The Topic as a Service feature — which lets teams self-serve topic creation within defined policy guardrails — received what Conduktor’s CTO described as “overwhelming praise” at Kafka Summit London 2024. [Stéphane Derosiaux, Medium/Conduktor, April 2024]

The Gateway also provides capabilities that extend beyond basic governance. In disaster recovery scenarios, the proxy architecture allows platform teams to failover backend cluster connections dynamically without requiring any client-side reconfiguration or application restarts — a meaningful operational advantage for multi-region deployments. [r/apachekafka, “DR for Kafka Cluster,” January 2025] The Gateway additionally supports virtual topic filtering, which executes server-side filtering logic without spinning up dedicated stream processors or creating physical duplicate topics. Practitioners evaluating this approach on Reddit noted it reduces network bandwidth and client-side processing load compared to a Kafka Streams or Flink implementation for the same problem. [r/apachekafka, “Is there any way to perform server-side filtering?”, August 2024]

There are two documented functional gaps. First, Conduktor has no native distributed tracing capability, which prevents it from being considered a complete observability platform. [Damaso Sanoja, Redpanda blog, March 2023] Second, while Azure Event Hubs is accessible via Kafka protocol compatibility, Azure’s native Schema Registry is not integrated — teams using Azure Event Hubs cannot access schema features within Conduktor Console. [Farbod Ahmadian, DataChef blog, November 2024]

One operational nuance: when Console’s internal index is incomplete or stale, it falls back to querying the Kafka cluster directly, which can result in slow page loads. [Conduktor support documentation, undated] The monitoring graph also averages brief metric spikes across large time windows, making short-lived spikes invisible at wider zoom levels. [Conduktor documentation, undated]

The Conduktor Testing product was deprecated in 2023 despite customer adoption, and the Terraform provider’s experimental generic resource is explicitly not recommended for production use.

Deployment and operations

Conduktor Console requires PostgreSQL 13+ as an external dependency — this is not optional. Minimum resource requirements are 2 CPU and 3 GB RAM for Console, plus 2 CPU and 4 GB RAM for Gateway if you are running the proxy component. A Kubernetes/Helm path is documented and functional. Docker deployment is supported; the image was reduced from 1.66 GB to 800 MB in 2023 as part of a documented effort to simplify onboarding. [Conduktor retrospective, 2023]

The retirement of the native desktop client has been a consistent source of friction for solo developers and small teams. Under the desktop model, installation required a standard ZIP extraction or an MSI installer. The Console model requires installing WSL2, configuring Docker on a Linux subsystem, downloading a specific compose file, placing it in a precisely named folder to avoid container configuration errors, and running container operations from the command line. Community commentary characterises this as an unreasonable setup burden for a tool whose primary use case — inspecting a local topic — does not require a centrally deployed web application. [r/apachekafka, “Open source clone of Conduktor Desktop,” May 2026] The consensus among practitioners on Reddit is that the Console architecture is well-suited to multi-tenant teams deploying to a shared server with Okta and RBAC, but it represents a meaningful barrier to entry for individual developers. This friction has driven some users toward native desktop alternatives and lightweight open-source web UIs that carry no containerisation dependency. [r/apachekafka, “I built a free, open-source desktop Kafka client,” 2025]

The Gateway introduces latency overhead: typically 2-10ms per message (1-5ms network hop plus 0.5-5ms message processing). For sub-10ms latency requirements, this overhead may be prohibitive. The proxy also becomes a potential single point of failure and requires HA deployment planning. [Kai Waehner, personal blog, October 2025]

Conduktor Cloud (their SaaS offering) failed to gain meaningful traction. In their 2023 retrospective, they attributed this to customers preferring on-premise deployments for security and privacy reasons, and to the absence of VPC peering and local agent support. Active Directory integration requires a specific workaround — setting the LDAP search filter to (sAMAccountName={0}) — when the default configuration returns an “invalid user” error.

Connecting Conduktor to a Strimzi-managed Kafka Connect cluster requires exposing the Connect REST service and disabling KafkaConnector resources. Practitioners have reported HTTP 500 errors and permission denied errors when attempting this integration without the workaround. [fathimaSheikh and mozarik, GitHub/Strimzi Discussions #8543, May-June 2023]

Access control and security

RBAC is Conduktor’s most consistently cited strength. The Console logs every user action — including produce, consume, and admin requests — across 70+ event types with user identity, IP address, timestamp, topic, and partition. [Conduktor documentation, undated]

SSO via OIDC and LDAP is available on all tiers including Community. SAML 2.0 requires the Enterprise plan. One independent comparison notes LDAP integration as “limited” relative to OIDC. [Hayato Shimizu, AxonOps blog, December 2025]

The Gateway’s field-level encryption and data masking operate at the wire level, meaning producers and consumers do not need to be modified. Key management integrates with AWS KMS, Azure Key Vault, GCP Cloud KMS, HashiCorp Vault, and Fortanix. The Gateway can additionally function as a secure intermediary for managed services like Confluent Cloud, allowing organisations to enforce custom security policies and access controls on top of a cloud-hosted cluster without modifying client applications. [r/apachekafka, “Is anyone using Confluent Cloud on a private dedicated network?”, 2023] [Conduktor documentation and product pages, undated]

The Gateway’s proxy architecture has drawn architectural commentary from competitors: a Lenses employee raised concerns that placing a component between clients and brokers “may add complexity and risk.” [Marios, Lenses Community Forum, January 2026] This is a trade-off worth evaluating.

User interface

Console uses a React-based interface. Practitioners describing it on Product Hunt have called the search function fast and praised the overall UX. [Mark Shannon, Johan Netzler, Product Hunt, ~2023].

Community feedback on Reddit is consistent with this assessment on the UX itself: a senior engineer described their personal experience as “highly positive due to its excellent user experience.” The same thread noted that their organisation abandoned the tool because of licensing fees rather than any functional dissatisfaction — a pattern that appears repeatedly across practitioner forums. [r/apachekafka, “The best Kafka Management tool,” 2025] The UI is widely regarded as Conduktor’s strongest attribute; the points of contention are almost entirely commercial and architectural rather than functional.

One independent reviewer notes that the “desktop app heritage shows in architecture,” though this appears to be a reference to product lineage rather than a specific UI deficiency. [Hayato Shimizu, AxonOps blog, December 2025]

The one documented UI limitation is the monitoring graph behaviour noted above: brief spikes averaged across a wide time window render as zero, making them invisible without narrowing the time range. [Conduktor documentation, undated]

Ecosystem

Conduktor integrates with AWS MSK (including IAM authentication, eliminating long-lived API key exposure), Confluent Platform and Cloud, Redpanda, Aiven, and Strimzi. Confluent-compatible and AWS Glue Schema Registries are both supported. ksqlDB and Kafka Connect management are included in the Console UI. A formal partnership integration with Redpanda positions Conduktor as a governance and management layer on top of Redpanda clusters. [Stéphane Derosiaux, Medium/Conduktor, undated]

The Azure gap is documented and specific: Azure Event Hubs is reachable via Kafka protocol compatibility, but Azure’s native Schema Registry is not integrated. [Farbod Ahmadian, DataChef blog, November 2024] Apache Flink support was not documented in any source reviewed for this article. [UNVERIFIED — needs source]

A Terraform provider is available for GitOps workflows. The experimental generic resource within that provider is not recommended for production. [Conduktor documentation, undated]

Customer support

Enterprise tier customers receive a dedicated Solutions Engineering team.

Community tier users receive public documentation, a community Slack, and best-effort email support. Conduktor switched from Intercom to Zendesk for customer support tooling in 2023, which they described as resulting in a “significant improvement in Quality of Service.” [Conduktor retrospective, 2023]

No named negative reviews of Conduktor’s support quality were found on any third-party review platform at time of research. G2, Capterra, PeerSpot, GetApp, and SourceForge each had zero verified user reviews at the time of writing.

Best for

Conduktor suits organisations running Kafka at multi-team scale where governance, compliance, and self-service are primary requirements. The documented customer base skews toward regulated industries — financial services, logistics, and healthcare — where field-level encryption, audit logging across 70+ event types, and compliance certifications (SOC2 Type II) are prerequisites rather than nice-to-haves.

Teams using AWS MSK who want IAM-native authentication and a governed self-service layer without building it in-house are a strong fit. Platform engineering teams managing Kafka access requests across 20 or more teams will find the Topic as a Service and ownership model design specifically relevant.

It is less well-suited to small teams or startups. Community feedback on Reddit is consistent on this point: the free tier restrictions added in version 1.43 (fewer allowed servers and users) and the requirement for a paid plan to connect to any SSL-enabled or multi-broker cluster make it impractical for staging environment access during early development. [r/apachekafka, “The best Kafka Management tool,” 2025; r/apachekafka, “Free tools to connect to multi-broker/SSL-enabled clusters,” 2022] Other scenarios where alternatives deserve a closer look: teams primarily on Azure Event Hubs who need schema registry integration, organisations requiring a stateless deployment model, or use cases that require distributed tracing.

Conduktor pricing

Conduktor operates a per-seat pricing model for its commercial tiers. A Community tier is available at no cost for teams getting started or running smaller deployments.

Pricing tiers

The Team Edition is priced at approximately $1,200 per seat per year (around $32/user/month). For an organisation with 100 users across three clusters, estimated list price ranges from $80,000 to $150,000 per year. Enterprise pricing is negotiated directly and includes the dedicated Solutions Engineering team, SAML 2.0 SSO, and additional compliance features. Exact Enterprise pricing is not published.

Per-seat pricing means that cost scales with headcount rather than with infrastructure. If your teams are growing faster than your cluster count, this is worth modelling explicitly before committing.

Free tier limitations

The Community tier has become more restrictive over successive releases. Version 1.43 reduced the number of allowed servers and users on the free plan. More significantly, connecting to any cluster configured with SSL/TLS or consisting of more than one broker node requires a paid subscription — which means the free tier cannot connect to most staging or production-equivalent environments. Community practitioners have described this as the point at which the free tier became unviable for startup use. [r/apachekafka, “The best Kafka Management tool,” 2025; r/apachekafka, “Free tools to connect to multi-broker/SSL-enabled clusters,” 2022]

In a public Reddit thread, Conduktor representatives characterised their licensing as “highly affordable” and attributed hesitation to confusion around volume discounts and floating licence options. [r/apachekafka, “Suggestions for UI for AWS managed Kafka?”, 2023] That framing is worth noting, but it does not address the specific free tier restrictions that prevent evaluation against secured clusters.

Free trial

Conduktor offers free sandboxes that allow evaluation without local Docker setup. A Community tier is available for teams that want to self-host without a commercial licence, subject to the tier restrictions noted above.

Conduktor competitors and alternatives

Conduktor occupies a reasonably well-defined segment — enterprise Kafka governance with a proxy component — but it competes across several dimensions with both open-source tools and commercial alternatives. Open-source tools like AKHQ and Kafbat are viable for teams where developer experience is the primary requirement and governance is handled separately. Commercial tools like Lenses and Kpow offer different trade-offs on pricing model, deployment architecture, and feature emphasis.

Best for	Tool	Type	Key functionalities	Deployment and ops	Access control	User interface	Pricing
Multi-team enterprise governance and compliance	Conduktor	Commercial	RBAC, Topic as a Service, Gateway proxy (encryption, masking, virtual topic filtering, DR failover), schema registry, Kafka Connect, ksqlDB	Self-hosted (requires PostgreSQL 13+); Kubernetes/Helm supported; SaaS limited	RBAC; OIDC/LDAP (all tiers); SAML 2.0 (Enterprise only)	React-based Console; NPS 80 at Kafka Summit 2024; widely praised by community	Per-seat (~$1,200/seat/year); Community tier free but restricted (no SSL, single-broker only)
Kafka operations and developer experience at cluster scale	Kpow (Factor House)	Commercial	Topic, consumer group, schema, connector management; advanced RBAC; audit log; WCAG-compliant UI; stable on clusters with up to 200k partitions	Stateless; no external database dependency; Docker, Helm, JAR	Advanced RBAC	Fully WCAG-compliant	Per-cluster pricing
Developer-focused data exploration and stream processing	Lenses	Commercial	Data exploration, SQL on streams, topology view, connector management	Self-hosted and SaaS	RBAC	Web UI	Per-licence (contact for pricing)
Lightweight developer UI for single-team or small-scale use	AKHQ / Kafbat	OSS	Topic, consumer group, schema management; basic ACL management; free SSL and multi-broker access	Self-hosted; low resource overhead; no external dependencies	ACL-based; limited RBAC	Web UI	Free (open-source)
Full Confluent Platform users	Confluent Control Center	Commercial (bundled)	Full Confluent Platform integration; ksqlDB, Kafka Connect, Schema Registry	Bundled with Confluent Platform; requires Confluent deployment	Confluent RBAC	Web UI	Included with Confluent Platform licence
Kafka operations visibility and JMX-based monitoring	AxonOps	Commercial (with OSS tier)	Monitoring, alerting, backup and restore, topic management	Self-hosted; Cassandra backend	Role-based	Web UI	Free tier; commercial tiers available

For a full comparison of Kafka UI and management tools in 2026, see Top Kafka UI tools in 2026: a practical comparison for engineering teams.

Frequently asked questions about Conduktor

How much does Conduktor cost, and is there a free tier?

The Team Edition is approximately $1,200 per seat per year. A free Community tier is available for self-hosted deployments, but it restricts the number of allowed servers and users and does not support connections to SSL-enabled or multi-broker clusters. Enterprise pricing is negotiated directly. For 100 users across three clusters, estimated list price is $80,000-$150,000 per year.

When is Conduktor a better choice than the alternatives?

Conduktor is a strong fit when multi-team Kafka governance, compliance certification (SOC2 Type II), field-level encryption, and self-service topic workflows are the primary requirements — particularly in regulated industries with PCI DSS, HIPAA, or GDPR obligations. The Gateway’s disaster recovery failover and virtual topic filtering capabilities are also relevant for platform teams managing high-availability or bandwidth-sensitive deployments.

When are the alternatives a better choice than Conduktor?

If your team is small, if the free tier’s SSL and multi-broker restrictions prevent you from evaluating the tool against your actual staging environment, if per-cluster pricing fits your scaling model better than per-seat, if you need a stateless deployment without a PostgreSQL dependency, or if Azure Schema Registry integration is required, alternatives are likely worth a closer look.

Does Conduktor support Azure Event Hubs?

Azure Event Hubs is accessible via Kafka protocol compatibility, but Azure’s native Schema Registry is not integrated. Teams using Event Hubs cannot access Conduktor’s schema features against the Azure native registry.

Is Conduktor Desktop still supported?

Conduktor Desktop was retired at end of 2025. Conduktor’s current product is Console, a centrally deployed web platform. The last available version of the desktop application no longer supports modern Kafka versions, rendering it obsolete for teams that have not frozen their infrastructure. Teams migrating from Desktop to Console should expect a change in deployment model — from a personal application to a shared platform — rather than a like-for-like upgrade. Community feedback notes that the Docker-based Console setup is better suited to centralised team deployments than to individual developer machines.

Can I use Conduktor’s free tier with a secured staging cluster?

No. Connecting to any cluster configured with SSL/TLS or consisting of more than one broker node requires a paid subscription. This makes the free tier unsuitable for evaluating Conduktor against a typical staging environment and is the most commonly cited reason developers on Reddit move to open-source alternatives like Kafbat or AKHQ for local and staging use.

How Airbnb uses Apache Kafka in production

Factor House — Sat, 30 May 2026 00:00:00 GMT

Airbnb runs Apache Kafka across at least six production systems simultaneously, from the analytics pipeline that ingests more than 35 billion events per day to the merge queue that serialises thousands of code changes across its engineering monorepos. Few companies have found this many distinct architectural roles for a single piece of infrastructure, which makes Airbnb’s Kafka story worth examining in detail.

The engineering problem Kafka solves at Airbnb is not a single one. It provides the event backbone for real-time personalisation, change data capture, distributed materialised views, and durable write-ahead logging, each with different throughput, ordering, and latency requirements handled through separate cluster and pipeline designs.

Company overview

Airbnb operates a two-sided marketplace connecting hosts and guests across more than 220 countries. At the scale Airbnb operates, the platform generates a continuous stream of user activity events: searches, listing views, booking requests, messages, and availability updates, each of which needs to reach multiple downstream systems in near real time.

Airbnb began building Kafka-backed streaming infrastructure around 2016-2017, when Youssef Francis and Jun He described their streaming logging pipeline at Kafka Summit New York 2017. The early architecture used Kafka as an event bus feeding Airstream, an internal Spark Streaming framework, into HBase and Hive. Over the following years the team expanded Kafka’s role substantially: adding a CDC system (SpinalTap, open-sourced in 2019), migrating real-time personalisation from Spark to Flink, building a Kafka Streams-based merge queue (Evergreen, presented at Kafka Summit London 2022), and adopting Kafka as the write-ahead log in their distributed key-value store (Mussel).

Period	Milestone
2016-2017	Airstream (Spark Streaming on Kafka) deployed for logging event ingestion; Kafka Summit NYC 2017 talk on reliable logging
2019	SpinalTap CDC open-sourced on GitHub; Kafka Summit NYC 2019 talk on streaming ingestion by Hao Wang
~2021	User Signals Platform launched with Apache Flink, replacing Spark Streaming for real-time personalisation
2022	Evergreen merge queue built on Kafka Streams; presented at Kafka Summit London 2022
2023	Riverbed framework in production processing 2.4 billion events/day across 50+ materialised views
2024-2025	Mussel V2 re-architecture: Kafka as WAL sustaining 100,000+ streaming writes/second on 100 TB+ tables

Airbnb’s Kafka use cases

Analytics event ingestion

The foundational use case is also the highest-volume one. Clients (mobile apps, web browsers) and online services publish logging events directly to Kafka. A Spark Streaming job built on Airstream, Airbnb’s internal streaming framework, reads from Kafka continuously, writes events to HBase for deduplication, then dumps them hourly into Hive partitions for downstream ETL and analytics jobs. The Hive-based data ingestion framework processes more than 35 billion Kafka event messages and more than 1,000 tables per day.

Kafka brokers were migrated from on-premises infrastructure to VPC-hosted infrastructure, producing a 4x throughput improvement. To support capacity planning, the team tracks QPS per Kafka partition and adds partitions proactively as event volumes grow.

Real-time personalisation via the User Signals Platform

Every user interaction on the Airbnb platform, a search query, a listing view, a booking, generates an event that flows into the User Signals Platform (USP). More than 100 Apache Flink jobs consume these Kafka events, transform them into structured “User Signals,” write the results to a key-value store, and re-emit signals as Kafka events for downstream consumer jobs handling session engagement and user segmentation. The USP service layer handles 70,000 queries per second.

The decision to use Flink rather than Spark Streaming was deliberate and driven by a specific latency constraint: Spark’s micro-batch model processed events in small batches rather than one at a time, which introduced delays that were incompatible with real-time personalisation requirements. Flink’s event-by-event processing model met those latency requirements.

Change data capture via SpinalTap

SpinalTap is Airbnb’s open-source CDC service, available on GitHub at airbnb/SpinalTap. It detects data mutations in MySQL, DynamoDB, and proprietary in-house storage systems, then propagates them as standardised events to Kafka using Apache Thrift serialisation, which supports both the Ruby and Java consumers that read from those topics.

SpinalTap supplies several downstream systems from a single Kafka stream of mutations. Search indexing: mutations flow from Kafka into Elasticsearch for products like review search and inbox search. Cache invalidation: downstream services subscribe to mutation events and evict or update entries in Memcached and Redis without blocking the request path. Service signalling: dependent services subscribe to data changes in near real time, with the Availability service monitoring Reservation changes as a concrete example.

SpinalTap provides at-least-once delivery with zero data-loss tolerance, per-record ordering (commit-order preservation), and epoch-based split-brain mitigation for high-availability deployments. A continuous validation pipeline runs in both pre-production and production environments.

Distributed materialised views via Riverbed

Some of Airbnb’s highest-traffic features require joining data across multiple distinct databases. Fetching that data at query time produces latency that is incompatible with user-facing response time requirements. Riverbed is Airbnb’s Lambda-like framework that solves this by pre-computing and maintaining distributed materialised views.

The streaming component of Riverbed consumes CDC events via Kafka. Rather than partitioning those events by their source record ID, Riverbed repartitions them in Kafka by the materialised view document ID. This ensures that all CDC events affecting a given materialised view document are routed to the same consumer and processed serially, eliminating concurrent-write race conditions without requiring distributed locking.

Riverbed currently powers 50+ materialised views, processes 2.4 billion events per day, and writes 350 million documents per day.

Write-ahead log in the Mussel key-value store

Mussel is Airbnb’s distributed key-value store for derived data. Rather than writing directly to the backend database, every write is first persisted to Kafka, which acts as the durable write-ahead log. Downstream Replayer and Write Dispatcher components consume from Kafka and apply writes to the backend database in order. In Mussel V1, the Kafka topic used 1,024 partitions aligned with the shard structure, so each shard’s data was processed by a single consumer in commit order.

Mussel V2, re-architected in 2024-2025, pairs a NewSQL backend with Kubernetes-native control and a stateless horizontally-scalable Dispatcher layer. Kafka continues to serve as the common durable log. The system sustains more than 100,000 streaming writes per second and supports tables exceeding 100 terabytes, with p99 read latencies under 25 milliseconds. During the V1-to-V2 migration, Kafka’s role as a shared log allowed both versions to run simultaneously and enabled data replay for validation.

Developer infrastructure: the Evergreen merge queue

Most of Airbnb’s engineering work happens inside large monolithic repositories. With thousands of changes being merged every day, there is a meaningful probability that changes passing automated checks independently will fail when integrated with concurrent changes on the mainline. Evergreen is Airbnb’s merge queue system that guarantees serializability of changes.

At its core, Evergreen uses an actor model built on Kafka Streams. A state machine applies a pure function on events and transforms them into actions executed by workers; workers may in turn produce additional events. The team selected Kafka Streams for its exactly-once processing guarantees (critical for a system where duplicate or missing merges would be costly), its built-in load balancing, and its minimal external dependencies. Janusz Kudelka and Joel Snyder from Airbnb’s Developer Infrastructure team presented the architecture and its production learnings at Kafka Summit London 2022.

Scale & throughput

Metric	Value
Daily events (analytics ingestion pipeline)	>35 billion
Daily events (Riverbed)	2.4 billion
Daily documents written (Riverbed)	350 million
Events per second (User Signals Platform)	>1 million
Flink jobs consuming Kafka (USP)	100+
USP service query rate	70,000 QPS
Mussel streaming write throughput	>100,000 writes/sec
Mussel largest table size	>100 terabytes
Mussel p99 read latency	<25 ms
Kafka partitions in Mussel V1	1,024
Tables ingested daily (data warehouse)	1,000+
Materialised views in Riverbed	50+

‍

Airbnb operates multiple Kafka clusters segmented by function. Youssef Francis and Jun He referenced separate clusters for analytics, change data capture, and inter-service communication as early as their 2017 Kafka Summit talk, a pattern consistent with the distinct pipeline architectures described above.

Airbnb’s Kafka architecture

Producer architecture

Producers span a wide range of application types. Mobile clients and web browsers publish logging events directly. Online services publish both logging events and mutation events (via SpinalTap’s binlog-parsing source layer). SpinalTap’s destination layer buffers outbound events in bounded in-memory queues and uses destination pools with thread-based partitioning to absorb traffic spikes without applying back-pressure upstream.

Serialisation varies by pipeline. SpinalTap uses Apache Thrift for cross-language support across Ruby and Java producers and consumers. The USP and Riverbed pipelines are Java-based.

Consumer architecture

Consumer group design reflects the isolation requirements of each system. The USP runs 100+ independent Flink jobs, each consuming from specific Kafka topics and maintaining its own consumer group state. Hot-standby Task Managers are provisioned to take over Kafka partition assignments immediately on failure, avoiding rebalancing delays.

For idempotency in the USP, the team made a deliberate storage-layer decision: rather than deduplicating events in the stream processor (which adds state complexity under at-least-once delivery), computed signals are written to an append-only key-value store where event timestamps serve as version keys. A later write for the same key with a higher timestamp wins, making the storage layer naturally idempotent.

Offset management in the logging ingestion pipeline is coupled to partition count monitoring. QPS per partition is tracked as the primary metric for determining when to add partitions.

Stream processing

Two stream processing systems run on top of Kafka.

Apache Flink powers the User Signals Platform. The Flink-based architecture was chosen specifically because Spark Streaming’s micro-batch model introduced event delays that made it unsuitable for real-time personalisation. To lower the barrier to authoring Flink jobs, the USP team built a config-driven layer: engineers define signal transformations declaratively, and a setup script generates the corresponding Flink job configuration, batch backfill files, and monitoring alerts automatically.

Kafka Streams powers Evergreen. The stateful actor model is implemented as a Kafka Streams topology, where the state machine and its transitions are expressed as stream-processing operations. Exactly-once processing semantics are a hard requirement: the merge queue must not execute a merge twice or skip one.

Kafka Connect ecosystem

SpinalTap functions as a custom producer framework rather than a standard Kafka Connect source connector: it parses MySQL binary logs and DynamoDB streams directly and writes standardised Thrift-encoded mutation events to Kafka. The downstream consumers (Elasticsearch indexers, cache invalidators, Hive exporters) are custom applications rather than Connect sink connectors. No use of standard Kafka Connect connectors is referenced in Airbnb’s public engineering writing.

Special techniques & engineering innovations

Partition alignment in Mussel (ordering without coordination)

Mussel V1 used 1,024 Kafka partitions aligned with the 1,024-shard structure of the backend store. Because each partition maps to exactly one shard, all writes to a shard flow through a single Kafka partition and are applied by a single consumer in order. This gives per-shard write ordering without needing a separate coordination layer or distributed lock.

Partitioning by document ID in Riverbed (race condition prevention)

CDC events in Riverbed are repartitioned in Kafka by the materialised-view document ID rather than the source database record ID. Because a materialised view document can be affected by mutations across multiple upstream tables, naive partitioning by source record would route concurrent updates to the same document to different consumers, creating write races. Routing by document ID ensures serial processing per document, which eliminates the race without requiring distributed locking or explicit conflict resolution.

Balanced Kafka reader (decoupling Spark parallelism from partition count)

In Airbnb’s early Spark Streaming logging pipeline, the degree of Spark task parallelism was directly tied to the number of Kafka partitions. Adding throughput capacity required repartitioning Kafka topics, which is an operationally disruptive change. Hao Wang’s team built a balanced Kafka reader that allows Spark parallelism to be configured independently of partition count, enabling flexible capacity increases without forced repartitioning.

Kafka as a migration backbone in Mussel V2

During the multi-version migration from Mussel V1 to V2, Kafka’s WAL role made it possible for both versions to operate simultaneously against the same event stream. The new Replayer and Write Dispatcher components could consume from the same Kafka log as V1, allowing engineers to validate V2 correctness by replaying production events against the new backend before cutting over. This reduced the risk of a hard migration cutover.

Operating Kafka at scale

Deployment model: Kafka is self-managed at Airbnb. The migration of broker infrastructure to VPC provided a 4x throughput improvement. No use of Amazon MSK or Confluent Cloud appears in any primary engineering sources.

Monitoring: Partition-level QPS is the primary capacity signal for the analytics ingestion pipeline. The USP generates per-job monitoring alerts automatically through the config-driven job setup. Mussel V2 provides namespace-level quotas and dashboards.

Upgrade and migration strategy: The Mussel V1-to-V2 migration used Kafka as the shared log to run both versions in parallel, supporting incremental validation and rollback. SpinalTap’s continuous validation pipeline runs in pre-production alongside production, allowing schema and routing changes to be tested against real mutation streams before deployment.

Developer experience: SpinalTap’s continuous validation pipeline, the USP’s config-driven job generator, and Evergreen’s Kafka Streams actor model all reflect a deliberate effort to abstract Kafka’s operational complexity away from application engineers. The goal in each case is to expose a declarative or event-driven interface while the platform team owns the Kafka infrastructure underneath.

Challenges & how they solved them

Latency incompatibility between Spark Streaming and real-time personalisation

Problem: The User Signals Platform was initially built on Spark Streaming, which processes events in micro-batches rather than one at a time. This introduced processing delays that could not be reduced below a threshold determined by the micro-batch interval, making the system unsuitable for real-time personalisation features.

Root cause: A fundamental processing model constraint in Spark Streaming, not a configuration or scaling issue.

Solution: Migrated to Apache Flink, which processes events one-by-one. The USP team also built a config-driven layer to absorb the additional complexity Flink introduces for new engineers.

Outcome: The USP now processes more than 1 million events per second across 100+ Flink jobs, serving the personalisation service layer at 70,000 QPS.

Concurrent-write race conditions in Riverbed

Problem: Materialised view documents in Riverbed are computed from data spread across multiple upstream tables. When mutations in different upstream tables triggered concurrent updates to the same materialised view document, writes from two consumers could interleave and produce a corrupted result.

Root cause: The partition key used for Kafka routing did not correspond to the unit of write exclusion (the materialised view document ID).

Solution: Repartitioned CDC events in Kafka by materialised view document ID. All updates to a given document are now routed to a single consumer and processed serially.

Outcome: Race conditions eliminated without distributed locking or explicit conflict detection.

Kafka partition count coupling in Spark Streaming

Problem: Airbnb’s Spark Streaming logging pipeline tied Spark task parallelism directly to Kafka partition count. Scaling throughput meant repartitioning Kafka topics, an operationally costly change that required coordinating producer and consumer restarts.

Root cause: The default Spark Kafka integration assigned one Spark task per Kafka partition.

Solution: Developed a custom balanced Kafka reader that maps Kafka partitions to Spark tasks independently, allowing Spark parallelism to be increased without changing partition count.

Outcome: The logging pipeline could grow event throughput without mandatory Kafka repartitioning operations.

One-record-at-a-time throughput in Evergreen

Problem: Evergreen’s Kafka Streams state machine processed one record at a time, which became a throughput bottleneck as the volume of daily merge requests grew. Replaying records for debugging purposes was also difficult.

Root cause: Kafka Streams’ processing model, combined with the actor pattern’s sequential state transitions.

Status: Identified as a known production challenge. Janusz Kudelka and Joel Snyder described it as a learning from operating Evergreen at Kafka Summit London 2022. No specific resolution is described in available public sources.

Full tech stack

Category	Tools	Notes
Message broker	Apache Kafka (self-managed, multiple clusters)	Separate clusters for analytics, CDC, and inter-service messaging
Stream processing	Apache Flink	User Signals Platform; 100+ jobs; config-driven job generation
Stream processing	Kafka Streams	Evergreen merge queue; exactly-once processing
Stream processing (historical)	Apache Spark Streaming / Airstream	Logging event ingestion pipeline; replaced by Flink for USP
CDC system	SpinalTap (open-source)	MySQL/DynamoDB mutation capture to Kafka; open-sourced 2019
Serialisation	Apache Thrift	SpinalTap event format; cross-language Ruby and Java support
Storage (operational)	Apache HBase	Event deduplication in logging ingestion pipeline
Storage (analytical)	Apache Hive	Data warehouse; hourly partition dumps from Kafka pipeline
Query engine	Apache Presto	Interactive querying on Hive tables
Orchestration	Apache Airflow	Batch job orchestration; Mussel V2 bulk ingestion via Airflow and S3
KV store	Mussel (internal, V1 and V2)	Derived data store; Kafka-backed write-ahead log
KV store backend (V2)	NewSQL (unspecified)	Mussel V2 backend; Kubernetes-native, horizontally scalable Dispatcher
Search indexing	Elasticsearch	CDC consumer via SpinalTap (review search, inbox search)
Caching	Memcached, Redis	Cache invalidation via SpinalTap CDC events
CDC sources	MySQL, DynamoDB	Primary data stores feeding SpinalTap
Business logic layer	GraphQL	Riverbed: aggregation logic for materialised view computation
Container orchestration	Kubernetes	Mussel V2 deployment model
Object storage	Amazon S3	Mussel V2 bulk ingestion staging

Key contributors

Name	Contribution
Hao Wang	Led Spark Streaming logging ingestion; presenter at Kafka Summit NYC 2019
Youssef Francis, Jun He	Presenters at Kafka Summit NYC 2017 on reliable logging with Kafka
Jad Abi-Samra, Litao Deng, Zuofei Wang	Authors of the SpinalTap engineering blog post
Kidai Kwon	Author of the User Signals Platform blog post
Amre Shakim	Lead author of the Riverbed blog posts; team also included Krish Chainani, Victor Chen, Yanxi Chen, Xiangmin Liang, Anton Panasenko, Sonia Stan, Peggy Zheng
Janusz Kudelka, Joel Snyder	Developer Infrastructure engineers; designed Evergreen and presented at Kafka Summit London 2022
Cong Zhu, Pala Muthiah, Jinyang Li, Ronnie Zhu, Gabe Lyons, Xu Zhang	Contributors to Spark Streaming logging ingestion pipeline

Key takeaways for your own Kafka implementation

Use partition key selection as an ordering contract, not just a routing decision. Mussel aligns partition count to shard count to guarantee per-shard ordering without coordination. Riverbed partitions by document ID rather than source record ID to make CDC processing race-free. In both cases, the partition key was chosen to match the unit of consistency required by the consuming system, not just to distribute load evenly.
Micro-batch stream processing is incompatible with some latency targets. Airbnb’s USP found that Spark Streaming’s micro-batch model had a latency floor that could not be reduced to meet real-time personalisation requirements. If your use case has a latency constraint below what a micro-batch interval can provide, Flink’s event-by-event processing model is worth the additional operational complexity.
Kafka as a write-ahead log simplifies distributed migrations. Mussel V2’s migration succeeded in part because Kafka already held a complete ordered log of writes. The new backend could consume the same stream independently, enabling parallel validation before cutover. If you are migrating a stateful system, having a durable Kafka WAL in place before the migration substantially reduces cutover risk.
Decouple stream processor parallelism from Kafka partition count. Airbnb’s balanced Kafka reader broke the coupling between Spark task count and partition count, allowing throughput to be increased without repartitioning. This is worth considering in any system where repartitioning is operationally costly, particularly where partition count affects ordering guarantees downstream.
Abstract Kafka complexity with a declarative layer if your engineering organisation is large. The USP config-driven Flink job generator, SpinalTap’s structured pipeline model, and Evergreen’s actor abstraction all serve the same goal: letting application engineers define what they need from a streaming pipeline without needing to understand Kafka consumer groups, offset management, or Flink topology design. At Airbnb’s scale, this separation of concerns is what allows Kafka to serve six distinct systems without six separate platform teams.

Sources and further reading

Hao Wang, “Scaling Spark Streaming for Logging Event Ingestion,” Airbnb Engineering Blog: https://medium.com/airbnb-engineering/scaling-spark-streaming-for-logging-event-ingestion-4a03141d135d
Hao Wang, Kafka Summit NYC 2019 talk: https://videos.confluent.io/watch/b94DNHsNfzTt8apLDmDLHP
Youssef Francis, Jun He, Kafka Summit NYC 2017: https://www.confluent.io/kafka-summit-nyc17/every-message-counts-kafka-foundation-highly-reliable-logging-airbnb/
Kidai Kwon, “Building a User Signals Platform at Airbnb,” Airbnb Engineering Blog: https://medium.com/airbnb-engineering/building-a-user-signals-platform-at-airbnb-b236078ec82b
Jad Abi-Samra, Litao Deng, Zuofei Wang, “Capturing Data Evolution in a Service-Oriented Architecture,” Airbnb Engineering Blog: https://medium.com/airbnb-engineering/capturing-data-evolution-in-a-service-oriented-architecture-72f7c643ee6f
SpinalTap on GitHub: https://github.com/airbnb/SpinalTap
Amre Shakim, “Riverbed: Optimizing Data Access at Airbnb’s Scale,” Airbnb Engineering Blog: https://medium.com/airbnb-engineering/riverbed-optimizing-data-access-at-airbnbs-scale-c37ecf6456d9
Xiangmin Liang, Sivakumar Bhavanari, Amre Shakim, “Riverbed data hydration — Part 1,” Airbnb Engineering Blog: https://medium.com/airbnb-engineering/riverbed-data-hydration-part-1-e7011d62d946
InfoQ, “Distributed Materialized Views: How Airbnb’s Riverbed Processes 2.4 Billion Daily Events”: https://www.infoq.com/news/2023/10/airbnb-riverbed-introduction/
Airbnb Engineering Blog, “Mussel — Airbnb’s Key-Value Store for Derived Data”: https://airbnb.tech/data/mussel-airbnbs-key-value-store-for-derived-data/
InfoQ, “Airbnb’s Mussel V2: Next-Gen Key Value Storage to Unify Streaming and Bulk Ingestion”: https://www.infoq.com/news/2025/10/airbnb-nextgen-kv-storage-mussel/
Janusz Kudelka, Joel Snyder, Kafka Summit London 2022: https://www.confluent.io/events/kafka-summit-london-2022/evergreen-building-airbnbs-merge-queue-with-kafka-streams/

Try Kpow with your own Kafka cluster: If you are running Kafka in production and want visibility into consumer lag, partition throughput, and topic health across multiple clusters, Kpow offers a free 30-day trial. You can connect it to any Kafka cluster in minutes and deploy via Docker, Helm, or JAR.

How Bytedance uses Apache Kafka in production

Factor House — Sat, 30 May 2026 00:00:00 GMT

ByteDance ran Apache Kafka at a scale that eventually made rebuilding it the more practical option. At peak, its streaming data layer handles tens of TB/s of consume throughput across applications that span short-video recommendation, real-time ML training, and cross-team data integration. Rather than continue tuning a standard Kafka deployment, ByteDance built ByteMQ (BMQ): a Kafka-compatible, cloud-native replacement that separates storage from compute and now runs 99.76% of what were formerly Kafka workloads, at roughly 70% lower resource cost. Understanding how the company got there, and what the architecture looks like today, is instructive for any team planning a streaming platform at similar scale.

Company overview

ByteDance operates the portfolio of applications behind TikTok, Douyin, Toutiao, and several other consumer products. The company’s platforms generate continuous high-volume user-interaction data: every view, like, share, and comment on TikTok becomes a signal that feeds recommendation models, content ranking, and advertising systems in near real time. That combination of volume and latency sensitivity pushed ByteDance toward event streaming infrastructure early in the company’s growth.

Kafka was adopted as the primary message bus for event and log collection. It served as the backbone for stream processing pipelines, online model training, and data integration across ByteDance’s business units. Over time, the throughput requirements outgrew what a standard Kafka deployment could serve cost-effectively, and the engineering team began work on a replacement. The timeline below covers the key milestones from the documented record.

Key Kafka milestones:

Pre-2022: Kafka operating as ByteDance’s primary event and log collection bus
September 2022: Monolith paper published, documenting Kafka’s role in TikTok’s real-time recommendation training pipeline
2023: StreamOps published at VLDB, describing runtime management of tens of thousands of Flink jobs against Kafka-compatible queues
November 2024: ByteMQ paper published at ACM SoCC; 99.76% Kafka-to-BMQ migration reported complete; 70% resource cost reduction achieved
February 2026: StreamShield paper published; Flink cluster now sustains 70,000+ concurrent streaming jobs across 11 million+ resource slots

ByteDance’s Kafka use cases

Event and log collection

Kafka served as ByteDance’s central data hub for collecting events and user-activity logs across its applications, covering online model training inputs, stream data processing, and real-time analytics. This is the foundational use case that drove initial adoption and still underpins the workloads now running on ByteMQ.

Real-time recommendation training

TikTok’s recommendation system is built on Monolith, ByteDance’s distributed real-time ML framework. Kafka plays a structural role in the training pipeline: one queue carries raw user-action events (views, likes, shares), and a second carries feature data. A Flink streaming job reads from both queues simultaneously, joining each user action with its corresponding features to produce labelled training examples. Those examples flow directly into Monolith’s training parameter server, which updates model weights continuously throughout the day. This architecture allows the recommendation model to adapt to shifting user behaviour without waiting for periodic batch retraining cycles.

Stream processing

ByteDance runs one of the largest documented Apache Flink deployments in production. As of early 2026, the cluster sustains over 70,000 concurrent streaming jobs backed by more than 11 million resource slots. The message queues underlying those jobs are now ByteMQ, but the workload pattern is a direct continuation of the Kafka-based streaming infrastructure that preceded it.

Data integration

BitSail, ByteDance’s open-source data integration engine, provides both Kafka source and Kafka sink connectors, enabling data synchronisation between Kafka topics and downstream stores. The system processes hundreds of trillions of records per day across ByteDance’s data estate, supporting batch, streaming, and incremental sync patterns.

Scale & throughput

Peak consume rate: Tens of TB/s (ByteMQ paper, ACM SoCC 2024)
Peak produce rate: Approximately one-fifth of the consume rate (ByteMQ paper, ACM SoCC 2024)
Kafka clusters migrated to BMQ: 99.76% (ByteMQ paper, ACM SoCC 2024)
Resource cost reduction post-migration: ~70% (ByteMQ paper, ACM SoCC 2024)
Concurrent Flink streaming jobs: 70,000+ (StreamShield paper, arXiv Feb 2026)
Flink resource slots managed: 11 million+ (StreamShield paper, arXiv Feb 2026)
Data synchronised per day (BitSail): Hundreds of trillions of records (BitSail GitHub repo)

The consume-to-produce ratio of roughly five-to-one reflects ByteDance’s fan-out consumption patterns, where multiple downstream consumers (recommendation, analytics, logging, model training) each read from the same upstream event streams.

ByteDance’s Kafka architecture

From Kafka to ByteMQ

ByteDance’s streaming architecture is best understood in two phases: Kafka as adopted, and ByteMQ as the current system. The transition happened because standard Kafka couples storage and compute: adding storage capacity requires adding brokers, and adding brokers for throughput also adds storage. At ByteDance’s scale, this relationship became expensive. The company’s response was to build ByteMQ, a Kafka-compatible system that breaks that coupling.

ByteMQ is Kafka API-compatible by design. Producers and consumers continue to use standard Kafka SDKs, and the Pub/Sub Engine within BMQ presents the familiar topic and consumer-group interface. This compatibility requirement was a deliberate engineering constraint, chosen to de-risk the migration. Switching infrastructure at the scale of tens of TB/s while keeping application code unchanged is a significant operational achievement.

ByteMQ architecture

Three architectural decisions define BMQ:

Storage-compute separation. Rather than writing message data to local broker disks, BMQ persists all data to ByteDance’s internal Federated Distributed File System (DFS). Brokers handle routing and API serving, while the DFS handles durability. This means brokers can be scaled independently of storage, and vice versa. In ByteDance’s own comparative benchmarks, a scale-out operation on clusters provisioned at approximately 1,000 TB of storage and 200 CPU cores required adding roughly 50 new BMQ brokers versus approximately 5 new brokers for an equivalent Kafka cluster: a reflection of how much more efficiently BMQ can redistribute load when brokers are not tied to their local disk state.

Adaptive resource scheduling. BMQ redistributes workloads across brokers and across multiple availability zones dynamically, balancing throughput and isolating failures without manual intervention. This addresses a common operational burden in large Kafka deployments, where partition reassignment is a heavyweight and often risky operation.

Historical data restructuring. BMQ restructures older streaming data into formats that are efficient for offline batch consumption. This allows both real-time stream processors and batch analytics jobs to consume from the same underlying data layer, without requiring separate pipelines or dedicated archival infrastructure.

Recommendation pipeline (Monolith)

The Monolith recommendation training pipeline uses Kafka as its inter-component transport with a specific two-queue architecture:

Action queue: carries user-action events from TikTok (views, likes, shares, follows)
Feature queue: carries pre-computed feature data from the feature store
A Flink online joiner reads from both queues simultaneously and joins each action event with its corresponding features to produce labelled training examples
Training examples are written to Monolith’s training parameter server (training-PS), which updates model weights in real time
The inference parameter server (inference-PS) serves live traffic and syncs periodically from the training-PS

This design keeps the training loop continuously fed from live user-interaction data, rather than waiting for a batch export cycle.

Producer architecture

ByteDance’s documented architecture does not describe specific Kafka producer configuration parameters (acks, batching, compression) in public sources. The Monolith pipeline uses Kafka producers to write action events and feature data into their respective queues as part of the online training loop.

Consumer architecture

Consumer groups in BMQ follow the same semantics as Kafka consumer groups: consumers are organised into groups, with each partition assigned to one consumer in the group at a time. The Kafka API compatibility layer in BMQ’s Pub/Sub Engine means existing consumer group code runs unchanged against BMQ.

Stream processing

Apache Flink is ByteDance’s primary stream processing engine. The Flink cluster is managed by two internal systems: StreamOps handles runtime lifecycle management (auto-scaling, straggler detection, automated recovery), and StreamShield provides resiliency mechanisms at the engine and cluster levels. Both systems interact with BMQ as the message source.

Special techniques & engineering innovations

Kafka-compatible replacement at scale (ByteMQ)

The ByteMQ migration is itself a notable engineering decision. Rather than adopting a third-party Kafka-compatible system or moving to a different messaging model, ByteDance built and operated its own replacement. The design choice to maintain full Kafka SDK compatibility meant that migration could proceed incrementally across 99.76% of clusters without requiring application changes. The 70% resource cost reduction suggests the storage-compute coupling in standard Kafka was contributing significantly to infrastructure spend at ByteDance’s scale.

Online continuous training via Kafka (Monolith)

The two-queue Kafka architecture in Monolith enables a training loop that runs continuously from live user interactions. The Flink join step is critical: by joining action events with features in a streaming job rather than storing features alongside events, the system keeps feature data up to date without having to re-embed stale feature values into the event stream at write time. This produces training examples that reflect the feature state at the time the action occurred rather than a fixed snapshot.

Collisionless embedding table

Monolith uses a hash-map-based embedding table rather than a fixed-size embedding table. Embeddings expire when unused (avoiding accumulation of long-tail entities), and frequency filtering prevents low-signal features from consuming memory. This keeps the embedding memory footprint bounded without sacrificing model quality from hash collisions, which are common in the fixed-size approach.

Region checkpointing (StreamShield)

Standard Flink checkpointing operates at the job level: a checkpoint failure triggers recovery for the entire job. StreamShield introduces region-level checkpointing, which narrows the recovery scope to the portion of the job that actually failed. This change improved checkpoint success rates in ByteDance’s production Flink cluster from 53.9% to 93.5%.

Adaptive shuffle (StreamShield)

StreamShield’s Adaptive Shuffle enables dynamic, load-aware data redistribution across Flink task managers. When a task manager becomes a bottleneck, the shuffle layer redistributes partitions to underutilised nodes without a full job restart. This reduces the frequency of hot-spot-induced failures under uneven workloads.

Operating Kafka at scale

StreamOps: runtime lifecycle management: ByteDance built StreamOps, published at VLDB 2023, as a cloud-native control plane for the Flink cluster. StreamOps runs three control policies concurrently across all streaming jobs: an auto-scaler that adjusts task manager allocation in response to throughput changes, a straggler detector that identifies slow-running tasks within jobs, and a job doctor that diagnoses and remediates common failure patterns automatically. StreamOps manages tens of thousands of concurrent jobs, many of which run continuously for days or longer against BMQ queues. Operating at that concurrency level without a centralised control plane would require proportionally more manual intervention per incident.

StreamShield: production resiliency: StreamShield, published February 2026, extends StreamOps with a resiliency layer built around four mechanisms: runtime optimisation (adaptive shuffle, autoscaling), fine-grained fault tolerance (region checkpointing, single-task recovery), hybrid replication (combining passive and active strategies for cluster-level fault tolerance), and a testing and release pipeline that includes chaos testing, micro-benchmarking, macro-benchmarking, and online probe tasks before each production deployment. As of the paper’s publication, StreamShield sustains over 70,000 concurrent streaming jobs and manages more than 11 million resource slots, with thousands of recovery events per day handled automatically.

Job startup overhead: StreamShield reduced job startup overhead from approximately 500 seconds to approximately 200 seconds at scale, measured in test configurations with thousands of task managers. Faster restarts reduce the window during which a failed job is not processing data, which matters for latency-sensitive pipelines.

Data integration with BitSail: ByteDance open-sourced BitSail, a distributed data integration engine that supports Kafka as both a source and a sink. It is deployed across cloud-native and on-premises environments within ByteDance, and the repository documents support for hundreds of trillions of daily records. For teams looking to evaluate it, the project is available at the BitSail GitHub repository.

Challenges & how they solved them

Kafka could not scale cost-effectively at ByteDance’s volume

Storage and compute are coupled in standard Kafka; adding storage requires adding brokers, and vice versa. At ByteDance’s volume, this relationship meant resource costs scaled proportionally with throughput in a way that became unsustainable. ByteDance built ByteMQ with storage-compute separation, using its internal Federated DFS as the storage layer, and maintained full Kafka API compatibility to allow migration without application changes. The migration reached 99.76% of Kafka clusters with approximately 70% reduction in resource costs.

Flink job failures at tens of thousands of concurrent jobs required extensive manual intervention

Job-level recovery scope meant any failure required restarting the full job, and without a centralised control plane, intervention at that concurrency level was proportionally costly. ByteDance built StreamOps for automated lifecycle management and StreamShield for region-level checkpointing and adaptive shuffle. Checkpoint success rate improved from 53.9% to 93.5%, job startup overhead fell from approximately 500 seconds to approximately 200 seconds, and thousands of daily recovery events are now handled automatically.

Recommendation model freshness lagged user behaviour

Batch retraining cycles introduced hours of delay between a user action and the corresponding model weight update. Monolith’s two-queue Kafka architecture feeds a Flink online joining step that produces training examples directly from live event streams, allowing model weights to update continuously throughout the day.

Fixed-size embedding tables caused hash collisions and degraded recommendation quality

Multiple distinct feature entities mapped to the same embedding slot when the fixed-size table was full. Monolith’s collisionless hash-map embedding table assigns each entity its own slot, with expirable embeddings and frequency filtering to keep the memory footprint bounded. This eliminates hash collisions without trading off model quality for sparse features.

Full tech stack

CategoryToolsNotesMessage broker (original)Apache KafkaEvent and log collection; inter-service messaging; ML pipeline transportMessage broker (current)ByteMQ (BMQ) — internalKafka API-compatible; storage-compute separated; backed by ByteDance’s Federated DFSPersistent storage (BMQ)ByteDance Federated DFS — internalDistributed file system backing BMQ’s storage layerStream processingApache Flink70,000+ concurrent jobs; real-time feature joining, analytics, online model trainingOnline ML trainingMonolith — internal (open source)Flink-based online joiner reads two Kafka queues to produce training examples for continuous recommendation model trainingData integrationBitSail — open sourceKafka source and sink; hundreds of trillions of records/dayStreaming runtime managementStreamOps — internalAuto-scaler, straggler detector, job doctor for Flink jobs at scaleStreaming resiliencyStreamShield — internalRegion checkpointing, adaptive shuffle, hybrid replication, chaos-tested release pipeline

Key contributors

Yancan Mao: Lead author on both ByteMQ (ACM SoCC 2024) and StreamOps (VLDB 2023). ByteMQ paper, StreamOps paper
Zhanghao Chen: Co-author, StreamOps. StreamOps paper
Ruohang Yin, Liyuan Lei, Peng Ye, Shengfu Zou, Shizheng Tang, Yunzhe Guo, Ye Yuan, Xiaochen Yu, Bo Wan, Yunfei Gong, Changli Gao, Guanghui Zhang, Jian Shen, Rui Shi: Co-authors, ByteMQ (ByteDance Inc.). ByteMQ paper
Yong Fang, Yuxing Han, Meng Wang, Yifan Zhang, Yue Ma, Chi Zhang: Authors, StreamShield. StreamShield paper

Key takeaways for your own Kafka implementation

Storage-compute coupling becomes expensive at high throughput. ByteDance’s decision to build a Kafka-compatible replacement rather than continue scaling standard Kafka was driven by the proportional growth in resource cost. If your Kafka storage and compute are scaling together but you primarily need one or the other, evaluating storage-offload options (tiered storage, object storage backends) earlier may avoid a more significant architectural change later.
Kafka API compatibility is a practical migration constraint. ByteMQ’s decision to maintain full Kafka SDK compatibility allowed ByteDance to migrate 99.76% of clusters without touching application code. If you are designing internal messaging infrastructure, preserving the Kafka protocol interface keeps your migration options open.
Real-time training pipelines benefit from separated queues with a streaming joiner. Monolith’s two-queue approach keeps raw events and feature data on separate Kafka topics, joined in real time by a Flink job. This pattern avoids embedding feature state into the event at write time, which would require re-ingesting events whenever features change.
Job-level recovery scope does not work at tens of thousands of concurrent jobs. ByteDance’s region checkpointing approach in StreamShield reduced the blast radius of individual failures and improved checkpoint success rates from 53.9% to 93.5%. If you are scaling a Flink deployment significantly, the granularity of your checkpointing and recovery scope is worth reviewing before it becomes a bottleneck.
A centralised control plane for streaming jobs reduces per-incident manual work. StreamOps manages auto-scaling, straggler detection, and automated recovery across ByteDance’s Flink cluster. At smaller concurrency levels, per-job monitoring is manageable; at tens of thousands of jobs, automation at the control-plane level is necessary.

Sources & further reading

Yancan Mao et al. (ByteDance) — ByteMQ: A Cloud-native Streaming Data Layer in ByteDance — ACM SoCC 2024: https://dl.acm.org/doi/10.1145/3698038.3698536
Yancan Mao, Zhanghao Chen et al. (ByteDance) — StreamOps: Cloud-Native Runtime Management for Streaming Services in ByteDance — VLDB 2023: https://dl.acm.org/doi/abs/10.14778/3611540.3611543
Yong Fang et al. (ByteDance) — StreamShield: A Production-Proven Resiliency Solution for Apache Flink at ByteDance — arXiv, February 2026: https://arxiv.org/abs/2602.03189
ByteDance AI Lab — Monolith: Real Time Recommendation System With Collisionless Embedding Table — arXiv, September 2022: https://arxiv.org/abs/2209.07663
ByteDance — BitSail: Distributed high-performance data integration engine — GitHub: https://github.com/bytedance/bitsail
Apache Software Foundation — Powered By Apache Kafka — ByteDance entry: https://kafka.apache.org/powered-by/

If you want visibility into the consumer groups, lag metrics, and topic throughput in your own Kafka cluster, give Kpow a try with a free 30-day trial. It connects to any Kafka cluster in minutes and deploys via Docker, Helm, or JAR.

‍

How Cloudflare uses Apache Kafka in production

Factor House — Sat, 30 May 2026 00:00:00 GMT

Cloudflare has been running Apache Kafka in production since 2014. By July 2022, the company had processed over one trillion messages on its primary inter-service message bus alone, across 14 distinct Kafka clusters and approximately 330 nodes. At peak, the HTTP analytics pipeline alone was ingesting 100 Gbps and 7.5 million requests per second through Kafka. Those numbers reflect more than raw traffic volume: they reflect how deeply Kafka is woven into Cloudflare’s control plane, analytics stack, logging infrastructure, and DNS pipeline.

Company overview

Cloudflare operates one of the largest networks in the world, providing content delivery, DDoS protection, DNS resolution, zero-trust security, and developer platform services to millions of customers. Its network spans hundreds of data centers globally and handles a significant share of internet traffic.

Kafka was first adopted in 2014, initially to power analytics, DDoS mitigation, logging, and metrics pipelines. As the company scaled and its microservice footprint grew, Kafka took on a broader role as the backbone for inter-service communication across the control plane.

Date	Event
2014	Kafka first adopted for analytics, DDoS mitigation, logging, and metrics
2018-03	Zstandard (zstd) compression adopted on the HTTP requests topic; HTTP analytics pipeline migrated from PostgreSQL to ClickHouse; analytics cluster at 106 brokers
2022-05	DNS per-record build pipeline launched; handling 250 DNS record changes per second, a 25x increase from initial deployment
2022-07	1 trillion inter-service messages milestone published; 14 clusters, approximately 330 nodes
2023-01	Intelligent offset-based consumer restart mechanism published
2023-03	Matt Boyle and Andrea Medda present “Tales of Kafka @Cloudflare” at QCon London
2024-01	Logging pipeline overview published; approximately 1 million log lines per second; OpenTelemetry Logs migration planned

Cloudflare’s Kafka use cases

Kafka at Cloudflare serves several distinct workloads, each with its own cluster or pipeline configuration.

Inter-service messaging via the Messagebus cluster

The largest and most visible use is inter-service communication across the control plane. Cloudflare’s Application Services team built a general-purpose cluster called “Messagebus” to decouple microservices by propagating resource lifecycle events: the creation, modification, or deletion of any resource is published as a Protobuf-encoded message and consumed by any interested downstream service.

The alert notification system (ANS) is one example built on this pattern. Teams produce to an alert topic with a YAML configuration and a single client import; the system handles delivery to Slack, Google Chat, webhooks, PagerDuty, and email automatically. Communication preferences management across multiple systems also runs on the same bus.

HTTP and DNS analytics

From 2014 onwards, Kafka has been the ingestion layer for HTTP request logs and DNS query logs from Cloudflare’s edge. In 2018, the HTTP analytics pipeline was processing an average of 6 million requests per second (peaking at 8 million), with the Kafka topic receiving up to 100 Gbps ingress and 7.5 million messages per second at peak.

Messages in this pipeline were encoded in Cap’n Proto format. Downstream consumers initially wrote aggregated results to PostgreSQL; the pipeline was later migrated to ClickHouse, where 106 Go consumers extract more than 100 fields per message for ingestion.

DDoS mitigation

Traffic telemetry from the edge is distributed to DDoS detection consumers via Kafka, giving the mitigation system a real-time feed of traffic patterns without tight coupling to the log collection infrastructure.

Internal logging pipeline

Service logs across Cloudflare’s infrastructure flow through a multi-stage pipeline into Kafka before reaching consumers. As of 2024, this pipeline handles approximately 1 million log lines per second. Kafka buffers logs at two core data centers (“log-a” and “log-b”), providing decoupling, fault tolerance, and up to eight hours of consumer outage tolerance before data loss risk arises.

DNS zone builds

DNS record changes submitted via the API trigger PostgreSQL events that are converted into Kafka messages consumed by the Zone Builder. The pipeline supports two consumer types: a full zone build scheduler and a per-record build scheduler. As of 2022, it handles an average of 250 DNS record changes per second, representing 25x growth from initial deployment.

Scale and throughput

Metric	Figure	Pipeline / date
Cumulative inter-service messages	Over 1 trillion	Messagebus cluster, ~8 years to July 2022
Total clusters	14	Across multiple data centers, July 2022
Total broker nodes	~330	All clusters, July 2022
HTTP requests topic peak ingress	100 Gbps / 7.5M req/sec	Analytics cluster, 2018
HTTP analytics cluster brokers	106 (replication factor: 3)	Analytics cluster, 2018
Log lines per second (logging pipeline)	~1 million	Logging cluster, 2024
DNS record changes per second	250 (25x growth from launch)	DNS pipeline, 2022
Minimum replication factor	3	All clusters

Cloudflare’s Kafka architecture

Cluster topology

Cloudflare runs 14 Kafka clusters across multiple data centers, deployed on bare metal alongside Kubernetes and databases in its control plane. The clusters are purpose-built: the general-purpose Messagebus cluster handles inter-service communication, while separate clusters serve analytics, logging, DNS, and other high-throughput workloads. Infrastructure is managed with Salt under a GitOps model.

Message format and topic design

The Messagebus cluster standardised on Protocol Buffers (Protobuf) as the message serialization format, chosen over JSON and Apache Avro for its strict typing, forward and backward compatibility guarantees, and multi-language code generation. The team enforces a single Protobuf message type per topic to prevent format incompatibility across consumers.

Schema governance runs through an internal “Messagebus Schema” service that maintains a central repository of all Protobuf schemas, maps team ownership to topics, and uses prototool to detect and flag breaking schema changes before they reach production. Teams are notified via chat integration when schema changes are proposed.

Messagebus-Client library

The Application Services team built an internal Go library called Messagebus-Client that wraps the Shopify Sarama Kafka client. It provides opinionated configuration defaults, automatic mTLS certificate rotation, and built-in Prometheus metrics exposure for both producers and consumers. Default producer metrics include message delivery success and failure rates; default consumer metrics include consumption success and error rates. This library is the standard entry point for Cloudflare’s internal teams producing and consuming on the Messagebus cluster.

Connector Framework

For workloads that require moving data from a source system into Kafka, or from Kafka into a sink, Cloudflare built a Connector Framework on top of Kafka connectors. New connector services are generated from Cookiecutter templates and configured entirely via environment variables: readers, writers, and transformations are declared without writing custom integration code. Each generated connector includes built-in metrics and alerting. The framework supports Kafka and Cloudflare’s internal Quicksilver key-value store as both source and sink targets.

Logging pipeline architecture

The internal logging pipeline follows a defined path: services emit logs via stdout/stderr, which are captured by systemd-journald and forwarded through syslog-ng. syslog-ng adds metadata (hostname, data center) and forwards logs to two core data centers. At those core data centers, logs are buffered in Kafka before downstream consumers pull from them.

Kafka partitioning in this pipeline uses a composite key of host name and service name. This guarantees message ordering within a partition, but creates uneven partition sizes because different machines generate different log volumes. As of 2024, the team was working on improving partition balancing at scale.

DNS pipeline architecture

The DNS zone build pipeline connects PostgreSQL to the Zone Builder via Kafka. When a DNS record is changed via the API, a PostgreSQL trigger emits an event that is converted to a Kafka message. The Zone Builder runs two consumer types: a full zone build scheduler for complete zone rebuilds, and a per-record build scheduler for incremental changes. The per-record approach reduced build times on large zones (1 million records) from approximately 34 seconds to 6-8 milliseconds, a 4,250x improvement.

Special techniques and engineering innovations

Offset-based consumer health checks

One of the more notable techniques Cloudflare published is their approach to detecting unhealthy Kafka consumers. A standard connectivity check confirms that a consumer is connected to the broker and can read from the topic, but does not confirm that the consumer is actively processing messages. Consumers can deadlock after a rebalancing event and remain connected while making no progress.

Cloudflare solved this by implementing a Kubernetes liveness probe that tracks committed offset advancement rather than connection state. The probe fails if a consumer’s committed offset has not changed since the previous check interval, indicating a stalled consumer. An in-memory map tracks offsets per partition. When a rebalance occurs, the system rebuilds the map using Sarama’s rebalance signal to ensure each replica only monitors its currently assigned partitions, preventing false failures during reassignment.

Compression with Zstandard

Cloudflare’s first compression choice was Snappy, which achieved a 2.25x reduction on the HTTP log topic and 2.6x on DNS logs. After benchmarking multiple algorithms, the team switched to Zstandard (zstd), which reached a 4.5x compression ratio on the HTTP requests topic. The switch saved hundreds of gigabits of network bandwidth and reduced storage footprint substantially.

During the compression work, the team discovered that Kafka was recompressing already-compressed message batches due to offset validation logic in the Sarama Go client library. They diagnosed the issue and contributed a fix upstream to the Sarama project.

Batch consumption for throughput spikes

For high-throughput consumers, particularly those that hit SLO breaches during traffic spikes, Cloudflare implemented batch consumption: consumers process multiple Kafka messages simultaneously before transformation and dispatch, rather than one message at a time. This approach absorbs production spikes without accumulating consumer lag.

OpenTelemetry integration in the SDK

After pandemic-era traffic growth caused a critical consumer to breach its SLA, the team added OpenTelemetry tracing directly into the Messagebus-Client SDK. This gave them visibility across the full processing chain and allowed them to pinpoint the specific bottlenecks contributing to the SLO breach: bucket writes and Kafka reads.

Operating Kafka at scale

Cloudflare runs its Kafka infrastructure on self-managed bare metal. Key operational practices across its 14 clusters include:

Monitoring: Every producer and consumer gets automatically generated Prometheus dashboards covering production rate, consumption rate, and partition skew. The primary alert for all consumers is high consumer lag, generated without manual configuration.

Observability: OpenTelemetry tracing is embedded in the Messagebus-Client SDK, enabling distributed traces that span from message production through broker delivery to consumer processing.

Fault tolerance targets: The logging pipeline Kafka buffer is sized to tolerate up to eight hours of total consumer outage before any data loss risk arises.

Developer experience: Internal wikis document adoption patterns. A ChatOps integration surfaces schema change notifications in team channels. The Connector Framework’s Cookiecutter templates generate production-ready connector services with minimal input. A newer tool called Gaia enables push-button provisioning of new Kafka-integrated services aligned with Cloudflare’s internal best practices.

Schema governance: The Messagebus Schema service acts as the schema registry for the Messagebus cluster. prototool checks for breaking changes before any schema reaches production.

Infrastructure provisioning: Salt with a GitOps model manages broker configuration and cluster provisioning.

Challenges and how they solved them

Schema coupling despite using a message bus

Early Kafka adoption at Cloudflare used JSON. Despite using Kafka as a decoupling layer, teams remained tightly coupled at the schema level because there was no enforcement of message contracts. The fix was a migration to Protobuf with a strict one-type-per-topic policy, combined with a client-side validation library that validates messages before they are published. This shifted schema errors from runtime consumer failures to producer-side build failures.

Silent consumer failures after rebalancing

Consumers that deadlocked after partition reassignment appeared healthy under basic connectivity checks while making no progress on their partitions. The team addressed this with the offset-comparison liveness probe described above, using Kubernetes to restart stalled consumers automatically. The approach reduced false positives and caught real deadlock conditions that had previously gone undetected.

Disk I/O contention from multiple lagging consumers

Multiple lagging consumers generating random read patterns on spinning disks caused significant I/O contention. The resolution was to migrate those clusters to SSDs. This was an infrastructure cost decision driven by consumer access patterns, and it was preceded by the compression work, which reduced the storage and bandwidth footprint that would otherwise have made the SSD migration more expensive.

SLO breach during pandemic-era traffic spike

A critical consumer fell behind its SLA when traffic surged during the COVID-19 pandemic. The team used OpenTelemetry tracing to identify the specific bottlenecks (bucket writes and Kafka reads) and resolved the lag by implementing batch consumption and adding observability into the SDK to prevent similar blind spots.

Client library over-configuration

Early versions of Messagebus-Client exposed too many configuration knobs. Teams that modified defaults occasionally introduced unintended side effects, including configuration changes that affected other parts of the pipeline. The lesson was to make opinionated defaults the path of least resistance and require explicit justification for deviations. Matt Boyle and Andrea Medda presented this as one of the core tradeoffs at QCon London 2023: the right balance between a highly configurable SDK and a standardised one depends on how much you are willing to invest in documentation and support.

Full tech stack

Category	Technology	Role
Message broker	Apache Kafka	Central message broker across 14 clusters: inter-service bus, analytics, logging, DNS, DDoS telemetry
Message serialization	Protocol Buffers (Protobuf)	Standard message format on the Messagebus cluster; strict typing and cross-language code generation
Message serialization (analytics)	Cap’n Proto	HTTP log encoding in the analytics Kafka pipeline (2018)
Kafka client library	Shopify Sarama	Go Kafka client wrapped by Messagebus-Client
Application language	Go	Primary language for all Kafka services, connectors, and the Messagebus-Client SDK
Application language	Rust	Protobuf schema code generation
Container orchestration	Kubernetes	Runs Kafka consumers; liveness probes power the offset-based consumer health check
Metrics	Prometheus	Auto-generated metrics for every Kafka producer and consumer via Messagebus-Client
Dashboards	Grafana	Visualisation of production rate, consumption rate, and partition skew per topic
Distributed tracing	OpenTelemetry / OpenTracing	Embedded in Messagebus-Client SDK for end-to-end trace visibility across producers and consumers
Analytics sink	ClickHouse	Receives HTTP log data extracted by Kafka consumers; replaced PostgreSQL for analytics aggregation
Compression	Zstandard (zstd)	Message compression on high-throughput topics; 4.5x ratio on the HTTP requests topic
Schema governance	prototool	Breaking change detection in the Messagebus Schema registry
Infrastructure provisioning	Salt	Broker configuration management under a GitOps model
Service scaffolding	Cookiecutter	Template engine for generating new Connector Framework service scaffolding
Internal service provisioning	Gaia	Push-button creation of new Kafka-integrated services according to Cloudflare’s best practices
Log forwarding	syslog-ng	Log collection stage feeding the Kafka logging pipeline; adds hostname and data center metadata
Log capture	systemd-journald	Captures service stdout/stderr and feeds syslog-ng
DNS source	PostgreSQL	Emits change triggers that are converted into Kafka messages for the DNS Zone Builder
Internal key-value store (connector sink)	Quicksilver	Cloudflare’s internal distributed key-value store; used as a Connector Framework sink target

Key contributors

Name	Role	Contribution
Matt Boyle	Engineering Manager, Application Services	Authored the 1 trillion messages post; co-presenter at QCon London 2023
Andrea Medda	Senior Systems Engineer, Cloudflare	Co-authored the intelligent consumer restart post; co-presenter at QCon London 2023
Chris Shepherd	Engineer, Cloudflare	Co-authored the intelligent consumer restart post
Ivan Babrou	Engineer, Cloudflare	Authored the Kafka compression post; led the zstd evaluation and Sarama upstream patch
Alex Bocharov	Engineer, Cloudflare	Authored the HTTP analytics pipeline migration post
Alex Fattouche	Engineer, Cloudflare	Authored the DNS build speed improvement post describing the Kafka-backed zone build pipeline
Colin Douch	Engineer, Cloudflare	Authored the logging pipeline overview post

Key takeaways for your own Kafka implementation

Invest in schema governance before you scale. Cloudflare ran into tight coupling between consumers despite using Kafka as a decoupling layer, because JSON offered no enforcement. Migrating to Protobuf with a strict one-type-per-topic rule and a central schema registry resolved it, but required a migration effort that would have been cheaper to do earlier.
Connection health is not the same as processing health. A Kubernetes liveness probe that checks TCP connectivity to the broker tells you very little about whether a consumer is making progress. Cloudflare’s offset-comparison approach, which checks whether committed offsets advance between intervals, catches deadlocked consumers that a simple connection check would miss entirely.
Measure compression algorithm choice at your actual message shape. Snappy was a reasonable starting point, but benchmarking on Cloudflare’s actual HTTP log messages showed that zstd delivered twice the compression ratio. The specific encoding format (Cap’n Proto, Protobuf, JSON) and message content distribution matter more than general benchmarks.
Opinionated client libraries reduce operational surface area, but documentation is the cost. The Messagebus-Client library made Kafka accessible to teams without deep Kafka expertise, and the default metrics meant every consumer was observable from day one. The tradeoff was that over-configurability in early versions caused unintended side effects. Cloudflare’s recommendation at QCon 2023 was to provide opinionated defaults and invest heavily in documentation so teams don’t deviate without understanding the implications.
Separate clusters by access pattern and criticality. Cloudflare’s 14 clusters are not all running the same workload, and the separation matters operationally. The analytics cluster at 106 brokers (2018) was optimised for throughput; the Messagebus cluster was optimised for reliability and schema governance. Running all workloads on a single cluster would have created both operational and performance coupling.

Sources and further reading

#	Source	Author	Date
1	Using Apache Kafka to process 1 trillion inter-service messages	Matt Boyle, Cloudflare Blog	2022-07
2	Squeezing the firehose: getting the most from Kafka compression	Ivan Babrou, Cloudflare Blog	2018-03
3	HTTP analytics for 6M requests per second using ClickHouse	Alex Bocharov, Cloudflare Blog	2018-03
4	Tales of Kafka at Cloudflare: lessons learnt on the way to 1 trillion messages	Matt Boyle, Andrea Medda — InfoQ	2023
5	Intelligent, automatic restarts for unhealthy Kafka consumers	Chris Shepherd, Andrea Medda — Cloudflare Blog	2023-01
6	An overview of Cloudflare’s logging pipeline	Colin Douch, Cloudflare Blog	2024-01
7	How we improved DNS record build speed by more than 4,000x	Alex Fattouche, Cloudflare Blog	2022-05
8	Tales of Kafka at Cloudflare: Andrea Medda and Matt Boyle at QCon London 2023	InfoQ editorial, QCon London 2023	2023-04

‍

If you are running Kafka in production and want better visibility into your cluster, consumer lag, and topic health, Kpow gives you a real-time control plane for Apache Kafka. You can connect it to any Kafka cluster in minutes and try it free for 30 days.

How DoorDash uses Apache Kafka in production

Factor House — Sat, 30 May 2026 00:00:00 GMT

DoorDash operates one of the largest food delivery platforms in the United States, coordinating millions of orders, restaurant partnerships, and Dasher assignments every day. Underpinning that coordination is an Apache Kafka deployment that spans five clusters, more than 2,500 topics, and around six billion messages per day on average — with peaks reaching twice that rate.

The company reached that scale through two distinct adoption waves: an initial migration from RabbitMQ in mid-2019 to stop task-processing outages, then a ground-up rebuild of its real-time data infrastructure under a platform called Iguazu that replaced Amazon Kinesis and SQS and brought data warehouse latency down from one day to a few minutes.

Company overview

DoorDash connects consumers, restaurants, and delivery drivers (Dashers) across the United States, Canada, Australia, and Japan. The platform processes a high volume of time-sensitive transactions — order placement, restaurant confirmation, Dasher assignment, and delivery completion — all of which depend on low-latency event propagation between microservices.

DoorDash first adopted Kafka in mid-2019 after repeated RabbitMQ failures began causing order checkout and Dasher dispatch outages under peak load. That initial migration, led by engineer Ashwin Kachhara, addressed the immediate reliability problem. A separate initiative, beginning around 2020 and led by Allen Wang (who had previously built Netflix’s Keystone Kafka pipeline), took a broader view: replacing multiple fragmented cloud-native queuing systems with a unified, Kafka-centred streaming platform.

Kafka milestones at DoorDash

Date	Event
Mid-2019	RabbitMQ and Celery outages drive adoption of Kafka for order checkout and Dasher assignment
Late 2019	Allen Wang joins from Netflix; begins building the Real-Time Streaming Platform (RTSP) team
~2020	Construction of Iguazu begins; Kafka replaces Amazon SQS and Kinesis as the central ingestion hub
2021	Riviera declarative ML feature framework goes into production (Kafka + Flink SQL + Redis)
2022	Iguazu reaches hundreds of billions of events per day; Allen Wang publishes the primary architecture post and presents at QCon San Francisco
2023	API-first Kafka topic governance replaces Terraform/Atlantis GitOps; 2,500+ topics, five clusters, six billion messages per day documented publicly
2024	Kafka multi-tenancy via OpenTelemetry context propagation; full self-serve CRUD for topics, users, and ACLs
2025	Fabricator (successor to Riviera) launched: 100+ pipelines, 500 ML features, 100 billion daily feature values

DoorDash’s Kafka use cases

Asynchronous task processing

The original use case. Before mid-2019, DoorDash routed order checkout and Dasher assignment work through RabbitMQ and Celery. Repeated broker failures under peak load caused customer-facing outages with direct revenue impact; RabbitMQ offered limited observability and required slow, manual recovery steps.

Ashwin Kachhara and his team replaced RabbitMQ with Kafka for asynchronous task processing. Kafka’s durable log, at-least-once delivery guarantees, and horizontal scalability eliminated the outage pattern. The migration was completed without downtime using a gradual traffic cutover approach.

Real-time event pipeline to Snowflake (Iguazu)

The more architecturally significant initiative. Before Iguazu, DoorDash maintained separate pipelines using Amazon SQS and Amazon Kinesis to feed its data warehouse, ML platform, and time-series metrics backend. Each pipeline used different technology and communication patterns, producing data warehouse latency of up to one day and significant operational overhead.

Iguazu replaced those fragmented pipelines with a single Kafka-centred platform. All microservices and mobile clients publish events via an HTTP-based Kafka proxy; Apache Flink jobs consume those topics and fan out to multiple destinations (Snowflake, Redis, Chronosphere) with format transformations applied per sink. End-to-end latency to Snowflake dropped from one day to a few minutes.

Use cases documented within Iguazu include:

Dasher assignment monitoring: near-real-time warehouse data lets the Dasher assignment team detect algorithm bugs within minutes rather than the next day
Mobile application health monitoring: checkout page load errors are routed to Chronosphere for operational alerting
Real-time user sessionization and behaviour analysis

Real-time ML feature generation (Riviera and Fabricator)

Delivery events are consumed from Kafka topics by Flink jobs to produce real-time ML features — for example, recent average restaurant wait times. The Riviera framework (and its 2025 successor, Fabricator) expresses these pipelines as a SQL query plus a YAML configuration file, routing Kafka topic data through Flink into Redis for low-latency reads by inference services. By 2025, the Fabricator platform ran more than 100 pipelines generating 500 unique features at a rate of over 100 billion feature values per day.

Search index replication

The Search Platform team built a continuous indexing pipeline where database changes propagate through Kafka to Flink applications, which then update Elasticsearch. Before this pipeline, a full reindex of the search catalogue took up to one week; the Kafka and Flink approach made it continuous and near-real-time.

Advertising budget pacing

The ads platform publishes a record to a Kafka topic when a campaign hits its daily spend limit. A Flink streaming job consumes that topic and writes an expiration flag to CockroachDB, embedding the budget-capped state into DoorDash’s in-house search index at the filtering stage. This change produced a 43% drop in search processor latency and a 45% reduction in discarded candidates compared to the previous approach.

Production testing via Kafka multi-tenancy

In 2024, DoorDash introduced a multi-tenant Kafka architecture to allow test traffic to flow through the same production topics and consumer instances as live traffic. OpenTelemetry context propagation carries a “doortest” tenant identifier on each Kafka record; consumers branch on that metadata without creating separate topics. This approach enables production-fidelity testing without topic proliferation.

Scale and throughput

Metric	Figure	Source
Kafka clusters	5	DoorDash Engineering Blog, December 2023
Topics	2,500+	DoorDash Engineering Blog, December 2023
Daily message volume	~6 billion messages	DoorDash Engineering Blog, December 2023
Average throughput	~4 million messages per minute	DoorDash Engineering Blog, December 2023
Peak throughput	~8 million messages per minute	DoorDash Engineering Blog, December 2023
Iguazu event volume	Hundreds of billions of events per day	Allen Wang, DoorDash Engineering Blog, August 2022
Delivery guarantee	99.99% (four nines)	Allen Wang, DoorDash Engineering Blog, August 2022
Data loss rate (async mode)	Less than 0.001%	Allen Wang, InfoQ / QCon Plus, June 2023
Flink daily processing volume	220 TB per day	Junaid Effendi, The Sequence (citing DoorDash engineering sources)
New topics provisioned per week	~100	DoorDash Engineering Blog, December 2023
Warehouse latency improvement	~1 day to a few minutes	Allen Wang, InfoQ / QCon Plus, June 2023
Fabricator daily feature values	100+ billion	DoorDash Engineering Blog, April 2025

DoorDash’s Kafka architecture

High-level overview

DoorDash runs its Kafka infrastructure on AWS. The five clusters are operated by the Real-Time Streaming Platform (RTSP) team. Public sources do not describe the specific role of each cluster (whether they are segmented by team, environment, or criticality), though all five are provisioned and governed through the same internal tooling.

The Iguazu architecture has three main layers:

Ingestion: An HTTP-based Kafka proxy (built on Confluent’s open-source Kafka REST Proxy) runs on Kubernetes and accepts events from microservices and mobile clients. The proxy provides a REST interface so internal producers do not need native Kafka client configuration.
Processing: Apache Flink jobs run as standalone Kubernetes services deployed via Helm charts, consuming Kafka topics and applying per-destination fan-out and format transformations.
Delivery: Events are written to S3 in Parquet format, then ingested into Snowflake via Snowpipe (triggered by Amazon SQS notifications). ML features go to Redis. Operational metrics go to Chronosphere.

Event ingestion layer: the Iguazu Kafka proxy

DoorDash extended the open-source Confluent Kafka REST Proxy with several additions for the Iguazu use case:

Multi-cluster routing (a single proxy endpoint routes to the appropriate cluster)
Asynchronous producing (decoupling publisher latency from Kafka ACK wait)
Metadata pre-fetching
HTTP header passthrough for OpenTelemetry context propagation

Schema management

All events are defined in Protocol Buffers. Schemas are stored in a central shared Protobuf Git repository and validated at build time in CI/CD pipelines — invalid schemas are caught before deployment, not at runtime. Confluent Schema Registry handles runtime schema lookup.

Where downstream sinks require Avro (for example, certain legacy connectors), DoorDash’s serialization library converts Protobuf to Avro transparently using Avro’s protobuf library. Producers remain Protobuf-native and are unaware of Avro consumers.

Every event flowing through Iguazu uses a standard envelope containing: creation time, source service, encoding method reference, and a schema registry pointer.

Producer architecture

For high-volume event topics without natural message keys (random partition assignment), DoorDash’s default producer configuration produced small, frequent batches and excessive broker CPU load. The RTSP team addressed this with two configuration changes:

Switching to Kafka’s built-in sticky partitioner (batching null-keyed records to the same partition until a batch is ready)
Increasing linger.ms to 50-100 ms to accumulate larger batches

The result was a 30-40% reduction in Kafka broker CPU utilisation on those topics.

For topics requiring delivery guarantees, DoorDash uses replication factor 2 with min.insync.replicas set to 1 — a deliberate choice to favour throughput and cost over maximum durability on non-critical event streams.

Consumer architecture

Consumer groups are managed centrally by the RTSP team. Topic readiness after provisioning is confirmed by polling Prometheus metrics via Chronosphere; the Minions orchestration service waits until topic metrics appear in Chronosphere before marking an onboarding workflow complete.

Stream processing

Apache Flink is the primary stream processing engine. DoorDash exposes two interfaces to internal users:

Flink DataStream API for custom stateful processing logic
Flink SQL with YAML configuration (via the Riviera and Fabricator frameworks) for declarative feature pipelines, where engineers write a SQL query and a YAML file rather than Flink code

Data delivery paths

Destination	Path
Snowflake (data warehouse)	Kafka topic → Flink job → S3 (Parquet) → SQS notification → Snowpipe → Snowflake
Redis (ML feature store)	Kafka topic → Flink SQL job (Riviera / Fabricator) → Redis
Chronosphere (operational metrics)	Kafka topic → Flink job → Chronosphere time-series backend
Elasticsearch (search index)	Database change event → Kafka topic → Flink job → Elasticsearch
CockroachDB (ad budget state)	Ad spend event → Kafka topic → Flink streaming job → CockroachDB

‍

S3 acts as a durable buffer independent of Kafka retention. If Snowflake suffers downtime, failed Snowpipe ingestions can be backfilled from S3 without depending on Kafka replayability.

Per-event isolation is a design principle: each event type has its own dedicated Flink application and its own Snowpipe instance. Failures are contained to a single event type rather than cascading across a shared pipeline.

Special techniques and engineering decisions

Declarative ML feature pipelines

The Riviera framework (2021) and its successor Fabricator (2025) allow data scientists to express Flink stream processing jobs as a SQL query plus a YAML configuration file. The framework generates the Flink job; engineers who would otherwise need to write DataStream API code can instead write a single YAML file. Feature development time was reduced from weeks to hours, and the feature engineering codebase shrank by 70%.

S3 as a durable overflow buffer

Finite Kafka retention creates a risk: if Snowflake is unavailable long enough for Kafka logs to roll off, data is lost. DoorDash addressed this by routing all Iguazu Flink jobs through S3 (Parquet) before Snowpipe ingestion. S3 is not subject to Kafka retention windows, so backfills remain possible regardless of Kafka log state.

OpenTelemetry-based Kafka multi-tenancy

Rather than creating separate topics per test scenario, DoorDash propagates OpenTelemetry context (tenant ID and route information) on every Kafka record header. Production and test traffic share the same topics and consumer group instances; consumers inspect the OTEL header to branch handling. This avoids topic proliferation while providing production-fidelity test conditions.

Protobuf-to-Avro transparent conversion

DoorDash maintains Protobuf as the canonical internal event format. Where downstream sinks require Avro, its serialization library converts on the fly. This allows the company to remain Protobuf-native without modifying producers when adding Avro-dependent sinks.

Operating Kafka at scale

Deployment

All five Kafka clusters run on AWS, managed by the RTSP team. Flink jobs are deployed as standalone Kubernetes services via Helm charts. Infrastructure is managed with Terraform.

Topic governance

DoorDash went through three generations of topic provisioning tooling:

Terraform / Atlantis (GitOps): Required infrastructure team review for every topic creation request. As the pace of new topics grew to roughly 100 per week, this created a significant support burden.
Infra Service + Minions (2023): Replaced GitOps with an HTTP API (Infra Service) and a Cadence-backed orchestration service (Minions) that automates end-to-end Iguazu onboarding — creating Kafka topics, launching Flink jobs, creating Snowflake objects, and opening pull requests, with Slack notifications at each step. Configuration options exposed to users are deliberately restricted to capacity-related settings (retention, partition count) to prevent common misconfigurations. Onboarding time reduced by 95%, from multiple days to under one hour, typically within 15 minutes.
Kafka Self-Serve (2024): Extended Infra Service to cover the full resource lifecycle — topics, user accounts, and ACLs — with auto-approval rules for routine, non-sensitive changes such as standard ACL grants or topic resizes within pre-set bounds.

Authentication and access control

DoorDash uses SASL/SCRAM for per-service Kafka authentication and ACLs for authorisation, controlling which producers and consumer groups can access each topic. The 2024 Kafka Self-Serve platform manages user account creation and ACL assignment through the same API-first workflow as topic creation.

Monitoring

Chronosphere is DoorDash’s operational metrics backend for Kafka. Topic readiness after provisioning is confirmed by polling Prometheus metrics through Chronosphere’s API — the Minions orchestration workflow waits on this before marking onboarding complete. Application-level operational events (for example, mobile checkout errors) are also routed through Iguazu Flink jobs to Chronosphere for alerting.

Schema evolution

Protobuf’s inherent backward and forward compatibility is the primary schema evolution mechanism. Schemas must pass CI/CD validation before merging; runtime enforcement is handled through Confluent Schema Registry. The Fabricator framework enforces backward- and forward-compatible evolution at the framework level.

Challenges and how DoorDash solved them

RabbitMQ reliability failures at peak load

Problem: RabbitMQ repeatedly went down under heavy order volume. Because order checkout and Dasher assignment ran through RabbitMQ and Celery, broker failures directly caused customer-facing outages. Recovery was slow and manual; observability was poor.

Solution: Replaced RabbitMQ with Kafka for asynchronous task processing using a gradual traffic cutover with no downtime. Kafka’s durable log and horizontal scalability eliminated the outage pattern.

Outcome: Task-processing outages from broker failures ceased; the system could scale horizontally to handle peak load.

Source: Ashwin Kachhara, DoorDash Engineering Blog, September 2020.

Fragmented SQS and Kinesis pipelines with one-day warehouse latency

Problem: Separate SQS and Kinesis pipelines for the data warehouse, ML platform, and metrics backend used incompatible technologies and communication paradigms, producing warehouse data latency of up to one day and significant operational overhead from managing disparate systems.

Solution: Consolidated onto a single Kafka + Flink platform (Iguazu). Kafka serves as the unified pub/sub hub; Flink applies per-destination fan-out with data format transformations.

Outcome: Warehouse latency from one day to a few minutes; a single platform team operates the infrastructure.

Source: Allen Wang, DoorDash Engineering Blog, August 2022.

High broker CPU on null-keyed event streams

Problem: For high-volume topics without message keys, the default round-robin partition assignment produced small, frequent batches, generating excessive CPU load on brokers.

Solution: Configured Kafka’s sticky partitioner and increased linger.ms to 50-100 ms to accumulate larger batches per partition before flushing.

Outcome: 30-40% reduction in Kafka broker CPU utilisation on affected topics.

Source: Allen Wang, August 2022 (via Shen Zhu engineering blog summary).

Gevent incompatibility with librdkafka in Python

Problem: A Python point-of-sale service used Gevent for async I/O via monkey-patching. Gevent cannot patch librdkafka (a C library), so native Kafka consumer usage blocked Gevent’s event loop.

Solution: Ran the Kafka consumer loop inside a dedicated Gevent greenlet. Blocking I/O calls inside the consumer were replaced with Gevent-compatible equivalents.

Outcome: The migrated service outperformed the previous Celery/Gevent worker under heavy I/O.

Source: Jessica Zhao and Boyang Wei, DoorDash Engineering Blog, February 2021.

Snowflake downtime risking data loss within Kafka retention windows

Problem: Kafka’s finite log retention creates a data loss risk if Snowflake is unavailable long enough for topic logs to roll off.

Solution: All Iguazu Flink jobs write to S3 in Parquet format before Snowpipe ingestion. S3 serves as a durable buffer independent of Kafka retention; backfills are possible from S3 regardless of Kafka log state.

Source: Allen Wang, InfoQ / QCon Plus, June 2023.

Manual topic provisioning becoming a bottleneck

Problem: GitOps-based topic creation required infrastructure team review for every new topic. At approximately 100 new topics per week, this created a significant support burden and slowed engineering teams waiting for approvals.

Solution: Replaced GitOps with an HTTP API (Infra Service) backed by Minions/Cadence orchestration, with auto-approval for routine requests.

Outcome: Iguazu onboarding time reduced by 95%, from multiple days to under one hour.

Source: DoorDash Engineering Blog, December 2023.

Full tech stack

Category	Technology	Role
Message broker	Apache Kafka	Central pub/sub and event streaming backbone; 5 clusters, 2,500+ topics
Stream processing	Apache Flink	DataStream API for custom logic; Flink SQL for declarative feature pipelines
Event ingestion proxy	Confluent Kafka REST Proxy (self-hosted, open source)	HTTP-based event ingestion layer (Iguazu Kafka Proxy) for microservices and mobile clients
Schema registry	Confluent Schema Registry	Runtime schema lookup and enforcement for Protobuf and Avro events
Schema format (primary)	Protocol Buffers (Protobuf)	Primary event schema format; defined in a central shared Git repository, validated at CI/CD build time
Schema format (secondary)	Avro	Auto-converted from Protobuf by DoorDash’s serialization library for sinks that require it
Intermediate storage	Amazon S3	Durable buffer for Parquet-format event data between Flink jobs and Snowflake
Warehouse ingestion	Amazon SQS + Snowpipe	SQS notifications trigger Snowpipe to ingest from S3 to Snowflake
Data warehouse	Snowflake	Analytical data warehouse for business intelligence and Dasher monitoring
Feature store	Redis	Sink for Riviera/Fabricator real-time ML feature pipelines; serves inference services
OLAP / internal analytics	Apache Pinot	Real-time OLAP for ad campaign reporting and Risk Platform dashboards; fed from Kafka
Monitoring backend	Chronosphere	Operational metrics and alerting; receives real-time events from Iguazu; confirms Kafka topic readiness after provisioning
Container orchestration	Kubernetes	Runs Flink jobs (as standalone services) and the Kafka REST Proxy
Deployment	Helm	Deploys Flink jobs on Kubernetes
Infrastructure-as-code	Terraform	Infrastructure provisioning for Kafka and surrounding systems
Workflow orchestration	Cadence	Workflow engine backing the Minions orchestration service for Iguazu onboarding
Internal UI	Retool	Internal management UI for Iguazu pipeline configurations
Authentication	SASL/SCRAM	Per-service Kafka authentication
Authorisation	Kafka ACLs	Controls which producers and consumer groups can access each topic
Observability / context	OpenTelemetry (OTEL)	Context propagation for Kafka multi-tenancy; carries tenant and route metadata on each record header
Search index	Elasticsearch	Continuously updated via Kafka + Flink indexing pipeline
Ad state storage	CockroachDB	Stores budget-capped flags written by the Kafka/Flink advertising pipeline
Batch processing	Apache Spark	Large-scale data lake transformations (separate from the Kafka streaming path)
Batch orchestration	Apache Airflow	Orchestrates batch pipelines
Feature pipeline orchestration	Dagster	DAG construction and orchestration within the Fabricator feature engineering framework
SQL query engine	Trino	Unified SQL query layer over the data lake

Key contributors

Name	Title / team	Contribution
Allen Wang	Tech Lead, Data Platform; founding member, Real-Time Streaming Platform	Architect of Iguazu; authored the primary August 2022 engineering blog post; presented at QCon San Francisco 2022 and QCon Plus 2022; previously built Netflix’s Keystone Kafka pipeline; contributed rack-aware consumer partition assignment to Apache Kafka
Ashwin Kachhara	Software Engineer, Kotlin and Go Platform team	Led the 2019 RabbitMQ-to-Kafka migration; authored the September 2020 engineering blog post; presented at the Bay Area Apache Kafka meetup and a Confluent event
Jessica Zhao	Software Engineer	Co-authored the February 2021 post on making Kafka consumer compatible with Gevent in Python
Boyang Wei	Software Engineer	Co-authored the February 2021 post on making Kafka consumer compatible with Gevent in Python
Carlos Herrera	Software Engineer, Developer Platform (Infrastructure Engineering)	Lead author on the March 2024 Kafka multi-tenancy blog post
Seed Zeng	Software Engineer, Storage team	Co-authored the August 2024 Kafka Self-Serve blog post
Kane Du	Software Engineer, Storage Self-Serve platform	Co-authored the August 2024 Kafka Self-Serve blog post
Donovan Bai	Software Engineer, Storage team (stateful systems including Kafka)	Co-authored the August 2024 Kafka Self-Serve blog post

Key takeaways for your own Kafka implementation

Use S3 as a durable layer between Kafka and your warehouse. Kafka retention is finite; if your downstream sink has an outage, you lose the ability to replay. Routing Flink output through S3 (Parquet) before Snowpipe decouples ingestion from Kafka retention windows and allows backfill regardless of Kafka log state.
Tune your producer for null-keyed high-volume topics. If you have event streams without natural message keys, round-robin partition assignment produces small, frequent batches and elevated broker CPU. Kafka’s sticky partitioner combined with a linger.ms of 50-100 ms is a low-risk configuration change that can meaningfully reduce broker load.
Restrict what users can configure when self-serving topics. DoorDash’s API-first governance deliberately exposes only capacity-related settings (retention, partition count) to internal users, hiding complex broker parameters. This design choice reduces misconfiguration without removing autonomy, and it scales better than a review-based GitOps model as topic volume grows.
Consider isolating Flink jobs per event type rather than sharing pipelines. Per-event isolation means a failure in one pipeline does not cascade to others. The trade-off is more Kubernetes deployments to manage; DoorDash addressed that with Helm and centralised orchestration through Minions.
If you run large-scale ML feature pipelines on Kafka, evaluate a declarative DSL layer. Riviera and Fabricator show that abstracting Flink SQL behind a YAML configuration significantly widens the pool of engineers who can create and maintain real-time feature pipelines, at the cost of framework complexity that the platform team must own.

Sources and further reading

Primary sources

Ashwin Kachhara, “Eliminating Task Processing Outages by Replacing RabbitMQ with Apache Kafka Without Downtime,” DoorDash Engineering Blog, September 2020.
Jessica Zhao and Boyang Wei, “How to Make Kafka Consumer Compatible with Gevent in Python,” DoorDash Engineering Blog, February 2021.
DoorDash Engineering, “Building A Declarative Real-Time Feature Engineering Framework,” DoorDash Engineering Blog, March 2021.
Satish, Danial, and Siddharth, “Building Faster Indexing with Apache Kafka and Elasticsearch,” DoorDash Engineering Blog, July 2021.
Allen Wang, “Building Scalable Real-Time Event Processing with Kafka and Flink,” DoorDash Engineering Blog, August 2022.
Allen Wang, “From Zero to a Hundred Billion: Building Scalable Real-Time Event Processing at DoorDash,” QCon San Francisco, October 2022.
Allen Wang, “Building Scalable Real Time Event Processing with Kafka and Flink,” InfoQ / QCon Plus (recorded), June 2023.
DoorDash Engineering (RTSP team), “API-First Approach to Kafka Topic Creation,” DoorDash Engineering Blog, December 2023.
DoorDash Engineering, “Introducing DoorDash’s In-House Search Engine,” DoorDash Engineering Blog, February 2024.
Carlos Herrera, Amit Gud, and Yunji Zhong, “Setting Up Kafka Multi-Tenancy,” DoorDash Engineering Blog, March 2024.
Seed Zeng, Kane Du, and Donovan Bai, “DoorDash Empowers Engineers with Kafka Self-Serve,” DoorDash Engineering Blog, August 2024.
DoorDash Engineering, “Introducing Fabricator: A Declarative Feature Engineering Framework,” DoorDash Engineering Blog, April 2025.

If you are running Kafka in production and want visibility into consumer lag, topic health, and cluster performance, Kpow gives you a UI and monitoring layer that works with any Kafka deployment. You can try it with a free 30-day trial and connect it to any cluster in minutes via Docker, Helm, or JAR.

How Goldman Sachs uses Apache Kafka in production

Factor House — Sat, 30 May 2026 00:00:00 GMT

Goldman Sachs runs Apache Kafka across at least three separate business divisions, each with a different deployment model and a different set of problems to solve. One team operates a resilient on-premises cluster designed around the assumption that failures will always occur. Another migrated its cluster to Amazon MSK to reduce operational overhead. A third uses Kafka as the messaging bus for a payments platform that must process instant transactions with exactly-once guarantees and strict ISO20022 schema compliance.

The result is less a single Kafka story and more three distinct engineering contexts that happen to share the same underlying technology. What connects them is a consistent focus on data integrity and availability in an industry where those properties are non-negotiable.

Company overview

Goldman Sachs is a global investment banking and financial services firm operating across investment banking, asset management, consumer banking, and transaction banking. The scale and latency requirements of financial markets mean the firm has long invested in real-time data infrastructure.

Goldman Sachs’s public Kafka work spans at least eight years of documented activity. Anton Gorshkov, Managing Director at Goldman Sachs Asset Management (GSAM), described the firm’s on-premises Kafka architecture at QCon New York as early as 2017. In the same year, the Global Investment Research (GIR) division began planning to refactor its on-premises stack, a process that ultimately culminated in a migration to Amazon MSK. The Transaction Banking (TxB) division separately built a payments platform on Amazon MSK and has presented on its monitoring and resiliency testing approach at Kafka Summit Europe 2021.

DateEvent2017Anton Gorshkov presents GSAM Core Platform Kafka architecture at QCon New York and Kafka Summit2018GIR begins refactoring its on-premises technology stack, starting the path toward AWS migration2021Sheikh Araf and Ameya Panse present TxB Kafka monitoring and resiliency approach at Kafka Summit Europe 20212023AWS blog post published detailing GIR’s completed migration from on-premises Kafka to Amazon MSK using MirrorMaker 2.0

Goldman Sachs’s Kafka use cases

GSAM Core Front Office Platform: financial transaction streaming

The Core Platform team within Goldman Sachs Asset Management uses Kafka as a pub-sub messaging platform for financial transaction events. Downstream consumers receive messages via three pathways: a direct sink to an in-memory RDBMS used for operational troubleshooting, a Spark Streaming job that processes events and feeds a second in-memory RDBMS queried via REST and Vert.x APIs, and a batch ETL job that persists the full event stream to a data lake for audit and governance purposes. Every message carries a globally unique identifier assigned by the upstream service, which makes idempotent replay practical in outage recovery scenarios.

Global Investment Research: publication workflow orchestration

The GIR division uses Kafka as the backbone for orchestrating research publication workflows. When a research document is ready for distribution, Kafka coordinates the handoff between approximately 12 microservices running across 30 instances, routing publications to the correct downstream channels: the client portal, the API, and mobile notifications.

Transaction Banking: instant payments processing

TxB uses Kafka as the inter-process messaging layer between the Payment Gateway and other TxB microservices. The division chose Kafka for this use case in part because Kafka’s native producer idempotency and schema registry integration map well to the requirements of instant payments: exactly-once processing guarantees and strict message format validation. In internal lab testing, Kafka outperformed traditional queuing technologies by approximately 175% for this workload.

Scale and throughput

Public scale figures are available for the GSAM Core Front Office Platform. As of 2017, the cluster handled approximately 1.5 TB per week of traffic, with peak production rates reaching around 1,500 messages per second.

For GIR, the cluster served approximately 12 microservices across 30 instances, though no message rate or throughput figures have been published.

No public scale figures are available for the TxB cluster.

Goldman Sachs’s Kafka architecture

GSAM Core Front Office Platform

The GSAM platform runs on an on-premises virtualised cluster spanning multiple data centres within the New York City metro area. The team’s design principle is to treat multiple co-located data centres as a single redundant logical data centre rather than as primary and backup sites. With inter-site network latency of approximately 4ms between New York City and New Jersey facilities, synchronous replication between sites is feasible and is the approach taken.

A minimum of three in-sync replicas is required at all times. There is no primary/backup distinction and no failover event to manage: all replicas are considered equivalent.

The platform uses the Kafka Connect API to attach downstream consumers to the cluster. Those consumers fall into three categories: a direct RDBMS sink for operational visibility, a Spark Streaming job that feeds a real-time query layer, and a batch ETL sink to a data lake.

Data protection is layered across four mechanisms. Starting from the most granular: synchronous disk-level replication via EMC Symmetrix Remote Data Facility (SRDF), asynchronous replication between sites, nightly batch replication, and tape backup. This arrangement is designed to cover failure scenarios ranging from a single VM loss (estimated to occur 1-5 times per year) to a full data centre outage (modelled as a once-in-20-years event).

Global Investment Research

GIR initially ran an on-premises Kafka cluster and migrated it to Amazon MSK beginning in 2018 and completing the cutover in the 2021-2023 period. Connectivity between the on-premises network and the MSK cluster in AWS uses AWS PrivateLink with Network Load Balancers, routed through a GS Transit Account VPC. Services use the Spring Kafka client library.

Avro is used for message serialisation, with a Schema Registry managing schema evolution. The Schema Registry itself was migrated to AWS as part of the same project, with the _schemas topic replicated via MirrorMaker 2 before the cluster cutover.

Transaction Banking

TxB runs its Payment Gateway on Amazon ECS with AWS Fargate as the compute layer. Amazon MSK handles all inter-process communication between the Payment Gateway and downstream TxB microservices. Schema compliance is enforced via a Schema Registry configured to validate all messages against ISO20022 specifications.

Stream processing

GSAM uses Apache Spark Streaming for stateful processing of events downstream of Kafka, feeding results to in-memory RDBMS instances that serve end-user-facing APIs.

Kafka Connect ecosystem

GSAM uses the Kafka Connect API to connect message sinks to its on-premises cluster. Documented sinks include the in-memory RDBMS and the batch ETL data lake sink. Specific connector implementations have not been published.

Special techniques and engineering innovations

Symmetric multi-datacenter replication without failover

The GSAM platform avoids the operational complexity of primary/backup data centre topologies by treating the NYC metro area as a single logical deployment zone. With synchronous replication and a minimum of three in-sync replicas spread across facilities, there is no failover scenario: if a broker or data centre becomes unavailable, the remaining in-sync replicas continue serving reads and writes without any reconfiguration. The team documented a range of actual failure frequencies: single VM failures happen 1-5 times per year with no impact; simultaneous loss of two VM hosts occurs approximately once a year with processing halted for some topics; a three-host or data centre-level failure is rare enough to be modelled as a once-in-several-years event.

Belt-and-suspenders data protection

Beyond Kafka’s built-in replication, the GSAM platform layers three additional data protection mechanisms: asynchronous replication, nightly batch replication, and tape backup. The most granular layer is synchronous disk-level replication via EMC SRDF, which mirrors data between storage arrays in different facilities. The team’s stated principle is that failure will always occur, and each protection layer is sized against a specific failure probability.

Globally unique identifiers for idempotent replay

Every message produced to the GSAM cluster is tagged with a globally unique identifier by the upstream service before it enters Kafka. If an outage requires messages to be resent, the identifier allows consumers to detect and discard duplicates without additional coordination or changes to consumer logic.

Atomic cutover migration with MirrorMaker 2

GIR evaluated two migration approaches for its move to Amazon MSK: an atomic cutover with a planned downtime window, and an incremental hybrid migration that would keep both clusters running in parallel. The team chose the atomic approach to avoid rewriting the Spring Kafka library configuration across all 12 services. The migration sequence was: replicate the Avro Schema Registry using a unidirectional MirrorMaker 2 stream, replicate all topics using a custom MM2 replication policy that stripped the default topic-name prefixes, stop all services, reconfigure DNS endpoints from on-premises brokers to MSK brokers, restart services, and validate end-to-end. Total downtime was approximately 2 hours within a 7-hour migration window.

ISO20022 schema validation with fail-fast enforcement

TxB enforces ISO20022 message format compliance at the producer and consumer level via Schema Registry integration. If a producer or consumer presents a message that does not conform to the registered schema, it is rejected before any further processing occurs. The team chose this fail-fast behaviour deliberately: in a payments context, processing a malformed payment message would have a worse outcome than rejecting it at the boundary.

Kafka producer idempotency for exactly-once payments

TxB uses Kafka’s native producer idempotency feature to achieve exactly-once delivery semantics for payment messages. The team notes that traditional queuing technologies do not provide native producer idempotency, making Kafka a better fit for this specific requirement without additional application-level deduplication logic.

Operating Kafka at scale

GSAM monitoring

The GSAM platform exposes a REST service that provides cluster insights: topic metadata, consumer lag, and in-sync replica counts. Metrics are collected at three levels: application, JVM, and infrastructure. All metrics feed into a time-series database with centralised alerting.

TxB monitoring and resiliency testing

TxB monitors its Kafka clusters using DataDog dashboards. A JMX agent sidecar collects metrics from producers and consumers, covering error rates, connection rates, latencies, and consumer lag, giving the team a live view across the entire service footprint.

Cluster health is additionally tracked by a custom heartbeat application that generates alerts in DataDog when it detects degraded cluster state.

Beyond monitoring, TxB runs regular game days where the team simulates various failure scenarios across the full client infrastructure to validate recovery behaviour and improve availability characteristics before those failures occur in production.

Deployment models

GSAM Core Platform runs self-managed Kafka on virtualised on-premises infrastructure. GIR migrated to Amazon MSK after 2018. TxB runs on Amazon MSK with compute on Amazon ECS and AWS Fargate.

Challenges and how they solved them

MirrorMaker 2 flush timeout too short during GIR migration

During topic replication in the GIR migration, the default MirrorMaker 2 flush timeout of 5 seconds caused failures. The team increased it to 30 seconds to accommodate the volume of data being replicated.

Message size limits exceeded in transit

Default settings for max.request.size and max.message.bytes were too small for some GIR messages during replication. Both parameters were increased in the MSK and MirrorMaker 2 configuration before the full cutover.

Network Load Balancer idle timeout shorter than Kafka client timeout

AWS Network Load Balancers have a default idle connection timeout of 350 seconds. The Kafka client default for connections.max.idle.ms is 540 seconds, meaning connections would be silently dropped by the NLB before the Kafka client detected and re-established them. GIR resolved this by setting connections.max.idle.ms to a value below 350 seconds on all client services.

Full tech stack

Category	Technology	Notes
Message broker	Apache Kafka (on-premises, virtualised)	GSAM Core Front Office Platform
Message broker	Amazon MSK	GIR (post-migration) and TxB instant payments platform
Schema registry	Avro + Schema Registry	Message serialisation and schema governance in GIR and TxB; ISO20022 validation in TxB
Stream processing	Apache Spark Streaming	Stateful processing downstream of Kafka in GSAM; feeds in-memory RDBMS query layer
Connectors	Kafka Connect	Message sink connectors in GSAM Core Platform
Migration tooling	Apache Kafka MirrorMaker 2.0	Topic and schema registry replication during GIR on-prem to MSK migration
Compute	Amazon ECS + AWS Fargate	Payment Gateway container compute in TxB
Networking	AWS PrivateLink + Network Load Balancers	Secure connectivity between on-premises network and MSK during GIR migration
Kafka client library	Spring Kafka	Used across GIR microservices
Monitoring	DataDog	Cluster monitoring dashboards and alerting in TxB
Metrics collection	JMX agent sidecar	Producer and consumer metrics collection in TxB
Storage replication	EMC Symmetrix Remote Data Facility (SRDF)	Synchronous disk-level replication between GSAM on-premises sites
Storage sinks	In-memory RDBMS	Real-time and troubleshooting query layer downstream of Kafka in GSAM
API layer	Vert.x / REST APIs	Application API layer serving data from in-memory RDBMS in GSAM

Key contributors

Name	Title / team	Contribution
Anton Gorshkov	Managing Director, GSAM Core Platform	Presented GSAM Kafka architecture and resilience design at QCon New York 2017 and Kafka Summit
Sheikh Araf	Associate, Transaction Banking	Presented TxB Kafka monitoring and resiliency testing at Kafka Summit Europe 2021
Ameya Panse	Associate, Transaction Banking	Presented TxB Kafka monitoring and resiliency testing at Kafka Summit Europe 2021
Zachary Whitford	Associate, Global Investment Research	Co-authored AWS blog post on GIR on-premises to MSK migration (2023)
Richa Prajapati	Vice President, Global Investment Research	Co-authored AWS blog post on GIR on-premises to MSK migration (2023)
Aldo Piddiu	Vice President, Global Investment Research	Co-authored AWS blog post on GIR on-premises to MSK migration (2023)

Key takeaways for your own Kafka implementation

Model your failure probability explicitly before designing replication. The GSAM team published specific failure rates for single-VM, dual-VM, and data centre-level failures. That grounding in actual observed frequencies is what justified the layered redundancy approach rather than over- or under-engineering it.
Treat multi-datacenter latency as a design input, not a constraint. The GSAM decision to use synchronous replication across NYC metro sites is only viable because the inter-site latency is approximately 4ms. If you are evaluating multi-site synchronous replication, measure your actual latency before committing to the topology.
Choose your migration strategy based on what you are willing to rewrite. GIR picked atomic cutover specifically to avoid modifying Spring Kafka configuration across 12 services. If your codebase can accommodate a gradual migration, the incremental approach gives you more rollback options. If it cannot, atomic cutover with thorough pre-migration replication may be simpler.
Network Load Balancer idle timeouts will silently drop Kafka connections. If you are running Kafka clients behind AWS NLBs, set connections.max.idle.ms below the NLB idle timeout rather than relying on the Kafka default of 540 seconds.
Schema enforcement at the boundary is a valid trade-off in high-stakes pipelines. TxB’s fail-fast approach rejects non-compliant payment messages before any processing occurs. For most event streaming workloads this would be too strict, but in payments, a rejected message is recoverable; a processed malformed payment is not.

Sources and further reading

When Streams Fail: Implementing a Resilient Apache Kafka Cluster at Goldman Sachs — Anton Gorshkov, InfoQ / QCon New York 2017
When Streams Fail: Kafka Off the Shore — Anton Gorshkov, InfoQ / QCon New York 2017 (video presentation)
How Goldman Sachs migrated from their on-premises Apache Kafka cluster to Amazon MSK — Zachary Whitford, Richa Prajapati, Aldo Piddiu, AWS Big Data Blog, March 2023
Monitoring and Resiliency Testing our Apache Kafka Clusters at Goldman Sachs — Sheikh Araf and Ameya Panse, Kafka Summit Europe 2021
How Transaction Banking Implemented an Instant Payment Architecture for High-Throughput, Low-Latency and 24x7 Availability — Goldman Sachs Developer Blog

If you are managing Kafka clusters across multiple teams or deployment environments, Kpow gives you a single view of broker health, consumer lag, schema registry state, and topic activity. You can connect it to any Kafka cluster in minutes and try it free for 30 days.

How Grab uses Apache Kafka in production

Factor House — Sat, 30 May 2026 00:00:00 GMT

Grab’s engineering team built one of Southeast Asia’s most sophisticated real-time data platforms on top of Apache Kafka, and then spent five years publishing exactly how they did it. The Coban team’s body of work is unusually candid: cross-AZ traffic costs that consumed half their Kafka budget, broker crashes that required manual intervention, partition rebalancing that took six hours and caused prolonged latency spikes. What makes Grab’s story worth studying is not just the scale, but the systematic way they worked through each operational constraint and wrote it up.

The platform processes more than 300 billion events per week at terabytes of ingress per hour. As of late 2023, the Coban control plane managed over 5,000 data streaming resources across every Grab vertical.

Company Overview

Grab is Southeast Asia’s leading superapp, operating across mobility, food delivery, package delivery, payments, and financial services in eight countries. The company serves hundreds of millions of consumers across more than 400 cities.

Kafka sits at the centre of Grab’s event-driven architecture. Every time a user books a ride, a series of events ripples through booking state machines, driver notification systems, rewards computation, personalisation pipelines, and analytics dashboards. The Coban team, whose name comes from a waterfall in Indonesia, was formed to build and operate the streaming infrastructure underneath all of it. Their mandate: provide a NoOps, managed platform for seamless, secure access to event streams in real time, for every team at Grab.

Key Kafka milestones:

Date	Event
2020-01	“Plumbing at Scale” published; Go-based Stream Processing Framework serving 300B+ events/week
2020-10	VPA with fixed pod count reduces Kafka consumer resource waste by 45%
2021	Kafka Connect initiative launched; Debezium CDC and MirrorMaker2 productionised
2021-H2	Company-wide chaos engineering validates Kafka platform resilience and DR failover
2022-02	Cross-VPC Kafka exposure via AWS VPC Endpoint Service published
2022-04	Kafka Connect blog: CDC, message mirroring, Azure Event Hubs connector
2022-12	Zero-trust mTLS + OPA + Strimzi security model deployed
2023-07	Closest-replica fetching rolled out; consumer cross-AZ cost reduced to near zero
2023-11	Coban control plane managing 5,000+ streaming resources; monthly active users quadrupled
2023-12	AWS NTH + EBS storage + Load Balancer Controller eliminate manual broker intervention
2025-09	AutoMQ adopted; partition migration time: 6 hours to seconds; 3x throughput
2025-10	ML predictive autoscaler for Flink: more than 35% CPU cost reduction
2025-11	Real-time data quality monitoring deployed across 100+ Kafka topics

Grab’s Kafka Use Cases

Kafka at Grab is not a single-purpose bus. The Coban platform serves transactional and analytical use cases simultaneously, across every vertical in the company.

Real-time event sourcing is the foundational use case. Every booking, payment, food order, and driver status change produces events that fan out to multiple downstream consumers. Booking state machines, reward point computation, feed generation, and personalisation all run asynchronously from a centralised Kafka event log. Services apply changes from this log to their own state independently, without blocking each other.

Change Data Capture eliminated a dual-write problem that was widespread across Grab’s MySQL-heavy backend. Services previously had to write to both MySQL and Kafka atomically, which required two-phase commit and created consistency risks. With Debezium connectors running on Kafka Connect, services write only to MySQL, and Coban’s CDC pipeline captures binlog changes and publishes them to Kafka automatically, including before-and-after row snapshots. GrabFood’s Elasticsearch index synchronisation was one of the most significant early adopters.

Disaster recovery and cross-cluster migration are handled through MirrorMaker2 deployed on Kafka Connect. Critical services can fail over across AWS regions with zero message loss and seamless offset resumption. Coban validated this during a company-wide chaos engineering campaign in 2021. MM2 has also been used to migrate streams from self-managed Kafka clusters into the Coban platform with zero producer or consumer downtime.

Real-time analytics pipelines connect Kafka to Apache Pinot. Partner Gateway API metrics flow from Kafka producers through Flink (for serialisation and transformation) into Pinot topics, where they power sub-second time-series dashboards and anomaly detection for partners monitoring their API integrations.

Data lake ingestion is one of Coban’s core responsibilities. The platform is the entry point to Grab’s data lake, ingesting events from all services for storage and batch or streaming analysis downstream.

Stream processing pipelines for ML features and business intelligence run as either Apache Flink jobs or Go-based Stream Processing Framework (SPF) pipelines on Kubernetes. Time-windowed aggregations, stream joins, filtering, and mapping are all standard operations. For example, personalising the Grab app landing page requires counting a user’s interactions with widget elements across a recent time window, pre-aggregated rather than computed on demand.

Real-time data quality monitoring was deployed in 2025. FlinkSQL jobs consume production Kafka topics, apply data contract rules (both syntactic and semantic), and halt propagation of invalid data before it cascades to downstream consumers. An LLM analyses Kafka stream schemas and anonymised sample data to recommend semantic validation rules, reducing the manual effort of defining per-field rules across hundreds of topics. As of late 2025, the system actively monitors more than 100 critical Kafka topics.

Scale

Grab’s streaming platform operates at a scale that makes conventional approaches expensive and brittle.

Metric	Value
Events processed per week	300 billion+ (as of 2020; platform has grown since)
Data ingress rate	Terabytes per hour
Streaming resources managed by Coban	5,000+ (topics, Flink pipelines, CDC pipelines, Kafka Connect connectors, Zeppelin notebooks)
Kafka topics monitored for data quality	100+
Cross-AZ traffic as share of Kafka cost (before 2023)	~50%
Kafka consumer resource reduction after VPA adoption	~45%
Heimdall API uptime (H1 2023)	Above 99.95%
End-to-end workflow success rate (H1 2023)	Above 90%

Architecture

Infrastructure layer. Grab’s platform is hosted on AWS in a single region spanning three Availability Zones. All Kafka brokers and clients are distributed across these three AZs for high availability. Each Kafka partition has three replicas, distributed rack-aware, with one replica per AZ.

Kafka runs on Amazon EKS using the Strimzi operator. Each production Kafka cluster runs on a dedicated EKS cluster, with one Kafka broker per dedicated worker node, enforced via Kubernetes taints and tolerations. Storage uses AWS EBS gp3 volumes dynamically provisioned via the EBS CSI driver. This replaced an earlier design using NVMe instance store volumes, which could not survive worker node replacement without manual intervention.

Kafka clusters are accessible across VPC boundaries via AWS Network Load Balancers, with each broker advertised by a private zonal DNS name and a distinct TCP port. This enables deterministic per-broker connections required by Kafka producers and consumers. For cross-account access (for example, GrabKios in its own AWS account), a VPC Endpoint Service is used in place of VPC peering.

Control plane. The Coban control plane has three tiers:

Component	Role
Coban UI	React SPA for self-service resource creation and monitoring
Heimdall	Go backend; exposes CRUD APIs with ABAC; converts UI actions to Terraform
Khone	Git repository; stores Terraform code and metadata; CI provisioner via Atlantis

‍

All streaming resources are declared as Terraform code under the hood. Heimdall abstracts this from users who prefer a UI or REST API, and Khone provides the Git-based audit trail, peer review workflow, and CI pipeline. The Coban UI shows real-time metrics for Kafka clusters and topic byte rates, and integrates with Grab’s monitoring stack.

Stream processing layer. Two frameworks handle stream processing at Grab. Apache Flink is used for stateful stream processing, including the data quality monitoring system, Partner Gateway analytics, and the ML feature pipelines. The Go-based Stream Processing Framework (SPF) runs as Kafka consumer pods on Kubernetes and handles hundreds of simpler pipelines with filtering, mapping, aggregation, and windowing operations.

AutoMQ adoption (2025). For a subset of clusters requiring high elasticity, Coban adopted AutoMQ, a Kafka-compatible broker with a shared EBS WAL and S3 storage architecture. With AutoMQ, partition reassignment no longer requires moving data between brokers, reducing migration time from six hours to seconds and delivering a 3x throughput improvement.

Special Techniques

Rack-aware closest-replica fetching. By default, Kafka consumers fetch from partition leaders, which reside in a different AZ 67% of the time. This generated cross-AZ traffic that represented approximately 50% of total Kafka platform cost. Coban upgraded their legacy Kafka clusters to version 3.1 and configured replica.selector.class=RackAwareReplicaSelector on brokers, with broker.rack set to the AZ ID. On the consumer side, the internal Golang Kafka SDK was updated so that teams can enable closest-replica fetching by exporting a single environment variable, with the SDK dynamically setting client.rack from EC2 instance metadata at startup. The same logic was applied to Flink pipelines and Kafka Connect connectors. After rollout, consumer cross-AZ traffic cost dropped to near zero. Note: up to 500ms of additional end-to-end latency can appear for non-latency-sensitive flows due to replication lag, so latency-sensitive pipelines continue to fetch from partition leaders.

Protobuf SerDes with Confluent Schema Registry. Grab standardised on Protobuf for all Kafka message serialisation and deserialisation. Confluent Schema Registry sits at the centre of the Kafka Connect ecosystem, enabling format conversions: Protobuf to Avro, Protobuf to JSON, and Protobuf to Parquet. For cross-cloud ingestion to Azure Event Hubs (which does not support Protobuf natively), Coban built an in-house converter that deserialises Protobuf bytes using a schema retrieved from the registry, traverses the message tree recursively, converts each field to a JSON node, and serialises the result.

MirrorMaker2 deployed per-connector on Kafka Connect. Rather than running the standard three-connector MM2 bundle, Coban deploys MirrorSourceConnector and MirrorCheckpointConnector independently on the Kafka Connect framework. This provides finer control over each connector and enables IaC management via Terraform, with offset mirroring handled separately so consumers can seamlessly resume from backup clusters.

Zero-trust mTLS and OPA authorisation. mTLS is enforced for all client-broker and broker-broker communications via Strimzi. Short-lived ephemeral certificates are issued by a HashiCorp Vault PKI engine, with a Root CA per environment signed down to cluster-level and client-level intermediate CAs. Open Policy Agent (OPA) is integrated with Kafka brokers to enforce topic-level, least-privilege authorisation: each client is whitelisted for the specific topics and permissions (produce or consume) it strictly needs.

ML-based predictive autoscaling for Flink. A time-series forecasting model trained on Kafka source topic throughput predicts CPU demand ahead of time. A separate regression model maps the forecast to required TaskManager CPU. By scaling vertically before demand spikes rather than reacting after, the system avoids the reactive scaling spirals common with HPA. Deployed to the majority of applicable Flink pipelines, the system delivered more than 35% reduction in cloud CPU cost.

LLM-assisted data contract rule generation. The data quality monitoring system uses an LLM to analyse Kafka stream schemas and anonymised sample data, recommending semantic test rules that would be impractical to define manually at scale across hundreds of topics. FlinkSQL auto-generates the test definitions from the approved contracts.

Operating Practices

Infrastructure-as-Code for all streaming resources. Every Kafka topic, cluster, Kafka Connect connector, Flink pipeline, and CDC pipeline is declared as Terraform. Changes go through Git MRs, peer review, and CI pipeline application via Atlantis. The Coban UI and Heimdall API abstract this for users who prefer not to write Terraform directly, but the underlying system remains fully auditable and reproducible.

Cruise Control for partition balancing. Cruise Control is deployed alongside each Kafka cluster for partition leader rebalancing. It is used during Kafka upgrades and rolling updates to maintain cluster balance.

AWS Node Termination Handler for graceful broker shutdown. NTH runs in Queue Processor mode, capturing ASG scale-in events, manual instance terminations, and EC2 maintenance events via SQS and EventBridge. When a node is going to be terminated, NTH cordons and drains it, which sends a SIGTERM to the Kafka pod and triggers a graceful shutdown. Strimzi’s terminationGracePeriodSeconds=180 gives brokers enough time to migrate all partition leaders, typically around 60 seconds for 600 partition leaders. The result is that unexpected node termination no longer requires engineer intervention.

Per-AZ consumer load monitoring. After enabling closest-replica fetching, CPU utilisation became skewed across AZs because consumers in well-provisioned AZs handled a disproportionate share of reads. Coban updated their internal Kafka SDK to expose AZ as a metric, enabling operators to proactively rebalance consumers across zones.

VPA with fixed pod count for Kafka consumer pipelines. SPF pipelines match pod count to the number of partitions in the source Kafka topic, fetched at runtime via the Kafka API. Vertical Pod Autoscaler handles CPU and memory allocation per pod based on historical load trends. This approach delivered approximately 45% reduction in total resource usage versus resources requested, compared to the previous HPA-based approach.

Self-service platform with high reliability targets. The Heimdall API maintained greater than 99.95% uptime in H1 2023. End-to-end workflow success rate for self-service resource creation, change, and deletion held above 90% during the same period, even as monthly active users nearly quadrupled.

Chaos engineering for DR validation. A company-wide chaos engineering campaign in 2021 validated the robustness of the Kafka platform, including MirrorMaker2-based cross-region failover. Coban’s Kafka demonstrated resilience across multiple chaos rounds.

Challenges and Solutions

Challenge: Cross-AZ network traffic consumed half the Kafka budget. In Grab’s initial design, Kafka consumers fetched from partition leaders by default. With three AZs and partition leaders distributed across them, any given consumer had a 67% chance of fetching from a different AZ. At Grab’s scale, this consumer-side cross-AZ traffic dominated the bill and represented roughly 50% of total Kafka platform cost.

Root cause: the default Kafka consumer configuration fetches only from partition leaders, regardless of replica proximity.

Solution: Coban upgraded to Kafka 3.1, configured RackAwareReplicaSelector on brokers, and updated the internal Golang SDK to expose a single environment variable that enables closest-replica fetching, setting client.rack dynamically from EC2 instance metadata. Applied to all Coban-managed pipelines and Kafka Connect connectors.

Outcome: consumer cross-AZ traffic cost dropped to near zero. Side effect to watch: end-to-end latency increases up to 500ms for flows that now read from replicas rather than leaders, due to replication lag. Latency-sensitive pipelines are configured to continue leader fetching.

Challenge: Kafka broker crashes on Kubernetes required manual engineer intervention. When EKS worker nodes were unexpectedly terminated (hardware failure, AWS maintenance), the Kafka pod could not restart on the replacement node for two reasons: the NLB target groups still pointed to the dead node, and the Kubernetes PVC remained bound to the now-missing NVMe instance store PV. The cluster ran degraded (two of three brokers) until a Coban engineer manually reconfigured the target groups and deleted the zombie PVC.

Root cause: NVMe instance store volumes are local to the EC2 instance and cannot be reattached to a replacement node. Static Kubernetes PV provisioning meant the PVC stayed bound to the missing volume.

Solution: migrated broker storage from NVMe instance store to AWS EBS gp3 volumes, dynamically provisioned via the EBS CSI driver and managed by Strimzi. Integrated AWS Load Balancer Controller (LBC) for dynamic NLB target group mapping. Added AWS NTH in Queue Processor mode to gracefully drain nodes before termination.

Outcome: unexpected node termination no longer requires engineer intervention. The cluster self-heals automatically.

Challenge: Partition rebalancing took six hours and caused prolonged latency spikes. In traditional Kafka, partition reassignment moves data between broker nodes. During rebalancing events, this caused multi-hour periods of elevated latency and significant operational risk. Over-provisioning based on peak usage also meant significant resource waste during off-peak periods.

Root cause: Kafka’s replication-based storage model couples partition data to specific broker nodes. Moving a partition requires copying potentially gigabytes of data between brokers.

Solution: adopted AutoMQ for a subset of clusters. AutoMQ uses a shared EBS WAL and S3 storage layer, where all brokers read from shared storage. Partition reassignment no longer requires moving data between brokers.

Outcome: partition migration time dropped from six hours to seconds. AutoMQ’s self-balancing mechanism periodically triggers rebalancing without the risk associated with traditional partition movement. 3x throughput improvement reported.

Challenge: Reactive HPA scaling spirals for Flink pipelines. HPA-triggered Flink restarts caused CPU and latency spikes. Those spikes triggered further HPA scale events, creating feedback loops that destabilised pipelines. The problem was compounded by the fact that Flink’s Kafka connector has fixed parallelism bounded by partition count, so horizontal scaling was often impossible, leaving only vertical scaling as an option.

Root cause: reactive scaling based on CPU or latency thresholds can amplify instability rather than resolve it, particularly for Kafka-bound Flink jobs where restart itself causes temporary load spikes.

Solution: ML-based predictive vertical autoscaler. A time-series model forecasts Kafka source topic throughput (which follows seasonal patterns). A regression model maps throughput to required CPU. TaskManager CPU is adjusted ahead of demand spikes.

Outcome: deployed to the majority of applicable Flink pipelines; more than 35% reduction in cloud CPU cost; scaling spirals eliminated.

Challenge: Invalid data silently propagated through Kafka to downstream consumers. Without effective data quality monitoring, bad data would enter Kafka topics and cascade to multiple downstream services before anyone detected the problem. Identifying the source, notifying the right team, and containing the blast radius was slow and manual.

Root cause: no systematic mechanism existed for declaring and testing data contracts on Kafka streams. Monitoring tools tracked throughput and lag but not data validity.

Solution: a data contract system where teams declare schemas and field-level semantic rules. FlinkSQL jobs auto-generated from these contracts consume production topics, detect violations in real time, and route errors to Grab’s observability platform (Genchi), Slack, and S3 sinks. An LLM helps generate semantic rules to reduce the manual definition burden.

Outcome: the system now monitors more than 100 critical Kafka topics and can immediately identify and halt propagation of invalid data across multiple streams.

Challenge: Dual-write inconsistency between MySQL and Kafka. Services were writing state changes to both MySQL and a Kafka topic. Keeping these two writes atomic required two-phase commit, was non-trivial to implement correctly, and impacted service availability. Additionally, some downstream consumers needed before-and-after row snapshots that event-based producers could not easily supply.

Root cause: publishing to Kafka from application code forces producers to solve distributed transaction problems that are not their core concern.

Solution: Debezium CDC connectors running on Kafka Connect. Services write only to MySQL. Debezium reads MySQL binlogs and publishes changes to Kafka, including before-and-after row snapshots. DB schema changes, migrations, and outages are handled via Debezium’s built-in binlog offset management.

Outcome: dual-write eliminated. GrabFood’s Elasticsearch index synchronisation was one of the most significant early adopters.

Tech Stack

Technology	Role
Apache Kafka	Core event log and message bus for all Grab verticals
Kafka Connect	Framework for CDC, message mirroring, and data sink/source connectors
Debezium	MySQL CDC connector on Kafka Connect for row-level change capture
MirrorMaker2	Cross-cluster message and offset mirroring for DR and live migrations
Apache Flink	Stateful stream processing: data quality tests, analytics pipelines, ML feature computation
Apache Zeppelin	Interactive notebook interface for stream data exploration
Apache Pinot	Real-time OLAP; ingests from Kafka for Partner Gateway metrics dashboards
Strimzi	CNCF Kafka on Kubernetes operator; mTLS, rolling upgrades, certificate management
Kubernetes / Amazon EKS	Container orchestration for Kafka brokers and consumer pipelines
AWS EBS gp3	Persistent storage for Kafka broker data volumes
AWS NLB + Load Balancer Controller	Per-broker network routing; dynamic target group mapping
AWS Node Termination Handler	Graceful Kafka broker shutdown on node termination events
Cruise Control	Partition leader rebalancing and cluster metrics
Zookeeper	Kafka cluster coordination
HashiCorp Vault	PKI engine for ephemeral mTLS certificate issuance
Open Policy Agent	Topic-level Kafka authorisation (least-privilege ACLs)
Protobuf + Confluent Schema Registry	Message serialisation, schema versioning, and format conversion
Terraform + Atlantis	IaC provisioning for all streaming resources
Golang Kafka SDK	Internal SDK encapsulating mTLS, AZ rack-awareness, and VPA config
AutoMQ	Kafka-compatible broker with EBS WAL + S3 shared storage for elastic clusters
Apache Hudi	Data lake table format; target sink for Kafka Connect pipelines
AWS S3	Object storage for data lake ingestion and AutoMQ storage layer
Amazon SQS + EventBridge	NTH event routing for graceful broker shutdown
React (Coban UI)	Self-service web portal for topic, pipeline, and connector management

Key Contributors

Name	Title / Team	Contribution
Karan Kamath	Coban team	“Plumbing at Scale” (2020); “Optimally Scaling Kafka Consumer Applications” (2020); “How Kafka Connect helps move data seamlessly” (2022)
Fabrice Harbulot	Coban, Platform Engineering	“Exposing a Kafka Cluster via a VPC Endpoint Service” (2022); “Zero trust with Kafka” (2022); “Zero traffic cost for Kafka consumers” (2023); “An Elegant Platform” (2023); “Kafka on Kubernetes: Reloaded for fault tolerance” (2023)
Quang Minh Tran	Coban	Co-author: “Zero traffic cost for Kafka consumers” (2023)
Thang Le	Coban	Co-author: “Kafka on Kubernetes: Reloaded for fault tolerance” (2023)
Minh Khoi Nguyen	Coban	Co-author: “An Elegant Platform” (2023)
Wenli Wan	Coban	Co-author: “How Kafka Connect helps move data seamlessly” (2022)
Thanh Tung Dao	Coban	Co-author: “How Kafka Connect helps move data seamlessly” (2022); “Zero trust with Kafka” (2022)
Shubham Badkur	Coban	“Optimally Scaling Kafka Consumer Applications” (2020)
Yuanzhe Liu	Coban	Lead author: real-time data quality monitoring (2025)

Sources

#	URL	Author	Publication	Date
1	https://engineering.grab.com/plumbing-at-scale	Karan Kamath	Grab Engineering Blog	Jan 2020
2	https://engineering.grab.com/optimally-scaling-kafka-consumer-applications	Shubham Badkur	Grab Engineering Blog	Oct 2020
3	https://engineering.grab.com/exposing-kafka-cluster	Fabrice Harbulot	Grab Engineering Blog	Feb 2022
4	https://engineering.grab.com/kafka-connect	Wenli Wan, Karan Kamath, Thanh Tung Dao	Grab Engineering Blog	Apr 2022
5	https://engineering.grab.com/zero-trust-with-kafka	Fabrice Harbulot, Thanh Tung Dao	Grab Engineering Blog	Dec 2022
6	https://engineering.grab.com/zero-traffic-cost	Fabrice Harbulot, Quang Minh Tran	Grab Engineering Blog	Jul 2023
7	https://engineering.grab.com/an-elegant-platform	Fabrice Harbulot, Minh Khoi Nguyen	Grab Engineering Blog	Nov 2023
8	https://engineering.grab.com/kafka-on-kubernetes	Fabrice Harbulot, Thang Le	Grab Engineering Blog	Dec 2023
9	https://engineering.grab.com/pinot-partnergateway-tech-blog	Grab Engineering	Grab Engineering Blog	2024
10	https://engineering.grab.com/ml-predictive-autoscaling-for-flink	Grab Engineering	Grab Engineering Blog	Oct 2025
11	https://www.automq.com/blog/how-grab-uses-automq-solve-kafka-challenges	AutoMQ Team	AutoMQ Blog	Sep 2025
12	https://www.infoq.com/news/2025/12/grab-kafka-data-quality/	InfoQ Staff	InfoQ	Dec 2025

‍

If you operate Kafka in production and want visibility into consumer lag, broker health, topic throughput, and rebalancing events without writing your own tooling, take a look at Kpow for Apache Kafka. It’s built for engineering teams who need operational control without the overhead.

How JPMorgan uses Apache Kafka in production

Factor House — Sat, 30 May 2026 00:00:00 GMT

JPMorgan Chase runs one of the largest disclosed Kafka deployments in financial services: 102 clusters, 510 nodes, 13,000 topics, and 1.5 PB of configured storage — figures the firm’s engineers shared publicly at Kafka Summit San Francisco in 2019. By 2021, that platform was ingesting 400 billion events per day in production.

The engineering problem at the centre of the deployment is coordination at firm scale: tens of thousands of applications across business lines, geographies, and regulatory boundaries, all needing reliable, low-latency event exchange without each team managing its own messaging infrastructure.

Company overview

JPMorgan Chase & Co. is a global financial services firm operating across investment banking, commercial banking, financial transaction processing, asset management, and retail banking. The firm employs more than 300,000 people and processes trillions of dollars in transactions annually, making data consistency, latency, and auditability first-order engineering constraints.

Apache Kafka entered JPMorgan Chase’s platform as part of a broader shift toward microservices and event-driven architecture. The firm’s CTO, Andrew J. Lang, described the goal as creating “a digital nervous system” connecting disconnected and siloed systems at scale. By January 2019, the firm publicly recognised Confluent’s Kafka distribution as part of its Hall of Innovation programme, confirming the Confluent Platform as the foundation of its streaming infrastructure.

Date	Event
January 2019	Confluent inducted into JPMorgan Chase Hall of Innovation, publicly confirming the Confluent Platform partnership. CTO Andrew J. Lang quoted in press release.
2019	Kafka Summit San Francisco: engineers Vishnu Balusu and Ashok Kadambala present the multi-tenant managed Kafka service, disclosing 102 clusters, 510 nodes, 13,000 topics, and 1.5 PB of storage.
2021	Kafka Summit Americas: Ashok Kadambala and Shreesha Hebbar present the Managed Kafka Connect architecture, citing 400 billion incoming events per day in production.

JPMorgan Chase’s Kafka use cases

JPMorgan Chase’s Kafka adoption started with straightforward infrastructure concerns and expanded into a firm-wide event backbone as the platform matured.

Log aggregation and metrics transport were the earliest use cases. Before the platform broadened in scope, teams used Kafka to consolidate operational logs and carry metadata and monitoring signals across services — a pattern that provided enough operational value to justify the investment before more complex streaming architectures were in place.

Microservices event bus is now the primary use case at firm scale. JPMorgan Chase operates approximately 8,000 microservices built on an internal framework called Photon. Kafka acts as the communications layer between those services, with Photon providing client-side resiliency on top of the open-source Kafka drivers: automatic consumer failover and a “phone-home” mechanism that reports runtime configurations (including batch size) to simplify production diagnosis.

Near real-time stream processing and data moving pipelines expanded Kafka’s role beyond service-to-service messaging. The firm uses Kafka to move data across cloud environments and datacentres in near real-time, supporting fan-in (many producers, centralised processing) and fan-out (one source, many downstream consumers) patterns. By 2021, this also encompassed data mesh architecture, with Kafka Connect serving as the data movement layer between domain-owned data stores.

Tenant-level mirroring extends Kafka’s reach across datacentres for teams that need their topics replicated geographically, either for disaster recovery or to serve consumers in different regions.

Scale and throughput

The scale figures JPMorgan Chase shared at Kafka Summit 2019 give a concrete picture of what “enterprise Kafka” looks like in a regulated financial institution.

Metric	Value	As of
Total clusters	102 (40 production)	2019
Total nodes	510	2019
Topics (total / production)	13,000 / 1,300	2019
Connected applications	400	2019
Configured storage	1.5 PB	2019
Incoming events per day	400 billion	2021
Kafka version	Apache Kafka 2.2.1 (Confluent 5.2.2)	2019

‍

The ratio of total to production clusters (102 to 40) reflects the firm’s multi-environment discipline: non-production clusters for development, testing, and staging are tracked and governed under the same platform rather than being managed ad hoc by individual teams. The gap between total topics (13,000) and production topics (1,300) tells a similar story: provisioning is self-service and relatively unconstrained, but production promotion is controlled.

JPMorgan Chase’s Kafka architecture

Cluster topology and multi-availability-zone design

Each Kafka cluster runs on five nodes with a replication factor of four. This configuration tolerates the simultaneous failure of two nodes, which matters in a regulated environment where cluster availability affects downstream financial processes. Every node in a cluster runs a dedicated ZooKeeper ensemble alongside the Kafka brokers, Schema Registry instances, and monitoring agents, keeping coordination local to the cluster rather than shared across the estate.

For clusters requiring higher fault tolerance, JPMorgan Chase uses a 2.5 AZ stretch cluster configuration: brokers are organised into four logical racks across two availability zones (two racks per AZ), with replication factor four and a minimum in-sync replica count of three. ZooKeeper quorum is placed in a separate region to ensure it remains available even if one AZ is lost. This setup provides continued write availability during a single AZ failure without requiring manual intervention.

Multi-tenancy and the control plane

Running 100+ clusters for 400 applications across a large enterprise requires a systematic approach to isolation and governance. JPMorgan Chase addresses this through logical namespace-based multi-tenancy: each application team gets a namespace that abstracts the underlying physical cluster. Entitlements, governance policies, and quota limits are enforced at the namespace level, while the physical infrastructure is shared.

A centralised, data-driven control plane sits above the clusters and provides a self-service API layer for topic provisioning, schema registration, Kafka Streams deployment, and quota management. All Kafka artifacts created by an application are maintained within that application’s namespace, making ownership explicit. The control plane also exposes a centralised data lineage view derived from the namespace metadata.

Kafka Connect architecture

JPMorgan Chase built a managed Kafka Connect service to handle near real-time data integration between Kafka and external systems. The design centres on a non-obvious deployment decision: instead of hosting Kafka Connect instances on shared provider infrastructure, the firm deploys Connect instances within each customer’s own Kubernetes namespace.

This placement solves the credential problem. When a Connect instance sits on shared infrastructure, source and sink systems must expose their credentials to the service provider. Deploying Connect into the customer namespace means the connectors authenticate to source and sink systems using the customer’s own credentials, with no cross-boundary credential sharing required. Each deployment gets a unique HTTPS URL (for example, connect1.jpmchase.net), along with dedicated configuration, offset, and status topics.

A centralised and federated control plane manages all Connect deployments across namespaces, keeping operational visibility consistent even as the deployment boundary sits inside each team’s Kubernetes environment.

Multi-datacenter replication

JPMorgan Chase uses MirrorMaker 2.0 for cross-datacenter topic replication. Both active-active and active-passive configurations are in use, with configurable topic patterns determining which topics are replicated and in which direction. Active-active replication supports geographically distributed consumers; active-passive configurations serve disaster recovery scenarios where a standby cluster needs to stay current.

Special techniques and engineering innovations

Connection Profiles abstract cluster addressing from application configuration. Rather than configuring a Kafka client with a list of broker addresses, applications reference a profile name (such as “NAD1700”). The control plane resolves that profile to the current broker addresses at runtime. When infrastructure changes — broker replacements, cluster migrations, IP reassignments — applications require no reconfiguration. The profile name is the stable handle.

Health Index is a metrics-based cluster availability score computed from broker and partition health signals and fed into Prometheus. When a cluster’s Health Index falls below a threshold, it is excluded from routing, and application-level resiliency controls react accordingly. This creates an automated circuit-breaker layer that sits between raw Kafka metrics and consumer behaviour without requiring each application team to implement their own health logic.

Custom Schema Registry authorisation extension adds operation-level access control to the Schema Registry REST API. A REST extension intercepts incoming requests and separates read operations (GET) from write operations (POST, PUT, DELETE), enforcing role-appropriate permissions before requests reach the schema storage layer. This prevents teams from unintentionally overwriting schemas owned by other namespaces while keeping read access open for discovery.

Federated ADFS OAuth integrates JPMorgan Chase’s existing Active Directory Federation Services identity provider with Kafka’s pluggable SASL/OAUTHBEARER authentication (KIP-255). Rather than managing a separate identity system for Kafka, the firm routes authentication through its existing corporate identity infrastructure, which simplifies credential lifecycle management and satisfies audit requirements around identity traceability.

Orchestrated broker patching replaces manual rolling restarts with an automated lifecycle management system. The process sequences broker updates, validates cluster health between each step, and monitors for under-replicated partitions before proceeding. If a step introduces replication lag or partition leadership instability, patching pauses until the cluster recovers, reducing the risk of data unavailability during routine maintenance.

Photon Framework Kafka client resiliency addresses gaps in the open-source Kafka client libraries for consumer failover scenarios. Photon, the internal Spring-Boot microservices framework, wraps the Kafka client with automatic failover logic and a “phone-home” capability that surfaces runtime configuration values — including batch size settings that commonly affect consumer throughput — for use in production diagnosis without requiring a service restart.

Operating Kafka at scale

JPMorgan Chase runs Confluent Platform on-premises in a hybrid configuration. The operational model is built around the centralised control plane rather than per-cluster management.

Self-service provisioning is the default path for teams. Topic creation, schema registration, quota adjustments, and Kafka Streams deployment all go through the control plane’s self-service API, keeping infrastructure teams out of routine provisioning workflows. Governance constraints — topic size limits, throughput quotas, namespace isolation — are encoded in the control plane and enforced automatically, so teams work within boundaries without needing to understand the physical cluster topology.

Metrics and observability rely on Prometheus for data collection. The Health Index pipeline feeds cluster-level signals into Prometheus, and the Photon Framework produces standard metrics and distributed traces across all Kafka producers and consumers. This gives platform operators a consistent observability interface across the 8,000 microservices that use Kafka, rather than requiring each team to instrument their Kafka clients independently.

Tenant quota enforcement is per-namespace: topic retention limits and producer/consumer throughput quotas are set at the namespace level and enforced by the control plane. All Kafka artifacts are owned by the namespace that created them, making quota attribution and governance auditing straightforward.

Managed Kafka Connect operations are handled through a federated control plane that spans all Connect deployments across customer namespaces. Each deployment has its own scoped configuration, offset, and status topics, so an issue in one deployment’s offset management does not affect others. The federated control plane provides operators with a consistent view of all Connect deployments without requiring direct access to each customer namespace.

Challenges and how they solved them

The Kafka Connect credential problem. When Kafka Connect runs on shared infrastructure, connectors that need to read from or write to source and sink systems must present credentials to a service provider outside the team’s trust boundary. For a regulated financial institution, this is unacceptable: credentials for core banking systems cannot be handed to a shared service operator. JPMorgan Chase’s resolution was to shift the deployment boundary: Connect instances run inside the customer’s Kubernetes namespace, so all authentication to source and sink systems uses credentials that remain within the customer’s control perimeter throughout.

Multi-tenant Kafka Streams application ID collisions. In a shared Kafka environment, Kafka Streams applications identify themselves using an application ID, which is also used to namespace internal topics (repartition and changelog topics). If two teams in different namespaces choose the same application ID, their internal topics collide. JPMorgan Chase identified this as an operational risk in a multi-tenant deployment at their scale. The control plane’s namespace model and self-service API provide the framework for enforcing application ID uniqueness within and across namespaces, though the specific enforcement mechanism was not detailed in public presentations.

Offset management across replicated clusters. Active-passive replication with MirrorMaker 2.0 introduces a consumer group offset synchronisation challenge: offsets in the primary cluster do not translate directly to offsets in the replica cluster, because topic partition assignments and log segment layouts may differ. JPMorgan Chase identified this as an operational complexity at their scale, particularly for clusters used in disaster recovery scenarios where consumers may need to resume from a specific position in the replica.

Operational burden at 100+ clusters. Managing individual clusters becomes untenable at the scale JPMorgan Chase operates. Monitoring, patching, quota enforcement, and health assessment across 100 clusters requires automation rather than manual workflows. The Health Index, centralised control plane, and orchestrated patching pipeline were all built in response to this operational pressure rather than as upfront design decisions.

Full tech stack

Category	Technology	Role
Message broker	Apache Kafka 2.2.1 via Confluent 5.2.2	Core distributed event streaming platform (version as of 2019)
Distribution	Confluent Platform (Enterprise)	Managed Kafka distribution providing Schema Registry, Kafka Connect, Control Center, and enterprise security features
Cluster coordination	ZooKeeper	Per-cluster coordination ensemble, co-located with each Kafka cluster
Schema management	Confluent Schema Registry	Schema registration and evolution, extended with a custom REST authorisation extension separating read and write access by role
Replication	MirrorMaker 2.0	Multi-datacenter topic replication in active-active and active-passive configurations
Data integration	Kafka Connect (Managed)	Near real-time data pipelines deployed in customer Kubernetes namespaces for fan-in, fan-out, and multi-cloud integration
Container runtime	Kubernetes	Runtime environment for Managed Kafka Connect instances, scoped per customer namespace
Metrics	Prometheus	Metrics collection from the Health Index pipeline and cluster monitoring agents
Authentication	Kerberos	Broker authentication for on-premises cluster environments
Identity federation	SASL/OAUTHBEARER (KIP-255) with ADFS	Federated identity via Active Directory Federation Services, integrated with Kafka’s pluggable authentication
Microservices framework	Photon Framework (internal)	Spring-Boot-based framework used by approximately 8,000 services; provides Kafka client resiliency, automatic failover, distributed tracing, and standardised metrics

Key contributors

Name	Title / team	Contribution
Vishnu Balusu	Managing Director, Cloud Services	Co-presenter at Kafka Summit SF 2019: “Secure Kafka at scale in true multi-tenant environment”
Ashok Kadambala	Global Messaging / Streaming Engineering	Co-presenter at Kafka Summit SF 2019 and Kafka Summit Americas 2021; architect of the managed Kafka and Managed Kafka Connect services
Shreesha Hebbar	Engineering Manager, Enterprise Messaging / Streaming	Co-presenter at Kafka Summit Americas 2021: “Changing landscapes in data integration — Kafka Connect for near real-time data moving pipelines”
Andrew J. Lang	Chief Technology Officer	Quoted in the Confluent Hall of Innovation press release on the role of the Confluent Platform in the firm’s event streaming strategy
Haiying Guo	Tech Partner	Co-author of the Next at Chase article describing the Photon Framework and its Kafka integration
Vidya Meyyappan	Product Manager, New Banking Architecture Delivery Platform	Co-author of the Next at Chase article describing the Photon Framework and its Kafka integration

Key takeaways for your own Kafka implementation

Separate your deployment boundary from your management boundary. JPMorgan Chase deploys Kafka Connect inside customer namespaces but manages all deployments from a centralised control plane. This keeps credentials within the team’s trust perimeter while preserving operational visibility for the platform team — a pattern worth considering any time a shared service needs access to systems that require privileged credentials.
Build indirection into client configuration early. Connection Profiles — stable name references that resolve to broker addresses at runtime — mean that infrastructure changes (IP changes, broker migrations, cluster splits) have no impact on application configuration. If you hard-code broker addresses into application configs today, you will pay the cost of that decision at every infrastructure change.
Encode governance as code in the control plane, not in process. Topic size limits, throughput quotas, and namespace isolation enforced automatically by a control plane are more reliable than guidelines enforced by review. At scale, the gap between what teams are supposed to do and what they actually do widens; a control plane that makes the wrong thing impossible is more durable than one that makes the right thing easy.
Design your cluster health model before you need it. JPMorgan Chase’s Health Index was built in response to the operational burden of managing 100+ clusters. A computed availability score that drives routing decisions is more useful than raw metrics alone, but it requires you to define what “healthy enough to route to” means for your workloads before you are in a position where that definition matters urgently.
Multi-tenant Kafka Streams requires namespace-scoped application ID enforcement. Application ID collisions in a shared cluster cause internal topic conflicts that are difficult to diagnose and resolve without downtime. If you are offering Kafka Streams as a shared platform capability, enforce application ID uniqueness at the provisioning layer rather than relying on teams to coordinate independently.

Sources and further reading

Vishnu Balusu and Ashok Kadambala (JP Morgan Chase): “Secure Kafka at scale in true multi-tenant environment” — Kafka Summit San Francisco 2019
Ashok Kadambala and Shreesha Hebbar (JP Morgan Chase): “Changing landscapes in data integration — Kafka Connect for near real-time data moving pipelines” — Kafka Summit Americas 2021
Confluent: “Confluent inducted into JPMorgan Chase Hall of Innovation” — January 2019
Haiying Guo and Vidya Meyyappan (JP Morgan Chase): “Driving native cloud adoption at scale through a microservice framework” — Next at Chase, 2021

If you are managing a Kafka deployment at a similar scale and want visibility into consumer lag, broker health, and topic configuration across your clusters, Kpow gives you a single interface for monitoring and managing Apache Kafka. You can connect it to any Kafka cluster in minutes and try it free for 30 days.

How LinkedIn uses Apache Kafka in production

Factor House — Sat, 30 May 2026 00:00:00 GMT

LinkedIn built Apache Kafka internally in 2010 to solve the problem of moving large volumes of event data reliably between internal systems. By 2019 the company was processing more than 7 trillion messages per day across over 100 clusters, 4,000 brokers, and 7 million partitions - one of the largest Kafka deployments publicly documented. Kafka connects virtually every system at LinkedIn, from member activity tracking and real-time search indexing to database replication and stream processing at scale.

Company overview

LinkedIn is a professional networking platform with over one billion members. Its products span hiring, marketing, learning, and news, and almost all of its personalisation, recommendations, and analytics workloads are underpinned by data pipelines.

Kafka was developed at LinkedIn in 2010 by Jay Kreps, Neha Narkhede, and Jun Rao. The motivation was straightforward: LinkedIn needed a unified, durable, high-throughput channel to move event data between its growing set of internal systems. It was open-sourced in June 2011 and donated to the Apache Software Foundation.

LinkedIn Kafka milestones:

Date	Event
2010	Apache Kafka developed internally at LinkedIn
June 2011	Kafka open-sourced and released to the Apache community
July 2011	Kafka processing 1 billion messages per day in production
2012	20 billion messages per day
July 2013	200 billion messages per day
2013	Apache Samza open-sourced by LinkedIn
2015	1 trillion messages per day; peak of 4.5 million messages per second; Burrow open-sourced
2016	Approximately 1.4 trillion messages per day; approximately 1,400 brokers; more than 2 PB per week
August 2017	Kafka Cruise Control open-sourced
2018	Migrated from Kafka MirrorMaker to Brooklin for all cross-cluster replication
2019	7 trillion messages per day; Brooklin and Cruise Control Frontend open-sourced
April 2022	Load-balanced Brooklin Mirror Maker published; Microsoft Azure migration paused
November 2022	TopicGC published
2023	Apache Beam adopted to unify batch and streaming pipelines

LinkedIn’s Kafka use cases

Kafka at LinkedIn is not limited to a single team or product area. It serves as the central data transport layer across the company.

Activity tracking is Kafka’s original use case at LinkedIn. The platform captures member activity events - pageviews, search queries, and ad impressions - and delivers them to both offline batch analytics pipelines (Hadoop) and real-time online services.

Real-time search indexing uses Kafka to deliver network-update events to LinkedIn’s search engine. According to the 2011 engineering post by Jay Kreps, updates become searchable within seconds of being posted.

Inter-service messaging runs asynchronously over Kafka topics, connecting LinkedIn’s microservices without direct coupling between them.

Database replication (Espresso CDC) placed Kafka on a latency-sensitive, mission-critical path. LinkedIn replaced MySQL replication with a Kafka-backed CDC pipeline for Espresso, its internal NoSQL document store, requiring no-data-loss guarantees and low transfer latencies.

Derived-data upload to Venice uses Kafka to asynchronously upload data into LinkedIn’s derived-data serving store, decoupling the write path from downstream consumers.

Stream processing with Samza relies entirely on Kafka as both input and output. Apache Samza, which LinkedIn open-sourced in 2013, consumes from and produces to Kafka for all real-time processing jobs. Samza also uses log-compacted Kafka topics as durable state backup stores.

Mobile event ingestion arrives via a REST interface built around the LiKafka client, allowing mobile devices and non-Java services to produce events into Kafka reliably without depending on the native Java client.

Cross-cluster and cross-datacenter aggregation is handled by Brooklin, which mirrors Kafka topics between clusters and datacenters to support centralised analytics and fault tolerance.

Log and metrics storage treats Kafka as the circulatory system of LinkedIn’s infrastructure, routing log data and system metrics between internal services.

Scale & throughput

LinkedIn’s Kafka footprint grew by a factor of approximately 1,200 in four years:

July 2011: 1 billion messages per day
2012: 20 billion messages per day
July 2013: 200 billion messages per day
2015: 1 trillion messages per day, with a peak throughput of 4.5 million messages per second

By 2019, LinkedIn had surpassed 7 trillion messages per day, spread across more than 100 clusters, 4,000 brokers, 100,000 topics, and 7 million partitions. Each message was consumed by approximately four applications on average.

Data volume reached 1.34 PB per week at the 1 trillion messages per day milestone in 2015. With approximately 1,400 brokers in a later configuration, LinkedIn received over 2 PB of data per week.

Brooklin, used for cross-cluster replication, was itself mirroring more than 7 trillion messages per day as of 2022.

The maximum message size is capped at 1 MB.

LinkedIn’s Kafka architecture

LinkedIn’s Kafka deployment is self-managed and multi-datacenter. The company operates its own data centers across US locations and Singapore.

Cluster topology

Each datacenter runs multiple Kafka clusters, each containing an independent set of data with distinct purposes. Activity tracking, Espresso replication, and metrics are kept in separate clusters. This isolation limits the blast radius of failures and allows cluster configurations to be tuned per workload.

The overall topology is tiered: producers write to a local Kafka cluster in their datacenter, data is replicated to aggregate clusters that consolidate events across datacenters, and consumers read from local copies. Data is copied only once per datacenter.

Cross-datacenter replication

LinkedIn replaced Kafka MirrorMaker (KMM) with Brooklin Mirror Maker (BMM) in 2018. A single Brooklin cluster handles hundreds of simultaneous datastreams, whereas the KMM approach required a separate cluster for each pipeline - an arrangement that became operationally unsustainable at LinkedIn’s scale.

Kafka release branches

LinkedIn maintains internal release branches of Apache Kafka, suffixed -li, that are kept close to upstream. These are not a fork: LinkedIn submits patches upstream via Kafka Improvement Proposals (KIPs), either before cherry-picking them into the LinkedIn branch or shortly after committing a hotfix internally. New internal releases are cut approximately every quarter.

Schema management

All data pipelines at LinkedIn are standardised on Apache Avro. The LiKafka client handles schema registration with a central Schema Registry, Avro encoding and decoding, and auditing automatically on both producer and consumer sides. Consumers receive schemas without needing to manage them explicitly.

Producer architecture

LinkedIn’s LiKafka client wraps the standard Kafka producer and adds schema registration, auditing, and large message support. Quota enforcement - per-producer bandwidth limits in bytes per second - is applied at the broker level to prevent any single application from saturating a cluster. Quotas can be exceeded by whitelisted users, and configuration changes take effect without a rolling broker restart.

Consumer architecture

Each Kafka message is consumed by approximately four applications on average. Offset management and consumer lag monitoring are handled through Burrow (described under Operating Kafka at scale). Consumer groups and lag are evaluated across every partition using a sliding-window model rather than static threshold comparisons.

Stream processing

Apache Samza handles all real-time stream processing at LinkedIn and depends on Kafka both as the event source and as a durable state store. Samza backs up stream processor state to log-compacted Kafka topics, enabling recovery after disk failures without replaying the entire topic history.

Kafka Connect ecosystem

LinkedIn built Gobblin, a Kafka-to-Hadoop ingestion framework, to continuously copy Kafka data into HDFS for offline analytics. Gobblin replaced an earlier framework called Camus. Brooklin also serves as a streaming bridge between Kafka and external systems including Azure Event Hubs and AWS Kinesis.

Special techniques & engineering innovations

Time-based replica lag thresholds: LinkedIn changed the Kafka replica lag calculation from a bytes-behind threshold to a time-based threshold. The bytes model caused large messages to incorrectly mark replicas as out of sync during normal operation. Switching to time prevents false positives from message size spikes.

Rack-aware replica placement: Replicas for a partition are placed on brokers in different datacenter racks. This prevents a top-of-rack switch failure from taking all replicas for a partition offline simultaneously.

TopicGC - automated unused topic garbage collection: LinkedIn built an internal service called TopicGC to detect unused Kafka topics, seal them (blocking reads and writes), disable mirroring, and delete them in batches of up to three concurrent deletions. A final usage check runs before each deletion step. In one of LinkedIn’s largest data pipelines, TopicGC reduced the topic count by approximately 20% and improved produce and consume performance by at least 30%.

Kafka as Espresso CDC backbone: Replacing MySQL replication with Kafka for Espresso required significantly stronger delivery guarantees than typical event streaming. LinkedIn tuned for no-data-loss delivery while maintaining the low transfer latencies Espresso’s online traffic requires.

LiKafka REST service for non-Java clients: The original REST interface had no guaranteed delivery. LinkedIn redesigned it so an event is only considered consumed after successful delivery to the destination topic, enabling reliable mobile and non-Java ingestion.

Load-balanced Brooklin partition assignment: The default Brooklin partition assignment distributed partitions by count rather than throughput. LinkedIn migrated Brooklin’s Kafka connectors from high-level consumer APIs to manual partition assignment APIs and introduced a throughput-based assignment strategy, reducing replication latency imbalance.

Kafka Audit - end-to-end message completeness verification: LiKafka clients emit per-topic message counts in 10-minute windows to a dedicated audit topic. The Kafka Audit Service aggregates counts from producers, each tier of the cluster topology, and critical consumers such as Hadoop, verifying that no messages are lost or duplicated across the pipeline.

Operating Kafka at scale

Deployment model: LinkedIn runs Kafka on its own hardware across multiple datacenters. As of early 2022, a planned migration to Microsoft Azure was paused.

Consumer lag monitoring with Burrow: Burrow evaluates consumer group health across every partition using a sliding window of offset commits - approximately 10 commits, representing roughly 10 minutes of lag history at LinkedIn’s typical commit rate. Rather than alerting on a fixed lag threshold, Burrow assesses the direction and rate of change of consumer lag to produce a health status: good, degraded, or stopped. It is exposed via an HTTP API and was open-sourced in 2015.

Cluster rebalancing with Cruise Control: Broker failures occur at LinkedIn’s scale on a daily basis. Manual SRE intervention for partition reassignment was not sustainable. Cruise Control continuously monitors disk, network, and CPU utilisation across all brokers and automatically reassigns partitions to meet pre-defined performance goals. It also handles self-healing on broker failure. LinkedIn open-sourced Cruise Control in August 2017. The Cruise Control Frontend (CCFE), a web dashboard for managing all Kafka clusters, was open-sourced in 2019.

Live availability monitoring with Kafka Monitor: Kafka Monitor runs synthetic producer and consumer applications against live clusters to continuously validate ordering, delivery, and data integrity guarantees and measure end-to-end latencies. It is also used to qualify new Kafka builds before they reach production.

Self-service topic management with Nuage: LinkedIn’s Nuage portal gives teams self-service access to Kafka topic lifecycle operations, including topic creation, deletion, configuration changes, metadata browsing, and ACL management. It delegates operations to the Kafka REST service.

Usage attribution with Bean Counter: Bean Counter tracks megabytes sent and received per application, attributing Kafka infrastructure costs to individual teams and providing an incentive for responsible usage.

Chaos testing with Simoorg: LinkedIn integrates Simoorg, its open-source failure inducer, into Kafka release validation. Simoorg introduces low-level failures such as dropped packets, low memory, failed disk writes, and process kills to verify that new Kafka builds behave correctly under real failure conditions.

Quarterly release cadence: LinkedIn cuts a new internal Kafka release approximately every quarter, tracking open-source trunk closely and upstreaming patches where possible.

Challenges & how they solved them

Kafka MirrorMaker data loss on upgrades and reboots

‍Root cause: The original KMM design could lose messages when a cluster was upgraded or a machine rebooted, because the delivery contract did not guarantee persistence before marking a message as consumed.

‍Solution: LinkedIn first redesigned the delivery contract so a message is only marked consumed after successful delivery to the destination. Later, in 2018, LinkedIn replaced all KMM clusters with Brooklin, which also eliminated the operational overhead of managing hundreds of separate pipeline-specific clusters.

‍Outcome: Reliable cross-cluster replication at scale, with a single Brooklin cluster handling hundreds of simultaneous datastreams.

ZooKeeper controller initialisation bottleneck from metadata bloat

‍Root cause: Accumulation of unused Kafka topics inflated ZooKeeper response payloads. For LinkedIn’s largest cluster, the ZooKeeper response size had reached 0.75 MB, close to the 1 MB limit that would trigger availability issues.

‍Solution: LinkedIn built TopicGC to automatically detect and delete unused topics in a controlled sequence, with a last-minute usage check before each deletion step.

‍Outcome: Topic count reduced by approximately 20% in one of LinkedIn’s largest pipelines; produce and consume performance improved by at least 30%.

Unbalanced cluster load and partition skew after broker failures

‍Root cause: Broker failures at LinkedIn’s scale occur daily. Manual partition reassignment by SREs was not a sustainable response.

‍Solution: LinkedIn built Cruise Control, which continuously monitors cluster resource utilisation and automatically reassigns partitions to meet disk, network, and CPU goals.

‍Outcome: Automated self-healing on broker failure; Cruise Control open-sourced and adopted across the industry.

Uneven load distribution across Brooklin replication tasks

‍Root cause: Brooklin’s original partition assignment distributed partitions by count rather than by throughput, causing some tasks to process significantly more data than others.

‍Solution: LinkedIn migrated to manual partition assignment APIs and implemented a throughput-based partition assignment strategy.

‍Outcome: Reduced replication latency imbalance across Brooklin tasks.

False replica out-of-sync alerts caused by large messages

‍Root cause: The bytes-of-lag threshold used to determine replica health could be exceeded by a single large message, incorrectly marking a healthy replica as out of sync.

‍Solution: LinkedIn switched the replica lag calculation to a time-based threshold.

‍Outcome: Fewer false positive alerts and more accurate replica health reporting.

Full tech stack

Category	Tools	Notes
Message broker	Apache Kafka (-li branches)	LinkedIn-internal release branches kept close to upstream; patches submitted via KIPs
Schema registry	LinkedIn Schema Registry	Central Avro schema store; integrated with LiKafka client on both producer and consumer sides
Serialisation	Apache Avro	Mandatory standard across all LinkedIn data pipelines
Kafka client	LiKafka	Custom client wrapping the OSS Kafka producer and consumer; adds schema management, auditing, and large message support
Stream processing	Apache Samza, Apache Beam, Apache Spark	Samza for real-time; Beam adopted in 2023 for unified batch and streaming; Spark for petabyte-scale batch
Cross-cluster replication	Brooklin (Brooklin Mirror Maker)	Replaced Kafka MirrorMaker in 2018; also bridges Kafka to Azure Event Hubs and AWS Kinesis
Kafka-to-Hadoop ingestion	Gobblin	Replaced Camus; continuously copies Kafka data to HDFS
Storage and serving	Espresso, Venice, Hadoop/HDFS	Espresso (NoSQL, CDC via Kafka), Venice (derived-data store), HDFS (offline analytics)
Cluster rebalancing	Kafka Cruise Control, Cruise Control Frontend (CCFE)	Automated partition rebalancing and self-healing; CCFE provides a central web dashboard
Consumer lag monitoring	Burrow	Sliding-window lag analysis; HTTP API; open-sourced 2015
Live cluster health	Kafka Monitor	Synthetic produce/consume tests; used in release qualification
Topic lifecycle	Nuage, TopicGC	Nuage: self-service portal for topic management; TopicGC: automated garbage collection of unused topics
Usage attribution	Bean Counter	Per-application MB tracking for cost attribution
Chaos testing	Simoorg	Open-source failure inducer; integrated into Kafka release validation
Real-time OLAP	Apache Pinot	Serves partition-level metrics collected by Cruise Control
Coordination	Apache ZooKeeper	Kafka broker metadata coordination

Key contributors

Name	Role	Contribution
Jay Kreps	Co-creator of Kafka at LinkedIn	Co-designed and built the original Kafka system; co-authored the 2011 open-source release post
Neha Narkhede	Co-creator of Kafka at LinkedIn	Co-designed and built the original Kafka system; later co-founded Confluent
Jun Rao	Co-creator of Kafka at LinkedIn	Co-designed and built the original Kafka system; later co-founded Confluent
Kartik Paramasivam	Engineering Lead, Kafka team	Authored “How we’re improving and advancing Kafka at LinkedIn” (2015); led quota, consumer, and reliability work
Todd Palino	Data Infrastructure SRE	Built Burrow; authored “Running Kafka at scale”; presented “More data centers, more problems” at Kafka Summit
Joel Koshy	Staff Engineer, Kafka team	Authored “Kafka ecosystem at LinkedIn”; presented “Kafkaesque days at LinkedIn in 2015” at Kafka Summit; led Nuage development
Efe Gencer	Kafka team	Created Cruise Control as an intern project in 2017
Joseph Lin	Streaming Infrastructure team	Co-invented TopicGC; implemented its original design (2022)
Lincong Li	Streaming Infrastructure team	Co-invented TopicGC; implemented its original design (2022)
Clark Haskins	Data Infrastructure SRE	Burrow core developer
Grayson Chao	Data Infrastructure SRE	Burrow core developer
Jon Bringhurst	Data Infrastructure SRE	Burrow core developer

Key takeaways for your own Kafka implementation

Separate clusters by workload type. LinkedIn runs multiple clusters per datacenter rather than a single shared cluster. This isolates failure domains and lets each cluster be tuned for its specific performance and durability requirements.
Automate rebalancing from day one. Manual partition reassignment does not scale. LinkedIn’s experience building Cruise Control demonstrates that continuous automated rebalancing is more reliable than on-demand SRE intervention, particularly as broker failure frequency increases with fleet size.
Monitor consumer lag by behaviour, not by threshold. Burrow’s sliding-window approach evaluates whether consumers are making progress relative to their recent history, rather than alerting when lag exceeds a fixed byte count. This reduces false positives and gives a more accurate picture of consumer health.
Invest in topic lifecycle management. Unused topics accumulate over time and can create real operational problems at scale. TopicGC’s results at LinkedIn - a 20% reduction in topic count and a 30% improvement in performance - demonstrate that proactive topic hygiene is worth automating.
Standardise serialisation across all pipelines early. LinkedIn’s universal adoption of Apache Avro and a central Schema Registry allowed teams to share infrastructure and tooling. Retrofitting schema standards onto an existing pipeline ecosystem is significantly harder.

Sources & further reading

[S1] Jay Kreps, LinkedIn Engineering Blog (2011): https://www.linkedin.com/blog/member/archive/open-source-linkedin-kafka
[S2] Todd Palino, “Running Kafka at scale”, LinkedIn Engineering Blog (March 2015): https://engineering.linkedin.com/kafka/running-kafka-scale
[S3] Joel Koshy, “Kafka ecosystem at LinkedIn”, LinkedIn Engineering Blog (~2016): https://www.linkedin.com/blog/engineering/open-source/kafka-ecosystem-at-linkedin
[S4] Kartik Paramasivam, “How we’re improving and advancing Kafka at LinkedIn”, LinkedIn Engineering Blog (September 2015): https://engineering.linkedin.com/apache-kafka/how-we_re-improving-and-advancing-kafka-linkedin
[S5] LinkedIn Engineering team, “Apache Kafka: more than 1 trillion messages”, LinkedIn Engineering Blog (2019): https://www.linkedin.com/blog/engineering/open-source/apache-kafka-trillion-messages
[S6] LinkedIn Streaming Infrastructure team, “Load-balanced Brooklin Mirror Maker”, LinkedIn Engineering Blog (April 2022): https://engineering.linkedin.com/blog/2022/load-balanced-brooklin-mirror-maker--replicating-large-scale-kaf
[S7] LinkedIn Streaming team, “Brooklin: near real-time data streaming at scale”, LinkedIn Engineering Blog (August 2019): https://engineering.linkedin.com/blog/2019/brooklin-open-source
[S8] Junaid Effendi, “LinkedIn data tech stack”, Substack (October 2024): https://www.junaideffendi.com/p/linkedin-data-tech-stack
[S9] Joseph Lin, Lincong Li, “TopicGC: how LinkedIn cleans up unused Kafka metadata”, LinkedIn Engineering Blog (November 2022): https://engineering.linkedin.com/blog/2022/topicgc_how-linkedin-cleans-up-unused-metadata-for-its-kafka-clu
[S10] Todd Palino, Clark Haskins, Grayson Chao, Jon Bringhurst, “Burrow: Kafka consumer monitoring reinvented”, LinkedIn Engineering Blog (June 2015): https://engineering.linkedin.com/apache-kafka/burrow-kafka-consumer-monitoring-reinvented
[S11] Efe Gencer + Kafka team, “Open-sourcing Kafka Cruise Control”, LinkedIn Engineering Blog (August 2017): https://engineering.linkedin.com/blog/2017/08/open-sourcing-kafka-cruise-control

If you are managing a Kafka deployment of your own, Kpow provides observability, control, and governance for Kafka clusters across self-managed, MSK, and Confluent environments. You can try it free for 30 days.

How PayPal uses Apache Kafka in production

Factor House — Sat, 30 May 2026 00:00:00 GMT

PayPal’s Apache Kafka deployment sits at a scale that most engineering teams will only read about: more than 85 clusters, over 1,500 brokers, and a peak throughput of 1.3 trillion messages per day during Retail Friday 2022 — roughly 21 million messages per second. Kafka underpins nearly every data pipeline at PayPal, from real-time fraud scoring to clickstream ingestion to database synchronisation, and the engineering challenges involved in running it reliably at that volume have shaped both internal tooling and the open-source project itself.

Company overview

PayPal is a global payments platform operating in more than 200 markets. At the time of writing, it processes hundreds of millions of transactions per year across consumer, merchant, and financial-services products including PayPal, Venmo, Braintree, and Xoom.

The company adopted Kafka in 2015, starting with a handful of isolated clusters for specific pipelines. Over the following years, payment volume and product surface area grew steadily, and Kafka evolved from a collection of point-to-point pipelines into a centralised event-streaming fabric used across dozens of teams and use cases.

Key Kafka milestones:

2015: Kafka introduced as isolated clusters for specific use cases
2016: Lambda architecture documented, combining Kafka with Spark Streaming for near-real-time analytics
2018: 400 billion messages per day across 40+ clusters in three geographically distributed data centres
2020: Approaching 1 trillion messages per day; multi-tenant fleet management formalised
2021: Zero-downtime migration of 20+ clusters (1,000+ broker and ZooKeeper nodes, 60+ MirrorMaker groups) across data centres
2022: 1.3 trillion messages per day at Retail Friday peak (21 million messages per second)
2023: Fleet documented at 85+ clusters, 1,500+ brokers, 20,000+ topics, approximately 2,000 MirrorMaker nodes, 99.99% availability

PayPal’s Kafka use cases

PayPal uses Kafka across seven primary use cases, each processing more than 100 billion messages per day.

First-party user behaviour tracking is one of the busiest pipelines. All clickstream and interaction events from PayPal’s web and mobile surfaces are ingested via Kafka and enriched before being distributed to downstream analytics and personalisation systems. The pipeline processes 18.8 billion messages per day in that use case alone, up from 9 billion before a reactive architecture rewrite in 2018.

Application health metrics are streamed through Kafka from brokers, ZooKeeper nodes, MirrorMakers, and Kafka clients themselves. A custom metrics library registers telemetry via Micrometer and forwards it to the SignalFX observability backend.

Database synchronisation uses Kafka as a change-data-capture transport, streaming operational database changes to downstream consumers for near-real-time replication across services.

Application log aggregation consolidates logs from across PayPal’s service estate via Kafka, feeding log-analysis and compliance tooling.

Risk detection and management depends on Kafka to move payment events to fraud-scoring and anomaly-detection models with low latency. Spark Streaming jobs consume directly from Kafka topics to support real-time risk decisions at transaction time.

Analytics and compliance pipelines use Kafka as the backbone for several analytical systems, including a consumer application that ingests 30 to 35 billion daily events from Kafka and publishes them to BigQuery on Google Cloud. Before this pipeline existed, the reporting latency for the underlying data was 12 hours; after the Kafka-based rewrite, it dropped to seconds.

Batch processing supplements the real-time pipelines. Event streams also feed macro-batch Spark jobs used for end-of-day reconciliation and reporting.

Scale and throughput

Metric	Value
Peak daily messages (Retail Friday 2022)	1.3 trillion
Peak message rate	21 million messages per second
Kafka clusters	85+
Brokers	1,500+
Topics	20,000+
MirrorMaker nodes	~2,000
Platform availability	99.99%
Messages per day per primary use case	100 billion+

‍

The growth trajectory is as informative as the peak numbers. PayPal started with isolated clusters in 2015, reached 400 billion messages per day by 2018, crossed 1 trillion by 2021, and then handled a 30% increase on top of that during the 2022 Retail Friday peak. Quarter-on-quarter traffic growth of around 30% was noted by Maulin Vasavada at Kafka Summit 2020, which helps explain why PayPal’s engineering approach has consistently prioritised operational automation over manual intervention.

PayPal’s Kafka architecture

Deployment model

PayPal runs Kafka brokers on bare metal for I/O performance reasons. ZooKeeper and MirrorMaker instances run on virtual machines. The production Kafka fleet spans multiple geographically distributed data centres, with clusters segmented by security zone within each data centre. Cluster placement is governed by data classification requirements and business criticality rather than a single flat topology.

The QA environment is hosted on Google Cloud Platform, with brokers spread across multiple GCP availability zones. The GCP QA environment maps directly to production cluster topology, follows the same security standards, and cost 75% less than the previous on-premises QA setup while delivering 40% better performance.

Kafka Config Service

One of the more consequential internal tools at PayPal is the Kafka Config Service: a highly available, stateless service that pushes bootstrap server addresses and standardised client configurations to applications. Rather than hard-coding broker IPs in application configs, each application queries the Config Service at startup.

This decouples application deployments from broker topology changes. When clusters are rebalanced, scaled, or reorganised, the Config Service absorbs the change and applications continue connecting without redeployment. It also reduces the operational burden on the SRE team, since broker address updates do not require coordinating application-side config changes across hundreds of teams.

Security and access control

SASL-based authentication is enforced across all clusters. Applications must authenticate and declare producer or consumer intent before connecting. Each topic onboarding request generates a unique authentication token scoped to that topic. The plaintext port that existed in the earlier deployment has been removed.

This replaced a configuration where applications could connect without identifying themselves. The shift to ACLs gave the Kafka team visibility into exactly which producers and consumers were using each topic, which both improved security posture and simplified capacity planning.

Topic onboarding

Application teams submit onboarding requests through an internal Onboarding Dashboard. A capacity analysis tool integrated into the workflow evaluates cluster placement before the topic is provisioned. MirrorMaker groups for any cross-zone replication requirements are set up as part of the same workflow.

Cross-cluster replication with MirrorMaker

PayPal uses approximately 2,000 MirrorMaker nodes to replicate events between clusters. Replication serves two purposes: disaster recovery (maintaining a mirror in a secondary data centre) and inter-security-zone communication (moving data from a high-classification zone to a lower one where consuming services operate).

In 2021, Lei Huang and Na Yang presented the data centre migration at Kafka Summit Americas. The team migrated 20+ clusters (1,000+ broker and ZooKeeper nodes, 60+ MirrorMaker groups) across data centres with zero service outage, zero message loss, and no application-side changes required. MirrorMaker groups provided the traffic cut-over path, with traffic gradually shifting from source clusters to destination clusters through the MirrorMaker pipeline.

Producer architecture

The BigQuery pipeline producer uses a batch size of 250,000 bytes and a linger time of 5 seconds to optimise throughput over latency. For the user behaviour tracking pipeline, a Chronicle Queue (memory-mapped file store) sits between the HTTP ingestion layer and the Kafka producer. When a Kafka consumer group rebalance causes the producer to become temporarily unavailable, events buffer to disk rather than being dropped. Once the rebalance completes, the Chronicle Queue drains into the Kafka producer in order.

Consumer architecture

The BigQuery consumer application runs on 75 production machines (2-core CPU, 8 GB RAM each) and is built on Project Reactor (reactive streams). Key configuration values: max.poll.records=1,000, fetch.min.bytes approximately 100 KB, max.partition.fetch.bytes approximately 1 MB. The application manages 300 partitions in production. GC was switched from Concurrent Mark Sweep to G1GC during benchmarking, which improved stability under high throughput. Sustained throughput of approximately 950,000 events per minute was achieved per instance.

Session timeout is set to 60 seconds, with a heartbeat interval of 20 seconds.

Stream processing

Spark Streaming jobs consume Kafka topics directly for risk detection and metrics aggregation pipelines. The user behaviour tracking pipeline uses Akka Streams (via the Squbs reactive framework) for in-process stream processing and fan-in from multiple HTTP sources. The MergeHub Akka primitive is used to dynamically merge HTTP connection streams into the downstream enrichment flow, so each new HTTP connection becomes its own sub-stream without reconfiguring the pipeline.

Client library ecosystem

PayPal maintains three internal client libraries:

A connectivity library for resilient connection to the Config Service and brokers
A monitoring library that registers Kafka client metrics via Micrometer and forwards them to SignalFX
A security library that manages SSL certificate lifecycle

Supported languages are Java, Python, Go, Scala, and Node.js, as well as applications built on the internal Squbs framework.

Special techniques and engineering innovations

Kafka Config Service is the most operationally significant internal tool. It removes the dependency between broker topology and application configuration, enabling infrastructure changes without application redeployment. At a scale of 85+ clusters, the operational leverage this provides is substantial.

Chronicle Queue buffering addresses a specific failure mode in the user behaviour tracking pipeline. Kafka consumer group rebalances can cause brief producer unavailability. Without a durable buffer, events arriving during a rebalance are either queued in memory (at risk of loss on restart) or dropped. Chronicle Queue writes to memory-mapped files on disk, providing durability at near-memory speeds. This pattern is not standard Kafka tooling; PayPal built it to meet the data loss requirements of a pipeline processing tens of billions of events per day.

Smart patching plugin checks under-replicated partition (URP) counts before initiating any broker patch. A patch proceeds only when the affected broker’s partitions are fully replicated. This gating condition enables parallel patching of multiple clusters concurrently, with single-broker restarts proceeding in sequence within each cluster. Before this plugin, patching at 1,500+ broker scale stretched maintenance windows to multiple days.

Targeted partition reassignment modifies the default Kafka reassignment behaviour. The default algorithm reassigns all partitions on a broker, which at PayPal’s partition counts produces very long rebalancing windows. PayPal’s modification restricts reassignment to only the under-replicated partitions on the affected broker, which dramatically shortens the rebalancing duration.

Upstream open-source contributions reflect the depth of operational experience the team has developed. PayPal authored three Kafka Improvement Proposals:

KIP-351: adds a --under-min-isr flag to kafka-topics.sh for identifying partitions below minimum in-sync replicas
KIP-427: adds an at-min-isr partition category metric to the broker
KIP-517: adds consumer polling behaviour metrics for observing max.poll.interval.ms compliance

All three address monitoring gaps that PayPal encountered running Kafka at scale before these metrics existed natively.

Operating Kafka at scale

Deployment model: Self-managed, on bare metal (brokers) and VMs (ZooKeeper, MirrorMaker), with QA on Google Cloud Platform.

Monitoring and alerting: The Kafka Metrics library collects metrics from brokers, ZooKeeper nodes, MirrorMakers, and Kafka clients. Metrics are registered via Micrometer and forwarded to SignalFX. Alert thresholds are tuned to fire on actionable conditions rather than informational noise. The team made deliberate decisions about which metrics to surface, discarding many default metrics that did not correlate reliably with user-facing issues.

Capacity management: Infrastructure is scaled ahead of peak periods. The topic onboarding workflow includes a capacity analysis step so that new topics are placed on clusters with headroom. Retail Friday planning involves expanding broker capacity to accommodate the expected traffic surge above the 21 million messages per second baseline.

Developer experience: The Onboarding Dashboard and Config Service form the main interface between application teams and the Kafka platform. Application teams do not need to know which cluster a topic lives on or manage their own broker bootstrap configurations; the platform handles that. This abstraction has enabled broad adoption across teams without requiring each team to develop deep Kafka operational knowledge.

Upgrade and migration strategy: The 2021 data centre migration demonstrated PayPal’s approach to large-scale infrastructure changes: use MirrorMaker pipelines to shift traffic gradually, maintain application transparency throughout, and validate at each step before cutting over. The same principles apply to broker version upgrades, where the URP-checking plugin provides the gate condition for rolling restarts.

Challenges and how they solved them

Challenge	Root cause	Solution	Outcome
Unauthenticated client connections	Early clusters had a plaintext port with no authentication requirement	SASL-based ACLs requiring clients to authenticate and declare producer or consumer intent; unique tokens per topic	Full visibility of producers and consumers; anonymous access eliminated
Broker patching at 1,500+ node scale	Default patching interrupted replication; sequential single-broker patching was impractical across 85+ clusters	URP-checking plugin gates patches on replication health; enables parallel cluster patching with single-broker restarts	Patching windows reduced from multiple days to hours
Long rebalancing after broker changes	Default reassignment algorithm moves all partitions on a broker, producing very long rebalancing windows at high partition counts	Modified reassignment to restrict to only under-replicated partitions on the affected broker	Dramatically shorter rebalancing duration
High QA environment cost	QA clusters mirrored production footprint on-premises	Rebuilt QA on Google Cloud Platform with multi-zone brokers	75% cost reduction; 40% performance improvement
Message loss during consumer group rebalances	Kafka producer back-pressure had no durable buffer; events arriving during a rebalance were at risk of loss	Chronicle Queue (memory-mapped disk buffer) inserted ahead of the Kafka producer in the user tracking pipeline	Zero message loss during rebalance events
Analytics latency of 12 hours	Reports depended on macro-batch Spark jobs running on multi-hour intervals	Kafka consumer pipeline publishing to BigQuery in real time using Project Reactor	Analytics latency reduced from 12 hours to seconds
Zero-downtime data centre migration at 1 trillion+ messages per day	Migrating 20+ clusters required cut-over without service interruption or data loss	60+ MirrorMaker groups used for gradual, transparent traffic migration with no application-side changes	0% service outage; 0% message loss

Full tech stack

Category	Tools	Notes
Message broker	Apache Kafka	Bare metal deployment for broker nodes
Coordination	Apache ZooKeeper	VM-deployed
Replication	MirrorMaker	~2,000 nodes on VMs; cross-DC and cross-security-zone replication
Stream processing	Apache Spark Streaming, Akka Streams (via Squbs), Project Reactor	Spark for risk/analytics pipelines; Squbs for user tracking; Reactor for BigQuery pipeline
Monitoring	SignalFX (via Micrometer and Kafka Metrics library)	Broker, ZooKeeper, MirrorMaker, and client metrics; alerting on threshold breaches
QA infrastructure	Google Cloud Platform (multi-zone)	Topic-for-topic mapping of production clusters; same security posture as production
Client languages	Java, Python, Go, Scala, Node.js	Internal reactive framework: Squbs (Akka Streams-based)
Persistent buffer	Chronicle Queue	Memory-mapped disk buffer ahead of Kafka producer in user tracking pipeline
Data sinks	BigQuery (Google Cloud), Teradata	BigQuery: real-time analytics pipeline; Teradata: batch and warehouse workloads
Config distribution	Internal Kafka Config Service	Stateless; distributes bootstrap addresses and client config to all applications
Certificate management	Internal security library	SSL certificate lifecycle management for all Kafka clients
Metrics instrumentation	Micrometer	Client-side metrics registration layer; forwards to SignalFX

Key contributors

Name	Role	Source
Maulin Vasavada	Software Developer and Architect, Kafka team, PayPal	Strata NY 2018; Kafka Summit 2020
Kevin Lu	Software Engineer, PayPal	Strata NY 2018
Na Yang	Engineering Lead, PayPal	Kafka Summit Americas 2021
Lei Huang	MTS2 Engineer, PayPal	Kafka Summit Americas 2021
Monish Koppa	Software Engineer, PayPal	PayPal developer blog, September 2023
Archit Agarwal	Engineer, PayPal	PayPal Tech Blog — consumer benchmarking
Sakshi Ganeriwal	Engineer, PayPal	PayPal Tech Blog — user behaviour tracking, 2018
Michael Zeltser	Engineer, PayPal	PayPal Tech Blog — fast data architecture, 2016

Key takeaways for your own Kafka implementation

Decouple application config from broker topology. PayPal’s Kafka Config Service, which pushes bootstrap addresses and client config to applications at runtime, is one of the most operationally significant tools in their stack. At any meaningful scale, hard-coded broker IPs become a maintenance liability. A centralised config distribution layer makes broker changes transparent to applications.
Add a durable buffer ahead of your producer if message loss during rebalances is unacceptable. Kafka consumer group rebalances are not exceptional events; they happen routinely as deployments roll, instances scale, and partitions reassign. If your producer pipeline has no durable buffer, events arriving during a rebalance window are at risk. Chronicle Queue (or an equivalent mechanism) can absorb the gap with minimal latency overhead.
Gate your patching and reassignment operations on under-replicated partition counts. PayPal’s URP-checking plugin and targeted reassignment modification both reflect the same principle: do not proceed with infrastructure changes while replication is degraded. Building this check into your automation, rather than relying on manual observation, is what enables parallel patching at scale.
Invest in your QA environment’s fidelity early. PayPal’s migration of QA to GCP delivered a 75% cost reduction and 40% performance improvement. A QA environment that accurately reflects production topology is the foundation for safe rollouts, and the cost of maintaining it does not have to scale linearly with production.
Contribute upstream when you find gaps in observability. KIP-351, KIP-427, and KIP-517 all addressed monitoring blind spots that PayPal encountered before these metrics existed in Kafka natively. If you are running Kafka at scale and hitting the edges of what the built-in metrics tell you, the upstream KIP process is a viable path to a permanent fix.

Sources and further reading

Primary sources:

Monish Koppa, “Scaling Kafka to Support PayPal’s Data Growth,” PayPal developer blog, September 2023 — https://developer.paypal.com/community/blog/scaling-kafka-to-support-paypals-data-growth/
Monish Koppa, “Scaling Kafka to Support PayPal’s Data Growth,” The PayPal Technology Blog, Medium — https://medium.com/paypal-tech/scaling-kafka-to-support-paypals-data-growth-a0b4da420fab
Maulin Vasavada, “Marching Toward a Trillion Kafka Messages per Day,” Kafka Summit 2020, Confluent — https://www.confluent.io/resources/kafka-summit-2020/marching-toward-a-trillion-kafka-messages-per-day-running-kafka-at-scale-at-paypal/
Lei Huang and Na Yang, “How Did We Move the Mountain? Migrating 1 Trillion+ Messages Per Day Across Data Centers at PayPal,” Kafka Summit Americas 2021, Confluent — https://www.confluent.io/events/kafka-summit-americas-2021/how-did-we-move-the-mountain-migrating-1-trillion-messages-per-day-across/
Kevin Lu, Maulin Vasavada, Na Yang, “Kafka at PayPal: Enabling 400 Billion Messages a Day,” Strata Data Conference NY 2018 — https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/69459.html
Archit Agarwal et al., “Scaling Kafka Consumer for Billions of Events,” The PayPal Technology Blog, Medium — https://medium.com/paypal-tech/kafka-consumer-benchmarking-c726fbe4000
Sakshi Ganeriwal, “Tracking User Behavior at Scale with Streaming Reactive Big Data Systems,” The PayPal Technology Blog, Medium, September 2018 — https://medium.com/paypal-tech/https-medium-com-paypal-engineering-tracking-user-behavior-at-scale-f0c584c4ddd4
Michael Zeltser, “From Big Data to Fast Data in Four Weeks — Part 2,” The PayPal Technology Blog, Medium, November 2016 — https://medium.com/paypal-tech/from-big-data-to-fast-data-in-four-weeks-or-how-reactive-programming-is-changing-the-world-part-2-29a9f7d48318

Kafka Improvement Proposals authored by PayPal:

KIP-351 — Add –under-min-isr option to describe topics command
KIP-427 — Add AtMinIsr topic partition category (new metric & TopicCommand option)
KIP-517 — Add consumer metrics to observe user poll behavior

If you are running Kafka in production and want deeper visibility into your brokers, consumer lag, and topic health, Kpow gives you a full-featured monitoring and management UI that connects to any Kafka cluster in minutes. You can try it free for 30 days.

How Reddit uses Apache Kafka in production

Factor House — Sat, 30 May 2026 00:00:00 GMT

Reddit runs one of the largest self-managed Kafka fleets in the industry: more than 500 brokers, over a petabyte of live data, and tens of millions of messages per second. In early 2026, the platform engineering team migrated that entire fleet from Amazon EC2 to Kubernetes with zero client-side connection-string changes, while the site remained live throughout.

Apache Kafka sits at the centre of Reddit’s event infrastructure, carrying clickstream data, real-time safety signals, vote integrity checks, and ads pipeline events across a platform with hundreds of millions of monthly active users.

Company overview

Reddit is a social news aggregation and discussion platform where users submit content, vote posts and comments up or down, and organise communities around shared interests. At the time of its 2019 infrastructure overhaul, Reddit reported 330 million monthly active users, 12 million posts per month, and 2 billion votes per month.

Reddit adopted Kafka as part of a broader infrastructure rationalisation that began around 2017, when the engineering team started migrating services from manually managed Puppet configurations to Terraform. Kafka and ZooKeeper were among the first services converted. The driver was operational complexity: broker replacement required a 14-step manual runbook, ZooKeeper and Kafka were provisioned with ad hoc scripts, and the team had no reliable way to reproduce or version configuration changes.

Date	Event
Pre-2017	Kafka on EC2 with manual Puppet configs; broker replacement required a 14-step runbook
2017	Large-scale migration to Terraform begins; Kafka and ZooKeeper are among the first services converted
January 2019	Krishnan Chandra presents Reddit’s Terraform-based Kafka management at HashiCorp; broker replacement reduced to three Terraform steps
2021	ksqlDB-backed event QA tool and vote-manipulation detection pipeline presented at Kafka Summit Americas
2021	REV1 real-time safety system (Kafka and Flink Stateful Functions) presented at Flink Forward Global
October 2023	REV2 (Rule Execution V2) published: per-action-type Protobuf Kafka topics, GitHub/S3 rule versioning, time-travel via offset resets
2025–2026	Entire Kafka fleet (500+ brokers, 1+ petabyte of live data) migrated from EC2 to Kubernetes using Strimzi; ZooKeeper-to-KRaft migration follows as a separate phase

Reddit’s Kafka use cases

Clickstream and event ingestion

Kafka’s longest-standing role at Reddit is as the backbone for clickstream and tracking-event ingestion. Events from web and mobile clients flow through load balancers to stateless application servers and into Kafka. From there, a stream-processing layer routes events to downstream consumers including Hive/S3 archival, BigQuery, and real-time safety systems. This pipeline was in place by 2019 and has remained the foundation of Reddit’s event infrastructure since.

Real-time event QA

Reddit’s data engineering team, led by Hannah Hagen and Paul Kiernan, built an internal QA tool that streams events directly from the production Kafka pipeline using ksqlDB. Engineers deploying a new app version can filter events by user ID, device ID, or interaction type and receive feedback on whether events are firing correctly within seconds. Before this tool, developers had to wait approximately two hours for events to appear in the data warehouse. The tool is backed by both ksqlDB and Kafka Streams.

Vote-manipulation detection

Reddit’s Anti-Evil Engineering team replaced hourly Airflow batch jobs with a ksqlDB streaming pipeline for detecting vote manipulation and derogatory content. Derek Hsieh presented the approach at Kafka Summit Americas 2021. Detection latency dropped from hours to minutes, with Kafka providing the continuous event stream that ksqlDB queries against.

Real-time safety actioning

Reddit’s real-time safety applications team uses Kafka as both the ingestion and egress layer for detecting and actioning policy-violating content. The first version of this system (REV1), presented at Flink Forward Global 2021 by Frédérique Mittelstaedt, Bhavani Balasubramanyam, and Vignesh Raja, used Kafka topics to carry post and comment events into Flink Stateful Functions, which dispatched messages to remote Python service endpoints executing Lua-based rules. Action messages then flowed to Kafka egress topics, where Safety Actioning Workers consumed them and applied the corresponding platform action.

The successor system, REV2, published by Reddit Safety Engineering in October 2023, extended this architecture with per-action-type Kafka egress topics, Protobuf-formatted action messages, GitHub-backed rule versioning with S3 distribution, and a time-travel feature implemented via consumer group offset resets.

Ads data pipeline

Reddit operates a dedicated Kafka engineering team for ads data infrastructure. This team builds and maintains Kafka consumers using Flink and Spark as the downstream processing layer, serving all of Reddit’s Ads engineering teams.

Scale and throughput

At the time of Reddit’s 2025-2026 EC2-to-Kubernetes migration, the Kafka fleet comprised:

Metric	Value
Brokers	500+
Live data held in Kafka	More than 1 petabyte
Throughput	Tens of millions of messages per second
Client services requiring reconfiguration	250+

‍

The 2019 platform context provides additional background: at the time Krishnan Chandra presented Reddit’s Terraform-based Kafka management, Reddit had 330 million monthly active users generating 12 million posts and 2 billion votes per month. No public breakdown of topic count, partition count, or consumer group count has been published by Reddit’s engineering team.

Reddit’s Kafka architecture

Deployment model

Reddit’s Kafka deployment has changed substantially since 2019. The original model was self-managed Kafka on Amazon EC2, provisioned through Terraform modules and Puppet, with operators applying changes directly from their workstations. By 2025, the team had migrated the entire fleet to Kubernetes, managed via the Strimzi operator.

Control plane

Reddit originally used ZooKeeper for Kafka metadata management. After completing the data-plane migration to Kubernetes, the team migrated the control plane from ZooKeeper to KRaft as a separate, sequenced phase.

Safety pipeline (REV2)

The REV2 architecture follows this sequence:

Kafka ingress topics receive post and comment events from upstream producers.
Flink Stateful Functions processes messages and dispatches them to remote Python service endpoints.
The Python endpoints execute Lua-based safety rules.
Rule outputs are written as Protobuf-formatted messages to per-action-type Kafka egress topics.
Safety Actioning Workers consume egress topics and apply the corresponding platform action.
Rule configurations are stored in S3, pushed from GitHub CI. A Kubernetes sidecar in each worker pod polls S3 for updates, enabling rule changes without a full service redeploy.
The time-travel feature resets consumer group offsets to replay historical content through newly published rules.

Schema management

Reddit uses Protobuf as the serialisation format for action messages on REV2 Kafka egress topics. Each action type has its own dedicated topic with a Protobuf schema enforced per topic.

Clickstream pipeline

As of 2019, the event ingestion path ran: mobile and web clients to load balancers, to stateless application servers, into Kafka, then to a stream-processing layer, with archival to Hive/S3 and analytics writes to BigQuery.

Stream processing

Reddit uses ksqlDB and Kafka Streams for the event QA tool and vote-manipulation detection, and Apache Flink Stateful Functions for the REV2 safety system.

Special techniques and engineering innovations

DNS facade for zero-downtime broker migration

Before moving any broker from EC2 to Kubernetes, Reddit introduced an intermediate DNS layer: client applications connected to infrastructure-controlled DNS records rather than directly to broker hostnames. This decoupled client connection strings from physical broker addresses, meaning no client application needed to update its configuration during the migration. Over 250 client services were effectively transparent to the change.

Forking Strimzi for a mixed-cluster transition

Strimzi does not natively support hybrid clusters spanning EC2 and Kubernetes. Reddit forked the operator and introduced targeted modifications:

Plaintext inter-broker listeners accessible from both environments
Shared ZooKeeper metadata management during the transition period
Consistent Cruise Control configuration across EC2 and Kubernetes brokers

After all EC2 brokers were decommissioned, Reddit removed the fork and switched to the standard Strimzi operator.

Broker ID space reorganisation

Strimzi requires low broker IDs for the brokers it manages. Because the existing EC2 brokers occupied the low-ID space, Reddit resolved this by temporarily doubling the cluster: new high-numbered EC2 brokers were added, Cruise Control drained partitions from the original low-numbered brokers, those brokers were decommissioned, and Kubernetes brokers were assigned the freed low IDs.

Reversibility as a design constraint

Each phase of the migration was required to be fully reversible before the team proceeded to the next. Reddit treated non-reversibility as a hard blocker. This constrained the migration timeline but ensured that any unexpected failure could be rolled back without data loss.

KRaft migration sequenced separately

Rather than migrating ZooKeeper to KRaft concurrently with the data-plane move to Kubernetes, Reddit treated them as separate phases: the data-plane migration was completed and stabilised first, and the ZooKeeper-to-KRaft migration followed as a distinct step.

Per-action-type Protobuf topics

In REV2, each safety action type has its own dedicated Kafka egress topic with a Protobuf schema enforced per topic. This gives the team granular per-action monitoring and tighter schema contracts, compared to a single multiplexed output topic.

Consumer group offset resets for retroactive rule application

REV2 implements a time-travel capability by resetting Kafka consumer group offsets. When a new safety rule is published, operators can replay historical content through it by rewinding the consumer group to an earlier offset, without needing a separate historical data store.

Operating Kafka at scale

Terraform-managed lifecycle (pre-Kubernetes)

By 2019, Reddit’s Terraform module had reduced broker replacement from a 14-step manual runbook to three operations: terraform taint the broker node, run plan, run apply. The new broker automatically registered with ZooKeeper and updated DNS records. A parallel module managed ZooKeeper ensemble provisioning by the same pattern. The module accepted cluster size and instance type as inputs and handled node discovery, ZooKeeper registration, and security group export automatically.

Kubernetes operator management (post-migration)

After the EC2-to-Kubernetes migration, Kafka clusters are managed declaratively via the Strimzi Kubernetes operator. Configuration changes and upgrades are applied through manifest updates rather than ad hoc commands.

Cruise Control for partition rebalancing

Reddit uses Cruise Control to automate partition reassignment during scaling events and broker replacements. During the 2025-2026 migration, Cruise Control was used to incrementally drain partitions from EC2 brokers before decommissioning them.

Challenges and how they solved them

Strimzi’s lack of support for mixed EC2/Kubernetes clusters

Problem: The team needed brokers in both EC2 and Kubernetes to coexist in a single cluster during migration so that partitions could be drained gradually rather than cut over all at once.

Root cause: Strimzi is designed to manage Kafka clusters entirely within Kubernetes and has no built-in support for inter-broker communication across EC2 and Kubernetes networking boundaries.

Solution: Reddit forked Strimzi and added support for plaintext inter-broker listeners accessible from both environments, shared ZooKeeper metadata management, and unified Cruise Control configuration. The fork ran for several weeks during migration, then was retired.

Outcome: Zero-downtime migration. No client application changed its connection string.

Client services tightly coupled to broker hostnames

Problem: 250+ client services held connection strings pointing directly to EC2 broker hostnames. Changing all of them simultaneously was not feasible.

Root cause: No abstraction layer existed between client configuration and physical broker addresses.

Solution: Reddit introduced a DNS facade layer before any broker movement. Clients were redirected to stable DNS names under Reddit’s control, while physical broker addresses changed independently.

Outcome: All 250+ client services continued operating without modification throughout the migration.

Broker ID space exhaustion

Problem: Strimzi requires low broker IDs for the brokers it manages. All low IDs were already in use by existing EC2 brokers.

Root cause: Kafka broker IDs are fixed at creation time and cannot be reassigned to a running broker.

Solution: Reddit doubled the cluster by provisioning new high-numbered EC2 brokers, used Cruise Control to drain all partitions from the original low-numbered brokers, decommissioned those brokers, then assigned the freed IDs to Kubernetes brokers.

Outcome: No disruption to producers or consumers during the ID space reorganisation.

Slow and error-prone safety rule deployment (REV1)

Problem: REV1 ran each safety rule as an independent process on Python 2.7 and raw EC2. There was no version control for rules, no staging environment, and no rule history. Deployment required engineers to SSH directly into hosts.

Root cause: REV1 was not built for operational scale. Rules were written and deployed outside of standard engineering workflows.

Solution: REV2 moved rules to GitHub, pushed configurations to S3 via CI, and introduced a Kubernetes sidecar that polls for updates, reducing rule deployment time by approximately 90% and eliminating direct host access.

Outcome: Rules can be updated and rolled back without a full service redeploy, with a complete audit trail in version control.

Two-hour instrumentation feedback lag

Problem: Engineers deploying new app versions had to wait approximately two hours before events appeared in the data warehouse, making it difficult to verify that instrumentation was correct immediately after a release.

Root cause: The data warehouse pipeline introduced significant processing latency between event production and queryable state.

Solution: Hannah Hagen and Paul Kiernan built a ksqlDB-backed web application that filters events directly from the live Kafka pipeline. Feedback is available in seconds rather than hours.

Outcome: Engineers can verify event instrumentation in near real-time during or immediately after a deployment.

Full tech stack

Category	Tools	Notes
Message broker	Apache Kafka	Self-managed; migrated from EC2 to Kubernetes in 2025-2026
Cluster management	Strimzi	Kubernetes operator for declarative Kafka management (post-migration)
Control plane	KRaft (post-migration), ZooKeeper (pre-migration)	ZooKeeper-to-KRaft migration completed as a separate phase after the data-plane migration
Partition rebalancing	Cruise Control	Used during broker replacement and migration partition draining
Stream processing	ksqlDB, Kafka Streams, Apache Flink (Stateful Functions)	ksqlDB and Kafka Streams for event QA and vote-manipulation detection; Flink Stateful Functions for REV2 safety system
Serialisation	Protobuf	Used for REV2 action messages on per-action-type Kafka egress topics
Infrastructure as code	Terraform	Kafka and ZooKeeper provisioning and broker lifecycle management from 2017 onwards (pre-Kubernetes)
Configuration management	Puppet	Pre-Terraform; Terraform modules were partially derived from existing Puppet configs
Container orchestration	Kubernetes	Hosts Flink, safety workers, Kafka brokers (post-migration), and rule-update sidecars
Cloud	Amazon Web Services (EC2, S3)	EC2 was the original broker host; S3 stores REV2 rule configurations and event archives
Rule language	Lua	Safety rules executed by Flink Stateful Functions remote Python endpoints in REV2
Remote function language	Python	REV2 remote function endpoints; Python 2.7 used in REV1 (deprecated)
CI and version control	GitHub	Version control for REV2 safety rules; CI pipeline pushes rule configs to S3
Batch processing (legacy)	Apache Airflow	Replaced by Kafka/ksqlDB streaming for safety detection
Data warehouse	BigQuery	Downstream analytics store fed from the Kafka event pipeline
Archival	Apache Hive, Amazon S3	Archival store for event data processed from Kafka
Downstream processing	Apache Spark	Used alongside Flink in the ads data pipeline

Key contributors

Name	Role	Contribution
Hannah Hagen	Senior Software Engineer, Data Engineering	Co-built the ksqlDB event QA tool; presented at Kafka Summit Americas 2021
Paul Kiernan	Staff Software Engineer	Co-built the ksqlDB event QA tool; presented at Kafka Summit Americas 2021
Derek Hsieh	Software Engineer III, Data Engineering	Led the ksqlDB vote-manipulation detection pipeline; presented at Kafka Summit Americas 2021
Frédérique Mittelstaedt	Engineering Manager, Real-Time Safety Applications	Led design of Reddit’s centralised streaming platform on Kafka and Flink; presented at Flink Forward Global 2021
Bhavani Balasubramanyam	Software Engineer, Anti-Evil Engineering	Developed real-time pipeline for violent content detection; co-presented at Flink Forward Global 2021
Vignesh Raja	Software Engineer	Worked on Flink Stateful Functions and Kafka for user safety; co-presented at Flink Forward Global 2021
Krishnan Chandra	Senior Software Engineer	Led Terraform-based Kafka infrastructure management; presented at HashiCorp 2019
Neven Miculinić	Senior Software Engineer, Messaging Infrastructure	Associated with the EC2-to-Kubernetes Kafka migration project (“Swapping the Engine Mid-Flight”, 2026)

Key takeaways for your own Kafka implementation

Introduce a DNS abstraction before migrating brokers. Reddit’s experience shows that decoupling client connection strings from physical broker addresses is what makes large-scale broker migrations feasible without client-side changes. If you are planning a hosting migration, build the DNS facade first.
Sequence control-plane and data-plane migrations separately. Reddit migrated brokers from EC2 to Kubernetes first, stabilised the deployment, and only then migrated from ZooKeeper to KRaft. Attempting both simultaneously increases the blast radius if something goes wrong.
Treat reversibility as a hard constraint, not a nice-to-have. Reddit required each migration phase to be fully reversible before proceeding. This slows the timeline but makes it possible to halt and recover if an unexpected issue arises mid-migration.
Use per-type topics with enforced schemas for action pipelines. The REV2 architecture uses a distinct Kafka egress topic per action type with a Protobuf schema. If you are building a Kafka-backed action or command pipeline, this pattern makes monitoring and schema evolution substantially more manageable than a multiplexed topic.
Kafka consumer group offsets are a reprocessing primitive. Reddit’s time-travel feature in REV2 demonstrates that resetting consumer group offsets to replay historical data is a practical operational technique, not only a disaster-recovery option. If your pipeline needs to apply new logic to historical events, this approach avoids the need for a separate historical data store.

Sources and further reading

Hannah Hagen and Paul Kiernan (Reddit) — Live Event Debugging With ksqlDB at Reddit, Kafka Summit Americas 2021
Hannah Hagen and Paul Kiernan (Reddit) — Kafka Summit Americas 2021 slide deck
Derek Hsieh (Reddit) — Catching Vote Manipulation at Reddit“, Kafka Summit Americas 2021
Frédérique Mittelstaedt, Bhavani Balasubramanyam, Vignesh Raja (Reddit) — Keeping Redditors Safe With Stateful Functions, Flink Forward Global 2021
InfoQ — Reddit’s REV2: Replacing a Batch Safety System with a Real-Time One Using Flink and Kafka, October 2023
Krishnan Chandra (Reddit) — How Reddit’s Large Scale Migration to Terraform, HashiCorp, January 2019
Reddit Engineering — Swapping the Engine Mid-Flight: How We Moved a Petabyte of Kafka Data from EC2 to Kubernetes, r/RedditEng, January 2026:
ByteByteGo — summary of Reddit’s Kafka migration
Red Hat Developer — Kafka Monthly Digest, February 2026

If you are monitoring a Kafka deployment at the scale described in this article, Kpow provides real-time visibility across brokers, topics, consumer groups, and schema registries. You can connect it to any Kafka cluster and try it free for 30 days.

How Robinhood uses Apache Kafka in production

Factor House — Sat, 30 May 2026 00:00:00 GMT

Robinhood’s Kafka platform handles 2.2 million messages per second across 6 clusters, 160 brokers, and 90,000 partitions, underpinning every step of its trading lifecycle from order routing to fraud detection to data lake ingestion. What makes its story particularly instructive is not the scale alone but the engineering choices the team made to reach it: a custom sidecar proxy to solve Python’s connection fan-in problem, a Postgres-backed dead letter queue that trades Kafka simplicity for richer replay tooling, and a 2025 decision to replace self-managed Kafka with WarpStream for logging workloads, cutting total costs by 45%.

Across fintech, Kafka is common. At Robinhood the challenge was operating a large, heterogeneous fleet of Python-based Kafka applications at financial-grade reliability without overwhelming a small platform team.

Company overview

Robinhood is a US-based brokerage and financial services platform offering commission-free trading in equities, ETFs, options, and cryptocurrency, along with a debit card and cash management product. As of 2025, the platform serves more than 14 million monthly active users.

Kafka has been part of Robinhood’s infrastructure since at least 2015, when engineer Jaren Glover joined and was assigned ownership of stabilising an existing Kafka, ZooKeeper, and Elasticsearch pipeline. Over the following four years, Glover scaled that infrastructure from 100,000 to 10 million users, presenting what he learned at SREcon Americas and Kafka Summit in 2019.

By 2017, the team had begun developing Faust, an open-source Python stream-processing library built on top of Apache Kafka. Faust was open-sourced in July 2018 and reflected how central Kafka had become to Robinhood’s product engineering.

Date	Milestone
2015	Kafka already in production; Jaren Glover takes ownership of stabilising the pipeline
2015-2019	Platform scales from 100,000 to 10 million users
2017-2018	Faust Python stream-processing library developed and open-sourced
2019	Data lake pipeline published: Kafka as primary databus, Secor archiving to S3
2019-2020	Brokerage system grows from 100k to 750k peak requests/second in six months; shard-aware Kafka filtering introduced
2021	Kafka Summit Americas: 6 clusters, 160 brokers, 1,400 topics, 90,000 partitions, 2.2M msg/sec confirmed publicly
2022	CDC pipeline using Debezium and Apache Hudi reduces data lake freshness from 24 hours to 5-15 minutes
2023	Consumer Proxy (kafkaproxy v2, Java + gRPC) presented at Confluent Current
2024	Kafka clusters migrated from EC2 to Kubernetes using a custom in-house operator
2025	Logging workloads migrated from self-managed Kafka to WarpStream; 45% total cost reduction

Robinhood’s Kafka use cases

The Streaming Platform team describes Kafka as involved in “almost every mission-critical step of Robinhood’s functionality.” There are 14 confirmed production use cases across product, data, and infrastructure teams.

Equities order routing and trade execution. Stock purchase events are written to both Kafka and Postgres. Multiple downstream services consume those events and process them using Kafka Streams with exactly-once semantics. The trade execution confirmation SLA is under one second, with 5-10 Kafka hops per trade depending on product type.

Crypto trading. The crypto backend uses shard-aware Kafka consumers. Each shard only processes messages belonging to users assigned to it, limiting the blast radius of any single consumer failure.

Self-clearing. Robinhood’s in-house clearing operations are listed as a distinct Kafka use case by the Streaming Platform team, though detailed architecture has not been published.

Market data distribution. External market data feeds are ingested via Kafka and consumed by internal Faust stream-processing applications.

Push notifications and messaging. Named explicitly as a production use case in the 2021 Kafka Summit Americas talk by Chandra Kuchi and Nick Dellamaggiore.

Fraud detection and risk. Faust processes risk signals in real time. Apache Flink handles stateful stream processing for fraud detection and shareholder position tracking, with more than 100 concurrent Flink deployments running in production.

Order execution quality monitoring and ad tracking. Both are listed as production Faust use cases, alongside newsfeed aggregation.

CDC into the data lake. Debezium captures WAL changes from thousands of Postgres RDS tables, encodes them as Avro records via Confluent Schema Registry, and writes them to Kafka. Apache Hudi Deltastreamer consumes those records with exactly-once semantics and writes into S3. Data freshness for core datasets improved from 24 hours to 5-15 minutes.

Card transaction authorisation (backup path). The backup payment authorisation service subscribes to asynchronous Kafka streams to cache each cardholder’s latest account state. The service must produce a decision within a 2-second window.

Investor eligibility (Say Technologies acquisition). Apache Flink determines investor eligibility with approximately 2-minute end-to-end latency.

Log analytics and observability. Application and infrastructure logs flow through Kafka (and since 2025, WarpStream) via the Vector log shipper to Humio for search and storage.

Scale and throughput

The figures below come from Chandra Kuchi and Nick Dellamaggiore’s Kafka Summit Americas 2021 presentation, reflecting the state of the platform at that time, with more recent additions from a 2025 WarpStream case study and a September 2024 Diginomica interview with Kuchi.

Metric	Value	Source / date
Kafka clusters	6	Kafka Summit Americas 2021
Brokers (total)	160	Kafka Summit Americas 2021
Topics	1,400	Kafka Summit Americas 2021
Partitions	90,000	Kafka Summit Americas 2021
Messages per second (all clusters)	2.2 million	Kafka Summit Americas 2021
API requests per second (largest cluster)	500,000	Kafka Summit Americas 2021
Inbound connections (largest cluster)	>100,000	Kafka Summit Americas 2021
Data processed per day	10+ TB	WarpStream case study, 2025
Monthly active users	14+ million	WarpStream case study, 2025
Concurrent Flink deployments	100+	Tony Chen, Current 2024
Postgres tables in CDC pipeline	Thousands	Robinhood Engineering blog, 2022
Inter-service latency (Kafka-connected)	20-50 ms	Chandra Kuchi, Diginomica 2024
Brokerage peak load growth (Dec 2019 to Jun 2020)	100k to 750k requests/sec (7x)	Edmond Wong and Nathan Ziebart, Robinhood blog, 2021
Faust throughput (2018 estimate)	Billions of events and terabytes of data per day	Ask Solem, Medium, 2018

Robinhood’s Kafka architecture

Cluster topology and hosting

Robinhood originally ran self-managed Confluent Platform on AWS EC2, provisioned with Saltstack. Brokers were spread across three AWS availability zones per cluster.

By 2024, the team had migrated all clusters to Kubernetes. Rather than adopting an existing Kafka Kubernetes operator, Robinhood built a custom in-house operator to manage the migration and ongoing operations. Mun Yong Jang and Nathan Moderwell presented details of this migration at Confluent Current 2024. The three-AZ layout was preserved in the new Kubernetes deployment.

Logging workloads were separated from the main Kafka clusters in 2025 and migrated to WarpStream, a diskless, S3-backed Kafka-compatible broker. WarpStream runs as three independent Kubernetes deployments (one per AZ), each with its own horizontal pod autoscaler, so logging capacity tracks market-hours traffic without manual intervention. Ethan Chen and Renan Rueda from Robinhood presented this architecture at Confluent Current New Orleans 2025.

Cloud provider: AWS throughout. S3 serves as primary data lake storage and as WarpStream’s backing store. Postgres runs on AWS RDS.

Schema management

Confluent Schema Registry is used for Avro encoding on the CDC pipeline. Debezium captures Postgres WAL and writes Avro-encoded change records to Kafka via the Schema Registry before downstream consumers read them into the data lake.

Producer architecture

All Python and Golang Kafka clients at Robinhood use kafkahood, an internal wrapper library built on librdkafka. kafkahood codifies standard defaults, client-side metrics collection, dead letter queue support, message serialisation, and feature-flag-based configuration rollouts. This means the Streaming Platform team can push new client defaults across the entire application fleet by flipping a feature flag rather than waiting for individual teams to update their library versions.

Producer connections from Python services are routed through kafkaproxy (described in detail in the Special Techniques section below).

Consumer architecture

Consumer lifecycle management is handled by the Consumer Proxy (kafkaproxy v2), a Java-based Kubernetes sidecar that runs alongside each application container. The proxy handles all Kafka consumer group coordination, rebalancing, commit management, and timeout management. It relays messages to the application container over gRPC. This design means a consumer group rebalance triggered by a scaling event does not affect the application container’s business logic, and a failure in the application container does not orphan an uncommitted Kafka offset.

kafkahood also provides a Postgres-backed dead letter queue for failed consumer events. Failed messages are inserted into Postgres rather than a secondary Kafka topic, and a CLI allows engineers to inspect, query, and fix records before republishing them to a retry topic.

Consumer lag, throughput, and error rates are surfaced per application as standard defaults through kafkahood’s client-side metrics, without each team needing to add their own instrumentation.

Stream processing

Faust (Python). Robinhood open-sourced Faust in 2018 as a Python port of the Kafka Streams API. In production, Faust applications handle risk signal processing, order quality monitoring, market data feed processing, newsfeed aggregation, ad tracking, event logging, and crypto feed processing. Faust uses RocksDB as a local embedded state store and aiokafka (which Robinhood also maintains as a fork) as its async Kafka client.

Apache Flink. Stateful stream processing for fraud detection, shareholder position tracking, investor eligibility determination, and data ingestion is handled by Flink. Robinhood runs more than 100 concurrent Flink deployments, managed by the Apache flink-kubernetes-operator. Tony Chen from the Streaming Platform team presented Robinhood’s Flink deployment practices, including a custom Checkpoint-Fetcher tool for automating recovery, at Confluent Current 2024.

Kafka Connect ecosystem

Debezium is used as the primary Kafka Connect source connector, capturing CDC events from Postgres RDS. It encodes changes as Avro records using Confluent Schema Registry and writes them to Kafka for consumption by Apache Hudi Deltastreamer on the data lake side. Secor (originally open-sourced by Pinterest) archives Kafka streams to S3 for the batch path of the data lake.

Special techniques and engineering innovations

Kafkaproxy as a sidecar for connection aggregation. Python’s process-based concurrency model creates a problem at Kafka scale: each gunicorn worker process opens its own set of TCP connections to brokers, which at Robinhood’s cluster sizes produced more than 100,000 inbound connections on the largest cluster. The first-generation kafkaproxy was a Rust-based Kubernetes sidecar that ran alongside each application pod and aggregated all producer and consumer connections from that pod’s worker pool into a single connection set to the brokers. This cut broker-side resource pressure from Python services substantially.

Consumer Proxy with gRPC message relay (v2). The second generation of kafkaproxy, presented by Tony Chen and Mun Yong Jang at Confluent Current 2023, replaced the Rust implementation with a Java-based sidecar using the standard Apache Kafka client library. The v2 proxy takes full ownership of the Kafka consumer lifecycle: it handles consumer group rebalancing, commit management, and timeout management, then relays raw messages to the application container over gRPC. This decoupling means the application team no longer needs to reason about consumer group behaviour, and the platform team maintains a single Java consumer implementation rather than N language-specific clients.

Shard-aware Kafka message routing. During the 2019-2020 period when brokerage traffic grew 7x, Robinhood introduced application-level sharding over Postgres. Each shard’s Kafka consumers filter messages on shared topics to only process events belonging to users assigned to that shard. For high-throughput flows where the per-message filtering overhead became significant, Robinhood created shard-specific Kafka topics to eliminate the filtering cost entirely.

CDC with exactly-once semantics into the data lake. The CDC pipeline chains Debezium (Postgres WAL capture), Confluent Schema Registry (Avro encoding), Kafka (transport), and Apache Hudi Deltastreamer (exactly-once writes into S3). Copy-on-write mode is used for raw tables to optimise columnar read performance. This pipeline reduced data freshness from 24 hours to 5-15 minutes across thousands of Postgres tables.

Postgres-backed dead letter queue. Rather than routing failed consumer events to a secondary Kafka topic, Robinhood writes them to a Postgres database. A client application with a CLI allows engineers to inspect and query failed records, apply fixes, and republish messages to a Kafka retry topic for safe reprocessing. Sreeram Ramji and Wenlong Xiong, who presented this pattern at Confluent Current 2022, noted that cross-team schema contracts make silent consumer failures particularly hazardous at Robinhood, which drove the choice of a more inspectable store over a native Kafka DLQ topic.

WarpStream for elastic logging clusters. Log throughput at Robinhood is highly cyclical: it peaks during US market hours and drops significantly in evenings and on weekends. Running fixed-capacity Kafka clusters for this workload was costly and wasteful. By migrating logging to WarpStream, the team eliminated inter-AZ data transfer costs entirely (WarpStream’s diskless architecture does not move data between availability zones) and gained per-AZ horizontal pod autoscaling. The result was a 45% reduction in total logging infrastructure cost, broken down as 36% compute savings, 13% storage savings, and 99% inter-AZ networking savings. The trade-off was an increase in end-to-end latency from 0.2 to 0.45 seconds, which was acceptable for the logging workload.

Feature-flag-based client library rollouts. kafkahood uses feature flags to control adoption of new client configuration defaults across the application fleet. A new default setting can be tested on a subset of services before rolling out broadly, with fast rollback if it causes latency or throughput regressions.

Operating Kafka at scale

Deployment model. Robinhood self-manages its Kafka infrastructure throughout its history, initially on EC2 with Saltstack and since 2024 on Kubernetes with a custom in-house operator. The WarpStream deployment for logging is also self-hosted (Kubernetes) rather than using WarpStream Cloud.

Observability. Client-side metrics collection is built into kafkahood as a standard default for all Python and Golang services. This provides per-application consumer lag, throughput, and error rates without requiring each team to instrument independently. The Vector log shipper routes application and infrastructure logs to Kafka (or WarpStream) for forwarding to Humio.

WarpStream Agent Groups for multi-tenant isolation. Within the logging WarpStream cluster, Agent Groups isolate traffic classes: VIP clients, SSL/TLS connections, and plaintext connections each run in separate groups. This prevents a high-throughput producer from saturating broker resources shared with other clients.

Flink checkpoint management. The Apache flink-kubernetes-operator had stability issues when reconciling more than 100 concurrent Flink deployments simultaneously. Robinhood tuned the operator’s concurrent reconciliation configuration and built a custom Checkpoint-Fetcher tool that automates locating the latest valid Flink checkpoint and updating the initialSavepointPath field in the Kubernetes manifest. This reduces human error during recovery and rollback operations. Automatic rollback to the last healthy application version via health checks is also implemented.

Confluent Platform usage. Chandra Kuchi confirmed in a September 2024 interview that Robinhood uses Confluent CP. The current status of Confluent’s commercial components post-Kubernetes migration has not been detailed in public sources.

Note: Robinhood’s SRE team has not published details of Kafka alerting thresholds, SLO definitions, quota management, or CI/CD pipelines for topic provisioning in any source reviewed for this article.

Challenges and how they solved them

Python connection fan-in at broker scale. Robinhood’s Python services use the gunicorn process-based worker model. Each worker process opens independent Kafka connections, and at hundreds of workers per cluster the broker inbound connection count exceeded 100,000 on the largest cluster, creating resource exhaustion and connection management overhead. The solution was kafkaproxy: a Kubernetes sidecar that aggregates all connections from a pod’s worker pool into a shared set, dramatically reducing broker-side connection counts. A second generation of the proxy (in Java) later extended this to full consumer lifecycle management via gRPC.

Divergent Kafka client libraries across languages. Separate Kafka client wrappers for Python and Golang were drifting apart in defaults, error handling, and observability. kafkahood standardised client behaviour across languages with a single librdkafka-based library. The v2 consumer proxy went further, centralising the consumer group logic in one Java implementation that serves both Golang and Python application containers via gRPC.

Failed consumer events with no safe retry path. Schema mismatches and deserialization errors cause consumer events to fail permanently unless there is a way to inspect and fix the message before replaying it. A secondary Kafka DLQ topic does not provide sufficient tooling for this. Robinhood built a Postgres-backed DLQ with a CLI for inspection and targeted republishing to a retry topic.

Data lake freshness of 24 hours. Batch ingestion from Postgres into S3 meant analytics consumers worked with day-old data. The CDC pipeline (Debezium, Confluent Schema Registry, Kafka, Apache Hudi Deltastreamer with exactly-once writes) reduced freshness to 5-15 minutes across thousands of Postgres tables.

7x brokerage traffic growth in six months. Requests grew from 100k to 750k per second between December 2019 and June 2020, overwhelming a single-shard brokerage system. Application-level sharding distributed load across Postgres shards, each with its own Kafka consumers. High-throughput flows received shard-specific Kafka topics to eliminate per-message filtering overhead.

Logging infrastructure cost and elasticity. Self-managed Kafka clusters sized for peak market-hours throughput sat largely idle in evenings and weekends, and incurred inter-AZ data transfer fees continuously. Migrating to WarpStream eliminated the inter-AZ networking cost entirely (99% reduction) and allowed per-AZ autoscaling to track actual throughput. Total cost fell 45%.

Managing 100+ concurrent Flink deployments. The Apache flink-kubernetes-operator struggled with concurrent reconciliation at this scale, and manual checkpoint management introduced human error during recovery. Robinhood tuned the operator’s concurrency settings, built the Checkpoint-Fetcher tool to automate manifest updates, and implemented health-check-based automatic rollback.

Full tech stack

Category	Tool	Role at Robinhood
Message broker	Apache Kafka (Confluent Platform)	Primary event broker across equities trading, crypto, self-clearing, market data, notifications, fraud detection, data lake ingestion
Message broker (logging)	WarpStream	Diskless, S3-backed Kafka-compatible broker for logging workloads; adopted 2025
Schema registry	Confluent Schema Registry	Schema management and Avro encoding for CDC change records from Debezium
Stream processing (Python)	Faust (open-sourced by Robinhood)	Risk/fraud detection, order quality monitoring, newsfeed aggregation, ad tracking, market data feed, crypto feed, event logging
Stream processing (JVM)	Apache Flink	Stateful processing: fraud detection, shareholder position tracking, investor eligibility; 100+ concurrent deployments
Kafka client wrapper	kafkahood (internal)	librdkafka-based library for Python and Golang; standardised defaults, client metrics, DLQ support, feature-flag rollouts
Consumer proxy	kafkaproxy / Consumer Proxy (internal)	Kubernetes sidecar (v1: Rust; v2: Java + gRPC) managing Kafka consumer lifecycle and connection aggregation
CDC connector	Debezium (Kafka Connect)	Captures WAL from Postgres RDS; writes Avro-encoded change records to Kafka
Data lake ingestion	Apache Hudi (Deltastreamer)	Exactly-once incremental writes from Kafka into S3 data lake
Data lake archival	Secor (Pinterest)	Archives Kafka streams to S3 for batch processing path
Batch processing	Apache Spark	Batch workloads on the data lake; also used for Hudi ingestion jobs
Query engine	Trino	SQL queries over the S3 data lake
Workflow orchestration	Apache Airflow	Orchestrates data pipeline jobs
Kubernetes operator	Apache Flink Kubernetes Operator	Manages 100+ concurrent Flink application deployments
Log shipping	Vector (Datadog)	Produces application and infrastructure logs to Kafka / WarpStream
Log storage and search	Humio	Receives logs from the Kafka/WarpStream pipeline
Embedded state store	RocksDB	Local state store used by Faust for stateful stream processing
Async Kafka client	aiokafka (Robinhood fork)	Async Kafka client used by Faust; maintained as a fork by Robinhood
Object storage	AWS S3	Data lake primary storage; backing store for WarpStream
Primary database	AWS RDS (Postgres)	Source for CDC pipeline; also backing store for Postgres DLQ
Container orchestration	Kubernetes	Hosts all services, Kafka clusters (post-2024 migration), and Flink deployments
Metastore	Apache Hive Metastore	Schema management for data lake tables
Cluster manager	Apache Hadoop YARN	Cluster management for Spark jobs

Key contributors

Name	Role	Contribution
Jaren Glover	Ops engineer, Robinhood (c. 2015-2019)	Stabilised and scaled Kafka infrastructure from 100,000 to 10 million users; presented at SREcon Americas 2019 and Kafka Summit 2019
Ask Solem	Software engineer, Robinhood	Created Faust, the Python stream-processing library built on Kafka; open-sourced July 2018
Vineet Goel	Software engineer, Robinhood	Co-authored the Faust introductory blog post and the “Adding Faust to your Existing Architecture” post
Chandra Kuchi	Engineering Manager, Data Infrastructure Lead, Streaming Platform	Presented “Taming a Massive Fleet of Python-based Kafka Apps at Robinhood” at Kafka Summit Americas 2021; cited in Diginomica (2024) on Robinhood’s Kafka and Flink architecture
Nick Dellamaggiore	Software engineer, Kafka Infrastructure, Robinhood	Co-presented at Kafka Summit Americas 2021
Tony Chen	Software engineer, Streaming Platform, Robinhood	Presented “Robinhood’s Kafkaproxy” at Current 2023 and “Robinhood’s Flink Deployment Practices” at Current 2024
Mun Yong Jang	Software engineer, Streaming Platform, Robinhood	Co-presented kafkaproxy at Current 2023; co-presented “Robinhood’s Kafka Journey from EC2 to Kubernetes” at Current 2024
Nathan Moderwell	Software engineer, Robinhood	Co-presented Kafka EC2-to-Kubernetes migration at Current 2024
Sreeram Ramji	Senior Staff Software Engineer, Data Infra and Experimentation	Co-presented “Dead Letter Queues for Kafka Consumers in Robinhood” at Current 2022; owns Kafka, Flink, Spark, Trino, Airflow, and Data Lake infrastructure
Wenlong Xiong	Software engineer, Streaming Platform and Batch Compute (formerly)	Co-presented DLQ architecture at Current 2022
Ethan Chen	Software engineer, Robinhood	Co-presented WarpStream logging migration at Current New Orleans 2025
Renan Rueda	Software engineer, Robinhood	Co-presented WarpStream logging migration at Current New Orleans 2025
Stephen Chang	Engineering Manager, Payments, Robinhood	Authored “Building a Resilient Card Transaction System” (2022), covering Kafka’s role in backup payment authorisation
Edmond Wong and Nathan Ziebart	Technical Leads, Brokerage Engineering, Robinhood	Co-authored “How we scaled Robinhood’s brokerage system for greater reliability” (2021), covering shard-aware Kafka filtering

Key takeaways for your own Kafka implementation

Connection fan-in is a real problem for process-based concurrency models. If you run Python services with gunicorn or uWSGI and connect them directly to Kafka, each worker process opens its own connections. At a few hundred workers per cluster this adds up to tens of thousands of inbound connections on your brokers. A sidecar proxy that aggregates connections from a single pod is one approach; Robinhood also considered async client libraries but chose the proxy model for the isolation benefits it provided.

Decoupling consumer lifecycle from application code simplifies failure handling. Robinhood’s Consumer Proxy moves all consumer group logic, rebalancing, commits, and timeouts into a separate sidecar. Your application container receives messages over gRPC only after they have been successfully consumed. This means application failures do not stall consumer groups, and consumer group rebalances triggered by scaling do not introduce latency into your application.

A Postgres-backed dead letter queue gives you more repair tooling than a DLQ topic. A secondary Kafka topic stores failed messages but does not give you a way to query them by failure reason, apply a fix, or selectively replay a subset. If your teams share schema contracts and schema mismatches are a real failure mode, the ability to inspect and fix records before replay is worth the additional infrastructure.

Separating workloads by their latency and cost profiles pays off at scale. Robinhood’s logging traffic and its trading traffic have very different latency requirements: 0.45 seconds is acceptable for logs but not for order routing. Migrating logging to WarpStream (with higher latency but lower cost) while keeping trading on self-managed Kafka let the team optimise each cluster for its actual workload, producing a 45% cost reduction on the logging side without touching the trading path.

A standardised internal Kafka client library is worth the investment early. kafkahood encodes sane defaults, observability, DLQ support, and feature-flag-controlled rollouts for all services. Teams get correct client behaviour without needing to understand all the underlying librdkafka configuration options. The payoff is proportional to the number of services you run: at 14+ million users and hundreds of services, a single misconfigured default can cause widespread consumer lag or data loss.

Sources and further reading

Title	Author(s)	Venue	Date
Robinhood swaps Kafka for WarpStream to tame logging workloads and costs	Ethan Chen, Renan Rueda (Robinhood); Jason Lauritzen (WarpStream)	WarpStream blog / Current New Orleans 2025	2025
Taming a Massive Fleet of Python-based Kafka Apps at Robinhood	Chandra Kuchi, Nick Dellamaggiore (Robinhood)	Kafka Summit Americas 2021	2021
Robinhood’s Kafkaproxy: Decoupling Kafka Consumer Logic from Application Business Logic	Tony Chen, Mun Yong Jang (Robinhood)	Confluent Current 2023	2023
Dead Letter Queues for Kafka Consumers in Robinhood	Sreeram Ramji, Wenlong Xiong (Robinhood)	Confluent Current 2022	2022
Faust: Stream Processing for Python	Ask Solem, Vineet Goel (Robinhood)	Robinhood Engineering / Medium	2018-07-31
Robinhood’s Kafka Journey from EC2 to Kubernetes	Mun Yong Jang, Nathan Moderwell (Robinhood)	Confluent Current 2024	2024
Robinhood’s Flink deployment practices	Tony Chen (Robinhood)	Confluent Current 2024	2024
Data lake at Robinhood	Robinhood Engineering	Robinhood Newsroom	2019-09-09
Robinhood invests in a stable and scalable future with Confluent	Derek du Preez (interviewing Chandra Kuchi)	Diginomica	2024-09-18
Scaling Robinhood crypto systems	Chirantan Mahipal, Hefu Chai, Xuan Zhang (Robinhood)	Robinhood Newsroom	2022-09-09
Fresher data lake on AWS S3	Robinhood Engineering	Robinhood Newsroom	2022-02-18
Building a resilient card transaction system	Stephen Chang (Robinhood)	Robinhood Newsroom	2022-10-12
How we scaled Robinhood’s brokerage system for greater reliability	Edmond Wong, Nathan Ziebart (Robinhood)	Robinhood Newsroom	2021-06-25
Faust: streaming at Robinhood with Ask Solem	SE Daily (interview with Ask Solem)	Software Engineering Daily	2018-09-05

‍

If you are running Kafka in production and want visibility into consumer lag, throughput, and cluster health across your brokers, give Kpow a try with a free 30-day trial. You can connect it to any Kafka cluster in minutes and deploy it via Docker, Helm, or JAR.

How Spotify used Apache Kafka in production

Factor House — Sat, 30 May 2026 00:00:00 GMT

Between roughly 2013 and 2017, Spotify ran one of the more thoroughly documented Kafka deployments in the industry. At peak, their event delivery system processed 700,000 events per second across five datacenters and fed every major feature on the platform — Discover Weekly, Year in Music, Spotify Wrapped, and Billboard’s streaming charts all relied on data moving through Kafka. In 2017, Spotify decommissioned their Kafka infrastructure entirely, migrating to Google Cloud Pub/Sub as part of a broader move to Google Cloud Platform. What makes this story worth reading is what happened in between: the architectural decisions Spotify made, the specific failure modes they encountered at scale, and the engineering trade-offs that led them to walk away from Kafka for a managed alternative.

Company overview

Spotify is a music and podcast streaming service founded in Stockholm in 2006. By 2015, when much of the documented Kafka work was underway, the platform had 60 million monthly active users, 30 million songs, and 1.5 billion playlists. By early 2016, that had grown to 100 million monthly active users across 61 markets.

The company adopted Kafka around 2013 to replace an ad-hoc log-shipping system that tailed files from service hosts and sent them toward a central Hadoop cluster. Event volume was growing fast enough that iterating on the existing system was no longer practical, and the team needed a message bus that could handle cross-datacenter delivery at scale.

Key Kafka milestones:

Date	Event
~2013	Adopted Apache Kafka 0.7 for event delivery
January 2015	Published Storm/Kafka personalization architecture — 3 billion+ events/day
2015	Production peak: 700,000 events/second across 5 datacenters
2015–2016	Evaluated Kafka 0.8 with MirrorMaker; encountered data loss and producer failure issues
March 2016	Announced migration to Google Cloud Pub/Sub; load-tested replacement at 2 million messages/second
February 2017	Kafka system fully decommissioned; Cloud Pub/Sub live in production
Q1 2019	Post-migration: 8 million events/second peak, 500 billion events/day, 350+ TB raw data/day
October 2021	EDI v3: 600+ event types, 70 TB/day compressed, Dataflow replacing Dataproc

Spotify’s Kafka use cases

Kafka sat at the centre of Spotify’s event delivery infrastructure (EDI) and served as the source of truth for all user interaction data. Every time a user played a song, skipped a track, viewed an ad, or performed a search, the client emitted an event that made its way through a Kafka producer, into a broker, and out to one or more downstream consumers.

Event delivery for analytics and features: The primary pipeline delivered event data to a central Hadoop cluster, where hourly Crunch/MapReduce ETL jobs converted raw tab-separated text into Apache Avro format for downstream analytics. This data fed Discover Weekly, Year in Music, and Spotify Wrapped, as well as Billboard’s streaming charts and the internal A/B testing infrastructure.

Real-time personalization via Apache Storm: In parallel with the batch path, Apache Storm topologies consumed from Kafka topics in real time. Topics carried event types including song completions and ad impressions. Each Storm topology subscribed to relevant topics, enriched events with entity metadata fetched from Cassandra (for example, attaching the genre of a completed song), grouped events by user, and computed user attributes for recommendation and ad-targeting models.

Log aggregation across data centers: Spotify operated four owned data centers plus capacity on Google Cloud Platform during this period. Kafka served as the collection point for service logs from all hosts, with a custom component handling cross-datacenter batching and forwarding.

Ownership was split along functional lines: the event delivery team owned the Kafka infrastructure and producer daemons, while feature teams (personalization, ads, data engineering) owned the Storm topologies and downstream consumers.

Scale and throughput

During the Kafka era, production throughput peaked at 700,000 events per second. The team load-tested a replacement system at 2,000,000 events per second — roughly triple their production peak — before committing to the migration.

The Storm cluster processing real-time events from Kafka consisted of 6 nodes with 24 cores per host and processed over 3 billion events per day by January 2015. Spotify ran this infrastructure across five datacenters simultaneously, with cross-datacenter forwarding handled by the Grouper component described below.

After the migration to Google Cloud Pub/Sub, the same event volume continued to grow:

Period	Metric	Value
2014–2015 (Kafka)	Peak production throughput	700,000 events/second
2015 (Kafka)	Real-time pipeline daily volume	3+ billion events/day
April 2017 (Pub/Sub)	Daily event volume	100 billion events/day
Q1 2019 (Pub/Sub)	Peak throughput	8 million events/second
Q1 2019 (Pub/Sub)	Daily raw data	350+ TB/day
Q1 2019 (Pub/Sub)	Daily event volume	500 billion events/day
2021 (Pub/Sub + Dataflow)	Distinct event types	600+
2021 (Pub/Sub + Dataflow)	Daily compressed data	70 TB/day

‍

The post-migration growth is notable: peak throughput went from 700K to 8 million events/second — more than an 11x increase — in roughly two years, without Kafka in the stack.

Spotify’s Kafka architecture

High-level architecture

The event delivery system had four principal components: a Syslog Producer on every service host, Kafka Brokers receiving those events, custom Grouper services for cross-datacenter forwarding, and an HDFS Consumer persisting data to Hadoop.

Apache Storm topologies ran alongside this, consuming from Kafka topics directly for real-time processing, with Cassandra serving as the metadata and user-profile store downstream.

Producer architecture

A daemon called the Kafka Syslog Producer ran on every service host. Its role was to tail log files and batch log lines toward the Kafka cluster. Events at this stage were treated as unstructured lines regardless of type — there was no schema enforcement at the producer level. The producer maintained a checkpoint (end-of-file marker) per log file to track delivery state; missing markers after host failure required manual intervention.

Batching and compression were applied at the Grouper stage rather than the producer, which simplified the per-host daemon but pushed cross-datacenter optimisation downstream.

Consumer architecture

Two consumer patterns ran in parallel from the same Kafka topics:

Batch consumers wrote events to HDFS for processing by hourly Crunch/MapReduce jobs. These converted tab-separated raw event data into Apache Avro format, partitioned by hour.

Real-time consumers used Kafka Consumer Spouts within Apache Storm topologies. Each topology version used a unique Kafka consumer group ID so that old and new versions could consume the same topic simultaneously during deployments. This allowed operators to run both versions in parallel and roll back without losing events. The team also tuned rebalancing.max.tries to reduce consumer rebalancing errors under their workload.

A Liveness Monitor tracked which service hosts were active during each hour by querying service discovery systems. This enabled the batch consumers to verify that all expected events for a given hour had been received before marking the partition complete.

Stream processing

Apache Storm handled real-time event processing in Spotify’s personalization and ad-targeting pipelines. Storm topologies subscribed to Kafka topics (e.g. song completions, ad impressions), fetched entity metadata from Cassandra, grouped events per user, and computed user attributes that fed recommendation and targeting models.

The 6-node Storm cluster (24 cores/host) processed over 3 billion events per day by early 2015 and supported use cases including Discover Weekly personalisation, ad targeting, and data visualisation.

The Grouper: cross-datacenter forwarding

Spotify built a custom component called the Grouper that consumed all event streams from a local datacenter and republished them as a single, compressed, efficiently batched topic for cross-datacenter transmission. Without the Grouper, raw Kafka streams would have saturated cross-datacenter links. The Grouper also allowed the team to enforce batching and compression consistently at the datacenter boundary rather than at each individual producer.

Special techniques and engineering innovations

Dual consumer groups for safe topology rollouts: Each new Storm topology version was deployed with a unique Kafka consumer group ID. Both the old and new topology versions consumed the full message stream simultaneously during a rollout window. If the new version misbehaved, operators could roll back without message loss and without manual reprocessing. This pattern decoupled deployment risk from data continuity.

Custom Grouper for cross-datacenter efficiency: Rather than forwarding per-topic streams cross-datacenter, the Grouper consumed multiple topics locally, merged and compressed them, and republished a single efficient stream. This was a pragmatic response to both bandwidth constraints and Kafka 0.7’s lack of native cross-datacenter replication.

Custom async-google-pubsub-client: When migrating to Cloud Pub/Sub, the official Google Java client could not meet Spotify’s throughput requirements. The team built an open-source, high-performance async Java client — async-google-pubsub-client — which ran in production for over a year before Google’s official libraries reached comparable performance.

Isolation per event type (post-Kafka lesson): A direct lesson from the Kafka era’s shared-channel problems was applied in the Pub/Sub architecture: each of 500+ event types received its own topic, ETL process, and final storage location. This prevented high-volume events from disrupting business-critical ones, and let the team assign different SLOs per event type (hours for high priority, up to 72 hours for low priority).

Operating Kafka at scale

Spotify’s Kafka deployment was entirely self-managed, running on owned datacenter infrastructure and configured via Puppet. The system spanned five datacenters, with the Grouper component managing cross-datacenter forwarding.

Observability: The team monitored consumer group lag and rebalancing events, and tuned rebalancing.max.tries to reduce noise from regular rebalancing errors. A dedicated Liveness Monitor tracked active service hosts by hour to detect gaps in event delivery before they became incidents.

Deployment and upgrades: Configuration changes were deployed via Puppet. The tight coupling between system components meant that even small changes to one component could cause system-wide outages, which the team described as difficult to recover from. There was no clean way to iterate on individual components in isolation.

Incident response: Missing end-of-file markers after host failure required manual intervention to unblock the Liveness Monitor and resume processing for the affected hour. The lack of automated recovery for this scenario was one of the documented pain points of the v1 architecture.

Developer experience: Spotify open-sourced docker-kafka (a combined Kafka and ZooKeeper Docker image) to give developers a local Kafka environment for testing. The repository is now archived.

Challenges and how they solved them

Kafka 0.7 had no reliable broker-level persistence

Problem: Kafka 0.7 did not support broker-level replication. All reliable persistence lived in Hadoop, making it the single point of failure for the entire event delivery system.

Root cause: Architectural limitation of the Kafka version in use. Broker-level replication was not available until later releases.

Solution: The team accepted the limitation and designed the system around HDFS as the persistence layer. This worked until the coupling between Hadoop availability and event delivery became operationally unacceptable.

Outcome: The design constraint accumulated into the technical debt that made the eventual migration necessary.

Cross-datacenter delivery at 700K events/second

Problem: Producers across five datacenters needed to deliver events to a central Hadoop cluster. Raw event streams would have exhausted cross-datacenter bandwidth, and producers needed distant datacenter acknowledgement before considering delivery complete.

Root cause: No native cross-datacenter replication in Kafka 0.7; bandwidth constraints at datacenter links.

Solution: Built the custom Grouper component to consume event streams locally, compress and batch them, and forward a single efficient stream per datacenter.

Outcome: The Grouper reduced cross-datacenter bandwidth and improved delivery reliability, at the cost of a custom component that the team had to maintain and operate.

Kafka 0.8 MirrorMaker silently dropped data

Problem: When the team evaluated upgrading to Kafka 0.8, MirrorMaker instances dropped data while reporting to the source cluster that mirroring had succeeded. The producer also required full service restarts after failure, with no automated recovery path.

Root cause: Bugs in the MirrorMaker coordination logic in the Kafka 0.8 version under evaluation.

Solution: Abandoned the Kafka 0.8 path entirely. Rather than invest further in debugging MirrorMaker, the team migrated to Google Cloud Pub/Sub and built a custom high-throughput Java client (async-google-pubsub-client) for the migration.

Outcome: A five-day consumer stability test of the Pub/Sub system showed a median end-to-end latency of 20 seconds with zero message loss. The Kafka system was decommissioned in February 2017.

No quality-of-service differentiation between event types

Problem: All event types shared the same delivery channel. A surge in high-volume, low-priority events could degrade delivery of business-critical event types, with no mechanism to prioritise one over another.

Root cause: The single shared Kafka pipeline had no per-event-type QoS controls.

Solution: The Pub/Sub architecture assigned each event type its own topic, ETL process, and storage location. Priority tiers determined SLOs: High (few hours), Normal (24 hours), Low (72 hours).

Outcome: By 2019, the team could state that “a noisy, broken, or blocked event type will not halt the rest of the system.”

Tight component coupling blocked iteration

Problem: Even small changes to one component caused system-wide outages that were hard to diagnose and recover from. The team could not improve the system incrementally.

Root cause: All components were tightly coupled. The Hadoop dependency for persistence, the Grouper’s cross-datacenter role, and the producer’s manual checkpoint logic created a fragile dependency graph.

Solution: A full system redesign, not an iterative fix. The new architecture decomposed the system into independent microservices with explicit interfaces, enabling independent deployment and failure isolation.

Outcome: By 2019, the event delivery system comprised approximately 15 microservices across ~2,500 VMs, each independently deployable.

Full tech stack

Category	Tools	Notes
Message broker (Kafka era)	Apache Kafka 0.7, evaluated 0.8	Decommissioned February 2017
Message broker (post-Kafka)	Google Cloud Pub/Sub	Per-event-type topics; 7-day retention
Stream processing (Kafka era)	Apache Storm	6-node cluster, 24 cores/host; consumed from Kafka topics
Stream/batch processing (post-Kafka)	Scio (Scala API for Apache Beam) on Google Dataflow	Unified batch and streaming; open-sourced by Spotify
Schema	Apache Avro	Applied via hourly ETL; no schema enforcement at Kafka producer level
Storage (Kafka era)	HDFS on Hadoop	Single point of failure in original architecture
Batch processing (Kafka era)	Crunch/MapReduce on Hadoop	Hourly Avro ETL from HDFS
Storage (post-Kafka)	Google Cloud Storage, BigQuery	GCS for hourly immutable partitions; BQ for analytics
Database	Apache Cassandra	User profile attributes and entity metadata for Storm enrichment
Coordination (Kafka era)	Apache ZooKeeper	Kafka dependency
Configuration management	Puppet	Replaced as part of broader cloud migration
Local development (Kafka era)	spotify/docker-kafka (Docker)	Combined Kafka + ZooKeeper image; now archived
Internal container orchestration	Helios	Spotify’s precursor to Kubernetes
Compute infrastructure	Google Compute Engine, Regional Managed Instance Groups	Post-migration; approximately 2,500 VMs
Deduplication	Google Dataproc, later Google Dataflow	Multi-week lookback deduplication using event message identifiers
Sensitive data handling	Google Dataflow	In-flight encryption for events containing personal data

Key contributors

Name	Role	Contribution
Igor Maravić	Software Engineer, Spotify	Led the event delivery migration; authored the three-part “Road to the Cloud” blog series; presented at QCon New York 2016
Neville Li	Data Infrastructure Engineer, Spotify	Created Scio, Spotify’s Scala API for Apache Beam; presented at QCon New York 2016
Kinshuk Mishra	Engineer, Spotify	Published the Storm/Kafka personalization architecture (2015)
Matt Brown	Engineer, Spotify	Co-authored the Cassandra personalization post (2015)
Bartosz Janota	Data Infrastructure Engineer, Spotify	Co-authored “Life in the Cloud” retrospective (2019)
Robert Stephenson	Senior Product Manager, Spotify	Co-authored the 2019 and 2021 EDI articles
Flavio Santos	Data Infrastructure Engineer, Spotify	Co-authored the EDI v3 migration article (2021)

Key takeaways for your own Kafka implementation

Operational burden compounds with scale. Spotify’s Kafka setup functioned well at 700K events/second but required increasing amounts of custom work (the Grouper, manual EOF recovery, tight Puppet-based configuration) to hold together. If you are building on Kafka, account for the operational investment required as throughput grows, especially if you are running across multiple datacenters.
Cross-datacenter Kafka without native replication requires custom solutions. Spotify built the Grouper specifically because Kafka 0.7 lacked reliable cross-datacenter capabilities. If you are running Kafka across regions today, verify which replication mechanisms you are relying on and test their failure modes explicitly — Spotify’s MirrorMaker evaluation is a reminder that silent data loss is a realistic failure mode.
Per-event-type isolation changes the operational model significantly. Spotify’s shared Kafka pipeline had no QoS controls, which meant any event type could affect all others. The isolation-per-event-type pattern they adopted post-migration (one topic per event type, independent SLOs) is applicable with Kafka: topic-per-event-type design increases operational surface area but limits blast radius when a single consumer or producer misbehaves.
Managed services trade operational control for operational simplicity. Spotify’s migration to Pub/Sub removed a significant operational burden and enabled 11x throughput growth without a proportional increase in infrastructure work. If you are evaluating self-managed Kafka against a managed alternative, Spotify’s trajectory is a concrete reference point for what that trade looks like at 700K+ events/second.
Decouple deployment risk from data continuity. Spotify’s dual consumer group pattern for Storm rollouts — where old and new topology versions consumed the same topic simultaneously — is a practical technique for any system where consumer logic changes frequently. It removes message loss as a rollback cost and lets you validate new consumers against live data before cutting over.

Sources, further reading and CTAs

Primary sources:

Igor Maravić, Spotify’s Event Delivery – The Road to the Cloud (Part I), Spotify Engineering, February 2016
Igor Maravić, Spotify’s Event Delivery – The Road to the Cloud (Part II), Spotify Engineering, March 2016
Kinshuk Mishra, How Spotify Scales Apache Storm, Spotify Engineering, January 2015
Kinshuk Mishra and Matt Brown, Personalization at Spotify using Cassandra, Spotify Engineering, January 2015
Neville Li, Big Data Processing at Spotify: The Road to Scio (Part 1), Spotify Engineering, October 2017
Bartosz Janota and Robert Stephenson, Spotify’s Event Delivery – Life in the Cloud, Spotify Engineering, November 2019
Flavio Santos and Robert Stephenson, Changing the Wheels on a Moving Bus, Spotify Engineering, October 2021
Tino Tereshko, Spotify’s journey to cloud: why Spotify migrated its event delivery system from Kafka to Google Cloud Pub/Sub, Google Cloud Blog, March 2016
Igor Maravić and Neville Li, Handling Streaming Data in Spotify, QCon New York 2016

Continue reading:

If you want to monitor and inspect your own Kafka clusters — consumer lag, topic throughput, partition health — give Kpow a try. It connects to any Kafka cluster in minutes and is available for a free 30-day trial. You can deploy it via Docker, Helm, or JAR.

How The New York Times uses Apache Kafka in production

Factor House — Sat, 30 May 2026 00:00:00 GMT

The New York Times is one of the largest English-language news publishers in the world, with a digital subscriber base in the millions and content operations spanning text, photography, video, and interactive journalism. Its publishing infrastructure must serve a website, iOS and Android applications, search, and personalisation systems — all simultaneously and with very low latency from the moment a journalist hits publish.

The Times operated multiple legacy content management systems for decades, each built independently with different APIs and schemas. Content from the 1920s was digitised via OCR; content from the late 1990s lived in a CMS that bore no resemblance to the one used a decade later. By 2017, the organisation had reached a point where the inconsistency between these systems was creating real engineering cost: every downstream service had to understand each upstream API individually, schema changes caused inconsistencies, and bootstrapping a new system against the historical content archive meant making individual API calls at a scale that created unpredictable load.

The trigger for adopting Kafka was the decision to build a unified publishing pipeline. Kafka’s two distinguishing properties — infinite event retention and globally ordered consumption — made it the only viable platform for treating the content log as a permanent store rather than a transient queue.

Timeline

Year	Milestone
Pre-2017	Multiple legacy CMS systems with inconsistent APIs; no unified content schema; full content archive inaccessible at scale without heavy API load
2017	Publishing Pipeline launched with Kafka as the Monolog; 166+ years of content archive stored in Kafka with infinite retention
September 2017	Boerge Svingen, Director of Engineering, publishes detailed technical writeup on the Confluent blog
October 2017	Svingen presents “The Source of Truth” at Kafka Summit NYC 2017

The New York Times’ Kafka use cases

Content publishing and distribution

The primary use case is the publishing pipeline. When any piece of content is ready for publication, the producing service writes it to the Monolog — a single-partition Kafka topic that acts as the canonical record of everything the Times has ever published. Downstream consumers read from this topic to power their own systems.

The downstream services include:

The website and native iOS/Android applications, which need published content available with minimal latency after a journalist publishes
The Elasticsearch cluster powering site search, which requires a denormalised view of content for indexing
Personalisation systems, which need to reprocess recent content to build recommendations
Content list services, both manually curated and query-driven

Before the Monolog, each of these systems maintained its own connection to upstream APIs. After the Monolog, they all consume from Kafka, with no knowledge of or dependency on the producing CMS.

Derived views: the Denormalized Log and Skinny Log

Two additional Kafka topics serve specific downstream needs:

The Denormalized Log is a multi-partition topic created by the Denormalizer service (described in the architecture section). It republishes each top-level asset together with all of its resolved dependencies — so Elasticsearch ingestion nodes receive a complete, self-contained payload rather than having to chase references. It is partitioned by top-level asset URI, allowing parallel ingestion.

The Skinny Log carries lightweight processing notifications — signals that an asset has been processed and is available — without the full content payload. Downstream systems use it for cache invalidation and for tracking service level objective (SLO) metrics: end-to-end latency from publish event to downstream availability.

Scale and throughput

The New York Times’ Kafka deployment is not high-frequency by the standards of financial services or ride-sharing platforms. The content corpus is editorial: text, images, and associated metadata. As of 2017, the full archive — every published asset since 1851 — totalled less than 100GB and fit in a single Kafka topic on a single disk.

Metric	Value
Total content corpus size	Under 100GB (as of 2017)
Historical span	All content since 1851 — 166+ years of archive
Monolog partitions	1 (single partition for total ordering)
Denormalized Log partitions	Multiple, partitioned by top-level asset URI
Retention policy	Infinite

‍

The small corpus size is a deliberate consequence of the data model: normalisation means each asset is stored once, with references handled via URI rather than duplication. The single-partition Monolog fits comfortably on a single disk, which makes full log replay from offset 0 a practical operational tool rather than a theoretical capability.

The New York Times’ Kafka architecture

The Monolog

The Monolog is a single-partition Kafka topic with infinite retention. It holds every published asset in chronological order, with one critical constraint: assets are topologically sorted so that every dependency appears in the log before any asset that references it. An image shared across multiple articles is written once; each article references that image by URI. A consumer reading from offset 0 will always encounter an image before any article that embeds it.

Assets are serialised as Protocol Buffer v3 (proto3) binaries. Every asset is identified by a structured URI: for example, nyt://article/577d0341-9a0a-46df-b454-ea0718026d30. This uniform identifier scheme applies regardless of asset type, making the reference graph consistent and resolvable.

The gateway service

No producer writes directly to the Monolog. All writes pass through a gateway service that validates each asset against the protobuf schema before allowing it into the log. A custom linter enforces forward and backward compatibility on every schema change, ensuring that new field additions or type modifications do not silently break existing consumers. Schema enforcement at the write boundary keeps the log clean from the start rather than requiring downstream consumers to handle malformed messages defensively.

The Denormalizer

The Denormalizer is a Java application built on the Kafka Streams API. It consumes the Monolog and maintains a local state store of the latest version of every asset, along with the reference graph linking assets together.

When a dependency is updated — for example, an image caption is corrected — the Denormalizer identifies every top-level asset that references that image and re-publishes those top-level assets to the Denormalized Log. This means downstream consumers such as Elasticsearch always receive the complete, current bundle of a top-level asset and all its dependencies in a single message, without needing to chase references themselves.

Elasticsearch ingestion

Each Elasticsearch ingestion node runs a Kafka Streams application that reads a partition of the Denormalized Log. It assembles JSON documents from the protobuf payloads and writes them to specific Elasticsearch shards. Because the Denormalized Log is partitioned by top-level asset URI, each ingestion node owns a deterministic subset of the content, enabling parallel ingestion without coordination between nodes.

Producer and consumer architecture

Producers: All content systems write through the gateway service. The gateway performs schema validation and then appends the asset to the single-partition Monolog. Producers are decoupled from all downstream consumers — they have no knowledge of how many consumers exist or what they do with the content.

Consumers: Each downstream system maintains its own consumer group and offset position in the relevant log (Monolog, Denormalized Log, or Skinny Log). Because Kafka retains events indefinitely, a consumer can always replay from offset 0 to rebuild its data store from scratch — which means stateless, immutable service deployments become practical.

GraphQL layer

A GraphQL schema is automatically generated from the protobuf definitions. This means the Kafka data model and the consumer API contract stay in sync without manual schema maintenance. The GraphQL schema exposes the same typed structure as the protobuf messages, giving API consumers a self-documenting interface derived directly from the canonical data model.

Special techniques and engineering innovations

Single-partition topic for total causal ordering

The decision to use a single partition for the Monolog is the most deliberate architectural departure from typical Kafka practice. Multi-partition topics offer parallelism and higher throughput, but they cannot guarantee global ordering across partitions. The Monolog requires total ordering: a referenced asset must always be visible before the asset that references it. With multiple partitions, an article could appear in one partition before its referenced image appears in another, breaking consumers that try to resolve the dependency graph in order. A single partition makes this impossible.

This is a reasonable trade-off given the data volume. Editorial content does not arrive at the rates that would make a single partition a throughput bottleneck.

Topological sort of the dependency graph

Before an asset is written to the Monolog, it is sorted topologically within its dependency graph. The Times described it as ensuring “you always see a referenced asset before the asset doing the referencing.” This is not handled by Kafka itself — it is enforced by the producing system before the write. The result is a log where consumers can process messages in sequence without defensive look-ahead or buffering.

Immutable service deployments via log replay

Because Kafka retains the full content history indefinitely, any downstream data store can be rebuilt from scratch by replaying from offset 0. The Times uses this property to enable immutable, stateless service deployments. Rather than running in-place database migrations when a schema or data change needs to be reflected downstream, a team can destroy its data store and replay the log. This eliminates a class of errors that arise from incremental state mutation over time.

Derived logs as first-class pipeline artefacts

The Denormalized Log and Skinny Log are not implementation details or optimisation hacks — they are first-class outputs of the pipeline, each designed to serve a specific downstream need. The Denormalized Log removes the reference-resolution burden from consumers that need complete assets. The Skinny Log gives lightweight notification consumers a low-overhead signal without the full payload. Designing explicit derived logs keeps individual consumer applications simple and reduces coupling between the core Monolog and the specific requirements of each downstream system.

Operating Kafka at scale

Deployment model: Kafka and ZooKeeper run on Google Cloud Platform (GCP) Compute instances. Application services — the gateway, the Denormalizer, and consumer applications — run in containers on Google Kubernetes Engine (GKE).

Service communication: Inter-service communication uses gRPC over Cloud Endpoints. Kafka Consumer API connections are made over SSL.

Schema governance: The custom protobuf linter that runs at write time is the primary schema governance mechanism. It checks forward and backward compatibility on every schema change before allowing it into the gateway, which means breaking changes are caught before they reach the log.

SLO tracking: The Skinny Log provides the signal for measuring end-to-end publish latency — from the moment an asset is written to the Monolog to the moment downstream systems confirm it is available. This gives the team an observable measure of pipeline health tied to a user-facing outcome.

Operational lesson: Boerge Svingen noted in the Software Engineering Daily interview that self-managing Kafka added meaningful operational burden. He indicated that a managed Kafka service would significantly reduce this overhead — a consideration worth weighing for teams planning similar log-based architectures.

Challenges and how they solved them

Legacy API fragmentation

Problem: Each of the Times’ legacy CMS systems had its own API, developed independently with no shared schema. Consumers had to maintain separate integration logic for each upstream system.

Root cause: Decades of independent CMS development without a unifying data contract.

Solution: The Monolog and its gateway enforce a single protobuf schema across all producers. Every piece of content — regardless of which CMS originated it — must conform to the same schema before it enters the log. Consumers only need to understand one format.

Outcome: Downstream services are decoupled from CMS internals entirely. A CMS change does not require downstream teams to update their integration code.

Historical content access at scale

Problem: Services that needed to bootstrap against historical content had to make individual API calls per asset. At 166 years of archives, this created unpredictable and unmanageable load on upstream APIs.

Root cause: API-based access is not designed for bulk historical retrieval. Each call adds latency and load that compounds at archive scale.

Solution: Log replay from Kafka offset 0. Because the Monolog retains all content indefinitely, any consumer can replay from the beginning to build or rebuild its data store without making API calls to upstream systems.

Outcome: A new downstream system can bootstrap against the full content archive by reading the Monolog from offset 0, with no coordination required with upstream teams and no risk of overloading upstream APIs.

In-place state mutation and schema drift

Problem: Services that maintained their own permanent state had to implement migration logic for each schema change, leading to inconsistencies between systems that had migrated at different times.

Root cause: Distributed mutable state creates divergence when schema changes are applied incrementally.

Solution: Log replay enables a different approach: instead of migrating state in place, a service can rebuild its data store from the log after a schema change. The Denormalizer and Elasticsearch ingestion nodes are designed with this assumption — they are stateless with respect to the log.

Outcome: Schema changes become a log replay operation rather than a coordinated migration across multiple live services.

Cloud messaging services as Kafka alternatives

Problem: The team evaluated Google Pub/Sub, AWS SNS/SQS, and Amazon Kinesis as potential alternatives to self-managed Kafka.

Root cause: Cloud-native messaging services are appealing for operational simplicity, but the Monolog requires two properties: infinite event retention and globally ordered consumption.

Solution: None of the evaluated cloud messaging services satisfied both requirements. Apache Kafka was the only platform that did, which drove the decision to self-manage it on GCP despite the operational overhead.

Outcome: A deliberate trade-off: greater operational burden in exchange for the architectural properties that make the log-as-source-of-truth model possible.

Full tech stack

Category	Tool / technology	Role
Message broker	Apache Kafka	Core event log — Monolog, Denormalized Log, Skinny Log
Stream processing	Kafka Streams (Java)	Denormalizer; Elasticsearch ingestion nodes
Serialisation	Protocol Buffers v3 (proto3)	Message format for all Kafka topics
API layer	GraphQL (auto-generated from protobuf)	Consumer-facing content API
Service communication	gRPC, Cloud Endpoints	Inter-service RPC; Kafka Consumer API over SSL
Cloud platform	Google Cloud Platform (GCP)	Kafka and ZooKeeper on Compute instances
Container orchestration	Kubernetes / GKE	Gateway service, Denormalizer, consumer applications
Search	Elasticsearch	Full-text search index, populated from the Denormalized Log
Cluster coordination	ZooKeeper	Kafka broker coordination
Schema governance	Custom protobuf linter	Forward/backward compatibility enforcement at write time

Key contributors

Name	Role	Contribution
Boerge Svingen	Director of Engineering, The New York Times	Primary author of the publishing pipeline architecture; author of the Confluent blog post; presenter at Kafka Summit NYC 2017
Martin Kleppmann	Researcher and author	Reviewed the Confluent blog post; cited for foundational log-based architecture concepts
Michael Noll	Confluent	Reviewed the Confluent blog post
Mike Kaminski	Not specified	Reviewed the Confluent blog post

Key takeaways for your own Kafka implementation

A single-partition topic is sometimes the right choice. If your use case requires total causal ordering across all events — not just within a partition — a single-partition topic removes the risk of cross-partition ordering violations. This only works if your throughput is low enough that a single partition is not a bottleneck, which is true for content-publishing patterns and similar workloads.
Enforce schema at the write boundary, not the read boundary. The Times’ gateway validates every asset against a protobuf schema before it enters the Monolog. Consumers never need to handle malformed messages because they cannot enter the log. Moving schema enforcement upstream reduces defensive complexity in every consumer you build.
Design explicit derived logs for specific consumer needs. Rather than burdening individual consumers with reference resolution or full-payload overhead, create purpose-built derived topics — a Denormalized Log for systems that need complete assembled assets, a Skinny Log for lightweight notification consumers. Each derived log keeps consumer applications simpler and reduces coupling to the source topic.
Infinite retention enables immutable deployments. If Kafka retains the full event history, downstream data stores become fully reproducible from the log. You can destroy and rebuild any materialised view from offset 0, which simplifies schema migrations and removes the need for coordinated in-place migration logic across services.
Evaluate managed Kafka against your operational capacity. The Times self-managed Kafka on GCP and found the overhead significant. If your team does not have deep Kafka operational experience, a managed service may be worth the cost — especially if the architectural properties you need (infinite retention, ordered consumption) are available in the managed offering you are evaluating.

Sources and further reading

Primary sources

Boerge Svingen, “Publishing with Apache Kafka at The New York Times,” Confluent Blog, September 6, 2017. https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/
Boerge Svingen, “The Source of Truth: Why the New York Times Stores Every Piece of Content Ever Published in Kafka,” Kafka Summit NYC 2017 (slides). https://www.slideshare.net/ConfluentInc/kafka-summit-nyc-2017-the-source-of-truth-why-the-new-york-times-stores-every-piece-of-content-ever-published-in-kafka
Boerge Svingen, Software Engineering Daily interview, October 30, 2017. https://softwareengineeringdaily.com/2017/10/30/kafka-at-ny-times-with-boerge-svingen/

Try Kpow with your Kafka cluster

If you are running a Kafka publishing pipeline and want visibility into consumer lag, topic throughput, and offset positions across your Monolog and derived topics, Kpow connects to any Kafka cluster and gives you that observability in a single interface. You can try it free for 30 days — deploy via Docker, Helm, or JAR, and connect to your cluster in minutes.

How Uber uses Apache Kafka in production

Factor House — Sat, 30 May 2026 00:00:00 GMT

Uber’s Kafka deployment is one of the most extensively documented in the industry, with the company publishing more than a dozen engineering blog posts and conference talks describing how its Streaming Platform team has evolved the infrastructure over a decade. By 2021, Uber was processing trillions of messages per day across tens of thousands of topics, with throughput that had grown from roughly one million to twelve million messages per second over five years. Apache Kafka sits at the centre of nearly every real-time system Uber operates, from matching riders with drivers to billing advertisers to detecting fraud.

Company overview

Uber operates a global ride-hailing, food delivery, and freight platform across more than 70 countries. At peak, thousands of trips are being coordinated simultaneously, each generating a continuous stream of GPS updates, payment events, and state changes from driver and rider applications. That volume, combined with strict latency requirements for matching and pricing, pushed Uber toward an event-driven architecture early in its growth.

Kafka was adopted in early 2015, beginning with a small cluster in a single region. Within a year, the platform was auditing approximately one trillion messages per day. By 2021, that figure had grown to trillions, and the team had built a suite of internal tools - uReplicator, Chaperone, uForwarder, and uGroup - on top of it. The timeline below traces the major milestones.

Timeline

Date	Milestone
Early 2015	Kafka adopted; small cluster in one region
November 2015	uReplicator deployed to production
January 2016	Chaperone auditing approximately 1 trillion messages per day
August 2016	uReplicator open-sourced
December 2016	Chaperone blog post published
June 2017	Hadoop Summit: real-time infrastructure scaled to trillions of events per day
February 2018	Dead letter queue / reliable reprocessing blog post published
March 2019	DBEvents CDC framework blog post published
April 2019	Kafka Summit SF: Kafka Cluster Federation and multi-region disaster recovery
January 2020	Kappa architecture blog post published
December 2020	Multi-region Kafka disaster recovery blog post published
August 2021	Consumer Proxy blog published; 200,000 partitions, 12 million messages per second
September 2021	Exactly-once ad event processing blog published
October 2021	uGroup consumer management framework blog published
2022-2023	Kafka tiered storage deployed to production workloads
April 2024	Kafka Summit London: Exactly-Once Stream Processing at Scale at Uber
July 2024	Kafka tiered storage blog published
February 2026	uForwarder open-sourced; 1,000+ consumer services onboarded

Uber’s Kafka use cases

Kafka underpins real-time operations across Uber’s core products and its internal data platform. The use cases below span multiple engineering teams and reflect how the system has expanded from transport to advertising and insurance.

Rider-driver matching and surge pricing

GPS location events from rider and driver applications flow into Kafka, where stream processing jobs analyse supply and demand in near real-time. The same event streams power dynamic pricing, which updates fares every few seconds. UberEats ETAs are also calculated from Kafka-sourced event streams. These pipelines operate with strict latency targets: the platform targets API latency below 5ms and 99.99% availability.

Fraud detection

Streaming analysis of transaction and session events runs continuously to identify fraud patterns. Bot detection and rider session analytics feed into similar pipelines.

Change data capture

The DBEvents framework reads MySQL binary logs via an internal tool called StorageTapper and streams change events into Kafka. Cassandra CDC events follow the same path. Downstream, these events land in Uber’s Hadoop data lake via Apache Hudi, which supports upsert-capable writes for incremental updates.

Ad event processing

Impressions and clicks from UberEats advertising flow through a Kafka pipeline that handles ad pacing, budget management, and customer billing. This pipeline operates with exactly-once guarantees, described in more detail under Special techniques below.

Async microservice queuing

More than 1,000 internal consumer services use Kafka as an asynchronous queue via the uForwarder Consumer Proxy rather than consuming directly from brokers. The proxy handles offset management and delivery, shielding application teams from Kafka consumer group mechanics.

Dead letter queues and retry pipelines

The Driver Injury Protection insurance product deducts per-mile premiums on each trip. When payment processing fails, events route through tiered retry topics with increasing delays before landing in a dead letter queue. Independent consumer groups maintain retry pipelines for each failure tier.

Data archival and real-time analytics

Kafka events sink to Apache Pinot for real-time OLAP queries and to Apache Hive for warehouse analysis. This combination gives Uber both low-latency query access to recent data and long-term historical analysis from the same event stream.

Kappa architecture backfill

Streaming and batch jobs share a single codebase. Rather than replaying historical data into Kafka for backfill runs, Uber models Apache Hive as a streaming source in Spark Structured Streaming, allowing the same job logic to run against historical data without reloading it into the cluster.

Scale and throughput

Metric	Value	Source
Messages per day	Trillions (multiple petabytes)	Uber Engineering Blog, 2020-2021
Messages per second (2021)	12 million	Consumer Proxy blog, August 2021
Messages per second (baseline)	1 million	Consumer Proxy blog (5 years prior)
Topics	Tens of thousands	Hadoop Summit 2017
Topics audited by Chaperone	20,000+	Hadoop Summit 2017
Partitions	200,000	Consumer Proxy blog, August 2021
Consumer services on uForwarder	1,000+	uForwarder blog, February 2026
Clusters	Dozens, across multiple regions	Kafka Cluster Federation talk, 2019
API latency target	Less than 5ms	Hadoop Summit 2017
Availability target	99.99%	Hadoop Summit 2017

‍

Throughput grew twelve-fold over five years. Much of that growth was absorbed by the Consumer Proxy rather than by adding partitions, since the proxy decouples partition count from consumer service concurrency.

Uber’s Kafka architecture

Cluster topology

Uber operates two tiers of Kafka clusters across multiple regions. Regional Clusters receive all producer traffic; producers always publish to their local region. Aggregate Clusters hold cross-region replicas and provide a unified global view of each topic. Replication between tiers is handled by uReplicator, Uber’s in-house replacement for Kafka MirrorMaker.

In addition to the regional/aggregate split, clusters are segmented by use case: separate clusters exist for logging, database changelogs, and high-reliability messaging. This isolation contains blast radius when one cluster has problems and allows retention and replication policies to be tuned per use case.

A Kafka Cluster Federation layer presents multiple physical clusters as a single logical cluster. A Kafka Proxy handles metadata routing, and a central Metadata Service maintains the registry of which topics live on which physical clusters.

Producer architecture

Producers publish through a Local Agent, a persistence layer that buffers messages on the producer side before writing to brokers. This improves durability by ensuring messages survive transient broker unavailability without being dropped at the client. Multi-language producer support covers Java, Go, Python, Node.js, and C++. An internal Kafka REST Proxy provides an HTTP interface for non-JVM producers; through internal optimisation work, its throughput was raised from 7,000 to 45,000 QPS per box.

Consumer architecture

Rather than having each microservice manage its own Kafka consumer group, Uber routes most consumer traffic through uForwarder, a push-based Consumer Proxy. uForwarder fetches messages from Kafka partitions and delivers them to consumer service endpoints via gRPC. This design separates partition count from processing concurrency: a consumer service can scale its processing threads independently of how many Kafka partitions the topic has. Offset commits are managed centrally by the proxy rather than by individual service instances.

uForwarder also handles out-of-order offset commits. Rather than blocking a partition on a slow message, it acknowledges individual messages and commits only contiguous completed ranges. This allows parallel processing within a partition without the risk of losing acknowledgement state.

Stream processing

Apache Flink is the primary stream processing engine for stateful workloads, including fraud detection and ad event aggregation. Apache Samza runs alongside Flink for some pipelines. Apache Spark Structured Streaming is used for the Kappa architecture backfill jobs described above.

Kafka Connect ecosystem

CDC ingestion uses StorageTapper, Uber’s internal MySQL binlog reader, to produce change events into Kafka. Downstream, sinks connect to Apache Hive, HDFS, and Apache Pinot. The Chaperone audit system also consumes every Kafka message as part of the observability pipeline.

Special techniques and engineering innovations

uReplicator: custom cross-cluster replication

Uber replaced Kafka MirrorMaker with uReplicator after MirrorMaker caused weekly production outages. The root cause was consumer group rebalancing: whenever a topic was added or changed, MirrorMaker would pause replication for five to ten minutes while rebalancing completed. uReplicator addresses this with Apache Helix for static partition assignment and a DynamicKafkaConsumer that eliminates rebalance-triggered pauses. New topics can be added to replication at runtime without restarting the cluster. uReplicator also applies header-based filters during replication to prevent cyclic data duplication across federated clusters.

Exactly-once ad event processing

Uber’s advertising billing pipeline uses a combination of Flink transactional producers, Kafka read_committed consumer isolation, two-minute checkpoint intervals, and per-record UUIDs to achieve end-to-end exactly-once delivery. Deduplication at the sink layer uses Apache Pinot’s native upsert capability and Hive keyed on the same UUIDs. This pipeline was built specifically to avoid double-counting billable advertising events.

Kafka tiered storage

Uber implemented Kafka tiered storage using the KIP-405 pluggable storage interface. Recent log segments remain on broker disk for low-latency access. Older segments are offloaded to remote storage (HDFS, S3, GCS, or Azure) transparently, without the consumer needing to know which tier a message is fetched from. This decouples storage retention from broker capacity: longer retention periods no longer require adding brokers or running separate data pipelines to external storage.

Kappa architecture backfill via Hive as a streaming source

When a streaming job needs to backfill historical data, replaying that data into Kafka adds significant cluster load and can disrupt real-time consumers. Uber avoids this by modelling Apache Hive as a streaming source in Spark Structured Streaming. The same job code handles both real-time Kafka consumption and historical Hive reads, and windowing semantics are preserved across the two modes.

Dead letter queue topology

Uber’s dead letter queue implementation uses a multi-tier retry topology: a main consumption topic feeds into retry topics with increasing delay intervals, and messages that exhaust retries land in a dead letter topic. Each tier uses Avro schemas and a leaky bucket pattern for flow control. Multiple independent consumer groups can maintain their own retry pipelines against the same underlying topics.

uGroup: consumer group visibility via __consumer_offsets decoding

Standard Kafka consumer group monitoring relies on active consumers reporting metrics. This misses consumer groups that are stopped or failing silently. uGroup is a streaming job that decodes Kafka’s internal __consumer_offsets topic directly, making all consumer group activity visible regardless of consumer state. It also tracks offset state across regions to support disaster recovery failover.

Operating Kafka at scale

End-to-end auditing with Chaperone

Chaperone is Uber’s audit system for Kafka pipelines. It consumes every message across all topics and uses ten-minute tumbling windows to compute count, p99 latency, and duplication metrics at four points in the pipeline: the proxy client, the proxy server, regional brokers, and aggregate brokers. It audits more than 20,000 topics and uses write-ahead logging and UUIDs to ensure that audit records themselves are written with exactly-once semantics. When a discrepancy appears, operators can pinpoint which pipeline tier introduced data loss or duplication.

Consumer group observability with uGroup

uGroup emits lag metrics and stuck-partition alerts for all consumer groups, including those that are not currently running. During a multi-region failover, uGroup provides the offset state mapping needed for consumers to resume from the correct position in the target region.

Schema governance with Schema-Service and Heatpipe

Uber maintains an in-house schema registry called Schema-Service that enforces backward-compatible Avro schema evolution. The Heatpipe library, used by producers, validates messages against the registered schema at ingestion time. This prevents malformed or schema-incompatible data from entering Kafka pipelines.

Topic ownership enforcement

Uber requires ownership metadata at topic creation. Automated tooling infers ownership where possible. This is part of a broader data culture initiative to ensure every dataset - including Kafka topics - has an identifiable owner who can be contacted during an incident.

Centralised upgrades via Consumer Proxy

Before uForwarder, each of Uber’s 1,000+ consumer services maintained its own Kafka client library, often across multiple languages. Upgrading Kafka client versions required coordinating changes across hundreds of services. With uForwarder, the Kafka consumer implementation is centralised in the proxy. Client upgrades happen in the proxy without requiring changes to the services it serves.

Deployment

Uber runs Kafka as a self-managed deployment across its own infrastructure in multiple geographic regions.

Challenges and how they solved them

Challenge	Solution	Outcome
Kafka MirrorMaker caused weekly outages due to rebalancing delays of five to ten minutes	Built uReplicator with Apache Helix for static partition assignment and a DynamicKafkaConsumer	Rebalancing delays eliminated; dynamic topic whitelisting without cluster restart
Scaling partitions: 1 msg/sec throughput required one partition per consumer thread, which would require millions of partitions across 1,000+ services	Consumer Proxy multiplexes a single partition across many service instances via gRPC	Throughput scaled from 1 million to 12 million messages per second without proportional partition growth
Head-of-line blocking from slow or poisoned messages halted entire partitions	Consumer Proxy detects stuck consumers and routes problem messages to DLQ; uForwarder adds active head-of-line blocking resolution	Individual slow messages no longer stall partition processing
Backfilling streaming jobs required replaying data into Kafka, creating significant cluster load	Kappa architecture models Hive as a streaming source in Spark Structured Streaming	Historical backfill runs without additional Kafka load
Ad billing double-counting risk from at-least-once delivery	Flink transactions, Kafka read_committed, per-record UUIDs, and Pinot upserts	End-to-end exactly-once delivery for billing events
Storage and compute were coupled: longer retention required adding broker capacity	Kafka tiered storage (KIP-405) offloads old segments to S3, GCS, or HDFS	Retention extended without adding brokers
Kafka REST Proxy throughput was insufficient at 7,000 QPS per box	Internal performance optimisations	Throughput raised to 45,000 QPS per box
No visibility into consumer groups that weren’t actively running	uGroup decodes __consumer_offsets directly	Full consumer group visibility including offline consumers
Offset mapping during multi-region failover with out-of-order replication	Active/Passive mode with periodic cross-region offset synchronisation and offset mapping tables	Consumers can resume from the correct position after a regional failover

Full tech stack

Category	Tools	Notes
Message broker	Apache Kafka	Self-managed, multi-region
Schema registry	Schema-Service (internal) + Heatpipe	Avro backward compatibility enforced at ingestion
Stream processing	Apache Flink, Apache Samza, Apache Spark Structured Streaming	Flink primary for stateful workloads
Cross-cluster replication	uReplicator (open-source, Uber-built)	Replaces MirrorMaker; Apache Helix for partition assignment
Consumer proxy	uForwarder (open-source, Uber-built)	Push-based; gRPC delivery; 1,000+ services
Connectors / CDC	StorageTapper (MySQL binlog reader, internal)	CDC to Kafka; downstream to Hive and Pinot
Real-time OLAP	Apache Pinot	Native upsert for exactly-once deduplication
Data lake / warehouse	Apache Hadoop, HDFS, Apache Hive	CDC and event archival; also used as streaming source
Upsert storage	Apache Hudi	Incremental CDC updates on HDFS
Tiered storage backends	HDFS, Amazon S3, Google Cloud Storage, Azure	KIP-405 pluggable interface
Monitoring / auditing	Chaperone (internal), uGroup (internal)	End-to-end audit; consumer group lag and DR offset tracking
Cluster coordination	Apache Helix, Apache ZooKeeper	uReplicator partition assignment; cluster state
Serialisation	Apache Avro	Enforced by Heatpipe + Schema-Service
Transport (Consumer Proxy)	gRPC / Protobuf	uForwarder to consumer service endpoints
HTTP producer interface	Kafka REST Proxy (internal)	Optimised to 45,000 QPS per box
Languages	Java, Go, Python, Node.js, C++	Multi-language producer support

Key contributors

Name	Role	Contribution
Chinmay Soman	Software Engineer, Streaming Platform	Led uReplicator design; authored the uReplicator blog post
Yuanchi Ning, Xiang Fu, Hongliang Xu	Streaming Platform engineers	Co-built uReplicator
Xiaobing Li	Software Engineer, Core Infrastructure	Co-authored Chaperone blog post
Ankur Bansal	Senior Software Engineer, Streaming Team	Co-authored Chaperone; presented at Hadoop Summit 2017
Mingmin Chen	Director of Engineering, SSD Team	Hadoop Summit 2017; co-authored DR blog; uGroup
Yupeng Fu	Principal Software Engineer, SSD/Streaming Team	Disaster recovery blog; uGroup; ad events; Kafka Cluster Federation talk
Xiaoman Dong	Senior Software Engineer, Streaming Data	Kafka Cluster Federation talk; uGroup
Ovais Tariq	Sr. Manager, Core Storage	Led DBEvents CDC framework
Amey Chaugule	Senior Software Engineer, Marketplace Experimentation	Authored Kappa Architecture blog
Qichao Chu, George Teo, Haitao Zhang, Zhifeng Chen	Streaming Data Team	Co-authored Consumer Proxy blog; led uForwarder
Jacob Tsafatinos, Yuriy Bondaruk, Yupeng Fu, James Kwon	Ads Platform / Ads Billing	Co-authored exactly-once ad events blog
Ning Xia	Software Engineer, Payments Team	Authored dead letter queue blog
Abhijeet Kumar, Kamal Chandraprakash, Satish Duggana	Kafka Team	Co-authored Kafka tiered storage blog; Satish Duggana is an Apache Kafka committer and PMC member
Roshan Naik, Si Lao	Uber Engineering	Presented Exactly-Once Stream Processing at Scale at Kafka Summit London 2024

Key takeaways for your own Kafka implementation

Replication tooling matters at scale. MirrorMaker’s consumer group rebalancing caused weekly outages for Uber. If you are replicating across clusters or regions, understand how your replication tool handles topic changes and partition rebalancing before it becomes a production problem.
Partition count and consumer concurrency are separate concerns. Uber’s Consumer Proxy approach shows that you do not have to create more partitions to scale consumer throughput. A proxy or multiplexing layer can decouple the two, which is relevant if you have many small consumers or need to avoid partition-count overhead.
Exactly-once requires a coordinated strategy across producers, consumers, and sinks. Uber’s ad billing pipeline combines Flink transactions, Kafka read_committed isolation, and per-record UUIDs with Pinot upserts. No single piece provides the guarantee on its own. If you need exactly-once for a high-stakes pipeline, plan the deduplication strategy at every tier from the start.
Tiered storage changes the broker-sizing conversation. Decoupling log retention from broker disk means you can extend retention without adding brokers. If your current retention policy is constrained by storage cost, tiered storage is worth evaluating before the next round of broker capacity planning.
Centralising consumer infrastructure simplifies client upgrades. Coordinating a Kafka client upgrade across hundreds of services in multiple languages is operationally expensive. If you are managing many consumer services, a shared consumer proxy or library layer reduces the coordination overhead significantly.

Sources and further reading

#	Source
1	Chinmay Soman et al. - uReplicator: Uber Engineering’s Robust Apache Kafka Replicator - Uber Engineering Blog, 2016-08-04
2	Xiaobing Li, Ankur Bansal - Chaperone: Audit Apache Kafka End-to-End - Uber Engineering Blog, 2016-12-08
3	Ankur Bansal, Mingmin Chen - How Uber Scaled Its Real-Time Infrastructure to Trillion Events Per Day - Hadoop Summit, 2017-06-14
4	Ovais Tariq, Nishith Agarwal - DBEvents: Uber’s Ingestion Framework for Database Events - Uber Engineering Blog, 2019-03-14
5	Yupeng Fu, Xiaoman Dong - Kafka Cluster Federation at Uber - Kafka Summit SF 2019
6	Amey Chaugule - Kappa Architecture: Uber’s Approach to Unifying Streaming and Batch - Uber Engineering Blog, 2020-01-23
7	Yupeng Fu, Mingmin Chen - Disaster Recovery for Multi-Region Kafka at Uber - Uber Engineering Blog, 2020-12-21
8	Qichao Chu, George Teo, Haitao Zhang, Zhifeng Chen - Kafka Async Queuing with Consumer Proxy - Uber Engineering Blog, 2021-08-31
9	Jacob Tsafatinos et al. - Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot - Uber Engineering Blog, 2021-09-23
10	Qichao Chu et al. - Introducing uGroup: Uber’s Consumer Management Framework - Uber Engineering Blog, 2021-10-21
11	Abhijeet Kumar et al. - Kafka Tiered Storage at Uber - Uber Engineering Blog, 2024-07-01
12	Zhifeng Chen, Haifeng Chen - Introducing uForwarder - Uber Engineering Blog, 2026-02-05
13	Roshan Naik, Si Lao - Exactly-Once Stream Processing at Scale in Uber - Kafka Summit London 2024
14	Ning Xia - Reliable Reprocessing: Dead Letter Queues for Apache Kafka - Uber Engineering Blog, 2018-02-16

‍

If you want to explore your own Kafka topics and consumer groups with the kind of visibility Uber has built internally, Kpow offers a free 30-day trial and connects to any Kafka cluster in minutes.

How Walmart uses Apache Kafka in production

Factor House — Sat, 30 May 2026 00:00:00 GMT

Walmart has been running Apache Kafka in production since at least 2016, and the scale of the deployment today reflects a decade of architectural iteration. As of June 2024, Walmart’s Kafka infrastructure processes trillions of messages per day, serves 25,000+ consumers across public and private clouds, and maintains 99.99% availability. The engineering problems Kafka solves at Walmart range from near-real-time search indexing and replenishment planning across 5,000+ stores to fraud detection on every online transaction and a Customer Data Platform ingesting around 40 billion events per day.

Company overview

Walmart is the world’s largest retailer by revenue, operating roughly 10,500 stores across 19 countries and a significant e-commerce business. Its technology organisation, Walmart Global Tech, manages the platform infrastructure that supports in-store systems, e-commerce, supply chain, and customer-facing applications simultaneously.

Kafka arrived at Walmart before 2016, initially on shared bare-metal clusters that replaced Apache Flume for log collection and tracking-pixel feeds. By late 2016, engineers Ning Zhang and Anil Kumar documented the first architectural shift: moving from shared clusters to a self-serving model where teams deployed dedicated clusters via OneOps, Walmart’s internal OpenStack-based platform. That shift established the architectural pattern Walmart has built on ever since: many purpose-built Kafka clusters rather than a single shared monolith.

Key milestones

Pre-2016: Kafka deployed on shared bare-metal clusters, replacing Flume for log collection and tracking pixel ingestion
Nov 2016: Migration to self-serving dedicated clusters via OneOps; 7 datacentres, 5 local clusters, 2 aggregation clusters
Oct 2016: Kafka becomes the backbone for the near-real-time search index and hundreds of microservices in the item-setup pipeline
Nov 2017: Kafka + Apache Druid analytics cluster reaches nearly 1 billion+ events per day
Jun 2019: Inventory system ingesting 500 million events per day across 5,000+ stores; Deepak Goyal presents Kafka Streams extensions at Kafka Summit NYC
2020–2021: Navinder Pal Singh Brar presents KIP-535 and KIP-562 at Kafka Summit; both merged into Apache Kafka
May 2022: Replenishment system reaches 85 GB/min throughput across 18 brokers processing close to 100 million SKUs in under 3 hours
Jul 2022: Cassandra CDC pipeline with Debezium and Redis Bloom filters deployed at 60,000 changes per second
Jun 2024: Messaging Proxy Service launched; trillions of messages per day, 25,000+ consumers, 99.99% availability

Walmart’s Kafka use cases

Walmart uses Kafka across multiple systems, each solving a distinct operational problem. The use cases described here are drawn from published engineering posts by named Walmart engineers between 2016 and 2024.

Near-real-time search index

Anil Kumar, a Global eCommerce Engineer at Walmart Labs, described Kafka as “the backbone for our New Near Real Time (NRT) Search Index, where changes are reflected on the site in seconds” in a 2016 Confluent post. The system processes billions of pricing and inventory updates per day. Before Kafka, search index latency was measured in hours; the Kafka-backed pipeline reduced that to seconds.

Item-setup pipeline and microservices backbone

Kafka connects hundreds of microservices in Walmart’s item-setup pipeline, covering product normalisation, offers validation, pricing, inventory, and logistics data. Teams in different geographies operate autonomously because Kafka decouples producers from consumers: each team publishes to its own topics without coordinating with downstream consumers. The architecture uses many small clusters with hundreds of topics rather than one shared cluster, limiting the blast radius of any individual failure.

Real-time inventory management

Suman Pattnaik, Director of Engineering at Walmart, described a real-time inventory system in a 2020 Confluent post that replaced batch-based inventory processes. The system ingests from more than 10 sources of event streaming data and maintains a denormalised canonical view of inventory positions across stores, e-commerce, and distribution centres.

Real-time replenishment

The replenishment platform, described by Pattnaik in a 2022 Confluent post, spans 5,000+ stores, 150+ distribution centres, and 1,000+ vendors across 24+ countries. Change data capture feeds a denormalised view that passes through a planning engine holding inventory positions, forecasts, and safety stocks. The system processed close to 100 million SKUs in under three hours at 85 GB/min throughput as of 2022.

Customer Data Platform (CDP)

The Customer Backbone team, led by Navinder Pal Singh Brar (Staff Engineer) and Deepak Goyal, built a multi-tenant platform on Kafka Streams that ingests customer activity events (search, add-to-cart, transactions), builds customer identity and state using RocksDB, and triggers ML models (bid models, fraud detection, omnichannel reorder) per-event or in batch. The CDP processes around 40 billion events per day and serves processed knowledge at under 10 ms (95th-percentile) latency.

Fraud detection

Walmart runs fraud detection on every online transaction. The fraud detection application uses Kafka Streams and was the primary driver behind Walmart’s open-source contributions KIP-535 and KIP-562 — described in detail in the challenges section below.

Event stream analytics

A Druid-based analytics cluster fed from Kafka via Apache Storm ingests nearly 1 billion+ events per day (2 TB of raw data) and serves sub-second OLAP query latencies, replacing Hadoop/Hive/Presto workflows that previously took hours. Amaresh Nayak documented this pipeline on the Walmart Global Tech Blog in November 2017.

Cassandra CDC

Scott Harvester and Nitin Chhabra documented a pipeline in July 2022 that uses Debezium to capture change data from Cassandra 4.x and publish it to Kafka, supporting high-volume e-commerce databases that generate 60,000 data changes per second.

Scale and throughput

Messages per day (platform-wide, 2024): Trillions (Ravinder Matte et al., Walmart Global Tech Blog, Jun 2024)
Kafka consumers (2024): 25,000+ (Ravinder Matte et al., Walmart Global Tech Blog, Jun 2024)
Platform availability (2024): 99.99% (Ravinder Matte et al., Walmart Global Tech Blog, Jun 2024)
Customer Data Platform events/day: ~40 billion (Navinder Pal Singh Brar, Strata Data Conference NY, Sep 2019)
CDP p95 serving latency: <10 ms (Navinder Pal Singh Brar, Strata Data Conference NY, Sep 2019)
Replenishment throughput (2022): 85 GB/min (Suman Pattnaik, Confluent Blog, May 2022)
Replenishment brokers: 18 brokers; 500+ partitions per topic (Suman Pattnaik, Confluent Blog, May 2022)
Replenishment SKUs processed per run: Close to 100 million in <3 hours (Suman Pattnaik, Confluent Blog, May 2022)
Inventory events/day (2019): 500 million (projected to 650 million) (Pattnaik and Subburaj, Kafka Summit SF 2019)
Druid analytics events/day (2017): Nearly 1 billion+ (2 TB raw) (Amaresh Nayak, Walmart Global Tech Blog, Nov 2017)
Cassandra CDC changes/second: 60,000 (Harvester and Chhabra, Walmart Global Tech Blog, Jul 2022)
Early topology (2016): 7 geographic datacentres; 5 local + 2 aggregation clusters (Ning Zhang, Walmart Global Tech Blog, Nov 2016)

Walmart’s Kafka architecture

Multi-cluster topology

Walmart has consistently favoured many purpose-built clusters over a shared monolith. The early 2016 architecture ran 5 local clusters and 2 aggregation clusters across 7 geographic datacentres. Producers wrote to local clusters only; consumers listened to both local and mirrored topics simultaneously. Patched MirrorMaker with complete topic renaming replicated data between sites, designed to degrade gracefully on datacentre failure and resume from the point of failure on recovery.

The item-setup pipeline used a separate “many small clusters with hundreds of topics” model (Anil Kumar, 2016). Each team maintained its own cluster, limiting cross-team coordination and blast radius.

WCNP and hybrid cloud (current)

Consumer applications now run as container images on WCNP (Walmart Cloud Native Platform), an enterprise-grade Kubernetes-based multi-cloud container orchestration framework spanning private cloud and public cloud (Azure and Google Cloud). The MPS architecture introduced in 2024 targets stateless consumer services that auto-scale on WCNP without touching Kafka partition counts.

Kafka Streams as a distributed NoSQL database

The Customer Backbone team uses Kafka Streams not just for stream processing but as the stateful data store for the Customer Data Platform. Each customer profile is maintained in RocksDB. Enriched profiles feed downstream Kafka Streams applications for recommendations and fraud detection, creating a graph of event-driven microservices. This pattern requires custom extensions to make Kafka Streams operationally stable at scale (see Special Techniques below).

Replenishment architecture

The replenishment system uses 18 Kafka brokers with 500+ partitions per topic. CDC feeds a denormalised view that passes through a planning engine. The system operates in an active-passive replication model between datacentres and includes a fallback REST service to a database if Kafka becomes unavailable.

Cassandra CDC pipeline

Debezium 1.9 reads Cassandra 4.x commit logs continuously and publishes changes to Kafka. A Redis + RedisBloom (Bloom filter) deduplication layer, partitioned hourly, handles the 9x record fan-out caused by 3-region, replication-factor-3 deployments. Walmart’s internal Data Acquisition Tool (DAQ) orchestrates the pipeline on Azure.

Druid analytics pipeline

Kafka delivers events to Apache Storm (using Trident), which enriches them via a custom log scraper and writes to Druid. Druid pre-aggregates (rollup) at ingestion and stores data in inverted-index, columnar format for sub-second OLAP queries.

Producer architecture

For the replenishment system, Suman Pattnaik documented the following producer configuration in 2022:

Custom partitioner using a murmur hash function to ensure each item-store combination lands on a single partition
linger.ms tuning for batching
batch.size set to 16,000 bytes
acks=all

Consumer architecture

Consumer configuration for the replenishment system includes: max.poll.records and max.poll.interval.ms explicitly tuned, auto-commit disabled, and session timeout and heartbeat interval explicitly set.

As of 2024, many consumer workloads are decoupled from direct Kafka consumption via the Messaging Proxy Service (described below).

Kafka Connect ecosystem

Kafka Connect is used to sink topics to HDFS for long-term storage, with max.poll.interval.ms, max.poll.records, and session.timeout.ms tuned for the HDFS connector. Debezium (acting as a Kafka Connect source) handles Cassandra CDC ingestion.

Special techniques and engineering innovations

Messaging Proxy Service (MPS)

The most significant architectural innovation in Walmart’s recent Kafka history is the Messaging Proxy Service, published by Ravinder Matte, Vilas Athavale, Sid Anand, and colleagues in June 2024.

The standard approach to scaling Kafka consumers is to increase partition count to match parallelism. The problem with this at Walmart’s scale is that consumer group rebalances become frequent and disruptive: pod scaling events, rolling deployments, or processing delays exceeding max.poll.interval.ms all trigger rebalances that cause consumer lag and SLA violations.

MPS inserts an HTTP proxy layer between Kafka and consumer services:

A single reader thread polls Kafka and writes to a bounded in-memory PendingQueue
A writer thread pool exposes stateless REST endpoints that consumer pods call
An Order Iterator preserves per-key ordering across the proxy boundary
A Dead Letter Queue handles poison-pill messages
A separate offset commit thread manages Kafka offsets, decoupling commit timing from processing

Consumer pods become stateless services that can auto-scale on Kubernetes without triggering Kafka rebalances. Consumer count and partition count are fully decoupled. According to the Walmart engineering team, this eliminated most rebalances in production, with the exception of rare cluster restarts or network events.

Cold Bootstrap for Kafka Streams recovery

Deepak Goyal presented this technique at Kafka Summit NYC 2019. Kafka Streams recovers stateful tasks by replaying changelog topics, but for the Customer Data Platform’s large RocksDB stores, those changelogs held gigabytes of data, making standby recovery very slow.

Cold Bootstrap replaces changelog replay with a direct RocksDB snapshot copy: when a node fails, the standby copies the active node’s RocksDB snapshot directly via JSch, then resumes from the saved offset. This achieves zero event loss with significantly faster recovery, and eliminates the need for indefinite changelog topic retention.

Dynamic repartitioning in Kafka Streams

Goyal’s team added support for elastic repartitioning of Kafka Streams state across new partitions at runtime, enabling scaling to any number of partitions and nodes without a full restart. This was not available in stock Kafka Streams at the time.

Rack/AZ-aware task assignment

Active and standby Kafka Streams tasks for the same partition are explicitly assigned to different racks or availability zones, improving resilience without additional hardware.

Availability-first Kafka Streams (KIP-535 and KIP-562)

Navinder Pal Singh Brar presented this work at Kafka Summit 2020. The fraud detection application requires Kafka Streams reads on every transaction, but Kafka Streams’ default behaviour blocks reads during consumer rebalances. At Walmart’s transaction volume, this was incompatible with latency requirements.

Walmart contributed two KIPs to the Apache Kafka project:

KIP-535: Read from replica standbys and read while rebalancing — explicitly trading some consistency for higher availability
KIP-562: Read from the store of a specific partition

Both were merged into Apache Kafka. Brar holds four US patents related to Kafka Streams and was named a Confluent Community Catalyst.

Custom partitioner for dirty-write prevention

The replenishment system uses a custom partitioner based on a murmur hash function, ensuring each item-store combination always lands on a single partition and is consumed by a single consumer. This prevents database deadlocks from concurrent writes that would occur if the same item-store pair were processed by multiple consumers simultaneously.

Bloom-filter deduplication for Cassandra CDC

Each Cassandra write in a 3-region, RF=3 cluster generates 9 CDC records. Out-of-order delivery and partial column updates compound the problem. Walmart uses Redis-backed RedisBloom Bloom filters partitioned hourly, with error rates tested at 1 in 1 million, to deduplicate records before they reach downstream consumers. Production configuration achieved 623,000 deduplicated records per minute across 3 nodes.

Walmart also enhanced Debezium to process Cassandra commit logs without waiting for log file completion, reducing CDC latency further.

Operating Kafka at scale

Deployment model

Walmart’s Kafka infrastructure spans self-managed clusters on private cloud (via OneOps historically, WCNP currently) and public cloud (Azure and Google Cloud). The deployment is hybrid: Kafka brokers remain within Walmart’s infrastructure while consumer workloads run on WCNP across clouds.

Monitoring stack (2016)

Ning Zhang documented the monitoring stack in December 2016. Walmart forked Yahoo’s Kafka Manager and added custom monitoring graphs. jmxtrans bridges Kafka JMX metrics to Graphite and Ganglia backends; Grafana is layered on top of Graphite for dashboards. Alerts covered: Kafka process down, under-replicated partitions, leader loss, low disk space, and high CPU utilisation.

No sourced material describes how the monitoring stack evolved post-2016 when Walmart moved to WCNP and Kubernetes.

Producer and consumer tuning

For the replenishment system, Pattnaik documented explicit tuning of linger.ms, batch.size (16,000 bytes), and acks=all on the producer side, with max.poll.records, max.poll.interval.ms, and session timeout configured on the consumer side.

Broker hardware

For the inventory system, Suman Pattnaik specified that brokers use multiple disks with RAID configurations for log.dirs storage, with consumer counts aligned to partition counts and CPU/memory maintained at around 50% utilisation headroom.

Developer experience

The self-serving OneOps model (documented by Ning Zhang in 2016) gave teams a GUI for deploying dedicated Kafka clusters, covering operations, monitoring, auto-repair, auto-replacement, and auto-scaling. Anil Kumar’s team built internal tooling for tracking pipeline flows, SLA metrics, message send/receive latencies per producer and consumer, and alerting on backlogs and throughput degradation. A Kafka REST proxy was deployed so that non-JVM services could produce and consume without native Kafka client libraries.

Challenges and how they solved them

Consumer rebalancing causing lag at scale

Ravinder Matte’s team described consumer group rebalancing as “the most common challenge in operationalising Kafka consumers at scale” in the June 2024 post. At Walmart’s scale — 25,000+ consumers — rebalances triggered by pod scaling, rolling deployments, or processing delays exceeding max.poll.interval.ms caused high consumer lag that disrupted SLAs.

Solution: The Messaging Proxy Service decouples Kafka partition consumption from consumer services. Most rebalances are now eliminated except for rare cluster restarts or network events.

Slow Kafka Streams standby recovery

Changelog topics holding gigabytes of state for the Customer Data Platform caused very slow recovery when a Kafka Streams node failed, as the standby had to replay the entire log.

Solution: Cold Bootstrap — the standby copies the active node’s RocksDB snapshot directly via JSch, then resumes from the saved offset. This avoids replaying the changelog, achieves zero event loss, and eliminates the need for indefinite changelog retention. (Deepak Goyal, Kafka Summit NYC 2019.)

Inability to elastically scale Kafka Streams

Kafka Streams lacked native repartitioning support, which prevented the Customer Data Platform from scaling its stateful applications without full restarts.

Solution: Walmart added dynamic repartitioning to Kafka Streams, redistributing state across new partitions at runtime. This was later contributed to the community via the KIP process. (Deepak Goyal, Kafka Summit NYC 2019.)

Availability vs. consistency in fraud detection

Kafka Streams’ default behaviour blocks reads during consumer rebalances. For a fraud detection application running on every transaction, this was incompatible with latency SLAs.

Solution: Walmart contributed KIP-535 and KIP-562 to Apache Kafka, explicitly choosing availability over strict consistency for this workload. “You’re basically trading consistency for availability,” Brar stated in a TechTarget interview. (Navinder Pal Singh Brar, Kafka Summit 2020.)

Shared cluster resource contention

Early shared bare-metal clusters suffered from capacity competition between tenants, no authentication or isolation, and buggy clients exhausting resources.

Solution: Migrated to dedicated clusters via OneOps, giving each team its own Kafka cluster with GUI-driven operations, auto-repair, and auto-scaling. (Ning Zhang, November 2016.)

Cassandra CDC record fan-out

A single Cassandra write in a 3-region, RF=3 deployment generates 9 CDC records, arriving out of order and containing only changed columns.

Solution: Debezium 1.9 in continuous log processing mode, combined with Redis-backed RedisBloom Bloom filters partitioned hourly, deduplicates records at 60,000 changes per second in production. Walmart also enhanced Debezium to process commit logs without waiting for file completion. (Scott Harvester and Nitin Chhabra, July 2022.)

Poison-pill messages causing head-of-line blocking

Unprocessable messages blocked partition consumption in direct consumer deployments.

Solution: The Messaging Proxy Service routes failed messages to a Dead Letter Queue with retry logic, isolating them from the main processing path. (Ravinder Matte et al., June 2024.)

Full tech stack

Category	Tools	Notes
Message broker	Apache Kafka	Kafka 0.10.1.0 documented in 2016; current version not specified in sourced material
Stream processing	Kafka Streams, Apache Storm (Trident), Apache Spark Streaming	Kafka Streams: Customer Data Platform, fraud detection, inventory; Storm: Druid ingestion; Spark: inventory and A/B testing
State store	RocksDB	Local state store for Kafka Streams applications in the CDP
Connectors	Kafka Connect (HDFS sink), Debezium 1.9 (Cassandra CDC source)	HDFS connector for long-term storage; Debezium enhanced for continuous log processing
Cross-datacenter replication	Kafka MirrorMaker (patched)	Internal patches added complete topic renaming to prevent name collisions across sites (2016)
HTTP interface	Kafka REST Proxy	Enables non-JVM producers and consumers (2016)
Analytics	Apache Druid	Sub-second OLAP queries on nearly 1 billion+ events/day; pre-aggregation at ingestion
Operational database	Apache Cassandra	Primary operational store for inventory and customer data; CDC source for Debezium
Deduplication	Redis, RedisBloom	Bloom-filter deduplication for Cassandra CDC fan-out at 60,000 changes/second
Storage sinks	Apache HDFS, Apache Hudi	HDFS via Kafka Connect for long-term retention; Hudi for lakehouse table format (as of 2023)
Monitoring (2016)	Kafka Manager (Yahoo fork, enhanced), jmxtrans, Graphite, Ganglia, Grafana	Custom monitoring graphs added to Kafka Manager; Grafana layered on Graphite
Cluster coordination	Apache ZooKeeper	Referenced in 2019 material; current coordination mechanism not specified
Deployment / infra (legacy)	OneOps (OpenStack-based)	Self-serve Kafka cluster deployment with GUI, auto-repair, and auto-scaling (pre-WCNP)
Deployment / infra (current)	WCNP (Walmart Cloud Native Platform)	Kubernetes-based multi-cloud orchestration spanning Azure and Google Cloud
Internal proxy	Messaging Proxy Service (MPS)	Internal HTTP proxy decoupling Kafka consumption from consumer pod scaling (2024)
Client libraries	Spring Kafka, Akka Streams	Spring Kafka in replenishment system; Akka Streams for early reactive microservice consumption (2016)
Public cloud	Azure, Google Cloud	Both used for WCNP workloads; Azure hosts Cassandra CDC pipeline

Key contributors

Ning Zhang (Software Engineer, Walmart Global Tech): Authored two foundational 2016 posts on Walmart’s Kafka ecosystem and multi-datacenter architecture
Anil Kumar (Global eCommerce Engineer, Walmart Labs): Authored the 2016 Confluent post on Kafka for item setup and near-real-time search indexing
Suman Pattnaik (Director of Engineering, Walmart): Architected real-time inventory and replenishment platforms; co-presented at Kafka Summit SF 2019; authored Confluent blog posts (2020, 2022)
Prasanna Subburaj (Engineer, Walmart): Co-presented “When Kafka meets the scaling and reliability needs of world’s largest retailer” at Kafka Summit SF 2019
Deepak Goyal (Engineer, Walmart Labs, Customer Backbone team): Developed Cold Bootstrap, dynamic repartitioning, and rack-aware task assignment for Kafka Streams; presented at Kafka Summit NYC 2019
Navinder Pal Singh Brar (Staff Engineer, Walmart Global Tech, Customer Data Platform): Founding member of Walmart’s CDP; contributed KIP-535 and KIP-562 to Apache Kafka; holds 4 US patents; Confluent Community Catalyst
Amaresh Nayak (Engineer, Walmart Global Tech): Authored the 2017 post on Druid + Kafka event stream analytics
Scott Harvester (Engineer, Walmart Global Tech): Co-authored the 2022 Cassandra CDC solution post
Nitin Chhabra (Engineer, Walmart Global Tech): Co-authored the 2022 Cassandra CDC post
Ravinder Matte (Engineer, Walmart Global Tech): Lead author of the 2024 Messaging Proxy Service post
Chris Kasten (VP of Walmart Cloud): Discussed Walmart’s real-time platform in Kafka Summit SF 2019 keynote Q&A

Key takeaways for your own Kafka implementation

Decouple consumers from partitions before you hit rebalance problems. Walmart’s Messaging Proxy Service (2024) addressed a challenge that many large-scale Kafka deployments encounter: consumer group rebalances at pod-scale. If you’re running containerised consumers on Kubernetes, consider whether a proxy layer that absorbs Kafka consumption and exposes stateless REST endpoints would reduce operational complexity before the problem becomes acute.
Consider Kafka Streams changelog recovery carefully when your state is large. Walmart’s Cold Bootstrap approach (replacing changelog replay with direct RocksDB snapshot copy via JSch) addressed a real recovery bottleneck. If your Kafka Streams state stores are measured in gigabytes, the default changelog replay strategy may result in slow standby recovery. Understanding this trade-off early lets you design around it.
Many small clusters often work better than one large shared cluster. From 2016, Walmart operated purpose-built clusters per team and per pipeline rather than a shared monolith. This limits blast radius, allows per-cluster tuning, and avoids cross-team coordination overhead in operations like broker upgrades and partition reassignments.
Custom partitioners are worth the complexity when ordering or write-isolation matters. Walmart’s murmur-hash partitioner, which routes each item-store combination to a single partition, prevented database deadlocks that would otherwise occur from concurrent writes. If your consumers write to a keyed data store, a custom partitioner aligned to the store’s key can eliminate a class of concurrency bugs.
Open-source contributions can close gaps faster than internal workarounds. Walmart’s KIP-535 and KIP-562 contributions (availability-first Kafka Streams for fraud detection) were eventually merged into Apache Kafka. Contributing upstream meant the solution became part of the core, reducing the maintenance burden of carrying a private fork.

Sources and further reading

Ning Zhang, “Tech Transformation: Real-time Messaging at Walmart,” Walmart Global Tech Blog, November 2016
Ning Zhang, “Kafka Ecosystem on Walmart’s Cloud,” Walmart Global Tech Blog, December 2016
Anil Kumar, “Apache Kafka for Item Setup at Walmart,” Confluent Blog (Walmart Labs), October 2016
Amaresh Nayak, “Event Stream Analytics at Walmart with Druid,” Walmart Global Tech Blog, November 2017
Suman Pattnaik and Prasanna Subburaj, “When Kafka Meets the Scaling and Reliability Needs of World’s Largest Retailer,” Kafka Summit SF 2019
Deepak Goyal, “Kafka Streams at Scale: Walmart’s Approach,” Kafka Summit NYC 2019
Navinder Pal Singh Brar, “Multi-tenant Kafka Streams CDP at Walmart,” Strata Data Conference New York, September 2019
Suman Pattnaik, “Real-time Inventory Management at Walmart Using Kafka,” Confluent Blog, May 2020
Navinder Pal Singh Brar, “Availability-first Kafka Streams at Walmart Scale,” Kafka Summit 2020
Sean Michael Kerner (citing Navinder Pal Singh Brar), “Kafka Users Northrop Grumman, Walmart Highlight Event Streaming,” TechTarget, August 2020
Suman Pattnaik, “How Walmart Uses Kafka for Real-time Omnichannel Replenishment,” Confluent Blog, May 2022
Scott Harvester and Nitin Chhabra, “Walmart’s Cassandra CDC Solution,” Walmart Global Tech Blog, July 2022
Samuel Guleff, “Lakehouse at Fortune 1 Scale,” Walmart Global Tech Blog, May 2023
Ravinder Matte, Vilas Athavale, Sid Anand et al., “Reliably Processing Trillions of Kafka Messages Per Day,” Walmart Global Tech Blog, June 2024

If you’re monitoring a Kafka environment at scale, Kpow offers a free 30-day trial that connects to any Kafka cluster in minutes and deploys via Docker, Helm, or JAR.

Data governance for Kafka: introducing lineage support in Factor Platform

Chad Harris — Fri, 29 May 2026 00:00:00 GMT

At Factor House, many of the teams we work with operate in highly regulated industries. NORD/LB, a German commercial bank, is a good illustration of why banking tends to be the most demanding of these. As an institution subject to both BCBS 239 and DORA, they are required to maintain clear, demonstrable data lineage across their environment: knowing where data originates, who owns it, and how it flows through their systems. Running Kafka at scale, they needed that lineage to be visible and actionable within the tools used to operate their streaming infrastructure, not isolated in a separate catalog that engineers rarely consult during day-to-day work.

This gap prompted us to build data governance support into Factor Platform, our unified management layer for streaming infrastructure. This article explains what we’ve built, why it matters for regulated organisations, and the approach we took to make it practical.

The regulatory context

European banks are subject to two frameworks that place direct requirements on data governance: BCBS 239 and DORA.

BCBS 239, the Basel Committee’s principles for effective risk data aggregation and risk reporting, requires systemically important financial institutions to demonstrate clear data lineage, defined data ownership, and the ability to trace data from source to consumption. For banks running event-driven architectures, this means being able to answer questions like: where does this data come from, who owns it, and does it contain regulated information?

DORA, the EU’s Digital Operational Resilience Act, adds further requirements around understanding ICT systems and their data dependencies as part of broader operational resilience obligations.

Both frameworks assume that an organisation has a coherent view of its data. In practice, for teams running Kafka at scale, that view has historically been difficult to maintain.

The problem with existing approaches

Most organisations with a data governance programme have invested in a data catalog. These catalogs accumulate metadata about datasets: ownership, classification, documentation, PII flags, and so on. That metadata is typically described using the OpenLineage standard, a vendor-neutral specification for representing datasets, jobs, and their relationships.

The challenge is that this metadata tends to live in a separate system from the Kafka tooling. Engineers operating Kafka can browse topics and schemas, but the governance context (who owns this schema, whether it contains PII, whether it meets the organisation’s data quality standards) isn’t visible alongside it. The two systems are not integrated, and bridging them has typically required additional infrastructure and dedicated integration work.

Our approach: the schema as the source of truth

Factor Platform’s lineage support is built around a straightforward observation: Kafka schemas already exist and are version-controlled. Rather than requiring a separate metadata store or additional infrastructure, we treat the schema itself as the place where governance metadata lives.

You annotate your existing Avro or JSON Schema definitions with lineage metadata - ownership, tags, documentation, PII classification, domain, and any custom attributes your organisation requires - using whatever field structure makes sense for your environment. Factor Platform then uses kJQ expressions to map that embedded metadata to valid OpenLineage Dataset facets, producing a structured, OpenLineage-conforming description of each dataset from your Schema Registry.

This means there is no additional system to run, no separate pipeline to maintain, and no duplication of your schema definitions. The metadata travels with the schema, which is where it belongs.

Mappings can be configured through the Factor Platform UI using a guided wizard, or through the API. Once in place, Factor Platform evaluates them on each schema observation cycle and surfaces the results across the interface. Datasets are versioned, so you have a full history of changes over time, including when an owner was added, when a tag changed, and when a new schema version introduced a new PII field.

What this makes possible

Bringing OpenLineage metadata into your Kafka operational environment changes what you can see and act on.

Identifying PII-tagged data. Column-level mappings allow individual fields within a schema to carry their own lineage metadata. A field like email or date_of_birth can be annotated with a PII flag and a classification. Factor Platform surfaces this when you browse schemas, so engineers and compliance teams can see at a glance which topics carry regulated fields.

Finding ownership gaps. Ownership is a required facet in many governance frameworks. When a schema has no owner mapping, or when the owner field is null or malformed, Factor Platform reports it as a dataset quality issue. This gives you a filtered view of all schemas in your environment that are missing the governance properties your organisation requires, surfacing them before an auditor does.

Catching schema quality issues early. Dataset mappings can mark fields as required. If an expected metadata field is absent or returns an unexpected value, Factor Platform flags it. This shifts schema governance from a periodic manual review process to a continuous, automated check that runs on every observation cycle.

Filtering and operating with context. Once lineage metadata is extracted, it becomes a first-class filter in the Factor Platform UI. You can browse schemas by domain, by owner, by tag, or by any custom attribute your mapping configuration exposes. For a team managing hundreds of schemas across multiple clusters, this makes governance questions significantly more practical to answer quickly.

Applying filters in Factor Platform using OpenLineage metadata

What’s coming next

The current release focuses on Schema Registry as the lineage source. In future releases, Factor Platform will also consume OpenLineage events from other producers in your stack, including Flink jobs and Iceberg tables, and will produce OpenLineage events that you can feed into other systems in your governance toolchain.

Speak to our team

Factor Platform is currently in early access. If you’re working in a regulated environment and want to understand how lineage support could work for your Kafka setup, book a time with our team.

Accelerating incident response: advanced filters, streaming search, and AI-powered queries

Factor House — Thu, 28 May 2026 00:00:00 GMT

Overview

In the first part of this series, we explored how establishing foundational data inspection allows teams to turn raw, unparsed data dumps into readable, shaped payloads. However, achieving visibility is only the first half of the equation. When a production incident occurs, engineering teams must optimize for speed.

Finding a rare anomaly across millions of Kafka messages requires high-velocity search capabilities. This article explores the root causes of friction during critical investigations and demonstrates how Kpow combines server-side filtering, artificial intelligence, and continuous scanning to accelerate incident response.

This is Part 2 of the Kafka Data Management with Kpow: Unlocking Engineering Productivity series. You can read the full strategy in the main series article and access the associated posts as they become available:

Part 1: Foundational Kafka Data Inspection: Shaping Payloads and Optimizing Visibility
Part 2: Accelerating Incident Response: Advanced Filters, Streaming Search, and AI-Powered Queries (This article)
Part 3: Triage, Repair, and Replay: Integrated Kafka Remediation Workflows
Part 4: Defense in Depth: Unifying RBAC and Data Policies for Transparent Governance

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Start your free 30-day trial or explore our live multi-cluster demo environment to see Kpow in action.

Problem: Friction in Incident Response

Application teams frequently face a Velocity Gap during production outages. High-volume topics process thousands of events per second. Locating a specific, rare failure requires sifting through millions of messages. To find these precise events, developers must construct complex search logic to match timestamps, evaluate nested array values, or isolate specific error codes.

Writing this logic introduces a steep syntax learning curve. Figuring out the exact code to filter deeply nested data takes time, which directly delays incident response. When engineers are forced to focus on how to query the data rather than analyzing the data itself, resolution times increase dramatically.

Limitations of Manual Search Workflows

Standard operational workflows often fail under the pressure of an active incident. Engineers using basic consumer scripts are forced to manually advance offsets, repeatedly executing commands and hoping to catch the right event before consumer timeouts occur.

This trial-and-error approach scales poorly. During an outage, developers struggle with basic JSON parsers that cannot handle complex type conversions or temporal math. To bypass the learning curve, teams rely on sharing massive text files of old, brittle query commands. Relying on static cheat sheets rather than dynamic search tools adds severe friction to the resolution process.

Advanced Filtering, AI, and Streaming Search in Kpow

Kpow eliminates the friction of manual polling and syntax struggles by providing a high-velocity search engine. By integrating advanced kJQ filtering, artificial intelligence, and automated query progression, Kpow transforms incident response into a frictionless workflow.

Precision with Advanced kJQ Filtering

To isolate highly specific events, Kpow provides advanced kJQ filtering. This JQ-like language executes directly on the server, allowing Kpow to rapidly slice through data before it ever reaches the browser. kJQ natively supports complex logical operators, type casting, and precise ISO 8601 duration mathematics.

For example, an engineer investigating un-audited, high-value provisional trades over the last two hours can execute the following advanced filter:

.value.partner.network == "VISA" and .value.trade.status == "provisional" and .value.trade.compliance.audit == false and .value.trade.price | to-double > 40 and .timestamp > now - pt120m

This single query isolates specific nested string values, evaluates a boolean field (compliance.audit), casts a string-based price to a double for mathematical comparison, and applies a strict 120-minute temporal window using record metadata.

Eliminating the Syntax Barrier with AI

While advanced kJQ is powerful, writing complex syntax during a high-pressure outage can be daunting. To make this functionality instantly accessible to all engineers, Kpow integrates Bring Your Own AI (BYO AI) capabilities supporting AWS Bedrock, OpenAI, Anthropic, and Ollama.

Operators can bypass the syntax learning curve entirely by using natural language prompts. If an engineer needs to find specific transactions, they simply type their request into the AI prompt interface:

Show me all trades where partner network is VISA

Kpow’s AI integration instantly translates this natural language request into a schema-aware, syntactically validated kJQ filter. This allows developers to interrogate complex data structures effortlessly.

Continuous Discovery with Streaming Search

Finding a rare event often means searching beyond the initial batch of results. Instead of forcing users to manually click to continue polling, Kpow offers Streaming Search.

By enabling the Streaming Search option, Kpow automatically and continuously progresses the query. The search runs persistently until the defined result limits are reached or the topic partitions are completely exhausted. This replaces manual polling with automated discovery, allowing engineers to define their search criteria via natural language or kJQ while Kpow handles the continuous data scanning.

Conclusion

In modern streaming architectures, minimizing the time to resolution requires highly automated, intelligent search capabilities. By combining advanced server-side data slicing with continuous streaming search, Kpow eliminates the need for manual offset tracking. Furthermore, by integrating AI models, Kpow removes the syntax learning curve, allowing any developer to instantly generate complex filters using natural language.

Once the problematic event is isolated, teams must take action to correct the pipeline. In the next part of this series, Triage, Repair, and Replay: Integrated Kafka Remediation Workflows, we will explore how Kpow enables teams to safely manage consumer offsets, re-inject dead-letter queue messages, and repair data payloads directly from the UI.

Next steps

Explore Kpow in your own environment with a free 30-day trial.

If you need assistance managing your Kafka environment, reach out to our engineering support team at support@factorhouse.io.

Defense in depth: unifying RBAC and data policies for transparent governance

Jaehyeon Kim — Thu, 28 May 2026 00:00:00 GMT

Overview

Throughout this series, we have explored how to shape data for foundational visibility, accelerate investigations with AI, and integrate workflows to repair broken pipelines. However, granting developers the power to inspect and modify production data introduces significant risk.

Balancing engineering velocity with regulatory compliance is a major enterprise challenge. Giving application teams access to production topics often exposes Personally Identifiable Information (PII) or sensitive financial records. This article explores the friction inherent in generic security models and demonstrates how Kpow unifies access control and payload redaction to safely empower developers.

This is Part 4 of the Kafka Data Management with Kpow: Unlocking Engineering Productivity series. You can read the full strategy in the main series article and access the associated posts below:

Part 1: Foundational Kafka Data Inspection: Shaping Payloads and Optimizing Visibility
Part 2: Accelerating Incident Response: Advanced Filters, Streaming Search, and AI-Powered Queries
Part 3: Triage, Repair, and Replay: Integrated Kafka Remediation Workflows
Part 4: Defense in Depth: Unifying RBAC and Data Policies for Transparent Governance (This article)

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Start your free 30-day trial or explore our live multi-cluster demo environment to see Kpow in action.

Problem: Compliance Risk in Production Environments

As organizations scale their streaming infrastructure, developers inevitably need to debug production issues. A failed transaction or a stalled consumer requires application teams to look at the exact data causing the problem.

This creates a severe “Compliance Gap”. Production topics contain sensitive data. Regulatory constraints (such as GDPR, HIPAA, or SOC2) strictly prohibit exposing unmasked PII, financial records, or secure tokens to broad engineering teams. Organizations must find a way to let developers fix their applications without violating these critical privacy laws.

Limitations of Manual Access Control

To mitigate compliance risks, platform administrators typically enforce a total lockdown. They block all direct access to production Kafka clusters for anyone outside of a small group of trusted infrastructure engineers.

While this solves the security problem, it creates a massive operational bottleneck. Developers are forced into disconnected, manual ticketing processes. When an incident occurs, an application engineer must submit a ticket and wait for a platform engineer to manually execute CLI scripts, extract the relevant records, sanitize the logs by hand, and return the safe data. This bureaucratic process destroys developer velocity and wastes valuable platform engineering time.

Unified Declarative Governance with Kpow

Kpow eliminates this operational bottleneck by introducing transparent governance. By utilizing a “Defense in Depth” approach, Kpow secures both administrative actions and data payloads using a shared, declarative YAML resource taxonomy.

Tier 1: Action-Level Control with Granular RBAC

The first layer of defense controls what actions a user can perform. Kpow provides highly granular Role Based Access Control (RBAC), allowing administrators to map Identity Provider roles to precise permissions.

Platform administrators can safely grant a TOPIC_INSPECT action to application teams (such as a kafka-readers role) so they can view data. Simultaneously, administrators can enforce an implicit deny or explicit Stage effect on sensitive mutations like TOPIC_CREATE or TOPIC_PRODUCE. If a user with read-only access attempts to execute an unauthorized infrastructure change, Kpow actively prevents the action and logs the attempt.

Tier 2: Payload-Level Control with Data Policies

Action-level access is necessary but insufficient on its own. If a developer is permitted to inspect a topic, they still cannot be allowed to see raw financial data. Kpow solves this by layering Data Policies directly on top of inspection rules.

Data Policies apply nested redaction to specific fields automatically. For example, consider an orders topic containing a sensitive order_id and a financial amount. Administrators can define a policy that applies a ShowFirst4 redaction to the order_id and a Full redaction to the amount field.

When an authorized developer inspects the topic to debug a stalled order, Kpow automatically renders the masked record. The developer can verify the operational status of the payload without ever exposing the restricted values.

Ensuring Fail-Safe Protection

To guarantee regulatory compliance, Kpow builds fail-safe mechanics into its transparent governance model. If a schema evolves and a previously simple string field becomes a complex nested object, Kpow utilizes a conservative fallback mechanism. Rather than risking exposure by failing to apply a partial redaction, the system automatically defaults to a Full redaction for that field.

Furthermore, Kpow intentionally disables String SerDes options in the UI when Data Policies are active. This architectural choice prevents users from bypassing JSON or Avro deserialization to read the raw, unmasked bytes directly from the topic.

Conclusion

Combining granular RBAC with automated Data Policies completely breaks the disconnected ticketing bottleneck. Platform administrators no longer need to act as manual data extractors. By mathematically guaranteeing data security at both the action and payload levels, platform teams can finally offer safe, self-service debugging in production environments.

This concludes our four-part series on enterprise Kafka data management. By closing the Visibility, Velocity, Remediation, and Compliance gaps, organizations can eliminate operational friction. Unifying data inspection, AI-powered search, pipeline repair, and transparent governance ultimately transforms Kafka from a complex infrastructure burden into an engine that unlocks true engineering productivity.

Next steps

Explore Kpow in your own environment with a free 30-day trial.

If you need assistance managing your Kafka environment, reach out to our engineering support team at support@factorhouse.io.

Foundational Kafka data inspection: shaping payloads and optimizing visibility

Jaehyeon Kim — Thu, 28 May 2026 00:00:00 GMT

Overview

While Kafka provides massive throughput, extracting actionable signal from high-volume streams presents a significant operational challenge that can easily disrupt enterprise engineering velocity. Among these challenges, the most significant is the “Visibility Gap”. This inherent opacity of streaming data leaves application teams struggling to parse complex payloads and locate specific events during critical incidents.

Turning raw Kafka payloads into readable, searchable data is the first step toward faster operational response. This article explores the root causes of data opacity and demonstrates how Kpow provides a structured, UI-driven approach to foundational data inspection to close the “Visibility Gap”.

This is Part 1 of the Kafka Data Management with Kpow: Unlocking Engineering Productivity series. You can read the full strategy in the main series article and access the associated posts as they become available:

Part 1: Foundational Kafka Data Inspection: Shaping Payloads and Optimizing Visibility (This article)
Part 2: Accelerating Incident Response: Advanced Filters, Streaming Search, and AI-Powered Queries
Part 3: Triage, Repair, and Replay: Integrated Kafka Remediation Workflows
Part 4: Defense in Depth: Unifying RBAC and Data Policies for Transparent Governance

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Start your free 30-day trial or explore our live multi-cluster demo environment to see Kpow in action.

Problem: Opacity in High-Volume Topics

Application teams often lack direct insight into their topic data. In a typical microservices environment, a single topic might process thousands of messages per second containing deeply nested JSON or Avro structures. During a routine investigation, developers face overwhelming visual noise when trying to read these massive payloads.

Furthermore, teams suffer from a lack of search context regarding partition-level scanning progress. When searching across millions of messages for a specific correlation ID, developers have no idea how far along a partition scan has progressed or if they have completely exhausted the available offsets. This lack of feedback turns event discovery into an opaque, uncertain process.

Limitations of Disconnected CLI Tools

To solve these challenges, engineers often rely on standard open-source consumer scripts. This approach immediately introduces significant configuration overhead. Because enterprise environments utilize various data formats, engineers are forced to manually supply complex schema registry flags and deserialization properties for every single query they execute.

Once the consumer is finally configured, the output creates another bottleneck. Terminal screens are flooded with raw, unparsed payloads. Without the ability to shape or filter the output effectively, it becomes nearly impossible to quickly identify the relevant fields required for an investigation.

Streamlined Inspection with Kpow

Kpow eliminates the friction of CLI-based investigations by providing a powerful data inspect engine for querying and filtering topic data. This enables application teams to query topics directly from the browser while shaping the data for maximum readability.

Instant Schema Inference with Auto SerDes

To remove manual configuration overhead, Kpow features an intelligent Auto SerDes option. Instead of manually specifying deserializers and registry URLs, application teams simply select a topic. Kpow automatically analyzes the stream, infers the correct format (such as Avro, Protobuf, or JSON), and deserializes the payload correctly. This capability provides immediate data inspection without requiring prior knowledge of topic serialization formats.

Filtering and Shaping Payloads with kJQ and Projection Expressions

Once data is accessible, finding the right events and improving payload readability become critical. Kpow addresses the search challenge by introducing kJQ filters, a fast, JQ-like filtering language built specifically for Kafka topics. Designed for high performance, kJQ can easily scan tens of thousands of messages per second, allowing developers to rapidly search and isolate specific events directly on the server.

After isolating the relevant records, users can refine the visual output using Projection Expressions. This feature enables developers to extract specific fields from a record by providing a comma-separated list of kJQ object identifiers.

You can apply projection expressions to both the key and value fields of Kafka records. This is especially useful when dealing with extremely large payloads where you only need a subset of fields. For example, applying .value.user_id, .value.transaction.status to a massive JSON payload collapses the result into a clean, targeted subset of data. This shapes the output for immediate human readability and eliminates visual clutter.

Transparent Query Context

To solve the problem of blind polling, Kpow surfaces detailed execution metrics directly in the UI. By expanding the Results Metadata Table within the query results toolbar, users gain instant feedback on partition-level scanning progress.

The table displays the query start and end offsets, the exact number of scanned records, and the remaining offsets available beyond the query window. This query transparency guarantees that engineers always know exactly how much of the topic has been searched and whether a partition is fully exhausted.

Optimizing Inspection for Enterprise Scale

To support fast, concurrent searches across large engineering teams, administrators can precisely tune Kpow’s underlying query infrastructure to match their specific cluster capacity and user demand.

Rather than relying on a rigid, one-size-fits-all approach, platform teams can adjust core configuration settings to balance developer velocity against broker load:

Worker Concurrency: Administrators can scale the number of active workers handling search requests, allowing multiple users to execute complex queries simultaneously without experiencing queuing bottlenecks.
Consumer Thread Pools: By increasing the number of consumer threads assigned to each worker, teams can enable parallel partition scanning to significantly improve overall query throughput.
Resource Protection: Configurable query timeouts ensure that long-running searches do not monopolize cluster resources, while tunable poll durations allow administrators to optimize performance for topics containing exceptionally large messages.

Conclusion

In modern streaming architectures, the ability to inspect and search live data quickly is no longer optional. Kpow reduces the time between detecting an issue and understanding it. By eliminating configuration friction and providing tools to shape nested payloads, Kpow transforms Kafka data inspection into a streamlined, self-service workflow. Developers no longer need to wrestle with terminal outputs or guess schema formats. With transparent query context and optimized concurrent workers, application teams gain the baseline insight required to maintain complex streaming architectures.

Once foundational readability is established, teams can tackle more complex, high-pressure scenarios. In the next part of this series, Accelerating Incident Response: Advanced Filters, Streaming Search, and AI-Powered Queries, we will explore how Kpow accelerates investigations with advanced kJQ filtering, automates continuous data scanning, and leverages AI to generate complex query syntax instantly.

Next steps

Explore Kpow in your own environment with a free 30-day trial.

If you need assistance managing your Kafka environment, reach out to our engineering support team at support@factorhouse.io.

Kafka Data Management with Kpow: Unlocking Engineering Productivity

Factor House — Thu, 28 May 2026 00:00:00 GMT

Overview

Event-driven architectures promise massive scalability and decoupled agility. In reality, the day-to-day experience of building and maintaining these systems often reveals underlying friction. While infrastructure teams have mastered the art of keeping brokers online, application teams frequently face obstacles when trying to access, debug, or correct the actual payloads flowing through their topics.

This disconnect creates a drag on engineering velocity. When a transaction fails in production, a downstream database crashes, or a poison pill stalls a consumer, developers need instant access to structured data. Instead, teams are slowed down by manual data configuration, forced to build custom tools to find specific events in high-volume streams, or blocked by restrictive permissions that prevent them from accessing the data they need.

Organizations require a structural shift away from fragmented command-line tasks toward a frictionless, self-service developer experience. This article outlines a comprehensive strategy to identify and mitigate the four critical gaps causing friction in enterprise Kafka data management, demonstrating how Kpow serves as the unified engine to unlock engineering productivity.

To see how these concepts translate into practical workflows, follow along with our upcoming four-part technical guide:

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Start your free 30-day trial or explore our live multi-cluster demo environment to see Kpow in action.

Four Critical Gaps in Enterprise Data Management

Unlocking developer productivity requires identifying and mitigating the operational bottlenecks that occur when teams interact with Kafka data.

Visibility Gap: Opacity of High-Volume Data

Application teams often lack immediate visibility into topic data. During an investigation, determining partition-level scanning progress can be difficult, and large, deeply nested JSON or Avro payloads are frequently hard to parse. Engineers often rely on standard CLI consumers, which require manual reconfiguration of schema registry properties for different topics and typically output unformatted payloads. Furthermore, configuring performant, secure inspection tools in containerized environments presents an ongoing challenge for platform administrators.

Velocity Gap: Friction of Incident Response

Finding a specific event in a high-volume topic often requires sifting through millions of messages. Isolating a single failure buried within deeply nested JSON payloads requires developers to write complex search queries or command-line filters. Figuring out the exact code to filter this data takes time, which directly delays incident response. Standard workflows force teams to write custom parsing scripts, manually step through offsets, or rely on shared text files of old query commands. During a high-pressure outage, this trial-and-error approach to finding data adds severe friction to the resolution process.

Remediation Gap: Fragmented Pipeline Repair

Fixing data pipelines extends beyond identifying a schema mismatch that stalls a consumer. Teams must frequently triage Dead Letter Queues (DLQs) for validation errors, or rewind consumer offsets when a downstream database crashes. However, extracting this data, unblocking the consumer, and repairing the pipeline involve separate, fragmented workflows. Developers may write custom scripts to find errors or bulk-publish missing headers, while simultaneously coordinating with platform engineers to manually reset consumer offsets via CLI commands. This fragmentation introduces delays in restoring data pipelines and extracting clean data for reporting.

Compliance Gap: Risk of Exposed Payloads

Organizations face strict regulatory constraints regarding Personally Identifiable Information (PII). Providing application teams with unrestricted access to production topics introduces compliance risks. To mitigate this, platform administrators often restrict access, requiring developers to use ticketing systems to request data. Platform engineers must then manually extract and sanitize logs before returning the data, creating an operational bottleneck that slows down development and debugging.

Kpow Solution: Unifying Data Workflows

Overcoming these gaps requires a unified platform that provides deep inspection capabilities, efficient search, integrated remediation, and robust data governance. Kpow addresses these requirements across four key operational dimensions.

Bridging Visibility with Foundational Data Inspection

Kpow brings structured clarity to cluster data through its Data Inspect capabilities. By leveraging Auto SerDes, Kpow infers schemas automatically, eliminating the manual configuration overhead associated with traditional consumers. Users can apply kJQ filters and Projection Expressions to extract specific nested fields, simplifying data analysis. Additionally, the Results Metadata Table provides transparent, partition-level query context and offset exhaustion metrics. Platform teams can also tune worker configuration variables to ensure stable, performant access across the enterprise.

Accelerating Velocity with AI and Streaming Search

To accelerate incident response, Kpow equips teams with powerful tools for high-velocity data analysis. Users can leverage advanced kJQ filtering for rapid, server-side data slicing, for example by evaluating date mathematics or extracting data from embedded JSON arrays. To make this advanced syntax instantly accessible and eliminate the traditional learning curve, Kpow integrates Bring Your Own AI (BYO AI) capabilities supporting AWS Bedrock, OpenAI, Anthropic, and Ollama. Operators can simply use natural language prompts to automatically generate schema-aware, syntactically validated kJQ filters. Furthermore, Streaming Search provides continuous, automatic query progression until result limits are reached or topics are completely exhausted.

Closing Remediation with Integrated Workflows

Kpow consolidates the remediation lifecycle for a wide range of pipeline failures. For serialization errors, users can leverage one of the deserialization options (such as Poison only) to easily isolate malformed data. If a downstream database fails, Kpow incorporates consumer group actions, allowing operators to explicitly skip, clear, or reset offsets, for example, by rewinding to a specific timestamp to replay historical data. For DLQ triage, native UI routing allows teams to send queried results directly to Data Produce. This enables operators to correct validation errors, append missing headers, and seamlessly re-inject messages, or generate safe tombstone records for targeted deletion. Throughout these workflows, Kpow supports data downloads in standard business formats for offline auditing as well.

Ensuring Compliance with Defense in Depth

Kpow helps reduce reliance on manual ticketing through transparent data governance, framing security as a two-tiered model using a shared YAML resource taxonomy. In the first tier, granular Role Based Access Control (RBAC) allows administrators to grant TOPIC_INSPECT permissions while explicitly denying or staging sensitive TOPIC_PRODUCE actions. In the second tier, Data Policies layer onto inspection rules to apply nested redaction (such as ShowLast4 on a credit card field). This preventative control enables self-service debugging in production while maintaining PII protection.

Achieving Engineering Productivity

Improving engineering productivity requires reducing the operational friction between developers and their streaming data. It demands a strategy that provides clear visibility, accelerates search workflows, integrates data and pipeline repair, and transparently secures both administrative actions and sensitive payloads. By adopting a unified workflow for Kafka data management, organizations can minimize administrative bottlenecks, enable self-service operations, and better leverage their streaming architecture.

Equipping teams with the right tools makes Kafka a more accessible and efficient data backbone. By streamlining data discovery, remediation, and compliance workflows, Kpow enables engineering organizations to focus their efforts on building resilient, real-time applications.

‍

Kafka scaling best practices: An in-depth primer

Factor House — Thu, 28 May 2026 00:00:00 GMT

Kafka is designed to scale, but it does not scale automatically. The decisions you make at the beginning, around partition counts, replication, consumer group design, and broker sizing, determine whether your cluster grows cleanly or accumulates operational debt that becomes expensive to unwind later.

This article pulls from published engineering work at LinkedIn, Uber, Pinterest, Cloudflare, Shopify, Wix, and Robinhood, alongside Apache KIPs and Confluent’s production documentation, to give you a grounded view of what scaling Kafka actually looks like in practice.

How to scale a Kafka cluster

Scaling a Kafka cluster means increasing its capacity across four separate dimensions: brokers, partitions, consumers, and storage. Each one is independent, and each has its own ceiling and operational cost.

Adding brokers increases raw throughput capacity, but by itself does nothing for existing topics. Kafka does not automatically rebalance partitions to new brokers. Only newly created topics land on a new broker; for everything already running, you need to trigger a reassignment explicitly, either via a manual kafka-reassign-partitions run or through a tool like Cruise Control. LinkedIn built Cruise Control specifically because manually managing partition distribution across a cluster of 4,000 brokers is not practical. Cruise Control treats broker load, disk utilisation, rack distribution, and leader balance as optimisation goals rather than manual inputs.

Increasing partition counts is the primary lever for producer and consumer throughput on a given topic, because partitions are the unit of parallelism. The standard formula is max(t/p, t/c), where t is target throughput, p is per-partition produce throughput, and c is per-partition consume throughput, measured on your hardware. The caveat is that for keyed topics you cannot increase partition count on a live topic without breaking key-to-partition routing, so the decision needs to be made before traffic arrives.

Scaling consumers is bounded by partition count within a consumer group. You can add consumer instances freely, but any instance in excess of the partition count sits idle. For slow-consumer workloads or when partition count cannot increase, the options are Confluent’s Parallel Consumer for in-process parallelism, or a consumer proxy architecture. Uber’s uForwarder, Wix’s gRPC fan-out proxy, and Robinhood’s Kafkaproxy sidecar each converge on the same pattern: consume once per topic, fan out to downstream consumers, and decouple throughput scaling from partition counts.

Scaling storage has historically meant scaling brokers proportionally. Tiered storage, now GA in Kafka 3.9 via KIP-405, changes this by offloading cold log segments to object storage. Uber has been running tiered storage in production for several years and describes faster broker recovery as one of the main operational wins: a new or replaced broker no longer needs to re-replicate terabytes of cold data to become current.

Each of these levers interacts with the others. What follows covers the specific configuration decisions, documented trade-offs, and patterns from production deployments that determine whether scaling each dimension is clean or operationally expensive.

Understanding the scale ceiling first

Before making configuration decisions, it helps to calibrate against what production Kafka clusters actually handle. LinkedIn operates more than 100 clusters with 4,000 brokers and roughly 7 million partitions, processing over 7 trillion messages per day. Pinterest pushes more than 800 billion messages per day at 15 million messages per second peak, across more than 2,000 brokers in AWS. Shopify peaked at 66 million messages per second during Black Friday and Cyber Monday.

These numbers are useful not because your cluster needs to match them, but because the teams that built them have documented what broke along the way.

Partitioning strategy

How many partitions to create

The canonical sizing formula, from Jun Rao’s Confluent post, remains the standard starting point: measure single-partition producer throughput (p) and single-partition consumer throughput (c) on your hardware, then calculate the required partition count as max(t/p, t/c) for your target throughput t. Add headroom beyond that, because consumer parallelism is bounded by partition count.

For ZooKeeper-backed clusters, Confluent’s documentation recommends staying below 4,000 partitions per broker and 200,000 per cluster. Beyond these thresholds, controller failover slows noticeably because the controller propagates LeaderAndIsr information serially. KRaft removes this bottleneck. Confluent’s lab benchmarks on a 2-million-partition cluster show drastically faster controlled-shutdown and uncontrolled-failure recovery compared to the ZooKeeper equivalent.

Even on KRaft, partition count is still bounded by per-broker file-descriptor limits, replica-fetcher threads, and metadata RAM. These are practical ceilings, not architectural ones.

Keyed topics and the repartitioning problem

For keyed topics, over-provision at creation time. Once a topic is in production, you cannot increase partition count without breaking the hash(key) % num_partitions mapping that routes messages to partitions. Confluent’s documentation is explicit about this. If you anticipate growth, it is considerably easier to start with more partitions than you need today than to deal with the downstream consequences of repartitioning a live topic.

There are also cases where the standard rule breaks down in the other direction. Netflix’s Real-Time Distributed Graph uses a separate Kafka topic per node and edge type so each can be tuned and scaled independently. The New York Times stores every article since 1851 in a single infinite-retention partition, accepting the per-topic throughput ceiling because publication rate is low. These are deliberate trade-offs, not mistakes.

Avoiding partition skew

Key distribution is the most common cause of hot partitions. LinkedIn’s answer to this was Cruise Control, a goal-based rebalancer that considers rack awareness, capacity, leader balance, and disk utilisation in its optimisation proposals. It is now the de facto partition rebalancer in the open ecosystem and ships with AWS MSK, Strimzi, and Canonical’s Charmed Kafka.

When adding brokers, be aware that Kafka does not automatically move existing partitions to new brokers. Only newly created partitions land on a new broker. You need to run Cruise Control or trigger a reassignment explicitly. When doing so, throttle the reassignment using --throttle <bytes/sec> to avoid saturating broker network and causing ISR shrinkage.

Producer configuration

Compression

Compression is one of the higher-leverage tuning options available on the producer side. The available codecs are none, gzip, snappy, lz4, and zstd. Confluent’s general recommendation is lz4 for performance and zstd when you want better compression ratios without the CPU cost of gzip.

Cloudflare published concrete benchmarks on their real rrdns.recordbatch workload: Snappy achieved around 43% of original size at 446 MB/s compression throughput, while LZ4 reached 40% of original size at 594 MB/s compression and 2,428 MB/s decompression. Zstandard at level 1 achieved 24% of original size at 409 MB/s compression. Cloudflare ultimately standardised on Snappy due to Go client compatibility issues with LZ4 at the time. Their post also found that combining Snappy with 1-second batching produced an 8x size reduction on a low-volume topic.

One point worth noting: if your producer and topic compression codecs do not match, the broker is forced to decompress and recompress each batch, which burns CPU. Keep them aligned.

Batching and `linger.ms`

In Apache Kafka 4.0, the default linger.ms was changed from 0 to 5 milliseconds. The upstream rationale is that in production the throughput and compression gains nearly always outweigh the additional latency. This is now the default rather than a tuning trick.

Robinhood’s migration to WarpStream for logging workloads illustrates the cost trade-off at the extreme end. By pushing batch sizes up significantly and accepting an increase in average produce latency from 0.2 seconds to 0.45 seconds, they achieved a 45% net cost reduction. That trade-off may not make sense for your workloads, but it does demonstrate how sensitive cost can be to batch sizing when storage is S3.

The counter-example is Shopify’s BFCM 2025 scale testing, which found that partition increases, not larger batches, were the lever needed to maintain data freshness during traffic spikes. Producer batching and partition count address different constraints.

Consumer group scaling

The partition-to-consumer ceiling

Within a consumer group, at most one consumer instance can be assigned to a partition. Excess consumers sit idle. This creates a hard ceiling: maximum parallelism equals the topic’s partition count. If you cannot or do not want to add partitions to a live keyed topic, you have a few options.

Confluent’s Parallel Consumer (open-source, Apache 2.0) lets a single consumer thread pool process messages from one partition in parallel, with three ordering modes: KEY, UNORDERED, and PARTITION. It is specifically aimed at slow-consumer workloads involving database calls or HTTP requests. The trade-off is that exactly-once semantics are more complex to manage.

At larger scale, a consumer proxy becomes the convergent answer. Uber’s uForwarder, Wix’s push-based gRPC fan-out proxy, and Robinhood’s Kafkaproxy sidecar all address the same fundamental problem: the partition count ceiling, head-of-line blocking, and fragmented multi-language client behaviour. Wix found they were consuming each produced byte roughly four times across different consumer groups and built a proxy that consumes each topic once and fans out via gRPC, reducing their Kafka bill by 30%.

Rebalancing

The legacy eager rebalancing protocol revokes all partitions from all consumers on every rebalance event. For large consumer groups, this can pause consumption for seconds to minutes.

KIP-429, introduced in Kafka 2.4 and the default since Kafka 3.0, provides incremental cooperative rebalancing via CooperativeStickyAssignor. Only the partitions that need to move are revoked during a rebalance; all others continue processing. In environments running Kafka 3.0 or later, there is little reason to stay on the eager assignor.

For Kubernetes deployments specifically, static membership (group.instance.id, KIP-345) is the other essential lever. A consumer that restarts with the same instance ID does not trigger a rebalance, provided it returns within session.timeout.ms. This matters in practice because autoscaling events that add or remove consumers can otherwise cause repeated rebalances that destabilise consumption, which is exactly the problem Robinhood ran into before building their proxy.

Monitoring consumer lag

LinkedIn’s Burrow remains one of the better approaches for lag monitoring. Rather than relying on static thresholds, it consumes the __consumer_offsets topic, tracks a sliding window per partition, and issues status judgements based on whether consumer offsets are advancing relative to broker offsets. This makes it far more reliable for wildcard consumers and MirrorMaker than threshold-based alerting.

Cloudflare’s pattern is also worth noting: they auto-create a high-lag alert for every topic at topic-creation time, using time-based lag rather than offset-based lag. Time-based lag measures the difference between when the last committed offset was produced and when it was consumed, which is more meaningful for SLOs than raw offset numbers.

Broker infrastructure

Sizing

Kafka brokers are not typically CPU-bound unless TLS, compression, or a high partition count is involved. Pinterest found that enabling TLS at scale changed the per-connection cost significantly enough to require raising their broker heap from 4 GB to 8 GB.

Memory allocation should prioritise the OS page cache over the JVM heap. Kafka relies heavily on the page cache for storing and serving messages. The standard recommendation is 4 to 8 GB of heap, with the rest of available RAM left for the page cache. Never co-locate Kafka with other memory-hungry processes.

For GC configuration, the LinkedIn and Confluent baseline uses G1GC with Xms and Xmx set to the same value to avoid heap-resize jitter, a MaxGCPauseMillis of 20, and InitiatingHeapOccupancyPercent of 35. Netflix switched to Generational ZGC on Java 21 for sub-millisecond pauses on large-heap brokers. That is worth considering if you have a clear reason to run heaps larger than 16 GB, but G1GC remains the sensible baseline.

At the OS level: set nofile to 100,000 or more, configure vm.swappiness=1 to prevent the kernel from swapping out page-cached log data, raise network buffers, and mount storage with noatime,nodiratime on XFS or ext4.

Rack-aware replication

Configure broker.rack to your AZ identifier so that replicas are distributed across zones with RF=3. Then enable follower fetching via replica.selector.class=org.apache.kafka.common.replica.RackAwareReplicaSelector and set client.rack on consumers. This allows consumers to prefer same-AZ replicas and can significantly reduce cross-AZ network costs, which are frequently the dominant cost line in cloud Kafka deployments.

KIP-881, available since Kafka 3.4, extends this further by allowing RangeAssignor and CooperativeStickyAssignor to use rack data when assigning partitions to consumers.

Tiered storage

Tiered storage separates compute and storage by offloading older log segments to object stores such as S3, GCS, or Azure Blob. KIP-405 reached GA in Kafka 3.9 (November 2024). Uber has been running tiered storage in production for several years and reports that the key wins are independent compute/storage scaling and faster broker recovery, since a new broker no longer needs to re-replicate terabytes of cold data on joining the cluster.

If you are evaluating greenfield analytics-style Kafka workloads at scale, tiered storage is worth considering up front rather than retrofitting later. The cost gravity in the ecosystem is moving toward S3-backed storage, and the options for doing so are broader than they were two years ago.

ZooKeeper to KRaft migration

This is not a future consideration. ZooKeeper mode was deprecated in Kafka 3.5 and removed in Kafka 4.0. Support for ZooKeeper-based clusters ends roughly November 2025. Confluent Platform 8.0 removed ZooKeeper. If you are running any ZooKeeper-backed Kafka cluster, migration planning should already be underway.

KRaft became production-ready in Apache Kafka 3.3 and reached full feature parity with ZooKeeper mode in Kafka 3.9. The migration path involves provisioning a dedicated KRaft controller quorum, running through a dual-write phase where metadata is written to both ZooKeeper and KRaft, then rolling brokers to KRaft mode one at a time before decommissioning the ZooKeeper ensemble.

A few things to be aware of before starting: there is no downgrade path once migration is finalised. SASL/SCRAM for controllers is not supported during migration. Brokers in dual-write mode use additional CPU and memory. And combined controller/broker mode is not supported for production migration targets.

Confluent Cloud completed the largest known migration, moving thousands of clusters to KRaft without breaching SLAs. One widely cited outcome from a 50-node financial services cluster: controller failover time dropped from 5 to 7 seconds to under 1 second.

How Kpow can help

Scaling a Kafka cluster involves a lot of moving parts, and visibility is frequently the gap. When you are troubleshooting a partition skew problem, investigating consumer lag across dozens of groups, or working through the state of a reassignment, having clear access to your cluster’s internal state saves significant time.

Kpow is a Kafka management and observability tool built by Factor House. It gives you a live view of consumer group lag, partition assignment, topic configuration, and broker health in a single interface. If you are running multiple clusters, it supports those as a unified view as well. For teams working through the scaling changes described in this article, having that visibility during and after configuration changes reduces the feedback loop considerably.

Viewing Consumer Groups and their metrics in Kpow

You can connect Kpow to any Kafka cluster in minutes and try it free for 30 days, whether you are running on bare metal, Kubernetes, or a managed service.

Summary

The decisions that constrain Kafka at scale tend to compound. Partition counts that seemed reasonable at launch become structural limits. Consumer groups on legacy rebalancing protocols create operational instability under autoscaling. ZooKeeper clusters that were not migrated before support ended become a liability.

Most of the patterns that work at scale, cooperative rebalancing, rack-aware follower fetching, KRaft, Cruise Control for partition balance, and tiered storage for cost, are well-documented and available without commercial dependencies. The engineering blogs from LinkedIn, Uber, Cloudflare, and others are primary sources worth reading in full if you are operating at significant scale or heading toward it.

Triage, repair, and replay: integrated Kafka remediation workflows

Jaehyeon Kim — Thu, 28 May 2026 00:00:00 GMT

In the second part of this series, we demonstrated how combining advanced filtering, artificial intelligence, and streaming search accelerates investigations to help teams respond to incidents quickly. However, locating bad data is only half the battle. Once an issue is identified, engineering teams must take action to fix the pipeline.

Resolving a stalled consumer, recovering from a downstream database crash, or fixing a dead-letter queue (DLQ) message requires managing both the consumer state and the data payload itself. This article explores the friction inherent in generic recovery workflows and demonstrates how Kpow unifies data extraction, state management, and payload correction into a single integrated process.

This is Part 3 of the Kafka Data Management with Kpow: Unlocking Engineering Productivity series. You can read the full strategy in the main series article and access the associated posts as they become available:

Part 1: Foundational Kafka Data Inspection: Shaping Payloads and Optimizing Visibility
Part 2: Accelerating Incident Response: Advanced Filters, Streaming Search, and AI-Powered Queries
Part 3: Triage, Repair, and Replay: Integrated Kafka Remediation Workflows (This article)
Part 4: Defense in Depth: Unifying RBAC and Data Policies for Transparent Governance

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Start your free 30-day trial or explore our live multi-cluster demo environment to see Kpow in action.

Problem: Fragmented Pipeline Repair

Fixing data pipelines extends beyond simply identifying a schema mismatch. A single malformed record (a poison pill) can crash a consumer thread. This causes the thread to catch the error, seek back to the problematic offset, and retry in an infinite loop. Because other partitions process normally, aggregated metrics often mask the failure, allowing lag to spike silently on a single partition.

Alternatively, a downstream sink failure might require a complete historical data replay. Resolving these diverse issues requires identifying the exact malformed payload, modifying consumer group offsets, and fixing the underlying data. Currently, these tasks exist in completely disconnected operational silos.

Limitations of Siloed Recovery

Generic workflows introduce severe friction when pipelines break. Developers are forced to context-switch, often writing one-off scripts to hunt down serialization errors or extract records for analysis.

Meanwhile, repairing the pipeline state requires administrative access. Developers must file tickets and wait for platform engineers to run the Kafka Consumer Group CLI commands manually to reset or skip offsets. This disconnected ticketing process creates dangerous race conditions. If the consumer application restarts before the manual CLI command executes, the command will fail or conflict with the active consumer state. Furthermore, this process only addresses the consumer offset, leaving the actual malformed data uncorrected and missing from the downstream system.

Integrated Remediation with Kpow

Kpow eliminates the Remediation Gap by consolidating data inspection, consumer state management, and payload correction into a single interface. This allows operators to seamlessly transition from finding an issue to completely resolving it.

Isolating Malformed Data

When a consumer stalls, operators can move directly from the stalled partition to the underlying data. By opening the Consumer Details tab, engineers can spot rapidly accumulating lag on a specific partition. Clicking the Inspect data action instantly opens the Data Inspect UI, pre-filtered to the affected partition, where operators can quickly apply a kJQ filter to pinpoint the exact offset causing the blockage.

Alternatively, if the problematic offset is unknown, Kpow provides powerful Deserialization Options to cut through the noise of healthy traffic. By setting the strategy to Poison only, operators can scan the topic to instantly filter out all successfully parsed records, isolating only the malformed data (such as an unexpected string instead of a required integer) that crashed the consumer.

Unblocking Consumers with Group Actions

Once the problematic offset is identified, operators must unblock the pipeline safely. Kpow’s Consumers UI provides robust state management capabilities. From the action menu, operators can execute a Skip offset at the partition level to immediately bypass the single poison pill. Kpow also supports Clear offset and Reset offset actions, which allow engineers to rewind a group to a specific offset, timestamp, or precise datetime to replay data after a sink failure.

Crucially, Kpow protects the integrity of the cluster through Scheduled Mutations. Rather than forcing manual CLI commands that clash with active consumers, Kpow safely schedules the offset change. The platform waits for the consumer group to scale down to an EMPTY state before automatically executing the mutation, completely preventing state conflicts and race conditions.

Repairing and Re-injecting Payloads

Skipping a poison pill unblocks the consumer, but the data is still lost to the downstream system. To achieve complete remediation, teams must correct the payload and re-inject it into the pipeline.

Kpow enables this through native UI routing. By highlighting the isolated bad record in the Data Inspect view and selecting Produce from the drop-down menu, operators route the message directly into the Data Produce form. The form opens pre-populated with the record’s details.

Developers can execute a quick manual edit to the JSON payload (for example, fixing an invalid “amount”: “ONE THOUSAND DOLLARS” field by replacing it with numeric 100) and produce the corrected message. This visually proves that the data was not just skipped, but successfully repaired and re-injected. For offline auditing, Kpow also offers robust downloading options to extract the problematic records in CSV, JSON, or EDN formats.

Conclusion

By unifying data extraction, consumer offset management, and payload correction, Kpow eliminates the need for manual scripts and cross-team ticketing during a production outage. Operators can seamlessly isolate a poison pill, safely skip the consumer offset using scheduled mutations, and repair the data payload in a single, cohesive workflow. This comprehensive approach restores data pipelines rapidly and guarantees that no critical messages are lost.

While these remediation capabilities are incredibly powerful, they must be governed securely. In the final part of this series, Defense in Depth: Unifying RBAC and Data Policies for Transparent Governance, we will explore how organizations can deploy these tools safely in production without exposing sensitive information or violating compliance constraints.

Next steps

Explore Kpow in your own environment with a free 30-day trial.

If you need assistance managing your Kafka environment, reach out to our engineering support team at support@factorhouse.io.

AKHQ: Review, pricing, and best alternatives in 2026

Factor House — Tue, 26 May 2026 00:00:00 GMT

Key takeaways

AKHQ is the most complete free Kafka UI available, covering multi-cluster management, Schema Registry, Kafka Connect, consumer group monitoring, ACL management, and LDAP/OIDC authentication in a single self-hosted tool.
Memory behaviour under high-throughput message tailing or large consumer group counts is a documented operational risk, with multiple out-of-memory reports at heap sizes up to 14 GB and no published fix.
The authentication layer is more mature than most free alternatives, but reaching fine-grained multi-cluster RBAC required Michelin to contribute code directly to the project; audit logging is absent throughout.
Reddit practitioners consistently frame AKHQ as a starting point for small teams, with a recurring pattern of switching to commercial tooling once teams feel pain around auditability, self-service workflows, or scale.
AKHQ has no data masking, no JMX metrics visualisation, no alerting, and no partition or replica management from the UI, making it unsuitable for regulated environments or operators who need those capabilities.
For teams that need enterprise RBAC, data masking, audit logging, or commercially backed support, Kpow is worth evaluating as an alternative.

What is AKHQ?

AKHQ (formerly KafkaHQ) is an open-source Kafka management UI maintained by Ludovic Dehon (tchiotludo) under the Apache 2.0 licence. The project is hosted at https://github.com/tchiotludo/akhq. It is self-hosted, JVM-based, and built on Micronaut.

A single AKHQ deployment connects to one or more Kafka clusters and exposes topic browsing, live message tailing, message production, consumer group monitoring, Schema Registry management, Kafka Connect management, ACL management, and role-based access control with LDAP and OIDC integration. AWS MSK IAM authentication is also supported.

AKHQ has no commercial edition, no hosted SaaS version, and no paid support tier. All functionality is available in the open-source release. Enterprise users including Michelin and La Redoute have contributed features directly to the project.

AKHQ review

Functionalities

AKHQ covers a broad surface area for a free tool. Named reviewers credit it with multi-cluster management, message browsing, live tailing, Schema Registry, Kafka Connect, ACL management, and RBAC in a single deployment. [German Osin, Medium / Towards Data Science, 2 Sep 2021]

The functional ceiling appears in a consistent set of areas. AKHQ does not support dynamic topic configuration, partition increase, replica change, Kafka Streams topology visualisation, or JMX metrics visualisation. [German Osin, same source; Zeenia Gupta, Platformatory blog, 5 Sep 2024] Protobuf support is partial: the schema registry integration is limited to the deserialiser side, and at least one user reports complete deserialization failure when using Protobuf with Apicurio Registry even when the descriptor-file flow works. [Zeenia Gupta, Platformatory blog, 5 Sep 2024; GitHub issue #1185] A related Avro bug causes AKHQ to fall back to raw message output when it cannot match a schema ID, leaving messages unparsed in the UI. [GitHub issue #1918, 2024]

ksqlDB integration is listed in AKHQ’s configuration options but is effectively non-functional against Confluent Cloud endpoints. Configuring a Confluent ksqlDB endpoint produces a null pointer exception on the HTTP client (Cannot invoke io.vertx.core.http.HttpClientRequest.response(...) because request is null). [sbourell, GitHub issue #1954, 24 Sep 2024] A separate discussion thread reports that ksqlDB does not appear in the menus at all despite correct configuration. [GitHub Discussion #1486]

There is also a production workflow friction point: the message production dialog closes after each message, requiring the operator to reopen it and refill all fields when producing multiple messages in sequence. [xarx00, GitHub Discussion #805, Sep 2021]

Data masking is absent. For any team handling PII or operating under HIPAA, PCI-DSS, or GDPR requirements, this is a hard compliance gap.

AKHQ also has no built-in support for dead-letter queue (DLT) retry patterns or inline message payload correction. These capabilities require separate tooling, and the gap has driven some teams toward purpose-built event-streaming platforms that handle error recovery alongside broker management. [r/apachekafka, GitHub: i_built_a_spring_boot_starter_for_kafka_message, 2025]

Deployment and operations

Docker Compose and Helm installation are two of the most consistently praised aspects of AKHQ. Getting a full local stack including Kafka, Schema Registry, Kafka Connect, and AKHQ running requires a single docker compose pull followed by docker compose up, giving access to everything at localhost:8080. [AKHQ docs, akhq.io/docs/] Connections, users, groups, and schema registry links are defined in YAML and deployed via Helm, making the configuration GitOps-friendly. [Factor House, same URL above, 2026] Community feedback echoes this: one practitioner on r/apachekafka described it as “open source, can be installed on prem using their docker image, and it’s very lightweight.” [Drazul_, r/apachekafka, 2025]

AKHQ 0.24.0 has been successfully deployed on Red Hat OpenShift 4 using Helm with Kafka 3.3.1; Rogerio Santos at Red Hat describes it as “an excellent complement to AMQ streams.” [Rogerio Santos, Red Hat Developer, 26 Jul 2023]

The operational risk is memory. A documented memory pattern on AWS ECS at 4 GB RAM shows “an ascending stair shape” with little or no deallocation between scans. [arnaud-ly, GitHub issue #1141, 5 Jul 2022] High-throughput topic tailing triggers a Direct buffer memory out-of-memory error even at 14 GB heap, which points to a missing backpressure mechanism rather than a simple heap sizing problem. [MichalPopielski, GitHub issue #1206, 13 Sep 2022] Neither issue has a published resolution. For clusters with many consumer groups, the AKHQ documentation itself recommends enabling HIDE_EMPTY and setting skip-consumer-groups: true as a workaround to keep startup responsive. [AKHQ docs, referenced in GitHub issue #271]

Security scan failures have also been reported against the official Docker image at version 0.24.0, related to vulnerabilities in bundled Java libraries. [GitHub issue #1771]

Access control and security

AKHQ supports LDAP, OIDC, HTTP basic authentication, and external role and attribute claim mapping. MSK IAM support was contributed and merged. [Zeenia Gupta, Platformatory blog, 5 Sep 2024; GitHub issue #810] This is a broader authentication story than most free Kafka UIs offer, and it is one of the primary reasons practitioners choose AKHQ over Kafdrop.

The limits become apparent under enterprise conditions. Michelin built their own resource-level RBAC layer on top of AKHQ because the default authorisation mechanisms were not granular enough for multi-cluster, multi-team use. Their contribution modified “all AKHQ features and authorization mechanisms (LDAP, OIDC, external claims)” and introduced an AKHQSecured annotation for fine-grained control. [Alexis Souquiere, Michelin engineering blog, 5 Mar 2024] An internal survey at Michelin in 2023 “revealed widespread dissatisfaction” with small issues that accumulated over time, and the team considered building an alternative before choosing to contribute instead. [Alexis Souquiere, same source]

Active auth bugs affect current releases. An Okta OIDC redirect loop sends authenticated users back to /ui/login when they click a topic linked to a restricted consumer group, affecting AKHQ 0.25.1. [ndemyanchuk-booking, GitHub issue #2131, 25 Mar 2025] A separate regression sends users to the login page despite a 200 OK on the initial login request. [GitHub issue #2055] LDAPS configuration also carries documented friction. [GitHub issues #887, #1041]

Audit logging is absent. Teams evaluating AKHQ alongside commercial alternatives will find that “AKHQ and Kafbat have basic RBAC but neither provides self-service workflows, team ownership models, audit logging, or the enterprise API that multi-team organisations need.” [Factor House, same URL above, 2026] Without audit logs, there is no record of who reset a consumer offset, modified a broker configuration, or produced a message - a meaningful constraint for any team operating under change-management or compliance requirements.

User interface

The UI is functional and covers its stated scope, but it is consistently described as less polished and less responsive than Conduktor or modern Kafbat. The structural complaints are recurring: search filters reset when navigating between views, and content reloads from the server rather than from a cached client-side state, which creates a sluggish experience when switching between topics or schemas. [xarx00, GitHub Discussion #805, Sep 2021] The maintainer acknowledged performance and navigation debt in the same thread, noting that schema registry caching would be a prerequisite for any direct topic-to-schema navigation improvement.

German Osin’s characterisation from 2021 remains consistent with more recent accounts: AKHQ “is not the most convenient” and users “will definitely need to allocate some time” to learn it. [German Osin, Medium / Towards Data Science, 2 Sep 2021] Comparison reviewers describe Kafbat’s UI as “more modern” than AKHQ’s. [Factor House, 2026] The Reddit community surfaces the same friction directly: one engineer summarised it as “you get what you pay for… since you pay nothing, you get a clunky UI/UX. It does its job, but leaves more to be desired (especially after you use Conduktor).” [roastedsun, r/apachekafka, 2025] A platform architect put it in practical terms: “if it’s a startup and you just want visibility and basic management I’d try Kafka UI or AKHQ first. If you’re feeling pain around ‘who changed what’ and ‘how do we let devs self-serve without blowing stuff up’ then look harder at Conduktor or Lenses.” [TellersTech, r/apachekafka, 2025]

On the positive side, a practitioner walkthrough from March 2026 frames AKHQ as “a control tower for the entire Kafka ecosystem,” removing the need to SSH into servers and run kafka shell scripts. [Naveen Mittal, Medium, 7 Mar 2026] Thoughtworks placed AKHQ in “Trial” status on their Technology Radar in 2022, citing topic browsing, Avro and Protobuf deserialization, and consumer group visibility as useful capabilities for teams working to understand data flows. [Thoughtworks Technology Radar, 29 Mar 2022] Day-to-day debugging tasks - message scanning, data discovery, and consumer group inspection - receive the most consistent praise from practitioners, and some larger engineering organisations run AKHQ alongside a commercial tool as a lightweight secondary viewer. [arcanumoid, r/apachekafka, 2025; r/apachekafka, urn8f3 thread, 2022]

Ecosystem

Schema Registry and Kafka Connect are well-supported and among the most cited reasons to choose AKHQ over lighter tools like Kafdrop. MSK IAM was added through a community contribution. Multi-cluster support is documented in production: Michelin uses AKHQ across on-premise factory clusters and cloud deployments to supervise topics, consumer groups, and connectors. [Alexis Souquiere, Michelin engineering blog, 5 Mar 2024] Rogerio Santos at Red Hat singles out multi-cluster support as a key advantage: “One of the features of AKHQ is its support for configuring multiple clusters, making it a convenient central GUI for managing multiple Kafka clusters.” [Red Hat Developer, 26 Jul 2023]

The ksqlDB gap is the notable exception. The integration exists in configuration and was listed as a 0.24.0 feature contributed by 30+ contributors, but it does not function reliably against Confluent Cloud endpoints, and at least two community threads report it being absent from the UI despite correct configuration. [sbourell, GitHub issue #1954, 2024; GitHub Discussion #1486; akhq.io LinkedIn post on the 0.24.0 release]

Customer support

AKHQ has no commercial support tier, no SLA, and no enterprise offering. The project is functionally a single-maintainer effort led by Ludovic Dehon, with major feature contributions from a small set of enterprise users. La Redoute’s distributed OSPO experience building an open-source Kafka product is documented; Michelin is a recognised core contributor. [La Redoute engineering blog, summarised via search result; Alexis Souquiere, Michelin engineering blog, 5 Mar 2024]

Open issues can wait months for a response. The memory leak reported in issue #1141 dates to July 2022 with no published fix or maintainer comment. The Confluent ksqlDB issue #1954 was labelled “wait for reply” with no visible maintainer response in the thread. Documentation covers the happy path well but is thin on production tuning guidance and auth edge cases, with HTTPS Schema Registry configuration appearing as a recurring community discussion topic. [GitHub Discussion #575]

Best for

AKHQ suits individual engineers and small platform teams that need real authentication (LDAP/OIDC), Schema Registry and Connect management, MSK IAM support, and multi-cluster visibility in one self-hosted tool at zero licensing cost. It is a credible production choice for OpenShift shops running AMQ Streams, and for teams who want GitOps-friendly YAML configuration and a mature Helm chart. Engineers replacing ad hoc CLI workflows - SSH sessions and kafka shell scripts - will find AKHQ a meaningful step forward. Some larger organisations also run it as a lightweight secondary viewer alongside a commercial platform, using it for quick message browsing while relying on the commercial tool for access control and governance.

It stops being a sufficient fit when teams grow beyond roughly five engineers sharing a cluster, when compliance requirements mandate data masking or audit logging, when operators need JMX metrics or alerting built into the UI, or when the cluster runs Confluent ksqlDB. Teams handling PII or operating in regulated industries should treat the absence of data masking as a hard constraint rather than a roadmap item. The operational overhead of maintaining a web-based server has also pushed some engineers toward desktop-native Kafka clients that require no Docker daemon to run; tools like KafkIO and Swifka have emerged to serve that preference. [r/apachekafka, rkquik thread, 2025; r/apachekafka, rash5y thread, 2025]

AKHQ pricing

AKHQ is free and open-source under the Apache 2.0 licence. There is no paid edition, no hosted SaaS option, and no commercial support offering.

Pricing tiers

Tier	Cost	Notes
Open source	Free	Full feature set; self-hosted only

Free trial

No trial period applies. The full project is available at https://github.com/tchiotludo/akhq with no feature restrictions or registration requirements.

AKHQ competitors and alternatives

AKHQ competes with a range of self-hosted and commercial Kafka management tools, from lightweight topic browsers like Kafdrop to full governance platforms like Kpow. The right fit depends on how much operational maturity, compliance capability, and support you need beyond what the open-source tier provides.

Tool	Best for	Type	Key functionalities	Deployment and ops	Access control	User interface	Pricing
AKHQ	Small teams needing LDAP/OIDC, Schema Registry, Connect, and MSK IAM at zero cost	OSS (Apache 2.0)	Topics, messages, Schema Registry, Connect, consumer groups, ACLs, multi-cluster, MSK IAM	Docker Compose, Helm, GitOps YAML; JVM memory issues at scale	LDAP, OIDC, basic, external claims; no audit logging	Functional; dated; filter state resets on navigation	Free
Kafbat	Teams wanting an actively developed OSS alternative with a more modern UI	OSS (Apache 2.0)	Similar to AKHQ; actively maintained AKHQ fork	Docker, Helm	LDAP, OIDC	More modern than AKHQ	Free
Kafdrop	Lightweight topic browsing with minimal setup	OSS (Apache 2.0)	Topic browse, message view, consumer groups	Docker; minimal dependencies	None	Simple, read-focused	Free
Conduktor	Multi-team enterprises needing data masking, field-level encryption, and audit logs	Commercial	Topics, Schema Registry, Connect, data masking, audit logs, ksqlDB, self-service workflows	Docker or Kubernetes; requires PostgreSQL; SaaS available	SSO, RBAC, audit logging; SAML on Enterprise tier	Polished, React-based	Community tier free; Team from ~$1,200/seat/year
Kpow (Factor House)	Teams needing enterprise RBAC, WCAG-compliant UI, and commercially backed support without per-seat pricing	Commercial	Topics, messages, Schema Registry, Connect, RBAC, MSK/GCP/Azure support	Stateless; Docker or Kubernetes; straightforward deployment	Advanced RBAC; SSO	WCAG-compliant; high performance at scale	Per-cluster pricing
Lenses	Teams needing SQL-based data exploration and a developer experience layer across Kafka	Commercial	SQL over topics, data observability, developer portal	Self-hosted or SaaS	RBAC, SSO	Developer-focused; SQL interface prominent	Contact for pricing
Redpanda Console	Redpanda-native teams or those wanting a lightweight browser for any Kafka-compatible cluster	OSS core / commercial	Topic browsing, consumer groups, Schema Registry; more limited Connect management	Docker, Kubernetes	Basic; enterprise auth on paid tiers	Clean, modern	OSS core free; enterprise pricing on request

For a broader comparison across all major Kafka UI tools, see the Factor House Kafka UI tools comparison.

Frequently asked questions about AKHQ

How much does AKHQ cost, and is there a free tier?

AKHQ is fully free under the Apache 2.0 licence. There is no paid tier, no SaaS option, and no commercial support offering. All features are available in the open-source release with no restrictions or registration required.

When is AKHQ a better choice than the alternatives?

AKHQ is the strongest free option when you need LDAP/OIDC authentication, Schema Registry and Connect management, MSK IAM support, and multi-cluster visibility in one self-hosted tool. It has a more mature auth story than Kafdrop and broader ecosystem coverage than Redpanda Console’s open-source tier.

When are the alternatives a better choice than AKHQ?

Consider a commercial alternative when you need data masking, audit logging, JMX metrics, alerting, or working ksqlDB integration. AKHQ’s single-maintainer governance model and documented out-of-memory behaviour under high-throughput tailing also make alternatives worth evaluating for large-scale or regulated deployments.

Is AKHQ actively maintained?

Active, but with a single primary maintainer. Ludovic Dehon leads development, with contributions from enterprise users including Michelin and La Redoute. Some issues sit without a maintainer response for months. There is no commercial entity behind the project and no SLA.

CMAK: Review, pricing, and best alternatives in 2026

Factor House — Tue, 26 May 2026 00:00:00 GMT

Key takeaways

CMAK (Cluster Manager for Apache Kafka) is a free, Apache 2.0-licensed tool originally built by Yahoo that provides multi-cluster management, partition reassignment, preferred-replica election, and optional JMX polling from a browser-based interface.
The project is functionally stalled: the last stable release (3.0.0.6) was tagged on 29 April 2022, and multiple pull requests and feature requests have gone unanswered since.
CMAK has a hard ZooKeeper dependency with no KRaft support. Apache Kafka 4.0, released on 18 March 2025, operates entirely without ZooKeeper, making CMAK incompatible with all clusters running that version.
Security is limited to LDAP basic auth and coarse feature flags: there is no SAML/OIDC, no per-topic RBAC, and no audit log.
For teams running KRaft clusters, needing message browsing, or requiring enterprise-grade access control, alternatives such as AKHQ, Kafdrop, Redpanda Console, or Kpow are worth evaluating.

What is CMAK?

CMAK (Cluster Manager for Apache Kafka), originally named Kafka Manager, is an open-source Kafka administration tool built at Yahoo and released under the Apache 2.0 licence. It provides a browser-based interface for managing one or more Kafka clusters from a single view.

The tool is built on Scala and the Play framework, and it requires a direct connection to a ZooKeeper ensemble to function. Its core feature set reflects the operational tasks Yahoo built it for: registering and monitoring clusters, creating and modifying topics, managing partitions, triggering preferred-replica elections, and polling JMX metrics from brokers.

CMAK is hosted at github.com/yahoo/CMAK. No commercial distribution, hosted offering, or paid support tier exists.

CMAK review

Functionalities

CMAK’s feature set is narrowly focused on cluster operations. Its official README enumerates the following capabilities: KMClusterManagerFeature, KMTopicManagerFeature, KMPreferredReplicaElectionFeature, and KMReassignPartitionsFeature. In practice, this covers multi-cluster registration, topic creation and configuration, partition reassignment, preferred-replica election, and optional JMX metric polling at the broker and topic level.

Practitioner reviews describe these features as effective for their intended purpose. A 2022 Towards Data Science overview states: “For the most part, CMAK is primarily an Ops tool. It is also really good at partition reassignment.” The same review notes that “multi-cluster management, dynamic topic configuration, partition creation, and replica change will cover most of your tasks.” User webodmin, writing on GitHub Issue #776 in 2023, described CMAK as “very useful and convenient” — this from someone who had run it in production until a KRaft migration made it unusable.

What CMAK does not cover is equally significant. The tool has no message-browsing UI, no schema registry integration, no Kafka Connect management, and no ksqlDB or Flink support. These absences reflect a scope deliberately limited to administrative operations rather than data-plane visibility.

One limitation that affects day-to-day operations is CMAK’s use of an internal caching layer rather than live data from broker APIs. Changes made to the cluster are not reflected in the UI immediately, because the interface polls for metadata on a fixed interval rather than subscribing to live state. This causes delays between an administrative action and its confirmation on-screen, which can mislead operators into believing a partition reassignment or offset reset has not taken effect. Tchiotludo, the author of AKHQ, cited this behaviour explicitly when describing why he built an alternative: “Kafka Manager have some kind of cache and you don’t see your change on real-time. During develop, it was confusing to don’t see your modifications” (r/apachekafka, 2019).

A related visibility gap appears during broker migrations. An operations specialist on r/devops noted that CMAK would report hours of catch-up replication completing in seconds, with no way to verify accuracy without querying the ZooKeeper shell directly: “CMAK would indicate hours of catch-up replication (by replicas) being completed in seconds whose correctness I could never figure out without going to the ZK shell” (Gathering opinions on Kafka management tools, 2021). For teams doing broker replacements or rebalances in production, this makes CMAK unreliable as a source of truth during the migration window.

One area where CMAK retains some value is as a learning tool. Its visual taxonomy of brokers, partitions, and replicas maps closely to Kafka’s internal model, and some practitioners recommend it specifically for building familiarity with core concepts. A systems engineer on r/apachekafka wrote: “Spend some time to understand CMAK, it is really good, and the functions on the page can help you check the knowledge points or keywords you missed when reading the document” (Suggestions for learning Kafka, 2025). This applies to ZooKeeper-era clusters; the tool provides no equivalent value on KRaft deployments where it cannot connect at all.

The most consequential functional gap as of 2025-2026 is the absence of KRaft support. CMAK connects to Kafka through a ZooKeeper endpoint. GitHub Issue #776 has accumulated requests from named users since 2022, including akamensky, webodmin, ceeeekay, meethigher, and azhurbilo, all of whom confirm CMAK is unusable on ZK-less deployments. Maintainer patelh acknowledged the problem in 2022: “Yep, we’ll phase it out but we’ll need some place to store metadata. Either directly in Kafka or RDBMS. Still to be designed.” As of May 2026, no implementation has shipped.

Practitioners hitting this wall in production confirm that the failure is hard and immediate. User dekanov on r/apachekafka wrote after standing up a test cluster: “I have just started my first Kraft test cluster. First problem is that it is not compatible with my Kafka Manager, which I can not get rid of entirely” (Running Kraft in production, 2023). The broader community has largely absorbed this as settled. Haarolean, a contributor to the Kafbat UI project, stated plainly on r/dataengineering: “CMAK is inactive for a couple years already” (Kafbat UI for Apache Kafka v1.0 is out, 2024).

Apache Kafka 4.0, released 18 March 2025, is the first major release to operate entirely without Apache ZooKeeper. CMAK is incompatible with any cluster running that version or later.

Deployment and operations

CMAK is distributed as source code requiring an sbt/Scala build. Docker images exist but are community-maintained (sheepkiller, TrivadisPF, intropro, hjben) rather than published by Yahoo. Kubernetes deployment is handled by the third-party eshepelyuk/cmak-operator Helm chart, not the upstream project.

Build friction is a recurring complaint. GitHub Issue #927 (December 2023, filed by zamek42) reports a compatibility problem on OpenJDK 17: “No JVMCI compiler found, but I use openjdk Java 17.” Issue #922 (July 2023, filed by SimonMikolajek) flags build dependency problems on current sbt/Scala toolchains.

For larger clusters with JMX enabled, the README requires manual thread-pool tuning: cmak.broker-view-thread-pool-size=<3 * number_of_brokers> and a corresponding update-interval formula. There is no documented autoscaling path.

Operational reliability has also been questioned. A widely-cited issue (#550, originally filed in 2018 by mimani) describes long-running instances hanging after 20-30 days with RejectedExecutionException errors caused by thread-pool exhaustion. This pattern has not been addressed in any subsequent release.

The last binary release is 3.0.0.6, tagged 29 April 2022 by contributor mcjyang. No stable release has followed in the three years since.

Access control and security

CMAK supports LDAP basic authentication and a coarse feature-flag model. The feature flags can be enabled or disabled globally but provide no per-user, per-cluster, or per-topic granularity.

The README warns explicitly: “Warning, you need to have SSL configured with CMAK (pka Kafka Manager) to ensure your credentials aren’t passed unencrypted… Note: LDAP is unencrypted and insecure.” SSL must be configured manually; the default setup transmits credentials in plaintext.

There is no SAML or OIDC integration. GitHub Issue #933 (September 2024, naganaidu-rezi) asks: “how to configure custom SAML for Kafka Manager?” — a direct signal that SSO is wanted and absent. In practice, teams that require SSO and MFA have had to place CMAK behind a reverse proxy wired to an identity provider. A DevOps engineer on r/devops described their setup: “We use it with OKTA to enable SSO and MFA for Spark UI, Kafka Manager, Kibana and Kubernetes dashboard” (How do you handle apps that do not have built-in SSO support?, 2021). This approach works but adds operational overhead, requires a correctly configured proxy, and still does not provide per-topic or per-cluster access controls — it secures the front door without addressing what happens inside. Issue #926 (October 2023, xiaobao623) reports that enabling ZooKeeper ACLs breaks CMAK’s connection entirely. Issue #932 (August 2024, qianghong000) covers TLS-to-ZK configuration complexity. There is no audit log.

For teams with compliance requirements, the access-control model is not enterprise-grade. It was designed for small, internally-trusted environments.

User interface

Practitioners describe the CMAK UI as functional and direct for operators already familiar with Kafka’s internal model. The 2022 Towards Data Science overview calls it “a good and fairly straightforward UI” that handles Ops tasks without unnecessary friction for experienced users.

The consistent criticism is that the UI is dated by comparison to modern single-page-application tools. Redpanda’s 2024 Kafka tools guide (vendor-authored, included here only for the descriptive claim it independently corroborates) describes it as “less intuitive and visually dated compared to newer tools… [lacking] advanced visualizations and interactive features.” The absence of a message-browsing view is the most frequently cited specific gap: almost every practitioner comparison uses Kafdrop’s topic-content screen as the counterexample.

There is no documented accessibility support.

Ecosystem

CMAK’s ecosystem coverage is minimal. It connects to Kafka through ZooKeeper; it has no native integration with Schema Registry, Kafka Connect, ksqlDB, Flink, or any cloud-managed Kafka service.

The ZooKeeper dependency is particularly limiting for cloud deployments. Users on GitHub Issue #776 confirm that managed services — MSK, Confluent Cloud, Aiven, Redpanda Cloud — either lock down or no longer expose ZooKeeper endpoints, making CMAK incompatible with the majority of managed Kafka offerings. GitHub Issue #934 (October 2024, stefannmih) specifically flags compatibility problems with Confluent distributions.

The eshepelyuk/cmak-operator maintainer, the author of the most widely used Kubernetes deployment path for CMAK, recommends AKHQ as the better-maintained alternative: “AKHQ project seems to be the most active open source tool for managing and monitoring Kafka clusters. It could be missing some functionality from CMAK, but their developers are open for feature requests and contributions.”

Customer support

CMAK has no commercial support offering. The only channel is GitHub Issues, and the maintainer response rate in 2023-2024 is low. Issues #925, #928, #929, #930, #931, and #933 all contain unanswered feature requests or release requests. A PR to add Kafka 3.3 compatibility (PR #924, submitted August 2023 by HenryCaiHaiying) remained unmerged as of the research date.

There is no documentation site, no community Slack or Discord, and no commercial-support option. The project has not been formally archived by Yahoo, but it has not received a stable release since April 2022.

Best for

CMAK suits small-to-medium teams running on-premises Kafka 2.x or ZooKeeper-era 3.x clusters, where the primary need is partition reassignment, preferred-replica election, and basic broker/topic visibility. It fits operators — SREs and platform engineers — who prefer a direct, Ops-oriented interface and have no compliance requirement for SSO, granular RBAC, or audit logging.

It is not a fit for any cluster running Kafka 3.3+ in KRaft mode, any Kafka 4.0 deployment, or any managed Kafka service where ZooKeeper is not user-accessible. It is also not appropriate for teams that need message inspection, schema management, Kafka Connect administration, or enterprise access controls.

CMAK pricing

CMAK is free and open-source software, released under the Apache 2.0 licence. There are no paid tiers, hosted offerings, or commercial distributions.

Pricing tiers

There is a single tier: free, self-hosted. You are responsible for all infrastructure, deployment, and operational costs.

Free trial

No trial is applicable. You can deploy CMAK directly from the GitHub repository or from a community-maintained Docker image.

CMAK competitors and alternatives

CMAK occupies a specific niche: free, ZooKeeper-era Kafka operations tooling. The wider ecosystem’s move toward KRaft and richer feature sets has eroded that niche substantially. Most practitioners evaluating a new deployment now consider AKHQ, Kafdrop, Redpanda Console, or a commercial option before settling on CMAK.

Best for	Tool	Type	Key functionalities	Deployment & ops	Access control	User interface	Pricing
ZK-era Kafka ops, partition reassignment	CMAK	OSS	Multi-cluster, partition reassignment, preferred-replica election, JMX polling	sbt build; community Docker and Helm only	LDAP; global feature flags	Functional, dated; cached state updates	Free
Full-featured OSS with schema registry and Connect	AKHQ	OSS	Message browsing, live tailing, schema registry, Kafka Connect, ACL management, multi-cluster	Docker, K8s Helm; actively maintained	OIDC, LDAP, per-topic RBAC	Modern SPA	Free
OSS multi-cluster management with advanced SSO	Kafbat UI	OSS	Message browsing, schema registry, multi-cluster, Avro/Protobuf/JSON support	Docker, K8s; actively maintained community fork	OAuth 2.0, Okta, Active Directory	Modern SPA	Free
Lightweight KRaft-compatible monitoring	Kafdrop	OSS	Topic browsing, consumer group monitoring, message inspection	Docker; lightweight footprint	Basic auth	Simple web UI	Free
Developer-focused UI with KRaft support	Redpanda Console	OSS / Commercial	Message browsing, schema registry, Kafka Connect, multi-cluster	Docker, K8s	SSO/RBAC (Enterprise tier only)	Modern React SPA	Free (Community); paid (Enterprise)
Enterprise Kafka tooling with advanced RBAC at scale	Kpow (Factor House)	Commercial	Full Kafka management, monitoring at scale	Stateless; straightforward deployment options	Advanced RBAC	Fully WCAG-compliant	Per-cluster pricing
Confluent Platform users	Confluent Control Center	Commercial	Full Kafka management, ksqlDB, schema registry, monitoring	Bundled with Confluent Platform	RBAC	Feature-rich	Paid (bundled)

For a broader comparison of Kafka management tools, see the Factor House Kafka tools guide.

What practitioners are moving to

Across r/apachekafka and r/devops, the migration path away from CMAK is well-trodden. AKHQ is the most frequently recommended direct replacement for teams that need live topic inspection alongside cluster administration — user thebrobotic on the CMAK replacement thread (2022) noted: “Started using this and I like it better than CMAK. Has more features like view data in a topic / live tail of a topic which is very nice.” Kafbat UI (a maintained community fork of the original Provectus UI) has gained ground for teams running multi-cluster environments that also need OAuth 2.0 or Active Directory integration without a reverse-proxy workaround. For teams with compliance requirements or operating Kafka at significant scale, the gap between CMAK’s feature set and commercial alternatives such as Kpow are substantial: per-topic RBAC, audit logging, and stateless deployment are not achievable in any configuration of CMAK.

Frequently asked questions about CMAK

How much does CMAK cost, and is there a free tier?

CMAK is free and open-source under the Apache 2.0 licence. There is no paid tier, hosted offering, or commercial support option. You self-host and bear all infrastructure costs.

When is CMAK a better choice than the alternatives?

CMAK suits operators managing on-premises Kafka 2.x or ZooKeeper-era 3.x clusters who need partition reassignment, preferred-replica election, and multi-cluster visibility, without budget for commercial tooling and without SSO or RBAC requirements.

When are the alternatives a better choice than CMAK?

If your cluster runs KRaft (Kafka 3.3+ or 4.0), uses a managed service, or requires message browsing, schema registry, SSO, per-topic RBAC, or an audit log, CMAK cannot meet those needs. AKHQ, Kafdrop, or a commercial option are better fits.

Is CMAK still actively maintained?

The last stable release is 3.0.0.6, dated 29 April 2022. Multiple PRs and feature requests have received no maintainer response since then. The project has not been formally archived, but it is not receiving new releases.

Does CMAK support Kafka 4.0?

No. Kafka 4.0 (released 18 March 2025) operates entirely without ZooKeeper. CMAK requires a ZooKeeper endpoint and has no KRaft support, so it is incompatible with Kafka 4.0 clusters.

Confluent Control Center: Review, pricing, and best alternatives in 2026

Factor House — Tue, 26 May 2026 00:00:00 GMT

Key takeaways

Confluent Control Center is built exclusively for Confluent Platform. It cannot monitor Amazon MSK, Redpanda, or Aiven clusters, and requires the proprietary Confluent Metrics Reporter JAR installed on each broker.
The next-generation Control Center, released with Confluent Platform 8.0 in May 2025, reduced startup time from up to 50 minutes to approximately one minute, but upgrading from the legacy version is a full migration that discards all historical metrics.
Pricing is the most frequently cited complaint across practitioner reviews; Control Center, multi-tenancy support, and encryption each carry costs beyond the base Confluent Platform licence.
IBM’s acquisition of Confluent, completed in March 2026, is actively prompting installed-base customers to evaluate alternatives, citing concerns about future pricing pressure and roadmap direction.
At large cluster scale, the legacy interceptor-based telemetry architecture creates measurable overhead: Control Center adds approximately 50 internal topics to broker metadata and can become unresponsive when the number of consumer groups is high. Many enterprise teams bypass it for production alerting, using Prometheus with Grafana instead.
For teams that need Kafka tooling across distributions, or that cannot accept Confluent’s pricing and ecosystem constraints, alternatives such as Kpow offer broader distribution support with more predictable per-cluster pricing.

What is Confluent Control Center?

Confluent Control Center is a web-based management and monitoring interface bundled with Confluent Platform, Confluent’s commercial Kafka distribution. It provides a single dashboard for monitoring brokers, topics, consumer groups, Kafka Connect workers, Schema Registry, ksqlDB, and Kafka Streams topologies across Confluent-managed clusters.

Two generations of Control Center are currently in active deployment. The legacy version, shipped with Confluent Platform 7.x and earlier, has been widely deployed since Confluent’s earliest commercial releases. The next-generation Control Center, released as generally available with Confluent Platform 8.0 in May 2025, replaces the internal Kafka Streams-based metrics pipeline with a Prometheus-based architecture and delivers substantially faster startup and improved partition scale limits.

Control Center is closed-source and requires an enterprise Confluent Platform licence. It is not available as a standalone product.

Confluent Control Center review

Functionalities

Control Center offers comprehensive monitoring within the Confluent ecosystem. Practitioners describe it as providing “a panoramic view into all the elements in Confluent Platform: brokers, consumers, Connect workers, ksqlDB objects.” [Ben, Confluent Community Forum, February 2021] The monitoring module has been described as “impressive” by at least one named enterprise reviewer. [Ahmed Emad, Territory Sales Leader at Sumerge, PeerSpot, January 2024]

The specific functional differentiators that justify Control Center over open-source alternatives are Kafka Streams topology visualisation and native ksqlDB development integration. No open-source Kafka UI provides a comparable native view of these Confluent-specific components.

Consumer lag monitoring has documented boundaries. Consumers using the assign() method rather than subscribe() are not tracked, and Metrics API lag values do not update during a rebalance. [Confluent documentation] A UI rendering bug also affects compound, nested Avro keys in the Topics > Messages view; because Control Center has no public issue tracker, there is no way for users to monitor for a fix. [Ben, Confluent Community Forum, February 2021]

Deployment and operations

The legacy Control Center has a well-documented reputation for slow startup and high resource consumption. Confluent’s own engineering blog acknowledged startup times of 15 to 50 minutes after a restart, and described customer ABANCA’s experience of service restarts as “tedious.” [Philip Wang and Surabhi Singh, Confluent Engineering Blog, May 2025] The next-generation release reduces startup to approximately one minute, increases partition scale support from 120,000 to 400,000, narrows metrics freshness to two to three minutes, and eliminates the requirement for a separate Kafka cluster to store metrics. [Philip Wang and Surabhi Singh, Confluent Engineering Blog, May 2025]

The upgrade path from legacy to next-gen is a full migration, not a standard upgrade. Historical metrics do not carry over, and Confluent recommends running both versions in parallel for 7 to 15 days. Starting with CP 8.0, Confluent Monitoring Interceptors are also removed; teams that embedded interceptors for producer and consumer tracing will need to reconfigure for the Prometheus-based architecture. [Confluent documentation; Confluent release notes]

Interceptor overhead at scale. In the legacy architecture, Control Center relies on client-side interceptors to collect producer and consumer telemetry. A practitioner on r/apachekafka warned that deploying Control Center with interceptors “will add like 50 topics to your kafka brokers.” [xkillac4, r/apachekafka] At high consumer group counts, this metadata burden can become structurally unworkable. One platform engineer on the same thread described the scale failure directly: “Interceptors destroy CCC with the number of consumers/consumer groups we have.” [lord_pirax, r/apachekafka] The same engineer noted that restarts without wiping state or changing the application ID were unreliable on their cluster. These are cluster-size-dependent conditions; the same engineer explicitly noted that “for people without crazy clusters: CCC will probably work fine.”

A documented practitioner response to this overhead is to decouple monitoring from Control Center entirely. On the same thread, an engineer stated: “I use Prometheus with the Confluent provided Grafana dashboards for all production deployments even when the Confluent Control Center is present.” [aerialbyte, r/apachekafka] This pattern of treating Control Center as an ad-hoc debugging tool while routing production alerting through external Prometheus pipelines is reported across multiple community discussions.

Several operational issues in the legacy version remain unresolved. A user running version 7.5.0 reported continuous RAM growth until all available memory was exhausted, with no vendor response. [NhatDuy11, Confluent Community Forum, November 2024] Control Center 7.6.0 on Confluent for Kubernetes was separately reported to stop after a few hours with RackId doesn't exist for process errors; the GitHub issue remained open with no assigned owner. [Mohamed Aziz Tousli, GitHub issue #305, confluentinc/confluent-kubernetes-examples, June 2024] A recurring pattern involves Kafka Streams entering a rebalancing loop on startup, leaving the UI on a “Loading data” spinner for 20 to 30 minutes. Confluent’s own mitigation recommends dedicated hardware at the level of an m4.2xl instance with 6 GB JVM heap and 8 stream threads, meaning Control Center can require compute comparable to the brokers themselves. [Elizabeth Bennett (Stitch Fix) and Xavier Léauté (Confluent), Google Groups, April 2017 — older source, flagged; deployment-sizing guidance confirmed in current Confluent documentation]

Horizontal Pod Autoscaling is not supported for Control Center pods in Kubernetes, and the cp-helm-charts repository was archived in February 2024, narrowing the supported Kubernetes deployment path to Confluent for Kubernetes (CFK). [Confluent documentation; GitHub, confluentinc/cp-helm-charts]

GitOps and configuration drift. A related operational concern for teams managing Control Center via CFK or Helm is configuration drift. Any cluster changes made through the Control Center UI — connector configurations, topic settings — are applied directly to the cluster state rather than through the declarative Git-managed manifest. One DevOps engineer noted the specific friction: “Only problem I have is not being able to easy persist my connector configs or being able to mount them as configmaps.” [r/devops] Teams running strict GitOps pipelines need to treat Control Center as a read-only observability surface in production, or accept that UI-driven changes will diverge from the declared state.

Access control and security

RBAC in Control Center covers the expected enterprise use cases, with audit logging available for authentication and authorisation events across Confluent-managed clusters. [Confluent documentation]

The most consequential security limitation for many organisations is that SAML SSO is not supported for self-managed, on-premises Control Center deployments. OIDC is the only supported SSO protocol for Confluent Platform; SAML is available only on Confluent Cloud. [Confluent documentation] Teams with a SAML-only identity provider have no supported path for SSO on self-managed deployments.

Initial RBAC setup requires the SystemAdmin role for the first role bindings, and in RBAC-enabled environments you cannot send only metrics to Control Center: full management must be enabled, which is an all-or-nothing constraint. [Confluent documentation] A scale limit of 10,000 rules per cluster adds friction for fine-grained access management at larger deployments. [Anonymous tech manager at a 1,001-5,000 employee tech services company, PeerSpot, August 2024] Principals who are Kafka super.user but have no RBAC role assignment cannot view or assign roles through the UI; the CLI must be used as a workaround. [Confluent documentation]

ACL configuration is a recurring pain point in community discussions. One engineer running Control Center in production described the experience as: “ACLs wise, was a pain, as for most of the confluent components, where ACLs are not really well documented.” [r/devops, Gathering opinions on Kafka management tools] This documentation gap is particularly notable given that RBAC is one of the headline enterprise features Confluent uses to justify the licence cost.

User interface

Practitioners who use Control Center within the full Confluent Platform describe it as feature-rich and appropriate for day-to-day operations. One named reviewer called it “more than just an ordinary topic checking tool,” while a named solutions architect offered a less favourable view: “From the control center perspective, there is a lot of room for improvement in the visualization.” [German Osin, Towards Data Science, September 2021; Ravi Bhati, Solutions Architect at a 10,000+ employee tech services company, PeerSpot, October 2021]

The production versus ad-hoc distinction is a consistent thread in community feedback. One platform engineer summarised the trade-off concisely: Control Center is “great for debugging” but “sadly not really reliable/stable/performant” as a continuous production observability tool. [lord_pirax, r/apachekafka] This matches the pattern of enterprise teams who keep Control Center available for incident investigation while routing routine alerting to Prometheus. A separate report from a practitioner running a local or dockerised instance described the experience as buggy, with errors not surfacing correctly, plugin integration issues, and no historical data available. [Accomplished-Map-984, r/apachekafka]

The next-generation UI has received positive feedback. A known WebKit issue in Safari causes authentication failures when browsing topic messages; the documented workaround is Chrome or Firefox. [Confluent documentation] In the legacy version, the “Initialising” spinner can persist for 20 to 30 minutes after a restart, which is most disruptive during incidents. [Google Groups, April 2017; pattern confirmed in current Confluent troubleshooting documentation]

Ecosystem

Control Center integrates natively with Kafka Connect, Schema Registry, ksqlDB, and Kafka Streams. Schema Registry integration in particular receives positive mention: one software architect credited it with “significantly enhancing our organisation’s data quality assurance.” [Gustavo Barbosa dos Santos, Software Architect at C&A Brasil, PeerSpot, January 2024]

The hard boundary is distribution compatibility. Control Center requires the Confluent Metrics Reporter JAR installed in broker classpaths to function fully. This prerequisite cannot be satisfied on Amazon MSK, Redpanda, or Aiven, and MSK’s native IAM authentication is not supported. The result is that Control Center is not viable for any team outside a pure Confluent Platform deployment. [Factor House, March 2026; Hayato Shimizu, AxonOps Blog, December 2025]

IBM’s acquisition of Confluent, completed in March 2026, has introduced meaningful uncertainty about pricing and roadmap direction. One VP Engineering at an unnamed fintech company stated: “We started evaluating alternatives the day the acquisition was announced. Not to leave Confluent, but to understand our options.” [Stéphane Derosiaux, Medium, December 2025]

Customer support

Enterprise tier users generally describe Confluent support as responsive. Below that tier, most issues are handled through documentation and community forums. “We continuously face issues, such as Kafka being down and slow responses from the support team,” noted one SDE II at Nutanix. [Mayank Aggarwal, PeerSpot, February 2025]

A recurring pattern across practitioner reviews is that the complexity of getting Control Center to work correctly requires deep engagement with Confluent professional services, and results have not always matched the investment: one reviewer described “many problems and limitations during implementation despite professional services.” [IT Security and Risk Management Associate at an IT services company, Gartner Peer Insights, January 2025 — anonymous, flagged; included for pattern]

Because Control Center is closed-source, there is no public issue tracker. A GitHub issue raised in September 2024 proposing that the README be updated to state clearly that an enterprise licence is required remained open with no vendor response, pointing to a recurring source of user confusion. [Mark Sallee, GitHub issue #105, confluentinc/control-center-images, September 2024] The Platinum support tier is not available; the maximum is Enterprise, with quarterly patch updates for the current version only. [Confluent documentation]

Pricing is the most consistently cited complaint across practitioner platforms. Multiple named reviewers describe multi-year contracts as “very costly” and rate pricing at 5 to 6 out of 10. “High fees while not offering features that match those of other tools” and “hundreds of configurations that application teams must understand” are both flagged as operational burdens. [PavanManepalli, Wells Fargo, PeerSpot, September 2025; RameshJogula, PeerSpot, April 2024; Praveeen Manvi, PeerSpot, March 2024; Clara Riva, PeerSpot, November 2023] Community feedback reinforces this: one engineer, asked why their team uses Control Center, replied simply: “we pay for it so might as well use it.” [r/devops, Gathering opinions on Kafka management tools] For teams running open-source Kafka rather than full Confluent Platform, the licensing cost of Control Center is rarely justified by features that modern open-source alternatives do not cover. [r/apachekafka community consensus]

Best for

Control Center is a well-matched tool for organisations running Confluent Platform on-premises or on their own infrastructure, with the full stack in place: Kafka Streams, ksqlDB, Kafka Connect, and Schema Registry. If Kafka Streams topology visualisation or native ksqlDB tooling integrated with your monitoring UI is a hard requirement, there is currently no open-source alternative that provides an equivalent view.

It suits large enterprise deployments where the Confluent ecosystem is already established and where in-house Kafka operations are not the preferred model. [Mayank Aggarwal, Nutanix, PeerSpot, February 2025] Community experience also suggests it is effective as an ad-hoc debugging and incident investigation tool, even for teams who have moved routine alerting elsewhere.

It is a weaker fit for teams on MSK, Redpanda, or Aiven (where it simply cannot function), for organisations that require SAML SSO for self-managed tooling, for cost-sensitive or smaller environments, and for teams running mixed-distribution environments or actively managing vendor lock-in risk in the wake of the IBM acquisition. Teams operating at high consumer group counts should evaluate the interceptor overhead against their broker capacity before committing to the legacy architecture.

Confluent Control Center pricing

Confluent Control Center is not priced or sold separately. It is bundled with Confluent Platform under an enterprise licence. Pricing is not published and requires direct engagement with Confluent’s sales team.

Pricing tiers

Confluent Platform is sold under an enterprise licence. Features including Control Center, multi-tenancy support, and encryption carry costs beyond the base tier, meaning the all-in cost of a production Control Center deployment is higher than the licence headline suggests. Multiple named practitioners rate pricing at 5 to 6 out of 10 and describe multi-year contracts as “very costly.” [RameshJogula, PeerSpot, April 2024; Clara Riva, PeerSpot, November 2023; Praveeen Manvi, PeerSpot, March 2024] Additional charges for scaling have also been noted: “They charge a lot for scaling, which makes it expensive.” [Mayank Aggarwal, Nutanix, PeerSpot, February 2025]

The Enterprise licence includes quarterly patch updates for the current version only. The Platinum support tier is not available for Control Center. [Confluent documentation]

Free trial

A time-limited evaluation of Confluent Platform may be available through Confluent’s website.

Confluent Control Center competitors and alternatives

For teams evaluating Kafka management tooling in 2026, the competitive landscape ranges from lightweight open-source options to commercial platforms designed for multi-distribution or multi-technology environments. The most important filtering decision is which Kafka distribution you run: if you are outside Confluent Platform, Control Center is not in the running.

Best for	Tool	Type	Key functionalities	Deployment and ops	Access control	User interface	Pricing
Full Confluent Platform users on-premises	Confluent Control Center	Commercial (bundled)	Kafka Streams topology, ksqlDB, Schema Registry, Kafka Connect, broker and consumer monitoring	Bundled with Confluent Platform; Kubernetes via CFK; HPA not supported	Confluent RBAC; OIDC only (no SAML on self-managed)	Web UI; legacy version slow to load; next-gen improved significantly	Included with Confluent Platform enterprise licence; extra costs for Control Center, multi-tenancy, encryption
Enterprise Kafka teams on any distribution	Kpow (Factor House)	Commercial	Topic, consumer group, schema, and connector management; advanced RBAC; audit log; WCAG-compliant UI	Stateless; no external database dependency; Docker, Helm, JAR; straightforward ops	Advanced RBAC; SAML, OIDC, LDAP	Fully WCAG-compliant; high performance at scale	Per-cluster pricing that scales without penalising team growth
Lightweight developer UI for small teams	AKHQ / Kafbat	OSS	Topic, consumer group, schema management; basic ACL management	Self-hosted; low resource overhead; no external dependencies	ACL-based	Web UI	Free
Kafka operations visibility across distributions	AxonOps	Commercial (with OSS tier)	Monitoring, alerting, backup and restore, topic management	Self-hosted; Cassandra backend	Role-based	Ops-focused web UI	Free tier; commercial tiers available
Data exploration and SQL on streams	Lenses	Commercial	Data exploration, SQL on streams, topology view, connector management	Self-hosted and SaaS	RBAC	Web UI	Per-licence (contact for pricing)

For a broader comparison of Kafka management tools in 2026, see Top Kafka UI tools in 2026: a practical comparison for engineering teams.

Frequently asked questions about Confluent Control Center

How much does Confluent Control Center cost, and is there a free tier?

Control Center is bundled with Confluent Platform under an enterprise licence. Pricing is not published. Named practitioners rate it 5-6 out of 10, describing multi-year contracts as very costly. Additional charges apply for Control Center, multi-tenancy, and encryption beyond the base licence.

When is Confluent Control Center a better choice than the alternatives?

If you run Confluent Platform on-premises with Kafka Streams and ksqlDB, Control Center provides native topology visualisation and ksqlDB tooling that no open-source alternative matches. The next-gen release (CP 8.0, May 2025) also resolves longstanding startup time and partition scaling complaints. It is also well-regarded as an ad-hoc debugging and investigation console even by teams who use external tools for production alerting.

When are the alternatives a better choice than Confluent Control Center?

If you run Amazon MSK, Redpanda, or Aiven, Control Center is not viable. Teams that need SAML SSO for self-managed deployments, have cost constraints, operate at high consumer group counts where interceptor overhead becomes a concern, or are managing mixed-distribution environments will find broader support and more predictable pricing elsewhere.

Does Confluent Control Center work with Amazon MSK?

No. MSK’s native IAM authentication is not supported, and the Confluent Metrics Reporter JAR (required for full Control Center functionality) cannot be installed on MSK broker classpaths. Control Center is not a viable option for MSK deployments.

What changed in the next-generation Control Center released with CP 8.0?

CP 8.0 (May 2025) cut startup from 15-50 minutes to ~1 minute, raised partition support from 120,000 to 400,000, and improved metrics freshness to 2-3 minutes. The new release also removes the legacy interceptor-based telemetry architecture in favour of Prometheus, which eliminates the interceptor overhead that affected large-scale deployments. Upgrading from the legacy version is a full migration; historical metrics do not carry over.

Kadeck: Review, pricing, and best alternatives in 2026

Factor House — Tue, 26 May 2026 00:00:00 GMT

Key takeaways

Kadeck is a commercial Kafka management tool from xeotek, available as a native desktop application and a Docker-based web/teams edition supporting Apache Kafka, Redpanda, and Amazon Kinesis.
Reddit practitioners consistently recommend it for local development, topic inspection, and debugging — especially on managed clusters like Amazon MSK.
The JavaScript QuickProcessor enables lightweight payload transformation and Dead Letter Queue recovery workflows without writing a full streaming application.
The free tier caps message display at 100 records and limits cluster connections to one; teams managing separate Dev, Test, Stage, and Prod clusters will hit that ceiling quickly.
The web edition runs exclusively in Docker with an online license check on every container start; teams in air-gapped environments need a separate offline activation process before deployment.
For teams evaluating commercially supported alternatives with advanced RBAC and enterprise deployment options, Kpow from Factor House is worth examining alongside Kadeck.

What is Kadeck?

Kadeck is a Kafka management and data exploration tool built by xeotek, a German software vendor. It comes in two editions: a desktop application for Linux, macOS, and Windows, and a web/teams edition distributed as a Docker image. Both connect to Apache Kafka clusters and provide a graphical interface for browsing topics, inspecting message payloads, managing consumer groups, and working with ACLs and Schema Registry configurations. Redpanda and Amazon Kinesis are supported alongside Apache Kafka.

The product is commercially licensed and is not open source. [Emil Koutanov, DZone, March 2020] Registration on the xeotek website is required before using the web edition in any substantive capacity.

Kadeck was originally started in 2019 as a solo project by its founder, Benjamin Buick. Buick has noted in Reddit discussions that “hardly any of my original code is left and the team has grown considerably” since then. [Benjamin Buick, r/apachekafka] The tool has maintained an active presence in the Apache Kafka subreddit, where Buick has directly engaged with practitioners to promote and clarify the product’s capabilities.

Kadeck review

Functionalities

Kadeck’s core strength is data exploration. It decodes Avro payloads via Confluent Schema Registry and presents structured records in a columnar layout, which is a practical improvement over reading raw bytes at the command line. A JavaScript-based Quick Processor lets users derive calculated fields from record values without writing a full streaming application. [Markus Günther, personal blog, mguenther.net, January 2021]

Reddit practitioners echo this assessment. In a thread on Kafka troubleshooting, user usualdev described their workflow as: “To check messages, topics and config I use Kadeck. There is another tool called Kafka Tool to check topics and messages as well. Both of these are cross platform.” [usualdev, r/apachekafka] In a thread about Kafka tools for AWS managed Kafka, user lite-beer-1620 noted: “The Kadeck GUI management & monitoring tools for Kafka and MKS are also really good (and free).” [lite-beer-1620, r/aws]

Dead Letter Queue reprocessing. One well-documented use case is recovering malformed payloads from a Dead Letter Queue (DLQ) topic. Buick described the workflow in detail on Reddit: consumer applications write unprocessable records to a designated DLQ topic with an error attribute attached; developers then use Kadeck’s Data Browser to isolate the failed records; the JavaScript QuickProcessor applies inline payload transformations; corrected records are previewed and re-ingested into the original source topic; and Kadeck’s “Delete up to here” command purges the processed DLQ entries to prevent duplicate processing. [Benjamin Buick, r/apachekafka] This is a practical alternative to writing a one-off consumer-producer script for each recovery incident, though it requires a paid plan to inspect more than 100 records per query.

Schema documentation. Kadeck includes an external metadata layer that lets teams document schema fields without modifying the Schema Registry or bumping a schema version. Buick described the capability in a thread on updating field documentation: “We have just recently added topic documentation capabilities to Kadeck.” [Benjamin Buick, r/apachekafka] This is useful for multi-team environments where consistent topic naming and field-level context need to be maintained outside the contract enforcement layer.

AI-assisted tuning. Kadeck 5.x introduced an AI Health Assistant for monitoring and cluster optimization. Independent practitioner assessment of this feature is limited: the most substantial commentary is a single Reddit reply describing it as “a game-changer for Kafka monitoring.” [Mariobmf, r/apachekafka] No independent technical review of the feature has been located.

Several functional limits are documented by independent practitioners. The free tier displays only 100 records per query. [Anonymous G2 reviewer, g2.com, estimated 2022-2024] The export function is inconsistent between formats: CSV exports respect the column filters applied in the record table, but JSON exports do not, which means the two formats carry different data from the same view. [Markus Günther, ibid.] There is no mechanism to stream derived or transformed data back into a Kafka topic — Kadeck is read-oriented. Kafka Streams and ksqlDB are not integrated, a limitation Günther described as worth noting for teams with processing-oriented workflows. [Markus Günther, ibid.]

There is also no built-in deduplication filter to restrict the view to the newest record per key. [Markus Günther, ibid.]

Deployment and operations

The web and teams edition ships exclusively as a Docker image. There are no RPM packages, native binaries, or Helm charts for server deployment. [Guy Shilo, idata.co.il, September 2020] For production Kubernetes deployments, xeotek’s own documentation recommends using a persistent external database.

Every container start triggers an online license validation call. For environments without outbound internet access, xeotek provides an offline activation path through the admin settings panel. [Guy Shilo, ibid.] Automated or ephemeral deployments — for example, spinning up Kadeck in a CI/CD pipeline — require handling a challenge-response license activation via xeotek’s Public API before the container becomes usable.

The desktop edition does not require Docker. It installs natively on Debian/Ubuntu, macOS, and Windows and requires a Java runtime. An independent installation guide published in March 2024 did not report significant issues with the process. [kifarunix.com, March 2024] Apple Silicon and ARM support were added for both editions in 2024.

Using the teams edition requires registering on the xeotek website to receive a team ID and secret key before the application is fully functional. [Guy Shilo, ibid.]

Scale and query performance

Because Apache Kafka does not natively index message payloads, searching for specific records requires sequential partition scans. For topics in the hundreds of gigabytes, this scanning pattern can cause significant performance issues in any desktop-class GUI tool.

A data engineer on r/dataengineering described the limitation directly: their team had Kadeck set up by the cluster hosting team, but found it unusable for large-scale ad-hoc queries, eventually routing high-volume topic data to a parallelized layer instead. [Efxod, r/dataengineering] This is not a Kadeck-specific problem — sequential partition scanning will degrade any UI-based tool at sufficient scale — but it is worth accounting for when evaluating Kadeck for production debugging scenarios involving high-throughput topics.

For large-scale analytical queries, the practical path is to bypass GUI visualizers entirely and use parallelized engines such as PySpark, Databricks, or external data lakes rather than expect any desktop-oriented tool to handle the load.

Access control and security

Kadeck Teams includes role- and group-based access control with LDAP and OpenID Connect integration. The free tier of the teams edition includes password policies and basic audit logs, alongside namespace support for topics, ACLs, and consumer groups. [Capterra listing content, capterra.com]

One Capterra reviewer described the group management as working well for separating project teams, noting that it prevented users from accessing data outside their scope. [Anonymous Product Owner, Capterra, September 2021] No independent practitioner feedback on SSO reliability, RBAC granularity, or the completeness of audit logs was found in any source reviewed during research.

Connection limits and certificate management

The free tier of the teams edition is limited to a single active cluster connection. Multi-broker clusters are supported within that single connection, but connecting to multiple clusters — for example, separate Dev, Test, Stage, and Prod environments — requires a paid plan.

This surfaced directly in a Reddit thread on Kafka visualizers. User Jalebibabyded described looking for an open-source alternative because “we have multiple brokers from which the data needs to be fetched and Kadeck freemium is a limiting there.” Buick clarified in the same thread: “We removed the broker restriction a while ago. Kadeck now supports clusters with multiple brokers even in the free version. However, the number of cluster connections is limited to one cluster (connection).” Jalebibabyded confirmed that was the sticking point: “Yes, that was the problem. Because I was looking at connecting to multiple clusters.” [Jalebibabyded and Benjamin Buick, r/dataengineering]

Certificate management carries a similar constraint. In a recent thread on Kafka UI tools, user Quirky-Design3856 noted: “Me using Kadeck, though free version only offer one cert at a time, its UX is perfect.” [Quirky-Design3856, r/apachekafka] For teams running multiple TLS-secured clusters with separate certificate authorities, this means either purchasing a commercial license or manually rotating credentials on each connection.

User interface

The UI receives consistent praise from the available review sample. Reviewers describe it as clean, modern, and a meaningful step up from CLI-based Kafka tooling. [David W., Solution Architect (IT Services), Capterra, August 2021; Christine U., SAP Consultant (Insurance), Capterra, August 2021; anonymous G2 reviewer, g2.com] One reviewer noted minor usability problems in earlier versions that the support team addressed quickly. [David W., ibid.] Another described it as making working with data from multiple sources straightforward, and cited the team features as time-saving in day-to-day work. [Christine U., ibid.]

Reddit commentary reinforces the UX narrative. The phrase “its UX is perfect” from Quirky-Design3856 — even in the context of raising a limitation — reflects the general tone of practitioner commentary on the interface. [Quirky-Design3856, r/apachekafka]

No independent practitioner assessments of the Kadeck 4.x or 5.x UI redesign were found within the last 36 months. All available feedback dates from 2021 or earlier.

Ecosystem

Kadeck supports Confluent Schema Registry for Avro payload decoding. Protocol Buffer support via Schema Registry was added in release 3.2.3. [Xeotek release notes, support.xeotek.com] Kafka Connect management and ACL configuration are included. Amazon Kinesis is listed as a supported data source alongside Apache Kafka and Redpanda.

Apache Pulsar is not supported, noted explicitly as a gap by at least one reviewer. [David W., Solution Architect, Capterra, August 2021] No independent practitioner feedback on ksqlDB, Apache Flink, Confluent Cloud, Amazon MSK, Aiven, or AWS Glue Schema Registry was found in any source reviewed.

Kadeck appears in the Confluent Community’s reference list of third-party GUI tools for Apache Kafka but receives no user discussion in that thread. [Confluent Community forum, forum.confluent.io]

Customer support

All three Capterra reviewers mentioned support in positive terms. One described receiving answers within minutes to questions that extended beyond Kadeck’s direct scope into general Kafka troubleshooting, and encouraged the team to “keep it up.” [Anonymous Product Owner, Capterra, September 2021] Another noted the support team reacted quickly. [David W., Capterra, August 2021] xeotek has documented conducting structured sessions with users to gather product feedback, and states that critical input from those sessions was incorporated into the Kadeck 4.0 redesign. [Xeotek blog, Kadeck.com]

The confidence limit here is the review sample itself: three reviews, all from 2021, all 5-star. No complaints about support quality appear in any source reviewed, but no commentary on enterprise tier support, SLA differences, or documentation quality was found either.

Best for

Kadeck suits small-to-medium development and QA teams that want a polished graphical interface for data exploration, debugging, and lightweight governance without writing custom scripts or living in the CLI. The desktop edition is a reasonable choice for individual developers on macOS, Windows, or Linux who need a native Kafka browser without a Docker dependency. The teams edition suits environments where a Docker deployment model is already established and group-based access control at the namespace level is sufficient.

Reddit community feedback broadly confirms this positioning: practitioners reach for Kadeck when they want a clean, accessible interface for inspecting topics and debugging consumer behaviour, particularly on managed clusters like Amazon MSK where CLI access is more cumbersome.

It is a less natural fit for teams requiring an open-source-licensed tool, for air-gapped environments that want a simpler deployment model without online license activation, for any workflow that requires streaming derived data back into Kafka topics, for teams managing multiple clusters on the free tier, or for production debugging scenarios where you need to inspect more than 100 records without a paid plan.

Kadeck pricing

The following describes what is confirmed in independent and vendor sources.

Pricing tiers

Kadeck Teams has a free tier: up to five users, with message display capped at 100 records per query. The free tier includes password policies, basic audit logs, namespace support, and group-based access control. Beyond five users, or for teams that need to remove the 100-record display cap, commercial paid tiers apply. Paid tier pricing is user-based, starting at $25 USD per user per month for Professional, and $32 USD per user per month on their flagship Enterprise plan. For a team of 100 engineers, the flagship plan would cost $3,200 USD per month, equivalent to $38,400 per year.

Free trial

A free tier is available for the teams edition within the five-user and 100-record limits described above. Registration on the xeotek website is required to obtain a team ID and access credentials before using the tool.

Kadeck competitors and alternatives

The Kafka management tooling market includes a mix of open-source projects and commercial products. Open-source options typically carry strong community familiarity but require self-managed deployment and tend to offer limited access control out of the box. Commercial tools trade some flexibility for stronger governance features, vendor support, and defined roadmaps.

Reddit discussions reflect this trade-off clearly. In a thread on UI tools for AWS-managed Kafka, developer JohnPreston72 noted a preference for commercial tooling once open-source options are outgrown: “I tried and used nearly all the open source ones. Winner for me was AKHQ before trying Conduktor. I now use Conduktor for work and well worth the money.” In the same thread, armanduco_ expressed satisfaction with AKHQ for lighter requirements: “In my company we use AKHQ, it’s simple but enough to retrieve some important information.” [JohnPreston72 and armanduco_, r/apachekafka]

Tool	Best for	Type	Key functionalities	Deployment and ops	Access control	User interface	Pricing
Kadeck	Dev/QA teams wanting a polished UI; individual desktop users	Commercial	Topic browsing, payload decoding (Avro, Protobuf), consumer group management, Kafka Connect, ACLs, Schema Registry, Amazon Kinesis, JavaScript QuickProcessor	Desktop (native) or Docker for web/teams; online license check on every start	RBAC, LDAP, OIDC; password policies and audit logs on free tier	Described as clean and modern; praised for UX by practitioners	Free: 5 users, 100 records, 1 cluster; paid tiers [UNVERIFIED]
AKHQ	Teams wanting a self-hosted open-source web UI	OSS	Topic management, Schema Registry, Kafka Connect, consumer group management, multi-cluster support	Docker or JVM; no managed hosting	Basic LDAP/OIDC auth; limited role granularity	Functional web UI	Free (Apache 2.0)
Kafbat	Teams wanting an actively maintained AKHQ fork	OSS	Similar to AKHQ; more active maintenance cadence post-2023	Docker or JVM	Basic auth/OIDC	Web UI, similar to AKHQ	Free (Apache 2.0)
Kpow (Factor House)	Enterprise teams needing advanced RBAC, strong performance at scale, and WCAG-compliant UI	Commercial	Topic management, consumer groups, Schema Registry, Kafka Connect, ACLs, RBAC, audit log	Stateless; Docker or Kubernetes; straightforward ops	Advanced RBAC; SSO; WCAG-compliant; trusted by HPE	Fully WCAG-compliant web UI	Per-cluster pricing; Community Edition free for individuals and organisations

For a broader comparison of Kafka management tools, see [our Kafka management tools guide] [placeholder — link to be added].

Frequently asked questions about Kadeck

How much does Kadeck cost, and is there a free tier?

Kadeck Teams has a free tier covering up to 5 users and 100 records displayed per query. It includes password policies, audit logs, and group access controls. Paid tier pricing is user-based, starting at $25 USD per user per month for Professional, and $32 USD per user per month on their flagship Enterprise plan. For a team of 100 engineers, the flagship plan would cost $3,200 USD per month, equivalent to $38,400 per year.

When is Kadeck a better choice than the alternatives?

Kadeck is worth considering if you want a native desktop client on macOS, Windows, or Linux without a Docker dependency, or if a small team needs a clean visual Kafka browser and basic group-based access control within the five-user free tier limit. Reddit practitioners particularly rate it for local development and debugging on managed Kafka clusters.

When are the alternatives a better choice than Kadeck?

If you need open-source licensing, a deployment model without an online license check, stream processing or write-back capability, Apache Pulsar support, multiple simultaneous cluster connections without a paid plan, or the ability to inspect more than 100 records per query without upgrading, other options are worth evaluating first.

Is Kadeck actively maintained?

xeotek released Apple Silicon and ARM support for both editions in 2024 and has documented a 5.x release including an AI Health Assistant. Independent practitioner discussion is sparse compared to tools like AKHQ or Kafbat, and the most recent located independent reviews date from 2021. Buick remains personally active in the Apache Kafka subreddit, where he has engaged directly with user questions and promoted new features.

Kafbat UI: Review, pricing, and best alternatives in 2026

Factor House — Tue, 26 May 2026 00:00:00 GMT

Key takeaways

Kafbat is a free, open-source Kafka UI forked from the abandoned Provectus kafka-ui project in early 2024, maintained by community volunteers with no commercial backing and no SLA.
The tool covers the core Kafka visibility surface well: multi-cluster management, Avro/Protobuf/JSON deserialization, Schema Registry, Kafka Connect, and CEL-based message filtering.
Regression bugs across minor version upgrades have repeatedly broken Confluent Cloud connectivity and Schema Registry serde auto-selection; many users on v1.3.0 choose to stay there rather than upgrade.
RBAC is available but shallow: configuration is error-prone and fails silently, with no team namespacing, no per-role data masking override, and no approval workflows.
For teams that need enterprise-grade access controls, reliable support, and a commercial SLA, Kpow from Factor House is worth evaluating alongside Kafbat.

What is Kafbat?

Kafbat is a web-based UI for Apache Kafka clusters. It is the active continuation of the Provectus kafka-ui project, which paused development in late 2023 and left a critical remote code execution vulnerability unpatched for approximately six months. In early 2024, core contributors led by Roman Zabaluev (GitHub: Haarolean) and germanosin forked the project to kafbat/kafka-ui and released v1.0.0 in March 2024.

The project operates under a GitHub Sponsors model. There is no commercial backer, no paid tier, and no SLA. Development velocity is determined by volunteer contributor interest and the priorities of the core maintainers.

The fork was well received across developer communities. On r/apachekafka, users expressed relief that the project was continuing and verified that their existing configurations carried over: “Looks like a fork, at least I tried it as is, and it is compatible with my old deployment” (Low-Iron6962, r/apachekafka, March 2024). The maintainer confirmed backward compatibility was a deliberate goal for the initial release.

At its core, Kafbat supports multi-cluster management, topic browsing, message inspection with Avro/Protobuf/JSON Schema deserialization, consumer group lag monitoring, Schema Registry integration, Kafka Connect management, and ACL administration. Version 1.3.0 (July 2025) added GCP IAM authentication and MCP (Model Context Protocol) support. Version 1.5.0 introduced live consumer lag updates.

Kafbat review

Functionalities

Kafbat’s core visibility features are solid for a free tool. Topic browsing, message inspection, and Schema Registry deserialization work reliably in stable releases. The move from Groovy-based scripting to CEL (Common Expression Language) filters is consistently praised: it removes the Groovy-based RCE risk while giving users a more expressive and readable filter syntax.

That said, the functionality surface has recurring rough edges. Consumer lag accuracy is unreliable when transactional producers are in use: end-of-transaction marker messages are counted in the lag calculation, causing persistently elevated readings that do not reflect actual consumer progress (krumft, GitHub Issue #1039, April 2025). Filtering behaviour under high message volumes has been described as “unstable and unpredictable” (Rajan Gaul, Product Hunt, approximately February 2026). CEL filters are not persisted between sessions, requiring users to re-enter them on every login (GitHub Issue #1401, October 2025).

There is no message replay capability from the UI, and no ability to add brokers, increase partitions, rebalance, or change replica counts from the interface (Zeenia Gupta, Platformatory, September 2024). These are open feature requests rather than bugs, but they matter for teams that need operational tooling, not just observational tooling.

This scope is broadly understood by community users. One r/apachekafka developer summarised it well: “Kafka UI is great if what you want is a simple CLI replacement… browse topics, consumer groups/lag, tweak configs, peek messages, and mess with Schema Registry without everyone needing 8 terminal tabs. Conduktor / Lenses are more ‘we want guardrails + governance + workflows’ than ‘give me a UI’” (TellersTech, r/apachekafka, approximately 2025). Kafbat is a strong observational console; it is not a change-management platform.

Deployment and operations

Docker deployment is fast: the project is launchable with a single command and accessible at localhost:8080 (Vorrawut Judasri, Medium, October 2024). A Helm chart is actively maintained for Kubernetes deployments, with quick-start documentation available. Some users bypass Docker entirely by downloading the JAR and running it directly from the terminal, which avoids the 4 GB RAM minimum the default Docker configuration recommends (vernochan, r/apachekafka, approximately 2025).

The main operational pain point is dynamic cluster configuration in Kubernetes. Users who attempt to add clusters through the UI in a Kubernetes deployment consistently receive 400 Bad Request validation errors. Static YAML configuration works; dynamic configuration does not. Documentation on how to deploy with dynamic config via Helm is described by multiple users as absent or insufficient (RaWqqq8, GitHub Issue #1637, January 2026).

Other operational issues to be aware of:

Spring Boot 3.4.4 reports unbound properties when cluster config is injected via environment variables, blocking AWS Secrets Manager and similar cloud-native secret patterns (Heniland, GitHub Issue #1045, April 2025).
AWS IAM authentication via STS temporary tokens requires manual JAAS config injection through environment variables, and credential rotation is not documented for this pattern (r/apachekafka, approximately 2025).
The Helm chart cannot mix yamlApplicationConfig with an existing secret, has no extraDeploy support, and does not expose deployment strategy configuration (Helm chart issues #43, #57, #47).
The default Docker configuration recommends 4 GB RAM minimum, which can catch teams running smaller instances.

Access control and security

Kafbat supports YAML-based RBAC, OAuth2 (Google, GitHub, Azure AD), LDAP/Active Directory, and basic auth. Active Directory support was added in v1.1.0 in response to demand carried over from the Provectus project. Community users cite this as a meaningful differentiator among free tools: “Unlike most free UIs, it comes with Active Directory integration for security which is a big plus for us” (Hopeful-Programmer25, r/apachekafka, approximately 2025). Data masking is available with REMOVE, REPLACE, and MASK policies. Audit logging ships as a built-in feature, writing events to a Kafka topic.

In practice, RBAC configuration is error-prone. The most common failure mode is a mismatch between the OAuth token attribute used for subject matching and the email or username format in the RBAC YAML, causing silent permission failures where users cannot see any clusters at all. The project’s own FAQ directs users to enable trace logging on io.kafbat.ui.service.rbac.extractor to diagnose the problem (kafbat RBAC FAQ; GitHub Discussion #290, April 2024).

The security record warrants scrutiny for compliance-sensitive teams. CVE-2025-49127 (CVSS 10.0) was introduced in kafbat’s own v1.0.0: the application accepted user-provided JMX endpoints without validation, and a 30-second scheduler automatically connected to them, allowing any unauthenticated user to trigger unsafe Java deserialization and execute arbitrary code. It was patched in v1.1.0 (SecureLayer7, July 2025). This follows CVE-2023-52251, an RCE inherited from Provectus that took approximately six months to be patched under the original maintainers.

Further gaps: there is no per-role data masking override (feature requested but not yet implemented; GitHub Issue #1311, September 2025), no team namespacing, no approval workflows, and no policy enforcement layer. The audit log feature is documented but inaccessible to users under RBAC due to a confirmed access denial bug that also generates continuous error spam in the server console (povigg, GitHub Discussion #587, October 2024).

User interface

Kafbat’s UI is consistently described as the most modern and visually clean among open-source Kafka UIs, and onboarding is fast for engineers already familiar with Kafka (multiple comparison sources, including Conduktor and Platformatory; Vorrawut Judasri, Medium, October 2024).

The main UX pain points are:

Filters reset on every session, requiring manual re-entry (GitHub Issue #1401).
No consumer groups or lag visibility in the Topics list; users must navigate away to find lag context (GitHub Issue #1405).
The Produce Message sidebar does not retain its “Keep Contents” state between uses (GitHub Issue #1535, November 2025).
With Kafka 4.x in KRaft mode, the UI shows inconsistent partition leaders on every page refresh because it queries brokers rather than the KRaft quorum controller; at least one user reported nearly aborting a production migration based on the false impression of leadership instability (elielfg, GitHub Issue #1513, November 2025).
There are no time-series graphs for consumer lag or message throughput; this has been an open feature request since early in the project (GitHub Issue #233).

Ecosystem

Kafbat covers the standard Confluent ecosystem: Schema Registry with Avro, Protobuf, and JSON Schema deserialization, Kafka Connect with connector and task management, and basic ksqlDB. Custom SerDe plugins are available, including an AWS Glue integration. GCP IAM authentication and MCP support were added in v1.3.0 (Release Discussion #1212, Haarolean, July 2025).

Cloud-managed Kafka compatibility is mixed. GCP Managed Kafka with SASL authentication works reliably, and Kafbat is the recommended replacement for the legacy Provectus image in that environment: multiple r/apachekafka users confirmed successful connections after switching container images (r/apachekafka, approximately late 2024 to early 2025). Confluent Cloud is a different story: connectivity broke in v1.4.x and v1.5.0 due to a metrics refactor that introduced three unhandled failure modes, leaving the cluster in a permanent INITIALIZING state. v1.3.0 works with the same configuration (miehar, GitHub Issue #1852, May 2026). Schema Registry OAuth2 authentication is not supported, blocking use with services such as Google Managed Schema Registry (GitHub Issue #1575, December 2025). Multiple Schema Registries per cluster are not supported (GitHub Discussion #569, October 2024). Flink integration is not available.

Customer support

Kafbat’s primary maintainer, Roman Zabaluev, is actively responsive on GitHub: triage typically happens within days, and there is at least one documented case of an Azure AD RBAC misconfiguration being diagnosed and resolved within 24 hours of the report (GitHub Discussion #290, April 2024). The GitHub Sponsors program offers priority bug handling for sponsors as a formal support incentive.

There is no commercial support tier, no SLA, and no dedicated support queue. Several issues remain open without resolution for months. The audit log RBAC discussion was closed without a confirmed fix for the user who reported it (GitHub Discussion #587, October 2024). For teams that cannot absorb an unresolved incident in production, the absence of an escalation path is a meaningful operational risk.

Best for

Kafbat is well suited to solo engineers and small teams of up to roughly five engineers who need cluster visibility for development or staging environments, are comfortable managing open-source tooling, and are running self-managed Kafka clusters rather than cloud-managed services. It is a practical starting point where budget is the primary constraint and the team can absorb configuration and maintenance overhead.

It is not a strong fit for teams where several engineers share a production cluster and need RBAC at team or namespace granularity, organisations with compliance requirements that depend on a clean server-side audit trail, teams using Confluent Cloud on v1.4.x or later, or any organisation that requires a commercial support channel or SLA.

Kafbat pricing

Kafbat is free and open-source under the Apache 2.0 licence. There is no paid tier and no enterprise edition.

Pricing tiers

Tier	Price	What is included
Open-source	Free	Full feature set; self-hosted; community support via GitHub Issues and Discussions

Free trial

There is nothing to trial: the full application is freely available. Deployment takes a few minutes via Docker or Kubernetes, with no registration required.

Kafbat competitors and alternatives

The open-source Kafka UI market includes a handful of actively maintained tools, from lightweight read-only viewers to fuller operational consoles. Commercial options offer stronger access controls, vendor support, and broader ecosystem integrations at the cost of licensing fees.

One trend shaping adoption in 2024 and 2025: Conduktor’s decision to restrict its community edition - reducing the number of allowed servers and users - pushed a portion of its user base toward fully open-source alternatives. On r/apachekafka, one developer noted “v1.43 ruined it by reducing the number of servers and users allowed” (Sure-Consideration33, r/apachekafka, approximately 2025). Kafbat and AKHQ have been the primary beneficiaries of that migration, though AKHQ has its own ceiling: “We used AKHQ, it’s a nice tool but you hit limitation quite quickly.” (Senior-Act-3761, r/apachekafka, approximately 2025). Teams caught between free tools that run out of headroom and commercial tools with restrictive licensing are the core audience Kafbat is currently serving.

Tool	Best for	Type	Key functionalities	Deployment and ops	Access control	User interface	Pricing
Kpow (Factor House)	Teams needing enterprise RBAC, compliance controls, and commercial support	Commercial	Multi-cluster Kafka, advanced RBAC, data masking, Schema Registry, Kafka Connect	Stateless; straightforward deployment; per-cluster pricing	Advanced RBAC; trusted by large enterprises including HPE	WCAG-compliant; clean and modern	Per-cluster
Kafbat	Small teams and dev environments on a budget	OSS	Multi-cluster, Avro/Protobuf/JSON, CEL filters, Schema Registry, Kafka Connect	Docker/Kubernetes; Helm chart; dynamic config issues in Kubernetes	YAML RBAC; OAuth2; LDAP; no team namespacing	Modern and clean; no filter persistence	Free
Conduktor	Enterprise teams needing governance and approval workflows	Commercial	Data masking, approval workflows, data quality, consumer group management	Cloud or self-hosted	Fine-grained RBAC; team namespacing; approval workflows	Polished; feature-rich	Paid tiers
Confluent Control Center	Teams standardised on Confluent Platform	Commercial	End-to-end monitoring, Stream Lineage, ksqlDB, Kafka Connect	Bundled with Confluent Platform	Integrated with Confluent RBAC	Comprehensive; can feel dense	Bundled with Confluent
AKHQ	Small teams needing a lightweight read-only view	OSS	Topic browsing, message inspection, consumer groups, Schema Registry	Docker/Kubernetes	Basic RBAC	Functional; less polished than Kafbat	Free
Redpanda Console	Teams on Redpanda or vanilla Kafka	BSL / Commercial	Topic management, message viewer, Schema Registry, Kafka Connect	Docker/Kubernetes; cloud option	Basic RBAC in BSL tier; more in paid tier	Modern and fast	Free (BSL, not Apache 2.0); paid (cloud)

For a broader side-by-side comparison, see Top Kafka UI tools in 2026: a practical comparison for engineering teams.

Frequently asked questions about Kafbat

How much does Kafbat cost, and is there a free tier?

Kafbat is fully free and open-source under Apache 2.0. There is no paid tier. The full application is self-hosted and takes minutes to deploy via Docker or Kubernetes, with no account or registration required.

When is Kafbat a better choice than the alternatives?

Kafbat suits small teams and dev environments where budget is the primary constraint and the team can absorb configuration overhead. If you are self-managing Kafka, need basic cluster visibility, and do not require an SLA or enterprise controls, Kafbat is a reasonable starting point.

When are the alternatives a better choice than Kafbat?

When you need team-level RBAC, a reliable audit trail, Confluent Cloud stability, or a commercial support channel. Kafbat has documented gaps in all four areas, and two critical unauthenticated RCEs (both CVSS 10.0) have been disclosed in the project’s history.

Is Kafbat secure?

Kafbat has had two CVSS 10.0 vulnerabilities disclosed: one inherited from Provectus, and CVE-2025-49127, which the kafbat team introduced in v1.0.0 and patched in v1.1.0. Running v1.1.0 or later and restricting network access to the UI are the minimum mitigations for production deployments.

Lenses.io: Review, pricing, and best alternatives in 2026

Factor House — Tue, 26 May 2026 00:00:00 GMT

Lenses.io is a commercial Kafka management platform that sits on top of your existing Kafka clusters, adding a SQL querying interface, topology visualisation, and a data catalog for multi-cluster environments. This review draws on practitioner feedback collected through mid-2026 to give you an honest picture of where the product delivers value and where it falls short.

Key takeaways

Lenses.io’s SQL Studio is its standout feature, giving non-Kafka-savvy team members a familiar interface for inspecting and troubleshooting topics without writing consumer code.
Multi-cluster topology and cross-cluster data lineage are genuine differentiators for organisations running several Kafka deployments across clouds or regions.
Deployment complexity, upgrade friction, and the absence of a high-availability option for the Lenses HQ control plane are recurring concerns in practitioner reviews through 2025-2026.
The community edition caps at two clusters and five users with basic authentication only; SSO, RBAC, and SAML all require the Team tier at a minimum of $4,000/year.
Lenses was acquired by Celonis in early 2022, leading to uncertainty about future development. The product has continued as a standalone offering and has received some version updates since then.
If your primary need is operational observability, audit logging, or data masking across Kafka clusters, Kpow by Factor House is worth evaluating alongside Lenses.io.

What is Lenses.io?

Lenses.io is a Kafka governance and data exploration platform. It connects to one or more Kafka clusters and surfaces topic data, schema information, consumer group state, and Kafka Connect configuration through a browser-based UI. Its primary differentiator is SQL Studio, a proprietary SQL interface that lets engineers and analysts query Kafka topics using familiar syntax without writing Kafka consumer code.

The platform also provides a topology view that maps data lineage across topics, connectors, and applications (including externally registered Flink jobs). A data catalog groups topics by logical domain and supports schema registry integration.

Lenses operates as a Kafka client; it does not sit in the data path and does not act as a proxy. The current architecture consists of a central Lenses HQ node and lightweight agents deployed per cluster. KRaft clusters are supported without modification, since the agent connects as a standard Kafka client.

Within the Kafka tooling ecosystem, practitioners draw a distinction between lightweight desktop clients designed for quick topic inspection and enterprise control planes that add governance, RBAC, and audit workflows. Lenses.io sits firmly in the latter category: its value becomes most apparent once multiple teams are sharing a cluster infrastructure and need guardrails around access, configuration changes, and data visibility.

The product is available in three tiers: a free community edition, a Team edition, and an Enterprise edition.

Lenses.io review

Functionalities

The SQL Studio is consistently the most praised aspect of Lenses.io in practitioner reviews. Juan Luis d. on G2 (November 2025) described it as “providing a straightforward way to troubleshoot problem cases and check topic metrics,” with the broader UI characterised as “accessible for both experienced individuals as well as those who are not very familiar with Kafka development.” Community discussion on r/apachekafka echoes this: engineers highlight that SQL Studio lowers the barrier for team members who are not comfortable with Kafka’s internal mechanics, replacing the need to search through millions of raw JSON, Avro, or Protobuf payloads with familiar query syntax.

The topology and lineage view is the second most cited strength. In multi-cluster environments where teams need a single pane of glass across producers, topics, connectors, and consumers, the topology view reduces the operational overhead of maintaining that picture manually.

Lenses.io also provides SQL Processors, Kubernetes-native stream processing engines built on Kafka Streams. These allow teams to define stateful and stateless processing rules in SQL, compiled and executed within a Kubernetes cluster, without maintaining a dedicated Flink or Spark infrastructure. For straightforward transformation and filtering tasks, this can be a practical option. However, the proprietary nature of SQL Processors is a meaningful consideration. Jose Manuel C. on G2 (November 2025) wrote: “Some of Lenses’ functionalities that are not open-source create a vendor lock-in, with SQL Processors being the most clear example… the tradeoff doesn’t pay off in the long run.” Teams considering SQL Processors as a core part of their architecture should weigh portability before committing.

The limitations are equally consistent across reviews. The ACL management interface has not advanced meaningfully across major releases: one unnamed G2 reviewer from the retail sector noted that version 6 “introduced a new interface, but the same issues remain: the ACLs listing is still too basic.” Bug resolution is perceived as slow, with Juan Luis d. noting “ongoing bugs within Lenses that remain unresolved, highlighting a potential lack of responsiveness in bug fixes.” One practitioner on r/apachekafka put it more directly: “I often felt like a QA for Lenses developers as a customer and it was exhausting.”

Schema Registry integration exists but was described by Berta m. on G2 (November 2025) as “not entirely optimal” in terms of schema inference accuracy.

Deployment and operations

Lenses.io supports Helm-based Kubernetes deployment, which is the recommended path. In practice, practitioners have encountered meaningful friction here.

The platform runs a server-side JVM backend alongside its web interface, which makes it a relatively resource-intensive deployment. This comes up frequently in community discussions: developers who only need to inspect local topics during development find the Docker and JVM overhead excessive compared to lighter alternatives. Starting with version 5, the community edition also restricts connections to single-node clusters, which has frustrated users attempting to test multi-broker SSL-enabled environments locally. Several have reported rolling back to version 4.3 to retain multi-broker support in their trial setups.

Upgrades to Lenses 6 HQ triggered Liquibase database migration errors in at least one documented case on the community forum (ask.lenses.io, November 2025), where a column-rename operation failed depending on the source version and database backend. A separate reported issue involved Traefik ingress login loops caused by the default secureSessionCookies: true setting, which breaks non-HTTPS environments; the workaround requires adding lensesHq.http.secureSessionCookies: false to values.yaml.

A third operational issue affects teams connecting Lenses to Amazon MSK: the default broker metrics refresh interval (approximately 5 seconds) generates an unexpectedly high volume of JMX requests against MSK’s OpenMetrics/Prometheus-backed JMX exporter. The recommended workaround is to set the interval to 30 seconds or higher (ask.lenses.io, April-May 2026).

Lenses HQ does not support high availability in the current architecture. Multiple G2 reviewers in 2025 identified this as a gap, particularly in production environments where the control plane itself needs to be resilient. The community edition further limits deployment to two clusters with five users and basic authentication only; a Helm chart for community edition was still a feature request as recently as May 2025.

Access control and security

SSO (Okta, Keycloak, OneLogin, Google, Azure/Entra ID) and RBAC are available but gated to the Team tier at $4,000/year minimum. The community edition provides only username/password authentication.

LDAP integration has had documented instability across recent releases. A connection-reset bug was fixed in v5.5.14 (December 2024). Version 5.5.6 (August 2024) introduced a breaking behaviour change: new LDAP users are no longer automatically created unless they belong to a mapped group, which silently broke access for teams relying on the previous behaviour.

Azure Entra ID SAML integration has a documented limitation: Azure exposes only Group UUIDs (not group names) via SAML, requiring administrators to use UUID strings as group identifiers within Lenses. Google SSO requires custom attribute mapping because Google does not expose user groups or organisation units to SAML applications by default.

The read-only account use case, a common requirement for giving stakeholders visibility without change permissions, was an unanswered question on the community forum as of August 2025, which suggests RBAC granularity for viewer-only roles is either not well-documented or not straightforward to configure in v6.

It is worth noting that identity provider support has historically been a point of friction with Lenses. The team behind Kowl (now Redpanda Console) cited Lenses’ lack of support for their preferred identity provider and group configurations as one of the motivating reasons they built their own tool.

User interface

The general UI receives consistently positive marks in practitioner reviews. Descriptions include “simple and effective” (unnamed reviewer, retail sector, G2, November 2025), “user friendly, easy to use” (unnamed reviewer, apparel sector, G2, November 2025), and praise for the ease with which non-engineers can interact with topic configurations.

The version 6 redesign refreshed the visual style but did not resolve the underlying capability gaps that existed in v5. The ACL management views remain limited, and the deployment experience was still described as “complicated” by reviewers writing after the v6 release. Documentation quality is also cited as a persistent concern, with multiple reviewers in 2025 describing it as “insufficient and unclear.”

Ecosystem

Lenses operates as a Kafka client and integrates with Schema Registry, Kafka Connect, and the Topology view can include externally registered Flink jobs for monitoring (though Flink is not natively managed through the platform). KRaft clusters are supported; the only known limitation is a minor display bug where the controller count is not shown correctly on KRaft clusters (confirmed by a Lenses team member on ask.lenses.io, July 2025).

The open-source Stream Reactor connector library extends Lenses with a range of sinks and sources. Active GitHub issues from late 2025 through early 2026 show a maintenance backlog for the S3 connector (ByteArrayConverter envelope restoration failures, ConnectionClosedException on S3 Source), an MQTT source SchemaParseException triggered by topic names containing hyphens, and Azure Data Lake Gen2 connector header errors.

Teams connecting to MSK Serverless with IAM authentication should note a documented access-denied issue: Lenses 6 Agent requires kafka-cluster:* IAM actions, not the broader kafka:* scope. A well-documented workaround exists on the community forum.

Celonis acquisition and product continuity

Celonis acquired Lenses.io in early 2022. At the time, the acquisition generated significant uncertainty within the Kafka community, with several engineers on r/dataengineering and r/apachekafka speculating that Celonis had bought the company primarily for its core technology and planned to discontinue the standalone product.

Lenses.io continues to operate as a standalone commercial product, and the cadence of version releases (including 5.3, 5.4, and the version 6 UI redesign) indicates that Celonis has maintained some active development. For teams evaluating the product, the acquisition risk appears lower than it did in 2022 and 2023, though Celonis has not publicly committed to a long-term standalone roadmap in specific terms.

Customer support

Community edition users rely on the public forum (ask.lenses.io) and documentation. The forum is actively monitored by a small number of Lenses team members, and response quality for technical issues is generally fair. Enterprise support SLAs are available at the Team and Enterprise tiers.

Independent review coverage is thin. Gartner Peer Insights has two ratings as of mid-2026, giving Lenses a 3.7/5 overall with a 50% willingness to recommend. TrustRadius and PeerSpot had insufficient or no reviews to report an aggregate score at the time of this research. Roadmap clarity and documentation quality were the most frequently cited support-related concerns in 2025 G2 reviews.

Best for

Lenses.io is a good fit for teams where the primary use case is giving business analysts or less Kafka-savvy engineers self-service access to Kafka topic data through a familiar SQL interface, without requiring them to write consumer code or understand Kafka internals. It is also well-suited to organisations running multiple Kafka clusters across regions or cloud accounts who need unified topology visibility and cross-cluster querying.

It is worth evaluating carefully if your organisation is small or budget-constrained (the step from the community edition to the Team tier is significant), if high availability for the control plane is a hard requirement, or if you are building on Kafka primarily for operational observability and governance rather than SQL-based data exploration.

Lenses.io pricing

Lenses.io uses a tiered model with a free community edition and two paid tiers. Pricing details are based on the Lenses pricing page and community forum sources.

Pricing tiers

Tier	Price	Clusters	Users	Notable limits
Community	Free	2	5	Basic authentication only; no SSO, no RBAC; single-node clusters only from v5
Team	From $4,000/year	Multiple	Multiple	SSO, SAML, RBAC, and team-level support included
Enterprise	Contact sales	Unlimited	Unlimited	Enterprise support SLAs, advanced governance

The jump from Community to Team is steep for smaller teams. SSO and any form of role-based access control require the paid Team tier, which starts at $4,000/year. The community edition’s two-cluster ceiling also limits its usefulness for anything beyond a single development or staging environment.

Self-hosting context

One recurring theme in community discussion is the financial case for self-hosting Kafka on Kubernetes with an operator like Strimzi, combined with a commercial management plane. Engineers on r/apachekafka have noted that managed Kafka pricing can change significantly with billing model updates: one example cited a partition-based fee change that would have increased monthly costs from $30 to $1,200 for a single team. Against that backdrop, the predictability of a fixed annual licence for Lenses (or an alternative like Kpow) can be attractive, even at the Team tier price point.

Free trial

A time-limited trial of the Team or Enterprise edition is not available from the Lenses website.

Lenses.io competitors and alternatives

Lenses.io occupies a specific niche: SQL-based self-service access and multi-cluster topology for teams investing in a DataOps model. Depending on your actual requirements, other tools may be a better fit. The table below covers the most relevant alternatives across open-source and commercial options.

Tool	Best for	Type	Key functionalities	Deployment and ops	Access control	User interface	Pricing
Conduktor	Governance-first organisations needing proxy-level policy enforcement, data masking, and audit trails	Commercial	Consumer group management, schema registry, Connect, gateway proxy for policy enforcement	Kubernetes, Docker; proxy sits in data path	SSO, RBAC, data masking, audit log	Clean and opinionated; strong governance UI	Tiered; community plan available
Kpow by Factor House	Platform teams needing stateless deployment, air-gap operation, per-cluster pricing, and comprehensive RBAC with audit logging	Commercial	Topic inspection with kJQ filtering, consumer lag, Kafka Streams topology, schema-aware deserialization, staged approval workflows	Docker, Kubernetes, JAR; stateless, no external DB; air-gap capable; deploys in minutes	SSO (Okta, Azure AD, Keycloak, LDAP, SAML, OAuth2), RBAC, data masking, full audit log	Focused and fast; no SQL layer	Community free; Enterprise from $3,950/year per cluster; 30-day trial, no credit card
AKHQ	Development teams wanting a zero-cost, self-hosted Kafka UI	OSS	Topic browser, consumer group management, schema registry, Connect	Docker, Kubernetes; self-managed	Basic auth, LDAP, OAuth2; limited fine-grained RBAC	Functional; suited to engineers	Free (self-hosted)
Kafbat UI (kafka-ui)	Teams wanting a lightweight, actively maintained OSS alternative to the archived kafka-ui	OSS	Topic/consumer group browser, schema registry, Connect, basic ACL management	Docker, Kubernetes; self-managed	Basic auth, OAuth2	Modern, configurable	Free (self-hosted)
Confluent Control Center	Teams fully committed to the Confluent Platform stack	Commercial	Cluster health, ksqlDB, stream lineage, connector management	Bundled with Confluent Platform; cloud or on-prem	Confluent RBAC	Polished; tightly coupled to Confluent	Included in Confluent Platform licensing
Redpanda Console	Teams using Redpanda or wanting a lightweight Kafka-compatible UI	OSS / Commercial	Topic browser, consumer groups, schema registry, Connect	Docker, Kubernetes; lightweight	Basic auth; SSO in enterprise tier	Minimal, fast	Free OSS; paid tiers for enterprise

For a more detailed comparison of Kafka UI tooling, see our guide to Kafka UI tools in 2026.

Frequently asked questions about Lenses.io

How much does Lenses.io cost, and is there a free tier?

The community edition is free and supports 2 clusters and 5 users with basic authentication only. From version 5, the community edition is also restricted to single-node clusters. SSO and RBAC require the Team tier, which starts at $4,000/year. Enterprise pricing is available on request.

When is Lenses.io a better choice than the alternatives?

Lenses.io is strongest when your team needs SQL-based self-service access to Kafka topics for non-engineering stakeholders, or when you need unified topology visibility and cross-cluster querying across multiple Kafka deployments. No comparable OSS tool provides an equivalent SQL Studio experience.

When are the alternatives a better choice than Lenses.io?

If your primary need is operational governance (audit logging, data masking, gateway-level policy enforcement), Conduktor’s proxy architecture is better suited. For teams whose main requirement is comprehensive RBAC, audit logging, and stateless deployment without an external database dependency, Kpow is worth evaluating. For small teams or single-cluster deployments, the community edition’s limits and the $4,000/year jump to SSO make the cost-to-value ratio difficult to justify against open-source alternatives.

Does Lenses.io support KRaft?

Yes. Lenses connects to Kafka clusters as a standard client, so KRaft clusters work without modification. There is a minor known display issue where the controller count is not shown correctly on KRaft clusters, confirmed by the Lenses team in July 2025.

Is there high availability for the Lenses control plane?

Not in the current architecture. Lenses HQ is a single-node deployment. Multiple practitioner reviews from 2025 identified the absence of native HA as a gap for production control-plane resilience. Manual workarounds are possible but are not a supported configuration.

Is Lenses.io still actively developed following the Celonis acquisition?

Yes. Celonis acquired Lenses.io in early 2022, which initially prompted concern in the community about product continuity. The product has continued as a standalone offering since then, with version updates including a full v6 UI redesign and ongoing connector and feature development. Community forum support from Lenses team members has remained active.

Redpanda Console: Review, pricing, and best alternatives in 2026

Factor House — Tue, 26 May 2026 00:00:00 GMT

Key takeaways

Redpanda Console’s message viewer is among the best available in any free Kafka UI: time-travel offset debugging, multi-format deserialization (Avro, Protobuf, JSON, XML, CBOR, MessagePack), and JavaScript-based filtering are standout capabilities for developer workflows.
The Observer Mode lets developers stream topic messages without joining a consumer group, avoiding accidental rebalancing on production offsets.
The tool is a viewer and browser, not an operational platform. There are no built-in broker metrics, no alerting, and no historical trend views; a separate Prometheus/Grafana stack is required for production monitoring.
RBAC, SSO, and data masking are locked behind a paid Redpanda Enterprise license. Teams running vanilla Apache Kafka or Amazon MSK who need access controls will pay for a Redpanda license even if they do not run Redpanda as their broker.
Multi-cluster management is absent at the broker tier. Teams managing dev, staging, and production clusters must deploy separate Console instances.
If you need production-grade access controls, multi-cluster support, and a fast-moving commercial roadmap, Kpow is worth evaluating as a dedicated Apache Kafka management tool.

What is Redpanda Console?

Redpanda Console is an open source web UI for inspecting and managing Apache Kafka-compatible clusters. It was originally developed by CloudHut under the name Kowl and acquired by Redpanda in April 2022. The product serves both Redpanda clusters (where it integrates with the Redpanda admin API for additional capabilities) and vanilla Apache Kafka, Amazon MSK, and Confluent Platform deployments.

The community edition is available under the Business Source License (BSL), which permits free use by internal teams but restricts commercial SaaS use. An enterprise edition adds RBAC, SSO (OIDC), and data masking behind a paid Redpanda Enterprise license.

The application is built on Go and React, and is distributed as a Docker image and a Helm chart.

Platform neutrality is a frequently confirmed strength among practitioners. Engineers running standard Apache Kafka report the tool works well outside of a Redpanda environment: “I’ve been using Redpanda’s console and it’s pretty awesome. You should be able to use it with any implementation of Kafka — that’s what I’m doing.” [vintage_px, r/apachekafka, 2024] This positions Console as a viable standardisation choice for teams running diverse broker configurations across on-premise, cloud-managed, and hybrid environments.

Redpanda Console review

Functionalities

Redpanda Console’s primary strength is its message viewer. The interface supports consuming, seeking, and filtering messages with a level of care that practitioners consistently describe as best-in-class for a free tool. One engineer who spent a year running Redpanda in production wrote that the interface made dedicated ad-hoc tools like Apache Zeppelin or custom query applications unnecessary. [Yaroslav Tkachenko, streamingdata.tech, July 2023]

Time-travel offset management, replay workflows, and JavaScript-based message filtering are well-regarded for developer debugging scenarios. Deserialization covers Avro, Protobuf, JSON, XML, CBOR, MessagePack, and binary (hex).

Protobuf without a schema registry is a less-discussed but practically useful capability. Teams with Protobuf-encoded topics that have no schema registry in their stack can configure Console with local proto descriptor maps, declaring which schema applies to which topic. The original Kowl maintainer confirmed: “Redpanda Console is capable of doing this — you just have to configure what Protos it should use for each topic.” [leventus93, r/apachekafka, 2022] This avoids the common workaround of running external CLI decoding scripts during incident response.

Observer Mode allows developers to stream and browse topic messages without joining an active consumer group. This matters in production contexts: joining a consumer group triggers a rebalance, which can affect downstream processing and pollutes offset tracking. Observer Mode sidesteps this entirely. Engineering teams reference it specifically for development-time debugging and lag investigation: “We mainly use scripts as we want automation. But we use Kowl/Redpanda Console for quickly viewing messages or checking lags during development or troubleshooting for instance.” [handstand2001, r/apachekafka, 2025]

Beyond the message viewer, the feature set thins out. The tool has no built-in broker metrics, no alerting, and no historical trend analysis. An independent comparison from January 2026 concluded that Console is suited to “browsing and basic management” and that any production monitoring setup requires an external stack. [KLogic, klogic.io, January 2026]

Consumer group management is present, but there is a known rendering bug: on some cluster configurations a partition is absent from the consumer group view in the UI while being correctly visible via the command-line tool rpk group describe. [freef4ll, GitHub Issue #447, August 2022]

Schema Registry support was historically read-only. A GitHub issue requesting write operations (creating, updating, and deleting schemas) was filed in August 2022 and closed, suggesting the capability was addressed, though the exact resolution is not confirmed in public sources. [sap1ens, GitHub Issue #434, August 2022] An intermittent display bug causing the Schema Registry Overview card to show “Not configured” despite correct configuration was filed in April 2026. [haoyukongTrackunit, GitHub Issue #2422, April 2026]

There is no ability to produce (write) messages to a topic via the UI. A feature request for this was filed in May 2021 and closed with no visible resolution. [tej1996nitrr, GitHub Issue #221, May 2021] The capability was discussed publicly on Reddit as far back as 2021, when the then-Kowl maintainer indicated it was on the roadmap — including key and value input fields, JSON validation, and confirmation dialogs — but it has not appeared in the stable release as of the research date. [Cell-i-Zenit and leventus93, r/apachekafka, 2021]

Console cannot set request quotas or trigger partition rebalancing. [Zeenia Gupta, platformatory.io, September 2024]

Deployment and operations

Docker and Helm deployments are straightforward for single-cluster setups. One engineer deployed Console on Amazon EKS via Helm with “minimal configuration required.” [Priyankar Prasad, medium.com, February 2023]

Local developer overhead is a recurring friction point for engineers who want ad-hoc topic inspection without running a persistent stack. Because Console is a server-side web application rather than a native desktop client, it requires Docker or a running server. Practitioners note the mismatch for lightweight diagnostic tasks: “Kafka UI, AKHQ, Redpanda Console — all great, but they’re web apps that need Docker or a server. On my work machine I don’t always have Docker running, and spinning up a container just to peek at a topic feels like overkill.” [r/apachekafka, 2025] This has driven interest in native desktop and TUI alternatives for local development loops, even among teams that run Console centrally for shared cluster access.

There is a known Helm upgrade bug: upgrading with console.enabled: true fails with “cannot patch ‘redpanda-console’ with kind Deployment” due to immutable label selectors. This is documented in the official Redpanda troubleshooting guides.

At scale, hardcoded timeouts become a meaningful operational problem. GetClusterInfo has a 6-second timeout for DescribeLogDirs and Metadata; GetTopicsOverview has a 5-second timeout for DescribeConfigs. On large AWS MSK clusters with IAM authentication, these limits are regularly exceeded because the overhead of the STS token exchange with IAM adds latency that the hardcoded values do not account for. Observed actual times of 4.9s and 5.7s were reported in April 2026. [grassiale, GitHub Issue #2410, April 2026] A separate timeout of 35 seconds in ListMessages also causes failures on slow or heavily filtered clusters; versions 3.5.2, 3.6.0, and 3.7.2 are all affected, and users have requested configurable timeout values. [cobolbaby, GitHub Issue #2432, May 2026]

Partial cluster failures expose a design limitation: when a single broker goes offline in a multi-node cluster, all consumer group queries fail entirely rather than degrading gracefully. The error reads “failed to list end offsets for topics: request ListOffsets has 1 separate shard errors.” This is a significant problem during incidents, when observability is most critical. [cobolbaby, GitHub Issue #2327, March 2026]

Kubernetes and TLS complexity is a documented risk for teams running Console in Kubernetes alongside the Redpanda Operator. Integrating with Knative channels, ingress controllers, or custom TLS configurations can lead to certificate validation failures against the Admin API on port 9644, hostname verification errors, and rpk debug bundle failures when localhost is not included in the certificate’s subject alternative names. One practitioner documented spending approximately 20 hours attempting to resolve these issues on a fresh GitOps-managed cluster before switching to Kafka with Strimzi, which completed the equivalent setup in under an hour: “After 3-4 hours (now total of about 20 hrs), I decided to cut losses with Redpanda Operator, and instead went with Strimzi.” [JuroOravec, r/kubernetes, 2024] Automated CD pipelines can encounter additional friction from Helm rate-limiting during FluxCD reconciliation of the Console HelmRelease.

Access control and security

RBAC and SSO (OIDC) are enterprise-only features. The free community tier has no access control of any kind. This is a significant limitation for teams managing shared clusters in multi-team or regulated environments.

The OIDC integration introduces a split authentication model that is worth understanding before committing to it. OIDC SSO for the Console UI is available across Redpanda Enterprise Self-Managed, BYOC, and Dedicated cloud deployments. However, OIDC authentication for the Kafka API, HTTP Proxy API, Admin API, and Schema Registry API is only available in Redpanda Enterprise Self-Managed as of early 2026. [r/redpanda, 2025] Teams on BYOC or Dedicated therefore operate two parallel security models: human operators authenticate through a centralised identity provider when accessing the Console UI, while automated client applications must fall back to mTLS, SASL/SCRAM, or OAuthBearer tokens when accessing the same APIs programmatically. This increases the maintenance surface and complicates auditing.

If the enterprise license expires at runtime, Redpanda Console shuts down entirely. If the license has already expired at startup, Console prints an error and exits. This behaviour creates production risk if license renewal is missed. [Redpanda documentation, docs.redpanda.com]

A security configuration bug filed in May 2026 reports that Kafka Connect requests ignore the Console base path setting. [malinskibeniamin, GitHub Issue #2440, May 2026]

Multi-tenant deployments requiring per-account payload visibility controls present an additional challenge. Teams co-mingling records from different accounts on shared topics, where access must be restricted by jurisdiction or contract, need data masking rules that redact values based on user group membership while preserving metadata for debugging. This capability is available in Console but is gated behind the enterprise license. [GitHub Issue discussion, r/apachekafka, 2021]

One comparison article summarised the commercial model plainly: “Every enterprise feature that matters (SSO, RBAC, data masking) requires a paid Redpanda Enterprise license. This creates a problematic dynamic for vanilla Apache Kafka or MSK users: you can use the viewer for free, but the moment you need governance, you’re paying for a Redpanda license even if you don’t run Redpanda.” [Factor House, factorhouse.io]

User interface

The UI is consistently described as modern, clean, and developer-focused. Practitioner feedback from 2022 onwards regularly describes it as the best-looking free Kafka UI available, with the message viewer praised as “extremely thoughtful.” [Yaroslav Tkachenko, streamingdata.tech, July 2023; rfernandez2007, Confluent Community Forum, February 2022]

Several active frontend bugs undermine this reputation on specific configurations. Pressing ? (the help shortcut) crashes the entire page with a React error when Console is connected to vanilla Apache Kafka rather than Redpanda. The root cause is repeated failed calls to the ListEnterpriseFeatures endpoint, which returns HTTP 501 “the Redpanda admin API must be configured to use this endpoint.” The bug affects versions 3.1.0, 3.3.0, and 3.5.3. [alexkau, GitHub Issue #2262, March 2026] A separate input field bug causes the search text field to replace typed input with a scientific notation number. [Wouter-M, GitHub Issue #2459, May 2026] Message previews could not expand or collapse after upgrading to Console 3.5+. [cobolbaby, GitHub Issue #2247, February 2026]

Users running Confluent Platform noted that the tool “felt optimised for Redpanda rather than Confluent Platform,” with credential propagation limitations for Kafka Connect. [whatsupbros, Confluent Community Forum, March 2023]

Ecosystem

Kafka Connect management is present: Console queries all configured Kafka Connect clusters and supports multi-cluster Kafka Connect setups (unlike broker-tier multi-cluster, which is not supported). Functionality covers connector discovery, bulk configuration operations, and basic task status monitoring. This distinction matters for teams with connector-heavy architectures who want a single pane for Connect management without needing separate instances per environment. [Redpanda documentation, docs.redpanda.com]

There is no native support for ksqlDB. A feature request was filed in 2021 and, as of the research date, there is no confirmed implementation. [GitHub Issue #177]

Multi-cluster management at the broker tier is not supported. A team managing development, staging, and production clusters must run three separate Console instances. Two GitHub issues requesting single-instance multi-cluster support were filed in 2021 and 2022; one remains open. [GitHub Issues #250, #349]

Support for Redpanda Connect (pipeline) instances via Console has been requested but is not yet available. [steffyd, GitHub Issue #2430, May 2026]

Licensing and community reception

The BSL licensing transition following the Redpanda acquisition of CloudHut in 2022 has been a persistent source of friction in the open-source community. Kowl was originally maintained under highly permissive terms that encouraged broad adoption. The shift to commercial licensing alienated a portion of the tool’s user base, particularly those who relied on it for sporadic or non-commercial use: “I was a heavy user of Kowl, but I do not like their new Redpanda license. Waiting to hear other people’s recommendations. I need a free version for sporadic usage.” [CasperTDK, r/apachekafka, 2022]

This pattern is not unique to Redpanda Console. Competing tools have followed similar trajectories: Conduktor restricted its community version in release v1.43, reducing the number of allowed servers and users, while Lenses charges approximately $3,650 per cluster annually for its commercial tier. The community response has been a cycle of open-source alternatives gaining rapid adoption — most recently Kafbat UI (the maintained Apache 2.0 fork of the original Kafka-UI project) — followed by those alternatives eventually reaching the same commercial crossroads.

For teams that need enterprise access controls and are committed to the Redpanda ecosystem, the licensing model is a reasonable trade-off. For teams running vanilla Apache Kafka or Amazon MSK who simply want governance features, the situation is less comfortable: the enterprise license funds Redpanda’s commercial interests regardless of whether the buyer runs a single Redpanda broker.

Customer support

Documentation at docs.redpanda.com is comprehensive, covering installation, configuration, security, Kubernetes deployment, and troubleshooting in detail. Active changelog entries confirm ongoing bug fixes and security patches.

GitHub Issues is the primary community support channel for bugs and feature requests, and engagement from the Redpanda team is visible across recent issues.

No practitioner accounts of enterprise support quality, response times, or account management were available in the public sources reviewed for this article.

Best for

Redpanda Console is best suited to:

Teams running Redpanda as their broker (self-managed or cloud). The native integration with the Redpanda admin API enables features that are unavailable on vanilla Kafka. [Hayato Shimizu, axonops.com, December 2025]
Solo developers and small teams who need a lightweight, visually clean message browser with strong deserialization support for a single cluster.
Developer debugging workflows: time-travel offset management, Observer Mode for non-intrusive topic inspection, offset rollback for replay scenarios, and JavaScript-based message filtering are the standout capabilities.
Platform teams hosting a shared single-cluster instance for internal developers, where the platform team handles SASL/TLS configuration and developers access a read-oriented UI.

It is a poor fit for teams that need multi-cluster management, production observability without a separate monitoring stack, governance features without paying for a Redpanda Enterprise license, reliable operation on large AWS MSK clusters with IAM authentication, or a native desktop option for lightweight local inspection.

Redpanda Console pricing

Redpanda Console is open source under the Business Source License. Internal use is free; commercial SaaS use requires a commercial agreement.

Pricing tiers

The community edition is free and covers the core message viewer, topic and consumer group management, Kafka Connect management, and Schema Registry browsing. RBAC, SSO (OIDC), and data masking are not included.

Enterprise features require a Redpanda Enterprise license. Specific pricing for the enterprise tier is not published publicly.

Free trial

There is no time-limited trial for the enterprise tier described in the available public documentation. Teams can run the community edition indefinitely without a license.

Redpanda Console competitors and alternatives

Redpanda Console occupies the developer-friendly free-tier end of the Kafka UI market. The landscape includes tools that match it on open-source availability (AKHQ, Kafbat UI) and commercial tools that extend further into governance, observability, and multi-cluster management. The right choice depends heavily on whether you need production-grade access controls and whether you run a single cluster or many.

Tool / Best for	Type	Key functionalities	Deployment and ops	Access control	User interface	Pricing
Redpanda Console Redpanda and single-cluster Kafka browsing	OSS (BSL)	Message viewer, deserialization (Avro, Protobuf, JSON, XML, CBOR), Observer Mode, Kafka Connect, Schema Registry	Docker, Helm; hardcoded timeouts at scale; Kubernetes TLS friction; hard shutdown on license expiry	Enterprise-only RBAC and SSO	Modern, clean; React crashes on vanilla Kafka with some versions	Free community tier; enterprise license required for RBAC/SSO
AKHQ / Kafbat Lightweight self-hosted Kafka UI	OSS (Apache 2.0)	Topic management, consumer groups, Schema Registry, Kafka Connect	Docker, Helm; low resource usage	No built-in RBAC	Functional; less polished than Console	Free
Lenses / AxonOps Teams needing full Kafka observability with governance	Commercial	Governance, data masking, audit trails, metrics	SaaS and self-hosted	Enterprise RBAC and SSO	Feature-rich; steeper learning curve	Commercial; higher price point
Kpow (Factor House) Production Kafka management with enterprise RBAC	Commercial	Topic management, consumer groups, Schema Registry, Kafka Connect, multi-cluster, audit logging	Self-hosted; stateless; straightforward deployment	Advanced RBAC, SSO, data masking	WCAG-compliant; high-performance at scale	Per-cluster pricing that does not penalise team growth
Confluent Control Center Confluent Platform users	Commercial (bundled)	Deep Confluent Platform integration, KSQL, metrics, replication	Bundled with Confluent Platform	Enterprise RBAC	Dated UX by comparison	Included with Confluent Platform license

For a broader comparison of Kafka UI tools in 2026, see Top Kafka UI tools in 2026: a practical comparison for engineering teams.

Frequently asked questions about Redpanda Console

How much does Redpanda Console cost, and is there a free tier?

The community edition is free under the BSL for internal use. RBAC, SSO, and data masking require a paid Redpanda Enterprise license. Pricing for the enterprise tier is not published publicly.

When is Redpanda Console a better choice than the alternatives?

It is the strongest free option for single-cluster Kafka or Redpanda setups where the primary need is message inspection and developer debugging. For Redpanda users specifically, native admin API integration adds capabilities unavailable elsewhere without cost. The Observer Mode and multi-format deserialization pipeline — including Protobuf without a schema registry — are capabilities that few free tools match.

When are the alternatives a better choice than Redpanda Console?

When you manage more than one cluster, need RBAC or SSO without a Redpanda license, require built-in metrics and alerting, run large AWS MSK clusters with IAM authentication where Console’s hardcoded timeouts cause regular failures, or want a native desktop client for lightweight local inspection without running Docker.

Does Redpanda Console work with vanilla Apache Kafka?

Yes, but with limitations. Some features rely on the Redpanda admin API and fail silently or throw errors on vanilla Kafka. The help shortcut (?) causes a full React crash on certain Console versions when connected to a non-Redpanda cluster. Practitioners confirm the tool works well with standard Apache Kafka for message browsing and consumer group management, but the gaps become more visible as teams move beyond basic inspection.

Is Redpanda Console open source?

It is source-available under the BSL. Internal use is free. Commercial SaaS deployments require a commercial agreement. The original maintainer (weeco of CloudHut) confirmed after the 2022 acquisition that “99% of all users can still use it” under the BSL. The licensing transition has nonetheless driven a portion of the original Kowl community toward fully open-source alternatives.

Top Kafka UI Tools in 2026: A Practical Comparison for Engineering Teams

Factor House — Tue, 26 May 2026 00:00:00 GMT

Managing Apache Kafka through the command line made sense when clusters were small and teams were smaller. That era is over. Modern Kafka deployments span multiple clusters, process millions of messages per second, and serve dozens of teams who need visibility into topics they don’t own. The CLI simply cannot provide the observability, governance, and operational efficiency that production environments demand.

This guide evaluates the leading Kafka UI tools against the criteria that actually matter for enterprise data engineering teams. We cover the commercial platforms, the vendor-specific options, and the open-source alternatives, with an honest assessment of where each excels and where each falls short.

What to Look for in a Kafka UI

Before diving into specific tools, it’s worth establishing what separates a production-grade Kafka UI from a basic message viewer. The gap between these categories has widened significantly as Kafka has moved from simple pub/sub to the central nervous system of enterprise data architectures.

Kafka distribution support matters more than most teams initially realise. Your UI needs to work with your specific flavour of Kafka, whether that’s vanilla Apache Kafka, AWS MSK with IAM authentication, Confluent Cloud, Redpanda, or Aiven. A tool that works beautifully with self-managed Kafka but can’t authenticate against MSK IAM is useless for half of modern deployments.

Governance and security have become non-negotiable. SOC 2, HIPAA, GDPR, and internal compliance frameworks require granular access controls, audit trails, and data masking. A Kafka UI is effectively a window into your organisation’s data, and treating security as an afterthought is increasingly untenable.

Multi-cluster management separates enterprise tools from development toys. Most organisations run separate clusters for development, staging, and production, often across multiple cloud providers. Switching between browser tabs or reconfiguring connections is not a sustainable workflow.

Operational architecture determines your total cost of ownership. Does the tool require an external PostgreSQL database? Does it need gigabytes of heap memory? Can it run in air-gapped environments? These questions matter when your SRE team is already stretched thin.

Serialisation support is where many tools quietly fail. Kafka stores bytes; the intelligence is in the serialisation layer. Your UI needs to handle Avro, Protobuf, JSON Schema, and ideally custom serialisers for AWS Glue or proprietary formats. A tool that chokes on nested schemas or schema drift is useless in production.

Streaming ecosystem breadth is increasingly relevant as data platforms expand beyond Kafka. Teams running Kafka Streams, Kafka Connect, ksqlDB, or Apache Flink need tooling that provides visibility across their entire streaming infrastructure, not just the broker layer.

The Tools Compared

Kpow: Enterprise Governance Without the Infrastructure Tax

Kpow was purpose-built for a problem most Kafka UIs ignore: delivering enterprise-grade governance and observability without introducing operational complexity that rivals the clusters you’re trying to manage.

The answer is architectural. Kpow is fully stateless. It stores all state in internal Kafka topics using Kafka Streams, which means it deploys as a single container with zero external dependencies. No PostgreSQL to maintain. No separate data store to back up, patch, and monitor. No data ever leaves your network control. For regulated industries, this simplicity translates directly into faster procurement and fewer moving parts for your security team to evaluate.

This architectural simplicity belies serious depth. Kpow’s Data Policies provide server-side masking of sensitive fields based on key names or patterns, ensuring PII never reaches the browser. This satisfies PCI-DSS and HIPAA requirements without the complexity of a proxy layer sitting in front of your brokers. Comprehensive audit logging captures every action: who viewed what data, who reset which offset, who changed which configuration, with optional Slack integration for ChatOps transparency.

Where Kpow genuinely pulls ahead of every other tool on this list is Kafka distribution compatibility. It offers native AWS MSK IAM authentication (not a workaround via SASL, but actual IAM integration), Confluent Cloud, Redpanda, Aiven, and AWS Glue Schema Registry support, all from a single deployment. If you’re running a multi-cloud or hybrid Kafka environment, and increasingly most enterprises are, Kpow provides a unified view across every cluster and every distribution without per-instance configuration.

Kpow’s kJQ filtering deserves specific attention. It provides JQ-based predicates for searching deeply nested data structures server-side, which engineers at organisations like Binance and Cash App have cited as critical for reducing incident resolution time. When you’re debugging a production issue at 2am, the ability to write precise queries against complex message payloads without writing disposable consumer code is a material operational advantage.

Teams can get started with a 30-day free trial of Kpow. There is transparent per-cluster pricing rather than per-user models that penalise growing teams. A free Community Edition is available for local testing.

Best for: Any team where Kafka is production infrastructure that must be governed, audited, and operated reliably, particularly in regulated industries, multi-cluster environments, or organisations where operational simplicity and vendor independence are strategic priorities.

AKHQ: Capable Open-Source, but Mind the Governance Gaps

AKHQ (formerly KafkaHQ) is the most established open-source Kafka UI option. It’s built on Micronaut and designed for configuration-as-code deployments.

AKHQ’s strength is its GitOps-native architecture. Connections, users, groups, and schema registry links can all be defined in YAML, making it straightforward to deploy consistently across environments using Helm charts. It supports LDAP, OAuth2/OIDC, and GitHub SSO, with regex-based topic filtering for access control.

The limitations become apparent at enterprise scale. AKHQ has no native data masking, which is a significant compliance gap for any team handling PII or operating under HIPAA, PCI-DSS, or GDPR requirements. Audit logging is limited to whatever your authentication provider captures, rather than comprehensive activity tracking within the tool itself. The UI has also received persistent criticism for responsiveness issues under load, which the maintainer has acknowledged but not fully resolved. For development and staging environments these gaps may be acceptable; for production governance, they leave real exposure.

Best for: Mid-size organisations with strong DevOps cultures who need proven open-source tooling and can accept the governance limitations, or as a complement to a commercial tool for non-production environments.

Kafka UI (Kafbat fork): Modern but Fragile

A critical warning first: if you’re still running the original provectuslabs/kafka-ui Docker image, you’re running abandoned software with known security vulnerabilities. The project was effectively unmaintained from late 2023, with a remote code execution vulnerability (CVE-2023-52251) taking six months to patch. The core maintainers forked the project to kafbat/kafka-ui, which is where active development continues.

The Kafbat fork offers the most approachable open-source interface, with multi-cluster management, Kafka Connect integration, and Avro/Protobuf/JSON support. It includes basic RBAC via YAML configuration and data masking with regex support.

The fundamental trade-off is sustainability. This is a community-maintained fork of an abandoned project. There’s no vendor to call during an incident, no SLA, and long-term development direction depends entirely on volunteer contributor interest. The project inherited a significant bug backlog from Provectus, and while the Kafbat team has been responsive, the provectus/kafka-ui abandonment is a cautionary tale about relying on open-source projects without commercial backing for production infrastructure tooling.

Best for: Startups and development environments where budget is the primary constraint and the team has capacity to manage configuration, maintenance, and the risk of project abandonment.

Redpanda Console: Fast Viewer, Locked Governance

Originally built as Kowl by CloudHut before Redpanda’s acquisition, Redpanda Console performs well as a message viewer. Written in Go, it delivers minimal memory footprint and near-instant startup, which is a meaningful advantage for developers running local stacks. Its automatic deserialisation heuristics for Protobuf, Avro, MessagePack, and JSON are solid.

The catch is the licensing model. The core viewer is free under a Business Source License, but every enterprise feature that matters (SSO, RBAC, data masking) requires a paid Redpanda Enterprise license. This creates a problematic dynamic for vanilla Apache Kafka or MSK users: you can use the viewer for free, but the moment you need governance, you’re paying for a Redpanda license even if you don’t run Redpanda. Without those enterprise features, it’s a browser for messages, not an operational tool.

Multi-cluster support is also limited compared to dedicated multi-cluster solutions, and MSK IAM authentication requires SASL workarounds rather than native integration.

Best for: Local development and debugging where you need a fast, lightweight message viewer. Not a realistic option for production governance unless you’re already a Redpanda customer.

Confluent Control Center: Powerful but Captive

Confluent Control Center provides the deepest integration for Confluent Platform users. Kafka Streams topology visualisation, ksqlDB development, and Replicator monitoring are tightly coupled to the Confluent ecosystem in ways that third-party tools have not replicated. If you’re fully committed to the Confluent ecosystem, it provides native observability across the platform.

The critical limitation is that commitment must be total. Control Center requires the Confluent Metrics Reporter JAR installed in broker classpaths and effectively mandates the Confluent ecosystem for full functionality. It cannot work with AWS MSK’s native IAM authentication, immediately disqualifying it for the growing number of organisations using MSK. It also doesn’t support Redpanda or Aiven deployments.

The resource footprint is substantial, with some deployments requiring as much compute as the Kafka brokers themselves. And because Control Center is bundled with Confluent Platform licensing, you’re not evaluating the UI in isolation; you’re evaluating an entire ecosystem commitment.

Best for: Organisations already fully committed to Confluent Platform who specifically need Kafka Streams and ksqlDB visualisation. Not viable, and not intended, for vanilla Apache Kafka, AWS MSK, or multi-vendor environments.

Conduktor: Broad Feature Set, Operationally Heavy

Conduktor has evolved from a desktop application into a web platform targeting enterprise data quality and governance. Its headline feature is the Gateway proxy architecture, a Kafka proxy layer that enables field-level encryption, data masking, and policy enforcement at the wire level without modifying producer applications.
‍
The platform offers RBAC with wildcard patterns, compliance-ready audit logging with SIEM integration, and data quality validation rules that catch schema violations before bad data pollutes topics.
‍
The operational cost is significant and should be evaluated carefully. Conduktor Console requires an external PostgreSQL database, adding another stateful dependency to provision, back up, patch, monitor, and secure. This stands in stark contrast to stateless architectures that deploy as a single container. The licensing model includes per-user pricing and tiered feature access, which means the cost scales with team size in ways that per-cluster pricing does not. For large platform teams, this distinction can be material.
‍
Best for: Enterprises that specifically need wire-level proxy capabilities for encryption or policy enforcement, and have the operational capacity to manage the additional infrastructure dependencies.

Lenses.io: Ambitious Scope, Mixed Execution

Lenses positions itself as a DataOps platform rather than a pure Kafka UI, with a proprietary SQL engine that lets users query and transform streaming data using SQL syntax. The SQL interface may appeal to business analyst roles or less technical team members who need access to Kafka data without writing Java or Scala consumer code. SQL Processors allow deploying continuous transformations to Kubernetes, though this introduces a proprietary abstraction layer over your streaming infrastructure.

The data catalog and lineage tracking provide searchable topic discovery and data flow visualisation across multi-cluster environments.

However, Lenses occupies a different price point that reflects its broader DataOps positioning, which may be difficult to justify if your primary need is Kafka observability and governance. Some users report deployment complexity, particularly in air-gapped environments. The SQL abstraction introduces a proprietary layer that creates its own form of lock-in. And recent user feedback on G2 and community forums has flagged concerns about product roadmap velocity and unresolved bugs across releases, which are signals worth monitoring when evaluating long-term investment.

Best for: Organisations where SQL-based streaming data access is a primary requirement, particularly for business analysts or less technical roles who need self-service Kafka access. Evaluate carefully if your core need is operational governance rather than data exploration.

Platform Compatibility Matrix

Enterprise deployments typically span multiple Kafka flavours. This compatibility matrix reflects verified documentation as of early 2026:

Tool	AWS MSK (IAM)	Confluent Cloud	Redpanda	Self-Managed	Multi-Cluster
Kpow	Native	Full	Yes	Yes	Unified view
Conduktor	Deep integration	Deep integration	Yes	Yes	Per-instance
Lenses	Yes	Yes	Yes	Yes	Global catalog
AKHQ	IAM Auth	Via SASL	Yes	Yes	Yes
Kafka UI (Kafbat)	Glue SerDe	Yes	Yes	Yes	Yes
Redpanda Console	Via SASL	Via SASL	Native	Yes	Limited
Confluent CC	No native support	Native	No	Requires components	Yes

‍

The absence of native AWS MSK IAM support in Confluent Control Center is a significant limitation for the growing number of organisations using MSK as their primary Kafka deployment.

Kpow is the only tool on this list that provides native integration with every major Kafka distribution, including AWS MSK IAM, Confluent Cloud, Redpanda, Aiven, and AWS Glue Schema Registry, from a single, stateless deployment. For organisations running heterogeneous Kafka environments, this eliminates the need for distribution-specific workarounds or multiple tool instances.

Why We Built Kpow

Factor House builds tooling for streaming data platforms. We started with Kpow because we saw a clear gap: existing Kafka UIs either required complex infrastructure to deliver governance, or treated governance as an afterthought to keep things simple. Our thesis was that these shouldn’t be mutually exclusive, and the adoption by teams at organisations like Binance, Cash App, and NORD/LB has validated that approach.

The streaming ecosystem is expanding beyond Kafka. Teams now run Kafka alongside Kafka Connect, Kafka Streams, and increasingly Apache Flink. We believe tooling should evolve with this reality rather than remaining siloed. Factor House’s roadmap extends Kpow’s operational model (stateless deployment, transparent pricing, compliance-first design) across the streaming stack with products like Flex for Apache Flink and Factor Platform.

Kpow delivers is the deepest combination of Kafka governance, distribution compatibility, and operational simplicity available, without the infrastructure overhead, vendor lock-in, or per-user pricing that comes with the alternatives.

Choosing the Right Tool for Your Team

For AWS MSK-primary environments: Kpow provides native IAM authentication with zero workarounds, making it the cleanest MSK integration available. Conduktor also offers strong MSK support but requires PostgreSQL infrastructure. AKHQ is the best free alternative with MSK IAM support, though without data masking.

For Confluent Platform shops: Use Confluent Control Center if you specifically need Kafka Streams topology and ksqlDB visualisation. For broader observability, governance, and multi-cluster management, Kpow delivers comparable or better capabilities without the ecosystem lock-in or resource overhead.

For multi-cloud or hybrid deployments: Kpow is the clear choice, offering the only truly unified multi-cluster view across every major Kafka distribution with transparent per-cluster pricing. No other tool matches this breadth from a single deployment.

For compliance-heavy enterprises: Kpow’s stateless architecture means no external database storing sensitive metadata, server-side data masking ensures PII never reaches browsers, and comprehensive audit logging satisfies SOC 2 and HIPAA requirements. Conduktor’s Gateway adds wire-level encryption if you need proxy capabilities, but evaluate whether that complexity is justified for your use case.

For cost-conscious teams: AKHQ offers the most proven free option with broad enterprise adoption. Kafbat provides a more modern UI but carries project sustainability risk. Kpow’s Community Edition is free for local testing with a single cluster and full features, which is worth evaluating before committing to open-source maintenance overhead.

For development and testing: Kpow Community Edition is free for local testing with no feature limitations. Redpanda Console and Kafbat are also solid options for local development.

Conclusion

The Kafka UI you choose has real consequences for your team’s operational efficiency, compliance posture, and long-term flexibility. The right decision depends on matching your tool to your actual constraints: regulatory requirements, Kafka distribution, deployment complexity, and team capacity.

Open-source tools have matured. AKHQ and Kafbat are production-viable for many organisations, though both carry governance limitations that matter at enterprise scale. The provectus/kafka-ui abandonment is a useful reminder that project health matters as much as feature sets.

For enterprises where governance isn’t optional, the commercial landscape offers clear trade-offs. Conduktor adds proxy capabilities at the cost of infrastructure complexity. Lenses provides SQL abstraction at a DataOps price point. Confluent Control Center delivers deep platform integration but demands total ecosystem commitment. Kpow provides the broadest Kafka distribution support, the simplest operational footprint, and the deepest compliance capabilities, without requiring external databases, vendor lock-in, or per-user pricing.

If Kafka is critical infrastructure for your organisation, start with a free trial of Kpow and see the difference for yourself.

Kpow Community Edition 🚀

Derek Troy-West — Fri, 22 May 2026 00:00:00 GMT

Kpow Community Edition is a free, developer focused toolkit for Apache Kafka clusters, schema registries, and connect installations.

This may be the worst-kept secret in Kafka-tooling land because thousands of engineers are already using Kpow Community Edition (CE).

After a soft launch at Current ’22, then a medium launch at Kafka Summit ’23, we’re making Kpow CE generally available to everyone, everywhere.

Starting today, individuals can use Kpow CE for free, even at work. Organisations can install Kpow CE in up to three non-production environments.

Each installation of Kpow CE can manage one Kafka Cluster, one Schema Registry, and one Connect cluster. See our feature matrix for more information.

Kpow for Apache Kafka

Apache Kafka is a transformative technology, every engineer should have access to tooling that makes Kafka a joy.

Built on our learnings from a decade of shipping systems with Kafka, we believe Kpow is the most powerful and intuitive Kafka Web UI available to engineers today.

Designed for enterprise integration from our first commit in 2018, Kpow runs securely in your network with zero data egress and a full suite of user authentication and authorization features including SAML, LDAP, OpenID, OAuth2, Okta, Keycloak, RBAC, Multi-Tenancy, Data-Masking, Audit Logs, and more.

Five years, 10k commits, and more than 800k Docker pulls later, Kpow now has tens of thousands of users in 100+ countries. Our focus on quality tooling sees Kpow stacked with features and our commitment to performance means that one instance of Kpow can manage up to a dozen Kafka clusters and associated resources.

Explore the full feature-set of Kpow in our live, multi-cluster demo environment.

Kpow Community Edition

Kpow CE is the free, light-weight version of Kpow that is packed with features to accelerate your Kafka experience (with none of the enterprise-y stuff, you know the drill).

If we have to give you one reason to install Kpow? Here it is..

Blazing Fast Multi-Topic Search!

Searching for data in Kafka can feel like trying to find the needle in a haystack. Kpow CE makes topic search easy with built-in support for JSON Query (JQ) predicates, e.g:

.key.id | endswith("4d4a") and
.value.trade.status == "final" and
.value.trade.price | to-double > 25.45 and
.value.partner.network == "AMEX" and
.headers.flag[0] == "audit"

Kpow’s multi-topic search easily scans tens of thousands of messages a second, finding the ones that match your JQ predicate in a flash.

Read more about Kpow’s implementation of JQ in our kJQ Documentation.

Monitor, Control, and Explore

Kpow CE covers the full surface area of Apache Kafka, from creating topics and resetting consumer group offsets to editing schema and restarting connectors.

Watch the Devoxx UK conference talk ‘Tune-Up your Kafka Tooling with Kpow Community’ and discover how to:

Set up Kpow CE with a Kafka Cluster, Schema Registry, and Kafka Connect. [3:00]
Skip the setup wizard and run Kpow CE with environment variables. [4:19]
Learn how Kpow generates unique metrics and insights while running airgapped and secure. [6:45]
Diagnose the root cause of production lag issues with multi-dimensional consumer metrics. [11:30]
Skip poison-pill messages and recompute topics with consumer offset management. [13:30]
Manage and monitor Kafka Connect clusters and schema registries. [15:22]
Find needle-in-a-haystack messages with blazing-fast multi-topic search and built in JQ predicates. [16:46]
Blend Clojure and JQ to make quick assertions about topic data with the kREPL. [22:18]

Tune-Up your Kafka Tooling!

Kpow CE works with any Kafka cluster v1.0.0+ including Confluent Cloud & Platform, AWS MSK, MSK Serverless, Redpanda, and clusters from providers like Aiven and Instaclustr.

Supported Kafka resource integrations include Confluent Schema Registry, AWS Glue, Apache Kafka Connect, Confluent Managed Connect, and MSK Connect.

Get started in minutes with the Kpow CE Docker container, our multi-arch build provides support for both ARM and x86.

Configure and launch Kpow CE:

Access Kpow CE setup wizard on http://localhost:3000
Follow the steps to get an individual or organization license.
Configure your Kafka Cluster, Schema Registry, Connect Cluster.
Launch Kpow CE!

See the configuration guide to learn how to configure Kpow CE with environment variables instead of using the setup wizard.

Running in Kubernetes? Use our Helm charts to deploy Kpow CE to your cluster.

We hope you find Kpow Community Edition useful. If you encounter any problems or techincal questions just raise an issue on the Kpow Github repository.

Interested in a commercial license? Contact us any time to discuss requirements and start a POC.

Derek Troy-West is a Co-Founder and CEO of Factor House.

Factor House build essential tools for modern engineers.

Best practices for Kafka data observability

Factor House — Mon, 18 May 2026 00:00:00 GMT

Key takeaways

Kafka data observability is distinct from broker monitoring. Broker metrics tell you the cluster is healthy; data observability tells you whether the data flowing through it is correct, fresh, complete, and structurally sound.
Consumer lag is your north-star metric, but raw offset thresholds are an unreliable way to measure it. Trend-based evaluation per consumer group, per partition, is more accurate.
Schema Registry with enforced compatibility in CI is non-negotiable above a handful of services. Schema drift causes silent consumer failures that broker dashboards will never surface.
End-to-end auditing, counting messages at every tier from producer to consumer, catches data loss that infrastructure monitoring alone cannot detect.
Tools like Kpow consolidate broker health, consumer lag, schema management, and Kafka Streams visibility into a single interface, reducing the operational overhead of maintaining separate observability tooling.

Introduction

Kafka’s operational surface area has two distinct layers. The first is the cluster itself: broker health, partition replication, JVM pressure, disk utilisation. The second is the data: whether messages are arriving on time, whether the schema is what consumers expect, whether any messages were lost in transit.

Most teams instrument the first layer thoroughly and underinvest in the second. The consequence is a pattern that practitioners have converged on calling “green dashboards, broken data.” The cluster looks healthy. CPU is normal. ISR is stable. And yet, somewhere downstream, a consumer is silently deserialising nulls because a field was renamed three hours ago.

PagerDuty’s August 2025 outage is the most thoroughly documented recent example of this failure mode. Brokers looked fine throughout the incident, but a producer-instance leak in a pekko-connectors-kafka integration caused approximately 4.2 million new producers to register per hour, roughly 84 times the normal rate. Brokers exhausted JVM heap tracking producer metadata, the cluster cascaded, and approximately 95% of incoming events were rejected over 38 minutes. PagerDuty’s own post-mortem explicitly names two causes: “previously minimal alerting on Kafka” and an “observability gap in Kafka producer and consumer telemetry including anomaly detection for unexpected workloads.”

Infrastructure monitoring alone could not have caught it. This guide covers the practices that close that gap.

What Kafka data observability actually means

Data observability, as applied to Kafka, asks five questions about data in motion:

Pillar	Question	Kafka-specific signals
Freshness	Is data arriving on time?	Consumer lag per group per partition, end-to-end latency from event-time to consume-time, watermark delay in Flink/Kafka Streams
Volume	Is the expected amount arriving?	MessagesInPerSec per topic, producer record counts, audit-tier message counts
Schema	Has the structure changed unexpectedly?	Schema Registry version count, compatibility-check failure rate, DLQ rate from deserialization errors
Distribution	Are field values within expected ranges?	Field-level rules in data contracts, null-rate spikes
Lineage	Where did this data come from, and where is it going?	Producer-to-topic-to-consumer graph, Kafka Connect transform-aware lineage, OpenLineage events

‍

The distinction from cluster monitoring is important. Monitoring tells you consumer lag spiked at 3:15 PM. Observability tells you consumer lag spiked at 3:15 PM, correlating with schema version 47 deployed at 3:12 PM, which introduced a backward-incompatible field removal causing consumer deserialization failures. The shift is from cluster-centric JMX metrics to data-centric signals.

12 best practices for Kafka data observability

1. Use trend-based consumer lag monitoring, not threshold alerts

Raw offset thresholds are a losing proposition for consumer lag alerting. A traffic spike will trip a fixed threshold even when the consumer is keeping up. Aggregating lag across partitions hides per-partition stalls, meaning one stuck partition out of ten will disappear into the average.

LinkedIn’s SRE team built Burrow specifically to address this. Rather than alerting on absolute lag values, Burrow evaluates each consumer group over a sliding window of committed offsets and classifies it as OK, WARN, or ERROR based on lag trend. Uber extended the same idea with uGroup, which decodes the __consumer_offsets topic directly because consumer-side metrics cannot fully account for all group activity, particularly when consumers are experiencing problems.

The practical implementation is to scrape Burrow’s REST API into Prometheus and alert on ERROR state persisting for five or more minutes. Conduktor reported going from 47 lag alerts per month (2 real) to 3 alerts per month (all real) after switching from threshold-based to rate-of-change alerting.

For more on this topic, see our consumer lag monitoring guide.

2. Track consumer lag as both a count and a time

Offset lag and time lag measure different things. “A consumer is 50,000 messages behind” depends entirely on the traffic shape of that topic. “A fraud-detection consumer is 30 seconds behind” is immediately actionable.

Most major platforms expose both. Datadog Data Streams Monitoring, the Confluent Metrics API, and AWS MSK CloudWatch all provide both OffsetLag and EstimatedTimeLag. Time-based lag is the SLO metric that matters to the business. Define your SLOs in time units, alert in time units.

A useful PromQL pattern for lag alerting that accounts for both lag and lack of progress:

max by(group, topic)(kafka_consumer_group_partition_lag) > 10000 and rate(kafka_consumer_fetch_manager_records_consumed_total[5m]) == 0

This fires on high lag combined with zero consumption progress, which filters out deployment blips and traffic spikes where the consumer is genuinely keeping up.

3. Enforce schema compatibility in CI before any producer deploys

Above roughly five services producing to Kafka, Schema Registry with enforced compatibility transitions from a nice-to-have to a requirement. The failure mode without it is well-documented: a producer deploys a new schema version that removes a field, consumers expect that field and crash when it is missing, messages accumulate unprocessed, lag grows into the millions, and alerts fire hours after the schema change that caused it.

The defaults that most practitioners converge on: BACKWARD for Avro and JSON Schema, BACKWARD_TRANSITIVE for Protobuf. The distinction matters for Protobuf because adding new message types is not forward-compatible, so transitive checking across all previous versions is required.

The enforcement point is at least as important as the compatibility mode. Community Schema Registry enforces only in the producer SDK, which any client speaking the Kafka wire protocol directly can bypass. Gate schema changes in CI with mvn schema-registry:test-compatibility or the Gradle equivalent, and fail the build on incompatibility. Disable auto.register.schemas in production so rogue producers cannot silently introduce new schemas.

One important operational note: Confluent Schema Registry defaults to BACKWARD, but AWS Glue Schema Registry defaults to DISABLED and Apicurio defaults to NONE. Always verify and set compatibility modes explicitly rather than relying on defaults.

4. Go beyond schema validation with data contracts

Schema compatibility validates structure. Data contracts extend that to semantics and SLAs.

The Open Data Contract Standard (ODCS), originally open-sourced by PayPal and now maintained under the Bitol project, extends schemas with data-quality rules, SLAs, security classifications, and ownership metadata. Confluent’s Data Contracts feature implements field-level validation using Common Expression Language (CEL) rules at serialization time, including validators like isEmail and isUuid, field transformations, and DLQ routing on rule failure.

The boundary on enforcement is worth understanding clearly. Community Schema Registry validates client-side only. Anything that speaks the Kafka wire protocol directly sidesteps the entire enforcement chain. Broker-side validation is available as a paid Confluent feature; Redpanda’s Wasm Data Transforms provide broker-side validation in open source. If you cannot use broker-side enforcement, supplement contract validation with audit-tier message counting (see practice 7).

5. Implement dead letter queues with rich metadata and active monitoring

A dead letter queue with no observability is a silent failure sink. Messages arrive, nobody sees them, and what should have been a detectable error becomes an invisible data hole.

The pattern that most teams have converged on, originally published by Uber Insurance Engineering: a tiered retry structure with a main topic, a series of count-based retry topics with exponential backoff (topic.retry.1 at one minute, topic.retry.2 at five minutes), and a final DLQ for manual review. Each DLQ message should carry headers recording the original topic, partition, offset, exception class, attempt count, and stack trace.

Kafka Connect has native DLQ support since version 2.0. Kafka Streams requires custom DeserializationExceptionHandler and ProductionExceptionHandler implementations.

The monitoring rules: alert on DLQ message rate above 10 messages per second for five minutes as a warning, and DLQ backlog above 1,000 messages for 15 minutes as critical. Feed DLQ message rates into ksqlDB to drive per-sink breach alerts.

6. Track end-to-end latency with OpenTelemetry trace context in message headers

Broker and consumer metrics cannot tell you how long a specific message spent in a topic waiting to be consumed. OpenTelemetry’s messaging semantic conventions solve this at the per-message level.

Producers inject a W3C traceparent header into each Kafka message. Consumers extract it and create a child span linked to the producer span. The gap between the producer span ending and the consumer span starting is the topic dwell time for that message. Relevant span attributes per the OpenTelemetry messaging semantic conventions include messaging.system=kafka, messaging.destination.name, messaging.kafka.partition, messaging.kafka.message.offset, and messaging.message.id.

For high-throughput topics, use tail-based sampling. Keep all error spans and all spans exceeding p99 latency; sample the rest. Strimzi shipped OpenTelemetry support for Kafka Connect, MirrorMaker 2, and the Kafka Bridge in 2023.

A practical note: OpenTelemetry messaging semantic conventions are still evolving as of mid-2026. Some span attributes including messaging.kafka.consumer.group and messaging.message.body.size are stable; others remain experimental. Pin your instrumentation library versions in production.

7. Build end-to-end auditing for true data-loss detection

This is the practice that separates mature streaming organisations from those that discover data loss during incident response. Broker metrics, consumer lag, and distributed traces do not tell you whether a message was dropped in transit between tiers. End-to-end auditing does.

The pattern, implemented by Uber as Chaperone and Netflix as Inca: every tier (proxy, broker, mirror, aggregate, consumer) emits an audit message to a dedicated audit topic with a tuple of message ID, tier name, count, and time window. A stream-processing job (Flink at Netflix, custom at Uber) keys by message ID, fans across tiers, and emits a missing-trace signal when an expected tier did not report within the time window.

Netflix’s Inca uses a Flink GlobalWindow with a custom trigger and reduces state via consumer offsets. It produces loss-rate, duplicate-rate, and end-to-end latency as SLO metrics, plus the IDs of lost messages to a Kafka topic for automated re-fetch.

Uber’s published example of why this matters: a dead-loop bug in uReplicator caused silent data loss. Neither uReplicator nor the Kafka brokers triggered any alerts. Only the end-to-end message counting job detected the loss. A pure broker-and-replicator-metric observability stack would never have surfaced it.

This level of investment is appropriate once you have more than 100 topics or more than 10 teams producing to Kafka. For earlier-stage deployments, the priority is practices 1 through 4.

8. Enforce topic naming conventions through CI

Topic naming is cheap to get right and expensive to get wrong. Kafka topics cannot be renamed after creation, so naming decisions made during initial development persist indefinitely.

The community-converged convention is hierarchical:

`....-by--v

Example

prod.public.fct.payments.payment_completed-by-order_id-v2`

Hard rules: lowercase only, choose either . or _ as the separator and use it consistently (mixing both creates collisions on JMX metric names), exclude any fields that might change over time such as team name, service name, or owner, and reserve the version suffix for backward-incompatible breaks only.

Disable auto.create.topics.enable on all production clusters. Provision topics through GitOps using Strimzi KafkaTopic custom resources, Confluent for Kubernetes, or an in-house provisioning API. Validate names against a regex in a pre-commit hook. DoorDash replaced its Terraform-based topic-creation flow with an in-house API and reduced real-time pipeline onboarding time by 95%.

9. Set retention and compaction policies with observability in mind

Compacted topics (cleanup.policy=compact) only retain the most recent value per key. Intermediate values may be deleted by compaction before a consumer reads them. This breaks audit-tier counting and any observability approach that relies on replaying message history.

Reserve compaction for state changelogs, which is what Kafka Streams uses internally. Use cleanup.policy=delete for event topics, with retention.ms set to your maximum replay window. If you need long retention without the cost of persistent SSD storage, tiered storage (available in Confluent Cloud and as an open-source implementation from Pinterest via KIP-405) lets you extend retention to object storage.

Track LogSegmentBytes per topic and alert on disk-pressure-driven retention truncation, which silently shortens your replay window without any explicit configuration change.

10. Add heartbeats and offset-translation verification for multi-cluster setups

In multi-cluster deployments, MirrorMaker 2’s replication of data is generally reliable. The harder problems are detecting replication lag, verifying offset translation, and coordinating client failover.

MirrorMaker 2’s MirrorHeartbeatConnector produces a heartbeat on every source cluster every five seconds. Alert if the heartbeat does not appear in the destination within replication_factor × interval. MirrorCheckpointConnector translates consumer offsets across clusters; verify by spot-checking that sync.group.offsets.enabled=true and that a test consumer can fail over cleanly.

Every published Kafka DR retrospective converges on the same lesson: replication is the solved part. Coordinating client failover during an actual incident, while also managing schema-registry consistency and consumer-group state across clusters, is where DR plans actually fail. Test your DR plan with current tooling and find the coordination gaps before investing in additional cross-cluster observability tooling.

11. Alert on SLOs and symptoms, not on infrastructure causes

URP (Under-Replicated Partitions) is the most commonly over-alerted Kafka metric. Todd Palino, who led SRE for Kafka at LinkedIn, has explicitly stated that URP does not map to an SLO and is often not actionable; it should be collected for forensics but should not page anyone.

The metrics that should page are those that map directly to user-visible impact: under-min-ISR partitions (genuine data-loss risk), consumer-group ERROR state from trend-based lag monitoring, DLQ rate breach, schema compatibility failure rate, and audit-tier loss-rate breach.

The practical approach:

Define SLOs first: “p99 end-to-end latency under 5 seconds for the payments topic” or “fraud-detection consumer lag under 30 seconds.”
Alert on SLO burn rate using multi-window burn-rate alerting.
Demote URP, ISR shrink, broker CPU, and request-handler idle ratio to dashboards available for forensics, not to PagerDuty.

Aggregate consumer group lag with max() per partition, not avg(). One stalled partition out of ten will be invisible in the average.

12. Treat data lineage as a runtime artifact

Static documentation of data lineage becomes inaccurate as soon as a pipeline changes. Runtime lineage, emitted as events as jobs execute, stays current.

OpenLineage’s specification uses Job, Run, and Dataset entities with Facets for schema, quality, and other metadata. It emits events natively from Airflow, dbt, Spark, and Flink to Marquez (the reference implementation) or DataHub (originally LinkedIn, now maintained by Acryl Data). DataHub treats Kafka topics as first-class datasets, parses Kafka Connect SMT configurations including RegexRouter and EventRouter to resolve the actual destination topic, and supports column-level lineage where schema-registry data is available.

Emit OpenLineage events from every job that touches Kafka. The producer-to-topic-to-consumer graph becomes queryable, auditable, and accurate in near-real time rather than a diagram that someone last updated six months ago.

Tooling stack

No single tool covers the full observability surface, but a comprehensive Kafka management platform like Kpow by Factor House gets close, combining broker monitoring, consumer lag, schema registry, DLQ management, and Kafka UI and management in one place. For teams that need additional capabilities, the ecosystem includes purpose-built tools across the following categories:

Category	Open source options	Commercial options
Broker/cluster monitoring	Prometheus + JMX Exporter, Kafka Exporter, Cruise Control, Strimzi	Kpow (Factor House), Confluent Control Center, Datadog, New Relic, Dynatrace
Consumer lag monitoring	Burrow, Kafka Lag Exporter	Kpow (Factor House), Confluent Metrics API, Datadog, AWS MSK CloudWatch
Schema registry	Apicurio, Karapace, Confluent Schema Registry (community)	Kpow (Factor House), Confluent Stream Governance, AWS Glue
Data contracts	ODCS, data-contract-cli	Confluent Data Contracts, Atlan, Soda Cloud
Distributed tracing	OpenTelemetry Collector + Jaeger/Tempo/SigNoz	Honeycomb, Datadog APM, Dynatrace
DLQ management	Kafka Connect native DLQ, Karafka	Kpow (Factor House), Confluent Control Center
Lineage and metadata	OpenLineage + Marquez, DataHub, Apache Amundsen	Factor Platform (Factor House), Acryl Data, Atlan, Collibra, Monte Carlo
Kafka UI and management	AKHQ, Kafka UI, Redpanda Console	Kpow (Factor House), Conduktor, Lenses.io
Multi-cluster replication	MirrorMaker 2, uReplicator	Confluent Replicator, Confluent Cluster Linking

‍

A reasonable starting point for a team with no existing investment: Prometheus + JMX Exporter + Kafka Exporter for cluster metrics, Burrow for consumer lag, Apicurio or Confluent Schema Registry community edition for schemas, OpenTelemetry Collector with Jaeger or Tempo for tracing, and OpenLineage with Marquez or DataHub for lineage. Add a Kafka UI for ad-hoc inspection and correlating data-level signals with cluster state. This broadly matches Cloudflare’s observability stack and the direction Shopify moved toward when consolidating off third-party tooling.

For teams evaluating their options more broadly, see our guide to the best Kafka monitoring tools.

How Kpow helps with Kafka data observability

Kpow by Factor House covers several of the observability concerns described in this article within a single deployable tool.

Consumer lag visibility. Kpow surfaces consumer group lag at the group, broker, topic, and partition level. It handles both active groups and EMPTY consumer groups, calculating lag for empty groups directly from start and end offsets via the AdminClient rather than relying on a cached snapshot. This matters in scenarios where a poison message has caused all instances of a consumer group to go offline; Kpow can still read the offsets and allow you to reset them without requiring the group to be running. Kpow also identifies simple consumers (those using manual partition assignment without group coordination), which appear in their own tab and are common in some Flink and Spark deployments.

Schema Registry management. Kpow integrates with Schema Registry, allowing you to inspect schema versions, view compatibility settings, and manage schemas directly from the UI. This is useful when investigating deserialization errors or verifying that compatibility modes are set correctly across environments.

Prometheus egress for alerting integration. Kpow exposes Prometheus endpoints following the OpenMetrics standard, making it straightforward to pipe Kafka metrics into your existing Grafana dashboards or AlertManager setup. Available endpoints include /metrics/v1 for all cluster metrics, /group-offsets/v1 for per-assignment group offset data, /offsets/v1 for topic partition offsets, and /streams/v1 for Kafka Streams metrics from connected agents. The group offset endpoint exposes group_assignment_delta, group_assignment_last_read, and group_assignment_offset at the partition assignment level, which gives you the granularity needed for accurate lag alerting.

Kafka Streams observability. Through the kpow-streams-agent, Kpow collects Kafka Streams application metrics and exposes them via the /streams/v1 and /streams/v1/state Prometheus endpoints. This provides visibility into Kafka Streams topology state alongside broker and consumer metrics.

Kafka Connect management. Kpow allows you to manage Kafka Connect connectors from the same interface, which reduces the context-switching involved when investigating pipeline issues that span brokers, consumers, and connectors.

Kpow connects to any Kafka cluster and deploys via Docker, Helm, or JAR. If you want to evaluate it against your current observability setup, you can try Kpow free for 30 days.

Implementation roadmap

The practices in this guide are not all equal in complexity or return on investment. A phased approach:

Within 30 days (foundation):

Deploy trend-based consumer lag monitoring (Burrow or equivalent) and replace any threshold-based lag alerts with ERROR/WARN classification per consumer group.
Stand up Schema Registry with compatibility set explicitly: BACKWARD for Avro/JSON Schema, BACKWARD_TRANSITIVE for Protobuf. Disable auto.register.schemas in production. Add compatibility checking to CI as a hard gate.
Disable auto.create.topics.enable on all production clusters. Move topic provisioning into GitOps.
Audit your current alerts. Demote URP, ISR shrink, and broker CPU to dashboards. Keep paging for under-min-ISR partitions, consumer-group ERROR, DLQ rate breach, and schema compatibility failure rate.

Within 90 days (data-level observability):

Instrument all producers and consumers with OpenTelemetry. Ensure trace context propagates in message headers. Export to Jaeger or Tempo with tail-based sampling.
Deploy tiered DLQ patterns for any consumer where message loss is unacceptable. Instrument DLQs with rich headers and alert on rate and backlog.
Define an SLO per business-critical topic. Track burn rate and alert on multi-window burn.
Emit OpenLineage events from any Kafka Connect, Flink, Spark, or Airflow job touching Kafka. Ingest into Marquez or DataHub.

Within 6 months (mature pattern, applicable above 100 topics or 10 producing teams):

Build or adopt end-to-end auditing: count messages at each tier and use a Flink or ksqlDB job to detect loss and duplication per message ID.
Adopt data contracts beyond schemas: field-level rules, SLAs, and ownership. Choose between Confluent Data Contracts (paid, broker-side enforcement), ODCS with Redpanda Data Transforms (open source, broker-side), or producer-side CI validation.
Consolidate dashboards so broker health, consumer lag, schema events, DLQ rates, and end-to-end traces are visible in a single view.

If you are experiencing more than three production incidents per quarter caused by schema changes, accelerate the schema enforcement work. If your mean time to detection for streaming pipeline incidents exceeds five minutes, end-to-end tracing and SLO-based alerting are the highest-leverage next steps.

Kafka message size best practice

Factor House — Mon, 18 May 2026 00:00:00 GMT

Every Kafka cluster operates under a set of message-size constraints, and getting them wrong costs you in ways that are not always immediately obvious: silent replication stalls, infinite producer retry loops, page-cache eviction cascades, and hard architectural ceilings imposed by cloud providers. This guide covers the sizing targets that production operators at scale converge on, how the configuration chain works, and what to do when your use case genuinely requires large messages.

Best practice for Kafka message size

The production consensus, backed by what LinkedIn, Cloudflare, Netflix, and Uber have published about their own clusters, is to keep individual messages well under 1 MB. The Apache Kafka broker default of message.max.bytes = 1,048,588 bytes (1 MiB plus 12 bytes of overhead, aligned under KAFKA-4203 with the Java producer’s max.request.size = 1,048,576) is the design point Kafka is optimised for. It is not a ceiling to design towards.

LinkedIn, which runs more than 100 clusters, 4,000 brokers, and 7 trillion messages per day, states this policy explicitly in their engineering blog: messages are capped at 1 MB, and anything that exceeds that is handled outside the standard message path using client-side fragmentation.

The practical sweet spot for most production workloads is significantly smaller. Public Kafka benchmarks from Aiven, Google Cloud, and LinkedIn typically use 100 bytes to 10 KB as representative message sizes. Confluent’s canonical performance benchmark uses 1 KB messages. These numbers are not arbitrary: LinkedIn’s 2014 benchmark noted that at 100-byte messages you saturate the network, making small records the harder case for a messaging system to optimise.

The 1 KB to 10 KB range is where Kafka’s batching, compression, and zero-copy I/O work most efficiently together. Messages in this range can be batched densely, compressed effectively at the batch level, and served from page cache without displacing hot data.

Kafka message size limits

Broker defaults

The broker’s message.max.bytes defaults to 1,048,588 bytes. This is the maximum size of a compressed record batch that the broker will accept. At the topic level, max.message.bytes overrides this per topic; the effective limit is whichever value is higher.

Cloud provider ceilings

If you are running on a managed Kafka service, the provider’s hard limits constrain what you can configure:

Provider	Ceiling
Apache Kafka (self-managed)	1 MB default; configurable upward
Confluent Cloud Basic / Standard	8 MB (8,388,608 bytes)
Confluent Cloud Dedicated	20 MB (20,971,520 bytes)
AWS MSK Serverless	8 MB; not user-configurable without a support case
Azure Event Hubs (Kafka API)	1 MB; no override

‍

These ceilings are architectural constraints, not soft guidelines. A design that relies on messages larger than 1 MB is incompatible with Azure Event Hubs. A design that needs more than 8 MB is incompatible with both Confluent Cloud Basic/Standard and AWS MSK Serverless. Validate these limits against your managed service tier before settling on an approach.

Sizing tiers

The guidance from Confluent, Red Hat, Strimzi, and operator engineering blogs converges on five tiers:

Tier	Size range	Recommended approach
Ideal	Less than 10 KB (typically 100 B – 10 KB)	Default Kafka tuning. Batching and compression dominate. The benchmark sweet spot for most production workloads.
Acceptable	10 KB – 100 KB	No configuration changes required. Tune batch.size to 64–128 KB and linger.ms to 10–20 ms to keep batches efficient.
Handle with care	100 KB – 1 MB	Individual messages can saturate per-partition batches at default config. Increase batch.size to 256 KB or higher and ensure compression is enabled. Messages larger than batch.size are sent unbatched.
Avoid unless necessary	1 MB – 10 MB	Requires raising all four configs in the producer/broker/consumer chain. Confluent’s own documentation describes very large messages as an anti-pattern in Kafka. Run these workloads on a dedicated cluster if other topics have low-latency SLAs.
Do not use inline	Over 10 MB	Use the claim-check pattern (store payload in S3 or GCS; put a pointer in Kafka) or client-side chunking. Allocating a 1 GB JVM chunk per 1 GB message on both the client and broker is not operationally viable.

How to configure larger messages

If the claim-check pattern is not viable and you need to raise the effective message size limit, you must update four interdependent configurations in concert. Missing any one of them produces failures that range from immediately visible to silently catastrophic.

The chain is:

producer max.request.size → topic max.message.bytes / broker message.max.bytes → broker replica.fetch.max.bytes → consumer max.partition.fetch.bytes

Configure them in this order to avoid silent failures:

1. Broker (requires restart for replica.fetch.max.bytes)

message.max.bytes=10485760 replica.fetch.max.bytes=10485760 socket.request.max.bytes=104857600

2. Topic (can be applied without a broker restart)

max.message.bytes=10485760

3. Producer

max.request.size=10485760 buffer.memory=67108864

4. Consumer

fetch.max.bytes=52428800 max.partition.fetch.bytes=10485760

Keep these values equal or monotonically non-decreasing along the chain.

What happens when the chain breaks

The failure mode depends on which link is misconfigured:

Producer over max.request.size: a synchronous RecordTooLargeException is thrown. The message never reaches the broker. This is the cleanest failure.
Producer under max.request.size but over broker message.max.bytes: the broker returns MESSAGE_TOO_LARGE. Modern Java producers auto-split the batch and retry. If batch.size itself exceeds message.max.bytes, you enter an infinite split-and-retry loop (KAFKA-8350). Always validate that batch.size does not exceed message.max.bytes.
replica.fetch.max.bytes smaller than message.max.bytes: historically this caused silent replication stalls, ISR shrinkage, and data loss under unclean leader election. Modern Kafka guarantees forward progress by always returning the first record batch, but replication latency degrades and a permanent replica-lag floor is introduced.
Consumer max.partition.fetch.bytes too small: the same forward-progress guarantee applies, but a consumer group can fall permanently behind on partitions containing large records.

Kafka Connect and MirrorMaker 2

For Kafka Connect and MirrorMaker 2, apply the large-message overrides via the producer.override.* and consumer.* prefixes in the connector configuration. The connector-level settings do not inherit from broker defaults automatically.

How to handle very small messages

The performance problem with small messages is a batching problem, not a size problem. Kafka compression operates at the batch level, not the message level. A compressor needs a sufficiently large sample of data to find repeated patterns; on a single 100-byte message, compression saves under 5% while still incurring the full CPU cost.

Apache Kafka 4.0 (released March 2025) changed linger.ms from 0 to 5 ms as a direct acknowledgement that the previous default was actively damaging batching efficiency. If you are on Kafka earlier than 4.0, set linger.ms explicitly. Do not rely on the zero default.

The standard production tuning for small-to-medium messages:

`# Producer
batch.size=131072 # 128 KB; default 16 KB is undersized for high-volume workloads
linger.ms=20 # or 10–100 depending on latency tolerance
compression.type=lz4 # or zstd if storage cost is the bottleneck
buffer.memory=67108864 # 64 MB

Topic

compression.type=producer # broker preserves whatever the producer used
min.insync.replicas=2`

To illustrate the leverage: Confluent’s throughput tuning benchmark shows that with default producer config, 8 KB records produce 23.58 MB/s at 927 ms average latency. With batch.size=200000, linger.ms=100, compression.type=lz4, and acks=1, the same workload achieves 94.89 MB/s at 4.92 ms average latency. That is a 4x throughput increase from producer configuration changes alone.

The JMX metric bufferpool-wait-ratio is a direct signal of buffer pressure. Values above 0.05 indicate that the producer is blocking on buffer allocation; increase buffer.memory to 64–256 MB if this occurs.

Key Kafka message configurations

`message.max.bytes` (broker)

Default: 1,048,588 bytes. The largest compressed record batch the broker accepts. Raising this without also raising replica.fetch.max.bytes is the most common source of silent replication degradation in misconfigured clusters.

`max.message.bytes` (topic)

Per-topic override of message.max.bytes. The effective limit for a given partition is the higher of the topic-level and broker-level values. Setting this per topic rather than at the broker level is the safer approach: it limits exposure and allows you to apply large-message config only where it is required.

`max.request.size` (producer)

Default: 1,048,576 bytes. The maximum size of a single produce request (the entire batch, post-compression). This must be raised before raising broker limits, or the producer will fail at the client before the broker ever sees the message.

`replica.fetch.max.bytes` (broker, follower)

Default: 1,048,576 bytes. Controls how many bytes a follower fetches per partition per request. This is the configuration most commonly forgotten when raising message size limits, and historically the most dangerous to misconfigure. Per Cloudera’s documentation: a broker can accept messages it cannot replicate if this value is smaller than message.max.bytes, which creates a data loss risk. Always set this equal to or larger than message.max.bytes.

`batch.size` (producer)

Default: 16,384 bytes (16 KB). Records larger than this value are sent as their own batch, unbatched. For workloads with messages in the 10–100 KB range, the default batch.size becomes the binding constraint on batching efficiency. Set to 64–256 KB for high-volume producers.

`linger.ms` (producer)

Default: 5 ms (Kafka 4.0+); 0 ms (pre-4.0). The maximum time the producer waits to fill a batch before sending. At 0 ms in low-to-medium throughput environments, most batches are single-message, which eliminates most of the benefit of compression.

`compression.type` (producer / topic)

Default: none at the producer, producer at the topic level (which means the broker preserves whatever the producer sent). At the topic level, you can override to a specific codec, but this forces broker-side recompression if producers and topics diverge. Setting the topic to producer and controlling compression at the producer is the lower-overhead approach.

Tips for achieving better performance

Use LZ4 as the default compression codec, switch to Zstandard when storage is the bottleneck

LZ4 offers the best throughput-per-CPU ratio for latency-sensitive workloads: compress speeds around 594 MB/s and decompress speeds around 2,428 MB/s on modern hardware. Zstandard at level 1 produces better compression ratios than LZ4 with acceptable throughput. At level 3 (the default), Zstd compresses around 242 MB/s but achieves ratios of roughly 24% of the original payload size for typical Kafka data. The KIP-390 measurements show that moving from Zstd level 3 to level 1 produces 32.7% more messages per second.

Cloudflare’s documented switch from no compression to Snappy to Zstandard reduced their topic size by 4.5x, freeing them from a pending hardware expansion. Their data was highly repetitive HTTP log payloads; your compression ratio will depend on your payload structure.

Gzip is the slowest codec in all benchmarks, with throughput around 830 msg/s in the HUMAN Security benchmark versus 3,400 for LZ4 and Snappy. The bottleneck is not CPU but lock contention in the gzip JNI binding. Prefer LZ4 or Zstd for all new workloads.

Switch from JSON to Avro or Protobuf

JSON is materially larger than its binary equivalents for the same logical record. A typical Avro or Protobuf payload is 30–50% the size of equivalent JSON. Cloudflare moved from JSON to Protobuf specifically because JSON made forward and backward compatibility harder to enforce and produced substantially larger messages.

Using Avro or Protobuf with a schema registry also reduces the per-message overhead: Confluent Schema Registry stores a 4-byte schema ID plus 1 magic byte per message, with the schema itself fetched once per consumer session and cached.

Keep heap small; let the OS manage page cache

Kafka brokers serve data primarily from Linux page cache via sendfile(), which bypasses the JVM heap entirely for message bodies. The JVM heap is used for request handling, metadata management, and format down-conversion. On a 64 GB broker, set JVM heap to 6–10 GB and let the operating system use the remainder as page cache. Setting vm.swappiness=1 prevents the kernel from swapping page cache to disk under memory pressure.

Large messages are particularly damaging in this model: a single large message can evict thousands of small hot records from page cache, converting subsequent reads from cache hits to physical disk reads.

Run large-message workloads on a dedicated cluster

The tuning required for large messages (larger replica.fetch.max.bytes, higher batch.size, longer linger.ms) conflicts with the tuning required for low-latency real-time traffic. Mixing workloads with significantly different size profiles on the same cluster means optimising for neither. If you have topics with 1 MB or larger messages and other topics with strict latency SLAs, use separate clusters.

Monitor the right JMX metrics

The metrics most directly relevant to message size behaviour:

RequestSizeAvg and RequestSizeMax: track against message.max.bytes. Set an alert at 75% of the broker limit on RequestSizeMax so you detect payload bloat before it becomes an incident.
RecordsPerRequestAvg: low values indicate poor batching.
bufferpool-wait-ratio: producer-side pressure on buffer allocation.
records-lag-max: correlate spikes with RequestSizeAvg to determine whether consumer lag is driven by large records.
Under-replicated partitions (URP): a sustained URP count is the primary signal of a replica.fetch.max.bytes misconfiguration for large messages.

For payloads that genuinely need to exceed 1 MB, use the claim-check pattern

Confluent’s own pattern documentation describes this as the standard approach: store the payload in an external store such as S3 or GCS, and publish only a small pointer record to Kafka. The pointer typically contains bucket, key, ETag, content type, size, and schema version. Sign URLs at read time with short TTLs. Manage object cleanup via lifecycle rules, and ensure delete.retention.ms on the Kafka topic is longer than the S3 grace period to avoid orphaned tombstones.

The claim-check pattern adds an external call in the read path, which introduces latency and a dependency on the external store’s availability. It is not suitable for stream-processing topologies where the external hop breaks the processing model. For those cases, in-band chunking using a library such as LinkedIn’s open-source li-apache-kafka-clients is the alternative: the producer fragments messages at a configurable segment size, and the consumer reassembles them before the application sees the record.

How Kpow helps you manage message sizes

Correctly sizing and configuring Kafka messages across a production cluster involves tracking multiple interdependent parameters across brokers, topics, and clients. Kpow is a commercial Kafka management UI and API from Factor House that surfaces the configurations and metrics most relevant to message-size operations without requiring you to instrument a separate observability stack.

Configuration visibility across brokers and topics

Kpow’s topic and broker configuration views show every config parameter alongside its current value, source (default, dynamic, or static), and importance. For message-size-relevant parameters specifically, the topic creation form displays the top five most common values set across your cluster for each config item, which makes it straightforward to spot drift in max.message.bytes across topics. You can view and edit message.max.bytes and replica.fetch.max.bytes at the broker level with appropriate RBAC permissions, and Kpow renders the equivalent kafka-topics.sh command if you prefer to manage changes through GitOps rather than through the UI.

Under-replicated partition detection

Kpow surfaces under-replicated partition counts with elapsed time since the URP was first detected. This is the primary operational signal for catching replica.fetch.max.bytes misconfiguration in large-message environments. The calculation handles offline brokers that are not visible to the AdminClient, so URP counts remain accurate even during partial cluster failures.

Message inspection for large-payload topics

For topics carrying large messages, Kpow’s Data Inspect feature runs server-side JQ-like queries across JSON, Avro, and Protobuf payloads. For topics with large messages, the SAMPLER_POLL_MS parameter can be increased from its 3,500 ms default to give consumers more time to fetch and batch records per poll. Message size configuration is directly relevant here: Kpow’s own documentation explicitly notes this tuning for large-message scenarios.

Throughput and lag metrics per topic

Kpow computes per-topic throughput in both messages per second and MB/s, partition lag, and consumer group offsets without requiring external Prometheus exporters. For clusters running on Confluent Cloud, it integrates with the Confluent Cloud Metrics API to surface retained bytes and active connection counts per topic, which helps you verify whether actual on-disk topic size matches your message-size and retention assumptions. Prometheus endpoints are also available per cluster and per topic for teams that want to pipe data into an existing Grafana stack.

Kpow runs as a single stateless Docker container and is compatible with Apache Kafka 1.0 and later, as well as managed services including Amazon MSK, Confluent Cloud, Azure Event Hubs, Aiven, and Redpanda. A free 30-day trial with full enterprise functionality is available. No credit card is required, and a free Community Edition is permanently available for local development environments.

For guidance on structuring and selecting message keys, which interacts with partitioning behaviour and message ordering, see our article on Kafka message key best practices.

Kafka partition key best practices

Factor House — Mon, 18 May 2026 00:00:00 GMT

The partition key is the most consequential design decision you make for a Kafka topic. It determines message ordering, consumer parallelism, log compaction behavior, and whether your cluster develops hot spots under load. Getting it wrong early is expensive to fix later.

This guide covers what partition keys actually do at the implementation level, common patterns and anti-patterns, and practical guidance on monitoring and migration. Tools like Kpow can help you observe partition behavior across your cluster, which is covered toward the end.

What partition keys are and what they do

A Kafka producer record is a (key, value, headers, timestamp) tuple. When the producer sends a record, the partitioner decides which partition it lands on. The default Java DefaultPartitioner follows this logic:

if (partition is explicitly specified) → use it else if (keyBytes != null) → partition = murmur2(keyBytes) % numPartitions else → sticky partitioner picks a partition until the batch fills

Murmur2 is a non-cryptographic 32-bit hash chosen for speed and distribution. The critical detail is the modulo operation: partition assignment is hash(key) % N, where N is the partition count. Any change to N reshuffles the entire mapping, which breaks ordering guarantees for existing keys. Every Kafka practitioner guide repeats this warning, and for good reason.

Null keys

Null-keyed records are handled differently depending on your Kafka version:

Pre-Kafka 2.4: round-robin per record, producing even distribution but small, inefficient batches.
Kafka 2.4+ (KIP-480): the UniformStickyPartitioner pins to one partition until the batch fills, then rotates. Better batching, but the original implementation worsened skew when a broker was slow.
Kafka 3.3+ (KIP-794): UniformStickyPartitioner is @Deprecated. The current recommendation is to remove the partitioner class configuration and set partitioner.ignore.keys=true instead.

Null keys are appropriate when ordering is irrelevant and downstream consumers have no per-entity locality requirements, such as fire-and-forget log shipping. They are a problem when the topic is compacted, when consumers expect per-entity ordering, or when you anticipate adding stateful stream processing later.

Log compaction

With cleanup.policy=compact, Kafka’s log-cleaner retains only the latest value per key per partition. This requires non-null keys. The broker exposes NoKeyCompactedTopicRecordsPerSec precisely so you can alert when null-keyed records land on a compacted topic. This metric should always be zero on compacted topics.

Compaction is also per-partition. If the same logical entity ends up under different keys due to a mutable field change, or on different partitions because partition count increased, compaction will not deduplicate across those entries. Compacted topics need immutable keys.

Kafka producer partition strategies

The table below covers the built-in options:

Strategy	When to use
Default (murmur2 on key, sticky on null)	General use; correct for most workloads
UniformStickyPartitioner (deprecated 3.3+)	High-throughput unkeyed writes pre-3.3 only. Never use for keyed traffic: it explicitly ignores keys
partitioner.ignore.keys=true (3.3+)	Modern equivalent of sticky for unkeyed writes
RoundRobinPartitioner	Forces round-robin even with keys present; rarely appropriate
Custom Partitioner	Priority routing, geographic affinity, multi-tenant isolation, weighted distribution

Custom partitioners give you precise control but come with real implementation complexity. The cross-language hashing problem described below is the canonical example of how easily a custom partitioning scheme can break. Confluent’s guidance is to stick with the default unless you have a clear, measurable reason to change.

The cross-language hashing problem

librdkafka (the C library underlying Python, Go, and other non-JVM Kafka clients) defaults to CRC32, not murmur2. A Python producer and a Java producer writing to the same topic with the same logical key will land records on different partitions unless you explicitly set partitioner=murmur2_random in librdkafka.

This matters in mixed-language environments, and it affects Kafka Connect tasks and CDC connectors, which are typically JVM-based and will use murmur2. If you have multiple producer languages writing to the same topic, pin the partitioner explicitly in every client.

Kafka partition key best practices

Choose a high-cardinality, immutable identifier

The most important thing you can do is choose the right key before the topic goes into production. The decision frames are:

What is the smallest entity that must stay ordered together? That entity’s ID is your key.
Is that field immutable? If it can change (email address, username, account status, IP address), do not use it as a key.
Does it have enough distinct values? For even distribution, you generally need at least 20 times as many distinct key values as partitions. With fewer than that, the binomial variance is high and some partitions will get significantly more load than others.

System-generated identifiers (UUIDs, customer IDs, aggregate IDs, order IDs) satisfy all three criteria. User-editable fields almost never do.

Avoid mutable fields

Using a mutable field as a partition key creates two compounding problems.

First, ordering breaks. Events for the same logical entity before and after the field changes hash to different partitions. Any stateful consumer that expects per-entity ordering will process those events out of sequence, silently.

Second, log compaction breaks. A compacted topic keyed by email accumulates stale entries whenever users change their email address. The old key and the new key are distinct, so the log-cleaner cannot merge them. The compacted log grows indefinitely without correctly representing the latest state of each entity.

The well-established guidance from Confluent, Cloudurable, and others: key compacted topics by an immutable, system-generated entity ID, never by anything the application or user can mutate.

Watch key cardinality

Low-cardinality keys (region, country, plan tier, status enum) guarantee hot partitions. If you only have five regions and thirty partitions, at most five partitions will ever receive traffic.

High-cardinality keys can also cause hot partitions when the distribution is skewed. Tenant IDs where one enterprise customer generates orders of magnitude more traffic than others, or user IDs with celebrity users, follow a Zipf distribution. The math looks fine on average, but in practice one or two keys drive the majority of throughput.

Both cases require intervention, but through different mechanisms.

Handle hot partitions with composite keys or salting

Composite keys combine two fields: tenantId|entityId, region|customerId, shardKey|primaryKey. This widens the cardinality of the routing key while preserving ordering for the inner entity. Use composite keys when one field alone does not have enough cardinality, or when one field has a skewed distribution.

Key salting appends a random bucket suffix to a hot key: userId#<0..K-1>. This distributes a single hot entity across K partitions. The trade-off is that per-entity ordering is lost across those salt buckets, so consumers must be designed to merge by the un-salted prefix. Salting works well when downstream processing is associative and order-insensitive (counts, sums, deduplication to a database).

Time-window bucketing adds a time component: userId|2025-10-21T10:00. This is useful for bursty single keys where the burst is time-bounded.

Dedicated topics for celebrity keys are the most expensive option but fully isolate their load. Reserve this for cases where the other approaches are not viable.

New Relic’s events pipeline is a documented example: the top 1.5% of query identifiers drove roughly 90% of events on their aggregation topic. Their fix was a composite key combining query ID with a time-window start time, spreading hot queries across partitions in time-bounded chunks.

Do not use `UniformStickyPartitioner` for keyed traffic

This warrants a specific call-out because it is a common mistake. UniformStickyPartitioner explicitly ignores partition keys. Records with the same key are not guaranteed to land on the same partition. The class has been @Deprecated since Kafka 3.3, but it appears in older documentation and can still be set explicitly. If you see UniformStickyPartitioner in your producer configuration and your topic has keyed traffic, remove it.

Over-provision partitions at topic creation

Adding partitions to a live topic with keyed traffic breaks the hash(key) % N mapping. Keys that previously routed to partition X will route to a different partition after the count changes. For stateful Kafka Streams applications, this is worse: state stores partition by the same hash, so an expansion also corrupts the state mapping.

Start with more partitions than you need today. It is much cheaper to over-provision upfront than to migrate later. The practical upper bound is around 4,000 partitions per broker on conservative deployments. Clusters with 100,000+ partitions have caused outages even under no traffic, because each partition represents file handles, metadata, and replica-fetcher threads.

Be careful with null keys on compacted topics

If your topic has cleanup.policy=compact, monitor NoKeyCompactedTopicRecordsPerSec. A value above zero means null-keyed records are reaching the topic and compaction is doing nothing for them. Producers sending null keys to a compacted topic are usually misconfigured. Alert on this metric; it should always be zero.

Standardise the hash function across all producers

If only one language produces to a topic, the default hash function is whatever that client uses. If multiple languages produce to the same topic and rely on consistent key-to-partition routing, you must pin the hash function explicitly. Set partitioner=murmur2_random in librdkafka-based clients so they match the JVM default. Verify CDC connectors and Kafka Connect tasks, which are JVM-based and use murmur2 regardless of other producer languages in your stack.

How to change your key strategy safely

Once a topic is in production, you cannot change the partition count or key field in place without breaking ordering. The documented migration pattern, used by teams at AppsFlyer and described in Confluent’s documentation, is:

Run a side consumer that reads the live topic and simulates the new key, computing what partition each record would land on. Collect distribution metrics to validate the new key before committing to it.
Create a new topic with the new partition count and key schema.
Dual-write via a feature flag. Producers emit to both the old and new topic simultaneously.
Bring up new consumers reading from the new topic. Verify they produce the same outputs as the old consumers.
Drain old consumers, cut traffic over, decommission the old topic.
Communicate the migration plan to all downstream teams before flipping the flag. Partition keys are a cross-team contract. Changing them affects every service that reads from the topic, every stateful processor, and potentially every database that receives derived output.

After the cutover, re-tune linger.ms, batch.size, and buffer.memory. Different keys have different temporal clustering properties, and batching dynamics change in ways that are not always immediately obvious.

Monitoring partition key health

The metrics that matter most for key-related issues:

Layer	Metric	What it tells you
Partition	kafka.cluster:type=Partition,topic={t},name=Size,partition={p}	Per-partition byte rate variance is the primary skew signal
Consumer	Per-partition LAG from kafka-consumer-groups.sh –describe or records-lag-max	Key skew often shows up in lag before it shows up in disk usage
Replication	UnderReplicatedPartitions, IsrShrinksPerSec	Hot partitions frequently manifest as ISR shrinks
Compaction	NoKeyCompactedTopicRecordsPerSec	Should be 0 on every compacted topic
Broker	BytesInPerSec,topic={t}	Top-line throughput comparison across topics

‍

Operationally, alert when the max-to-average per-partition byte rate ratio exceeds 1.5 to 2.0. A ratio under 1.2 generally requires no action. A ratio above 2.0 is worth treating as urgent: investigate whether to apply a composite key, salting, or a dedicated topic for celebrity keys.

For topic-level partition counts referenced in the Kafka topic partition best practices guide, the same thresholds apply: per-broker partition counts approaching 3,500 warrant attention regardless of traffic levels.

How Kpow helps with partition key monitoring

Choosing a good partition key is a design-time decision, but validating and maintaining it is an operational one. Kpow provides the visibility that makes this practical at scale.

Within Kpow’s consumer group views, you can inspect per-partition lag across all consumer groups on a topic, which is the most direct signal of key skew in production. When one or two partitions consistently lag behind the rest, the distribution analysis starts there. Kpow’s multi-cluster support means you can compare partition health across environments in a single view rather than running kafka-consumer-groups.sh queries against each cluster separately.

For compacted topics, Kpow’s topic inspection surfaces message-level metadata including keys, which makes it straightforward to identify producers sending null-keyed records to compacted topics before NoKeyCompactedTopicRecordsPerSec metrics are fully instrumented.

Kpow also supports broker-level health monitoring, so when a hot partition starts manifesting as ISR shrinks or under-replicated partitions, the signal is visible alongside the consumer-group lag data that points to the likely cause.

You can try Kpow for yourself with a free 30-day trial. It connects to any Kafka cluster and deploys via Docker, Helm, or JAR.

Summary

The decisions that matter most, in order of when you make them:

Before topic creation: choose an immutable, high-cardinality, system-generated identifier. Estimate distinct(key) / num_partitions and target at least 20. Over-provision partitions; it is much cheaper than migrating later.
When building producers: standardise the hash function across all client languages. Verify CDC connectors explicitly. Do not use UniformStickyPartitioner on keyed topics.
In production: monitor per-partition byte rate variance and consumer-group lag skew. Alert on max/avg ratio above 1.5. Alert on NoKeyCompactedTopicRecordsPerSec > 0 for compacted topics.
When skew appears: identify whether it is a data distribution problem or a broker placement problem. For data: composite keys first, then salting, then dedicated topics for intractable cases.
When migration is necessary: new topic, dual-write, validation, cutover. Never change partition count on a live keyed topic in place.

Kafka cluster management: A practical guide for engineers

Factor House — Mon, 11 May 2026 00:00:00 GMT

Key takeaways

The decisions you make at cluster creation time, specifically partition counts, replication factor, broker sizing, and KRaft vs ZooKeeper, determine whether you scale cleanly or accumulate operational debt that is difficult to unwind.
Apache Kafka 4.0 (released March 2025) removed ZooKeeper entirely. KRaft is now the only supported mode. If you are still running ZooKeeper-based Kafka 3.x, your migration path is mandatory.
The most impactful monitoring signals are under-replicated partitions (URPs), ActiveControllerCount, OfflinePartitionsCount, and consumer lag trend. Alert on these; log everything else.
Producer compression combined with sensible batching is the highest-leverage performance tuning available without architectural changes.
Tools like Kpow can provide visibility into partition health and URP status, and allow you to manage partition reassignments and leader elections without dropping to the CLI.

Understanding Kafka at a conceptual level is one thing. Operating it in production is something else entirely. At a handful of topics and a few brokers, the configuration feels manageable. Once you are running dozens of topics across multiple teams, with different retention requirements, latency SLOs, and replication topologies, the operational surface area expands considerably. Partition count decisions become difficult to reverse. Monitoring gaps surface during incidents rather than during planning. Rebalancing a cluster without causing ISR shrinkage requires care. None of this is obvious from the documentation alone.

This guide covers the practical decisions and operational patterns that matter most when running Kafka clusters at scale: how to size and structure clusters from day one, what to watch for in production, how to tune producers, consumers, and brokers, and how the shift to KRaft has changed the management picture.

What is Kafka cluster management?

Kafka cluster management covers the end-to-end operational work of keeping one or more Kafka clusters running correctly: provisioning and sizing brokers, choosing partition strategies and replication settings, managing consumer groups and offsets, handling rolling operations and partition reassignments, monitoring for cluster health, tuning configuration for performance, enforcing access control, and planning for the architectural shifts like the ZooKeeper-to-KRaft migration.

It is not a single task but a continuous discipline. The configuration choices you make at topic creation time constrain what you can do later. The monitoring infrastructure you build before incidents determines how quickly you can diagnose them. And the operational practices you establish early, things like how you handle rolling restarts, how you reassign partitions, and how you gate cluster mutations, determine whether your cluster remains manageable as it grows.

How architecture and sizing decisions affect cluster management

Start with three brokers across three availability zones

The baseline production configuration is three brokers, one per availability zone, with replication.factor=3 and min.insync.replicas=2. This is the smallest topology that survives a single broker failure while still allowing producers with acks=all to make progress. RF=2 only survives zero failures during maintenance, which is not a reasonable production posture. Note that increasing the replication factor on a live topic adds meaningful network and disk pressure. Get it right at topic creation.

For storage, the choice is straightforward for latency-sensitive workloads: NVMe/SSD or fast provisioned EBS (gp3 or io2 on AWS). HDD is acceptable only for cold archive workloads where tiered storage is in use. A single historical consumer reading from disk on a shared HDD broker can cause a 43% drop in producer throughput, based on Stanislav Kozlovski’s analysis of KIP-405 storage trade-offs. The memory guidance is consistent across most large operators: 4-8 GB JVM heap, with the remainder of RAM left to the OS page cache.

Partition count is the most consequential and least reversible decision

For keyed topics, you cannot increase the partition count later without breaking hash(key) % num_partitions ordering. Confluent’s official guidance states up to approximately 4,000 partitions per broker and 200,000 per cluster on ZooKeeper-based deployments. On KRaft, lab tests demonstrated stable operation at 2 million partitions per cluster, though real-world production clusters typically run in the hundreds of thousands.

For sizing, Confluent’s formula is a reasonable starting point: minimum partitions = max(target_throughput / per_partition_producer_throughput, target_throughput / per_partition_consumer_throughput). On modern hardware with LZ4 compression and acks=all, single-partition producer throughput is typically 10-50 MB/s; consumer throughput is 50-100+ MB/s. Add 2-3x headroom for hot-key skew.

For small clusters under six brokers, starting at approximately 3x the broker count in partitions is a reasonable rule. For larger clusters above twelve brokers, 2x the broker count tends to work. Do not under-partition either: a single-partition topic caps you at one consumer instance for that topic-level parallelism.

Cluster isolation by workload type

LinkedIn, Netflix, Pinterest, and Cloudflare all operate many smaller clusters rather than a single large one. Netflix explicitly maintains a rule of a maximum of 200 brokers per cluster, preferring more clusters of moderate size. The operational argument is compelling: a single large cluster creates a blast radius that affects every consumer and producer on it. Isolating by traffic class (tracking, logging, metrics, CDC) limits the impact of a misconfigured producer or a runaway topic.

Multi-cluster topology comes before tool selection

If you are planning for geo-replication or disaster recovery, choose your topology first: active-passive, active-active, or aggregation. Then select the tool. MirrorMaker 2 (MM2) is the open-source option; it runs on Kafka Connect and handles topic replication via MirrorSourceConnector, offset translation via MirrorCheckpointConnector, and heartbeats via MirrorHeartbeatConnector. The main operational catch is that offset sync is periodic (default 60 seconds), so consumers may reprocess up to that interval’s messages on failover. Make your consumers idempotent regardless of which replication tool you use, and test failover at least quarterly. At LinkedIn or Uber scale, MM2 becomes an operational burden; purpose-built tools like Brooklin (LinkedIn) or uReplicator (Uber) were built to handle per-partition error isolation that MM2 cannot provide.

For Confluent Platform or Cloud users, Cluster Linking provides byte-for-byte replication that preserves offsets exactly, at the cost of vendor lock-in.

Day-to-day cluster operations

Rolling restarts

Rolling restarts are the standard approach for applying broker configuration changes or upgrades without downtime. The procedure is straightforward: drain, restart, wait for ISR recovery, repeat. Before restarting a broker, confirm that UnderReplicatedPartitions is zero cluster-wide. If it is not, wait. After restarting, wait for the broker to fully rejoin the ISR before proceeding to the next node. Restarting a broker while another is still recovering from a previous restart risks losing min.insync.replicas headroom, which will cause producers with acks=all to start receiving NotEnoughReplicasException.

# Check cluster-wide under-replicated partitions before each step

kafka-topics.sh --bootstrap-server localhost:9092 \

--describe --under-replicated-partitions

‍

# Graceful broker shutdown

kafka-server-stop.sh

Partition reassignment

Adding brokers to a cluster does not automatically rebalance existing partitions. Only new partitions land on new brokers by default. After adding capacity, you need to explicitly reassign partitions using either Cruise Control (LinkedIn’s open-source rebalancing tool, the production standard) or kafka-reassign-partitions.sh. Always throttle reassignments to avoid saturating broker network and triggering ISR shrinkage during the move.

# Generate a reassignment plan for specified topics

kafka-reassign-partitions.sh \

--bootstrap-server localhost:9092 \

--topics-to-move-json-file topics.json \

--broker-list "0,1,2,3" \

--generate

‍

# Execute with throttle (bytes/sec)

kafka-reassign-partitions.sh \

--bootstrap-server localhost:9092 \

--reassignment-json-file reassign.json \

--throttle 50000000 \

--execute

‍

# Verify completion

kafka-reassign-partitions.sh \

--bootstrap-server localhost:9092 \

--reassignment-json-file reassign.json \

--verify

Cruise Control automates this with goal-driven rebalancing: it accounts for rack awareness, leader replica distribution, bytes-in distribution, and disk capacity targets, reducing the manual overhead significantly on clusters with dozens of brokers.

Consumer group and offset management

The most common operational tasks around consumer groups are offset resets (after a deployment rollback, a poison message event, or a schema change) and lag investigation. Disable auto-commit for any consumer doing meaningful processing. The robust pattern is enable.auto.commit=false, process the message, then commit explicitly after the result is durable.

For investigating consumer group state:

# List all consumer groups

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list

‍

# Describe a specific group, including lag per partition

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \

--group my-consumer-group --describe

‍

# Reset offsets to latest (requires group to be inactive)

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \

--group my-consumer-group \

--topic my-topic \

--reset-offsets --to-latest \

--execute

Static membership (KIP-345) is worth enabling for stateful consumers. Set group.instance.id to a stable per-instance identifier. When the consumer restarts, the broker waits for session.timeout.ms before triggering a rebalance, and when the consumer rejoins with the same instance ID, it receives the same partition assignments back. This eliminates rebalances caused by rolling deployments. When using static membership, increase session.timeout.ms to 10-30 minutes to prevent false-positive rebalances.

For the assignor, switch to CooperativeStickyAssignor (KIP-429, available since Kafka 2.4) to avoid the stop-the-world rebalance behaviour of the eager protocol. On Kafka 4.0+, the next-generation rebalance protocol (KIP-848) is GA and moves rebalance coordination entirely server-side, measured by Instaclustr at up to 20x faster than eager rebalancing.

Observing clusters for health

Effective cluster monitoring distinguishes between three things: what you need to page on, what warrants a ticket, and what you log and review periodically.

Alert-worthy signals

These are the metrics that indicate active cluster failure or imminent unavailability:

kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions: Sustained non-zero across many brokers indicates a broker is offline or severely degraded. This is the single most critical cluster-level metric.
kafka.controller:type=KafkaController,name=ActiveControllerCount: Must be exactly 1 cluster-wide. A value of 0 or greater than 1 requires immediate investigation.
kafka.controller:type=KafkaController,name=OfflinePartitionsCount: Must be 0. Any offline partitions mean producers and consumers cannot access that data.
kafka.cluster:type=Partition,name=UnderMinIsr: Partitions below min.insync.replicas. Producers with acks=all cannot write to these partitions.
Consumer lag trending upward: A consumer that is falling further behind over time, not just experiencing a momentary spike, is an application-level incident.

For consumer lag monitoring, LinkedIn’s Burrow is the most robust approach in the open-source space, and often included in Kafka management tools. Rather than alerting on raw offset distance (which produces false positives during batch catch-up), Burrow evaluates lag over a sliding window of commits and classifies consumer groups as OK, WARNING, or ERROR based on whether the lag trend is decreasing, stable, or growing. This significantly reduces alert fatigue. For more detail on consumer lag monitoring strategies and tooling, see how to monitor Kafka consumer lag.

Ticket-worthy signals

These indicate degradation worth investigating before it becomes a page:

kafka.server:type=ReplicaManager,name=IsrShrinksPerSec / IsrExpandsPerSec: Frequent ISR shrink/expand events on a single broker usually mean that broker has a disk or network issue. Sustained shrinks across all brokers usually mean aggregate load or a network problem.
kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent: Below 0.7 (30% utilisation) means performance is degrading. Below 0.5 means the broker is approaching saturation.

Monitoring tooling

The standard production stack is JMX exported to Prometheus via the JMX Exporter, visualised in Grafana. OpenTelemetry Collector with the JMX receiver is increasingly common as an alternative. For companies already running Datadog, the Kafka integration covers the core broker and consumer metrics. Confluent Control Center is the natural choice on Confluent Platform.

For visibility into cluster health and Kafka monitoring more broadly, the key principle from engineering teams at Cloudflare and Pinterest is to avoid alert fatigue: broad threshold alerts on per-topic metrics fire too frequently to be actionable. Focus alerts on the signals listed above, and treat per-topic byte rates and partition commit latencies as log-level observability.

Tuning producers, consumers, and brokers for performance

Producer configuration

Compression combined with larger batch sizes is the highest-leverage producer tuning available. Cloudflare’s production data is instructive: Snappy plus approximately 1-second batching reduced a metrics topic from 800 Mbps to 170 Mbps. At 100 TB/month, a 3x compression ratio represents a 67% cost reduction on network egress, replication, storage, and consumer transfer.

Key producer settings:

# Compression: lz4 for speed, zstd for ratio

compression.type=lz4

‍

# Larger batch size (default 16 KB is too small for high throughput)

batch.size=131072

‍

# Wait for batches to fill before sending

linger.ms=10

‍

# Durability

acks=all

enable.idempotence=true

‍

# Compatible with idempotence

max.in.flight.requests.per.connection=5

One critical gotcha: if your producer compression.type does not match the topic’s compression.type, the broker decompresses and recompresses every batch. Set the topic-level configuration to producer to pass through the producer’s codec, or ensure they match explicitly.

For schema format, Avro and Protobuf compress dramatically better than JSON and have cheaper deserialisation. Shopify and Cloudflare both use Protobuf as their standard wire format.

Consumer configuration

# Process explicitly, commit explicitly

enable.auto.commit=false

‍

# Tune for your processing capacity

max.poll.records=500

‍

# Batch reading for throughput

fetch.min.bytes=1024

‍

# Cooperative rebalancing

partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

‍

# Static membership (for stateful consumers)

group.instance.id=consumer-instance-1

session.timeout.ms=600000

Set max.poll.records to a value you can reliably process within max.poll.interval.ms. If processing a batch takes longer than max.poll.interval.ms, the consumer is removed from the group and a rebalance triggers.

Broker configuration

# Increase for high-connection environments

num.network.threads=8

‍

# Scale with disk count

num.io.threads=16

‍

# Larger socket buffers for high-bandwidth networks

socket.send.buffer.bytes=1048576

socket.receive.buffer.bytes=1048576

Leave log.flush.interval.messages and log.flush.interval.ms at their defaults. Kafka relies on OS page cache and replication for durability. Forcing fsync eliminates that benefit and significantly reduces throughput.

At the OS level, set vm.swappiness=1 to prevent the kernel from consuming page cache for swap, vm.max_map_count=262144 or higher (each log segment requires two memory-mapped areas, and the default of 65536 is quickly exhausted on brokers with thousands of partitions), and ensure file descriptor limits are set to 100,000 or above. Mount filesystems with noatime and prefer XFS for large disks.

JVM heap should be 6-8 GB with G1GC. Never exceed 32 GB or you lose compressed ordinary object pointers, which has a meaningful impact on GC overhead.

Security and multi-tenancy

ACLs

Kafka ACLs bind principals (users, service accounts) to resources (topics, consumer groups, clusters) with specific operations (Read, Write, Create, Describe, Alter, Delete). They work well at moderate scale. As the number of teams and topics grows, the number of ACL rules grows correspondingly, and ACL management can become a significant operational overhead.

# Grant a service account read access to a topic

kafka-acls.sh --bootstrap-server localhost:9092 \

--add \

--allow-principal User:my-service \

--operation Read \

--topic my-topic

‍

# List ACLs for a topic

kafka-acls.sh --bootstrap-server localhost:9092 \

--list --topic my-topic

Multi-tenancy patterns

At the cluster level, multi-tenancy in Kafka typically means some combination of namespace conventions for topics (for example, team-name.topic-name), ACLs restricting access to namespaced resources, and quota configurations to prevent any single tenant from saturating broker network or request handler threads.

Kafka quotas allow you to set byte-rate limits and request-rate limits per client ID or user principal:

# Set a producer byte-rate quota for a client

kafka-configs.sh --bootstrap-server localhost:9092 \

--alter \

--add-config 'producer_byte_rate=10485760' \

--entity-type clients \

--entity-name my-producer-client

For larger organisations, dedicated clusters per business domain or per environment (development, staging, production) provide better blast radius containment than a multi-tenant shared cluster.

KRaft and what it changes for cluster management

Why ZooKeeper was removed

ZooKeeper was a separately operated dependency: a second cluster to deploy, secure, monitor, and patch. It also became a scaling bottleneck. Every metadata change, broker registration, partition leader election, ACL update, and topic configuration update, flowed through ZooKeeper. For a 10,000-partition cluster, controller failover required reading all partition metadata from ZooKeeper during initialisation, adding approximately 20 seconds to the unavailability window during an unclean failure.

How KRaft works

KRaft replaces ZooKeeper with a Raft-based consensus protocol implemented inside Kafka itself. A dedicated controller quorum, typically three or five nodes, maintains an internal __cluster_metadata topic where each metadata change is written as an event. Brokers consume this metadata stream like a regular Kafka topic. Controller failover is near-instantaneous because the incoming active controller already has all committed metadata in memory. Confluent migrated all Confluent Cloud clusters to KRaft without customer-visible downtime. Aiven migrated 15,000 servers over three months with zero downtime.

The operational benefits are concrete: Confluent reports approximately 40% reduction in cluster setup time and the ability to scale to millions of partitions in a single cluster (demonstrated at 2 million in lab conditions).

Migration path if you are still on ZooKeeper

Kafka 4.0 (March 2025) removed ZooKeeper support entirely. If you are running Kafka 3.x with ZooKeeper, your path is:

Upgrade to Kafka 3.9 (the bridge release with the most complete migration tooling).
Provision a dedicated KRaft controller quorum with zookeeper.metadata.migration.enable=true.
Roll restart brokers in migration mode. The active KRaft controller copies all metadata from ZooKeeper to __cluster_metadata.
Complete the dual-write phase where KRaft is authoritative and ZooKeeper is a safety net. You can still roll back at this point.
Finalize: reconfigure brokers for KRaft only. No rollback is possible after this step.

Use kafka-migration-check for preflight and status checks, and monitor progress via the JMX MBean kafka.controller:type=KafkaController,name=ZkMigrationState. The final state should be MIGRATION_COMPLETED.

Budget two to four weeks of dry-run testing in a non-production environment and one to three months for a full production rollout depending on cluster count.

Key constraints: migration only supports isolated mode (dedicated controllers), not combined mode. JBOD support arrived in Kafka 3.7, so older KRaft versions did not support multiple log directories. Kafka 4.0 clients require brokers at version 2.1 or higher; upgrade all clients before moving brokers to 4.x.

Managing topic partitions with Kpow

Kpow provides a UI layer for the partition management operations that are otherwise handled entirely through the CLI. This is particularly useful in environments where you want to limit direct CLI access for most operators, or where you need an audit trail of partition mutations.

Topic and partition creation

Kpow’s topic creation form lets you specify partition count, replication factor, and any additional configuration values. If you are not sure what configuration values are appropriate, the interface surfaces documentation inline, including the top five most common values currently set across your cluster for each parameter. This can be useful when creating topics that should match the configuration patterns already established for similar topics.

If you prefer not to grant Kpow mutation permissions but still want the UI for planning, the form generates an equivalent kafka-topics.sh command that updates reactively as you fill in the form.

Partition reassignment and leader election

From the Topics page, Kpow supports partition reassignment at the individual partition level: you select a topic partition, click the reassignment action, and choose target replicas. In-progress reassignments are visible in a dedicated Reassignment tab, where you can also cancel any in-flight reassignment. Full cluster and full topic reassignment support is on the roadmap.

For leader election, Kpow supports both preferred and unclean election types. Preferred election is the standard operation for rebalancing leader distribution after a rolling restart. Unclean election is a last-resort option that allows a replica not in the ISR to become leader, with the trade-off of potential data loss.

URP detection

Kpow provides enhanced URP detection that correctly identifies partitions where the in-sync replica count is below the configured replication factor, even when a broker is offline and not visible to the AdminClient. URP totals are displayed on both the Brokers and Topics pages, and if the count is greater than zero, a detailed table expands to list every affected topic and partition. This makes it practical to identify URP scope quickly during an incident without running kafka-topics.sh –under-replicated-partitions manually across multiple brokers.

Consumer group management

Kpow’s Groups UI handles consumer group offset management across the full range of dimensions: whole-group, host-level, topic-level, and partition-level. Offset resets are scheduled and held until the consumer group reaches an EMPTY state, which prevents modifications to an active group from causing consistency issues. For static consumer groups, you can also remove individual members to trigger a rebalance without waiting for session.timeout.ms to expire.

Moving from reactive to proactive cluster management

Most Kafka problems do not arrive suddenly. ISR shrink rates creep up before brokers start failing. Consumer lag trends upward before an application team reports an issue. Request handler idle percentages decline gradually before a broker becomes saturated. The pattern is almost always visible in the metrics before it becomes an incident, provided you are looking in the right place with the right resolution.

The practical shift toward proactive cluster management involves three things. First, instrumenting the signals that matter at the cluster level, the ones listed in the monitoring section above, and ensuring they flow into an alerting layer before they become visible to end users. Second, running rebalancing and maintenance operations on a schedule rather than in response to imbalance. Tools like Cruise Control are designed to run continuously and maintain goal-based cluster health rather than waiting for manual intervention. Third, maintaining an observation layer that gives you a complete picture of partition health, consumer group state, and broker configuration across all clusters in one place, rather than assembling that picture from multiple CLI commands during an incident.

If you want to explore what a unified observation layer looks like in practice, Kpow offers a free 30-day trial that you can connect to any Kafka cluster in minutes and deploy via Docker, Helm, or JAR.

Checklist: greenfield cluster in 2026

For reference, here are the baseline configuration choices validated against the practices of teams at LinkedIn, Pinterest, Netflix, Cloudflare, and Stripe:

Kafka 4.0+, KRaft only, isolated mode: three dedicated controllers plus three or more brokers across three AZs.
replication.factor=3, min.insync.replicas=2, acks=all, enable.idempotence=true.
NVMe/SSD or provisioned EBS; 6-8 GB JVM heap; rest to OS page cache.
LZ4 producer compression, batch.size=131072, linger.ms=10-20.
CooperativeStickyAssignor and group.instance.id on every stateful consumer.
Cruise Control (or equivalent) for rebalancing from day one.
JMX to Prometheus to Grafana, plus Burrow for consumer lag trend analysis.
Tiered storage enabled at topic creation for analytics or long-retention topics.
enable.auto.commit=false; commit offsets explicitly after processing is durable.
Avro or Protobuf with a Schema Registry.

Scale up brokers when sustained BytesInPerSec plus ReplicationBytesInPerSec exceeds 60% of NIC capacity, or when RequestHandlerAvgIdlePercent drops below 0.5. Consider adding a second cluster when approaching 200 brokers in one cluster, when a single noisy tenant repeatedly causes cross-tenant impact, or when regulatory data residency requirements make cluster isolation necessary.

Kafka topic partition best practices

Factor House — Mon, 11 May 2026 00:00:00 GMT

Partitions are the foundational unit of parallelism, ordering, storage, and consumer-group fan-out in Kafka. Every performance and reliability decision you make about a topic flows from how partitions work at the broker level. Getting partition count wrong at topic creation is costly because the recoverable and unrecoverable failure modes are asymmetric: adding partitions is mechanically trivial, but on a keyed topic it permanently breaks ordering guarantees for every downstream consumer.

This guide covers what partitions actually do, how to size them correctly, what breaks when you get it wrong, and what the most operationally mature Kafka deployments in production actually do.

What Kafka topic partitions actually do

A Kafka topic is a logical label. The partition is the real unit of work: a physically replicated, append-only log on disk. Four properties stack on top of that single primitive.

Ordering. Kafka guarantees message order within a single partition only. There is no global order across partitions. This has been true since the original 2011 paper by Kreps, Narkhede and Rao, and it has not changed. If your consumers depend on ordering guarantees, they depend on partition-level ordering, not topic-level ordering.

Parallelism. Producers and brokers write to different partitions fully in parallel. CPU-intensive work like compression scales with partition count. On the consumer side, parallelism is strictly bounded: one consumer thread per partition per consumer group. Extra consumers beyond the partition count sit idle.

Consumer-group assignment. A group coordinator broker assigns each partition to exactly one consumer in the group. The default assignor since Kafka 3.0 is CooperativeStickyAssignor (KIP-429), which performs incremental rebalancing instead of revoking all partitions from all consumers on every membership change. This does not change the ceiling, but it makes living at the ceiling less operationally painful.

Log segment mechanics. Each partition is a directory on disk. That directory contains a sequence of segment files (.log plus .index and .timeindex companions). The active segment is the only one currently being written to. A segment rolls when it hits log.segment.bytes (default 1 GB) or log.segment.ms (default 7 days). Critically, Kafka holds an open file descriptor to every segment in every partition, including inactive ones. That is the mechanical basis for the “more partitions = more open file descriptors” constraint. Cloudera’s documented formula for minimum file descriptors is (number of partitions) × (partition size / segment size). A starting point of 100,000 file descriptors per broker is the common recommendation from both Cloudera and LinkedIn.

Kafka topic partition best practices

Size partitions for two years of peak throughput, not today’s traffic

The single most consequential partition sizing decision is the time horizon you plan against. Because adding partitions to a keyed topic breaks per-key ordering guarantees permanently, the production pattern at every large-scale Kafka deployment (LinkedIn, Confluent, Shopify, Cloudflare) is to over-provision at topic creation and avoid –alter on keyed topics entirely.

The canonical formula, from Jun Rao’s 2015 Confluent post and still the foundation of Confluent’s official documentation:

partitions ≥ max(t/p, t/c)

Where t is your target throughput, p is per-partition produce throughput, and c is per-partition consume throughput.

A more complete version for production sizing:

P_writes = ceil(W / P_part_safe) # P_part_safe = 60–70% of measured peak per-partition produce throughput

P_consume = ceil(W / C_per_consumer) # consumer compute/IO bound, often 1–5 MB/s in practice

P_consumers = N_consumers_at_peak # physical parallelism you need to run

P = max(P_writes, P_consume, P_consumers) × 1.5–2 # growth headroom

Round the result up to a number with many divisors: 12, 24, 30, 60 are common choices. Avoid primes because they don’t divide evenly across brokers, which creates leadership imbalance. Document the partition count and the assumptions (peak MB/s, consumer count, retention) on the topic itself. These assumptions are what a future engineer needs when evaluating whether to alter the topic or create a new one.

Benchmark per-partition throughput in your actual cluster

What is realistic for per-partition throughput? The answer is highly workload-specific, which makes the formula only as good as the input numbers you use.

Confluent’s published guidance states that a single partition can generally sustain “10s of MB/sec.” LinkedIn’s Kreps benchmark on three commodity machines achieved ~50 MB/s sustained per partition on a six-partition topic. Conduktor’s 2025 guidance puts modern Kafka (3.0+) at 50–100+ MB/s per partition with proper tuning, calling earlier estimates conservative.

Netflix Keystone, operating at approximately 2 trillion messages per day, uses 0.5–1 MB/s per partition as their planning constant. A 10 MB/s topic gets 10 partitions. The reason the number is conservative relative to what Kafka can physically sustain is that the bottleneck is rarely the broker: it is usually the consumer.

Uber’s message-queue workload demonstrates this precisely. Their consumer logic involved synchronous RPC calls to payment processors, which limited each partition to approximately one event per second. Achieving 1,000 events/s required 1,000 partitions. The per-partition throughput number in your formula should reflect your slowest consumer path, not your broker’s disk ceiling.

Run kafka-producer-perf-test.sh with realistic message sizes and acks=all against your actual cluster before committing to a partition count. Take 60–70% of the peak result as your safe planning constant to account for bursts and replication pressure.

Account for replication factor in your broker load estimate

Replication factor multiplies broker-side cost directly. With RF=3, every partition leader has two follower replicas continuously fetching from it. Producing at 50 MB/s creates 100 MB/s of replication fetch traffic on top of the write.

Pinterest measured 50% background CPU on a 10,000-partition test cluster with RF=3 and no actual production load, purely from follower fetching. This is the cost that the older “100 partitions per broker” rule of thumb was trying to capture, and it is still present regardless of what version of Kafka you are running.

Standard configuration for production: RF=3, min.insync.replicas=2, acks=all. RF=2 is the floor; RF=1 is benchmark-only. Increasing RF on a live topic is more disruptive than adding partitions; it introduces immediate network and disk pressure that can affect in-sync replica sets across the cluster. Treat replication factor changes as maintenance operations, not on-the-fly adjustments.

Rack-awareness (KIP-881, Kafka 3.4+) extends replica placement into the CooperativeStickyAssignor so consumers can prefer same-AZ replicas via follower fetching. At cloud scale, this materially reduces inter-AZ data transfer costs. Pinterest spreads replicas across three availability zones to survive two simultaneous broker failures per cluster.

Understand the consumer-parallelism ceiling and its current exceptions

The rule is exact: the number of active consumer threads in one consumer group cannot exceed the number of partitions of the topics it subscribes to. Extras sit idle. They still incur join/leave overhead and hold a slot in the group.

Under-provisioning consumers is the more common and more visible failure mode: consumer lag rises until you scale up to the partition ceiling. Over-provisioning consumers is silently wasteful and is easy to miss without partition-level lag monitoring.

Two Kafka features reduce the operational pain of this ceiling without eliminating it:

Static group membership (group.instance.id, KIP-345) allows a consumer to leave and rejoin within session.timeout.ms without triggering a rebalance. This is the standard pattern for stateful consumers running Kubernetes rolling deployments, where the pod restart cycle would otherwise trigger repeated rebalances.

KIP-848 (default in Kafka 4.0) redesigns the rebalance protocol around a broker-side assignor, eliminating the global rebalance barrier so offsets can commit even mid-rebalance.

KIP-932 Share Groups (early access in 4.0, preview in 4.1, GA in 4.2) is the first feature that actually relaxes the ceiling itself. Share groups allow multiple consumers per partition with per-record acknowledgement, providing queue semantics on top of Kafka. If your consumer concurrency is bounded by partition count and you cannot add more partitions (because the topic is keyed), share groups are the right next step. They are not yet recommended for production clusters as of 4.1.

Know what breaking partition count actually breaks

Adding partitions to a topic runs in seconds: kafka-topics.sh –alter –partitions N. The mechanics work. What breaks depends on whether the topic is keyed.

For topics with null keys or round-robin partitioning, adding partitions is safe. Producers are unaffected; consumer groups rebalance automatically.

For keyed topics, adding partitions permanently breaks murmur2(key) % numPartitions for every key. Old data stays where it was; Kafka is append-only and immutable. New data with the same key may land on a different partition. From that point forward, per-key ordering guarantees are broken across the alter boundary. Stateful stream processors using Kafka Streams or ksqlDB use the partition as the key/state co-location boundary; RocksDB state stores, changelog topics, and repartition topics are all keyed by partition. Repartitioning the source topic does not reshuffle existing state.

Compacted topics have an additional failure mode: if a key was previously hashed to partition 2 and now hashes to partition 5, two values exist for the same key in different partitions. Log compaction does not see across partition boundaries, so the “latest value per key” semantic breaks.

Kafka does not support reducing partition count. This is documented explicitly. Plan upward.

The standard pattern for keyed topics that have genuinely outgrown their partition count: create a new topic with the desired partition count, dual-write from producers to both, drain consumers off the old topic, then decommission it. Cloudflare, Shopify, and Confluent all follow this pattern. It is operationally heavier but preserves per-key ordering guarantees and stateful consumer correctness.

For Kafka Streams pipelines, design downstream repartition() operations into the topology so the source topic’s partition count is decoupled from processing parallelism.

Understand the KRaft-era cluster partition limits

The ZooKeeper-era rules (4,000 partitions per broker, 200,000 per cluster) were driven by a specific bottleneck: controller failover. When a ZooKeeper controller failed, the new controller had to load the full cluster state from ZooKeeper before serving traffic. With 200,000 partitions, that took 14 seconds in Kafka 1.1. It grew linearly with partition count.

KRaft (KIP-500, production-ready in 3.3, mandatory in 4.0) replaces ZooKeeper with an event-sourced metadata log. Follower controllers hold the current state in memory. Failover is near-instantaneous regardless of partition count. Confluent’s lab has demonstrated a 2-million-partition cluster running on KRaft. Instaclustr created approximately 600,000 partitions on a single KRaft broker.

These numbers are not operating targets. They are ceiling demonstrations. The constraints that now bind you are resource-level, not metadata-level:

File descriptors: two per segment per partition; can blow past default ulimits quickly at high retention and throughput. Minimum ulimit recommendation is 100,000 per broker.
vm.max_map_count: Kafka memory-maps index files; the Linux default of 65,530 is a hard ceiling at modest partition counts. Set it to 1,000,000 or higher for high-partition-density clusters.
JVM heap and OS page cache: standard recommendation is 4–8 GB JVM heap, the rest for page cache.
Replica-fetcher threads: each follower fetches from leaders; more partitions demand more fetcher threads.
Producer and consumer client memory: Jun Rao’s guidance is to allocate at least a few tens of KB per partition being produced. A 100,000-partition topic can exhaust producer buffer.
Replication CPU: RF=3 essentially triples steady-state broker work.

The practical consequence for 2026: if you are on ZooKeeper, the 200K cluster cap is real and you should migrate to KRaft before evaluating partition counts. If you are on KRaft, the previous caps no longer bind you, but the resource costs that motivated the conservative rules still accumulate. Most production clusters still target hundreds to low-thousands of partitions per broker, not millions.

Choose numbers with good divisor properties

Avoid partition counts that are prime numbers. Kafka distributes partition leadership across brokers; prime numbers don’t divide evenly, producing leadership imbalance that over-stresses specific brokers.

Prefer numbers with many divisors: 12, 24, 30, 48, 60. These divide cleanly across 2, 3, 4, 6, and 12 brokers, giving you flexibility as your cluster scales and making rebalancing predictable.

The “3 × number of brokers” rule of thumb has a kernel of truth: it balances partition leadership at the time of topic creation. It is not a substitute for the throughput formula and it does not account for consumer parallelism requirements.

Set cluster-level guardrails, not just per-topic settings

Broker configuration defaults affect every topic on the cluster. Platform teams should enforce a consistent baseline:

RF=3, min.insync.replicas=2, acks=all on all production topics.
File descriptor ulimit ≥ 100,000 per broker.
vm.max_map_count ≥ 1,000,000 on Linux brokers.
4–8 GB JVM heap, remainder for OS page cache.
CooperativeStickyAssignor and group.instance.id for stateful consumers on Kubernetes.
Per-partition lag and throughput monitoring, not just per-topic. Consumer group lag at the topic level masks partition skew.

LinkedIn’s engineering team built custom tooling specifically because, as they documented, “Kafka does not natively provide partition throughput metrics.” At scale, per-topic aggregate metrics are not sufficient to detect which partitions are saturated.

Avoid common anti-patterns

Using one partition for global ordering. This is almost always the wrong design. Global ordering requires 1 partition, which means 1 broker handles all writes and 1 consumer in any group handles all reads. You need per-key ordering, not global ordering, and a key-based partitioner with enough partitions to distribute load achieves that.

Hot keys. The right partition count does not help you if your key distribution is skewed. One merchant with 90% of payment events in a merchant_id-keyed topic reproduces all the failure modes of a single-partition topic on whichever broker holds that partition. Key selection and partition count are coupled problems. See our separate guide on Kafka partition key best practices for detailed coverage.

Treating partition count as set-and-forget. Size for one to two years of peak growth, then review quarterly. A topic sized correctly at 1× traffic is almost certainly under-partitioned at 50× traffic, and the fix becomes progressively more disruptive the longer it is deferred.

Using null keys when ordering is required. When a producer omits message keys, sticky or round-robin partitioning is used. If downstream consumers have ordering assumptions, those assumptions silently break. This is a common bug, particularly when producers are refactored across teams.

Increasing replication factor as a quick fix. It works mechanically, but the disk and network pressure shock can trigger ISR shrinks across the cluster. Plan it as a maintenance operation.

Running ZooKeeper in 2026. ZooKeeper mode was deprecated in Kafka 3.5 and removed in 4.0. If you are still running it, the 200,000-partition cluster cap is a real operational constraint, and you are accumulating tech debt against a hard end-of-life.

What well-known organisations actually do

The following is a representative sample of how production Kafka deployments at scale approach partitioning. The common thread is that all of them treat partition count as a design parameter set upfront, not a runtime knob.

LinkedIn runs 100+ clusters, 4,000+ brokers, 100,000+ topics, and 7+ million partitions handling more than 7 trillion messages per day (2023 figures). They run Cruise Control on every cluster for partition rebalancing and goal-based optimisation. They built custom tooling to emit per-partition throughput metrics specifically because Kafka does not expose them natively, and because partition-count-based load distribution was wrong at their scale.

Netflix Keystone handles approximately 2 trillion messages per day and uses 0.5–1 MB/s per partition as their planning constant. Their architectural principle for partition scaling and configuration changes is instructive: they treat them as simulated failures, failing traffic over to a new cluster rather than reconfiguring the live one. A max of ~200 nodes per cluster; beyond that, they provision a new cluster.

Shopify on Black Friday / Cyber Monday 2024 hit 66 million messages per second peak. Their engineering team explicitly identified partition increases, not larger batch sizes, as the lever to maintain ETL data freshness during traffic spikes. Their CDC architecture creates one Kafka topic per logical database table, partitioned by primary key, deliberately decoupling source database sharding from consumer-side partitioning.

Uber operates at trillions of messages per day. They hit the hard consumer-parallelism limit when their payment processing consumers could handle approximately one event per second per partition. At 1,000 events/s throughput requirements, that meant 1,000 partitions per topic, and the ZooKeeper-era 200K cluster cap bounded them to roughly 200 topics per cluster at that scale. Their solution (uForwarder) was a push-based gRPC consumer proxy that decoupled consumer concurrency from partition count, anticipating what KIP-932 share groups formalise.

Cloudflare runs 14 clusters with ~330 nodes. Their public partition lessons: back-of-the-napkin throughput math before a topic reaches production is not optional; partition skew is real and they have had incidents from it; and Snappy compression gave them 2.25× ingress reduction on their highest-throughput topic without increasing producer or consumer CPU.

How Kpow helps you manage Kafka topic partitions

Partition decisions span topic creation, ongoing monitoring, and operational intervention when something is off. Kpow covers each of those phases.

Topic creation with guardrails. Kpow’s topic creation UI lets you set partition count, replication factor, and all topic-level configuration before a topic reaches production. It exposes cluster-default configuration for each option inline and shows the top-5 most common values set across your cluster, which makes it easier to stay consistent with your existing topology. If you want to review a configuration option before setting it (for example, cleanup.policy or compression.type), the documentation accordion surfaces the config item’s type, default, and allowed values without leaving the form.

If you prefer not to grant Kpow direct mutation access, the Topic Create form generates the equivalent kafka-topics.sh command reactively as you fill it in, which you can pipe directly into your cluster tooling.

Configuration management across topics. Kpow’s topic configuration view gives you a filterable table of configuration across all topics in a cluster. You can filter by topic name, config key, source (dynamic vs. default), importance, and whether the item is read-only. Editing a config item brings up the same inline documentation, making it straightforward to review the implications of a change before committing it.

Under-replicated partition detection. Kpow tracks under-replicated partitions (URPs) at both the broker and topic level. Its URP calculation correctly identifies partitions where the in-sync replica count is below the configured replication factor even when a broker is offline and not visible to the AdminClient, a subtle but important distinction for accurate cluster health visibility. URP totals appear on both the Brokers and Topics pages; if the count is above zero, a detailed table lists every affected topic and partition.

Partition replica management. From the Topic Details page, Kpow lets you elect preferred or unclean leaders for individual partitions and manage partition reassignments. During a reassignment, you can monitor progress and cancel an in-flight operation from the Reassignment tab. Full cluster-wide reassignment is on the product roadmap.

Consumer group lag at partition granularity. The consumer group topology view in Kpow shows lag at the group, host, topic, and partition level. This matters because per-topic aggregate lag masks partition skew, a common failure mode where one partition is saturated while others are idle, and the topic-level metric looks acceptable. Kpow also surfaces lag for empty consumer groups, which is relevant when a poison message has taken a consumer group offline and you need to reset offsets before restarting.

Offset management. When you need to reset, clear, or skip offsets at the partition level, including after a partition count change or a consumer migration to a new topic version, Kpow handles this from the Consumers workflow UI. Offset mutations are scheduled and execute once the consumer group reaches the EMPTY state, which prevents accidental offset resets on running consumers.

If you want to see how Kpow fits into your Kafka operations workflow, you can start a free 30-day trial.

Summary

Partition count is a design-time decision with long operational consequences. The practical guidance distills to:

Size for one to two years of peak throughput using the throughput formula with measured per-partition numbers from your actual cluster.
Round up to a number with good divisor properties; avoid primes.
On keyed topics, over-provision rather than alter. Adding partitions breaks per-key ordering permanently.
RF=3, min.insync.replicas=2, acks=all is the standard baseline. Treat replication factor changes as maintenance operations.
The ZooKeeper-era 200K partition cluster cap is a KRaft-era non-issue. The resource constraints (file descriptors, mmap limits, replication CPU) that motivated conservative rules are not.
Monitor lag at the partition level. Topic-level aggregate lag hides partition skew.
When a keyed topic has genuinely outgrown its partition count, create a new topic and dual-write rather than altering.

The Complete Guide to Kafka Change Data Capture (CDC)

Factor House — Fri, 08 May 2026 00:00:00 GMT

Change Data Capture (CDC) tracks row-level changes in a database and publishes them as events. When paired with Apache Kafka, CDC turns your databases into real-time event streams without modifying application code or adding polling overhead.

This guide covers CDC fundamentals, compares the dominant implementation patterns, walks through a working PostgreSQL-to-Kafka setup, and shows where CDC fits in a broader data mesh architecture.

Key takeaways

CDC captures row-level changes from databases and streams them as events to Kafka, eliminating the latency and resource waste of traditional batch ETL.
Log-based CDC reads the database transaction log directly, capturing all changes including deletes with near-zero impact on production workloads.
Debezium and Kafka Connect JDBC are the two main CDC patterns for Kafka; Debezium is the preferred choice for most production use cases that require complete change capture and low latency.
CDC fits naturally in enterprise data mesh architectures, letting domain teams publish database changes as Kafka events without modifying application code.
Keeping CDC pipelines running reliably at scale requires operational tooling: Kpow provides real-time visibility into connector health, consumer lag, and automatic restarts across your entire Kafka environment.

What is Kafka change data capture (CDC)?

Traditional data integration relies on batch ETL: extract everything from the source, transform it, load it into the target. This approach has well-known problems. It introduces latency (hours or days), wastes resources by re-reading unchanged data, and puts load on the source database during extraction windows.

CDC solves these problems by capturing only the rows that changed (inserts, updates, deletes) and streaming them as individual events. The source database is read through its internal replication mechanism, typically the write-ahead log (WAL), so the impact on production workloads is minimal.

In practice, CDC gives you:

Sub-second propagation of database changes to downstream systems
A complete, ordered history of every mutation, useful for audit trails and temporal queries
Decoupled producers and consumers, since changes flow through Kafka topics rather than direct database connections
The ability to rebuild derived datastores by replaying the change log from a known offset

For real-time data pipelines, CDC eliminates the gap between “data at rest” and “data in motion.” Your analytical systems, search indexes, caches, and microservices all see changes as they happen rather than in delayed batches.

Log-based versus query-based CDC for Kafka

There are two fundamental approaches to capturing changes from a database.

Log-based CDC reads the database’s internal transaction log (WAL in PostgreSQL, binlog in MySQL, redo log in Oracle). This is the preferred method. The database already writes these logs for crash recovery and replication, so CDC reads them with near-zero overhead. Log-based CDC captures all changes including deletes, preserves the exact ordering of transactions, and works without schema modifications.

Query-based CDC periodically polls the source table using SQL queries, typically filtering by a last_modified timestamp column. This is simpler to set up but has significant limitations: it cannot reliably detect deletes, requires a timestamp or incrementing column on every tracked table, misses rapid intermediate changes between polls, and puts read load on the source database.

For most production use cases, log-based CDC is the correct choice. Query-based CDC can work for simple, append-only tables where deletes are not a concern.

Choosing a Kafka CDC connector

The two most common ways to implement CDC with Kafka are Debezium (log-based) and the Kafka Connect JDBC Source Connector (query-based). They serve different purposes.

Debezium CDC for Kafka

Debezium is an open-source CDC platform built on Kafka Connect. It reads database transaction logs directly and produces change events to Kafka topics.

Key characteristics:

Reads WAL/binlog/redo logs directly; no schema changes required on source tables
Captures inserts, updates, and deletes with full before/after images of each row
Provides exactly-once semantics when combined with Kafka’s transactional features
Supports PostgreSQL, MySQL, MongoDB, SQL Server, Oracle, Db2, and others
Emits a structured envelope containing the operation type, before state, after state, source metadata, and transaction info
Handles initial snapshots of existing data before switching to log streaming

Kafka Connect JDBC

The JDBC Source Connector uses SQL queries to poll for changes at a configured interval.

Key characteristics:

Requires a timestamp column, an incrementing column, or both to detect changes
Cannot capture deletes (the row is gone before the next poll)
Simpler setup: no database-level replication configuration needed
Higher latency, bounded by the poll interval
Puts periodic query load on the source database
Works with any JDBC-compatible database

Debezium versus Kafka Connect JDBC

Use Debezium when you need complete change capture (including deletes), low latency, minimal source database impact, and accurate ordering of changes. This covers most CDC use cases.

Use the JDBC Source Connector when you have a simple append-only or update-only table, cannot configure database-level replication permissions, or need a quick prototype before investing in log-based CDC infrastructure.

Working Example: PostgreSQL to Kafka with Debezium

The following walks through a complete setup using PostgreSQL’s logical replication and Debezium.

Step 1: Configure PostgreSQL for Logical Replication

Edit postgresql.conf:

wal_level = logical
max_replication_slots = 4
max_wal_senders = 4

Create a replication user and grant permissions:

CREATE ROLE debezium WITH REPLICATION LOGIN PASSWORD 'dbz_password';
GRANT USAGE ON SCHEMA public TO debezium;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO debezium;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO debezium;

Create a publication for the tables you want to track:

CREATE PUBLICATION dbz_publication FOR TABLE orders, customers, products;

Step 2: Deploy Kafka Connect with Debezium

A docker-compose.yml snippet for the Kafka Connect worker with the Debezium PostgreSQL connector plugin:

kafka-connect:
  image: debezium/connect:2.5
  ports:
    - "8083:8083"
  environment:
    BOOTSTRAP_SERVERS: kafka:9092
    GROUP_ID: connect-cluster
    CONFIG_STORAGE_TOPIC: connect-configs
    OFFSET_STORAGE_TOPIC: connect-offsets
    STATUS_STORAGE_TOPIC: connect-status
    KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
    VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
  depends_on:
    - kafka
    - postgres

Step 3: Register the Debezium Connector

Submit the connector configuration via the Kafka Connect REST API:

{
  "name": "pg-cdc-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres",
    "database.port": "5432",
    "database.user": "debezium",
    "database.password": "dbz_password",
    "database.dbname": "app_db",
    "topic.prefix": "cdc",
    "schema.include.list": "public",
    "table.include.list": "public.orders,public.customers,public.products",
    "publication.name": "dbz_publication",
    "slot.name": "debezium_slot",
    "plugin.name": "pgoutput",
    "publication.autocreate.mode": "filtered",
    "snapshot.mode": "initial",
    "tombstones.on.delete": true,
    "key.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "key.converter.schemas.enable": false,
    "value.converter.schemas.enable": false
  }
}

curl -X POST http://localhost:8083/connectors \
  -H "Content-Type: application/json" \
  -d @pg-cdc-connector.json

Step 4: Verify the Pipeline

Check the connector status:

curl -s http://localhost:8083/connectors/pg-cdc-connector/status | jq .

You should see the connector and its tasks in RUNNING state. Insert a row into the source table:

INSERT INTO orders (customer_id, product_id, quantity, total)
VALUES (1, 42, 3, 149.97);

Consume from the CDC topic to see the change event:

kafka-console-consumer --bootstrap-server kafka:9092 \
  --topic cdc.public.orders \
  --from-beginning \
  --max-messages 1

The resulting event will contain an envelope with op: "c" (create), the after field with the full row state, and source metadata including the LSN (Log Sequence Number), transaction ID, and timestamp.

A Debezium change event for an insert looks like this:

{
  "before": null,
  "after": {
    "id": 1001,
    "customer_id": 1,
    "product_id": 42,
    "quantity": 3,
    "total": 149.97
  },
  "source": {
    "version": "2.5.0.Final",
    "connector": "postgresql",
    "name": "cdc",
    "ts_ms": 1704067200000,
    "db": "app_db",
    "schema": "public",
    "table": "orders",
    "lsn": 234567890,
    "txId": 5678
  },
  "op": "c",
  "ts_ms": 1704067200123
}

For updates, op is "u" and both before and after are populated. For deletes, op is "d" and after is null.

CDC events in Kafka: structure and consumption

Every change Debezium captures is published to Kafka as a structured CDC event. Understanding the shape of these events is essential for building reliable consumers.

Each CDC event in Kafka carries an envelope with four key fields:

op indicates the operation type: "c" for create (insert), "u" for update, "d" for delete, and "r" for a read during the initial snapshot.
before contains the full row state before the change. This is null for inserts and snapshot reads.
after contains the full row state after the change. This is null for deletes.
source provides metadata from the database, including the connector name, table, transaction ID, log sequence number (LSN), and commit timestamp.

Consumers that process CDC events in Kafka typically follow one of two patterns. Filter-and-apply consumers inspect the op field and route each event to the appropriate handler (insert, update, or delete logic). Event-sourcing consumers treat the full stream as an ordered log and reconstruct state by replaying events from a known offset, making it straightforward to rebuild derived data stores after failures or schema migrations.

When publishing CDC events to Kafka, topic naming conventions matter for downstream discoverability. A common pattern is {prefix}.{schema}.{table}, such as cdc.public.orders. This makes it easy for consumers to subscribe to specific tables and for schema registry tooling to apply per-topic compatibility rules.

Kafka CDC in an enterprise data mesh architecture

In a data mesh, domain teams own their data products and expose them through well-defined interfaces. CDC fits naturally into this model because it allows teams to publish their database changes as domain events without building custom event-producing application code.

A typical architecture looks like this:

Each domain team runs a Debezium connector for its database. The change events land in Kafka topics namespaced by domain (e.g., cdc.orders.*). A schema registry enforces contracts on the event format. Downstream teams consume from these topics to build analytics stores, search indexes, materialized views, audit logs, or ML feature stores.

This decouples data producers from consumers entirely. The Orders team does not need to know that the Analytics team reads their changes, and vice versa. Kafka acts as the durable, replayable log that connects the domains.

In practice, you also need governance over topic naming, schema evolution rules, data classification, and access controls. CDC topics often carry sensitive data (customer PII, payment details), so encryption, RBAC, and audit logging are essential at the Kafka layer.

Common Kafka CDC pitfalls

Replication slot growth. If a consumer falls behind or a connector stops, PostgreSQL retains WAL segments for the replication slot indefinitely. This can fill the disk. Monitor pg_replication_slots and set max_slot_wal_keep_size as a safety limit.

Schema evolution. When a source table schema changes (added columns, type changes), the CDC events change shape. Use a schema registry with compatibility checks (backward, forward, or full) to prevent breaking downstream consumers. Debezium integrates with Confluent Schema Registry and Apicurio.

Snapshot handling. On first startup, Debezium takes an initial snapshot of the existing data. For large tables, this can take hours and produce a burst of messages. Plan capacity accordingly and consider using snapshot.mode=no_data if you only need changes going forward.

Ordering guarantees. Debezium produces events in commit order within a single table partition. If you need strict ordering across tables or across partitions of the same table, you need to handle this in your consumer logic or use single-partition topics (which limits throughput).

Tombstone records. Debezium emits a tombstone (null value) after a delete event by default. This is required for Kafka log compaction to work correctly, removing the key entirely from the compacted topic. Make sure your consumers handle null values.

Monitoring Kafka CDC pipelines with Kpow

CDC connectors are long-running processes, and they fail silently more often than you would like. A connector task might enter a FAILED state because of a WAL slot issue, a schema change, or a network partition. Without monitoring, these failures go unnoticed until downstream systems start serving stale data.

Kpow gives you real-time visibility into Kafka Connect clusters alongside the rest of your Kafka infrastructure.

Reviewing Kafka Connect connector and task status in Kpow

Specifically for CDC pipelines, Kpow helps with:

Connector and task health. Kpow surfaces the status of every connector and task (RUNNING, PAUSED, FAILED, UNASSIGNED) in a single view. You can set up Prometheus alerts via Kpow’s metrics endpoint to fire when a connector task enters an error state, so your team gets notified immediately rather than discovering the failure through downstream symptoms.

Consumer lag tracking. The CDC topics produced by Debezium are consumed by downstream services. Kpow tracks consumer group lag across all topics and partitions, letting you see at a glance whether any consumer is falling behind. For CDC, rising lag means your derived datastore is diverging from the source of truth.

Automatic connector restarts. Kpow can automatically restart failed connectors at configurable intervals. You specify which connectors to auto-restart (by exact name or wildcard pattern), and Kpow monitors them at one-minute intervals. All restart actions are logged in the audit trail and can be forwarded to Slack for team visibility.

Operational controls. From Kpow’s UI or OpenAPI-based REST API, you can pause, resume, restart, or delete connectors, inspect task stack traces when errors occur, and view or edit connector configurations. This gives your on-call engineers a single interface for triaging CDC issues without needing to hit the Kafka Connect REST API directly.

Multi-cluster support. If you run CDC connectors across multiple Kafka clusters or environments (dev, staging, production), Kpow manages them all from a single instance with role-based access controls and audit logging.

CDC is one of those patterns where the setup is the easy part. Keeping it running reliably at scale, across multiple source databases and dozens of connectors, is where the operational complexity lives. Kpow reduces that complexity by consolidating connector health, consumer lag, topic throughput, and cluster metrics into a single tool.

You can try Kpow with a free 30 day trial or explore the full documentation at docs.factorhouse.io.

Kafka CDC FAQ

Is a managed CDC or custom Kafka consumers less work?

Managed CDC tools like Debezium handle snapshotting, ordering, schema changes, and replication slot management out of the box. Custom Kafka consumers require you to implement all of this logic yourself. For most teams, managed CDC is significantly less work to operate at scale.

Is it possible to implement CDC without managing Kafka infrastructure?

Yes. Managed Kafka services such as Confluent Cloud, Amazon MSK, or Aiven for Apache Kafka handle cluster provisioning, scaling, and operations for you. Debezium also offers a cloud service. This lets teams focus on CDC pipeline configuration rather than infrastructure management.

How to deal with “schema-change runbook and pre-deploy contract checks”?

Register every CDC topic with a schema registry and enforce compatibility rules (backward or full) before deploying schema changes. This ensures downstream consumers are notified of changes in advance. Pair this with a runbook covering steps to update connectors and consumers when breaking changes are unavoidable.

Enhanced Under-Replicated Partition Detection in Kpow

Factor House — Wed, 06 May 2026 00:00:00 GMT

Overview

In a distributed system like Apache Kafka, data is partitioned and replicated across multiple brokers to ensure high availability and fault tolerance. A partition is considered an under-replicated partition (URP) when the number of in-sync replicas (ISRs) falls below the configured replication factor. This scenario can arise from various issues, including broker failures, network partitions, or high load on specific brokers.

The presence of URPs is a significant concern as it indicates a degradation in your topics’ fault tolerance. If another broker fails before the cluster recovers, you risk permanent data loss. A key challenge in Kafka management is accurately detecting these URPs in real-time, especially during common operational events like a broker failure. Standard monitoring methods can sometimes lag, creating a temporary but dangerous blind spot where a cluster appears healthy even though its resilience has been compromised.

This is precisely the challenge that Kpow’s enhanced URP detection is designed to solve. By providing a more accurate and immediate assessment of your cluster’s true fault tolerance, this feature delivers significant benefits. It gives you the confidence to act quickly on reliable data, the ability to proactively mitigate risks before they escalate, and ultimately, the power to ensure the resilience and durability of your critical data pipelines.‍

Enhancement of URP detection is implemented in Release 94.5. For an overview of all the changes, check out the release note: Release 94.5: New Factor House docs, enhanced data inspection, and URP & KRaft improvements.Release 94.5: New Factor House docs, enhanced data inspection, and URP & KRaft improvements.

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Start your free 30-day trial or explore our live multi-cluster demo environment to see Kpow in action.

Enhanced calculation for more accurate health monitoring

Ensuring the fault tolerance of your Kafka clusters requires a precise and accurate count of under-replicated partitions. A challenging but common operational scenario can arise where a broker becomes unavailable, yet the overall cluster health status does not immediately reflect this change. This can mask a critical degradation in data durability, leading operations teams to believe their cluster is healthier than it actually is. Making decisions based on this incomplete information can delay necessary interventions.

To provide a more reliable and trustworthy view, Kpow has enhanced its calculation for under-replicated partitions. Instead of calculating replication status by iterating through each broker—a method that can be incomplete if a broker is offline and unreachable—our new calculation iterates directly through every topic-partition defined in the cluster.

This partition-centric approach provides a more comprehensive and authoritative view of the cluster’s state. It is precisely this change that allows the system to correctly detect partitions with fewer in-sync replicas than the configured replication factor, even when brokers are offline and not reported by the AdminClient.

This enhancement ensures that Kpow’s health monitoring is a precise reflection of your cluster’s real-time condition. It gives you the confidence to trust the metrics you see and act decisively to investigate and resolve replication issues, thereby maintaining a resilient and robust system.

Surfacing URP details in Kpow

This vital health information, now powered by our more accurate calculation, continues to be clearly presented on both the Brokers and Topics pages of the user interface. We’ve retained this dual perspective as it remains essential for diagnosing problems from different angles—whether you’re investigating a single problematic broker or assessing the health of a critical application’s topic.

On both pages, summary statistics displays the total number of under-replicated partitions. If this count is greater than zero, it serves as an immediate visual alert. A detailed table automatically appears, listing all affected topics along with their specific URPs. This allows you to quickly identify which topics are at risk and gather the necessary context to restore full replication.

On the Brokers Page:

On the Topics Page:

To further strengthen monitoring and alerting capabilities, new Prometheus metrics have been introduced to track under-replicated partitions. These metrics integrate seamlessly with your existing observability stack and provide more granular insights for automated alerting and historical trend analysis:

broker_urp: The total number of under replicated topic partitions belonging to this broker.
topic_urp: The total number of under replicated partitions belonging to this topic.
topic_urp_total: The total number of under replicated partitions of all topics in the Kafka cluster.

Conclusion

Accurate and timely detection of under-replicated partitions (URPs) is fundamental to maintaining a resilient and reliable Apache Kafka cluster. With its enhanced calculation, Kpow provides a more precise and immediate understanding of your cluster’s health by correctly identifying replication issues, particularly in scenarios involving broker failures. This enhanced detection, combined with detailed visibility in the Kpow UI and new Prometheus metrics for automated alerting, empowers you to proactively address replication issues, mitigate the risk of data loss, and ensure the continuous high performance of your real-time data pipelines. This feature update reaffirms Kpow’s commitment to providing comprehensive and intuitive tooling for Kafka management and monitoring.

How to monitor Kafka consumer lag: 5 options

Gaurav Bhatt — Wed, 06 May 2026 00:00:00 GMT

Kafka consumer lag is one of the most common operational signals teams rely on to understand whether their event-driven systems are healthy. It is also one of the most common misunderstandings. When teams refer to consumer lag operationally, it means the following to them:

Consumer lag represents how far a consumer group is behind the latest data in a Kafka topic.
Lag is not inherently bad, but unbounded or growing lag usually points to a system issue.
Monitoring lag effectively requires understanding partitions, offsets, and consumer group behavior.
There are multiple ways to monitor consumer lag, ranging from built-in tooling to dedicated monitoring products.
Reducing lag often involves addressing downstream constraints, not Kafka itself.

If you are seeing lag increase, stay flat, or behave unexpectedly, it is not always obvious whether you are looking at a real problem, a temporary backlog, or a misleading metric. This article explains what Kafka consumer lag is, why it happens, and several practical ways to monitor it.

What is Kafka consumer lag?

Kafka consumer lag is the difference between:

the latest offset written to a partition (the log end offset), and
the offset most recently committed by a consumer group for that partition.

In practical terms, lag indicates how many records a consumer group has yet to process. If a topic partition has a latest offset of 10,000 and the consumer group has committed offset 9,500, the lag for that partition is 500.

Lag is tracked per partition, but it is often aggregated to give a topic-level or consumer-group-level view.

It is important to note that lag is not a measure of time. A lag of 10,000 messages could represent milliseconds of work or several hours, depending on message size, processing cost, and consumer throughput.

Causes of consumer lag

Consumer lag occurs whenever producers write data faster than consumers can process it. This can happen for many reasons, including the following.

Insufficient consumer capacity

If there are fewer consumer instances than partitions, some partitions cannot be processed in parallel and will inevitably fall behind. Lag can also build up when consumers are under-provisioned in terms of CPU, memory, or I/O, even if the number of instances matches the partition count. In these cases, consumers may technically be healthy but unable to sustain the required throughput.

Slow downstream systems

Kafka promotes decoupling, but consumers frequently interact with external systems such as databases, APIs, or object storage. When these downstream dependencies become slow or intermittently unavailable, consumer processing time increases and messages spend longer waiting to be handled. Over time, this reduced throughput shows up as growing lag, even if the Kafka cluster itself is operating normally.

Rebalances and restarts

Consumer group rebalances temporarily pause consumption while partitions are reassigned. Occasional rebalances are expected, but repeated rebalances—often caused by frequent deployments, crashes, or unstable group membership—can significantly reduce effective processing time. In these situations, lag may fluctuate or accumulate despite sufficient consumer capacity.

Uneven partition load

Kafka distributes records by partition, but traffic is rarely uniform across them. A small number of partitions may receive a disproportionate share of messages, creating so-called “hot partitions.” Even when overall throughput appears healthy, lag can accumulate on these partitions and increase end-to-end processing latency for downstream systems.

Intentional backlog

In some scenarios, consumer lag is expected and planned for. Batch-oriented consumers, replay jobs, or backfill processes often operate behind the head of the log by design. In these cases, lag reflects scheduling or workflow decisions rather than a system failure and should be interpreted in that context.

Benefits of consumer lag monitoring

Monitoring consumer lag provides early signals that your system is under pressure. Unlike error logs or alerts triggered after a failure, lag typically increases before users notice an issue.

Effective lag monitoring helps you:

detect throughput mismatches between producers and consumers
identify which consumer groups are affected
understand whether lag is temporary or growing over time
correlate processing delays with deployments or infrastructure changes

When combined with partition-level visibility and operational context, lag metrics can also help distinguish between expected backlogs and genuine processing bottlenecks. For example, a brief spike during a rebalance may be benign, while steadily increasing lag after a deployment can point to a configuration or capacity issue. Used correctly, lag becomes a diagnostic tool rather than just a number on a dashboard, supporting investigation and decision-making instead of generating noise.

How to monitor Kafka consumer lag

There are several ways to monitor consumer lag, each with different levels of effort and visibility.

1. Using a Kafka UI like Kpow

Tools such as Kpow provide consumer group visibility out of the box, including per-partition lag, historical trends, rebalance activity, and group state transitions.

Kpow connects directly to a Kafka cluster and automatically discovers brokers, topics, partitions, and consumer groups. A key differentiator of Kpow is that it generates metrics internally without relying on JMX. This means the data it exposes is consistent across Kafka resources, avoiding the gaps and inconsistencies that often arise from JMX-based scraping pipelines. For teams integrating Kafka observability into Grafana dashboards, this makes Kpow a high-fidelity telemetry source - the approach is described in detail in Beyond JMX: Supercharging Grafana Dashboards with High-Fidelity Metrics.

In addition to lag visualisation, Kpow provides operational context that is difficult to reconstruct from metrics alone, such as rebalance history and consumer group lifecycle events. This helps distinguish between temporary backlogs caused by rebalances and sustained lag driven by throughput constraints.

Kpow also exposes administrative capabilities commonly used during troubleshooting, including consumer group offset inspection and reset operations. These actions are available directly in the UI and documented in the Kpow management guide.

You can see an example of consumer group exploration in the publicly available demo instance.

For teams that want to explore this approach, Kpow is available with a free 30-day trial.

2. Using Kafka’s command-line tools

Kafka ships with built-in scripts for inspecting consumer group offsets, such as:

kafka-consumer-groups.sh \
    --bootstrap-server broker:9092 \
    --describe \
    --group my-consumer-group

This output shows committed offsets, log end offsets, and lag per partition. While useful for ad-hoc inspection, it is not well suited to continuous monitoring or historical analysis.

3. Using JMX metrics

Kafka exposes consumer metrics via JMX, including lag-related measurements. Teams often scrape these metrics into Prometheus and visualize them in Grafana.

This approach offers flexibility but requires careful configuration, metric selection, and alert tuning. It also shifts the responsibility for correctness and interpretation onto the engineering team.

4. Building custom consumer instrumentation

Some teams instrument lag tracking directly within their consumers by periodically querying committed offsets and log end offsets using the Kafka Admin API.

This can provide fine-grained control, but it introduces additional complexity and maintenance overhead. In practice, most teams reserve this approach for specialized scenarios.

5. Cloud-managed monitoring integrations

Managed Kafka offerings such as Amazon MSK integrate with CloudWatch, exposing offset and lag-related metrics at the service level. These integrations can be helpful but often lack the granularity needed for diagnosing consumer behavior at the application level.

Best practices for monitoring consumer lag

Monitor trends, not just absolute values

A stable lag that remains constant over time may be acceptable, depending on your use case. A steadily increasing lag almost always requires investigation. Set up trend-based alerts that fire when lag has been growing continuously for a defined period - for example, increasing for more than five minutes - rather than alerting on any non-zero value. This reduces alert fatigue while catching genuine problems early.

Watch lag by partition

Aggregate lag can hide partition-level issues. A single slow or hot partition can impact processing latency even if total group lag looks manageable. Ensure your monitoring system exposes per-partition lag so you can identify which partitions are falling behind and investigate the cause - whether that is skewed message distribution, a slow consumer instance, or an upstream producer spike.

Correlate lag with deployments and rebalances

Sudden changes in lag often coincide with consumer restarts, rebalances, or configuration changes. Observability is strongest when metrics are combined with deployment events and group state changes. Tools that surface rebalance history alongside lag trends make it significantly easier to diagnose whether a lag spike is transient or structural.

Avoid alerting on every spike

Short-lived spikes are common during rebalances or traffic bursts and are often self-correcting within seconds. Alerts should focus on sustained lag growth rather than momentary increases. Using a rolling window or rate-of-change metric - rather than a raw threshold - typically produces more actionable alerts. Reserve high-priority alerts for situations where lag exceeds a threshold and remains elevated for a meaningful duration.

How to reduce consumer lag

Scale consumers horizontally

Adding consumer instances increases parallelism, as long as there are enough partitions to support it. Each partition in Kafka can only be assigned to one consumer within a group, so the maximum useful parallelism is bounded by the partition count. If you have already scaled to match partition count and lag persists, the bottleneck is likely in processing logic or downstream dependencies rather than consumer capacity.

Optimize consumer processing

Profiling message handling, batching operations, and reducing synchronous calls can significantly improve throughput. Common gains come from reducing per-message latency through asynchronous I/O, pre-fetching lookups, or processing records in micro-batches rather than one at a time. Even modest improvements in per-message processing time can translate to substantial lag reduction at scale.

Tune Kafka fetch and commit settings

Configuration such as max.poll.records, fetch.min.bytes, and commit frequency can affect how efficiently consumers process data. Increasing max.poll.records allows consumers to fetch more messages per poll cycle, which can improve throughput in high-volume scenarios. However, increasing this value also extends the time between commits, so it should be balanced against your at-least-once delivery requirements and processing latency targets.

Address downstream bottlenecks

Lag is often a symptom of slow databases, APIs, or storage systems. Improving these dependencies - through connection pooling, caching, batching writes, or upgrading infrastructure - can have a larger impact on lag reduction than any Kafka-side tuning. Always profile your consumer to understand where time is being spent before making changes.

Common misconceptions about consumer lag

Consumer lag is widely referenced, but often misinterpreted. A few common misunderstandings are worth addressing directly.

Lag is not a reliability indicator on its own

High lag does not automatically mean messages are being lost or skipped. Kafka retains data independently of consumer progress. A consumer group can fall behind and still process every record correctly once it catches up, provided retention limits are not exceeded.

Conversely, low or zero lag does not guarantee correctness. Consumers can commit offsets prematurely, encounter silent processing failures, or drop messages downstream while still appearing “caught up” from Kafka’s perspective.

Lag should be interpreted alongside error rates, commit behaviour, and downstream outcomes.

Lag can decrease without work being done

Lag shrinking does not always mean consumers are processing records faster. During rebalances, offset resets, or consumer group recreation, committed offsets can move forward abruptly. This can make lag appear to recover even though records were skipped or reassigned.

If lag drops suddenly without a corresponding increase in consumer throughput, it is worth verifying offset commit patterns and rebalance history.

Aggregate lag hides operational detail

Most dashboards default to total or average lag across a topic. While useful at a glance, these aggregates can obscure issues in individual partitions or consumer instances. A single partition with sustained lag is often enough to increase end-to-end processing delay for dependent systems. Monitoring at the partition level is essential for accurate diagnosis.

Closing thoughts

Kafka consumer lag is a simple metric that carries a lot of operational context. Understanding what it represents, why it changes, and how to monitor it effectively can make the difference between reactive firefighting and proactive system management.

If you want to explore consumer lag monitoring with a dedicated Kafka UI, you can try Kpow with a free 30-day trial.

Integrate Kpow with WarpStream

Jaehyeon Kim — Wed, 06 May 2026 00:00:00 GMT

Overview

WarpStream is a Kafka-compatible, BYOC (Bring Your Own Cloud) streaming data platform built directly on top of object storage (such as AWS S3). It delivers high durability, exceptional elasticity, and significant cost savings by eliminating the need for local disks and traditional broker replication.

WarpStream operates on a highly efficient split architecture: a Control Plane (managed by WarpStream) handles cluster lifecycles, IAM, and metadata, while your Data Plane (the Agents deployed in your own cloud) handles the actual streaming traffic.

While WarpStream eliminates the infrastructure headache of managing Kafka brokers, engineering teams still need a dedicated, intuitive interface to monitor, explore, and manage their Kafka data in real-time. That is where Kpow comes in. By leveraging standard Kafka protocols, Kpow connects directly to your Data Plane endpoints and BYOC Schema Registry, delivering a unified, single-pane-of-glass experience without the need for proprietary plugins or sidecars.

Kpow connects natively to a wide range of Kafka vendors and managed service providers. See our Kafka Providers documentation to learn more.

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Start your free 30-day trial or explore our live multi-cluster demo environment to see Kpow in action.

Prerequisites

To connect Kpow to WarpStream, you must have the following resources provisioned:

A running WarpStream Data Plane: Your WarpStream Agents must be deployed and running in your cloud environment.
Kafka Connection Details: Your Virtual Cluster Bootstrap URL (exposed via your Data Plane, not the Control Plane URL).
Kafka Authentication: Your SASL username and password (generated in the WarpStream Console).
WarpStream Schema Registry (Optional): The Schema Registry URL and its associated credentials.
A Kpow Enterprise License: Get a free 30-day trial.

Quick Start

The fastest way to connect Kpow to WarpStream is using Docker.

Run the following command in your terminal, replacing the placeholder values with your specific cluster details:

docker run -p 3000:3000 -d --name kpow \
  --env ENVIRONMENT_NAME="WarpStream Kafka" \
  --env BOOTSTRAP="<WARPSTREAM_VIRTUAL_CLUSTER_BOOTSTRAP_URL>" \
  --env SECURITY_PROTOCOL="SASL_SSL" \
  --env SASL_MECHANISM="PLAIN" \
  --env SASL_JAAS_CONFIG='org.apache.kafka.common.security.plain.PlainLoginModule required username="<SASL_USERNAME>" password="<SASL_PASSWORD>";' \
  --env REPLICATION_FACTOR="1" \
  --env NUM_PARTITIONS="1" \
  --env LICENSE_ID="<LICENSE_ID>" \
  --env LICENSE_CODE="<LICENSE_CODE>" \
  --env LICENSEE="<LICENSEE>" \
  --env LICENSE_EXPIRY="<LICENSE_EXPIRY>" \
  --env LICENSE_SIGNATURE="<LICENSE_SIGNATURE>" \
  factorhouse/kpow:latest

Notes

License details: The license details can be obtained from your signup email or via the Factor House license portal.
Cluster Authentication: This guide demonstrates connecting using SASL/PLAIN, which is the standard for credentials generated in the WarpStream Console. Kpow also fully supports SCRAM, Mutual TLS (mTLS), and OAuthBearer for cluster authentication. For a detailed walkthrough, see the WarpStream provider guide.
Internal Topic Configuration (Crucial): Because WarpStream’s diskless architecture writes directly to object storage rather than utilizing traditional broker replication, the replication factor acts as a logical durability configuration. You must configure Kpow to use REPLICATION_FACTOR="1". We also highly recommend setting NUM_PARTITIONS="1" to conserve your WarpStream logical partition limits, as Kpow’s internal telemetry topics do not require multi-partition compute fanout.
Authorization configuration: For brevity, Kpow authorization configuration has been omitted. See Simple Access Control to enable necessary user actions.

Once the container starts, open a browser and navigate to http://localhost:3000. You will immediately see your WarpStream topics, consumer groups, and cluster telemetry.

Ecosystem Integration

If you use WarpStream’s BYOC Schema Registry, you can seamlessly integrate it into Kpow by appending the following environment variables to your deployment command.

WarpStream Schema Registry

WarpStream’s Schema Registry requires Basic Authentication (USER_INFO).

By default, Kpow uses an optimized “Version 2” observation strategy that queries the bulk schema listing endpoint (/schemas). Depending on your WarpStream BYOC version and deployment, this bulk endpoint may not be implemented (which will return a 404 Not Found error in Kpow’s logs). If you experience this, you may need to instruct Kpow to fall back to the legacy “Version 1” strategy (/subjects) by setting SCHEMA_REGISTRY_OBSERVATION_VERSION=“1”. See the Schema Registry configuration docs for more details.

  --env SCHEMA_REGISTRY_NAME="WarpStream Schema Registry" \
  --env SCHEMA_REGISTRY_URL="<SCHEMA_REGISTRY_URL>" \
  --env SCHEMA_REGISTRY_AUTH="USER_INFO" \
  --env SCHEMA_REGISTRY_USER="<SCHEMA_USERNAME>" \
  --env SCHEMA_REGISTRY_PASSWORD="<SCHEMA_PASSWORD>" \
  --env SCHEMA_REGISTRY_OBSERVATION_VERSION="1"

Managed Data Pipelines

WarpStream’s “Managed Data Pipelines” (their Kafka Connect alternative) is not currently supported by Kpow.

Production Deployment

When you are ready to move from a local Docker test to a production deployment, we recommend the following paths:

Kubernetes

For deploying Kpow to Kubernetes clusters, we recommend using our official Helm Charts. You can store your Aiven ca.pem file as a Kubernetes Secret and securely mount it into the Kpow pod.

Bare Metal / VM

If you prefer running Kpow directly on a Virtual Machine, you can download the Kpow JAR file.

Kpow JAR Quickstart

Conclusion

Kpow provides a powerful, single pane of glass view into your WarpStream Data Plane. By using standard Kafka protocols and respecting WarpStream’s unique diskless architecture configurations, you can unify your virtual clusters and Schema Registry environments in minutes.

Explore these features in your own environment with a free 30-day trial of Kpow.

If you need assistance with your WarpStream integration, reach out to our engineering support team at support@factorhouse.io.

Introduction to Factor House Local

Factor House — Wed, 06 May 2026 00:00:00 GMT

Factor House Local is a collection of pre-configured Docker Compose environments that demonstrate modern data platform architectures. Each setup is purpose-built around a specific use case and incorporates widely adopted technologies such as Kafka, Flink, Spark, Iceberg, and Pinot. These environments are further enhanced by enterprise-grade tools from Factor House: Kpow, for Kafka management and control, and Flex, for seamless integration with Flink.

Data Stack

Kafka Development & Monitoring with Kpow

This stack provides a comprehensive, locally deployable Apache Kafka environment designed for robust development, testing, and operations. It utilizes Confluent Platform components, featuring a high-availability 3-node Kafka cluster, Zookeeper, Schema Registry for data governance, and Kafka Connect for data integration.

The centerpiece of the stack is Kpow (by Factorhouse), an enterprise-grade management and observability toolkit. Kpow offers a powerful web UI that provides deep visibility into brokers, topics, and consumer groups. Key features include real-time monitoring, advanced data inspection using kJQ (allowing complex queries across various data formats like Avro and Protobuf), and management of Schema Registry and Kafka Connect. Kpow also adds critical enterprise features such as Role-Based Access Control (RBAC), data masking/redaction for sensitive information, and audit logging.

Ideal For: Building and testing microservices, managing data integration pipelines, troubleshooting Kafka issues, and enforcing data governance in event-driven architectures.

Unified Analytics Platform (Flex, Flink, Spark, Iceberg & Hive Metastore)

This architecture establishes a modern Data Lakehouse that seamlessly integrates real-time stream processing and large-scale batch analytics. It eliminates data silos by allowing both Apache Flink (for streaming) and Apache Spark (for batch) to operate on the same data.

The foundation is built on Apache Iceberg tables stored in MinIO (S3-compatible storage), providing ACID transactions and schema evolution. A Hive Metastore, backed by PostgreSQL, acts as the unified catalog for both Flink and Spark. The PostgreSQL instance is also configured for Change Data Capture (CDC), enabling real-time synchronization from transactional databases into the lakehouse.

The stack includes Flex (by Factorhouse), an enterprise toolkit for managing and monitoring Apache Flink, offering enhanced security, multi-tenancy, and deep insights into Flink jobs. A Flink SQL Gateway is also included for interactive queries on live data streams.

Ideal For: Unified batch and stream analytics, real-time ETL, CDC pipelines from operational databases, fraud detection, and interactive self-service analytics on a single source of truth.

Apache Pinot Real-Time OLAP Cluster

This stack deploys the core components of Apache Pinot, a distributed OLAP (Online Analytical Processing) datastore specifically engineered for ultra-low-latency analytics at high throughput. Pinot is designed to ingest data from both batch sources (like S3) and streaming sources (like Kafka) and serve analytical queries with millisecond response times.

Ideal For: Powering real-time, interactive dashboards; user-facing analytics embedded within applications (where immediate feedback is crucial); anomaly detection; and rapid A/B testing analysis.

Centralized Observability & Data Lineage

This stack provides a complete solution for understanding both system health and data provenance. It combines Marquez, the reference implementation of the OpenLineage standard, with the industry-standard monitoring suite of Prometheus, Grafana, and Alertmanager.

At its core, OpenLineage enables automated data lineage tracking for Kafka, Flink, and Spark workloads by providing a standardized API for emitting metadata about jobs and datasets. Marquez consumes these events to build a living, interactive map of your data ecosystem. This allows you to trace how datasets are created and consumed, making it invaluable for impact analysis and debugging. The Prometheus stack complements this by collecting time-series metrics from all applications, visualizing them in Grafana dashboards, and using Alertmanager to send proactive notifications about potential system issues.

Ideal For: Tracking data provenance, performing root cause analysis for data quality issues, monitoring the performance of the entire data platform, and providing a unified view of both data lineage and system health.

Factor House Local Labs

The Factor House Local labs are a series of 12 hands-on tutorials designed to guide developers through building real-time data pipelines and analytics systems. The labs use a common dataset of orders from a Kafka topic to demonstrate a complete, end-to-end workflow, from data ingestion to real-time analytics.

The labs are organized around a few key themes:

Lab 1 - Streaming with Confidence:

Learn to produce and consume Avro data using Schema Registry. This lab helps you ensure data integrity and build robust, schema-aware Kafka streams.

Lab 2 - Building Data Pipelines with Kafka Connect:

Discover the power of Kafka Connect! This lab shows you how to stream data from sources to sinks (e.g., databases, files) efficiently, often without writing a single line of code.

Labs 3, 4, 5 - From Events to Insights:

Unlock the potential of your event streams! Dive into building real-time analytics applications using powerful stream processing techniques. You’ll work on transforming raw data into actionable intelligence.

Labs 6, 7, 8, 9, 10 - Streaming to the Data Lake:

Build modern data lake foundations. These labs guide you through ingesting Kafka data into highly efficient and queryable formats like Parquet and Apache Iceberg, setting the stage for powerful batch and ad-hoc analytics.

Labs 11, 12 - Bringing Real-Time Analytics to Life:

See your data in motion! You’ll construct reactive client applications and dashboards that respond to live data streams, providing immediate insights and visualizations.

Overall, the labs provide a practical, production-inspired journey, showing how to leverage Kafka, Flink, Spark, Iceberg, and Pinot together to build sophisticated, real-time data platforms.

Conclusion

Factor House Local is more than just a collection of Docker containers; it represents a holistic learning and development ecosystem for modern data engineering.

The pre-configured stacks serve as the ready-to-use “what,” providing the foundational architecture for today’s data platforms. The hands-on labs provide the practical “how,” guiding users step-by-step through building real-world data pipelines that solve concrete problems.

By bridging the gap between event streaming (Kafka), large-scale processing (Flink, Spark), modern data storage (Iceberg), and low-latency analytics (Pinot), Factor House Local demystifies the complexity of building integrated data systems. Furthermore, the inclusion of enterprise-grade tools like Kpow and Flex demonstrates how to operate these systems with the observability, control, and security required for production environments.

Whether you are a developer looking to learn new technologies, an architect prototyping a new design, or a team building the foundation for your next data product, Factor House Local provides the ideal starting point to accelerate your journey.

KIP-1150 Diskless Topics: Rethinking Storage and Cloud Costs in Kafka

Factor House — Wed, 06 May 2026 00:00:00 GMT

Overview

Operating Apache Kafka in public clouds at massive scale has traditionally meant grappling with high infrastructure costs, specifically around block storage and inter-Availability Zone (AZ) network transfer. While the introduction of Tiered Storage (KIP-405) helped offload historical data, active replication still forces engineers to pay a premium for high-performance disks and network bandwidth.

Recently accepted as a consensus document, KIP-1150: Diskless Topics represents a major architectural pivot for the Apache Kafka community. By proposing native object storage for active segments, Diskless Topics aim to fundamentally change how data is stored and replicated. If successfully implemented, this architecture will bring true cloud-native scalability and cost-efficiency to Kafka.

Blending object storage semantics with real-time streaming requires a shift in operational thinking. In this article, we will explore what KIP-1150 proposes, the cloud infrastructure problems it targets, how it aligns with broader industry trends, and the new monitoring complexities teams will need to manage as the design becomes a reality.

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Start your free 30-day trial or explore our live multi-cluster demo environment to see Kpow in action.

Key takeaways

Reduction in cross-AZ costs: Diskless topics bypass standard broker-to-broker replication, significantly reducing network fees. However, this trades replication costs for object storage API and transfer fees.
“Diskless” is like “Serverless”: It does not mean zero disks. Brokers will still use disks for KRaft metadata, caching, and batch metadata, but user data is written through to highly durable object storage.
Cost vs. Latency tradeoff: Operators will have the ability to choose between low-latency classic block-storage topics and highly cost-optimized diskless topics on a per-topic basis.
New monitoring horizons: Abstracting storage away from local disks into remote object storage requires entirely new visibility strategies for caching, ingestion engines, and object store performance.

Background: Shift to Decoupled Architecture

To fully understand the necessity of KIP-1150, we must look at three converging forces reshaping the data landscape. The most immediate catalyst is the financial friction between traditional Kafka architecture and modern cloud economics. Beyond pure cost optimization, this evolution is in line with the broader industry shift toward separating compute from storage, alongside mounting market pressure from new object-storage-first streaming vendors. Together, these factors make Diskless Topics a critical step forward.

Traditional Architecture vs. Cloud Economics

Kafka was originally designed for commodity hardware. It relies on low-durability local block storage and direct broker-to-broker replication to ensure high availability and data safety. However, modern public clouds financially penalize this specific design. Major cloud providers like AWS and GCP charge heavily for cross-Availability Zone network traffic, often between $0.01 to $0.02 per GiB. Furthermore, provisioned block storage is considerably more expensive than object storage like Amazon S3 or Google Cloud Storage.

While the introduction of Tiered Storage (KIP-405) was a fantastic milestone, it only solved the cost problem for inactive, historical data. Active data still requires expensive local disk provisioning and incurs high replication network costs.

Industry Shift to Decoupling

This friction between legacy architecture and cloud pricing is not unique to event streaming. The broader data ecosystem solved this exact problem years ago by fundamentally decoupling compute resources from storage layers. Modern data warehouse solutions like Snowflake, Amazon Redshift, and Google BigQuery revolutionized cloud analytics by moving their ultimate source of truth to highly durable, low-cost object storage. This architectural shift proved that data platforms could achieve massive scale and cost efficiency without sacrificing reliability.

Market Pressure in Event Streaming

Naturally, this separation of compute and storage has now arrived in the real-time streaming space. Recognizing the financial burden of running traditional Kafka in the cloud, several alternative vendors capitalized on this architectural gap to build object-storage-first streaming engines. Platforms like AutoMQ and WarpStream built Kafka-compatible engines backed entirely by object storage. Similarly, StreamNative recently expanded beyond Apache Pulsar to offer a Kafka-compatible engine that also leverages a decoupled storage architecture.

KIP-1150 as Kafka’s Native Response

Rather than allowing alternative platforms to dictate the future of cloud-native streaming, the Apache Kafka community accepted KIP-1150 as a roadmap to incorporate decoupled innovation natively. By planning to utilize object storage for active segments, KIP-1150 addresses these industry forces simultaneously. It tackles the financial friction of cloud block storage, aligns Kafka with the principle of separating compute from storage, and provides a robust answer to emerging competitors.

What are KIP-1150 Diskless Topics?

KIP-1150 introduces a blueprint for a new type of topic equipped with a distinct ingestion engine. Instead of relying purely on local block storage for durability and replication, this proposed engine writes data through directly to object storage.

It is important to clarify the terminology. “Diskless” does not mean broker disks vanish entirely. Disks are still utilized for KRaft metadata, caching consumer data, and short-term staging. However, the primary source of truth and data durability completely shifts from the local disk to the remote object store.

What does KIP-1150 solve?

KIP-1150 directly targets the most significant infrastructure pain points that platform engineers face today.

Drastically reducing replication transfer costs

By relying on object storage for durability, brokers will no longer need to constantly mirror gigabytes of data across availability zones to maintain replication factors. This largely bypasses the expensive cross-AZ network transfer bills that plague large-scale cloud deployments. It is worth noting that costs are not entirely eliminated. Operators trade heavy broker-to-broker network fees for object store API and data transfer costs, which are generally much cheaper but still require architectural consideration.

Infrastructure optimization

Operators can provision brokers with extremely small, highly performant disks (or even memory-backed storage) merely for caching. Relying on cheap object storage for actual data retention drastically reduces monthly block storage spend.

Decoupled scaling

In traditional Kafka, scaling out a cluster means physically moving large volumes of data between broker disks, a process that is notoriously slow and network-intensive. With Diskless Topics, the canonical data safely resides in object storage. Brokers can scale up and rebalance significantly faster because they only need to update metadata and warm up their caches, completely bypassing heavy data migration.

How will Diskless Topics work?

KIP-1150 is a consensus and architectural document. Its primary goal is to align the community on the need and the high-level concept. The actual codebase implementation is deliberately split into proposed follow-up designs, including KIP-1163 (Diskless Core) and KIP-1164 (Diskless Coordinator).

At an architectural level, Diskless Topics are envisioned to operate via an ingestion engine running parallel to the classic topic engine. Data will remain accessible to consumers both from the ingestion engine cache and directly from tiered storage.

Anticipated Write Path

To maintain reasonable performance while writing to high-latency object storage, the ingestion engine cannot simply block producer requests waiting for S3 HTTP responses. Instead, the architecture will heavily rely on local staging. When a producer sends data, the ingestion engine will likely use local memory or a highly optimized disk cache to stage the incoming records. Depending on the configured durability guarantees, the broker might acknowledge the producer immediately after local staging, or wait for an asynchronous background process to flush these batches to the remote object store. Balancing fast producer acknowledgments with the strict durability guarantees of object storage is where the real engineering complexity lies.

What complexities does KIP-1150 introduce?

While the financial and operational benefits of Diskless Topics are highly appealing, they introduce new tradeoffs and visibility challenges.

Ambitious backward compatibility

The KIP states a goal of being entirely backwards compatible with existing Kafka APIs, including strict ordering guarantees, idempotency, and consumer group semantics. Achieving these semantics over a remote object-storage hot path is a massive, non-trivial engineering challenge. Ensuring consistency when the primary storage layer operates asynchronously over HTTP will require careful design and likely extensive real-world testing once implemented.

Latency Trade-offs

Writing through to an object store naturally introduces higher produce latency compared to writing directly to a local NVMe drive. Applications that require strict, ultra-low microsecond latency will likely need to stick to classic block-storage topics. Platform teams will need to carefully categorize workloads to determine which topics should be diskless and which must remain classic.

Obscured Storage Visibility

Troubleshooting local disk bottlenecks is a well-understood science. However, troubleshooting object storage API rate limits, write-through cache misses, and ingestion engine latency introduces a totally new paradigm. Traditional monitoring strategies relying on simple JMX disk metrics will no longer show the complete picture.

Factor House is actively preparing for these new topologies. As storage paradigms shift from local disks to remote object stores, we recognize that traditional monitoring tools will leave operations teams with significant blind spots. Factor House is actively tracking the development of KIP-1150 and its sub-KIPs to ensure Kpow will provide seamless, purpose-built observability into remote storage latencies, cache hit rates, and overall diskless topic performance.

Looking Ahead: Future of Diskless Kafka

KIP-1150 serves as the foundational layer for a highly anticipated roadmap of future Kafka enhancements. Once the core diskless functionality is implemented, it unlocks several exciting possibilities:

Lakehouse (e.g., Apache Iceberg) Integration: By writing data natively to object storage, Kafka paves the way for massively parallel analytical processing on at-rest topic data using open table formats.
Topic Type Changing: Future updates aim to allow operators to dynamically convert topics back and forth between classic and diskless configurations as business needs evolve.
Heterogeneous Clusters: Because data will live securely in object storage, brokers in a cluster will no longer need to be identical clones of each other. Operators could provision distinct types of hardware for specific jobs, such as using high-CPU machines dedicated solely to ingesting incoming data, or high-memory machines exclusively for serving consumers, to maximize resource efficiency.

Our take at Factor House

KIP-1150 represents a critical proposed evolution for Apache Kafka. By embracing the separation of compute and storage, it lays the groundwork for Kafka to remain a highly competitive and cost-effective streaming engine regardless of shifting cloud architecture trends. If successful, it offers a massive win for organizations looking to drastically reduce their cloud infrastructure costs while achieving easier, faster cluster scalability.

We believe that operational tooling must evolve at the same pace as the infrastructure it monitors. The transition from block storage to object storage introduces significant complexities. We are incredibly excited about this evolving era of cloud-native Kafka, and we are committed to updating Kpow right alongside Kafka’s core architecture to ensure you maintain total visibility and control.

Next steps

Explore Kpow in your own environment with a free 30-day trial.

If you need assistance managing your Kafka environment or preparing for upcoming architecture shifts, reach out to our engineering support team at support@factorhouse.io.

KIP-932 Queues for Kafka: Bridging the Gap Between Streaming and Messaging

Jaehyeon Kim — Wed, 06 May 2026 00:00:00 GMT

Software engineers have long debated the architectural trade-offs between event streaming platforms and traditional message queues. Do you need the durable, replayable logs of Apache Kafka, or the point-to-point, competing-consumer flexibility of RabbitMQ or AWS SQS? Often, the answer was to use both. This approach unfortunately led to complex and sprawling infrastructure.

With the introduction of KIP-932: Queues for Kafka, the Apache Kafka community has dramatically reshaped this conversation. By introducing “Share Groups”, Kafka now natively supports cooperative, point-to-point message queuing without sacrificing its underlying distributed log architecture.

Blending queue semantics with log storage is not without its trade-offs. In this article, we will explore what KIP-932 is, the problems it solves, how to implement it, and the new operational complexities teams will need to manage as they adopt it.

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Start your free 30-day trial or explore our live multi-cluster demo environment to see Kpow in action.

Key takeaways

Share Groups break the 1:1 rule: Multiple consumers can now concurrently process messages from a single partition. This unlocks elastic scalability without the need for aggressive over-partitioning.
Record-level acknowledgments: Instead of committing bulk offsets, consumers now acquire temporary locks on individual messages and acknowledge them upon successful processing.
Elimination of head-of-line blocking: Slow-to-process messages no longer block the entire partition because other consumers in the Share Group can continue processing subsequent messages.
New monitoring requirements: Managing states like “Acquired,” “Available,” and “Archived” requires new visibility paradigms. Factor House is actively building features to help engineers manage these new complexities as KIP-932 matures.

Background: Logs vs Queues in Kafka

To fully appreciate KIP-932, we must look at the historical divide between traditional messaging queues and event streaming logs.

Traditional Queues (e.g., RabbitMQ, ActiveMQ, SQS) operate on a point-to-point basis. When a message is published to a queue, multiple worker services can listen to that queue. When a worker grabs a message, it is hidden from others, processed, and then fundamentally destroyed or removed upon acknowledgment. This allows for massive, dynamic scalability of consumers. If your queue backs up, you simply spin up more worker instances.

Apache Kafka, on the other hand, is a distributed commit log. Messages are appended immutably to partitions. To read data, applications use Consumer Groups. In a traditional Consumer Group, Kafka enforces a strict rule: there can only be one active consumer per partition within a group.

If a topic has three partitions, you can only have three active consumers processing in parallel. If you spin up a fourth consumer, it sits completely idle. To scale up message processing throughput in Kafka, engineers have historically been forced to over-partition their topics. For instance, they might create 100 partitions just in case they need 100 consumers in the future. This taxes broker resources, increases ZooKeeper or KRaft metadata overhead, and complicates cluster management.

What is KIP-932 queues for Kafka?

KIP-932 is a major architectural upgrade that brings point-to-point, cooperative queuing semantics natively to Apache Kafka. It introduces Share Groups, a new consumption model where multiple consumers can dynamically collaborate to process messages from the exact same partition.

Traditional Kafka Consumer Groups track progress using a single numerical offset to indicate how far down the log a consumer has read. In contrast, Share Groups track the state of individual records. When a consumer in a Share Group fetches a message, it does not just read it; it locks it. The message remains in the Kafka log, but the broker knows not to hand that specific message to any other consumer in the Share Group until the lock expires or the message is explicitly acknowledged.

This effectively turns a Kafka partition into a high-throughput, competing-consumer queue. It blends the best of both streaming and messaging worlds.

What does KIP-932 solve?

KIP-932 directly targets several long-standing pain points that developers face when building scalable microservices on top of Kafka.

Solving the partition bottleneck

The most immediate benefit of KIP-932 is the separation of storage scale from processing scale. Previously, your partition count dictated your maximum concurrency limit. With Share Groups, you can have a single partition topic being processed concurrently by 50 different microservice instances. Applications can now dynamically scale horizontally in response to traffic spikes, such as Black Friday sales. Administrators no longer need to modify topic configurations or rebalance partition leaders.

Eradicating head-of-line blocking

In a traditional consumer group, partitions are processed sequentially. If Consumer A reads Message 1, and processing that message requires a slow third-party API call taking 15 seconds, all subsequent messages are stuck waiting. This “head-of-line blocking” degrades latency. With KIP-932, Consumer A can lock and process Message 1 while other consumers immediately fetch and process the next messages concurrently.

Infrastructure consolidation

Historically, architectures required a two-tier messaging strategy. Kafka was used as the central nervous system for high-throughput, durable event streaming, while tools like RabbitMQ were deployed at the edge for task routing and worker queues. KIP-932 enables teams to deprecate legacy messaging systems. This reduces infrastructure sprawl, cuts licensing costs, and simplifies the tech stack to just Apache Kafka.

How does KIP-932 work in Kafka?

Under the hood, KIP-932 fundamentally alters how Kafka brokers and clients agree on what data has been read. It achieves this via Record-Level Acknowledgement and Acquisition Locks.

When a Share Group is utilized, records cycle through distinct states:

Available: The message is sitting in the partition and is ready to be processed.
Acquired: A Share Consumer fetches the message. The broker applies a temporary lock (configurable, but defaulting to 30 seconds). During this window, the message is invisible to other consumers in the same Share Group.
Acknowledged: The consumer successfully processes the message and sends an ACCEPT acknowledgment to the broker. The message is marked as complete for that group.
Released or Archived: If the consumer fails and sends a REJECT, or if the lock times out because the consumer crashed, the broker releases the lock. This returns the message to the Available state for another consumer to try. If it hits a predefined retry limit, it transitions to Archived (Kafka’s native equivalent of a Dead Letter Queue).

What complexities does KIP-932 introduce?

While the benefits of Share Groups are immense, they introduce entirely new paradigms of state and error handling. Traditional Kafka monitoring strategies will no longer suffice.

Obscured State Management and Visibility

For years, monitoring Kafka consumers meant tracking a single metric: Consumer Lag. If the partition log ended at offset 100, and your consumer was at offset 90, your lag was 10. It was simple arithmetic.

With Share Groups, a group might have Acknowledged offsets 1, 2, 4, and 7, while offsets 3 and 6 are currently Acquired (locked), and offset 5 is Available. The concept of a single “lag” number is obsolete; you now have an in-flight state matrix.

Factor House is actively building the tools to manage the new complexities introduced by KIP-932. Because Share Groups operate on an entirely different set of methods and properties at the AdminClient level, traditional monitoring tools that rely on simple offset math will essentially break or fail to show the complete picture. Recognizing that Share Groups represent an “entirely new world,” our engineering team is currently designing a dedicated, purpose-built experience within our market-leading toolkit, Kpow. Rather than forcing this new paradigm into legacy consumer group views, our goal is to provide a tailored interface that translates this complex state matrix into clear, actionable insights. This ensures your operations team will not be left guessing about in-flight states.

Dealing with Poison Pills and Archiving

In standard Kafka, a “poison pill” (a malformed message that crashes your consumer) would halt partition processing entirely until an engineer intervened. KIP-932 automatically handles this via the group.share.delivery.attempt.limit configuration. If a message continually crashes consumers, its lock expires, it gets retried, and eventually, it is moved to the Archived state.

However, Kafka natively provides very little tooling to inspect or manage these Archived messages once they are sidelined. At Factor House, we are carefully evaluating how to bridge this gap. As KIP-932 matures and the APIs stabilize, we are looking at how to extend Kpow’s industry-leading data inspection and troubleshooting workflows to support these new archiving mechanics. Our vision is to ensure engineers will have the seamless ability to isolate, inspect, and recover archived records just as easily as they manage standard topic data today.

Loss of Strict Ordering Guarantees

Because multiple consumers are pulling from the same partition concurrently, strict chronological ordering is completely broken. If order matters for your business logic (e.g., processing an “Account Created” event before an “Account Updated” event for the same user ID), Share Groups are not the right tool. Engineers must carefully evaluate their domain logic to ensure their microservices are genuinely idempotent and order-independent before migrating to KIP-932 queues.

Ecosystem Maturity

KIP-932 requires brand new API classes (like KafkaShareConsumer in Java) to handle the new acknowledgment protocols (ACCEPT, RELEASE, REJECT). While Confluent and the Apache community have prioritized the Java ecosystem, wrappers like librdkafka (which power Python, Go, C++, and .NET) are taking longer to catch up. Organizations with polyglot architectures will face a transition period where only their Java-based microservices can leverage Share Groups. This could potentially lead to fragmented architectural patterns in the short term.

Implementing Share Groups requires a modern Kafka deployment and specific client-side code changes. Instead of relying on a standard KafkaConsumer, your application must utilize the new Share Group APIs.

Prerequisites:

Apache Kafka 4.0 or newer.
A compatible client (currently Java via KafkaShareConsumer).

Step 1: Enable Share Groups on the Broker

Ensure your broker configurations have the share group protocols enabled. This is enabled by default in newer KRaft-based Kafka releases.

Step 2: Instantiate the Share Consumer

In your Java application, replace your traditional consumer with the new KafkaShareConsumer class.

Step 3: Configure the Group ID

Instead of using group.id, you bind your consumers together using the share.group.id property.

Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("share.group.id", "my-first-share-group");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

KafkaShareConsumer<String, String> consumer = new KafkaShareConsumer<>(props);
consumer.subscribe(Collections.singletonList("kip932-demo-topic"));

Step 4: Handle Acknowledgements

Update your application code logic to utilize the new record-level APIs. When a record is processed, explicit acknowledgments dictate what happens to the message next.

while (true) {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));
    for (ConsumerRecord<String, String> record : records) {
        try {
            // Process the message
            System.out.printf("Processing key = %s, value = %s%n", record.key(), record.value());

            // Explicitly accept the record upon success
            consumer.acknowledge(record, AcknowledgeType.ACCEPT);
        } catch (Exception e) {
            // Reject the record so it can be retried by another consumer
            consumer.acknowledge(record, AcknowledgeType.REJECT);
        }
    }
    // Commit the acknowledgments to the broker
    consumer.commitSync();
}

By deploying multiple instances of this Java application, you will instantly see Kafka distribute the messages concurrently among the active instances. This completely bypasses the traditional partition limits.

Our take at Factor House

KIP-932 is undeniably a milestone for Apache Kafka. By successfully merging event streaming with point-to-point queuing, it empowers engineering teams to vastly simplify their infrastructure and scale worker processes dynamically without jumping through the hoops of over-partitioning.

We believe that operational tooling must evolve at the same pace as the infrastructure. The transition from simple offset tracking to complex, record-level state management introduces significant blind spots if teams rely on legacy monitoring tools. We are incredibly excited about this new era of Kafka, but we also recognize that such a fundamental shift requires equally evolved tooling.

Because Share Groups introduce an entirely new set of methods and properties, we are actively developing the next generation of Kpow features to tackle this challenge head-on. Rather than trying to patch over legacy consumer group views, we are building a completely distinct and tailored UI experience to illuminate these new mechanics. Stay tuned as we continue to roll out dedicated support to help you safely and effectively navigate this exciting new streaming landscape.

Next steps

Explore Kpow in your own environment with a free 30-day trial.

If you need assistance managing your Kafka environment, reach out to our engineering support team at support@factorhouse.io.

RBAC for Kafka: How to Implement and Key Considerations

Moslem Chalfouh — Wed, 06 May 2026 00:00:00 GMT

Key takeaways

Role-Based Access Control (RBAC) replaces unmanageable per-user ACL rules with role abstractions that scale across teams, topics, and clusters.
Kafka’s native ACLs and RBAC via a management tool protect different layers. ACLs govern data plane access (producers, consumers, admin clients); RBAC controls what operators can do through the management UI. Both are typically needed.
Management tools like Kpow connect to Kafka using their own service account, not as the logged-in user. This separation is what makes RBAC enforcement possible without granting every operator direct cluster access.
A working RBAC setup with Kpow can run in under an hour. This guide walks you through every step.

What is Kafka RBAC?

Role-Based Access Control (RBAC) is a permission model that groups access rights into roles, and then assigns users to those roles. In a Kafka context, it determines what authenticated users can do through a management interface, rather than assigning permissions directly to individuals.

I’ve seen the pattern multiple times. A team starts with five topics and three services. They write a handful of ACL rules. It works. Then the platform grew to ten teams, 100+ topics, producers and consumers multiplying across environments. Suddenly there were thousands of user-to-resource rules, nobody knew who had access to what, and onboarding a new engineer meant writing rules one by one. That’s when ACLs stop being a solution and become the problem.

Role-Based Access Control uses a different model. Rather than assigning permissions directly between individual users and resources, it groups permissions into roles such as viewer, operator, or kafka-admin, and then assigns users to those roles. A single role change propagates to everyone assigned to it, and a single audit query shows you who can do what.

In the Kafka ecosystem, RBAC doesn’t replace the broker’s native authorization. It layers on top of it, typically through a management tool like Kpow that sits between your operators and the cluster. The broker still enforces ACLs at the protocol level. RBAC controls what humans can do through the operations interface.

Here’s the core difference, visualized:

In this example, eight rules are reduced to two role assignments for three users. In larger environments, the reduction in individual rules can become even more significant.

Kafka authorization: RBAC vs ACLs

ACLs can be effective at certain scales. The key consideration is whether they fit the operational and governance needs of your environment as it grows.

Kafka ships with StandardAuthorizer (KRaft) or AclAuthorizer (ZooKeeper). Both enforce allow/deny rules at the broker level: who can produce, consume, create topics, manage groups. They work directly on the Kafka protocol. Every client connection is evaluated against these rules.

RBAC, as implemented by management tools like Kpow, operates at a different layer. It controls what authenticated users can do through the management UI: inspect topics, query data, create or delete resources, edit ACLs. The tool connects to Kafka using a service account with the necessary privileges. The logged-in user never touches the broker directly.

This means they’re complementary, not competing. Most teams need both. ACLs for the data plane. RBAC for the operational control plane.

Dimension	Kafka ACLs	RBAC
What it protects	Direct broker access (producers, consumers, admin clients)	Operations UI access (inspect, query, mutate resources)
Granularity	Per-user, per-resource, per-operation	Per-role, per-resource taxon, per-action
Scale	Rules grow linearly with users × resources	Rules grow with roles (typically 3 to 10)
Audit	Broker logs (verbose, unstructured)	Dedicated audit log (provided by the management tool)
Central management	No, per-cluster only	Yes, one config across managed clusters
Onboarding a new user	Write N ACL rules	Assign one role
Typical adopter	Every Kafka installation	Teams managing authorization at scale
Deny semantics	Explicit DENY rules supported, DENY takes precedence over ALLOW	Default-deny with allow-only policies. If no policy matches, access is denied.

Three layers of Kafka governance

Governance discussions often focus on two layers, but there is a third worth considering.

The first layer is Kafka ACLs. They sit at the broker level and control what every client can do at the protocol level. Every producer, consumer, and admin tool goes through this gate. ACLs are not optional in a serious production environment.

The second layer is RBAC. It controls what your operators can do through the management UI. Without it, every person with Kpow access implicitly has the same privileges as Kpow’s service account. That’s a broad blast radius for a misconfigured delete operation.

The third layer is multi-tenancy. This is about isolating teams and resources from each other within a shared cluster. Who can see which topics. Which namespaces are off-limits. Kpow supports multi-tenancy as a feature separate from RBAC. See the Kpow multi-tenancy documentation.

Platform engineers who operate shared Kafka infrastructure need all three layers working together. Each one solves a different problem. None of them replaces the others.

Why implement Kafka RBAC?

The technical comparison is useful. But the real driver is usually a business question: can we prove who had access to what, and when?

Governance and compliance

Auditors don’t want to read thousands of ACL entries. They want role definitions and assignment logs. RBAC gives you a clean answer: “These three roles exist. Here’s who is assigned to each. Here’s the audit trail of every action.” That conversation takes five minutes instead of five days.

Fine-grained authorization at the UI level

Not every operator needs the same access. A developer on the payments team should inspect payment topics and query messages for debugging. They should not be able to delete production topics or edit ACLs. With Kpow RBAC, I defined a viewer role with exactly three permissions TOPIC_INSPECT, TOPIC_DATA_QUERY, TOPIC_DATA_DOWNLOAD and a kafka-admin role with full CRUD plus ACL_EDIT. Two roles. Clean separation. Done.

Scaling authorization across teams

This is where ACLs collapse. When a new engineer joins, you assign them a role. When someone changes teams, you change their role. When a team is decommissioned, you remove the role. No per-user rules to track. No orphaned ACLs rotting in the broker config.

Central management for multiple clusters

If you operate Kafka in dev, staging, and production, maintaining separate ACL sets per cluster is painful. Kpow’s RBAC config is external to the broker. One YAML file defines your roles, and Kpow applies them to whichever cluster it connects to.

Authentication method flexibility

Kafka supports SASL/PLAIN, SASL/SCRAM, SASL/GSSAPI (Kerberos), mTLS, and OAuthBearer for client authentication. Kpow adds its own auth layer on top supporting Jetty (file-based), OpenID Connect (which covers OAuth 2.0 providers like Okta and Azure AD), SAML, and LDAP. So yes, Kafka does support OAuth.

What to consider before implementing RBAC

Before you start, give some consideration to the following five choices. Missteps can introduce security weaknesses that become apparent later, including during incidents.

1. Principle of Least Privilege (PoLP)

‍Start with zero permissions and add only what each role requires. I made the mistake of testing with an overly generous viewer role first. It worked, but it also let read-only users see data they shouldn’t. Strip it down. The right question isn’t “what could this role need?” It’s “what’s the minimum this role must have to do their job?”

2. Multi-tenancy and resource isolation

If multiple teams share a cluster, your RBAC policies need resource-level scoping. Kpow’s taxon system supports this. You can restrict a role to specific topics by name or wildcard. But you have to define policies at every taxon depth (more on this in the implementation section). Don’t assume a cluster-level wildcard covers topic-level operations. It doesn’t.

3. Integration with your identity provider (IdP)

‍Kpow supports four auth providers: jetty, openid, saml, and LDAP (configured via the Jetty provider). Choose based on what you already run. If you have Okta or Azure AD, go OpenID. If you want a quick local setup for testing, go Jetty. The one thing you cannot do is reuse Kafka’s SASL users directly. Kpow’s auth is a separate layer.

4. Data masking and sensitive topics

‍RBAC controls who can query topic data through the UI. But consider whether some topics contain PII or financial data that no role should access in plain text. Kpow’s data policies allow sensitive field values to be redacted, even for users who have TOPIC_DATA_QUERY permission.

5. Role explosion

RBAC should reduce the number of individual rules. If you end up with 30 roles for 30 teams, you’ve rebuilt ACLs with extra steps. Aim for 3 to 7 generic roles (viewer, operator, admin, maybe a data-steward) and use resource scoping for team isolation. The moment you create a role named payments-team-staging-read-only-except-topic-x, stop. Rethink.

How to implement RBAC

Kpow adds a user-facing access control layer on top of your Kafka cluster. Kpow’s RBAC lets you define what each user role can do within the Kpow UI: creating topics, querying data, editing consumer groups, and so on.

The two systems are independent. Kafka ACLs govern what client applications (including Kpow’s own service account) can do at the broker. Kpow RBAC governs what your human operators can do through Kpow. Both are worth configuring; neither replaces the other.

How it works

Kpow RBAC maps roles from your identity provider to Allow, Deny, or Stage permissions on specific Kafka resources. Roles come from whatever authentication provider you’re using: Jetty (file, LDAP, or DB), SAML, or OpenID/OAuth. The RBAC configuration itself lives in a YAML file:

RBAC_CONFIGURATION_FILE=/path/to/rbac-config.yaml

Without RBAC configured, the default effect for all actions is an implicit Deny. With it enabled, you define exactly which roles can do what.

Defining policies

Each policy specifies a resource, an effect, a list of actions, and the role it applies to:‍

`authorized_roles:
- “*” # Allow all authenticated users into the UI

admin_roles:
- “kafka-admin”

policies:
# Read-only access for viewers
- role: “viewer”
effect: “Allow”
resource: [“cluster”, “*”]
actions: [“TOPIC_INSPECT”]

- role: “viewer”
effect: “Allow”
resource: [“cluster”, “”, “topic”, “”]
actions: [“TOPIC_INSPECT”, “TOPIC_DATA_QUERY”, “TOPIC_DATA_DOWNLOAD”]

# Full control for admins
- role: “kafka-admin”
effect: “Allow”
resource: [“cluster”, “”, “topic”, “”]
actions: [“TOPIC_INSPECT”, “TOPIC_CREATE”, “TOPIC_EDIT”, “TOPIC_DELETE”,
“TOPIC_DATA_QUERY”, “TOPIC_DATA_DOWNLOAD”, “TOPIC_DATA_PRODUCE”]`

A few things worth understanding before you write your first config:

Role names are from your auth provider, not usernames. The role field must match the role assigned in your identity provider (e.g. the role in your Jetty realm file, or a group from your SAML IdP), not an individual username.

Taxon depth matters. Resources follow a four-part taxonomy: [DOMAIN_TYPE, DOMAIN_ID, OBJECT_TYPE, OBJECT_ID]. A policy scoped to ["cluster", "*"] only covers cluster-level actions. Topic-level actions (querying data, producing messages) require a separate policy scoped to ["cluster", "*", "topic", "*"]. You need entries at each depth you want to cover.

Deny takes precedence. Where multiple policies apply to the same resource, Deny effects always win. Anything without a matching Allow is implicitly denied.

Debugging access issues

Kpow surfaces permission errors directly in the UI when a user is denied access to a resource. For deeper debugging, every action is recorded in Kpow’s audit log topic (__oprtr_audit_log), which includes the principal, the resource taxon, and the effect applied.

For the full list of available actions and detailed configuration options, see the Kpow RBAC documentation.

Try RBAC for free with Kpow by Factor House

ACLs work well for some environments, but often become harder to manage over time. Without RBAC in your management tooling, every operator inherits the full permissions of the tool’s service account, regardless of their actual role.

Kpow gives you what’s missing:

Fine-Grained Authorization. Define exactly what each role can see, query, create, and delete down to specific topics and action types.
Seamless Integration. Plug into your existing identity provider OpenID Connect (Okta, Azure AD), SAML, LDAP, or file-based Jetty auth. No changes to your Kafka cluster configuration.
Operational Control. Every action flows through RBAC policies, every action is audited, and sensitive operations can be staged for approval before execution.

Give Kpow a try for yourself with a free 30-day trial. You can connect it to any Kafka cluster in minutes and deploy via Docker, Helm, or JAR.

Beyond JMX: Supercharging Grafana Dashboards with High-Fidelity Metrics

Jaehyeon Kim — Fri, 17 Apr 2026 00:00:00 GMT

Overview

In Part 1 of our observability series, we demonstrated how to close the Context Gap using Kpow’s unified diagnostic interface. Resolving active incidents quickly is crucial, but mature engineering teams also require robust historical analysis, capacity planning, and automated alerting. For these tasks, teams rely on monitoring stacks like Prometheus and Grafana.

However, standard Kafka dashboards frequently suffer from a persistent issue known as the Quality Gap. They are flooded with raw, low-level metrics that provide technical depth but lack the context required to understand true business impact.

This guide explores how to close the Quality Gap by transitioning from scraping low-level infrastructure data to utilizing high-fidelity, pre-calculated telemetry that instantly reveals environment health.

This is Part 2 of the Kafka Observability with Kpow: Driving Operational Excellence series.

Part 1: Rapid Kafka Diagnostics: A Unified Workflow for Root Cause Analysis
Part 2: Beyond JMX: Supercharging Grafana Dashboards with High-Fidelity Metrics (This article)
Part 3: Operational Transparency: Real-Time Audit Trail integrated with Webhooks

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Start your free 30-day trial or explore our live multi-cluster demo environment to see Kpow in action.

Problem: Quality Gap of Raw Metrics

The standard approach for routing Kafka metrics into Prometheus is to deploy a JMX Exporter. While this successfully moves data from the broker to the dashboard, it creates a significant Quality Gap.

Raw JMX metrics provide deep technical visibility but offer very little business context. Metrics such as raw message offsets or byte counters are difficult to interpret during a critical incident. They do not directly convey user impact or service degradation.

In practice, the most meaningful operational signals are derived metrics. For example, knowing a consumer’s exact lag reflects data freshness and downstream SLA risk. Knowing the active throughput delta of a topic indicates whether a producer has silently failed.

Attempting to compute these high-fidelity metrics from raw JMX offsets using PromQL is notoriously difficult, fragile, and often inaccurate. As a result, teams are left with noisy, low-quality dashboards that hinder incident response and make automated alerting unreliable.

Solution: Kpow as a High-Fidelity Telemetry Engine

Kpow eliminates the Quality Gap by acting as a centralized, high-fidelity metrics engine. The unique advantage of Kpow is that it calculates its own telemetry by directly observing your Kafka cluster and related resources.

This architectural choice provides several distinct advantages:

No JMX Dependency: Because Kpow does not rely on Kafka’s internal JMX metrics, it offers frictionless installation and configuration. You do not need to deploy sidecars or manage complex regular expression filters. Instead, Kpow exposes several dedicated, OpenMetrics-compliant egress endpoints. You simply configure your Prometheus scraper to target specific paths, such as the core telemetry endpoint, topic offsets, or consumer group offsets, as detailed in the Prometheus Integration Overview.
Derived, Actionable Metrics: Kpow handles the heavy computational lifting. From its direct observations, it calculates complex, derived metrics that represent true business impact. As outlined in the Kpow Metrics Glossary, these include precise consumer group lag (group_offset_lag), production throughput rates (topic_end_delta), and under-replicated partition counts (topic_urp_total). These are exposed as clean, highly labeled time-series data ready for immediate visualization.
Unified Scope: This frictionless pattern extends beyond just Kafka brokers. Kpow automatically calculates and exposes metrics across your entire streaming ecosystem. With granular endpoint structures, you gain comprehensive observability into Kafka Connect, Schema Registry, Kafka Streams, and ksqlDB. This provides full-stack visibility without requiring additional agents or custom exporters.

Putting it into Practice: Ready-to-Use Grafana Dashboards

To demonstrate the power of high-fidelity telemetry, we have published four ready-to-use Grafana dashboard templates. These dashboards consume Kpow’s Prometheus endpoint to provide immediate, actionable visibility across your infrastructure.

You can instantly import these dashboards into your environment directly from the Grafana Community Gallery, or explore the source templates and setup instructions in our dedicated Factor Telemetry GitHub repository.

1. Kafka Environment Health

Designed for Platform Teams, this dashboard provides a high-level macro view of overall cluster stability and capacity.

Rather than relying on raw byte counts, it surfaces derived operational health indicators. It tracks total online brokers, overall data on disk, total topics, and total consumer groups. It also visualizes cluster-wide production and consumption rates, and provides a detailed breakdown of topic activity and consumer group health (Stable, Rebalancing, Empty) to give you an instant read on the environment’s status.

🌐 Import from Grafana Gallery (ID: 25103) | 📁 View JSON Template

2. Kafka Topic Diagnostics

Designed for data engineers and platform administrators, this dashboard provides granular visibility into the data layer.

It tracks aggregate metrics like total topics, total replica disk usage, cluster-wide read/write throughput, and non-preferred leaders. Most importantly, it visualizes per-topic production and consumption rates over time, topic size growth, and isolates the exact topics experiencing consumer lag or Under Replicated Partitions (URPs) through detailed diagnostic tables.

🌐 Import from Grafana Gallery (ID: 25104) | 📁 View JSON Template

3. Kafka Consumer Group Deep Dive

Designed for Application Teams, this dashboard focuses on micro-level Service Level Agreement (SLA) monitoring.

Instead of generic host metrics, it visualizes the exact state of your data consumption. Key metrics include precise total lag (group_offset_lag) and real-time consumption rates (group_offset_delta). It details total assigned members and hosts, and features a clear status table tracking the exact state of every consumer group to help engineers spot stalling applications before downstream users are impacted.

🌐 Import from Grafana Gallery (ID: 25105) | 📁 View JSON Template

4. Kafka Connect Operations

Data pipeline reliability depends heavily on integration health. This dashboard targets Kafka Connect deployments, replacing tedious API queries with instant visual feedback.

It tracks aggregate summary statistics alongside individual Connector and Task states. By mapping state labels directly to distinct visual alerts (RUNNING, PAUSED, FAILED, UNASSIGNED, UNREACHABLE), teams can immediately detect stalled integrations and isolate whether the failure exists at the connector or task level.

🌐 Import from Grafana Gallery (ID: 25106) | 📁 View JSON Template

Conclusion

Standard monitoring approaches leave teams struggling with noisy, low-level data. This Quality Gap makes historical analysis difficult and accurate alerting nearly impossible.

By bypassing raw JMX metrics and leveraging Kpow’s self-calculated telemetry, you can instantly upgrade your Grafana dashboards with high-fidelity, actionable insights. This approach provides frictionless installation while delivering precise business context across Kafka, Kafka Connect, Schema Registry, and ksqlDB.

With real-time diagnostics and high-fidelity metrics established, the final step to operational maturity is administrative accountability. In Part 3: Operational Transparency, we will demonstrate how to close the Governance Gap by streaming Kpow’s real-time audit trail directly into your team’s communication channels via Kpow’s webhook integration.

Next steps

Explore Kpow in your own environment with a free 30-day trial.

If you need assistance managing your Kafka environment, reach out to our engineering support team at support@factorhouse.io.

Kafka Observability with Kpow: Driving Operational Excellence

Jaehyeon Kim — Fri, 17 Apr 2026 00:00:00 GMT

Overview

In the modern enterprise, Apache Kafka has evolved from a simple messaging queue into the central nervous system of the technology stack. It facilitates real-time data flow across microservices, databases, and analytics engines. However, the distributed nature of Kafka introduces significant operational complexity. When performance degrades or a data pipeline stalls, the impact is spread instantly across the entire organization. For engineering and platform teams, the challenge is rarely a lack of data, but rather a lack of actionable insight.

Many organizations find themselves trapped in a reactive cycle: resolving incidents only after they have impacted downstream consumers and spending hours manually correlating logs with technical metrics. To break this cycle, teams need a structured approach to operational excellence. This article introduces a comprehensive strategy designed to overcome the three critical gaps inherent in traditional Kafka monitoring, demonstrating how Kpow transforms operations into proactive observability.

To implement this strategy in your own environment, look out for our upcoming three-part operational guide:

Part 1: Rapid Kafka Diagnostics: A Unified Workflow for Root Cause Analysis
Part 2: Beyond JMX: Supercharging Grafana Dashboards with High-Fidelity Metrics
Part 3: Operational Transparency: Real-Time Audit Trail integrated with Webhooks

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Start your free 30-day trial or explore our live multi-cluster demo environment to see Kpow in action.

Three Critical Gaps in Traditional Kafka Monitoring

Achieving a mature observability posture is frequently hindered by three fundamental gaps in how Kafka environments are managed.

The Context Gap: The Difficulty of Correlating Distributed Data

In a distributed system, a single issue often manifests in multiple places. Infrastructure metrics (such as broker CPU, memory, or disk I/O) typically reside in one monitoring tool, while application-level performance data (such as consumer group lag or producer throughput) resides in another. During a production incident, engineers must manually aggregate and correlate these data points to find the root cause. Without a unified view, it is difficult to determine if a slow consumer is the result of a saturated broker, a network bottleneck, or a logic error within the application itself. This absence of context leads to fragmented investigations and extended resolution times.

The Quality Gap: The Limitation of Raw Technical Metrics

Standard Kafka monitoring often relies on raw broker JMX metrics, which provide deep technical visibility but limited business context. Metrics such as message offsets or byte counters are difficult to interpret during incidents and do not directly convey user impact. In practice, the more meaningful signal is consumer group lag, ideally expressed as time-based delay, which reflects data freshness and downstream SLA risk. Many legacy, JMX-centric monitoring setups fail to compute or surface these higher-fidelity, derived metrics. As a result, teams are left with noisy dashboards that hinder incident response and make long-term capacity and performance trend analysis unreliable.

The Governance Gap: The Risk of Opaque Administrative Changes

As Kafka adoption grows, more teams gain the ability to interact with the cluster. In many organizations, critical administrative actions (such as managing topics, editing Kafka ACLs, or resetting consumer offsets) occur in an architectural black box. When a configuration change leads to an outage, there is often no centralized audit trail to identify what was changed or who performed the action. This lack of transparency introduces significant operational risk and makes it difficult to satisfy security or compliance requirements. Without a formal record of changes, teams are forced to spend valuable time retracing steps rather than resolving issues.

The Kpow Solution: Implementing the Strategy

To overcome these gaps, organizations require a platform that integrates real-time diagnostics, high-fidelity metrics, and administrative transparency into a single, cohesive workflow. Kpow addresses these requirements across three key operational dimensions.

Closing the Context Gap with a Unified Diagnostic UI

Kpow provides a comprehensive, real-time interface designed specifically for the complexities of Kafka, serving as the primary single pane of glass for the organization. Rather than navigating between terminal screens and disconnected monitoring tools, engineering teams gain an immediate, holistic view of cluster health. The UI allows users to visualize the intricate relationships between brokers, topics, and consumers in one place. By providing this unified context, Kpow enables teams to move from observing a symptom to identifying a cause in seconds. This turns a complex, multi-team investigation into a streamlined and methodical diagnostic process.

Closing the Quality Gap with the Kpow Prometheus Endpoint

While Kpow’s UI provides a high-resolution view for immediate diagnostics, mature operations can also require long-term data retention to understand historical trends. Kpow bridges this gap by acting as a high-fidelity data source for your existing monitoring stack. Through its dedicated Prometheus integration, Kpow exposes critical, pre-calculated data points, such as accurate consumer group lag in seconds, that are otherwise difficult to extract. This allows teams to populate Grafana dashboards with business-relevant insights, ensuring that long-term historical analysis and capacity planning are based on the true health of the data pipelines, not just raw JMX noise.

Closing the Governance Gap with Real-Time Webhook Integrations

Operational excellence is built on a foundation of accountability and transparency. Kpow solves the governance challenge by providing an automated, real-time audit log of every action taken within the platform. Through its webhook integration, Kpow can stream a live record of administrative events (such as truncating topics, deleting schemas, or editing connector configurations) directly to the communication tools your team already uses (including Slack or Microsoft Teams). This ensures that every stakeholder has visibility into cluster changes as they happen, transforming Kafka administration from an opaque process into a fully auditable and transparent operation.

Achieving Operational Maturity

Transitioning from reactive troubleshooting to proactive excellence requires a shift in how you view your infrastructure. It requires a strategy that provides immediate context, ensures high-quality data for both real-time and historical analysis, and enforces administrative governance. By utilizing a unified interface for diagnostics, enriching your existing monitoring stack with better metrics, and automating your audit trail, you can significantly reduce operational risk and improve the reliability of your most critical data systems.

This article has outlined the strategic foundation. For the practical implementation, stay tuned for our upcoming series of operational guides. We encourage you to revisit this space as we release step-by-step workflows designed to help you put these principles into practice.

Operational Transparency: Real-Time Audit Trail Integrated with Webhooks

Jaehyeon Kim — Fri, 17 Apr 2026 00:00:00 GMT

Overview

In Part 1 of our observability series, we demonstrated how to close the “Context Gap” using Kpow’s unified diagnostic interface. In Part 2, we addressed the “Quality Gap” by supercharging Grafana dashboards with high-fidelity, pre-calculated metrics.

Now, we turn our attention to the final hurdle of mature Kafka operations: transparent governance. Observability goes beyond tracking system health. It also requires a clear understanding of user actions. As Kafka adoption grows across an organization, administrative actions like managing topics, editing ACLs, or resetting consumer offsets become a major operational risk.

This guide explores how to close the “Governance Gap” by moving from an opaque black box of CLI commands to a fully transparent, real-time audit trail integrated directly into your team’s communication channels.

This is Part 3 of the Kafka Observability with Kpow: Driving Operational Excellence series.

Part 1: Rapid Kafka Diagnostics: A Unified Workflow for Root Cause Analysis
Part 2: Beyond JMX: Supercharging Grafana Dashboards with High-Fidelity Metrics
Part 3: Operational Transparency: Real-Time Audit Trail integrated with Webhooks (This article)

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Start your free 30-day trial or explore our live multi-cluster demo environment to see Kpow in action.

Problem: Architectural Black Box

Consider a standard outage scenario. A critical topic is mysteriously deleted, or a rogue script resets consumer offsets and triggers a massive wave of data reprocessing.

The immediate pain is that the engineering team is completely blind. There is no central record of who executed the change, when it happened, or why. Incident resolution stalls while platform engineers ask around to see if anyone changed anything that day.

Furthermore, this lack of traceability presents a severe compliance risk. For regulated industries, operating critical data infrastructure without an immutable, searchable record of administrative changes fails basic security and compliance audits.

Generic Solution: Bureaucracy & Brittle Parsing

To manage administrative risk and establish an audit trail, organizations without a dedicated tool typically fall back on a combination of two generic approaches, both of which introduce significant friction.

Option A (Bureaucracy): Platform teams lock down the cluster completely. Every minor configuration change must go through a disconnected ticketing process, waiting for an approval. Without an automated approval workflow, this manual gatekeeping effectively kills developer velocity and ruins any chance of providing a self-service data platform.

Option B (Log Parsing): Teams attempt to parse Kafka’s complex, raw server logs to piece together an audit trail. This is a brittle, highly technical process. It is difficult to maintain and almost impossible to alert on in real time, meaning you only find out about a destructive change after the incident has already occurred.

Solution: Data Governance and Webhook Integration

Kpow solves the “Governance Gap” seamlessly by providing a complete framework built on two pillars: preventative access control and real-time observability.

First, Kpow’s Role-Based Access Control (RBAC) allows you to safely delegate self-service capabilities by mapping your organization’s existing user groups to specific Kafka actions. Instead of locking down the cluster entirely, you can enforce granular permissions. For example, you can grant application teams full control to edit, produce to, and inspect their specific domain topics (such as payments_*), while explicitly denying access to sensitive audit topics. Meanwhile, platform administrators retain the global authority to manage Kafka ACLs and oversee cluster-wide infrastructure.

Second, as part of its Data Governance capabilities, Kpow automatically records every authorized user action in a secure audit log. This captures state-changing actions known as mutations (such as creating topics, resetting offsets, or modifying ACLs) as well as read-only queries (such as searching topic data via Data Inspect).

Moreover, instead of forcing administrators to constantly check a centralized dashboard to see these logs, Kpow’s Webhook Integration pushes these events instantly to the communication tools your team already lives in. Kpow seamlessly integrates with Slack, Microsoft Teams, and any custom platform via generic HTTP webhooks.

Putting it into Practice: Configuring a Slack Audit Trail

The following step-by-step walkthrough demonstrates how to turn Kafka administrative actions into instant Slack alerts.

Step 1: Configure the Slack App and Webhook

To integrate Kpow with Slack, you need to create a Slack App and generate an incoming webhook URL.

Create a Slack app: Navigate to the Slack API website and click on “Create New App”. Choose to create it “From scratch”.

Name your app and choose a workspace: Provide a name for your application and select the Slack workspace you want to post messages to.

Enable incoming webhooks: In your app’s settings page, go to “Incoming Webhooks” under the “Features” section. Toggle the feature on and then click “Add New Webhook to Workspace”.

Select a channel: Choose the channel where you want the Kpow notifications to be posted and click “Allow”.

Copy the webhook URL: After authorizing, you will be redirected back to the webhook configuration page. Copy the newly generated webhook URL. This URL is what you will use to configure Kpow.

Step 2: Launch Kpow with Webhook Environment Variables

Once you have your webhook URL, you can configure Kpow using a few simple environment variables.

Setting WEBHOOK_VERBOSITY to MUTATIONS ensures that Kpow only sends alerts for state-changing actions, preventing notification fatigue from standard data queries.

WEBHOOK_PROVIDER: slack
WEBHOOK_URL: <SLACK-WEBHOOK-URL>
# Accepts MUTATIONS, QUERIES, or ALL (default: MUTATIONS)
WEBHOOK_VERBOSITY: MUTATIONS

Step 3: Verify the Integration

To test the integration, perform a state-changing action in Kpow, such as creating and deleting a topic.

View audit logs

After performing these actions, you can verify they have been logged by navigating to Settings > Audit log in the Kpow UI.

On the Slack channel, you should see messages detailing the actions. Each message includes information such as the user who performed the action, the type of action (e.g., create-topic), and the cluster environment name.

Conclusion

When platform teams lack visibility into administrative actions, they are forced to rely on restrictive bureaucracy or brittle log parsing. Both options hinder developer productivity and extend incident resolution times.

By combining granular RBAC with streaming Kpow’s built-in audit log directly into Slack, Microsoft Teams, or custom webhooks, you replace the architectural black box with transparent, automated governance. Every critical mutation is safely restricted and instantly visible to the team, satisfying compliance requirements while enabling developers to work securely and autonomously.

This concludes our three-part series on driving operational excellence. By combining unified UI diagnostics (Part 1), high-fidelity Prometheus telemetry (Part 2), and transparent governance by real-time webhook integration (Part 3), you have the blueprint to transition your Kafka operations from a reactive guessing game into a proactive, mature observability posture.

Next steps

Explore Kpow in your own environment with a free 30-day trial.

If you need assistance managing your Kafka environment, reach out to our engineering support team at support@factorhouse.io.

Rapid Kafka Diagnostics: A Unified Workflow for Root Cause Analysis

Jaehyeon Kim — Fri, 17 Apr 2026 00:00:00 GMT

Overview

In distributed systems, the biggest barrier to resolving incidents is rarely a lack of data. It is the Context Gap.

When a data pipeline stalls, operational signals are often fragmented. Infrastructure metrics like broker CPU are isolated in one dashboard, while application performance data like consumer lag resides in another. Engineers are forced to manually correlate these disconnected views, wasting critical minutes switching between terminals and monitoring platforms to piece together a coherent picture of the system.

Kpow eliminates this friction by providing a unified, real-time interface that bridges the gap between infrastructure, data, and consumers.

In this guide, we demonstrate exactly how Kpow closes the Context Gap. Using a real-world “Zombie Consumer” scenario, we walk through a streamlined diagnostic workflow that allows you to move from a vague symptom to a definitive root cause and resolution.

This is Part 1 of the Kafka Observability with Kpow: Driving Operational Excellence series. You can read the full strategy in the main series article and access the associated posts as they become available.

Part 1: Rapid Kafka Diagnostics: A Unified Workflow for Root Cause Analysis (This article)
Part 2: Beyond JMX: Supercharging Grafana Dashboards with High-Fidelity Metrics
Part 3: Operational Transparency: Real-Time Audit Trail integrated with Webhooks

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Start your free 30-day trial or explore our live multi-cluster demo environment to see Kpow in action.

Scenario: Silent Stall

Imagine you are managing an orders-fulfillment service consuming from an orders topic with three partitions.

The alert comes in from customer support. Some customers are complaining that their orders haven’t been completed. It is not a total system outage. Most orders are flowing through and generating shipping labels, but a persistent number of requests seem to be vanishing into a black hole.

You immediately check your primary monitoring dashboard. The Kubernetes pod status is Running. The overall service throughput looks active. There are no critical alerts for broker failures or network outages. According to your infrastructure tools, the system is healthy.

In reality, a legacy upstream system has injected a “poison pill” message into the topic. Specifically, a JSON record where the amount field was sent as a string (“ONE THOUSAND DOLLARS”) rather than the expected numeric representation. This schema mismatch has hit Partition 2. It caused that specific consumer thread to crash, catch the error, seek back to the problematic offset, and retry in an infinite loop.

Because other partitions are processing normally, the aggregated metrics mask the failure. To the Kafka broker, the stuck consumer looks alive because it is still sending heartbeats. To the customers whose orders landed on Partition 2, the service is dead.

Traditional Investigation: Fragmented Tools

Diagnosing this kind of partial failure usually forces you to piece together clues from multiple disconnected sources, wasting valuable time.

Infrastructure Check

Your first instinct is to check the infrastructure metrics in Datadog or Grafana. You look for CPU spikes or memory leaks, but you find nothing. Because the majority of the consumer instances are running without an issue, the aggregated metrics hide the single stuck instance. The failure is completely invisible at the infrastructure level.

Tooling Gap (Skipping the Producer)

Logically, your next step should be to verify the Producer: is data actually entering the system? However, native CLI tools like kafka-topics.sh provide configuration details, not real-time throughput. To check “messages per second” in the terminal requires hacky shell scripts or manual offset math. Because this is slow and painful, you are forced to skip the producer check entirely and jump straight to the consumer, leaving a critical blind spot in your diagnosis.

Consumer CLI Query

You switch context to kafka-consumer-groups.sh. The output shows the group state is Stable, which is technically true but misleading. However, looking closely at the partition list, you finally spot the anomaly: the LAG for Partition 2 is skyrocketing while the others remain near zero. You now know where the blockage is, but because the CLI cannot correlate this with live data, you have no clue what caused it.

Log Dive

Finally, you open Splunk or Elasticsearch to hunt for the root cause. You find a stream of Schema Mismatch exceptions intermingled with thousands of successful processing logs. You can see the error type, but application logs often redact sensitive payloads or fail to capture the specific offset context. You know the consumer is choking on specific data, but you cannot see the message causing the choke. You cannot fix data you cannot see.

Kpow Workflow: Unified Diagnostic Workflow

To address this, we replace the fragmented toolset with a unified workflow in Kpow. The following steps demonstrate how to troubleshoot the failure by tracing the path from the cluster down to the specific message. We then resolve the issue immediately, all within a single interface.

Step 1: Broker Metrics - Ruling Out Infrastructure

Because only some orders are missing, it is unlikely that the brokers are the root cause. However, it is best practice to perform a quick health check to be certain.

In the Overview tab, review the summary statistics and metrics graphs. You can see that the cluster throughput is active and there are no indicators of under-replicated partitions. The brokers are running fine.

Step 2: Topic Exploration - Spotting Deviations

Navigate to the Topics menu.

In the Overview tab, select the orders topic. At first glance, there is no clear evidence of an issue based on the summary statistics and metrics. The aggregate data looks healthy, and it is because the majority of partitions are still processing traffic.

Switch to the Details tab to investigate further. Look at the topic partitions table. Here you identify the discrepancy immediately. In Partition 2, messages are being written continuously, but the read rate is zero. Traffic is entering the partition but nothing is leaving.

Step 3: Consumer Analysis - Identifying Root Cause

Navigate to the Consumers menu and select the relevant consumer group.

In the Overview tab, you can spot an issue immediately. The total consumer lag is increasing, even though no idle members are detected and messages continue to be consumed globally.

Switch to the Details tab for a granular view. You can now see that one specific group member is stuck on a particular offset. It is not reading any new data, and the lag for that specific partition is accumulating rapidly. To confirm what is blocking the pipeline, use the action menu on the right and select Inspect data.

This opens the Data inspect view with the affected topic partition pre-selected. You can use kJQ to filter the specific offset under investigation. By clicking the search button, you can confirm that the message value is malformed, causing the application logic to crash on this record.

Step 4: Resolution - Executing Staged Mutations

Now that the poison pill is identified, you need to unblock the partition.

Go back to the Consumers menu. In the action menu for the stuck group member, select Skip offset.

This action initiates a Staged Mutation, and its status is marked as Scheduled. Note that, for Kpow to safely apply this offset change, the consumer group status must be Empty to prevent state conflicts.

Stop your consumer application instances. Kpow detects that the group has stopped and automatically applies the staged mutation.

Once the mutation status shows as succeeded, restart your consumer application. You can now see the consumers reading from all partitions as usual. The stuck lag drains immediately, and the missing orders begin to process.

Conclusion

In the “Silent Stall” scenario, we saw that relying on fragmented tools creates a critical context gap, forcing engineers to waste time manually correlating logs, metrics, and CLI output.

Kpow solves this with a unified workflow. By integrating consumer lag, data inspection, and resolution tools into a single interface, we transformed a complex investigation into a linear path. We didn’t just identify the poison pill; we resolved it immediately using Staged Mutations without leaving the platform.

In Part 2: Beyond JMX, we will show you how to close the Quality Gap by feeding high-fidelity Kafka metrics directly into your Grafana dashboards.

Next steps

Explore Kpow in your own environment with a free 30-day trial.

If you need assistance managing your Kafka environment, reach out to our engineering support team at support@factorhouse.io.

Integrate Kpow with StreamNative Cloud

Jaehyeon Kim — Tue, 31 Mar 2026 00:00:00 GMT

Overview

StreamNative delivers a fully managed, Kafka-native cloud service designed for exceptional elasticity and enterprise-grade resilience. However, as with any high-performance data infrastructure, engineering teams still require an intuitive way to monitor, manage, and explore their live data streams.

Kpow serves as a versatile engineering toolkit for this modern streaming environment. Fully compatible with StreamNative Kafka out of the box, Kpow connects directly to your managed brokers and Schema Registry using standard Kafka protocols. This provides a unified, single-pane-of-glass experience without the need for proprietary plugins, sidecars, or complex custom configurations.

Kpow connects natively to a wide range of Kafka vendors and managed service providers. See our Kafka Providers documentation to learn more.

About Factor House

Factor House is a leader in real-time data tooling, empowering engineers with innovative solutions for Apache Kafka® and Apache Flink®.

Our flagship product, Kpow for Apache Kafka, is the market-leading enterprise solution for Kafka management and monitoring.

Start your free 30-day trial or explore our live multi-cluster demo environment to see Kpow in action.

Prerequisites

To connect Kpow to StreamNative Cloud, you must have the following resources provisioned:

A running StreamNative Kafka cluster.
Connection Details: Your Kafka Bootstrap Server address and, optionally, Schema Registry URL.
Authentication: A Service Account email and its associated JWT Token.
Authorization: The Service Account should be assigned a sufficiently privileged role, such as instance-owner , to grant Kpow the broad administrative access required for effective management.
A Kpow Enterprise License: Get a free 30-day trial.

Quick Start

The fastest way to connect Kpow to StreamNative Cloud is using Docker.

Run the following command in your terminal, replacing the placeholder values with your specific cluster details:

docker run -d -p 3000:3000 \
  --env ENVIRONMENT_NAME="StreamNative Kafka" \
  --env BOOTSTRAP="[BOOTSTRAP_SERVER_ADDRESS]:9093" \
  --env SECURITY_PROTOCOL="SASL_SSL" \
  --env SASL_MECHANISM="PLAIN" \
  --env SASL_JAAS_CONFIG='org.apache.kafka.common.security.plain.PlainLoginModule required username="[SERVICE_ACCOUNT_EMAIL]" password="[JWT_TOKEN]";' \
  --env LICENSE_ID="<LICENSE_ID>" \
  --env LICENSE_CODE="<LICENSE_CODE>" \
  --env LICENSEE="<LICENSEE>" \
  --env LICENSE_EXPIRY="<LICENSE_EXPIRY>" \
  --env LICENSE_SIGNATURE="<LICENSE_SIGNATURE>" \
  factorhouse/kpow:latest

Notes

License details: The license details can be obtained from your signup email or via the Factor House license portal.
Authorization configuration: For brevity, Kpow authorization configuration has been omitted. See Simple Access Control to enable necessary user actions.

Once the container starts, navigate to http://localhost:3000. You will see an overview of your OCI topics, brokers, and consumer groups.

Configuration Details

Connecting to StreamNative Cloud requires understanding a few specific authentication nuances.

JWT Authentication via SASL/PLAIN

StreamNative uses JWT tokens for authentication. While standard Kafka often uses OAUTHBEARER for tokens, StreamNative leverages the SASL/PLAIN mechanism where the “username” is your Service Account email and the “password” is the raw JWT token.

Access Control

As noted in the prerequisites, Kpow connects using the instance-owner role to fetch metadata and execute management actions against the StreamNative APIs. Once Kpow is connected, you can configure Kpow’s internal Authorization (via RBAC or Simple Access Control) to restrict exactly what individual engineers or teams are allowed to see and do within the UI.

Ecosystem Integration

Integrating Kpow with the StreamNative Schema Registry requires two specific configuration adjustments due to how the proxy is implemented.

Basic Authentication

The StreamNative Schema Registry relies on Basic Authentication (USER_INFO). However, it only validates the password (which must be your JWT token). The username can be any string.

Observation Version Requirement

By default, Kpow uses an highly optimized “Version 2” observation strategy for Confluent-compatible registries. This strategy hits the GET /schemas endpoint to download all schemas in one efficient bulk request.

However, StreamNative Cloud’s Schema Registry proxy does not implement this bulk endpoint. It only implements the older /subjects endpoints. Because the bulk endpoint does not exist, StreamNative will return a 404 Not Found error.

In order to fix, you must tell Kpow to use the legacy “Version 1” observation strategy. This forces Kpow to query /subjects first, and then fetch individual schemas one by one. You do this by setting SCHEMA_REGISTRY_OBSERVATION_VERSION=“1”.

To attach the Schema Registry, append the following environment variables to your docker run command:

  --env SCHEMA_REGISTRY_NAME="StreamNative Schema Registry" \
  --env SCHEMA_REGISTRY_URL="[SCHEMA_REGISTRY_URL]" \
  --env SCHEMA_REGISTRY_AUTH="USER_INFO" \
  --env SCHEMA_REGISTRY_USER="any-user" \
  --env SCHEMA_REGISTRY_PASSWORD="[JWT_TOKEN]" \
  --env SCHEMA_REGISTRY_OBSERVATION_VERSION="1"

Production Deployment

When you are ready to move from a local Docker test to a production deployment, we recommend the following paths:

Kubernetes

For deploying Kpow to Kubernetes clusters running alongside your infrastructure, we recommend using our official Helm Charts.

Bare Metal / VM

If you prefer running Kpow directly on a Virtual Machine, you can download the Kpow JAR file.

Kpow JAR Quickstart

Conclusion

Kpow provides a powerful, single pane of glass view into your StreamNative Cloud infrastructure. By using standard Kafka protocols and addressing the specific nuances of StreamNative’s proxy endpoints, you can unify your Kafka clusters and Schema Registry environments in minutes.

Explore these features in your own environment with a free 30-day trial of Kpow.

If you need assistance with your StreamNative integration, reach out to our engineering support team at support@factorhouse.io.

What the IBM Confluent acquisition means for Kafka users

Chad Harris — Thu, 26 Mar 2026 00:00:00 GMT

On March 17, 2026, IBM completed its $11 billion acquisition of Confluent. The deal was announced in December and closed quickly. Since then, the question for anyone running Kafka in production has been straightforward: what changes, and what should we do about it?

Early Signals

The early signs are not reassuring. Hundreds of former Confluent employees have appeared on LinkedIn with #opentowork since the acquisition closed. Unofficial reports suggest the number may be as high as 800. There’s been no official statement from IBM or Confluent on the scale of the cuts.

For engineering and platform teams, the concern isn’t the layoffs themselves but what they imply about product velocity. Confluent’s value has always been driven by deep engineering investment: in the Kafka core, in the connector ecosystem, in Schema Registry, in the cloud platform. Fewer engineers means slower bug fixes, slower feature development, and less community engagement on KIPs. If you depend on Confluent’s managed connectors or cloud-native features, the pace at which those improve is directly relevant to your operational risk.

What History Tells Us

Two precedents are worth studying, though neither is a perfect analogy.

The first is Oracle’s acquisition of Sun Microsystems in 2010, which brought MySQL into Oracle’s hands. Over time the open-source release cadence slowed, and most Linux distributions switched their default database to MariaDB. Oracle never killed MySQL, but it deprioritised it. The key difference: Oracle’s core business (Oracle Database) competed directly with MySQL, giving Oracle an incentive to let it stagnate. IBM has no equivalent competing product, so the incentive structure is different.

The more instructive precedent is Red Hat’s decision, under IBM ownership, to replace CentOS Linux with CentOS Stream in late 2020. CentOS had been a free, stable, binary-compatible rebuild of RHEL. Thousands of organisations ran production workloads on it precisely because it tracked a stable RHEL release. CentOS Stream changed the model to a rolling preview of the next RHEL version. The product still existed, but the contract it had with its users (stability equivalent to RHEL) was broken. Organisations that had standardised on CentOS faced a forced migration they hadn’t planned for.

The pattern in both cases is the same: the open-source project survived, but the terms of engagement changed in ways that imposed real costs on users who had built deep operational dependencies.

Where the Risk Actually Sits

Apache Kafka the open-source project is likely safe. It has multiple companies invested in its development. AWS, Redpanda, Aiven, and others all have commercial interests in the Kafka protocol continuing to thrive. IBM has publicly stated its acquisition rationale depends on Kafka’s openness and adoption.

The risk is in the commercial and proprietary layer. Specifically:

Confluent-managed connectors. If your data pipelines use Confluent’s fully managed or licensed source and sink connectors, you have a hard dependency. There’s no direct equivalent on other platforms. The open-source Kafka Connect framework is portable, but the managed hosting and operational wrapper around it is not.

Schema Registry. Confluent Schema Registry is the de facto standard for schema management in Kafka environments, but it’s a Confluent product, not part of Apache Kafka. Alternative implementations exist (Apicurio Registry from Red Hat and AWS Glue Schema Registry being the most mature) but migration involves changing client configurations, compatibility testing, and potentially reworking CI/CD pipelines that integrate with the registry API.

Confluent Cloud platform dependencies. Features like cluster linking, Stream Governance, the Confluent Terraform provider, and the Confluent CLI tooling don’t have 1:1 replacements in the broader ecosystem. If your operational workflows are built around these, a migration is materially harder than “just point at a different broker.”

Pricing and licensing. This is the quietest risk but potentially the most impactful. IBM has a well-established pattern of monetising enterprise customers through licensing changes. Confluent’s current pricing may not survive contact with IBM’s enterprise sales model.

What To Do Now

You don’t need to migrate anything today. But you should know your exposure. A few concrete questions worth putting to your platform team:

Which Confluent-specific features are we actually using, versus standard Kafka APIs?
If we had to move to MSK, Redpanda, or Aiven, what would break? What would need to be re-implemented?
How much of our CI/CD and data pipeline tooling is tied to Confluent-specific features, versus standard APIs like Schema Registry or Kafka Admin that other providers also support?
Are our monitoring and operational workflows coupled to Confluent Control Center?

That last point is sometimes overlooked. If your engineers’ daily Kafka workflows (topic inspection, consumer group management, ACL configuration, data inspection) run through Confluent tooling, that’s both a technical dependency and a familiarity dependency. It’s one of the things that makes a provider migration feel harder than it technically is.

This is an area where decoupling is straightforward. Vendor-agnostic Kafka management tools exist specifically to work across providers. Factor House’s Kpow, for example, works with Confluent Cloud, AWS MSK, Redpanda, Aiven, and self-managed Kafka. Regardless of which tool you choose, the underlying point is the same: building your observability and operational workflows on a tool that is independent of your Kafka provider means one less thing to migrate if your provider changes.

What To Watch

A few signals that will indicate how this acquisition is playing out over the coming months:

Kafka open-source contribution cadence. Track commit activity from Confluent-affiliated contributors on the Apache Kafka repo. A sustained decline would suggest engineering resources are being redirected.
KIP activity. Kafka Improvement Proposals are the mechanism for protocol-level changes. If Confluent-driven KIPs slow down or shift toward IBM enterprise use cases, that tells you something about roadmap priorities.
Confluent Cloud pricing changes. Any movement toward consumption-based pricing with enterprise minimums, or changes to the free tier, would signal IBM’s monetisation strategy taking hold.
Connector ecosystem investment. Watch whether the managed connector catalogue continues to expand, or whether investment shifts toward IBM’s own integration tooling (App Connect, MQ).

None of this is cause for panic. But the cost of understanding your dependencies now is low, and the cost of discovering them under pressure is high.