Reduce Kafka spend and operational risk: practical techniques for cluster consolidation

This session, co-presented with Chad Harris, Solutions Architect at Factor House, starts by acknowledging the original use case Kafka was built for: solving the difficult problem of data integration by consolidating pipelines across operational systems and data warehouses into a single distributed commit log. Kafka is now mainstream well beyond that, powering event-driven services, real-time streaming compute, and analytics and ML workloads as well. As more services integrate onto it and it becomes the backbone of an organization's data flow, the question eventually becomes whether a single Kafka cluster is still enough, or whether it's time to move to multiple clusters. In practice we've seen two common but problematic patterns: staying on one monolithic cluster because it's easier, or the opposite extreme of every team getting its own cluster, ending up with 50 clusters before long.

The signal that you've reached this point is usually a degraded cluster: under-replicated partitions, high network egress cost, disk under pressure, CPU pressure on some brokers. At that point you need to understand the impact on the services and workloads riding on the cluster. Large organizations typically define service tiers — Tier 0/1 services that are core to the business and need high availability and tight SLOs, down to Tier 3/4 supporting or analytical services that can tolerate more relaxed SLOs. You also need to weigh data criticality (regulated data versus internal versus public data) and your disaster recovery posture for each workload — recovery time objective and recovery point objective — before deciding how to respond to a degraded cluster. As Leslie Lamport put it, a distributed system is one in which the failure of a machine you didn't even know existed can render your own computer unusable — and with Kafka, that failure can take out the critical services depending on it.

A single Kafka cluster at scale often can't serve every use case well. Low-latency workloads need one kind of cluster configuration; latency-tolerant analytical workloads are often just paying low-latency prices for no benefit, since you pay for the low-latency capability across every workload sharing that cluster, even the ones that don't need it. Regulated industries sometimes push toward physically separate clusters too — not because a regulation literally requires it, but because it's much easier to prove data segregation to an auditor when a sensitive workload runs on its own cluster. Global organizations running services across regions need regional clusters for latency and reliability, and acquisitions are a very common way organizations end up with multiple Kafka clusters — and sometimes multiple Kafka providers at once, each with its own schema registry and tooling. It's not unusual to see an organization running two or more Kafka providers after a merger, and we've seen cases of five providers in one organization, including a mix of self-hosted and managed.

The good news is Kafka has had over a decade to mature since it was open-sourced, and the Kafka wire protocol going mainstream means many vendors — Confluent, AWS MSK, Aiven, Redpanda, WarpStream (since acquired by Confluent), AutoMQ, Bufstream, StreamNative, and others — implement compatible brokers in different languages and architectures, which makes migrating between them, while still hard, at least protocol-compatible. Kafka itself has grown tiered storage and diskless offerings, and there's a growing set of options to ingest directly from brokers into Apache Iceberg for analytical workloads, removing a lot of the operational overhead of separately managed ingestion pipelines — not yet proven at scale in production everywhere, but a space worth watching.

Consolidating doesn't necessarily mean moving everything onto one cluster — it means matching each workload to the right cluster for its needs. The approach: understand your main cost, reliability, and DR drivers; understand what cluster types are available for your use cases; review your service tiering, data criticality, and DR strategy; and then plan the migration incrementally and strategically.

On cost specifically: a lot of organizations don't understand what's actually driving their Kafka bill. Stanislav Kozlovski's Kafka cost calculator (from the Kafkanated podcast) is a useful reference for breaking this down — for a typical 6-broker cluster on AWS, network transfer cost is usually one of the biggest cost drivers. Analytical workloads sharing a cluster with critical low-latency services are often a large share of that network cost, and could instead move to a cluster that writes straight into an Iceberg layer without touching disk at all. Beyond network, cost is also driven by infrastructure choice (instance types, availability-zone distribution, disk types), configuration (compression, retention, cross-AZ transfer, partition counts), and the cluster type itself.

Configuration is the fastest lever, and it buys you time rather than solving the underlying problem. Compression reduces network and disk bytes — benchmark it yourself rather than trusting vendor numbers, and make sure your batch size and linger.ms are tuned so you're compressing a full batch rather than single records. Fetch-from-follower (introduced around Kafka 2.4, though historically not all providers exposed it) lets a consumer read from an in-sync replica in its own availability zone instead of the partition leader across zones, avoiding cross-AZ transfer cost when your Kafka and consumer pods share an availability zone. Quotas let you throttle less-critical workloads so they don't consume disproportionate cluster resources.

Once configuration has bought you time, migration is the real fix, and it is hard. Planning starts with understanding your topology in detail: does order matter for a given topic, what's your replication and rollback strategy, is schema registry migration part of the plan (schema registries are often harder to migrate than the Kafka cluster itself, and it's frequently the afterthought that trips teams up because most schema registries aren't compatible with each other), do you need to preserve consumer offsets, and can you tolerate duplicates. Consumer offset preservation across a migration is possible today, but only through proprietary vendor tooling — WarpStream Orbit, Redpanda shadowing, AutoMQ's equivalent, and likely more soon — which helps you migrate into that vendor, not out of it. KIP-1279, which introduces cluster mirroring with offset-preserving replication as a first-class part of open-source Apache Kafka itself, is one of the most exciting developments to watch here, because it would make that kind of migration vendor-neutral.

Understanding your topology in detail — ideally with something like a graph database over your infrastructure-as-code or topic configuration, labeling topics and services by tier — lets you see which topics have the most Tier 0 producers and consumers, and which topics are safe to practice a migration on first. You never want your first migration attempt to be a Tier 0 topic; start with something that can tolerate a bit of downtime, and make sure you've identified every producer and consumer of a topic before you migrate it, since missing one causes an outage.

There are three broad migration orderings, chosen based on whether order and historical data matter. Producer-first: migrate producers to the target cluster, accept that you won't replay old history, let consumers drain the source topic, then migrate consumers and decommission the source. This is a straightforward approach we've used in production, for example migrating Iceberg Kafka Connect connectors onto an analytical cluster where historical replay wasn't required. Consumer-first: works well when duplicates and reordering are tolerable, such as feeding a data lake — deploy a consumer against the target cluster (or have the existing consumer read from both), switch the producer to the target, drain the source, then decommission it. When order does matter, you need replication — start MirrorMaker 2 (never MirrorMaker 1) replicating source to target, migrate consumers to the target, and rely on MirrorMaker 2's offset translation to keep consumer offsets consistent between clusters. Monitoring replication lag closely is critical here, and it's something MirrorMaker 2 isn't good at out of the box — you'll need to build or buy additional tooling for it. Once MirrorMaker 2 has fully drained the topic, you switch the producer to the target. The really hard part is that if you care about order, the producer needs a short window of downtime while the replication lag drains to zero before cutover — you can narrow that window with techniques like briefly failing producer writes while producers retry, giving MirrorMaker time to catch up, which is essentially what vendor tools like WarpStream Orbit do under the hood — but the downtime is never fully zero with open-source MirrorMaker 2, and even vendor tools like Confluent Cluster Linking still have a brief cutover gap.

We've noticed this MirrorMaker 2 lag-monitoring gap ourselves at Factor House, and we're building tooling to close it — a sneak peek of both a new terminal UI and MirrorMaker replication monitoring that will help you see the health of a migration in real time and make an informed call about when to cut over producers or consumers, or just confirm your DR replication between regions is healthy. Beyond MirrorMaker, if you're moving within a single vendor's ecosystem, use their native tooling — Confluent Cluster Linking, AWS MSK Replicator, Redpanda Shadowing, WarpStream Orbit — they'll save you significant time and effort.

Ultimately, at scale a single Kafka cluster won't fit every use case, and configuration tuning only delays that reality. Kafka migration is hard and requires the right tooling and monitoring, which doesn't come for free — but pick the right cluster type for each workload. If your use case can tolerate a bit of extra latency, like a quarter of a second, move it to a cheaper S3-backed or diskless cluster type; several of those now post P99 latencies around 100 milliseconds, which is good enough for a lot of workloads people assume need to be low-latency but actually don't.

On the audience Q&A: asked about the Confluent/IBM acquisition, the honest read is a mix of nervousness and cautious optimism — IBM didn't acquire a dying technology, and there are early positive signs they're stewarding Kafka and its open-source community well, but a lot of people are watching closely over the next 6-12 months before making decisions. Interestingly, several vendors — including WarpStream/Confluent, IBM, and Redpanda — are actively working on easier, closer-to-one-click migration tooling, which is itself an acknowledgment that customers want real optionality between providers.

Asked whether any vendor is genuinely good at migrations: none of them are better or worse than each other at helping you migrate out — they're all excellent at helping you migrate in, because that's what's in their interest. Some vendors have real proprietary advantages, like Confluent's Kora engine, which let them ship S3-backed topics well ahead of open-source Kafka. The practical approach is to use whichever vendor's platform you like, but consciously avoid the proprietary features that lock you in, so that if you ever need the option to leave, you still have it — even if you probably won't exercise it.

On diskless clusters: the driver is cost, since spinning disk or attached SSD is the single most expensive component of running Kafka, and it's common to over-scale broker count purely to get more attached disk, ending up with far more compute than you need. Object storage is much cheaper than broker-attached storage, so we're seeing a lot of migrations move analytical, latency-tolerant workloads onto diskless or S3-backed cluster types. Migrations in this direction would likely be happening faster if cluster migration itself weren't still so hard — once offset-preserving replication (like KIP-1279 aims to standardize) is mainstream, expect that trend to accelerate. Some newer offerings, like AWS MSK Express brokers, use a hybrid approach: a small amount of broker-attached storage for active, low-latency data, offloaded to S3 as quickly as possible, which keeps attached storage costs down while preserving low latency for the workloads that need it.

On whether migration is worth the cost: it depends entirely on your current spend and the expected saving. Migrations are not free, and can be genuinely expensive, but there is also a cost to not migrating. The way to think about it is a straightforward payback calculation: what would the migration cost, what would the ongoing saving be, and how many years would it take to pay that off. If a migration costs a million dollars to save a hundred thousand a year, that's a ten-year payback and probably not worth it; if it pays back within months or a year or two, it's absolutely worth pursuing. Every organization needs to run that math for its own numbers and forecast.