A Kafka use case is an architectural pattern with the numbers to prove it works. The recurring ones are real-time analytics, event sourcing, change data capture, log aggregation and asynchronous communication between microservices, and what a platform engineer actually needs is not the list but the reference architectures behind it: how Kafka sits alongside Flink, Spark or a warehouse, and what happened when someone ran that shape at volume. The hub indexes those deep dives; this page is the argument layer.
Architectural validation and design patterns
The strongest validation is breadth inside one company. PayPal runs Kafka for seven primary use cases, and each of the seven processes over 100 billion messages per day. One cluster technology carrying analytics, synchronisation and application messaging at that volume is the argument for the pattern in a single fact.
The pairing with a stream processor is now the default shape rather than an advanced variant. Teams running Kafka in production overwhelmingly run processing on top of it, and practitioner communities around Flink are full of teams actively using both. The decoupling pattern underneath is always the same: services publish events to topics, and consumers subscribe without the producer knowing they exist, which is what lets each side change independently.
Our position on where this is heading is explicit: Kafka as the streaming backbone, Flink as the processing layer, and Iceberg as the lakehouse table format is rapidly becoming the default modern data architecture. The use-case list above is converging on that one shape, with each pattern occupying a different slice of it. The evidence is already in the reference deployments: more than 100 Flink jobs consume Kafka events in Airbnb’s User Signals Platform alone, and the adoption pattern we see across the market is teams arriving for Kafka and adding Flink within the same planning cycle.
The cautionary pattern is the one that funds most migrations: Netflix’s Content Finance services hit cascading failures and inconsistent state because synchronous RPC calls created hard dependencies between services, and the fix was exactly this architecture, Kafka-based events with Flink doing enrichment and reordering. The use case was not “we want streaming”. It was “our call graph is a failure graph”.
Scale and performance metrics
The published production numbers set the ceiling. LinkedIn, where Kafka originated, runs more than 100 Kafka clusters on 4,000+ brokers, carrying 100,000+ topics and 7+ million partitions, handling more than 7 trillion messages per day on 2023 figures. DoorDash averages around 4 million messages per minute. Those numbers are not targets, they are proof the architecture does not run out of road.
Sizing guidance for ordinary clusters is more useful than the records. For clusters above twelve brokers, partition counts around twice the broker count tend to work as a starting point for a topic. Capacity planning tracks three consumption rates: network, disk and broker count, and the plan is whichever of the three runs out first.
The discipline that keeps costs sane is restraint. Scaling triggers partition movement and rebalancing that temporarily makes the cluster slower, exactly when someone thought it would get faster, so a cluster scales for headroom or real throughput growth and for nothing else.
Read benchmark numbers with their assumptions attached. The public Kafka benchmarks from Aiven, Google Cloud and LinkedIn typically use message sizes between 100 bytes and 10 KB, which is exactly Kafka’s sweet spot, so a benchmark proves what Kafka does with benchmark-shaped traffic. If your records are 200 KB documents, no published throughput number transfers.
And copy the big deployments’ partition discipline rather than their totals, the practice the topic vs partition page owns: the interesting number in a capacity review is never the cluster total, it is partitions per broker and headroom per topic.
Business justification
The business case for Kafka almost always starts as a complaint about batch. Shopify’s initial deployment replaced a plain log-file batch system whose delay between an event happening and the event being visible in dashboards and Hadoop was the whole problem. The same move plays out in conservative industries: in 2020 a handful of engineers at a major North American bank concluded batch workloads were insufficient and lobbied leadership to build Kafka for event streaming. The pattern for the funding conversation is identical both times: name the decision being made late because the data arrives late, and price that delay.
The REST-to-events argument is about failure coupling rather than speed. A synchronous call chain is only as available as its least available service, while events through Kafka let a slow consumer lag without taking the producer down with it. Once Kafka is in the picture, its purpose extends to enabling event-driven collaboration between operational services, which is a capability argument rather than a cost one.
Change data capture is often the cleanest first project to fund. PayPal uses Kafka as a CDC transport, streaming operational database changes for near-real-time replication across services, a use case with an obvious before and after and no application rewrite. Analytics pipelines follow the same logic, with warehouse and stream tooling consuming from Kafka once the events exist.
For the stakeholder deck, three numbers do different jobs. Adoption comes first: over 80% of Fortune 100 companies use Apache Kafka, which retires the “is this proven” question in one line. For latency, PayPal’s Kafka-based rewrite of its analytics and compliance pipeline took reporting time from 12 hours to seconds, the before-and-after shape every batch replacement pitch needs. And on cost, Notion’s Kafka pipeline delivers net savings of over $1 million per year, which answers the CFO’s version of the question. Longevity closes it: Shopify has run Kafka for data movement since 2014, and by 2019 JPMorgan Chase had 400 applications connected. This is not a bet on a trend.
My one caution for the pitch: do not oversell the mechanism. Kafka does not make a slow team fast or a broken data model sound, and the failure stories in this series are all misconfigurations and missing observability rather than Kafka failing. Fund the use case and the operational discipline together, or the 12-hours-to-seconds slide becomes someone else’s incident review.
FAQ
What are some common use cases for Apache Kafka?
The recurring production use cases are real-time analytics pipelines, event-driven communication between microservices, change data capture from operational databases, event sourcing, and log aggregation. The pattern underneath all five is the same: services publish events to durable topics, and any number of consumers read them independently. The reference deployments on this page are these five patterns at volume.
What is Apache Kafka commonly used for in data engineering?
In data engineering specifically, Kafka is the ingestion and transport backbone: CDC streams carry database changes into the platform, application events flow to the warehouse and lakehouse, and stream processors like Flink consume from Kafka to build derived datasets in real time. It replaces the nightly batch window with a continuous feed, which is the 12-hours-to-seconds change PayPal’s analytics rewrite documents.
Is Apache Kafka an ETL tool?
No. Kafka is the transport and buffer layer ETL runs across, not the ETL tool itself. Extraction is done by producers and CDC connectors, transformation by stream processors such as Flink or Kafka Streams, and loading by sink connectors. What Kafka changes about ETL is the T’s timing: transformation happens continuously on the stream instead of in a batch window.