Kafka Streams is a Java library for stream processing that runs inside your own application, using Kafka itself for coordination rather than a separate processing cluster. Its stateful operations keep their working data in local RocksDB state stores, backed by changelog topics in Kafka, and that local state is where most production surprises live. This page is the streams topic’s root; the documentation map and ksqlDB sit under it, and the hub holds the wider cluster.
Operational and production concerns
State stores are rebuilt when instances move. When a Streams instance joins or leaves the consumer group, the group rebalances, and a task landing on a new host must restore its RocksDB state from the changelog topic before processing resumes. In cloud environments this happens constantly: Kafka Streams consumers at Pinterest restart frequently through host restarts, rolling deployments and spot instance preemptions, each one triggering a full group rebalance and RocksDB state rebuild. Standby replicas and static group membership exist precisely to shrink that window.
Stability under load comes from a small set of configurations: num.stream.threads sizes processing parallelism within an instance, and commit.interval.ms trades end-to-end latency against commit overhead. Exactly-once semantics are enabled with processing.guarantee=exactly_once_v2, which wraps processing and offset commits in Kafka transactions. A poison pill still needs explicit handling: Kafka Streams requires a custom deserialization exception handler, because unlike Spring Kafka it does not catch deserialization failures for you.
State store capability is still evolving. Versioned key-value stores (KIP-914) keep multiple record versions per key, so timestamped reads can return the latest record as of a given time, with improved processing semantics when used from the DSL.
The scale of real Streams estates surprises people who think of it as the toy option. One enterprise we know of runs roughly 400 Kafka Streams applications alongside heavy Spark usage: the library model is not the stepping stone to a “real” processing cluster, it is the workhorse in parallel with one. At that count, the operational concerns above stop being edge cases and become weekly events, which is why the state-store and rebalance behaviour deserves the attention this section gives it.
The failure modes are also not all solved upstream. Airbnb’s Evergreen, the flagship public Kafka Streams deployment, has its one-record-at-a-time throughput identified as a known production challenge in public talks, with no published resolution. Adopting the library means adopting its constraints, and the honest posture is to know which ones are still open.
Architecture and integration
The architectural choice Kafka Streams represents is processing without a processing cluster. The library runs in your microservice’s own JVM, scales by running more instances of that service, and needs no job submission, no cluster manager and no separate operational surface. That is the low-ops trade it offers against standalone engines.
The line where a standalone engine wins is well defined. Apache Flink is preferable when you need to join Kafka streams with non-Kafka sources, when batch and streaming workloads must share processing logic, or when the scale of state exceeds what RocksDB on a single JVM can comfortably handle. Inside those limits, the library model carries serious production workloads.
The topology patterns in real deployments go well beyond word counts. Airbnb’s Evergreen implements a stateful actor model as a Kafka Streams topology with exactly-once processing semantics. The New York Times runs its Denormalizer and Elasticsearch ingestion nodes on Kafka Streams, using it to join and reshape published content on the way to search. Pinterest uses it for in-process stream processing in monetisation and metrics paths. The shared shape: stateful joins, windowed aggregations and enrichment running inside the services that own the data, with Kafka as the only shared infrastructure.
My colleague Thomas Crowley’s framing is the one I use: Flink is complementary to Kafka, not a replacement for Kafka Streams, and increasingly essential as the streaming ecosystem grows. The question is never which one is better, it is which side of the well-defined line above your workload sits on. Plenty of estates run both, and the maturity path we see teams walk goes from first experiments with the library through to business-critical platforms running both engines against the same cluster.
One war story from our own engineering, because subtle is the word for how stateful topologies fail. Supporting Kafka 4.0.0, a change of ours initially appeared to work: the state store was invoked for deltas and individual aggregation data points appeared in the tables. But no messages were being sent downstream in the topology. Every intermediate signal said healthy while the output said nothing at all. With stateful stream processing, verify the output topic, not the state store. The store filling up is not the same thing as the topology working.
FAQ
What is the purpose of Kafka Streams?
Kafka Streams exists to do stream processing without a processing cluster. It is a Java library that runs inside your own application, reading from Kafka topics, transforming, joining and aggregating records, and writing results back, with state kept in local RocksDB stores backed by changelog topics. Its purpose relative to engines like Flink or Spark is operational: the same class of stateful processing, scaled by running more instances of your own service, with Kafka as the only shared infrastructure.