Kafka vs Spark: what each does and how they work together
ComparisonsKafka and Spark do different jobs and are usually used together. Apache Kafka is an event streaming platform that stores streams of events durably and lets applications publish and read them. Apache Spark is an engine that processes data at scale, and its Structured Streaming API can read a Kafka topic as its input. The question “Kafka or Spark” usually means “Kafka Streams or Spark” or “Kafka plus which processing engine”, and the answer to either depends on what already runs in your organization.
What each one is for
The Apache Kafka documentation says Kafka combines three capabilities so an event streaming use case can run end to end on one platform. The first two are publishing and subscribing to streams of events, including continuous import and export of data from other systems, and storing streams of events durably and reliably for as long as you want. The complete Kafka guide covers what that means in operation, and What is Apache Kafka? is the short introduction.
The Apache Spark documentation describes Spark as “a unified analytics engine for large-scale data processing”, with APIs in Java, Scala, Python and R. Spark does not store your events. It reads them from somewhere, processes them and writes the result somewhere.
How Spark reads Kafka
Spark reads Kafka through the Kafka source of Structured Streaming, with options such as subscribe and startingOffsets set on the stream. The Kafka integration guide says the Kafka source does not commit any offset and that Structured Streaming manages which offsets are consumed internally. In practice, a Spark job reading Kafka behaves as a consumer that keeps its own position in a checkpoint, which has consequences for monitoring. Where Spark keeps its position explains them.
Where the confusion comes from
- Both say “streaming”. Kafka streams events between systems. Spark Structured Streaming processes them.
- Kafka has its own processing library. Kafka Streams is a processing library that ships with Kafka, covered in the Kafka Streams section of the guide, and it is a different tool from Spark.
- Both appear in the same architecture diagrams. Company architecture write-ups on this site often show Kafka feeding Spark, for example Airbnb’s Kafka architecture, which describes Kafka as an event bus feeding an internal Spark Streaming framework.
Choosing a processing engine for Kafka data
If the choice is a processing engine, see Spark Structured Streaming vs Flink. If the choice is already made, the question becomes how to watch the pipeline, which is covered by the Spark monitoring tools ranking and the Kafka consumer lag tools comparison.
This page is published by Factor House, which makes Kpow for Kafka and Flex for Apache Flink. Factor House has no Spark product.
FAQ
Is Kafka a replacement for Spark?
No. Kafka stores and moves streams of events and Spark processes data. A common design has Spark Structured Streaming reading a Kafka topic and writing to a table or another topic.
Can Spark read from Kafka without extra software?
Spark reads Kafka through its Kafka source, which is a separate dependency from the core Spark package. The integration guide explains how to deploy it with your application.
Does Spark commit offsets to Kafka?
No. The Spark documentation says the Kafka source does not commit any offset, so the read position is in the query checkpoint.
Does Factor House offer Spark tools?
No. Factor House’s products manage Kafka and Flink. Spark appears only in its open source Factor House Local labs.