News, guides, and engineering deep dives.
Practical guidance on Kafka, Flink, Iceberg, and real-time data.
Quick Search for Produced Messages
Kpow now links produced message output to Search by Key, so you can quickly search for produced messages in a single click.
Improved Error Reporting
Kpow 36 improves UI performance and error reporting, making issues easier to diagnose and resolve quickly.
OKTA single sign-on authentication
Kpow now supports OKTA authentication for single sign-on, simplifying login and access management for teams.
Best tools to monitor Spark Structured Streaming jobs that read Kafka
Five ways to monitor a Spark Structured Streaming job that reads Kafka, scored on rates, offsets behind latest, history, alerting and portability. Factor House has no Spark product and is not scored.
Kafka vs Spark: what each does and how they work together
Kafka stores and moves streams of events. Spark processes data at scale. Kafka and Spark are not alternatives, and most pipelines use both. What each is for, how Spark reads Kafka, and where the confusion comes from.
PySpark Kafka error: Failed to find data source: kafka
PySpark cannot read Kafka and fails with Failed to find data source: kafka. The cause is the missing Kafka connector package, and the fix is adding it with --packages at submit time.
Spark Structured Streaming Kafka offsets and lag: where Spark keeps its position
Spark Structured Streaming does not commit offsets to Kafka, so a consumer group view shows no lag for it. Where Spark keeps its position, how to measure lag from Spark, and how to restart safely.
Spark Structured Streaming from Kafka to Iceberg: how to write and keep tables healthy
How to write a Kafka topic into an Apache Iceberg table with Spark Structured Streaming, which settings matter for partitioned tables and commit rate, and what maintenance a streaming table needs.
Spark Structured Streaming vs Flink: how to choose for Kafka pipelines
Spark Structured Streaming and Apache Flink both read Kafka at scale. Where they differ on execution model, state, fault tolerance and operations, and when each fits. Factor House makes tools for Flink, not Spark.