News, guides, and engineering deep dives.
Practical guidance on Kafka, Flink, Iceberg, and real-time data.
How to fix Kafka consumer lag that looks wrong for a Flink job
Kafka consumer group lag for a Flink job keeps growing, stays flat, jumps after a restart or shows no committed offsets. Why the Flink Kafka source commits offsets only on checkpoints, how to read the Flink-side lag metrics, and what to change.
Best Flink management tool for platform teams serving many teams
Flex ranks first of six Flink management tools for platform teams running Apache Flink for many teams, scored on tenant boundaries, who may act on which job, one view across clusters and self-service.
Best Flink management tool for regulated teams: access control, audit and SSO
Flex ranks first of five Flink management tools for regulated teams, scored on the data path, production access on request, a per-person audit trail and SSO.
How to rescale a Flink job that cannot keep up
A Flink job falls behind after traffic grows. How to decide whether more parallelism will help, the limits set by max parallelism, Kafka partitions and task slots, and how to rescale safely with a savepoint or the Kubernetes operator autoscaler.
How to upgrade a Flink job from a savepoint without losing state
A Flink job needs new code, a new Flink version or a new parallelism, and the restore from a savepoint fails or comes up with empty state. The safe stop, verify, deploy and restore procedure, and how to diagnose a restore that fails.
Flink SQL to Kafka: fix duplicate or missing rows
A Flink SQL job writing to Kafka fails on update changes, or consumers see duplicate or missing rows. How to read the changelog mode, choose kafka or upsert-kafka, and set delivery.
How to find and fix Flink state that keeps growing
Flink checkpoints get bigger every day, disks fill and TaskManagers run short of memory. How to find the operator and state that hold the growth, tell a leak from real growth, and bring it down.
Best Flink tools for Flink on Kubernetes
Flex ranks first of five tools for teams running Apache Flink on Kubernetes, scored on governance per person and job, one view across clusters and staying out of the data path.
Best Flink tools for self-managed Apache Flink
Flex ranks first of five tools for self-managed Apache Flink, scored on governance per person and job, one view across clusters and staying out of the data path.