Apache Flink: the complete guide

Companies running Apache Flink in production and the architectures behind their deployments, indexed as we publish them.

How Netflix runs more than 30,000 Apache Flink jobs across Keystone, its Data Mesh SQL Processor, and a real-time distributed graph.

How Uber runs more than 2,000 Apache Flink SQL jobs processing over 4 trillion messages a day, from AthenaX through to IngestionNext, its Flink-to-Hudi streaming data lake.

How Lyft runs Apache Flink across pricing and matching features, fraud detection, and one of its largest streaming jobs, moving 80 gigabytes of events a minute from Kinesis to S3.

How Booking.com's Security Platform Services team runs Apache Flink as the engine behind an internal security-as-a-service platform, scaling to more than 250 jobs under Ververica Platform.

How Pinterest runs 130+ Apache Flink jobs across eight multitenant YARN clusters for image-similarity detection, experiment analytics, ad-budget enforcement, and CDC ingestion into Iceberg.

How Airbus's AirSense unit processes more than 2 billion aircraft-position events a day with Apache Flink, fusing multi-provider ADS-B feeds into real-time global flight tracking.

How Alibaba built Blink, its internal Apache Flink fork, to handle 4 billion records a second during Double 11, then merged its isolation and checkpointing innovations into Apache Flink 1.9 and 1.10.

Which checkpoint stage is slow, the Flink metric that proves it, and the one configuration change to try first.

How to find the root exception, tell a transient failure from a permanent one, and set a restart strategy that stops the loop.

How to find the operator and state that hold the growth, tell a leak from legitimate growth, and bring it down with TTL, incremental checkpoints, RocksDB and key design.

How to find the busy operator behind the backpressured ones, tell a slow sink, skewed keys, a slow external call and too little parallelism apart, and fix the right one.

The safe stop, verify, deploy and restore procedure, and how to diagnose a restore that fails on operator uids, state schema, max parallelism or claim mode.

How to decide whether more parallelism will help, the limits set by max parallelism, Kafka partitions and task slots, and how to rescale with a savepoint or the Kubernetes operator autoscaler.

How to read currentInputWatermark, currentOutputWatermark and numLateRecordsDropped to tell an idle Kafka partition, wrong timestamps, the wrong out-of-orderness bound and unread late data apart, and fix the right one.

What JobManager high availability protects, how to confirm it is configured and working, and how to fix a failover that loops or a cluster that came back without its jobs.

Why the Flink Kafka source commits offsets only on completed checkpoints, how to read pendingRecords, the per-partition currentOffset gauge and records-lag-max, and how to fix failing commits, stale offsets and shared group ids.

How to read the changelog mode with EXPLAIN, choose the kafka or upsert-kafka connector, handle keys and tombstones, and set the delivery guarantee that decides what a restart writes.

Six Flink management tools scored on tenant boundaries, who may act on which job, one view across clusters, the data path and self-service for application teams.

Four Flink UIs scored on submitting, stopping and restoring jobs, inspecting them, reading exceptions and logs, one view across clusters, sign-in and the data path.