I spent close to two decades building distributed systems and streaming platforms before I co-founded Factor House in 2019, including several years advising banks on Kafka and Cassandra deployments. One thing that’s stayed consistent across all of it is how different that infrastructure looks on a slide compared to how a team actually runs it once it’s carrying real production traffic. So a few weeks ago I started going back through how the industry’s most demanding teams run Apache Flink themselves - reading their own engineering blogs and conference talks rather than relying on secondhand write-ups. Netflix is the first one worth putting in writing: more than 30,000 Flink jobs running across multiple AWS regions, ranging from stateless routing to multi-terabyte stateful jobs for ads, recommendations, and games.
That fleet did not arrive at once. It grew out of a single stream processing platform, Keystone, that Netflix migrated from Apache Samza to Flink in the mid-2010s, and it has kept evolving as the company’s business has grown well beyond video streaming into live events, games, and advertising. Everything below comes from Netflix’s own engineering blog, its team’s conference talks, and one piece of trade reporting that named its source directly - I’ve listed all ten sources at the end.
The engineering problem underneath all of this is a familiar one from my own time running these systems, just at a scale most teams never encounter: how do you give tens of thousands of independent streaming jobs, each with its own resource profile and failure characteristics, the compute they need without a human tuning each one by hand. Netflix’s answer touches nearly every layer of its Flink usage, from how jobs are autoscaled to how state is checkpointed to S3 to how late-arriving events land in the right date partition of a table.
Company overview
Netflix operates a global streaming entertainment service used by subscribers in more than 190 countries, and in recent years has expanded into ad-supported plans, live events, and mobile games. That expansion multiplied the number of independent microservices producing data about member activity, and made it harder to recognise when the same member’s actions across different services and devices were related.
Netflix’s real-time data infrastructure predates this expansion. Netflix’s Mantis platform, used for operational stream processing, went into production in 2014. The company’s analytics-focused stream processing, originally built on Apache Samza, was rebuilt on Apache Flink and rebranded Keystone in the mid-2010s, becoming the foundation Netflix has extended ever since.
| Date | Milestone |
|---|---|
| 2014 | Netflix Mantis, the operational stream processing platform, goes into production |
| 2017-04 | Monal Daxini keynotes Flink Forward San Francisco on Keystone’s move to Apache Flink |
| 2018-04 | Steve Wu presents the completed Samza-to-Flink migration at Flink Forward, citing 3 trillion events and 12 PB moved per day |
| 2018-09 | Netflix publishes the Keystone stream processing platform architecture on the Netflix TechBlog |
| 2019-10 | Netflix presents event-time partitioning into Iceberg at Flink Forward Europe |
| 2023-11 | Netflix introduces the Flink-SQL-based Data Mesh Processor |
| 2024 | Netflix reports 15,000+ Flink jobs and 60+ PB processed daily at Confluent’s Current conference |
| 2025-10 | Netflix publishes its real-time distributed graph architecture, built on Flink |
| 2026-08 | Netflix reports 30,000+ Flink jobs across multiple AWS regions and its move toward a community-built autoscaler |
Netflix’s Apache Flink use cases
Keystone - stream processing as a service
Keystone is Netflix’s self-serve platform for building managed stream processing applications, first described in detail by Zhenzhong Xu on the Netflix TechBlog. It combines a Kafka-backed routing and messaging service for near-real-time microservice events with a Stream Processing as a Service (SPaaS) layer on top of Flink, letting internal teams configure a job through a web UI rather than operating their own cluster. Keystone’s first production use case was Netflix’s video-view measurement pipeline, and it has since grown into the shared platform underneath most of Netflix’s other Flink use cases.
Real-time distributed graph - connecting member activity across services
As Netflix expanded into ad-supported plans, live events, and games, member activity became spread across many independent microservices that were hard to connect. As Adrian Taruc and James Dalton explain, Flink jobs consume Kafka event records from Netflix’s API Gateway, filter and project the fields they need, enrich records using side inputs, convert events into graph nodes and edges, deduplicate overlapping updates within time windows, and publish the results into Netflix’s Data Mesh platform for storage and serving.
Data Mesh SQL Processor - transformations without hand-built job graphs
Teams building on Netflix’s Data Mesh platform originally had a fixed set of low-level Processors for projection, filtering, unioning, and field renaming, and had to drop into Flink’s DataStream API for anything more complex - a steep learning curve, and chaining several Processors together added runtime overhead from the extra Flink jobs and intermediate Kafka topics involved. As Guil Pires, Mark Cho, Mingliang Liu, and Sujay Jain describe, the Data Mesh SQL Processor is a platform-managed, parameterized Flink Job built on Flink’s Table API that converts schematized sources into tables automatically, so teams write Flink SQL directly instead. It now underpins thousands of use cases on the platform.
Take fraction - a real-time replacement for a batch join
Netflix’s personalization models depend on “take fraction,” the ratio of plays to impressions. Computing it had required a batch pipeline with a delay of hours to days. As Sonali Sharma and Shriya Arora presented at QCon San Francisco, Netflix’s Data Personalization team rebuilt it as a stateful Flink streaming application that joins plays against impressions in real time, so models can train and make decisions on fresher data.
Event-time partitioning - keeping playback data queryable by the day it happened
Netflix’s playback data has a long tail of events that arrive late. To let analysts query all of a given day’s playback data together regardless of when each event was actually processed, Julia Bennett described at Flink Forward Europe how Netflix built dynamic, event-time-based partitioning in Flink that writes into an Apache Iceberg table, handling deduplication and late or out-of-order events at the partition level.
Scale and throughput
| Year | Metric | Source |
|---|---|---|
| 2018 | 3 trillion events and 12 PB moved per day through Keystone; 2,000 routing jobs running 200,000 parallel operators; 125 million hours streamed daily | Steve Wu, Flink Forward |
| 2019 | Roughly 1,000 streams and data pipelines monitored | Allen Wang, QCon San Francisco |
| 2024 | 15,000+ Flink jobs; 60+ PB processed daily; cluster sizes up to 2,000-way parallelism; per-job state from stateless up to 4 TB | Mark Cho, Mingliang Liu, Current |
| 2025 | Kafka topics feeding the real-time graph pipeline run up to roughly 1 million messages/second per topic; graph output exceeds 5 million node-and-edge records per second | Adrian Taruc, James Dalton, Netflix TechBlog |
| 2026 | 30,000+ Flink jobs across multiple AWS regions; one team cut annual compute cost 58% (about $1.1M) after adopting the community autoscaler | Netflix TechBlog |
Within individual jobs, state size has itself been an optimization target: Netflix’s take-fraction join reduced retained state from roughly 5 TB to roughly 2 TB through on-the-fly summarization, then cut it by a further 25% by consolidating repeated events into composite types.
Netflix’s Apache Flink architecture
Keystone’s platform layer
Keystone runs its Flink jobs on Netflix Titus, its container runtime, deployed through Spinnaker. State lives across AWS RDS, S3, and ZooKeeper, with sinks into Kafka, Elasticsearch, and Hive. The platform follows a declarative reconciliation model: a single source of truth for state, a control-plane/data-plane split, idempotent operations that let jobs self-heal, and per-job isolation to support many tenants on shared infrastructure.
A custom control plane instead of the Kubernetes Operator for Flink
As Mark Cho and Mingliang Liu presented at Current, Netflix runs its own custom control plane rather than the open-source Kubernetes Operator for Flink, a choice driven by the scale and diversity of onboarding needed to support tens of thousands of jobs.
Streaming data infrastructure topology
Event producers write into “Fronting” Kafka clusters. Flink jobs perform ETL, projection, and filtering on that data, and results land in downstream sinks that include S3, Elasticsearch, and separate “Consumer” Kafka clusters for other services to read from.
The Data Mesh SQL Processor’s query layer
The SQL Processor sits on Flink’s Table API, which automatically converts schematized upstream sources into tables so a user’s SQL query becomes a Flink job without manual job-graph construction. It includes an interactive query mode for sampling live data and validates queries in real time by inspecting the Calcite query tree the SQL compiles to.
The real-time distributed graph’s three layers
The graph pipeline separates into ingestion/processing, storage, and serving layers. Events flow from the Netflix API Gateway into Kafka, then into Flink jobs that transform them into graph nodes and edges, then into Data Mesh for persistence and downstream serving.
Special techniques and engineering innovations
Two generations of autoscaler, running side by side
Netflix’s original in-house autoscaler, built around 2019, ran as a Mantis streaming job consuming Atlas container-level metrics, and scaled every operator in a job together as a single cluster-level decision - achieving 25-45% resource savings on the jobs it fit, but unable to handle multi-operator stateful DAGs or tune individual operators. As Samuel Yeboah, Francesco Di Chiara, and Mingliang Liu wrote up in 2026, Netflix has since adopted the Apache Flink community’s autoscaler, which estimates each operator’s “true processing rate” by dividing its observed throughput by its busy fraction, then uses that to compute per-vertex parallelism from the job’s DAG. Making the community autoscaler work at Netflix’s scale required JobManager metric caching and server-side filtering, forward-connection subgraph detection to preserve in-memory handovers between operators, async-sink backpressure detection, and running at a 0.45 target utilization rather than the community default of 0.7. Netflix now runs both autoscalers in production and is steadily converging its fleet onto the open-source one.
This is the part of the post I found most useful, and it’s rarer than it should be: a team writing publicly that the tool they built in-house hit a real ceiling, and choosing to move toward the community alternative instead of just patching around it. Autoscaling and per-job resource visibility at this kind of scale is exactly the operational gap we spend our time on at Factor House, so it’s a good, concrete illustration of a pattern I see in most large Flink estates once they get past a handful of jobs.
State-size reduction for stateful joins
For the take-fraction join, Netflix used Flink’s CoProcessFunction with explicit state management instead of windows, applied on-the-fly summarization to shrink retained state, and consolidated repeated events into composite types for a further reduction. Checkpoint-based recovery combined with a bounded 4-hour event-expiry window let the job reach 99th-percentile data completeness.
Event-time partitioning into Iceberg
Netflix’s event-time partitioning implementation handles exactly-once semantics through deduplication of incoming streams and append-only writes, together with partition-level handling for late and out-of-order events, so a Flink job can keep writing correctly into a table even as events for a given day continue arriving well after that day has passed.
S3 checkpoint tuning at Keystone’s scale
Sequential checkpoint filenames caused S3 to cluster related objects into the same partition, creating write hot-spots once Keystone reached thousands of parallel operators. Netflix fixed this by inserting a randomized 4-character hex string into checkpoint filenames to spread writes across S3 partitions. Separately, a low threshold for inlining operator state meant many small checkpoint files were written per job; raising that threshold from 1 KB to 1 MB let the JobManager consolidate acknowledgments into a single “uber” checkpoint file. Netflix also identified that Flink’s Checkpoint Stream Factory was re-resolving its S3 path roughly every 30 seconds, generating around 13,000 redundant metadata requests; as reported at the time, Netflix contributed a fix so the factory initializes once, which landed in Flink 1.2.1.
Operating Apache Flink at scale
External, out-of-band monitoring
As Allen Wang described at QCon San Francisco, Netflix’s Real-Time Data Infrastructure team runs a monitoring service that observes the streaming pipeline from outside rather than relying only on internal metrics: verifying heartbeat messages end to end, cross-checking ZooKeeper metadata against live broker metadata, tracking consumer offsets and lag, and flagging stuck consumers through staleness detection.
Distributed tracing through a dedicated Kafka cluster
A separate Kafka cluster, referred to internally as Inca, captures trace messages at the point of production, processing, and consumption. Stream processing jobs then correlate those traces by message ID to compute message-loss rate, duplicate rate, and end-to-end latency across the pipeline.
Automated remediation
A “First Responder” system automatically relaunches stateless jobs when a stuck-consumer alert fires, without waiting for a human to intervene. A separate regression-based autoscaler using quadratic and linear regression on traffic patterns adjusts job capacity automatically ahead of demand changes.
Per-job isolation through workflow orchestration
Netflix’s newer autoscaler runs as one Temporal workflow per Flink job, rather than a shared batch process across all jobs, specifically so that one job’s scaling failure cannot affect any other job - a direct response to an earlier single-batch model that had created bottlenecks affecting the whole fleet at once.
Conservative utilization targets
Netflix runs the community autoscaler at a 0.45 target utilization, well below the community’s own 0.7 default, trading some resource efficiency for additional stability headroom across a fleet this large. It’s a small number to bury in a blog post, but it tells you something honest about how they think about risk at this scale: they gave up headroom on purpose rather than chase the community default.
Challenges and how they solved them
Homegrown autoscaler couldn’t handle stateful, multi-operator jobs
As Netflix’s Flink usage grew beyond simple routing jobs into complex, stateful pipelines for ads, recommendations, and games, its original autoscaler stopped being sufficient.
Problem: Netflix’s in-house autoscaler could only reason about whole clusters through container-level metrics, so it scaled every operator in a job together.
Root cause: It had been built for Netflix’s earlier generation of largely stateless or single-operator Flink jobs, and was never designed for complex, stateful, multi-operator DAGs.
Solution: Netflix adopted the Apache Flink community’s autoscaler, which computes a per-operator true processing rate and scales each vertex in a job’s graph independently.
Outcome: Netflix now runs both autoscalers in production while converging onto the open-source one; one team alone reported a 58% (about $1.1M annual) reduction in compute cost, published by Samuel Yeboah, Francesco Di Chiara, and Mingliang Liu on the Netflix TechBlog in 2026.
External metrics silently missed real scaling problems
Even after adopting a more capable autoscaler, the metrics feeding it turned out to have their own blind spots.
Problem: Container-level CPU metrics could look healthy even when a job was fully saturated, so the platform failed to catch jobs that needed more resources.
Root cause: A job could be completely busy without any of that load showing up as CPU utilization, and separately, an internal networking migration silently degraded the accuracy of the metrics the autoscaler depended on - a problem that only became visible in production much later.
Solution: Netflix moved to job-level, operator-aware metrics computed from Flink’s own throughput and busy-time signals rather than infrastructure-level CPU counters, with JobManager-side metric caching and filtering to keep those signals reliable at scale.
Outcome: More accurate, operator-level autoscaling decisions, and eventual identification of the networking-migration regression that had been degrading metric accuracy.
S3 checkpoint writes hot-spotted and wasted metadata calls
At Keystone’s early scale of 2,000 routing jobs and 200,000 parallel operators, the checkpointing path itself became a bottleneck.
Problem: Checkpoint writes to S3 slowed down, and the platform was issuing around 13,000 redundant metadata requests every 30 seconds.
Root cause: Sequential checkpoint filenames caused S3 to place related objects in the same partition, a low state-size threshold produced many small separate checkpoint files instead of consolidated ones, and Flink’s Checkpoint Stream Factory was re-resolving its S3 path on a fixed interval instead of once.
Solution: Netflix randomized checkpoint filenames to spread S3 writes, raised the state-size inlining threshold from 1 KB to 1 MB so the JobManager could write single “uber” checkpoint files, and contributed a fix so the Checkpoint Stream Factory initializes once.
Outcome: Reduced S3 hot-spotting and eliminated the redundant metadata call volume, presented by Steve Wu at Flink Forward in 2018.
Real-time joins needed batch-level correctness without batch latency
Replacing a trusted batch metric with a streaming equivalent meant matching its accuracy while cutting its latency from hours to seconds.
Problem: Computing take fraction accurately in real time required handling late-arriving events, deduplicating after checkpoint restarts, and validating results without a batch job to compare against.
Root cause: The prior batch pipeline could tolerate hours of delay to let data settle before computing the ratio; a streaming replacement had to reach comparable completeness while events were still arriving out of order.
Solution: Netflix built a stateful Flink application using CoProcessFunction, checkpoint-based recovery, and a bounded 4-hour event-expiry window, monitoring watermark progression to track completeness over time.
Outcome: 99th-percentile data completeness at streaming latency, presented by Sonali Sharma and Shriya Arora at QCon San Francisco in 2019.
Full tech stack
| Category | Tools | Notes |
|---|---|---|
| Stream processing | Apache Flink | Core engine for Keystone SPaaS, the Data Mesh SQL Processor, the real-time distributed graph, take-fraction joins, and event-time partitioned Iceberg writes |
| Messaging | Apache Kafka | Event backbone: “Fronting” ingest clusters, Keystone’s Messaging Service, and source topics feeding Flink jobs |
| Prior stream processor | Apache Samza | Keystone’s original engine, migrated to Flink |
| Table format | Apache Iceberg | Target for event-time-partitioned playback data written from Flink |
| Container runtime | Netflix Titus | Runs Keystone’s Flink jobs |
| Deployment | Spinnaker | Continuous deployment engine for Keystone’s Flink jobs |
| Operational stream processing | Netflix Mantis | Netflix’s operational stream platform; also the original execution substrate for Netflix’s in-house Flink autoscaler |
| Metrics | Atlas | Netflix’s metrics system, consumed by the original in-house autoscaler |
| Workflow orchestration | Temporal | Orchestrates one autoscaler workflow per Flink job |
| Application framework | Spring Boot | Framework for the newer autoscaler service |
| Object storage | AWS S3 | Checkpoint/snapshot store and data warehouse storage layer |
| Relational storage | AWS RDS | State storage for Keystone’s control plane |
| Coordination | ZooKeeper | Metadata/coordination store for Keystone and stream monitoring validation |
| Search | Elasticsearch | Downstream sink for processed stream data |
| Data warehouse | Apache Hive | Downstream sink/warehouse target for processed stream data |
Key contributors
| Name | Role | Contribution |
|---|---|---|
| Zhenzhong Xu | Netflix, Real-time Data Infrastructure / Mantis | Authored the Keystone platform architecture post; helped build Keystone, the managed Flink platform, Mantis, and the managed Kafka platform |
| Monal Daxini | Netflix, Stream Processing engineering leadership | Keynoted Flink Forward San Francisco 2017 on Keystone’s move to Flink |
| Steve Wu | Netflix, Senior Software Engineer | Presented Flink-on-AWS optimizations, including the S3 checkpointing and metadata request fixes, at Flink Forward |
| Allen Wang | Netflix, Real-Time Data Infrastructure | Architected Netflix’s multi-cluster Kafka infrastructure; presented monitoring and tracing architecture at QCon San Francisco 2019 |
| Sonali Sharma | Netflix, Data Engineer, Data Personalization | Co-presented the take-fraction stateful Flink join at QCon San Francisco 2019 |
| Shriya Arora | Netflix, Senior Software Engineer | Co-presented the take-fraction stateful Flink join at QCon San Francisco 2019 |
| Julia Bennett | Netflix | Presented streaming event-time partitioning with Flink and Iceberg at Flink Forward Europe 2019 |
| Guil Pires | Netflix, Data Platform | Co-authored the Streaming SQL in Data Mesh blog post |
| Mark Cho | Netflix, Data Platform / Flink Platform | Co-authored Streaming SQL in Data Mesh; co-presented the 15,000-job Flink platform talk at Current 2024 |
| Mingliang Liu | Netflix, Data Platform / Flink Platform | Co-authored Streaming SQL in Data Mesh and the two-autoscalers post; co-presented at Current 2024 |
| Sujay Jain | Netflix, Data Platform | Co-authored Streaming SQL in Data Mesh |
| Adrian Taruc | Netflix | Co-authored the real-time distributed graph architecture post |
| James Dalton | Netflix | Co-authored the real-time distributed graph architecture post |
| Samuel Yeboah | Netflix, Flink Platform | Co-authored “A Tale of Two Flink Autoscalers” |
| Francesco Di Chiara | Netflix, Flink Platform | Co-authored “A Tale of Two Flink Autoscalers” |
Key takeaways for your own Apache Flink implementation
These are the patterns from Netflix’s own account that I’d point any team running (or planning to run) Flink at meaningful scale toward:
- Plan for two generations of autoscaler, not one. Netflix’s own container-level autoscaler worked well until jobs became stateful and multi-operator; if you expect your Flink usage to grow past simple stateless jobs, budget for a later migration to operator-aware, DAG-level autoscaling rather than assuming your first approach will scale indefinitely.
- Treat your scaling metrics as something to validate, not trust. Netflix found that CPU-based metrics could look healthy while a job was fully saturated, and that an unrelated infrastructure change had quietly degraded metric accuracy for a long time before anyone noticed. Build metrics that reflect what Flink itself is doing (throughput, busy fraction) rather than relying solely on infrastructure-level signals.
- Isolate scaling decisions per job. Netflix moved from a shared batch scaling process to one workflow per job specifically to stop one job’s failure from affecting others - a pattern worth adopting before a shared control loop becomes a single point of failure across your fleet.
- Checkpoint storage details matter at scale. Sequential filenames, small-file thresholds, and how often a checkpoint factory re-resolves its storage path all became real bottlenecks for Netflix once it reached thousands of parallel operators; these are cheap to get right early and expensive to discover under load.
- Match state management to the correctness bar you actually need. Netflix’s take-fraction join used explicit
CoProcessFunctionstate and a bounded event-expiry window rather than default windowing, because a fixed correctness target (99th-percentile completeness) mattered more than the simplicity of an out-of-the-box windowing approach.
I’ll keep working through the rest of the list and write up the next one as it’s ready.
Sources and further reading
Primary sources
- Samuel Yeboah, Francesco Di Chiara, Mingliang Liu, “A Tale of Two Flink Autoscalers” - https://netflixtechblog.com/a-tale-of-two-flink-autoscalers-e9f6a1b1492b (2026)
- Zhenzhong Xu, “Keystone Real-time Stream Processing Platform” - https://netflixtechblog.com/keystone-real-time-stream-processing-platform-a3ee651812a (2018)
- Guil Pires, Mark Cho, Mingliang Liu, Sujay Jain, “Streaming SQL in Data Mesh” - https://netflixtechblog.com/streaming-sql-in-data-mesh-0d83f5a00d08 (2023)
- Adrian Taruc, James Dalton, “How and Why Netflix Built a Real-Time Distributed Graph: Part 1” - https://netflixtechblog.com/how-and-why-netflix-built-a-real-time-distributed-graph-part-1-ingesting-and-processing-data-80113e124acc (2025)
- Mark Cho, Mingliang Liu, “Building a Scalable Flink Platform: A Tale of 15,000 Jobs at Netflix,” Current - https://current.confluent.io/2024-sessions/building-a-scalable-flink-platform-a-tale-of-15-000-jobs-at-netflix (2024)
- Allen Wang, “Monitoring and Tracing @ Netflix Streaming Data Infrastructure,” QCon San Francisco - https://www.infoq.com/presentations/netflix-streaming-data-infrastructure/ (2019)
- Sonali Sharma, Shriya Arora, stream processing with Flink at Netflix, QCon San Francisco - https://www.infoq.com/presentations/netflix-event-stream-flink (2019)
- Julia Bennett, “Streaming Event-Time Partitioning With Apache Flink and Apache Iceberg,” Flink Forward Europe - https://www.ververica.com/blog/flink-forward-preview-event-time-partitioning-with-apache-flink-apache-iceberg-netflix (2019)
- Monal Daxini, “Stream Processing with Flink at Netflix,” Flink Forward San Francisco - https://www.youtube.com/watch?v=sPB8w-YXX1s (2017)
- Alex Woodie, reporting on Steve Wu (Netflix), “How Netflix Optimized Flink for Massive Scale on AWS,” BigDATAwire/HPCwire - https://hpcwire.com/bigdatawire/2018/04/30/how-netflix-optimized-flink-for-massive-scale-on-aws/ (2018)
I read every one of these directly rather than working from a summary, and I’ve linked the specific claim to its source throughout the piece above, not just in this list.
Enterprise tooling for Apache Flink
If you are running Flink at anywhere close to this scale, or expect to, the operational picture Netflix describes here - per-job autoscaling, checkpoint tuning, and cross-job visibility - is exactly the space my team and I build for at Factor House. It’s part of why I find write-ups like Netflix’s worth this much attention: Flex, our enterprise UI and API for Apache Flink, exists to make that picture manageable without building all of the tooling above in-house.