Skip to content
Migrating to open source Kafka.
Sep 9, 1pm SGT. Register

How Uber uses Apache Flink in production

Flink
Derek Troy-West·September 5, 2026·20 min read

Last time out, I wrote up how Netflix runs Apache Flink across more than 30,000 jobs. Uber’s own story with Flink goes back further and reads differently: it walked away from Storm, Samza, and Spark Streaming in 2016 after comparing how each behaved under a heavy backlog, then spent the next several years building its own SQL-on-Flink platform before anyone outside the company had heard of Flink SQL. That history is closer to my own - I spent close to two decades building distributed systems and streaming platforms, including several years advising banks on Kafka and Cassandra deployments, before co-founding Factor House in 2019.

I read 18 sources for this one: Uber’s own engineering blog, conference talks by named Uber engineers at Flink Forward, QCon, and Kafka Summit, and one academic paper Uber’s team co-authored for SIGMOD. Where I couldn’t confirm a claim against a named source - a couple of blog pages had gone missing since they were first indexed, and one contributor’s title was only available through LinkedIn rather than a byline - I’ve left it out rather than guess. All 18 sources are listed at the end.

The headline figure is over 4 trillion messages a day across more than 2,000 Flink SQL jobs on Uber’s internal managed platform. What’s more interesting than the scale itself is how Uber got there: not by adopting a single Flink deployment pattern, but by building three generations of tooling on top of Flink - AthenaX, then uWorc, then a fully managed Flink-as-a-Service platform - each aimed at getting Flink out of the hands of a small platform team and into the hands of anyone who could write SQL.

Company overview

Uber operates a global ride-hailing and delivery marketplace connecting riders, drivers, restaurants, and merchants, generating a continuous stream of location, order, pricing, and marketplace-balancing events across the cities it serves. That combination of real-time logistics and a large advertising business built on top of it (UberEats Ads) is what makes Uber’s Flink footprint unusually broad: the same platform underpins ML feature pipelines, ad billing, geospatial anomaly detection, and data lake ingestion.

Uber’s move to Flink followed a direct framework comparison rather than a single triggering incident. In 2016, evaluating Storm, Samza, and Spark Streaming against a real production backlog, Uber’s streaming team found Storm needed several hours to recover from backpressure where Flink needed about 20 minutes, and that Spark Streaming used five to ten times the memory of an equivalent Flink job. That comparison is what took Flink from evaluation to Uber’s primary stream processing engine.

Date Milestone
2016 Uber evaluates Storm, Samza, Spark Streaming, and Flink against a live backlog and adopts Flink as its primary stream processor
2017-10 Uber open-sources AthenaX, its SQL-on-Flink platform, already running 220+ applications in production
2019-05 Michelangelo Palette establishes Flink SQL as a first-class path for near-real-time ML features
2020-12 uWorc, Uber’s no-code workflow orchestrator abstracting Flink and Airflow, is described in detail at 10,000+ deployed workflows
2021-09 Uber publishes its real-time, exactly-once ad event processing system for UberEats Ads
2024-01 Uber’s Flink Platform team describes a mature, fully managed “Flink as a Service” internal offering
2024 Uber reports 4 trillion+ messages/day across 2,000+ Flink SQL jobs at Flink Forward Berlin
2025-12 Uber publishes IngestionNext, its Flink-to-Hudi streaming data lake ingestion system

Uber’s first major Flink platform was built to get analysts and product teams off of hand-written streaming jobs. As Haohui Mai, Bill Liu, and Naveen Cherukuri described when they open-sourced it, AthenaX lets users submit a SQL query against a Kafka topic and have it compiled into a Flink job automatically, with a master service validating and packaging the SQL before deploying it to YARN. Within its first six months in production it was running 220+ applications, powering UberEATS Restaurant Manager order and earnings dashboards and UberPOOL trip-routing and traffic-condition analysis.

Michelangelo Palette - near-real-time ML feature pipelines

Uber’s Michelangelo ML platform needs training and serving features to stay consistent with each other even when one is computed in real time and the other from a batch warehouse. As Amit Nene and Eric Chen presented at QCon.ai, Palette uses Flink SQL to transform Kafka event streams into features that are written simultaneously to an online Cassandra store for serving and an offline Hive warehouse for training, with parity guarantees between the two paths.

Marketplace Intelligence - real-time geospatial features

Uber’s surge-pricing and demand-forecasting models need fresh, per-location signals at a scale most feature stores aren’t built for. As Feng Xu and Gang Zhao described, the Marketplace Intelligence team’s Flink pipeline generates 54 real-time geospatial features per H3 hexagon per minute, across roughly 5 million hexagons spanning 1,500 cities, feeding directly into forecasting and pricing models.

Real-time exactly-once ad event processing

UberEats Ads bills advertisers off event counts, which makes duplicate or lost events a direct revenue problem. As Jacob Tsafatinos, Yuriy Bondaruk, Yupeng Fu, and James Kwon wrote up, Uber built a three-job Flink pipeline (aggregation, attribution, then a union-and-load step) running active-active across two regions, processing hundreds of millions of ad impression, click, and order-attribution events a week with exactly-once guarantees.

Geospatial situation detection

Detecting a localized anomaly - a traffic incident, an unusual demand spike - at global scale means running complex event pattern matching across a constantly shifting set of geographic regions. As Teng “Niel” Hu presented at Flink Forward Virtual, Uber combines FlinkCEP with its own H3 hexagonal spatial indexing library and clustering techniques to detect these patterns without pre-defining every region a pattern could occur in.

IngestionNext - streaming ingestion into the data lake

Uber’s data lake was historically fed by scheduled Spark batch jobs, which meant hours of latency between an event happening and it being queryable. As Xinli Shang and co-authors describe, IngestionNext re-platforms that ingestion path onto Flink writing directly into Apache Hudi, cutting freshness from hours to minutes across the Delivery, Rider, Mobility, Finance, and Marketing Analytics organizations, while also reducing compute cost relative to the batch jobs it replaced.

Scale and throughput

Year Metric Source
2017 Over 1 trillion messages/day through Kafka feeding Flink; AthenaX running 220+ applications, processing up to several million messages/second on just eight YARN containers Mai, Liu, Cherukuri, Uber Engineering Blog
2020 “Several thousand” Flink jobs, growing roughly 30% year over year; trillions of messages and petabytes of data generated per day across the real-time platform Yupeng Fu, Chinmay Soman, SIGMOD ’21
2021 Marketplace feature pipeline: 120,000 events/second input across 5 million hexagons in 1,500 cities; write throughput scaled from 150 QPS to 300,000 QPS per datacenter; query P99 latency improved from 150ms to 10ms Feng Xu, Gang Zhao, Uber Engineering Blog
2021 Ad event pipeline: hundreds of millions of ad events per week, 1-minute tumbling windows, 2-minute checkpoint interval Tsafatinos, Bondaruk, Fu, Kwon, Uber Engineering Blog
2024 Over 4 trillion messages/day across 2,000+ Flink SQL jobs on Uber’s managed Flink platform Yang Yang, Sai Sharath Dandi, Flink Forward Berlin
2025 IngestionNext: petabyte-scale data, hundreds of datasets, thousands of concurrent ingestion jobs; freshness cut from hours to minutes; 25% compute cost reduction versus the batch jobs it replaced Xinli Shang et al., Uber Engineering Blog

Within the marketplace feature pipeline specifically, Uber’s own optimization work cut per-message size from 451 to 237 bytes and reduced deduplicated message rate from 120,000/second to 8,000/second before it ever reached the write path - a reminder that a lot of Flink’s scale story at Uber has been won on the producer side of each job, not just in cluster sizing.

AthenaX’s original deployment model packages and deploys compiled Flink jobs to YARN, with a master service handling SQL validation. As Yupeng Fu and Chinmay Soman describe in their SIGMOD ’21 paper, Uber’s platform has since organized around three layers - Platform, Job Management, and Infrastructure - with automated resource scaling and rule-based failure recovery sitting on top of that original YARN foundation.

FlinkSQL on Calcite

Both AthenaX and Uber’s later managed Flink SQL service compile SQL through Apache Calcite into Flink jobs, rather than requiring users to write against Flink’s DataStream API directly. This is the layer that lets Uber’s uWorc orchestrator (below) treat Flink as just another no-code building block.

Kafka in, Pinot and Hudi out

Across nearly every use case above, the same basic data flow repeats: Kafka carries the raw event stream in, Flink transforms it, and the result lands in whichever downstream system fits the consumer - Pinot for real-time OLAP serving, Hive for offline training data, Cassandra for online ML feature lookups, or, since IngestionNext, Hudi for the data lake itself.

IngestionNext’s three-layer design

IngestionNext separates its data plane (Kafka to Flink to Hudi), control plane (job lifecycle management across thousands of concurrent ingestion jobs), and a dedicated resilience layer for regional failover and fallback, so that a regional outage doesn’t require manually restarting thousands of individual jobs.

Exactly-once through checkpoint-aligned commits

The ad event pipeline ties its two-phase commit protocol to a 2-minute Flink checkpoint interval, using keyed state for deduplication. IngestionNext takes a related approach further downstream: it extends Hudi’s own metadata to embed Flink checkpoint IDs, so that a failover between regions can recover to a consistent point without Flink and Hudi disagreeing about what’s already been committed.

Special techniques and engineering innovations

AthenaX’s upstream contributions

Building AthenaX wasn’t purely an internal exercise - Uber contributed group windows and complex data type support back to Apache Flink 1.3 as part of the work, rather than maintaining those capabilities as a private fork.

As Sriharsha Chintalapani and Sandeep Karmakar describe, uWorc lets users build a pipeline through a drag-and-drop UI over JSON templates, without needing to know whether the engine underneath is Flink or Airflow. By the time it was documented in detail, uWorc had more than 10,000 workflows deployed out of 15,000+ pipelines company-wide, and cut typical workflow creation time from hours to under five minutes.

Protobuf as a first-class citizen

At Uber’s scale of 4 trillion-plus messages a day, serialization format matters. As Yang Yang and Sai Sharath Dandi presented at Flink Forward Berlin, Uber’s managed Flink platform moved from Avro-formatted Kafka topics to treating Protobuf as a first-class format, for performance and cost reasons at that message volume.

Micro-optimizations that moved the needle before the write path

Uber’s Marketplace Intelligence team found that several small, unglamorous changes compounded into large gains: compressing UUIDs from 128 bits to a single byte, switching from POJOs to Tuples, encoding hexagon IDs as longs instead of strings (a 50% processing-time reduction on its own), reusing objects to avoid unnecessary cloning, and replacing Flink’s native sliding-window operator with a custom FlatMap-based implementation to cut serialization and garbage-collection overhead.

Row-group-level Parquet merging

IngestionNext’s small-file problem - a familiar one to anyone streaming into a data lake - wasn’t solved by tuning file sizes alone. Uber merges at the Parquet row-group level rather than record by record, a roughly 10x improvement over record-level merging, combined with round-robin polling and per-partition quotas at the connector level to handle partition skew caused by garbage-collection pauses upstream in Kafka.

Data lineage through OpenLineage

As Zhenqiu Huang presented at Flink Forward Berlin, Uber integrated Flink’s own lineage graph and listener mechanism (built on FLIP-314) with OpenLineage, giving platform teams a standard way to trace a dataset back through the Flink jobs that produced it.

As Na Yang and Yusheng Chen described at Uber’s internal Flink meetup, Uber’s Flink Platform team now runs Flink as a fully managed service: developers and data scientists submit Flink SQL jobs without provisioning or operating any Flink infrastructure themselves, a meaningfully different operating model from AthenaX’s original self-service-but-still-YARN-aware design.

Multi-tenancy and noisy-neighbor isolation

Running thousands of jobs from many different teams on shared infrastructure creates a resource-isolation problem that doesn’t show up at smaller scale. Na Yang, who leads Uber’s Streaming Data and Flink team, has spoken about resource isolation between tenants as an ongoing area of platform work rather than a solved problem - a detail I appreciated, since it’s rare for a platform team to describe multi-tenancy as still evolving rather than already finished.

Coordinated, phased version upgrades

As Anand Kotriwal, Yupeng Fu, Aparajita Pandey, and Charu Jain described, upgrading Flink jobs from Java 8 to Java 11 was done as part of a broader, multi-system version upgrade (moving through Java, then Scala 2.12, then Spark 3.0, before touching Flink), specifically to de-risk the rollout by isolating which system change was responsible for any regression.

Regional failover as a first-class design goal

Both the ad event pipeline (active-active across two regions with a union-and-load reconciliation step) and IngestionNext (a dedicated resilience layer for regional failover and fallback) treat losing a region as something the pipeline design accounts for directly, rather than an operational procedure bolted on afterward.

Challenges and how they solved them

Storm and Spark Streaming couldn’t recover fast enough under load

Before settling on Flink, Uber ran a direct comparison of the streaming frameworks available to it at the time.

Problem: Under a large input backlog, Storm needed several hours to recover from backpressure, and Spark Streaming used five to ten times the memory of an equivalent Flink job.

Root cause: Storm’s backpressure recovery model and Spark Streaming’s micro-batch memory footprint weren’t built for the kind of sustained, high-volume backlog Uber’s production traffic could generate.

Solution: Uber adopted Flink as its primary stream processing engine after the comparison.

Outcome: A 20-minute backpressure recovery time versus Storm’s several hours, presented by Danny Yuan, Uber’s Real-time Streaming Lead, at QCon San Francisco in 2016.

Ad billing risked double-counting the same event

UberEats Ads bills off event counts directly, so any duplicate processing has a direct financial consequence.

Problem: A single ad impression, click, or order-attribution event could be counted more than once if a job restarted or retried mid-stream.

Root cause: Standard at-least-once delivery semantics don’t guarantee an event is billed exactly once across job restarts and retries.

Solution: Uber generates a UUID idempotency key per event, combines Flink’s two-phase commit protocol with Kafka’s read_committed transactional reads, and relies on Pinot’s native upsert support for a final layer of downstream deduplication.

Outcome: Exactly-once ad billing across a three-job, two-region pipeline processing hundreds of millions of events a week, published by Jacob Tsafatinos, Yuriy Bondaruk, Yupeng Fu, and James Kwon in 2021.

The marketplace feature pipeline hit network and memory limits at 120,000 events/second

Generating 54 features per hexagon per minute across 5 million hexagons meant the pipeline’s own message overhead became the bottleneck before cluster sizing did.

Problem: Message size and per-event object overhead were limiting throughput well below what the underlying compute could otherwise support.

Root cause: Verbose encodings (string-based UUIDs and hexagon IDs, POJO-based records, native sliding windows) were adding serialization, network, and garbage-collection cost on every message.

Solution: Uber compressed UUIDs, switched hexagon IDs from strings to longs, moved from POJOs to Tuples, reused objects instead of cloning them, and replaced native sliding windows with a custom FlatMap implementation.

Outcome: Deduplicated message rate fell from 120,000/second to 8,000/second, and write throughput scaled from 150 QPS to 300,000 QPS per datacenter, published by Feng Xu and Gang Zhao in 2021.

Streaming into the data lake created a small-file problem

Moving from scheduled batch writes to continuous streaming writes changes how files accumulate in a data lake.

Problem: Traditional record-by-record Parquet merging was too slow to keep up with continuous streaming writes at Uber’s scale, and Kafka partition skew caused by upstream garbage-collection pauses made the problem worse.

Root cause: Streaming ingestion produces many small files far more frequently than batch ingestion does, and record-level merge strategies weren’t designed for that write pattern.

Solution: Uber merges at the Parquet row-group level instead of record by record, and added connector-level fairness (round-robin polling and per-partition quotas) to handle partition skew.

Outcome: Roughly a 10x improvement in ingestion performance over record-level merging, published by Xinli Shang and co-authors in December 2025.

A regional failover is exactly the moment when a Flink job and its downstream Hudi table are most likely to disagree about the last consistent state.

Problem: Without a shared reference point, a failover between regions risked duplicating or losing data if Flink and Hudi recovered to different points.

Root cause: Flink’s own checkpoint IDs weren’t visible to Hudi’s commit metadata, so the two systems had no common way to agree on which writes were durable.

Solution: Uber extended Hudi’s metadata to embed Flink’s checkpoint IDs directly, giving both systems a shared, deterministic recovery point.

Outcome: Deterministic recovery across regional failovers without duplicate or lost writes, as part of IngestionNext’s resilience layer.

Full tech stack

Category Tools Notes
Stream processing Apache Flink Core engine for AthenaX, uWorc, Michelangelo Palette, marketplace features, ad event processing, and IngestionNext
Messaging Apache Kafka Event backbone feeding nearly every Flink job described above
SQL layer Apache Calcite Compiles SQL into Flink jobs inside AthenaX and Uber’s managed Flink SQL service
Real-time serving Apache Pinot OLAP serving layer downstream of Flink; native upsert support used for ad-event deduplication
Data warehouse Apache Hive Offline training-data and backfill sink for ML feature pipelines
Data lake table format Apache Hudi Streaming data-lake format written directly by Flink in IngestionNext
Distributed storage HDFS Long-term archival and deep storage
Query engine Presto Interactive SQL over Pinot and HDFS
Resource orchestration YARN Original deployment target for AthenaX-era Flink jobs
Key-value store Docstore Uber’s in-house store used for order and event lookups in the ad pipeline
ML feature store Cassandra Online feature serving for Michelangelo Palette
Geospatial indexing H3 Uber’s open-source hexagonal indexing library, used in marketplace features and FlinkCEP situation detection
Workflow orchestration Airflow Batch workflow engine, abstracted alongside Flink inside uWorc
Runtime Java 11, Scala 2.12 Versions Flink jobs were upgraded to from Java 8 in the 2024 coordinated upgrade

Key contributors

Name Role Contribution
Danny Yuan Uber, Real-time Streaming Lead Presented the 2016 framework comparison that led Uber to adopt Flink
Haohui Mai Uber, Software Engineer, Streaming Analytics Co-authored the AthenaX architecture post
Bill Liu Uber, Software Engineer, Streaming Analytics Co-authored the AthenaX architecture post
Naveen Cherukuri Uber, Engineering Manager, Streaming Analytics Co-authored the AthenaX architecture post
Amit Nene Uber, Michelangelo ML Platform Co-presented Flink’s role in Michelangelo Palette at QCon.ai 2019
Eric Chen Uber, Michelangelo ML Platform Co-presented Flink’s role in Michelangelo Palette at QCon.ai 2019
Feng Xu Uber, Sr. Software Engineer II, Marketplace Intelligence Co-authored the marketplace feature-pipeline scaling post
Gang Zhao Uber, former Sr. Software Engineer II, Marketplace Intelligence Co-authored the marketplace feature-pipeline scaling post
Jacob Tsafatinos Uber, Software Engineer Co-authored the exactly-once ad event processing post
Yuriy Bondaruk Uber, Sr. Software Engineer II Co-authored the exactly-once ad event processing post
Yupeng Fu Uber, Principal Software Engineer Co-authored the ad event post, the SIGMOD ’21 paper, and the Java upgrade post
James Kwon Uber, Software Engineer II Co-authored the exactly-once ad event processing post
Chinmay Soman Uber, Inc. Co-authored the SIGMOD ’21 real-time infrastructure paper
Sriharsha Chintalapani Uber, Sr. Staff Software Engineer, Data Platforms Co-authored the uWorc architecture post
Sandeep Karmakar Uber, former lead Product Manager, data/ML platform Co-authored the uWorc architecture post
Teng “Niel” Hu Uber, Software Engineer Presented FlinkCEP-based geospatial situation detection at Flink Forward Virtual 2020
Na Yang Uber, Flink Platform Engineering Manager Co-presented “Flink as a Service @ Uber” and spoke separately on multi-tenancy
Yusheng Chen Uber, Staff Engineer Co-presented “Flink as a Service @ Uber”
Yang Yang Uber, Sr. Staff Engineer Presented the Protobuf-on-managed-Flink work at Flink Forward Berlin 2024
Sai Sharath Dandi Uber, Sr. Software Engineer Co-presented the Protobuf-on-managed-Flink work at Flink Forward Berlin 2024
Zhenqiu Huang Uber, Software Engineer Presented the OpenLineage/Flink data lineage integration at Flink Forward Berlin 2024
Anand Kotriwal Uber, Sr. Software Engineer Co-authored the Java 8 to 11 upgrade post
Aparajita Pandey Uber, Software Engineer Co-authored the Java 8 to 11 upgrade post
Charu Jain Uber, Engineering Manager Co-authored the Java 8 to 11 upgrade post
Xinli Shang Uber Co-authored the IngestionNext architecture post
Jing Li Uber Co-authored the IngestionNext architecture post
Peter Huang Uber Co-authored the IngestionNext architecture post
Jack Song Uber Co-authored the IngestionNext architecture post
Jing Zhao Uber Co-authored the IngestionNext architecture post

These are the patterns from Uber’s own account that I’d point any team running Flink across many teams, not just one, toward:

  • Plan for the platform layer to outlive any single Flink version. Uber has rebuilt the layer around Flink twice - AthenaX, then uWorc, then a fully managed Flink-as-a-Service model - while the underlying engine kept evolving underneath. Treat the SQL/orchestration layer your users touch as a separate long-lived investment from whichever Flink version happens to sit underneath it.
  • Push cost optimization upstream of the cluster, not just into it. Uber’s biggest wins in its marketplace pipeline - UUID compression, encoding changes, custom windowing - all happened in how messages were represented before they ever reached the write path, not in adding more compute.
  • Tie exactly-once guarantees to what actually gets billed or counted. Uber’s ad-event and IngestionNext designs both treat checkpoint alignment as a correctness mechanism, not just a recovery mechanism - useful framing if your own Flink jobs feed anything with a dollar figure attached.
  • Solve multi-tenancy as an ongoing problem, not a one-time fix. Uber’s own Flink Platform lead describes resource isolation between tenants as continuing platform work even after thousands of jobs are already running - a healthier framing than treating it as solved once and never revisited.
  • Sequence multi-system upgrades deliberately. Uber’s Java 8 to 11 migration moved through Java, then Scala, then Spark, before touching Flink, specifically so a regression could be traced to one system change at a time rather than a tangle of simultaneous ones.

I’ll keep working through the rest of the list and write up the next one as it’s ready.

Sources and further reading

Primary sources

I read every one of these directly rather than working from a summary, and linked the specific claim to its source throughout the piece above, not just in this list.

If you’re running Flink across more teams than one platform group can babysit by hand - which is exactly the situation Uber’s three generations of tooling were built to solve - that’s the same operational gap my team and I built Flex to close: an enterprise UI and API for Apache Flink covering job visibility, checkpoint health, and multi-tenant operations, so you don’t have to build all of the platform layer above in-house before anyone outside your team can safely run a Flink job.