Last time out, I wrote up how Netflix runs Apache Flink across more than 30,000 jobs. Uber’s own story with Flink goes back further and reads differently: it walked away from Storm, Samza, and Spark Streaming in 2016 after comparing how each behaved under a heavy backlog, then spent the next several years building its own SQL-on-Flink platform before anyone outside the company had heard of Flink SQL. That history is closer to my own - I spent close to two decades building distributed systems and streaming platforms, including several years advising banks on Kafka and Cassandra deployments, before co-founding Factor House in 2019.
I read 18 sources for this one: Uber’s own engineering blog, conference talks by named Uber engineers at Flink Forward, QCon, and Kafka Summit, and one academic paper Uber’s team co-authored for SIGMOD. Where I couldn’t confirm a claim against a named source - a couple of blog pages had gone missing since they were first indexed, and one contributor’s title was only available through LinkedIn rather than a byline - I’ve left it out rather than guess. All 18 sources are listed at the end.
The headline figure is over 4 trillion messages a day across more than 2,000 Flink SQL jobs on Uber’s internal managed platform. What’s more interesting than the scale itself is how Uber got there: not by adopting a single Flink deployment pattern, but by building three generations of tooling on top of Flink - AthenaX, then uWorc, then a fully managed Flink-as-a-Service platform - each aimed at getting Flink out of the hands of a small platform team and into the hands of anyone who could write SQL.
Company overview
Uber operates a global ride-hailing and delivery marketplace connecting riders, drivers, restaurants, and merchants, generating a continuous stream of location, order, pricing, and marketplace-balancing events across the cities it serves. That combination of real-time logistics and a large advertising business built on top of it (UberEats Ads) is what makes Uber’s Flink footprint unusually broad: the same platform underpins ML feature pipelines, ad billing, geospatial anomaly detection, and data lake ingestion.
Uber’s move to Flink followed a direct framework comparison rather than a single triggering incident. In 2016, evaluating Storm, Samza, and Spark Streaming against a real production backlog, Uber’s streaming team found Storm needed several hours to recover from backpressure where Flink needed about 20 minutes, and that Spark Streaming used five to ten times the memory of an equivalent Flink job. That comparison is what took Flink from evaluation to Uber’s primary stream processing engine.
| Date | Milestone |
|---|---|
| 2016 | Uber evaluates Storm, Samza, Spark Streaming, and Flink against a live backlog and adopts Flink as its primary stream processor |
| 2017-10 | Uber open-sources AthenaX, its SQL-on-Flink platform, already running 220+ applications in production |
| 2019-05 | Michelangelo Palette establishes Flink SQL as a first-class path for near-real-time ML features |
| 2020-12 | uWorc, Uber’s no-code workflow orchestrator abstracting Flink and Airflow, is described in detail at 10,000+ deployed workflows |
| 2021-09 | Uber publishes its real-time, exactly-once ad event processing system for UberEats Ads |
| 2024-01 | Uber’s Flink Platform team describes a mature, fully managed “Flink as a Service” internal offering |
| 2024 | Uber reports 4 trillion+ messages/day across 2,000+ Flink SQL jobs at Flink Forward Berlin |
| 2025-12 | Uber publishes IngestionNext, its Flink-to-Hudi streaming data lake ingestion system |
Uber’s Apache Flink use cases
AthenaX - self-service SQL on Flink
Uber’s first major Flink platform was built to get analysts and product teams off of hand-written streaming jobs. As Haohui Mai, Bill Liu, and Naveen Cherukuri described when they open-sourced it, AthenaX lets users submit a SQL query against a Kafka topic and have it compiled into a Flink job automatically, with a master service validating and packaging the SQL before deploying it to YARN. Within its first six months in production it was running 220+ applications, powering UberEATS Restaurant Manager order and earnings dashboards and UberPOOL trip-routing and traffic-condition analysis.
Michelangelo Palette - near-real-time ML feature pipelines
Uber’s Michelangelo ML platform needs training and serving features to stay consistent with each other even when one is computed in real time and the other from a batch warehouse. As Amit Nene and Eric Chen presented at QCon.ai, Palette uses Flink SQL to transform Kafka event streams into features that are written simultaneously to an online Cassandra store for serving and an offline Hive warehouse for training, with parity guarantees between the two paths.
Marketplace Intelligence - real-time geospatial features
Uber’s surge-pricing and demand-forecasting models need fresh, per-location signals at a scale most feature stores aren’t built for. As Feng Xu and Gang Zhao described, the Marketplace Intelligence team’s Flink pipeline generates 54 real-time geospatial features per H3 hexagon per minute, across roughly 5 million hexagons spanning 1,500 cities, feeding directly into forecasting and pricing models.
Real-time exactly-once ad event processing
UberEats Ads bills advertisers off event counts, which makes duplicate or lost events a direct revenue problem. As Jacob Tsafatinos, Yuriy Bondaruk, Yupeng Fu, and James Kwon wrote up, Uber built a three-job Flink pipeline (aggregation, attribution, then a union-and-load step) running active-active across two regions, processing hundreds of millions of ad impression, click, and order-attribution events a week with exactly-once guarantees.
Geospatial situation detection
Detecting a localized anomaly - a traffic incident, an unusual demand spike - at global scale means running complex event pattern matching across a constantly shifting set of geographic regions. As Teng “Niel” Hu presented at Flink Forward Virtual, Uber combines FlinkCEP with its own H3 hexagonal spatial indexing library and clustering techniques to detect these patterns without pre-defining every region a pattern could occur in.
IngestionNext - streaming ingestion into the data lake
Uber’s data lake was historically fed by scheduled Spark batch jobs, which meant hours of latency between an event happening and it being queryable. As Xinli Shang and co-authors describe, IngestionNext re-platforms that ingestion path onto Flink writing directly into Apache Hudi, cutting freshness from hours to minutes across the Delivery, Rider, Mobility, Finance, and Marketing Analytics organizations, while also reducing compute cost relative to the batch jobs it replaced.
Scale and throughput
| Year | Metric | Source |
|---|---|---|
| 2017 | Over 1 trillion messages/day through Kafka feeding Flink; AthenaX running 220+ applications, processing up to several million messages/second on just eight YARN containers | Mai, Liu, Cherukuri, Uber Engineering Blog |
| 2020 | “Several thousand” Flink jobs, growing roughly 30% year over year; trillions of messages and petabytes of data generated per day across the real-time platform | Yupeng Fu, Chinmay Soman, SIGMOD ’21 |
| 2021 | Marketplace feature pipeline: 120,000 events/second input across 5 million hexagons in 1,500 cities; write throughput scaled from 150 QPS to 300,000 QPS per datacenter; query P99 latency improved from 150ms to 10ms | Feng Xu, Gang Zhao, Uber Engineering Blog |
| 2021 | Ad event pipeline: hundreds of millions of ad events per week, 1-minute tumbling windows, 2-minute checkpoint interval | Tsafatinos, Bondaruk, Fu, Kwon, Uber Engineering Blog |
| 2024 | Over 4 trillion messages/day across 2,000+ Flink SQL jobs on Uber’s managed Flink platform | Yang Yang, Sai Sharath Dandi, Flink Forward Berlin |
| 2025 | IngestionNext: petabyte-scale data, hundreds of datasets, thousands of concurrent ingestion jobs; freshness cut from hours to minutes; 25% compute cost reduction versus the batch jobs it replaced | Xinli Shang et al., Uber Engineering Blog |
Within the marketplace feature pipeline specifically, Uber’s own optimization work cut per-message size from 451 to 237 bytes and reduced deduplicated message rate from 120,000/second to 8,000/second before it ever reached the write path - a reminder that a lot of Flink’s scale story at Uber has been won on the producer side of each job, not just in cluster sizing.
Uber’s Apache Flink architecture
From YARN-based AthenaX to a managed Flink SQL layer
AthenaX’s original deployment model packages and deploys compiled Flink jobs to YARN, with a master service handling SQL validation. As Yupeng Fu and Chinmay Soman describe in their SIGMOD ’21 paper, Uber’s platform has since organized around three layers - Platform, Job Management, and Infrastructure - with automated resource scaling and rule-based failure recovery sitting on top of that original YARN foundation.
FlinkSQL on Calcite
Both AthenaX and Uber’s later managed Flink SQL service compile SQL through Apache Calcite into Flink jobs, rather than requiring users to write against Flink’s DataStream API directly. This is the layer that lets Uber’s uWorc orchestrator (below) treat Flink as just another no-code building block.
Kafka in, Pinot and Hudi out
Across nearly every use case above, the same basic data flow repeats: Kafka carries the raw event stream in, Flink transforms it, and the result lands in whichever downstream system fits the consumer - Pinot for real-time OLAP serving, Hive for offline training data, Cassandra for online ML feature lookups, or, since IngestionNext, Hudi for the data lake itself.
IngestionNext’s three-layer design
IngestionNext separates its data plane (Kafka to Flink to Hudi), control plane (job lifecycle management across thousands of concurrent ingestion jobs), and a dedicated resilience layer for regional failover and fallback, so that a regional outage doesn’t require manually restarting thousands of individual jobs.
Exactly-once through checkpoint-aligned commits
The ad event pipeline ties its two-phase commit protocol to a 2-minute Flink checkpoint interval, using keyed state for deduplication. IngestionNext takes a related approach further downstream: it extends Hudi’s own metadata to embed Flink checkpoint IDs, so that a failover between regions can recover to a consistent point without Flink and Hudi disagreeing about what’s already been committed.
Special techniques and engineering innovations
AthenaX’s upstream contributions
Building AthenaX wasn’t purely an internal exercise - Uber contributed group windows and complex data type support back to Apache Flink 1.3 as part of the work, rather than maintaining those capabilities as a private fork.
uWorc - Flink and Airflow behind one no-code interface
As Sriharsha Chintalapani and Sandeep Karmakar describe, uWorc lets users build a pipeline through a drag-and-drop UI over JSON templates, without needing to know whether the engine underneath is Flink or Airflow. By the time it was documented in detail, uWorc had more than 10,000 workflows deployed out of 15,000+ pipelines company-wide, and cut typical workflow creation time from hours to under five minutes.
Protobuf as a first-class citizen
At Uber’s scale of 4 trillion-plus messages a day, serialization format matters. As Yang Yang and Sai Sharath Dandi presented at Flink Forward Berlin, Uber’s managed Flink platform moved from Avro-formatted Kafka topics to treating Protobuf as a first-class format, for performance and cost reasons at that message volume.
Micro-optimizations that moved the needle before the write path
Uber’s Marketplace Intelligence team found that several small, unglamorous changes compounded into large gains: compressing UUIDs from 128 bits to a single byte, switching from POJOs to Tuples, encoding hexagon IDs as longs instead of strings (a 50% processing-time reduction on its own), reusing objects to avoid unnecessary cloning, and replacing Flink’s native sliding-window operator with a custom FlatMap-based implementation to cut serialization and garbage-collection overhead.
Row-group-level Parquet merging
IngestionNext’s small-file problem - a familiar one to anyone streaming into a data lake - wasn’t solved by tuning file sizes alone. Uber merges at the Parquet row-group level rather than record by record, a roughly 10x improvement over record-level merging, combined with round-robin polling and per-partition quotas at the connector level to handle partition skew caused by garbage-collection pauses upstream in Kafka.
Data lineage through OpenLineage
As Zhenqiu Huang presented at Flink Forward Berlin, Uber integrated Flink’s own lineage graph and listener mechanism (built on FLIP-314) with OpenLineage, giving platform teams a standard way to trace a dataset back through the Flink jobs that produced it.
Operating Apache Flink at scale
Flink as a fully managed internal service
As Na Yang and Yusheng Chen described at Uber’s internal Flink meetup, Uber’s Flink Platform team now runs Flink as a fully managed service: developers and data scientists submit Flink SQL jobs without provisioning or operating any Flink infrastructure themselves, a meaningfully different operating model from AthenaX’s original self-service-but-still-YARN-aware design.
Multi-tenancy and noisy-neighbor isolation
Running thousands of jobs from many different teams on shared infrastructure creates a resource-isolation problem that doesn’t show up at smaller scale. Na Yang, who leads Uber’s Streaming Data and Flink team, has spoken about resource isolation between tenants as an ongoing area of platform work rather than a solved problem - a detail I appreciated, since it’s rare for a platform team to describe multi-tenancy as still evolving rather than already finished.
Coordinated, phased version upgrades
As Anand Kotriwal, Yupeng Fu, Aparajita Pandey, and Charu Jain described, upgrading Flink jobs from Java 8 to Java 11 was done as part of a broader, multi-system version upgrade (moving through Java, then Scala 2.12, then Spark 3.0, before touching Flink), specifically to de-risk the rollout by isolating which system change was responsible for any regression.
Regional failover as a first-class design goal
Both the ad event pipeline (active-active across two regions with a union-and-load reconciliation step) and IngestionNext (a dedicated resilience layer for regional failover and fallback) treat losing a region as something the pipeline design accounts for directly, rather than an operational procedure bolted on afterward.
Challenges and how they solved them
Storm and Spark Streaming couldn’t recover fast enough under load
Before settling on Flink, Uber ran a direct comparison of the streaming frameworks available to it at the time.
Problem: Under a large input backlog, Storm needed several hours to recover from backpressure, and Spark Streaming used five to ten times the memory of an equivalent Flink job.
Root cause: Storm’s backpressure recovery model and Spark Streaming’s micro-batch memory footprint weren’t built for the kind of sustained, high-volume backlog Uber’s production traffic could generate.
Solution: Uber adopted Flink as its primary stream processing engine after the comparison.
Outcome: A 20-minute backpressure recovery time versus Storm’s several hours, presented by Danny Yuan, Uber’s Real-time Streaming Lead, at QCon San Francisco in 2016.
Ad billing risked double-counting the same event
UberEats Ads bills off event counts directly, so any duplicate processing has a direct financial consequence.
Problem: A single ad impression, click, or order-attribution event could be counted more than once if a job restarted or retried mid-stream.
Root cause: Standard at-least-once delivery semantics don’t guarantee an event is billed exactly once across job restarts and retries.
Solution: Uber generates a UUID idempotency key per event, combines Flink’s two-phase commit protocol with Kafka’s read_committed transactional reads, and relies on Pinot’s native upsert support for a final layer of downstream deduplication.
Outcome: Exactly-once ad billing across a three-job, two-region pipeline processing hundreds of millions of events a week, published by Jacob Tsafatinos, Yuriy Bondaruk, Yupeng Fu, and James Kwon in 2021.
The marketplace feature pipeline hit network and memory limits at 120,000 events/second
Generating 54 features per hexagon per minute across 5 million hexagons meant the pipeline’s own message overhead became the bottleneck before cluster sizing did.
Problem: Message size and per-event object overhead were limiting throughput well below what the underlying compute could otherwise support.
Root cause: Verbose encodings (string-based UUIDs and hexagon IDs, POJO-based records, native sliding windows) were adding serialization, network, and garbage-collection cost on every message.
Solution: Uber compressed UUIDs, switched hexagon IDs from strings to longs, moved from POJOs to Tuples, reused objects instead of cloning them, and replaced native sliding windows with a custom FlatMap implementation.
Outcome: Deduplicated message rate fell from 120,000/second to 8,000/second, and write throughput scaled from 150 QPS to 300,000 QPS per datacenter, published by Feng Xu and Gang Zhao in 2021.
Streaming into the data lake created a small-file problem
Moving from scheduled batch writes to continuous streaming writes changes how files accumulate in a data lake.
Problem: Traditional record-by-record Parquet merging was too slow to keep up with continuous streaming writes at Uber’s scale, and Kafka partition skew caused by upstream garbage-collection pauses made the problem worse.
Root cause: Streaming ingestion produces many small files far more frequently than batch ingestion does, and record-level merge strategies weren’t designed for that write pattern.
Solution: Uber merges at the Parquet row-group level instead of record by record, and added connector-level fairness (round-robin polling and per-partition quotas) to handle partition skew.
Outcome: Roughly a 10x improvement in ingestion performance over record-level merging, published by Xinli Shang and co-authors in December 2025.
Flink and Hudi could disagree about what had actually been committed
A regional failover is exactly the moment when a Flink job and its downstream Hudi table are most likely to disagree about the last consistent state.
Problem: Without a shared reference point, a failover between regions risked duplicating or losing data if Flink and Hudi recovered to different points.
Root cause: Flink’s own checkpoint IDs weren’t visible to Hudi’s commit metadata, so the two systems had no common way to agree on which writes were durable.
Solution: Uber extended Hudi’s metadata to embed Flink’s checkpoint IDs directly, giving both systems a shared, deterministic recovery point.
Outcome: Deterministic recovery across regional failovers without duplicate or lost writes, as part of IngestionNext’s resilience layer.
Full tech stack
| Category | Tools | Notes |
|---|---|---|
| Stream processing | Apache Flink | Core engine for AthenaX, uWorc, Michelangelo Palette, marketplace features, ad event processing, and IngestionNext |
| Messaging | Apache Kafka | Event backbone feeding nearly every Flink job described above |
| SQL layer | Apache Calcite | Compiles SQL into Flink jobs inside AthenaX and Uber’s managed Flink SQL service |
| Real-time serving | Apache Pinot | OLAP serving layer downstream of Flink; native upsert support used for ad-event deduplication |
| Data warehouse | Apache Hive | Offline training-data and backfill sink for ML feature pipelines |
| Data lake table format | Apache Hudi | Streaming data-lake format written directly by Flink in IngestionNext |
| Distributed storage | HDFS | Long-term archival and deep storage |
| Query engine | Presto | Interactive SQL over Pinot and HDFS |
| Resource orchestration | YARN | Original deployment target for AthenaX-era Flink jobs |
| Key-value store | Docstore | Uber’s in-house store used for order and event lookups in the ad pipeline |
| ML feature store | Cassandra | Online feature serving for Michelangelo Palette |
| Geospatial indexing | H3 | Uber’s open-source hexagonal indexing library, used in marketplace features and FlinkCEP situation detection |
| Workflow orchestration | Airflow | Batch workflow engine, abstracted alongside Flink inside uWorc |
| Runtime | Java 11, Scala 2.12 | Versions Flink jobs were upgraded to from Java 8 in the 2024 coordinated upgrade |
Key contributors
| Name | Role | Contribution |
|---|---|---|
| Danny Yuan | Uber, Real-time Streaming Lead | Presented the 2016 framework comparison that led Uber to adopt Flink |
| Haohui Mai | Uber, Software Engineer, Streaming Analytics | Co-authored the AthenaX architecture post |
| Bill Liu | Uber, Software Engineer, Streaming Analytics | Co-authored the AthenaX architecture post |
| Naveen Cherukuri | Uber, Engineering Manager, Streaming Analytics | Co-authored the AthenaX architecture post |
| Amit Nene | Uber, Michelangelo ML Platform | Co-presented Flink’s role in Michelangelo Palette at QCon.ai 2019 |
| Eric Chen | Uber, Michelangelo ML Platform | Co-presented Flink’s role in Michelangelo Palette at QCon.ai 2019 |
| Feng Xu | Uber, Sr. Software Engineer II, Marketplace Intelligence | Co-authored the marketplace feature-pipeline scaling post |
| Gang Zhao | Uber, former Sr. Software Engineer II, Marketplace Intelligence | Co-authored the marketplace feature-pipeline scaling post |
| Jacob Tsafatinos | Uber, Software Engineer | Co-authored the exactly-once ad event processing post |
| Yuriy Bondaruk | Uber, Sr. Software Engineer II | Co-authored the exactly-once ad event processing post |
| Yupeng Fu | Uber, Principal Software Engineer | Co-authored the ad event post, the SIGMOD ’21 paper, and the Java upgrade post |
| James Kwon | Uber, Software Engineer II | Co-authored the exactly-once ad event processing post |
| Chinmay Soman | Uber, Inc. | Co-authored the SIGMOD ’21 real-time infrastructure paper |
| Sriharsha Chintalapani | Uber, Sr. Staff Software Engineer, Data Platforms | Co-authored the uWorc architecture post |
| Sandeep Karmakar | Uber, former lead Product Manager, data/ML platform | Co-authored the uWorc architecture post |
| Teng “Niel” Hu | Uber, Software Engineer | Presented FlinkCEP-based geospatial situation detection at Flink Forward Virtual 2020 |
| Na Yang | Uber, Flink Platform Engineering Manager | Co-presented “Flink as a Service @ Uber” and spoke separately on multi-tenancy |
| Yusheng Chen | Uber, Staff Engineer | Co-presented “Flink as a Service @ Uber” |
| Yang Yang | Uber, Sr. Staff Engineer | Presented the Protobuf-on-managed-Flink work at Flink Forward Berlin 2024 |
| Sai Sharath Dandi | Uber, Sr. Software Engineer | Co-presented the Protobuf-on-managed-Flink work at Flink Forward Berlin 2024 |
| Zhenqiu Huang | Uber, Software Engineer | Presented the OpenLineage/Flink data lineage integration at Flink Forward Berlin 2024 |
| Anand Kotriwal | Uber, Sr. Software Engineer | Co-authored the Java 8 to 11 upgrade post |
| Aparajita Pandey | Uber, Software Engineer | Co-authored the Java 8 to 11 upgrade post |
| Charu Jain | Uber, Engineering Manager | Co-authored the Java 8 to 11 upgrade post |
| Xinli Shang | Uber | Co-authored the IngestionNext architecture post |
| Jing Li | Uber | Co-authored the IngestionNext architecture post |
| Peter Huang | Uber | Co-authored the IngestionNext architecture post |
| Jack Song | Uber | Co-authored the IngestionNext architecture post |
| Jing Zhao | Uber | Co-authored the IngestionNext architecture post |
Key takeaways for your own Apache Flink implementation
These are the patterns from Uber’s own account that I’d point any team running Flink across many teams, not just one, toward:
- Plan for the platform layer to outlive any single Flink version. Uber has rebuilt the layer around Flink twice - AthenaX, then uWorc, then a fully managed Flink-as-a-Service model - while the underlying engine kept evolving underneath. Treat the SQL/orchestration layer your users touch as a separate long-lived investment from whichever Flink version happens to sit underneath it.
- Push cost optimization upstream of the cluster, not just into it. Uber’s biggest wins in its marketplace pipeline - UUID compression, encoding changes, custom windowing - all happened in how messages were represented before they ever reached the write path, not in adding more compute.
- Tie exactly-once guarantees to what actually gets billed or counted. Uber’s ad-event and IngestionNext designs both treat checkpoint alignment as a correctness mechanism, not just a recovery mechanism - useful framing if your own Flink jobs feed anything with a dollar figure attached.
- Solve multi-tenancy as an ongoing problem, not a one-time fix. Uber’s own Flink Platform lead describes resource isolation between tenants as continuing platform work even after thousands of jobs are already running - a healthier framing than treating it as solved once and never revisited.
- Sequence multi-system upgrades deliberately. Uber’s Java 8 to 11 migration moved through Java, then Scala, then Spark, before touching Flink, specifically so a regression could be traced to one system change at a time rather than a tangle of simultaneous ones.
I’ll keep working through the rest of the list and write up the next one as it’s ready.
Sources and further reading
Primary sources
- Danny Yuan, “Stream Processing and Analytics with Flink,” QCon San Francisco - https://qconsf.com/sf2016/sf2016/presentation/stream-processing-analytics-flink.html (2016)
- Haohui Mai, Bill Liu, Naveen Cherukuri, “AthenaX: Uber’s Open Source Streaming Analytics Platform” - https://www.uber.com/en-IN/blog/athenax/ (2017)
- Amit Nene, Eric Chen, Michelangelo Palette feature platform, QCon.ai San Francisco - https://www.infoq.com/presentations/michelangelo-palette-uber/ (2019)
- Feng Xu, Gang Zhao, “Building Scalable Streaming Pipelines for Near Real-Time Features” - https://www.uber.com/en-DE/blog/building-scalable-streaming-pipelines/ (2021)
- Jacob Tsafatinos, Yuriy Bondaruk, Yupeng Fu, James Kwon, “Real-Time Exactly-Once Ad Event Processing with Apache Flink, Kafka, and Pinot” - https://www.uber.com/en-DE/blog/real-time-exactly-once-ad-event-processing/ (2021)
- Yupeng Fu, Chinmay Soman, “Real-time Data Infrastructure at Uber,” SIGMOD ’21 - https://arxiv.org/abs/2104.00087 (2021)
- Sriharsha Chintalapani, Sandeep Karmakar, “uWorc: Uber’s No-Code Workflow Orchestrator” - https://www.uber.com/blog/no-code-workflow-orchestrator/ (2020)
- Teng “Niel” Hu, geospatial situation detection with FlinkCEP, Flink Forward Virtual - https://www.youtube.com/watch?v=9U8ksIqrgLM (2020)
- Na Yang, Yusheng Chen, “Flink as a Service @ Uber,” Uber x Apache Flink Meetup - https://www.meetup.com/uberevents/events/298077002/ (2024)
- Yang Yang, Sai Sharath Dandi, Protobuf on Uber’s managed Flink platform, Flink Forward Berlin - https://www.flink-forward.org/berlin-2024/recordings (2024)
- Zhenqiu Huang, Flink data lineage with OpenLineage, Flink Forward Berlin - https://www.flink-forward.org/berlin-2024/recordings (2024)
- Anand Kotriwal, Yupeng Fu, Aparajita Pandey, Charu Jain, “Lucene Version Upgrade” - https://www.uber.com/us/en/blog/lucene-version-upgrade/ (2024)
- Xinli Shang, Jing Li, Peter Huang, Jack Song, Jing Zhao, “From Batch to Streaming: Accelerating Data Freshness in Uber’s Data Lake” - https://www.uber.com/en-DE/blog/from-batch-to-streaming-accelerating-data-freshness-in-ubers-data-lake/ (2025)
I read every one of these directly rather than working from a summary, and linked the specific claim to its source throughout the piece above, not just in this list.
Enterprise tooling for Apache Flink
If you’re running Flink across more teams than one platform group can babysit by hand - which is exactly the situation Uber’s three generations of tooling were built to solve - that’s the same operational gap my team and I built Flex to close: an enterprise UI and API for Apache Flink covering job visibility, checkpoint health, and multi-tenant operations, so you don’t have to build all of the platform layer above in-house before anyone outside your team can safely run a Flink job.