Last time out, I wrote up how Uber walked away from Storm, Samza, and Spark Streaming to build its own SQL-on-Flink platform. Alibaba’s story with Apache Flink goes a layer deeper than either that piece or the Netflix one before it: rather than adopting the open-source engine and building on top of it, Alibaba’s Search and Recommendation Team decided in 2015 that Flink’s underlying model was worth building on directly, forked it into an internal engine called Blink, spent the next four years hardening it against Alibaba’s own scale, and then handed a large share of the result back to the Apache Flink project in 2019. Several of the engineers behind that work ended up on the Apache Flink PMC, where some remain today. A company’s internal fork becoming a substantial share of the open-source project itself is the part of this story that pulled me in, and it sits close to the ground I spent my own two decades on, building distributed systems and streaming platforms before co-founding Factor House in 2019.
I read 16 sources for this one: Alibaba’s own Cloud Community blog and Medium publications, the official Apache Flink blog (co-authored in two cases by Alibaba engineers), Flink Forward conference talks by named Alibaba engineers, and Alibaba Cloud’s own product documentation for its GeminiStateBackend feature. Two carried gaps I’ve flagged rather than papered over: one contributor’s title is corroborated only through search-indexed conference bios rather than a single stable bio page, and the state-backend documentation doesn’t carry an individual byline or a specific year for one of its performance claims. All 16 are listed at the end.
What makes Alibaba worth reading closely is the scale at which its choices got tested. Flink now handles more than a trillion events a day at Alibaba, spiking during its annual Double 11 shopping festival to figures most engineering teams will never need to plan for: 4 billion records a second, 7 terabytes a second, across more than 35,000 concurrent jobs. Decisions Alibaba made about job isolation, checkpointing, and state storage under that kind of load didn’t stay internal. Several of them became part of Flink itself.
Company overview
Alibaba Group is a Chinese multinational technology conglomerate operating e-commerce platforms (Taobao and Tmall), cloud computing (Alibaba Cloud), and logistics (Cainiao), among other businesses. Its scale is most visible in its annual Double 11 (11.11) Global Shopping Festival, which by 2020 was closing at US$74.1 billion in gross merchandise volume (GMV) for the day.
Alibaba’s relationship with Flink began in 2015, when its Search and Recommendation Team surveyed next-generation big data engines to rebuild index building and feature engineering as a single stream-batch system. As Wang Feng (Mowen) described in a 2019 talk at the Yunqi Computing Conference Developer Summit, Flink’s “batch on streaming” model, treating batch as a bounded special case of streaming rather than simulating streaming on top of a batch engine the way Spark does, won the team over. The Flink of 2015 wasn’t mature enough for Alibaba’s needs as it stood, so the team began building an internal fork, later named Blink. That platform went into large-scale production in 2016, supporting that year’s Double 11 festival.
| Date | Milestone |
|---|---|
| 2009 | The Stratosphere research project is founded at Technische Universität Berlin, Flink’s predecessor |
| 2014 | Stratosphere is donated to the Apache Software Foundation and renamed Flink, becoming a top-level Apache project |
| 2015 | Alibaba’s Search and Recommendation Team selects Flink and begins building Blink, its internal fork |
| 2016 | Blink goes into large-scale production for search and recommendation, supporting that year’s Double 11 |
| 2017 | Double 11 processes a peak of 470 million events/second; two Alibaba real-time infrastructure engineers become Apache Flink committers |
| 2018 | Alibaba Cloud opens its Flink-based StreamCompute platform to external enterprise customers; Alibaba organizes the first Flink Forward Asia conference, in Beijing |
| 2019-01 | Alibaba acquires dataArtisans/Ververica, the company founded by Flink’s original creators |
| 2019-02 | The official Apache Flink blog announces the plan to merge Blink back into Apache Flink |
| 2019-08 | Apache Flink 1.9.0 ships with the first Blink features merged in, roughly 1.5 million lines of code |
| 2020 | Apache Flink 1.10 completes the Blink merge; that year’s Double 11 peaks at 4 billion records/second and 7 TB/second across 35,000+ jobs |
| 2022 | Yuan Mei, head of Alibaba’s Storage Engine Team, co-authors the official Apache Flink blog post on generic log-based incremental checkpoints |
| 2024 | Apache Flink marks its 10th anniversary; Wang Feng gives the “Past, Present and Future of Apache Flink” keynote at Flink Forward Asia in Jakarta |
Alibaba’s Apache Flink use cases
Real-time search and recommendation - the platform that started it all
This was the original use case that brought Flink into Alibaba. In 2015 the Search and Recommendation Team rebuilt index building and feature engineering as a unified stream-batch pipeline on top of Flink, as Wang Feng recounted, going into large-scale production the following year. The same platform now also runs real-time machine learning for search and recommendation ranking, feeding ranking models with fresher signal than a batch pipeline could provide.
The Double 11 GMV dashboard - a second-by-second view of the festival
As Alibaba Tech describes, Alibaba’s real-time computing platform powers the Double 11 “big screen,” the live dashboard that displays group-wide gross merchandise volume second by second as the shopping festival runs. It’s the most visible face of Alibaba’s Flink usage, and the one under the most concentrated load: the platform has to keep updating that number continuously through the exact traffic spikes described under Scale below.
Stream-batch unified marketing analytics for Tmall
For years, Tmall’s core marketing decision-making system ran separate batch and streaming pipelines, built and maintained independently, at roughly double the development effort and a standing risk that the real-time and offline numbers would disagree. During Double 11 2020, Flink’s stream-batch unification replaced both with a single pipeline for the first time, jointly built by Alibaba’s Flink Team and Data Platform Team. As Wang Feng reported, the unified pipeline ran twice as fast and queried four times faster than the two separate engines it replaced.
Flink CDC - unifying data collection into lakes and warehouses
Flink CDC, led by Jark Wu’s team, as Wu describes, unifies data collection, computation, and ETL into a single Flink-based pipeline, replacing a separate Debezium-plus-Kafka setup for capturing database changes. It syncs changes out of systems like MySQL directly into data lakes and warehouses such as Hudi, removing a whole class of glue infrastructure that previously sat between the database and the analytics layer.
Ad fraud detection, logistics tracking, and infrastructure monitoring
As Wang Feng’s account of Alibaba Cloud’s real-time compute platform notes, the same Flink-based real-time computing platform underpins several narrower but continuously running systems: real-time advertisement anti-fraud detection, real-time order tracking and feedback for Cainiao (Alibaba’s logistics arm), and ECS attack detection alongside monitoring and alerting across Alibaba’s infrastructure footprint.
Scale and throughput
| Year | Metric | Source |
|---|---|---|
| 2017 (Double 11) | 470 million events/second peak; 168.2 billion RMB in GMV for the day | Alibaba Tech, Medium (2018) |
| 2019 | More than 1 trillion events/day; 472 million transactions/second at peak; petabyte-level cumulative state; cluster scale grown from several hundred servers at adoption to tens of thousands | Wang Feng, Yunqi Computing Conference Developer Summit (2019) |
| 2019 (Blink merge) | Roughly 1.5 million lines of code merged into Apache Flink 1.9.0 | Apache Flink blog / Alibaba Cloud blog (2019) |
| 2020 (Double 11) | 4 billion records/second peak; 7 TB/second peak data volume; 35,000+ real-time computing jobs; 1.5 million+ CPU cluster scale; US$74.1 billion GMV | Alibaba Cloud Community blog (2020) |
| GeminiStateBackend key-value separation | Deployment throughput up 50-70%+; average compute utilization up 50%, reaching 100-200% in typical scenarios, verified during a Double 11 festival | Alibaba Cloud documentation |
Alibaba’s own account doesn’t give a precise current job or node count outside Double 11 peaks, or an absolute figure for total state size beyond “petabyte-level” - I’ve left both as gaps rather than estimate them.
Alibaba’s Apache Flink architecture
The Blink runtime - a foundation for job isolation
As Alibaba Tech describes, Blink restructured Flink’s distributed architecture so that each job gets its own JobMaster and its own TaskManager, rather than sharing a single JobMaster and a pool of TaskManagers across every job on the cluster, and paired that with a dedicated ResourceManager that communicates with JobMasters to adjust resources in real time. That decoupled job scheduling from resource management, and it’s the foundation that eventually let Flink run natively on both YARN and, later, Kubernetes.
That’s the same job-isolation problem I’ve watched bite teams running any shared multi-tenant cluster, Kafka included: the fix generally has to happen at the platform layer, not by asking every team sharing the cluster to write more careful code.
Flink SQL and the Blink planner
Alibaba’s real-time data infrastructure team built an extensively enhanced Table API and SQL query processor on top of Blink: dynamic tables, stream-stream joins, lookup joins, window aggregation, and “retraction” for eventual consistency with batch results, backed by cost-based query optimization. As Stephan Ewen, Fabian Hueske, and Xiaowei Jiang described on the official Apache Flink blog, this became the basis of what’s known as the Blink planner, merged into Apache Flink 1.9 and 1.10 and now the primary developer-facing API for stream and batch pipelines across Alibaba.
Deployment model - from YARN to Kubernetes
As Alibaba Cloud Community describes, Flink at Alibaba runs on Hadoop-based infrastructure, using YARN for resource management and scheduling and HDFS for checkpoint and state persistence as well as batch data storage, and also runs natively on Kubernetes since the FLIP-6 resource-management rework that Blink’s architecture made possible.
The surrounding data platform - MaxCompute, Hologres, DataHub, and DataWorks
As Alibaba Cloud’s documentation for building a real-time data warehouse on Flink and Hologres lays out, Alibaba Cloud’s Realtime Compute for Apache Flink is one piece of a broader unified data warehouse architecture. MaxCompute is the offline batch data warehouse; Hologres is a Hybrid Serving and Analytical Processing (HSAP) system used as the real-time serving layer downstream of Flink, supporting up to 100 million transactions per second; DataHub handles streaming data ingestion, built on Alibaba’s Apsara distributed operating system and integrating with MaxCompute, Flink, Hologres, OSS, and Tablestore; and DataWorks ties the whole thing together as an end-to-end data integration, development, and governance layer.
Special techniques and engineering innovations
Incremental checkpointing
As Alibaba Tech explains, rather than consolidating all historical state at every checkpoint, Blink stores only the incremental changes since the previous one. That cut checkpoint time meaningfully and removed a major source of failover-recovery latency, and it was one of the pieces contributed back to Apache Flink as part of the merge.
Asynchronous I/O
A redesigned Async-IO read path in Blink lets the engine issue and manage many concurrent external-storage requests via callback rather than blocking on each one synchronously. Alibaba Tech reported this increased computation throughput 10 to 100 times over in relevant workloads.
Credit-based network flow control
As Wang Feng described, Alibaba’s architecture work also contributed credit-based network flow control into Flink’s distributed runtime, part of the same broader effort to keep the engine responsive under the backpressure conditions Alibaba’s own traffic patterns routinely created.
Operating Apache Flink at scale
State-access latency monitoring
As Li Yu (“Jueding”) and Tang Yun (“Chagan”) presented at Flink Forward Asia 2021, before Flink 1.13 there was no built-in way to monitor state-access performance. Alibaba engineers introduced state latency tracking, sampling read and write latency into a histogram, with an explicit tuning tradeoff between sampling interval, measurement accuracy and performance overhead, and how much historical data to retain against memory usage.
RocksDB version governance
Alibaba engineers, working as Flink community contributors, drove an in-place RocksDB upgrade from 5.17 to 6.20 inside Flink’s state backend, as Li Yu and Tang Yun detailed, picking up ARM support and finer-grained write-buffer memory accounting. It’s treated as ongoing governance work rather than a one-time fix, consistent with how the same team has kept revisiting state-backend internals over several years.
Benchmarking with Nexmark
As Alibaba Cloud’s GeminiStateBackend documentation notes, Alibaba uses Nexmark, a streaming benchmark, to compare state backend performance, Gemini against RocksDB, before and after making changes, rather than relying on production traffic alone to validate a tuning decision.
Idle-capacity reuse for batch workloads
As Wang Feng described, at Alibaba’s scale, more than 1.5 million CPUs of cluster capacity, batch workloads are deliberately scheduled onto the same real-time cluster’s idle nighttime capacity rather than provisioned separately, running through the same unified engine and resource pool instead of a second dedicated batch fleet.
Challenges and how they solved them
Isolating tenants on a shared cluster
Before Blink, Flink’s architecture had a single JobMaster and shared TaskManagers across a cluster, which meant a single misbehaving job, one throwing an unhandled task exception, could crash a TaskManager process and take down unrelated jobs sharing it. At Alibaba’s multi-tenant scale, that was unacceptable from the first year of large-scale production.
Problem: one bad job could crash a shared TaskManager and take unrelated jobs down with it.
Root cause: the original Flink runtime had no isolation boundary at the job level, one JobMaster and a shared TaskManager pool serving every job on the cluster.
Solution: Blink gave every job its own JobMaster and TaskManager, paired with a dedicated ResourceManager that adjusts resources per job in real time.
Outcome: this became Alibaba’s isolation model through years of subsequent scale growth, and was among the pieces of Blink’s architecture later contributed back to Apache Flink, as Alibaba Tech recounted.
Checkpoints stalling under Double 11 traffic spikes
Standard aligned checkpoints require every upstream barrier to reach an operator before it snapshots. Under backpressure, exactly the condition a Double 11 traffic spike creates, that can stall a checkpoint indefinitely.
Problem: aligned checkpoints could stall under heavy backpressure, precisely when Alibaba most needed them to succeed.
Root cause: checkpoint completion was tied to how fast barriers could physically flow through already-congested channels.
Solution: unaligned checkpoints (FLIP-76) snapshot in-flight channel and output-buffer state alongside operator state, so barriers can be pushed through to the sink immediately regardless of backpressure, with Flink automatically switching between aligned and unaligned modes based on an alignment timeout.
Outcome: presented as a 2020 streaming-engine-kernel achievement at Flink Forward Asia 2020 by Wang Feng (Mowen).
RocksDB memory growing unpredictably under container limits
Flink doesn’t cap the number of user-declared states or column families per operator, and operators can share a slot, so memory pressure from RocksDB-backed state could accumulate unpredictably in containerized deployments.
Problem: RocksDB memory usage grew unpredictably under Kubernetes resource limits, and a write-buffer manager pre-emptively flushing small memtables produced large numbers of small SST files with poor read and write performance.
Root cause: no per-operator ceiling on declared state, combined with a shared write-buffer manager and block cache across operators in a slot.
Solution: RocksDB was upgraded from 5.17 to 6.20 inside Flink for finer-grained write-buffer memory accounting, alongside new configuration validation for arena-block size to stop premature memtable flushes.
Outcome: presented by Li Yu (“Jueding”) and Tang Yun (“Chagan”) at Flink Forward Asia 2021.
Compaction driving latency spikes in checkpointing
RocksDB’s own incremental checkpoint still required periodic compaction per subtask, and the state backend otherwise sat idle until a checkpoint barrier physically arrived at each operator.
Problem: checkpoint durability was tied directly to compaction timing, creating unpredictable latency spikes at scale, while idle time before a checkpoint barrier arrived went unused.
Root cause: the state backend’s only way to acknowledge a checkpoint was to finish materializing full state, which compaction could delay.
Solution: the generic log-based incremental checkpoint, or changelog state backend, co-authored by Yuan Mei (Head of Alibaba’s Storage Engine Team) and Roman Khachatryan: operators continuously log state changes to a changelog in addition to the state tables, and can acknowledge a checkpoint as soon as that log write is durable, while full-state materialization happens asynchronously in the background.
Outcome: in a published ValueState workload benchmark, 99.9th-percentile checkpoint duration dropped from 10 seconds to 1 second, at the cost of a larger full checkpoint (19.6GB up from 25.6GB) and longer worst-case recovery (20-21 seconds up to 35-65 seconds). Publishing that worst-case recovery-time cost alongside the headline win, rather than only the number that flatters the change, is the kind of detail I look for as a sign a benchmark is being reported honestly.
Local disk becoming a capacity ceiling for large state
Deployments with large amounts of state kept hitting local-disk capacity limits, and the only lever available was adding more parallelism or resources purely to get more aggregate disk.
Problem: large-state deployments hit local-disk ceilings with no way to add storage independently of compute.
Root cause: state size was tied directly to whatever local disk happened to be attached to a given TaskManager.
Solution: GeminiStateBackend’s compute-storage separation decouples state size from local node disk entirely, with adaptive hot and cold tiered storage on top.
Outcome: verified during a Double 11 Shopping Festival as part of Gemini’s broader production validation.
I’ve watched more than one platform team quietly “solve” running out of local disk by throwing more nodes at the problem instead of fixing the underlying coupling. Alibaba’s fix here is the more honest version of that same story: decouple the two properly instead of scaling around the symptom.
Dual-stream joins bottlenecking on state storage
Dual-stream and multi-stream joins are one of the harder state-storage bottlenecks in stream processing generally, particularly with low join hit rates or long values.
Problem: joins across multiple streams bottlenecked on state storage, especially with long values or low hit rates.
Root cause: a single LSM tree storing both short keys and long values together forces reads of unrelated value data just to resolve a join key.
Solution: GeminiStateBackend’s adaptive key-value separation splits keys from values within the LSM tree, with no manual tuning required.
Outcome: verified during a Double 11 Shopping Festival: deployment throughput increased 50-70% or more, and average compute utilization increased 50%, reaching 100-200% in typical scenarios.
Full tech stack
| Category | Tools | Notes |
|---|---|---|
| Stream/batch engine | Apache Flink, Blink | Blink was Alibaba’s internal fork (2015-2019) providing the isolated per-job runtime, incremental checkpointing, and async I/O; merged into Apache Flink 1.9-1.10 |
| SQL and data integration | Flink SQL (Blink planner), Flink CDC | Flink SQL is the primary developer-facing API for stream and batch pipelines; Flink CDC, led by Jark Wu’s team, unifies data collection, computation, and ETL for database-to-lake/warehouse sync |
| State storage | GeminiStateBackend, RocksDB (versions 5.17 to 6.20 tracked) | Gemini is the default LSM-tree-based state backend for Alibaba Cloud’s Realtime Compute for Apache Flink, with compute-storage separation and key-value separation; RocksDB is the alternate backend, tuned for memory control and ARM support |
| Resource management | Hadoop YARN, Kubernetes | YARN handles resource management and scheduling for on-premises clusters; Kubernetes is the native scheduling target since the FLIP-6 rework |
| Distributed storage | HDFS | Persistence for Flink checkpoints and savepoints, and for batch (offline) data storage |
| Real-time serving/OLAP | Hologres | HSAP system used as the real-time serving and query layer downstream of Flink, supporting up to 100 million TPS |
| Offline data warehouse | MaxCompute | Alibaba Cloud’s batch data warehouse, paired with Flink for the batch side of the unified architecture |
| Streaming ingestion | DataHub | Streaming data ingestion service built on the Apsara distributed OS, integrating Flink with MaxCompute, Hologres, OSS, and Tablestore |
| Data platform orchestration | DataWorks | End-to-end data integration, development, and governance layer tying Flink, MaxCompute, and Hologres together |
| Messaging | Kafka, RocketMQ | Upstream message-queue sources that Flink jobs read from and track offsets and state against |
Key contributors
| Name | Role | Contribution |
|---|---|---|
| Wang Feng (Mowen) | Senior Technical Expert, Alibaba Computing Platform Division; initiator of Apache Flink Community China | Authored or presented several primary sources used here, including “Why Did Alibaba Choose Apache Flink Anyway?”, the 2020 Double 11 stream-batch unification writeup, and the Flink 10th-anniversary keynote |
| Xiaowei Jiang | Senior Director, Alibaba; led the StreamCompute Platform and later the Hologres team | Co-authored the official Apache Flink blog post on Blink’s contribution back to Flink; gave “A Year of Blink at Alibaba” and a Flink Forward Berlin keynote |
| Jark Wu | Apache Flink PMC member and Committer; head of Flink SQL and Flink CDC, Alibaba Cloud | Built out Flink SQL semantics and features; leads Flink CDC development |
| Kurt Young | Apache Flink and Apache Druid PMC member; engineering lead, Alibaba Group | Led Alibaba’s SQL engine team built on Flink, serving nearly all Alibaba business units; presented on the 1.9.0 Blink planner at Flink Forward |
| Shaoxuan Wang | Apache Flink Committer and PMC member; senior engineering manager, Alibaba | Led development of Apache Flink SQL |
| Jincheng Sun | ASF Member; Apache Flink PMC and Committer; Staff Engineer, Alibaba | Started and led development of PyFlink, Flink’s Python API |
| Li Yu (“Jueding”) | ASF Member; Apache Flink and HBase PMC; Senior Technical Expert, Alibaba | Co-presented Flink state-backend and RocksDB improvements at Flink Forward Asia 2021 |
| Tang Yun (“Chagan”) | Apache Flink Committer; Technical Expert, Alibaba | Co-presented Flink state-backend and RocksDB improvements at Flink Forward Asia 2021 |
| Yuan Mei | Engine Architect of Apache Flink; Head of Alibaba’s Storage Engine Team | Co-authored the official Apache Flink blog post on generic log-based incremental checkpoints; presented Fault Tolerance 2.0 at Flink Forward Asia 2021 |
| Song Xintong | Senior development engineer, Alibaba | Authored an in-depth breakdown of Flink’s resource-management mechanism |
Key takeaways for your own Apache Flink implementation
These are the patterns from Alibaba’s own account that I’d point any team running Flink at serious multi-tenant scale toward:
- Isolate jobs at the runtime level, not just the code level. Alibaba’s original shared-TaskManager model meant one bad job could take down others sharing it; Blink’s fix was giving every job its own JobMaster and TaskManager, an infrastructure-layer decision no amount of careful application code could substitute for.
- Treat checkpoint improvements as tradeoffs, not free wins. Every checkpoint technique Alibaba published, unaligned checkpoints and the changelog state backend alike, came with an explicit cost in larger checkpoints or longer worst-case recovery alongside the headline win. That’s the number worth asking for before adopting either yourself.
- Separate compute from storage before large state becomes an operational fire. GeminiStateBackend’s compute-storage separation exists because Alibaba kept hitting local-disk ceilings and solving it by over-provisioning parallelism. Decoupling the two earlier avoids that cycle entirely.
- Unify real-time and offline pipelines once the drift between them becomes the real cost. Tmall’s marketing decision system ran separate batch and streaming pipelines for years, at roughly double the development effort and a standing risk the two would disagree, before stream-batch unification replaced both with one.
- Upstreaming your own fixes compounds. Several of the engineers who built Blink’s isolation and checkpointing model ended up on the Apache Flink PMC; the fixes Alibaba needed for its own scale are now defaults every Flink user gets for free.
I’ll keep working through the rest of the list and write up the next one as it’s ready.
Sources and further reading
Primary sources
- Stephan Ewen, Fabian Hueske, Xiaowei Jiang, “Batch as a Special Case of Streaming and Alibaba’s Contribution of Blink” - https://flink.apache.org/2019/02/13/batch-as-a-special-case-of-streaming-and-alibabas-contribution-of-blink/ (2019)
- Wang Feng, “Four Billion Records Per Second: Stream-Batch Integration Implementation of Alibaba Cloud Realtime Compute for Apache Flink During Double 11” - https://www.alibabacloud.com/blog/four-billion-records-per-second-stream-batch-integration-implementation-of-alibaba-cloud-realtime-compute-for-apache-flink-during-double-11_596962 (2020)
- Alibaba Tech, “Alibaba Blink: Real-Time Computing for Big-Time Gains” - https://alibabatech.medium.com/alibaba-blink-real-time-computing-for-big-time-gains-707fdd583c26 (2018)
- Wang Feng, “Why Did Alibaba Choose Apache Flink Anyway?”, Yunqi Computing Conference Developer Summit - https://alibaba-cloud.medium.com/why-did-alibaba-choose-apache-flink-anyway-4cf0da145ff8 (2019)
- Alibaba Cloud Realtime Compute Team, “Flink Forward Asia 2020: Keynote Summary” - https://www.alibabacloud.com/blog/flink-forward-asia-2020---keynote-summary_597131 (2021)
- Alibaba Cloud, “What’s All Involved with Blink Merging with Apache Flink” - https://www.alibabacloud.com/blog/whats-all-involved-with-blink-merging-with-apache-flink_595401 (2019)
- Alibaba Cloud, GeminiStateBackend product documentation - https://www.alibabacloud.com/help/en/flink/user-guide/geministatebackend (accessed 2026)
- Roman Khachatryan, Yuan Mei, “Improving Speed and Stability of Checkpointing with Generic Log-Based Incremental Checkpoints” - https://flink.apache.org/2022/05/30/improving-speed-and-stability-of-checkpointing-with-generic-log-based-incremental-checkpoints/ (2022)
- Li Yu (“Jueding”), Tang Yun (“Chagan”), “Flink State Backend Improvements and Evolution in 2021,” Flink Forward Asia 2021 - https://www.alibabacloud.com/blog/flink-state---backend-improvements-and-evolution-in-2021_599218 (2022)
- Alibaba Cloud Community, “Deploy Apache Flink Natively on YARN or Kubernetes” - https://www.alibabacloud.com/blog/deploy-apache-flink-natively-on-yarn-or-kubernetes_595189 (undated)
- Song Xintong, “Data Warehouse In-Depth: Interpretation of Flink Resource Management Mechanism” - https://alibaba-cloud.medium.com/data-warehouse-in-depth-interpretation-of-flink-resource-management-mechanism-5c13b531abfa (undated)
- Jark Wu (referenced), “The Next Step of Flink CDC” - https://www.alibabacloud.com/blog/601210 (2023)
- Alibaba Cloud, “Build a Real-Time Data Warehouse Based on Flink and Hologres” - https://www.alibabacloud.com/help/en/flink/realtime-flink/use-cases/build-real-time-data-warehouse-based-on-flink-hologres (undated)
- Wang Feng, “The Past, Present and Future of Apache Flink,” Flink Forward Asia keynote - https://www.alibabacloud.com/blog/the-past-present-and-future-of-apache-flink_601867 (2024)
- Kurt Young, “What’s New in 1.9.0 Blink Planner,” Flink Forward - https://www.slideshare.net/slideshow/whats-new-in-190-blink-planner-kurt-young-alibaba/185807449 (2019)
I read every one of these directly rather than working from a summary, and linked the specific claim to its source throughout the piece above, not just in this list.
Enterprise tooling for Apache Flink
Alibaba’s story is unusual because it faced exactly the operational gap Flex exists to close, but from the vendor side rather than the customer side: platform-layer tooling for running Flink across many teams and jobs, not just one. Alibaba built that tooling itself, at a scale few companies will ever reach, and gave a meaningful share of it back to the open-source project. Most teams don’t have the headcount to build a Blink of their own, and that’s exactly the gap my team and I built Flex to close: an enterprise UI and API for Apache Flink covering job visibility, checkpoint health, and multi-tenant operations, so you get some of what Alibaba built in-house without having to build it yourself.