I spent close to two decades building distributed systems and streaming platforms before co-founding Factor House in 2019, including several years advising banks on Kafka and Cassandra deployments, so I read a lot of company engineering blogs looking for the gap between how a streaming platform looks on a slide and how a team actually runs it once it is carrying real production traffic. Lyft is the second stop in that reading project, after Netflix’s Flink fleet: eight sourced pieces, mostly Lyft’s own engineering blog spanning 2021 to 2026, plus one conference talk and Lyft’s own open-source repository, all read directly rather than taken from a secondhand summary.
What makes Lyft’s account worth writing up is how candid it is about the parts that went wrong. Most vendor-adjacent writing about a streaming migration reads as a highlight reel. Lyft’s own engineers instead published a five-day silent data-loss incident down to the specific Flink internals that caused it, and a multi-quarter operator migration that admits to a downtime regression along the way. Apache Flink sits underneath a genuinely wide set of jobs at Lyft: dynamic-pricing feature generation, a fraud-detection model, an ML feature-serving layer, and one of the company’s largest streaming jobs, a persistence pipeline moving 80 gigabytes of events a minute from Kinesis to S3. The common thread across all of it is a platform team that kept rebuilding its own deployment infrastructure rather than accept its limits, and eventually decided the better rebuild was to delete their own code and adopt the community’s instead.
Company overview
Lyft operates a ridesharing marketplace connecting drivers and passengers across the United States and Canada. As Sherin Thomas put it in her QCon London talk, nearly every decision the app makes in the moment, matching a rider to a nearby driver, estimating an ETA, setting a price that reflects current supply and demand, depends on the freshest possible read of what is happening on the platform right now, which is what pulls a ridesharing marketplace toward stream processing in the first place.
Lyft’s Marketplace organization first turned to stream processing in 2017, when its dynamic-pricing system ran on a cronjob-based DAG that recomputed prices once a minute and carried several minutes of latency end to end as a result. That latency, plus a multi-week cycle to hand-write each new ML feature, is what pushed the team toward Apache Beam running on Flink, the foundation most of the systems in this piece grew out of.
| Date | Milestone |
|---|---|
| 2017 | Lyft’s Marketplace team begins replacing its cronjob-based pricing DAG with an Apache Beam pipeline on Flink |
| 2019 | Lyft open-sources its self-built Flink Kubernetes Operator, flinkk8soperator |
| 2020 | Lyft moves off one shared Flink cluster onto per-application Kubernetes deployments; Sherin Thomas presents Lyft’s Dryft self-service streaming framework at QCon London |
| Feb 2021 | Lyft publishes its first “Gotchas of Stream Processing” post, on data skewness |
| Mar 2021 | Lyft publishes the architecture of its Feature Service, unifying batch and streaming ML feature serving |
| Sep 2022 | Lyft publishes the evolution of its Marketplace streaming pipelines, including the split into feature-generation and model-execution pipelines |
| Jun 2023 | Lyft publishes its profiling and performance-improvement playbook for streaming pipelines |
| Aug 2023 | Lyft identifies and fixes a five-day Kinesis-connector watermark deadlock, filed upstream as FLINK-29099 |
| Sep 2026 | Lyft completes a multi-quarter migration of its entire streaming fleet to the open-source Apache Flink Kubernetes Operator |
Lyft’s Flink use cases
Lyft runs Flink, sometimes directly and sometimes through Apache Beam’s Flink runner, across at least five distinct systems: real-time pricing and matching features, fraud detection, a shared ML feature-serving layer, a self-service framework for building new streaming features, and large-scale event persistence.
Marketplace pricing and matching feature engineering
Lyft’s Marketplace organization, the team responsible for dynamic pricing and driver-rider matching, runs Apache Beam pipelines on the Flink runner to turn raw ride and app events into the ML features those models consume. As Rakesh Kumar described it, the team replaced a cronjob-based DAG that computed pricing once a minute, with several minutes of inherent latency baked in, with a Beam-on-Flink pipeline that cut feature-generation code from 10,000 lines to 4,000 and cut ML feature development time in half.
Real-time fraud detection
Lyft’s research scientists built a convolutional neural network that fingerprints sequences of user actions in event-time order, such as a passenger requesting a ride, contacting the driver, then cancelling, a pattern associated with phishing scams targeting drivers. Sherin Thomas walked through the specific case at QCon London in 2020: a scammer poses as Lyft support, talks a driver into cancelling a ride mid-pickup, then extracts a two-factor code under the guise of “verifying” the driver’s identity. Streaming, rather than batch, matters here because the model needs the last several actions a person took, in the order they happened, to catch the pattern before the damage is done.
ML feature serving for training and online inference
Lyft built a shared Feature Service so that a feature, once defined, could be computed either in a Flyte batch job or a Flink streaming job and made available identically to both a model-training pipeline and a low-latency online prediction request. Vinay Kakade and Shiraz Zaman describe it hosting several thousand features across fraud detection, driver dispatch, pricing, location projections, growth, and customer-support models, with feature definitions and validations shared between the training and serving paths so the two cannot silently drift apart.
Dryft: a self-service streaming framework
Because most of Lyft’s streaming users are data and research scientists rather than distributed-systems engineers, the Streaming Compute team built a framework, referred to in Sherin Thomas’s talk as Dryft, that lets a user submit a one-page declarative SQL configuration and have the platform handle query analysis, resource sizing, and Flink job generation on their behalf. Around 90% of feature-definition use cases at Lyft are expressed purely in Flink SQL through this framework.
High-volume event persistence
A dedicated Flink job persists a large majority of all events generated at Lyft, streaming them from Kinesis and writing Parquet files to S3 for downstream analytics and offline processing. Seth Saperstein describes it as one of Lyft’s largest streaming jobs by throughput, and it is the job at the center of the watermark-deadlock incident covered below.
Scale and throughput
| Year | Metric | Source |
|---|---|---|
| 2017 | Legacy pricing pipeline: cronjob DAG with minutes of inherent latency; rewrite cut feature-generation code from 10,000 to 4,000 lines | Evolution of Streaming Pipelines in Lyft’s Marketplace |
| ~2019-2020 | Roughly 60 Flink jobs ran on a single shared EC2 cluster, with capacity manually estimated per job | Sherin Thomas, QCon London 2020 |
| 2021 | Feature Service hosts several thousand features, serves millions of requests per minute at single-digit millisecond latency, with 99.99%+ availability | ML Feature Serving Infrastructure at Lyft |
| 2022 | Feature-generation pipeline ingests tens of millions of events per second; model-execution pipeline handles tens of thousands of events per minute; end-to-end latency held under 1 minute through peak-load events | Evolution of Streaming Pipelines in Lyft’s Marketplace |
| 2023 | Kinesis-to-S3 persistence job runs at 80 gigabytes per minute average throughput with a parallelism of 1,800 | Where’s My Data |
| 2026 | Hundreds of critical streaming jobs migrated to the open-source Apache Flink Kubernetes Operator; autoscaling right-sizes a fleet described as overprovisioned by a few million dollars a year | Rerouting the Stream |
A parallelism of 1,800 configured for a single Flink application, rather than the size of an underlying cluster, gives a sense of how much of Lyft’s total event volume that one persistence job is carrying on its own.
Lyft’s Flink architecture
Event ingestion and persistence
Platform events land on AWS Kinesis streams and are separately persisted to S3, both for the offline analytics use case and as the durable source that other Flink jobs consume from. The persistence job itself writes Parquet directly, keeping the S3 copy in a format downstream batch and ad hoc analytics tooling can read without a conversion step.
Marketplace pipeline topology
The Marketplace pipeline architecture went through two rounds of decomposition. It started as one Beam pipeline handling ingestion, feature generation, and model execution together. Once that stopped scaling to larger markets, Lyft split it into a feature-generation pipeline optimized for throughput on a larger cluster, and a model-execution pipeline isolated on a smaller one, since the two had different performance profiles and different release cadences. As usage grew further, Lyft decomposed the feature-generation side again into a standardized two-tier design: a first tier handling ingestion, filtering, hydration, and metadata tagging that publishes clean signals to Kafka topics, and a second tier that aggregates those Kafka topics into standard, shareable ML features.
Dryft’s control plane and fanout pattern
Dryft’s control plane takes a user’s declarative SQL configuration, runs query analysis to estimate the compute the job needs, and generates the underlying Flink job without the user touching Kubernetes or Flink configuration directly. Feature-materializing jobs write once to a Kinesis stream, and a separate, stateless “feature fanout” job, itself built on Dryft and running on Flink, reads that stream and writes out to Kafka, DynamoDB, Druid, Hive, and Elasticsearch. Keeping the fanout job separate means a user’s application logic never has to change when a new sink is added or an existing one’s write pattern needs adjusting.
Feature Service storage and serving path
The Feature Service stores each feature’s most recent value in DynamoDB, using DynamoDB’s conditional writes for optimistic locking so that concurrent updates from distributed callers resolve deterministically by timestamp. DynamoDB values are replicated to Hive for the large historical reads a training job needs, and to Elasticsearch for feature search and inspection. Redis sits in front of DynamoDB as a write-through cache, which is what gets the read path down to the single-digit-millisecond latency the online inference callers need.
Deployment topology: from one shared cluster to the Apache operator
Lyft’s deployment model went through three distinct phases. It began with one shared EC2 cluster running every Flink job together, a setup Sherin Thomas described in her QCon London talk as “multi tenancy hell,” where a single misbehaving job could take the whole cluster down and capacity had to be hand-calculated ahead of time. Lyft then built its own Kubernetes-native control plane, later open-sourced as flinkk8soperator, giving every Flink application its own dedicated job manager and task manager pods. In 2026, Lyft retired that operator entirely in favor of the community-maintained Apache Flink Kubernetes Operator, using a translation layer in its internal deploy API so that existing Jsonnet-based application configs, written against the legacy FlinkApplication custom resource, continue to work unchanged against the Apache operator’s FlinkDeployment resource.
Special techniques and engineering innovations
Key salting for hot regions
Marketplace data is naturally keyed by geography, and some cities generate far more ride events than others. As Rakesh Kumar describes, Lyft salts keys, using random.choice in the pipeline code, before the heaviest per-region processing steps, then strips the salt and re-keys by region before any downstream join, so that a handful of dense cities cannot dominate a handful of task slots.
Two-stage distributed aggregation
For region-level aggregates, Lyft partitions first by geohash and aggregates at that finer grain, then repartitions by region to combine the geohash-level results. It costs an extra network shuffle, but it distributes heavy per-shard computation across more nodes than a single region-level partition would allow.
Deliberate reshuffling to break operator fusion
Flink’s default behavior fuses adjacent operators to avoid unnecessary data movement, which is usually a win, but it can leave data pinned to a poorly partitioned upstream key. Lyft found that some Kinesis shards were skewed and Flink was fusing operators in a way that left resources underutilized elsewhere in the job, so the team added an explicit reshuffle after ingestion to force a redistribution, trading some network overhead for materially lower end-to-end latency.
CRD translation layer for the operator migration
As Maheep Myneni, Arda Kuyumcu, and Prem Santosh Udaya Shankar describe, rather than rewrite every team’s Jsonnet-defined application configuration to target the Apache operator’s native FlinkDeployment schema, Lyft’s deploy API translates the legacy FlinkApplication resource into a FlinkDeployment at deploy time: flattening fields like jarName into a nested jarURI, defaulting the upgrade mode to last-state, promoting inline pod settings into a proper PodTemplateSpec, and injecting legacy environment variables like APP_NAME so existing application code keeps working. It is the kind of unglamorous compatibility shim that made an otherwise risky wholesale migration reversible one application at a time.
Sidecar isolation for Python SDK harness memory
Lyft’s Apache Beam workloads run Python user code in an SDK harness that lives outside the JVM’s own memory accounting, and for years Lyft had simply carved out a share of JVM overhead for it. That worked until autotuning arrived and started reclaiming memory it saw Flink itself was not using, OOM-killing task managers across the Beam fleet in the process. Lyft’s fix was to move the Python harness out of the Flink container entirely, into its own sidecar container with its own resource limits, so autotuning and the Python process stopped competing for the same memory pool. I’ve seen the same “hidden overhead” problem show up in enterprise Kafka deployments with sidecar-heavy service meshes, and Lyft’s fix, giving the thing you cannot observe its own isolated allocation, is the right instinct in both settings.
Two-tier autotuning and autoscaling policy
Autotuning and in-place autoscaling do not compose cleanly, because resizing a job’s resources requires a restart while rescaling its parallelism does not. Lyft resolved the tension by splitting its fleet: business-critical pricing and routing jobs keep in-place autoscaling and turn autotuning off, while less-critical jobs run both autotuning and restart-based autoscaling.
Operating Flink at scale
Profiling toolchain
As Rakesh Kumar details, Lyft pairs Pyflame and async-profiler flame graphs for its Python-based Beam pipelines with standard JVM profilers for pure-Flink applications, and layers Flink’s own dashboard on top, using its per-operator CPU-busyness view and built-in flame graph (enabled with rest.flamegraph.enabled: true) to narrow a slow pipeline down to a specific operator before reaching for a heavier tool.
Metrics, alerting, and the most common failure mode
Beyond standard throughput and latency metrics, Lyft alerts on checkpoint size, checkpoint failure, and pipeline restarts, and separately tracks business-level SLA metrics, since a pipeline’s system metrics can look healthy while a downstream business metric quietly degrades. Across their own postmortems, Lyft’s engineers rank data skewness, a hot key or hot shard, as the cause in roughly 80% of the performance problems they have diagnosed, ahead of oversized windows, slow external-service calls, and serialization overhead.
Checkpoint tuning and availability-zone placement
Lyft tunes checkpoint intervals against each pipeline’s SLA rather than using one default, has tested unaligned checkpoints without regressions, and actively purges state tied to global or unbounded windows to keep checkpoint size from growing indefinitely. Because, in their words, a streaming pipeline “can’t be partially down,” task managers are kept within a single availability zone to minimize the network latency between them.
Incremental, reversible migration
As Lyft’s Streaming Compute team recounts, the 2026 operator migration moved one application at a time, starting in staging, using dynamic configuration so any app that hit trouble could be rolled straight back to the legacy operator without a redeploy. When Lyft hit a bug in the still-developing FlinkBlueGreenDeployments CRD, a Flink configuration-field-renaming issue, the team filed it and contributed a fix upstream rather than patching around it in a private fork, a choice they credit with changing the team’s posture from maintaining the operator to using and occasionally improving it.
Elastic node provisioning with Karpenter
Autotuning made Lyft’s static, ratio-constrained node pools obsolete, since the same application could need a very different CPU-to-memory ratio from one tuning cycle to the next. Lyft adopted Karpenter to provision EC2 nodes on demand to match what each pod actually needs, combined with a custom “most-allocated” Kubernetes scheduler that packs pods onto already-busy nodes first, letting underutilized nodes get shed as autoscaling restarts move pods around.
Challenges and how they solved them
A five-day silent data loss in the Kinesis connector
For over a month, a Lyft on-call rotation kept hitting the same outage on their largest persistence job on a roughly biweekly cadence: throughput would silently collapse, and nothing short of a hard restart with no savepoint would bring it back.
Problem: A single subtask, out of 1,800, stopped consuming its Kinesis shard entirely, for five days, before anyone noticed a throughput drop.
Root cause: Extended CPU throttling caused the subtask’s local watermark to fall silent long enough to be marked idle. Because Flink’s event-time-alignment mechanism excludes idle subtasks from the global watermark calculation, and the global watermark had meanwhile moved backward once the throttled subtask’s own stale reading dropped out, the subtask ended up too far ahead of a watermark it could no longer see updates to, a genuine deadlock rather than ordinary lag.
Solution: Lyft traced the exact sequence through Flink’s Kinesis consumer internals, filed and fixed the bug upstream as FLINK-29099, decomposed the offending application into smaller ones to reduce CPU-throttling risk, and added per-shard monitoring comparing production against consumption.
Outcome: Published by Seth Saperstein in August 2023, including the shard-level data that fell out of Kinesis’s 7-day retention window before the root cause was found, meaning the incident cost real, if small, unrecoverable data loss.
A pricing pipeline that stopped scaling
Problem: After a successful rollout to smaller markets, with full data parity against the legacy system and 60% lower latency, Lyft’s first Beam-on-Flink pricing pipeline saw latency climb again and throughput suffer once it reached larger markets.
Root cause: One pipeline was doing two jobs with very different resource and release profiles: lightweight feature generation and CPU- and memory-intensive model execution, sharing the same operators and the same deploy cadence.
Solution: Lyft split the pipeline in two, a high-throughput feature-generation pipeline on a larger cluster and an isolated model-execution pipeline on a smaller one, each scaled and released independently.
Outcome: Stable end-to-end latency under one minute, sustained through peak-load events like New Year’s Eve and Halloween, as Rakesh Kumar reported in 2022.
A homegrown Kubernetes operator that ran out of road
Problem: Lyft’s own Flink Kubernetes Operator, architected in 2020, exposed a single memory knob for non-JVM overhead with no real observability into what Beam’s Python SDK harness actually needed, and its savepoint-based upgrade path had no retry logic, so a transient savepoint failure on a large-state job could escalate into a full stateless restart.
Root cause: Both were reasonable design choices for the smaller, more uniform fleet Lyft had in 2020, but they did not hold up as the fleet grew larger and more varied.
Solution: Migrate the entire fleet to the open-source Apache Flink Kubernetes Operator, which offers an explicit last-state upgrade mode that restores from high-availability metadata or the latest checkpoint by design, and, layered with autotuning, real per-job memory right-sizing instead of a single hand-tuned fraction.
Outcome: A multi-quarter migration that Lyft’s Streaming Compute team says minimized the operator-maintenance burden that used to accompany every Flink version upgrade, and an autoscaler now right-sizing a fleet they describe as having been overprovisioned by a few million dollars a year. I’d add that giving up on maintaining your own control-plane fork is a genuinely hard call for a platform team to make, and the honesty in Lyft’s writeup about how long it took to get there is more useful to a reader than a cleaner story would have been.
Deploy-time downtime got worse before it got better
Problem: Once migrated onto the Apache operator, every deploy cost Lyft an average of 3 to 6 minutes of downtime, and closer to 20 minutes for the largest jobs.
Root cause: The Apache operator manages only job manager pods directly and lets the job manager coordinate task manager teardown and startup afterward, in sequence, whereas Lyft’s own operator had managed both lifecycles together specifically to minimize that gap.
Solution: Adopt FlinkBlueGreenDeployments, a CRD that keeps the old and new FlinkDeployment running side by side until cutover. It was still under active development when Lyft started testing it, and the team hit and fixed a configuration-field-renaming bug, contributing the fix back upstream.
Outcome: Zero-downtime deploys restored for Lyft’s most latency-sensitive jobs, and, per Lyft’s own account, a concrete example of the benefit of running on infrastructure other companies are also actively improving.
Full tech stack
| Category | Tools | Notes |
|---|---|---|
| Stream processing | Apache Flink, Apache Beam | Flink is the execution engine; Beam is the pipeline SDK for Lyft’s Python-based Marketplace pipelines, run on the Flink runner |
| Kubernetes control plane | Apache Flink Kubernetes Operator (current), flinkk8soperator (retired 2026) |
Lyft’s own operator is open source on GitHub; superseded by the community-maintained Apache operator in 2026 |
| Event transport | AWS Kinesis, Apache Kafka | Kinesis for primary ingestion; Kafka as the intermediate topic layer in the standardized Marketplace feature pipelines |
| Storage | AWS S3, DynamoDB, Redis, Hive, Elasticsearch, Druid | S3 for persisted events and Flink checkpoints/savepoints; DynamoDB as the Feature Service system of record with Redis as a read cache; Hive, Elasticsearch, and Druid as replication targets for training, search, and analytics |
| Orchestration | Flyte | Runs scheduled batch feature-extraction jobs against Lyft’s Hive data warehouse |
| Serialization | Protocol Buffers | Cut one pipeline’s data-transfer volume by roughly 20% versus its prior format |
| Infrastructure | Kubernetes (AWS EKS), Karpenter, Jsonnet | Karpenter provisions nodes on demand post-autotuning; Jsonnet defines Flink application deployment specs |
Key contributors
| Name | Role | Contribution |
|---|---|---|
| Maheep Myneni, Arda Kuyumcu, Prem Santosh Udaya Shankar | Streaming Compute team, Lyft | Co-authored the 2026 writeup of Lyft’s migration to the Apache Flink Kubernetes Operator |
| Seth Saperstein | Lyft Engineering | Investigated and wrote up the five-day Kinesis-connector watermark deadlock, and filed the upstream Flink fix |
| Rakesh Kumar | Marketplace team, Lyft | Authored the “Gotchas of Stream Processing” and “Evolution of Streaming Pipelines” series covering Marketplace’s pipeline architecture |
| Vinay Kakade, Shiraz Zaman | Lyft Engineering | Co-authored the Feature Service architecture writeup |
| Sherin Thomas | Senior Software Engineer, Lyft | Presented Lyft’s Dryft framework and its Kubernetes-based Flink deployment model at QCon London 2020 |
| Anirudh Adkar, Ravi Magham | Streaming Compute team, Lyft | Credited with guidance throughout the 2026 operator migration |
Key takeaways for your own Flink implementation
These are the patterns from Lyft’s own account that I’d point any team running Flink at meaningful scale toward:
- Split pipelines the moment their scaling requirements diverge. Lyft’s first pricing pipeline worked fine in small markets and fell over in large ones because feature generation and model execution had different resource profiles bolted into one job. Splitting them, not adding more compute to the combined pipeline, was what fixed it.
- Idle-subtask logic in a source connector deserves the same scrutiny as your own business logic. The five-day data-loss incident happened entirely inside Flink’s Kinesis consumer internals, not in Lyft’s own pipeline code. Understanding a connector’s watermark and idleness handling before it fails under load beats debugging it after.
- A compatibility-translation layer can make an otherwise risky migration reversible. Translating legacy CRDs to the new operator’s schema at deploy time, rather than rewriting every application’s config up front, let Lyft migrate one app at a time with an instant rollback path.
- Autoscaling and autotuning solve different problems and can conflict. Lyft’s two-tier policy, favoring uninterrupted autoscaling for latency-sensitive jobs and allowing autotuning’s restarts everywhere else, is a more honest answer than picking one feature and disabling the other fleet-wide.
- Filing the bug upstream is cheaper than living with it. Lyft’s BlueGreen and Kinesis-consumer fixes both landed in the open-source project within days of being reported, work Lyft would otherwise have carried as a permanent patch on a private fork.
I’ll keep working through the rest of the list of companies running Flink in production.
Sources and further reading
Primary sources
- Maheep Myneni, Arda Kuyumcu, and Prem Santosh Udaya Shankar, “Rerouting the Stream: How Lyft Moved to the Apache Flink Operator” - https://eng.lyft.com/rerouting-the-stream-how-lyft-moved-to-the-apache-flink-operator-36f20246d250 (2026)
- Seth Saperstein, “Where’s My Data - A Unique Encounter with Flink Streaming’s Kinesis Connector” - https://eng.lyft.com/wheres-my-data-a-unique-encounter-with-flink-streaming-s-kinesis-connector-6da3b11b164a (2023)
- Rakesh Kumar, “Gotchas of Streaming Pipelines: Profiling & Performance Improvements” - https://eng.lyft.com/gotchas-of-streaming-pipelines-profiling-performance-improvements-301439f46412 (2023)
- Rakesh Kumar, “Evolution of Streaming Pipelines in Lyft’s Marketplace” - https://eng.lyft.com/evolution-of-streaming-pipelines-in-lyfts-marketplace-74295eaf1eba (2022)
- Vinay Kakade and Shiraz Zaman, “ML Feature Serving Infrastructure at Lyft” - https://eng.lyft.com/ml-feature-serving-infrastructure-at-lyft-d30bf2d3c32a (2021)
- Rakesh Kumar, “Gotchas of Stream Processing: Data Skewness” - https://eng.lyft.com/gotchas-of-stream-processing-data-skewness-cfba58eb45d4 (2021)
- Sherin Thomas, “Machine Learning through Streaming at Lyft,” QCon London, transcript via InfoQ - https://www.infoq.com/presentations/ml-streaming-lyft/ (2020)
- Lyft,
flinkk8soperator(official GitHub repository) - https://github.com/lyft/flinkk8soperator (created 2019)
Enterprise tooling for Apache Flink
If you are running Flink at anywhere close to the scale Lyft describes here, the operational picture in this piece, per-subtask backpressure, deploy health across hundreds of jobs, memory contention between the JVM and a Python SDK harness, is exactly the space my team and I build for at Factor House. It’s part of why I find write-ups like Lyft’s worth this much attention: Flex, our enterprise UI and API for Apache Flink, exists to make that picture manageable without building all of the tooling above in-house.
This architecture is one of 2 production Flink architectures covered in the Flink guide.