Skip to content
Migrating to open source Kafka.
Sep 9, 1pm SGT. Register

How Pinterest uses Apache Flink in production

Flink
Derek Troy-West·September 5, 2026·22 min read

I spent close to two decades building distributed systems and streaming platforms before co-founding Factor House in 2019, including several years advising banks on Kafka and Cassandra deployments, and the thing I keep coming back to in this reading project is how differently a streaming platform looks once real production traffic is running through it compared to how it looks on a slide. After Netflix’s fleet of 30,000-plus Apache Flink jobs and Uber’s decade-long build-out of its own Flink SQL platform, Pinterest is the third stop: eighteen sourced pieces, almost all Pinterest’s own engineering blog spanning 2019 to 2026, plus two conference talks and one official GitHub repository, all read directly rather than taken from a secondhand summary.

What makes Pinterest’s account worth reading is how specific it stays even as the scale grows. Its engineers do not just say a job got faster. They say a job’s cross-host network traffic fell 60% after pinning subtasks to the same host as their downstream operators, or that a four-month, 30-team cluster retuning effort cut platform cost 40% while onboarding 40% more jobs with zero incidents. Flink now sits underneath image-similarity detection, real-time experiment analysis, ad-budget enforcement, and a next-generation change-data-capture pipeline into Apache Iceberg, running across 130-plus jobs and eight multitenant YARN clusters that together process more than 20 million messages a second.

The engineering problem underneath most of it is a familiar one from my own time running these systems, just applied to a company whose core product is inherently visual and combinatorial: how do you take a signal, whether it is a duplicate image, an experiment metric, or a database row change, and get it from the point it is created to the point it needs to be acted on in seconds rather than hours, without a human hand-tuning every pipeline that carries it.

Company overview

Pinterest is a visual discovery platform where hundreds of millions of people save, search, and shop across a corpus that had grown to 300 billion saved pins by 2021, according to its own Content Quality team. At that scale, connecting a person to relevant content, and keeping unsafe or duplicated content out of the system, depends on understanding what is happening across the platform close to the moment it happens.

Pinterest’s first production Flink application went live in October 2019: a real-time experiment analytics pipeline built by the Data Engineering team, described in Parag Kesar and Ben Liu’s own post as “the first Flink-based application in production at Pinterest.” It replaced a daily batch pipeline that could take more than 10 hours to surface a metric regression in a live experiment, with a streaming pipeline that could catch the same regression within minutes. Everything Pinterest has built on Flink since has grown out of that first deployment.

Date Milestone
2019-10 Pinterest’s first Flink-based application in production: real-time experiment analytics for the Data Engineering team
2019-10 Pinterest engineers speak at Flink Forward Berlin 2019, the company’s debut at the conference
2020-11 Visual Signals Infrastructure moves from a Lambda to a Kappa architecture as Near-real-time Galaxy (NRTG), built on Flink and Kafka
2021-03 Pinterest publishes its Flink Deployment Framework, built on Hermez and the Job Submission Service
2021-07 UnifiedSource ships, stitching live Kafka data and S3-backed historical replay into a single Flink source
2022-08 Rainie Li and Kanchi Masalia present Pinterest’s Flink platform at Flink Forward San Francisco: 300 million messages a second, 100-plus applications
2023-07 Pinterest publishes its cluster-tuning results: 40% cost reduction after a four-month, 30-team rollout with zero incidents
2023-11 Pinterest’s unified PubSub client, PSC, reaches broad production maturity across the Flink fleet
2024 Pinterest presents its PubSub platform automation at Confluent Current: roughly 2,000 Kafka brokers across 50-plus clusters
2026-02 Pinterest publishes its next-generation CDC database ingestion framework, built on Flink and Apache Iceberg

Near-real-time image similarity - fewer duplicates, faster moderation

Pinterest’s Content Quality team uses Flink to detect near-duplicate images shortly after they are created, feeding both recommendation quality and content moderation. As Shaji Chennan Kunnummel and Iaroslav Tymchenko describe, the pipeline uses a stream-stream join to synchronize the arrival of separate embedding streams before computing similarity, replacing an offline Spark, LSH, and TensorFlow batch pipeline that had taken hours to surface a match. The Flink pipeline calls Pinterest’s Scorpion model-serving layer at up to 500,000 instances a second at peak and writes cluster relationships into Zen, Pinterest’s graph store.

Explainability mattered as much as speed here. A system deciding to take down content for looking too similar to something else needs a traceable reason, not just a black-box score, which is exactly the kind of constraint I saw over and over advising banks on stream processing: speed is worthless if you cannot show your working when someone asks why a decision was made.

Real-time experiment analytics - catching regressions in minutes, not hours

Pinterest’s Data Engineering team built the company’s first production Flink application to catch statistically significant metric drops or gains across 200 to 300 active experiment groups, replacing a daily batch pipeline with 10-plus hour latency. Per Kesar and Liu’s account, the pipeline uses Flink’s Broadcast State pattern to filter a live event stream against a set of recently ramped experiment activations refreshed every 10 seconds, runs on 8 masters and 50 EC2 c5d.9xlarge workers at a parallelism of 256, and applies a t-test with Bonferroni correction to keep results statistically sound under repeated testing.

Ads budget enforcement - real-time spend against a live limit

Pinterest’s ads infrastructure runs on the same Xenon Flink platform to calculate ad spend against budget limits and produce advertiser reporting in real time, part of the flagship set of use cases Rainie Li and Kanchi Masalia listed at Flink Forward San Francisco in 2022. The Ads Data Infrastructure team’s pipeline for this also surfaced one of the more instructive production incidents in Pinterest’s Flink history, covered below under challenges.

Near-real-time Galaxy - one streaming path instead of two

Pinterest’s visual signal generation, covering roughly 50 distinct ingestion pipelines for user uploads, crawled images, shopping content, and video keyframes, originally ran as a Lambda architecture: a nightly Hadoop batch job alongside a separate real-time path through PinLater. As Ankit Patel describes, Pinterest replaced both with Near-real-time Galaxy (NRTG), a single Kappa-style streaming path built on Flink and Kafka, where developers write Flink jobs against NRTG’s own design patterns and annotations while the framework hides most of the underlying infrastructure.

Large-scale user sequences - real-time signal for HomeFeed

Pinterest’s User Understanding and Index Core Entity teams built a Flink-based pipeline to index user action sequences and enrich them in real time, feeding HomeFeed recommendation and sequential modeling. As Zefan Fu and co-authors describe, the system targets sub-2-second latency from a user’s action to a service response being able to use it, backed by RocksDB as the underlying key-value store.

Next-generation database ingestion - CDC into Iceberg

Pinterest’s Logging Platform team built a Flink-based pipeline to replace a legacy database ingestion system that carried more than 24 hours of latency and lacked row-level delete support. As Liang Mou and co-authors describe, Debezium and TiCDC capture changes from MySQL, KVStore, and TiDB into Kafka, Flink parses and transforms those change events into CDC Iceberg tables on S3 near-real-time, and Spark periodically upserts them into base Iceberg tables using a bucketed merge that avoids full shuffles.

Scale and throughput

Year Metric Source
2019 200-300 active experiment groups; parallelism of 256 across 8 masters and 50 workers; roughly 100 GB checkpoint size Parag Kesar, Ben Liu, Pinterest Engineering
2021 300 billion pins saved; roughly 100 images created per second (200 at peak); ML-serving layer handles 500,000 instances/second at peak Shaji Chennan Kunnummel, Iaroslav Tymchenko, Pinterest Engineering
2022 300 million messages/second; 150 TB of data per second; 100-plus Flink applications; 179 deployments a day Rainie Li, Kanchi Masalia, Flink Forward San Francisco
2023 130-plus Flink jobs across 8 multitenant YARN clusters; more than 20 million messages/second processed platform-wide; largest single job uses 2,000-plus cores Divye Kapoor et al., Pinterest Engineering
2023 More than 80% reduction in Flink application restarts caused by remediable client exceptions after adopting PSC Jeff Xiang, Vahid Hashemian, Jesus Zuniga, Pinterest Engineering
2024 Roughly 2,000 Kafka brokers across 50-plus clusters feeding the platform Jeff Xiang, Vahid Hashemian, Confluent Current
2026 CDC-to-Kafka latency typically under 1 second; end-to-end online-to-offline latency cut from 24-plus hours to 15 minutes-1 hour Liang Mou et al., Pinterest Engineering

The cluster-tuning work Pinterest published in 2023 also reset the platform’s cost baseline: individual jobs became 50-90% cheaper to run after a combination of colocation constraints and CPU-isolation changes, on top of a 10% further saving from an EC2 i3-to-i4i hardware upgrade and 5% from input-stream optimization.

Xenon - Flink on multitenant YARN

Xenon is Pinterest’s Flink-based stream-processing platform, running on multitenant production YARN clusters on AWS EC2. By 2023 it hosted 130-plus jobs across 8 clusters, a scale that shaped nearly every operational decision covered below.

UnifiedSource - one log spanning live and historical data

Lu Niu and Chen Qin describe UnifiedSource, a custom Flink source extending RichParallelSourceFunction that runs a FlinkKafkaConsumer and a MercedFileSource within each subtask, aligning partitions so the two are treated as one continuous, seekable log spanning live Kafka data and S3-backed historical data persisted by Merced. This gives Flink jobs at Pinterest replay and backfill capability that Kafka’s own retention window cannot provide on its own.

Near-real-time Galaxy’s Kappa architecture

As Ankit Patel describes, NRTG sits between Flink and the rest of Pinterest’s visual signal pipelines as a Kappa-style middleware layer, replacing the earlier Lambda split between a nightly Hadoop job and a separate real-time PinLater path with a single streaming path that developers extend through NRTG’s own design patterns rather than building infrastructure from scratch each time.

PSC - one client over Kafka and MemQ

As Jeff Xiang, Vahid Hashemian, and Jesus Zuniga describe, Pinterest’s Logging Platform team built PSC (PubSub Client), a unified client abstraction that sits between applications and two backends: native Kafka, and MemQ, Pinterest’s own S3-backed cloud-native PubSub system designed by Ambud Sharma. PSC ships FlinkPscProducer and FlinkPscConsumer connectors and identifies resources through a “Resource Name” scheme built for automated service discovery, and is open-sourced on Pinterest’s GitHub.

CDC to Iceberg - the next-generation ingestion path

As Liang Mou and co-authors describe, the database ingestion pipeline chains Debezium and TiCDC (capturing changes from MySQL, KVStore, and TiDB) into Kafka, then Flink (parsing and transforming those events into CDC Iceberg tables on S3), then Spark (periodically upserting into primary-key-hash bucketed base Iceberg tables through a bucketed merge that avoids full shuffles), with a separate bootstrap pipeline for loading historical dumps.

Special techniques and engineering innovations

Stream-stream joins for embedding synchronization

As Shaji Chennan Kunnummel and Iaroslav Tymchenko describe, the image-similarity pipeline uses a stream-stream join to synchronize the arrival of multiple embedding streams before triggering a similarity computation, rather than assuming both streams arrive in lockstep.

Broadcast State and IntervalJoin for dynamic experiment filtering

Pinterest’s experiment-analytics job combines Flink’s Broadcast State pattern, refreshed every 10 seconds against recently ramped experiments, with an IntervalJoin buffering activation records over a fixed 3-day window against the event stream by user ID, followed by 15-minute tumbling windows for aggregation. Kesar and Liu flag the fixed interval as a known limitation and describe a planned migration to Flink’s CoProcessFunction for more dynamic windowing.

Colocation constraints for cost and network savings

Divye Kapoor and co-authors describe adopting Flink 1.11’s colocation constraints, pinning the i-th subtask of an operator to the same host as the i-th subtask of its downstream operator. That cut cross-host network traffic 60% and CPU needs by more than half, which in turn let Pinterest reduce job parallelism and cut per-job cost 50-90%.

Guaranteed and burst CPU capacity through CGroups

Rather than reserving 3x a job’s expected traffic as headroom, a policy that had left clusters running at roughly 33% utilization under normal conditions, Pinterest moved to CGroups soft CPU limits: each host guarantees 24 of 32 cores (75%) to steady-state job traffic, with the remaining 25% shared as burst capacity across all tenants. That change alone enabled a 20% cluster-size reduction.

I’ve watched more Flink estates than I can count get this exact tradeoff wrong in one direction or the other, either over-provisioning for safety until nobody can justify the bill, or under-provisioning until one noisy job takes the cluster down. Pinterest’s soft-limit approach, with a real number attached to the burst allocation, is a more honest way to make that tradeoff than most teams manage.

As Jeff Xiang, Vahid Hashemian, and Jesus Zuniga describe, PSC’s FlinkPscProducer and FlinkPscConsumer connectors include checkpoint-migration logic so a running Flink job’s underlying client library could be swapped to PSC without losing offset or state continuity, letting Pinterest migrate more than 90% of its Java applications, and 100% of its Flink applications, without a coordinated stop-the-world cutover.

Self-service diagnosis with DrSquirrel

Fanshu Jiang and Lu Niu built DrSquirrel after observing that “90% of the stats we spend time on are either benign or simply unrelated to the root cause” and that “80% of the issue-fixing follows a pattern.” A custom Flink build emits metrics and logs to Kafka, a dedicated FlinkJobWatcher Flink job joins them into 5-minute job-health snapshots, and a Dropwizard service enriches that with YARN ResourceManager and Flink REST API data, surfaced through a React frontend with an Elasticsearch-backed log timeline, exception grouping with linked runbooks, and per-operator backpressure and GC visualizations.

Deployment governance through Hermez and JSS

Rainie Li describes a deployment pipeline where Bazel builds trigger through Jenkins CI, JARs land in S3 and YAML configs in Artifactory, and Hermez, Pinterest’s internal CD system, launches jobs through the Job Submission Service, which schedules to YARN and deduplicates by job name to prevent an accidental double-write to Kafka. Standardized launch, kill, savepoint, and resume operations, plus hotfix-style config updates without a full rebuild, sit on top of that pipeline.

A four-month, cross-team cluster retuning

Pinterest’s 2023 cluster-tuning rollout was run as a cross-team effort: retuned jobs over four months in partnership with more than 30 client teams and a 200-plus member client community, reporting zero incidents through the rollout and into the first half of 2023.

Schema-change governance across four systems

Yisheng Zhou and co-authors describe a three-phase rollout for schema changes across the CDC pipeline, moving from schema divergence to code convergence to data convergence, restricted to additive-only changes such as new nullable columns or safe numeric widening, with stable numeric column identifiers. Every change flows through PR-based review, detected both by a DDL-triggered workflow and a daily diff job against the Iceberg catalog as a safety net, with observability spanning Kafka traffic, Flink and Spark job health, evolution latency, and null-column counts.

Coordinating a schema change across a CDC source, a stream processor, a batch engine, and a table format without anyone’s job silently breaking is one of the genuinely hard problems in this space, and it’s one most teams underestimate until the day a well-meaning column rename takes down three downstream consumers at once.

Automated remediation through PSC

As Jeff Xiang, Vahid Hashemian, and Jesus Zuniga report, PSC’s in-client handling of known Kafka and MemQ exceptions lets Flink jobs recover from remediable errors on their own rather than waiting for a human to restart them, a change Pinterest credits with an 80%-plus reduction in Flink application restarts and roughly 275 FTE-hours saved a year.

Challenges and how they solved them

A single runaway user stalled checkpoints

Pinterest’s earliest production Flink job, real-time experiment analytics, hit a checkpoint problem within its first year in production.

Problem: Checkpoints began failing intermittently under severe backpressure.

Root cause: A single abnormal user was producing roughly 100 times more messages than its peers, stalling the flow of checkpoint barriers through the job.

Solution: Pinterest capped per-user message throughput in the upstream filter job.

Outcome: Stable checkpointing restored, as described by Parag Kesar and Ben Liu, an early lesson in treating traffic-shape assumptions as something to defend against rather than take for granted.

Kafka’s short retention blocked replay and backfill

As Pinterest’s Flink fleet grew, the gap between Kafka’s retention window and the need to replay or backfill historical data became a recurring blocker.

Problem: Kafka’s retention of hours to a few days meant jobs could not replay or backfill data beyond that window, and naive load testing by rewinding Kafka offsets had caused a 20x spike in infrastructure cost.

Root cause: Kafka was being asked to double as both a live event bus and a long-term historical store, a role it was never built for.

Solution: Pinterest built UnifiedSource, stitching S3-backed historical data persisted through Merced onto the live Kafka tail as one continuous log, and switched replay traffic to read from Merced directly.

Outcome: Replay from Merced ran 6x faster than replay from Kafka, per Lu Niu and Chen Qin, without the cost blowup of the earlier approach.

Skewed-volume joins stalled watermarks

UnifiedSource solved the replay problem but exposed a second one once jobs began joining topics of very different sizes.

Problem: Joining a high-volume topic against a much smaller one caused watermark progression to stall and triggered backpressure failures.

Root cause: Subtasks reading the high-volume topic raced ahead of subtasks reading the low-volume topic, so the job’s overall watermark could not advance correctly.

Solution: Pinterest added topic-level rate limiting alongside an allreduce-based, GlobalAggregateManager-driven mechanism to synchronize watermarks across subtasks.

Outcome: Stable watermark progression across skewed joins, part of the same UnifiedSource effort that solved the replay problem.

Multitenancy created noisy neighbors and an unsustainable burst policy

By 2023, Pinterest’s multitenant clusters were showing the strain of years of ad hoc capacity decisions.

Problem: Fully unisolated containers let one job’s CPU usage degrade another’s performance, undersized vCore reservations created unthrottled hot nodes, and a policy of reserving 3x expected traffic per job left clusters running at roughly 33% utilization under normal conditions.

Root cause: Capacity planning had been done per job rather than per cluster, with no shared mechanism for isolating tenants or reclaiming unused headroom.

Solution: Pinterest introduced CGroups soft CPU limits, a 75%/25% guaranteed-to-burst core split per host, and Flink 1.11 colocation constraints, rolled out across more than 30 client teams over four months.

Outcome: 40% lower platform cost, 40% more onboarded jobs, and zero incidents through the rollout, published by Divye Kapoor and co-authors in July 2023.

Pinterest’s Ads Data Infrastructure team traced a slow, hard-to-reproduce memory problem in a production Flink pipeline back to a specific library behavior.

Problem: Multiple operators in an ads-reporting Flink pipeline were throwing direct out-of-memory errors, visible as a distinctive “staircase” pattern of steadily growing memory use.

Root cause: A ChronicleMap-based off-heap cache did not free its backing memory when the map reference was dropped, because the cached objects lived outside the normal Flink task lifecycle across task restarts.

Solution: Sanchay Javeria added explicit cleanup logic to the operator’s close() method to release the off-heap memory when a task actually ended.

Outcome: The leak stopped, and Pinterest published the debugging process in September 2023 as a reference for diagnosing off-heap memory issues that standard JVM heap metrics do not show.

Legacy database ingestion could not keep up

Pinterest’s older database ingestion system had fallen well behind what analytics, ML, and product teams needed from fresh data.

Problem: The legacy system carried more than 24 hours of end-to-end latency, re-copied entire tables even when daily change rates were often under 5%, and had no row-level delete support.

Root cause: The system was built around periodic full-table snapshots rather than incremental change capture.

Solution: Pinterest built a CDC pipeline through Debezium and TiCDC into Kafka, Flink, and Iceberg, using primary-key-hash bucketing and a shuffle-avoiding bucket join for the Spark upsert stage, governed by the three-phase schema-evolution rollout described above.

Outcome: End-to-end latency fell to 15 minutes to 1 hour, reported by Liang Mou and co-authors in February 2026, alongside a compute-cost reduction of more than 40%.

Full tech stack

Category Tools Notes
Stream processing Apache Flink Core engine for image similarity, experiment analytics, ads budgeting, NRTG, user sequences, and CDC ingestion; Flink 1.11 specifically for colocation constraints
Messaging Apache Kafka Primary event-transport layer feeding nearly every Flink job
Messaging MemQ Pinterest’s S3-backed PubSub system for ML training data and faster Flink replay
Client library PSC (PubSub Client) Unified client over Kafka and MemQ, with dedicated Flink producer/consumer connectors
Storage/replay Merced Kafka-to-S3 exactly-once persistence layer underlying UnifiedSource
Logging agent Singer At-least-once event delivery into Kafka
State backend RocksDB Flink state backend with incremental checkpointing; also Pinterest’s key-value store for signal and user-sequence data
Resource manager YARN on AWS EC2 (i3, later i4i) Hosts Pinterest’s multitenant Flink clusters
Graph storage Zen Stores image-cluster relationships from the similarity pipeline
ML serving Scorpion Model-serving layer invoked from the image-similarity Flink job
Search Manas LSH-term candidate lookups for image similarity
Build and deploy Bazel, Jenkins, S3, Artifactory, Job Submission Service, Hermez Flink build and deployment toolchain
Diagnostics DrSquirrel Self-service Flink job health and root-cause diagnosis tool
Signal platform Galaxy / NRTG Kappa-architecture visual signal generation framework
Key-value store Rockstore Legacy Lambda-path storage and Galaxy Signal Service RPC layer
CDC source Debezium, TiCDC Change-data-capture from MySQL, KVStore, and TiDB into Kafka
Batch processing Apache Spark Periodic upserts into base Iceberg tables; legacy batch layer being replaced
Table format Apache Iceberg CDC and base table storage on S3
Resource isolation CGroups Soft CPU limiting on multitenant YARN hosts

Key contributors

Name Role Contribution
Shaji Chennan Kunnummel Software Engineer, Content Quality Co-wrote the near-duplicate image-detection Flink pipeline post
Iaroslav Tymchenko Software Engineer, Content Quality Co-wrote the near-duplicate image-detection Flink pipeline post
Parag Kesar Software Engineer, Data Engineering Co-built and co-wrote Pinterest’s first production Flink application
Ben Liu Software Engineer, Data Engineering Co-built and co-wrote Pinterest’s first production Flink application
Rainie Li Software Engineer / Senior Engineering Manager, Stream Processing Platform Wrote the Flink Deployment Framework post; co-presented Pinterest’s Flink platform at Flink Forward San Francisco 2022
Kanchi Masalia Software Engineer, Stream Processing Platform Co-presented Pinterest’s Flink platform at Flink Forward San Francisco 2022
Lu Niu Software Engineer, Stream Processing Platform Co-built DrSquirrel and UnifiedSource; spoke at Flink Forward Global 2021
Chen Qin Software Engineer, Stream Processing Platform Co-built UnifiedSource
Fanshu Jiang Software Engineer, Stream Processing Platform Co-built DrSquirrel
Divye Kapoor Senior Staff Software Engineer Founder and tech lead of NRTG; contributor on the cluster-tuning post
Ankit Patel Software Engineer, Content Acquisition and Media Platform Wrote the Lambda-to-Kappa NRTG migration post
Zefan Fu Software Engineer, User Understanding Co-built the large-scale user-sequences pipeline
Jeff Xiang Software Engineer, Logging Platform Co-built and co-wrote PSC; presented at Confluent Current 2024; maintains the PSC GitHub repo
Vahid Hashemian Software Engineer, Logging Platform Co-built and co-wrote PSC; presented at Confluent Current 2024; maintains the PSC GitHub repo
Jesus Zuniga Software Engineer, Logging Platform Co-built and co-wrote PSC; maintains the PSC GitHub repo
Ambud Sharma Tech Lead and Engineering Manager, Logging Platform Designed MemQ and wrote up its architecture
Liang Mou Staff / Senior Staff Software Engineer, Logging Platform Co-built the next-generation CDC/Iceberg ingestion framework
Yisheng Zhou Software Engineer II, Logging Platform Co-built the CDC framework’s schema-evolution system
Sanchay Javeria Software Engineer, Ads Data Infrastructure Diagnosed and wrote up the direct-memory-leak incident

These are the patterns from Pinterest’s own account that I’d point any team running Flink at real production scale toward:

  • Build a replay path before you need one. Pinterest’s UnifiedSource exists because Kafka’s retention window could not support backfill or replay, and a naive workaround (rewinding Kafka offsets) had already caused a 20x cost spike before the real fix shipped. If your jobs will ever need to reprocess history, design the storage tier for that on purpose rather than discovering the gap under pressure.
  • Treat multitenant capacity planning as its own discipline, not a side effect of individual job tuning. Pinterest’s noisy-neighbor and burst-capacity problems came from capacity decisions made per job rather than per cluster; the fix, CGroups soft limits plus colocation constraints plus a defined guaranteed-to-burst ratio, only worked because it was rolled out cluster-wide.
  • Cap the traffic shapes you cannot control. Pinterest’s earliest Flink job learned this the hard way when one abnormal user’s traffic stalled checkpoints for everyone else; a simple per-user throughput cap fixed what looked at first like a Flink-internals problem.
  • Off-heap memory needs its own debugging playbook. Standard JVM heap metrics did not show Pinterest’s ChronicleMap leak; if you use off-heap caches inside a Flink operator, plan for how you would diagnose a leak there specifically, not just in managed heap.
  • Schema evolution across a CDC-to-table-format pipeline needs its own governance, not an afterthought. Pinterest’s three-phase, additive-only rollout with dual push and pull detection exists because a schema change touching four different systems at once is exactly where things quietly break.

I’ll keep working through the rest of the list and write up the next one as it’s ready.

Sources and further reading

Primary sources

I read every one of these directly rather than working from a summary, and I’ve linked the specific claim to its source throughout the piece above, not just in this list.

If Pinterest’s multitenant capacity planning, self-service diagnosis tooling, and cross-system schema governance sound like the parts of running Flink that never quite finish, that operational gap is exactly the space my team and I build for at Factor House. Flex, our enterprise UI and API for Apache Flink, exists to give teams that same job-level and cluster-level visibility Pinterest had to build in-house, without building all of it yourselves first.

Keep reading

Guides

Apache Iceberg: the complete guide

Apache Iceberg is an open table format bringing ACID transactions, schema evolution, and time travel to huge analytic tables in S3. This hub indexes what we've written about running it in production.

September 6, 2026·2 min read
Iceberg

Iceberg use cases

Companies running Apache Iceberg in production, the architectures behind their deployments, and what to read next. Indexed as we publish new use-case research.

September 6, 2026·2 min read