Before Factor House I spent eight years as a software engineer at Square, later Block, most of it on streaming and data infrastructure, including a production migration of our own analytics tables onto Apache Iceberg. This is the fourth in a short run of Iceberg write-ups I’ve done, after Netflix, Apple, and Airbnb, and Pinterest’s account is a good one to add to that list because it isn’t a single migration story: it’s two different platform teams, a database-CDC ingestion framework and a machine-learning feature pipeline, both leaning on the same Iceberg primitives for very different reasons. I read six primary sources for this piece directly rather than through a secondhand summary: four Pinterest Engineering Blog posts on Medium, the README of Pinterest’s own internal Iceberg fork on GitHub, and a LinkedIn profile used only to corroborate one contributor’s role.
The number that stood out most: Pinterest’s legacy database-ingestion pipeline reprocessed entire tables daily even though only about 5% of rows had actually changed, and took over 24 hours to do it. The rebuilt version, running on Iceberg, gets base-table latency down to 15 minutes to an hour. On the machine-learning side, a similar story plays out at a different scale: a feature backfill job that used to take 140 days end to end now takes 26. The underlying engineering problem connecting both is the same one most teams eventually hit once a table format has to serve both streaming writers and batch readers at once: how do you let CDC events, backfills, and schema changes land continuously without forcing every reader to either wait for a lock or read half-written data.
Company overview
Pinterest is a visual discovery platform, and its Logging Platform and ML Infrastructure teams both operate what one of Pinterest’s own engineering posts describes as “petabyte-scale data and thousands of pipelines.” Two largely separate efforts inside that platform ended up converging on Apache Iceberg: a rebuild of the change-data-capture pipeline that feeds Pinterest’s data warehouse from its operational MySQL and TiDB databases, and a rebuild of the backfill and feature-joining infrastructure that feeds Pinterest’s recommendation and ads models.
| Date | Milestone |
|---|---|
| 2025-05 | Pinterest publishes “How Pinterest Accelerates ML Feature Iterations via Effective Backfill,” describing Feature Backfiller v2 on Iceberg |
| 2025-06 | Pinterest publishes “Scaling Pinterest ML Infrastructure with Ray,” describing Iceberg-bucketed feature joins for Ray training jobs |
| 2026-02 | Pinterest publishes “Next Generation DB Ingestion at Pinterest,” describing a CDC-based ingestion framework writing into Iceberg CDC and base tables |
| 2026-05 | Pinterest’s internal pinterest/iceberg GitHub fork shows a pinterest-release-1.3.0 branch as its most recently updated release line |
| 2026-06 | Pinterest publishes “Automated Schema Evolution in Pinterest’s Next-Generation DB Ingestion Framework,” describing a three-phase schema convergence model for the same CDC platform |
Pinterest’s Apache Iceberg use cases
Next-generation DB ingestion (Logging Platform)
Pinterest’s Logging Platform team replaced a batch-oriented database-ingestion pipeline that dumped and reprocessed entire tables, even on days when only a small fraction of rows had changed, with a change-data-capture framework writing directly into Iceberg. As Liang Mou, Yisheng Zhou, Elizabeth (Vi) Nguyen, and Owen Zhang describe, the new pipeline captures changes from MySQL via Debezium and from TiDB via TiCDC, streams them through Kafka, and writes them into two kinds of Iceberg tables: an append-only CDC table holding every change event, and a deduplicated base table mirroring current state.
Automated schema evolution for the same CDC platform
Once the CDC framework was live, keeping Iceberg table schemas in sync with upstream MySQL and TiDB schema changes became its own problem. Yisheng Zhou, Liang Mou, Gabriel Raphael Garcia Montoya, and Istvan Podor describe a follow-on automation layer that propagates new nullable columns, numeric precision widening, and column renames through Kafka, Flink, Spark, and Iceberg’s own schema definitions without manual coordination.
ML feature backfill for recommendation and ads models (Feature Backfiller v2)
On the machine-learning side, Iceberg replaced Hive as the table format underneath the system that joins and materializes features into training datasets for Pinterest’s recommendation and ads models. Kartik Kapur, Matthew Jin, and Qingxian Lai describe Feature Backfiller v2, a two-stage backfill redesign built on Iceberg’s partitioning and snapshot model.
Dynamic feature joining for Ray training pipelines
The most recent piece layers a Ray-based training infrastructure on top of the same Iceberg tables. Andrew Yu, Jiahuan Liu, Qingxian Lai, and Kritarth Anand describe a custom Bucket Join Datasource that lets Ray training jobs merge features and labels from multiple Iceberg tables at runtime, and persist transformed features for reuse instead of recomputing them for every training iteration.
I’ve seen this same pattern from the other side of the fence: at Square, we kept discovering that the same partitioning and snapshot primitives that made streaming ingestion reliable turned out to be exactly what a completely unrelated ML or backfill workload needed too, once someone thought to check.
Scale and throughput
- CDC ingestion latency: events land in Kafka in under 1 second; the CDC Iceberg table reflects them within 5 minutes; the deduplicated base Iceberg table reaches 15 minutes to 1 hour, compared with the prior batch system’s 24+ hour latency.
- Reprocessing waste eliminated: the legacy batch pipeline reprocessed entire tables daily even though only about 5% of rows changed per day.
- Compute savings from bucket joins: the CDC ingestion pipeline’s bucket-join technique cut compute cost by more than 40%.
- Feature Backfiller v2 job size: individual backfill jobs shuffle up to 90 TiB of data.
- Feature Backfiller v2 storage: Iceberg-backed compaction and format changes reduced table sizes to as little as 25% of their original size.
- Feature Backfiller v2 speed: up to a 3x increase in overall backfill speed; dynamic partition insertion alone cut per-partition write time from 12 hours to 1 hour, a 12x improvement; the full two-stage redesign cut end-to-end backfill time from 140 days to 26 days, an 82% improvement.
- Ray feature-joining iteration time: feature-experimentation cycle time dropped from days to hours, roughly a 10x improvement.
- Pinterest’s own posts describe the underlying platform as operating at “petabyte-scale data and thousands of pipelines,” but none of the sources I read name a total Iceberg table count or an aggregate data volume specifically for Iceberg - I’m not including a company-wide figure here rather than borrowing one from an unrelated, pre-Iceberg description of Pinterest’s Presto and Hive fleet.
Pinterest’s Apache Iceberg architecture
Storage and table types
Both the CDC ingestion pipeline and the ML feature pipeline store their Iceberg tables on Amazon S3. The CDC framework uses two table types side by side: an append-only CDC table that acts as a time-series ledger of every change event, and a base table that holds a deduplicated, current-state mirror of the source database.
Merge-on-read over copy-on-write
Pinterest’s Logging Platform team standardized on merge-on-read rather than copy-on-write for these tables, citing copy-on-write’s higher storage cost as outweighing its benefits for a workload built around continuous upserts.
CDC ingestion path
Change events flow from MySQL (via Debezium) and TiDB (via TiCDC) into Kafka, where a Flink job parses them and writes into the Iceberg CDC tables in near real time. A separate Spark job runs periodic, watermark-based MERGE INTO operations that upsert from the CDC tables into the base Iceberg tables, alongside historical backfill.
Catalog
Pinterest’s own posts describe validating schemas against “the Iceberg catalog API” and comparing Iceberg table metadata directly, but neither post names a specific catalog implementation - I couldn’t confirm from the sourced material whether Pinterest runs a Hive Metastore, AWS Glue, or a REST catalog for these tables, so I’m flagging that as a real gap rather than guessing.
ML feature pipeline architecture
Feature Backfiller v2 uses Iceberg tables for both the initial full backfill, via dynamic partition insertion, and the incremental phase that follows it, with rollback implemented as a metadata-only operation through Iceberg snapshot cherry-picking. On top of that, the Ray-based feature-joining system adds four components: a file resolver that maps files across tables by bucket, a Bucket Join Datasource that issues one Ray read task per partition, three joiner strategies (a default two-step concatenation, a Pandas-based path, and a fully custom path), and a partition-mapping layer for tables whose partition-key names don’t line up.
Special techniques and engineering innovations
Bucketed partitioning for shuffle-free upserts
The base CDC-ingestion tables are partitioned with bucket(100, id), letting upserts run in parallel across 100 partitions. Where a merge would otherwise force a full shuffle of the much larger base table, Pinterest’s “bucket join” technique first creates a temporary bucketed copy of the CDC table, so the merge only has to shuffle data that’s already bucketed the same way. Writes also use WRITE DISTRIBUTED BY PARTITION specifically to hold down the small-file proliferation that frequent upserts would otherwise cause.
Additive-only schema evolution with stable column IDs
The automated schema-evolution framework only permits semantically safe, additive changes: new nullable columns, numeric precision widening (never narrowing), and column renaming, the last of which relies on Iceberg’s own stable numeric column identifiers rather than column names. Type conversions like a string becoming an integer, primary-key changes, and precision narrowing are all explicitly rejected rather than attempted.
A three-phase schema convergence model
Rolling out a schema change without breaking a running pipeline is sequenced into three phases: schema divergence, where the Iceberg schema updates immediately and existing jobs keep running, writing null into the new column; code convergence, where Spark’s code deploys first to make backfill possible, followed by Flink; and data convergence, where a backfill job completes and the table reaches full consistency inside an explicit SLA window.
Dual-path schema drift detection
Upstream schema drift is caught two ways at once: a push-based path where DDL change-data messages trigger an automatic diff against the Iceberg catalog, and a pull-based path where a daily comparison job acts as a safety net for anything the push path missed.
Bucketed, sorted, and delta-encoded ML feature tables
ML feature tables use bucketing, local bucket sorting, and delta encoding together, with patterns like bucket(request_id, 16), specifically to make storage-partitioned joins fast during training-time data loading.
Snapshot-based rollback for ML backfills
Feature Backfiller v2 recovers from bad data by cherry-picking data partitions from earlier Iceberg snapshots, a metadata-only operation that makes rollback fast regardless of how much underlying data a partition holds.
Dynamic partition insertion
Rather than looping through partitions one at a time within a backfill epoch, the redesigned pipeline writes multiple partitions in a single pass. Pinterest’s own post credits this specifically as the main driver of the 12x per-partition write-time improvement described above.
I’d put the bucket-join technique and dynamic partition insertion in the same category: both are ways of getting Iceberg’s partitioning to do the shuffle-avoidance work that would otherwise fall on a much more expensive full-table operation, just applied to two very different workloads.
Operating Apache Iceberg at scale
Schema-change governance
Schema changes go through a PR-based rollout process with versioning and an audit trail, backed by stable column identifiers that persist across revisions. Ambiguous CREATE TABLE diffs from self-serve MySQL changes are resolved at build time using Skeema, backstopped by a deployment-time audit trail checked against MySQL binlog DDL history for anything Skeema can’t disambiguate on its own.
CI/CD and sequential execution
Flink deployments go through CI/CD staging validation before reaching production, and the ingestion and schema-evolution workflows are deliberately restricted to one running at a time, specifically to avoid races between concurrent schema or data changes.
Monitoring
The schema-evolution framework tracks schema-evolution latency, Kafka-to-Flink lag, Spark watermark delay, and null-column counts, alongside general pipeline health metrics.
Recovery paths
Spark failures resume from the last recorded watermark rather than reprocessing from scratch. A full recovery path also exists through an onboarding system that keeps backup snapshots and can roll the Spark watermark back if needed.
Challenges and how they solved them
Batch ingestion latency and wasted reprocessing
Pinterest’s legacy database-ingestion pipeline was a batch system, and it showed in both directions: high latency and wasted work.
Problem: data latency exceeded 24 hours, and the pipeline reprocessed entire tables daily.
Root cause: the batch design had no way to identify or ingest only the rows that had actually changed, so it always processed the whole table, even though only about 5% of rows changed on a typical day.
Solution: a CDC-based pipeline (Debezium and TiCDC into Kafka, then Flink and Spark into Iceberg CDC and base tables) that only ever processes actual change events.
Outcome: base-table latency down to 15 minutes to 1 hour, per Pinterest’s Logging Platform team’s own account.
Expensive full-table shuffles during merges
Once ingestion moved to CDC, upserting changes into the much larger base table introduced its own cost problem.
Problem: merging CDC changes into base tables forced a full shuffle of the entire base table on every merge.
Root cause: a standard MERGE INTO has no way to know that only a bucketed subset of the base table actually needs touching.
Solution: the bucket-join technique creates a temporary bucketed copy of the CDC table so the merge only shuffles data already aligned to the base table’s own bucketing.
Outcome: more than a 40% reduction in compute cost for the ingestion pipeline.
Coordinating schema changes across four systems at once
A schema change to a source database has to propagate through Kafka, Flink, Spark, and Iceberg without any of them falling out of sync.
Problem: manual, uncoordinated schema updates across those four systems risked drift, with some components applying a change before others were ready for it.
Root cause: no single framework owned the sequencing of a schema change across ingestion, transport, processing, and storage.
Solution: an automated schema-evolution framework reusing the same code-generation logic for both initial onboarding and later evolution, sequenced through the three-phase schema convergence model described above.
Outcome: schema changes now roll out with an explicit, SLA-bounded consistency window instead of an unmanaged race between components, per Pinterest’s Logging Platform team.
Ambiguous, self-serve MySQL schema diffs
Letting engineers make their own MySQL schema changes without a centralized data team introduced a new class of ambiguity.
Problem: self-serve CREATE TABLE diffs could be interpreted more than one way, risking an incorrect schema change reaching Iceberg.
Root cause: MySQL’s own DDL doesn’t always map unambiguously onto the additive-only, semantically-safe change set the Iceberg schema-evolution framework permits.
Solution: Skeema resolves the diff at build time, and any case Skeema can’t resolve is checked at deployment time against the MySQL binlog’s own DDL history.
Outcome: ambiguous schema changes are caught before they reach a production Iceberg table, rather than after.
Slow, expensive ML feature backfills
Before Feature Backfiller v2, Pinterest’s ML feature pipeline was a serious bottleneck on how fast new features could be tested.
Problem: end-to-end feature backfills took up to 140 days, and per-partition writes alone took up to 12 hours.
Root cause: the original backfill design wrote one partition at a time within each data epoch instead of writing multiple partitions in a single pass.
Solution: a two-stage backfill redesign combined with dynamic partition insertion into Iceberg tables.
Outcome: end-to-end backfill time down to 26 days (an 82% improvement) and per-partition write time down to 1 hour (a 12x improvement), per Kartik Kapur, Matthew Jin, and Qingxian Lai’s account.
Recomputing features on every training iteration
Even after backfills got faster, Pinterest’s ML training pipelines were still redoing feature transformations that hadn’t actually changed.
Problem: recomputing the same feature transformations for every training iteration was expensive and slowed down experimentation.
Root cause: there was no shared, reusable store of already-transformed features that a new training run could read from directly.
Solution: a Bucket Join Datasource lets Ray training jobs read and cache transformed features straight from Iceberg-bucketed tables, reusing them across training iterations and production systems instead of recomputing them each time.
Outcome: feature-experimentation cycle time dropped from days to hours, roughly a 10x improvement, per Andrew Yu, Jiahuan Liu, Qingxian Lai, and Kritarth Anand.
Full tech stack
| Category | Tools | Notes |
|---|---|---|
| Table format | Apache Iceberg | CDC, base, and ML feature tables; merge-on-read, schema evolution, snapshot rollback, and bucketed partitioning are the specific features exercised |
| Object storage | Amazon S3 | Underlying storage for all Iceberg tables described |
| CDC capture | Debezium | Captures change events from source MySQL databases |
| CDC capture | TiCDC | Captures change events from source TiDB databases |
| Streaming transport | Apache Kafka | Carries CDC change events between capture and stream processing |
| Stream processing | Apache Flink | Parses and writes CDC records into Iceberg CDC tables in near real time; generates transformation code per schema version during schema evolution |
| Batch processing | Apache Spark | Runs periodic watermark-based MERGE INTO upserts into base Iceberg tables, historical backfill, and the Feature Backfiller v2 pipeline |
| Distributed ML training | Ray | Reads Iceberg-bucketed tables through a custom Bucket Join Datasource to join features and labels at training time |
| Schema tooling | Skeema | Resolves ambiguous MySQL CREATE TABLE diffs as part of the schema-evolution safety controls |
| Source database | MySQL | Operational database captured via Debezium |
| Source database | TiDB | Operational database captured via TiCDC |
Key contributors
| Name | Role | Contribution |
|---|---|---|
| Liang Mou | Staff / Sr Staff Software Engineer, Logging Platform | Co-authored both the DB ingestion and schema-evolution posts |
| Yisheng Zhou | Software Engineer II, Logging Platform | Co-authored both the DB ingestion and schema-evolution posts |
| Elizabeth (Vi) Nguyen | Software Engineer I, Logging Platform | Co-authored “Next Generation DB Ingestion at Pinterest” |
| Owen Zhang | Senior Software Engineer, Logging Platform | Co-authored “Next Generation DB Ingestion at Pinterest” |
| Gabriel Raphael Garcia Montoya | Staff Software Engineer | Co-authored the automated schema-evolution post |
| Istvan Podor | Staff Software Engineer | Co-authored the automated schema-evolution post |
| Kartik Kapur | Tech Lead, Sr Software Engineer | Co-authored the ML feature backfill post |
| Matthew Jin | Sr Software Engineer | Co-authored the ML feature backfill post |
| Qingxian Lai | Staff Software Engineer, AI Platform | Co-authored both the ML feature backfill and Ray posts |
| Andrew Yu | Staff Software Engineer | Co-authored the Ray infrastructure post |
| Jiahuan Liu | Staff Software Engineer | Co-authored the Ray infrastructure post |
| Kritarth Anand | Staff Software Engineer | Co-authored the Ray infrastructure post |
Key takeaways for your own Apache Iceberg implementation
These are the patterns from Pinterest’s own account that I’d point any team building CDC or ML feature pipelines on Iceberg toward:
- Match your partitioning to your merge pattern before you optimize anything else. Pinterest’s
bucket(100, id)partitioning on base tables is what makes both parallel upserts and the bucket-join technique possible in the first place - the partitioning decision comes before the shuffle-avoidance trick, not after. - Restrict schema evolution to what’s actually safe, not what’s technically possible. Allowing only additive, non-narrowing changes and rejecting type conversions outright is a deliberate constraint, not a limitation Pinterest’s engineers were forced into - it’s what makes the automated rollout safe to run without a human in the loop.
- Sequence a schema rollout as three separate phases, not one atomic switch. Pinterest’s schema-divergence, code-convergence, data-convergence sequence is worth copying directly if you’re propagating schema changes across more than two systems at once.
- Snapshot-based rollback is worth designing for even outside disaster recovery. Cherry-picking older snapshots gave Feature Backfiller v2 a metadata-only recovery path for routine bad-data cleanup, not just catastrophic failure.
- Cache transformed features, don’t just cache raw data. The Ray bucket-join system’s biggest iteration-speed win came from persisting already-transformed features for reuse, which is a different (and in this case bigger) win than simply caching source rows.
I’ll keep working through the rest of the list and write up the next one as it’s ready.
Sources and further reading
Primary sources
- Liang Mou, Yisheng Zhou, Elizabeth (Vi) Nguyen, Owen Zhang, “Next Generation DB Ingestion at Pinterest” - https://medium.com/pinterest-engineering/next-generation-db-ingestion-at-pinterest-66844b7153b7 (2026)
- Yisheng Zhou, Liang Mou, Gabriel Raphael Garcia Montoya, Istvan Podor, “Automated Schema Evolution in Pinterest’s Next-Generation DB Ingestion Framework” - https://medium.com/pinterest-engineering/automated-schema-evolution-in-pinterests-next-generation-db-ingestion-framework-36c5c07070de (2026)
- Kartik Kapur, Matthew Jin, Qingxian Lai, “How Pinterest Accelerates ML Feature Iterations via Effective Backfill” - https://medium.com/pinterest-engineering/how-pinterest-accelerates-ml-feature-iterations-via-effective-backfill-d67ea125519c (2025)
- Andrew Yu, Jiahuan Liu, Qingxian Lai, Kritarth Anand, “Scaling Pinterest ML Infrastructure with Ray: From Training to End-to-End ML Pipelines” - https://medium.com/pinterest-engineering/scaling-pinterest-ml-infrastructure-with-ray-from-training-to-end-to-end-ml-pipelines-4038b9e837a0 (2025)
- Pinterest,
pinterest/icebergGitHub repository (fork ofapache/iceberg) - https://github.com/pinterest/iceberg (accessed 2026)
I read every one of these directly, and linked the specific claim to its source the first time it comes up above, rather than leaving attribution to this list alone.
Enterprise tooling for Apache Iceberg
Pinterest built its own schema-convergence automation, its own bucket-join workaround, and its own Ray integration layer largely from scratch, which is exactly the kind of engineering investment most teams adopting Iceberg can’t make themselves. That gap is what my team and I are building Iglu for at Factor House: enterprise tooling for Apache Iceberg. Iglu is currently in preview with select customers - if the operational problems Pinterest describes here sound familiar, request preview access and tell us what you’re running into.