I spent close to nine years as a software engineer at Square, later Block, before joining Factor House, and one of the things I carried with me from that time is a production migration off a proprietary warehouse onto Apache Iceberg, streaming Cassandra-style data into it the same way you’d stream Kafka into a lakehouse today. That background is exactly why Netflix’s own account of its Iceberg usage caught my attention: Netflix’s big data platform team built Iceberg in the first place, largely because Hive tables kept breaking down against the realities of S3. I read ten primary sources for this piece: seven Netflix TechBlog posts, two conference talks (including the one where Iceberg was first introduced publicly), and the README of a Netflix-published migration tool on GitHub, rather than relying on anyone else’s summary of them. This is the first Iceberg use-case piece I’ve written for this site, so I won’t pretend it’s part of an existing series - it’s simply the first one worth putting in writing, and the most fitting one to start with, since Netflix is where the project itself began.
What stands out across all of it is less any single number and more how many different Netflix teams ended up leaning on the same handful of Iceberg primitives - its snapshot history and its per-file column statistics - to solve problems that, on the surface, look unrelated: reprocessing late-arriving billing events, moving Cassandra data into the warehouse, and cleaning up unused media assets. By 2023, Netflix engineers were describing the underlying migration as exabyte-scale, moving roughly 300 petabytes of data off Hive tables entirely. The engineering problem underneath most of what follows is a familiar one from my own migration work, just several orders of magnitude larger: how do you keep a shared data warehouse correct and fast when thousands of independent pipelines are reading and writing it concurrently, without a central team hand-holding every table.
Company overview
Netflix operates a global streaming entertainment service used by members in more than 190 countries, and has more recently expanded into ad-supported plans, live events, and games. Every part of that business, from A/B test analysis to content acquisition to fraud detection, depends on a shared data warehouse that by 2020 already held hundreds of petabytes in Amazon S3, with more petabytes added daily.
That warehouse was originally built on Hive tables, and Hive’s assumptions about directory listings and atomic renames didn’t hold up on an eventually consistent object store like S3. Ryan Blue, an engineer on Netflix’s big data platform team, and Owen O’Malley of Cloudera introduced Iceberg’s alternative: a snapshot-based, file-level metadata model, publicly at the Strata Data Conference in New York in September 2018. Netflix subsequently donated the project to the Apache Software Foundation, and it has anchored the company’s data warehouse ever since.
| Date | Milestone |
|---|---|
| 2018-06 | Netflix publishes Metacat, its federated metadata service, on the Netflix TechBlog |
| 2018-09 | Ryan Blue and Owen O’Malley introduce Iceberg at Strata Data Conference New York |
| 2018-11 | Apache Iceberg enters the Apache Software Foundation Incubator |
| 2020-05 | Apache Iceberg graduates to an Apache Top-Level Project |
| 2020-12 | Netflix publishes AutoOptimize, its Iceberg storage-optimization system |
| 2021-07 | Netflix publishes Data Mesh-powered Studio operational reporting into Iceberg |
| 2023-11 | Netflix publishes its three-part Psyberg series and its Maestro-based Incremental Processing Solution |
| 2023-11 | Netflix presents its exabyte-scale, Iceberg-only migration at AWS re:Invent |
| 2024-03 | Netflix publishes its Iceberg-backed media asset Garbage Collector |
| 2026-05 | Netflix publishes the evolution of its Cassandra-to-Iceberg data movement engine |
Netflix’s Apache Iceberg use cases
Every use case below traces back to the same underlying decision: once Iceberg tables carried real snapshot and per-file metadata, individual teams stopped building bespoke change-detection or cleanup logic and started reading that metadata directly.
Incremental Processing Solution (IPS) - a shared change-capture primitive for Maestro
Netflix Maestro is the company’s workflow orchestrator, serving “thousands of users” including data scientists, data engineers, and business analysts across tens of thousands of workflows. As Jun He, Yingyi Zhang, and Pawan Dixit described in November 2023, the Incremental Processing Solution (IPS) extends Maestro with a lightweight incremental-processing building block, so that any workflow can get data freshness (micro-batch scheduling with state tracking), data accuracy (reprocessing all late-arriving data instead of relying on a fixed lookback window), and managed backfill without writing custom change-capture logic.
Psyberg - closing the gap on late-arriving membership and finance data
Netflix’s Membership and Finance Data Engineering team powers and reconciles many of the metrics in Netflix’s financial reports, which makes late-arriving signup, plan-change, and cancel events a real accuracy risk. Abhinaya Shetty and Bharath Mummadisetty explained that before Psyberg existed, an on-call data engineer had to manually triage audit alerts and re-trigger a sequential chain of hourly ETL jobs by hand, often without being the subject-matter expert for the affected pipeline. Psyberg automates that detection by reading Iceberg’s own metadata instead.
Cassandra-to-Iceberg data movement - feeding analytics from Member, Billing, and Recommendations
Apache Cassandra backs some of Netflix’s most critical services - Member, Billing, Recommendations, and Subscriptions - and Guil Pires and co-authors described how that data reaches Iceberg for analytics and operational use. The original connector, Casspactor, handled roughly 1,200 data movements a day and around 3 PB of data, working off Cassandra’s existing S3 backups rather than querying the online database directly.
Studio operational reporting - Data Mesh CDC into the data warehouse
As Netflix’s own film and television Studio scaled up its production pipeline, teams needed near-real-time visibility into studio data spread across many microservices. Andrew Nguonly and co-authors described how Data Mesh captures change data from Studio’s microservice databases, enriches it through the Studio Edge GraphQL data graph, and writes the result into Iceberg tables in the Netflix Data Warehouse for ad hoc and scheduled reporting.
Media asset lifecycle management - the Garbage Collector
Netflix Studios generate about 2 PB of data a week, and internal research found at least 40% of it is never read again. Vinay Kawade and co-authors from the Media Infrastructure & Storage Platform team explained how their Garbage Collector queries an Iceberg copy of Cassandra object metadata every day to work out which S3 objects are eligible for deletion under TTL or soft-delete policies, rather than querying the live, latency-sensitive Cassandra database.
The exabyte-scale migration off Hive
At AWS re:Invent 2023, Ashwin Kayyoor and Rakesh Veeramacheneni, both Netflix engineers, presented the company’s move to an Iceberg-only data warehouse, describing it as an exabyte-scale migration that moved roughly 300 petabytes of data off Hive tables, alongside supporting ecosystem work such as an Iceberg REST catalog and secure Iceberg tables.
I’ve spent a lot of my own career on exactly this kind of migration - moving a live system off one storage layer onto another without customers noticing - and the detail that struck me most is how many of Netflix’s Iceberg use cases (IPS, Psyberg, the Garbage Collector) only became possible after the migration itself: once the data was in Iceberg, every downstream team could read its snapshot history instead of building their own.
Scale and throughput
- Data warehouse size: hundreds of petabytes in Amazon S3 as of 2020, with additional petabytes ingested daily.
- Migration scope: roughly 300 petabytes moved as part of an exabyte-scale migration off Hive, per Netflix’s AWS re:Invent 2023 talk.
- Cassandra-to-Iceberg movement: the original Casspactor connector handled about 1,200 data movements a day, moving roughly 3 PB from Cassandra into Iceberg.
- Studio media assets: about 2 PB generated per week, with storage costs that had been growing roughly 50% year over year before the Garbage Collector shipped.
- AutoOptimize results: an 80% reduction in the total number of small files, a 72% reduction in file replacements, a 22% reduction in partition scans, roughly 60% faster queries, and 70% less compute than the previous merge system it replaced.
- Maestro adoption: 100% of Netflix’s scheduled workflow load runs on Maestro, with IPS available to all of it.
Netflix’s Apache Iceberg architecture
The Iceberg-backed data warehouse
Netflix’s data warehouse sits on Amazon S3, with Iceberg as the table format read and written by Spark, Presto, and Hive-compatible engines. That single table format is what lets teams as different as Membership Data Engineering and the Media Infrastructure & Storage Platform team build on the same underlying primitives.
Metacat - the metadata layer underneath every engine
Metacat, described by Ajoy Majumdar and Zhen Li back in 2018, is Netflix’s federated metadata service, originally built to unify Hive, S3, Druid, Elasticsearch, Redshift, Snowflake, and MySQL metadata so that Pig, Spark, Presto, and Hive could all treat the warehouse as one system. It predates Netflix’s Iceberg adoption, but per the 2023 AWS re:Invent talk it still plays an observability role across Netflix’s current Iceberg ecosystem.
Maestro - the orchestration substrate
Maestro is the workflow-as-a-service layer that runs all of Netflix’s scheduled data pipelines, and it’s the platform IPS, Psyberg’s scheduling, and the Garbage Collector’s daily cleanup jobs are all built on top of.
Data Mesh - CDC and streaming ingest into Iceberg
Data Mesh is Netflix’s configuration-driven, self-service change-data-capture platform: users wire up sources, reusable processors, and sinks without managing streaming infrastructure by hand. It writes into Iceberg both for Studio operational reporting and, separately, to stream the Garbage Collector’s delete-eligible rows out to a Kafka queue for processing.
The layered Cassandra Analytics engine
The newer Cassandra-to-Iceberg engine that replaced Casspactor is deliberately layered: a Cassandra Analytics Wrapper reads Cassandra’s own S3 backups directly and produces standard Spark DataFrames, and a “Connector Factory” model lets each Cassandra-backed data abstraction - Key Value, Time Series, Graph - build its own data-model-aware connector on that shared foundation instead of bolting post-processing onto a one-size-fits-all connector.
Special techniques and engineering innovations
AutoOptimize and AutoAnalyze - event-driven, entropy-guided compaction
Anupom Syam described AutoOptimize as a Service/Actors split: a Redis-backed decision service that observes table-change events and decides what to do, and a pool of long-running Spark “actor” jobs that do the merging, sorting, and compaction. Rather than scanning every changed partition, AutoOptimize computes a “Partition Entropy” statistic from a mean-squared-error formula over file sizes, and skips a partition entirely if it hasn’t drifted far enough from its target file size to be worth touching. When a partition does need work, AutoOptimize picks between a Knuth/Plass line-breaking algorithm and a modified First-Fit-Decreasing bin-packing algorithm to compute the minimum file replacement needed, and prioritizes across tenants using a Redis-backed Reliable Priority Queue.
Psyberg’s snapshot and partition metadata trick
Instead of querying underlying data to work out what changed, Psyberg reads two of Iceberg’s own metadata tables - snapshots, which records the operation type and partitions touched by each commit, and partitions, which stores per-partition, per-column value ranges - to detect exactly which partitions were affected by late-arriving event-time or processing-time data. It layers its own session and high-watermark metadata tables on top to track each pipeline’s progress independently.
IPS’s lightweight ICDC table
IPS creates a derived Iceberg table, which the team calls an ICDC table, that references only the new data-file pointers added since a source table’s last-seen snapshot, without copying any data. Because Iceberg already stores per-file column min/max statistics, the same lightweight table also exposes the changed value range for free - giving any of Maestro’s workflows both change-data capture and change-range capture from the metadata layer alone.
Streaming Kafka into a lakehouse via Kafka Connect, which is part of what I work on day to day, taught me to be suspicious of any change-capture design that needs a second copy of the data just to know what changed. IPS’s ICDC table is the cleanest version of that idea I’ve read: it captures change purely through Iceberg’s existing metadata, with nothing duplicated.
Shadow-mode validation for a mission-critical cutover
Before cutting over from Casspactor to the new Cassandra Analytics engine, Netflix ran the new engine in shadow alongside the old one on real production workloads, continuously diffing the two Iceberg outputs’ row sets in both directions until reaching 100% row-for-row parity.
Staged, reversible bulk table migration
The public Netflix-Skunkworks/hive2iceberg-migration tooling advances each Hive table through a Preprocessor, a multi-level (L1/L2/L3) Communicator that notifies table owners and downstream users ahead of time, a Migrator that performs the conversion, a Shadower that keeps a _hive-suffixed copy in sync during a probation window, and a Reverter that can roll a table back to Hive if something surfaces during that window.
Operating Iceberg at scale
Maestro carries 100% of the scheduled workload
Every one of Netflix’s scheduled data pipelines runs on Maestro, which is what let the company add IPS as a shared capability rather than a bolt-on available to only a subset of workflows.
Three-pillar migration governance: validation, visibility, safety
Netflix’s Cassandra-to-Iceberg engine cutover was governed by three explicit pillars: Validation (the shadow-mode row-parity testing described above), Visibility (dashboards tracking migrated-versus-remaining table counts, runtime, and cost comparisons, plus alerting on dependency failures), and Safety (automatic fallback to the legacy engine on failure). Framing a migration around those three concerns rather than just “does the new system work” is a discipline I recognize from cluster consolidation work of my own, where the failure mode that actually hurts a team isn’t the migration itself, it’s losing visibility into it partway through.
Staged communication and a probation window before dropping legacy tables
Bulk Hive-to-Iceberg migrations don’t drop the original table the moment the new one exists. Table owners get staged notifications ahead of the cutover, and the legacy table stays available, shadow-synced, for a two-week probation window with a revert path, before it’s finally dropped.
Separating soft-delete latency from hard-delete cost
Netflix’s media asset governance keeps a low-latency (under 15ms at the 99th percentile) soft-delete API, backed by Cassandra metadata, entirely separate from the asynchronous, autoscaled, Kafka-driven hard-deletion path, whose daily list of eligible objects comes from querying Iceberg rather than the live Cassandra database.
Challenges and how they solved them
Hive’s architecture broke down at Netflix’s scale on S3
By the mid-2010s, Netflix’s Hive-based warehouse was running up against fundamental limits: no atomic commits, no isolation between concurrent readers and writers, and query planning that depended on slow, eventually consistent S3 directory listings.
Problem: Hive tables gave no atomic commits and no reader/writer isolation, and their directory-listing-based query planning was slow and inconsistent on S3.
Root cause: Hive’s table format assumed a POSIX-like filesystem with atomic renames and consistent directory listings, neither of which S3 provides.
Solution: Ryan Blue and Owen O’Malley designed Iceberg’s snapshot-based, file-level metadata model, which tracks individual data files rather than relying on directory listings and requires an atomic commit for every table change.
Outcome: presented publicly as “Introducing Iceberg: Tables designed for object stores” at Strata Data Conference New York in September 2018, citing 10-100x faster query planning than the approach it replaced.
Casspactor’s fragile, composite view of Cassandra backups
Problem: Casspactor assembled its view of “which Cassandra backup exists and is complete” from several independent systems, and a single node replacement in a region could silently break data movement for that entire region; it also couldn’t handle large, skewed partitions without running out of memory.
Root cause: backup-completeness metadata was scattered across multiple systems with their own failure modes and update cadences, rather than living in one authoritative place.
Solution: the new Cassandra Analytics Wrapper reads backup completeness directly from the S3 backup files themselves, and processes partitions at the Spark executor level to handle skew that previously caused out-of-memory failures.
Outcome: validated through shadow-mode row-diffing until reaching 100% parity, delivering storage and compute savings Netflix describes as being in the order of USD millions, published by Guil Pires and co-authors in 2026.
Manual, error-prone recovery from late-arriving membership and finance data
Problem: a late-arriving signup, plan-change, or cancel event required an on-call data engineer to manually triage audit logs and re-trigger a sequential chain of hourly ETL jobs by hand, often without being the subject-matter expert for the affected pipeline.
Root cause: Netflix’s sequential, stateful hourly pipelines had no automated way to detect which specific partitions were affected by late-arriving data.
Solution: Psyberg automates that detection by reading Iceberg’s own snapshot and partition metadata tables instead of the underlying data.
Outcome: published as a three-part Netflix TechBlog series by Abhinaya Shetty and Bharath Mummadisetty of the Membership and Finance Data Engineering team, November 2023.
Point incremental-processing solutions didn’t generalize
Problem: internal incremental-processing libraries such as Psyberg only worked for the specific use cases they were built for, and were tightly coupled to each team’s business logic - which meant high migration and maintenance costs for any other team that wanted similar capability.
Root cause: incremental change capture had been implemented repeatedly, bespoke, inside individual pipelines rather than once as a shared platform primitive.
Solution: IPS generalizes incremental change capture into a lightweight, decoupled Maestro building block available to any of Maestro’s thousands of users.
Outcome: published by Jun He, Yingyi Zhang, and Pawan Dixit of the Data Platform team, November 2023.
Full tech stack
| Category | Tools | Notes |
|---|---|---|
| Table format | Apache Iceberg | Table format for the Netflix Data Warehouse; queried by Spark, Presto, and Hive-compatible engines |
| Orchestration | Netflix Maestro | Workflow-as-a-service orchestrator running 100% of Netflix’s scheduled workflows; hosts the Incremental Processing Solution |
| Incremental processing | Psyberg | Membership and Finance Data Engineering’s incremental ETL framework, built on Iceberg’s own snapshot and partition metadata |
| Streaming / CDC | Data Mesh | Configuration-driven, self-service CDC streaming pipeline platform that writes into Iceberg sinks |
| Metadata | Metacat | Federated metadata service unifying Hive/S3, Druid, Elasticsearch, Redshift, Snowflake, and MySQL metadata; also used for Iceberg-ecosystem observability |
| Operational database | Apache Cassandra | System of record for Member, Billing, Recommendations, and Subscriptions data; source for Iceberg data movement |
| Data movement | Casspactor / Cassandra Analytics Wrapper | Connector engines that move Cassandra backup data into Iceberg tables |
| Object storage | Amazon S3 | Underlying storage for the Iceberg data warehouse and for Cassandra backups |
| Messaging | Apache Kafka | Event backbone for the Garbage Collector’s delete queue and Netflix’s broader streaming ingest |
| Storage optimization | AutoOptimize / AutoAnalyze | Iceberg table storage optimization service (file merge, sort, compaction, metadata layout) |
| Data graph | Studio Edge | GraphQL data graph used to enrich Studio CDC events before they land in Iceberg |
| Compute | Apache Spark | Compute engine for AutoOptimize’s Actors, the Cassandra Analytics Wrapper, and general Iceberg table processing |
Key contributors
| Name | Role | Contribution |
|---|---|---|
| Jun He | Netflix, Data Platform | Co-authored the Incremental Processing Solution post |
| Yingyi Zhang | Netflix, Data Platform | Co-authored the Incremental Processing Solution post |
| Pawan Dixit | Netflix, Data Platform | Co-authored the Incremental Processing Solution post |
| Abhinaya Shetty | Netflix, Membership and Finance Data Engineering | Co-authored the Psyberg series |
| Bharath Mummadisetty | Netflix, Membership and Finance Data Engineering | Co-authored the Psyberg series |
| Guil Pires | Netflix | Co-authored the Cassandra Data Movement evolution post |
| Anupom Syam | Netflix | Authored the AutoOptimize / data warehouse storage post |
| Ajoy Majumdar | Netflix | Co-authored the Metacat post |
| Zhen Li | Netflix | Co-authored the Metacat post |
| Ashwin Kayyoor | Netflix, Senior Distributed Systems Engineer | Co-presented Netflix’s exabyte-scale Iceberg migration at AWS re:Invent 2023 |
| Rakesh Veeramacheneni | Netflix | Co-presented Netflix’s exabyte-scale Iceberg migration at AWS re:Invent 2023 |
| Ryan Blue | Netflix, big data platform team (at the time) | Co-created Apache Iceberg; co-presented its public introduction at Strata NY 2018 |
Key takeaways for your own Apache Iceberg implementation
These are the patterns from Netflix’s own account that I’d point any team toward, especially anyone still running a change-capture library on top of a data lake rather than reading the table format’s own metadata.
- Push change detection down into the table format, not up into your pipelines. Both Psyberg and IPS get late-arrival and change-range detection almost for free by reading Iceberg’s
snapshotsandpartitionsmetadata tables directly, instead of maintaining a separate change-tracking system alongside the data. - Generalize a working point solution before every team reinvents it. Netflix had to build IPS specifically because Psyberg-style libraries were too tightly coupled to individual teams’ business logic to reuse - worth deciding on purpose rather than discovering after the fact.
- Validate a migration by diffing outputs, not by trusting a code review. The Cassandra-to-Iceberg cutover ran in shadow against the legacy engine until the two systems’ row sets matched exactly, which is a materially higher bar than “the new code passed its tests.”
- Give a migrated table a genuine way back, not just a one-way door. The Hive-to-Iceberg tooling’s two-week probation window with an active shadow sync and a revert path meant a bad migration was a rollback, not an incident.
- Let storage optimization be event-driven, not scheduled. AutoOptimize only touches a partition when its own entropy metric says it’s drifted enough to be worth the cost, which is a cheaper default than periodic compaction jobs that run whether or not anything changed.
I’ll keep working through the rest of the list of companies running Iceberg at this kind of scale.
Sources and further reading
Primary sources
- Jun He, Yingyi Zhang, Pawan Dixit, “Incremental Processing using Netflix Maestro and Apache Iceberg” - https://netflixtechblog.com/incremental-processing-using-netflix-maestro-and-apache-iceberg-b8ba072ddeeb (2023)
- Abhinaya Shetty, Bharath Mummadisetty, “Streamlining Membership Data Engineering at Netflix with Psyberg” - https://netflixtechblog.com/1-streamlining-membership-data-engineering-at-netflix-with-psyberg-f68830617dd1 (2023)
- Guil Pires, Jennifer Prince, Jose Camacho, Ken Kurzweil, Phanindra Chunduru, “The Evolution of Cassandra Data Movement at Netflix” - https://netflixtechblog.com/the-evolution-of-cassandra-data-movement-at-netflix-6e13329c80a1 (2026)
- Andrew Nguonly, Armando Magalhães, Obi-Ike Nwoke, Shervin Afshar, Sreyashi Das, Tongliang Liu, Wei Liu, Yucheng Zeng, “Data Movement in Netflix Studio via Data Mesh” - https://netflixtechblog.com/data-movement-in-netflix-studio-via-data-mesh-3fddcceb1059 (2021)
- Vinay Kawade, Obi-Ike Nwoke, Vlad Sydorenko, Priyesh Narayanan, Shannon Heh, Shunfei Chen, “Navigating the Netflix Data Deluge: The Imperative of Effective Data Management” - https://netflixtechblog.medium.com/navigating-the-netflix-data-deluge-the-imperative-of-effective-data-management-e39af70f81f7 (2024)
- Anupom Syam, “Optimizing data warehouse storage” - https://netflixtechblog.com/optimizing-data-warehouse-storage-7b94a48fdcbe (2020)
- Ajoy Majumdar, Zhen Li, “Metacat: Making Big Data Discoverable and Meaningful at Netflix” - https://netflixtechblog.com/metacat-making-big-data-discoverable-and-meaningful-at-netflix-56fb36a53520 (2018)
- Ashwin Kayyoor, Rakesh Veeramacheneni, “Netflix’s journey to an Apache Iceberg-only data lake” (NFX306), AWS re:Invent - https://www.youtube.com/watch?v=jMFMEk8jFu8 (2023)
- Owen O’Malley, Ryan Blue, “Introducing Iceberg: Tables designed for object stores,” Strata Data Conference New York - https://conferences.oreilly.com/strata/strata-ny-2018/public/schedule/detail/69503.html (2018)
- Netflix-Skunkworks, “hive2iceberg-migration” - https://github.com/Netflix-Skunkworks/hive2iceberg-migration/blob/main/README.md (accessed 2026)
Enterprise tooling for Apache Iceberg
Netflix built most of what’s described above in-house, which is exactly the kind of engineering investment most teams adopting Iceberg can’t make themselves. That gap is what my team and I are building Iglu for at Factor House: enterprise tooling for Apache Iceberg. If Iceberg’s snapshot and partition metadata are already doing real work in your own pipelines, request preview access and tell us what you’re running into.