After eight years at Square, most of it spent on the streaming and data infrastructure behind Cash App, including a production migration of our own analytics tables onto Apache Iceberg, I’ve been going back through how other large engineering organizations run Iceberg once it’s carrying real production weight rather than a proof of concept. Apple is the first one worth writing up: eight of its own engineers co-authored a peer-reviewed paper on the row-level operations framework they built into Iceberg and Spark, describing production tables holding tens of petabytes of data and tens of millions of files. Everything below comes from that paper, from a keynote and interview given by the engineering manager who led the work, and from two more recent conference talks by other named Apple engineers - six sources in total, read directly rather than through a secondhand summary, and I’ve listed all of them at the end. This is the first Iceberg use-case piece I’ve put together for this series, so there’s no back catalogue to point you to yet.
The engineering problem underneath Apple’s work is one I recognize from my own time on it, just under more regulatory pressure than most teams face: how do you delete or update individual rows scattered across petabyte-scale tables, cheaply and correctly, when the traditional data-lake answer is to rewrite an entire partition for even a single changed record. Apple’s answer became a core part of how Iceberg itself now handles row-level changes, not just a private workaround.
Company overview
Apple runs consumer hardware, software, and services at a scale that generates enormous volumes of operational data across its product lineup. A dedicated Data Lakehouse team builds and operates a managed lakehouse on Apache Iceberg, with its own Iceberg REST Catalog implementation, serving both analytics and machine-learning workloads company-wide.
Iceberg adoption at Apple grew from a small, early bet into what engineering manager Russell Spitzer described as “wall-to-wall” coverage. As he put it in an interview after eventually leaving Apple for Snowflake: “When I joined the company, it was almost nowhere. Back then, no one knew what Iceberg was. You had to explain why it was something you wanted. Now, everyone knows what Iceberg is.” The specific driver that pushed Apple from being a general Iceberg user to becoming one of its deepest contributors was regulatory: GDPR and the EU’s Digital Markets Act both require deleting or updating individual records on request, and Hive-style partition rewrites made that prohibitively expensive at Apple’s scale.
| Date | Milestone |
|---|---|
| Undated | Russell Spitzer joins Apple from DataStax and begins working on what he later described as “the then nascent Apache Iceberg project” |
| 2024-05-02 | Spitzer delivers “Iceberg Development at Apple” at Subsurface Live, describing wall-to-wall adoption and the GDPR/DMA-driven row-level operations work |
| 2024 | “Petabyte-Scale Row-Level Operations in Data Lakehouses,” co-authored by eight Apple engineers plus Iceberg co-creator Ryan Blue, is published in PVLDB Volume 17, confirming production deployment at tens of petabytes and tens of millions of files |
| 2024-06 | Spitzer leaves Apple for Snowflake, remaining an Apache Iceberg PMC member |
| 2024-10-03 | The Register publishes an interview with Spitzer describing Iceberg’s adoption at Apple as “wall-to-wall” |
| 2025-05 | Apple engineers present two sessions at the first in-person Apache Iceberg Summit: Nicholas Mehr on cross-engine interoperability, and Parth Chandra with Huaxin Gao on accelerating Spark with Iceberg and Apache DataFusion Comet |
Apple’s Apache Iceberg use cases
Regulatory row-level deletes across the lakehouse
The use case that pulled Apple furthest into Iceberg’s internals was compliance. As Russell Spitzer explained at Subsurface Live 2024, GDPR and DMA requests require deleting or updating individual records, often scattered sparsely across many partitions of a table. Hive’s partition-level model meant rewriting an entire partition to satisfy a single-row deletion request - workable at small scale, ruinously expensive once tables reach petabytes. Apple’s engineering team built out Iceberg’s row-level delete, update, and merge support specifically to make sparse, record-level compliance operations cheap enough to run routinely.
Change data capture and upsert pipelines at petabyte scale
Beyond compliance, the same row-level framework serves ordinary data-engineering workloads: change data capture, incremental materialized view refresh, and slowly changing dimensions, all of which need to handle updates and deletes, not just appends. As Anton Okolnychyi and his co-authors describe, the framework was explicitly designed to support daily batch jobs touching a large share of a table’s partitions, hourly micro-batches modifying a small set of records, and streaming upserts, all against the same underlying tables.
Cross-engine analytics on the same tables
Iceberg’s appeal at Apple isn’t limited to any one compute engine. As Nicholas Mehr described at the 2025 Apache Iceberg Summit, Iceberg’s interoperability lets Spark, Trino, and Flink all read and write the same tables, with Kafka feeding streaming ingestion into Iceberg for combined batch and streaming workloads - letting different teams pick the engine that suits their job without forking the underlying data.
Scale and throughput
- Production deployment: tens of petabytes of data and tens of millions of files, per Apple’s own VLDB paper describing the row-level operations framework as “widely deployed in production” at that scale.
- Individual table sizes: range from hundreds of megabytes up to many petabytes, reflecting how broadly Iceberg is used across teams rather than concentrated in one warehouse.
- Maintenance time: operations that “used to take something like two hours” now complete in “several minutes,” per Spitzer’s Subsurface 2024 talk.
- Query time: some aggregate queries improved from “over an hour” to “the second range” through metadata-level pushdowns that skip scanning data files entirely, per the same talk.
- Benchmark methodology (Apple’s own, not a production figure): the VLDB paper’s evaluation populated a TPC-DS
store_salestable at scale factor 1000 (2.8 billion records) and ran it on an 8-node Amazon EKS cluster (16 cores, 64GB RAM per node) with Spark 3.5.1 and Iceberg 1.5.0.
Introducing storage-partitioned joins to that same benchmark produced roughly an order-of-magnitude improvement in write performance for both of Iceberg’s materialization strategies - a bigger jump than I expected from what’s essentially a smarter join plan, and a reminder of how much shuffle overhead petabyte-scale MERGE operations can otherwise carry.
Apple’s Apache Iceberg architecture
Cloud deployment and the Iceberg REST Catalog
Apple’s Data Lakehouse team runs its managed lakehouse with its own Iceberg REST Catalog implementation, and the row-level operations benchmark in Apple’s VLDB paper ran on Amazon EKS. By the 2025 Iceberg Summit, Nicholas Mehr described Apple evaluating emerging AWS-managed Iceberg metadata support - a reference to AWS S3 Tables, which launched as a managed Iceberg catalog and storage option in late 2024 - as a way to simplify some of that large-scale workflow management.
Multi-engine compute topology
Spark, Trino, and Flink all operate against the same Iceberg tables at Apple, per Mehr’s 2025 talk. Spark carries the heaviest load: it’s the engine used for the row-level operation writes, table maintenance, and - notably - for distributed metadata planning itself. As Okolnychyi et al. describe, when a query’s manifest-skipping isn’t effective enough on its own, Iceberg can offload manifest reading and evaluation to a Spark cluster rather than a single coordinator process, a technique the paper notes Spark currently uses “for distributed maintenance and planning.”
Streaming ingestion into Iceberg tables
Kafka feeds streaming ingestion into Iceberg tables, combined with Iceberg’s native support for both batch and streaming upserts, per Mehr’s 2025 talk - the same row-level machinery built for compliance deletes also carries ordinary streaming writes.
Metadata layout designed for incremental commits
Apple’s engineers were core contributors to Iceberg’s persistent-tree metadata layout: a catalog pointer to a metadata file, to a manifest list, to individual manifests, to data and delete files. Because manifests aren’t bound to specific partitions and can be shared across snapshots, Iceberg can produce a new snapshot by inheriting nearly all of the prior metadata unchanged, which is what makes commits fast even against tables with tens of millions of files.
Special techniques and engineering innovations
Combining copy-on-write and merge-on-read in one table
Apple’s engineers extended Iceberg to support two distinct ways of encoding a change - copy-on-write (rewriting and swapping whole data files) and merge-on-read (writing small delete files that are merged with the base data at read time) - and, critically, designed them to be usable together within the same table. A daily batch job touching half a table’s partitions can use copy-on-write, while a sparse, compliance-driven single-row delete can use merge-on-read, without picking one strategy for the whole table up front.
Position and equality delete files
Merge-on-read comes in two flavors. Position deletes reference a data file and a row position, and load into a Roaring bitmap at read time - cheap to apply, well suited to micro-batch and streaming workloads. Equality deletes reference column values directly and can be written without scanning the target table at all, which makes them the fastest option for very sparse upserts, but they’re more expensive for readers to apply and, as of the VLDB paper, can’t yet be compacted across sequence numbers.
Storage-partitioned joins
Row-level operations against petabyte-scale tables need to join a target table against a set of incoming changes, and a naive join means shuffling both sides across the cluster. Apple’s engineers generalized Spark’s existing bucket-join mechanism into storage-partitioned joins, which co-locate data using Iceberg’s own partition transforms instead of requiring identical hash bucketing on both sides, plus a “partially clustered distribution” technique that splits oversized, skewed partitions into smaller pieces each side of the join can process independently.
This is the part of Apple’s paper that’s closest to what my own team dealt with migrating Cash App’s ledger tables onto Iceberg: the join strategy matters far more than the delete-encoding choice once you’re past a few hundred gigabytes, and it’s easy to blame slow MERGE performance on the wrong layer of the stack.
Runtime filtering to cut write amplification
When a row-level operation’s predicate involves a subquery that can’t be pushed down to Iceberg for file skipping during planning, Spark now evaluates a lightweight filter subquery at runtime to identify which file groups actually contain matches, before the expensive rewrite step runs - avoiding the cost of eagerly rewriting files that turn out not to have any matching rows at all.
Bitmap-based cardinality checks for MERGE
The SQL standard requires validating that a MERGE’s ON condition doesn’t match more than one target row per source row. Apple’s initial implementation used a local sort by synthetic row ID to check for duplicates, which caused costly spills to disk under load; the team switched to a compressed bitmap instead, avoiding the sort entirely and reducing memory pressure.
Operating Apache Iceberg at scale
Distributed table maintenance as a routine job, not a special case
Expiring old snapshots, rewriting manifests, and compacting data and delete files all run as distributed Spark jobs at Apple rather than single-process operations - a requirement once tables reach tens of millions of files, and part of what took maintenance windows from roughly two hours down to several minutes.
Compaction strategy differs by delete type
Position deletes can be compacted without reading the underlying data files at all, which is cheap. Equality deletes, as of Apple’s 2024 paper, can’t yet be compacted across sequence numbers and instead have to be converted into position deletes first - a more expensive path that does require reading data. Apple’s own evaluation found that a minor compaction of position deletes after ten iterations of updates restored read performance to within 14% of the pre-modification baseline, a 45% improvement over the uncompacted state, at only 23% of the cost of a full eager rewrite of the same data.
Compliance deletes as a routine production operation
Perhaps the most operationally significant shift is not a specific technique but a posture: Apple treats GDPR- and DMA-style row deletes as an ordinary, frequent production operation rather than an occasional, manually-run exception - which is exactly why the maintenance-time and query-time improvements above mattered enough to invest real engineering effort into.
Challenges and how they solved them
Partition-level rewrites were too costly for regulatory deletes
Hive-style data lakes could only update or delete data at the partition level, which collided directly with the granularity that compliance regulation actually requires.
Problem: Satisfying a GDPR or DMA deletion request for even a single record meant rewriting an entire partition under Hive’s model.
Root cause: GDPR and DMA requests are frequently sparse - individual records scattered across many partitions - and the partition-replacement model handles that by rewriting far more data than actually changed.
Solution: Apple’s engineers built and adopted Iceberg’s row-level DELETE, UPDATE, and MERGE support, using merge-on-read for sparse compliance-driven changes so only the affected rows are touched.
Outcome: Maintenance operations that previously took roughly two hours dropped to several minutes, per Russell Spitzer’s Subsurface Live 2024 talk.
Shuffle-heavy joins slowed row-level operations at petabyte scale
Running DELETE, UPDATE, or MERGE against multi-petabyte tables meant shuffling both the target table and the incoming changes across the cluster.
Problem: Row-level operations at Apple’s scale were spending a large share of their time on shuffle I/O rather than the actual work of finding and applying changes.
Root cause: Spark’s traditional hash-based bucket joins require identical bucketing on both sides of a join and don’t generalize to Iceberg’s more flexible partition-transform-based clustering, so joins between a target table and its changes still triggered full shuffles.
Solution: Apple’s engineers designed storage-partitioned joins, co-locating data via Iceberg’s own partition transforms, plus a partially clustered distribution mechanism to split skewed partitions into independently processable pieces.
Outcome: Roughly an order-of-magnitude improvement in write performance for both eager and lazy materialization, measured directly in Apple’s own VLDB benchmark.
Write amplification from rewriting files that didn’t actually match
Row-level operations with complex predicates - subqueries in particular - couldn’t be pushed down to Iceberg for file skipping during planning, so early implementations rewrote more files than necessary.
Problem: Operations that only actually matched a small fraction of a table’s data files were still eagerly rewriting far more files than needed.
Root cause: Complex subquery predicates can only be evaluated inside Spark itself, not by the Iceberg connector during query planning, which ruled out file-level pruning based on those predicates ahead of time.
Solution: Apple’s engineers added runtime filtering, letting Spark evaluate a lightweight filter subquery mid-plan to identify which file groups actually contain matches, before the expensive rewrite runs.
Outcome: A measurable reduction in unnecessarily rewritten data for operations targeting a small subset of files, demonstrated in Apple’s own evaluation against a case affecting only 25% of a table’s data files.
Read performance degraded as delete files accumulated
Under merge-on-read, every additional round of updates adds more delete files that readers have to reconcile at query time - and that cost climbs faster for equality deletes than for position deletes.
Problem: Query performance steadily degraded with each additional round of updates as delete files accumulated, without any table maintenance in between.
Root cause: Delete files are immutable additions rather than in-place edits, so without periodic compaction, the number of delete files a scan must merge grows without bound.
Solution: Apple built targeted compaction paths: cheap, data-file-free compaction for position deletes, a more expensive data-scanning path to convert accumulated equality deletes into position deletes, and the option to fall back to a single eager (copy-on-write) pass to reset a table to zero deletes without a full compaction.
Outcome: In Apple’s own micro-batch evaluation, a minor compaction of position deletes after ten iterations restored read performance to within 14% of baseline - a 45% improvement over the uncompacted state - at only 23% of the cost of a full eager rewrite, per the VLDB paper.
Full tech stack
| Category | Tools | Notes |
|---|---|---|
| Table format | Apache Iceberg | Core format for Apple’s lakehouse; extended with row-level delete/update/merge support, position and equality delete files, and distributed maintenance |
| Batch/stream compute | Apache Spark (3.5.1 in Apple’s own benchmark) | Primary engine for writes, table maintenance, distributed metadata planning, and the row-level operation extensions (storage-partitioned joins, runtime filtering, adaptive writes) |
| Query engine | Trino | Additional query engine reading the same Iceberg tables |
| Stream/batch processing | Apache Flink | Additional engine operating against the same Iceberg tables |
| Streaming ingestion | Apache Kafka | Feeds streaming writes into Iceberg tables alongside batch workloads |
| Catalog | Iceberg REST Catalog | Catalog layer for Apple’s managed Lakehouse service |
| Cloud infrastructure | Amazon EKS | Used to run Apple’s own row-level operations benchmark |
| Emerging catalog/storage | AWS-managed Iceberg metadata (e.g. S3 Tables) | Evaluated by Apple as of the 2025 Iceberg Summit talk to simplify large-scale workflows |
| Native execution | Apache DataFusion Comet | Arrow-based native Spark execution engine Apple engineers were integrating to accelerate row-level operations |
| File format | Apache Parquet | Underlying columnar format for Iceberg data and delete files throughout Apple’s evaluation |
Key contributors
| Name | Role | Contribution |
|---|---|---|
| Anton Okolnychyi | Apache Iceberg PMC member and committer; Apache Spark contributor (at Apple during this work) | Lead author of Apple’s VLDB paper on petabyte-scale row-level operations; designed the framework itself |
| Russell Spitzer | Engineering manager, Apache Iceberg PMC member (at Apple during this work; later Snowflake) | Co-authored the VLDB paper; gave the “Iceberg Development at Apple” keynote at Subsurface Live 2024 |
| Chao Sun | Apache Iceberg PMC member (at Apple during this work) | Co-authored the VLDB paper |
| Kazuyuki Tanimura | Apple | Co-authored the VLDB paper |
| Szehon Ho | Apple | Co-authored the VLDB paper |
| Yufei Gu | Apple | Co-authored the VLDB paper |
| Vishwanath Lakkundi | Apple | Co-authored the VLDB paper |
| DB Tsai | Apple | Co-authored the VLDB paper |
| Ryan Blue | Co-creator of Apache Iceberg (at Tabular during this work) | Co-authored the VLDB paper alongside the Apple team |
| Nicholas Mehr | Software engineer, Apple | Presented on Iceberg’s cross-engine interoperability at Apple at the 2025 Apache Iceberg Summit |
| Parth Chandra | Software engineer, Apple | Co-presented on accelerating Spark with Iceberg and Apache DataFusion Comet at the 2025 Apache Iceberg Summit |
| Huaxin Gao | Software engineer, Apple; committer on Apache Spark, Iceberg, Parquet, and DataFusion | Co-presented the same DataFusion Comet session |
Key takeaways for your own Apache Iceberg implementation
These are the patterns from Apple’s own account that I’d point any team running (or planning to run) Iceberg at meaningful scale toward:
- Don’t force a single materialization strategy on a whole table. Apple’s engineers designed copy-on-write and merge-on-read to be usable together in the same table, matching the strategy to the shape of each specific job rather than picking one approach and living with its weaknesses everywhere.
- Check your join strategy before you blame the delete encoding. Storage-partitioned joins produced roughly an order-of-magnitude write-performance improvement in Apple’s own benchmark - a bigger factor than the choice between position and equality deletes, and the first place I’d look if a MERGE is underperforming at scale.
- Compaction cadence should match delete type, not a single fixed schedule. Position deletes compact cheaply without reading data; equality deletes are more expensive and, per Apple’s paper, couldn’t yet be compacted across sequence numbers at all - budget maintenance time accordingly rather than treating all delete files the same way.
- Treat regulatory deletion requirements as a capacity-planning input, not an edge case. Apple’s entire row-level operations investment traces back to needing GDPR and DMA deletes to run cheaply and routinely at scale - if your table format can’t do that affordably today, it will become the loudest complaint in the room the first time a real deletion request lands on a petabyte-scale table.
- Distribute your maintenance jobs, not just your queries. Apple runs snapshot expiry, manifest rewriting, and compaction as distributed Spark jobs specifically because single-process maintenance stopped being viable once tables reached tens of millions of files.
I’ll keep working through the rest of the list and write up the next one as it’s ready.
Sources and further reading
Primary sources
- Anton Okolnychyi, Chao Sun, Kazuyuki Tanimura, Russell Spitzer, Ryan Blue, Szehon Ho, Yufei Gu, Vishwanath Lakkundi, DB Tsai, “Petabyte-Scale Row-Level Operations in Data Lakehouses,” Proceedings of the VLDB Endowment, Vol. 17, No. 12 - https://vldb.org/pvldb/vol17/p4159-okolnychyi.pdf (2024)
- Russell Spitzer, “Keynote Address: Iceberg Development at Apple,” Subsurface Live - https://www.youtube.com/watch?v=kOsJ0cRV4YI (2024)
- Reporting on Russell Spitzer’s Subsurface Live 2024 talk, “How Iceberg Powers Data and AI Applications at Apple, Netflix, LinkedIn, and Other Leading Companies,” Qlik Blog - https://www.qlik.com/blog/how-iceberg-powers-data-and-ai-applications-at-apple-netflix-linkedin-and (accessed 2026)
- Named interview with Russell Spitzer, “Are the table format wars entering the final chapter?,” The Register - https://www.theregister.com/2024/10/03/apache_iceberg_russell_spitzer_interview/ (2024)
- Nicholas Mehr, “Iceberg as the Modern Table Standard at Apple,” Databricks Data + AI Summit session listing - https://databricks.com/dataaisummit/session/iceberg-modern-table-standard-apple (2025)
- Huaxin Gao, Parth Chandra, “Accelerating Spark with Iceberg and DataFusion Comet: Challenges, Performance Insights, and Roadmap,” Iceberg Summit - https://www.youtube.com/watch?v=kd4iqN0dmog (2025)
I read every one of these directly rather than working from a summary, and I’ve linked the specific claim to its source throughout the piece above, not just in this list.
Getting visibility into Iceberg tables at this scale
Apple’s investment here (combining materialization strategies in one table, distributing maintenance, and treating compliance deletes as routine) is the exact kind of production-grade Iceberg operation that’s hard to see clearly without dedicated tooling. That visibility problem is what my team and I are building Iglu for at Factor House: it’s currently in preview with select customers, so if the operational picture Apple describes here sounds familiar, request preview access and we’ll get you in.