I spend most of my working time on open table formats and the streaming infrastructure that feeds them, and Airbnb’s own account of moving its data warehouse ingestion off Tez and Hive onto Apache Iceberg is one of the more concrete migration write-ups I’ve come across, so it’s worth a proper look. Before writing this, I read five primary sources directly rather than working from summaries: Airbnb’s own engineering blog post on the migration, a design discussion Airbnb posted to the Apache Iceberg developer mailing list, a Trino Summit talk description and recording from two Airbnb engineers, and an earlier Airbnb architecture post used only for background.
Airbnb’s pipeline was ingesting more than 35 billion Kafka event messages a day into over 1,000 warehouse tables before the migration even started, which puts the numbers below in context. The underlying engineering problem is one every team running a Hive-based warehouse at that scale eventually hits: the Hive Metastore and the Hive table format weren’t built to carry the partition and metadata load of a system ingesting tens of billions of events daily, and Apache Iceberg was designed specifically to remove that class of bottleneck.
Company overview
Airbnb runs a global two-sided marketplace connecting hosts and guests, and its data platform ingests the logging and event data that product, growth, and analytics teams depend on for everything from search ranking to pricing. By 2016, Airbnb’s data infrastructure already spanned two HDFS clusters holding a combined 11 petabytes, alongside multi-petabyte storage on Amazon S3, with Presto as the primary ad-hoc query engine over Hive-managed tables (James Mayfield, Krishna Puttaswamy, et al., Airbnb Tech Blog, 2016). That scale is background context rather than an Iceberg-era figure, but it explains why the Hive Metastore eventually became a constraint worth re-architecting around.
The event-ingestion pipeline that later moved to Iceberg is fed by Airstream, Airbnb’s Spark Streaming-based framework that publishes logging events from services and clients to Kafka and lands them into the warehouse in near real time. As partition counts grew, the Hive Metastore itself became a bottleneck, and Airbnb engineers Ronnie Zhu, Edgar Rodriguez-Diaz, Jason Xu, Gustavo Torres, Kerim Oktay, and Xu Zhang led the effort to move that pipeline onto Apache Spark 3 running on Apache Iceberg tables, published in detail on the Airbnb Tech Blog in September 2022.
| Date | Milestone |
|---|---|
| Feb 2016 | Airbnb’s data infrastructure spans two HDFS clusters at 11PB combined, plus multi-petabyte S3 storage, with Presto as the primary ad-hoc query engine over Hive tables |
| Mar 2021 | Airbnb engineer Gustavo Torres Torres posts a design discussion to the Apache Iceberg developer mailing list on Airflow-Iceberg sensor integration, indicating the Hive-to-Iceberg migration is already underway |
| Sep 2022 | Airbnb publishes “Upgrading Data Warehouse Infrastructure at Airbnb,” documenting the completed migration of its event-ingestion pipeline to Spark 3 on Iceberg |
| Dec 2023 | Airbnb engineers Jinyang Li and Tingting Ma present Trino’s workload-management redesign at Trino Summit, describing Trino as Airbnb’s main ad-hoc analytics engine |
| Apr 2025 | Airbnb is represented among attending companies at the inaugural Iceberg Summit in San Francisco |
Airbnb’s Apache Iceberg use cases
Airbnb’s sourced Iceberg usage centers on one pipeline, but the two changes it made there are each worth separating out, since they solve different problems.
Airstream - Kafka-to-warehouse event ingestion
Airstream is the framework that turns Airbnb’s Kafka event stream into warehouse tables. It was already handling more than 35 billion messages a day across 1,000-plus tables before the migration, running on Apache Spark Streaming (Ronnie Zhu et al., Airbnb Tech Blog, 2022). An earlier post on the same framework’s Kafka-reading layer, Hao Wang, “Scaling Spark Streaming for Logging Event Ingestion,” Airbnb Tech Blog, 2018, describes the throughput problem this ingestion path was already built to handle, well before the Iceberg migration. Moving the tables it writes into from Hive to Iceberg didn’t change what Airstream does; it changed what the tables underneath it can tolerate at that message volume.
Consolidating hourly and daily tables into one Iceberg table
Before the migration, Airbnb maintained two parallel tables for the same underlying data: one partitioned hourly, one partitioned daily, purely because Hive and its metastore couldn’t cheaply serve both granularities from a single table. Iceberg’s native partition-spec evolution let Airbnb collapse those into one table carrying multiple partition specs, then migrate the existing data via compaction rather than a full rewrite - at a resource saving of more than 90% over rewriting the table from scratch (Ronnie Zhu et al., 2022). I’ve seen this exact pattern in other Hive-only warehouses I’ve worked on migrating: two tables, two sets of downstream jobs, double the operational surface for one dataset, all just to serve two query granularities cheaply. It’s precisely the kind of maintenance tax partition-spec evolution was built to remove.
Scale and throughput
- Kafka events ingested per day: more than 35 billion messages, across more than 1,000 warehouse tables
- Compute savings from the Spark 3 + Iceberg migration: more than 50% reduction in compute, with a 40% reduction in job elapsed time, versus the prior Tez-on-Hive stack
- Compaction efficiency: more than 90% resource savings when consolidating hourly and daily tables via partition-spec evolution, compared with a full table rewrite
Airbnb hasn’t published a total Iceberg table count, an aggregate data volume figure across its Iceberg tables, or partition/file counts specific to this migration, so I’m not going to estimate them here - the figures above are the ones Airbnb itself has put a number on.
Airbnb’s Apache Iceberg architecture
Deployment and storage
Airbnb’s data warehouse runs on AWS, with Amazon S3 as the underlying object store, having previously depended on HDFS for at least part of the pipeline described below (Ronnie Zhu et al., 2022).
Compute layer: Apache Spark 3 replacing Tez on Hive
Apache Spark 3 is the batch engine behind the migrated ingestion workload, and it’s Spark 3’s Adaptive Query Execution (AQE) specifically, not just the engine swap, that Airbnb credits with a meaningful share of the efficiency gain - AQE lets a job pick its shuffle-partition count dynamically instead of running with the fixed count Tez required.
Catalog: Hive Metastore, minus the partition load
Iceberg tables at Airbnb still register in the Hive Metastore, but partition information itself is no longer stored there, which removes the single biggest source of metastore load that a partition-heavy Hive table generates. Iceberg tables also don’t depend on S3’s list-after-write consistency, which had previously forced Airbnb into a workaround: write to a temporary HDFS cluster first, then move the data to S3 via an optimized distcp job. Neither Airbnb source confirms whether a REST or other non-Hive catalog is used anywhere else in the warehouse, so I’m treating Hive Metastore as the confirmed catalog for this pipeline specifically, not a warehouse-wide claim.
Interactive queries: Trino
Trino is Airbnb’s main interactive compute engine for offline ad-hoc analytics, and engineers Jinyang Li and Tingting Ma presented a workload-management redesign for it - query cost forecasting and workload-aware scheduling - at Trino Summit 2023. That talk describes Trino’s cluster architecture in detail but doesn’t itself state that Trino reads the same Iceberg tables Airstream writes to; I’m treating that connection as a reasonable architectural inference from the ingestion post, not a directly sourced claim, and flagging it as such rather than asserting it outright.
Special techniques and engineering innovations
Partition-spec evolution instead of a full table rewrite
Rather than migrating the consolidated hourly/daily table by rewriting all of its historical data under a new layout, Airbnb used Iceberg’s native support for changing a partition spec in place - switching from an hourly (ds/hr) spec to a daily (ds) spec - and let compaction handle the transition over time. That’s the design choice behind the greater than 90% resource saving mentioned above, and it’s a good example of Iceberg’s partition evolution doing real migration work rather than just being a schema-flexibility feature on paper.
Designing Iceberg-aware Airflow sensors ahead of the Hive Metastore
In a March 2021 message to the Apache Iceberg developer mailing list, Airbnb engineer Gustavo Torres Torres laid out the design problem: Airflow’s existing partition sensors were built around Hive’s partition-appearance semantics, and a straight port to Iceberg tables would either need the Hive Metastore API modified so Iceberg partition-sensing stays transparent to users, or a distinct sensor that makes clear it’s sensing over an Iceberg table rather than a Hive one. The 2022 migration post confirms Airbnb ultimately reworked partition-readiness detection around a signal-based sensor mechanism, consistent with that earlier design discussion. In my own experience planning table-format migrations, this kind of orchestration-layer work is the easiest to under-budget for, since it never shows up in the storage or compute numbers you’re using to justify the project.
Challenges and how they solved them
Hive Metastore load from partition-heavy queries
As Airbnb’s ingestion volume grew, so did the number of partitions any given query needed to touch - pulling a month of hourly partitions, for instance, meant the Hive Metastore had to serve metadata for every one of them on every query.
Problem: The Hive Metastore became a bottleneck for partition-heavy queries as table partition counts grew.
Root cause: Hive stores partition metadata centrally in the metastore, so metastore load scales directly with partition count and query breadth.
Solution: Move the affected tables to Iceberg, which tracks partition information in its own metadata layer rather than the Hive Metastore.
Outcome: Partition metadata load on the Hive Metastore was removed for the migrated tables, as described in Airbnb’s September 2022 migration post.
The S3 list-after-write workaround
Before the migration, writing directly to S3 carried a consistency risk that Airbnb had to design around rather than rely on.
Problem: S3’s list-after-write consistency behavior wasn’t reliable enough for the ingestion pipeline to write directly and safely.
Root cause: Hive-style tables depend on directory listing to discover new files, and S3 listing consistency couldn’t guarantee that behavior at Airbnb’s write volume.
Solution: Write output to a temporary HDFS cluster first, then move it to S3 via an optimized distcp job, as an interim workaround ahead of the Iceberg migration.
Outcome: Moving to Iceberg tables removed the dependency on S3 listing entirely, eliminating both the consistency risk and the extra distcp step, per Ronnie Zhu et al., 2022.
Inconsistent schema evolution across engines
Problem: Schema changes propagated inconsistently across the engines reading and writing the same tables, creating data quality issues.
Root cause: Hive’s schema evolution semantics aren’t uniformly implemented or enforced across every engine that touches a table.
Solution: Adopt Iceberg, whose schema evolution is standardized at the table-format level rather than left to each engine’s own interpretation.
Outcome: More predictable schema-evolution behavior across engines reading the migrated tables, per Airbnb’s 2022 account.
Static shuffle partitioning under the old Spark configuration
Problem: Jobs ran with a fixed 200 shuffle partitions regardless of actual data volume, producing inefficient runs on datasets that didn’t match that assumption - and, separately, output files that were too small once compression was applied.
Root cause: A static shuffle-partition count can’t adapt to the wide variance in shuffle data size that a workload like this produces run to run.
Solution: Turn on Spark 3’s Adaptive Query Execution to let the shuffle-partition count vary dynamically (Airbnb observed a range of 5 to 1,500), and separately raise the advisory shuffle partition size to counter the small-file problem. A related fix, backporting the SPARK-35239 patch for empty-input handling, was also needed for Spark versions below 3.2 to work correctly in this pipeline.
Outcome: A representative medium-sized job’s runtime dropped from 12 minutes 36 seconds to 4 minutes 34 seconds, contributing to the migration’s overall greater than 50% compute saving and 40% reduction in job elapsed time, per Ronnie Zhu et al., 2022.
Full tech stack
| Category | Tools | Notes |
|---|---|---|
| Table format | Apache Iceberg | Replaces Hive tables for the migrated ingestion pipeline; removes partition metadata from the Hive Metastore and the S3 list-after-write dependency |
| Batch compute | Apache Spark 3 | Primary engine for the ingestion workload; Adaptive Query Execution drives dynamic shuffle-partition sizing |
| Prior batch compute | Apache Tez (on Hive) | Replaced by Spark 3 for this workload |
| Catalog / metastore | Apache Hive (Metastore) | Retained as the catalog after the Iceberg migration; no longer stores partition metadata |
| Object storage | Amazon S3 | Underlying warehouse storage |
| Prior storage layer | HDFS | Used as a temporary staging cluster in the pre-Iceberg distcp-to-S3 workaround |
| Event streaming | Apache Kafka | Source of the event stream ingested into the warehouse (35B+ messages/day) |
| Ingestion framework | Airstream | Airbnb-built, Spark Streaming-based framework that ingests Kafka events into the warehouse |
| Interactive query engine | Trino | Airbnb’s main ad-hoc analytics engine, with a dedicated workload-management redesign presented at Trino Summit 2023 |
| Orchestration | Apache Airflow | Partition-sensing and signal mechanisms reworked for Iceberg tables |
Key contributors
| Name | Role | Contribution |
|---|---|---|
| Ronnie (Huirong) Zhu | Staff Software Engineer, Airbnb | Lead author, “Upgrading Data Warehouse Infrastructure at Airbnb” |
| Edgar Rodriguez-Diaz | Staff Software Engineer, Airbnb | Co-author of the same post |
| Gustavo Torres (Torres) | Software/Data Engineer, Airbnb | Co-author of the same post; authored the March 2021 Apache Iceberg dev-list design discussion on Airflow-Iceberg sensor integration |
| Jason Xu | Airbnb | Co-author, “Upgrading Data Warehouse Infrastructure at Airbnb” |
| Kerim Oktay | Airbnb | Co-author, “Upgrading Data Warehouse Infrastructure at Airbnb” |
| Xu Zhang | Airbnb | Co-author, “Upgrading Data Warehouse Infrastructure at Airbnb” |
| Jinyang Li | Software Engineer, Airbnb | Co-presenter, “Trino workload management,” Trino Summit 2023 |
| Tingting Ma | Airbnb | Co-presenter, “Trino workload management,” Trino Summit 2023 |
Key takeaways for your own Iceberg implementation
These are the patterns from Airbnb’s own account that I’d point any team planning a Hive-to-Iceberg migration toward:
- Partition-spec evolution can replace a full rewrite, not just a schema tweak. Airbnb consolidated two parallel tables into one by changing the partition spec in place and compacting, rather than rewriting history - a greater than 90% resource saving that a full rewrite wouldn’t have offered.
- Moving partition metadata out of the Hive Metastore is often the real win, not the table format itself. Airbnb’s bottleneck was metastore load under partition-heavy queries; Iceberg’s separate metadata layer, not any single query-performance feature, is what relieved it.
- Design your orchestration layer’s Iceberg awareness before you migrate, not after. Airbnb’s Airflow sensors needed a deliberate redesign to sense Iceberg partitions correctly, and that design conversation happened on a public mailing list a year and a half before the migration was published as complete.
- Adaptive Query Execution and the table-format migration are separable wins that compound. A meaningful share of Airbnb’s compute and runtime savings came from Spark 3’s AQE tuning shuffle partitions dynamically, independent of the Iceberg switch itself - both are worth measuring on their own.
Airbnb’s account only covers one pipeline in detail, so there’s more of their platform I’d like to see written up; I’ll keep reading whatever they publish next.
Sources and further reading
Primary sources
- Ronnie Zhu, Edgar Rodriguez-Diaz, Jason Xu, Gustavo Torres, Kerim Oktay, Xu Zhang, “Upgrading Data Warehouse Infrastructure at Airbnb” - https://medium.com/airbnb-engineering/upgrading-data-warehouse-infrastructure-at-airbnb-a4e18f09b6d5 (2022)
- Hao Wang, “Scaling Spark Streaming for Logging Event Ingestion” - https://medium.com/airbnb-engineering/scaling-spark-streaming-for-logging-event-ingestion-4a03141d135d (2018)
- Gustavo Torres Torres, Apache Iceberg developer mailing list, “Airflow Integration” design discussion - https://www.mail-archive.com/[email protected]/msg01949.html (2021)
- Jinyang Li, Tingting Ma, “Trino workload management” - Trino Summit 2023, https://www.youtube.com/watch?v=qZejzyxT2fo (2023)
- Trino Summit organizers, 2023 speaker lineup announcement - https://trino.io/blog/2023/11/22/trino-summit-2023-nears-lineup.html (2023)
- James Mayfield, Krishna Puttaswamy, et al., “Data Infrastructure at Airbnb” (background only) - https://medium.com/airbnb-engineering/data-infrastructure-at-airbnb-8adfb34f169c (2016)
Enterprise tooling for Apache Iceberg
Airbnb’s migration post is candid about how much of this work was custom engineering: designing partition-spec evolution into a migration path, building Iceberg-aware Airflow sensors from scratch, tuning Spark’s shuffle behavior by hand. That’s the exact gap my team and I are building Iglu to close - a single interface for inspecting Iceberg tables, tracking snapshots, and managing maintenance across catalogs, so teams don’t have to stitch together query engines and CLI tools to answer the same questions Airbnb’s engineers had to script around. Iglu is currently in active development and previewed to select customers. If that’s useful to you, get in touch about early access.