Journey to an open lakehouse with Iceberg

Today I'm going to talk about a real-world experience: a journey to migrate some data assets and workloads over to an open lakehouse, specifically around Apache Iceberg. I'll go through a bit of history on how we got to the lakehouse, look at the table formats that power it, talk about Apache Iceberg as the leading format, walk through the migration my team went through, and close with a look toward the future.

Throughout the last few decades we've moved through different styles of data architecture. From the late 1980s we had data warehouses, a solution to get operational and external data sources into a highly curated, well-defined structure, mainly for business intelligence and reporting. Later, in the 2010s, with the rise of MapReduce from Google, there was a drive to bring in all sorts of sources — not just operational data stores, but images, video, clickstreams, log files — into what's called a data lake, a concept introduced around 2010 by James Dixon of Pentaho. With the rise of Hadoop platforms, the data lake became mainstream, and there was a huge effort to build them.

I worked across at least two of these efforts myself in past organizations, and one thing you keep hearing is how many of them failed in practice. Researching for this talk, I found a paper published a couple of months ago called "What Went Wrong with Data Lakes?", which captured what it called the seven deadly sins of data lakes. Three are most relevant here: ingestion without purpose — uploading all your data to the lake and working out the schema and reporting later; schema avoidance — uploading sources as-is instead of building well-modeled data marts, which created a lot of downstream issues; and governance as an afterthought, since once a lot of data is sitting in object storage you need to know whether it's available for consumption, who owns it, and what its sensitivity is.

So it's not surprising that companies wanted a better solution. The main driver of the next wave was Databricks, with the concept of the data lakehouse: combining the benefits of a data warehouse and a data lake into a single architecture. Both warehouses and lakes had issues around ACID properties — consistent, atomic changes when writing to tables — and a lakehouse gives you the well-defined schemas and ACID transactions you'd expect from a warehouse, while still letting you bring in other data types the way you would in a data lake. That combination gives you governance across use cases that go beyond BI and reporting, into machine learning, streaming analytics, and data science.

Back to the specific experience I want to talk about. In some organizations you'll have not just a data warehouse but also a data lakehouse or data platform, like Databricks or one from your cloud provider. The problem my team ran into, as part of the data platform group, was a lot of data duplication: every application had its own operational data store, and teams wanted to build reporting, analytics, and BI use cases across different data platforms, so some product teams ended up duplicating data across Snowflake and Databricks and maintaining separate replicas. It became hard to know what the right source of truth was for a given use case. That duplication increased storage costs, created vendor lock-in, and made governance more complex — as a data owner with regulated data sitting in both Snowflake and Databricks, you now have to duplicate governance and role-based access control.

With the lakehouse, we started thinking about what a good approach would look like: a single source of definition to govern data quality, ownership, and permissions, one storage layer with multiple compute engines able to access it — in our case Snowflake and Databricks, plus workloads on Apache Spark in AWS — and we wanted to base this on open standards, even where we were otherwise tied to specific vendors.

So we started looking at open table formats, around 2022–2023. At the time, Apache Hive, created around 2010, was the standard table format for data lakes, and it had real issues: no ACID transactions, awkward schema management, poor partitioning and time-travel support, and performance problems. We evaluated Apache Hudi, Apache Iceberg, and Delta Lake as the most mature contenders, looking closely at Iceberg and Delta Lake in particular. This was a genuinely difficult call — sometimes framed as a "war," but really it was a lot of confusion about which format to use. Apache Iceberg was shaping up to be the format that would win out, especially given the openness of its specification, and interoperability layers like Delta UniForm and Apache XTable emerged, letting you write a table in one format and generate corresponding metadata so it could also be read in another. We settled on Apache Iceberg: our data could sit in S3, since we're on AWS, and be queried by multiple compute engines — at the time that meant Snowflake, Databricks, Apache Flink, Athena, and Spark.

Our data was sitting in Snowflake, so for Iceberg we needed to understand its main components: a metadata layer over a set of files, a catalog you need to choose, and the actual data files, which were locked inside Snowflake, so we needed to get that data out into our S3 layer. One thing to consider is the different types of catalogs involved — technical catalogs like Hive's Hive Metastore, Iceberg's own catalog spec, Delta Lake moving toward catalog-managed tables, vendor catalogs like Databricks' Unity Catalog or AWS's Glue Catalog, and business-level catalogs like DataHub, which we were also introducing. Choosing your catalog is one of the most important decisions once you commit to Iceberg. We wanted one that supports Iceberg's published REST specification for catalog access and is supported across multiple platform vendors, plus credential vending — scoped, short-lived credentials instead of static ones — federation across multiple catalogs, role-based access control, and support for branching and versioning. In the end, we went with the AWS Glue Catalog.

Next: how do you actually start ingesting into Apache Iceberg? We were migrating data in Snowflake that originated mostly from Apache Kafka, plus some scheduled batch loads. There are at least nine documented ways to stream Kafka topics into Iceberg — Kafka Connect, Apache Spark, or Apache Flink can all consume from Kafka and write to Iceberg with full support (Spark was actually the first engine to support Iceberg); some Kafka providers now offer ingestion as a built-in feature, like Confluent's TableFlow, WarpStream, Redpanda, and, more recently, Amazon MSK Express brokers; there's Kafka-protocol-native ingestion from providers like AutoMQ and Bufstream; and, as always, you can schedule batch jobs or do change data capture from relational databases using Flink or Debezium. We focused strategically on Apache Kafka, migrating the subset of warehouse data that originated from Kafka into Iceberg as a starting point, using the Iceberg Kafka Connector deployed on our own AWS infrastructure, with the AWS Glue Catalog as the main catalog and AWS Lake Formation as the management layer on top of S3. One interesting choice we made: one Kafka Connect connector per topic, across 1,000+ topics — effectively its own Kafka Connect cluster per topic, and a fair amount of operational overhead to manage, though we automated a lot of it. The alternative is a single Kafka Connect cluster running multiple connectors; we chose full isolation instead.

The next important piece is schema management. If you're using Kafka you likely have a schema registry, but as part of the migration we wanted to retain all the metadata around the Kafka records themselves, so we could backfill, leverage the Iceberg datasets fully, and republish back to Kafka topics if needed. So we used CloudEvents as a standard envelope for our event publishing, with some custom CloudEvent types alongside the standard ones. In Kafka Connect, using converters and single message transforms, every Kafka record in Iceberg ends up carrying the full metadata for that record: a value struct (the structured, typed view of the record using Iceberg types) and value bytes (the raw bytes, if you're publishing Avro or Protobuf to Kafka); event metadata like the event ID and event time; Kafka metadata like the source cluster, topic, partition, offset, timestamp, and headers; and archival metadata about the archiving process itself, like the received-at time. That's a fairly custom schema, but the payoff was that all 1,000+ Kafka topics shared the same schema in the raw layer for Iceberg, which made it much easier for downstream teams to understand and consume the tables.

Once the connector was committing files into Iceberg via the Glue Catalog, we configured Snowflake — which was just starting to support Iceberg at the time — to access the Glue Catalog through a catalog integration, refreshing its view of the catalog periodically. We had a five-minute commit interval into Iceberg, so you have to think about your SLOs in terms of data freshness; Iceberg wasn't built as a streaming solution, it's a table format for large batch-oriented datasets, but in a modern architecture low-latency availability still matters for near-real-time use cases. If your commit interval is too short in a high-throughput streaming context, you accumulate a lot of small files that then need maintaining, so we had to balance data freshness against file volume. My goal was to have the platform handle this automatically, similar to how an application developer doesn't usually think about how data is laid out on disk in a relational database — a DBA handles that. Three years ago that kind of managed maintenance wasn't really available, so we built it ourselves, scheduling Amazon EMR jobs per Iceberg table, with some jitter so they didn't all fire at once and overwhelm the cluster.

We've since been building a product at Factor House called Iglu, which gives visibility into your Iceberg tables — surfacing metrics and problems, particularly around maintenance, especially if you're ingesting from Kafka, plus visibility into the Kafka side. If your commit interval is short, you accumulate lots of small files, and you need to compact them in time so your query engine stays performant, since the more small files you have, the worse your query performance, as all that data lives in S3 and has to be scanned. Right now it integrates with the AWS Glue Catalog and Confluent TableFlow, showing schema versions and diffs across snapshots, and broken down by namespace. Iglu is in early access now at factorhouse.io.

On automation: this depends a lot on how your organization uses infrastructure as code. GitOps is fairly mainstream now, using Git as the source of truth for infrastructure definitions. We built an internal tool for this, using Terraform modules to provision Iceberg tables on AWS with Glue. Product engineers could define Kafka topics declaratively, so the data platform team wasn't a bottleneck for approvals — with thousands of engineers across the org, you really don't want your team to be the single point of approval. Engineers could deploy via GitOps, declare they wanted a new Kafka topic and that they wanted it ingested into Iceberg, which would trigger provisioning of the Iceberg connector, the correct Glue configuration, and the right Lake Formation permissions automatically.

Once you're running a large-scale migration across thousands of topics or tables, you need observability and metrics. Iceberg supports metric reporters you can configure through the connector, pushing commit and scan metrics to your logs, and from the metadata you can also derive table-level metrics, like the percentage of small files — a strong signal of coming query performance problems. During a migration it's important to surface errors clearly: connector task errors, such as a connector failing because of a backward-incompatible schema change; Kafka consumption lag, to catch ingestion into Iceberg falling behind; and data freshness, since your data warehouse sits downstream and only refreshes its view of the catalog periodically — with a five-minute commit interval on top of that, you're looking at up to ten minutes of end-to-end freshness lag in the worst case. Data quality matters too: since we were running the open source Iceberg connector with our own converter and several single message transforms, we had to sample and validate against Kafka to confirm nothing changed in transit.

We were also very intentional about not replicating relational database tables directly into the lakehouse — we wanted that to go through integration events published to Kafka, or Kinesis, or some other commit log, scoped to each service's domain, rather than uploading whole database tables wholesale, especially with hundreds of services each owning its own database. And given the volume of tables, we used a shared responsibility model: product teams were on call for issues they caused themselves, like a misconfigured topic or a backward-incompatible schema change, while the data platform team owned infrastructure failures, monitored separately. That split made the practice sustainable for the platform team long term.

On planning a migration to Iceberg, there are really three strategies: in-place migration, if you're coming from Hive tables, since Iceberg has migration primitives for that; a full rewrite, which isn't practical once you have live consumers and terabytes of data; or — what we did — an online migration: keep the legacy pipeline running into Snowflake, start ingesting from Kafka into new Iceberg tables in parallel, backfill historical data from Snowflake into Iceberg, run completeness and accuracy checks, and gradually cut consumers over via views. Be strategic about sequencing — we migrated 1,000+ topics, but incrementally, prioritized by the most important tables, avoiding all-or-nothing thinking, and planning for backfilling historical data and reprocessing if something goes wrong. We ran into issues with very wide Protobuf schemas in Snowflake — a thousand-plus columns can cause real problems, and AWS Glue has its own limits we bumped into. Enforce schema contracts on your Iceberg tables from day one, especially coming from Kafka, and use vended credentials for your catalogs, since scoped, short-lived access based on role is much easier to manage than static credentials.

During Q&A, an attendee asked about our specific architectural approach toward CDC for Iceberg tables in production. Karel said he hadn't personally run CDC into Iceberg in production, though he's familiar with the space: in his experience, they used Debezium to capture changes from relational databases into Kafka and ingested from there into Iceberg. CDC generates inserts, updates, and deletes, and Iceberg gives you two write modes for handling that — copy-on-write, which rewrites the affected data file with the changes applied, and merge-on-read, where inserts, updates, and deletes are captured separately, with updates and deletes producing what Iceberg calls delete files. If you have, say, 512MB of Parquet data and only one row changes, you don't want to rewrite the whole file for that one update, which is where merge-on-read is the better fit. Besides Debezium, there are tools that ingest straight from relational databases into Iceberg, like AWS DMS, and SaaS options like Fivetran, but Karel said he'd still generally prefer integration events over replicating database tables directly.

Asked whether Iglu was AWS-only and whether Azure and GCP would be supported in future, Karel said Factor House intends to support all the major cloud providers, though right now the focus is AWS Glue and Confluent TableFlow. Asked whether an AI agent might monitor and update Iceberg tables, Karel said that's a real possibility, especially from an observability standpoint, where there's a lot happening industry-wide with AI agents doing monitoring and sometimes remediation — though he'd be cautious about an agent actively updating a table itself, as opposed to monitoring and flagging issues. Asked what else he'd suggest for gathering metrics beyond piping them to Kafka for telemetry, and whether batch pipelines get monitored too, Karel pointed to Iceberg's documented metric reporters, which report commit and scan metrics and can be configured directly in Apache Spark or Flink, on top of what Iceberg's own metadata already surfaces.

Asked whether migrating to Iceberg requires a whole new ingestion layer and new Kafka topic schemas, Karel confirmed you do need a new ingestion layer — in their case, with data locked in Snowflake, they ingested via AWS Glue and S3, though they could instead have used Snowflake's own Iceberg support and kept the data inside Snowflake, which would still have required a new ingestion path since the legacy pipeline wrote straight into Snowflake. But schemas didn't need to change on the Kafka side — in Kafka Connect they enriched the metadata around the record using a single message transform in the converter, without changing the underlying Kafka schema itself. One caveat: there's a big difference between the decisions made three years ago and what's possible today, since Snowflake now has full native support for Iceberg, so a migration done today might look at different options.

I'll close on the medallion architecture — raw ingestion layer, then cleaning and deduplication — which is still very relevant with Iceberg, especially because table partitioning has to match your query access patterns; if you partition one way at ingestion time but consumers need to query a different way, you'll need to revisit that partitioning. And one thing to keep in mind: ultimately, we shouldn't have to think about Iceberg at all — something Ryan Blue, one of Iceberg's creators, said back in 2022. Unfortunately, here in 2026, we still do, though things are converging: Delta Lake and Iceberg are moving toward each other, and the next version of Iceberg, v4, is specifically focused on converging its metadata structure with Delta Lake's, to reduce the remaining friction between the two formats.