Before Factor House I spent eight years as a software engineer at Square, later Block, most of it on streaming and data infrastructure, including a production migration onto Apache Iceberg. This is the fifth in a run of Iceberg write-ups I’ve done, after Netflix, Apple, Airbnb, and Pinterest, and LinkedIn’s account is worth adding to that list for a specific reason: LinkedIn is also the company that built and open-sourced Apache Kafka, so watching how the same engineering culture approaches a completely different problem, table sprawl across a data lakehouse, is a useful comparison. I read nine primary sources directly for this piece: three of LinkedIn’s own engineering blog posts, the official linkedin/openhouse GitHub repository, a peer-reviewed SIGMOD paper LinkedIn co-authored with Microsoft and University of Maryland researchers, and conference talks and a speaker profile from the engineer who has led this work since 2023.
The headline figure is that LinkedIn’s OpenHouse control plane now manages more than 300,000 Apache Iceberg tables holding over one exabyte of data. The engineering problem underneath that number is a familiar one at a different order of magnitude: once a company has enough teams creating their own tables on their own schedules, nobody can answer basic questions like which tables exist, who owns them, whether they’re being maintained, or whether they comply with retention and access policy, without a system that makes table management declarative rather than ad hoc.
Company overview
LinkedIn is a professional networking platform operating at a scale that has historically forced it to build rather than buy much of its data infrastructure, Apache Kafka and Apache Samza both originated there. Its Hadoop-based data lake ingests, by its own account, hundreds of terabytes of data daily from thousands of Kafka topics, and by 2023 an internal review found that 65% of its tables were self-managed, meaning no consistent operational support existed for maintenance, governance, or lifecycle policy. That gap is what led to OpenHouse, a declarative control plane for Iceberg tables that LinkedIn built internally starting in 2022 and open-sourced in 2024.
| Date | Milestone |
|---|---|
| 2021-01 | LinkedIn publishes FastIngest, its first documented production use of Apache Iceberg |
| 2023-07 | LinkedIn publishes “Taking Charge of Tables,” introducing OpenHouse internally |
| 2023-10 | Sumedh Sakdeo presents OpenHouse at a Big Data Meetup hosted at LinkedIn’s own office |
| 2024-03 | LinkedIn open-sources OpenHouse, reporting 3,500+ managed tables and 550+ daily active users |
| 2024-09 | Sumedh Sakdeo presents OpenHouse at Current 2024 in Austin |
| 2025-04 | AutoComp, a joint LinkedIn/Microsoft/University of Maryland compaction paper, is published ahead of SIGMOD-Companion ’25 |
| 2025-04 | Sumedh Sakdeo presents “Taking Charge of Tables with OpenHouse” at Iceberg Summit 2025 |
| 2026-05 | Sumedh Sakdeo reports 300,000+ tables and over one exabyte of data under OpenHouse management at FORMATS’26 |
LinkedIn’s Apache Iceberg use cases
OpenHouse - a declarative control plane for Iceberg tables
OpenHouse is the core of LinkedIn’s Iceberg story: a system that lets teams define a table, its schema, and its metadata declaratively, then reconciles the table’s actual state against that declaration automatically. As Sumedh Sakdeo, Lei Sun, Sushant Raikar, Stanislav Pak, and Abhishek Nath describe, self-service table creation dropped from a 2-3 week onboarding process involving multiple teams to something a developer can do in seconds.
FastIngest - low-latency ingestion from Kafka into Iceberg
Before OpenHouse existed, LinkedIn’s first production Iceberg use case was ingestion. Zihan Li, Sudarshan Vasudevan, Lei Sun, and Shirshanka Das describe FastIngest, an Apache Gobblin-based pipeline that reads from Kafka and writes into Iceberg tables, replacing a batch MapReduce job that had previously been the only way to get read/write isolation on top of Hive.
AI and LLM training data governance
More than 1,000 datasets have been onboarded to OpenHouse specifically to support AI use cases, including large language model training, according to Sumedh Sakdeo’s open-source announcement. The point isn’t that Iceberg does anything AI-specific here, it’s that OpenHouse’s existing governance, retention, and access controls apply to model-training data the same way they apply to any other managed table, without a separate carve-out.
Cross-geography table replication
OpenHouse extends Apache Gobblin with a component the team calls IcebergDistcp, replicating Iceberg tables between geographies at snapshot granularity on a schedule. Replica tables are marked read-only, so writes and updates only ever happen against the primary copy, which keeps LinkedIn from having to reason about conflicting writes landing in two regions at once.
Scale and throughput
| Year | Metric | Source |
|---|---|---|
| 2021 | Kafka-to-lake ingestion latency cut from ~45 minutes to ~5 minutes | FastIngest, LinkedIn Engineering Blog |
| 2023 | 65% of LinkedIn’s tables were self-managed, with no consistent operational support | “Taking Charge of Tables,” LinkedIn Engineering Blog |
| 2024 | 3,500+ managed OpenHouse tables in production; 550+ daily active users; dbt time-to-market cut by over 6 months | “Open Sourcing OpenHouse,” LinkedIn Engineering Blog |
| 2025 | 21,000 tables onboarded to automated compaction, projected to reach 100,000; compaction cut small files (under 128MB) as far as 44% | AutoComp, SIGMOD-Companion ’25 |
| 2026 | 300,000+ tables under OpenHouse management, holding over one exabyte of data | Sumedh Sakdeo, FORMATS’26 |
That’s roughly a 100x growth in managed table count in two years, which is the kind of curve that makes automated maintenance non-optional rather than a nice-to-have. LinkedIn also ingests, by its own account, hundreds of terabytes of data daily from thousands of Kafka topics into the underlying Hadoop lake that most of these tables sit on top of.
LinkedIn’s Apache Iceberg architecture
Table Service and Metastore Catalog
OpenHouse’s Table Service is a RESTful web service running on Kubernetes behind an Envoy proxy, handling table provisioning and declarative metadata management. Its Metastore Catalog exposes Iceberg’s own catalog APIs, so Apache Spark, Trino, and Apache Flink can all read and write the same managed tables without each engine needing its own bespoke integration.
House Database Service and managed namespace
Table state itself, the declared configuration versus the observed reality, is tracked in a House Database Service backed by MySQL and Spring Data JPA. Tables live inside a managed HDFS namespace with a purpose-built permissioning scheme, rather than being scattered across arbitrary paths that individual teams control directly.
Governance and discovery integration
Access control runs through Open Policy Agent, and metadata changes propagate out from OpenHouse, the operational catalog, to DataHub, LinkedIn’s own discovery and governance catalog. That split matters: OpenHouse owns the operational lifecycle of a table, DataHub owns making it findable and understandable to people who aren’t the table’s owner.
Special techniques and engineering innovations
Asynchronous metadata publishing
FastIngest’s central trick is decoupling write throughput from Iceberg commit latency. Instead of committing to the table inline on every write, a separate metadata pipeline consumes Gobblin Metadata Change Events off Kafka, aggregates them, and periodically registers batched commits against the Iceberg table. That’s what let LinkedIn move from a ~45-minute batch cadence to ~5-minute streaming ingestion without running into the contention that frequent inline commits would otherwise cause.
IcebergDistcp for snapshot-consistent replication
Rather than replicating files at arbitrary points in time, IcebergDistcp replicates at snapshot granularity on a schedule, so a replica table’s state always corresponds to a specific, consistent point in the primary table’s history.
Marking replicas read-only is the same instinct behind Kafka’s MirrorMaker, which I’ve spent a fair amount of time operating: never let two copies accept writes for the same logical dataset, because resolving the conflict after the fact costs far more than preventing it up front.
AutoComp’s budget-aware compaction scheduling
Instead of compacting every table on a fixed schedule, AutoComp models the expected benefit of compacting each table and weighs that against a cluster-wide compute budget measured in TB-hours, then dynamically picks which tables to compact each iteration. The paper reports compacting roughly 2,500 tables per iteration inside a 226 TB-hour budget.
I’ve built compaction tooling before, and the part of AutoComp that stands out to me is treating compaction as a resource-allocation problem across the whole table fleet rather than a per-table cron job. That’s a genuinely different framing, and it only makes sense once you’re operating at a scale where compute contention between tables is itself the bottleneck.
Operating Apache Iceberg at scale
Declarative reconciliation as the operating model
OpenHouse’s control plane continuously reconciles the table state a team has declared against the state it observes, and orchestrates whatever data services are needed to close the gap. That removes table maintenance as a task any individual team has to remember to do.
Governance applied by default, not opted into
Column tagging, audit event instrumentation, and Open Policy Agent-driven access rules apply to every table OpenHouse manages, rather than being something a team has to configure for itself.
Open research problems at exabyte scale
At FORMATS’26, Sumedh Sakdeo laid out five open research problems that OpenHouse’s own operators are still working through: query result divergence across engines reading the same Iceberg tables, the cost of duplicating data between online and offline systems via CDC, the safety model needed before LLM-driven agents can write directly to production tables, the coupling between compaction, sort order, and partitioning that AutoComp only partly addresses, and the gap between stream-processing freshness and batch-processing completeness. None of these are solved problems yet, which is worth knowing if you’re assuming a control plane like OpenHouse means every operational question is closed. The agent-guardrails question is the one I’d watch most closely: giving an LLM-driven process write access to a production table without a branch-validate-merge safety net first is the kind of shortcut that looks fine right up until a bad write ships to every downstream consumer.
Challenges and how they solved them
Hive’s lack of read/write isolation blocked low-latency ingest
LinkedIn needed to shrink Kafka-to-lake ingestion latency from tens of minutes to single digits, and the table format it was using stood in the way.
Problem: the existing Hive table format gave no read/write isolation guarantee, so readers could see partially-written data if writes happened too frequently.
Root cause: the previous pipeline avoided this entirely by batching writes far apart in time, which is what produced the ~45-minute latency floor.
Solution: LinkedIn adopted Apache Iceberg for FastIngest, using its snapshot isolation to let asynchronous metadata publishing commit small, frequent batches safely.
Outcome: ingestion latency dropped to about 5 minutes, as reported by Zihan Li, Sudarshan Vasudevan, Lei Sun, and Shirshanka Das on LinkedIn’s engineering blog in January 2021.
Small-file proliferation degraded query performance
As OpenHouse’s table count grew into the tens of thousands, Iceberg’s own write pattern started working against LinkedIn.
Problem: Iceberg’s append-only, log-structured writes across a growing table fleet produced an increasing number of small files, degrading query performance and increasing storage and metadata overhead. A synthetic TPC-DS benchmark in the AutoComp paper showed a 1.53x query performance hit after data modifications, before compaction ran.
Root cause: manual, per-team compaction scheduling couldn’t keep pace with LinkedIn’s table count, and had no way to account for each table’s actual write and query pattern - a baseline of manual compaction alone only brought small files (under 128MB) from 83% down to 62% of files.
Solution: LinkedIn, together with Microsoft and University of Maryland researchers, built AutoComp, an automated, budget-aware compaction framework integrated into OpenHouse’s own data services.
Outcome: production deployment cut the share of small files as far as 44%, and auto-tuned compaction thresholds showed up to a 2x query time reduction on a TPC-DS workload, per Anja Gruenheid, Jesús Camacho-Rodríguez, Carlo Curino, Raghu Ramakrishnan, Stanislav Pak, Sumedh Sakdeo, Lenisha Gandhi, Sandeep K. Singhal, Pooja Nilangekar, and Daniel J. Abadi, SIGMOD-Companion ’25.
Full tech stack
| Category | Tools | Notes |
|---|---|---|
| Table format | Apache Iceberg | Built on Iceberg 1.2.0 per the OpenHouse repository; underpins every managed table OpenHouse governs |
| Batch processing | Apache Spark | Primary compute engine for batch workloads and OpenHouse’s own data services |
| Interactive query | Trino | Ad hoc SQL access to OpenHouse-managed Iceberg tables |
| Stream processing | Apache Flink | Reads and writes OpenHouse-managed Iceberg tables |
| Ingestion and replication | Apache Gobblin | Extended for FastIngest’s Kafka-to-Iceberg pipeline and for IcebergDistcp cross-geography replication |
| Storage | HDFS | Underlying distributed storage for OpenHouse’s managed table namespace |
| Metadata store | MySQL / Spring Data JPA | Backs the House Database Service’s table-state tracking |
| Deployment | Kubernetes / Envoy / Helm | Runs and exposes the OpenHouse Table Service |
| Access control | Open Policy Agent | Enforces table-level governance policy |
| Discovery catalog | DataHub | Receives propagated metadata for search and discovery outside OpenHouse itself |
| File format | ORC | Written directly by FastIngest’s ingestion pipeline |
Key contributors
| Name | Role | Contribution |
|---|---|---|
| Sumedh Sakdeo | Principal Staff Software Engineer, LinkedIn | Led OpenHouse’s development; lead or co-author of both LinkedIn Engineering Blog posts on OpenHouse; co-author of the AutoComp paper; conference speaker at Iceberg Summit 2025, Current 2024, and FORMATS’26 |
| Lei Sun | Founding engineer on OpenHouse, LinkedIn | Co-authored “Taking Charge of Tables” (2023) and FastIngest (2021) |
| Zihan Li | Lead author, FastIngest (2021) | |
| Sudarshan Vasudevan | Co-author, FastIngest (2021) | |
| Shirshanka Das | Co-author, FastIngest (2021) | |
| Stanislav Pak | Co-author, “Taking Charge of Tables” (2023) and AutoComp (2025) | |
| Sushant Raikar | Co-author, “Taking Charge of Tables” (2023) | |
| Abhishek Nath | Co-author, “Taking Charge of Tables” (2023) | |
| Anja Gruenheid | Microsoft | Co-author, AutoComp (2025) |
| Daniel J. Abadi | University of Maryland | Co-author, AutoComp (2025) |
Key takeaways for your own Apache Iceberg implementation
These are the patterns from LinkedIn’s own account that I’d point any team running Iceberg at meaningful scale toward:
- Treat table management as a control-plane problem, not a per-team responsibility. OpenHouse’s core idea, a declarative catalog reconciling desired state against observed state, is what let table onboarding drop from weeks to seconds. If your organization is still relying on individual teams to remember to run maintenance jobs, that’s the gap OpenHouse was built to close.
- Decouple write throughput from commit latency for high-frequency ingestion. FastIngest’s asynchronous metadata publishing is a specific, reusable pattern: batch your metadata commits separately from your data writes if inline commits are causing contention.
- Budget compaction like a shared resource, not a per-table chore. AutoComp’s TB-hour budget model is worth copying directly once you have enough tables that naive fixed-schedule compaction starts competing with itself for compute.
- Build governance into the platform, not into individual pipelines. Column tagging, audit events, and access policy applied at the OpenHouse layer mean a team can’t accidentally skip governance the way they could if it were bolted onto each pipeline separately.
- Don’t assume a mature control plane means every problem is solved. LinkedIn’s own team is still working through open questions on cross-engine consistency and agent-driven writes at 300,000+ tables, which says something honest about where the ceiling actually is right now.
I’ll keep working through the rest of the list and write up the next one as it’s ready.
Sources and further reading
Primary sources
- Zihan Li, Sudarshan Vasudevan, Lei Sun, Shirshanka Das, “FastIngest: Low-latency Gobblin with Apache Iceberg and ORC format” - https://www.linkedin.com/blog/engineering/open-source/fastingest-low-latency-gobblin (2021)
- Sumedh Sakdeo, Lei Sun, Sushant Raikar, Stanislav Pak, Abhishek Nath, “Taking Charge of Tables: Introducing OpenHouse for Big Data Management” - https://www.linkedin.com/blog/engineering/data-management/taking-charge-of-tables--introducing-openhouse-for-big-data-mana (2023)
- Sumedh Sakdeo, “Open Sourcing OpenHouse: A Control Plane for Managing Tables in a Data Lakehouse” - https://www.linkedin.com/blog/engineering/open-source/open-sourcing-openhouse (2024)
- LinkedIn,
linkedin/openhouseGitHub repository - https://github.com/linkedin/openhouse (accessed 2026) - Anja Gruenheid, Jesús Camacho-Rodríguez, Carlo Curino, Raghu Ramakrishnan, Stanislav Pak, Sumedh Sakdeo, Lenisha Gandhi, Sandeep K. Singhal, Pooja Nilangekar, Daniel J. Abadi, “AutoComp: Automated Data Compaction for Log-Structured Tables in Data Lakes” - https://arxiv.org/abs/2504.04186 (2025)
- Sumedh Sakdeo, “Full-Stack Lakehouse Architecture at Exabyte Scale: Lessons and Research Opportunities from LinkedIn’s OpenHouse-Managed Iceberg,” FORMATS’26 - https://dataformats.org/program.html (2026)
- Sumedh Sakdeo, Sessionize speaker profile (Iceberg Summit 2025, Current 2024) - https://sessionize.com/sumedh-sakdeo/ (accessed 2026)
I read every one of these directly, and linked the specific claim to its source the first time it comes up above, rather than leaving attribution to this list alone.
Enterprise tooling for Apache Iceberg
LinkedIn built OpenHouse because nobody else had solved declarative table management at its scale, and most teams adopting Iceberg don’t have the engineering capacity to build their own control plane from scratch. That’s exactly the gap my team and I are building Iglu for at Factor House: enterprise tooling for Apache Iceberg. Iglu is currently in preview with select customers - if the governance and compaction problems LinkedIn describes here sound familiar, request preview access and tell us what you’re running into.