News, guides, and engineering deep dives.
Practical guidance on Kafka, Flink, Iceberg, and real-time data.
How Datadog uses Apache Kafka in production
A deep-dive into Datadog's Kafka architecture — covering use cases, scale, engineering decisions, and key contributors across hundreds of clusters.
Apache Kafka architecture: a complete guide to internals, components, and deployment
A complete guide to Apache Kafka architecture: internals, components, KRaft, replication, consumers, Connect, Streams, and deployment options.
How Netflix uses Apache Kafka in production
A deep-dive into Netflix's Kafka architecture — covering the Keystone pipeline, Data Mesh platform, scale figures from 700 billion to 2 trillion events per day, and the engineering decisions behind it.
How New Relic uses Apache Kafka in production
A deep-dive into New Relic's Kafka architecture — covering use cases, scale, engineering decisions and key contributors.
How Notion uses Apache Kafka in production
A deep-dive into Notion's Kafka architecture — covering use cases, scale, engineering decisions, and key contributors across their data lake and AI pipelines.
How PagerDuty uses Apache Kafka in production
A deep-dive into PagerDuty's Kafka architecture, covering event ingestion, notification scheduling, task execution, and the engineering decisions behind each.
How Pinterest uses Apache Kafka in production
A deep-dive into Pinterest's Kafka architecture — covering use cases, scale, engineering decisions, and key contributors. From 15 million to 40 million messages per second across 3,000 brokers.
How Salesforce uses Apache Kafka in production
A deep-dive into Salesforce's Kafka architecture — covering use cases, scale, engineering decisions and key contributors across a fleet of 100+ clusters processing 3+ trillion events per day.
How Shopify uses Apache Kafka in production
A deep-dive into Shopify's Kafka architecture — covering CDC at 100,000 records/sec, Kubernetes deployment, the Sarama Go client library, and BFCM scale engineering.