Chad Harris
- Using Apache Kafka since 0.8.0 (2012)
- 18 years across engineering, architecture and engineering leadership
- 6+ years at Square / Block, Engineering Manager to Engineering Leader
- PCI-compliant, high-volume transactional platforms
In short
Chad Harris is a Solutions Architect at Factor House who has worked with Apache Kafka since 2012 and previously led engineering on PCI-compliant payments systems at Square (Block).
Chad Harris is a Solutions Architect at Factor House, bringing 18 years of experience across software engineering, application architecture, and engineering leadership. He has deep hands-on expertise with Apache Kafka, high-volume transactional systems, and PCI-compliant architectures, most recently as an Engineering Leader at Block (formerly Square). At Factor House, Chad works directly with global enterprise customers to help them improve how they manage, govern, and observe their real-time data.
Expertise
Chad's expertise spans distributed systems, real-time data infrastructure, and enterprise application architecture. He has over six years of production experience with Apache Kafka and extensive background in NoSQL data stores including DynamoDB and Cassandra. His specialisation includes building highly scalable, tokenised security systems and high-volume transactional platforms built to PCI compliance standards. Beyond the technical, Chad is a seasoned engineering leader with a track record of mentoring teams and applying agile and lean methodologies in ways that deliver practical business outcomes.
Experience
Chad is currently a Solutions Architect at Factor House. Before that, he spent over six years at Square (now Block), progressing from Engineering Manager to Engineering Leader. Prior to Square, he was Head of Engineering at Verrency, a payments technology company, where he led engineering across a high-security, high-availability fintech platform. Earlier roles include Technical Lead and Senior Software Engineer positions at Reece Australia, IOOF Holdings, National Australia Bank, and AIA, as well as consulting engagements across financial services and government.
Education
- University of Newcastle
Talks, appearances and mentions
Work published, hosted or co-presented by someone other than Factor House.
- Talk June 10, 2026 · Factor House and Aiven
Things that go bump in the night: Kafka operational issues
co-presented with Hugh Evans, Aiven
Real-world Kafka operational failures, from subtle misconfigurations to full-scale incidents, and the debugging workflows that help. Co-presented with Aiven, whose audience the session was run for.
- Video August 14, 2025 · Factor House
Chad Harris on Factor House's software
recorded while at Block, as a customer
Recorded while Chad was at Block, before he joined Factor House: a practitioner's account of running the tooling as a customer.
Writing and talks for Factor House
- Video September 16, 2026 · Factor House
Apache Kafka RBAC & multi-tenancy: Kpow demo
Scoping a virtual cluster view down to a single team's topics and metrics in Kpow, governed by SSO and fine-grained role-based access control.
- Video September 16, 2026 · Factor House
Apache Kafka broker monitoring & configuration: Kpow demo
Monitoring broker disk, throughput and replication health, filtering and exporting broker/topic tables, editing broker configuration under RBAC, and managing KRaft controller info and ACLs in Kpow.
- Video September 16, 2026 · Factor House
Apache Kafka topic management: Kpow demo
Truncating data by offset, electing leaders, increasing partitions, break-glass configuration changes like retention, and topic-level ACLs and reassignments in Kpow.
- Video September 16, 2026 · Factor House
Apache Kafka consumer group monitoring & lag: Kpow demo
Tracking consumer group stability over time, breaking lag down to the partition level, safely resetting offsets on a running group, and using group topology to trace lag back to a host or topic in Kpow.
- Video September 16, 2026 · Factor House
Apache Kafka Connect monitoring & task management: Kpow demo
Monitoring connector and task state, historical health charts, deploying new connector instances from the UI, and filtering to bulk-restart a subset of Kafka Connect tasks in Kpow.
- Video September 16, 2026 · Factor House
Apache Kafka schema registry management: Kpow demo
Viewing and editing schemas, creating new revisions, updating compatibility settings, and creating or deleting subjects across Confluent, Karapace, MSK, and other schema registries in Kpow.
- Video September 16, 2026 · Factor House
Apache Kafka cluster health monitoring: Kpow demo
An overall cluster health score tracked over time, a real example of partition data skew from a poor partitioning key, cost-saving cleanup signals, and what's coming next for Signals in Kpow.
- Video September 16, 2026 · Factor House
Apache Kafka data inspection & search: Kpow demo
Filtering topic data with kJQ, running high-volume streaming searches with no scan limit, narrowing scans by partition or key, and downloading, cloning, or producing result sets to other topics in Kpow.
- Video September 16, 2026 · Factor House
Apache Kafka data masking & PII protection: Kpow demo
Last-four, full, and email-domain masking rules applied during data inspection, and the data policy playground for testing redaction rules like show-first and hashing in Kpow.
- Video September 16, 2026 · Factor House
Apache Kafka tenant configuration: Kpow demo
Admin versus team-member tenant access, scoping a team to a single tenant by default, and configuring tenancy through a simple YAML file in Kpow.
- Video September 16, 2026 · Factor House
Apache Kafka audit logging: Kpow demo
How the audit log lives on a Kafka topic you can inspect directly, tracing an entry back to the exact query and data it exposed, and forwarding audit events to Slack, Teams, or a SIEM via webhook in Kpow.
- Video September 16, 2026 · Factor House
Apache Kafka access policies & SSO: Kpow demo
How access policies map out what a role can do, and how SSO integration with OAuth, SAML, Entra ID, and other providers drives permission assignment from existing identity roles and groups in Kpow.
- Video September 16, 2026 · Factor House
Apache Kafka temporary access & just-in-time permissions: Kpow demo
Configuring time-boxed, API-driven permissions that expire automatically, and wiring temporary policies to a service desk like Jira or ServiceNow for just-in-time approval workflows in Kpow.
- Video September 11, 2026 · Factor House
Kpow CLI, terminal UI, and agentic skills
A first look at Kpow's new CLI, terminal UI, and the agentic skills that let an AI assistant query, diagnose, and operate Kafka through Kpow under your existing SSO and RBAC controls.
- Article August 19, 2026 · Factor House
Dead letter queues in Kafka: a consumer-side, Kafka-only approach
The DLQ pattern for plain Kafka consumers: Kafka-only topics scoped per consumer group, replay through a retry topic rather than the main topic, and why using retries to ride out a downstream outage is the wrong move.
- Video August 7, 2026 · Factor House
Kpow Signals: automated operational insights
Introducing Signals, a new Kpow feature that continuously monitors your Kafka clusters for misconfigurations and early warning signs across brokers, topics, and consumer groups.
- Article June 27, 2026 · Factor House
Kafka UI: The Ultimate Guide
A 27-minute guide to what a Kafka UI is for, and what operators actually need visual control over: topics, consumers, brokers and connectors.
- Talk June 25, 2026 · Factor House webinar
Kafka operational issues and how to survive them
Four real production incidents, walked through end to end: a consumer group with tens of thousands of idle members overwhelming its coordinator, a config change that silently failed to roll back and filled a broker's disk six months later, a partition increase that left messages unread, and a 20-minute poll interval that turned one stuck batch into a 2am page.
The recording of the 25 June run. The same talk also ran on 29 April, and on 10 June with Aiven (both listed above).
- Video May 29, 2026 · Factor House
Data governance for Apache Kafka: lineage support in Factor Platform
A walkthrough of the OpenLineage metadata support for Apache Kafka in Factor Platform, presented to camera.
- Article May 29, 2026 · Factor House
Data governance for Kafka: introducing lineage support in Factor Platform
How OpenLineage metadata makes data ownership, PII classification and lineage visible by default in a Kafka environment.
- Talk April 29, 2026 · Factor House webinar
Kafka production failures, live tech talk
The first run of the Kafka production failures talk, delivered for a North American audience.
Registration page for a session that has ended.
- Post April 8, 2026 · LinkedIn
On StreamNative forking Kafka: why the Ursa engine changes the math
50 reactions, 2 comments, 3 reposts
"I always liked the idea of Pulsar and the Ursa engine. But there was just one problem with it for me: it wasn't Kafka." On why competing with the Kafka ecosystem is a Sisyphean task, and what changes once a challenger implements the Kafka protocol instead.
- Post March 27, 2026 · LinkedIn
Four questions to ask your platform team after the IBM–Confluent deal
80 reactions, 8 comments, 7 reposts
"You don't need to migrate anything today. But you should know your exposure." Four questions on Confluent-specific features versus standard Kafka APIs, what would break on a move to MSK, Redpanda or Aiven, and how much CI/CD and monitoring is coupled to Confluent tooling.
- Article March 26, 2026 · Factor House
What the IBM Confluent acquisition means for Kafka users
Assessing lock-in risk across Schema Registry, managed connectors and operational tooling after IBM's $11B acquisition.
Chad also posts on LinkedIn.
Talks
Kafka Migrating to open source Kafka: cutting TCO without the operational burden
Chad Harris (Factor House) and Justin George (NetApp Instaclustr) work a real total-cost-of-ownership comparison across Confluent, Amazon MSK, and managed open source Kafka.
September 9, 2026
Kafka Reduce Kafka spend and operational risk: practical techniques for cluster consolidation
In this recorded webinar, Karel Sague and Chad Harris from Factor House share a practical framework for cutting Kafka costs and operational risk through cluster consolidation and migration.
August 27, 2026
Kafka Kafka operational issues: how to survive them
In this recorded session, Chad Harris, Solutions Architect at Factor House, walks through real Kafka operational failures and the debugging workflows that actually help in production.
June 25, 2026Latest articles
MirrorMaker 2 on Connect: migrating off connect-mirror-maker.sh
Moving MirrorMaker 2 off connect-mirror-maker.sh onto Kafka Connect: replication progress doesn't migrate, offset translation is off by default, and when not to bother.
September 16, 2026Apache Flink: the complete guide
Apache Flink is a distributed stream processing framework for stateful computation over unbounded and bounded data. This hub indexes what we have written about running it in production.
September 5, 2026Flink use cases
Companies running Apache Flink in production, the architectures behind their deployments, and what to read next. Indexed as we publish new use-case research.
September 5, 2026Kpow vs AKHQ
AKHQ is free and costs operator time. Kpow is licensed per cluster with a published price. How the two differ on masking, audit, and support.
August 30, 2026Kpow vs CMAK
CMAK connects through ZooKeeper and stops at Kafka 4.0. Kpow bills per cluster and talks to brokers. What each one costs, and which one fits your team.
August 30, 2026Kpow vs Confluent Control Center
Control Center needs a Confluent JAR in the broker classpath. Kpow runs against any distribution, per cluster, at a published price. Which fits which team.
August 30, 2026Kpow vs Kadeck
Kadeck bills per user, Kpow bills per cluster. What each one costs, what each needs in order to start, and which one fits your team.
August 30, 2026Kpow vs Kafbat UI
Kafbat UI is free and carries real RBAC, masking and an audit log. Kpow is licensed per cluster with support behind it. Which fits which team.
August 30, 2026Kpow vs Kafdrop
Kafdrop is free with no tier above it. Kpow is licensed per cluster. What each one does well, where each runs out, and which fits your cluster.
August 30, 2026Kpow vs Lenses.io
Lenses.io bills by capability with a user cap. Kpow bills per cluster. How the pricing, the control plane and the exit differ, and which fits which team.
August 30, 2026Kpow vs Offset Explorer
Offset Explorer is licensed per named user and installed on a laptop. Kpow is licensed per cluster and shared. What each costs, and which fits which team.
August 30, 2026Kpow vs Redpanda Console
Redpanda Console is free, and its governance is licensed to the broker vendor. Kpow prices per cluster and publishes the price. Which one fits which team.
August 30, 2026Apache Kafka vs Confluent Kafka
The broker is the same. What differs is licensing, bundled components, deployment models and support. Where the Apache line sits, what Confluent adds, and the dependency questions to ask.
August 29, 2026Kafka-compatible cloud brokers
Redpanda and the Kafka-compatible brokers, evaluated from the operator's seat: compatibility proof, operational claims, TCO against tiered-storage Kafka, and benchmarks.
August 29, 2026Confluent Kafka in Docker
Running Confluent's Kafka images and the official apache/kafka image in Docker: a working compose file, the advertised-listeners fix, log levels and custom Connect images.
August 29, 2026Data governance policy examples for streaming platforms
Worked data governance policy examples for Kafka: schema compatibility rules, topic lifecycle, PII classification tiers, ACL and RBAC templates, and lineage requirements.
August 29, 2026Debezium vs Kafka Connect
Debezium and Kafka Connect are not alternatives: Debezium is a CDC connector family that runs on Connect. The real decisions are log-based CDC vs JDBC polling, and Connect cluster vs Debezium Server.
August 29, 2026Kafka deployment automation
Deployment automation for Kafka: zero-downtime rolling changes, declarative topics and ACLs, Kubernetes operators, Terraform and GitOps for cluster state.
August 29, 2026The difference between Kafka and RabbitMQ
The difference between Kafka and RabbitMQ is the data model: a replayable log against a delete-on-acknowledge queue. Storage, routing, scaling and when to pick which.
August 29, 2026How RabbitMQ works
How RabbitMQ works, mapped to Kafka concepts: exchanges and bindings, per-message acknowledgement, competing consumers, quorum queues and dead lettering.
August 29, 2026Kafka ACL
A Kafka ACL allows or denies a principal an operation on a resource from a host. Syntax, copy-paste commands, production best practices and GitOps automation.
August 29, 2026Kafka authentication
Kafka authentication verifies every client and broker connection with SASL or mutual TLS. Listener and JAAS blueprints, mechanism choice, rotation and troubleshooting.
August 29, 2026Kafka consumer groups
A Kafka consumer group shares the work of consuming a topic, one partition per member. Troubleshooting lag and rebalances, the CLI cheat sheet, coordinator architecture and offset commits.
August 29, 2026Kafka Connect
Kafka Connect moves data between Kafka and external systems through source and sink connectors. What a deployment is made of, the operational tasks, and where the sub-pages go deeper.
August 29, 2026Kafka Connect MongoDB example
A production Kafka Connect MongoDB example in both directions: source connector via change streams, sink connector with idempotent writes, DLQ settings and secrets handled properly.
August 29, 2026Kafka Connect pricing
Kafka Connect is free open source software. The cost is in running it: managed per-task and per-GB billing, network surcharges, or the engineering hours of self-hosting. The TCO math, honestly.
August 29, 2026Kafka Tool download
Kafka Tool is the former name of Offset Explorer. What each download actually installs, the production jobs a Kafka GUI has to do, and the enterprise constraints to check before connecting one.
August 29, 2026Kafka UI and console comparisons
Every major Kafka UI and console compared head to head: Kpow against each, and each against the others. Pricing, plan limits and where each one runs out.
August 29, 2026Kafka vs other brokers
How Kafka compares with RabbitMQ and other message brokers: retention models, routing, use cases and operational trade-offs, from the operator's seat.
August 29, 2026Kafka vs RabbitMQ performance
Kafka and RabbitMQ perform differently because their storage models differ. Throughput and latency behaviour, durability trade-offs, benchmark methodology and the workload facts that decide the fit.
August 29, 2026Managed vs unmanaged databases
Managed vs unmanaged databases for teams running Kafka: operational overhead, SLAs, cost architecture, connector and CDC integration, control and security.
August 29, 2026Multi-tenant architecture
A multi-tenant Kafka architecture shares one cluster across teams with quotas, ACLs and naming conventions. Isolation, namespaces, chargeback and topology.
August 29, 2026RBAC roles
RBAC roles bundle permissions into named sets like viewer, operator and admin. How role definitions, resource patterns and operation mappings work across the Kafka ecosystem.
August 29, 2026Kafka stream governance
Stream governance applies data governance to data in motion: schemas enforced at produce time, lineage across topics and jobs, catalogs, and access and quality rules on live streams.
August 29, 2026What is a data governance policy?
A data governance policy is an enforceable rule for how data is structured, accessed, retained and traced. On Kafka it is implemented as configuration and code, not documents.
August 29, 2026What is envelope encryption?
Envelope encryption encrypts data with a local data key, then wraps that key with a KMS-held key. On Kafka it is the pattern that makes per-field encryption work at full throughput.
August 29, 2026What is Kafka Connect?
Kafka Connect streams data between Kafka and other systems as managed, fault-tolerant tasks. The internal architecture, production scaling, error handling and custom development.
August 29, 2026What is Kafka rebalancing?
Kafka rebalancing redistributes a consumer group's partitions when membership changes. The triggers, the three timeouts, cooperative rebalancing, KIP-848 and the metrics that explain incidents.
August 29, 2026What is RabbitMQ?
RabbitMQ explained for Kafka operators: the AMQP queue model, broker-side routing, push delivery, streams, and where it fits beside a Kafka deployment.
August 29, 2026What is Redpanda?
Redpanda reimplements the Kafka wire protocol in a single C++ binary. The architecture, the drop-in compatibility boundaries, and how to evaluate it against tiered-storage Kafka.
August 29, 2026Kafka: The Complete Guide
Apache Kafka is a distributed event streaming platform that stores ordered, replayable records in partitioned topics. This hub covers Kafka fundamentals, operations, governance and tooling.
August 21, 2026
Guides Apache Kafka architecture: a complete guide
A complete guide to Apache Kafka architecture: internals, components, KRaft, replication, consumers, Connect, Streams, and deployment options.
August 21, 2026Kafka brokers in production
A Kafka broker is the server that stores partition logs and serves produce and fetch traffic. Configuration, troubleshooting, JMX metrics, maintenance and network tuning for production.
August 21, 2026Kafka consumers in production
A Kafka consumer reads records from topic partitions, tracking its own offset. The configuration that decides message loss, rebalance troubleshooting, and poll-loop patterns that survive production.
August 21, 2026Kafka in Docker
Running Kafka in Docker: the official images, reliable Docker Compose topologies, the advertised.listeners trap that breaks local connections, and why a single-node container is not a deployment.
August 21, 2026Kafka offsets
A Kafka offset is a record's position in its partition, and a committed offset is a consumer group's bookmark. Reading lag from CURRENT-OFFSET and LOG-END-OFFSET, plus the reset strategies.
August 21, 2026Kafka producers in production
A Kafka producer appends records to topic partitions. Configuration and tuning with real numbers, idempotence and delivery guarantees, and the client-library decision that quietly matters most.
August 21, 2026Kafka Streams
Kafka Streams is a Java library for stream processing that runs inside your application, with state in local RocksDB stores and changelog topics. The operational realities, and where Flink wins.
August 21, 2026Kafka Streams documentation, mapped
The Kafka Streams documentation divides into four layers: API reference, configuration surface, state store internals, and the upgrade guide. Which layer answers which production question.
August 21, 2026Kafka topics
A Kafka topic is a named, append-only log, and in production it is the unit you operate on. The CLI verbs, partition and replication mechanics, retention, compaction and the troubleshooting moves.
August 21, 2026A Kafka topic example, fully specified
A production Kafka topic example: the exact creation command with durability properties, the same topic as Terraform and Strimzi code, the naming convention that scales, and the schema contract.
August 21, 2026Kafka topic vs partition
A topic is the logical name; a partition is the physical log. How replication, ordering, parallelism and key hashing follow the physical unit, and why keyed topics lock their count.
August 21, 2026A production Kafka tutorial
A Kafka tutorial for people who run it in production: zero-downtime upgrades, broker tuning, layered security, troubleshooting signals, and the client settings that decide delivery guarantees.
August 21, 2026Kafka use cases
The production Kafka use cases with the numbers behind them: real-time analytics, event-driven microservices, change data capture, event sourcing and log aggregation, plus the business case for each.
August 21, 2026Kafka KRaft
KRaft replaces ZooKeeper with a Raft quorum built into Kafka. The migration path and deadlines, controller sizing, the real scale limits, and day-2 operations for KRaft clusters.
August 21, 2026ksqlDB on Kafka
ksqlDB is the streaming SQL layer for Kafka: streams and tables in SQL, persistent queries with state in internal topics. The push-vs-pull trap, and where it stands as Confluent shifts to Flink.
August 21, 2026Kafka with Spring Boot
Spring Boot integrates with Kafka through spring-kafka: KafkaTemplate, @KafkaListener and auto-configuration. Production setup, dead letter topics, non-blocking retries, and listener tuning.
August 21, 2026Dead letter queues in Kafka: a consumer-side, Kafka-only approach
How to implement dead letter queues in Kafka: consumer-side, Kafka-only, scoped per consumer group, and why retry topics for transient failures are an anti-pattern.
August 19, 2026What is Apache Kafka?
Apache Kafka is an open-source distributed event streaming platform that stores records in ordered, partitioned, replayable logs. This is what it is, how the pieces fit, and when to choose it.
August 13, 2026Best free Kafka UI tools in 2026
Compare the best free Kafka UI and management tools in 2026: Kpow Community Edition, Conduktor Console Community, Lenses Community Edition, AKHQ, and Kafbat UI.
August 1, 2026
Comparisons Kafdrop: pricing and alternatives
Kafdrop review for 2026: strengths, limitations, pricing, and the best alternatives for platform and data engineers running production Kafka clusters.
June 27, 2026
Guides Kafka dashboard: features that matter
A Kafka dashboard gives you real-time visibility into consumer lag, broker health, and partition state. Here's what to look for and how Kpow delivers it in production.
June 27, 2026
Guides Kafka management console: what to look for
A Kafka management console gives your team full control of topics, consumers, schemas, and connectors from one UI. See what to look for and how Kpow delivers it.
June 27, 2026
Guides Kafka message key best practices
A technical guide to Kafka message key best practices covering partitioning, ordering guarantees, hot keys, log compaction, and serialization for production systems.
June 27, 2026
Guides Kafka security architecture for production
Kafka ships insecure by default. Learn how to build a production-ready Kafka security architecture covering TLS encryption, SASL authentication, ACLs, audit logging, and network isolation.
June 27, 2026
Comparisons Kafka UI: The Ultimate Guide
A Kafka UI is a web interface for managing Apache Kafka, giving operators visual control over topics, consumers, brokers, and connectors without the CLI.
June 27, 2026
Guides Dead letter queues in Kafka: patterns and pitfalls
How to implement a dead letter queue in Apache Kafka, with Spring Kafka, Connect, and Streams examples, and the production failure modes to avoid.
June 26, 2026
Comparisons Best Kafka management tools for 2026
Compare the 10 best Kafka management tools for 2026, including Kpow, AKHQ, Conduktor, and Confluent Control Center. Covers pricing, RBAC, and deployment requirements.
June 17, 2026
Comparisons Best Kafka monitoring tools for 2026
Compare 12 Kafka monitoring tools for 2026, from enterprise-grade Kpow to open-source AKHQ and Prometheus. Covers deployment, pricing, and key trade-offs.
June 17, 2026
Guides Kafka broker monitoring
How to monitor Kafka brokers: key JMX metrics, alerting thresholds, process monitoring scripts, and common issues with step-by-step diagnosis.
June 4, 2026
Guides Kafka cluster monitoring
What to monitor at the Kafka cluster level: key JMX metrics, multi-broker collection, alerting thresholds, capacity signals, and a health check script.
June 4, 2026
Guides Kafka consumer monitoring and performance tuning
Learn which Kafka consumer metrics matter most, how to interpret them, and which configuration changes will improve performance and reduce lag.
June 4, 2026
Guides Kafka monitoring: a guide for platform engineers
A practical guide to Kafka monitoring for platform engineers: the metrics that matter, alert thresholds, JVM tuning, consumer lag, and KRaft changes.
June 4, 2026
Guides A detailed guide to Kafka producer monitoring
A practical guide to Kafka producer metrics, JMX collection, alerting thresholds, and diagnostic scripts for Java-based Kafka producers.
June 4, 2026
Kafka How Adidas uses Apache Kafka in production
A deep-dive into Adidas's Kafka architecture — covering observability at 100 billion messages per day, self-service topic provisioning, and custom GoLang tooling.
June 2, 2026
Kafka How Apple uses Apache Kafka in production
A deep-dive into Apple's Kafka architecture — covering their managed internal platform, Strimzi on EKS, tiered storage, zero-data-movement balancing, and mTLS migration.
June 2, 2026
Kafka How Barclays uses Apache Kafka in production
A deep-dive into Barclays' Kafka architecture — covering dual-environment deployment on AWS and IBM Z-Linux, operating practices, and the broader streaming stack.
June 2, 2026
Kafka How Datadog uses Apache Kafka in production
A deep-dive into Datadog's Kafka architecture — covering use cases, scale, engineering decisions, and key contributors across hundreds of clusters.
June 2, 2026
Kafka How Netflix uses Apache Kafka in production
A deep-dive into Netflix's Kafka architecture, covering the Keystone pipeline, Data Mesh platform, scale figures from 700 billion to 2 trillion events per day, and the engineering decisions behind it.
June 2, 2026
Kafka How New Relic uses Apache Kafka in production
A deep-dive into New Relic's Kafka architecture — covering use cases, scale, engineering decisions and key contributors.
June 2, 2026
Kafka How Notion uses Apache Kafka in production
A deep-dive into Notion's Kafka architecture — covering use cases, scale, engineering decisions, and key contributors across their data lake and AI pipelines.
June 2, 2026
Kafka How PagerDuty uses Apache Kafka in production
A deep-dive into PagerDuty's Kafka architecture, covering event ingestion, notification scheduling, task execution, and the engineering decisions behind each.
June 2, 2026
Kafka How Pinterest uses Apache Kafka in production
A deep-dive into Pinterest's Kafka architecture — covering use cases, scale, engineering decisions, and key contributors. From 15 million to 40 million messages per second across 3,000 brokers.
June 2, 2026
Kafka How Salesforce uses Apache Kafka in production
A deep-dive into Salesforce's Kafka architecture — covering use cases, scale, engineering decisions and key contributors across a fleet of 100+ clusters processing 3+ trillion events per day.
June 2, 2026
Kafka How Shopify uses Apache Kafka in production
A deep-dive into Shopify's Kafka architecture — covering CDC at 100,000 records/sec, Kubernetes deployment, the Sarama Go client library, and BFCM scale engineering.
June 2, 2026
Kafka How Tencent uses Apache Kafka in production
A deep-dive into Tencent's Kafka architecture — covering their federated cluster design, 20 trillion messages per day, KIP contributions, and tiered storage at Tencent Cloud.
June 2, 2026
Kafka How Wix uses Apache Kafka in production
A deep-dive into Wix's Kafka architecture: 66 billion daily messages, 2,200+ microservices, the Greyhound SDK, Confluent Cloud migration, and operating 500,000+ partitions across 4 regions.
June 2, 2026
Comparisons Conduktor: pricing and alternatives
Conduktor review for 2026: pricing, strengths, deployment trade-offs, and how it compares to alternatives for enterprise Kafka governance teams.
June 1, 2026
Kafka How Airbnb uses Apache Kafka in production
A deep-dive into Airbnb's Kafka architecture — covering six production systems, 35+ billion daily events, SpinalTap CDC, Flink-based personalisation, and Kafka as a write-ahead log.
May 30, 2026
Kafka How Bytedance uses Apache Kafka in production
ByteDance ran Kafka at tens of TB/s before replacing it with ByteMQ, a Kafka-compatible platform separating storage from compute. How it works, and why migrating cut resource cost by roughly 70%.
May 30, 2026
Kafka How Cloudflare uses Apache Kafka in production
A deep-dive into Cloudflare's Kafka architecture: use cases at trillion-message scale, 14 clusters, internal tooling decisions, and the engineering lessons behind a decade of Kafka operations.
May 30, 2026
Kafka How DoorDash uses Apache Kafka in production
A deep dive into DoorDash's Kafka architecture, covering the Iguazu event platform, Flink-based ML feature pipelines, self-serve topic governance, and hundreds of billions of daily events.
May 30, 2026
Kafka How Goldman Sachs uses Apache Kafka in production
A deep-dive into Goldman Sachs's Kafka architecture — covering use cases across three divisions, migration to Amazon MSK, resilience design, and key engineering decisions.
May 30, 2026
Kafka How Grab uses Apache Kafka in production
A deep-dive into Grab's Kafka architecture — how the Coban team built a terabyte-per-hour streaming platform serving 300 billion events a week across GrabFood, GrabPay, mobility, and more.
May 30, 2026
Kafka How JPMorgan uses Apache Kafka in production
A deep dive into JPMorgan Chase's Kafka architecture, covering multi-tenant cluster design, managed Kafka Connect, the Photon Framework, and decisions behind a large-scale deployment.
May 30, 2026
Kafka How LinkedIn uses Apache Kafka in production
A deep-dive into LinkedIn's Kafka architecture, covering use cases, scale, engineering decisions, and key contributors.
May 30, 2026
Kafka How PayPal uses Apache Kafka in production
A deep-dive into PayPal's Kafka architecture — covering use cases, scale, engineering decisions, and key contributors across a fleet handling 1.3 trillion messages per day.
May 30, 2026
Kafka How Reddit uses Apache Kafka in production
A deep-dive into Reddit's Kafka architecture — covering use cases, scale, engineering decisions and key contributors.
May 30, 2026
Kafka How Robinhood uses Apache Kafka in production
A deep-dive into Robinhood's Kafka architecture: use cases, scale, and engineering decisions. Robinhood processes 2.2 million messages per second across equities, crypto, and fraud detection.
May 30, 2026
Kafka How Spotify used Apache Kafka in production
A deep-dive into Spotify's Kafka architecture — covering their event delivery system, 700K events/second scale, engineering decisions, and why they ultimately migrated to Google Cloud Pub/Sub.
May 30, 2026
Kafka How The New York Times uses Kafka
A deep-dive into The New York Times' Kafka publishing pipeline, covering the Monolog architecture, single-partition design, Kafka Streams usage, and treating Kafka as a permanent content store.
May 30, 2026
Kafka How Uber uses Apache Kafka in production
A deep-dive into Uber's Kafka architecture - covering use cases, scale, engineering decisions, and key contributors. From one region to trillions of messages a day.
May 30, 2026
Kafka How Walmart uses Apache Kafka in production
A deep-dive into Walmart's Kafka architecture — covering real-time inventory, fraud detection, the Customer Data Platform, and the Messaging Proxy Service handling trillions of messages per day.
May 30, 2026
Product Data lineage support in Factor Platform
Learn how Factor Platform brings OpenLineage metadata into your Kafka environment, making data ownership, PII classification, and lineage visible by default.
May 29, 2026
Guides Kafka scaling best practices: An in-depth primer
A practical guide to scaling Apache Kafka in production, covering partitioning strategy, consumer group design, broker sizing, KRaft migration, and more.
May 28, 2026
Comparisons AKHQ: pricing and alternatives
AKHQ review for 2026: features, known limitations, pricing, and the best alternatives for teams that need more than open-source tooling.
May 26, 2026
Comparisons CMAK: pricing and alternatives
CMAK is a free, open-source Kafka admin tool from Yahoo. This review covers features, KRaft limitations, security gaps, and the best alternatives for 2026.
May 26, 2026
Comparisons Confluent Control Center: pricing and alternatives
An honest technical review of Confluent Control Center in 2026, covering features, deployment, pricing, and the best alternatives for Kafka teams.
May 26, 2026
Comparisons Kadeck: pricing and alternatives
Kadeck review for 2026: features, deployment, pricing, and how it compares to AKHQ, Kafbat, Conduktor, and Kpow for Kafka management teams.
May 26, 2026
Comparisons Kafbat UI: pricing and alternatives
A practical review of Kafbat, the open-source kafka-ui fork — covering features, deployment, security, pricing, and best alternatives in 2026.
May 26, 2026
Comparisons Lenses.io review: pricing and alternatives
Lenses.io review for 2026: honest assessment of SQL Studio, deployment complexity, pricing, and when to consider alternatives like Conduktor or Kpow.
May 26, 2026
Comparisons Redpanda Console: pricing and alternatives
Redpanda Console reviewed for 2026: features, pricing, limitations, and the best alternatives for engineering teams running Apache Kafka or Redpanda.
May 26, 2026
Comparisons Top Kafka UI tools in 2026: a practical comparison
Honest comparison of Kafka UI tools for enterprise teams. We evaluate AKHQ, Kafbat, Redpanda Console, Conduktor, Confluent Control Center, and Kpow.
May 26, 2026
Guides Best practices for Kafka data observability
12 best practices for Kafka data observability covering consumer lag monitoring, schema enforcement, end-to-end auditing, DLQs, and lineage, with an implementation roadmap.
May 18, 2026
Guides Kafka message size best practice
How large should Kafka messages be in production? Covers sizing tiers, the four-config chain, compression codecs, and patterns for handling payloads above 1 MB.
May 18, 2026
Guides Kafka partition key best practices
How Kafka partition keys work, what makes a good key, and practical guidance on cardinality, hot partitions, compaction, cross-language hashing, and safe key migration.
May 18, 2026
Guides Kafka cluster management: a practical guide
A practical guide to Kafka cluster management: architecture sizing, day-to-day operations, performance tuning, KRaft migration, and monitoring for production clusters.
May 11, 2026
Guides Kafka topic partition best practices
Size Kafka topic partitions correctly from day one. Covers the throughput formula, the keyed topic asymmetry, KRaft-era limits, and operational best practices.
May 11, 2026The complete guide to Kafka change data capture
Learn how to implement change data capture with Kafka using Debezium. Includes working PostgreSQL CDC examples, architecture patterns, and monitoring.
May 8, 2026
Industry What the IBM-Confluent deal means for Kafka users
IBM's $11B Confluent acquisition raises questions for Kafka users. Assess your lock-in risk across Schema Registry, managed connectors, and operational tooling.
March 26, 2026
Guides Is your data stack ready for the EU Data Act?
The EU Data Act takes effect in September 2025, with major implications for teams running Kafka. This explores what it means for engineers, and how Kpow can help ensure compliance.
October 30, 2025