Most Kafka tutorials teach you what a topic is. If you are past that, what you actually need is the production layer: how to change a running cluster without downtime, which signals to trust when something breaks, and which client settings decide whether messages survive. That is this page. For the concepts themselves, start with what Apache Kafka is and come back.
Production and scale details
A production Kafka cluster has to change while it is running. Configuration changes and version upgrades are applied with a rolling restart: brokers restart one at a time while partition leadership moves to in-sync replicas on the remaining brokers, so producers and consumers stay connected throughout. An upgrade that requires stopping the whole cluster is a design failure, not a maintenance window.
Scaling a traditional Kafka cluster is not instant. Adding brokers triggers partition reassignment, which physically copies data between broker disks over the network, and that movement competes with production traffic for bandwidth. Capacity in practice is set by partition count, consumer group design and broker sizing well before it is set by hardware.
Broker tuning operates at three layers. The JVM layer is garbage collection: long GC pauses cause brokers to miss heartbeats and drop out of the cluster. At the operating system layer, file descriptor limits and the page cache do the damage: Kafka holds a file handle per log segment, and a broker at the default OS limit stops accepting connections. On the network, socket buffer sizes cap throughput on high-latency links between brokers and clients.
Production security is layered, not a single control. A hardened deployment isolates the network with VPC isolation, security groups and private listeners, encrypts all traffic with TLS, authenticates clients with SASL or mutual TLS, and authorizes every principal with ACLs or an external authorizer. A multi-tenant cluster additionally needs role-based authorization at the topic, consumer group and connector level, audit logging, and integration with an enterprise identity provider over SAML, OIDC or LDAP.
Most of what pages like this list as “advanced configuration” is really the same discipline applied in different places. Most Kafka problems are not Kafka bugs. They are misconfigurations, missing observability, or reasonable decisions made without full context, and the production topics in this section, upgrades, tuning, security, are where those reasonable decisions get made. The quick-win checks I give teams are unglamorous for exactly that reason: offset reset defaults, poll interval limits, DLQ loops, retention sizing, message size limits, linger.ms, client library choice, transactional producer usage. None of them is an architecture change, and every one has caused a real incident somewhere.
The security layer has its own version of this. The practical failure mode in Kafka authorization is not a breach, it is permission creep: ACLs accumulate over time through broad grants and principals nobody ever revokes, until the ACL list describes the org chart of three years ago. Audit it like you audit retention.
Troubleshooting and operations
Kafka troubleshooting starts from a small set of signals. The four most impactful are under-replicated partitions, ActiveControllerCount, OfflinePartitionsCount, and the consumer lag trend. Broker CPU, memory and network throughput tell you that a problem exists, and rarely tell you what caused it.
An under-replicated partition is a partition whose follower replicas have fallen behind the leader and dropped out of the in-sync replica set. The usual causes are a failing broker, saturated disk I/O, or network congestion between brokers. UnderReplicatedPartitions above zero for more than five minutes is a critical alert condition, because writes to affected partitions are one broker failure away from data loss.
Consumer lag is the distance, measured in offsets, between the newest record in a partition and the position a consumer group has committed. Lag has to be tracked at partition granularity, and reading it right is its own discipline: a group-level total hides one stuck partition behind healthy ones, and the stuck partition is usually the story. Clearing lag safely means first distinguishing a slow consumer, which needs scaling or tuning, from a stuck consumer, which is blocked on a poison message or a rebalance loop and does not improve with more instances.
A dead broker is survivable by design. Every partition it led elects a new leader from the in-sync replicas on other brokers, and clients rediscover leadership automatically. Replacing the broker means bringing up a new one and letting replication rebuild its data, either by reusing the dead broker’s id or by reassigning its partitions. No data is lost provided the replication factor was at least three and producers wrote with acks=all.
Of everything on this page, disk is the one I want tattooed somewhere. Running out of disk is about the worst state a Kafka cluster can be in, and it is very hard to recover from: you cannot write, you often cannot cleanly delete, and every recovery action needs the resource you have run out of. Retention sizing is not housekeeping, it is the thing standing between you and that state.
The published incident record backs the severity. One post-incident analysis of a Kubernetes-hosted Kafka failure documents a cascading broker crash that pushed consumer lag to 14 hours across 10,000 topics. And the failure modes are not always inside Kafka: PagerDuty’s staging Kafka hosts went intermittently unresponsive for tens of seconds at a time, producing client connectivity failures, under-replicated partitions and leader elections, with the root cause outside the brokers entirely. When the signals fire together like that, resist the reflex to fix Kafka first. Find what the host is doing.
Architecture and code integration
Client configuration decides the delivery guarantee, and the defaults favour throughput over safety. Setting acks=all makes the partition leader wait for every in-sync replica to confirm a write before acknowledging it. Combined with retries and idempotence enabled on the producer, this prevents both message loss and duplication through transient failures. Production producers set these values explicitly rather than trusting defaults.
A schema registry manages the contract between producers and consumers. Every message schema is versioned in one central place and checked for compatibility before a producer can publish a change, so a field removal or type change that would break downstream consumers is rejected at build or publish time instead of discovered in production. In a microservices architecture this is what allows teams to evolve their events independently.
Modern Kafka clusters run in KRaft mode, which replaces ZooKeeper with a Raft quorum built into Kafka itself. The result is one system to deploy, secure and monitor instead of two, and faster controller failover on large clusters.
Prometheus and Grafana are the most common self-hosted monitoring stack for Kafka. Broker-side alerts cover under-replicated partitions, active controller count and offline partitions. On the producer side, the alerts that matter are delivery failures, back pressure, queue delay and retry rate, each mapping to a distinct failure mode in the client rather than the cluster.
I gave a talk built entirely on this section’s theme: four real production incidents, every one of them survivable with the configuration and monitoring described above, none of them survived gracefully at the time. The pattern across all four was identical. A reasonable decision made without full context, a missing or misread signal, and then a response, usually scaling Kafka itself, that made the underlying problem worse before the real cause surfaced.
That is why my advice on this section is to treat alert rules as code you ship, not dashboards you admire. Concrete thresholds beat vibes: a producer retry-rate alert firing when the five-minute rate exceeds 10 for three minutes tells you something specific is wrong at the client, before the broker-side graphs move at all. The delivery numbers achievable when the whole chain is configured deliberately are real: DoorDash holds data loss under 0.001% in async mode, at billions of messages a day. Not because Kafka is magic. Because every one of the settings on this page was chosen on purpose.
FAQ
How hard is Kafka to learn?
Kafka’s concepts take an afternoon. A topic is an append-only log, a partition is its unit of parallelism and ordering, producers write, and consumer groups share the reading. What takes longer is operating it: the failure modes, the configuration interactions and the recovery procedures this page covers. The learning curve is operational, not conceptual, which is why a production tutorial spends its time on upgrades, troubleshooting and monitoring rather than on the API.
Is Kafka a part of DevOps?
Kafka is infrastructure, not a DevOps tool. It is typically owned by a platform or data engineering team and operated with the same discipline DevOps applies elsewhere: configuration under version control, changes through CI, monitoring from day one. Application teams own their producers and consumers; the cluster itself is a shared platform with an operations rota.