A Kafka broker is a single Kafka server that stores partition data on disk and serves produce and fetch requests from clients. Brokers form a cluster, replicate partitions between each other, and elect new partition leaders when one fails.
I am a Solutions Architect at Factor House, and most of what follows comes from broker incidents I have either worked or watched closely. This page covers configuration, troubleshooting, monitoring, maintenance, metadata and network tuning, in that order. The wider picture sits in the Kafka architecture guide and the hub.
Configuration parameters
Every broker reads its identity and behaviour from server.properties at startup. Identity comes first: broker.id names the broker uniquely within the cluster, node.id does the same job on KRaft clusters, and log.dirs lists the disks where partition data lives.
The listener pair is the most misconfigured entry in the file. listeners is the address the broker binds. advertised.listeners is the address the broker hands back to clients in metadata responses, and it is the one clients actually connect to. When a client reaches the bootstrap address fine and then times out, the advertised address is almost always the problem, because the broker returned a hostname the client cannot resolve or route to.
Retention is the other block worth knowing cold. log.retention.hours or log.retention.ms bounds retention by time, log.retention.bytes bounds it per partition by size, and log.segment.bytes sets the segment size that retention deletes in units of. Retention applies per topic partition, and a topic-level override always beats the broker default.
The reason I treat broker configuration as an operational surface rather than a set-and-forget file is an incident of ours. An MSK cluster upgrade of ours failed and rolled back, and the provisioned storage throughput reverted while the replica fetcher and IO thread counts we had raised did not, so the config drift ran silently for six months. Nothing was wrong until the cluster came under recovery load, and then everything was. After any rollback, diff the running configuration against your baseline. The cluster management guide covers the wider discipline.
Troubleshooting and error codes
Connection drops usually trace to one of three places. The advertised listener returning an unreachable address, an idle connection reaped by connections.max.idle.ms after ten minutes of silence, or a load balancer in front of the brokers with a shorter idle timeout than the clients expect. A producer retries through these invisibly until it cannot, which is why the first symptom is often a latency spike rather than an error.
Out-of-sync replicas are a health signal, not an error. A follower that has not caught up to the leader within replica.lag.time.max.ms, 30 seconds by default, is dropped from the in-sync replica set, and the partition shows as under-replicated. Brief ISR shrink during restarts is normal. Sustained under-replication points at a broker that cannot keep up, a saturated disk, or a network path problem.
Unclean leader election is the trade nobody should make silently. With unclean.leader.election.enable at its default of false, a partition whose in-sync replicas are all gone goes offline rather than electing a stale follower. Setting it true restores availability at the price of acknowledged writes. If you flip it during an incident, write down what you lost.
Crash loops nearly always start on disk. A full volume in log.dirs kills the broker, restart triggers log recovery, recovery fails or the disk fills again, and the loop continues. The other frequent cause is a heap misconfiguration that turns startup log recovery into an OutOfMemoryError. Read the first fatal log line, not the most recent one.
Metrics and monitoring
Four JMX metrics cover most broker incidents. kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions should be zero. kafka.controller:type=KafkaController,name=ActiveControllerCount must sum to exactly one across the cluster. kafka.controller:type=KafkaController,name=OfflinePartitionsCount must be zero. And kafka.network:type=RequestChannel,name=RequestQueueSize growing means requests are arriving faster than the IO threads can clear them.
Behind those sit the saturation signals. kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercent below about 0.3 means the IO thread pool is running out of headroom. Request latency is kafka.network:type=RequestMetrics,name=TotalTimeMs with request=Produce or request=Fetch, and its queue-time component tells you whether the time went to waiting or to work. Throughput is BytesInPerSec and BytesOutPerSec under kafka.server:type=BrokerTopicMetrics. CPU and disk utilisation come from the host, not from Kafka, and most teams scrape all of it into Prometheus with the JMX exporter.
Per-metric thresholds and collection setups are in the broker monitoring guide, and the monitoring tools comparison covers what to run on top.
Cluster maintenance
A rolling restart is safe when it is boring. Restart one broker at a time, let controlled shutdown migrate partition leadership away first, and wait for under-replicated partitions to return to zero before touching the next broker. Skipping the wait is how a routine restart becomes an outage, because the second broker down can take a partition’s last in-sync replica with it.
Your tooling has to treat restarts as routine too. Our co-founder and CEO Derek Troy-West made Kpow’s AWS integration more lenient with rolling restarts, because a broker that disappears for ninety seconds during a planned roll is not an incident, and tooling that pages on every planned restart trains people to ignore the page that matters.
Decommissioning runs in the opposite order. Move every partition off the broker first with kafka-reassign-partitions.sh, confirm it holds no replicas, then shut it down, and on a KRaft cluster unregister it so the controller quorum stops expecting it back. Replacing a dead broker is the same mechanics reversed: bring up a new node with the dead broker’s id and the cluster re-replicates its partitions onto it.
Metadata and state
On disk, a broker is a directory tree of partition logs. Each partition is a set of segment files, a .log file of records plus .index and .timeindex files mapping offsets and timestamps to file positions. Writes append to the active segment only, older segments are immutable until retention deletes them, and reads are served through the operating system page cache, which is why brokers want memory well beyond the JVM heap.
Cluster state lives elsewhere. On a modern cluster the broker registers with the KRaft controller quorum and follows the internal __cluster_metadata topic to learn topic configurations, partition assignments and leadership. Older clusters did the same through ZooKeeper. The KRaft guide covers the quorum, the migration and its day-2 operations. Consumer progress is cluster state too, stored on the brokers in the internal __consumer_offsets topic and covered in the offsets guide.
Network tuning
The knobs stack in layers. At the socket layer, socket.send.buffer.bytes and socket.receive.buffer.bytes size the TCP buffers, which matters most on high-latency links between sites. socket.request.max.bytes caps a single request at 100 MB by default, and message.max.bytes caps a record batch at roughly 1 MB, a limit producers and consumers have to agree with.
Above the sockets sit the thread pools. num.network.threads moves bytes on and off the wire, and num.io.threads does the disk work. In my talk on Kafka operational issues I put it plainly: the replica fetcher count controls how many threads exist to fetch data, the IO thread count controls the IO thread pool, and both settings move disk and network performance together. They were also the two settings whose drift caused the incident above, so tune them with the disk they will hit in mind, not just the CPU in front of it.
Quotas are the guard rail on all of it. Tom Crowley, our founding engineer, keeps the taxonomy crisp: network bandwidth quotas are defined by byte-rate thresholds and have been available since Kafka 0.9, and request rate quotas are defined by CPU utilization thresholds, as a percentage of network and I/O thread time, and have been available since 0.11. A request rate quota of n percent means n percent of one thread’s time, so the total capacity is the combined thread count times one hundred. Set quotas before one client can queue everyone else’s requests behind its own.
FAQ
What does a Kafka broker do?
A Kafka broker stores partition data on disk as segment files, serves produce and fetch requests from clients, replicates partitions to other brokers, and follows cluster metadata from the KRaft controller quorum, or from ZooKeeper on older versions.
Why can clients reach the bootstrap server but not the broker?
Because advertised.listeners is the address the broker hands back to clients in metadata responses, and it is the one clients actually connect to. When the broker advertises a hostname the client cannot resolve or route to, the bootstrap connection works and everything after it times out.
How do I restart a Kafka broker safely?
Restart one broker at a time, let controlled shutdown migrate partition leadership away first, and wait for under-replicated partitions to return to zero before touching the next broker. Skipping the wait is how a routine restart becomes an outage.
What is an under-replicated partition?
A partition where a follower has not caught up to the leader within replica.lag.time.max.ms, 30 seconds by default, and has been dropped from the in-sync replica set. Brief shrink during restarts is normal, and sustained under-replication points at a broker, disk or network problem.
