A Kafka consumer reads records from topic partitions, tracking its own position with an offset rather than having the broker track delivery. Consumers work in groups: a consumer group is a set of consumers sharing the work of consuming a topic, with each partition assigned to exactly one member at a time. That assignment rule is the constraint behind almost everything else on this page, and behind most of the incidents in the hub’s troubleshooting material.
Core documentation and references
Three configuration parameters do most of the damage when misunderstood. session.timeout.ms and its companion heartbeat interval decide how fast the group notices a dead consumer, and production deployments set them deliberately rather than accepting defaults: PayPal’s consumer architecture runs a 60 second session timeout with a 20 second heartbeat interval. max.poll.interval.ms decides how long a consumer may go between polls before the group assumes it has died. And enable.auto.commit decides whether offsets commit on a timer or under your control. For any consumer doing meaningful processing, the setting is enable.auto.commit=false, with an explicit commit only after the result is durable.
The official Apache Kafka documentation and the consumer Javadoc remain the reference for the full parameter list. The parameters above are the ones that decide whether messages are lost, duplicated or stuck.
A pattern from my own incident history that belongs next to any parameter reference: some of the worst consumer incidents I have been close to were only resolved through vendor support tickets, and the root causes turned out to be things no dashboard was surfacing, like a consumer group carrying far too many inactive members. The parameters in this section are not trivia. They are the difference between an incident you can read from your own metrics and one where you are paying someone else to tell you what your group state is.
Troubleshooting and operations
The two most common operational tasks around consumer groups are offset resets and lag investigation, and both start from the same measurement. Consumer lag is the offset distance between the newest record in a partition and the group’s committed position, and monitoring it per group and per partition is one of the most operationally important signals in any Kafka-backed system. Per partition matters: a group-level number averages a stuck partition against healthy ones and hides it. The reset mechanics themselves live on the offsets page.
Rebalance storms are the classic lag generator. Every membership change pauses consumption while partitions are reassigned, so frequent or slow rebalances let lag build even when every consumer is individually healthy. The causes are usually processing that exceeds max.poll.interval.ms, timeouts tuned too aggressively, or deployment patterns that constantly restart group members. This is old knowledge operationalised early: Spotify monitored consumer group lag and rebalancing events and tuned rebalancing retries specifically to cut the noise from routine rebalances.
One of the four incidents from my talk lives exactly here, and it is the one I retell most because every instinct in it was reasonable. A consumer group had accumulated a huge number of members, most of them idle, and a bigger group means more work for the coordinator on every membership change. Lag climbed steadily. The response was the reflex response, scale out, add more consumers, which made the group bigger, which made the rebalances slower. Rebalances took even longer, consumer lag increased further, and one broker, the one hosting the group coordinator, hit 100% CPU while every other broker in the cluster sat healthy. That asymmetry is the tell. When one broker is on fire and its peers are fine, look for a coordinator problem, not a capacity problem, and count the group’s members before adding another one.
The other operational hole is consumers that die without dying: still running, not consuming. Cloudflare’s fix for silent consumer failures is worth copying at any scale, an offset-comparison liveness probe, with Kubernetes restarting any consumer whose committed offsets have stopped advancing. Lag alerts tell you a consumer fell behind. The liveness probe tells you it stopped moving at all, and it converts a 3am page into an automatic restart.
Code and implementation
The consumer client is not thread-safe, so the correct pattern is a single-threaded poll loop: one thread owns the consumer, calls poll() on a tight cadence, and hands records to workers if parallelism is needed. Production systems run exactly this shape, down to a scheduler implemented as a single-threaded consumer poll loop. What breaks the pattern is doing slow work inside the loop: max.poll.interval.ms is a failure-detection timeout, not a processing budget, and stretching it to accommodate slow processing lets one slow message take out an entire partition.
Offset commits decide the failure semantics. A synchronous commit blocks until the broker confirms, giving certainty at a throughput cost, while an asynchronous commit keeps the loop moving and risks a small window of repeated records after a crash. Committing after processing yields at-least-once delivery, committing before yields at-most-once. Whichever is chosen, consumers should be idempotent, because at-least-once is the practical default and duplicates will eventually arrive.
Exactly-once across topics is a producer-side transaction, not a consumer setting. The transactional API wraps produced records and consumer offset commits in one transaction: initTransactions(), begin, produce and send the offsets, then commit or abort. Failed records need a destination as well, and the dead letter queue pattern is implemented at the consumer application, Kafka Connect connector, or Kafka Streams topology layer; the Spring wiring is on the Spring Boot page.
The 2am incident from my talk is the definitive max.poll.interval.ms story. A team set it to 20 minutes to accommodate application-level pre-batching, and it worked, for 12 months. Then a large poison-pill batch exceeded the window. The consumer was kicked from the group mid-processing, the partition was reassigned, the next consumer picked up the same batch, processed for 20 minutes, got kicked, and the group settled into an infinite loop of consumer failure and rebalances that no restart would clear, because the poison batch was always waiting at the committed offset. Twelve months of “working” is what makes this configuration dangerous: the timeout is a failure detector, and setting it to 20 minutes means your failure detector fires after 20 minutes.
The design that avoids the whole class: keep the poll loop fast, move slow work behind it, and give failures somewhere to go, a dedicated retry topic per consumer so replay is isolated and scales independently. If processing can genuinely take minutes, that is queue work dispatched from the loop, not work done inside it.