A Kafka consumer group is a set of consumers that share the work of consuming a topic. Each partition is assigned to exactly one member of the group at a time, so partitions are the unit of parallelism, and a broker acting as group coordinator manages membership and assignments.
Consumer groups are where application code meets cluster health, and in my experience they are the least observed part of most Kafka deployments. One of the incidents I talk about most started as a support ticket asking the vendor to scale the cluster faster. The vendor came back with the actual problem: far too many members in one consumer group. Nobody had metrics on group membership size or coordinator request rates, so nobody could have diagnosed it in-house. If you monitor only brokers and topics, this page is the layer you are blind to. Where consumer groups sit in the wider platform is mapped in the complete Kafka guide.
Troubleshooting consumer lag and rebalances
Lag is the offset distance between the newest record on a partition and the last record the group has committed. Rising lag means consumption is falling behind production, and frequent or slow rebalances are one of its common causes, because partitions sit unowned while assignments settle. Our guide to monitoring Kafka consumer lag covers the measurement side, and what is Kafka rebalancing goes deep on the rebalance mechanics.
Consumer-group detail showing members, per-partition assignment, and members holding no assignment.
Checking lag is one command:
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-groupThe output lists CURRENT-OFFSET, LOG-END-OFFSET and LAG per partition, with the owning consumer.
Rebalance loops have a short list of causes. A consumer that takes longer between polls than the poll interval allows is considered failed and evicted, controlled by: max.poll.interval.ms, default 300000 (5 minutes). A consumer that misses heartbeats for longer than the session timeout is evicted the same way, controlled by: session.timeout.ms, default 45000 (45 seconds). A JVM GC pause longer than the session timeout is enough to trigger eviction and a rebalance.
Rolling restarts need static membership. Setting group.instance.id on each consumer makes it a static member, so a bounce that returns within the session timeout does not trigger a rebalance at all.
The worst consumer-group incident I have worked had exactly this shape. Rebalances kept taking longer, lag kept climbing, and one broker sat at 100% CPU while the others looked fine. That broker was the group coordinator. The team’s first instinct was to add brokers, and it made things worse, because more partition movement meant more reassignment and rebalance activity landing on the coordinator that was already saturated. The arithmetic underneath it: the group covered around 100 topics of 10 partitions each, so only 1,000 assignments could ever be active, and roughly 39,000 group members sat idle while still heartbeating to the coordinator. The lesson we took from it: scale consumer groups by logical consumer boundary, one group per real consumer concern, never one giant group across everything.
The CLI cheat sheet
The kafka-consumer-groups.sh tool covers the routine group operations. The most common tasks in practice are offset resets and lag investigation.
# list all groups
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
# describe a group: per-partition offsets, lag, members
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group
# active members and their assignments
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --members
# group state (Stable, PreparingRebalance, Empty, ...)
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-group --state
# delete inactive groups
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --delete --group my-groupOffset resets preview by default and execute only when told to. --dry-run is the default, --execute applies, and --export emits CSV:
# rewind a topic to the earliest offset (preview first)
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
--reset-offsets --group my-group --topic topic1 --to-earliest --dry-run
# then apply
bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
--reset-offsets --group my-group --topic topic1 --to-earliest --executeOther reset scenarios: --to-latest, --to-datetime YYYY-MM-DDThh:mm:ss.sss, --shift-by n, --by-duration PnDTnHnMnS, --to-offset, --from-file. Resets require the group to have no active members, and manual deletion works only on groups with no active members.
The CLI answers are point-in-time, and that is the limit teams hit as adoption grows. In one case study we published, a national food-distribution group ran Kafka behind its e-commerce platform with open-source tooling and CLI scripts, and tasks like moving offsets were manual, painful work that could not keep pace once a dozen teams depended on the cluster. The commands on this page are the right tool for a single investigation. Once offset resets become routine operations across many groups, they need tooling with a continuous view and an audit trail rather than a terminal history.
Architecture and partition assignment
The group coordinator is a broker with a second job. It tracks membership through heartbeats, stores committed offsets in the internal __consumer_offsets topic, and triggers reassignment when members join or leave. Coordinator load grows with member count, which is why very large groups rebalance slowly.
Assignment strategies decide who gets which partition. The client config partition.assignment.strategy selects among RangeAssignor, RoundRobinAssignor, StickyAssignor and CooperativeStickyAssignor (consumer configuration reference). The cooperative variant revokes only the partitions that actually move, so a membership change no longer stops every consumer in the group. Incremental cooperative rebalancing shrinks the blast radius of a failure, though a poison-pill record still blocks its own partition.
Scale is capped by partitions. One partition serves at most one group member, so a topic with three partitions supports at most three active consumers in the group and any extra members sit idle. Scaling consumption beyond that means more partitions, which is a topic design decision, not a group setting. KIP-932 share groups relax the one-consumer-per-partition rule for queue-style workloads.
For stateful consumers, the combination that avoids most rebalance pain is CooperativeStickyAssignor plus group.instance.id on every member.
Partition count increases are not zero cost for consumers, and you need to know where your consumers will start reading before you scale. I have watched a team scale a service by raising partition counts, and because their consumers ran auto.offset.reset=latest and took longer than expected to rebalance onto the new partitions, hundreds of messages produced to those partitions in the gap were never read. The lesson we took from it: treat a partition increase as a consumer-side change too, and confirm every group’s reset behaviour before the topology changes, not after.
Offset commits in application code
Auto-commit trades safety for convenience. With enable.auto.commit=true (the default) the consumer commits in the background on an interval, which can commit offsets for records the application has not finished processing. For any consumer doing meaningful work, disable it: enable.auto.commit=false, and commit explicitly after the result is durable.
Synchronous versus asynchronous commits. commitSync retries until it succeeds or fails hard, and is the right call on rebalance and shutdown paths. commitAsync is cheaper per record batch and suits the steady state, with commitSync as the final commit.
Poison pills need an exit. A record that fails processing forever blocks its partition, because the group cannot commit past it. The standard pattern is an error tolerance plus a dead letter queue topic, so the bad record is captured and the partition moves on. If the DLQ write itself fails, the offset should not be committed, and an alert should fire.
max.poll.interval.ms is a failure detection timeout, not a processing budget. I have seen a team raise it to 20 minutes so they could pre-batch thousands of messages into one unit of work, and it ran fine for 12 months, right up until a large batch and a poison pill pushed processing past the limit and the group started evicting healthy consumers. Pre-batching in the application also fights Kafka’s own batching, so one slow message stalls everything behind it. On failure handling, the pattern I recommend is a dedicated retry stream per topic and consumer pair, with the DLQ reserved as the give-up path that needs manual intervention, so failures stay attributable to a specific consumer. And keep auto.offset.reset=latest in production, with offset resets a deliberate, manual operation, because an innocent consumer group ID rename under earliest reprocesses the topic from the beginning.
FAQ
How many consumers can a Kafka consumer group have?
As many as you like, but only as many active as the topic has partitions. Each partition is assigned to exactly one member at a time, so a topic with three partitions supports three active consumers and any extra members sit idle.
Should I use Kafka’s auto-commit?
Not for consumers doing meaningful work. Auto-commit can commit offsets for records the application has not finished processing. Set enable.auto.commit=false and commit explicitly after the result is durable.
How do I check consumer lag?
Run kafka-consumer-groups.sh --describe --group <group> against a broker. The output lists CURRENT-OFFSET, LOG-END-OFFSET and LAG per partition with the owning consumer.