Best tools to monitor Kafka consumer lag

Kafka consumer lag monitoring tools fall into five groups: the kafka-consumer-groups.sh CLI, client-side JMX metrics, open-source lag exporters and evaluators (Burrow, kafka_exporter, KMinion), Kafka UIs (Kpow, AKHQ, Kafbat UI), and managed-service consoles such as Amazon MSK CloudWatch metrics, Confluent Cloud, Datadog and Conduktor.

Which one fits depends on five things: where lag is measured, whether you get lag in messages or in time, whether you can see it per partition, whether alerts fire on a sustained trend rather than every spike, and whether the tool shows enough context to find the cause. Kpow is Factor House’s product, and I work at Factor House as a Solutions Architect, so it is scored on the same rubric and the same sources as every other option. The complete Kafka guide covers the wider tooling picture.

What to score a consumer lag tool on

Lag is the gap between a partition’s log end offset and the offset a consumer group has reached. Every tool on this page reports some version of that number. They differ on five criteria, and each one maps to a way a lag dashboard has let a real on-call engineer down.

1. Where the lag is measured: client-side or cluster-side

Client-side lag comes from the consumer process itself. The Java consumer exposes it as a JMX metric:

kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{client-id}", attribute records-lag-max

The Apache Kafka monitoring documentation notes that this value “is based on current offset and not committed offset”, and it only exists while the consumer is running. If the process crashes, hangs, or sits in a long rebalance, the metric stops reporting at the moment you most need it.

Cluster-side lag is calculated from outside the application: the group’s committed offsets, read through the Admin API or the __consumer_offsets topic, subtracted from each partition’s end offset. It keeps reporting when every consumer is dead, which is the scenario that matters. In Kpow’s own consumer group documentation the reason is stated directly: offsets for an EMPTY group are fetched from the AdminClient, which “is critical for when a poison message causes an entire consumer group to go offline.” Managed metrics have their own gaps here. Amazon’s MSK consumer lag documentation says lag metrics “are emitted only if a consumer group is in a STABLE or EMPTY state”, so a group stuck rebalancing goes quiet in CloudWatch.

2. Offset lag or time lag

“50,000 messages behind” means nothing until you know the throughput. On a topic taking 100,000 messages a second it is half a second of delay. On a topic taking ten a minute it is days. Time lag, the age of the oldest unconsumed record or an estimate of it, is the number an on-call engineer can compare against an SLA. Few tools compute it, because doing it properly means sampling record timestamps or interpolating offset commits over time. Each option below is scored on which of the two it actually reports.

3. Per-partition detail, not a group average

A group-level lag total averages one stuck partition against its healthy siblings and hides it. A poison pill, a hot key or an unbalanced assignment shows up as one partition climbing while the rest sit near zero. Our heaviest Kpow users ask for group offset lag at topic-partition granularity in their Prometheus metrics for exactly this reason.

4. Alerting on a sustained trend, not a spike

Lag spikes are normal. A batch job lands, a deploy triggers a rebalance, and lag jumps and drains within minutes. Alerting on a fixed threshold pages people for all of that. The useful alert fires when lag keeps growing for longer than your longest normal rebalance or batch, or when a partition’s committed offset stops moving while lag is above zero. Burrow builds this in, and the Prometheus-based tools let you write it yourself.

5. Enough context to find the cause

Lag tells you something is wrong, not why. In a talk on Kafka operational incidents I walked through a service that scaled to 400 instances, each starting 100 consumers in one shared group. With 100 topics of 10 partitions there were only 1,000 assignments, so 39,000 members sat idle, the group coordinator broker hit 100% CPU, rebalances slowed, and lag climbed. The team doubled the instance count, which made it worse. The missing signal was consumer group membership size, and the broker-side signals that would have shown the overloaded coordinator are compared in the best tools to monitor Kafka broker health. A lag tool that also shows member counts, idle members and assignment balance next to the lag line saves you from scaling the wrong thing.

Consumer lag tools compared

Diagnosing a specific lag spike, rather than choosing a tool, is covered step by step in how to monitor Kafka consumer lag. The table scores each option on the criteria above, in the same order.

Tool	Where lag is measured	Offset or time lag	Per partition	Alerting	Visible when consumers die	Source
kafka-consumer-groups.sh	Cluster-side, from committed offsets at the moment you run it	Offset lag only	Yes, one row per partition	None, it is a one-off command with no history	Yes, it describes groups with no active members	Apache Kafka docs
Consumer JMX (records-lag-max)	Client-side, from the consumer’s fetch position	Offset lag only	Yes, records-lag per partition, plus a per-client max	Through whatever scrapes JMX	No, the metric disappears with the process	Apache Kafka docs
Burrow	Cluster-side, from committed offsets over a sliding window	Offset lag, but its headline output is a status per group and partition (OK, WARNING, STALLED, STOPPED)	Yes, evaluated per partition	Built-in email and HTTP notifiers, no thresholds to tune	Yes, a group that stops committing is marked STOPPED	GitHub: linkedin/Burrow
kafka_exporter	Cluster-side, Prometheus exporter	Offset lag, documented as approximate	Yes, kafka_consumergroup_lag per partition	Prometheus Alertmanager rules you write	Yes	GitHub: danielqsj/kafka_exporter
KMinion	Cluster-side, Admin API or the offsets topic	Offset lag. Its end-to-end probe measures roundtrip latency on its own topic, not your consumers’ time lag	Configurable, per partition or per topic	Prometheus Alertmanager rules you write	Yes, and it exports empty-member counts	GitHub: redpanda-data/kminion
Kafka Lag Exporter	Cluster-side	Offset lag plus interpolated time lag	Yes	Prometheus rules you write	Yes	GitHub: seglo/kafka-lag-exporter, archived February 2024
Kpow	Cluster-side, including EMPTY groups	Offset lag, plus consumption rate and last-read timestamp per assignment. No time lag metric	Yes, by group, topic, partition, host and broker	Prometheus egress endpoints for Alertmanager or Grafana	Yes, lag is calculated for EMPTY groups	Kpow docs
AKHQ	Cluster-side, in the UI	Offset lag	Yes	Not described in its docs, pair it with an exporter	Yes	akhq.io
Kafbat UI	Cluster-side, in the UI	Offset lag	Yes, combined and per partition	Not described in its docs, pair it with an exporter	Yes	GitHub: kafbat/kafka-ui
Amazon MSK (CloudWatch)	Cluster-side, managed	Both: OffsetLag, SumOffsetLag, MaxOffsetLag, EstimatedTimeLag, EstimatedMaxTimeLag	Yes	CloudWatch alarms	Only for STABLE or EMPTY groups, so not during a long rebalance	AWS MSK docs
Confluent Cloud	Cluster-side, managed	Offset lag through the Metrics API (consumer_lag_offsets), and the Cloud Console	Not stated in the documentation page cited	Through the Metrics API and third-party monitors	Not stated in the documentation page cited	Confluent’s documentation, “Monitor Kafka Consumer Lag in Confluent Cloud”
Datadog Data Streams Monitoring	Service-side, correlated with traces and logs	Consumer lag plus end-to-end pathway latency	Per service and queue	Out-of-the-box monitor templates	Depends on the Kafka integration feeding it	Datadog’s documentation, Data Streams Monitoring overview
Conduktor Console	Cluster-side	Both: offset lag, and a time lag its docs describe as the estimated time to catch up based on consume rate	Yes, lag per partition and per member	Threshold alerts on OffsetLag or TimeLag to Slack, Teams, email or webhook	Yes	Conduktor’s documentation, Alerts and Consumer groups pages

How the options score

On time lag, Amazon MSK, Conduktor and Datadog lead the field, and Kafka Lag Exporter showed how to interpolate it from open-source parts before it was archived. Kpow does not compute time lag. It exposes consumption rate (group_assignment_delta) and the last time a group was observed reading a partition (group_assignment_last_read), which answer “is it moving” but not “how many minutes behind”.

On visibility when consumers die, every cluster-side tool passes and client-side JMX fails. Burrow goes furthest, because it turns “stopped committing” into its own STOPPED status rather than leaving you to infer it from a flat line.

On alerting without noise, Burrow is the only option that evaluates trends out of the box. kafka_exporter, KMinion and Kpow all hand you Prometheus metrics and leave the rule to you, which is more work and more control. Conduktor’s alerts are threshold-based, with time lag as one of the thresholds.

On context and action, the UIs win. Kpow, AKHQ and Kafbat UI put lag next to the group’s members and let you reset offsets from the same screen, and the best tools to reset consumer group offsets compares that half of the job on its own rubric. Kpow adds member-level lag by host and broker, and its Signals checks flag unbalanced assignments and idle members, the signal that was missing in the 39,000-idle-member incident. KMinion is the strongest exporter on this criterion because it exports empty-member counts.

Alerting on lag without the noise

Whatever exporter you choose, write two alerts: one for sustained growth and one for a stalled partition. Here is the shape for kafka_exporter’s metrics. The window is the part to tune, and it should be longer than your longest normal rebalance or batch job.

groups:
  - name: kafka-consumer-lag
    rules:
      # Lag has grown steadily for the whole window on one partition
      - alert: KafkaConsumerLagGrowing
        expr: deriv(kafka_consumergroup_lag[15m]) > 0
        for: 15m
      # Committed offset has not moved while lag is above zero
      - alert: KafkaConsumerPartitionStalled
        expr: delta(kafka_consumergroup_current_offset[10m]) == 0 and kafka_consumergroup_lag > 0
        for: 10m

The stalled-partition rule is the one that catches a poison pill: one partition’s offset stops, its lag rises, and every other partition in the group looks fine. How to find and skip a poison pill walks through confirming and clearing it. Burrow’s STALLED status encodes the same idea without the PromQL. If you run Kpow, the per-assignment offsets on its /group-offsets/v1 endpoint and partition end offsets on /offsets/v1 support the same two rules.

Each option in detail

How Factor House approaches consumer lag

Kpow treats lag as the start of an investigation rather than the end of one. The Kpow consumer lag product page shows the views: lag by group, topic, partition, host and broker at once, with Kafka Streams applications detected automatically. The consumer group documentation covers lag for EMPTY groups and the offset actions (reset, clear, skip) you reach from the same screen. You can check this against the per-partition criterion in the live Kpow demo: open any consumer group and compare its lag by partition, host and broker with the group total. The Prometheus integration is how lag gets to Alertmanager and Grafana, and the metrics glossary lists every lag and offset metric.

When the lag is on one partition and the offset has stopped moving, the cause is often a record the consumer cannot process. Kafka consumer monitoring and performance tuning covers the other consumer-side causes, and what is Kafka rebalancing? covers the lag that rebalances create on their own. For the wider monitoring stack beyond lag, see the best Kafka monitoring tools.

Kpow live demo

See consumer lag per partition, live

Open the Kpow demo and watch lag by group and partition on a running cluster, then drill from a lagging group to the topic behind it.

Built for platform and data engineers running Kafka in production.

Try the Kpow demo→

FAQ

What is the best tool to monitor Kafka consumer lag?

For a team on Prometheus, kafka_exporter or KMinion for metrics plus Burrow-style trend alerts is the strongest open-source stack. If you need lag in minutes rather than messages, Amazon MSK and Conduktor compute time lag. If you want lag next to the controls to fix it, a UI such as Kpow puts both on one screen.

Why is records-lag-max not enough for lag alerting?

It is reported by the consumer process, so it stops when the consumer crashes, hangs or sits in a long rebalance. It is also based on the fetch position, not the committed offset. Pair it with a cluster-side tool that reads committed offsets.

What is the difference between offset lag and time lag?

Offset lag is the number of messages between a group’s committed offset and the partition’s end. Time lag is how long the oldest unconsumed record has been waiting. Time lag is the one you can hold against an SLA, because it does not depend on throughput.

How do I stop consumer lag alerts from firing on every spike?

Alert on sustained growth over a window longer than your normal rebalances and batch jobs, and separately on a partition whose committed offset has stopped moving while lag is above zero. Burrow’s evaluation rules do this for you, and a Prometheus for: clause does it for exporter metrics.

Related reading

• Kafka: the complete guide
• Kafka consumer lag: how to find the cause and monitor it
• Kafka offsets
• What is Kafka rebalancing?
• Kafka consumers in production