How to inspect Kafka topics, messages and consumer groups from the terminal

Inspecting Kafka from the terminal means answering five questions with the tools already on the box: are the controllers and brokers healthy, is any partition under-replicated, what is in the topic, is the consumer group keeping up, and are the connectors running.

On Apache Kafka 4.x the answers come from kafka-metadata-quorum.sh, kafka-topics.sh, kafka-consumer-groups.sh and the Kafka Connect REST API, with kcat for reading messages. Everything below is read-only until the offset reset step, and the steps run in the order that makes sense during an incident. The commands were checked against the Apache Kafka 4.3 operations documentation and the 4.3.1 tool source. The complete Kafka guide covers the cluster concepts behind each step.

When the terminal beats the browser

The terminal wins when it is the only thing that can reach the cluster: an SSH session on a jump host inside the VPC, a broker you are already logged into, a cluster with no web UI deployed, or an incident where the UI itself is the thing that is down. It also wins when you want the exact output in a ticket.

From my talk on Kafka operational incidents: “Don’t panic, think before you act, and don’t scale Kafka as a reflex during an incident.” Look first, and change one thing at a time once you know the cause.

Set up the connection once per session. The Kafka scripts read client settings from a properties file, so put the credentials of a read-only principal in one file rather than on every command line:

# admin.properties
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="ops-readonly" password="<password>";

Then set two variables so every command below stays short:

export BS=broker-1.internal:9093
export CFG=admin.properties

kcat uses librdkafka property names instead, and reads ~/.config/kcat.conf by default or a file passed with -F:

# kcat.conf
bootstrap.servers=broker-1.internal:9093
security.protocol=SASL_SSL
sasl.mechanisms=SCRAM-SHA-512
sasl.username=ops-readonly
sasl.password=<password>

Check the brokers, then find the topic and its partitions

Start with the controller quorum, because nothing else is reliable if the metadata leader is missing. The status view prints the leader, its epoch, the high watermark and the worst follower lag:

kafka-metadata-quorum.sh --bootstrap-server $BS --command-config $CFG describe --status

MaxFollowerLag and MaxFollowerLagTimeMs near zero means the controllers agree. The replication view lists every controller and broker with its log end offset, lag and last caught-up time, with brokers shown as observers:

kafka-metadata-quorum.sh --bootstrap-server $BS --command-config $CFG describe --replication

To see which brokers are registered, and whether any is fenced:

kafka-cluster.sh list-endpoints --bootstrap-server $BS --command-config $CFG --include-fenced-brokers

On Kafka 4.0 and 4.1, kafka-cluster.sh takes --config instead of --command-config. On a 3.x cluster still running ZooKeeper, skip the quorum commands. Everything from here on works the same.

Next, look for partitions that are not fully replicated. An empty result is the healthy answer:

kafka-topics.sh --bootstrap-server $BS --command-config $CFG --describe --under-replicated-partitions
kafka-topics.sh --bootstrap-server $BS --command-config $CFG --describe --under-min-isr-partitions
kafka-topics.sh --bootstrap-server $BS --command-config $CFG --describe --unavailable-partitions

The three flags are increasingly serious. Under-replicated means a follower has fallen out of the ISR. Under min ISR means producers using acks=all are being rejected on that partition. Unavailable means the partition has no leader. If every under-replicated partition has the same broker missing from its Isr list, the problem is that broker, and Kafka brokers in production covers what to check on it. In the incidents talk I made the point that under-replicated partitions are “useful for telling you something is wrong, but not why”, so treat this as the question to take to the broker, not the answer.

Now the topic itself. The describe output gives the partition count, replication factor, config overrides, and a leader, replica list and ISR per partition:

kafka-topics.sh --bootstrap-server $BS --command-config $CFG --describe --topic orders

The config the topic actually runs with, including broker defaults it inherits:

kafka-configs.sh --bootstrap-server $BS --command-config $CFG --describe --entity-type topics --entity-name orders --all

Whether data is arriving at all: print the latest offset per partition, wait a minute, and print it again. The earliest offsets tell you how much retention is holding.

kafka-get-offsets.sh --bootstrap-server $BS --command-config $CFG --topic orders --time latest
kafka-get-offsets.sh --bootstrap-server $BS --command-config $CFG --topic orders --time earliest

And how much disk each partition uses on each broker, as JSON:

kafka-log-dirs.sh --bootstrap-server $BS --command-config $CFG --describe --topic-list orders

Search and filter messages

To read messages safely, never join the application’s consumer group. If you pass the application’s group id to a console consumer, it joins that group, triggers a rebalance, takes partitions away from the real consumers, and can commit offsets for them.

With no --group, the console consumer creates a throwaway group named console-consumer- plus a random number and turns auto-commit off, which is exactly what you want for a peek. Five records from partition 0, starting at offset 1200, with keys, offsets and timestamps:

kafka-console-consumer.sh --bootstrap-server $BS --consumer.config $CFG \
  --topic orders --partition 0 --offset 1200 --max-messages 5 \
  --property print.key=true --property print.offset=true --property print.timestamp=true

From Kafka 4.2, --command-config and --formatter-property are the preferred spellings of --consumer.config and --property. The old ones are marked deprecated but still work in 4.3. The console producer, kafka-get-offsets.sh and the other scripts beside it are in Kafka CLI commands for day-2 operations.

kcat is faster for this, because it is a small native binary rather than a JVM start per command, and without -G it does not use a consumer group at all. The last 20 records on partition 0, then exit:

kcat -F kcat.conf -C -t orders -p 0 -o -20 -e -f 'offset %o key %k: %s\n'

To follow a topic live from the current end, the way tail -f follows a log:

kcat -F kcat.conf -C -t orders -o end -f '%p:%o %k %s\n'

To filter by a field value, ask kcat for a JSON envelope per record (topic, partition, offset, ts, key, payload) and select with jq:

kcat -F kcat.conf -C -t orders -o beginning -e -J \
  | jq -c 'select((.payload | fromjson? | .status) == "FAILED") | {partition, offset, key}'

For a time window, kcat takes start and end timestamps in milliseconds:

kcat -F kcat.conf -C -t orders -o s@1790038800000 -o e@1790042400000 -J | jq -c '.payload'

Every one of these filters runs on your side of the connection, so every record in the range crosses the network to the jump host before jq throws most of them away. That is fine for a few thousand records and slow for a few days of a busy topic. If you need field-level search over large ranges regularly, server-side search is the better tool, and the best tools to search messages across Kafka topics compares them. Among terminal tools, Yozefu adds a query language and never commits offsets, but still filters on the client.

My own line from the incidents talk explains why this step is worth the effort: “Data inspection tooling is essential for handling lost-message complaints, since the large majority of the time, the message was on Kafka all along.” Finding the record, with its partition and offset, usually ends the argument about whether Kafka lost it.

Decode Avro and JSON with Schema Registry

JSON needs nothing extra: the commands above print it, and jq formats it. Avro and Protobuf are binary, so the console consumer prints unreadable bytes. The Apache Kafka distribution ships no Avro deserializer. Confluent’s kafka-avro-console-consumer comes with Confluent’s Schema Registry packages, and kcat can decode Avro itself against a registry:

kcat -F kcat.conf -C -t payments -s value=avro -r https://schema-registry.internal:8081 -o -5 -e

Use -s avro to decode both key and value. Avro support in kcat is an optional build dependency (libserdes), so a distribution package may not include it. kcat does not decode Protobuf.

When a consumer fails with a deserialization error, check whether the bytes are in the registry’s framing at all. Confluent-framed records start with a zero magic byte followed by a four-byte schema id, so the first bytes of the raw value tell you whether the producer used the registry serializer:

kcat -F kcat.conf -C -t payments -o -1 -e -f '%s' | xxd | head -1

A value that starts with 00 and then four bytes of schema id is registry-framed. Anything else was written by a different serializer, and how to diagnose a Kafka deserialization error takes it from there.

Check a consumer group’s lag and state, and reset offsets

Lag per partition comes from the group’s committed offsets, so this works even when every consumer is down:

kafka-consumer-groups.sh --bootstrap-server $BS --command-config $CFG --describe --group order-service

Read the LAG column per partition, not as a total. One partition climbing while the rest sit near zero usually means a record the consumer cannot process, and how to find and skip a poison pill covers that case. Every partition climbing together means the consumers are too slow or not running.

The group’s state and member count:

kafka-consumer-groups.sh --bootstrap-server $BS --command-config $CFG --describe --group order-service --state
kafka-consumer-groups.sh --bootstrap-server $BS --command-config $CFG --describe --group order-service --members --verbose

A state of Empty means no consumer is connected. A group that keeps showing PreparingRebalance is rebalancing instead of consuming, which what is Kafka rebalancing? explains. A member count far above the partition count means idle members. In one incident I described in the talk, a group grew to tens of thousands of members for 1,000 partition assignments, and the group coordinator broker ended up at 100% CPU.

To watch lag move rather than take a snapshot, wrap the describe in watch:

watch -n 10 "kafka-consumer-groups.sh --bootstrap-server $BS --command-config $CFG --describe --group order-service"

That is a stopgap for the length of an incident, and it has no history behind it. If you find yourself running it regularly, lag belongs on a dashboard that keeps the trend and raises an alert, which the best Kafka monitoring tools compares.

Resetting offsets is the first step on this page that changes anything. My advice from the talk: “Use latest in production and make offset resets a deliberate, manual operation.” The tool supports that. The Apache documentation says to make sure the group’s consumers are inactive first, and without --execute the command only prints the plan. Preview a reset to a point in time, export it to a file you can roll back from, then apply it:

kafka-consumer-groups.sh --bootstrap-server $BS --command-config $CFG \
  --reset-offsets --group order-service --topic orders --to-datetime 2026-09-22T01:00:00.000

kafka-consumer-groups.sh --bootstrap-server $BS --command-config $CFG \
  --reset-offsets --group order-service --topic orders --to-datetime 2026-09-22T01:00:00.000 \
  --export > order-service-reset-plan.csv

kafka-consumer-groups.sh --bootstrap-server $BS --command-config $CFG \
  --reset-offsets --group order-service --topic orders --to-datetime 2026-09-22T01:00:00.000 --execute

To skip one record on one partition, name the partition after the topic and shift by one:

kafka-consumer-groups.sh --bootstrap-server $BS --command-config $CFG \
  --reset-offsets --group order-service --topic orders:3 --shift-by 1 --execute

Check Kafka Connect connector status

Connect has no script in bin/ for status. It has a REST API on each worker, documented in the Kafka Connect user guide. List the connectors, then ask each one for its state and the state of its tasks:

export CONNECT=http://connect-1.internal:8083
curl -s $CONNECT/connectors
for c in $(curl -s $CONNECT/connectors | jq -r '.[]')
do
  curl -s $CONNECT/connectors/$c/status | jq -c '{name, connector: .connector.state, tasks: [.tasks[].state]}'
done

A FAILED connector or task carries a trace field with the exception. Read it before restarting anything:

curl -s $CONNECT/connectors/iceberg-sink/status | jq -r '.tasks[] | select(.state == "FAILED") | .trace' | head -20

Restarting only the failed instances is one call:

curl -s -X POST "$CONNECT/connectors/iceberg-sink/restart?includeTasks=true&onlyFailed=true"

A restart fixes a transient failure. It does not fix a bad config, a missing table or a database that refuses connections, and restarting a connector in that state only produces the same failure again. How to diagnose and fix a failed Kafka Connect connector walks through reading the trace.

Doing it under SSO and RBAC

Everything above assumes a jump host with a credential that can read every topic and, for the reset, write offsets for any group. That is how most teams run it, and it is the weak point. Tom Crowley, our founding engineer, described the pattern in 2021: someone “must jump through the hoops of configuring the VPN, connecting to the jumpbox, and making sure they execute the right combination of bash commands against the Kafka cluster”, and “the jumpbox generally has full access to the Kafka cluster, and there is no audit log recording the actions being committed.” Nothing in the scripts records who ran a reset, and nothing stops a read-only task from becoming a write.

The Kpow terminal UI is Factor House’s answer to doing the same jobs from a terminal without that credential. It talks to a Kpow server rather than to the brokers, and you sign in with fh auth login through your OpenID provider, so the Kpow role-based access control rules that apply in the web UI apply in the terminal. In the Kpow CLI, terminal UI and agentic skills demo I show the broker disk, config, KRaft and ACL views, the topics list, a Data Inspect query on value.destination, failing Kafka Connect connectors, and consumer groups with offset reset. The Data Inspect query runs on the Kpow server, so the terminal receives the matching records rather than the whole range. From that demo: “your CLI users, terminal UI users, and agentic users all get the same control and governance in place”.

brew install factorhouse/tap/fh
fh config configure
fh auth login
fh tui

It has limits worth knowing before you plan around it. It needs a running Kpow deployment. RBAC, multi-tenancy and audit logging are Kpow Enterprise features, and Community Edition connects without authentication. It cannot produce messages, so kafka-console-producer.sh stays in your toolkit for that. If you are choosing between it and the open-source terminal UIs, the best Kafka terminal UIs scores all of them on the same rubric, and the best Kafka CLI tools does the same for kcat, kcl and the fh command line.

Kpow live demo

Watch the same inspection in the Kpow terminal UI

Brokers, topics, a Data Inspect query, consumer groups and failing connectors, driven from the keyboard under your existing SSO and RBAC.

Built for platform and data engineers running Kafka in production.

Watch the demo→

FAQ

How to view Kafka topics?

List them with kafka-topics.sh --bootstrap-server <broker> --list, then describe one with --describe --topic <name> to see its partitions, leaders, replicas and ISR. Add --command-config <file> with your client credentials on a secured cluster.

How do I read messages from a Kafka topic in the terminal?

Use kafka-console-consumer.sh with --topic, --partition, --offset and --max-messages, or kcat with -C -t <topic> -o -20 -e for the last 20 records. Do not pass the application’s group id, or you will join its consumer group and can move its offsets.

Can I filter Kafka messages by field value from the command line?

Yes, by piping kcat’s JSON output (-J) through jq and selecting on the decoded payload. The filtering happens on your machine, so it suits small ranges. Yozefu adds a query language in the terminal, and server-side search tools avoid pulling every record across the network.

Related reading

• Kafka: the complete guide
• Kafka brokers in production
• Kafka consumer lag: how to find the cause and monitor it
• How to find and skip a poison pill in Kafka
• How to diagnose and fix a failed Kafka Connect connector
• How to diagnose a Kafka deserialization error