Best tools to search messages across Kafka topics

Searching messages across Kafka topics means reading records from one or more topics, decoding them, and keeping only the ones that match a condition, because Kafka stores an append-only log with no index on message content. The tools that do it are the Apache CLI (kafka-console-consumer.sh) and kcat piped through jq, the Kafka UIs with a search feature (Kpow with its kJQ filter language, Kafbat UI, AKHQ, Kafka Magic, Conduktor and Redpanda Console), SQL layers such as Lenses SQL Studio and ksqlDB, and offloading topics into a search index. Kpow is Factor House’s product, and I work at Factor House as a Solutions Architect, so it is scored on the same rubric and the same sources as every other option.

Why every Kafka search is a scan

A Kafka topic is a set of partition logs addressed by offset and, through the time index, by timestamp. Nothing inside Kafka indexes the key or the value. So “find the order with ID 8841” always means reading records from a starting point, deserializing each one, and testing it against the condition. Every tool on this page does that. They differ in where the reading and testing happens, how much of the topic they read, and what they let you write as a condition.

The search matters more than its reputation suggests. In my operational issues webinar I said that if I had a dollar for every time I heard a lost message complaint, only to find the message was actually on Kafka all along, I would have a lot more money than I do. Most of the time the message is there, and the incident is really a search problem. Six criteria decide how fast a tool answers it, and every option is scored against them in this order.

Scan scope and cost. Because every search reads records, the tool should let you bound what it reads: a time window, a starting offset, a single partition, an exact key, and a limit on results and on records scanned. An unbounded scan of a large topic is slow for you and adds fetch load on the brokers serving it, so a tool that only reads from the beginning is the wrong tool for a production topic.

Where the filter runs. Either the tool’s server consumes the records and sends only matches to your browser, or every record crosses to your machine and the filter runs there. A CLI pipeline is the second kind: kcat fetches and prints every record, and jq or grep discards most of them locally.

Filter language. A substring match finds a line containing “false”. A structured filter tests .value.trade.compliance.audit == false on a specific field, with type casts and time comparisons. The first finds candidates, and the second finds the record.

Multi-topic scope. Incidents rarely stay in one topic. A tool that searches several topics in one query, by list, by regex, or by the topics a consumer group reads, saves one query per topic.

Deserialization. The records have to be decoded before any field-level filter can run. Avro, Protobuf and JSON Schema with a schema registry, plus plain JSON and strings, cover most clusters, and a topic you do not know the format of needs automatic detection or trial and error.

Access control, masking and audit on results. Searching production data during an incident often means looking at customer data. The tool should control who can query which topics, mask sensitive fields in results, and record who searched what. The Kafka data masking tools comparison scores the masking side on its own.

The options, compared

Every cell comes from the tool’s own documentation or source code. “Not documented” means it does not appear in the tool’s own material, not that the feature is absent.

Tool	Filter language	Where the filter runs	Scan scope controls	Multi-topic in one query	Deserialization	Access control and masking on results	Source
kafka-console-consumer.sh with grep or jq	None built in. Pipe to grep or jq	Your machine, after every record is fetched	Start offset, partition, max messages, timeout	Yes, by topic regex with --include	Configurable deserializers, output as text	Only Kafka ACLs on the consuming user	Apache Kafka quickstart
kcat with jq	None built in. Its README pipes to jq	Your machine	Offsets, partition, message count, exit at end	Not as a search, one pipeline per topic in practice	Avro with a schema registry, per its README	Only Kafka ACLs	kcat README
Kpow (kJQ)	kJQ, a jq-like language with type casts and time arithmetic, plus AI filter generation (Enterprise)	Kpow’s server, which returns only matches to the browser	Slice, bounded window, partition and key modes, result and scan limits, streaming search	Yes, a topic list, a topic regex, or a consumer group’s topics	Avro, Protobuf, JSON Schema, JSON, strings and more, with automatic detection, custom serdes (paid)	RBAC per topic, data masking policies and a user-action audit log (Enterprise)	Kpow data inspect docs
Kafbat UI	CEL expressions	Kafbat’s backend	Not documented beyond the message view’s own options	Not documented	Schema registry serde, Protobuf file serde and others	Free RBAC with a message read permission, and masking	Kafbat UI features
AKHQ	Match on key, value, header key or header value by equals, contains or not contains	AKHQ’s backend	Per topic data view	No, search is per topic. Live tail takes several topics	Avro and Protobuf with a registry	Free RBAC, regex masking	AKHQ repository
Kafka Magic	JavaScript over message fields	Not documented	Not documented	No, its query guide searches a topic	JSON and Avro	Community Edition free, Professional licensed	Kafka Magic query guide
Conduktor Console	Global search, field search, JavaScript filters, JQ projection, SQL on topics	Conduktor’s server	Not documented in the pages checked	Not documented	Not documented in the pages checked	Masking and audit not in the free tier	Conduktor docs: Topics, Community Edition (text only)
Lenses SQL Studio	SQL snapshot queries	Lenses	SQL clauses	Yes, a query can join topics	Not documented in the pages checked	Data policies mask fields by name	Lenses docs: SQL concepts, Data policies (text only)
ksqlDB	SQL pull and push queries over a declared stream	ksqlDB servers	SQL clauses	Through joins between declared streams	The format declared on the stream	Not documented in the pages checked	ksqlDB queries documentation
Redpanda Console	JavaScript push filters per record	Redpanda Console’s backend	Not documented in the pages checked	Not documented	JSON, Avro, Protobuf, CBOR, XML, MessagePack, text and binary	RBAC needs an enterprise licence	Redpanda docs: Programmable Push Filters (text only)
Offload to a search index	The index’s query language	The index	Whatever is retained in the index	Yes, across everything you index	Decoded on the way in by a sink connector	The index’s own access control	Kafka Connect sink to Elasticsearch or OpenSearch

Here is how the scores work out. On filter language, Kpow’s kJQ, Conduktor’s JavaScript and SQL, Lenses SQL, ksqlDB and Redpanda Console’s JavaScript all express structured conditions, and AKHQ’s match modes and the CLI’s grep do not. On where the filter runs, every UI here filters in its own backend, and only the CLI pipelines pull every record to your machine. On multi-topic search in one query, Kpow documents a topic list, a regex and a consumer-group mode, Lenses and ksqlDB get there through joins, and the console consumer can read several topics by regex but filters nothing. On scan scope, Kpow documents the most controls. On masking, Kafbat UI and AKHQ include it free, while Kpow, Conduktor and Redpanda put it or RBAC behind a paid tier. The search index beats all of them on query speed once data is in it, at the cost of running a second system and only finding what you indexed.

How Factor House approaches it

Finding data is the reason most people install Kpow in the first place. Derek, our co-founder and CEO, described it on a podcast as the thing everyone uses Kpow for: you put messages into Kafka, you do not know what happened to them, and you use the data inspector to find out. Kpow’s data inspect is built around the criteria above. kJQ gives a structured filter that runs on Kpow’s server, the four query modes and the scan limit bound what gets read, and one query can span a topic list, a regex or everything a consumer group consumes. Each record shows its schema ID and deserializer, and records that fail to deserialize can be kept and flagged or listed on their own, which turns a poison message hunt into a query. The full procedure for that case is in how to find and skip a poison pill in Kafka, and once found, dead-lettered records are the subject of Kafka DLQ tools.

A kJQ filter for failed orders with more than three retries in the last thirty minutes looks like this:

.value.status == "failed" and .value.retryCount | to-long > 3 and .timestamp > now - pt30m

For teams that do not want to learn the syntax during an incident, Kpow Enterprise can generate the filter from a plain-English request using your own AI model, as covered in the post on accelerating incident response with AI queries. Results can be exported, copied, or produced to another topic, which is how a found message gets replayed. The kJQ manual has the full operator reference.

The test worth running in a Kpow demo is a real cross-topic search: pick a topic regex that covers several topics, write a kJQ filter on one field, bound it to the last hour, and see how long it takes to get from the question to the matching records.

For the step-by-step on running a query in Kpow once you have picked a tool, see how to query a Kafka topic.

Kpow live demo

Search Kafka messages with kJQ

Open the Kpow demo and filter topic data with kJQ, with automatic deserialization and no consumer to write.

Built for platform and data engineers running Kafka in production.

Try the Kpow demo→

FAQ

How do I search for a message across multiple Kafka topics?

Use a tool that takes several topics in one query: Kpow’s data inspect accepts a topic list, a topic regex or a consumer group, and Lenses SQL and ksqlDB can join topics. With the CLI, kafka-console-consumer.sh --include '<regex>' reads several topics but filtering is up to grep or jq on your machine.

Can I search Kafka messages by field value?

Yes, with a structured filter: kJQ in Kpow, CEL in Kafbat UI, JavaScript in Conduktor, Kafka Magic and Redpanda Console, or SQL in Lenses and ksqlDB. AKHQ matches key, value and headers by equals or contains rather than by field.

Why is searching Kafka slow?

Kafka has no index on message content, so every search reads and decodes records from a starting point. Bound the search with a time window, partition or key, and use a tool that filters on its server rather than on your laptop. If you search the same history repeatedly, sink it into a search index.

Is there a free tool to search Kafka messages?

kafka-console-consumer and kcat with jq are free, as are Kafbat UI and AKHQ. Kpow Community Edition is free for up to three clusters and includes kJQ and multi-topic search.

The rest of the operational picture is in the complete Kafka guide.

Related reading

• How to query a Kafka topic
• Kafka topic vs partition
• Kafka offsets
• Kafka consumers in production
• Kafka: the complete guide
• How to find and skip a poison pill in Kafka