Kafka AI troubleshooting means giving an AI agent read access to your cluster’s live state, through a CLI, an API or an MCP server, and asking it the questions an on-call engineer asks: which connectors are failing and why, whether a service is consuming, and why lag is growing.
The question usually arrives from an application team as “my service isn’t getting data”. I used to get asked that a lot, and half the time the answer is that the service is getting data and something else is wrong with it. An agent with the right access can prove which half you are in within a minute, with offsets and timestamps rather than a guess. This guide covers the access to give it, the questions to ask, how to check its answer, and where to stop it. The complete Kafka guide holds the wider operations picture.
What AI-assisted Kafka troubleshooting is
An agent in this sense is a model such as Claude working in a loop with tools: it runs a command or calls an API, reads the result, decides what to look at next, and repeats until it can answer. Troubleshooting with one follows the same diagnostic sequence you would run by hand, with the agent running the commands and reading the output.
That is different from the other thing people mean by “Kafka AI”, which is Kafka as the event backbone for AI applications. It is also different from an AI feature inside a Kafka tool, such as Kpow’s bring-your-own-model filter generation in Data Inspect, which turns a plain-English question into a message filter and is covered in accelerating incident response with AI-powered queries. This page is about an external agent investigating a live cluster.
What an agent needs to be useful
An agent needs three things before it can diagnose anything: read access to the cluster’s state, a tool that returns that state in a form it can parse, and context about what to look for.
Most Kafka incidents are not Kafka bugs. In my talk on Kafka operational incidents I put it as misconfigurations that made sense at the time, missing observability that hid a real signal, or a reasonable decision made without full context. An agent has the same failure mode as a person: if it cannot see connector task traces or committed offsets, it guesses.
Give the agent its own read-only identity before anything else. On plain Kafka that is a principal with Describe access and little more, and the Apache Kafka authorization documentation is clear that a resource with no matching ACL is denied to everyone but super users. Describe on topics and groups lets the agent list offsets, fetch a group’s committed offsets and describe the group, and Describe on the cluster lets it list groups:
kafka-acls.sh --bootstrap-server broker:9092 --command-config admin.properties \
--add --allow-principal User:ai-readonly --operation Describe --topic '*'
kafka-acls.sh --bootstrap-server broker:9092 --command-config admin.properties \
--add --allow-principal User:ai-readonly --operation Describe --group '*'
kafka-acls.sh --bootstrap-server broker:9092 --command-config admin.properties \
--add --allow-principal User:ai-readonly --operation Describe --clusterLeave Read off the groups. The same documentation lists Read on the group as the permission an offset commit needs, so a principal holding it can move a group’s committed offsets. Add Read on topics only if you want the agent to see message contents.
The tool can be the Kafka scripts, a CLI like kcl whose --help-json flag hands an agent its whole command tree, the Kafka Connect REST API, one of the Kafka MCP servers, or a vendor CLI with structured output. Prefer JSON over text tables wherever the tool offers it. Context can come from a packaged skill or from a file you write yourself naming your clusters, your conventions and which consumer groups belong to which team.
If you are still choosing between packaged skills, MCP servers and raw CLI access, Kafka agent skills compared scores each option on what the agent can see, what it can change, and whether it runs under your own SSO and RBAC.
That's what's really important about agentic workflows: an agent is only as good as the context you give it.
Chad Harris, Solutions Architect at Factor House
Root cause analysis: failed connectors, stuck consumers and poison pills
Start with an open question and let the agent work down from it. “Are my connectors healthy?” is a good first prompt because the answer is in one place. The Kafka Connect REST API lists every connector, and the status endpoint returns the connector state, every task’s state, the worker it runs on and the error if it failed, per the Kafka Connect user guide:
curl -s http://connect:8083/connectors
curl -s http://connect:8083/connectors/<name>/statusA good agent checks every task, not only the connector, because a connector can report RUNNING while its tasks have FAILED. When I asked this question in the September demo, the agent found three connectors not moving data, pulled the exception from each failed task and read it, rather than stopping at the word FAILED.
For stuck consumers, the question is which partition stopped. Ask the agent to describe the group and compare CURRENT-OFFSET with LOG-END-OFFSET per partition, then run it again a minute later:
kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group <group>One partition whose committed offset has not moved while the rest advance is the signature of a poison pill: a record the consumer cannot process, retried forever. How to find and skip a poison pill covers confirming and clearing it. Ask the agent to name the partition and offset, then stop it there, because skipping a record is a data decision for the team that owns the consumer.
Telling a Kafka fault from an external-system fault
Many connector failures sit in the external system rather than in the connector. The most useful thing an agent can tell you is who has to fix it, and the stack trace usually says so.
In the demo, two Iceberg sink connectors I had just deployed against Databricks Unity Catalog were exiting on an unrecoverable exception. The agent read the trace and concluded that Unity Catalog did not support the endpoint the connector used to create the Iceberg table, and that restarting would fail until that changed. The fix it proposed was to create the tables by hand in Unity Catalog and turn off table auto-creation, and then the connectors start. The third, a Debezium Postgres source, failed with connection refused, meaning the database was down or unreachable, and again a restart would not help.
Ask for the verdict in that shape: whose fault, what evidence, and whether a restart can change anything. How to diagnose and fix a failed Kafka Connect connector walks the same diagnosis by hand, including the config setting behind table auto-creation.
Proving whether a service is consuming
“Why isn’t my service getting data?” is the question an agent handles best, because the answer is a set of facts the cluster already holds. Ask it to establish, in order:
- Messages are arriving on the topic: the log end offset is moving.
- The group exists and is Stable, with the members you expect.
- The group’s committed offsets are advancing, and how far behind the log end they are.
- When the group last committed.
The group-level facts come from two describe options on the same tool:
kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group <group> --state
kafka-consumer-groups.sh --bootstrap-server broker:9092 --describe --group <group> --membersIn the demo the answer for an airline analytics service was that it was getting data: messages were being produced to the topic, the group was stable, it was two messages behind, and the last commit was at the time of recording. That moves the investigation out of Kafka and into the service itself, which is where it belonged. The agent also handed back the CLI command to run, so the check could be repeated without the agent next time. Ask for that from any agent.
Consumer lag and rebalance diagnosis
Ask for two lag readings a minute or so apart, because a single lag number cannot tell you whether a group is falling behind or catching up, and have the agent report the direction per partition. How to monitor Kafka consumer lag covers the metrics for doing this continuously. If you are asking an agent for two readings because nothing is recording lag over time, fix that first: the best Kafka monitoring tools compares what keeps the history and fires the alert.
Rebalances show up as a group state other than Stable and a member list that keeps changing. In the demo, the agent noticed from repeated observations that the service instance had been restarting, and suggested temporary latency during rebalances as a lead. That is a hypothesis, not a finding, and it should be checked against the member list and the application’s own restart history. What is Kafka rebalancing? explains which configs turn restarts into rebalance storms.
Be careful with the fix the agent proposes for lag. In one incident from my talk, a team doubled a service to 800 instances to fix lag when the real cause was tens of thousands of idle consumer members overloading the group coordinator. An agent asked “how do I reduce lag?” will propose scaling out too, unless you ask it to count members against partitions first.
Where AI troubleshooting stops
Stop the agent at the diagnosis, and make the change yourself. OWASP’s LLM06:2025 Excessive Agency describes the failure as an application that “fails to independently verify and approve high-impact actions”, and the MCP specification says there should always be a human able to deny a tool call.
Three rules keep an agent useful without letting it hurt you:
- Read-only identity first. Grant write access only for a specific, reviewed task, and take it away afterwards. How to let an AI agent operate Kafka safely sets up that identity step by step.
- Verify the conclusion against the metric it cites. If the agent says the group is two messages behind, run the describe yourself and read the number. If it says the catalog rejected a call, read the trace.
- Treat a restart recommendation with suspicion. When the fault is in a database, a catalog or a downstream API, a restart changes nothing, and automated restarts in a loop add load to the thing that is already down.
Client-side controls help but are not a boundary. Claude Code evaluates permission rules in the order deny, ask, allow, so you can deny destructive commands outright:
{
"permissions": {
"deny": ["Bash(kafka-topics.sh *--delete*)", "Bash(kafka-consumer-groups.sh *--reset-offsets*)"]
}
}The Claude Code permissions documentation also notes that a deny rule does not match the same program called by path or inside sh -c. That makes the identity the agent signs in with the control that holds. kcl adds one more layer: when stdin is not a terminal, its destructive prompts answer no and it prints the plan instead, so an agent gets a dry run by default.
How Factor House approaches it
Kpow’s approach is the one in the demo: agent skills that teach an agent how to query Kafka through the Kpow fh CLI, which calls the Kpow API. In Claude, a question such as “are my connectors healthy?” loads a Kpow skill for Connect status, which tells the agent how to query the CLI, and the workflow takes around 30 seconds to a minute. The CLI’s --output json flag is the form agents read.
The governance comes from Kpow rather than from the agent. The CLI signs in through your deployment’s OpenID provider, so the agent works under your existing SSO, role-based access control and tenancy, and an RBAC policy can stage an action for an administrator’s approval through staged mutations instead of letting it run. There is no separate agent mode to switch on, because the control is the role: a policy allows, denies or stages each action, so a read-only agent is one whose role allows the reads and stages or denies every write. The CLI does not produce or consume messages, so an agent working through it can see offsets, lag and connector traces, but not message contents. For those, the Data Inspect documentation covers searching topics in the Kpow UI and terminal UI.
The Connect and consumer group views that the CLI reads are the same ones in the Kpow UI: Kafka Connect management shows connector and task state with stack traces, and consumer groups shows lag per partition and member, which is where to check the agent’s answer.
Product demo · 11 min
Kpow CLI, terminal UI, and agentic skills
Chad Harris previews Kpow's new CLI and terminal UI for Apache Kafka, plus the agentic skills that let an AI assistant query, diagnose, and operate Kafka through Kpow under your own SSO and RBAC.
Kpow live demo
Check the agent's evidence yourself
Open the Kpow demo to look at connector and task state, consumer group offsets and lag side by side, the same evidence an agent should cite.
Built for platform and data engineers running Kafka in production.
Try the Kpow demoGetting started
Whatever tools you use, set up in this order.
- Create a read-only identity for the agent: a Kafka principal with Describe ACLs, plus Read on topics only if it should see messages, or a read-only role in whatever platform fronts your cluster.
- Give the agent one tool with structured output: the Kafka scripts and the Connect REST API, kcl, an MCP server scoped to read, or a vendor CLI with a JSON output flag.
- Write a short context file: cluster names and which is production, the Connect URL, naming conventions, and which consumer groups belong to which team.
- Add deny rules in the agent’s settings for delete and reset commands, as a second line behind the identity.
- Ask an open question on a non-production cluster first, such as “are my connectors healthy?”, and check every claim in the answer against the command output.
- Ask the agent to end each answer with the commands it ran, so you can rerun them yourself.
On Kpow, steps 1 and 2 are the fh CLI signed in with fh auth login under a Kpow role that can read and not write, with --output json for the agent. Install it from Factor House’s Homebrew tap with brew install factorhouse/tap/fh. The recording is on the Kpow CLI, terminal UI and agentic skills demo page.
FAQ
Can an AI agent fix Kafka issues automatically?
It can if you give it write access, and you should not start there. An agent is reliable at gathering evidence and proposing a cause, and less reliable at judging whether a restart, an offset reset or a config change is safe. Keep it read-only and make changes yourself, or route them through an approval step such as a staged mutation.
What tool can I use to view Kafka messages?
From the command line, kcat and kcl consume and print messages, and the bundled kafka-console-consumer.sh does the same. A Kafka UI such as Kpow lets you search message contents with filters across topics. Kpow’s CLI does not consume messages, so an agent working through it sees offsets and metadata, not payloads.
Can AI diagnose a failed Kafka Connect connector?
Yes, and it is one of the better jobs for an agent, because the status endpoint returns each task’s state and stack trace in JSON. The value is in reading the trace and saying whether the fault is in the connector or in the external system, which decides whether a restart can help.
What is the difference between AI for Kafka operations and Kafka for AI?
AI for Kafka operations means an agent investigating or managing a Kafka cluster, which is what this page covers. Kafka for AI means using Kafka to feed data and events to AI applications, a separate architecture question.