How to let an AI agent operate Kafka safely
GovernanceLetting an AI agent operate Kafka safely means giving it its own scoped identity, read-only access by default, a human approval step before any destructive change, an audit record of every action, and no raw sensitive payloads in its context.
The request usually arrives as “can we point Claude at the cluster?”, and the honest answer is yes, once the agent is treated as a new principal with less trust than the engineer who asked. The steps below apply whether the agent reaches Kafka through an MCP server, a vendor CLI, a REST proxy or a shell, and they follow the order you would set them up in. The Kafka governance section of the complete Kafka guide holds the underlying reference pages.
The two layers to secure
An agent reaches Kafka through two connections, and each needs its own control.
The first is from the AI client to the agent’s interface: the MCP server, CLI or API the agent calls. This layer decides who the agent is acting for, which tools it can see, and whether a person confirms a call before it runs.
The second is from that interface to the cluster: the Kafka principal, TLS and SASL settings, and the ACLs the broker enforces. This layer is the backstop. If everything above it fails, a principal with Describe-only ACLs still cannot delete a topic.
Most incidents with agents will come from treating the first layer as sufficient. An MCP server’s tool list is advisory to the model. The MCP tools specification says clients “MUST consider tool annotations to be untrusted unless they come from trusted servers”, and a broker ACL is the only control the model cannot argue with.
Authenticate the agent through your identity provider
Sign the agent in through the same OAuth 2.x or OpenID Connect provider your engineers use (Okta, Microsoft Entra ID, Keycloak, AWS IAM Identity Center), so it acts as a named identity with a short-lived token. Never paste a long-lived admin key into an agent’s config file.
The Model Context Protocol’s authorization specification makes authorization optional, and says that when it is supported, servers on an HTTP transport should follow its OAuth 2.1-based flow, while stdio servers “retrieve credentials from the environment”. Two rules from it matter for Kafka:
- the MCP server must validate that a token was issued for it, and must not pass the token it received through to another API
- the client must use PKCE, so an intercepted authorization code is useless
A local stdio server inherits whatever credentials sit in its environment. That is acceptable for a laptop pointed at a dev cluster and unacceptable for production, where the credential should be a dedicated principal, covered below. The Kafka SSO tools comparison covers which tools can front Kafka with your identity provider.
Authenticate to the brokers properly
Whatever the interface is, its connection to Kafka should use the same authentication as any production client: SASL/SCRAM or mutual TLS, over TLS. Create a separate SCRAM user for the agent so its traffic is attributable:
kafka-configs.sh --bootstrap-server broker1:9093 --command-config admin.properties \
--alter --add-config 'SCRAM-SHA-512=[password=<from your secret store>]' \
--entity-type users --entity-name agent-diagnosticsThe agent’s client config then carries that user:
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="agent-diagnostics" password="<from your secret store>";With mutual TLS, map the certificate’s distinguished name to a stable principal with ssl.principal.mapping.rules, as covered on Kafka authentication. Either way, the broker now sees User:agent-diagnostics, not a shared service account. The mechanisms are documented in Apache Kafka’s SASL authentication guide.
Least privilege: a dedicated identity for each agent
Give every agent, or every agent use case, its own principal, and grant it only the operations its job needs. A diagnostics agent needs to describe topics, read configs and read consumer group offsets. It does not need to read records, and it never needs Delete or Alter.
Apache Kafka’s authorization documentation is the reference for which operation each request needs. Fetching committed offsets needs Describe on the group and the topic, not Read, so a principal with these ACLs can report lag per partition without consuming a single message:
kafka-acls.sh --bootstrap-server broker1:9093 --command-config admin.properties \
--add --allow-principal User:agent-diagnostics \
--operation Describe --operation DescribeConfigs --topic '*'
kafka-acls.sh --bootstrap-server broker1:9093 --command-config admin.properties \
--add --allow-principal User:agent-diagnostics \
--operation Describe --group '*'Kafka denies anything without a matching ACL: “If a resource (R) does not have any ACLs defined, meaning that no ACL matches the resource, Kafka will restrict access to that resource.” Add explicit deny rules anyway, because a broad User:* allow written for someone else would otherwise reach the agent, and deny always wins:
kafka-acls.sh --bootstrap-server broker1:9093 --command-config admin.properties \
--add --deny-principal User:agent-diagnostics \
--operation Delete --operation Alter --operation AlterConfigs --topic '*'Check what the principal ended up with:
kafka-acls.sh --bootstrap-server broker1:9093 --command-config admin.properties \
--list --principal User:agent-diagnosticsThe full ACL syntax, including prefixed patterns, is on Kafka ACL, and RBAC roles covers how the same idea is packaged as roles in management tools.
Default to read-only, and gate writes behind approval
Start every agent read-only and add writes one at a time, each with a human approval step in front of it. When I needed to reverse engineer some Terraform in our own AWS account, I created a read-only role first so I could inspect everything without any chance of deleting it. An agent is in the same position on its first day: it is exploring an environment it does not understand.
Diagnosis covers most of the value anyway. In our demo of an AI assistant working against Kafka, it traced two failed connectors to a catalog limitation and concluded that “restarting the connector will just fail until that’s fixed”. It changed nothing, and the fix was a human decision.
When you do add writes, put the approval in the platform, not in the prompt. The guidance is consistent across neutral sources:
- the MCP tools specification says clients should “Present confirmation prompts to the user for operations, to ensure a human is in the loop”
- OWASP’s LLM06 Excessive Agency names excessive functionality, excessive permissions and excessive autonomy as root causes, and advises against open-ended extensions such as running a shell command
A confirmation prompt in the AI client is useful, but it is a click-through for a tired engineer. A staged change that an administrator approves in the platform, with the request recorded, is a control. If your tooling has no approval step, keep the agent read-only and have it print the command for a person to run. For break-glass access during an incident, grant a write for a fixed window and let it expire, rather than widening the agent’s standing permissions.
Keep sensitive data out of the model
Anything a tool returns becomes model context, and that context leaves your network for the model provider. In the demo I said an agent is only as good as the context you give it. The corollary is that it only leaks what you give it.
Three controls, in order of strength:
- Withhold Read on topics that carry personal or payment data. Without Read, the agent’s principal cannot consume those records at all, and the Describe-only ACLs above already do this.
- Choose tools with no record-reading capability, or switch those tools off. Several MCP servers expose consume tools that return raw messages, and some let you leave them out.
- Mask at the source of the read. If the agent must look at records, route the read through a layer that redacts fields server-side before the result is returned.
Finding which topics and fields carry PII in the first place is covered step by step in how to find and mask PII in Kafka topics.
Audit every agent action alongside human actions
Record every tool call the agent makes, with its identity, in the same audit trail as human actions. An engineer at Current London 2025 put the requirement in one line, recorded by Derek, our co-founder and CEO: “If an LLM is making decisions in my pipeline, I want to know what it saw, why it acted, and how to stop it fast.”
Kafka gives you part of this. The broker’s authorizer logs decisions by principal, and in Apache Kafka’s StandardAuthorizer denials are logged at INFO and allowed requests at DEBUG, so a dedicated agent principal makes its denied attempts easy to find. It does not record which prompt or tool call led to the request. That record has to come from the interface layer: the MCP server’s tool-call log, or the audit log of the platform the agent works through. The layers and the tools for each are compared in Kafka audit logging tools.
Review the agent’s trail the way you would a new engineer’s for the first few weeks: what it read, what it tried that was denied, and what it changed.
Multi-tenancy: limit which clusters and tenants the agent can see
Scope the agent to the clusters and resources of the team that owns it. On a shared cluster, use prefixed ACLs so a team’s agent sees only that team’s topics:
kafka-acls.sh --bootstrap-server broker1:9093 --command-config admin.properties \
--add --allow-principal User:agent-payments \
--operation Describe --operation DescribeConfigs \
--topic payments. --resource-pattern-type prefixedGive each environment its own principal too. An agent with credentials for dev, staging and production in one config file is one mistaken prompt away from running a dev fix in production. Multi-tenant architecture covers naming conventions and quotas that make prefix scoping reliable.
If you are still choosing the interface the agent will use, the options differ widely on these controls. Best Kafka MCP servers scores every current option on read and write scope, permission model, audit, approval and message exposure.
How Factor House approaches it
Kpow gives agents the same interface engineers use and puts them under the same governance, rather than adding a separate agent product with its own permission model. In the September 2026 demo I put it this way: “your CLI users, terminal UI users, and agentic users all get the same control and governance in place - you don’t need to do anything additional to your existing OIDC workflows.”
There are three ways in, and all three go through the Kpow API as the identity that signed in. fh, the Factor House command-line interface, covers almost everything the web application does, apart from producing and consuming messages. Agent skills drive that CLI, which is how Claude works through Kpow in the demo. The Kpow MCP server runs on version 2 of the Kpow API and signs in through the same OpenID provider, which is the reason OIDC is the sign-in for all three. The steps above map onto Kpow Enterprise like this:
- Identity:
fh auth loginsigns in to your OpenID provider with the authorization code flow and PKCE, and the authentication documentation covers the providers. - Least privilege: Kpow RBAC grants Allow, Deny or Stage per action on a cluster, topic, group, connector or schema subject, and actions you do not grant are implicitly denied.
- Approval: the Stage effect turns an action into a staged mutation that an admin approves or denies in the UI, and
fhreports “Mutation staged for admin approval” instead of running it. Temporary policies grant an action for a fixed window, which is the break-glass path. - Masking: data policies redact fields in Data Inspect results server-side, and the Kpow product page states the unmasked value never reaches the browser or the API response.
- Audit: every user action goes to the audit log, and the webhook integration can send mutations, Data Inspect queries or both to Slack, Teams or any endpoint, marked with whether the request came through the API or the UI.
- Tenancy: multi-tenancy limits what a role can see, and
fhselects a tenant with--tenant-id.
A policy for an agent’s role looks like this. It stages topic creation and consumer group edits for the payments groups, and grants nothing else, so Data Inspect (TOPIC_INSPECT) and topic edits stay denied:
policies:
- resource: ["cluster", "*"]
effect: "Stage"
actions: ["TOPIC_CREATE"]
role: "kafka-agent"
- resource: ["cluster", "*", "group", "payments_*"]
effect: "Stage"
actions: ["GROUP_EDIT"]
role: "kafka-agent"These governance controls are Enterprise features, described on the Kpow governance and audit page. They cover an agent because each one is enforced on the server, not in the client that made the call. The Kpow product page is explicit about masking: the unmasked value never reaches the browser or the API response, so a policy applies whether the record is read through the UI or the REST API, and an agent reading through the CLI or the MCP server is reading through that API.
Kpow live demo
See an agent work under your existing RBAC
Watch Claude diagnose failing connectors and a consumer that looks stuck through Kpow's CLI, signed in with the same SSO, RBAC and tenancy as the web UI.
Built for platform and data engineers running Kafka in production.
Watch the demoA checklist before you give an agent production access
Run through this list in order, and do the first pass against a non-production cluster that mirrors production’s ACLs.
- The agent has its own identity in your identity provider and its own Kafka principal, with no shared or admin credentials in its config.
- The broker connection uses SASL/SCRAM or mutual TLS over TLS.
- ACLs grant only Describe and DescribeConfigs, explicit deny rules cover Delete, Alter and AlterConfigs, and
kafka-acls.sh --list --principalshows nothing else. - In non-production, ask the agent to delete a scratch topic and confirm the broker refuses it with an authorization error.
- The interface runs read-only, or every write passes through an approval step in the platform.
- Topics carrying personal or payment data are either unreadable by the agent or read through server-side masking.
- Every tool call lands in an audit trail your team reviews, and the agent’s denied attempts are visible in the broker’s authorizer log.
- The agent sees only its own team’s clusters and topic prefixes, with a separate principal per environment.
- You know how to revoke it: disable the identity, remove the ACLs, and confirm the next call fails.
FAQ
Should an AI agent have write access to production Kafka?
Not by default. Start read-only, which covers most of the diagnostic value, and add individual writes only behind an approval step in the platform, with each write recorded in the audit log. If your tooling cannot stage a change for approval, have the agent print the command and let a person run it.
How do I stop an AI agent from deleting Kafka topics?
Enforce it at the broker. Give the agent its own principal, grant only Describe and DescribeConfigs, and add an explicit deny ACL for Delete on topics, which overrides any broader allow. A read-only flag on an MCP server helps, but the ACL is the control the agent cannot bypass.
Does MCP require OAuth?
Not always. The MCP authorization specification makes authorization optional. When an HTTP-based MCP server supports it, the specification’s flow is based on OAuth 2.1 with PKCE, and local stdio servers take credentials from their environment instead. For production Kafka, prefer an HTTP server behind your identity provider.
Can I audit what an AI agent did on my Kafka cluster?
Yes, if you set it up. A dedicated Kafka principal makes the agent’s requests identifiable in the broker’s authorizer log, and the interface it uses must log each tool call with the identity behind it. Confluent’s managed MCP server, for example, writes each tool call to the Confluent Cloud audit log, and Kpow’s audit log records actions made through its API.