Kafka authentication verifies the identity of every client and broker connection before any data moves. Kafka supports two families of mechanisms, SASL (GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER) and mutual TLS, configured per listener. For new deployments without existing Kerberos infrastructure, SCRAM-SHA-512 over TLS is the most practical starting point. Authentication is one layer of the security model covered across the complete Kafka guide.
Authentication config is where I have seen the most 2am pain for the least architectural reason. Most Kafka operational problems are not Kafka bugs. They are misconfigurations, missing observability, or reasonable decisions made without full context, and auth sits at the top of that list. As I put it before a recent talk on exactly this: the failures I have seen rarely announce themselves. A subtle misconfiguration. A metric that looks fine until it does not. Treat every listener and JAAS block on this page as something you test in a staging cluster before it touches production.
Concrete configuration blueprints
Kafka authentication is configured in three places that must agree: the listeners, the JAAS login modules, and the client properties.
A client connection configured for SASL_SSL with keystore and truststore, and broker logs for the authenticated connection.
Listeners bind a security protocol to a port. The four options are PLAINTEXT, SSL, SASL_PLAINTEXT and SASL_SSL, mapped per listener:
listeners=CLIENT://:9092,BROKER://:9093
listener.security.protocol.map=CLIENT:SASL_SSL,BROKER:SASL_SSL
inter.broker.listener.name=BROKERJAAS carries the credentials. Brokers can configure it either as a static JAAS file with a KafkaServer section, or inline in server.properties with the listener-and-mechanism prefixed property, which is the more operable form because it needs no JVM flag:
listener.name.sasl_ssl.scram-sha-256.sasl.jaas.config=\
org.apache.kafka.common.security.scram.ScramLoginModule required \
username="admin" \
password="admin-secret";When JAAS is defined at more than one level, the broker property wins over the listener-named static section, which wins over the plain KafkaServer section.
Clients set the same pair on their side: sasl.mechanism and sasl.jaas.config in producer, consumer or admin client properties.
Moving a live cluster from PLAINTEXT to SASL_SSL is done with listeners rather than a big-bang switch. The Apache documentation’s incremental procedure runs four phases, each an incremental bounce: open the secured port alongside the plaintext one, move clients to it, switch inter-broker traffic, then close the plaintext port. A plaintext port stays open until the final phase so brokers and clients keep communicating, and each bounce waits for restarted replicas to rejoin the ISR before the next node.
The blueprint pattern we point enterprise teams at is JPMorgan’s: federated identity through Active Directory Federation Services wired into Kafka’s pluggable SASL/OAUTHBEARER support (KIP-255). It shows that the mechanism list below is not a menu of equals. If your organisation already runs a central identity provider, OAUTHBEARER is the option that lets Kafka join it instead of maintaining a parallel credential store.
Enterprise authentication mechanisms
SASL/SCRAM. SCRAM-SHA-256 and SCRAM-SHA-512 (RFC 5802) authenticate with username and password without ever sending the password in cleartext, and Kafka stores the salted credentials in the cluster metadata log. Credentials are created before first broker start with kafka-storage.sh format --add-scram, and dynamically afterwards:
bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter \
--add-config 'SCRAM-SHA-256=[iterations=8192,password=alice-secret]' \
--entity-type users --entity-name alice --command-config client.propertiesThe default iteration count is 4096. Because SCRAM protects the exchange but not the wire, it is run over TLS in production, security protocol: SASL_SSL.
Mutual TLS. With ssl.client.auth=required on the broker, clients present certificates signed by a CA in the broker truststore, and the certificate’s distinguished name becomes the principal. Kafka expects keys and certificates in keystores, PKCS12 format, managed with keytool. The operational cost of mTLS is certificate lifecycle: issuing, distributing and renewing client certificates before they expire.
SASL/OAUTHBEARER. Kafka’s default OAUTHBEARER implementation creates and validates unsecured JWTs and is suitable only for non-production use. Production use runs the OAuth 2.0 integration against an identity provider, with the broker validating tokens against the IdP’s JWKS endpoint and clients fetching tokens from the token endpoint:
listener.name.<listener>.oauthbearer.sasl.oauthbearer.jwks.endpoint.url=https://example.com/oauth2/v1/keys
sasl.oauthbearer.token.endpoint.url=https://example.com/oauth2/v1/tokenOAUTHBEARER is increasingly common in cloud-native deployments because it removes per-cluster credential stores entirely, identity lives in the IdP. Our read on where this is heading: SASL/OAUTHBEARER is on track to become the standard authentication mechanism for cloud-native Kafka deployments. mTLS is not going anywhere for broker-to-broker links, but new client-facing listeners in identity-provider shops are increasingly OAuth-shaped.
SASL/GSSAPI (Kerberos). The right choice where a Kerberos realm (typically Active Directory) already exists, and rarely adopted without one.
Production troubleshooting and edge cases
Handshake failures are the common failure class. The two standard diagnostic moves are testing the TLS layer from outside the JVM: openssl s_client -connect broker:9093, and turning on the JVM’s own TLS debugging: -Djavax.net.debug=ssl. Between them they separate certificate problems (expired, wrong SAN, untrusted CA) from SASL problems (wrong mechanism, bad credentials, mechanism not enabled on the listener).
Credential rotation differs by mechanism. SCRAM credentials update dynamically through kafka-configs.sh, and updated credentials apply to new connections without a broker restart. TLS certificates rotate by updating keystores and truststores, and dynamic broker configuration allows keystore updates without a restart. Rotating the CA itself is the hard case, it needs a trust-both window where old and new CAs are in every truststore.
Authentication has a measurable performance cost, and it is mostly TLS. Production figures from our Pinterest architecture analysis put TLS memory overhead at roughly 122 KB per SSL connection at scale, which adds up on brokers carrying tens of thousands of connections. The handshake is the expensive moment, steady-state encryption overhead is modest on modern JVMs, and Java 11 and later perform significantly better with TLS enabled than Java 8, part of the runtime picture in our Java compatibility and evolution strategy.
The compliance trap we warn about: teams enable DEBUG-level authorizer logs to satisfy an audit requirement, then discover that on a busy cluster processing 10,000 requests per second those logs generate gigabytes per hour, so they get switched off and the log captures denials but not successful access. That is not an audit trail. Plan the audit story separately from broker log levels.
Downstream authorization: from identity to ACLs
Authentication produces a principal, and everything after that is authorization. How the principal is derived depends on the mechanism. SCRAM and PLAIN use the username. Mutual TLS uses the certificate’s full distinguished name by default, shortened with mapping rules in server.properties:
ssl.principal.mapping.rules=RULE:^CN=(.*?),OU=ServiceUsers.*$/$1/,DEFAULTKerberos principals map through sasl.kerberos.principal.to.local.rules. Custom logic goes through principal.builder.class.
The resulting principal, written User:alice, is what ACLs bind to. Kafka ACLs grant or deny operations on resources, topics, consumer groups, transactional ids and the cluster itself, to exactly these principals. Getting the mapping right matters operationally because an mTLS deployment that skips mapping rules ends up writing ACLs against full distinguished names, which are brittle and unreadable. The ACL model, syntax and best practices have their own page: Kafka ACL, and role-level access design sits in RBAC roles. In a shared cluster, principals also anchor the isolation model described in multi-tenant architecture.
Two field notes from the ACL boundary. PayPal removed the plaintext port that existed in its earlier deployment when it moved to SASL-based ACLs, and got a capacity-planning win alongside the security one, because authenticated intent per topic gives you visibility into who actually uses what. And the cheapest lesson on this page: missing the DESCRIBE permission produces cryptic “unknown topic” errors even when the topic exists. We have watched teams burn hours on that dead end.