A multi-tenant Kafka architecture shares one cluster among multiple teams, applications or customers, with quotas, ACLs and naming conventions keeping each tenant’s traffic and data isolated from the others. Apache Kafka ships every isolation primitive this requires: client quotas, prefixed ACLs and per-topic configuration. This page covers the governance corner of the topic in depth. For the whole operational picture, start with the complete Kafka guide.
I push back when teams treat multi-tenancy as a compromise. As I argued in my talk on Kafka operational issues, Kafka is designed to be a multi-tenanted, highly scalable, shared piece of infrastructure. It is meant for sharing data across your organization. GitOps and cloud vendors have made spinning up a new cluster so easy that I have seen organizations scale prematurely, where every team gets their own Kafka cluster or every service gets its own Kafka cluster, and you end up with a proliferation of clusters nobody governs. I encourage teams to only scale clusters when they really need to, generally for headroom or for genuine throughput demand. The rest of this page is about doing shared infrastructure properly instead of avoiding it.
Resource isolation and noisy neighbors
The noisy neighbor problem is one tenant consuming enough broker resources that other tenants degrade. Kafka’s answer is client quotas, which are enforced per broker and throttle a client group once it exceeds its allowance.
Two quota types exist in Apache Kafka. Network bandwidth quotas cap byte rates per client group: producer_byte_rate and consumer_byte_rate. Request rate quotas cap the share of broker request-handler and network thread time a client group may use: request_percentage. Request rate quotas often matter more in shared clusters, because a client that hammers the broker with small requests burns CPU without moving many bytes.
Quotas attach to a user principal, a client id, or the pair of both, and the most specific match wins. Setting one is a single command:
bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter \
--add-config 'producer_byte_rate=1024,consumer_byte_rate=2048,request_percentage=200' \
--entity-type users --entity-name user1Kafka also supports broker-side connection limits, a cap on new connection rate, connections per broker, and connections per IP address. Topic creation storms are governed separately by the controller_mutation_rate quota (KIP-599).
When quotas stop being enough, the topology question comes next. Our Kafka cluster management guide puts the threshold for considering a second cluster at around 200 brokers, at repeated cross-tenant impact from one noisy tenant, or at regulatory data-residency requirements that force physical separation.
The scale ceiling on a shared cluster is higher than most teams assume. In KRaft mode the old ZooKeeper-era limit of roughly 4,000 partitions per broker no longer applies, and our monitoring guide still treats 10,000 to 20,000 partitions per broker as the sensible practical upper bound for most workloads. The stronger precedent is PayPal: by 2020 its deployment was approaching 1 trillion messages per day, and that is the point at which it formalised multi-tenant fleet management rather than splitting into per-team clusters. Robinhood attacked the noisy-neighbor problem from the client side, using a sidecar proxy to aggregate connections from each pod because the isolation was worth more than async client libraries.
Security and access control
Tenant isolation is only as strong as the security layers under it. Kafka security in a shared cluster follows a defense-in-depth model with three categories that mirror how any client-server data system is secured. Encryption of traffic between clients and brokers and between brokers, with TLS. Authentication of every connection, with SASL mechanisms (GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, OAUTHBEARER) or mutual TLS. Authorization of every operation, with ACLs evaluated by the broker’s authorizer.
For distinguishing tenants, each tenant authenticates as its own principal. SCRAM gives each tenant a username and password stored in the cluster metadata log. Mutual TLS gives each tenant a client certificate whose distinguished name maps to a principal via ssl.principal.mapping.rules. OAUTHBEARER delegates identity to an OAuth 2.0 identity provider, which suits organizations that already run one.
Authorization in a multi-tenant cluster leans on prefixed ACLs, covered in the next section, because per-topic ACL grants do not scale to hundreds of tenant topics. The full ACL model has its own page: Kafka ACL, role-level grouping in RBAC roles, and the broader security stack in Kafka authentication and our published Kafka security architecture guide.
The failure mode we see most in shared-cluster authorization is permission creep: ACLs accumulate over time through broad grants and principals that never get revoked, and nobody can say who still needs what. PayPal’s answer is the strictest version of the pattern: no unauthenticated client connections at all, with SASL-based ACLs that require every client to authenticate and declare producer or consumer intent before connecting.
Namespace and topic management
Kafka has no built-in namespace object. Namespaces are a convention built from hierarchical topic names plus prefixed ACLs that enforce them. The Apache Kafka multi-tenancy documentation recommends a naming structure per team or per project: <organization>.<team>.<dataset>.<event-name>, for example acme.infosec.telemetry.logins.
Three enforcement mechanisms exist, and they stack:
- Prefixed ACLs (KIP-290) restrict a tenant to topics under its own prefix. Granting a tenant write access to everything under its namespace is one command:
bin/kafka-acls.sh --bootstrap-server localhost:9092 \
--add --allow-principal User:Alice \
--producer \
--resource-pattern-type prefixed --topic acme.infosec.-
A custom
CreateTopicPolicy(KIP-108, configured withcreate.topic.policy.class.name) rejects topic creations that break the naming pattern, which catches cases a prefix cannot express. -
Denying topic creation to normal users entirely and provisioning topics through an external automated process, commonly a GitOps workflow of the kind covered in deployment automation.
Alongside these, auto.create.topics.enable=false in the broker configuration stops topics appearing as a side effect of a client typo, though the Apache documentation notes it should not be the only control. Topic renames do not exist in Kafka, a rename is create, migrate, delete, which is one more reason to get the naming convention right before tenants onboard.
Tenant counts get much higher than most architecture discussions assume. One global retailer we work with runs four to five thousand tenants across its Kafka clusters, standardised on Avro. At that density, naming conventions and topic lifecycle rules stop being hygiene and become the only way the platform team can reason about the topology at all.
Monitoring and chargeback
Per-tenant visibility comes from the same JMX metrics that drive cluster monitoring, filtered by principal or client id. Kafka exposes quota metrics per client group, so throttling shows up in monitoring before tenants complain. Topic-partition size is trackable per topic with the JMX metric: kafka.log.Log.Size.<TOPIC-NAME>. With a tenant-prefixed naming convention, summing those sizes per prefix gives storage per tenant, which is the basis for chargeback.
The commonly documented open-source stack for this is JMX metrics exported to Prometheus and rendered in Grafana, with Burrow for consumer lag trend analysis. Alerting on a tenant approaching its quota or storage envelope turns chargeback data into an early warning rather than a billing argument. What to watch per tenant follows the same signal set as our Kafka data observability best practices.
Managed services also need monitoring. Kafka health is a shared responsibility, and the provider’s dashboard does not know what your consumer lag means for your business. For chargeback at real scale, JPMorgan is the reference pattern we point to: faced with more than 100 clusters, it built a Health Index, a metrics-based availability score fed into Prometheus, with a centralised control plane and orchestrated patching on top.
Cluster topology decisions
Logical multi-tenancy means one shared cluster with quotas, ACLs and naming keeping tenants apart. Physical multi-tenancy means a cluster per tenant or per environment. The trade is operational: a shared cluster amortizes brokers, monitoring and upgrades across every tenant, while separate clusters buy hard isolation at the cost of running many of everything.
A multi-tenancy view scoping one cluster per tenant, with per-tenant quota and usage.
Most organizations start logical and split physically along one of three lines: scale (the roughly 200 broker guidance above), blast radius (a tenant that repeatedly hurts others), or regulation (data residency that mandates separation). At very large scale, some operators put a proxy layer in front of many physical clusters so that tenants see one logical cluster. Tencent’s published architecture supported up to 60 physical clusters per logical cluster with 10 brokers per physical cluster behind such a layer.
Tiered storage (KIP-405, production-ready since Kafka 3.9) shifts older log segments to object storage, which changes the topology math for tenants with long retention requirements, since storage stops being the reason to give a heavy tenant its own hardware.
When teams do split clusters, I want it to be for the right axis. Splitting by governance domain looks tidy on an org chart, but that is an inefficient way to do it because governance domains change all the time. Netflix’s split is the kind that earns its keep: its original single-cluster design saw consumer fan-out degrade producer ingest, so it moved to a two-tier topology with fronting clusters for producers and consumer clusters for consumers, a split that follows the workload rather than the org chart.