A Kafka topic is a named, append-only log that producers write to and consumers read from. In a live cluster the topic is also the unit you operate on: everything routine, creating, configuring, inspecting, resetting, happens per topic. This page is the working reference for those operations in the order you reach for them: the CLI commands, the mechanics underneath, the configuration that matters, and the troubleshooting moves. The topic vs partition page separates the logical name from the physical log, the topic example page is a full production specification, and the hub holds the rest of the cluster picture.
At production scale a topic is also an operational unit with a cost. DoorDash manages more than 2,500 topics, and by 2021 Salesforce’s fleet carried over 50,000. Topic design at that scale is system design: a real deployment’s topics encode the domain, a canonical state topic, a real-time changes topic, an event topic per business action, an audit log, an alerts feed, and each one carries its own partition count, replication, retention and access control.
One more scale number reframes the page. As of 2019, JPMorgan Chase’s deployment held 13,000 topics, of which 1,300 were production. Read that ratio again: for every production topic there were nine that were tests, experiments, migrations that finished, or projects that did not. Topics are cheap to create and expensive to reason about, and nobody deletes them, so an estate’s topic list drifts toward archaeology. My advice from watching this happen is to treat topic creation as cheap and topic EXISTENCE as a liability, with an owner, a purpose and a review date, because the question “what is this topic and does anything still read it” gets asked eventually, usually during an incident.
Command line operations
kafka-topics.sh is the tool, and four verbs cover the routine work. --create makes a topic, with --partitions and --replication-factor setting the layout at birth. --list names every topic the cluster holds. --describe shows one topic’s partition layout, leader placement and in-sync replicas. --delete removes it, subject to delete.topic.enable on the brokers. Each command takes --bootstrap-server, and per-topic configuration overrides ride on --config key=value.
The same operations exist above the CLI. Topic management surfaces, whether an API or a UI, cover the identical verbs: creating and deleting topics, setting replication factor and partition count at creation, reconfiguring, inspecting partition layout and replica placement, and reading messages with offset, key and header detail. The operational point is not which surface you use but that create-time flags are the ones that matter: partition count and replication factor are structural, and everything else is adjustable later.
My honest position on the CLI, having lived in it: it is complete, and completeness is not the constraint. At scale, the difference between a surface that shows you under-replicated partitions clearly and one that forces an operator back to kafka-topics.sh mid-incident is the difference between a five-minute fix and a thirty-minute incident. The CLI is the right tool for scripting and the wrong shape for diagnosis, because every question costs a command and the answers do not sit next to each other.
Volume compounds the argument. DoorDash provisions roughly 100 new topics per week, and at those rates topic operations are an automated pipeline with humans reviewing exceptions, which is why the create-time flags above belong in code review rather than in someone’s shell history. Also trust the shipped tooling’s edges: the ecosystem’s own partition-reassignment tools have had mixed experiences historically, our co-founder Derek Troy-West among those observing it, so rehearse reassignment on a non-production cluster before you need it on the production one.
Inside a topic: partitions, replication, retention
A topic’s data lives in its partitions, and each partition is an append-only log stored as segments on broker disk. Replication happens at the partition level: every partition has a leader and one or more replicas, the leader handles all produce and fetch requests, and the replicas fetch from the leader to stay in sync. A topic as a whole is never replicated; its partitions are, each with its own leader placement.
Retention is the configuration that decides what a topic costs. Kafka’s default log retention is 7 days, and on high-throughput topics time-based retention alone is a disk-exhaustion risk: retention.bytes must be set alongside retention.ms, capping the on-disk size even where that means deleting data earlier than intended. Compaction is the other cleanup policy, and it serves a different job: for topics with cleanup.policy=compact, Kafka retains only the most recent record per key, with a background log cleaner thread comparing dirty segments against clean ones. Delete-based retention suits event streams where history expires. Compaction suits state, where the latest value per key is the whole point.
Durability is a three-setting contract. The standard production configuration is replication factor 3 with min.insync.replicas=2 and producers writing acks=all. Each partition maintains an in-sync replica set, the replicas considered current with the leader, and min.insync.replicas is the floor on how many of them must acknowledge a write before it counts. Two of the three settings live on the topic, one on the producer, and the guarantee only exists when all three agree. One number in the ISR mechanics is worth knowing cold: the default replica.lag.time.max.ms is 30 seconds, the window a follower gets before it is dropped from the in-sync set. That is the clock behind every under-replicated-partition alert. For what disciplined settings look like as policy, Netflix’s distributed tracing system Inca runs its trace topics on a dedicated cluster at replication factor 3 with min ISR 2: the standard contract, applied deliberately, per workload.
The retention story from my talk is the slow one, and slow is what makes it dangerous. A high-throughput topic without retention.bytes does not fail on deploy day. The disk fills over weeks, gradually, nothing alarming on any single day, until the broker halts operations, and recovery is the hardest job in Kafka operations because every recovery step needs disk you no longer have. So my rule is monitor disk usage per topic, not just per broker, and set retention.bytes alongside retention.ms on every high-throughput topic. Per topic matters because a broker-level disk graph tells you the patient is dying without telling you which topic is the tumour.
Two settings in this section fail silently when they disagree with each other, and both incidents are documented. Historically, replica.fetch.max.bytes smaller than message.max.bytes caused silent replication stalls: the leader accepted a message the followers could not fetch, the ISR shrank without an error naming the cause, and under unclean leader election that meant data loss. And New Relic hit slow topic consumption on startup because segment.ms was not aligned with retention.ms, leaving large log segments full of mostly expired data that had to be scanned. The pattern in both: topic settings are a system, not a list, and the failure lives in the interaction, which no single setting’s documentation warns you about.
Partition counts also move for operational reasons, not just design ones. Shopify’s Game Days load exercises revealed that analytics topics needed higher partition counts to keep data fresh through Black Friday traffic spikes, and partition increases became a standard pre-BFCM checklist item. The lesson is that a topic’s partition count is a capacity setting you should rehearse changing, on unkeyed topics, before the traffic arrives. Keyed topics do not give you that freedom, for reasons the topic vs partition page covers.
Production concerns
Topic design decides blast radius. A topic-per-event-type layout increases operational surface area, more topics to configure and watch, and in exchange it limits the damage when a single producer or consumer misbehaves: the failure stays inside one event type instead of contaminating a shared stream. Spotify’s deployment is the published example of choosing that trade deliberately.
The failure modes are rarely exotic, and topics are where the configurations that cause them live. The gap that lets them persist is visibility: in most organisations, non-specialists cannot check whether the correct data is even flowing into a topic, so a wrong or stale feed looks identical to a healthy one until something downstream breaks.
There is a threshold where informal topic management stops working. Once an organisation passes roughly 100 topics or 10 teams producing to Kafka, investment in end-to-end auditing is appropriate, because at that point nobody holds the full topic inventory in their head and provenance questions, what writes to this, what reads it, is it still used, need tooling rather than tribal knowledge.
The environment question is part of topic design too, because almost every team running Kafka in production runs at least three clusters: development, staging and production. The same topic exists three times, and the production concerns in this section apply to the copy nobody watches. A staging topic with production-sized throughput and default retention is the disk incident you were not monitoring for. Whatever topic standards you adopt, apply them across all three clusters, or the standard is a suggestion.
Limits and standards
Topic count reaches the brokers as partition count, and partitions cost memory and file handles on every broker that hosts them. The commonly cited practical limit for older Kafka versions before 2.6 is roughly 4,000 partitions per broker, and conservative deployments still plan around that figure. The historical ceiling was starker: Uber’s ZooKeeper-era 200,000-partition cluster cap bounded them to roughly 200 topics per cluster at their scale, which is the arithmetic that pushed partition-count discipline into topic design. The KRaft page covers how the modern controller moved those limits.
My broker-count heuristic surprises people: do not exceed roughly 36 brokers in one cluster. A 36-broker cluster handles a phenomenal number of topics, partitions and message throughput, and past that size the operational overhead of the cluster itself, rebalancing, upgrades, failure correlation, grows faster than the capacity does. If 36 brokers with disciplined partition counts cannot carry your load, the answer is usually a second cluster along a domain boundary, not broker 37.
Two topic-level settings spring their traps late, which is why I put them on the creation checklist rather than the tuning list. max.message.bytes set generously “to be safe” caps the topic’s throughput later, when the topic is popular and there is no real fix short of migrating it. And for large clusters, anything under 100 partitions on a busy topic starts to produce uneven load distribution across brokers, so lowball partition counts do not just cap parallelism, they skew the cluster. Both are five-second decisions at creation and migration projects afterwards.
Two standards keep multi-tenant clusters navigable. Naming conventions, because a 50,000-topic fleet without them is unsearchable, and schema enforcement at the topic level, because compatibility checks at publish time are what let hundreds of teams share a cluster without breaking each other’s consumers. The topic example page shows both as concrete artefacts.
Troubleshooting and inspection
Topic troubleshooting is mostly consumer-group troubleshooting seen from the other side. The two most common tasks are offset resets and lag investigation, and lag is read per partition against this topic: current offset versus log-end offset, for every group consuming it. Group-level tooling that aggregates across all of a topic’s partitions gives the view no single client can, and it is particularly useful for spotting partition lag skew, where one partition trails the rest. Group behaviour itself, rebalances and the coordinator, lives on the consumer page.
Skewed lag with healthy consumers usually means skewed leadership. When partition leaders have bunched onto one broker, that broker carries the topic’s whole load. The fix is preferred leader election, kafka-leader-election.sh --type preferred, or a partition reassignment where the layout itself is wrong.
The honest limit of the standard metric set: most teams track consumer lag, broker CPU and memory, network throughput and under-replicated partitions, and those tell you a problem exists without telling you its cause. Every metric in the list is a symptom that several different diseases share. The diagnostic step that follows is always the same, and it is this section’s real method: localise. Which topic, which partition, which broker. A cluster-wide graph moving is a fact; one partition on one broker diverging from its siblings is a diagnosis.
Leadership skew is the canonical example because it is invisible in aggregates. Pinterest documented it at fleet scale: as their cluster fleet grew, partition leadership distribution became uneven and individual brokers were overloaded while cluster-level metrics looked liveable. Per-broker leader counts are the check, preferred leader election is the cheap fix, and it belongs in your runbook as routine maintenance, not as an incident response.
FAQ
What is a topic in Apache Kafka?
A topic is a named, append-only log that producers write records to and consumer groups read from. It is divided into partitions, which are the physical logs that carry its data, its ordering guarantee and its parallelism. A topic is not a queue: records are retained by the topic’s retention policy, not deleted on delivery, so many independent consumers can read the same topic at their own pace. In production the topic is also the unit of configuration and governance, which is what the rest of this page covers.