A topic is the logical name; a partition is the physical thing. Producers write to and consumers subscribe to the topic, which itself stores nothing: its data lives in partitions, the actual append-only logs on broker disk. A topic with twelve partitions is twelve logs, each with its own offset sequence. Everything else on this page, replication, ordering, parallelism, scaling, follows from which of the two levels a mechanism attaches to. The hub holds the full cluster picture.
Core differences and architecture
Replication follows the physical unit. Every partition has a leader and zero or more followers spread across brokers, the leader handles all produce and fetch requests for that partition, and the followers fetch from it to stay in sync. Topics are never replicated as a whole. The corollary for fault tolerance is partition-level too: more brokers means a broker failure impacts fewer partitions, because each broker hosts a smaller share of the total.
The physicality has a cost that shapes cluster operations. Because partition data is coupled to specific broker disks, moving a partition means physically copying its data between brokers, potentially gigabytes per partition over the network. That is why scaling a cluster out is slow, and why partition placement decisions outlive the meeting where they were made.
The most direct consequence of the distinction: a single-partition topic caps you at one consumer instance for that topic’s parallelism, no matter how many machines you own.
The logical-versus-physical distinction sounds academic until parallelism gets welded to it. Airbnb’s Spark Streaming logging pipeline tied Spark task parallelism directly to Kafka partition count, so scaling processing throughput meant operationally costly topic repartitioning, a physical storage operation, to solve a compute problem. Their eventual fix was a custom balanced reader mapping partitions to Spark tasks independently, so Spark parallelism could grow without touching partition count. The design lesson generalises: any system that inherits its parallelism one-to-one from partition count has coupled itself to the physical layer, and will eventually pay a physical price for a logical change.
Production and scaling impact
Parallelism is the partition’s job. Kafka’s assignment model gives each partition to exactly one consumer in a group at a time, which preserves per-partition ordering and simultaneously caps how many consumers can share the work: partition count is the maximum useful size of a consumer group. That cap is what historically drove over-partitioning, creating 100 partitions in case 100 consumers were ever needed, with every partition costing broker resources whether used or not. Share groups (KIP-932, queues for Kafka) relax exactly this constraint, letting more consumers than partitions share work, six consumers on three partitions, for workloads that do not need ordering.
Ordering is a per-partition guarantee and nothing more. Kafka guarantees record order within a single partition, never across a topic. Routing decides what shares a partition: a record’s key is hashed, hash(key) % num_partitions, so all records with one key land in one partition in order. That is how ordering per customer, per account or per device is built.
The two rules meet in one sharp edge: for keyed topics, you cannot increase the partition count later without breaking hash(key) % num_partitions, because the same key suddenly routes to a different partition and its history splits across two. Partition count on a keyed topic is a decision made once, at creation, with the growth already priced in.
The keyed-topic edge is not theoretical. One of the four incidents in my talk: a team increased the partition count on their topics to scale a service, a completely reasonable throughput decision, made without the one piece of context that mattered, that the topics were keyed and consumers relied on the routing. Messages were silently skipped, workflows got stuck, SLAs broke, and Kafka raised no error and no alert, because from Kafka’s side nothing was wrong. hash(key) % num_partitions had simply started giving different answers, and in-flight keys’ histories split across old and new partitions mid-read. Silence is the defining feature of this failure. Every other scaling mistake in Kafka announces itself somewhere. This one only shows up downstream, as missing work.
Which is why the practice at the big deployments is the whole lesson of this page in one line: over-provision partitions at creation, and do not alter keyed topics. Two working numbers for the creation decision: estimate distinct keys per partition and target at least 20, so key skew cannot concentrate load, and on large clusters keep busy topics at 100 partitions or more, because under that, load distribution across brokers goes uneven. Decide as if you cannot change it, because on a keyed topic, you effectively cannot.
FAQ
Why does Kafka need partitions?
Partitions are how one topic scales past one machine and one reader. Each partition is an independent append-only log that can live on a different broker, so a topic’s traffic spreads across the cluster, and each partition is consumed by exactly one member of a consumer group, so adding partitions is what allows adding consumers. They are also the ordering boundary: Kafka guarantees order within a partition, which is what makes per-key ordering possible at all.
How many partitions should a Kafka topic have?
Decide from three inputs, at creation, as the section above details. Throughput: enough partitions that your consumers keep up. Keys: at least 20 distinct keys per partition so skew cannot concentrate load. Cluster shape: busy topics at 100 partitions or more on large clusters, brokers below 10,000 to 20,000 partitions each. Then over-provision, because on a keyed topic you effectively cannot change the count later.