Confluent’s Kafka Docker images (cp-kafka and the surrounding cp-* component images) package Apache Kafka and its ecosystem for containerised use, most commonly a docker-compose stack for local development that mirrors a production cluster’s configuration. Apache Kafka has also shipped its own official image, apache/kafka, since Kafka 3.7.
Almost every team running Kafka in production runs at least three clusters: development, staging, and production. The compose stack on your laptop is in practice a fourth environment, and it is only worth keeping while it mirrors the other three. A local environment that shares nothing with production configuration cannot tell you much about production configuration.
A ready-to-copy docker-compose.yml
A useful local stack runs more than one broker. The compose files engineers actually keep around run a KRaft-mode cluster together with a schema registry, Kafka Connect and a UI, so the local environment exercises the same components as production. Factor House’s own factorhouse-local stack runs this as a three-broker, ZooKeeper-less cluster brought up with a single docker compose pull and docker compose up. If plain Apache Kafka in a container is the starting point, Kafka Docker covers the image-agnostic ground, and the rest of the operational surface is mapped in the complete Kafka guide.
A minimal single-node KRaft broker on the official image, with the env vars the apache/kafka image documents:
services:
broker:
image: apache/kafka:latest
ports:
- "9092:9092"
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093KRaft needs no ZooKeeper container. The controller runs in-process, declared through KAFKA_PROCESS_ROLES and KAFKA_CONTROLLER_QUORUM_VOTERS. KRaft became available in Kafka 3.3 and reached full feature parity in 3.9.
Advertised listeners, the classic failure
Two configs do different jobs, and mixing them up is the single most common way a local Kafka breaks. Per the Apache Kafka configuration reference: listeners is the list of URIs the broker binds and listens on, while advertised.listeners is the set of addresses the broker publishes for clients to connect to, used exactly when the bind address is not the address clients should dial.
Inside a compose network, containers reach the broker by service name. The host machine reaches it through a published port. One listener cannot serve both names, so the working pattern is two listeners on two ports:
KAFKA_LISTENERS: INTERNAL://:9092,EXTERNAL://:29092,CONTROLLER://:9093
KAFKA_ADVERTISED_LISTENERS: INTERNAL://broker:9092,EXTERNAL://localhost:29092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNALThe client bootstraps against one address, then reconnects to whatever the broker advertises. When a container-internal hostname gets advertised on the external listener, host clients receive an address they cannot resolve and the connection fails after bootstrap. The same misconfiguration in reverse is a security problem in real deployments: an internal hostname or an unencrypted PLAINTEXT listener exposed to external clients.
I have spent years running Kafka at companies of every size, and the failures I have seen rarely announce themselves. A subtle misconfiguration. A metric that looks fine until it does not. Consumer lag that spikes at 2am with no obvious cause. An advertised-listeners mistake behaves exactly like that class of failure. The stack comes up, bootstrap succeeds, every container is green, and the problem only appears when the first client outside the compose network follows the advertised address and cannot resolve it.
Log levels and configuration overrides
Container images map broker properties to environment variables so the same image serves any configuration. The rule on the official apache/kafka image: the variable starts with KAFKA_, and dots in the property name become underscores. num.partitions becomes KAFKA_NUM_PARTITIONS, log.retention.hours becomes KAFKA_LOG_RETENTION_HOURS. Confluent’s component images follow the same convention with per-component prefixes, for example SCHEMA_REGISTRY_ for the schema registry and CONNECT_ for Kafka Connect workers, documented in Confluent’s image reference.
Log verbosity works the same way. Raising the root log level to DEBUG through the image’s log4j environment variables is the quickest way to see why a client cannot connect, and it belongs in the local compose file, never in a production manifest.
A caution from our published security architecture guide applies here: the gap between “audit logging is enabled” and “audit logging is actionable” is wider than most teams expect. Debug logging in containers has the same property. DEBUG on the root logger answers a connection question in minutes, and it also produces volume nobody reads. Raise it to answer a specific question and set it back afterwards.
Customising Kafka Connect images
A stock Connect image ships with no third-party connectors. Two ways to add one locally:
Extend the image. A short Dockerfile starts from the Connect base image and installs the connector at build time. On Confluent’s cp-kafka-connect image the installer is Confluent’s own confluent-hub tool. Published examples pin exact versions, for instance a lab running confluentinc/cp-kafka-connect:7.8.0, because a floating tag changes under you between pulls.
Mount the plugin. The connector JARs go in a volume mounted at the worker’s plugin.path. Published local stacks use this for JDBC, S3 and Iceberg connectors, and purpose-built CDC images exist: our complete guide to Kafka CDC runs debezium/connect:2.5 with the PostgreSQL connector already in place.
Local is where a connector’s configuration gets proven before it reaches the production deployment pipeline. The compose stack exists so a connector change can be tested in minutes rather than through a deployment.
Thomas Crowley, one of our engineers, gets this question in our community Slack and deliberately does not prescribe one pattern. Some teams build a custom image, others use a sidecar pattern, an init container with an emptyDir volume, or a plain volume mount. In his experience the choice is usually dictated by organisational requirements rather than by the tooling, for instance where a derived image is itself an organisational requirement. That is also why our own docs decline to name a single recommended approach.
FAQ
What is Confluent Kafka used for?
In the context this page covers, Confluent’s Kafka images (cp-kafka and the cp-* component images) are used to run Apache Kafka and its ecosystem components in containers, most commonly as a docker-compose stack for local development that mirrors production configuration. The official apache/kafka image has covered the broker-only case since Kafka 3.7.
Is Kafka an ETL tool?
Kafka itself is the streaming platform. The data-movement work this page touches is done by Kafka Connect, whose workers load source and sink connectors, for example JDBC, S3, Iceberg and Debezium CDC connectors, added to a stock Connect image by extending it or mounting the JARs at plugin.path.