Kafka Connect is a framework for reliably streaming data between Apache Kafka and other systems. It runs connectors as managed, fault-tolerant tasks across a group of worker processes, so data pipelines survive individual worker failures without custom code. This page covers the internals. The surrounding pieces live on the Kafka Connect topic page and in the complete Kafka guide.
The design-validation question behind this search is usually “do serious teams actually run this on shared infrastructure”. In our write-up of JPMorgan Chase’s Kafka platform, one of the largest disclosed deployments in financial services, the credential problem was the sticking point: connectors on shared Connect infrastructure needed privileged database credentials. Their answer was to deploy Connect instances inside each customer team’s own Kubernetes namespace, so the framework stays standard while the blast radius of a credential stays contained. The framework is proven at that scale. The design work is in where you draw the isolation boundaries.
Internal architecture
A connector is a definition, and tasks are its unit of parallelism. When a connector is submitted, the framework splits its work into tasks up to the configured maximum, and the tasks are distributed across the workers in the Connect group.
One connector split into tasks across workers, and the internal config, offset and status topics.
Offset management. Connect tracks progress in dedicated Kafka topics rather than local state. A distributed Connect group stores connector configurations, source offsets and connector status in three internal topics, named by the worker configs: config.storage.topic, offset.storage.topic and status.storage.topic (worker configuration reference). Source connectors commit their own source-position offsets to the offset topic. Sink connectors use ordinary Kafka consumer group offsets.
Rebalancing. When workers join or leave, connectors and tasks are reassigned. The current protocol rebalances incrementally and cooperatively rather than stopping the world: only the tasks that must move are revoked, and the worker config scheduled.rebalance.max.delay.ms (default 300000, 5 minutes) holds a departed worker’s tasks unassigned for a grace period so a bounced worker can reclaim them without a full reshuffle. The protocol mode is set by connect.protocol (default sessioned).
Production scaling and operations
Distributed mode is the production deployment. Standalone mode runs a single worker with file-based offsets and no failover, which suits development and one-off jobs. A distributed group of two or more workers shares state through the internal topics above and reassigns tasks when a worker dies.
Worker sizing is JVM sizing. Each worker runs its assigned tasks in one JVM, so memory and CPU are allocated per worker and tuned like any other JVM service. Connectors that buffer large batches need heap headroom.
Monitoring runs through JMX. Workers expose connector and task states, throughput and error rates as JMX metrics, and a standard production setup exports them to Prometheus. The signals worth paging on are FAILED tasks, and task counts below what the connector was configured to run.
My advice on sizing Connect is the same as for Kafka clusters generally: only scale when you truly need to, which in practice means for headroom or for more throughput. For a sense of what a mature deployment looks like, Shopify runs roughly 150 Debezium connectors across 12 Kubernetes pods. That ratio is worth sitting with. The operational unit is the worker, not the connector, and a dozen well-sized workers can carry a very large integration surface if the monitoring on FAILED tasks is trustworthy.
Integration and error handling
Error handling is configured per sink connector. By default any bad record fails the task: errors.tolerance=none. Setting errors.tolerance=all skips problem records, and a dead letter queue captures them for inspection when a DLQ topic is named: errors.deadletterqueue.topic.name. Error context can be attached as record headers with errors.deadletterqueue.context.headers.enable=true (sink connector configuration reference). Kafka Connect has had native DLQ support since version 2.0, and it is the one integration path that needs no custom code to get a DLQ. The patterns and pitfalls are covered in our guide to dead letter queues in Kafka.
Converters handle serialization. key.converter and value.converter translate between Connect’s internal representation and bytes on the topic, which is how one connector can work with JSON, Avro or Protobuf without changing its code. A schema registry backs the Avro and Protobuf converters.
Delivery guarantees differ by direction. Sink connectors are at-least-once by default, and idempotent write strategies in the sink make retries safe. Source connectors can run exactly-once where the cluster enables it, via the worker config exactly.once.source.support, which writes source records and their offsets in transactions and fences out zombie task generations.
The strongest validation I can offer for the DLQ and delivery guarantee machinery above is that it is what teams standardise on after trying the alternatives. Shopify rebuilt its change data capture system on Kafka Connect and Debezium in 2021, replacing an in-house pipeline, precisely because progress tracking, failover and error handling come with the framework instead of being re-implemented per integration. When you evaluate Connect, weigh it against the cost of owning those three problems yourself forever.
Custom development
A custom connector implements the Connect plugin API: a Connector class that defines and splits the work, and a Task class that moves the records. The same plugin mechanism covers custom converters and single message transforms.
Plugins load in isolation. Every path entry in the worker’s plugin.path gets its own classloader, so a connector that ships an old version of a common library does not break a neighbouring connector that ships a new one. Existing open ecosystems already cover most systems: JDBC, S3, Iceberg and the Debezium CDC family are the common examples, so custom development is usually reserved for in-house systems.
Custom development is rarer than teams expect, and it is usually a sign of a genuinely unusual system rather than a gap in the ecosystem. Airbnb is the instructive case: they built a custom balanced Kafka reader that maps partitions to Spark tasks independently, so Spark parallelism could grow without changing partition count. That is a real constraint the plugin ecosystem does not solve, and it justified the build. If your reason for a custom connector is that the existing one’s config looks awkward, the classloader isolation above means you can run the stock connector safely and spend the engineering elsewhere.
FAQ
What is the purpose of Kafka Connect?
Kafka Connect exists so data pipelines between Kafka and external systems do not need custom code. It runs connectors as managed, fault-tolerant tasks across worker processes, tracks their progress in Kafka itself, and reassigns work when a worker fails.
Is Kafka an ETL tool?
Not by itself. Kafka Connect covers the extract and load halves by streaming data in and out of Kafka, and single message transforms handle light per-record changes. Heavier transformation belongs in a stream processor reading from the topics.
Does Kafka Connect guarantee exactly-once delivery?
It depends on direction. Sink connectors are at-least-once by default, made safe by idempotent writes. Source connectors can run exactly-once where the cluster enables exactly.once.source.support, which writes records and offsets in transactions.