Kafka Connect is Apache Kafka’s framework for moving data between Kafka and external systems without writing custom integration code. Source connectors pull data from systems such as databases into Kafka topics. Sink connectors push data from topics into systems such as object storage or search indexes. Where Kafka itself fits in a streaming platform is covered in the complete Kafka guide.
Connect ships with Apache Kafka itself. Connector plugins for specific systems are installed separately and loaded from the worker’s plugin path.
Most Kafka problems I get called into are not Kafka bugs. They are misconfigurations, missing observability, or reasonable decisions made without full context, and Connect concentrates all three. It is the part of the platform where a JSON document you submitted at 4pm is moving production data by 4:01, so the gap between “the connector says RUNNING” and “the data is actually flowing correctly” deserves more attention than it usually gets.
What a Connect deployment is made of
A Kafka Connect deployment has three moving parts: the worker process, the connector plugins installed on it, and the connector configurations submitted to it.
A Connect cluster view showing connectors, their tasks and their status.
Workers are the JVM processes that run connectors. The reference documentation for worker and connector configuration is the Kafka Connect section of the Apache Kafka documentation.
Connector plugins are the system-specific implementations. Widely used examples include JDBC connectors for relational databases, connectors for object storage such as S3, the MongoDB connector, and the Debezium family of change data capture source connectors. Plugins are installed by placing their jars on the path named in the worker’s plugin configuration: plugin.path. Each plugin loads in an isolated classloader so conflicting dependency versions do not collide.
Connector configurations are JSON or properties documents naming the connector class, the topics involved and the connection details for the external system. Connection strings and credentials belong in externalised secrets rather than plain configuration files.
A note on converters. Converters translate between Connect’s internal data format and the bytes written to Kafka. The worker default applies unless a connector overrides it, set by: key.converter and value.converter.
Common operational tasks
Day-to-day Connect work happens through three surfaces: the deployment mode, the metrics the workers expose, and the REST API.
Running distributed. Standalone mode runs one worker with local state and suits development. Distributed mode runs a group of workers that share connector and task state through Kafka itself, so a failed worker’s tasks move to the survivors. High availability comes from running distributed mode with more than one worker.
Monitoring. Workers expose JMX metrics for connector and task state, throughput and error counts, and these are commonly scraped into Prometheus and graphed, the same pipeline described in our guide to Kafka alerting with Prometheus and Alertmanager. The states worth alerting on are FAILED tasks and connectors whose task count has dropped. Consumer-side symptoms of a struggling sink usually show first as lag, covered in how to monitor Kafka consumer lag.
Driving the REST API. Each worker exposes a REST API for managing connectors. The default listener is: http://:8083, set by the worker config listeners (Connect worker configuration reference). The API creates, updates, pauses, resumes, restarts and deletes connectors, and reports connector and task status including stacktraces for failures. Kafka management tools in the Kpow category expose these same operations through a UI on top of the API.
Scale changes what “managing Connect” means. Derek, our co-founder, has fielded enterprise evaluations built around exactly this: a platform team running many Connect clusters, potentially thousands of connectors, asking how to stop one team deleting another team’s connectors and how to scope access by LDAP group. At that size the REST API stops being something a person drives by hand and becomes something you govern, with ACLs on who can create, restart and delete, and an audit trail of who did. Plan for that from the first cluster, because retrofitting access control onto a hundred running connectors is much harder than starting with it.
Where the sub-pages go deeper
This page is the topic parent. The architecture and internals live on what is Kafka Connect. A worked production configuration lives on Kafka Connect MongoDB example. The CDC tooling decision lives on Debezium vs Kafka Connect, and the managed-versus-self-hosted cost question lives on Kafka Connect pricing.
FAQ
What is the purpose of Kafka Connect?
Kafka Connect moves data between Kafka and external systems without custom integration code. Source connectors pull data from systems such as databases into topics, and sink connectors push data from topics into systems such as object storage or search indexes.
Do Netflix use Kafka?
Yes, Netflix runs Kafka as a core part of its data platform. Our analysis of Netflix’s Kafka architecture covers how it is deployed and operated at that scale.
Is Kafka Connect highly available?
In distributed mode, yes. Workers share connector and task state through Kafka itself, so a failed worker’s tasks move to the surviving workers. Standalone mode runs a single worker with local state and suits development rather than production.