Best Kafka tools for Instaclustr managed Kafka Connect
ComparisonsThe best Kafka tool for teams using Instaclustr managed Kafka Connect is one that reaches the Connect REST API through the Connect cluster’s own firewall rule with the credential and certificate from its Connection Info page, shows each connector’s tasks and the stack trace of the one that failed rather than per-worker totals, pauses and restarts the right thing around a custom connector sync, decides per person who may create or change a connector whose configuration holds credentials and records who did, and runs as one container beside the cluster, out of the data path. Kpow, Kafbat UI, AKHQ, Lenses, the Instaclustr console with the Connect REST API, and Conduktor each cover part of that. Scored on the six weighted criteria explained below the rankings, Kpow ranks first with 98 out of 110, ahead of Kafbat UI at 71 and AKHQ at 65; Conduktor, listed last, totals 67.
Tools compared
| Rank | Tool | Total (out of 110) | Out of the data path | Connector operations | Production access on request | Audit trail per person | Instaclustr Connect setup | Directory and Kafka sign-in | Cost a year, one cluster (modelled) |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Kpow | 98 | One container, no external database, not a proxy | Task state and traces, pause, capped auto-restart | Temporary policies, staged approvals, masking | Every action and data read, by user | Own Instaclustr guide; permissive SSL or a truststore | SAML, OpenID, LDAP | $7,380 |
| 2 | Kafbat UI | 71 | One stateless container | Task status, restarts, free RBAC | Read-only clusters, no approvals | Topic or console log, no view | No Connect guide; one truststore field | OAuth2, OIDC, LDAP | $8,640 |
| 3 | AKHQ | 65 | One stateless container | Task status, single-task restart | Regex groups, no approvals | Opt-in topic, no reads | Instaclustr's own guide with the truststore | LDAP, OIDC | $8,640 |
| 4 | Instaclustr console and the Connect REST API | 61 | Nothing to deploy | Every REST operation, by curl | One credential, no approvals | Worker logs only | Provider's own control plane | Basic authentication, no directory | $11,520 |
| 5 | Lenses | 60 | HQ on PostgreSQL plus an agent per cluster | Task health, Connect alert rules | Global masking, no approvals | Readable in the product | Generic Connect, no guide | SSO on paid tiers | $6,880 for 15 users; 25 is a quote |
| 6 | Conduktor | 67 | Console on PostgreSQL; Gateway, a proxy, for data-level controls | Failed-task auto-restart with history, alerts | Masking exemptions, owner approval | 70+ event types in the UI | Generic Connect, no guide | LDAP, OIDC | $32,880; $122,880 with Gateway Core and Protect |
The tools, ranked for Instaclustr managed Kafka Connect
Rank 1 Kpow
98 out of 110 Total
Try Kpow in the live demo No signup needed.
- Cost a year
- $4,500 per Kafka cluster with 100 users included, plus about $2,880 in operator time, so $7,380 on one cluster (modelled)
- On Instaclustr Connect
- Documented settings for the managed Connect cluster, beside Kafka and Karapace
- Deployment
- One container or JAR, no external database
- Out of the data path ×3 weight, this criterion counts 3 times toward the total
- 9 out of 10
- Connector operations ×2 weight, this criterion counts 2 times toward the total
- 9 out of 10
- Production access on request ×2 weight, this criterion counts 2 times toward the total
- 9 out of 10
- Audit trail per person ×2 weight, this criterion counts 2 times toward the total
- 9 out of 10
- Instaclustr Connect setup
- 8 out of 10
- Directory and Kafka sign-in
- 9 out of 10
Why these scores for Kpow
- Out of the data path 9 out of 10
- It is one container or JAR whose snapshots, metrics and audit log live in topics on your own cluster, and it reaches Instaclustr’s Connect cluster through the Connect REST API and Kafka as an ordinary client, so nothing sits between your connectors and the brokers.
- Connector operations 9 out of 10
- Connector and task state sit side by side with task stack traces, individual tasks restart from the task table, connectors pause and resume from the row actions, and auto-restart of named or wildcard connectors runs on a one-minute interval with a 10-minute window and a cap of 50 restarts per interval, each restart written to the audit log; it is held below 10 because Conduktor restarts only the failed tasks automatically and keeps a restart history, and the JMX exporter goes deeper on per-task metrics.
- Production access on request 9 out of 10
- Temporary policies grant time-boxed access that an admin or a change system calling the Kpow API can create, staged mutations hold any action for approval, and data policies mask fields in inspection, though masking is per resource rather than per viewer.
- Audit trail per person 9 out of 10
- Every action is recorded with the user from the identity provider and the policy that allowed it, including data inspect queries, with a seven-day view in the product, the record written to an audit topic on your own cluster, and webhooks that send it to a SIEM for long-term retention.
- Instaclustr Connect setup 8 out of 10
- Factor House’s Instaclustr guide gives the exact Connect settings, basic authentication from the Connection Info page and a separate firewall rule for the Connect cluster, and Kpow keeps reconnecting when startup validation is off; it scores 8, level with AKHQ, because the guide sets permissive SSL where AKHQ’s guide loads the cluster’s truststore, although Kpow also takes a Connect truststore.
- Directory and Kafka sign-in 9 out of 10
- People sign in with SAML, OpenID or LDAP, and Kpow connects to Kafka with the same SASL or TLS settings as any client and to each Connect REST API over HTTPS with basic authentication.
On Instaclustr managed Kafka Connect. Kpow’s Instaclustr documentation adds the managed Connect cluster with CONNECT_REST_URL pointing at a Connect node on port 8083, CONNECT_AUTH=BASIC with the username and password from the Connection Info page, and CONNECT_PERMISSIVE_SSL=true, which the guide says Instaclustr often requires. The same page says the Kpow host’s address must be added to the Connect cluster’s firewall rules as well as the Kafka cluster’s. The Kafka Connect configuration adds truststore settings for the REST API, several Connect clusters per Kafka cluster with CONNECT_RESOURCE_IDS, and auto-restart.
Where it falls short. Its Instaclustr guide disables certificate validation for the Connect REST API rather than loading the cluster’s certificate, so a team that wants the server checked sets the truststore variables itself. Its auto-restart does not keep a restart history beyond the audit log, and alert rules on connector state go through Prometheus and Alertmanager rather than being built in. Kpow governs people working through Kpow, so the REST credential Instaclustr issues still opens the API to anyone who holds it. RBAC, masking, staged mutations and the full audit log need Kpow Enterprise; Community Edition is free for 3 clusters and 10 users.
Compare Kpow vs Kafbat UIKpow vs AKHQ
Rank 2 Kafbat UI
71 out of 110 Total
- Cost a year
- $0 licence, about $8,640 in operator time (modelled)
- On Instaclustr Connect
- Connect by URL; one truststore field for every TLS endpoint
- Sign-in
- OAuth2, OIDC and LDAP, free
- Out of the data path ×3 weight, this criterion counts 3 times toward the total
- 9 out of 10
- Connector operations ×2 weight, this criterion counts 2 times toward the total
- 6 out of 10
- Production access on request ×2 weight, this criterion counts 2 times toward the total
- 4 out of 10
- Audit trail per person ×2 weight, this criterion counts 2 times toward the total
- 6 out of 10
- Instaclustr Connect setup
- 5 out of 10
- Directory and Kafka sign-in
- 7 out of 10
Why these scores for Kafbat UI
- Out of the data path 9 out of 10
- It is one stateless container with no database and no proxy, the same pass as Kpow.
- Connector operations 6 out of 10
- It shows connector and task status, creates and edits connectors, and restarts a connector or its failed tasks, across several Connect clusters with free RBAC, but documents no auto-restart.
- Production access on request 4 out of 10
- RBAC grants actions per resource and a cluster can be set read-only, but there is no approval step, no time-boxed grant, and its masking applies the same way to every viewer.
- Audit trail per person 6 out of 10
- Its audit log names the logged-in user and records reads when the level is set to ALL, but it writes to a topic or the console with no view in the product, so reading the trail is something you build.
- Instaclustr Connect setup 5 out of 10
- Instaclustr’s guide for UI for Apache Kafka, the project Kafbat UI continues, does not configure the managed Connect cluster and warns that clusters using both Kafka Connect and Schema Registry over TLS can hit connection issues because the UI has one truststore field, worked around by merging truststores with keytool.
- Directory and Kafka sign-in 7 out of 10
- It supports OAuth2 and OIDC, including Microsoft Entra ID, and LDAP or Active Directory, and its documentation does not list SAML.
On Instaclustr managed Kafka Connect. Kafbat UI lists Connect clusters beside each Kafka cluster in its configuration, each with its own URL and credentials, and its feature list covers Kafka Connect next to topic browsing, consumer groups and schemas. The Kafbat UI review covers its RBAC and release history.
Where it falls short. Nothing restarts a failed connector automatically, there is no way to grant production access for an hour and have it expire or to hold a change for approval, and the single truststore field means the Connect and Karapace certificates have to be merged into one file. Its modelled running cost on one cluster is 6 engineer-hours a month, $8,640 a year at $120 an hour.
Rank 3 AKHQ
65 out of 110 Total
- Cost a year
- $0 licence, about $8,640 in operator time (modelled)
- On Instaclustr Connect
- Instaclustr's own guide, with basic authentication and the cluster's truststore
- Security default
- Disabled until you enable it
- Out of the data path ×3 weight, this criterion counts 3 times toward the total
- 9 out of 10
- Connector operations ×2 weight, this criterion counts 2 times toward the total
- 5 out of 10
- Production access on request ×2 weight, this criterion counts 2 times toward the total
- 3 out of 10
- Audit trail per person ×2 weight, this criterion counts 2 times toward the total
- 4 out of 10
- Instaclustr Connect setup
- 8 out of 10
- Directory and Kafka sign-in
- 6 out of 10
Why these scores for AKHQ
- Out of the data path 9 out of 10
- It is one stateless container with no database and no proxy, the same pass as Kpow.
- Connector operations 5 out of 10
- It lists connector and task status and restarts a connector or a single task, with no failed-tasks-only restart and no auto-restart.
- Production access on request 3 out of 10
- Groups bind actions to resources by regex, but there is no approval step or time-boxed grant, masking is global, and without the JWT signing secret the restriction is in the UI only.
- Audit trail per person 4 out of 10
- Audit events are opt-in to a Kafka topic, reads are not recorded, and there is no view for the trail.
- Instaclustr Connect setup 8 out of 10
- Instaclustr’s November 2023 guide connects AKHQ 0.24.0 to a managed Connect cluster with basic authentication and the truststore.jks file downloaded from the Connect cluster’s Connection Info page, so the server certificate is verified, a clean documented pass.
- Directory and Kafka sign-in 6 out of 10
- It supports LDAP, OIDC and header authentication from a proxy, does not list SAML, and ships with security disabled until you enable it.
On Instaclustr managed Kafka Connect. Instaclustr’s blog published a guide to AKHQ with Instaclustr for Apache Kafka in November 2023, adding the Connect cluster under the Kafka connection with its URL, the Connect username and password, and the truststore, after adding the machine’s address to the Connect cluster’s firewall rules. The console then shows each connector’s tasks and configuration. The AKHQ review covers the rest.
Where it falls short. Security is off until you configure it, the audit trail is an opt-in topic that does not record reads, and a failed connector stays failed until someone restarts it. Its modelled running cost on one cluster is 6 engineer-hours a month, $8,640 a year at $120 an hour.
Compare Kpow vs AKHQAKHQ review
Rank 4 Instaclustr console and the Connect REST API
instaclustr.com
61 out of 110 Total
- Cost a year
- $0 licence and nothing to run; connector work with curl is modelled at 8 hours a month, $11,520 (modelled)
- On Instaclustr Connect
- Provisions the cluster, firewall rules, credentials and custom connector sync
- Deployment
- Nothing to deploy
- Out of the data path ×3 weight, this criterion counts 3 times toward the total
- 10 out of 10
- Connector operations ×2 weight, this criterion counts 2 times toward the total
- 6 out of 10
- Production access on request ×2 weight, this criterion counts 2 times toward the total
- 1 out of 10
- Audit trail per person ×2 weight, this criterion counts 2 times toward the total
- 2 out of 10
- Instaclustr Connect setup
- 10 out of 10
- Directory and Kafka sign-in
- 3 out of 10
Why these scores for Instaclustr console and the Connect REST API
- Out of the data path 10 out of 10
- There is nothing to deploy beyond the Connect cluster Instaclustr runs, which is the only option here that scores 10.
- Connector operations 6 out of 10
- The REST API has every operation, including restarting only the failed tasks, and the console adds per-worker metrics, but nothing watches connector state, nothing restarts automatically, and each operation is a curl call.
- Production access on request 1 out of 10
- Anyone holding the REST credential and allowed through the firewall can create, change or delete any connector, with no approval step, no expiring grant and no masking.
- Audit trail per person 2 out of 10
- The workers’ own logs, which Instaclustr can ship to a topic on the Kafka cluster, are the only record, and nothing ties a call to a person in the directory.
- Instaclustr Connect setup 10 out of 10
- It is the provider’s own control plane, where the Connect cluster’s firewall rules, credentials, certificates and the Connected Clusters view that sets up firewalls and truststores to other clusters all live, so it is the clear best on this criterion.
- Directory and Kafka sign-in 3 out of 10
- The REST API takes the basic authentication credential from the Connection Info page over HTTPS, with no directory sign-in for the people calling it.
What it covers. Instaclustr’s documentation names the Kafka Connect REST API as the primary interface for its managed Connect, reached with a TLS-capable client such as curl or Postman on any node, all of which have equal functionality. Its console provisions the cluster with an optional Private Network add-on, keeps the firewall rules, issues the credentials and server certificates on the Connection Info page, syncs custom connectors from the team’s own bucket, can ship worker logs to the Kafka cluster, and shows per-worker metrics. The Kafka Connect user guide lists the REST endpoints every tool on this page calls.
Where it falls short. It is a toolkit, not a tool: status has to be polled, failures noticed and restarts scripted, Instaclustr’s own examples pass curl’s -k flag, which skips certificate verification, and one credential grants all of it to whoever holds it. Its modelled running cost is 8 engineer-hours a month, $11,520 a year at $120 an hour.
Compare How to diagnose and fix a failed Kafka Connect connector
Rank 5 Lenses
lenses.io
60 out of 110 Total
- Cost a year
- Team is $4,000 for up to 15 users on one cluster, $6,880 with operator time; 25 engineers needs a custom quote (modelled)
- On Instaclustr Connect
- Generic Connect support; no guide from either vendor
- Deployment
- HQ on PostgreSQL plus an agent and database per cluster
- Out of the data path ×3 weight, this criterion counts 3 times toward the total
- 4 out of 10
- Connector operations ×2 weight, this criterion counts 2 times toward the total
- 7 out of 10
- Production access on request ×2 weight, this criterion counts 2 times toward the total
- 4 out of 10
- Audit trail per person ×2 weight, this criterion counts 2 times toward the total
- 7 out of 10
- Instaclustr Connect setup
- 5 out of 10
- Directory and Kafka sign-in
- 7 out of 10
Why these scores for Lenses
- Out of the data path 4 out of 10
- It runs a central HQ on PostgreSQL plus an agent and an agent database beside every cluster, and HQ has no high-availability option.
- Connector operations 7 out of 10
- It shows task health, metrics and stack traces, restarts a connector or a single task, and has Connect-specific alert rules built in, but documents no auto-restart.
- Production access on request 4 out of 10
- Its masking is the strictest view-time model, global with no escape even for admins, but no approval step or time-boxed grant is described.
- Audit trail per person 7 out of 10
- Audit logs can be read in the product, with no need to build a consumer first.
- Instaclustr Connect setup 5 out of 10
- It manages Kafka Connect clusters in general through its agent, which would need its own firewall rule on the Connect cluster, and neither Lenses nor Instaclustr documents the pairing, so the setup is the reader’s to verify.
- Directory and Kafka sign-in 7 out of 10
- SSO spans Okta, Keycloak, OneLogin, Google and Entra ID, with basic authentication only on Community.
On Instaclustr managed Kafka Connect. Lenses manages several Connect clusters through its agent, with connector and task health, metrics and built-in alert rules for Connect, and reaches Instaclustr as it reaches any Kafka and Connect cluster, from an agent allowed through the firewall rules. The Lenses review covers its tiers and deployment.
Where it falls short. A central HQ on PostgreSQL plus an agent and an agent database for every cluster is a lot to run beside a Connect cluster that Instaclustr already operates, and a failed connector waits for a person to restart it. The Team licence stops at 15 users on one cluster, so a larger team is on a custom quote.
Compare Kpow vs LensesLenses review
Rank 6 Conduktor
conduktor.io
67 out of 110 Total
- Cost a year
- 25 Console seats at $1,200 is $30,000 plus $2,880 operator time, so $32,880; Gateway Core adds $60,000 and Gateway Protect, which carries encryption and masking, a further $30,000 (modelled)
- On Instaclustr Connect
- Generic Connect support; no guide from either vendor
- Deployment
- Console on PostgreSQL 13+; data-level controls through Gateway, a proxy
- Out of the data path ×3 weight, this criterion counts 3 times toward the total
- 3 out of 10
- Connector operations ×2 weight, this criterion counts 2 times toward the total
- 9 out of 10
- Production access on request ×2 weight, this criterion counts 2 times toward the total
- 6 out of 10
- Audit trail per person ×2 weight, this criterion counts 2 times toward the total
- 8 out of 10
- Instaclustr Connect setup
- 5 out of 10
- Directory and Kafka sign-in
- 7 out of 10
Why these scores for Conduktor
- Out of the data path 3 out of 10
- Console needs PostgreSQL 13 or later, and its encryption, data-level masking and Virtual Clusters only work when client traffic goes through Gateway, a proxy in the data path.
- Connector operations 9 out of 10
- Conduktor’s Kafka Connect documentation describes connector and task views, an offsets tab, built-in connector alerts, and an auto-restart that checks every minute and, for self-managed connectors, restarts only the failed tasks with a 10-minute window and a history of each restart, level with Kpow on this criterion.
- Production access on request 6 out of 10
- Masking can exempt users or groups, which beats every other tool here on who sees unmasked data, and cross-team access requests are approved by the owning team, but no expiring grant is described and topic creation that passes policy is a direct API call.
- Audit trail per person 8 out of 10
- Console logs produce, consume and admin requests across more than 70 event types with user, IP and timestamp, browsable in the UI and exported as CloudEvents.
- Instaclustr Connect setup 5 out of 10
- Console manages Kafka Connect clusters in general by URL with authentication, and neither Conduktor nor Instaclustr documents the managed Connect cluster with it, so the setup is the reader’s to verify.
- Directory and Kafka sign-in 7 out of 10
- Its SSO configuration covers LDAP and OIDC, with guides for Okta, Entra ID and Keycloak, and does not describe SAML.
On Instaclustr managed Kafka Connect. Conduktor Console connects to a Connect cluster by its REST URL, and from its Kafka Connect page creates connectors from the installed plugin classes, pauses, resumes, restarts or deletes one connector or several at once, shows committed offsets, sets alerts and enables auto-restart. The Conduktor review covers the rest.
Where it falls short. Console needs PostgreSQL. Conduktor’s data-level controls, such as encryption, masking of the data itself and virtual clusters for multi-tenancy, run in Gateway, a Kafka proxy that client applications connect through, and Connect workers are Kafka clients too, so routing Instaclustr’s workers through Gateway would put it in front of every connector’s reads and writes. On AWS Marketplace, Conduktor Enterprise lists Console at $1,200 a seat for the first 100 seats, Gateway Core, which carries virtual clusters, at $60,000 a year, and Gateway Protect, which carries encryption and masking, at $30,000 a year.
Compare Conduktor review
What teams using Instaclustr managed Kafka Connect need
This page is about NetApp Instaclustr’s managed Kafka Connect: a Connect cluster that Instaclustr provisions and runs, attached to an Instaclustr Kafka cluster or to another Kafka cluster, whose connectors are managed through the standard Kafka Connect REST API. Instaclustr’s documentation names that REST API as the primary interface, reached with an HTTP client such as curl or Postman on any node of the Connect cluster. The brokers, Karapace and the rest of the Instaclustr platform are ranked in Best Kafka tools for NetApp Instaclustr managed Kafka, and Kafka Connect in general in Best Kafka tools for Apache Kafka Connect.
A network path of its own. Instaclustr blocks external traffic by default, and the Connect cluster has firewall rules separate from the Kafka cluster’s and Karapace’s, so a tool needs its host’s address added to each of the three it talks to, as Kpow’s Instaclustr guide spells out. The Connect workers have the same problem in the other direction. In Factor House’s public workshop-london-26 material, attaching a Connect cluster to an Instaclustr Kafka cluster added the Connect nodes’ private addresses to the broker firewall automatically but did not touch Karapace’s, and because the workers reached the public Karapace endpoint over the internet, it was their public addresses that had to be allowed; private ones produced timeouts. A connector whose converter cannot reach the schema registry fails in its task, which is where a tool has to show it.
Certificates, or the lack of a check. The Connect REST API is served over HTTPS on port 8083, and Instaclustr’s examples call it with curl’s -k option, which the curl manual defines as --insecure: the connection goes ahead without verifying the server’s certificate. Instaclustr publishes the Connect cluster’s server certificates on the Connection Info page for clients that do verify. A tool that only offers a permissive mode copies the shortcut; one that takes a truststore lets the team check that it is talking to its own Connect cluster.
Worker totals are not task state. The Instaclustr console’s Kafka Connect monitoring is per worker: connectors and tasks assigned to each node, connector and task startup attempts and startup failures, rebalance counts, the epoch and leader each worker believes in, whether each node’s REST API answers, and a latency measured with synthetic records only when the Connect cluster is attached to an Instaclustr Kafka cluster. Apache Kafka’s monitoring reference lists task-startup-failure-total as a worker metric, separate from the per-task status metric and the per-connector connector-failed-task-count. A task that starts cleanly and fails an hour later on a record its sink rejects is not a failed start, so it does not appear in startup failure counts; Instaclustr’s troubleshooting guide sends engineers to the REST API’s status endpoint for that, and a management tool has to read the same endpoint for every connector.
Plugin syncs restart every worker. Custom connectors on Instaclustr are uploaded to the team’s own cloud storage bucket and loaded with the console’s Sync button, and for new Instaclustr accounts the custom connector feature has to be enabled by Instaclustr support first. A sync copies the files to every Connect node and restarts the worker process on each, which Instaclustr warns may disrupt live connectors, suggesting they be paused first. The Kafka Connect user guide describes pause as stopping message processing while the tasks keep their resources, so they resume quickly. While workers restart, Kafka Connect’s incremental cooperative rebalancing, introduced in KIP-415, lets the leader wait up to scheduled.rebalance.max.delay.ms, five minutes by default, before reassigning a departed worker’s tasks. A task can therefore read unassigned rather than failed for a few minutes after a sync, and restarting it in that window does nothing useful; the useful sequence is pause, sync, wait for every worker to rejoin, resume, then read task state and restart only what actually failed.
Connector configs carry passwords. A connector’s configuration holds the credentials it uses: in the workshop’s lab, the Avro converter’s settings carry the Karapace URL and its basic authentication username and password. KIP-297 records that Connect’s connector configurations hold plaintext passwords, stored in cleartext in its internal topics in distributed mode, and that the REST API exposes them over unsecured connections. Instaclustr serves the REST API over HTTPS, but it is reached with one credential from the Connection Info page, so everyone who holds it can change or delete any connector and read the configuration values the API returns. A shared tool decides who sees and edits those values; Kpow shows sensitive config values redacted and does not send them in plaintext when a config is edited (Kafka Connect management).
Connect pointed somewhere else. An Instaclustr Connect cluster can also target a Kafka cluster outside Instaclustr, such as Amazon MSK, configured with the bootstrap servers and connection security settings only, and Instaclustr’s documentation says it does not support TLS client authentication for that link. A tool should then show the Connect cluster beside the Kafka cluster it writes to, wherever that is; in Kpow the Connect settings sit in the same environment as that Kafka cluster’s, and one Kafka cluster can carry several Connect clusters (Kafka Connect configuration).
No tool here leads on every point. The Instaclustr console has nothing to deploy and is the only option that scores 10 on setup, Conduktor is level with Kpow on connector operations and keeps a restart history, and Lenses has built-in alert rules for Connect. Kpow governs people working through Kpow, so the REST credential Instaclustr issues still opens the API to anyone who holds it, and limiting who has it and which addresses the Connect firewall allows stays a job in the Instaclustr console.
Where Kpow has run on Instaclustr managed Kafka Connect
No Kpow customer has described running Kpow against Instaclustr’s managed Kafka Connect in public, so this section names none and the page ranks tools on the requirements above. The public record is Factor House’s own workshop material, which runs Kpow against Connect clusters provisioned on Instaclustr, and the partnership between the two companies described on the NetApp Instaclustr page.
-
Beyond the CLI workshop, Kafka Connect lab
- Custom connector from S3
- Firewall rules
- Kpow API with tenants
- Connector config with Karapace
The workshop’s Kafka Connect lab provisions an Instaclustr Connect cluster with custom connectors enabled, uploads a data generator JAR to the participant’s own S3 bucket and loads it with the console’s Sync button. Kpow then creates one connector from its UI by importing a JSON config whose Avro converter carries the Karapace URL and basic authentication credentials, and a second through the Kpow Enterprise API with a tenant header, so the API call is scoped to the AppTeam tenant. The lab notes that linking Connect to the Kafka cluster updates the broker firewall but not Karapace’s, so the Connect nodes’ public addresses have to be allowed by hand.
How a team runs Kpow with Instaclustr managed Kafka Connect
Allowing Kpow through. Kpow runs as one Docker container or Java JAR in a network the team controls, with no external database, and its host’s public address goes on the Connect cluster’s firewall rules as well as the Kafka cluster’s and Karapace’s (Instaclustr provider page). With CONNECT_STARTUP_VALIDATION=false, Kpow starts even when the Connect REST API cannot be reached, shows the error in the UI and keeps reconnecting, so a missing firewall rule shows up as a message rather than a container that will not start (Kafka Connect configuration).
Connecting the Connect cluster. The settings come from the Connect cluster’s Connection Info page: CONNECT_REST_URL as https://<node IP>:8083, CONNECT_AUTH=BASIC with CONNECT_BASIC_AUTH_USER and CONNECT_BASIC_AUTH_PASS, and either CONNECT_PERMISSIVE_SSL=true as in Kpow’s Instaclustr guide or CONNECT_SSL_TRUSTSTORE_LOCATION, CONNECT_SSL_TRUSTSTORE_TYPE and CONNECT_SSL_TRUSTSTORE_PASSWORD with a truststore built from the cluster’s certificates. A second Connect cluster on the same Kafka cluster is added with CONNECT_RESOURCE_IDS and a prefix per cluster (Kafka Connect configuration).
Creating connectors, custom ones included. The create form lists the plugins installed on the workers, so a custom connector appears once Instaclustr’s Sync has loaded it, with each field’s documentation beside it and form errors shown as the config is entered. A connector can be created from the form or exported as a REST call for a deployment pipeline (Kafka Connect management). In the workshop, one connector is created by importing a JSON config in the form and another through the Kpow Enterprise API with a tenant header, so an automated deployment runs inside one team’s tenant and its permissions.
Around a plugin sync. Before syncing a new connector version, the connectors on the cluster are paused from their row actions, and after every worker has rejoined they are resumed and their tasks read from the task table, where a failed task’s stack trace opens and the task restarts from the same row. With CONNECT_AUTO_RESTART set to connector names or wildcards, Kpow restarts failed connectors and their failing tasks, waits 10 minutes by default before trying the same connector again (CONNECT_AUTO_RESTART_WINDOW_MS) and restarts at most 50 per one-minute interval (CONNECT_AUTO_RESTART_LIMIT), and each automatic restart is written to the audit log as the kpow_system user (Kafka Connect configuration).
Deciding who may do what. Kpow’s authorization actions split Connect into CONNECT_CREATE, CONNECT_ALTER_STATE (pause, stop, resume and restart), CONNECT_EDIT_CONFIG, CONNECT_DELETE and CONNECT_INSPECT, and RBAC sets Allow, Deny or Stage per action, so the on-call role can pause and restart connectors without being able to read or edit their configs. Tenants give each team a view limited to its own connectors, topics and consumer groups. People sign in through SAML, OpenID Connect or LDAP, so every connector action carries a person from the company directory rather than the one Instaclustr credential.
Watching state and keeping the record. Kpow exports connector state counts per Connect cluster, including connect_connector_failed_total and connect_connector_task_failed_total, to Prometheus (listed in the metrics glossary), which fills the gap between Instaclustr’s per-worker startup counts and a task that fails while running. The audit log records every connector action with the user from the identity provider, in a topic on the team’s own Kafka cluster.
Product demo · 2 min
Kafka Connect monitoring and tasks: Kpow demo
Chad Harris walks through Kafka Connect in Kpow: connector and task state at a glance, historical health charts, deploying connectors from the UI, and bulk-restarting a subset of tasks.
Kpow live demo
See the Kpow Connect views before you add your Instaclustr firewall rule
The live Kpow demo needs no signup and runs on two Amazon MSK clusters rather than Instaclustr. Switch the cluster picker to MSK Primary to see its Kafka Connect cluster, with connectors, tasks and their configuration, then open the audit log topic on MSK Secondary.
For platform teams running connectors on Instaclustr.
Try the Kpow demoFAQ
What is the best Kafka tool for Instaclustr managed Kafka Connect?
On this page’s rubric, Kpow, with 98 of 110 points: it connects with the settings in its own Instaclustr guide, shows connector and task state with stack traces, pauses, restarts and auto-restarts connectors with an audit entry, redacts sensitive config values, splits connector permissions per action and per team, and runs as one container beside the cluster with no external database. Kafbat UI is the highest-scoring free option, and the Instaclustr console is the easiest to set up.
Does Kpow support Instaclustr managed Kafka Connect?
Yes. Kpow’s Instaclustr documentation says it manages a managed Kafka Connect cluster in an Instaclustr environment natively, through the Connect REST API with basic authentication, alongside the Kafka cluster over SASL/SCRAM and the Karapace Schema Registry add-on.
Do I need a separate firewall rule for Kafka Connect on Instaclustr?
Yes. Instaclustr keeps separate firewall rules for the Kafka cluster, the Karapace Schema Registry and the Kafka Connect cluster, and the address of the machine running the tool has to be added to each one it uses. If the Connect workers reach Karapace through its public endpoint, their public addresses also need to be on Karapace’s allowed list.
Why does Kpow’s Instaclustr guide use permissive SSL?
The guide says Instaclustr often requires permissive SSL for its Connect REST endpoints, so it sets CONNECT_PERMISSIVE_SSL=true, which disables certificate validation, as curl’s -k option does in Instaclustr’s own examples. Instaclustr publishes the Connect cluster’s server certificates on the Connection Info page, and Kpow takes a truststore for the Connect REST API, so a team that wants the certificate checked can configure that instead.
Why don’t the Instaclustr console’s Connect metrics show my failed task?
The console’s Connect metrics are per worker and count startup attempts and startup failures, so a task that started cleanly and failed later does not appear in them. Read the connector’s status through the REST API, or in a tool that shows task state and the stack trace, as covered in how to diagnose and fix a failed Kafka Connect connector.
Is a free tool enough for Instaclustr managed Kafka Connect?
For one team, often. Kafbat UI and AKHQ show connectors and tasks and restart them at no licence cost, and Instaclustr has published a guide for AKHQ with its managed Connect. Neither restarts failed connectors automatically, grants access that expires or holds a change for approval. Kpow Community Edition is also free, for 3 clusters and 10 users. Free options at any team size are compared in the best free Kafka UI tools.
How these tools were scored
Four of the six criteria are the ones Factor House scores on every page for teams that share a Kafka cluster, the fifth is connector operations, and the sixth is specific to Instaclustr. They are listed here in order of weight. Each criterion is scored 0 to 10: 10 where a tool is the only one here doing it or clearly the best, 8 for a clean documented pass, 5 or 6 for partial support or support that needs work the reader must verify, 1 to 4 for a weak or indirect form, and 0 where it is absent. The weights add up to 11, so totals are out of 110. Scores on the data path, connector operations, production access, audit and sign-in are the same as each tool’s scores on Best Kafka tools for Apache Kafka Connect, because Instaclustr’s managed Connect exposes the standard Connect REST API; the Instaclustr console and REST API option carries that page’s scores for the Connect REST API and scripts.
1. Out of the data path (counts three times). The tool should run in your own environment, reach Kafka as an ordinary client and Connect through its REST API, and keep no data outside your own cluster. Scored lower: tools that need an external database of their own, and tools whose controls work only when application or connector traffic passes through a vendor’s proxy.
2. Connector operations (counts twice). Whether the tool shows task state with stack traces, restarts a connector, a task or only the failed tasks, pauses and resumes connectors, restarts failed connectors automatically with a limit, creates and validates connectors, and reaches several Connect clusters.
3. Production access on request (counts twice). Whether an engineer can be granted access to production for one task and have it expire, whether a destructive change can be held for a second person’s approval, and whether sensitive fields can be masked from people who do not need them.
4. Audit trail per person (counts twice). Whether the tool records each action, reads included, against the person from the identity provider, and whether that record can be read in the product and sent to the systems that keep it long term.
5. Instaclustr Connect setup (counts once). Whether a guide from the tool’s vendor or from Instaclustr covers connecting to the managed Connect cluster, with its firewall rule, basic authentication and certificates; whether the server certificate is verified; and whether the tool copes with a Connect cluster that is not yet reachable. A guide that verifies the certificate or a tool that offers both modes scores 8, a generic Connect connection with no guide 5, and the provider’s own control plane 10.
6. Directory and Kafka sign-in (counts once). Whether people sign in through SAML, OpenID Connect or LDAP, and whether the tool connects to Kafka with the same SASL or TLS settings as any client and to the Connect REST API over HTTPS with authentication.
Costs are modelled for one production Kafka cluster with its Connect cluster and 25 engineers at $120 per engineer hour, using the same hours per tool class as Factor House’s other comparison pages. Tools with a licence carry the published price plus 2 hours a month to run. The open-source UIs carry 6 hours a month, $8,640 a year, to run, secure and keep current, and connector work with curl against the REST API carries 8 hours a month, $11,520. Instaclustr’s own charge for the Connect cluster is the same whichever tool is used, so it is left out. Kpow’s $7,380 uses the published price of $4,500 per cluster with 100 users included. Lenses publishes a Team price of $4,000 a year for up to 15 users on one cluster, so 25 engineers is a custom quote. Conduktor’s Console is $1,200 a seat on AWS Marketplace, and its Gateway Core and Gateway Protect prices are added for the data-level controls. Kpow Community Edition is free for 3 clusters and 10 users, so a 25-engineer team is on Enterprise.
The criteria map onto Instaclustr’s managed Kafka Connect in the figure below.
Every option is scored from 0 to 10 on each criterion, from the evidence and sources this page cites, and the reason for each score is on its card. The criteria are weighted: Out of the data path counts three times, Connector operations counts twice, Production access on request counts twice, Audit trail per person counts twice, Instaclustr Connect setup counts once and Directory and Kafka sign-in counts once, for a total out of 110. Out of the data path counts three times. Every Connect task is itself a Kafka producer or consumer, so a tool whose controls work through a proxy sits in front of every connector's reads and writes, and a tool that needs a database of its own is one more stateful service beside a Connect cluster that Instaclustr already runs for you. Connector operations, production access on request and the per-person audit trail count twice: Instaclustr hands the team one REST credential, and whoever holds it can create, change or delete any connector and read configs that carry passwords, so who may do that, and who did, decides whether a shared tool can be pointed at production Connect at all, and a tool that cannot read task state and restart the right thing leaves the team on curl. Instaclustr Connect setup and directory and Kafka sign-in count once. This page is published by Factor House, which makes Kpow. Every option is scored on the same rubric and the same sources: Kpow's per-criterion scores are set the same way as every other option's and are not adjusted, and the weights apply to every option alike. Kpow ranks first on its total of 98 out of 110. The other options follow by total. Conduktor is listed last whatever its total; on its total of 67 it would place third.
Related reading
- Kafka: the complete guide
- Kafka Connect
- Best Kafka tools for NetApp Instaclustr managed Kafka
- Best Kafka tools for Apache Kafka Connect
- Best tools to monitor Kafka Connect connectors
- How to diagnose and fix a failed Kafka Connect connector
- Best Kafka tools for Aiven
- Best Kafka UI tools for Amazon MSK