MirrorMaker 2 on Connect: migrating off connect-mirror-maker.sh

Key takeaways

• Dedicated mode is not deprecated, and the reason to move is not the replication — it’s the control plane. If you don’t already run Kafka Connect, the trade is usually bad, because you give up per-flow isolation.
• Replication progress does not migrate. A new source connector starts at offset zero with auto.offset.reset=earliest, so seed every partition’s offsets before you start it, or the target gets a second copy of everything.
• sync.group.offsets.enabled defaults to false: MM2 writes checkpoint records but does not commit translated offsets to the target’s consumer groups.
• Translation is unavailable for groups that were already behind when MM2 started, and it only lands on groups that are idle on the target. Let every group reach zero lag before you flip.
• Set tasks.max deliberately: it defaults to 1, so omitting it hands every partition to a single task.
• Do not run old and new over the same topics. They share internal topics and heartbeats; split by topic set, or freeze and cut over.

Introduction

A lot of people ask me about this migration, and nearly all of them started in the same place. That is not a coincidence. Apache’s MirrorMaker 2 documentation only covers the dedicated cluster — its opening line is that the following sections “describe how to configure and run a dedicated MirrorMaker cluster,” and running MM2 inside an existing Connect cluster is deferred to a KIP. Dedicated mode is also the easier start: one file, one script, no Connect cluster to stand up or size.

Then they hit the ceiling. There is no management API, so changing anything means editing the file and restarting the process, and for most of dedicated mode’s life that restart could silently not take effect. Scope is next, then the argument for and against moving; everything after that is what breaks on the way.

Scope

You run connect-mirror-maker.sh with an mm2.properties. You want those three connectors deployed on a Kafka Connect cluster you operate, so you get the REST API, per-connector lifecycle control, and the monitoring and tooling your Connect cluster already has. That’s the whole article.

Out of scope, deliberately: MirrorMaker 1 (removed in Kafka 4.0; it has none of the machinery below, so migrating from it is building a new deployment, not converting one) and operator- or service-managed platforms like Strimzi’s KafkaMirrorMaker2 and MSK Connect. Those hand the Connect worker to something else, which removes the two fixes this article is mostly about. If you’re on one, read the seeding step in the playbook and ask your platform whether you can do it; if not, design the cutover around a frozen window instead.

Throughout, the example is a one-way flow from us-west to us-east, with a consumer group called orders-api.

You need Kafka 3.6 or later on the Connect side. The step this article depends on is altering connector offsets: MirrorSourceConnector.alterOffsets() and the PATCH /connectors/{name}/offsets route both first appear in Kafka 3.6. KIP-875’s read and reset endpoints arrived earlier, in 3.5, so reading offsets is not the problem — writing them is. Below 3.6 the seeding strategy is unavailable and the freeze-window path is your only option.

Should you do this at all?

Dedicated mode is not deprecated. It still ships in Kafka 4.3, it is what Apache’s MirrorMaker 2 documentation describes, and the Kafka team explicitly rejected abandoning its distributed mode. If somebody told you connect-mirror-maker.sh is going away, they were wrong, and you can stop reading.

So the reason to move is not that dedicated mode is broken. It is that you don’t get a control plane. There is no management API: the config file is the control plane, so changing anything means editing mm2.properties and restarting the process. KIP-710 was implemented in 3.5.0, so dedicated.mode.enable.internal.rest exists — but it registers only the internal REST surface (InternalMirrorResource: writing task configurations and fencing zombies), and none of the public management API. Even with the flag on there is no /connectors, no /connectors/{name}/config, and no offsets endpoint. Whether that was deliberate is murky — KIP-710’s author called it “an oversight in the KIP-710 design/implementation,” and the maintainer who fixed the fallout concluded the public API had been “intentionally left out of the implementation to avoid security problems,” because internal endpoints are secured by default while the connector-config endpoint is not. If you are waiting for dedicated mode to grow an API, don’t. Two consequences followed:

• KAFKA-15372 (fixed 3.6.2 and 3.7.0) — on a multi-node cluster, a rolling restart could silently drop configuration changes. Leadership ping-pongs: each node restarts, misses the leadership window, and fails to apply its config. The workaround was to fully stop and start the whole deployment. Enabling the internal REST did not help, because the connector-config endpoint was not part of KIP-710.
• KAFKA-10582 (fixed 3.5.0) — a new source topic got its remote topic created but never started replicating until MM2 was restarted. Users reported this on 2.5.1, 2.6.1, 2.7.0, 3.0.0 and 3.3.1, and the fix only takes effect if you also set dedicated.mode.enable.internal.rest=true. That is roughly three years in which scaling dedicated mode out quietly broke new-topic replication.

In Connect-managed mode that whole class of problem disappears, because PUT /connectors/{name}/config is an ordinary REST request and Connect forwards it to the leader for you.

Four things follow, permanently:

1. Change config without restarting replication. In dedicated mode, editing topics, groups, intervals or tasks.max means bouncing the process: replication stops, then catches up. PUT /connectors/{name}/config applies to a running Connect cluster, and Connect rebalances tasks itself.
2. Per-connector and per-task control. Pause the checkpoint connector without stopping replication. Restart one failed task. Read connector status. Dedicated mode gives you logs and JMX, and nothing else.
3. Offset management. GET, PATCH and DELETE /connectors/{name}/offsets is the only supported way to skip a poison record, rewind a topic, or recover a connector that has started re-replicating. Dedicated mode has no supported way at all; people hand-edit internal topics, which is as bad as it sounds.
4. A desired-state surface. Runbook steps, CI checks and GitOps reconcilers all need an API. A file plus a restart is not programmable.

That said, this is operational leverage, and leverage only pays if you operate at a rate where it matters. If your replication config changes a few times a year, you have never needed to intervene in a connector’s offsets, and you don’t already run Connect, the migration buys you very little and costs you per-flow isolation. The strongest reason to move is usually not capability at all: it is estate consolidation — MM2 as a tenant of the Connect platform you already run, monitor, secure and upgrade, rather than a second process with its own fleet and no API — or a platform that only offers Connect-managed MM2.

Move if any of the four capabilities above is something you have wanted and could not have; or you already run Connect and want MM2 on the same control plane; or your platform leaves you no choice.

Don’t move if your replication config is stable, you have never needed to intervene in a connector’s offsets, and standing up or joining a Connect cluster would be new work. Dedicated mode is supported and it works. This is a trade, not an upgrade. Here is what you would give up:

• Per-flow isolation. Each flow currently gets its own logical Connect cluster: its own config, offset and status topics, and its own worker group. On a shared Connect cluster every MM2 connector shares one worker fleet and one connect-offsets topic, so a hot flow can starve the others, and all replication progress lives in one place.
• One file for the whole topology. Fan-out, aggregation and active/active fall out of mm2.properties. On Connect, topology becomes a deployment-placement decision.
• Worker state on the target, and heartbeat wiring that just happens.

KIP-710’s authors put it plainly when they rejected documenting distributed dedicated mode as unsupported: “Since the dedicated mode of MM2 provides essential improvements over the vanilla Connect mode (namely in terms of configuration and operations), distributed mode should be fixed instead of abandoned.” They were right. That is why this article is about how to move rather than whether to.

Why this migration is dangerous

connect-mirror-maker.sh is not a wrapper — it’s a Connect driver. It creates one internal Connect cluster per replication flow (a worker plus a herder each), and that cluster’s elected leader writes the three connector configs itself. The worker’s config store, offset store, and status store live on the target cluster, named after the source: mm2-configs.<source>.internal, mm2-offsets.<source>.internal, mm2-status.<source>.internal, with group.id = <source>-mm2 (MirrorMakerConfig).

So the aliases are the deployment’s identity, and the driver’s state is what makes replication resume where it left off. In Connect-managed mode the connectors become ordinary configs in your Connect cluster, their progress moves to your connect-offsets topic, and their identity becomes the connector name you chose.

The consequence is the whole article: configuration can be retyped; state cannot be moved. Almost every serious failure here comes from something that lived in the driver’s state and was left behind.

One thing that doesn’t change: the legacy metrics. Same kafka.connect.mirror namespace in both modes, because it’s the same connector classes — the widely repeated claim that the MBean prefix differs is wrong, and traces back to a name in KIP-382 that never shipped. What does change is the newer names: KIP-1280 moved MM2 onto KIP-877’s Connect metric names in 4.3 (kafka.connect:type=plugins,connector=<connector>,task=<task>,source=…), and that connector label is the connector’s own name — always MirrorSourceConnector in dedicated mode, whatever you named it in Connect-managed mode. Dashboards built on the new names break across this migration even though the legacy namespace does not.

The eight catches that change what you do

Catch 1 — Replication progress does not migrate, and the naive move duplicates everything

What breaks. You deploy the connectors, point them at the same source and target, and the target fills with a second copy of every record from the beginning of time.

Why. The source task stores its position in the Connect worker’s offset store under {cluster, topic, partition} and seeks to storedOffset + 1. In dedicated mode that store is mm2-offsets.<source>.internal on the target; on your Connect cluster it’s connect-offsets, and it’s empty for these connectors. MM2 also sets auto.offset.reset=earliest unless you override it — it is a putIfAbsent, so consumer.auto.offset.reset wins — and with no stored offset that means “start at zero”.

Do this. Seed every partition before replication starts, via the API that MirrorSourceConnector.alterOffsets() implements for exactly this. Connect will refuse unless the connector is stopped: modifyConnectorOffsetsChecks requires target state STOPPED and zero tasks, and otherwise returns a 400 reading “Connectors must be in the STOPPED state before their offsets can be modified.” Setting enabled=false does not satisfy it — that is MM2’s own switch, and it leaves Connect’s target state at RUNNING.

CONN=http://connect:8083/connectors/<connector>

curl -sS -X PUT $CONN/stop          # required before offsets can be altered

curl -sS -X PATCH $CONN/offsets -H 'Content-Type: application/json' -d '{
  "offsets": [
    {"partition": {"cluster": "us-west", "topic": "orders", "partition": 0},
     "offset":    {"offset": 14829371}}
  ]}'

curl -sS -X PUT $CONN/resume        # begin replicating from the seeded position

The keys are cluster (the source alias), topic, partition; the offset is the last source offset already replicated. Get the values by freezing producers, confirming the old deployment is at zero lag, and seeding HWM - 1 per partition — or by decoding the old mm2-offsets.<source>.internal topic.

Then prove it: after resume, confirm target high-water marks do not move until new records are produced. A connector that “catches up” in ten seconds on a huge topic did not catch up.

Catch 2 — Offset translation is off by default in the sense that matters

What breaks. Your runbook says consumers will resume near where they left off. After cutover they start wherever auto.offset.reset puts them.

Why. MirrorCheckpointConnector always writes checkpoint records to <source>.checkpoints.internal on the target. It only commits translated offsets into the target’s __consumer_offsets when sync.group.offsets.enabled=true, and that defaults to false. Checkpoints without it are a data source, not an action. Kafka clients never read that topic on their own. The documentation isn’t wrong here; it is describing a flag that is off.

Do this. Choose deliberately. Automatic: set sync.group.offsets.enabled=true on the checkpoint connector. Manual: translate with RemoteClusterUtils.translateOffsets() and apply the result yourself — there is no Apache CLI for the translation, so at scale this means a small tool, then kafka-consumer-groups.sh --reset-offsets --from-file.

Catch 3 — Translation only lands on idle groups, and is unavailable for lagging ones

What breaks. Some groups get sensible offsets, some get offsets behind their real progress, and some get nothing at all — silently.

Why. Three mechanisms. (a) The checkpoint task only records target offsets for groups in state EMPTY, and alterConsumerGroupOffsets fails with UnknownMemberIdException if the group is active on the target. (b) OffsetSyncStore refuses to translate when the newest sync is ahead of the group’s committed offset — the javadoc says translation is unavailable “if replication started after the position of the consumer group,” so a group already behind when MM2 started gets nothing. (c) Elsewhere it is approximate by design: when the group is ahead of the newest usable sync, MM2 translates to at most one record past that sync, preferring re-delivery over data loss. offset.lag.max (default 100) bounds sync frequency, and the config doc gives the arithmetic: “Partition Count x offset.lag.max = Approximate duplicated record count.”

Do this. Design consumers for at-least-once across the cut. Let groups reach zero lag before flipping. Use a fresh group ID for every rehearsal — resetting offsets does not clear MM2’s checkpoint state, so stale checkpoints keep winning.

Catch 4 — tasks.max defaults to 1, and the ten-partition rule you may have read is obsolete

What breaks. Two different things that get conflated. Set tasks.max too low and one task carries every partition, so replication lags well behind what the cluster can absorb. On Kafka 3.4.0 and earlier there was a second, worse failure: assigning more than ten partitions to a task could stall offset translation for those partitions permanently.

Why. tasks.max defaults to 1, so omitting it hands every partition to a single task — the usual cause of “MM2 is slower than it should be”. The ten-partition problem was a bug, not a design limit. OffsetSyncWriter caps in-flight offset-sync records with MAX_OUTSTANDING_OFFSET_SYNCS = 10, a producer backpressure buffer that has nothing to do with partition counts. In KAFKA-12558 the partition state was mutated before the send was confirmed, so once that buffer saturated the sync was never retried — “additional offset syncs are unlikely to arrive for a long time, if ever.” Fixed in 3.4.1 and 3.5.0 — the KAFKA-12558 ticket lists a 3.3.3 fix version, but no 3.3.3 release ever shipped (3.3 ended at 3.3.2), so 3.4.1 is the first release with the fix. The constant is still 10 because it is a buffer, not a cap.

Do this. Set tasks.max from your partition count and throughput target, and confirm it under load rather than trusting a formula. Only if you are stuck on a version older than the fix should you keep it to ten partitions per task.

Catch 5 — You cannot cleanly run old and new side by side over the same topics

What breaks. The plan says “run both and compare.” Instead: duplicated heartbeat records, two writers on one checkpoint topic, and — if both are dedicated-mode — one silently winning the config for the whole cluster pair.

Why. With matching aliases the deployments share mm2-offset-syncs.<peer>.internal, heartbeats, and <source>.checkpoints.internal. Apache’s docs warn that MM2 processes sharing a target cluster share configuration: “either the topic foo or the topic bar is replicated, but not both.” And you can’t partially disable checkpointing — emit.checkpoints.enabled=false is rejected by validation, so the whole connector must be disabled.

Do this. Prefer a freeze-window cutover (see the playbook). If you must run in parallel, split by topic set, prove the sets are disjoint by listing them, and set heartbeats.replication.enabled=false on exactly one side.

Catch 6 — Aliases and separator are baked in, and alias drift disables cycle detection

What breaks. Renaming an alias after go-live renames every replicated topic and orphans the internal topics. Worse: if the two directions ever disagree on the alias strings, cycle detection stops working — each side re-replicates the other’s prefixed topics, names grow by a prefix per hop, and at Kafka’s 249-character topic-name limit (Topic.MAX_NAME_LENGTH) topic creation fails and every connector task dies (Instaclustr).

Why. The alias appears in the internal topic names, the worker group.id, the default topic prefix, metric tags, and the cluster key in every stored offset. Cycle detection works by parsing the alias back out of a topic name, so it depends on every configuration agreeing on those strings.

Do this. Treat aliases and separator as immutable, keep them short (they eat the 249-character budget), and add an automated check that every connector for a pair agrees on both aliases. That’s a correctness check, not style. If you choose IdentityReplicationPolicy instead — the usual choice when consumers must keep unprefixed names — accept that cycle detection is impossible by construction, and note that heartbeat topics are renamed even then.

Catch 7 — offset-syncs.topic.location defaults to source, and the topic is named for the peer

What breaks. The connector fails on startup needing write access to a cluster your policy treats as read-only. Or you look for the offset-syncs topic and can’t find it.

Why. The default puts mm2-offset-syncs.<targetAlias>.internal on the source cluster — note the target alias in the name. Setting location=target gives mm2-offset-syncs.<sourceAlias>.internal on the target instead. Moving it later means a new topic and losing all offset-sync history.

Do this. Choose explicitly, document it, and grant the ACLs it implies on both clusters. One-way door.

Catch 8 — replication.factor for data topics defaults to 2

What breaks. Replicated topics are created RF 2 on a three-broker target. Nobody notices until a broker fails.

Why. MirrorSourceConfig.REPLICATION_FACTOR_DEFAULT = 2 — neither the source’s RF nor the target broker default. MM2 never copies min.insync.replicas either, and never updates the RF of a topic it didn’t create.

Do this. Set replication.factor explicitly and audit it with kafka-topics.sh --describe after first replication. It’s invisible in MM2’s metrics.

Also check

These are real, but they don’t change the migration plan, so they get a line each here rather than their own sections.

• Internal topic RFs default to 3 — and a config key on the wrong connector is a silent no-op, because ConfigDef.validateAll() ignores unknown keys and PUT /config still returns 200.
• MM2 recreates a remote topic you delete and resumes from its stored offset, so a bare delete leaves a topic with only recent data and a healthy-looking lag metric.
• Topic config sync will delete target-side tuning. use.defaults.from=target issues DELETE ops for anything at default on the source, every 600s. It never copies min.insync.replicas; it does revert your retention.ms.
• ACL sync is narrow — topic LITERAL only, WRITE dropped, ALL downgraded to READ. Its “no authorizer” diagnostic points at the source and logs at INFO.
• source.cluster.alias is required; target.cluster.alias silently defaults to "target" — a wrong value there corrupts metric tags and checkpoint attribution without erroring.
• A “one-way” deployment still runs a reverse herder, because heartbeats need it. Inventory by listing topics and *-mm2 consumer groups, not by reading the file.
• MM2 ships no replication-lag metric. You derive it. Heartbeat lag (the replicated <source>.heartbeats topic on the target) and connector metrics fail differently — keep both alerts. Legacy metric names go away in Kafka 5.0.
• Worker-level config is invisible to translation: config.providers, plugin version, internal-topic settings, and exactly.once.source.support (a worker setting, not a connector key). EOS can silently degrade to at-least-once.

Playbook

1. Inventory what’s actually running. Not the file — it may be years out of date, and unknown keys were silently discarded. List both clusters’ internal topics, *-mm2 consumer groups, every consumer group with its lag (lagging groups get no clean translation — catch 3), and partition counts for the tasks.max calculation.

2. Decide the target. A Connect cluster dedicated to MM2 is simpler to reason about; a shared one is cheaper but couples MM2’s throughput to its neighbours and hands you its worker config. Either way, record where replication progress will live, because that’s catch 1.

3. Translate config, with one owner per key. The rows that actually bite:

mm2.properties	Connector JSON	Goes on
us-west.bootstrap.servers / us-east.bootstrap.servers	source.cluster.bootstrap.servers / target.cluster.bootstrap.servers	all three
us-west.consumer.* / us-east.producer.*	consumer.* (or source.consumer.*) / producer.* (or target.producer.*)	all three / source
us-west->us-east.topics and .topics.exclude	topics, topics.exclude	source and checkpoint, identical
us-west->us-east.groups and .groups.exclude	groups, groups.exclude	checkpoint only
replication.policy.class, replication.policy.separator, offset-syncs.topic.location	same keys	all three, identical
replication.factor / checkpoints.topic.replication.factor / offset-syncs.topic.replication.factor	same keys	source / checkpoint / source
tasks.max	tasks.max	source and checkpoint (heartbeat is always one task)
exactly.once.source.support	worker config	Connect worker, not a connector

Two rules. Every key needs an owner — validate each connector against the plugin’s own validator and diff “keys I set” against “keys the connector knows,” because a misplaced key is silent. And don’t use producer.override.* — that configures Connect-created clients, not MM2’s internal ones.

4. Pre-flight the Connect cluster. Worker version matches the MM2 classes (one plugin location); internal topics exist, compacted, with the RF you want (remember connect-offsets now holds your replication progress); config.providers configured; ACLs granted — including CreateTopics/Write on the source if you kept offset-syncs.topic.location=source; and tasks.max set from your partition count and throughput target rather than left at the default of 1.

5. Seed, then prove nothing is re-read. Create the connector, PUT /connectors/<connector>/stop — offsets cannot be altered while it is running — then freeze producers. Confirm the old deployment is at zero lag. Seed HWM - 1 per partition (catch 1) and PUT /connectors/<connector>/resume. Record target high-water marks, wait one full refresh.topics.interval.seconds (600s), and re-check: they must be unchanged. If they moved, stop and re-seed.

6. Validate translation in a shadow group. Take a group that is not yet in production — orders-api in this example — and consume, commit and stop it on the source. Wait emit.checkpoints.interval.seconds + sync.group.offsets.interval.seconds (60 + 60) plus margin. Compare its target offsets against source. If zero or absent, work through: flag off, group still active on target, connector not running or unauthorized, offset precedes the earliest sync, topics filter mismatch.

7. Cut over in this order. Stop source producers → wait for source consumers to reach zero lag and commit → wait for MM2 to translate those offsets → update target ACLs and start target producers → start target consumers → stop the old deployment. The order is not arbitrary: producers early means dropped records, consumers early means MM2 can’t write into an active group and you get extra re-delivery.

8. Decommission and roll back. Keep the old internal topics — mm2-offsets.<source>.internal, mm2-offset-syncs.<peer>.internal, <source>.checkpoints.internal — for the full confidence window. Rollback is not a clean restart: the old process resumes from its own offsets and re-replicates everything the new one did. Plan for that duplicate window, or re-seed the old deployment’s offsets the same way you seeded the new one.

Checklist

• GET /connectors/<connector>/offsets returns an offset for every topic-partition in the filter.
• Target high-water marks are unchanged after a full refresh.topics.interval.seconds with producers still frozen.
• tasks.max is set explicitly and confirmed under load, not left at the default of 1.
• replication.factor set explicitly, and --describe on a new remote topic shows the intended RF.
• sync.group.offsets.enabled=true, or a documented manual translation that has been run end to end.
• A shadow group translated correctly, compared with kafka-consumer-groups.sh --describe on both clusters.
• Every cutover group had zero lag on the source before the flip, or is documented as accepting re-delivery.
• Internal topics exist on the expected clusters with the intended RF and cleanup.policy=compact.
• replication.policy.*, both aliases, and offset-syncs.topic.location are identical across all connectors for the pair — checked programmatically.
• topics and topics.exclude are identical between source and checkpoint connectors.
• A config diff of each connector shows no keys its ConfigDef doesn’t own.
• Heartbeat lag is under threshold, and a lag alert has been fired and resolved artificially.
• Topic configs re-read on the target after one sync interval; target-local configs are either excluded or accepted as transient.
• The topic-recreation runbook exists and has been walked through by whoever is on call.
• Old internal topics retained with a written deletion date; rollback runbook states the duplicate window.

Closing

Migrate for the control plane, not for the replication. Keep the aliases stable, seed the offsets before you start the new connectors, and let every consumer group reach zero lag before you flip. Do that and the move is boring, which is the point.

Further reading

Kafka’s behaviour is cited inline to the connect/mirror classes on GitHub. The references actually worth your time, and why:

• Ryanne Dolan, KIP-382: MirrorMaker 2.0 — the original design, and the reason a driver exists at all. Note that its MBean name (kafka.mirror.connect) never shipped, so don’t cite it for JMX.
• Daniel Urban, KIP-710 — one nested Connect node per replication flow. Implemented in 3.5.0, but only the internal REST surface, which is why dedicated mode still has no management API. The most useful KIP for understanding what you are moving between.
• Omnia Ibrahim, KIP-690 and KIP-716 — internal-topic naming and offset-syncs location. Read these if you are confused about why a topic name contains the other cluster’s alias.
• Apache Kafka geo-replication docs — the operational source of truth for dedicated mode: config-sharing conflicts, metrics, exactly-once requirements.
• Greg Harris in KAFKA-19607 — the cutover sequence and the “reach zero lag or tolerate re-delivery” rule, from the person who maintains this code. KAFKA-12558 (Alan Ning) is where the ten-partitions-per-task limit comes from.
• Instaclustr’s field notes — the only write-up I found that documents alias drift silently breaking cycle detection, plus independent confirmation of the replication.factor=2 default.

Related reading

• Kafka Connect
• Kafka consumer offsets