Mirror Maker

An example compose.yaml using Mirror Maker with Tansu:

• a and b are brokers using the SQLite storage engine
• mirror is Apache Mirror Maker setup to replicate all topics from a to b

In this example, b could be used as a standby in the case of failure of a. Fetchers could be directed to use b if the load on a became too heavy.

This example could be adapted to use another storage engine on b (e.g., S3 or PostgreSQL).

compose.yaml

---
services:
  a:
    image: ghcr.io/tansu-io/tansu
    environment:
      ADVERTISED_LISTENER_URL: tcp://a:9092
      CLUSTER_ID: a
      RUST_LOG: warn
      STORAGE_ENGINE: sqlite:///data/a.db
    volumes:
      - ./data/:/data/
  b:
    image: ghcr.io/tansu-io/tansu
    environment:
      ADVERTISED_LISTENER_URL: tcp://b:9092
      CLUSTER_ID: b
      RUST_LOG: warn
      STORAGE_ENGINE: sqlite:///data/b.db
    volumes:
      - ./data/:/data/
  mirror:
    image: apache/kafka:4.1.0
    command: ["/opt/kafka/bin/connect-mirror-maker.sh", "/srv/mm2.properties"]
    configs:
      - source: mm2_properties
        target: /srv/mm2.properties
    links:
      - a
      - b
configs:
  mm2_properties:
    file: mm2.properties

mm2.properties

# specify any number of cluster aliases
clusters = A, B

# connection information for each cluster
# This is a comma separated host:port pairs for each cluster
# for e.g. "A_host1:9092, A_host2:9092, A_host3:9092"
A.bootstrap.servers = a:9092
B.bootstrap.servers = b:9092

# enable and configure individual replication flows
A->B.enabled = true

# regex which defines which topics gets replicated. For eg "foo-.*"
A->B.topics = .*

B->A.enabled = false
B->A.topics = .*

# Setting replication factor of newly created remote topics
replication.factor=1

checkpoints.topic.replication.factor=1
heartbeats.topic.replication.factor=1
offset-syncs.topic.replication.factor=1

offset.storage.replication.factor=1
status.storage.replication.factor=1
config.storage.replication.factor=1

sync.topic.acls.enabled = false
replication.policy.class=org.apache.kafka.connect.mirror.IdentityReplicationPolicy

admin.timeout.ms = 5000
refresh.topics.interval.seconds = 5