S3

Features of the S3 storage engine for Tansu:

• Multiple brokers can use the same S3 bucket using conditional writes (both to check for the existence of an object before creating it and when overwriting an existing object)
• Each broker is completely stateless acting as the leader for any topic partitiion, transaction or consumer group

Configuration

Enabled in the broker by using the --storage-engine=s3://tansu parameter or setting the STORAGE_ENGINE environment variable. Where tansu is the name of a bucket.

Example

STORAGE_ENGINE="s3://tansu"

MinIO

An example compose.yaml, using MinIO as a S3 compatible storage service:

---
services:
  tansu:
    image: ghcr.io/tansu-io/tansu
    environment:
      ADVERTISED_LISTENER_URL: tcp://tansu:9092
      RUST_BACKTRACE: 1
      RUST_LOG: debug,tansu_sans_io=warn,tansu_model=warn
      STORAGE_ENGINE: "s3://tansu/"
      AWS_ACCESS_KEY_ID: "minioadmin"
      AWS_SECRET_ACCESS_KEY: "minioadmin"
      AWS_DEFAULT_REGION: "eu-west-2"
      AWS_ENDPOINT: "http://minio:9000"
      AWS_ALLOW_HTTP: "true"
    links:
      - minio
  minio:
    image: quay.io/minio/minio
    command: server /data --console-address ":9001"
    volumes:
      - minio:/data
    ports:
      - 9000:9000
      - 9001:9001
    environment:
      MINIO_PROMETHEUS_AUTH_TYPE: public
    healthcheck:
      test: timeout 5s bash -c ':> /dev/tcp/127.0.0.1/9000' || exit 1
      interval: 5s
      retries: 1
      start_period: 5s
      timeout: 5s
volumes:
  minio:
    driver: local