PostgreSQL Storage Engine

Features of the PostgreSQL storage engine for Tansu:

Batches of Kafka records are inserted using copy data
Deletion of messages older than retention.ms where the topic configuration cleanup.policy includes delete
Compaction of duplicate message keys on topics having a cleanup.policy that includes compact
Tables are partitioned on each topic partition for efficiency
Foreign key relationships are used to cascade deletion of topics and their associated metadata
Prepared statements used by the broker are cached and reused by subsequent connections
Multiple brokers can be used with the same PostgreSQL database
Each broker is completely stateless acting as the leader for any topic partitiion, transaction or consumer group

On startup the broker does not create the Tansu schema on PostgreSQL.

Enabled in the broker by using the --storage-engine parameter with a PostgreSQL Connection URI or setting the STORAGE_ENGINE environment variable.

Examples

Simple connection URI to as PostgreSQL database running on localhost:

STORAGE_ENGINE="postgres://postgres:postgres@localhost"

compose.yaml

A compose.yaml, showing PostgreSQL with an initialization script in ./etc/init.d containing 010-schema.sql. The broker isn't started until the db service is in a healthy condition:

---
services:
  tansu:
    image: ghcr.io/tansu-io/tansu
    environment:
      ADVERTISED_LISTENER_URL: tcp://tansu:9092
      STORAGE_ENGINE: postgres://postgres:postgres@db
    depends_on:
      db:
        condition: service_healthy
  db:
    image: postgres:18
    environment:
      PGUSER: postgres
      POSTGRES_PASSWORD: postgres
    ports:
      - 5432:5432
    volumes:
      - db:/var/lib/postgresql/data
      - ./etc/initdb.d/:/docker-entrypoint-initdb.d/
    healthcheck:
      test: /usr/bin/pg_isready
      interval: 5s
      timeout: 10s
      retries: 5
    pull_policy: missing
volumes:
  db:
    driver: local