Apache Kafka® compatible broker

PostgreSQL, SQLite or S3 storage

AVRO, JSON or Protobuf schema validation

Iceberg or Delta Lake support

Open Source Apache licensed

Get startedView on GitHub
broker.rs
Cargo.toml
async move {
    Frame::response(
        Header::Response { correlation_id },
        self.response_for(client_id, body, correlation_id)
            .await?,
        api_key,
        api_version,
    )
    .map_err(Into::into)
}
.instrument(span)
.await

Introduction

Getting started

Tansu is a drop-in replacement for Apache Kafka with PostgreSQL, SQLite, S3 or memory storage engines. Without the cost of broker replicated storage for durability. Licensed under the Apache License. Written in 100% safe 🦺 async 🚀 Rust 🦀.

Installation

Step-by-step guides to setting up Tansu with various storage engines.

Storage Engines

Explore pluggable storage engines supporting PostgreSQL, S3 or memory.

Quick start

Tansu is published on crates.io, and installed using cargo, part of the Rust systems programming language. You can easily install cargo via rustup.

Tansu uses cargo features to enable the storage engines:

FeatureDescription
libsqlSQLite storage
postgresPostgreSQL storage
dynostoreS3 compatible and memory storage

Similarly, support for the Apache Iceberg or Delta Lake open table formats can be enabled via:

FeatureDescription
icebergApache Iceberg open table format support
deltaDelta Lake open table support

We will use the SQLite storage engine for this quick start:

cargo install tansu --features libsql

Alternatively, you can enable all the features with:

cargo install tansu --all-features

Tansu is a single executable, creating a 40-150MB binary depending on the features enabled.

Once cargo has finished building, you can verify which storage engines are enabled with:

tansu --help

Assuming you've just included SQLite support you will see:

Storage engines: libsql

Now, we will start the broker:

tansu --storage-engine=sqlite://tansu.db

Tansu uses the RUST_LOG, environment variable if you want to enable a lot of logging you can use:

RUST_LOG=debug tansu --storage-engine=sqlite://tansu.db

Leaving the broker running, switching to another shell, we can maintain topics using the tansu topic subcommand:

tansu topic --help

Using the topic command, we can create, delete or list topics available on the broker.

tansu topic create test

We can verify the topic has been created on the broker with:

tansu topic list | jq '.[].name'

We can use the tansu cat subcommand to produce data to the topic:

echo '{"value": "Hello World!"}' | tansu cat produce test

We can consume from the topic using:

tansu cat consume test

Which will output:

[{"key":null,"value":"Hello World!"}]

Getting help

Thank you for your interest in Tansu. Have a question? Please Contact us.

Submit an issue

Please raise any bug reports as an Issue on our GitHub.

Next
Features