Why
At Callisto Labs, we are working towards expertise in the real-time data streaming ecosystem. In order to have a complete understanding of the tools available, we decided to build our own Kafka client in Rust. The language choice was obvious for us, but our main reasons were:
- No C lib dependencies, just a self contained executable.
- Tokio as the asynchronous runtime enables performant concurrency and lightweight green-threads that can outclass OS threads.
- Type and memory safety bring table-stake reliability and resiliency to your distributed systems.
- With speed being necessary in real-time contexts, native Rust is always in competition to be the most performant tool implementations.
- Minimal resource usage make for small containers that spin up quickly.
Our library is named samsa and we are releasing version 1.0, complete with Producers, Consumers, and Consumer Groups as well as support for Compression, TLS, SASL. We are proud to say this is the most complete open source Kafka client in Rust.
What
We dug deep into the Kafka protocol and implemented each of the core functionalities. Using nom as the parser combinator library was key to keeping that code concise and readable. Rust's type conversion primitives and trait system helped out networking code stay simple and reusable across multiple protocols.
Samsa supports TLS and SASL in order for our users to securely connect to their production clusters. Furthermore we support Gzip compression to assist with keeping network transmissions as small as possible. Today, users can set up individual producers and consumers, as well as full fledged consumer groups with automatic rebalancing.
Goals
Going forward, we hope to continue to support this project and optimize for speed. With building for simplicity and no optimizations, our initial benchmarks show that samsa producers and consumers can read and write at a rate of 500k short messages each second. The memory usage for a consumer reading at full speed is 4-5Mb. With the appropriate optimizations we can get these benchmarks to be even more impressive.
We plan to see a number of tools built using samsa. Our own plans involve such; we are already developing tools that allow end users consume to Kafka streams en masse. We are using samsa along with a custom stream processing library, also written in pure Rust.