kafka-rs

🦀 Kafka-RS: Educational Kafka Implementation in Rust

Rust License: MIT Rust Version

An educational Apache Kafka-compatible message broker implementation in Rust, designed for learning how Kafka works under the hood. Built with Domain-Driven Design (DDD) principles to demonstrate clean architecture patterns in systems programming.

🎯 Purpose

This project serves as an educational resource for understanding:

⚠️ Educational Use Only: This implementation prioritizes learning and clarity over production performance. It uses in-memory storage and simplified algorithms.

✨ Features

🔌 Kafka Protocol Compatibility

🏗️ Domain-Driven Design Architecture

📚 Educational Focus

🚀 Quick Start

Prerequisites

Installation & Running

# Clone the repository
git clone https://github.com/elsonwu/kafka-rs.git
cd kafka-rs

# Run the server (default port 9092)
cargo run --bin kafka-rs

# Or with custom port
cargo run --bin kafka-rs -- --port 9093

# Run tests
cargo test

# Run with logging
RUST_LOG=info cargo run --bin kafka-rs

The server will start and listen on localhost:9092 (or your specified port).

🧪 Testing with KafkaJS

const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'test-app',
  brokers: ['localhost:9092']
});

const producer = kafka.producer();
const consumer = kafka.consumer({ groupId: 'test-group' });

async function example() {
  // Connect
  await producer.connect();
  await consumer.connect();
  
  // Produce messages
  await producer.send({
    topic: 'test-topic',
    messages: [
      { key: 'key1', value: 'Hello Kafka-RS!' },
      { key: 'key2', value: 'Educational implementation' }
    ]
  });
  
  // Consume messages
  await consumer.subscribe({ topic: 'test-topic' });
  await consumer.run({
    eachMessage: async ({ partition, message }) => {
      console.log({
        partition,
        offset: message.offset,
        key: message.key?.toString(),
        value: message.value?.toString()
      });
    }
  });
}

example().catch(console.error);

📖 Documentation

Comprehensive documentation is available in the docs/ folder:

Core Documentation

Deep Dive Documentation

These deep dive documents capture the intricate details of Kafka’s operation that you can only discover by implementing a compatible system from scratch.

🏗️ Architecture

graph TB
    subgraph "Infrastructure Layer"
        A[TCP Server] --> B[Protocol Handler]
        B --> C[Message Encoding/Decoding]
    end
    
    subgraph "Application Layer"
        D[Use Cases] --> E[DTOs]
    end
    
    subgraph "Domain Layer"
        F[Entities] --> G[Value Objects]
        G --> H[Domain Services]
        H --> I[Repository Interfaces]
    end
    
    A --> D
    D --> F

Domain Layer (src/domain/)

Application Layer (src/application/)

Infrastructure Layer (src/infrastructure/)

🧪 Testing

# Run all Rust tests
cargo test

# Run specific test suite
cargo test --test integration_tests

# Run with output
cargo test -- --nocapture

# Run benchmarks (if available)
cargo bench

Kafka Client Integration Test

This project includes a comprehensive integration test using real Kafka JavaScript clients to verify protocol compatibility:

# Prerequisites: Node.js 18+ required
cd integration/kafka-client-test
npm install

# Start the Kafka server (in another terminal)
cargo run --release -- --port 9092

# Run the integration test
npm test

The integration test:

Test Coverage

🛠️ Development

Quick Development Commands

We provide a Makefile with common development tasks:

# Show all available commands
make help

# Format code (fixes formatting issues automatically)
make format

# Check formatting without making changes  
make check

# Run linting with clippy
make lint

# Run all tests
make test

# Build the project
make build

# Build in release mode
make build-release

# Start the Kafka server
make server

# Fix formatting and linting issues
make fix

# Run integration tests with real Kafka clients
make integration-test

# Run pre-commit checks (format + lint + test)
make pre-commit

# Run CI checks locally
make ci

Development Workflow

  1. Before making changes:

    make ci  # Ensure everything is working

  2. After making changes:

    make fix      # Auto-fix formatting and linting
make test     # Run tests
make ci       # Final check before commit

  3. Before committing:

    make pre-commit  # Comprehensive pre-commit checks

Formatting Configuration

The project uses rustfmt.toml for consistent code formatting. The CI will automatically check formatting and provide helpful error messages if issues are found.

Project Structure

kafka-rs/
├── src/
│   ├── domain/          # Business logic layer
│   ├── application/     # Use case orchestration
│   ├── infrastructure/  # I/O and external systems
│   ├── lib.rs          # Library interface
│   └── main.rs         # Binary entry point
├── docs/               # Comprehensive documentation
├── tests/              # Rust integration tests
├── integration/        # External client integration tests
│   └── kafka-client-test/  # KafkaJS compatibility tests
└── .github/workflows/  # CI/CD pipeline

Contributing

This is an educational project, but contributions are welcome:

  1. Fork the repository
  2. Create a feature branch with descriptive name (e.g., feat/consumer-groups)
  3. Make your changes with tests and documentation
  4. Use conventional commits (e.g., feat:, fix:, chore:, docs:)
  5. Run make pre-commit to ensure code quality
  6. Submit a pull request with conventional commit title

Conventional Commit Examples:

Design Principles

📋 Supported Kafka APIs

API Key Status Description
Produce 0 ✅ Send messages to topics
Fetch 1 ✅ Consume messages from topics
Metadata 3 ✅ Discover topics and brokers
OffsetCommit 8 ✅ Commit consumer offsets
OffsetFetch 9 ✅ Fetch committed offsets

Limitations (Educational Simplifications)

📊 Performance

While not optimized for production use, the implementation can handle:

🔍 Monitoring & Debugging

# Enable debug logging
RUST_LOG=debug cargo run --bin kafka-rs

# Enable trace logging (verbose)
RUST_LOG=trace cargo run --bin kafka-rs

# JSON structured logging
RUST_LOG=info cargo run --bin kafka-rs 2>&1 | jq

🤝 Community & Learning

This project is part of learning Rust systems programming and distributed systems concepts:

📜 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

⭐ Star this repo if you find it useful for learning Kafka internals or Rust systems programming!

📚 Learning Resources

Built with ❤️ and 🦀 for educational purposes.

This site is open source. Improve this page.