Why Kafka Failed Us

Kafka is a masterpiece of engineering. It handles millions of messages, provides strong durability guarantees, and has an ecosystem to match. But it's designed for log storage, not real-time event distribution.

Our requirements were different:

• 10M+ events per second sustained throughput
• Sub-10ms end-to-end latency (p99)
• Dynamic fan-out: one event to thousands of consumers
• No persistence: events are ephemeral, loss is acceptable

We tried to bend Kafka to our will. We tweaked batch sizes, partition counts, consumer configurations. At 5M events/sec, latency spiked to 200ms. The JVM garbage collector was fighting us. We needed a different approach.

Design Principles

beamline is built on three core ideas:

1. Zero-copy networking: Events touch memory once, then hit the wire
2. Lock-free data structures: No mutex contention on the hot path
3. NUMA-aware allocation: Pin goroutines to cores, eliminate cache thrashing

The Architecture

A beamline cluster consists of brokers and routers. Brokers handle ingestion. Routers handle distribution. The separation allows independent scaling.

// Producer publishes to any broker
producer.Publish(event, routingKey)

// Broker stores in memory-mapped ring buffer
broker.ingest(event)

// Router subscribes to brokers, fans out to consumers
router.subscribe(pattern, consumerChan)

// Consumer receives filtered events
for event := range consumerChan {
    process(event)
}

Ring Buffers and Cache Lines

The heart of beamline is a lock-free ring buffer implementation optimized for x86 cache lines (64 bytes):

type RingBuffer struct {
    buffer   []Event
    writeIdx atomic.Uint64
    readIdx  atomic.Uint64
    mask     uint64
}

func (r *RingBuffer) Write(e Event) bool {
    idx := r.writeIdx.Add(1) - 1
    pos := idx & r.mask
    
    // Spin until slot is free (lock-free)
    for !r.buffer[pos].IsEmpty() {
        runtime.Gosched()
    }
    
    r.buffer[pos] = e
    return true
}

Each event is exactly 64 bytes, aligning perfectly with cache lines. False sharing eliminated. Contention reduced to atomic increments.

Networking: epoll on Steroids

Standard Go net/http couldn't keep up. We built on gnet—an event-driven networking framework using epoll on Linux and kqueue on BSD.

A single beamline broker handles 2M events/sec per core. With 16 cores, that's 32M events/sec per machine. We run 3 brokers for redundancy.

Latency Distribution

In production:

• p50 latency: 2.3ms
• p99 latency: 8.7ms
• p99.9 latency: 15ms

The p99.9 outliers are usually GC pauses in consumer applications, not beamline itself.

When to Use beamline

beamline is not a Kafka replacement. It makes tradeoffs:

Use Kafka when you need durability. Use beamline when you need speed and can tolerate loss.

Lessons Learned

Building high-performance systems in Go requires understanding the runtime. The garbage collector is your friend until it isn't. Avoid allocations on hot paths. Pool your objects. Profile relentlessly.

Sometimes the right tool doesn't exist. Build it.