jalla: Universal LLM Middleware in Go

The Provider Fragmentation Problem

You start with OpenAI. GPT-4 works great. Then Claude drops and suddenly it's better at reasoning. Then Gemini 1.5 Pro arrives with a 2M token context window. Your code is now a mess of different client libraries, different request formats, different response shapes.

Every provider implements their own API. They're all roughly the same, but different enough to break your code.

We needed a universal adapter. One endpoint. One request format. Switch providers by changing a config flag.

Architecture: The Adapter Pattern at Scale

jalla is built around a simple interface:

type Provider interface {
    Complete(ctx context.Context, req CompletionRequest) (*CompletionResponse, error)
    StreamComplete(ctx context.Context, req CompletionRequest) (chan StreamChunk, error)
    Embed(ctx context.Context, req EmbeddingRequest) (*EmbeddingResponse, error)
}

Each provider implements this interface, translating between OpenAI's request/response format and their native APIs. The HTTP layer never knows which provider it's talking to.

Go Implementation

The core translation layer for Anthropic looks like this:

func (a *AnthropicAdapter) Complete(ctx context.Context, req CompletionRequest) (*CompletionResponse, error) {
    // Convert OpenAI messages to Anthropic format
    anthropicReq := anthropic.MessagesRequest{
        Model:     mapModel(req.Model),
        MaxTokens: req.MaxTokens,
        Messages:  convertMessages(req.Messages),
        System:    extractSystem(req.Messages),
    }
    
    resp, err := a.client.CreateMessages(ctx, anthropicReq)
    if err != nil {
        return nil, fmt.Errorf("anthropic api error: %w", err)
    }
    
    // Convert back to OpenAI format
    return &CompletionResponse{
        ID:      resp.ID,
        Object:  "chat.completion",
        Created: time.Now().Unix(),
        Model:   req.Model,
        Choices: []Choice{{
            Index: 0,
            Message: Message{
                Role:    "assistant",
                Content: resp.Content[0].Text,
            },
            FinishReason: string(resp.StopReason),
        }},
    }, nil
}

Why Go?

Go's concurrency model is perfect for this. Thousands of concurrent streams, each handled by a lightweight goroutine. No callback hell. No event loops. Just clean, blocking code that the runtime schedules efficiently.

The binary is a single static executable. No runtime dependencies. No Python environment hell. Deploy it anywhere.

Usage

# Point at jalla instead of OpenAI
export OPENAI_BASE_URL=http://localhost:8080/v1
export OPENAI_API_KEY=jalla-api-key

# Use any OpenAI-compatible client
curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "claude-3-5-sonnet",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

Production Hardening

Real deployments need more than just translation:

Rate Limiting: Token bucket per API key, per model, per IP
Retry Logic: Exponential backoff with jitter for provider failures
Circuit Breakers: Fail fast when a provider is down
Streaming: SSE streams that multiplex multiple provider connections

jalla handles all of this transparently. Your application code stays clean.

The Result

Switching from GPT-4 to Claude to Gemini is now a configuration change. A/B test models in production. Route high-context tasks to Gemini, reasoning tasks to Claude, cost-sensitive tasks to local models. One codebase, infinite flexibility.

Sometimes the most valuable code is the code that disappears.

ARTICLE

001