gut: Building Gomputer Use Tooling

The Computer Use Revolution

LLMs can write code, but can they use software? The computer use paradigm—giving AI agents the ability to see screens and interact with UIs—opens entirely new capabilities. Customer support that actually clicks through your app. Automated testing that navigates like a real user. Data entry bots that work with legacy systems.

Computer use isn't about replacing human judgment. It's about extending AI from the text domain into the visual, interactive world.

Most computer use implementations are Python-based, resource-heavy, and hard to deploy. We built gut in Go for production environments.

Core Abstractions

gut provides three primitives:

// Computer represents a virtual desktop
type Computer interface {
    Screenshot(ctx context.Context) (*Screenshot, error)
    Click(x, y int) error
    Type(text string) error
    Key(key string) error
    Scroll(direction Direction, amount int) error
}

// Agent reasons about actions
type Agent interface {
    Observe(ctx context.Context, screenshot *Screenshot) (*Action, error)
    Learn(experience Experience) error
}

// Task is a high-level goal
type Task interface {
    Done(screenshot *Screenshot) (bool, error)
    Reward() float64
}

Vision: Beyond OCR

Early computer use systems relied on OCR—extracting text from screenshots and reasoning about positions. This fails with icons, diagrams, and modern web apps where text is often SVG.

gut uses a vision-language model (VLM) that processes screenshots directly. The model outputs structured JSON describing UI elements and their locations:

{
  "elements": [
    {
      "type": "button",
      "text": "Submit",
      "bbox": [120, 340, 200, 380],
      "confidence": 0.97
    },
    {
      "type": "input",
      "placeholder": "Email address",
      "bbox": [120, 280, 400, 320],
      "confidence": 0.94
    }
  ]
}

Execution Loop

The agent runs a simple loop:

Capture screenshot
VLM processes image → UI elements
LLM reasons about goal → action
Execute action on computer
Check if task is complete
Repeat

func (a *Agent) Run(ctx context.Context, task Task) error {
    for {
        screenshot, _ := a.computer.Screenshot(ctx)
        
        if done, _ := task.Done(screenshot); done {
            return nil
        }
        
        action, _ := a.Observe(ctx, screenshot)
        if err := action.Execute(a.computer); err != nil {
            a.Learn(Experience{Action: action, Error: err})
        }
        
        time.Sleep(a.pollInterval)
    }
}

Resilience Strategies

Computer use is brittle. A button moves by 5 pixels and your script breaks. gut implements several resilience patterns:

Visual anchors: Find reference points that rarely change
Retry with backoff: Elements might not be ready immediately
Action verification: Screenshot after action to confirm effect
Failure recovery: If stuck, try alternative approaches

Deployment Models

gut supports multiple deployment targets:

Docker: Headless Chromium in container
VM: Full desktop environment
Browser Extension: Agent runs in user's actual browser
SaaS: Managed browser farms

Security Considerations

Running AI agents that can click and type is a security nightmare. gut implements strict sandboxing:

Network isolation (no external calls)
Filesystem restrictions (read-only or tmpfs)
Rate limiting on sensitive operations
Audit logging of all actions

The Future

Computer use is in its infancy. Today's agents can navigate simple UIs. Tomorrow they'll handle complex multi-step workflows, adapt to layout changes, and collaborate with human operators. gut is our bet on that future—fast, reliable, and deployable at scale.

ARTICLE

001