Vercel Labs open-sources agent-desktop for computer use by coding agents

Vercel Labs released agent-desktop, an open-source CLI and C library, written in Rust, that lets AI agents see and operate desktop applications on macOS, Windows, and Linux. Instead of taking screenshots and guessing where to click based on pixels, the tool reads an app's real UI structure through the operating system's accessibility tree and returns a JSON snapshot listing every clickable element with a stable reference ID. An agent acts on a reference, then takes a new snapshot to confirm the result, rather than relying on image matching that can break when a UI changes slightly.

The tool runs headless by default, meaning it can act through the accessibility API without moving the mouse or stealing focus from whatever the user is doing on screen; a separate flag allows physical input when needed. It ships 56 operational commands covering observation, clicking, typing, keyboard and mouse control, window management, notifications, and clipboard access, plus language bindings so programs can call it in-process instead of launching the CLI each time.

Chris Tate of Vercel Labs announced the project on X, describing it as similar to the company's existing agent-browser tool but for an entire computer rather than just a browser, and said it can run in the background while a person keeps working, with multiple agents each controlling their own cursor in parallel. The project is released under the Apache 2.0 license.

Why it matters

It gives coding agents a more reliable way to operate real desktop software instead of guessing from screenshots, which could unlock more automation beyond the browser.

Source: Vercel Labs on GitHub