agent-desktop

agent-desktop lets AI agents inspect and operate desktop apps locally on macOS. It's for developers building agents that need to work in Finder, Safari, Xcode or other apps with an accessibility tree. It reads the controls an app exposes to the operating system, so agents can identify buttons and text fields without relying on screenshot coordinates.

The tool checks targets again before acting. It rejects missing or ambiguous elements rather than picking a likely match, and checks that controls are visible, enabled and ready for the requested action. By default, accessibility-based interactions block unintended changes to focus, the pointer, keyboard or clipboard. Physical mouse and keyboard interaction is also available.

For complex interfaces such as Slack and VS Code, agents can request a compact overview and inspect only the relevant region, reducing the UI text sent to a model. Beyond clicking and typing, it handles screenshots, clipboard access, notifications and app or window management. Structured JSON responses include errors and recovery hints.

Chromium apps such as Obsidian and Discord can expose their web content to Playwright, Puppeteer or agent-browser through Chrome DevTools Protocol. Native menus and dialogs remain accessible through the OS interface. Sessions support multiple agents and keep traces locally; an HTML export can include the action timeline and screenshots.

Written in Rust, it provides a standalone CLI without runtime dependencies and a library callable from languages including Python and Swift. It's open source under Apache 2.0. Desktop automation support is available on macOS.