agent-desktop

A local desktop automation tool for AI agents on macOS that targets accessibility controls and connects Chromium apps to browser automation tools.

agent-desktop lets developers give AI agents control of macOS apps such as Finder, Safari and Xcode. It identifies controls through the app's accessibility tree, so agents can act on buttons and text fields by their UI structure.

  • Checked interactions: Recheck targets before acting, reject missing or ambiguous matches, and verify that controls are visible, enabled and ready for the requested action.
  • Focused inspection: Request a compact overview of dense interfaces such as Slack or VS Code, then inspect the relevant area to reduce the UI text sent to a model.
  • Desktop controls: Click, type, capture screenshots, access the clipboard and manage apps and windows, with structured responses that include errors and recovery hints.
  • Chromium app access: Connect web content in Chromium apps to Playwright, Puppeteer or agent-browser through Chrome DevTools Protocol, while using accessibility controls for native menus and dialogs.
  • Session traces: Keep action traces locally and export an HTML timeline with screenshots; separate or shared sessions support multiple agents.

Accessibility-based actions block unintended focus, pointer, keyboard and clipboard side effects by default. Physical mouse and keyboard interaction is also available.

The Rust tool provides a standalone CLI without runtime dependencies and a library callable from languages including Python and Swift. It's open source under Apache 2.0. Desktop automation is supported on macOS and depends on apps exposing an accessibility tree.

Similar to agent-desktop