Self-Operating Computer
Open-source AI computer-use framework for macOS, Windows and Linux, with LLaVA through Ollama or cloud models from OpenAI, Google and Anthropic.
Self-Operating Computer lets a vision-capable AI model control a desktop by reading the screen and choosing mouse and keyboard actions to carry out a goal. It's a Python framework for developers and researchers exploring AI agents that work through application interfaces. The code uses the MIT license.
The framework runs on macOS, Windows and Linux, with an X server required on Linux. On macOS, the terminal needs Screen Recording and Accessibility permissions. Desktop control happens on your machine; where the model processes screen input depends on the backend you choose. LLaVA runs locally through Ollama. Cloud models require provider API access and send model inputs to the selected service.
Supported models include GPT-4o, GPT-4.1 and o1, along with Gemini Pro Vision, Claude 3 and Qwen-VL. That choice lets you compare different models on computer-use tasks within the same framework. The local LLaVA route has very high error rates, so it's a base for experimentation rather than a dependable desktop assistant.
Beyond reading screenshots, the framework supports optical character recognition to help models select interface elements by their text. Its Set-of-Mark mode uses YOLOv8 button detection to mark visual targets, and developers can replace the detector with their own trained model. Voice input lets you speak the objective you want the agent to attempt.
