Agent-S Documentation

Agent S: an open agentic framework that uses computers like a human

11,192 stars Python Python 1 file · ~6,033 tokens #agent-computer-interface#ai-agents#computer-automation#computer-use#computer-use-agent#cua#grounding#gui-agents
RAW Doc

1. Project Overview & Quickstart (simular-ai/Agent-S)

File: README.md

Agent S: Use Computer Like a Human

🏆 Agent S3: First to Surpass Human Performance on OSWorld (72.60%)

🥳 Updates

  • 2026/07/30: The Agent S3 paper "Scaling Agents for Computer Use" is accepted to TMLR 2026!
  • 2025/12/15: Agent S3 is the first to surpass human-level performance on OSWorld with an impressive score of 72.60%!
  • 2025/10/02: Released Agent S3 and its technical paper, setting a new SOTA of 69.9% on OSWorld (approaching 72% human performance), with strong generalizability on WindowsAgentArena and AndroidWorld! It is also simpler, faster, and more flexible.
  • 2025/08/01: Agent S2.5 is released (gui-agents v0.2.5): simpler, better, and faster! New SOTA on OSWorld-Verified!
  • 2025/07/07: The Agent S2 paper is accepted to COLM 2025! See you in Montreal!
  • 2025/04/27: The Agent S paper won the Best Paper Award 🏆 at ICLR 2025 Agentic AI for Science Workshop!
  • 2025/04/01: Released the Agent S2 paper with new SOTA results on OSWorld, WindowsAgentArena, and AndroidWorld!
  • 2025/03/12: Released Agent S2 along with v0.2.0 of gui-agents, the new state-of-the-art for computer use agents (CUA), outperforming OpenAI's CUA/Operator and Anthropic's Claude 3.7 Sonnet Computer-Use!
  • 2025/01/22: The Agent S paper is accepted to ICLR 2025!
  • 2025/01/21: Released v0.1.2 of gui-agents library, with support for Linux and Windows!
  • 2024/12/05: Released v0.1.0 of gui-agents library, allowing you to use Agent-S for Mac, OSWorld, and WindowsAgentArena with ease!
  • 2024/10/10: Released the Agent S paper and codebase!

Table of Contents

  1. 💡 Introduction
  2. 🎯 Current Results
  3. 🛠️ Installation & Setup
  4. 🚀 Usage
  5. 🤝 Acknowledgements
  6. 💬 Citation

💡 Introduction

Welcome to Agent S, an open-source framework designed to enable autonomous interaction with computers through Agent-Computer Interface. Our mission is to build intelligent GUI agents that can learn from past experiences and perform complex tasks autonomously on your computer.

Whether you're interested in AI, automation, or contributing to cutting-edge agent-based systems, we're excited to have you here!

🎯 Current Results

On OSWorld, Agent S3 alone reaches 66% in the 100-step setting, already exceeding the previous state of the art of 63.4% (GTA1 w/ GPT-5). With the addition of Behavior Best-of-N, performance climbs even higher to 72.6%, surpassing human-level performance on OSWorld (~72%)!

Agent S3 also demonstrates strong zero-shot generalization! On WindowsAgentArena, accuracy rises from 50.2% using only Agent S3 to 56.6% by selecting from 3 rollouts. Similarly on AndroidWorld, performance improves from 68.1% to 71.6%

🛠️ Installation & Setup

Prerequisites

  • Single Monitor: Our agent is designed for single monitor screens
  • Security: The agent runs Python code to control your computer - use with care
  • Supported Platforms: Linux, Mac, and Windows

Installation

To install Agent S3 without cloning the repository, run

bash
pip install gui-agents

If you would like to test Agent S3 while making changes, clone the repository and install using

text
pip install -e .

Don't forget to also brew install tesseract! Pytesseract requires this extra installation to work.

API Configuration

Option 1: Environment Variables

Add to your .bashrc (Linux) or .zshrc (MacOS):

bash
export OPENAI_API_KEY=<YOUR_API_KEY>
export ANTHROPIC_API_KEY=<YOUR_ANTHROPIC_API_KEY>
export HF_TOKEN=<YOUR_HF_TOKEN>

Option 2: Python Script

python
import os
os.environ["OPENAI_API_KEY"] = "<YOUR_API_KEY>"

Supported Models

We support Azure OpenAI, Anthropic, Gemini, Open Router, and vLLM inference. See models.md for details.

Grounding Models (Required)

For optimal performance, we recommend UI-TARS-1.5-7B hosted on Hugging Face Inference Endpoints or another provider. See Hugging Face Inference Endpoints for setup instructions.

🚀 Usage

⚡️ Recommended Setup:
For the best configuration, we recommend using OpenAI gpt-5-2025-08-07 as the main model, paired with UI-TARS-1.5-7B for grounding.

CLI

Note, this is running Agent S3, our improved agent, without bBoN.

Run Agent S3 with the required parameters:

bash
agent_s \
    --provider openai \
    --model gpt-5-2025-08-07 \
    --ground_provider huggingface \
    --ground_url http://localhost:8080 \
    --ground_model ui-tars-1.5-7b \
    --grounding_width 1920 \
    --grounding_height 1080

Local Coding Environment (Optional)

For tasks that require code execution (e.g., data processing, file manipulation, system automation), you can enable the local coding environment:

bash
agent_s \
    --provider openai \
    --model gpt-5-2025-08-07 \
    --ground_provider huggingface \
    --ground_url http://localhost:8080 \
    --ground_model ui-tars-1.5-7b \
    --grounding_width 1920 \
    --grounding_height 1080 \
    --enable_local_env

⚠️ WARNING: The local coding environment executes arbitrary Python and Bash code locally on your machine. Only use this feature in trusted environments and with trusted inputs.

Required Parameters

  • --provider: Main generation model provider (e.g., openai, anthropic, etc.) - Default: "openai"
  • --model: Main generation model name (e.g., gpt-5-2025-08-07) - Default: "gpt-5-2025-08-07"
  • --ground_provider: The provider for the grounding model - Required
  • --ground_url: The URL of the grounding model - Required
  • --ground_model: The model name for the grounding model - Required
  • --grounding_width: Width of the output coordinate resolution from the grounding model - Required
  • --grounding_height: Height of the output coordinate resolution from the grounding model - Required

Optional Parameters

  • --model_temperature: The temperature to fix all model calls to (necessary to set to 1.0 for models like o3 but can be left blank for other models)

Grounding Model Dimensions

The grounding width and height should match the output coordinate resolution of your grounding model:

  • UI-TARS-1.5-7B: Use --grounding_width 1920 --grounding_height 1080
  • UI-TARS-72B: Use --grounding_width 1000 --grounding_height 1000

Optional Parameters

  • --model_url: Custom API URL for main generation model - Default: ""
  • --model_api_key: API key for main generation model - Default: ""
  • --ground_api_key: API key for grounding model endpoint - Default: ""
  • --max_trajectory_length: Maximum number of image turns to keep in trajectory - Default: 8
  • --enable_reflection: Enable reflection agent to assist the worker agent - Default: True
  • --enable_local_env: Enable local coding environment for code execution (WARNING: Executes arbitrary code locally) - Default: False

Local Coding Environment Details

The local coding environment enables Agent S3 to execute Python and Bash code directly on your machine. This is particularly useful for:

  • Data Processing: Manipulating spreadsheets, CSV files, or databases
  • File Operations: Bulk file processing, content extraction, or file organization
  • System Automation: Configuration changes, system setup, or automation scripts
  • Code Development: Writing, editing, or executing code files
  • Text Processing: Document manipulation, content editing, or formatting

When enabled, the agent can use the call_code_agent action to execute code blocks for tasks that can be completed through programming rather than GUI interaction.

Requirements:

  • Python: The same Python interpreter used to run Agent S3 (automatically detected)
  • Bash: Available at /bin/bash (standard on macOS and Linux)
  • System Permissions: The agent runs with the same permissions as the user executing it

Security Considerations:

  • The local environment executes arbitrary code with the same permissions as the user running the agent
  • Only enable this feature in trusted environments
  • Be cautious when the agent generates code for system-level operations
  • Consider running in a sandboxed environment for untrusted tasks
  • Bash scripts are executed with a 30-second timeout to prevent hanging processes

`gui_agents` SDK

First, we import the necessary modules. AgentS3 is the main agent class for Agent S3. OSWorldACI is our grounding agent that translates agent actions into executable python code.

python
import pyautogui
import io
from gui_agents.s3.agents.agent_s import AgentS3
from gui_agents.s3.agents.grounding import OSWorldACI
from gui_agents.s3.utils.local_env import LocalEnv  # Optional: for local coding environment

# Load in your API keys.
from dotenv import load_dotenv
load_dotenv()

current_platform = "linux"  # "darwin", "windows"

Next, we define our engine parameters. engine_params is used for the main agent, and engine_params_for_grounding is for grounding. For engine_params_for_grounding, we support custom endpoints like HuggingFace TGI, vLLM, and Open Router.

python
engine_params = {
  "engine_type": provider,
  "model": model,
  "base_url": model_url,           # Optional
  "api_key": model_api_key,        # Optional
  "temperature": model_temperature # Optional
}

# Load the grounding engine from a custom endpoint
ground_provider = "<your_ground_provider>"
ground_url = "<your_ground_url>"
ground_model = "<your_ground_model>"
ground_api_key = "<your_ground_api_key>"

# Set grounding dimensions based on your model's output coordinate resolution
# UI-TARS-1.5-7B: grounding_width=1920, grounding_height=1080
# UI-TARS-72B: grounding_width=1000, grounding_height=1000
grounding_width = 1920  # Width of output coordinate resolution
grounding_height = 1080  # Height of output coordinate resolution

engine_params_for_grounding = {
  "engine_type": ground_provider,
  "model": ground_model,
  "base_url": ground_url,
  "api_key": ground_api_key,  # Optional
  "grounding_width": grounding_width,
  "grounding_height": grounding_height,
}

Then, we define our grounding agent and Agent S3.

python
# Optional: Enable local coding environment
enable_local_env = False  # Set to True to enable local code execution
local_env = LocalEnv() if enable_local_env else None

grounding_agent = OSWorldACI(
    env=local_env,  # Pass local_env for code execution capability
    platform=current_platform,
    engine_params_for_generation=engine_params,
    engine_params_for_grounding=engine_params_for_grounding,
    width=1920,  # Optional: screen width
    height=1080  # Optional: screen height
)

agent = AgentS3(
    engine_params,
    grounding_agent,
    platform=current_platform,
    max_trajectory_length=8,  # Optional: maximum image turns to keep
    enable_reflection=True     # Optional: enable reflection agent
)

Finally, let's query the agent!

python
# Get screenshot.
screenshot = pyautogui.screenshot()
buffered = io.BytesIO() 
screenshot.save(buffered, format="PNG")
screenshot_bytes = buffered.getvalue()

obs = {
  "screenshot": screenshot_bytes,
}

instruction = "Close VS Code"
info, action = agent.predict(instruction=instruction, observation=obs)

exec(action[0])

Refer to gui_agents/s3/cli_app.py for more details on how the inference loop works.

OSWorld

To deploy Agent S3 in OSWorld, follow the OSWorld Deployment instructions.

💬 Citations

If you find this codebase useful, please cite:

text
@misc{Agent-S3,
      title={The Unreasonable Effectiveness of Scaling Agents for Computer Use}, 
      author={Gonzalo Gonzalez-Pumariega and Vincent Tu and Chih-Lun Lee and Jiachen Yang and Ang Li and Xin Eric Wang},
      year={2025},
      eprint={2510.02250},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2510.02250}, 
}

@misc{Agent-S2,
      title={Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents}, 
      author={Saaket Agashe and Kyle Wong and Vincent Tu and Jiachen Yang and Ang Li and Xin Eric Wang},
      year={2025},
      eprint={2504.00906},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2504.00906}, 
}

@inproceedings{Agent-S,
    title={{Agent S: An Open Agentic Framework that Uses Computers Like a Human}},
    author={Saaket Agashe and Jiuzhou Han and Shuyu Gan and Jiachen Yang and Ang Li and Xin Eric Wang},
    booktitle={International Conference on Learning Representations (ICLR)},
    year={2025},
    url={https://arxiv.org/abs/2410.08164}
}

Star History

File: gui_agents/s1/README.md

Agent S: Using Computers Like a Human

🥳 Updates

  • 2025/01/22: The Agent S paper is accepted to ICLR 2025!
  • 2025/01/21: Released v0.1.2 of gui-agents library, with support for Linux and Windows!
  • 2024/12/05: Released v0.1.0 of gui-agents library, allowing you to use Agent-S for Mac, OSWorld, and WindowsAgentArena with ease!
  • 2024/10/10: Released Agent S paper and codebase!

Table of Contents

  1. 💡 Introduction
  2. 🎯 Current Results
  3. 🛠️ Installation
  4. 🚀 Usage
  5. 🙌 Contributors
  6. 💬 Citation

💡 Introduction

Welcome to Agent S, an open-source framework designed to enable autonomous interaction with computers through Agent-Computer Interface. Our mission is to build intelligent GUI agents that can learn from past experiences and perform complex tasks autonomously on your computer.

Whether you're interested in AI, automation, or contributing to cutting-edge agent-based systems, we're excited to have you here!

🎯 Current Results

🛠️ Installation & Setup

❗Warning❗: If you are on a Linux machine, creating a conda environment will interfere with pyatspi. As of now, there's no clean solution for this issue. Proceed through the installation without using conda or any virtual environment.

Clone the repository:

text
git clone https://github.com/simular-ai/Agent-S.git

Install the gui-agents package:

text
pip install gui-agents

Set your LLM API Keys and other environment variables. You can do this by adding the following line to your .bashrc (Linux), or .zshrc (MacOS) file.

text
export OPENAI_API_KEY=<YOUR_API_KEY>

Alternatively, you can set the environment variable in your Python script:

text
import os
os.environ["OPENAI_API_KEY"] = "<YOUR_API_KEY>"

We also support Azure OpenAI, Anthropic, and vLLM inference. For more information refer to ../../models.md.

Setup Retrieval from Web using Perplexica

Agent S works best with web-knowledge retrieval. To enable this feature, you need to setup Perplexica:

  1. Ensure Docker Desktop is installed and running on your system.

  2. Navigate to the directory containing the project files.

    bash
    cd Perplexica
 git submodule update --init

  3. Rename the sample.config.toml file to config.toml. For Docker setups, you need only fill in the following fields:

    • OPENAI: Your OpenAI API key. You only need to fill this if you wish to use OpenAI's models.

    • OLLAMA: Your Ollama API URL. You should enter it as http://host.docker.internal:PORT_NUMBER. If you installed Ollama on port 11434, use http://host.docker.internal:11434. For other ports, adjust accordingly. You need to fill this if you wish to use Ollama's models instead of OpenAI's.

    • GROQ: Your Groq API key. You only need to fill this if you wish to use Groq's hosted models.

    • ANTHROPIC: Your Anthropic API key. You only need to fill this if you wish to use Anthropic models.

      Note: You can change these after starting Perplexica from the settings dialog.

    • SIMILARITY_MEASURE: The similarity measure to use (This is filled by default; you can leave it as is if you are unsure about it.)

  4. Ensure you are in the directory containing the docker-compose.yaml file and execute:

    bash
    docker compose up -d

  5. Next, export your Perplexica URL. This URL is used to interact with the Perplexica API backend. The port is given by the config.toml in your Perplexica directory.

    bash
    export PERPLEXICA_URL=http://localhost:{port}/api/search

  6. Our implementation of Agent S incorporates the Perplexica API to integrate a search engine capability, which allows for a more convenient and responsive user experience. If you want to tailor the API to your settings and specific requirements, you may modify the URL and the message of request parameters in agent_s/query_perplexica.py. For a comprehensive guide on configuring the Perplexica API, please refer to Perplexica Search API Documentation

For a more detailed setup and usage guide, please refer to the Perplexica Repository.

Setup Paddle-OCR Server

Switch to a new terminal where you will run Agent S. Set the OCR_SERVER_ADDRESS environment variable as shown below. For a better experience, add the following line directly to your .bashrc (Linux), or .zshrc (MacOS) file.

text
export OCR_SERVER_ADDRESS=http://localhost:8000/ocr/

Run the ocr_server.py file code to use OCR-based bounding boxes.

text
cd Agent-S
python gui_agents/utils/ocr_server.py

You can change the server address by editing the address in gui_agents/s1/utils/ocr_server.py file.

❗Warning❗: The agent will directly run python code to control your computer. Please use with care.

🚀 Usage

CLI

Run agent_s on your computer using:

text
agent_s1 --model gpt-4o

This will show a user query prompt where you can enter your query and interact with Agent S. You can use any model from the list of supported models in models.md.

`gui_agents` SDK

To deploy Agent S on MacOS or Windows:

text
import pyautogui
import io
from gui_agents.core.AgentS import GraphSearchAgent
import platform

if platform.system() == "Darwin":
  from gui_agents.aci.MacOSACI import MacOSACI, UIElement
  grounding_agent = MacOSACI()
elif platform.system() == "Windows":
  from gui_agents.aci.WindowsOSACI import WindowsACI, UIElement
  grounding_agent = WindowsACI()
elif platform.system() == "Linux":
  from gui_agents.aci.LinuxOSACI import LinuxACI, UIElement
  grounding_agent = LinuxACI()
else:
  raise ValueError("Unsupported platform")

engine_params = {
    "engine_type": "openai",
    "model": "gpt-4o",
}

agent = GraphSearchAgent(
  engine_params,
  grounding_agent,
  platform="ubuntu",  # "macos", "windows"
  action_space="pyautogui",
  observation_type="mixed",
  search_engine="Perplexica"
)

# Get screenshot.
screenshot = pyautogui.screenshot()
buffered = io.BytesIO() 
screenshot.save(buffered, format="PNG")
screenshot_bytes = buffered.getvalue()

# Get accessibility tree.
acc_tree = UIElement.systemWideElement()

obs = {
  "screenshot": screenshot_bytes,
  "accessibility_tree": acc_tree,
}

instruction = "Close VS Code"
info, action = agent.predict(instruction=instruction, observation=obs)

exec(action[0])

Refer to cli_app.py for more details on how the inference loop works.

Downloading the Knowledege Base

Agent S2 uses a knowledge base that continually updates with new knowledge during inference. The knowledge base is initially downloaded when initializing GraphSearchAgent. The knowledge base is stored as assets under our GitHub Releases. The GraphSearchAgent initialization will only download the knowledge base for your specified platform and agent version (e.g s1, s2). If you'd like to download the knowledge base programmatically, you can use the following code:

text
download_kb_data(
    version="s2",
    release_tag="v0.2.2",
    download_dir="kb_data",
    platform="linux"  # "darwin", "windows"
)

This will download Agent S2's knowledge base for Linux from release tag v0.2.2 to the kb_data directory. Refer to our GitHub Releases or release tags that include the knowledge bases.

OSWorld

To deploy Agent S in OSWorld, follow the OSWorld Deployment instructions.

WindowsAgentArena

To deploy Agent S in WindowsAgentArena, follow the WindowsAgentArena Deployment instructions.

🙌 Contributors

We’re grateful to all the amazing people who have contributed to this project. Thank you! 🙏

💬 Citation

text
@misc{agashe2024agentsopenagentic,
      title={Agent S: An Open Agentic Framework that Uses Computers Like a Human}, 
      author={Saaket Agashe and Jiuzhou Han and Shuyu Gan and Jiachen Yang and Ang Li and Xin Eric Wang},
      year={2024},
      eprint={2410.08164},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2410.08164}, 
}

2. Official Technical Reference & Guides (simular-ai/simular-ai.github.io)

File: docs/API/SaveScreenshot.md

title: SaveScreenshot parent: API

API documentation

SaveScreenshot(element:fileName:directory)

Takes a screenshot of an element on the screen or the whole screen, and saves the screenshot as a PNG to a file.

swift
func SaveScreenshot(
        element: UIElement? = nil,
        fileName: String? = nil,
        directory: String? = nil
    )

Parameters

element

If provided, limit the screenshot to the frame of the element. Default is the whole screen.

fileName

Name for the image name, default is “simularSavedImage.png”.

directory

Directory to save the image, default at desktop.

Return Value

None

Discussion

Examples:

  • Instruction: Save screenshot of whole screen to screenshot.png
swift
SaveScreenshot(fileName: "screenshot.png")
  • Instruction: Screenshot the group element and save to clipboard
swift
SaveScreenshot(element: group)
  • Instruction: Screenshot the table element and save to /User/path/to/screenshot/
swift
SaveScreenshot(element: table, directory: "/User/path/to/screenshot/")

File: docs/API/ScreenshotToClipboard.md

ScreenshotToClipboard(element)

Take a screenshot of an element or the current page and save it to the system clipboard

swift
func ScreenshotToClipboard(element: UIElement? = nil)

Parameters

element

If provided, limit the screenshot to the frame of the element. Default is the whole screen.

Return Value

None

Discussion

Examples:

  • Instruction: take screenshot and save to clipboard
swift
ScreenshotToClipboard()
  • Instruction: get the table element on the page, save a screenshot of it to clipboard
swift
var elements = GetElements(elementRoles: ["table"])
ScreenshotToClipboard(element: elements[0])

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#agent-computer-interface Browse agent-computer-interface catalog → #ai-agents Browse ai-agents catalog → #computer-automation Browse computer-automation catalog → #computer-use Browse computer-use catalog → #computer-use-agent Browse computer-use-agent catalog → #cua Browse cua catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →