Technical Documentation: microsoft/UFO

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (README.md) Copy Section

**UFO³** : Weaving the Digital Agent Galaxy

🎯 Choose Your Path Copy Chapter

| ### UFO³ Multi-Device Agent Galaxy
✨ NEW & RECOMMENDED

Perfect for:

• 🔗 Cross-device collaboration workflows
• 📊 Complex multi-step automation
• 🎯 DAG-based task orchestration
• 🌍 Heterogeneous platform integration

Key Features:

• Constellation: Task decomposition into executable DAGs
• Dynamic DAG editing for adaptive workflow evolution
• Asynchronous execution with parallel task coordination
• Unified AIP protocol for secure agent communication

📖 Galaxy Documentation →
📖 Galaxy Quick Start → ⭐ Online Docs | ### UFO² Desktop AgentOS
STABLE & BATTLE-TESTED

Perfect for:

• 💻 Single Windows automation
• ⚡ Quick task execution
• 🎓 Learning agent basics
• 🛠️ Simple workflows

Key Features:

• Deep Windows OS integration
• Hybrid GUI + API actions
• Proven reliability
• Easy setup
• Can serve as Galaxy device agent

📖 UFO² Documentation → |
| --- | --- |

🎬 See UFO³ Galaxy in Action Copy Chapter

Watch how UFO³ Galaxy orchestrates complex workflows across multiple devices:

🌟 What's New in UFO³? Copy Chapter

Evolution Timeline Copy Section

%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#E8F4F8','primaryTextColor':'#1A1A1A','primaryBorderColor':'#7CB9E8','lineColor':'#A8D5E2','secondaryColor':'#B8E6F0','tertiaryColor':'#D4F1F4','fontSize':'16px','fontFamily':'Segoe UI, Arial, sans-serif'}}}%%
graph LR
    A["<b>🎈 UFO</b><br/><span style='font-size:14px'>February 2024</span><br/><span style='font-size:13px; color:#666'><i>GUI Agent for Windows</i></span>"] 
    B["<b>🖥️ UFO²</b><br/><span style='font-size:14px'>April 2025</span><br/><span style='font-size:13px; color:#666'><i>Desktop AgentOS</i></span>"]
    C["<b>🌌 UFO³ Galaxy</b><br/><span style='font-size:14px'>November 2025</span><br/><span style='font-size:13px; color:#666'><i>Multi-Device Orchestration</i></span>"]
    
    A -->|Evolve| B
    B -->|Scale| C
    
    style A fill:#E8F4F8,stroke:#7CB9E8,stroke-width:2.5px,color:#1A1A1A,rx:15,ry:15
    style B fill:#C5E8F5,stroke:#5BA8D0,stroke-width:2.5px,color:#1A1A1A,rx:15,ry:15
    style C fill:#A4DBF0,stroke:#3D96BE,stroke-width:2.5px,color:#1A1A1A,rx:15,ry:15

🚀 UFO³ = **Galaxy** (Multi-Device Orchestration) + **UFO²** (Device Agent) Copy Section

UFO³ introduces Galaxy, a revolutionary multi-device orchestration framework that coordinates intelligent agents across heterogeneous platforms. Built on five tightly integrated design principles:

1. 🌟 Declarative Decomposition into Dynamic DAG - Requests decomposed into structured DAG with TaskStars and dependencies for automated scheduling and runtime rewriting

2. 🔄 Continuous Result-Driven Graph Evolution - Living constellation that adapts to execution feedback through controlled rewrites and dynamic adjustments

3. ⚡ Heterogeneous, Asynchronous & Safe Orchestration - Capability-based device matching with async execution, safe locking, and formally verified correctness

4. 🔌 Unified Agent Interaction Protocol (AIP) - WebSocket-based secure coordination layer with fault tolerance and automatic reconnection

5. 🛠️ Template-Driven MCP-Empowered Device Agents - Lightweight toolkit for rapid agent development with MCP integration for tool augmentation

🎓 Migration Path Copy Section

For UFO² Users:

1. ✅ Keep using UFO² – Fully supported, actively maintained
2. 🔄 Gradual adoption – Galaxy can use UFO² as Windows device agent
3. 📈 Scale up – Move to Galaxy when you need multi-device capabilities
4. 📚 Learning resources – Migration Guide

✨ Capabilities at a Glance Copy Chapter

🌌 Galaxy Framework – What's Different? Copy Section

| #### 🌟 Constellation Planning

User Request
     ↓
ConstellationAgent
     ↓
  [Task DAG]
   /   |   \
Task1 Task2 Task3
(Win) (Linux)(Mac)

Benefits:

• Cross-device dependency tracking
• Parallel execution optimization
• Cross-device dataflow management | #### 🎯 Device Assignment

Selection Criteria
  • Platform
  • Resource
  • Task requirements
  • Performance history
        ↓
  Auto-Assignment
        ↓
  Optimal Devices

Smart Matching:

• Capability-based selection
• Real-time resource monitoring
• Dynamic reallocation | #### 📊 Orchestration

Task1 → Running  ✅
Task2 → Pending  ⏸️
Task3 → Running  🔄
        ↓
   Completion
        ↓
   Final Report

Orchestration:

• Real-time status updates
• Automatic error recovery
• Progress tracking with feedback |
  | --- | --- | --- |

🪟 UFO² Desktop AgentOS – Core Strengths Copy Section

UFO² serves dual roles: standalone Windows automation and Galaxy device agent for Windows platforms.

As Galaxy Device Agent:

• Receives tasks from ConstellationAgent via Galaxy orchestration layer
• Executes Windows-specific operations using proven UFO² capabilities
• Reports status and results back to TaskOrchestrator
• Participates in cross-device workflows seamlessly

🚀 Quick Start Guide Copy Chapter

Choose your path and follow the detailed setup guide:

| ### 🌌 Galaxy Quick Start

For cross-device orchestration

# 1. Install
pip install -r requirements.txt

# 2. Configure ConstellationAgent
copy config\galaxy\agent.yaml.template config\galaxy\agent.yaml
# Edit and add your API keys

# 3. Configure devices
# Edit config\galaxy\devices.yaml to register your devices

# 4. Start device agents (with platform flags)
# Windows: Start server + client
# Linux: Start server + MCP servers + client  
# Mobile (Android): Start server + MCP servers + client
# See platform-specific guides for detailed setup

# 5. Launch Galaxy
python -m galaxy --interactive

📖 Complete Guide:

• Galaxy README – Architecture & concepts
• Online Quick Start – Step-by-step tutorial
• Windows Device Setup
• Linux Device Setup
• Mobile Device Setup – Android agent setup
• Configuration – Device pool configuration | ### 🪟 UFO² Quick Start

For Windows automation

# 1. Install
pip install -r requirements.txt

# 2. Configure
copy config\ufo\agents.yaml.template config\ufo\agents.yaml
# Edit and add your API keys

# 3. Run
python -m ufo --task <task_name>

📖 Complete Guide:

• UFO² README – Full documentation
• Configuration Guide – LLM setup
• Advanced Features – Multi-action, RAG |
  | --- | --- |

📋 Common Configuration Copy Section

Both frameworks require LLM API configuration. Choose your provider:

OpenAI Configuration

For Galaxy (config/galaxy/agent.yaml):

CONSTELLATION_AGENT:
  REASONING_MODEL: false
  API_TYPE: "openai"
  API_BASE: "https://api.openai.com/v1/chat/completions"
  API_KEY: "sk-your-key-here"
  API_MODEL: "gpt-4o"

For UFO² (config/ufo/agents.yaml):

VISUAL_MODE: True
API_TYPE: "openai"
API_BASE: "https://api.openai.com/v1/chat/completions"
API_KEY: "sk-your-key-here"
API_MODEL: "gpt-4o"

Azure OpenAI Configuration

For Galaxy (config/galaxy/agent.yaml):

CONSTELLATION_AGENT:
  REASONING_MODEL: false
  API_TYPE: "aoai"
  API_BASE: "https://YOUR-RESOURCE.openai.azure.com"
  API_KEY: "your-azure-key"
  API_MODEL: "gpt-4o"
  API_DEPLOYMENT_ID: "your-deployment-id"

For UFO² (config/ufo/agents.yaml):

VISUAL_MODE: True
API_TYPE: "aoai"
API_BASE: "https://YOUR-RESOURCE.openai.azure.com"
API_KEY: "your-azure-key"
API_MODEL: "gpt-4o"
API_DEPLOYMENT_ID: "your-deployment-id"

💡 More LLM Options: See Model Configuration Guide for Qwen, Gemini, Claude, and more.

📚 Documentation Structure Copy Chapter

| ### 🌌 Galaxy Documentation

• Galaxy Framework Overview ⭐ Start Here – Architecture & technical concepts
• Quick Start Tutorial – Get running in minutes
• Galaxy Client – Device coordination and API
• Constellation Agent – Task decomposition and planning
• Task Orchestrator – Execution engine
• Task Constellation – DAG structure
• Agent Registration – Device registry
• Configuration Guide – Setup and device pools

📖 Technical Documentation:

• AIP Protocol – WebSocket messaging

• Session Management – Session lifecycle

• Visualization – Real-time monitoring

• Events & Observers – Event system | ### 🪟 UFO² Documentation

• UFO² Overview – Desktop AgentOS architecture

• Installation – Setup & dependencies

• Configuration – LLM & RAG setup

• Usage Guide – Running UFO²

• Advanced Features – Multi-action, RAG, etc.

• Automator Guide – Hybrid GUI + API

• Benchmarks – WAA & OSWorld results

📖 Online Docs:

• Complete Documentation
• Model Support
• RAG Configuration |
  | --- | --- |

📢 Latest Updates Copy Chapter

2025-11 – UFO³ Galaxy Framework Released 🌌 Copy Section

Major Research Breakthrough: Multi-Device Orchestration System

• 🌟 Declarative DAG Decomposition: TaskConstellation structure for workflow logic and dependencies
• 🔄 Dynamic Graph Evolution: Living constellation that adapts through controlled rewrites
• 🎯 Heterogeneous Orchestration: Safe, asynchronous execution with capability-based device matching
• 🔌 Unified AIP Protocol: WebSocket-based secure agent coordination with fault tolerance
• 🛠️ MCP-Empowered Agent Framework: Template-driven toolkit for rapid device agent development
• 📄 Research Paper: UFO³: Weaving the Digital Agent Galaxy

Key Features:

• First multi-device orchestration framework for GUI agents
• Result-driven adaptive execution instead of rigid workflows
• Model Context Protocol (MCP) integration for tool augmentation
• Formally verified correctness and concurrency safety guarantees

2025-04 – UFO² v2.0.0 Copy Section

• 📅 UFO² Desktop AgentOS released
• 🏗️ Enhanced architecture with AgentOS concept
• 📄 Technical Report published
• ✅ Entered Long-Term Support (LTS) status

2024-02 – Original UFO Copy Section

• 🎈 First UFO release - UI-Focused agent for Windows
• 📄 Original Paper
• 🌍 Wide media coverage and adoption

📚 Citation Copy Chapter

If you use UFO³ Galaxy or UFO² in your research, please cite the relevant papers:

UFO³ Galaxy Framework (2025) Copy Section

@article{zhang2025ufo3,
  title={UFO$^3$: Weaving the Digital Agent Galaxy}, 
  author = {Zhang, Chaoyun and Li, Liqun and Huang, He and Ni, Chiming and Qiao, Bo and Qin, Si and Kang, Yu and Ma, Minghua and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei},
  journal = {arXiv preprint arXiv:2511.11332},
  year    = {2025},
}

UFO² Desktop AgentOS (2025) Copy Section

@article{zhang2025ufo2,
  title   = {{UFO2: The Desktop AgentOS}},
  author  = {Zhang, Chaoyun and Huang, He and Ni, Chiming and Mu, Jian and Qin, Si and He, Shilin and Wang, Lu and Yang, Fangkai and Zhao, Pu and Du, Chao and Li, Liqun and Kang, Yu and Jiang, Zhao and Zheng, Suzhen and Wang, Rujia and Qian, Jiaxu and Ma, Minghua and Lou, Jian-Guang and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei},
  journal = {arXiv preprint arXiv:2504.14603},
  year    = {2025}
}

Original UFO (2024) Copy Section

@article{zhang2024ufo,
  title   = {{UFO: A UI-Focused Agent for Windows OS Interaction}},
  author  = {Zhang, Chaoyun and Li, Liqun and He, Shilin and Zhang, Xu and Qiao, Bo and Qin, Si and Ma, Minghua and Kang, Yu and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei and Zhang, Qi},
  journal = {arXiv preprint arXiv:2402.07939},
  year    = {2024}
}

🌐 Media & Community Copy Chapter

Media Coverage:

• 微软正式开源UFO²，Windows桌面迈入「AgentOS 时代」
• Microsoft's UFO: Smarter Windows Experience
• 下一代Windows系统曝光
• More coverage →

Community:

• 💬 GitHub Discussions
• 🐛 Issue Tracker
• 📧 Email: [email protected]
• 📺 YouTube Channel

🎨 Related Projects & Research Copy Chapter

Microsoft Research:

• TaskWeaver – Code-first LLM agent framework for data analytics and task automation

GUI Agent Research:

• LLM-Brained GUI Agents Survey – Comprehensive survey of GUI automation agents
• Interactive Survey Site – Explore latest GUI agent research and developments

Multi-Agent Systems:

• UFO³ Galaxy represents a novel approach to multi-device orchestration, introducing the Constellation framework for coordinating heterogeneous agents across platforms
• Builds on multi-agent coordination research while addressing unique challenges of cross-device GUI automation

Benchmarks:

• Windows Agent Arena (WAA) – Evaluation benchmark for Windows automation agents
• OSWorld – Cross-application task evaluation suite

💡 FAQ Copy Chapter

🤔 Should I use Galaxy or UFO²?

Start with UFO² if:

• You only need Windows automation
• You want quick setup and learning
• Tasks are relatively simple

Choose Galaxy if:

• You need cross-device coordination
• Tasks are complex and multi-step
• You want advanced orchestration
• You're comfortable with active development

Hybrid approach if:

• You want best of both worlds
• Some tasks are simple (UFO²), some complex (Galaxy)
• You're gradually migrating

⚠️ Will UFO² be deprecated?

No! UFO² has entered Long-Term Support (LTS) status:

• ✅ Actively maintained
• ✅ Bug fixes and security updates
• ✅ Performance improvements
• ✅ Full community support
• ✅ No plans for deprecation

UFO² is the stable, proven solution for Windows automation.

🔄 How do I migrate from UFO² to Galaxy?

Migration is gradual and optional:

1. Phase 1: Learn – Understand Galaxy concepts
2. Phase 2: Experiment – Try Galaxy with non-critical tasks
3. Phase 3: Hybrid – Use both frameworks
4. Phase 4: Migrate – Gradually move complex tasks to Galaxy

No forced migration! Continue using UFO² as long as it meets your needs.

See Migration Guide for details.

🎯 Can Galaxy do everything UFO² does?

Functionally: Yes. Galaxy can use UFO² as a Windows device agent.

Practically: It depends.

• For simple Windows tasks: UFO² standalone is easier and more streamlined
• For complex workflows: Galaxy orchestrates UFO² with other device agents

Recommendation: Use the right tool for the job. UFO² can work standalone or as Galaxy's Windows device agent.

📊 How mature is Galaxy?

Status: Active Development 🚧

Stable:

• ✅ Core architecture
• ✅ DAG orchestration
• ✅ Basic multi-device support
• ✅ Event system

In Development:

• 🔨 Advanced device types
• 🔨 Enhanced monitoring
• 🔨 Performance optimization
• 🔨 Extended documentation

Recommendation: Great for experimentation and non-critical workflows.

🔧 Can I extend or customize?

Both frameworks are highly extensible:

UFO²:

• Custom actions and automators
• Custom knowledge sources (RAG)
• Custom control detectors
• Custom evaluation metrics

Galaxy:

• Custom agents
• Custom device types
• Custom orchestration strategies
• Custom visualization components

See respective documentation for extension guides.

🤝 How can I contribute?

We welcome contributions to both UFO² and Galaxy!

Ways to contribute:

• 🐛 Report bugs and issues
• 💡 Suggest features and improvements
• 📝 Improve documentation
• 🧪 Add tests and examples
• 🔧 Submit pull requests

See CONTRIBUTING.md for guidelines.

⚠️ Disclaimer & License Copy Chapter

Disclaimer: By using this software, you acknowledge and agree to the terms in DISCLAIMER.md.

License: This project is licensed under the MIT License.

Trademarks: Use of Microsoft trademarks follows Microsoft's Trademark Guidelines.

Documents Docs Index (`documents/docs/index.md`) Copy Section

Welcome to UFO³ Documentation

📖 About This Documentation Copy Chapter

Welcome to the official documentation for UFO³, Microsoft's open-source framework for intelligent automation across devices and platforms. Whether you're looking to automate Windows applications or orchestrate complex workflows across multiple devices, this documentation will guide you through every step.

What you'll find here:

• 🚀 Quick Start Guides – Get up and running in minutes
• 📚 Core Concepts – Understand the architecture and key components
• ⚙️ Configuration – Set up your agents and models
• 🔧 Advanced Features – Deep dive into powerful capabilities
• 💡 FAQ – Common questions and troubleshooting

🎯 Choose Your Path Copy Chapter

UFO³ consists of two complementary frameworks. Choose the one that best fits your needs, or use both together!

🤔 Decision Guide Copy Section

🌟 What's New in UFO³? Copy Chapter

UFO³ is a scalable, universal cross-device agent framework that enables you to develop new device agents for different platforms and applications. Through the Agent Interaction Protocol (AIP), custom device agents can seamlessly integrate into UFO³ Galaxy for coordinated multi-device orchestration.

Evolution Timeline Copy Section

%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#E8F4F8','primaryTextColor':'#1A1A1A','primaryBorderColor':'#7CB9E8','lineColor':'#A8D5E2','secondaryColor':'#B8E6F0','tertiaryColor':'#D4F1F4','fontSize':'16px','fontFamily':'Segoe UI, Arial, sans-serif'}}}%%
graph LR
    A["<b>🎈 UFO</b><br/><span style='font-size:14px'>February 2024</span><br/><span style='font-size:13px; color:#666'><i>GUI Agent for Windows</i></span>"] 
    B["<b>🖥️ UFO²</b><br/><span style='font-size:14px'>April 2025</span><br/><span style='font-size:13px; color:#666'><i>Desktop AgentOS</i></span>"]
    C["<b>🌌 UFO³ Galaxy</b><br/><span style='font-size:14px'>November 2025</span><br/><span style='font-size:13px; color:#666'><i>Multi-Device Orchestration</i></span>"]
    
    A -->|Evolve| B
    B -->|Scale| C
    
    style A fill:#E8F4F8,stroke:#7CB9E8,stroke-width:2.5px,color:#1A1A1A,rx:15,ry:15
    style B fill:#C5E8F5,stroke:#5BA8D0,stroke-width:2.5px,color:#1A1A1A,rx:15,ry:15
    style C fill:#A4DBF0,stroke:#3D96BE,stroke-width:2.5px,color:#1A1A1A,rx:15,ry:15

🚀 UFO³ = **Galaxy** (Multi-Device Orchestration) + **UFO²** (Device Agent) Copy Section

UFO³ introduces Galaxy, a revolutionary multi-device orchestration framework that coordinates intelligent agents across heterogeneous platforms. Built on five tightly integrated design principles:

1. 🌟 Declarative Decomposition into Dynamic DAG - Requests decomposed into structured DAG with TaskStars and dependencies for automated scheduling and runtime rewriting

2. 🔄 Continuous Result-Driven Graph Evolution - Living constellation that adapts to execution feedback through controlled rewrites and dynamic adjustments

3. ⚡ Heterogeneous, Asynchronous & Safe Orchestration - Capability-based device matching with async execution, safe locking, and formally verified correctness

4. 🔌 Unified Agent Interaction Protocol (AIP) - WebSocket-based secure coordination layer with fault tolerance and automatic reconnection

5. 🛠️ Template-Driven MCP-Empowered Device Agents - Lightweight toolkit for rapid agent development with MCP integration for tool augmentation

🎓 Migration Path Copy Section

For UFO² Users:

1. ✅ Keep using UFO² – Fully supported, actively maintained
2. 🔄 Gradual adoption – Galaxy can use UFO² as Windows device agent
3. 📈 Scale up – Move to Galaxy when you need multi-device capabilities
4. 📚 Learning resources – Migration Guide

✨ Capabilities at a Glance Copy Chapter

🌌 Galaxy Framework – What's Different? Copy Section

🌟 Constellation Planning

User Request
     ↓
ConstellationAgent
     ↓
  [Task DAG]
   /   |   \
Task1 Task2 Task3
(Win) (Linux)(Mac)

Benefits:

• Cross-device dependency tracking
• Parallel execution optimization
• Cross-device dataflow management

🎯 Device Assignment

Selection Criteria
  • Platform
  • Resource
  • Task requirements
  • Performance history
        ↓
  Auto-Assignment
        ↓
  Optimal Devices

Smart Matching:

• Capability-based selection
• Real-time resource monitoring
• Dynamic reallocation

📊 Orchestration

Task1 → Running  ✅
Task2 → Pending  ⏸️
Task3 → Running  🔄
        ↓
   Completion
        ↓
   Final Report

Orchestration:

• Real-time status updates
• Automatic error recovery
• Progress tracking with feedback

🪟 UFO² Desktop AgentOS – Core Strengths Copy Section

UFO² serves dual roles: standalone Windows automation and Galaxy device agent for Windows platforms.

As Galaxy Device Agent:

• Receives tasks from ConstellationAgent through Galaxy orchestration layer
• Executes Windows-specific operations using proven UFO² capabilities
• Reports status and results back to TaskOrchestrator
• Seamlessly participates in cross-device workflows

🏗️ Architecture Copy Chapter

UFO³ Galaxy – Multi-Device Orchestration Copy Section

📖 Learn More →

UFO² – Desktop AgentOS Copy Section

📖 Learn More →

🚀 Quick Start Copy Chapter

Ready to dive in? Follow these guides to get started with your chosen framework:

🌌 Galaxy Quick Start (Multi-Device Orchestration) Copy Section

Perfect for complex workflows across multiple devices and platforms.

# 1. Install dependencies
pip install -r requirements.txt

# 2. Configure agents (see detailed guide for API key setup)
copy config\galaxy\agent.yaml.template config\galaxy\agent.yaml
copy config\ufo\agents.yaml.template config\ufo\agents.yaml

# 3. Start device agents
python -m ufo.server.app --port 5000
python -m ufo.client.client --ws --ws-server ws://localhost:5000/ws --client-id device_1 --platform windows

# 4. Launch Galaxy
python -m galaxy --interactive

📖 Complete Galaxy Quick Start Guide →
⚙️ Galaxy Configuration Details →

🪟 UFO² Quick Start (Windows Automation) Copy Section

Perfect for Windows-only automation tasks with quick setup.

# 1. Install
pip install -r requirements.txt

# 2. Configure (add your API keys)
copy config\ufo\agents.yaml.template config\ufo\agents.yaml

# 3. Run
python -m ufo --task <task_name>

📖 Complete UFO² Quick Start Guide →
⚙️ UFO² Configuration Details →

📚 Documentation Navigation Copy Chapter

🎯 Getting Started Copy Section

Start here if you're new to UFO³:

🏗️ Core Architecture Copy Section

Understand how UFO³ works under the hood:

⚙️ Configuration & Setup Copy Section

Configure your agents, models, and environments:

🎓 Tutorials & Examples Copy Section

Learn through practical examples in the documentation:

🔧 Advanced Topics Copy Section

Deep dive into powerful features:

🛠️ Development & Extension Copy Section

Customize and extend UFO³:

❓ Support & Troubleshooting Copy Section

Get help when you need it:

📊 Feature Matrix Copy Chapter

🎯 Use Cases & Examples Copy Chapter

Explore what you can build with UFO³:

🌌 Galaxy Use Cases (Cross-Device) Copy Section

Perfect for complex, multi-device workflows:

• Cross-Platform Data Pipelines: Extract from Windows Excel → Process on Linux → Visualize on Mac
• Distributed Testing: Run tests on Windows → Deploy to Linux → Update mobile app
• Multi-Device Monitoring: Collect logs from multiple devices → Aggregate centrally
• Complex Automation: Orchestrate workflows across heterogeneous platforms

🪟 UFO² Use Cases (Windows) Copy Section

Perfect for Windows automation and rapid task execution:

• Office Automation: Excel/Word/PowerPoint report generation and data processing
• Web Automation: Browser-based research, form filling, data extraction
• File Management: Organize, rename, convert files based on rules
• System Tasks: Windows configuration, software installation, backups

🌐 Community & Resources Copy Chapter

📺 Media & Videos Copy Section

Check out our official deep dive of UFO on YouTube.

Media Coverage: Copy Section

• 微软正式开源UFO²，Windows桌面迈入「AgentOS 时代」
• Microsoft's UFO: Smarter Windows Experience
• 下一代Windows系统曝光：基于GPT-4V

💬 Get Help & Connect Copy Section

• 📖 Documentation: You're here! Browse the navigation above
• 💬 Discussions: GitHub Discussions
• 🐛 Issues: GitHub Issues
• 📧 Email: [email protected]

🎨 Related Projects Copy Section

• TaskWeaver – Code-first LLM agent framework
• Windows Agent Arena – Evaluation benchmark
• GUI Agents Survey – Latest research

📚 Research & Citation Copy Chapter

UFO³ is built on cutting-edge research in multi-agent systems and GUI automation.

Papers Copy Section

If you use UFO³ in your research, please cite:

UFO³ Galaxy Framework (2025)

@article{zhang2025ufo3,
  title={UFO$^3$: Weaving the Digital Agent Galaxy}, 
  author = {Zhang, Chaoyun and Li, Liqun and Huang, He and Ni, Chiming and Qiao, Bo and Qin, Si and Kang, Yu and Ma, Minghua and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei},
  journal = {arXiv preprint arXiv:2511.11332},
  year    = {2025},
}

UFO² Desktop AgentOS (2025)

@article{zhang2025ufo2,
  title   = {{UFO2: The Desktop AgentOS}},
  author  = {Zhang, Chaoyun and Huang, He and Ni, Chiming and Mu, Jian and Qin, Si and He, Shilin and Wang, Lu and Yang, Fangkai and Zhao, Pu and Du, Chao and Li, Liqun and Kang, Yu and Jiang, Zhao and Zheng, Suzhen and Wang, Rujia and Qian, Jiaxu and Ma, Minghua and Lou, Jian-Guang and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei},
  journal = {arXiv preprint arXiv:2504.14603},
  year    = {2025}
}

Original UFO (2024)

@article{zhang2024ufo,
  title   = {{UFO: A UI-Focused Agent for Windows OS Interaction}},
  author  = {Zhang, Chaoyun and Li, Liqun and He, Shilin and Zhang, Xu and Qiao, Bo and Qin, Si and Ma, Minghua and Kang, Yu and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei and Zhang, Qi},
  journal = {arXiv preprint arXiv:2402.07939},
  year    = {2024}
}

📖 Read the Papers →

🗺️ Roadmap & Future Copy Chapter

UFO² Desktop AgentOS (Stable/LTS) Copy Section

• ✅ Long-term support and maintenance
• ✅ Windows device agent integration
• 🔜 Enhanced device capabilities
• 🔜 Picture-in-Picture mode

UFO³ Galaxy (Active Development) Copy Section

• ✅ Constellation Framework
• ✅ Multi-device coordination
• 🔄 Mobile, Web, IoT agents
• 🔄 Interactive visualization
• 🔜 Advanced fault tolerance

Legend: ✅ Done | 🔄 In Progress | 🔜 Planned

⚖️ License & Legal Copy Chapter

• License: MIT License
• Disclaimer: Read our disclaimer
• Trademarks: Microsoft Trademark Guidelines
• Contributing: Contribution Guidelines

🚀 Ready to Start? Copy Chapter

Choose your framework and begin your automation journey:

🌌 Start with Galaxy Copy Section

For multi-device orchestration

🪟 Start with UFO² Copy Section

For Windows automation

📖 Explore the Documentation Copy Section

Core Concepts | Configuration | FAQ | GitHub

Documents Docs Aip Overview (`documents/docs/aip/overview.md`) Copy Section

Agent Interaction Protocol (AIP)

The orchestration model requires a communication substrate that remains correct under continuous DAG evolution, dynamic agent participation, and fine-grained event propagation. Legacy HTTP-based coordination approaches (e.g., A2A, ACP) assume short-lived, stateless interactions, incurring handshake overhead, stale capability views, and fragile recovery when partial failures occur mid-task. These assumptions make them unsuitable for the continuously evolving workflows and long-running reasoning loops characteristic of UFO².

Design Overview Copy Chapter

AIP serves as the nervous system of UFO², connecting the ConstellationClient, device agent services, and device clients under a unified, event-driven control plane. It is designed as a lightweight yet evolution-tolerant protocol to satisfy six goals:

Design Goals:

• (G1) Maintain persistent bidirectional sessions to eliminate per-request overhead
• (G2) Unify heterogeneous capability discovery via multi-source profiling
• (G3) Ensure fine-grained reliability through heartbeats and timeout managers for disconnection and failure detection
• (G4) Preserve deterministic command ordering within sessions
• (G5) Support composable extensibility for new message types and resilience strategies
• (G6) Provide transparent reconnection and task continuity under transient failures

Five-Layer Architecture Copy Chapter

To meet these requirements, AIP adopts a persistent, bidirectional WebSocket transport and decomposes the orchestration substrate into five logical strata, each responsible for a distinct aspect of reliability and adaptability. The architecture establishes a complete substrate where L1 defines semantic contracts, L2 provides transport flexibility, L3 implements protocol logic, L4 ensures operational resilience, and L5 delivers deployment-ready orchestration primitives.

Architecture Diagram:

The following diagram illustrates the five-layer architecture and the roles of each component:

Layer 1: Message Schema Layer Copy Section

Defines strongly-typed, Pydantic-validated contracts (ClientMessage, ServerMessage) for message direction, purpose, and task transitions. All messages are validated at schema level, preventing malformed messages from entering the protocol pipeline, enabling early error detection and simplifying debugging.

Layer 2: Transport Abstraction Layer Copy Section

Provides protocol-agnostic Transport interface with production-grade WebSocket implementation. The abstraction layer allows swapping transports without changing protocol logic, supporting future protocol evolution.

Layer 3: Protocol Orchestration Layer Copy Section

Implements modular handlers for registration, task execution, heartbeat, and command dispatch. Each handler is independently testable and replaceable, supporting composable extensibility (G5) while maintaining ordered state transitions (G4).

Related Documentation:

• Complete message reference
• Protocol implementation details

Layer 4: Resilience and Health Management Layer Copy Section

!!!warning "Fault Tolerance"
This layer guarantees fine-grained reliability (G3) and seamless task continuity under transient disconnections (G6), preventing cascade failures.

Encapsulates reliability mechanisms ensuring operational continuity under failures:

→ Resilience implementation details

Layer 5: Endpoint Orchestration Layer Copy Section

Provides role-specific facades integrating lower layers into deployable components. These endpoints unify connection lifecycle, task routing, and health monitoring across roles, reinforcing G1–G6 through consistent implementation of lower-layer capabilities.

Endpoint Integration Benefits:

• ✅ Connection lifecycle management (G1, G6)
• ✅ Role-specific protocol variants (G5)
• ✅ Health monitoring integration (G3)
• ✅ Task routing and session management (G4)

→ Endpoint setup guide

Architecture Benefits Copy Chapter

Together, these layers form a vertically integrated stack that enables UFO² to maintain correctness and availability under challenging conditions:

AIP transforms distributed workflow execution into a coherent, safe, and adaptive system where reasoning and execution converge seamlessly across diverse agents and environments.

Core Capabilities Copy Chapter

Agent Registration & Profiling Copy Section

Each agent is represented by an AgentProfile combining data from three sources for comprehensive capability discovery, supporting heterogeneous capability unification (G2):

Benefits of Multi-Level Profiling:

• ✅ Accurate task allocation based on real-time capabilities (G2)
• ✅ Transparent adaptation to environmental changes (e.g., GPU availability)
• ✅ No manual updates needed when device state changes
• ✅ Informed scheduling decisions at scale

!!!tip "Dynamic Profile Updates"
Client telemetry continuously refreshes, so the orchestrator always sees current device state—critical for GPU-aware scheduling or cross-device load balancing (G2).

→ See detailed registration flow

Task Dispatch & Result Delivery Copy Section

AIP uses long-lived WebSocket sessions that span multiple task executions, eliminating per-request connection overhead and preserving context (G1).

Task Execution Sequence:

The following sequence diagram shows the complete lifecycle of a task from assignment to completion, including intermediate execution steps and state updates:

sequenceDiagram
    participant CC as ConstellationClient
    participant DAS as Device Service
    participant DAC as Device Client
    
    CC->>DAS: TASK message (TaskStar)
    DAS->>DAC: Stream task payload
    DAC->>DAC: Execute using MCP tools
    DAC->>DAS: Stream execution logs
    DAS->>CC: TASK_END (status, logs, results)
    CC->>CC: Update TaskConstellation
    CC->>CC: Notify ConstellationAgent

Each arrow represents a message exchange, with vertical lifelines showing the temporal ordering of events. Note how logs stream back during execution, enabling real-time monitoring.

!!!warning "Asynchronous Execution"
Tasks execute asynchronously. The orchestrator may assign multiple tasks to different devices simultaneously, with results arriving in non-deterministic order.

Related Documentation:

• Message format details
• TaskConstellation documentation
• TaskStar (task nodes) documentation

Command Execution Copy Section

Within each task, AIP executes individual commands deterministically with preserved ordering, enabling precise control and error handling (G4).

Command Structure:

Execution Guarantees:

• ✅ Sequential execution within a session (deterministic order) (G4)
• ✅ Command batching supported (reduces network overhead)
• ✅ Structured results with status codes and error details
• ✅ Timeout propagation for precise recovery strategies (G3)

Command Batching Example:

{
  "actions": [
    {"tool_name": "click", "parameters": {"target": "File"}, "call_id": "1"},
    {"tool_name": "click", "parameters": {"target": "Save As"}, "call_id": "2"},
    {"tool_name": "type", "parameters": {"text": "document.pdf"}, "call_id": "3"}
  ]
}

All three commands sent in one message, executed sequentially.

→ See command execution protocol

Message Protocol Overview Copy Chapter

All AIP messages use Pydantic models for automatic validation, serialization, and type safety.

Bidirectional Message Types Copy Section

Message Correlation:

Every message includes:

• timestamp: ISO 8601 formatted
• request_id / response_id: Unique identifier
• prev_response_id: Links responses to requests
• session_id: Session context

→ Complete message reference

Resilient Connection Protocol Copy Chapter

!!!warning "Network Instability Handling (G3, G6)"
AIP ensures continuous orchestration even under transient network failures or device disconnections through fine-grained reliability mechanisms and transparent reconnection.

Device Disconnection Flow Copy Section

Connection State Transitions:

This state diagram illustrates how devices transition between connection states and the actions triggered at each transition:

stateDiagram-v2
    [*] --> CONNECTED
    CONNECTED --> DISCONNECTED: Connection lost
    DISCONNECTED --> CONNECTED: Reconnection succeeds
    DISCONNECTED --> [*]: Timeout / Manual removal
    
    note right of DISCONNECTED
        • Excluded from scheduling
        • Tasks marked FAILED
        • Auto-reconnect triggered
    end note

The DISCONNECTED state acts as a quarantine zone where the device is temporarily removed from the scheduling pool while auto-reconnection attempts are made. If reconnection fails after timeout, the device is permanently removed.

ConstellationClient Disconnection Copy Section

!!!danger "Bidirectional Fault Handling"
When the ConstellationClient disconnects, all Device Agent Services:

1. Receive termination signal  
2. **Abort all ongoing tasks** tied to that client  
3. Prevent resource leakage and zombie processes  
4. Maintain end-to-end consistency

Guarantees:

• ✅ No orphaned tasks
• ✅ Synchronized state across client-server boundary
• ✅ Rapid recovery when connection restored (G6)
• ✅ Consistent TaskConstellation state (G4)

→ See resilience implementation

Extensibility Mechanisms Copy Chapter

AIP provides multiple extension points for domain-specific needs without modifying the core protocol, supporting composable extensibility (G5).

1. Protocol Middleware Copy Section

Add custom processing to message pipeline:

from aip.protocol.base import ProtocolMiddleware

class AuditMiddleware(ProtocolMiddleware):
    async def process_outgoing(self, msg):
        log_to_audit_trail(msg)
        return msg
    
    async def process_incoming(self, msg):
        log_to_audit_trail(msg)
        return msg

2. Custom Message Handlers Copy Section

Register handlers for new message types:

protocol.register_handler("custom_type", handle_custom_message)

3. Transport Layer Copy Section

Pluggable transport (default: WebSocket) (G5):

from aip.transport import CustomTransport
protocol.transport = CustomTransport(config)

→ See extensibility guide

Integration with UFO² Ecosystem Copy Chapter

AIP abstracts network/device heterogeneity, allowing the orchestrator to treat all agents as first-class citizens in a single event-driven control plane.

Related Documentation:

• TaskConstellation (DAG orchestrator)

• ConstellationAgent (orchestration agent)

• MCP Integration Guide

• Configuration System
  Next Steps:

• 📖 Message Reference - Complete message type documentation

• 🔧 Protocol Guide - Implementation details and best practices

• 🌐 Transport Layer - WebSocket configuration and optimization

• 🔌 Endpoints - Endpoint setup and usage patterns

• 🛡️ Resilience - Connection management and fault tolerance

Summary Copy Chapter

AIP transforms distributed workflow execution into a coherent, safe, and adaptive system where reasoning and execution converge seamlessly across diverse agents and environments.

Key Takeaways:

By decomposing orchestration into five logical layers—each addressing specific reliability and adaptability concerns—AIP enables UFO² to maintain correctness and availability under DAG evolution (G4, G5), agent churn (G3, G6), and heterogeneous execution environments (G1, G2).

Documents Docs Client Overview (`documents/docs/client/overview.md`) Copy Section

UFO Client Overview

The UFO Client runs on target devices and serves as the execution layer of UFO's distributed agent system. It manages MCP (Model Context Protocol) servers, executes commands deterministically, and communicates with the Agent Server through the Agent Interaction Protocol (AIP).

Quick Start: Jump to the Quick Start Guide to connect your device. Make sure the Agent Server is running first.

🎯 What is the UFO Client? Copy Chapter

graph LR
    subgraph "Agent Server (Brain)"
        Reasoning[High-Level Reasoning]
        Planning[Task Planning]
        Strategy[Strategy Selection]
    end
    
    subgraph "Agent Client (Hands)"
        Execution[Command Execution]
        Tools[Tool Management]
        Reporting[Status Reporting]
    end
    
    subgraph "Device Environment"
        Apps[Applications]
        Files[File System]
        UI[User Interface]
    end
    
    Reasoning -->|Directives| Execution
    Planning -->|Commands| Execution
    Strategy -->|Tasks| Execution
    
    Execution --> Tools
    Tools --> Apps
    Tools --> Files
    Tools --> UI
    
    Reporting -->|Results| Reasoning
    
    style Reasoning fill:#bbdefb
    style Execution fill:#c8e6c9
    style Tools fill:#fff9c4

The UFO Client is a stateless execution agent that:

Stateless Design Philosophy: The client focuses purely on execution. All reasoning and decision-making happens on the server, allowing independent updates to server logic and client tools, simple client architecture, intelligent orchestration of multiple clients, and resource-efficient operation.

Architecture: The UFO Client is part of UFO's distributed server-client architecture, where it handles command execution and resource access while the Agent Server handles orchestration and decision-making. See Server-Client Architecture for the complete design rationale, communication protocols, and deployment patterns.

🏗️ Architecture Copy Chapter

The client implements a layered architecture separating communication, execution, and tool management for maximum flexibility and maintainability.

graph TB
    subgraph "Communication"
        WSC[WebSocket Client<br/>AIP Protocol]
    end
    
    subgraph "Orchestration"
        UFC[UFO Client]
        CM[Computer Manager]
    end
    
    subgraph "Execution"
        COMP[Computer]
        MCPM[MCP Manager]
    end
    
    subgraph "Tools"
        LOCAL[Local MCP Servers]
        REMOTE[Remote MCP Servers]
    end
    
    WSC --> UFC
    UFC --> CM
    CM --> COMP
    COMP --> MCPM
    MCPM --> LOCAL
    MCPM --> REMOTE
    
    style WSC fill:#bbdefb
    style UFC fill:#c8e6c9
    style COMP fill:#fff9c4
    style MCPM fill:#ffcdd2

Core Components Copy Section

For detailed component documentation:

• WebSocket Client - AIP protocol implementation
• UFO Client - Execution orchestration
• Computer Manager - Multi-computer management
• Device Info Provider - System profiling
• MCP Integration - MCP server management (comprehensive documentation)

🚀 Key Capabilities Copy Chapter

1. Deterministic Command Execution Copy Section

The client executes commands exactly as specified without interpretation or reasoning, ensuring predictable behavior.

sequenceDiagram
    participant Server
    participant Client as UFO Client
    participant Computer
    participant Tool as MCP Tool
    
    Server->>Client: COMMAND (AIP)
    Client->>Computer: Execute Command
    Computer->>Computer: Lookup Tool
    Computer->>Tool: Execute with Timeout
    Tool-->>Computer: Result
    Computer-->>Client: Aggregated Result
    Client-->>Server: COMMAND_RESULTS (AIP)

Execution Flow:

Execution Guarantees:

• Isolation: Each tool runs in separate thread pool
• Timeouts: Configurable timeout (default: 6000 seconds/100 minutes)
• Fault Tolerance: One failed tool doesn't crash entire client
• Thread Safety: Concurrent tool execution supported
• Error Reporting: Structured errors returned to server

2. MCP Server Management Copy Section

The client manages a collection of MCP (Model Context Protocol) servers to provide diverse tool access for automation tasks. The client is responsible for registering, managing, and executing these tools, while the Agent Server handles command orchestration. See Server-Client Architecture for how MCP integration fits into the overall architecture.

MCP Server Categories:

Data Collection Servers gather information from the device:

Example Tools: get_system_info(), list_running_apps(), capture_screenshot(), get_ui_tree()

Action Servers perform actions on the device:

Example Tools: click_button(label), type_text(text), open_application(name), execute_command(cmd)

Server Types:

Example MCP Server Configuration:

mcp_servers:
  data_collection:
    - name: "system_info"
      type: "local"
      class: "SystemInfoServer"
    - name: "ui_detector"
      type: "local"
      class: "UIDetectionServer"
  
  action:
    - name: "gui_automation"
      type: "local"
      class: "GUIAutomationServer"
    - name: "file_ops"
      type: "remote"
      url: "http://localhost:8080/mcp"

See MCP Integration for comprehensive MCP server documentation.

3. Device Profiling Copy Section

The client automatically collects and reports device information to enable the server to make intelligent task routing decisions.

Device Profile Structure:

{
  "device_id": "device_windows_001",
  "platform": "windows",
  "platform_type": "computer",
  "os_version": "10.0.22631",
  "system_info": {
    "cpu_count": 8,
    "memory_total_gb": 16.0,
    "disk_total_gb": 512.0,
    "hostname": "DESKTOP-ABC123",
    "ip_address": "192.168.1.100"
  },
  "supported_features": [
    "gui_automation",
    "cli_execution",
    "browser_control",
    "office_integration",
    "windows_apps"
  ],
  "installed_applications": [
    "Chrome",
    "Excel",
    "PowerPoint",
    "VSCode"
  ],
  "screen_resolution": "1920x1080",
  "connected_at": "2025-11-05T10:30:00Z"
}

Profile Usage on Server:

graph LR
    Client[Client Detects<br/>Device Info]
    Server[Server Stores<br/>Profile]
    Route[Server Routes<br/>Tasks]
    
    Client -->|Report Profile| Server
    Server -->|Match Requirements| Route
    Route -->|Dispatch Task| Client
    
    style Client fill:#bbdefb
    style Server fill:#c8e6c9
    style Route fill:#fff9c4

Server Uses Profile For:

See Device Info Provider for detailed profiling documentation.

4. Resilient Communication Copy Section

Robust, fault-tolerant communication with the server using strongly-typed AIP messages.

Connection Lifecycle:

stateDiagram-v2
    [*] --> Disconnected
    Disconnected --> Connecting: Initiate Connection
    Connecting --> Registering: WebSocket Established
    Registering --> Connected: Registration Success
    Connecting --> Disconnected: Connection Failed
    Registering --> Disconnected: Registration Failed
    
    Connected --> Heartbeating: Start Heartbeat Loop
    Heartbeating --> Heartbeating: Send/Receive Heartbeat
    Heartbeating --> Disconnected: Heartbeat Timeout
    Heartbeating --> Disconnected: WebSocket Closed
    
    Disconnected --> Connecting: Retry (Exponential Backoff)
    
    note right of Connected
        • Receive commands
        • Execute tasks
        • Report results
    end note
    
    note right of Heartbeating
        Default interval: 30s
        Timeout: 60s
    end note

Connection Features:

Message Types:

See WebSocket Client and AIP Protocol for protocol details.

📋 Workflow Examples Copy Chapter

Client Initialization & Registration Copy Section

sequenceDiagram
    participant Main as Client Main
    participant MCP as MCP Manager
    participant WSC as WebSocket Client
    participant Server
    
    Main->>MCP: Initialize MCP Servers
    MCP-->>Main: Server Registry Ready
    
    Main->>WSC: Create Client & Connect
    WSC->>Server: WebSocket Connect
    Server-->>WSC: Connection Established
    
    WSC->>WSC: Collect Device Info
    WSC->>Server: REGISTRATION
    Server-->>WSC: REGISTRATION_ACK
    
    WSC->>WSC: Start Heartbeat Loop
    
    loop Every 30 seconds
        WSC->>Server: HEARTBEAT
        Server-->>WSC: HEARTBEAT_ACK
    end
    
    Note over WSC,Server: Ready to Execute Commands

Initialization Steps:

Command Execution Flow Copy Section

sequenceDiagram
    participant Server
    participant Client as UFO Client
    participant Comp as Computer
    participant Tool as MCP Tool
    
    Server->>Client: COMMAND<br/>{type: "click_button", args: {...}}
    Client->>Comp: execute_command()
    Comp->>Comp: find_tool("click_button")
    
    alt Tool Found
        Comp->>Tool: execute(args)
        Note over Tool: Thread Pool Execution<br/>6000s timeout
        Tool-->>Comp: Success
        Comp-->>Client: Result
        Client-->>Server: COMMAND_RESULTS<br/>{status: "completed"}
    else Tool Not Found
        Comp-->>Client: Error
        Client-->>Server: ERROR<br/>{error: "Tool not found"}
    end

🖥️ Platform Support Copy Chapter

The client supports multiple platforms with platform-specific tool implementations.

Platform Detection:

• Automatic: Detected via platform.system() on startup
• Override: Use --platform flag to specify manually
• Validation: Server validates platform matches task requirements

Platform-Specific Example:

Windows:

# Windows-specific tools
tools = [
    "open_windows_app(name='Excel')",
    "execute_powershell(script='Get-Process')",
    "read_registry(key='HKLM\\Software')"
]

Linux:

# Linux-specific tools
tools = [
    "execute_bash(command='ls -la')",
    "install_package(name='vim')",
    "control_systemd(service='nginx', action='restart')"
]

⚙️ Configuration Copy Chapter

Command-Line Arguments Copy Section

Start the UFO client with:

python -m ufo.client.client [OPTIONS]

Available Options:

Quick Start Command:

# Minimal command (default server)
python -m ufo.client.client --ws --client-id my_device

# Production command (custom server)
python -m ufo.client.client \
  --ws \
  --client-id device_production_01 \
  --ws-server ws://ufo-server.company.com:5000/ws \
  --max-retries 10 \
  --log-level INFO

UFO Configuration Copy Section

The client inherits settings from config_dev.yaml:

Key Configuration Sections:

Sample Configuration:

client:
  heartbeat_interval: 30  # seconds
  command_timeout: 6000   # seconds (100 minutes)
  max_concurrent_tools: 10

mcp_servers:
  data_collection:
    - name: system_info
      type: local
      enabled: true
  action:
    - name: gui_automation
      type: local
      enabled: true
      settings:
        click_delay: 0.5
        typing_speed: 100  # chars per minute

logging:
  level: INFO
  format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
  file: "logs/client.log"

See Configuration Guide for comprehensive documentation.

⚠️ Error Handling Copy Chapter

The client is designed to handle various failure scenarios gracefully without crashing.

Connection Failures Copy Section

stateDiagram-v2
    [*] --> Attempting
    Attempting --> Connected: Success
    Attempting --> Failed: Error
    
    Failed --> Waiting: Exponential Backoff
    Waiting --> Attempting: Retry (2^n seconds)
    
    Failed --> [*]: Max Retries Exceeded
    
    note right of Waiting
        Retry Delays:
        1st: 2s
        2nd: 4s
        3rd: 8s
        4th: 16s
        5th: 32s
    end note

Connection Error Handling:

Tool Execution Failures Copy Section

Protection Mechanisms:

Error Handling Example:

# Client automatically handles tool errors
try:
    result = tool.execute(args)
    return {"status": "success", "result": result}
except TimeoutError:
    return {"status": "error", "error": "Tool execution timeout"}
except Exception as e:
    return {"status": "error", "error": str(e)}

Server Disconnection Copy Section

Graceful Shutdown Process:

1. Detect Disconnection - WebSocket connection lost
2. Stop Heartbeat - Terminate keep-alive loop
3. Cancel Pending Tasks - Abort in-progress commands
4. Attempt Reconnection - Use exponential backoff
5. Clean Shutdown - If max retries exceeded

✅ Best Practices Copy Chapter

Development Best Practices Copy Section

1. Use Unique Client IDs

# Bad: Generic ID
--client-id client_001

# Good: Descriptive ID
--client-id device_win_dev_john_laptop

2. Start with INFO Logging

# Development: WARNING for normal operation (default)
--log-level WARNING

# Debugging: DEBUG for troubleshooting
--log-level DEBUG

3. Test MCP Connectivity First

# Verify MCP servers are accessible before running client
from ufo.client.mcp.mcp_server_manager import MCPServerManager

manager = MCPServerManager()
# Test server creation from configuration

Production Best Practices Copy Section

1. Use Descriptive Client IDs

# Include environment, location, purpose
--client-id device_windows_production_office_01
--client-id device_linux_staging_lab_02

2. Configure Automatic Restart

systemd (Linux):

[Unit]
Description=UFO Agent Client
After=network.target
    
[Service]
Type=simple
User=ufo
WorkingDirectory=/opt/ufo
ExecStart=/usr/bin/python3 -m ufo.client.client \
  --ws \
  --client-id device_linux_prod_01 \
  --ws-server ws://ufo-server.internal:5000/ws \
  --log-level INFO
Restart=always
RestartSec=10
    
[Install]
WantedBy=multi-user.target

PM2 (Cross-platform):

{
  "apps": [{
    "name": "ufo-client",
    "script": "python",
    "args": [
      "-m", "ufo.client.client",
      "--ws",
      "--client-id", "device_win_prod_01",
      "--ws-server", "ws://ufo-server.internal:5000/ws",
      "--log-level", "INFO"
    ],
    "cwd": "C:\\ufo",
    "restart_delay": 5000,
    "max_restarts": 10
  }]
}

3. Monitor Connection Health

# Check logs for connection status
tail -f logs/client.log | grep -E "Connected|Disconnected|ERROR"

Security Best Practices Copy Section

!!! warning "Security Considerations"

| Practice | Description | Implementation |
|----------|-------------|----------------|
| **Use WSS** | Encrypt WebSocket communication | `wss://server:5000/ws` instead of `ws://` |
| **Validate Server** | Verify server certificate | Configure SSL/TLS verification |
| **Restrict Tools** | Limit MCP server access | Only enable necessary tools |
| **Least Privilege** | Run with minimum permissions | Create dedicated user account |
| **Network Isolation** | Use firewalls and VPNs | Restrict server access to internal network |

🎓 Documentation Map Copy Chapter

Getting Started Copy Section

Component Details Copy Section

Related Documentation Copy Section

🔄 Client vs. Server Copy Chapter

Understanding the clear division between client and server responsibilities is crucial for effective system design.

Responsibility Matrix:

Workflow Comparison:

graph TB
    subgraph "Server Workflow"
        S1[Receive User Request]
        S2[Reason About Task]
        S3[Plan Execution Steps]
        S4[Select Target Device]
        S5[Send Commands]
    end
    
    subgraph "Client Workflow"
        C1[Receive Command]
        C2[Lookup Tool]
        C3[Execute Tool]
        C4[Return Result]
    end
    
    S1 --> S2
    S2 --> S3
    S3 --> S4
    S4 --> S5
    S5 -.->|AIP| C1
    C1 --> C2
    C2 --> C3
    C3 --> C4
    C4 -.->|AIP| S5
    
    style S1 fill:#bbdefb
    style S2 fill:#bbdefb
    style S3 fill:#bbdefb
    style C1 fill:#c8e6c9
    style C2 fill:#c8e6c9
    style C3 fill:#c8e6c9

Decoupled Architecture Benefits:

• Independent Updates: Modify server logic without touching clients
• Flexible Deployment: Run clients on any platform
• Scalability: Add more clients without server changes
• Maintainability: Simpler client code, easier debugging
• Testability: Test client and server independently

🚀 Next Steps Copy Chapter

1. Run Your First Client

# Follow the quick start guide
python -m ufo.client.client \
  --ws \
  --client-id my_first_device \
  --ws-server ws://localhost:5000/ws

👉 Quick Start Guide

2. Understand Registration Process

Learn how clients register with the server, device profile structure, and registration acknowledgment.

👉 Server Quick Start - Start server and connect clients

3. Explore MCP Integration

Learn about MCP servers, configure custom tools, and create your own MCP servers.

👉 MCP Integration

4. Configure for Your Environment

Customize MCP servers, adjust timeouts and retries, and configure platform-specific settings.

👉 Configuration Guide

5. Master the Protocol

Deep dive into AIP messages, understand message flow, and error handling patterns.

👉 AIP Protocol

Documents Docs Galaxy Overview (`documents/docs/galaxy/overview.md`) Copy Section

UFO³ — Weaving the Digital Agent Galaxy

🚀 What is UFO³ Galaxy? Copy Chapter

UFO³ Galaxy is a revolutionary cross-device orchestration framework that transforms isolated device agents into a unified digital ecosystem. It models complex user requests as Task Constellations (星座) — dynamic distributed DAGs where nodes represent executable subtasks and edges capture dependencies across heterogeneous devices.

🎯 The Vision Copy Section

Building truly ubiquitous intelligent agents requires moving beyond single-device automation. UFO³ Galaxy addresses four fundamental challenges in cross-device agent orchestration:

🔄 Asynchronous Parallelism
Enabling concurrent task execution across multiple devices while maintaining correctness through event-driven coordination and safe concurrency control

⚡ Dynamic Adaptation
Real-time workflow evolution in response to intermediate results, transient failures, and runtime observations without workflow abortion

🌐 Distributed Coordination
Reliable, low-latency communication across heterogeneous devices via WebSocket-based Agent Interaction Protocol with fault tolerance

🛡️ Safety Guarantees
Formal invariants ensuring DAG consistency during concurrent modifications and parallel execution, verified through rigorous proofs

🏗️ Architecture Copy Chapter

Layered Design Copy Section

UFO³ Galaxy follows a hierarchical orchestration model that separates global coordination from local execution. This architecture enables scalable cross-device orchestration while maintaining consistent control and responsiveness across diverse operating systems and network environments.

🎛️ Hierarchical Control Plane

ConstellationClient serves as the global control plane, maintaining a live registry of all connected device agents with their:

• Capability profiles and system specifications
• Runtime health metrics and availability status
• Current load and resource utilization

This registry enables intelligent task placement based on device capabilities, avoiding mismatches between task requirements and device capacity.

Each device hosts a device agent server that manages local orchestration through persistent WebSocket sessions with ConstellationClient. The server:

• Maintains execution contexts on the host
• Provides unified interface to underlying tools via MCP servers
• Handles task execution, telemetry streaming, and resource monitoring

Clean separation: Global orchestration policies are decoupled from device-specific heterogeneity, providing consistent abstraction across endpoints with different OS, hardware, or network conditions.

🔄 Orchestration Flow

1. DAG Synthesis: ConstellationClient invokes the Constellation Agent to construct a TaskConstellation—a dynamic DAG encoding task decomposition, dependencies, and device mappings
2. Device Assignment: Each TaskStar (DAG node) is assigned to suitable device agents based on capability profiles and system load
3. Asynchronous Execution: The Constellation Orchestrator executes the DAG in an event-driven manner:
   • Task completions trigger dependent nodes
   • Failures prompt retry, migration, or partial DAG rewrites
   • Workflows adapt to real-time system dynamics (device churn, network variability)

Result: Highly parallel and resilient execution that sustains workflow completion even as subsets of devices fail or reconnect.

🔌 Cross-Agent Communication

The Agent Interaction Protocol (AIP) handles all cross-agent interactions:

• Agent registration and capability synchronization
• Task dispatch and progress reporting
• Result aggregation and telemetry streaming

Built on persistent WebSocket channels, AIP provides:

• Lightweight: Minimal overhead for control messages
• Bidirectional: Full-duplex communication between client and agents
• Multiplexed: Concurrent message streams over single connection
• Low-latency: Fast propagation of control signals and state updates
• Resilient: Maintains global consistency despite intermittent connectivity

Together, these design elements form a cohesive foundation for orchestrating large-scale, heterogeneous, and adaptive workflows across a resilient multi-device execution fabric.

✨ Core Design Principles Copy Chapter

UFO³ Galaxy realizes cross-device orchestration through five tightly integrated design principles:

1. 🌟 Declarative Decomposition into Dynamic DAG (Task Constellation) Copy Section

Natural-language or programmatic requests are decomposed by the Constellation Agent into a structured DAG of TaskStars (nodes) and TaskStarLines (edges) that encode workflow logic, dependencies, and device assignments. This declarative structure is amenable to automated scheduling, introspection, and dynamic modification throughout execution.

Key Benefits:

• 📋 Declarative structure for automated scheduling
• 🔍 Runtime introspection for workflow visibility
• ✏️ Dynamic rewriting throughout execution
• 🔄 Automated orchestration across heterogeneous devices

graph LR
    A[User Intent] --> B[Constellation Agent]
    B --> C[Task Constellation DAG]
    C --> D[TaskStar 1<br/>Windows]
    C --> E[TaskStar 2<br/>Linux GPU]
    C --> F[TaskStar 3<br/>Linux CPU]
    C --> G[TaskStar 4<br/>Mobile]
    E --> H[TaskStar 5]
    F --> H
    G --> H

Learn more →

2. 🔄 Continuous, Result-Driven Graph Evolution Copy Section

The Task Constellation is a living data structure that evolves in response to execution feedback. Intermediate outputs, transient failures, and new observations trigger controlled rewrites—adding diagnostic TaskStars, creating fallbacks, rewiring dependencies, or pruning completed nodes—so the system adapts dynamically instead of aborting on errors.

Adaptation Mechanisms:

• 🩺 Diagnostic TaskStars added for debugging
• 🛡️ Fallback creation for error recovery
• 🔗 Dependency rewiring for workflow optimization
• ✂️ Node pruning after completion

The Constellation Agent operates in two modes:

• Creation Mode: Synthesizes initial DAG from user request with device-aware task decomposition
• Editing Mode: Incrementally refines constellation based on task completion events and runtime feedback

Learn more →

3. 🎯 Heterogeneous, Asynchronous, and Safe Orchestration Copy Section

Each Task Star is matched to the most suitable device agent via rich Agent Profiles reflecting OS, hardware capabilities, and installed tools. The Constellation Orchestrator executes tasks asynchronously, allowing multiple TaskStars to progress in parallel.

Safety Guarantees:

• 🔒 Safe assignment locking prevents race conditions
• 📅 Event-driven scheduling monitors DAG readiness
• ✅ DAG consistency checks maintain structural integrity
• 🔄 Batched edits ensure atomicity
• 📐 Formal verification reinforces correctness
• ⏱️ Timeout protection prevents deadlocks

These mechanisms collectively ensure high efficiency without compromising reliability.

Learn more →

4. 🔌 Unified Agent Interaction Protocol (AIP) Copy Section

Built atop persistent WebSocket channels, AIP provides a unified, secure, and fault-tolerant layer for the entire agent ecosystem.

Core Capabilities:

• 📝 Agent registry with capability profiles
• 🔐 Session management for secure communication
• 📤 Task dispatch with intelligent routing
• 🎯 Coordination primitives for distributed workflows
• 💓 Heartbeat monitoring for health tracking
• 🔌 Automatic reconnection under network fluctuations
• 🔄 Retry mechanisms for reliability

Architecture Benefits:

• 🪶 Lightweight interface for easy integration
• 🧩 Extensible design supports new agent types
• 🛡️ Fault tolerance ensures continuous operation

This protocol abstracts OS and network heterogeneity, enabling seamless collaboration among agents across desktops, servers, and edge devices, while allowing new agents to integrate seamlessly into the UFO³ ecosystem.

Learn more →

5. 🛠️ Template-Driven Framework for Device Agents Copy Section

To democratize agent creation, UFO³ provides a lightweight development template and toolkit for rapidly building new device agents.

Development Framework:

• 📄 Capability declaration defines agent profiles
• 🔗 Environment binding connects to local systems
• 🧩 MCP server integration for tool augmentation
• 🔧 Modular design accelerates development

Model Context Protocol (MCP) Integration:

• 🎁 Tool packages via MCP servers
• 🔌 Plug-and-play capability extension
• 🌐 Cross-platform tool standardization
• 🚀 Rapid prototyping of new agents

This modular architecture maintains consistency across the constellation while enabling developers to extend UFO³ to new platforms (mobile, web, IoT, embedded systems, etc.) with minimal effort.

🔌 Extensibility: UFO³ is designed as a universal framework that supports developing new device agents for different platforms (mobile, web, IoT, embedded systems, etc.) and applications. Through the Agent Interaction Protocol (AIP), custom device agents can seamlessly integrate into UFO³ Galaxy for coordinated multi-device automation. Want to build your own device agent? See our Creating Custom Device Agents tutorial to learn how to extend UFO³ to new platforms.

Learn more → | MCP Integration →

🎯 Key Capabilities Copy Chapter

🌐 Cross-Device Collaboration Copy Section

Execute workflows that span Windows desktops, Linux servers, GPU clusters, mobile devices, and edge nodes—all from a single natural language request.

⚡ Asynchronous Parallelism Copy Section

Automatically identify parallelizable subtasks and execute them concurrently across devices through:

• Event-driven scheduling that continuously monitors DAG topology for ready tasks
• Non-blocking execution with Python asyncio for maximum concurrency
• Dynamic adaptation that integrates new tasks without interrupting running execution

Result: Dramatically reduced end-to-end latency compared to sequential execution.

🛡️ Safety & Consistency Copy Section

• Three formal invariants (I1-I3) enforced at runtime for DAG correctness
• Safe assignment locking prevents race conditions during concurrent modifications
• Acyclicity validation ensures no circular dependencies
• State merging algorithm preserves execution progress during dynamic edits
• Timeout protection prevents deadlocks from agent failures

🔄 Dynamic Workflow Evolution Copy Section

• Dual-mode operation: Separate creation and editing phases with controlled transitions
• Feedback-driven adaptation: Task completion events trigger intelligent constellation refinement
• LLM-powered reasoning: ReAct architecture for context-aware DAG modifications
• Undo/redo support: ConstellationEditor with command pattern for safe interactive editing

👁️ Rich Observability Copy Section

• Real-time constellation visualization with DAG topology updates
• Event bus with publish-subscribe pattern for monitoring task progress
• Detailed execution logs with markdown trajectory support
• Task status tracking (pending, running, completed, failed, cancelled)
• Dependency graph inspection and validation tools

🎨 Use Cases Copy Chapter

🖥️ Software Development & Deployment Copy Section

"Clone the repo on my laptop, build the Docker image on the GPU server, deploy to staging, and run the test suite on the CI cluster."

Workflow DAG:

graph LR
    A[Clone<br/>Windows] --> B[Build<br/>Linux GPU]
    B --> C[Deploy<br/>Linux Server]
    C --> D[Test<br/>Linux CI]

📊 Data Science Workflows Copy Section

"Fetch the dataset from cloud storage, preprocess on the Linux workstation, train the model on the A100 node, and generate a visualization dashboard on my Windows machine."

Workflow DAG:

graph LR
    A[Fetch<br/>Any] --> B[Preprocess<br/>Linux]
    B --> C[Train<br/>Linux GPU]
    C --> D[Visualize<br/>Windows]

📝 Cross-Platform Document Processing Copy Section

"Extract data from Excel on Windows, process with Python scripts on Linux, generate PDF reports, and send summary emails."

Workflow DAG:

graph LR
    A[Extract<br/>Windows] --> B[Process<br/>Linux]
    B --> C[Generate PDF<br/>Windows]
    B --> D[Send Email<br/>Windows]

🔬 Distributed System Monitoring Copy Section

"Collect server logs from all Linux machines, analyze for errors, generate alerts, and create a consolidated report."

Workflow DAG:

graph LR
    A[Collect Logs<br/>Linux 1] --> D[Analyze Errors<br/>Any]
    B[Collect Logs<br/>Linux 2] --> D
    C[Collect Logs<br/>Linux 3] --> D
    D --> E[Generate Report<br/>Windows]

🏢 Enterprise Automation Copy Section

"Query the database on the server, process the results, update Excel spreadsheets on Windows, and generate PowerPoint presentations."

Workflow DAG:

graph LR
    A[Query DB<br/>Linux] --> B[Process Data<br/>Any]
    B --> C[Update Excel<br/>Windows]
    B --> D[Create PPT<br/>Windows]

🗺️ Documentation Structure Copy Chapter

🚀 [Quick Start](../getting_started/quick_start_galaxy.md) Copy Section

Get UFO³ Galaxy up and running in minutes with our step-by-step guide

👥 [Galaxy Client](client/overview.md) Copy Section

Device coordination, connection management, and ConstellationClient API

🧠 [Constellation Agent](constellation_agent/overview.md) Copy Section

LLM-driven task decomposition, DAG creation, and dynamic workflow evolution

⚙️ [Constellation Orchestrator](constellation_orchestrator/overview.md) Copy Section

Asynchronous execution engine, event-driven coordination, and safety guarantees

📊 [Task Constellation](constellation/overview.md) Copy Section

DAG structure, TaskStar nodes, TaskStarLine edges, and constellation editor

🆔 [Agent Registration](agent_registration/overview.md) Copy Section

Device registry, agent profiles, and registration flow

🌐 [Agent Interaction Protocol](../aip/overview.md) Copy Section

WebSocket messaging, protocol specification, and communication patterns

⚙️ [Configuration](../configuration/system/galaxy_devices.md) Copy Section

Device pools, capabilities, and orchestration policies

🚦 Getting Started Copy Chapter

Ready to build your Digital Agent Galaxy? Follow these steps:

1. Install UFO³ Copy Section

# Clone the repository
git clone https://github.com/microsoft/UFO.git
cd UFO

# Install dependencies
pip install -r requirements.txt

2. Configure Device Pool Copy Section

Create configuration files in config/galaxy/:

config/galaxy/devices.yaml - Define your devices:

devices:
  - device_id: "windowsagent"
    server_url: "ws://localhost:5005/ws"
    os: "windows"
    capabilities:
      - "web_browsing"
      - "office_applications"
      - "file_management"
    metadata:
      location: "home_office"
      os: "windows"
      performance: "medium"
    max_retries: 5
    
  - device_id: "linux_agent_1"
    server_url: "ws://localhost:5001/ws"
    os: "linux"
    capabilities:
      - "server"
      - "python"
      - "docker"
    metadata:
      os: "linux"
      performance: "high"
      logs_file_path: "/root/log/log1.txt"
    auto_connect: true
    max_retries: 5
    
  - device_id: "mobile_agent_1"
    server_url: "ws://localhost:5002/ws"
    os: "android"
    capabilities:
      - "mobile"
      - "adb"
      - "ui_automation"
    metadata:
      os: "android"
      performance: "medium"
      device_type: "smartphone"
    auto_connect: true
    max_retries: 5

config/galaxy/constellation.yaml - Configure runtime settings:

# Constellation Runtime Settings
CONSTELLATION_ID: "my_constellation"
HEARTBEAT_INTERVAL: 30.0  # Heartbeat interval in seconds
RECONNECT_DELAY: 5.0  # Delay before reconnecting in seconds
MAX_CONCURRENT_TASKS: 6  # Maximum concurrent tasks
MAX_STEP: 15  # Maximum steps per session

# Device Configuration
DEVICE_INFO: "config/galaxy/devices.yaml"

# Logging Configuration
LOG_TO_MARKDOWN: true

See Galaxy Configuration for complete documentation.

3. Start Device Agents Copy Section

On each device, launch the Agent Server. For detailed setup instructions, see the respective quick start guides:

On Windows:

See Windows Agent (UFO²) Quick Start →

On Linux:

See Linux Agent Quick Start →

On Mobile (Android):

See Mobile Agent Quick Start →

4. Launch Galaxy Client Copy Section

Interactive Mode:

python -m galaxy --interactive

Direct Request:

python -m galaxy "Your cross-device task here"

Programmatic API:

from galaxy.galaxy_client import GalaxyClient

async def main():
    client = GalaxyClient(session_name="my_session")
    await client.initialize()
    result = await client.process_request("Your task request")
    await client.shutdown()

For detailed instructions, see the Quick Start Guide.

🔧 System Components Copy Chapter

UFO³ Galaxy consists of several integrated components working together:

Core Components Copy Section

Supporting Infrastructure Copy Section

For detailed component documentation, see the respective sections in Documentation Structure.

Technology Stack Copy Section

🌟 From Devices to Constellations to Galaxy Copy Chapter

UFO³ represents a paradigm shift in intelligent automation:

• Single Device → Isolated agents operating within one OS
• Task Constellation → Coordinated multi-device workflows for one task
• Digital Agent Galaxy → Interconnected constellations spanning your entire digital estate

Over time, multiple constellations can interconnect, weaving together agents, devices, and capabilities into a self-organizing Digital Agent Galaxy. This design elevates cross-device automation from a brittle engineering challenge to a unified orchestration paradigm, where multi-device workflows become naturally expressive, paving the way for large-scale, adaptive, and resilient intelligent ubiquitous computing systems.

📊 Performance Monitoring & Evaluation Copy Chapter

UFO³ Galaxy provides comprehensive performance monitoring and evaluation tools to analyze multi-device workflow execution:

Automated Metrics Collection Copy Section

Galaxy automatically collects detailed performance metrics during execution through an event-driven observer pattern:

• Task Metrics: Execution times, success rates, bottleneck identification
• Constellation Metrics: DAG statistics, parallelism analysis, critical path computation
• Modification Metrics: Dynamic editing patterns and adaptation frequency
• Device Metrics: Per-device performance and resource utilization

All metrics are captured in real-time without impacting execution performance and saved to structured JSON files for programmatic analysis.

Trajectory Report Copy Section

Galaxy automatically generates a comprehensive Markdown trajectory report (output.md) documenting the complete execution lifecycle:

logs/galaxy/<task_name>/output.md

This human-readable report includes:

• Step-by-step execution timeline with agent actions
• Interactive DAG topology visualizations showing constellation evolution
• Detailed task execution logs with results and errors
• Device connection status and coordination events
• Complete before/after constellation states at each step

The trajectory report provides visual debugging and workflow understanding, complementing the quantitative result.json metrics.

Result JSON Format Copy Section

After each session, Galaxy also generates a comprehensive result.json file containing:

logs/galaxy/<task_name>/result.json

This file includes:

• Complete session metadata and execution timeline
• Task-by-task performance breakdown
• Constellation statistics (parallelism ratio, critical path, max concurrency)
• Modification history showing DAG evolution
• Final results and outcomes

Example Key Metrics:

Performance Analysis Tools Copy Section

import json

# Load session results
with open("logs/galaxy/task_32/result.json", 'r') as f:
    result = json.load(f)

# Extract key metrics
metrics = result["session_results"]["metrics"]
task_stats = metrics["task_statistics"]
const_stats = result["session_results"]["final_constellation_stats"]

print(f"✅ Success Rate: {task_stats['success_rate'] * 100:.1f}%")
print(f"⏱️  Avg Task Duration: {task_stats['average_task_duration']:.2f}s")
print(f"🔀 Parallelism Ratio: {const_stats['parallelism_ratio']:.2f}")

Documentation:

• Trajectory Report Guide - Complete guide to the human-readable execution log with DAG visualizations
• Performance Metrics Guide - Comprehensive metrics documentation with analysis examples
• Result JSON Reference - Complete schema reference and programmatic access guide

📚 Learn More Copy Chapter

• Research Paper: UFO³: Weaving the Digital Agent Galaxy (Coming Soon)
• UFO² (Desktop AgentOS): Documentation
• UFO (Original): GitHub Repository

🤝 Contributing Copy Chapter

We welcome contributions! Whether you're building new device agents, improving orchestration algorithms, or enhancing the protocol, check out our Contributing Guide on GitHub.

📄 License Copy Chapter

UFO³ Galaxy is released under the MIT License.

Documents Docs Linux Overview (`documents/docs/linux/overview.md`) Copy Section

LinuxAgent: CLI Task Executor

LinuxAgent is a specialized lightweight agent designed for executing command-line instructions on Linux systems. It demonstrates how a standalone device agent can leverage the layered FSM architecture and server-client design to perform intelligent, iterative task execution in a CLI-based environment.

Quick Links:

• New to Linux Agent? Start with the Quick Start Guide
• Using as Sub-Agent in Galaxy? See Using Linux Agent as Galaxy Device

Architecture Overview Copy Chapter

LinuxAgent operates as a single-agent instance that interacts with Linux systems through command-line interface (CLI) commands. Unlike the two-tier architecture of UFO (HostAgent + AppAgent), LinuxAgent uses a simplified single-agent model optimized for shell-based automation.

Core Responsibilities Copy Chapter

LinuxAgent provides the following capabilities for Linux CLI automation:

Command-Line Execution Copy Section

LinuxAgent interprets user requests and translates them into appropriate shell commands for execution on Linux systems.

Example: User request "Check disk space and create a backup" becomes:

1. Execute df -h to check disk space
2. Execute tar -czf backup.tar.gz /data to create backup

System Information Collection Copy Section

The agent can proactively gather system-level information to inform decision-making:

• Memory usage (free -h)
• Disk space (df -h)
• Process status (ps aux)
• Hardware configuration (lscpu, lshw)

Iterative Task Execution Copy Section

LinuxAgent executes tasks iteratively, evaluating execution outcomes at each step and determining the next action based on results and LLM reasoning.

Error Handling and Recovery Copy Section

The agent monitors command execution results (stdout, stderr, exit codes) and can adapt its strategy when errors occur.

Key Characteristics Copy Chapter

• Scope: Single Linux system (CLI-based automation)
• Lifecycle: One instance per task session
• Hierarchy: Standalone agent (no child agents)
• Communication: Direct MCP server integration
• Control: 3-state finite state machine with 3-phase processing pipeline

Execution Workflow Copy Chapter

sequenceDiagram
    participant User
    participant LinuxAgent
    participant LLM
    participant MCPServer
    participant Linux
    
    User->>LinuxAgent: "Check disk space and create backup"
    LinuxAgent->>LinuxAgent: State: CONTINUE
    LinuxAgent->>LLM: Send prompt with request & context
    LLM-->>LinuxAgent: Return command: df -h
    LinuxAgent->>MCPServer: execute_command: df -h
    MCPServer->>Linux: Execute command
    Linux-->>MCPServer: stdout + stderr
    MCPServer-->>LinuxAgent: Execution result
    LinuxAgent->>LinuxAgent: Update memory
    
    LinuxAgent->>LLM: Send prompt with previous result
    LLM-->>LinuxAgent: Return command: tar -czf ...
    LinuxAgent->>MCPServer: execute_command: tar -czf backup.tar.gz /data
    MCPServer->>Linux: Execute command
    Linux-->>MCPServer: stdout + stderr
    MCPServer-->>LinuxAgent: Execution result
    LinuxAgent->>LinuxAgent: State: FINISH
    LinuxAgent-->>User: Task completed

Comparison with UFO Agents Copy Chapter

Design Principles Copy Chapter

LinuxAgent exemplifies a minimal viable design for single-agent systems with a small state set (only 3 states for deterministic control flow), modular strategies (clear separation between LLM interaction, action execution, and memory updates), well-defined commands (atomic CLI operations isolated in MCP server layer), proactive information gathering (on-demand system info collection), and traceable execution (complete logging of commands, results, and state transitions).

Deep Dive Topics Copy Chapter

Explore the detailed architecture and implementation:

• State Machine - 3-state FSM lifecycle and transitions
• Processing Strategy - 3-phase pipeline (LLM, Action, Memory)
• MCP Commands - CLI execution and system information commands

Use Cases Copy Chapter

LinuxAgent is ideal for:

• System Administration: Automated system maintenance and monitoring
• DevOps Tasks: Deployment scripts, log analysis, configuration management
• Data Processing: File operations, text processing, batch jobs
• Monitoring & Alerts: System health checks and automated responses
• Cross-Device Workflows: As a sub-agent in Galaxy multi-device orchestration

!!!tip "Galaxy Integration"
LinuxAgent can serve as a device agent in Galaxy's multi-device orchestration framework, executing Linux-specific tasks as part of cross-platform workflows alongside Windows and other devices.

See [Using Linux Agent as Galaxy Device](as_galaxy_device.md) for configuration details.

Implementation Location Copy Chapter

The LinuxAgent implementation can be found in:

ufo/
├── agents/
│   ├── agent/
│   │   └── customized_agent.py          # LinuxAgent class definition
│   ├── states/
│   │   └── linux_agent_state.py         # State machine implementation
│   └── processors/
│       ├── customized/
│       │   └── customized_agent_processor.py  # LinuxAgentProcessor
│       └── strategies/
│           └── linux_agent_strategy.py  # Processing strategies

Next Steps Copy Chapter

To understand LinuxAgent's complete architecture:

1. State Machine - Learn about the 3-state FSM
2. Processing Strategy - Understand the 3-phase pipeline
3. MCP Commands - Explore CLI command execution

For deployment and configuration, see the Getting Started Guide.

Documents Docs Mcp Overview (`documents/docs/mcp/overview.md`) Copy Section

MCP (Model Context Protocol) - Overview

What is MCP? Copy Chapter

MCP (Model Context Protocol) is a standardized protocol that enables UFO² agents to interact with external tools and services in a unified way. It provides a tool execution framework where agents can:

• Collect system state through data collection servers
• Execute actions through action servers
• Extend capabilities through custom MCP servers

graph TB
    subgraph Agent["UFO² Agent"]
        HostAgent[HostAgent]
        AppAgent[AppAgent]
    end
    
    Computer["Computer<br/>(MCP Tool Manager)"]
    
    subgraph DataServers["Data Collection Servers"]
        UICollector["UICollector<br/>• Screenshots<br/>• Window Info"]
        HWInfo["Hardware Info<br/>• CPU/Memory<br/>• System State"]
    end
    
    subgraph ActionServers["Action Servers"]
        UIExecutor["UIExecutor<br/>• Click/Type<br/>• UI Automation"]
        CLIExecutor["CLI Executor<br/>• Shell Commands"]
        COMExecutor["COM Executor<br/>• API Calls"]
    end
    
    HostAgent --> Computer
    AppAgent --> Computer
    Computer --> DataServers
    Computer --> ActionServers
    
    style Agent fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
    style Computer fill:#fff3e0,stroke:#f57c00,stroke-width:2px
    style DataServers fill:#e8f5e9,stroke:#388e3c,stroke-width:2px
    style ActionServers fill:#fce4ec,stroke:#c2185b,stroke-width:2px

MCP serves as the execution layer in UFO²'s architecture. While agents make decisions about what to do, MCP servers handle how to do it by providing concrete tool implementations.

Key Concepts Copy Chapter

1. Two Server Types Copy Section

MCP servers in UFO² are categorized into two types based on their purpose:

Server Selection Model:

• Data Collection Servers: Automatically invoked by the framework to gather context and build observation prompts. Not selectable by LLM.
• Action Servers: LLM agent actively selects which action tool to execute at each step based on the task. Only action tools are LLM-selectable.

How Action Tools Reach the LLM: Each action tool's Annotated type hints and docstring are automatically extracted and converted into structured instructions that appear in the LLM's prompt. The LLM uses these instructions to understand what each tool does, what parameters it requires, and when to use it. Therefore, developers should write clear, comprehensive docstrings and type annotations - they directly impact the LLM's ability to use the tool correctly.

2. Server Deployment Models Copy Section

UFO² supports three deployment models for MCP servers:

3. Namespace Isolation Copy Section

Each MCP server has a namespace that groups related tools together:

# Example: HostAgent configuration
HostAgent:
  default:
    data_collection:
      - namespace: UICollector          # Namespace for UI detection tools
        type: local
    action:
      - namespace: HostUIExecutor       # Namespace for UI automation tools
        type: local
      - namespace: CommandLineExecutor  # Namespace for CLI tools
        type: local

Tool Key Format: {tool_type}::{tool_name}

• Example: data_collection::screenshot - Screenshot tool in data_collection
• Example: action::click - Click tool in action
• Example: action::run_shell - Shell command in action

Key Features Copy Chapter

1. GUI + API Dual-Mode Automation Copy Section

UFO² supports both GUI automation and API-based automation simultaneously. Each agent can register multiple action servers, combining:

• GUI Automation: Windows UI Automation (UIA) - clicking, typing, scrolling when visual interaction is needed
• API Automation: Direct COM API calls, shell commands, REST APIs for efficient, reliable operations

The LLM agent dynamically chooses the best action at each step based on task requirements, reliability, speed, and availability.

Example: Word Document Automation

AppAgent:
  WINWORD.EXE:
    action:
      - namespace: WordCOMExecutor      # API: Fast, reliable
      - namespace: AppUIExecutor        # GUI: Visual navigation fallback  
      - namespace: CommandLineExecutor  # Shell: File operations

LLM's Dynamic Selection:

Task: "Create a report with a 3x2 table and bold the title"

Step 1: Insert table
  → LLM selects: WordCOMExecutor::insert_table(rows=3, cols=2)
  → Reason: API is fast, reliable, no GUI navigation needed

Step 2: Navigate to Design tab  
  → LLM selects: AppUIExecutor::click_input(id="5", name="Design")
  → Reason: Visual navigation, COM API doesn't expose tab selection

Step 3: Type table header
  → LLM selects: AppUIExecutor::set_edit_text(id="cell_1_1", text="Product")
  → Reason: GUI interaction needed for cell input

Step 4: Bold title text
  → LLM selects: WordCOMExecutor::select_text(text="Report Title")
  →            WordCOMExecutor::set_font(font_size=16)
  → Reason: API is more reliable than GUI button clicking

Step 5: Save as PDF
  → LLM selects: WordCOMExecutor::save_as(file_ext=".pdf")
  → Reason: One API call vs. multiple GUI clicks (File → Save As → Format → PDF)

Why Hybrid Automation Matters:

• APIs: ~10x faster, deterministic, no visual dependency
• GUI: Handles visual elements, fallback when API unavailable
• LLM Decision: Chooses optimal approach per step, not locked into one mode

2. Multi-Server Per Agent Copy Section

Each agent can register multiple action servers, each providing a different set of tools:

HostAgent Example:

HostAgent:
  default:
    data_collection:
      - UICollector           # Automatically invoked
    action:
      - HostUIExecutor        # LLM selects: Window selection
      - CommandLineExecutor   # LLM selects: Launch apps, shell commands

AppAgent Example (Word-specific):

AppAgent:
  WINWORD.EXE:
    data_collection:
      - UICollector           # Automatically invoked
    action:
      - WordCOMExecutor       # LLM selects: insert_table, select_text, save_as
      - AppUIExecutor         # LLM selects: click_input, set_edit_text
      - CommandLineExecutor   # LLM selects: run_shell

HardwareAgent Example (Cross-platform):

HardwareAgent:
  default:
    data_collection:
      - HardwareCollector     # Auto-invoked (HTTP remote)
    action:
      - HardwareExecutor      # LLM selects: touch_screen, swipe, press_key (HTTP remote)

At each step, the LLM sees all available action tools and selects the most appropriate one.

3. Process Isolation Copy Section

MCP servers can run:

• In-process (local): Fast, low overhead
• HTTP (remote): Process isolation, cross-platform, distributed
• Stdio (child process): Sandboxed execution, clean resource management

4. Namespace Isolation Copy Section

Each MCP server has a unique namespace that groups related tools together, preventing naming conflicts and enabling modular organization. See Namespace Isolation section above for details.

Architecture Copy Chapter

MCP Server Lifecycle Copy Section

graph TB
    Start([MCP Server Lifecycle])
    
    Config["1. Configuration Loading<br/>(mcp.yaml)"]
    Manager["2. MCPServerManager<br/>Creates BaseMCPServer"]
    ServerStart["3. Server.start()<br/>• Local: Get from registry<br/>• HTTP: Build URL<br/>• Stdio: Spawn process"]
    Register["4. Computer Registration<br/>• List tools from server<br/>• Register in tool registry"]
    Execute["5. Tool Execution<br/>• Agent sends Command<br/>• Computer routes to tool<br/>• MCP server executes"]
    Reset["6. Server.reset() (optional)<br/>Reset server state"]
    
    Start --> Config
    Config --> Manager
    Manager --> ServerStart
    ServerStart --> Register
    Register --> Execute
    Execute --> Reset
    
    style Start fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
    style Config fill:#e1f5fe,stroke:#0277bd,stroke-width:2px
    style Manager fill:#fff9c4,stroke:#f57f17,stroke-width:2px
    style ServerStart fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
    style Register fill:#fce4ec,stroke:#c2185b,stroke-width:2px
    style Execute fill:#e0f2f1,stroke:#00695c,stroke-width:2px
    style Reset fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px

Component Relationships Copy Section

graph TB
    subgraph Architecture["MCP Architecture"]
        Registry["MCPRegistry (Singleton)<br/>• Stores server factories<br/>• Lazy initialization"]
        
        Manager["MCPServerManager (Singleton)<br/>• Creates server instances<br/>• Maps server types to classes<br/>• Manages server lifecycle"]
        
        subgraph ServerTypes["MCP Server Types"]
            Local["Local MCP Server<br/>(In-Process)"]
            HTTP["HTTP MCP Server<br/>(Remote)"]
            Stdio["Stdio MCP Server<br/>(Child Process)"]
        end
        
        Computer["Computer (Per Agent)<br/>• Manages multiple MCP servers<br/>• Routes commands to tools<br/>• Maintains tool registry"]
        
        Registry --> Manager
        Manager --> ServerTypes
        Manager --> Computer
    end
    
    style Architecture fill:#fafafa,stroke:#424242,stroke-width:2px
    style Registry fill:#e1f5fe,stroke:#01579b,stroke-width:2px
    style Manager fill:#fff9c4,stroke:#f57f17,stroke-width:2px
    style ServerTypes fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px
    style Local fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
    style HTTP fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
    style Stdio fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
    style Computer fill:#fce4ec,stroke:#880e4f,stroke-width:2px

Built-in MCP Servers Copy Chapter

UFO² comes with several built-in MCP servers that cover common automation scenarios:

Data Collection Servers Copy Section

Action Servers Copy Section

!!!example "Tool Examples"
```python
# Data Collection: Take a screenshot
{
"tool_type": "data_collection",
"tool_name": "take_screenshot",
"parameters": {"region": "active_window"}
}

# Action: Click a button
{
    "tool_type": "action",
    "tool_name": "click",
    "parameters": {"control_label": "Submit"}
}

# Action: Run a shell command
{
    "tool_type": "action",
    "tool_name": "run_shell",
    "parameters": {"bash_command": "notepad.exe"}
}
```

Agent-Specific Configurations Copy Chapter

Different agents can have different MCP configurations based on their roles:

# HostAgent: System-level operations
HostAgent:
  default:
    data_collection:
      - namespace: UICollector
        type: local
    action:
      - namespace: HostUIExecutor
        type: local
      - namespace: CommandLineExecutor
        type: local

# AppAgent: Application-specific operations
AppAgent:
  WINWORD.EXE:  # Word-specific configuration
    data_collection:
      - namespace: UICollector
        type: local
    action:
      - namespace: AppUIExecutor
        type: local
      - namespace: WordCOMExecutor  # Word COM API
        type: local
        reset: true  # Reset when switching documents

# HardwareAgent: Remote hardware monitoring
HardwareAgent:
  default:
    data_collection:
      - namespace: HardwareCollector
        type: http  # Remote server
        host: "localhost"
        port: 8006
        path: "/mcp"

# MobileAgent: Android device automation
MobileAgent:
  default:
    data_collection:
      - namespace: MobileDataCollector
        type: http  # Remote server
        host: "localhost"
        port: 8020
        path: "/mcp"
    action:
      - namespace: MobileExecutor
        type: http
        host: "localhost"
        port: 8021
        path: "/mcp"

Configuration Hierarchy:

Agent configurations follow this hierarchy:

1. Agent Name (e.g., HostAgent, AppAgent)
2. Sub-type (e.g., default, WINWORD.EXE)
3. Tool Type (e.g., data_collection, action)
4. Server List (array of server configurations)

Key Features Copy Chapter

1. Process Isolation with Reset Copy Section

Some MCP servers support state reset to ensure clean execution:

AppAgent:
  WINWORD.EXE:
    action:
      - namespace: WordCOMExecutor
        type: local
        reset: true

**When to use reset:**

- Server state is cleared when switching contexts
- Prevents state leakage between tasks
- Useful for stateful tools (e.g., COM APIs)

### 2. Thread Isolation

MCP tools execute in **isolated thread pools** to prevent blocking:

```python
# From Computer class
self._executor = ThreadPoolExecutor(max_workers=10)
self._tool_timeout = 6000  # 100 minutes

Benefits:

• Prevents blocking the main event loop
• Protects WebSocket connections from timeouts
• Enables concurrent tool execution

Timeout Protection: If a tool takes longer than 6000 seconds, it will be cancelled and return a timeout error. Adjust _tool_timeout for long-running operations.

3. Dynamic Server Management Copy Section

Add or remove MCP servers at runtime:

# Add a custom server
await computer.add_server(
    namespace="CustomTools",
    mcp_server=custom_server,
    tool_type="action"
)

# Remove a server
await computer.delete_server(
    namespace="CustomTools",
    tool_type="action"
)

4. Tool Introspection Copy Section

Use meta tools to discover available tools:

# List all action tools
tool_call = MCPToolCall(
    tool_key="action::list_tools",
    tool_name="list_tools",
    parameters={"tool_type": "action"}
)

result = await computer.run_actions([tool_call])
# Returns: List of all available action tools

For more details on introspection capabilities, see Computer - Meta Tools.

Configuration Files Copy Chapter

MCP configuration is located at:

config/ufo/mcp.yaml

For detailed configuration options, see:

• MCP Configuration Guide - Complete configuration reference
• System Configuration - MCP-related system settings
• MCP Reference - MCP-specific settings

Use Cases Copy Chapter

1. UI Automation Copy Section

# Agent that automates UI interactions
HostAgent:
  default:
    data_collection:
      - namespace: UICollector        # Detect UI elements
    action:
      - namespace: HostUIExecutor     # Click, type, scroll

2. Document Processing Copy Section

# Agent specialized for Word documents
AppAgent:
  WINWORD.EXE:
    data_collection:
      - namespace: UICollector        # Read document structure
    action:
      - namespace: WordCOMExecutor    # Insert tables, format text

3. Multi-Device Coordination Copy Section

# Agent that coordinates tasks across devices
ConstellationAgent:
  default:
    action:
      - namespace: ConstellationEditor  # Create and assign tasks

4. Remote Hardware Monitoring Copy Section

# Agent that monitors remote hardware
HardwareAgent:
  default:
    data_collection:
      - namespace: HardwareCollector
        type: http
        host: "192.168.1.100"
        port: 8006

5. Android Device Automation Copy Section

# Agent that automates Android devices via ADB
MobileAgent:
  default:
    data_collection:
      - namespace: MobileDataCollector
        type: http
        host: "localhost"  # Or remote Android automation server
        port: 8020
        path: "/mcp"
    action:
      - namespace: MobileExecutor
        type: http
        host: "localhost"
        port: 8021
        path: "/mcp"

Getting Started Copy Chapter

To start using MCP in UFO²:

1. Understand the two server types - Read about Data Collection and Action servers
2. Configure your agents - See Configuration Guide for setup details
3. Use built-in servers - Explore available Local Servers
4. Create custom servers - Follow the Creating Custom MCP Servers Tutorial
5. Deploy remotely - Learn about Remote Servers deployment

Related Documentation Copy Chapter

• Data Collection Servers - Read-only observation tools
• Action Servers - State-changing execution tools
• Configuration Guide - How to configure MCP for agents
• Local Servers - Built-in MCP servers
• Remote Servers - HTTP and Stdio deployment
• Creating Custom MCP Servers Tutorial - Step-by-step guide to building custom servers
• Computer - MCP tool execution layer
• Agent Client - Client architecture overview
• Agent Overview - UFO² agent system architecture

Design Philosophy:

MCP in UFO² follows the separation of concerns principle:

• Agents decide what to do (high-level planning)
• MCP servers implement how to do it (low-level execution)
• Computer manages the routing between them (middleware)

This architecture enables flexibility, extensibility, and maintainability.

Documents Docs Mobile Overview (`documents/docs/mobile/overview.md`) Copy Section

MobileAgent: Android Task Executor

MobileAgent is a specialized agent designed for executing tasks on Android mobile devices. It leverages the layered FSM architecture and server-client design to perform intelligent, iterative task execution in mobile environments through UI interaction and app control.

Quick Links:

• New to Mobile Agent? Start with the Quick Start Guide - Set up your first Android device agent in 10 minutes
• Using as Sub-Agent in Galaxy? See Using Mobile Agent as Galaxy Device
• Deep Dive: State Machine • Processing Strategy • MCP Commands

Architecture Overview Copy Chapter

MobileAgent operates as a single-agent instance that interacts with Android devices through UI controls and app management. Unlike the two-tier architecture of UFO (HostAgent + AppAgent), MobileAgent uses a simplified single-agent model optimized for mobile device automation, similar to LinuxAgent but with visual interface support.

Core Responsibilities Copy Chapter

MobileAgent provides the following capabilities for Android device automation:

UI Interaction Copy Section

MobileAgent interprets user requests and translates them into appropriate UI interactions on Android devices through screenshots analysis and control manipulation.

Example: User request "Search for restaurants on Maps" becomes:

1. Take screenshot and identify app icons
2. Launch Google Maps app
3. Identify search field control
4. Type "restaurants" into search field
5. Tap search button

Visual Context Understanding Copy Section

The agent captures and analyzes device screenshots to understand the current UI state:

• Screenshot capture (clean and annotated)
• Control identification and labeling
• UI hierarchy parsing
• App detection and recognition

App Management Copy Section

MobileAgent can manage installed applications:

• List installed apps (user apps and system apps)
• Launch apps by package name or app name
• Switch between apps
• Monitor current app state

Iterative Task Execution Copy Section

MobileAgent executes tasks iteratively, evaluating execution outcomes at each step and determining the next action based on results and LLM reasoning.

Error Handling and Recovery Copy Section

The agent monitors action execution results and can adapt its strategy when errors occur, such as controls not found or apps not responding.

Key Characteristics Copy Chapter

• Scope: Single Android device (UI-based automation)
• Lifecycle: One instance per task session
• Hierarchy: Standalone agent (no child agents)
• Communication: MCP server integration via ADB
• Control: 3-state finite state machine with 4-phase processing pipeline
• Visual: Screenshot-based UI understanding with control annotation

Execution Workflow Copy Chapter

/* Detailed source-code truncated for AI context efficiency. */

Comparison with Other Agents Copy Chapter

Design Principles Copy Chapter

MobileAgent exemplifies mobile-specific design considerations:

• Visual Context: Screenshot-based UI understanding with control annotation for precise interaction
• Control Caching: Efficient control information caching to reduce ADB overhead
• Touch-based Interaction: Specialized actions for mobile gestures (tap, swipe, long-press)
• App-centric Navigation: Focus on app launching and switching rather than window management
• Minimal State Set: 3-state FSM for deterministic control flow
• Modular Strategies: Clear separation between data collection, LLM interaction, action execution, and memory updates
• Traceable Execution: Complete logging of screenshots, actions, and state transitions

Deep Dive Topics Copy Chapter

Explore the detailed architecture and implementation:

• State Machine - 3-state FSM lifecycle and transitions
• Processing Strategy - 4-phase pipeline (Data Collection, LLM, Action, Memory)
• MCP Commands - Mobile UI interaction and app management commands
• As Galaxy Device - Using Mobile Agent in multi-device workflows

Technology Stack Copy Chapter

ADB (Android Debug Bridge) Copy Section

MobileAgent relies on ADB for all device interactions:

• Screenshot Capture: adb shell screencap for visual context
• UI Hierarchy: adb shell uiautomator dump for control information
• Touch Input: adb shell input tap/swipe for user interaction
• Text Input: adb shell input text for typing
• App Control: adb shell monkey for app launching
• Device Info: adb shell getprop for device properties

MCP Server Architecture Copy Section

Two separate MCP servers handle different responsibilities:

1. Data Collection Server (Port 8020):

   • Screenshot capture
   • UI tree retrieval
   • App list collection
   • Control information gathering
   • Device information

2. Action Server (Port 8021):

   • Touch actions (tap, swipe)
   • Text input
   • App launching
   • Key press events
   • Control clicking

Both servers share a singleton MobileServerState for efficient caching and coordination.

Use Cases Copy Chapter

MobileAgent is ideal for:

• Mobile App Testing: Automated UI testing across different apps
• Cross-App Workflows: Tasks spanning multiple mobile applications
• Data Entry: Automated form filling and text input
• App Navigation: Exploring and interacting with mobile UIs
• Mobile Productivity: Automating repetitive mobile tasks
• Cross-Device Workflows: As a sub-agent in Galaxy multi-device orchestration

!!!tip "Galaxy Integration"
MobileAgent can serve as a device agent in Galaxy's multi-device orchestration framework, executing Android-specific tasks as part of cross-platform workflows alongside Windows and Linux devices.

See [Using Mobile Agent as Galaxy Device](as_galaxy_device.md) for configuration details.

Requirements Copy Chapter

Hardware Copy Section

• Android device or emulator
• USB connection (for physical devices) or network connection (for emulators)
• USB debugging enabled on the device

Software Copy Section

• ADB (Android Debug Bridge) installed and accessible
• Android device with API level 21+ (Android 5.0+)
• Python 3.8+
• Required Python packages (see requirements.txt)

Implementation Location Copy Chapter

The MobileAgent implementation can be found in:

ufo/
├── agents/
│   ├── agent/
│   │   └── customized_agent.py          # MobileAgent class definition
│   ├── states/
│   │   └── mobile_agent_state.py        # State machine implementation
│   └── processors/
│       ├── customized/
│       │   └── customized_agent_processor.py  # MobileAgentProcessor
│       └── strategies/
│           └── mobile_agent_strategy.py # Processing strategies
├── prompter/
│   └── customized/
│       └── mobile_agent_prompter.py     # Prompt construction
├── module/
│   └── sessions/
│       └── mobile_session.py            # Session management
└── client/
    └── mcp/
        └── http_servers/
            └── mobile_mcp_server.py     # MCP server implementation

Next Steps Copy Chapter

To understand MobileAgent's complete architecture:

1. State Machine - Learn about the 3-state FSM
2. Processing Strategy - Understand the 4-phase pipeline
3. MCP Commands - Explore mobile UI interaction commands
4. As Galaxy Device - Configure for multi-device workflows

For deployment and configuration, see the Quick Start Guide (coming soon).

Documents Docs Server Overview (`documents/docs/server/overview.md`) Copy Section

Agent Server Overview

The Agent Server is the central orchestration engine that transforms UFO into a distributed multi-agent system, enabling seamless task coordination across heterogeneous devices through persistent WebSocket connections and robust state management.

New to the Agent Server? Start with the Quick Start Guide to get up and running in minutes.

What is the Agent Server? Copy Chapter

The Agent Server is a FastAPI-based asynchronous WebSocket server that serves as the communication hub for UFO's distributed architecture. It bridges constellation orchestrators, device agents, and external systems through a unified protocol interface.

Core Responsibilities Copy Section

Why Use the Agent Server?

• Centralized Control: Single point of orchestration for multi-device workflows
• Protocol Abstraction: Clients communicate via AIP, hiding network complexity
• Async by Design: Non-blocking execution enables high concurrency
• Platform Agnostic: Supports Windows, Linux, macOS (in development)

The Agent Server is part of UFO's distributed server-client architecture, where it handles orchestration and state management while Agent Clients handle command execution. See Server-Client Architecture for the complete design rationale and communication patterns.

Architecture Copy Chapter

The server follows a clean separation of concerns with distinct layers for web service, connection management, and protocol handling.

Architectural Overview Copy Section

Component Interaction Diagram:

graph TB
    subgraph "Web Layer"
        FastAPI[FastAPI App]
        HTTP[HTTP API]
        WS[WebSocket /ws]
    end
    
    subgraph "Service Layer"
        WSM[Client Manager]
        SM[Session Manager]
        WSH[WebSocket Handler]
    end
    
    subgraph "Clients"
        DC[Device Clients]
        CC[Constellation Clients]
    end
    
    FastAPI --> HTTP
    FastAPI --> WS
    
    HTTP --> SM
    HTTP --> WSM
    WS --> WSH
    
    WSH --> WSM
    WSH --> SM
    
    DC -->|WebSocket| WS
    CC -->|WebSocket| WS
    
    style FastAPI fill:#e1f5ff
    style WSM fill:#fff4e1
    style SM fill:#f0ffe1
    style WSH fill:#ffe1f5

This layered design ensures each component has a single, well-defined responsibility. The managers maintain state while the handler implements protocol logic.

Core Components Copy Section

Component Documentation:

• Session Manager - Session lifecycle and background execution
• Client Connection Manager - Connection registry and client tracking
• WebSocket Handler - AIP protocol message handling
• HTTP API - REST endpoint specifications

Key Capabilities Copy Chapter

1. Multi-Client Coordination Copy Section

The server supports two distinct client types with different roles in the distributed architecture.

Client Type Comparison:

Device Clients

• Execute tasks locally on Windows/Linux machines
• Report hardware specs and real-time status
• Respond to commands via MCP tool servers
• Stream execution logs back to server

See Agent Client Overview for detailed client architecture.

Constellation Clients

• Orchestrate multi-device workflows from a central point
• Dispatch tasks to specific target devices via target_id
• Coordinate complex cross-device DAG execution
• Aggregate results from multiple devices

Both client types connect to /ws and register using the REGISTER message. The server differentiates behavior based on client_type field. For the complete server-client architecture and design rationale, see Server-Client Architecture.

See Quick Start for registration examples.

2. Session Lifecycle Management Copy Section

Unlike stateless HTTP servers, the Agent Server maintains session state throughout task execution, enabling multi-turn interactions and result callbacks.

Session Lifecycle State Machine:

stateDiagram-v2
    [*] --> Created: create_session()
    Created --> Running: Start execution
    Running --> Completed: Success
    Running --> Failed: Error
    Running --> Cancelled: Disconnect
    Completed --> [*]
    Failed --> [*]
    Cancelled --> [*]
    
    note right of Running
        Async background execution
        Non-blocking server
    end note

Lifecycle Stages:

!!!warning "Platform-Specific Sessions"
The SessionManager creates different session types based on the target platform:
- Windows: WindowsSession with UI automation support
- Linux: LinuxSession with bash automation
- Auto-detected or overridden via --platform flag

Session Manager Responsibilities:

• ✅ Platform abstraction: Hides Windows/Linux differences
• ✅ Background execution: Non-blocking async task execution
• ✅ Callback routing: Delivers results via WebSocket
• ✅ Resource cleanup: Cancels tasks on disconnect
• ✅ Result caching: Stores results for HTTP retrieval

3. Resilient Communication Copy Section

The server implements the Agent Interaction Protocol (AIP), providing structured, type-safe communication with automatic failure handling.

Protocol Features:

Disconnection Handling Flow:

sequenceDiagram
    participant Client
    participant Server
    participant SM as Session Manager
    
    Client-xServer: Connection lost
    Server->>SM: Cancel sessions
    SM->>SM: Cleanup resources
    Server->>Server: Remove from registry
    
    Note over Server: Client can reconnectwith same client_id

!!!danger "Important: Session Cancellation on Disconnect"
When a client disconnects (device or constellation), all associated sessions are immediately cancelled to prevent orphaned tasks and resource leaks.

4. Dual API Interface Copy Section

The server provides two API styles to support different integration patterns: real-time WebSocket for agents and simple HTTP for external systems.

WebSocket API (AIP-based)

Purpose: Real-time bidirectional communication with agent clients

!!!example "WebSocket Connection"
```python
import websockets

async with websockets.connect("ws://localhost:5000/ws") as ws:
    # Register as device client
    await ws.send(json.dumps({
        "message_type": "REGISTER",
        "client_id": "windows_agent_001",
        "client_type": "device",
        "metadata": {"platform": "windows", "gpu": "NVIDIA RTX 3080"}
    }))
```

HTTP REST API

Purpose: Task dispatch and monitoring from external systems (HTTP clients, CI/CD, etc.)

!!!example "HTTP Task Dispatch"
```bash
# Dispatch task to device
curl -X POST http://localhost:5000/api/dispatch
-H "Content-Type: application/json"
-d '{
"client_id": "my_windows_device",
"request": "Open Notepad and type Hello World",
"task_name": "test_task_001"
}'

# Response: {"status": "dispatched", "session_id": "session_abc123", "task_name": "test_task_001"}

# Retrieve results
curl http://localhost:5000/api/task_result/test_task_001
```

See HTTP API Reference for complete endpoint documentation.

Workflow Examples Copy Chapter

Complete Task Dispatch Flow Copy Section

End-to-End HTTP WebSocket Device Execution:

sequenceDiagram
    participant EXT as External System
    participant HTTP as HTTP API
    participant SM as Session Manager
    participant WSH as WebSocket Handler
    participant DC as Device Client
    
    EXT->>HTTP: POST /api/dispatch{client_id, request, task_name}
    HTTP->>SM: create_session()
    SM->>SM: Create platform session
    SM-->>HTTP: session_id
    HTTP-->>EXT: 200 {session_id, task_name}
    
    SM->>WSH: send_task(session_id, task)
    WSH->>DC: TASK message (AIP)
    DC-->>WSH: ACK
    
    rect rgb(240, 255, 240)
        Note over DC: Background Execution
        DC->>DC: Execute via MCP tools
        DC->>DC: Generate results
    end
    
    DC->>WSH: COMMAND_RESULTS
    WSH->>SM: on_result_callback()
    SM->>SM: Cache results
    
    DC->>WSH: TASK_END (COMPLETED)
    WSH->>SM: on_task_end()
    
    EXT->>HTTP: GET /task_result/{session_id}
    HTTP->>SM: get_results()
    SM-->>HTTP: results
    HTTP-->>EXT: 200 {results}

The green box highlights async execution on the device side, which doesn't block the server.

Multi-Device Constellation Workflow Copy Section

Constellation Client Coordinating Multiple Devices:

sequenceDiagram
    participant CC as Constellation Client
    participant Server as Agent Server
    participant D1 as Device 1 (GPU)
    participant D2 as Device 2 (CPU)
    
    CC->>Server: REGISTER (constellation)
    Server-->>CC: HEARTBEAT (OK)
    
    Note over CC: Plan multi-device DAG
    
    CC->>Server: TASK (target: device_1)Subtask 1: Image processing
    Server->>D1: TASK (forward)
    
    CC->>Server: TASK (target: device_2)Subtask 2: Data extraction
    Server->>D2: TASK (forward)
    
    par Parallel Execution
        D1->>D1: Process image on GPU
        D2->>D2: Extract data from DB
    end
    
    D1->>Server: COMMAND_RESULTS
    Server->>CC: COMMAND_RESULTS (from device_1)
    
    D2->>Server: COMMAND_RESULTS
    Server->>CC: COMMAND_RESULTS (from device_2)
    
    Note over CC: Combine results,Update DAG
    
    D1->>Server: TASK_END
    D2->>Server: TASK_END
    Server->>CC: TASK_END (both tasks)

The server acts as a message router, forwarding tasks to target devices and routing results back to the constellation orchestrator. See Constellation Documentation for more details on multi-device orchestration.

Platform Support Copy Chapter

The server automatically detects client platforms and creates appropriate session implementations.

Supported Platforms:

Platform Auto-Detection:

The server automatically detects the client's platform during registration. You can override this globally with the --platform flag when needed for testing or specific deployment scenarios.

python -m ufo.server.app --platform windows  # Force Windows sessions
python -m ufo.server.app --platform linux    # Force Linux sessions
python -m ufo.server.app                     # Auto-detect (default)

!!!warning "When to Use Platform Override"
Use --platform override when:
- Testing cross-platform sessions without actual devices
- Running server in container different from target platform
- Debugging platform-specific session behavior

For more details on platform-specific implementations, see Windows Agent and Linux Agent.

Configuration Copy Chapter

The server runs out-of-the-box with sensible defaults. Advanced configuration inherits from UFO's central config system.

Command-Line Arguments Copy Section

python -m ufo.server.app [OPTIONS]

Available Options:

!!!example "Configuration Examples"
```bash
# Development: Local-only with debug logging
python -m ufo.server.app --local --log-level DEBUG --port 8000

# Production: External access, info logging
python -m ufo.server.app --host 0.0.0.0 --port 5000 --log-level INFO

# Testing: Force Linux sessions
python -m ufo.server.app --platform linux --port 9000
```

UFO Configuration Inheritance Copy Section

The server uses UFO's central configuration from config_dev.yaml:

See Configuration Guide for comprehensive config documentation.

Monitoring & Operations Copy Chapter

Health Monitoring Copy Section

Monitor server status and performance using HTTP endpoints.

Health Check Endpoints:

# Server health and uptime
curl http://localhost:5000/api/health

# Response:
# {
#   "status": "healthy",
#   "online_clients": [...]
# }

# Connected clients list
curl http://localhost:5000/api/clients

# Response:
# {
#   "online_clients": ["windows_001", "linux_002", ...]
# }

For comprehensive monitoring strategies including performance metrics collection, log aggregation patterns, alert configuration, and dashboard setup, see Monitoring Guide.

Error Handling Copy Section

The server handles common failure scenarios gracefully to maintain system stability.

Disconnection Handling Matrix:

!!!note "Reconnection Support"
Clients can reconnect with the same client_id. The server will re-register the client and restore heartbeat monitoring, but will not restore previous sessions (sessions are ephemeral).

Best Practices Copy Chapter

Development Environment Copy Section

Optimize your development workflow with these recommended practices.

Recommended Development Configuration:

# Isolate to localhost, enable detailed logging
python -m ufo.server.app \
  --host 127.0.0.1 \
  --port 5000 \
  --local \
  --log-level DEBUG

Development Checklist:

• Use --local flag to prevent external access
• Enable DEBUG logging for detailed traces
• Monitor logs in separate terminal: tail -f logs/ufo_server.log
• Test with single device before adding multiple clients
• Use HTTP API for quick task dispatch testing
• Verify heartbeat monitoring with client disconnection

!!!example "Development Testing Pattern"
```bash
# Terminal 1: Start server with debug logging
python -m ufo.server.app --local --log-level DEBUG

# Terminal 2: Connect device client
python -m ufo.client.client --ws --ws-server ws://127.0.0.1:5000/ws

# Terminal 3: Dispatch test task
curl -X POST http://127.0.0.1:5000/api/dispatch \
  -H "Content-Type: application/json" \
  -d '{"client_id": "windowsagent", "request": "Open Notepad", "task_name": "test_001"}'
```

Production Deployment Copy Section

The default configuration is not production-ready. Implement these security and reliability measures.

Production Architecture:

graph LR
    Internet[Internet]
    LB[Load Balancernginx/HAProxy]
    SSL[SSL/TLSTermination]
    
    subgraph "UFO Server Cluster"
        S1[Server Instance 1:5000]
        S2[Server Instance 2:5001]
        S3[Server Instance 3:5002]
    end
    
    Monitor[MonitoringPrometheus/Grafana]
    PM[Process Managersystemd/PM2]
    
    Internet --> LB
    LB --> SSL
    SSL --> S1
    SSL --> S2
    SSL --> S3
    
    PM -.Manages.-> S1
    PM -.Manages.-> S2
    PM -.Manages.-> S3
    
    S1 -.Metrics.-> Monitor
    S2 -.Metrics.-> Monitor
    S3 -.Metrics.-> Monitor
    
    style LB fill:#ffe1f5
    style SSL fill:#fff4e1
    style Monitor fill:#f0ffe1

Production Checklist:

Example Nginx Configuration:

upstream ufo_server {
    server localhost:5000;
}

server {
    listen 443 ssl;
    server_name ufo-server.example.com;
    
    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;
    
    # WebSocket endpoint
    location /ws {
        proxy_pass http://ufo_server;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_read_timeout 3600s;
    }
    
    # HTTP API
    location /api/ {
        proxy_pass http://ufo_server;
        proxy_set_header Host $host;
    }
}

Scaling Strategies Copy Section

The server can scale horizontally for high-load deployments, but requires careful session management.

Scaling Patterns:

!!!warning "Current Limitation"
The current implementation stores session state in-memory. For horizontal scaling, use sticky sessions (client affinity) in your load balancer to route clients to consistent server instances. Future: Shared state backend (Redis) for true stateless horizontal scaling.

Troubleshooting Copy Chapter

Common Issues Copy Section

Issue: Clients Can't Connect

# Symptom: Connection refused
Error: WebSocket connection to 'ws://localhost:5000/ws' failed

# Diagnosis:
1. Check server is running: curl http://localhost:5000/api/health
2. Verify port: netstat -an | grep 5000
3. Check firewall: sudo ufw status

# Solution:
# Start server with correct host binding
python -m ufo.server.app --host 0.0.0.0 --port 5000

Issue: Sessions Not Executing

# Symptom: Task dispatched but no results

# Diagnosis:
1. Check server logs for errors
2. Verify client is connected: curl http://localhost:5000/api/clients
3. Check target_id matches client_id

# Solution:
# Ensure client_id in request matches registered client
curl -X POST http://localhost:5000/api/dispatch \
  -d '{"client_id": "correct_client_id", "request": "test", "task_name": "test_001"}'

Issue: Memory Leak / High Memory Usage

# Symptom: Server memory grows over time

# Diagnosis:
1. Check session cleanup in logs
2. Monitor /api/health for session count
3. Profile with memory_profiler

# Solution:
# Ensure clients send TASK_END to complete sessions
# Restart server periodically (systemd handles this)
# Implement session timeout (future feature)

Debug Mode Copy Section

!!!example "Enable Maximum Verbosity"
```bash
# Ultra-verbose debugging
python -m ufo.server.app
--log-level DEBUG
--local
--port 5000 2>&1 | tee debug.log

# Watch logs in real-time
tail -f debug.log | grep -E "(ERROR|WARNING|Session|WebSocket)"
```

Documentation Map Copy Chapter

Explore related documentation to deepen your understanding of the Agent Server ecosystem.

Getting Started Copy Section

Architecture & Components Copy Section

Operations Copy Section

Related Documentation Copy Section

Next Steps Copy Chapter

Follow this recommended sequence to master the Agent Server:

1. Run the Server (5 minutes)

• Follow the Quick Start Guide
• Verify server responds to /api/health

2. Connect a Client (10 minutes)

• Use Device Client
• Verify registration in server logs
• Check /api/clients endpoint

3. Dispatch Tasks (15 minutes)

• Use HTTP API to send tasks
• Retrieve results via /api/task_result
• Observe WebSocket message flow in logs

4. Understand Architecture (30 minutes)

• Read Session Manager internals
• Study WebSocket Handler protocol implementation
• Review AIP Protocol message types

5. Deploy to Production (varies)

• Set up reverse proxy (nginx)
• Configure SSL/TLS
• Implement monitoring
• Test failover scenarios

Documents Docs Ufo2 Overview (`documents/docs/ufo2/overview.md`) Copy Section

UFO² — Windows AgentOS

UFO² is a Windows AgentOS that reimagines desktop automation as a first-class operating system abstraction. Unlike traditional Computer-Using Agents (CUAs) that rely on screenshots and simulated inputs, UFO² deeply integrates with Windows OS through UI Automation APIs, application-specific introspection, and hybrid GUI–API execution—enabling robust, efficient, and non-disruptive automation across 20+ real-world applications.

What is UFO²? Copy Chapter

UFO² addresses fundamental limitations of existing desktop automation solutions:

Traditional RPA (UiPath, Power Automate):
❌ Fragile scripts that break with UI changes
❌ Requires extensive manual maintenance
❌ Limited adaptability to dynamic environments

Current CUAs (Claude, Operator):
❌ Visual-only inputs with high cognitive overhead
❌ Miss native OS APIs and application internals
❌ Lock users out during automation (poor UX)

UFO² AgentOS:
✅ Deep OS Integration — Windows UIA, Win32, WinCOM APIs
✅ Hybrid GUI–API Actions — Native APIs + fallback GUI automation
✅ Continuous Knowledge Learning — RAG-enhanced from docs & execution history
✅ Picture-in-Picture Desktop — Parallel automation without user disruption
✅ 10%+ better success rate than state-of-the-art CUAs

Core Architecture Copy Chapter

UFO² implements a hierarchical multi-agent system optimized for Windows desktop automation:

Two-Tier Agent Hierarchy Copy Section

Design Philosophy:

• HostAgent handles WHAT (which application) and WHEN (task sequencing)
• AppAgent handles HOW (UI/API interaction) and WHERE (control targeting)
• Blackboard facilitates inter-agent communication without tight coupling
• State Machines ensure deterministic execution flow and error recovery

!!!info "Learn More"
- HostAgent Documentation — 7-state FSM, desktop orchestration, AppAgent lifecycle
- AppAgent Documentation — 6-state FSM, UI automation, hybrid action execution
- Agent Architecture — Three-layer design principles

Key Innovations Copy Chapter

1. Deep OS Integration 🔧 Copy Section

UFO² embeds directly into Windows OS infrastructure:

• UI Automation (UIA): Introspects accessibility trees for standard controls
• Win32 APIs: Low-level window management and process control
• WinCOM: Interacts with Office applications (Excel, Word, Outlook)
• Hybrid Detection: Fuses UIA metadata + visual grounding for non-standard UI elements

!!!tip "Hybrid Control Detection"
Combines Windows UIA APIs with vision models (OmniParser) to detect both standard and custom UI controls—bridging structured accessibility trees and pixel-level perception.

📖 [Control Detection Guide](core_features/control_detection/overview.md)

2. Unified GUI–API Action Layer ⚡ Copy Section

Traditional CUAs simulate mouse/keyboard only. UFO² chooses the best execution method:

GUI Actions (fallback):
click, type, select, scroll → Reliable for any application

Native APIs (preferred):

• Excel: xlwings for direct cell/chart manipulation
• Outlook: win32com for email operations
• PowerPoint: python-pptx for slide editing
  → 51% fewer LLM calls via speculative multi-action execution

Model Context Protocol (MCP) Servers:
Extensible framework for adding application-specific APIs without modifying agent code.

!!!info "Learn More"
📖 Hybrid Actions Guide • MCP Integration

3. Continuous Knowledge Substrate 📚 Copy Section

UFO² learns from three knowledge sources without model retraining:

Result: Agents improve autonomously by retrieving relevant context at execution time.

!!!info "Knowledge Integration"
📖 Knowledge Substrate Overview
📖 Learning from Help Documents
📖 Experience Learning

4. Speculative Multi-Action Execution 🚀 Copy Section

Reduce LLM latency by predicting and validating action sequences:

Traditional Approach:
1 LLM call → 1 action → observe → repeat → High latency

UFO² Speculative Execution:
1 LLM call → predict N actions → validate with UI state → execute all → 51% fewer queries

Validation Mechanism:
Lightweight control-state checks ensure predicted actions remain valid before execution.

!!!example "Efficiency Gain"
Task: "Fill form fields A1–A10 with sequential numbers"

- **Traditional CUA:** 10 LLM calls (1 per field) → ~30 seconds  
- **UFO² Speculative:** 1 LLM call predicts all 10 actions → ~8 seconds

📖 [Multi-Action Execution Guide](core_features/multi_action.md)

5. Picture-in-Picture Desktop 🖼️ Copy Section

Problem: Existing CUAs lock users out during automation (poor UX).

UFO² Solution: Nested virtual desktop via Windows Remote Desktop loopback:

• User Desktop: Continue working normally
• Agent Desktop (PiP): Automation runs in parallel sandboxed environment
• Zero Interference: User and agent don't compete for mouse/keyboard

Implementation:
Built on Windows native remote desktop infrastructure—secure, isolated, non-disruptive.

!!!success "User Experience"
Users can continue email, browsing, or coding while UFO² automates Excel reports in the background PiP desktop.

System Components Copy Chapter

Processing Pipeline Copy Section

Both HostAgent and AppAgent execute a 4-phase processing cycle:

!!!info "Processing Details"
📖 Strategy Layer — Processing framework and dependency chain
📖 State Layer — FSM design principles

Command System Copy Section

Commands are dispatched through MCP (Model Context Protocol) servers:

HostAgent Commands:

• Desktop Capture: capture_desktop_screenshot
• Window Management: get_desktop_app_info, get_app_window
• Process Control: launch_application, close_application

AppAgent Commands:

• Screenshot: capture_screenshot, annotate_screenshot
• UI Inspection: get_control_info, get_ui_tree
• UI Interaction: click, set_edit_text, wheel_mouse_input
• Control Selection: select_control_by_index, select_control_by_name

!!!info "Command Architecture"
📖 Command Layer — MCP integration and command dispatch
📖 MCP Servers — Server architecture and custom server creation

Configuration Copy Chapter

UFO² integrates with a centralized YAML-based configuration system:

# config/ufo/host_agent_config.yaml
host_agent:
  visual_mode: true                  # Enable screenshot-based reasoning
  max_subtasks: 10                   # Maximum subtasks per session
  llm_config:
    model: "gpt-4o"
    temperature: 0.0

# config/ufo/app_agent_config.yaml
app_agent:
  visual_mode: true                  # Enable UI screenshot analysis
  control_backend: "uia"             # UI Automation (uia) or Win32 (win32)
  max_steps: 20                      # Maximum steps per subtask

!!!tip "Complete Configuration Guide"
For detailed configuration options, model setup, and advanced customization:

📖 **[Configuration & Setup](../configuration/system/overview.md)** — Complete system configuration reference  
📖 **[Model Setup](../configuration/models/overview.md)** — LLM provider configuration (OpenAI, Azure, Gemini, Claude, etc.)  
📖 **[MCP Configuration](../configuration/system/mcp_reference.md)** — MCP server and extension configuration

Quick Start Copy Chapter

Basic Usage Copy Section

UFO² is designed to be run from the command line:

Interactive Mode:

# Start UFO² in interactive mode
python -m ufo --task <your_task_name>

Example:

python -m ufo --task excel_demo

This will prompt you to enter your request interactively:

Welcome to use UFO🛸, A UI-focused Agent for Windows OS Interaction.
Please enter your request to be completed🛸: Create a chart from Sheet1 data in Excel

Direct Request Mode:

# Execute with a specific request directly
python -m ufo --task <your_task_name> -r "<your_request>"

Example:

python -m ufo --task excel_demo -r "Open Excel and create a chart from Sheet1 data"

!!!tip "Complete Setup Guide"
For detailed installation, configuration, and advanced usage options, see the Quick Start Guide.

What Happens Under the Hood Copy Section

1. Session creates HostAgent with user request
2. HostAgent captures desktop, selects "Microsoft Excel", launches app
3. HostAgent creates AppAgent for Excel, delegates subtask
4. AppAgent captures Excel UI, identifies chart insertion control
5. AppAgent executes hybrid action (API if available, GUI fallback)
6. AppAgent reports completion to HostAgent
7. HostAgent verifies task, returns success to Session

!!!tip "Next Steps"
📖 Getting Started Guide
📖 Creating Your AppAgent

Documentation Navigation Copy Chapter

Core Concepts Copy Section

• HostAgent — Desktop orchestrator with 7-state FSM
• AppAgent — Application executor with 6-state FSM
• Agent Types — Platform-specific implementations
• Evaluation Agent — Automated testing and benchmarking

Advanced Features Copy Section

• Hybrid Actions — GUI–API execution layer
• Control Detection — UIA + visual grounding
• Knowledge Substrate — RAG-enhanced learning
• Multi-Action Execution — Speculative action planning
• Follower Mode — Human-in-the-loop execution
• Batch Mode — Bulk task processing

System Architecture Copy Section

• Device Agent Overview — Three-layer architecture
• State Layer — FSM design principles
• Strategy Layer — Processing framework
• Command Layer — MCP integration

Development Copy Section

• Creating AppAgent — Custom agent development
• MCP Servers — Building custom MCP servers
• Configuration — System configuration reference
• Prompts — Prompt engineering guide

Benchmarking & Logs Copy Section

• Benchmark Overview — WindowsAgentArena, OSWorld
• Performance Logs — Execution logs and debugging

Research Impact Copy Chapter

UFO² demonstrates that system-level integration and architectural design matter more than model size alone:

!!!success "Key Findings"
- 10%+ improvement over Claude/Operator on WindowsAgentArena
- 51% fewer LLM calls via speculative multi-action execution
- Robust to UI changes through hybrid UIA + visual detection
- Continuous learning without model retraining via RAG
- Non-disruptive UX via Picture-in-Picture desktop

Research Paper:
📄 UFO²: A Grounded OS Agent for Windows

Get Started Copy Chapter

Ready to explore UFO²? Choose your path:

!!!info "Learning Paths"
🚀 New Users: Start with Quick Start Guide
🔧 Developers: Read Creating AppAgent
🏗️ System Architects: Study Device Agent Architecture
📊 Researchers: Check Benchmark Results

Next: HostAgent Deep Dive → Understand desktop orchestration

🌐 Media Coverage Copy Chapter

Check out our official deep dive of UFO on this Youtube Video.

UFO sightings have garnered attention from various media outlets, including:

• 微软正式开源UFO²，Windows桌面迈入「AgentOS 时代」
• Microsoft's UFO abducts traditional user interfaces for a smarter Windows experience
• 🚀 UFO & GPT-4-V: Sit back and relax, mientras GPT lo hace todo🌌
• The AI PC - The Future of Computers? - Microsoft UFO
• 下一代Windows系统曝光：基于GPT-4V，Agent跨应用调度，代号UFO
• 下一代智能版 Windows 要来了？微软推出首个 Windows Agent，命名为 UFO！
• Microsoft発のオープンソース版「UFO」登場！　Windowsを自動操縦するAIエージェントを試す

📚 Citation Copy Chapter

If you build on this work, please cite the AgentOS framework:

UFO² – The Desktop AgentOS (2025)
https://arxiv.org/abs/2504.14603

@article{zhang2025ufo2,
  title   = {{UFO2: The Desktop AgentOS}},
  author  = {Zhang, Chaoyun and Huang, He and Ni, Chiming and Mu, Jian and Qin, Si and He, Shilin and Wang, Lu and Yang, Fangkai and Zhao, Pu and Du, Chao and Li, Liqun and Kang, Yu and Jiang, Zhao and Zheng, Suzhen and Wang, Rujia and Qian, Jiaxu and Ma, Minghua and Lou, Jian-Guang and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei},
  journal = {arXiv preprint arXiv:2504.14603},
  year    = {2025}
}

UFO – A UI‑Focused Agent for Windows OS Interaction (2024)
https://arxiv.org/abs/2402.07939

@article{zhang2024ufo,
  title   = {{UFO: A UI-Focused Agent for Windows OS Interaction}},
  author  = {Zhang, Chaoyun and Li, Liqun and He, Shilin and Zhang, Xu and Qiao, Bo and Qin, Si and Ma, Minghua and Kang, Yu and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei and Zhang, Qi},
  journal = {arXiv preprint arXiv:2402.07939},
  year    = {2024}
}

🎨 Related Projects Copy Chapter

• TaskWeaver — a code‑first LLM agent for data analytics: https://github.com/microsoft/TaskWeaver
• LLM‑Brained GUI Agents: A Survey: https://arxiv.org/abs/2411.18279 • GitHub • Interactive site

❓Get Help Copy Chapter

• ❔GitHub Issues (preferred)
• For other communications, please contact [email protected]

Documents Docs Configuration Models Overview (`documents/docs/configuration/models/overview.md`) Copy Section

Supported Models

UFO supports a wide variety of LLM models and APIs. You can configure different models for HOST_AGENT, APP_AGENT, BACKUP_AGENT, and EVALUATION_AGENT in the config/ufo/agents.yaml file to optimize for performance, cost, or specific capabilities.

Available Model Integrations Copy Chapter

Model Selection Guide Copy Chapter

By Use Case Copy Section

For Production Deployments:

• Primary: OpenAI GPT-4o or Azure OpenAI (enterprise features)
• Cost-optimized: GPT-4o-mini for APP_AGENT, GPT-4o for HOST_AGENT
• Privacy-sensitive: Ollama (local models)

For Development & Testing:

• Fast iteration: Gemini 2.0 Flash (high speed, low cost)
• Local testing: Ollama with llama2 or similar
• Budget-friendly: DeepSeek or Qwen models

For Specialized Tasks:

• Computer control: OpenAI Operator (CUA model)
• Code generation: DeepSeek-Coder or Claude
• Long context: Gemini 1.5 Pro (large context window)

By Capability Copy Section

Vision Support (Screenshot Understanding):

• ✅ OpenAI GPT-4o, GPT-4-turbo
• ✅ Azure OpenAI (vision-enabled deployments)
• ✅ Google Gemini (all 1.5+ models)
• ✅ Claude 3+ (all variants)
• ✅ Qwen-VL models
• ⚠️ Ollama (llava models only)
• ❌ DeepSeek (text-only)

JSON Schema Support:

• ✅ OpenAI / Azure OpenAI
• ✅ Google Gemini
• ⚠️ Limited: Claude, Qwen, Ollama

Configuration Architecture Copy Chapter

Each model is implemented as a separate class in the ufo/llm directory, inheriting from the BaseService class in ufo/llm/base.py. All models implement the chat_completion method to maintain a consistent interface.

Key Configuration Files:

• config/ufo/agents.yaml: Primary agent configuration (HOST, APP, BACKUP, EVALUATION, OPERATOR)
• config/ufo/system.yaml: System-wide LLM parameters (MAX_TOKENS, TEMPERATURE, etc.)
• config/ufo/prices.yaml: Cost tracking for different models

Multi-Provider Setup Copy Chapter

You can mix and match providers for different agents to optimize cost and performance:

# Use OpenAI for planning
HOST_AGENT:
  API_TYPE: "openai"
  API_MODEL: "gpt-4o"

# Use Azure OpenAI for execution (cost control)
APP_AGENT:
  API_TYPE: "aoai"
  API_MODEL: "gpt-4o-mini"

# Use Claude for evaluation
EVALUATION_AGENT:
  API_TYPE: "claude"
  API_MODEL: "claude-3-5-sonnet-20241022"

Getting Started Copy Chapter

1. Choose your LLM provider from the table above
2. Follow the provider-specific documentation to obtain API keys
3. Configure config/ufo/agents.yaml with your credentials
4. Refer to the Quick Start Guide to begin

For detailed configuration options:

• Agent Configuration Guide - Complete configuration reference
• System Configuration - LLM parameters and behavior
• Quick Start Guide - Step-by-step setup

Documents Docs Configuration System Overview (`documents/docs/configuration/system/overview.md`) Copy Section

Configuration Architecture

UFO² features a modern, modular configuration system designed for flexibility, maintainability, and backward compatibility. This guide explains the overall architecture and design principles.

Design Philosophy Copy Chapter

The configuration system follows professional software engineering best practices:

Separation of Concerns Copy Section

Configuration files are organized by domain rather than monolithic structure:

• Agent configurations (agents.yaml) - LLM settings for different agents → Agent Config Guide
• System configurations (system.yaml) - Execution and runtime settings → System Config Guide
• RAG configurations (rag.yaml) - Knowledge retrieval settings → RAG Config Guide
• MCP configurations (mcp.yaml) - Model Context Protocol servers → MCP Config Guide
• Pricing configurations (prices.yaml) - Cost tracking for different models → Pricing Config Guide
• Third-party configurations (third_party.yaml) - External agent integration (LinuxAgent, HardwareAgent) → Third-Party Config Guide

Type Safety + Flexibility Copy Section

Hybrid approach combining:

• Fixed typed fields - IDE autocomplete, type checking, and IntelliSense
• Dynamic YAML fields - Add new settings without code changes

Example:

# Type-safe access (recommended)
config = get_ufo_config()
max_step = config.system.max_step  # IDE autocomplete!
api_model = config.app_agent.api_model

# Dynamic access (for custom fields)
custom_value = config.CUSTOM_FEATURE_FLAG
new_setting = config["NEW_YAML_KEY"]

# Backward compatible (legacy code still works)
max_step_old = config["MAX_STEP"]

Backward Compatibility Copy Section

Zero breaking changes - existing code continues to work:

• Old configuration paths still supported (ufo/config/)
• Old access patterns still work (config["MAX_STEP"])
• Automatic migration warnings guide users to new structure

Your existing code will continue to work without any modifications. The system automatically falls back to legacy paths and access patterns. See the Migration Guide for details on upgrading to the new structure.

Auto-Discovery Copy Section

No manual file registration needed:

• All *.yaml files in config/ufo/ are automatically loaded
• Files are merged intelligently with deep merging
• Environment-specific overrides (*_dev.yaml, *_test.yaml) supported

Directory Structure Copy Chapter

UFO/
├── config/                          ← New Configuration Root (Recommended)
│   ├── ufo/                        ← UFO² Configurations
│   │   ├── agents.yaml             # LLM agent settings
│   │   ├── agents.yaml.template    # Template for setup
│   │   ├── system.yaml             # System and runtime settings
│   │   ├── rag.yaml                # RAG knowledge settings
│   │   ├── mcp.yaml                # MCP server configurations
│   │   ├── prices.yaml             # Model pricing
│   │   └── third_party.yaml        # Third-party agents (optional)
│   │
│   ├── galaxy/                     ← Galaxy Configurations
│   │   ├── agent.yaml              # Constellation agent settings
│   │   ├── agent.yaml.template     # Template for setup
│   │   ├── constellation.yaml      # Constellation runtime settings
│   │   └── devices.yaml            # Device/client configurations
│   │
│   ├── config_loader.py            # Modern config loader
│   └── config_schemas.py           # Type definitions
│
└── ufo/config/                      ← Legacy Path (Still Supported)
    └── config.yaml                 # Old monolithic config

Galaxy Configuration Files Copy Chapter

The Galaxy constellation system has its own set of configuration files in config/galaxy/:

Galaxy Configuration Structure Copy Section

/* Detailed source-code truncated for AI context efficiency. */

Galaxy Configuration Loading Copy Section

# Load Galaxy configurations
from config.config_loader import get_galaxy_config

# Load Galaxy configuration (includes agent and constellation settings)
galaxy_config = get_galaxy_config()

# Access agent configuration (LLM settings)
agent_config = galaxy_config.agent.constellation_agent

# Access constellation runtime settings
constellation_settings = galaxy_config.constellation

# Or use raw dict access for backward compatibility
constellation_id = galaxy_config["CONSTELLATION_ID"]

Galaxy vs UFO Configuration:

• UFO Configurations (config/ufo/) - Single-agent automation settings
• Galaxy Configurations (config/galaxy/) - Multi-device constellation settings
• Both systems can coexist in the same project

Configuration Loading Process Copy Chapter

Priority Chain Copy Section

The configuration system uses a clear priority chain (highest to lowest):

1. New modular configs - config/{module}/*.yaml
2. Legacy monolithic config - {module}/config/config.yaml
3. Environment variables - Runtime overrides

When the same setting exists in multiple locations, the new modular config takes precedence over legacy configs. Values are merged with later sources overriding earlier ones.

Loading Algorithm Copy Section

def load_config():
    # Step 1: Start with environment variables (lowest priority)
    config_data = dict(os.environ)
    
    # Step 2: Load legacy config if exists (middle priority)
    if exists("ufo/config/config.yaml"):
        legacy_data = load_yaml("ufo/config/config.yaml")
        merge(config_data, legacy_data)
    
    # Step 3: Load new modular configs (highest priority)
    for yaml_file in discover("config/ufo/*.yaml"):
        new_data = load_yaml(yaml_file)
        merge(config_data, new_data)
    
    # Step 4: Create typed config object
    return UFOConfig.from_dict(config_data)

Deep Merging Copy Section

Configuration files are merged recursively, allowing you to split configurations across multiple files without duplication:

# config/ufo/agents.yaml
HOST_AGENT:
  API_TYPE: "openai"
  API_MODEL: "gpt-4o"

# config/ufo/custom.yaml (added later)
HOST_AGENT:
  TEMPERATURE: 0.5  # Added to HOST_AGENT

# Result: HOST_AGENT has all three fields

Fields from later files are added to (not replacing) earlier configurations.

File Organization Patterns Copy Chapter

Split by Domain (Current Approach) Copy Section

config/ufo/
├── agents.yaml      # All agent LLM configs
├── system.yaml      # All system settings
├── rag.yaml         # All RAG settings
├── mcp.yaml         # All MCP servers
└── prices.yaml      # Model pricing

Advantages: Easy to find related settings, clear separation of concerns, good for documentation.

Alternative: Split by Agent Copy Section

config/ufo/
├── host_agent.yaml       # HOST_AGENT config
├── app_agent.yaml        # APP_AGENT config
├── system.yaml           # Shared system config
└── rag.yaml              # Shared RAG config

Advantages: Agent-specific settings isolated, easy to customize per agent, good for multi-agent scenarios.

Both patterns work! The loader auto-discovers and merges all YAML files.

Environment-Specific Overrides Copy Chapter

Support for development, testing, and production environments:

# Base configuration
config/ufo/agents.yaml          # All environments

# Environment-specific overrides
config/ufo/agents_dev.yaml      # Development only
config/ufo/agents_test.yaml     # Testing only
config/ufo/agents_prod.yaml     # Production only

Activation:

# Set environment
export UFO_ENV=dev              # Linux/Mac
$env:UFO_ENV = "dev"            # Windows PowerShell

# Configuration loads:
# 1. agents.yaml (base)
# 2. agents_dev.yaml (overrides)

Type System Copy Chapter

Fixed Types (Recommended) Copy Section

Provides IDE autocomplete and type safety:

@dataclass
class SystemConfig:
    max_step: int = 50
    max_retry: int = 20
    temperature: float = 0.0
    # ...

# Usage - IDE knows the types!
config.system.max_step        # int
config.system.temperature     # float

Dynamic Types (Flexible) Copy Section

For custom or experimental settings. Learn more about adding custom fields in the Extending Configuration guide.

Example:

# In YAML
MY_CUSTOM_FEATURE: True
NEW_EXPERIMENTAL_SETTING: "value"

# In code - dynamic access
if config.MY_CUSTOM_FEATURE:
    setting = config.NEW_EXPERIMENTAL_SETTING

Hybrid Approach Copy Section

Best of both worlds:

class SystemConfig:
    # Fixed fields
    max_step: int = 50
    
    # Dynamic extras
    _extras: Dict[str, Any]
    
    def __getattr__(self, name):
        # Try extras for unknown fields
        return self._extras.get(name)

Migration Warnings Copy Chapter

The system provides clear warnings when using legacy paths:

⚠️  LEGACY CONFIG PATH DETECTED: UFO

Using legacy config: ufo/config/
Please migrate to:   config/ufo/

Quick migration:
  mkdir -p config/ufo
  cp ufo/config/*.yaml config/ufo/

Or use migration tool:
  python -m ufo.tools.migrate_config

These warnings appear once per session and guide you to migrate to the new structure.

Best Practices Copy Chapter

Recommended Practices:

• Use modular files - Split by domain or agent
• Use typed access - config.system.max_step over config["MAX_STEP"]
• Add templates - Provide .template files for sensitive data
• Document custom fields - Add comments in YAML
• Use environment overrides - For dev/test/prod differences

Anti-Patterns to Avoid:

• Mix old and new - Migrate fully to new structure
• Put secrets in YAML - Use environment variables instead
• Duplicate settings - Leverage deep merging
• Break backward compat - Keep config["OLD_KEY"] working
• Hardcode paths - Use config system

Configuration Lifecycle Copy Chapter

graph LR
    A[Application Start] --> B[Load Environment Vars]
    B --> C[Check for Legacy Config]
    C --> D[Load New Modular Configs]
    D --> E[Deep Merge All Sources]
    E --> F[Apply Transformations]
    F --> G[Create Typed Config Object]
    G --> H[Cache for Reuse]
    H --> I[Application Running]

Next Steps Copy Chapter

UFO Configuration Guides Copy Section

• Agent Configuration - LLM and API settings for all agents
• System Configuration - Runtime and execution settings
• RAG Configuration - Knowledge retrieval and learning settings
• MCP Configuration - Model Context Protocol servers
• Pricing Configuration - LLM cost tracking
• Third-Party Configuration - External agent integration (LinuxAgent, HardwareAgent)
• Migration Guide - How to migrate from old to new config
• Extending Configuration - How to add new configuration options

Galaxy Configuration Guides Copy Section

• Galaxy Constellation Configuration - Runtime settings for constellation orchestrator
• Galaxy Devices Configuration - Device definitions and capabilities
• Galaxy Agent Configuration - LLM configuration for constellation agent

API Reference Copy Chapter

For detailed API documentation of configuration classes and methods, see:

• config.config_loader.ConfigLoader - Configuration loading and caching
• config.config_schemas.UFOConfig - UFO configuration schema
• config.config_schemas.GalaxyConfig - Galaxy configuration schema
• config.config_loader.get_ufo_config() - Get UFO configuration instance
• config.config_loader.get_galaxy_config() - Get Galaxy configuration instance

Documents Docs Galaxy Agent Registration Overview (`documents/docs/galaxy/agent_registration/overview.md`) Copy Section

🌟 Agent Registration & Profiling - Overview

Agent Registration is the cornerstone of the AIP (Agent Interaction Protocol) initialization process. It enables dynamic discovery, capability advertisement, and intelligent task allocation across distributed constellation agents.

📋 Introduction Copy Chapter

An overview of the Constellation Agent architecture showing the registration and profiling system.

At the core of AIP's initialization process is the ConstellationClient (implemented as ConstellationDeviceManager), which maintains a global registry of active agents. Any device agent service that exposes a WebSocket endpoint and implements the AIP task dispatch and result-return protocol can be seamlessly integrated into UFO, providing remarkable extensibility.

The multi-source profiling pipeline enables transparent capability discovery and safe adaptation to environmental drift without direct administrator intervention.