---
### 🪟 UFO² Desktop AgentOS – Core Strengths
UFO² serves dual roles: **standalone Windows automation** and **Galaxy device agent** for Windows platforms.
| Feature | Description | Documentation |
|---------|-------------|---------------|
| **Deep OS Integration** | Windows UIA, Win32, WinCOM native control | [Learn More](https://microsoft.github.io/UFO) |
| **Hybrid Actions** | GUI clicks + API calls for optimal performance | [Learn More](https://microsoft.github.io/UFO/automator/overview) |
| **Speculative Multi-Action** | Batch predictions → **51% fewer LLM calls** | [Learn More](https://microsoft.github.io/UFO/advanced_usage/multi_action) |
| **Visual + UIA Detection** | Hybrid control detection for robustness | [Learn More](https://microsoft.github.io/UFO/advanced_usage/control_detection/hybrid_detection) |
| **Knowledge Substrate** | RAG with docs, demos, execution traces | [Learn More](https://microsoft.github.io/UFO/advanced_usage/reinforce_appagent/overview/) |
| **Device Agent Role** | Can serve as Windows executor in Galaxy orchestration | [Learn More](./galaxy/README.md) |
**As Galaxy Device Agent:**
- Receives tasks from ConstellationAgent via Galaxy orchestration layer
- Executes Windows-specific operations using proven UFO² capabilities
- Reports status and results back to TaskOrchestrator
- Participates in cross-device workflows seamlessly
---
## 🚀 Quick Start Guide
Choose your path and follow the detailed setup guide:
---
## 📢 Latest Updates
### 2025-11 – UFO³ Galaxy Framework Released 🌌
**Major Research Breakthrough:** Multi-Device Orchestration System
- 🌟 **Declarative DAG Decomposition**: TaskConstellation structure for workflow logic and dependencies
- 🔄 **Dynamic Graph Evolution**: Living constellation that adapts through controlled rewrites
- 🎯 **Heterogeneous Orchestration**: Safe, asynchronous execution with capability-based device matching
- 🔌 **Unified AIP Protocol**: WebSocket-based secure agent coordination with fault tolerance
- 🛠️ **MCP-Empowered Agent Framework**: Template-driven toolkit for rapid device agent development
- 📄 **Research Paper**: [UFO³: Weaving the Digital Agent Galaxy](https://arxiv.org/abs/2511.11332)
**Key Features:**
- First multi-device orchestration framework for GUI agents
- Result-driven adaptive execution instead of rigid workflows
- Model Context Protocol (MCP) integration for tool augmentation
- Formally verified correctness and concurrency safety guarantees
### 2025-04 – UFO² v2.0.0
- 📅 UFO² Desktop AgentOS released
- 🏗️ Enhanced architecture with AgentOS concept
- 📄 [Technical Report](https://arxiv.org/pdf/2504.14603) published
- ✅ Entered Long-Term Support (LTS) status
### 2024-02 – Original UFO
- 🎈 First UFO release - UI-Focused agent for Windows
- 📄 [Original Paper](https://arxiv.org/abs/2402.07939)
- 🌍 Wide media coverage and adoption
---
## 📚 Citation
If you use UFO³ Galaxy or UFO² in your research, please cite the relevant papers:
### UFO³ Galaxy Framework (2025)
```bibtex
@article{zhang2025ufo3,
title={UFO$^3$: Weaving the Digital Agent Galaxy},
author = {Zhang, Chaoyun and Li, Liqun and Huang, He and Ni, Chiming and Qiao, Bo and Qin, Si and Kang, Yu and Ma, Minghua and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei},
journal = {arXiv preprint arXiv:2511.11332},
year = {2025},
}
```
### UFO² Desktop AgentOS (2025)
```bibtex
@article{zhang2025ufo2,
title = {{UFO2: The Desktop AgentOS}},
author = {Zhang, Chaoyun and Huang, He and Ni, Chiming and Mu, Jian and Qin, Si and He, Shilin and Wang, Lu and Yang, Fangkai and Zhao, Pu and Du, Chao and Li, Liqun and Kang, Yu and Jiang, Zhao and Zheng, Suzhen and Wang, Rujia and Qian, Jiaxu and Ma, Minghua and Lou, Jian-Guang and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei},
journal = {arXiv preprint arXiv:2504.14603},
year = {2025}
}
```
### Original UFO (2024)
```bibtex
@article{zhang2024ufo,
title = {{UFO: A UI-Focused Agent for Windows OS Interaction}},
author = {Zhang, Chaoyun and Li, Liqun and He, Shilin and Zhang, Xu and Qiao, Bo and Qin, Si and Ma, Minghua and Kang, Yu and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei and Zhang, Qi},
journal = {arXiv preprint arXiv:2402.07939},
year = {2024}
}
```
---
## 🌐 Media & Community
**Media Coverage:**
- [微软正式开源UFO²，Windows桌面迈入「AgentOS 时代」](https://www.jiqizhixin.com/articles/2025-05-06-13)
- [Microsoft's UFO: Smarter Windows Experience](https://the-decoder.com/microsofts-ufo-abducts-traditional-user-interfaces-for-a-smarter-windows-experience/)
- [下一代Windows系统曝光](https://baijiahao.baidu.com/s?id=1790938358152188625)
- **[More coverage →](./ufo/README.md#-tracing-the-stars)**
**Community:**
- 💬 [GitHub Discussions](https://github.com/microsoft/UFO/discussions)
- 🐛 [Issue Tracker](https://github.com/microsoft/UFO/issues)
- 📧 Email: [ufo-agent@microsoft.com](mailto:ufo-agent@microsoft.com)
- 📺 [YouTube Channel](https://www.youtube.com/watch?v=QT_OhygMVXU)
---
## 🎨 Related Projects & Research
**Microsoft Research:**
- **[TaskWeaver](https://github.com/microsoft/TaskWeaver)** – Code-first LLM agent framework for data analytics and task automation
**GUI Agent Research:**
- **[LLM-Brained GUI Agents Survey](https://github.com/vyokky/LLM-Brained-GUI-Agents-Survey)** – Comprehensive survey of GUI automation agents
- **[Interactive Survey Site](https://vyokky.github.io/LLM-Brained-GUI-Agents-Survey/)** – Explore latest GUI agent research and developments
**Multi-Agent Systems:**
- **UFO³ Galaxy** represents a novel approach to multi-device orchestration, introducing the Constellation framework for coordinating heterogeneous agents across platforms
- Builds on multi-agent coordination research while addressing unique challenges of cross-device GUI automation
**Benchmarks:**
- **[Windows Agent Arena (WAA)](https://github.com/nice-mee/WindowsAgentArena)** – Evaluation benchmark for Windows automation agents
- **[OSWorld](https://github.com/nice-mee/WindowsAgentArena/tree/2020-qqtcg/osworld)** – Cross-application task evaluation suite
---
## 💡 FAQ
🤔 Should I use Galaxy or UFO²?
**Start with UFO²** if:
- You only need Windows automation
- You want quick setup and learning
- Tasks are relatively simple
**Choose Galaxy** if:
- You need cross-device coordination
- Tasks are complex and multi-step
- You want advanced orchestration
- You're comfortable with active development
**Hybrid approach** if:
- You want best of both worlds
- Some tasks are simple (UFO²), some complex (Galaxy)
- You're gradually migrating
⚠️ Will UFO² be deprecated?
**No!** UFO² has entered **Long-Term Support (LTS)** status:
- ✅ Actively maintained
- ✅ Bug fixes and security updates
- ✅ Performance improvements
- ✅ Full community support
- ✅ No plans for deprecation
UFO² is the stable, proven solution for Windows automation.
🔄 How do I migrate from UFO² to Galaxy?
Migration is **gradual and optional**:
1. **Phase 1: Learn** – Understand Galaxy concepts
2. **Phase 2: Experiment** – Try Galaxy with non-critical tasks
3. **Phase 3: Hybrid** – Use both frameworks
4. **Phase 4: Migrate** – Gradually move complex tasks to Galaxy
**No forced migration!** Continue using UFO² as long as it meets your needs.
See [Migration Guide](./documents/docs/getting_started/migration_ufo2_to_galaxy.md) for details.
🎯 Can Galaxy do everything UFO² does?
**Functionally: Yes.** Galaxy can use UFO² as a Windows device agent.
**Practically: It depends.**
- For **simple Windows tasks**: UFO² standalone is easier and more streamlined
- For **complex workflows**: Galaxy orchestrates UFO² with other device agents
**Recommendation:** Use the right tool for the job. UFO² can work standalone or as Galaxy's Windows device agent.
📊 How mature is Galaxy?
**Status: Active Development** 🚧
**Stable:**
- ✅ Core architecture
- ✅ DAG orchestration
- ✅ Basic multi-device support
- ✅ Event system
**In Development:**
- 🔨 Advanced device types
- 🔨 Enhanced monitoring
- 🔨 Performance optimization
- 🔨 Extended documentation
**Recommendation:** Great for experimentation and non-critical workflows.
🔧 Can I extend or customize?
**Both frameworks are highly extensible:**
**UFO²:**
- Custom actions and automators
- Custom knowledge sources (RAG)
- Custom control detectors
- Custom evaluation metrics
**Galaxy:**
- Custom agents
- Custom device types
- Custom orchestration strategies
- Custom visualization components
See respective documentation for extension guides.
🤝 How can I contribute?
We welcome contributions to both UFO² and Galaxy!
**Ways to contribute:**
- 🐛 Report bugs and issues
- 💡 Suggest features and improvements
- 📝 Improve documentation
- 🧪 Add tests and examples
- 🔧 Submit pull requests
See [CONTRIBUTING.md](./CONTRIBUTING.md) for guidelines.
---
## ⚠️ Disclaimer & License
**Disclaimer:** By using this software, you acknowledge and agree to the terms in [DISCLAIMER.md](./DISCLAIMER.md).
**License:** This project is licensed under the [MIT License](LICENSE).
**Trademarks:** Use of Microsoft trademarks follows [Microsoft's Trademark Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
---
## 📖 About This Documentation
Welcome to the official documentation for **UFO³**, Microsoft's open-source framework for intelligent automation across devices and platforms. Whether you're looking to automate Windows applications or orchestrate complex workflows across multiple devices, this documentation will guide you through every step.
**What you'll find here:**
- 🚀 **[Quick Start Guides](getting_started/quick_start_galaxy.md)** – Get up and running in minutes
- 📚 **[Core Concepts](galaxy/overview.md)** – Understand the architecture and key components
- ⚙️ **[Configuration](configuration/system/agents_config.md)** – Set up your agents and models
- 🔧 **[Advanced Features](ufo2/core_features/multi_action.md)** – Deep dive into powerful capabilities
- 💡 **[FAQ](faq.md)** – Common questions and troubleshooting
---
## 🎯 Choose Your Path
UFO³ consists of two complementary frameworks. Choose the one that best fits your needs, or use both together!
| Framework | Best For | Key Strength | Get Started |
|-----------|----------|--------------|-------------|
| **🌌 Galaxy** ✨ NEW & RECOMMENDED | Cross-device workflows Complex automation Parallel execution | Multi-device orchestration DAG-based planning Real-time monitoring | [Quick Start →](getting_started/quick_start_galaxy.md) |
| **🪟 UFO²** ⚡ STABLE & LTS | Windows automation Quick tasks Learning basics | Deep Windows integration Hybrid GUI + API Stable & reliable | [Quick Start →](getting_started/quick_start_ufo2.md) |
### 🤔 Decision Guide
| Question | Galaxy | UFO² |
|----------|:------:|:----:|
| Need cross-device collaboration? | ✅ | ❌ |
| Complex multi-step workflows? | ✅ | ⚠️ Limited |
| Windows-only automation? | ✅ | ✅ Optimized |
| Quick setup & learning? | ⚠️ Moderate | ✅ Easy |
| Stable & reliable? | 🚧 Active Dev | ✅ LTS |
---
## 🌟 What's New in UFO³?
**UFO³ is a scalable, universal cross-device agent framework** that enables you to develop new device agents for different platforms and applications. Through the **Agent Interaction Protocol (AIP)**, custom device agents can seamlessly integrate into UFO³ Galaxy for coordinated multi-device orchestration.
### Evolution Timeline
```mermaid
%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#E8F4F8','primaryTextColor':'#1A1A1A','primaryBorderColor':'#7CB9E8','lineColor':'#A8D5E2','secondaryColor':'#B8E6F0','tertiaryColor':'#D4F1F4','fontSize':'16px','fontFamily':'Segoe UI, Arial, sans-serif'}}}%%
graph LR
A["🎈 UFO February 2024 GUI Agent for Windows"]
B["🖥️ UFO² April 2025 Desktop AgentOS"]
C["🌌 UFO³ Galaxy November 2025 Multi-Device Orchestration"]
A -->|Evolve| B
B -->|Scale| C
style A fill:#E8F4F8,stroke:#7CB9E8,stroke-width:2.5px,color:#1A1A1A,rx:15,ry:15
style B fill:#C5E8F5,stroke:#5BA8D0,stroke-width:2.5px,color:#1A1A1A,rx:15,ry:15
style C fill:#A4DBF0,stroke:#3D96BE,stroke-width:2.5px,color:#1A1A1A,rx:15,ry:15
```
### 🚀 UFO³ = **Galaxy** (Multi-Device Orchestration) + **UFO²** (Device Agent)
UFO³ introduces **Galaxy**, a revolutionary multi-device orchestration framework that coordinates intelligent agents across heterogeneous platforms. Built on five tightly integrated design principles:
1. **🌟 Declarative Decomposition into Dynamic DAG** - Requests decomposed into structured DAG with TaskStars and dependencies for automated scheduling and runtime rewriting
2. **🔄 Continuous Result-Driven Graph Evolution** - Living constellation that adapts to execution feedback through controlled rewrites and dynamic adjustments
3. **⚡ Heterogeneous, Asynchronous & Safe Orchestration** - Capability-based device matching with async execution, safe locking, and formally verified correctness
4. **🔌 Unified Agent Interaction Protocol (AIP)** - WebSocket-based secure coordination layer with fault tolerance and automatic reconnection
5. **🛠️ Template-Driven MCP-Empowered Device Agents** - Lightweight toolkit for rapid agent development with MCP integration for tool augmentation
| Aspect | UFO² | UFO³ Galaxy |
|--------|------|-------------|
| **Architecture** | Single Windows Agent | Multi-Device Orchestration |
| **Task Model** | Sequential ReAct Loop | DAG-based Constellation Workflows |
| **Scope** | Single device, multi-app | Multi-device, cross-platform |
| **Coordination** | HostAgent + AppAgents | ConstellationAgent + TaskOrchestrator |
| **Device Support** | Windows Desktop | Windows, Linux, macOS, Android, Web |
| **Task Planning** | Application-level | Device-level with dependencies |
| **Execution** | Sequential | Parallel DAG execution |
| **Device Agent Role** | Standalone | Can serve as Galaxy device agent |
| **Complexity** | Simple to Moderate | Simple to Very Complex |
| **Learning Curve** | Low | Moderate |
| **Cross-Device Collaboration** | ❌ Not Supported | ✅ Core Feature |
| **Setup Difficulty** | ✅ Easy | ⚠️ Moderate |
| **Status** | ✅ LTS (Long-Term Support) | ⚡ Active Development |
### 🎓 Migration Path
**For UFO² Users:**
1. ✅ **Keep using UFO²** – Fully supported, actively maintained
2. 🔄 **Gradual adoption** – Galaxy can use UFO² as Windows device agent
3. 📈 **Scale up** – Move to Galaxy when you need multi-device capabilities
4. 📚 **Learning resources** – [Migration Guide](./getting_started/migration_ufo2_to_galaxy.md)
---
## ✨ Capabilities at a Glance
### 🌌 Galaxy Framework – What's Different?
#### 🌟 Constellation Planning
```
User Request
↓
ConstellationAgent
↓
[Task DAG]
/ | \
Task1 Task2 Task3
(Win) (Linux)(Mac)
```
**Benefits:**
- Cross-device dependency tracking
- Parallel execution optimization
- Cross-device dataflow management
#### 🎯 Device Assignment
```
Selection Criteria
• Platform
• Resource
• Task requirements
• Performance history
↓
Auto-Assignment
↓
Optimal Devices
```
**Smart Matching:**
- Capability-based selection
- Real-time resource monitoring
- Dynamic reallocation
#### 📊 Orchestration
```
Task1 → Running ✅
Task2 → Pending ⏸️
Task3 → Running 🔄
↓
Completion
↓
Final Report
```
**Orchestration:**
- Real-time status updates
- Automatic error recovery
- Progress tracking with feedback
---
### 🪟 UFO² Desktop AgentOS – Core Strengths
UFO² serves dual roles: **standalone Windows automation** and **Galaxy device agent** for Windows platforms.
| Feature | Description | Documentation |
|---------|-------------|---------------|
| **Deep OS Integration** | Windows UIA, Win32, WinCOM native control | [Learn More](ufo2/overview.md) |
| **Hybrid Actions** | GUI clicks + API calls for optimal performance | [Learn More](ufo2/core_features/hybrid_actions.md) |
| **Speculative Multi-Action** | Batch predictions → **51% fewer LLM calls** | [Learn More](ufo2/core_features/multi_action.md) |
| **Visual + UIA Detection** | Hybrid control detection for robustness | [Learn More](ufo2/core_features/control_detection/hybrid_detection.md) |
| **Knowledge Substrate** | RAG with docs, demos, execution traces | [Learn More](ufo2/core_features/knowledge_substrate/overview.md) |
| **Device Agent Role** | Can serve as Windows executor in Galaxy orchestration | [Learn More](galaxy/overview.md) |
**As Galaxy Device Agent:**
- Receives tasks from ConstellationAgent through Galaxy orchestration layer
- Executes Windows-specific operations using proven UFO² capabilities
- Reports status and results back to TaskOrchestrator
- Seamlessly participates in cross-device workflows
---
## 🏗️ Architecture
### UFO³ Galaxy – Multi-Device Orchestration
| Component | Role |
|-----------|------|
| **ConstellationAgent** | Plans and decomposes tasks into DAG workflows |
| **TaskConstellation** | DAG representation with TaskStar nodes and dependencies |
| **Device Pool Manager** | Matches tasks to capable devices dynamically |
| **TaskOrchestrator** | Coordinates parallel execution and handles data flow |
| **Event System** | Real-time monitoring with observer pattern |
[📖 Learn More →](galaxy/overview.md)
### UFO² – Desktop AgentOS
| Component | Role |
|-----------|------|
| **HostAgent** | Desktop orchestrator, application lifecycle management |
| **AppAgents** | Per-application executors with hybrid GUI–API actions |
| **Knowledge Substrate** | RAG-enhanced learning from docs & execution history |
| **Speculative Executor** | Multi-action prediction for efficiency |
[📖 Learn More →](ufo2/overview.md)
---
## 🚀 Quick Start
Ready to dive in? Follow these guides to get started with your chosen framework:
### 🌌 Galaxy Quick Start (Multi-Device Orchestration)
Perfect for complex workflows across multiple devices and platforms.
```bash
# 1. Install dependencies
pip install -r requirements.txt
# 2. Configure agents (see detailed guide for API key setup)
copy config\galaxy\agent.yaml.template config\galaxy\agent.yaml
copy config\ufo\agents.yaml.template config\ufo\agents.yaml
# 3. Start device agents
python -m ufo.server.app --port 5000
python -m ufo.client.client --ws --ws-server ws://localhost:5000/ws --client-id device_1 --platform windows
# 4. Launch Galaxy
python -m galaxy --interactive
```
**📖 [Complete Galaxy Quick Start Guide →](getting_started/quick_start_galaxy.md)**
**⚙️ [Galaxy Configuration Details →](configuration/system/galaxy_devices.md)**
### 🪟 UFO² Quick Start (Windows Automation)
Perfect for Windows-only automation tasks with quick setup.
```bash
# 1. Install
pip install -r requirements.txt
# 2. Configure (add your API keys)
copy config\ufo\agents.yaml.template config\ufo\agents.yaml
# 3. Run
python -m ufo --task
```
**📖 [Complete UFO² Quick Start Guide →](getting_started/quick_start_ufo2.md)**
**⚙️ [UFO² Configuration Details →](configuration/system/agents_config.md)**
---
## 📚 Documentation Navigation
### 🎯 Getting Started
Start here if you're new to UFO³:
| Guide | Description | Framework |
|-------|-------------|-----------|
| [Galaxy Quick Start](getting_started/quick_start_galaxy.md) | Set up multi-device orchestration in 10 minutes | 🌌 Galaxy |
| [UFO² Quick Start](getting_started/quick_start_ufo2.md) | Start automating Windows in 5 minutes | 🪟 UFO² |
| [Linux Agent Quick Start](getting_started/quick_start_linux.md) | Automate Linux systems | 🐧 Linux |
| [Mobile Agent Quick Start](getting_started/quick_start_mobile.md) | Automate Android devices via ADB | 📱 Mobile |
| [Choosing Your Path](choose_path.md) | Decision guide for selecting the right framework | Both |
### 🏗️ Core Architecture
Understand how UFO³ works under the hood:
| Topic | Description | Framework |
|-------|-------------|-----------|
| [Galaxy Overview](galaxy/overview.md) | Multi-device orchestration architecture | 🌌 Galaxy |
| [UFO² Overview](ufo2/overview.md) | Desktop AgentOS architecture and concepts | 🪟 UFO² |
| [Task Constellation](galaxy/constellation/overview.md) | DAG-based workflow representation | 🌌 Galaxy |
| [ConstellationAgent](galaxy/constellation_agent/overview.md) | Intelligent task planner and decomposer | 🌌 Galaxy |
| [Task Orchestrator](galaxy/constellation_orchestrator/overview.md) | Execution engine and coordinator | 🌌 Galaxy |
| [AIP Protocol](aip/overview.md) | Agent communication protocol | 🌌 Galaxy |
### ⚙️ Configuration & Setup
Configure your agents, models, and environments:
| Topic | Description | Framework |
|-------|-------------|-----------|
| [Agent Configuration](configuration/system/agents_config.md) | LLM and agent settings | Both |
| [Galaxy Devices](configuration/system/galaxy_devices.md) | Device pool and capability management | 🌌 Galaxy |
| [Model Providers](configuration/models/overview.md) | Supported LLMs (OpenAI, Azure, Qwen, etc.) | Both |
### 🎓 Tutorials & Examples
Learn through practical examples in the documentation:
| Topic | Description | Framework |
|-------|-------------|-----------|
| [Creating App Agents](tutorials/creating_app_agent/overview.md) | Build custom application agents | 🪟 UFO² |
| [Multi-Action Prediction](ufo2/core_features/multi_action.md) | Efficient batch predictions | 🪟 UFO² |
| [Knowledge Substrate](ufo2/core_features/knowledge_substrate/overview.md) | RAG-enhanced learning | 🪟 UFO² |
### 🔧 Advanced Topics
Deep dive into powerful features:
| Topic | Description | Framework |
|-------|-------------|-----------|
| [Multi-Action Prediction](ufo2/core_features/multi_action.md) | Batch actions for 51% fewer LLM calls | 🪟 UFO² |
| [Hybrid Detection](ufo2/core_features/control_detection/hybrid_detection.md) | Visual + UIA control detection | 🪟 UFO² |
| [Knowledge Substrate](ufo2/core_features/knowledge_substrate/overview.md) | RAG-enhanced learning | 🪟 UFO² |
| [Constellation Agent](galaxy/constellation_agent/overview.md) | Task planning and decomposition | 🌌 Galaxy |
| [Task Orchestrator](galaxy/constellation_orchestrator/overview.md) | Execution coordination | 🌌 Galaxy |
### 🛠️ Development & Extension
Customize and extend UFO³:
| Topic | Description |
|-------|-------------|
| [Project Structure](project_directory_structure.md) | Understand the codebase layout |
| [Creating Custom Device Agents](tutorials/creating_device_agent/overview.md) | Build device agents for new platforms (mobile, web, IoT, etc.) |
| [Creating App Agents](tutorials/creating_app_agent/overview.md) | Build custom application agents |
| [Contributing Guide](about/CONTRIBUTING.md) | How to contribute to UFO³ |
### ❓ Support & Troubleshooting
Get help when you need it:
| Resource | What You'll Find |
|----------|------------------|
| [FAQ](faq.md) | Common questions and answers |
| [GitHub Discussions](https://github.com/microsoft/UFO/discussions) | Community Q&A |
| [GitHub Issues](https://github.com/microsoft/UFO/issues) | Bug reports and feature requests |
---
## 📊 Feature Matrix
| Feature | UFO² Desktop AgentOS | UFO³ Galaxy | Winner |
|---------|:--------------------:|:-----------:|:------:|
| **Windows Automation** | ⭐⭐⭐⭐⭐ Optimized | ⭐⭐⭐⭐ Supported | UFO² |
| **Cross-Device Tasks** | ❌ Not supported | ⭐⭐⭐⭐⭐ Core feature | Galaxy |
| **Setup Complexity** | ⭐⭐⭐⭐⭐ Very easy | ⭐⭐⭐ Moderate | UFO² |
| **Learning Curve** | ⭐⭐⭐⭐⭐ Gentle | ⭐⭐⭐ Moderate | UFO² |
| **Task Complexity** | ⭐⭐⭐ Good | ⭐⭐⭐⭐⭐ Excellent | Galaxy |
| **Parallel Execution** | ❌ Sequential | ⭐⭐⭐⭐⭐ Native DAG | Galaxy |
| **Stability** | ⭐⭐⭐⭐⭐ Stable | ⭐⭐⭐ Active dev | UFO² |
| **Monitoring Tools** | ⭐⭐⭐ Logs | ⭐⭐⭐⭐⭐ Real-time viz | Galaxy |
| **API Flexibility** | ⭐⭐⭐ Good | ⭐⭐⭐⭐⭐ Extensive | Galaxy |
---
## 🎯 Use Cases & Examples
Explore what you can build with UFO³:
### 🌌 Galaxy Use Cases (Cross-Device)
Perfect for complex, multi-device workflows:
- **Cross-Platform Data Pipelines**: Extract from Windows Excel → Process on Linux → Visualize on Mac
- **Distributed Testing**: Run tests on Windows → Deploy to Linux → Update mobile app
- **Multi-Device Monitoring**: Collect logs from multiple devices → Aggregate centrally
- **Complex Automation**: Orchestrate workflows across heterogeneous platforms
### 🪟 UFO² Use Cases (Windows)
Perfect for Windows automation and rapid task execution:
- **Office Automation**: Excel/Word/PowerPoint report generation and data processing
- **Web Automation**: Browser-based research, form filling, data extraction
- **File Management**: Organize, rename, convert files based on rules
- **System Tasks**: Windows configuration, software installation, backups
---
## 🌐 Community & Resources
### 📺 Media & Videos
Check out our official deep dive of UFO on [YouTube](https://www.youtube.com/watch?v=QT_OhygMVXU).
### Media Coverage:
- [微软正式开源UFO²，Windows桌面迈入「AgentOS 时代」](https://www.jiqizhixin.com/articles/2025-05-06-13)
- [Microsoft's UFO: Smarter Windows Experience](https://the-decoder.com/microsofts-ufo-abducts-traditional-user-interfaces-for-a-smarter-windows-experience/)
- [下一代Windows系统曝光：基于GPT-4V](https://baijiahao.baidu.com/s?id=1790938358152188625)
### 💬 Get Help & Connect
- **📖 Documentation**: You're here! Browse the navigation above
- **💬 Discussions**: [GitHub Discussions](https://github.com/microsoft/UFO/discussions)
- **🐛 Issues**: [GitHub Issues](https://github.com/microsoft/UFO/issues)
- **📧 Email**: [ufo-agent@microsoft.com](mailto:ufo-agent@microsoft.com)
### 🎨 Related Projects
- **[TaskWeaver](https://github.com/microsoft/TaskWeaver)** – Code-first LLM agent framework
- **[Windows Agent Arena](https://github.com/nice-mee/WindowsAgentArena)** – Evaluation benchmark
- **[GUI Agents Survey](https://vyokky.github.io/LLM-Brained-GUI-Agents-Survey/)** – Latest research
---
## 📚 Research & Citation
UFO³ is built on cutting-edge research in multi-agent systems and GUI automation.
### Papers
If you use UFO³ in your research, please cite:
**UFO³ Galaxy Framework (2025)**
```bibtex
@article{zhang2025ufo3,
title={UFO$^3$: Weaving the Digital Agent Galaxy},
author = {Zhang, Chaoyun and Li, Liqun and Huang, He and Ni, Chiming and Qiao, Bo and Qin, Si and Kang, Yu and Ma, Minghua and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei},
journal = {arXiv preprint arXiv:2511.11332},
year = {2025},
}
```
**UFO² Desktop AgentOS (2025)**
```bibtex
@article{zhang2025ufo2,
title = {{UFO2: The Desktop AgentOS}},
author = {Zhang, Chaoyun and Huang, He and Ni, Chiming and Mu, Jian and Qin, Si and He, Shilin and Wang, Lu and Yang, Fangkai and Zhao, Pu and Du, Chao and Li, Liqun and Kang, Yu and Jiang, Zhao and Zheng, Suzhen and Wang, Rujia and Qian, Jiaxu and Ma, Minghua and Lou, Jian-Guang and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei},
journal = {arXiv preprint arXiv:2504.14603},
year = {2025}
}
```
**Original UFO (2024)**
```bibtex
@article{zhang2024ufo,
title = {{UFO: A UI-Focused Agent for Windows OS Interaction}},
author = {Zhang, Chaoyun and Li, Liqun and He, Shilin and Zhang, Xu and Qiao, Bo and Qin, Si and Ma, Minghua and Kang, Yu and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei and Zhang, Qi},
journal = {arXiv preprint arXiv:2402.07939},
year = {2024}
}
```
**📖 [Read the Papers →](https://arxiv.org/abs/2504.14603)**
---
## 🗺️ Roadmap & Future
### UFO² Desktop AgentOS (Stable/LTS)
- ✅ Long-term support and maintenance
- ✅ Windows device agent integration
- 🔜 Enhanced device capabilities
- 🔜 Picture-in-Picture mode
### UFO³ Galaxy (Active Development)
- ✅ Constellation Framework
- ✅ Multi-device coordination
- 🔄 Mobile, Web, IoT agents
- 🔄 Interactive visualization
- 🔜 Advanced fault tolerance
**Legend:** ✅ Done | 🔄 In Progress | 🔜 Planned
---
## ⚖️ License & Legal
- **License**: [MIT License](https://github.com/microsoft/UFO/blob/main/LICENSE)
- **Disclaimer**: [Read our disclaimer](https://github.com/microsoft/UFO/blob/main/DISCLAIMER.md)
- **Trademarks**: [Microsoft Trademark Guidelines](https://www.microsoft.com/legal/intellectualproperty/trademarks)
- **Contributing**: [Contribution Guidelines](about/CONTRIBUTING.md)
---
## 🚀 Ready to Start?
Choose your framework and begin your automation journey:
### 🌌 Start with Galaxy
**For multi-device orchestration**
[](getting_started/quick_start_galaxy.md)
### 🪟 Start with UFO²
**For Windows automation**
[](getting_started/quick_start_ufo2.md)
### 📖 Explore the Documentation
[Core Concepts](galaxy/overview.md) | [Configuration](configuration/system/agents_config.md) | [FAQ](faq.md) | [GitHub](https://github.com/microsoft/UFO)
---
From Single Agent to Digital Galaxy UFO³ - Weaving the Future of Intelligent Automation
---
---
### Documents Docs Aip Overview (`documents/docs/aip/overview.md`)
# Agent Interaction Protocol (AIP)
The orchestration model requires a communication substrate that remains **correct under continuous DAG evolution**, **dynamic agent participation**, and **fine-grained event propagation**. Legacy HTTP-based coordination approaches (e.g., A2A, ACP) assume short-lived, stateless interactions, incurring handshake overhead, stale capability views, and fragile recovery when partial failures occur mid-task. These assumptions make them unsuitable for the continuously evolving workflows and long-running reasoning loops characteristic of UFO².
## Design Overview
AIP serves as the **nervous system** of UFO², connecting the ConstellationClient, device agent services, and device clients under a unified, event-driven control plane. It is designed as a lightweight yet evolution-tolerant protocol to satisfy six goals:
**Design Goals:**
- **(G1)** Maintain persistent bidirectional sessions to eliminate per-request overhead
- **(G2)** Unify heterogeneous capability discovery via multi-source profiling
- **(G3)** Ensure fine-grained reliability through heartbeats and timeout managers for disconnection and failure detection
- **(G4)** Preserve deterministic command ordering within sessions
- **(G5)** Support composable extensibility for new message types and resilience strategies
- **(G6)** Provide transparent reconnection and task continuity under transient failures
| Legacy HTTP Coordination | AIP WebSocket-Based Design |
|--------------------------|----------------------------|
| ❌ Short-lived requests | ✅ Persistent sessions (G1) |
| ❌ Stateless interactions | ✅ Session-aware task management |
| ❌ High latency overhead | ✅ Low-latency event streaming |
| ❌ Poor reconnection support | ✅ Seamless recovery from disconnections (G6) |
| ❌ Manual state synchronization | ✅ Automatic DAG state propagation |
| ❌ Fragile partial failures | ✅ Fine-grained reliability (G3) |
## Five-Layer Architecture
To meet these requirements, AIP adopts a persistent, bidirectional WebSocket transport and decomposes the orchestration substrate into **five** logical strata, each responsible for a distinct aspect of reliability and adaptability. The architecture establishes a complete substrate where **L1** defines semantic contracts, **L2** provides transport flexibility, **L3** implements protocol logic, **L4** ensures operational resilience, and **L5** delivers deployment-ready orchestration primitives.
**Architecture Diagram:**
The following diagram illustrates the five-layer architecture and the roles of each component:
### Layer 1: Message Schema Layer
Defines strongly-typed, Pydantic-validated contracts (`ClientMessage`, `ServerMessage`) for message direction, purpose, and task transitions. All messages are validated at schema level, preventing malformed messages from entering the protocol pipeline, enabling early error detection and simplifying debugging.
| Responsibility | Implementation | Supports |
|----------------|----------------|----------|
| Message contracts | Pydantic models with validation | Human-readable + machine-verifiable |
| Structured metadata | System info, capabilities | Unified capability discovery (G2) |
| ID correlation | Explicit request/response linking | Deterministic ordering (G4) |
### Layer 2: Transport Abstraction Layer
Provides protocol-agnostic `Transport` interface with production-grade WebSocket implementation. The abstraction layer allows swapping transports without changing protocol logic, supporting future protocol evolution.
| Feature | Benefit | Goals |
|---------|---------|-------|
| Configurable pings/timeouts | Connection health monitoring | G3 |
| Large payload support | Handles complex task definitions | G1 |
| Decoupled transport logic | Future extensibility (HTTP/3, gRPC) | G5 |
| Low-latency persistent sessions | Eliminates per-request overhead | G1 |
### Layer 3: Protocol Orchestration Layer
Implements modular handlers for registration, task execution, heartbeat, and command dispatch. Each handler is independently testable and replaceable, supporting composable extensibility (G5) while maintaining ordered state transitions (G4).
| Component | Purpose | Design |
|-----------|---------|--------|
| `AIPProtocol` base | Common handler infrastructure | Extensible base class |
| Handler modules | Registration, tasks, heartbeat, commands | Pluggable handlers |
| Middleware hooks | Logging, metrics, authentication | Composable extensions (G5) |
| State transitions | Ordered message processing | Deterministic ordering (G4) |
**Related Documentation:**
- [Complete message reference](./messages.md)
- [Protocol implementation details](./protocols.md)
### Layer 4: Resilience and Health Management Layer
!!!warning "Fault Tolerance"
This layer guarantees fine-grained reliability (G3) and seamless task continuity under transient disconnections (G6), preventing cascade failures.
Encapsulates reliability mechanisms ensuring operational continuity under failures:
| Component | Mechanism | Goals |
|-----------|-----------|-------|
| `HeartbeatManager` | Periodic keepalive signals | G3 |
| `TimeoutManager` | Configurable timeout policies | G3 |
| `ReconnectionStrategy` | Exponential backoff with jitter | G6 |
| Session recovery | Automatic state restoration | G6 |
[→ Resilience implementation details](./resilience.md)
### Layer 5: Endpoint Orchestration Layer
Provides role-specific facades integrating lower layers into deployable components. These endpoints unify connection lifecycle, task routing, and health monitoring across roles, reinforcing G1–G6 through consistent implementation of lower-layer capabilities.
| Endpoint | Role | Responsibilities |
|----------|------|------------------|
| `ConstellationEndpoint` | Orchestrator | Global agent registry, task assignment, DAG coordination |
| `DeviceServerEndpoint` | Server | WebSocket connection management, task dispatch, result aggregation |
| `DeviceClientEndpoint` | Executor | Local task execution, MCP tool invocation, telemetry reporting |
**Endpoint Integration Benefits:**
- ✅ Connection lifecycle management (G1, G6)
- ✅ Role-specific protocol variants (G5)
- ✅ Health monitoring integration (G3)
- ✅ Task routing and session management (G4)
[→ Endpoint setup guide](./endpoints.md)
## Architecture Benefits
Together, these layers form a vertically integrated stack that enables UFO² to maintain **correctness and availability** under challenging conditions:
| Challenge | How AIP Addresses It | Layers Involved |
|-----------|----------------------|-----------------|
| **DAG Evolution** | Deterministic ordering, extensible message types | L1, L3, L4, L5 (G4, G5) |
| **Agent Churn** | Heartbeats, reconnection, session recovery | L4, L5 (G3, G6) |
| **Heterogeneous Environments** | Persistent sessions, multi-source profiling | L1, L2, L5 (G1, G2) |
| **Transient Failures** | Timeout management, automatic recovery | L4 (G3, G6) |
| **Protocol Evolution** | Transport abstraction, middleware hooks | L2, L3 (G5) |
AIP transforms distributed workflow execution into a **coherent, safe, and adaptive system** where reasoning and execution converge seamlessly across diverse agents and environments.
## Core Capabilities
### Agent Registration & Profiling
Each agent is represented by an **AgentProfile** combining data from three sources for comprehensive capability discovery, supporting heterogeneous capability unification (G2):
| Source | Provider | Information |
|--------|----------|-------------|
| **User Config** | ConstellationClient | Endpoint URLs, user preferences, device identity |
| **Service Manifest** | Device Agent Service | Supported tools, capabilities, operational metadata |
| **Client Telemetry** | Device Agent Client | OS, hardware specs, GPU status, runtime metrics |
**Benefits of Multi-Level Profiling:**
- ✅ Accurate task allocation based on real-time capabilities (G2)
- ✅ Transparent adaptation to environmental changes (e.g., GPU availability)
- ✅ No manual updates needed when device state changes
- ✅ Informed scheduling decisions at scale
!!!tip "Dynamic Profile Updates"
Client telemetry continuously refreshes, so the orchestrator always sees current device state—critical for GPU-aware scheduling or cross-device load balancing (G2).
[→ See detailed registration flow](./protocols.md)
### Task Dispatch & Result Delivery
AIP uses **long-lived WebSocket sessions** that span multiple task executions, eliminating per-request connection overhead and preserving context (G1).
**Task Execution Sequence:**
The following sequence diagram shows the complete lifecycle of a task from assignment to completion, including intermediate execution steps and state updates:
```mermaid
sequenceDiagram
participant CC as ConstellationClient
participant DAS as Device Service
participant DAC as Device Client
CC->>DAS: TASK message (TaskStar)
DAS->>DAC: Stream task payload
DAC->>DAC: Execute using MCP tools
DAC->>DAS: Stream execution logs
DAS->>CC: TASK_END (status, logs, results)
CC->>CC: Update TaskConstellation
CC->>CC: Notify ConstellationAgent
```
Each arrow represents a message exchange, with vertical lifelines showing the temporal ordering of events. Note how logs stream back during execution, enabling real-time monitoring.
| Stage | Message Type | Content |
|-------|-------------|---------|
| Assignment | `TASK` | TaskStar definition, target device, commands |
| Execution | (internal) | MCP tool invocations, local computation |
| Reporting | `TASK_END` | Status, logs, evaluator outputs, results |
!!!warning "Asynchronous Execution"
Tasks execute asynchronously. The orchestrator may assign multiple tasks to different devices simultaneously, with results arriving in non-deterministic order.
**Related Documentation:**
- [Message format details](./messages.md)
- [TaskConstellation documentation](../galaxy/constellation/task_constellation.md)
- [TaskStar (task nodes) documentation](../galaxy/constellation/task_star.md)
### Command Execution
Within each task, AIP executes **individual commands** deterministically with preserved ordering, enabling precise control and error handling (G4).
**Command Structure:**
| Field | Purpose | Example |
|-------|---------|---------|
| `tool_name` | Tool/action name | `"click_input"` |
| `parameters` | Typed arguments | `{"target": "Save Button", "button": "left"}` |
| `tool_type` | Category | `"action"` or `"data_collection"` |
| `call_id` | Unique identifier | `"cmd_001"` |
**Execution Guarantees:**
- ✅ **Sequential execution** within a session (deterministic order) (G4)
- ✅ **Command batching** supported (reduces network overhead)
- ✅ **Structured results** with status codes and error details
- ✅ **Timeout propagation** for precise recovery strategies (G3)
**Command Batching Example:**
```json
{
"actions": [
{"tool_name": "click", "parameters": {"target": "File"}, "call_id": "1"},
{"tool_name": "click", "parameters": {"target": "Save As"}, "call_id": "2"},
{"tool_name": "type", "parameters": {"text": "document.pdf"}, "call_id": "3"}
]
}
```
All three commands sent in one message, executed sequentially.
[→ See command execution protocol](./protocols.md)
## Message Protocol Overview
All AIP messages use **Pydantic models** for automatic validation, serialization, and type safety.
### Bidirectional Message Types
| Direction | Message Type | Purpose |
|-----------|--------------|---------|
| **Client → Server** | `REGISTER` | Initial capability advertisement |
| | `COMMAND_RESULTS` | Return command execution results |
| | `TASK_END` | Notify task completion |
| | `HEARTBEAT` | Keepalive signal |
| | `DEVICE_INFO_RESPONSE` | Device telemetry update |
| **Server → Client** | `TASK` | Task assignment |
| | `COMMAND` | Command execution request |
| | `DEVICE_INFO_REQUEST` | Request telemetry refresh |
| | `HEARTBEAT` | Keepalive acknowledgment |
| **Bidirectional** | `ERROR` | Error condition reporting |
**Message Correlation:**
Every message includes:
- `timestamp`: ISO 8601 formatted
- `request_id` / `response_id`: Unique identifier
- `prev_response_id`: Links responses to requests
- `session_id`: Session context
[→ Complete message reference](./messages.md)
## Resilient Connection Protocol
!!!warning "Network Instability Handling (G3, G6)"
AIP ensures **continuous orchestration** even under transient network failures or device disconnections through fine-grained reliability mechanisms and transparent reconnection.
### Device Disconnection Flow
**Connection State Transitions:**
This state diagram illustrates how devices transition between connection states and the actions triggered at each transition:
```mermaid
stateDiagram-v2
[*] --> CONNECTED
CONNECTED --> DISCONNECTED: Connection lost
DISCONNECTED --> CONNECTED: Reconnection succeeds
DISCONNECTED --> [*]: Timeout / Manual removal
note right of DISCONNECTED
• Excluded from scheduling
• Tasks marked FAILED
• Auto-reconnect triggered
end note
```
The `DISCONNECTED` state acts as a quarantine zone where the device is temporarily removed from the scheduling pool while auto-reconnection attempts are made. If reconnection fails after timeout, the device is permanently removed.
| Event | Orchestrator Action | Device Action |
|-------|---------------------|---------------|
| **Device disconnects** | Mark as `DISCONNECTED` Exclude from scheduling Trigger auto-reconnect (G6) | N/A |
| **Reconnection succeeds** | Mark as `CONNECTED` Resume scheduling | Session restored (G6) |
| **Disconnect during task** | Mark tasks as `FAILED` Propagate to ConstellationAgent Trigger DAG edit | N/A |
### ConstellationClient Disconnection
!!!danger "Bidirectional Fault Handling"
When the **ConstellationClient** disconnects, all Device Agent Services:
1. Receive termination signal
2. **Abort all ongoing tasks** tied to that client
3. Prevent resource leakage and zombie processes
4. Maintain end-to-end consistency
**Guarantees:**
- ✅ No orphaned tasks
- ✅ Synchronized state across client-server boundary
- ✅ Rapid recovery when connection restored (G6)
- ✅ Consistent TaskConstellation state (G4)
[→ See resilience implementation](./resilience.md)
## Extensibility Mechanisms
AIP provides multiple extension points for domain-specific needs without modifying the core protocol, supporting composable extensibility (G5).
### 1. Protocol Middleware
Add custom processing to message pipeline:
```python
from aip.protocol.base import ProtocolMiddleware
class AuditMiddleware(ProtocolMiddleware):
async def process_outgoing(self, msg):
log_to_audit_trail(msg)
return msg
async def process_incoming(self, msg):
log_to_audit_trail(msg)
return msg
```
### 2. Custom Message Handlers
Register handlers for new message types:
```python
protocol.register_handler("custom_type", handle_custom_message)
```
### 3. Transport Layer
Pluggable transport (default: WebSocket) (G5):
```python
from aip.transport import CustomTransport
protocol.transport = CustomTransport(config)
```
[→ See extensibility guide](./protocols.md)
## Integration with UFO² Ecosystem
| Component | Integration Point | Benefit |
|-----------|-------------------|---------|
| **MCP Servers** | Command execution model aligns with MCP message formats | Unified interface for system actions and LLM tool calls |
| **TaskConstellation** | Real-time state synchronization via AIP messages | Planning DAG always reflects distributed execution state |
| **Configuration System** | Agent endpoints, capabilities managed via UFO² config | Centralized management, type-safe validation |
| **Logging & Monitoring** | Comprehensive logging at all protocol layers | Debugging, performance monitoring, audit trails |
AIP abstracts network/device heterogeneity, allowing the orchestrator to treat all agents as **first-class citizens** in a single event-driven control plane.
**Related Documentation:**
- [TaskConstellation (DAG orchestrator)](../galaxy/constellation/task_constellation.md)
- [ConstellationAgent (orchestration agent)](../galaxy/constellation_agent/overview.md)
- [MCP Integration Guide](../mcp/overview.md)
- [Configuration System](../configuration/system/system_config.md)
**Next Steps:**
- 📖 [Message Reference](./messages.md) - Complete message type documentation
- 🔧 [Protocol Guide](./protocols.md) - Implementation details and best practices
- 🌐 [Transport Layer](./transport.md) - WebSocket configuration and optimization
- 🔌 [Endpoints](./endpoints.md) - Endpoint setup and usage patterns
- 🛡️ [Resilience](./resilience.md) - Connection management and fault tolerance
## Summary
AIP transforms distributed workflow execution into a **coherent, safe, and adaptive system** where reasoning and execution converge seamlessly across diverse agents and environments.
**Key Takeaways:**
| Aspect | Impact | Goals |
|--------|--------|-------|
| **Persistence** | Long-lived connections reduce overhead, maintain context | G1 |
| **Low Latency** | WebSocket enables real-time event propagation | G1 |
| **Capability Discovery** | Multi-source profiling unifies heterogeneous agents | G2 |
| **Reliability** | Heartbeats, timeouts, auto-reconnection ensure graceful degradation | G3, G6 |
| **Determinism** | Sequential command execution, explicit ID correlation | G4 |
| **Extensibility** | Middleware hooks, pluggable transports, custom handlers | G5 |
| **Developer UX** | Strongly-typed messages, clear errors reduce integration effort | G5 |
By decomposing orchestration into five logical layers—each addressing specific reliability and adaptability concerns—AIP enables UFO² to maintain **correctness and availability** under DAG evolution (G4, G5), agent churn (G3, G6), and heterogeneous execution environments (G1, G2).
---
### Documents Docs Client Overview (`documents/docs/client/overview.md`)
# UFO Client Overview
The **UFO Client** runs on target devices and serves as the **execution layer** of UFO's distributed agent system. It manages MCP (Model Context Protocol) servers, executes commands deterministically, and communicates with the Agent Server through the Agent Interaction Protocol (AIP).
**Quick Start:** Jump to the [Quick Start Guide](./quick_start.md) to connect your device. Make sure the [Agent Server](../server/quick_start.md) is running first.
---
## 🎯 What is the UFO Client?
```mermaid
graph LR
subgraph "Agent Server (Brain)"
Reasoning[High-Level Reasoning]
Planning[Task Planning]
Strategy[Strategy Selection]
end
subgraph "Agent Client (Hands)"
Execution[Command Execution]
Tools[Tool Management]
Reporting[Status Reporting]
end
subgraph "Device Environment"
Apps[Applications]
Files[File System]
UI[User Interface]
end
Reasoning -->|Directives| Execution
Planning -->|Commands| Execution
Strategy -->|Tasks| Execution
Execution --> Tools
Tools --> Apps
Tools --> Files
Tools --> UI
Reporting -->|Results| Reasoning
style Reasoning fill:#bbdefb
style Execution fill:#c8e6c9
style Tools fill:#fff9c4
```
**The UFO Client is a stateless execution agent that:**
| Capability | Description | Benefit |
|------------|-------------|---------|
| **🔧 Executes Commands** | Translates server directives into concrete actions | Deterministic, reliable execution |
| **🛠️ Manages MCP Servers** | Orchestrates local and remote tool interfaces | Extensible tool ecosystem |
| **📊 Reports Device Info** | Provides hardware and software profile to server | Intelligent task assignment |
| **📡 Communicates via AIP** | Maintains persistent WebSocket connection | Real-time bidirectional communication |
| **🚫 Remains Stateless** | Executes directives without high-level reasoning | Independent updates, simple architecture |
**Stateless Design Philosophy:** The client focuses purely on execution. All reasoning and decision-making happens on the server, allowing independent updates to server logic and client tools, simple client architecture, intelligent orchestration of multiple clients, and resource-efficient operation.
**Architecture:** The UFO Client is part of UFO's distributed **server-client architecture**, where it handles command execution and resource access while the [Agent Server](../server/overview.md) handles orchestration and decision-making. See [Server-Client Architecture](../infrastructure/agents/server_client_architecture.md) for the complete design rationale, communication protocols, and deployment patterns.
---
## 🏗️ Architecture
The client implements a **layered architecture** separating communication, execution, and tool management for maximum flexibility and maintainability.
```mermaid
graph TB
subgraph "Communication"
WSC[WebSocket Client AIP Protocol]
end
subgraph "Orchestration"
UFC[UFO Client]
CM[Computer Manager]
end
subgraph "Execution"
COMP[Computer]
MCPM[MCP Manager]
end
subgraph "Tools"
LOCAL[Local MCP Servers]
REMOTE[Remote MCP Servers]
end
WSC --> UFC
UFC --> CM
CM --> COMP
COMP --> MCPM
MCPM --> LOCAL
MCPM --> REMOTE
style WSC fill:#bbdefb
style UFC fill:#c8e6c9
style COMP fill:#fff9c4
style MCPM fill:#ffcdd2
```
### Core Components
| Component | Responsibility | Key Features | Documentation |
|-----------|---------------|--------------|---------------|
| **WebSocket Client** | AIP communication | • Connection management • Registration • Heartbeat monitoring • Message routing | [Details →](./websocket_client.md) |
| **UFO Client** | Execution orchestration | • Command execution • Result aggregation • Error handling • Session management | [Details →](./ufo_client.md) |
| **Computer Manager** | Multi-computer abstraction | • Computer instance management • Namespace routing • Resource isolation | [Details →](./computer_manager.md) |
| **Computer** | Tool management | • MCP server registration • Tool registry • Execution isolation • Thread pool management | [Details →](./computer.md) |
| **MCP Server Manager** | MCP lifecycle | • Server creation • Configuration loading • Connection pooling • Health monitoring | [MCP Documentation →](../mcp/overview.md) |
| **Device Info Provider** | System profiling | • Hardware detection • Capability reporting • Platform identification • Feature enumeration | [Details →](./device_info.md) |
For detailed component documentation:
- [WebSocket Client](./websocket_client.md) - AIP protocol implementation
- [UFO Client](./ufo_client.md) - Execution orchestration
- [Computer Manager](./computer_manager.md) - Multi-computer management
- [Device Info Provider](./device_info.md) - System profiling
- [MCP Integration](../mcp/overview.md) - MCP server management (comprehensive documentation)
---
## 🚀 Key Capabilities
### 1. Deterministic Command Execution
The client executes commands **exactly as specified** without interpretation or reasoning, ensuring predictable behavior.
```mermaid
sequenceDiagram
participant Server
participant Client as UFO Client
participant Computer
participant Tool as MCP Tool
Server->>Client: COMMAND (AIP)
Client->>Computer: Execute Command
Computer->>Computer: Lookup Tool
Computer->>Tool: Execute with Timeout
Tool-->>Computer: Result
Computer-->>Client: Aggregated Result
Client-->>Server: COMMAND_RESULTS (AIP)
```
**Execution Flow:**
| Step | Action | Purpose |
|------|--------|---------|
| 1️⃣ **Receive** | Get structured command from server via AIP | Ensure well-formed input |
| 2️⃣ **Route** | Dispatch to appropriate computer instance | Support multi-namespace execution |
| 3️⃣ **Lookup** | Find tool in MCP registry | Dynamic tool resolution |
| 4️⃣ **Execute** | Run tool in isolated thread pool | Fault isolation and timeout protection |
| 5️⃣ **Aggregate** | Combine results from multiple tools | Structured response format |
| 6️⃣ **Return** | Send results back to server via AIP | Complete the execution loop |
**Execution Guarantees:**
- **Isolation**: Each tool runs in separate thread pool
- **Timeouts**: Configurable timeout (default: 6000 seconds/100 minutes)
- **Fault Tolerance**: One failed tool doesn't crash entire client
- **Thread Safety**: Concurrent tool execution supported
- **Error Reporting**: Structured errors returned to server
### 2. MCP Server Management
The client manages a collection of **MCP (Model Context Protocol) servers** to provide diverse tool access for automation tasks. The client is responsible for registering, managing, and executing these tools, while the [Agent Server](../server/overview.md) handles command orchestration. See [Server-Client Architecture](../infrastructure/agents/server_client_architecture.md#client-command-execution-and-resource-access) for how MCP integration fits into the overall architecture.
**MCP Server Categories:**
**Data Collection Servers** gather information from the device:
| Server Type | Tools Provided | Use Cases |
|-------------|---------------|-----------|
| **System Info** | CPU, memory, disk stats | Resource monitoring |
| **Application State** | Running apps, windows | Context awareness |
| **Screenshot** | Screen capture | Visual verification |
| **UI Element Detection** | Control trees, accessibility | UI automation |
Example Tools: `get_system_info()`, `list_running_apps()`, `capture_screenshot()`, `get_ui_tree()`
**Action Servers** perform actions on the device:
| Server Type | Tools Provided | Use Cases |
|-------------|---------------|-----------|
| **GUI Automation** | Keyboard, mouse, clicks | UI interaction |
| **Application Control** | Launch, close, focus | App management |
| **File System** | Read, write, delete | File operations |
| **Command Execution** | Shell commands | System automation |
Example Tools: `click_button(label)`, `type_text(text)`, `open_application(name)`, `execute_command(cmd)`
**Server Types:**
| Type | Deployment | Pros | Cons |
|------|------------|------|------|
| **Local MCP Servers** | Run in same process via FastMCP | Fast, no network overhead | Limited to local capabilities |
| **Remote MCP Servers** | Connect via HTTP/SSE | Scalable, shared services | Network latency, external dependency |
**Example MCP Server Configuration:**
```yaml
mcp_servers:
data_collection:
- name: "system_info"
type: "local"
class: "SystemInfoServer"
- name: "ui_detector"
type: "local"
class: "UIDetectionServer"
action:
- name: "gui_automation"
type: "local"
class: "GUIAutomationServer"
- name: "file_ops"
type: "remote"
url: "http://localhost:8080/mcp"
```
See [MCP Integration](../mcp/overview.md) for comprehensive MCP server documentation.
### 3. Device Profiling
The client automatically collects and reports **device information** to enable the server to make intelligent task routing decisions.
**Device Profile Structure:**
```json
{
"device_id": "device_windows_001",
"platform": "windows",
"platform_type": "computer",
"os_version": "10.0.22631",
"system_info": {
"cpu_count": 8,
"memory_total_gb": 16.0,
"disk_total_gb": 512.0,
"hostname": "DESKTOP-ABC123",
"ip_address": "192.168.1.100"
},
"supported_features": [
"gui_automation",
"cli_execution",
"browser_control",
"office_integration",
"windows_apps"
],
"installed_applications": [
"Chrome",
"Excel",
"PowerPoint",
"VSCode"
],
"screen_resolution": "1920x1080",
"connected_at": "2025-11-05T10:30:00Z"
}
```
**Profile Usage on Server:**
```mermaid
graph LR
Client[Client Detects Device Info]
Server[Server Stores Profile]
Route[Server Routes Tasks]
Client -->|Report Profile| Server
Server -->|Match Requirements| Route
Route -->|Dispatch Task| Client
style Client fill:#bbdefb
style Server fill:#c8e6c9
style Route fill:#fff9c4
```
**Server Uses Profile For:**
| Use Case | Example Logic |
|----------|--------------|
| **Platform Matching** | Route Excel task to Windows device |
| **Capability Filtering** | Only send browser tasks to devices with Chrome |
| **Load Balancing** | Distribute tasks based on CPU/memory |
| **Failure Recovery** | Reassign task if device disconnects |
See [Device Info Provider](./device_info.md) for detailed profiling documentation.
### 4. Resilient Communication
Robust, fault-tolerant communication with the server using strongly-typed AIP messages.
**Connection Lifecycle:**
```mermaid
stateDiagram-v2
[*] --> Disconnected
Disconnected --> Connecting: Initiate Connection
Connecting --> Registering: WebSocket Established
Registering --> Connected: Registration Success
Connecting --> Disconnected: Connection Failed
Registering --> Disconnected: Registration Failed
Connected --> Heartbeating: Start Heartbeat Loop
Heartbeating --> Heartbeating: Send/Receive Heartbeat
Heartbeating --> Disconnected: Heartbeat Timeout
Heartbeating --> Disconnected: WebSocket Closed
Disconnected --> Connecting: Retry (Exponential Backoff)
note right of Connected
• Receive commands
• Execute tasks
• Report results
end note
note right of Heartbeating
Default interval: 30s
Timeout: 60s
end note
```
**Connection Features:**
| Feature | Description | Configuration |
|---------|-------------|---------------|
| **Auto Registration** | Registers with server on connect | Device ID, platform, capabilities |
| **Exponential Backoff** | Smart retry on connection failure | Max retries: 5 (default) |
| **Heartbeat Monitoring** | Keep-alive mechanism | Interval: 30s (configurable) |
| **Graceful Reconnection** | Resume operation after disconnect | Auto-reconnect on network recovery |
**Message Types:**
| Message | Direction | Purpose |
|---------|-----------|---------|
| `REGISTRATION` | Client → Server | Register device with capabilities |
| `REGISTRATION_ACK` | Server → Client | Confirm registration |
| `HEARTBEAT` | Client ↔ Server | Keep connection alive |
| `COMMAND` | Server → Client | Execute task command |
| `COMMAND_RESULTS` | Client → Server | Return execution results |
| `ERROR` | Client → Server | Report execution errors |
See [WebSocket Client](./websocket_client.md) and [AIP Protocol](../aip/overview.md) for protocol details.
---
## 📋 Workflow Examples
### Client Initialization & Registration
```mermaid
sequenceDiagram
participant Main as Client Main
participant MCP as MCP Manager
participant WSC as WebSocket Client
participant Server
Main->>MCP: Initialize MCP Servers
MCP-->>Main: Server Registry Ready
Main->>WSC: Create Client & Connect
WSC->>Server: WebSocket Connect
Server-->>WSC: Connection Established
WSC->>WSC: Collect Device Info
WSC->>Server: REGISTRATION
Server-->>WSC: REGISTRATION_ACK
WSC->>WSC: Start Heartbeat Loop
loop Every 30 seconds
WSC->>Server: HEARTBEAT
Server-->>WSC: HEARTBEAT_ACK
end
Note over WSC,Server: Ready to Execute Commands
```
**Initialization Steps:**
| Step | Action | Details |
|------|--------|---------|
| 1️⃣ **Parse Args** | Process command-line arguments | `--client-id`, `--ws-server`, `--platform` |
| 2️⃣ **Load Config** | Load UFO configuration | MCP servers, tools, settings |
| 3️⃣ **Init MCP** | Initialize MCP server manager | Create local/remote servers |
| 4️⃣ **Create Managers** | Create computer manager | Register MCP servers with computers |
| 5️⃣ **Connect** | Establish WebSocket connection | Connect to server |
| 6️⃣ **Register** | Send device profile | Platform, capabilities, system info |
| 7️⃣ **Heartbeat** | Start keep-alive loop | Default: 30s interval |
| 8️⃣ **Listen** | Wait for commands | Ready for task execution |
### Command Execution Flow
```mermaid
sequenceDiagram
participant Server
participant Client as UFO Client
participant Comp as Computer
participant Tool as MCP Tool
Server->>Client: COMMAND {type: "click_button", args: {...}}
Client->>Comp: execute_command()
Comp->>Comp: find_tool("click_button")
alt Tool Found
Comp->>Tool: execute(args)
Note over Tool: Thread Pool Execution 6000s timeout
Tool-->>Comp: Success
Comp-->>Client: Result
Client-->>Server: COMMAND_RESULTS {status: "completed"}
else Tool Not Found
Comp-->>Client: Error
Client-->>Server: ERROR {error: "Tool not found"}
end
```
---
## 🖥️ Platform Support
The client supports multiple platforms with platform-specific tool implementations.
| Platform | Status | Features | Native Tools |
|----------|--------|----------|--------------|
| **Windows** | ✅ **Full Support** | • UI Automation (UIAutomation API) • COM API integration • Office automation • Windows-specific apps | PowerShell, Registry, WMI, Win32 API |
| **Linux** | ✅ **Full Support** | • Bash automation • X11/Wayland GUI tools • Package managers • Linux applications | bash, apt/yum, systemd, xdotool |
| **macOS** | 🚧 **In Development** | • macOS applications • Automator integration • AppleScript support | osascript, Automator, launchctl |
| **Mobile** | 🔮 **Planned** | • Touch interface • Mobile apps • Gesture control | ADB (Android), XCTest (iOS) |
**Platform Detection:**
- **Automatic**: Detected via `platform.system()` on startup
- **Override**: Use `--platform` flag to specify manually
- **Validation**: Server validates platform matches task requirements
**Platform-Specific Example:**
**Windows:**
```python
# Windows-specific tools
tools = [
"open_windows_app(name='Excel')",
"execute_powershell(script='Get-Process')",
"read_registry(key='HKLM\\Software')"
]
```
**Linux:**
```python
# Linux-specific tools
tools = [
"execute_bash(command='ls -la')",
"install_package(name='vim')",
"control_systemd(service='nginx', action='restart')"
]
```
---
## ⚙️ Configuration
### Command-Line Arguments
Start the UFO client with:
```bash
python -m ufo.client.client [OPTIONS]
```
**Available Options:**
| Option | Type | Default | Description | Example |
|--------|------|---------|-------------|---------|
| `--client-id` | `str` | `client_001` | Unique client identifier | `--client-id device_win_001` |
| `--ws-server` | `str` | `ws://localhost:5000/ws` | WebSocket server URL | `--ws-server ws://192.168.1.10:5000/ws` |
| `--ws` | `flag` | `False` | **Enable WebSocket mode** (required) | `--ws` |
| `--max-retries` | `int` | `5` | Connection retry limit | `--max-retries 10` |
| `--platform` | `str` | Auto-detect | Platform override | `--platform windows` |
| `--log-level` | `str` | `WARNING` | Logging verbosity | `--log-level DEBUG` |
**Quick Start Command:**
```bash
# Minimal command (default server)
python -m ufo.client.client --ws --client-id my_device
# Production command (custom server)
python -m ufo.client.client \
--ws \
--client-id device_production_01 \
--ws-server ws://ufo-server.company.com:5000/ws \
--max-retries 10 \
--log-level INFO
```
### UFO Configuration
The client inherits settings from `config_dev.yaml`:
**Key Configuration Sections:**
| Section | Purpose | Example |
|---------|---------|---------|
| **MCP Servers** | Define data collection and action servers | `mcp_servers.data_collection`, `mcp_servers.action` |
| **Tool Settings** | Tool-specific parameters | Timeouts, retries, API keys |
| **Logging** | Log levels, formats, destinations | File logging, console output |
| **Platform Settings** | OS-specific configurations | Windows UI automation settings |
**Sample Configuration:**
```yaml
client:
heartbeat_interval: 30 # seconds
command_timeout: 6000 # seconds (100 minutes)
max_concurrent_tools: 10
mcp_servers:
data_collection:
- name: system_info
type: local
enabled: true
action:
- name: gui_automation
type: local
enabled: true
settings:
click_delay: 0.5
typing_speed: 100 # chars per minute
logging:
level: INFO
format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
file: "logs/client.log"
```
See [Configuration Guide](../configuration/system/overview.md) for comprehensive documentation.
---
## ⚠️ Error Handling
The client is designed to handle various failure scenarios gracefully without crashing.
### Connection Failures
```mermaid
stateDiagram-v2
[*] --> Attempting
Attempting --> Connected: Success
Attempting --> Failed: Error
Failed --> Waiting: Exponential Backoff
Waiting --> Attempting: Retry (2^n seconds)
Failed --> [*]: Max Retries Exceeded
note right of Waiting
Retry Delays:
1st: 2s
2nd: 4s
3rd: 8s
4th: 16s
5th: 32s
end note
```
**Connection Error Handling:**
| Scenario | Client Behavior | Configuration |
|----------|----------------|---------------|
| **Initial Connection Failed** | Exponential backoff retry | `--max-retries` (default: 5) |
| **Connection Lost** | Attempt reconnection | Automatic |
| **Max Retries Exceeded** | Exit with error code | Log error, exit |
| **Server Unreachable** | Log error, retry | Backoff between retries |
### Tool Execution Failures
**Protection Mechanisms:**
| Mechanism | Purpose | Default Value |
|-----------|---------|---------------|
| **Thread Pool Isolation** | Prevent one tool from blocking others | Enabled |
| **Execution Timeout** | Kill hung tools | 6000 seconds (100 minutes) |
| **Exception Catching** | Graceful error handling | All tools wrapped |
| **Error Reporting** | Notify server of failures | Structured error messages |
**Error Handling Example:**
```python
# Client automatically handles tool errors
try:
result = tool.execute(args)
return {"status": "success", "result": result}
except TimeoutError:
return {"status": "error", "error": "Tool execution timeout"}
except Exception as e:
return {"status": "error", "error": str(e)}
```
### Server Disconnection
**Graceful Shutdown Process:**
1. **Detect Disconnection** - WebSocket connection lost
2. **Stop Heartbeat** - Terminate keep-alive loop
3. **Cancel Pending Tasks** - Abort in-progress commands
4. **Attempt Reconnection** - Use exponential backoff
5. **Clean Shutdown** - If max retries exceeded
---
## ✅ Best Practices
### Development Best Practices
**1. Use Unique Client IDs**
```bash
# Bad: Generic ID
--client-id client_001
# Good: Descriptive ID
--client-id device_win_dev_john_laptop
```
**2. Start with INFO Logging**
```bash
# Development: WARNING for normal operation (default)
--log-level WARNING
# Debugging: DEBUG for troubleshooting
--log-level DEBUG
```
**3. Test MCP Connectivity First**
```python
# Verify MCP servers are accessible before running client
from ufo.client.mcp.mcp_server_manager import MCPServerManager
manager = MCPServerManager()
# Test server creation from configuration
```
### Production Best Practices
**1. Use Descriptive Client IDs**
```bash
# Include environment, location, purpose
--client-id device_windows_production_office_01
--client-id device_linux_staging_lab_02
```
**2. Configure Automatic Restart**
**systemd (Linux):**
```ini
[Unit]
Description=UFO Agent Client
After=network.target
[Service]
Type=simple
User=ufo
WorkingDirectory=/opt/ufo
ExecStart=/usr/bin/python3 -m ufo.client.client \
--ws \
--client-id device_linux_prod_01 \
--ws-server ws://ufo-server.internal:5000/ws \
--log-level INFO
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
```
**PM2 (Cross-platform):**
```json
{
"apps": [{
"name": "ufo-client",
"script": "python",
"args": [
"-m", "ufo.client.client",
"--ws",
"--client-id", "device_win_prod_01",
"--ws-server", "ws://ufo-server.internal:5000/ws",
"--log-level", "INFO"
],
"cwd": "C:\\ufo",
"restart_delay": 5000,
"max_restarts": 10
}]
}
```
**3. Monitor Connection Health**
```bash
# Check logs for connection status
tail -f logs/client.log | grep -E "Connected|Disconnected|ERROR"
```
### Security Best Practices
!!! warning "Security Considerations"
| Practice | Description | Implementation |
|----------|-------------|----------------|
| **Use WSS** | Encrypt WebSocket communication | `wss://server:5000/ws` instead of `ws://` |
| **Validate Server** | Verify server certificate | Configure SSL/TLS verification |
| **Restrict Tools** | Limit MCP server access | Only enable necessary tools |
| **Least Privilege** | Run with minimum permissions | Create dedicated user account |
| **Network Isolation** | Use firewalls and VPNs | Restrict server access to internal network |
---
## 🎓 Documentation Map
### Getting Started
| Document | Purpose | When to Read |
|----------|---------|--------------|
| [Quick Start](./quick_start.md) | Connect your device quickly | First time setup |
| [Server Quick Start](../server/quick_start.md) | Understand server-side setup | Before running client |
### Component Details
| Document | Component | Topics Covered |
|----------|-----------|----------------|
| [WebSocket Client](./websocket_client.md) | Communication layer | AIP protocol, connection management |
| [UFO Client](./ufo_client.md) | Orchestration | Session tracking, command execution |
| [Computer Manager](./computer_manager.md) | Multi-computer abstraction | Namespace management, routing |
| [Computer](./computer.md) | Tool management | MCP registry, execution |
| [Device Info](./device_info.md) | System profiling | Hardware detection, capabilities |
| [MCP Integration](./mcp_integration.md) | MCP servers | Server types, configuration |
### Related Documentation
| Document | Topic | Relevance |
|----------|-------|-----------|
| [Server Overview](../server/overview.md) | Server architecture | Understand the other half |
| [AIP Protocol](../aip/overview.md) | Communication protocol | Deep dive into messaging |
| [Configuration](../configuration/system/overview.md) | UFO configuration | Customize behavior |
---
## 🔄 Client vs. Server
Understanding the **clear division** between client and server responsibilities is crucial for effective system design.
**Responsibility Matrix:**
| Aspect | Client (Execution) | Server (Orchestration) |
|--------|-------------------|------------------------|
| **Primary Role** | Execute directives deterministically | Reason about tasks, plan actions |
| **State Management** | Stateless (no session memory) | Stateful (maintains sessions) |
| **Reasoning** | None (pure execution) | Full (high-level decision-making) |
| **Tools** | MCP servers (local/remote) | Agent strategies, prompts, LLMs |
| **Communication** | Device ↔ Server (AIP) | Multi-client coordination |
| **Updates** | Tool implementation changes | Strategy and logic updates |
| **Complexity** | Low (simple execution loop) | High (complex orchestration) |
| **Dependencies** | MCP servers, system APIs | LLMs, databases, client registry |
**Workflow Comparison:**
```mermaid
graph TB
subgraph "Server Workflow"
S1[Receive User Request]
S2[Reason About Task]
S3[Plan Execution Steps]
S4[Select Target Device]
S5[Send Commands]
end
subgraph "Client Workflow"
C1[Receive Command]
C2[Lookup Tool]
C3[Execute Tool]
C4[Return Result]
end
S1 --> S2
S2 --> S3
S3 --> S4
S4 --> S5
S5 -.->|AIP| C1
C1 --> C2
C2 --> C3
C3 --> C4
C4 -.->|AIP| S5
style S1 fill:#bbdefb
style S2 fill:#bbdefb
style S3 fill:#bbdefb
style C1 fill:#c8e6c9
style C2 fill:#c8e6c9
style C3 fill:#c8e6c9
```
**Decoupled Architecture Benefits:**
- Independent Updates: Modify server logic without touching clients
- Flexible Deployment: Run clients on any platform
- Scalability: Add more clients without server changes
- Maintainability: Simpler client code, easier debugging
- Testability: Test client and server independently
---
## 🚀 Next Steps
**1. Run Your First Client**
```bash
# Follow the quick start guide
python -m ufo.client.client \
--ws \
--client-id my_first_device \
--ws-server ws://localhost:5000/ws
```
👉 [Quick Start Guide](./quick_start.md)
**2. Understand Registration Process**
Learn how clients register with the server, device profile structure, and registration acknowledgment.
👉 [Server Quick Start](../server/quick_start.md) - Start server and connect clients
**3. Explore MCP Integration**
Learn about MCP servers, configure custom tools, and create your own MCP servers.
👉 [MCP Integration](../mcp/overview.md)
**4. Configure for Your Environment**
Customize MCP servers, adjust timeouts and retries, and configure platform-specific settings.
👉 [Configuration Guide](../configuration/system/overview.md)
**5. Master the Protocol**
Deep dive into AIP messages, understand message flow, and error handling patterns.
👉 [AIP Protocol](../aip/overview.md)
---
### Documents Docs Galaxy Overview (`documents/docs/galaxy/overview.md`)
# UFO³ — Weaving the Digital Agent Galaxy
From isolated device agents to interconnected constellations — Building the Digital Agent Galaxy
---
## 🚀 What is UFO³ Galaxy?
**UFO³ Galaxy** is a revolutionary **cross-device orchestration framework** that transforms isolated device agents into a unified digital ecosystem. It models complex user requests as **Task Constellations** (星座) — dynamic distributed DAGs where nodes represent executable subtasks and edges capture dependencies across heterogeneous devices.
### 🎯 The Vision
Building truly ubiquitous intelligent agents requires moving beyond single-device automation. UFO³ Galaxy addresses four fundamental challenges in cross-device agent orchestration:
**🔄 Asynchronous Parallelism**
Enabling concurrent task execution across multiple devices while maintaining correctness through event-driven coordination and safe concurrency control
**⚡ Dynamic Adaptation**
Real-time workflow evolution in response to intermediate results, transient failures, and runtime observations without workflow abortion
**🌐 Distributed Coordination**
Reliable, low-latency communication across heterogeneous devices via WebSocket-based Agent Interaction Protocol with fault tolerance
**🛡️ Safety Guarantees**
Formal invariants ensuring DAG consistency during concurrent modifications and parallel execution, verified through rigorous proofs
---
## 🏗️ Architecture
UFO³ Galaxy Layered Architecture — From natural language to distributed execution
### Layered Design
UFO³ Galaxy follows a **hierarchical orchestration model** that separates global coordination from local execution. This architecture enables scalable cross-device orchestration while maintaining consistent control and responsiveness across diverse operating systems and network environments.
#### 🎛️ Hierarchical Control Plane
**ConstellationClient** serves as the **global control plane**, maintaining a live registry of all connected device agents with their:
- Capability profiles and system specifications
- Runtime health metrics and availability status
- Current load and resource utilization
This registry enables intelligent task placement based on device capabilities, avoiding mismatches between task requirements and device capacity.
Each device hosts a **device agent server** that manages local orchestration through persistent WebSocket sessions with ConstellationClient. The server:
- Maintains execution contexts on the host
- Provides unified interface to underlying tools via MCP servers
- Handles task execution, telemetry streaming, and resource monitoring
**Clean separation**: Global orchestration policies are decoupled from device-specific heterogeneity, providing consistent abstraction across endpoints with different OS, hardware, or network conditions.
#### 🔄 Orchestration Flow
1. **DAG Synthesis**: ConstellationClient invokes the **Constellation Agent** to construct a TaskConstellation—a dynamic DAG encoding task decomposition, dependencies, and device mappings
2. **Device Assignment**: Each TaskStar (DAG node) is assigned to suitable device agents based on capability profiles and system load
3. **Asynchronous Execution**: The **Constellation Orchestrator** executes the DAG in an event-driven manner:
- Task completions trigger dependent nodes
- Failures prompt retry, migration, or partial DAG rewrites
- Workflows adapt to real-time system dynamics (device churn, network variability)
**Result**: Highly parallel and resilient execution that sustains workflow completion even as subsets of devices fail or reconnect.
#### 🔌 Cross-Agent Communication
The **Agent Interaction Protocol (AIP)** handles all cross-agent interactions:
- Agent registration and capability synchronization
- Task dispatch and progress reporting
- Result aggregation and telemetry streaming
Built on persistent WebSocket channels, AIP provides:
- **Lightweight**: Minimal overhead for control messages
- **Bidirectional**: Full-duplex communication between client and agents
- **Multiplexed**: Concurrent message streams over single connection
- **Low-latency**: Fast propagation of control signals and state updates
- **Resilient**: Maintains global consistency despite intermittent connectivity
Together, these design elements form a cohesive foundation for orchestrating large-scale, heterogeneous, and adaptive workflows across a resilient multi-device execution fabric.
---
## ✨ Core Design Principles
UFO³ Galaxy realizes cross-device orchestration through **five tightly integrated design principles**:
### 1. 🌟 Declarative Decomposition into Dynamic DAG (Task Constellation)
Natural-language or programmatic requests are decomposed by the **Constellation Agent** into a structured DAG of **TaskStars** (nodes) and **TaskStarLines** (edges) that encode workflow logic, dependencies, and device assignments. This declarative structure is amenable to automated scheduling, introspection, and dynamic modification throughout execution.
**Key Benefits:**
- 📋 **Declarative structure** for automated scheduling
- 🔍 **Runtime introspection** for workflow visibility
- ✏️ **Dynamic rewriting** throughout execution
- 🔄 **Automated orchestration** across heterogeneous devices
```mermaid
graph LR
A[User Intent] --> B[Constellation Agent]
B --> C[Task Constellation DAG]
C --> D[TaskStar 1 Windows]
C --> E[TaskStar 2 Linux GPU]
C --> F[TaskStar 3 Linux CPU]
C --> G[TaskStar 4 Mobile]
E --> H[TaskStar 5]
F --> H
G --> H
```
[Learn more →](constellation/overview.md)
### 2. 🔄 Continuous, Result-Driven Graph Evolution
The **Task Constellation** is a **living data structure** that evolves in response to execution feedback. Intermediate outputs, transient failures, and new observations trigger controlled rewrites—adding diagnostic TaskStars, creating fallbacks, rewiring dependencies, or pruning completed nodes—so the system adapts dynamically instead of aborting on errors.
**Adaptation Mechanisms:**
- 🩺 **Diagnostic TaskStars** added for debugging
- 🛡️ **Fallback creation** for error recovery
- 🔗 **Dependency rewiring** for workflow optimization
- ✂️ **Node pruning** after completion
The **Constellation Agent** operates in two modes:
- **Creation Mode**: Synthesizes initial DAG from user request with device-aware task decomposition
- **Editing Mode**: Incrementally refines constellation based on task completion events and runtime feedback
[Learn more →](constellation_agent/overview.md)
### 3. 🎯 Heterogeneous, Asynchronous, and Safe Orchestration
Each **Task Star** is matched to the most suitable device agent via rich **Agent Profiles** reflecting OS, hardware capabilities, and installed tools. The **Constellation Orchestrator** executes tasks asynchronously, allowing multiple TaskStars to progress in parallel.
**Safety Guarantees:**
- 🔒 **Safe assignment locking** prevents race conditions
- 📅 **Event-driven scheduling** monitors DAG readiness
- ✅ **DAG consistency checks** maintain structural integrity
- 🔄 **Batched edits** ensure atomicity
- 📐 **Formal verification** reinforces correctness
- ⏱️ **Timeout protection** prevents deadlocks
These mechanisms collectively ensure **high efficiency without compromising reliability**.
[Learn more →](constellation_orchestrator/overview.md)
### 4. 🔌 Unified Agent Interaction Protocol (AIP)
Built atop persistent **WebSocket channels**, AIP provides a unified, secure, and fault-tolerant layer for the entire agent ecosystem.
**Core Capabilities:**
- 📝 **Agent registry** with capability profiles
- 🔐 **Session management** for secure communication
- 📤 **Task dispatch** with intelligent routing
- 🎯 **Coordination primitives** for distributed workflows
- 💓 **Heartbeat monitoring** for health tracking
- 🔌 **Automatic reconnection** under network fluctuations
- 🔄 **Retry mechanisms** for reliability
**Architecture Benefits:**
- 🪶 **Lightweight interface** for easy integration
- 🧩 **Extensible design** supports new agent types
- 🛡️ **Fault tolerance** ensures continuous operation
This protocol **abstracts OS and network heterogeneity**, enabling seamless collaboration among agents across desktops, servers, and edge devices, while allowing new agents to integrate seamlessly into the UFO³ ecosystem.
[Learn more →](../aip/overview.md)
### 5. 🛠️ Template-Driven Framework for Device Agents
To **democratize agent creation**, UFO³ provides a **lightweight development template and toolkit** for rapidly building new device agents.
**Development Framework:**
- 📄 **Capability declaration** defines agent profiles
- 🔗 **Environment binding** connects to local systems
- 🧩 **MCP server integration** for tool augmentation
- 🔧 **Modular design** accelerates development
**Model Context Protocol (MCP) Integration:**
- 🎁 **Tool packages** via MCP servers
- 🔌 **Plug-and-play** capability extension
- 🌐 **Cross-platform** tool standardization
- 🚀 **Rapid prototyping** of new agents
This modular architecture maintains consistency across the constellation while enabling developers to extend UFO³ to new platforms (mobile, web, IoT, embedded systems, etc.) with minimal effort.
**🔌 Extensibility:** UFO³ is designed as a **universal framework** that supports developing new device agents for different platforms (mobile, web, IoT, embedded systems, etc.) and applications. Through the **Agent Interaction Protocol (AIP)**, custom device agents can seamlessly integrate into UFO³ Galaxy for coordinated multi-device automation. **Want to build your own device agent?** See our [Creating Custom Device Agents tutorial](../tutorials/creating_device_agent/overview.md) to learn how to extend UFO³ to new platforms.
[Learn more →](agent_registration/overview.md) | [MCP Integration →](../mcp/overview.md)
---
## 🎯 Key Capabilities
### 🌐 Cross-Device Collaboration
Execute workflows that span Windows desktops, Linux servers, GPU clusters, mobile devices, and edge nodes—all from a single natural language request.
### ⚡ Asynchronous Parallelism
Automatically identify parallelizable subtasks and execute them concurrently across devices through:
- **Event-driven scheduling** that continuously monitors DAG topology for ready tasks
- **Non-blocking execution** with Python `asyncio` for maximum concurrency
- **Dynamic adaptation** that integrates new tasks without interrupting running execution
Result: Dramatically reduced end-to-end latency compared to sequential execution.
### 🛡️ Safety & Consistency
- **Three formal invariants** (I1-I3) enforced at runtime for DAG correctness
- **Safe assignment locking** prevents race conditions during concurrent modifications
- **Acyclicity validation** ensures no circular dependencies
- **State merging** algorithm preserves execution progress during dynamic edits
- **Timeout protection** prevents deadlocks from agent failures
### 🔄 Dynamic Workflow Evolution
- **Dual-mode operation**: Separate creation and editing phases with controlled transitions
- **Feedback-driven adaptation**: Task completion events trigger intelligent constellation refinement
- **LLM-powered reasoning**: ReAct architecture for context-aware DAG modifications
- **Undo/redo support**: ConstellationEditor with command pattern for safe interactive editing
### 👁️ Rich Observability
- Real-time constellation visualization with DAG topology updates
- Event bus with publish-subscribe pattern for monitoring task progress
- Detailed execution logs with markdown trajectory support
- Task status tracking (pending, running, completed, failed, cancelled)
- Dependency graph inspection and validation tools
---
## 🎨 Use Cases
### 🖥️ Software Development & Deployment
*"Clone the repo on my laptop, build the Docker image on the GPU server, deploy to staging, and run the test suite on the CI cluster."*
**Workflow DAG:**
```mermaid
graph LR
A[Clone Windows] --> B[Build Linux GPU]
B --> C[Deploy Linux Server]
C --> D[Test Linux CI]
```
### 📊 Data Science Workflows
*"Fetch the dataset from cloud storage, preprocess on the Linux workstation, train the model on the A100 node, and generate a visualization dashboard on my Windows machine."*
**Workflow DAG:**
```mermaid
graph LR
A[Fetch Any] --> B[Preprocess Linux]
B --> C[Train Linux GPU]
C --> D[Visualize Windows]
```
### 📝 Cross-Platform Document Processing
*"Extract data from Excel on Windows, process with Python scripts on Linux, generate PDF reports, and send summary emails."*
**Workflow DAG:**
```mermaid
graph LR
A[Extract Windows] --> B[Process Linux]
B --> C[Generate PDF Windows]
B --> D[Send Email Windows]
```
### 🔬 Distributed System Monitoring
*"Collect server logs from all Linux machines, analyze for errors, generate alerts, and create a consolidated report."*
**Workflow DAG:**
```mermaid
graph LR
A[Collect Logs Linux 1] --> D[Analyze Errors Any]
B[Collect Logs Linux 2] --> D
C[Collect Logs Linux 3] --> D
D --> E[Generate Report Windows]
```
### 🏢 Enterprise Automation
*"Query the database on the server, process the results, update Excel spreadsheets on Windows, and generate PowerPoint presentations."*
**Workflow DAG:**
```mermaid
graph LR
A[Query DB Linux] --> B[Process Data Any]
B --> C[Update Excel Windows]
B --> D[Create PPT Windows]
```
---
## 🗺️ Documentation Structure
### 🚀 [Quick Start](../getting_started/quick_start_galaxy.md)
Get UFO³ Galaxy up and running in minutes with our step-by-step guide
### 👥 [Galaxy Client](client/overview.md)
Device coordination, connection management, and ConstellationClient API
### 🧠 [Constellation Agent](constellation_agent/overview.md)
LLM-driven task decomposition, DAG creation, and dynamic workflow evolution
### ⚙️ [Constellation Orchestrator](constellation_orchestrator/overview.md)
Asynchronous execution engine, event-driven coordination, and safety guarantees
### 📊 [Task Constellation](constellation/overview.md)
DAG structure, TaskStar nodes, TaskStarLine edges, and constellation editor
### 🆔 [Agent Registration](agent_registration/overview.md)
Device registry, agent profiles, and registration flow
### 🌐 [Agent Interaction Protocol](../aip/overview.md)
WebSocket messaging, protocol specification, and communication patterns
### ⚙️ [Configuration](../configuration/system/galaxy_devices.md)
Device pools, capabilities, and orchestration policies
---
## 🚦 Getting Started
Ready to build your Digital Agent Galaxy? Follow these steps:
### 1. Install UFO³
```bash
# Clone the repository
git clone https://github.com/microsoft/UFO.git
cd UFO
# Install dependencies
pip install -r requirements.txt
```
### 2. Configure Device Pool
Create configuration files in `config/galaxy/`:
**`config/galaxy/devices.yaml`** - Define your devices:
```yaml
devices:
- device_id: "windowsagent"
server_url: "ws://localhost:5005/ws"
os: "windows"
capabilities:
- "web_browsing"
- "office_applications"
- "file_management"
metadata:
location: "home_office"
os: "windows"
performance: "medium"
max_retries: 5
- device_id: "linux_agent_1"
server_url: "ws://localhost:5001/ws"
os: "linux"
capabilities:
- "server"
- "python"
- "docker"
metadata:
os: "linux"
performance: "high"
logs_file_path: "/root/log/log1.txt"
auto_connect: true
max_retries: 5
- device_id: "mobile_agent_1"
server_url: "ws://localhost:5002/ws"
os: "android"
capabilities:
- "mobile"
- "adb"
- "ui_automation"
metadata:
os: "android"
performance: "medium"
device_type: "smartphone"
auto_connect: true
max_retries: 5
```
**`config/galaxy/constellation.yaml`** - Configure runtime settings:
```yaml
# Constellation Runtime Settings
CONSTELLATION_ID: "my_constellation"
HEARTBEAT_INTERVAL: 30.0 # Heartbeat interval in seconds
RECONNECT_DELAY: 5.0 # Delay before reconnecting in seconds
MAX_CONCURRENT_TASKS: 6 # Maximum concurrent tasks
MAX_STEP: 15 # Maximum steps per session
# Device Configuration
DEVICE_INFO: "config/galaxy/devices.yaml"
# Logging Configuration
LOG_TO_MARKDOWN: true
```
See [Galaxy Configuration](../configuration/system/galaxy_devices.md) for complete documentation.
### 3. Start Device Agents
On each device, launch the Agent Server. For detailed setup instructions, see the respective quick start guides:
**On Windows:**
See [Windows Agent (UFO²) Quick Start →](../getting_started/quick_start_ufo2.md)
**On Linux:**
See [Linux Agent Quick Start →](../getting_started/quick_start_linux.md)
**On Mobile (Android):**
See [Mobile Agent Quick Start →](../getting_started/quick_start_mobile.md)
### 4. Launch Galaxy Client
**Interactive Mode:**
```bash
python -m galaxy --interactive
```
**Direct Request:**
```bash
python -m galaxy "Your cross-device task here"
```
**Programmatic API:**
```python
from galaxy.galaxy_client import GalaxyClient
async def main():
client = GalaxyClient(session_name="my_session")
await client.initialize()
result = await client.process_request("Your task request")
await client.shutdown()
```
For detailed instructions, see the [Quick Start Guide](../getting_started/quick_start_galaxy.md).
---
## 🔧 System Components
UFO³ Galaxy consists of several integrated components working together:
### Core Components
| Component | Location | Responsibility |
|-----------|----------|----------------|
| **GalaxyClient** | `galaxy/galaxy_client.py` | Session management, user interaction, orchestration coordination |
| **ConstellationClient** | `galaxy/client/constellation_client.py` | Device management, connection lifecycle, task assignment |
| **ConstellationAgent** | `galaxy/agents/constellation_agent.py` | LLM-driven DAG synthesis and evolution, state machine control |
| **TaskConstellationOrchestrator** | `galaxy/constellation/orchestrator/` | Asynchronous execution, event coordination, safety enforcement |
| **TaskConstellation** | `galaxy/constellation/task_constellation.py` | DAG data structure, validation, and modification APIs |
| **DeviceManager** | `galaxy/client/device_manager.py` | WebSocket connections, heartbeat monitoring, message routing |
### Supporting Infrastructure
| Component | Purpose |
|-----------|---------|
| **Event Bus** | Publish-subscribe system for constellation events |
| **Observer Pattern** | Event listeners for visualization and synchronization |
| **Device Registry** | Centralized device information and capability tracking |
| **Agent Profile** | Device metadata and capability declarations |
| **MCP Servers** | Tool augmentation via Model Context Protocol |
For detailed component documentation, see the respective sections in [Documentation Structure](#documentation-structure).
### Technology Stack
| Layer | Technologies |
|-------|-------------|
| **Programming** | Python 3.10+, asyncio, dataclasses |
| **Communication** | WebSockets, JSON-RPC |
| **LLM Integration** | OpenAI API, Azure OpenAI, Gemini, Claude, Custom Models |
| **Tool Augmentation** | Model Context Protocol (MCP) |
| **Configuration** | YAML, Pydantic models |
| **Logging** | Python logging, Rich console, Markdown trajectory |
| **Testing** | pytest, mock agents |
---
## 🌟 From Devices to Constellations to Galaxy
UFO³ represents a paradigm shift in intelligent automation:
- **Single Device** → Isolated agents operating within one OS
- **Task Constellation** → Coordinated multi-device workflows for one task
- **Digital Agent Galaxy** → Interconnected constellations spanning your entire digital estate
Over time, multiple constellations can interconnect, weaving together agents, devices, and capabilities into a self-organizing **Digital Agent Galaxy**. This design elevates cross-device automation from a brittle engineering challenge to a unified orchestration paradigm, where multi-device workflows become naturally expressive, paving the way for large-scale, adaptive, and resilient intelligent ubiquitous computing systems.
---
## 📊 Performance Monitoring & Evaluation
UFO³ Galaxy provides comprehensive performance monitoring and evaluation tools to analyze multi-device workflow execution:
### Automated Metrics Collection
Galaxy automatically collects detailed performance metrics during execution through an event-driven observer pattern:
- **Task Metrics**: Execution times, success rates, bottleneck identification
- **Constellation Metrics**: DAG statistics, parallelism analysis, critical path computation
- **Modification Metrics**: Dynamic editing patterns and adaptation frequency
- **Device Metrics**: Per-device performance and resource utilization
All metrics are captured in real-time without impacting execution performance and saved to structured JSON files for programmatic analysis.
### Trajectory Report
Galaxy automatically generates a comprehensive Markdown trajectory report (`output.md`) documenting the complete execution lifecycle:
```
logs/galaxy//output.md
```
This human-readable report includes:
- Step-by-step execution timeline with agent actions
- Interactive DAG topology visualizations showing constellation evolution
- Detailed task execution logs with results and errors
- Device connection status and coordination events
- Complete before/after constellation states at each step
The trajectory report provides visual debugging and workflow understanding, complementing the quantitative `result.json` metrics.
### Result JSON Format
After each session, Galaxy also generates a comprehensive `result.json` file containing:
```
logs/galaxy//result.json
```
This file includes:
- Complete session metadata and execution timeline
- Task-by-task performance breakdown
- Constellation statistics (parallelism ratio, critical path, max concurrency)
- Modification history showing DAG evolution
- Final results and outcomes
**Example Key Metrics:**
| Metric | Description | Use Case |
|--------|-------------|----------|
| `parallelism_ratio` | Efficiency of parallel execution (total_work / critical_path) | Optimization target |
| `critical_path_length` | Minimum possible execution time | Theoretical performance limit |
| `average_task_duration` | Mean task execution time | Baseline performance |
| `modification_count` | Number of dynamic DAG edits | Adaptability analysis |
### Performance Analysis Tools
```python
import json
# Load session results
with open("logs/galaxy/task_32/result.json", 'r') as f:
result = json.load(f)
# Extract key metrics
metrics = result["session_results"]["metrics"]
task_stats = metrics["task_statistics"]
const_stats = result["session_results"]["final_constellation_stats"]
print(f"✅ Success Rate: {task_stats['success_rate'] * 100:.1f}%")
print(f"⏱️ Avg Task Duration: {task_stats['average_task_duration']:.2f}s")
print(f"🔀 Parallelism Ratio: {const_stats['parallelism_ratio']:.2f}")
```
**Documentation:**
- **[Trajectory Report Guide](./evaluation/trajectory_report.md)** - Complete guide to the human-readable execution log with DAG visualizations
- **[Performance Metrics Guide](./evaluation/performance_metrics.md)** - Comprehensive metrics documentation with analysis examples
- **[Result JSON Reference](./evaluation/result_json.md)** - Complete schema reference and programmatic access guide
---
## 📚 Learn More
- **Research Paper**: [UFO³: Weaving the Digital Agent Galaxy](https://arxiv.org/) *(Coming Soon)*
- **UFO² (Desktop AgentOS)**: [Documentation](../ufo2/overview.md)
- **UFO (Original)**: [GitHub Repository](https://github.com/microsoft/UFO)
---
## 🤝 Contributing
We welcome contributions! Whether you're building new device agents, improving orchestration algorithms, or enhancing the protocol, check out our Contributing Guide on GitHub.
---
## 📄 License
UFO³ Galaxy is released under the MIT License.
---
Transform your distributed devices into a unified digital collective.
UFO³ Galaxy — Where every device is a star, and every task is a constellation.
---
### Documents Docs Linux Overview (`documents/docs/linux/overview.md`)
# LinuxAgent: CLI Task Executor
**LinuxAgent** is a specialized lightweight agent designed for executing command-line instructions on Linux systems. It demonstrates how a standalone device agent can leverage the layered FSM architecture and server-client design to perform intelligent, iterative task execution in a CLI-based environment.
**Quick Links:**
- New to Linux Agent? Start with the [Quick Start Guide](../getting_started/quick_start_linux.md)
- Using as Sub-Agent in Galaxy? See [Using Linux Agent as Galaxy Device](as_galaxy_device.md)
## Architecture Overview
LinuxAgent operates as a single-agent instance that interacts with Linux systems through command-line interface (CLI) commands. Unlike the two-tier architecture of UFO (HostAgent + AppAgent), LinuxAgent uses a simplified single-agent model optimized for shell-based automation.
## Core Responsibilities
LinuxAgent provides the following capabilities for Linux CLI automation:
### Command-Line Execution
LinuxAgent interprets user requests and translates them into appropriate shell commands for execution on Linux systems.
**Example:** User request "Check disk space and create a backup" becomes:
1. Execute `df -h` to check disk space
2. Execute `tar -czf backup.tar.gz /data` to create backup
### System Information Collection
The agent can proactively gather system-level information to inform decision-making:
- Memory usage (`free -h`)
- Disk space (`df -h`)
- Process status (`ps aux`)
- Hardware configuration (`lscpu`, `lshw`)
### Iterative Task Execution
LinuxAgent executes tasks iteratively, evaluating execution outcomes at each step and determining the next action based on results and LLM reasoning.
### Error Handling and Recovery
The agent monitors command execution results (`stdout`, `stderr`, exit codes) and can adapt its strategy when errors occur.
## Key Characteristics
- **Scope**: Single Linux system (CLI-based automation)
- **Lifecycle**: One instance per task session
- **Hierarchy**: Standalone agent (no child agents)
- **Communication**: Direct MCP server integration
- **Control**: 3-state finite state machine with 3-phase processing pipeline
## Execution Workflow
```mermaid
sequenceDiagram
participant User
participant LinuxAgent
participant LLM
participant MCPServer
participant Linux
User->>LinuxAgent: "Check disk space and create backup"
LinuxAgent->>LinuxAgent: State: CONTINUE
LinuxAgent->>LLM: Send prompt with request & context
LLM-->>LinuxAgent: Return command: df -h
LinuxAgent->>MCPServer: execute_command: df -h
MCPServer->>Linux: Execute command
Linux-->>MCPServer: stdout + stderr
MCPServer-->>LinuxAgent: Execution result
LinuxAgent->>LinuxAgent: Update memory
LinuxAgent->>LLM: Send prompt with previous result
LLM-->>LinuxAgent: Return command: tar -czf ...
LinuxAgent->>MCPServer: execute_command: tar -czf backup.tar.gz /data
MCPServer->>Linux: Execute command
Linux-->>MCPServer: stdout + stderr
MCPServer-->>LinuxAgent: Execution result
LinuxAgent->>LinuxAgent: State: FINISH
LinuxAgent-->>User: Task completed
```
## Comparison with UFO Agents
| Aspect | LinuxAgent | HostAgent | AppAgent |
|--------|------------|-----------|----------|
| **Platform** | Linux (CLI) | Windows Desktop | Windows Applications |
| **States** | 3 (CONTINUE, FINISH, FAIL) | 7 states | 6 states |
| **Architecture** | Single-agent | Parent orchestrator | Child executor |
| **Interface** | Command-line | Desktop GUI + Shell | Application GUI + API |
| **Processing Phases** | 3 phases | 4 phases | 4 phases |
| **MCP Tools** | execute_command, get_system_info | Desktop commands | UI + API commands |
## Design Principles
LinuxAgent exemplifies a minimal viable design for single-agent systems with a small state set (only 3 states for deterministic control flow), modular strategies (clear separation between LLM interaction, action execution, and memory updates), well-defined commands (atomic CLI operations isolated in MCP server layer), proactive information gathering (on-demand system info collection), and traceable execution (complete logging of commands, results, and state transitions).
## Deep Dive Topics
Explore the detailed architecture and implementation:
- [State Machine](state.md) - 3-state FSM lifecycle and transitions
- [Processing Strategy](strategy.md) - 3-phase pipeline (LLM, Action, Memory)
- [MCP Commands](commands.md) - CLI execution and system information commands
## Use Cases
LinuxAgent is ideal for:
- **System Administration**: Automated system maintenance and monitoring
- **DevOps Tasks**: Deployment scripts, log analysis, configuration management
- **Data Processing**: File operations, text processing, batch jobs
- **Monitoring & Alerts**: System health checks and automated responses
- **Cross-Device Workflows**: As a sub-agent in Galaxy multi-device orchestration
!!!tip "Galaxy Integration"
LinuxAgent can serve as a device agent in Galaxy's multi-device orchestration framework, executing Linux-specific tasks as part of cross-platform workflows alongside Windows and other devices.
See [Using Linux Agent as Galaxy Device](as_galaxy_device.md) for configuration details.
## Implementation Location
The LinuxAgent implementation can be found in:
```
ufo/
├── agents/
│ ├── agent/
│ │ └── customized_agent.py # LinuxAgent class definition
│ ├── states/
│ │ └── linux_agent_state.py # State machine implementation
│ └── processors/
│ ├── customized/
│ │ └── customized_agent_processor.py # LinuxAgentProcessor
│ └── strategies/
│ └── linux_agent_strategy.py # Processing strategies
```
## Next Steps
To understand LinuxAgent's complete architecture:
1. [State Machine](state.md) - Learn about the 3-state FSM
2. [Processing Strategy](strategy.md) - Understand the 3-phase pipeline
3. [MCP Commands](commands.md) - Explore CLI command execution
For deployment and configuration, see the [Getting Started Guide](../getting_started/quick_start_linux.md).
---
### Documents Docs Mcp Overview (`documents/docs/mcp/overview.md`)
# MCP (Model Context Protocol) - Overview
## What is MCP?
**MCP (Model Context Protocol)** is a standardized protocol that enables UFO² agents to interact with external tools and services in a unified way. It provides a **tool execution framework** where agents can:
- **Collect system state** through data collection servers
- **Execute actions** through action servers
- **Extend capabilities** through custom MCP servers
```mermaid
graph TB
subgraph Agent["UFO² Agent"]
HostAgent[HostAgent]
AppAgent[AppAgent]
end
Computer["Computer (MCP Tool Manager)"]
subgraph DataServers["Data Collection Servers"]
UICollector["UICollector • Screenshots • Window Info"]
HWInfo["Hardware Info • CPU/Memory • System State"]
end
subgraph ActionServers["Action Servers"]
UIExecutor["UIExecutor • Click/Type • UI Automation"]
CLIExecutor["CLI Executor • Shell Commands"]
COMExecutor["COM Executor • API Calls"]
end
HostAgent --> Computer
AppAgent --> Computer
Computer --> DataServers
Computer --> ActionServers
style Agent fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Computer fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style DataServers fill:#e8f5e9,stroke:#388e3c,stroke-width:2px
style ActionServers fill:#fce4ec,stroke:#c2185b,stroke-width:2px
```
MCP serves as the **execution layer** in UFO²'s architecture. While agents make decisions about *what* to do, MCP servers handle *how* to do it by providing concrete tool implementations.
## Key Concepts
### 1. Two Server Types
MCP servers in UFO² are categorized into two types based on their purpose:
| Type | Purpose | Examples | Side Effects | LLM Selectable? |
|------|---------|----------|--------------|-----------------|
| **Data Collection** | Retrieve system state Read-only operations | UI detection, Screenshot, System info | ❌ None | ❌ **No** - Auto-invoked |
| **Action** | Modify system state State-changing operations | Click, Type text, Run command | ✅ Yes | ✅ **Yes** - LLM chooses |
**Server Selection Model:**
- **[Data Collection Servers](./data_collection.md)**: Automatically invoked by the framework to gather context and build observation prompts. Not selectable by LLM.
- **[Action Servers](./action.md)**: LLM agent actively selects which action tool to execute at each step based on the task. Only action tools are LLM-selectable.
**How Action Tools Reach the LLM**: Each action tool's `Annotated` type hints and docstring are automatically extracted and converted into structured instructions that appear in the LLM's prompt. The LLM uses these instructions to understand what each tool does, what parameters it requires, and when to use it. Therefore, developers should write clear, comprehensive docstrings and type annotations - they directly impact the LLM's ability to use the tool correctly.
### 2. Server Deployment Models
UFO² supports three deployment models for MCP servers:
| Model | Description | Benefits | Trade-offs |
|-------|-------------|----------|------------|
| **Local (In-Process)** | Server runs in same process as agent | Fast (no IPC overhead), Simple setup | Shares process resources |
| **HTTP (Remote)** | Server runs as HTTP service (e.g., Port 8006) | Process isolation, Language-agnostic | Network overhead |
| **Stdio (Process)** | Server runs as child process using stdin/stdout | Process isolation, Bidirectional streaming | Platform-specific |
### 3. Namespace Isolation
Each MCP server has a **namespace** that groups related tools together:
```yaml
# Example: HostAgent configuration
HostAgent:
default:
data_collection:
- namespace: UICollector # Namespace for UI detection tools
type: local
action:
- namespace: HostUIExecutor # Namespace for UI automation tools
type: local
- namespace: CommandLineExecutor # Namespace for CLI tools
type: local
```
**Tool Key Format**: `{tool_type}::{tool_name}`
- Example: `data_collection::screenshot` - Screenshot tool in data_collection
- Example: `action::click` - Click tool in action
- Example: `action::run_shell` - Shell command in action
## Key Features
### 1. GUI + API Dual-Mode Automation
**UFO² supports both GUI automation and API-based automation simultaneously.** Each agent can register multiple action servers, combining:
- **GUI Automation**: Windows UI Automation (UIA) - clicking, typing, scrolling when visual interaction is needed
- **API Automation**: Direct COM API calls, shell commands, REST APIs for efficient, reliable operations
**The LLM agent dynamically chooses the best action at each step** based on task requirements, reliability, speed, and availability.
**Example: Word Document Automation**
```yaml
AppAgent:
WINWORD.EXE:
action:
- namespace: WordCOMExecutor # API: Fast, reliable
- namespace: AppUIExecutor # GUI: Visual navigation fallback
- namespace: CommandLineExecutor # Shell: File operations
```
**LLM's Dynamic Selection:**
```
Task: "Create a report with a 3x2 table and bold the title"
Step 1: Insert table
→ LLM selects: WordCOMExecutor::insert_table(rows=3, cols=2)
→ Reason: API is fast, reliable, no GUI navigation needed
Step 2: Navigate to Design tab
→ LLM selects: AppUIExecutor::click_input(id="5", name="Design")
→ Reason: Visual navigation, COM API doesn't expose tab selection
Step 3: Type table header
→ LLM selects: AppUIExecutor::set_edit_text(id="cell_1_1", text="Product")
→ Reason: GUI interaction needed for cell input
Step 4: Bold title text
→ LLM selects: WordCOMExecutor::select_text(text="Report Title")
→ WordCOMExecutor::set_font(font_size=16)
→ Reason: API is more reliable than GUI button clicking
Step 5: Save as PDF
→ LLM selects: WordCOMExecutor::save_as(file_ext=".pdf")
→ Reason: One API call vs. multiple GUI clicks (File → Save As → Format → PDF)
```
**Why Hybrid Automation Matters:**
- **APIs**: ~10x faster, deterministic, no visual dependency
- **GUI**: Handles visual elements, fallback when API unavailable
- **LLM Decision**: Chooses optimal approach per step, not locked into one mode
### 2. Multi-Server Per Agent
Each agent can register **multiple action servers**, each providing a different set of tools:
**HostAgent Example:**
```yaml
HostAgent:
default:
data_collection:
- UICollector # Automatically invoked
action:
- HostUIExecutor # LLM selects: Window selection
- CommandLineExecutor # LLM selects: Launch apps, shell commands
```
**AppAgent Example (Word-specific):**
```yaml
AppAgent:
WINWORD.EXE:
data_collection:
- UICollector # Automatically invoked
action:
- WordCOMExecutor # LLM selects: insert_table, select_text, save_as
- AppUIExecutor # LLM selects: click_input, set_edit_text
- CommandLineExecutor # LLM selects: run_shell
```
**HardwareAgent Example (Cross-platform):**
```yaml
HardwareAgent:
default:
data_collection:
- HardwareCollector # Auto-invoked (HTTP remote)
action:
- HardwareExecutor # LLM selects: touch_screen, swipe, press_key (HTTP remote)
```
**At each step, the LLM sees all available action tools and selects the most appropriate one.**
### 3. Process Isolation
MCP servers can run:
- **In-process** (local): Fast, low overhead
- **HTTP** (remote): Process isolation, cross-platform, distributed
- **Stdio** (child process): Sandboxed execution, clean resource management
### 4. Namespace Isolation
Each MCP server has a unique namespace that groups related tools together, preventing naming conflicts and enabling modular organization. See [Namespace Isolation](#3-namespace-isolation) section above for details.
## Architecture
### MCP Server Lifecycle
```mermaid
graph TB
Start([MCP Server Lifecycle])
Config["1. Configuration Loading (mcp.yaml)"]
Manager["2. MCPServerManager Creates BaseMCPServer"]
ServerStart["3. Server.start() • Local: Get from registry • HTTP: Build URL • Stdio: Spawn process"]
Register["4. Computer Registration • List tools from server • Register in tool registry"]
Execute["5. Tool Execution • Agent sends Command • Computer routes to tool • MCP server executes"]
Reset["6. Server.reset() (optional) Reset server state"]
Start --> Config
Config --> Manager
Manager --> ServerStart
ServerStart --> Register
Register --> Execute
Execute --> Reset
style Start fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style Config fill:#e1f5fe,stroke:#0277bd,stroke-width:2px
style Manager fill:#fff9c4,stroke:#f57f17,stroke-width:2px
style ServerStart fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
style Register fill:#fce4ec,stroke:#c2185b,stroke-width:2px
style Execute fill:#e0f2f1,stroke:#00695c,stroke-width:2px
style Reset fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
```
### Component Relationships
```mermaid
graph TB
subgraph Architecture["MCP Architecture"]
Registry["MCPRegistry (Singleton) • Stores server factories • Lazy initialization"]
Manager["MCPServerManager (Singleton) • Creates server instances • Maps server types to classes • Manages server lifecycle"]
subgraph ServerTypes["MCP Server Types"]
Local["Local MCP Server (In-Process)"]
HTTP["HTTP MCP Server (Remote)"]
Stdio["Stdio MCP Server (Child Process)"]
end
Computer["Computer (Per Agent) • Manages multiple MCP servers • Routes commands to tools • Maintains tool registry"]
Registry --> Manager
Manager --> ServerTypes
Manager --> Computer
end
style Architecture fill:#fafafa,stroke:#424242,stroke-width:2px
style Registry fill:#e1f5fe,stroke:#01579b,stroke-width:2px
style Manager fill:#fff9c4,stroke:#f57f17,stroke-width:2px
style ServerTypes fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px
style Local fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style HTTP fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style Stdio fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px
style Computer fill:#fce4ec,stroke:#880e4f,stroke-width:2px
```
## Built-in MCP Servers
UFO² comes with several **built-in MCP servers** that cover common automation scenarios:
### Data Collection Servers
| Namespace | Purpose | Key Tools | Platform |
|-----------|---------|-----------|----------|
| **UICollector** | UI element detection | `get_control_info`, `take_screenshot`, `get_window_list` | Windows |
| **HardwareCollector** | Hardware information | `get_cpu_info`, `get_memory_info` | Cross-platform |
| **MobileDataCollector** | Android device observation | `capture_screenshot`, `get_ui_tree`, `get_device_info`, `get_mobile_app_target_info` | Android (ADB) |
### Action Servers
| Namespace | Purpose | Key Tools | Platform |
|-----------|---------|-----------|----------|
| **HostUIExecutor** | UI automation (host-level) | `click`, `type_text`, `scroll` | Windows |
| **AppUIExecutor** | UI automation (app-level) | `click`, `type_text`, `set_edit_text` | Windows |
| **CommandLineExecutor** | CLI execution | `run_shell` | Cross-platform |
| **WordCOMExecutor** | Word automation | `insert_table`, `select_text`, `format_text` | Windows |
| **ExcelCOMExecutor** | Excel automation | `insert_cell`, `select_range`, `format_cell` | Windows |
| **PowerPointCOMExecutor** | PowerPoint automation | `insert_slide`, `add_text`, `format_shape` | Windows |
| **ConstellationEditor** | Multi-device coordination | `create_task`, `assign_device` | Cross-platform |
| **BashExecutor** | Linux commands | `execute_bash` | Linux |
| **MobileExecutor** | Android device control | `tap`, `swipe`, `type_text`, `launch_app`, `click_control` | Android (ADB) |
!!!example "Tool Examples"
```python
# Data Collection: Take a screenshot
{
"tool_type": "data_collection",
"tool_name": "take_screenshot",
"parameters": {"region": "active_window"}
}
# Action: Click a button
{
"tool_type": "action",
"tool_name": "click",
"parameters": {"control_label": "Submit"}
}
# Action: Run a shell command
{
"tool_type": "action",
"tool_name": "run_shell",
"parameters": {"bash_command": "notepad.exe"}
}
```
## Agent-Specific Configurations
Different agents can have **different MCP configurations** based on their roles:
```yaml
# HostAgent: System-level operations
HostAgent:
default:
data_collection:
- namespace: UICollector
type: local
action:
- namespace: HostUIExecutor
type: local
- namespace: CommandLineExecutor
type: local
# AppAgent: Application-specific operations
AppAgent:
WINWORD.EXE: # Word-specific configuration
data_collection:
- namespace: UICollector
type: local
action:
- namespace: AppUIExecutor
type: local
- namespace: WordCOMExecutor # Word COM API
type: local
reset: true # Reset when switching documents
# HardwareAgent: Remote hardware monitoring
HardwareAgent:
default:
data_collection:
- namespace: HardwareCollector
type: http # Remote server
host: "localhost"
port: 8006
path: "/mcp"
# MobileAgent: Android device automation
MobileAgent:
default:
data_collection:
- namespace: MobileDataCollector
type: http # Remote server
host: "localhost"
port: 8020
path: "/mcp"
action:
- namespace: MobileExecutor
type: http
host: "localhost"
port: 8021
path: "/mcp"
```
**Configuration Hierarchy:**
Agent configurations follow this hierarchy:
1. **Agent Name** (e.g., `HostAgent`, `AppAgent`)
2. **Sub-type** (e.g., `default`, `WINWORD.EXE`)
3. **Tool Type** (e.g., `data_collection`, `action`)
4. **Server List** (array of server configurations)
## Key Features
### 1. Process Isolation with Reset
Some MCP servers support **state reset** to ensure clean execution:
```yaml
AppAgent:
WINWORD.EXE:
action:
- namespace: WordCOMExecutor
type: local
reset: true
**When to use reset:**
- Server state is cleared when switching contexts
- Prevents state leakage between tasks
- Useful for stateful tools (e.g., COM APIs)
### 2. Thread Isolation
MCP tools execute in **isolated thread pools** to prevent blocking:
```python
# From Computer class
self._executor = ThreadPoolExecutor(max_workers=10)
self._tool_timeout = 6000 # 100 minutes
```
**Benefits**:
- Prevents blocking the main event loop
- Protects WebSocket connections from timeouts
- Enables concurrent tool execution
**Timeout Protection:** If a tool takes longer than 6000 seconds, it will be cancelled and return a timeout error. Adjust `_tool_timeout` for long-running operations.
### 3. Dynamic Server Management
Add or remove MCP servers at runtime:
```python
# Add a custom server
await computer.add_server(
namespace="CustomTools",
mcp_server=custom_server,
tool_type="action"
)
# Remove a server
await computer.delete_server(
namespace="CustomTools",
tool_type="action"
)
```
### 4. Tool Introspection
Use meta tools to discover available tools:
```python
# List all action tools
tool_call = MCPToolCall(
tool_key="action::list_tools",
tool_name="list_tools",
parameters={"tool_type": "action"}
)
result = await computer.run_actions([tool_call])
# Returns: List of all available action tools
```
For more details on introspection capabilities, see [Computer - Meta Tools](../client/computer.md#meta-tools).
## Configuration Files
MCP configuration is located at:
```
config/ufo/mcp.yaml
```
For detailed configuration options, see:
- [MCP Configuration Guide](configuration.md) - Complete configuration reference
- [System Configuration](../configuration/system/system_config.md) - MCP-related system settings
- [MCP Reference](../configuration/system/mcp_reference.md) - MCP-specific settings
## Use Cases
### 1. UI Automation
```yaml
# Agent that automates UI interactions
HostAgent:
default:
data_collection:
- namespace: UICollector # Detect UI elements
action:
- namespace: HostUIExecutor # Click, type, scroll
```
### 2. Document Processing
```yaml
# Agent specialized for Word documents
AppAgent:
WINWORD.EXE:
data_collection:
- namespace: UICollector # Read document structure
action:
- namespace: WordCOMExecutor # Insert tables, format text
```
### 3. Multi-Device Coordination
```yaml
# Agent that coordinates tasks across devices
ConstellationAgent:
default:
action:
- namespace: ConstellationEditor # Create and assign tasks
```
### 4. Remote Hardware Monitoring
```yaml
# Agent that monitors remote hardware
HardwareAgent:
default:
data_collection:
- namespace: HardwareCollector
type: http
host: "192.168.1.100"
port: 8006
```
### 5. Android Device Automation
```yaml
# Agent that automates Android devices via ADB
MobileAgent:
default:
data_collection:
- namespace: MobileDataCollector
type: http
host: "localhost" # Or remote Android automation server
port: 8020
path: "/mcp"
action:
- namespace: MobileExecutor
type: http
host: "localhost"
port: 8021
path: "/mcp"
```
## Getting Started
To start using MCP in UFO²:
1. **Understand the two server types** - Read about [Data Collection](data_collection.md) and [Action](action.md) servers
2. **Configure your agents** - See [Configuration Guide](configuration.md) for setup details
3. **Use built-in servers** - Explore available [Local Servers](local_servers.md)
4. **Create custom servers** - Follow the [Creating Custom MCP Servers Tutorial](../tutorials/creating_mcp_servers.md)
5. **Deploy remotely** - Learn about [Remote Servers](remote_servers.md) deployment
## Related Documentation
- [Data Collection Servers](data_collection.md) - Read-only observation tools
- [Action Servers](action.md) - State-changing execution tools
- [Configuration Guide](configuration.md) - How to configure MCP for agents
- [Local Servers](local_servers.md) - Built-in MCP servers
- [Remote Servers](remote_servers.md) - HTTP and Stdio deployment
- [Creating Custom MCP Servers Tutorial](../tutorials/creating_mcp_servers.md) - Step-by-step guide to building custom servers
- [Computer](../client/computer.md) - MCP tool execution layer
- [Agent Client](../client/overview.md) - Client architecture overview
- [Agent Overview](../ufo2/overview.md) - UFO² agent system architecture
**Design Philosophy:**
MCP in UFO² follows the **separation of concerns** principle:
- **Agents** decide *what* to do (high-level planning)
- **MCP servers** implement *how* to do it (low-level execution)
- **Computer** manages the routing between them (middleware)
This architecture enables flexibility, extensibility, and maintainability.
---
### Documents Docs Mobile Overview (`documents/docs/mobile/overview.md`)
# MobileAgent: Android Task Executor
**MobileAgent** is a specialized agent designed for executing tasks on Android mobile devices. It leverages the layered FSM architecture and server-client design to perform intelligent, iterative task execution in mobile environments through UI interaction and app control.
**Quick Links:**
- **New to Mobile Agent?** Start with the [Quick Start Guide](../getting_started/quick_start_mobile.md) - Set up your first Android device agent in 10 minutes
- **Using as Sub-Agent in Galaxy?** See [Using Mobile Agent as Galaxy Device](as_galaxy_device.md)
- **Deep Dive:** [State Machine](state.md) • [Processing Strategy](strategy.md) • [MCP Commands](commands.md)
## Architecture Overview
MobileAgent operates as a single-agent instance that interacts with Android devices through UI controls and app management. Unlike the two-tier architecture of UFO (HostAgent + AppAgent), MobileAgent uses a simplified single-agent model optimized for mobile device automation, similar to LinuxAgent but with visual interface support.
## Core Responsibilities
MobileAgent provides the following capabilities for Android device automation:
### UI Interaction
MobileAgent interprets user requests and translates them into appropriate UI interactions on Android devices through screenshots analysis and control manipulation.
**Example:** User request "Search for restaurants on Maps" becomes:
1. Take screenshot and identify app icons
2. Launch Google Maps app
3. Identify search field control
4. Type "restaurants" into search field
5. Tap search button
### Visual Context Understanding
The agent captures and analyzes device screenshots to understand the current UI state:
- Screenshot capture (clean and annotated)
- Control identification and labeling
- UI hierarchy parsing
- App detection and recognition
### App Management
MobileAgent can manage installed applications:
- List installed apps (user apps and system apps)
- Launch apps by package name or app name
- Switch between apps
- Monitor current app state
### Iterative Task Execution
MobileAgent executes tasks iteratively, evaluating execution outcomes at each step and determining the next action based on results and LLM reasoning.
### Error Handling and Recovery
The agent monitors action execution results and can adapt its strategy when errors occur, such as controls not found or apps not responding.
## Key Characteristics
- **Scope**: Single Android device (UI-based automation)
- **Lifecycle**: One instance per task session
- **Hierarchy**: Standalone agent (no child agents)
- **Communication**: MCP server integration via ADB
- **Control**: 3-state finite state machine with 4-phase processing pipeline
- **Visual**: Screenshot-based UI understanding with control annotation
## Execution Workflow
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Comparison with Other Agents
| Aspect | MobileAgent | LinuxAgent | AppAgent |
|--------|-------------|------------|----------|
| **Platform** | Android Mobile | Linux (CLI) | Windows Applications |
| **States** | 3 (CONTINUE, FINISH, FAIL) | 3 states | 6 states |
| **Architecture** | Single-agent | Single-agent | Child executor |
| **Interface** | Mobile UI (touch-based) | Command-line | Desktop GUI |
| **Processing Phases** | 4 phases (with DATA_COLLECTION) | 3 phases | 4 phases |
| **Visual** | ✓ Screenshots + Annotations | ✗ Text-only | ✓ Screenshots + Annotations |
| **MCP Tools** | UI controls + App management | CLI commands | UI controls + API |
| **Input Method** | Touch (tap, swipe, type) | Keyboard commands | Mouse + Keyboard |
| **Control Identification** | UI hierarchy + bounds | N/A | UI Automation API |
## Design Principles
MobileAgent exemplifies mobile-specific design considerations:
- **Visual Context**: Screenshot-based UI understanding with control annotation for precise interaction
- **Control Caching**: Efficient control information caching to reduce ADB overhead
- **Touch-based Interaction**: Specialized actions for mobile gestures (tap, swipe, long-press)
- **App-centric Navigation**: Focus on app launching and switching rather than window management
- **Minimal State Set**: 3-state FSM for deterministic control flow
- **Modular Strategies**: Clear separation between data collection, LLM interaction, action execution, and memory updates
- **Traceable Execution**: Complete logging of screenshots, actions, and state transitions
## Deep Dive Topics
Explore the detailed architecture and implementation:
- [State Machine](state.md) - 3-state FSM lifecycle and transitions
- [Processing Strategy](strategy.md) - 4-phase pipeline (Data Collection, LLM, Action, Memory)
- [MCP Commands](commands.md) - Mobile UI interaction and app management commands
- [As Galaxy Device](as_galaxy_device.md) - Using Mobile Agent in multi-device workflows
## Technology Stack
### ADB (Android Debug Bridge)
MobileAgent relies on ADB for all device interactions:
- **Screenshot Capture**: `adb shell screencap` for visual context
- **UI Hierarchy**: `adb shell uiautomator dump` for control information
- **Touch Input**: `adb shell input tap/swipe` for user interaction
- **Text Input**: `adb shell input text` for typing
- **App Control**: `adb shell monkey` for app launching
- **Device Info**: `adb shell getprop` for device properties
### MCP Server Architecture
Two separate MCP servers handle different responsibilities:
1. **Data Collection Server** (Port 8020):
- Screenshot capture
- UI tree retrieval
- App list collection
- Control information gathering
- Device information
2. **Action Server** (Port 8021):
- Touch actions (tap, swipe)
- Text input
- App launching
- Key press events
- Control clicking
Both servers share a singleton `MobileServerState` for efficient caching and coordination.
## Use Cases
MobileAgent is ideal for:
- **Mobile App Testing**: Automated UI testing across different apps
- **Cross-App Workflows**: Tasks spanning multiple mobile applications
- **Data Entry**: Automated form filling and text input
- **App Navigation**: Exploring and interacting with mobile UIs
- **Mobile Productivity**: Automating repetitive mobile tasks
- **Cross-Device Workflows**: As a sub-agent in Galaxy multi-device orchestration
!!!tip "Galaxy Integration"
MobileAgent can serve as a device agent in Galaxy's multi-device orchestration framework, executing Android-specific tasks as part of cross-platform workflows alongside Windows and Linux devices.
See [Using Mobile Agent as Galaxy Device](as_galaxy_device.md) for configuration details.
## Requirements
### Hardware
- Android device or emulator
- USB connection (for physical devices) or network connection (for emulators)
- USB debugging enabled on the device
### Software
- ADB (Android Debug Bridge) installed and accessible
- Android device with API level 21+ (Android 5.0+)
- Python 3.8+
- Required Python packages (see requirements.txt)
## Implementation Location
The MobileAgent implementation can be found in:
```
ufo/
├── agents/
│ ├── agent/
│ │ └── customized_agent.py # MobileAgent class definition
│ ├── states/
│ │ └── mobile_agent_state.py # State machine implementation
│ └── processors/
│ ├── customized/
│ │ └── customized_agent_processor.py # MobileAgentProcessor
│ └── strategies/
│ └── mobile_agent_strategy.py # Processing strategies
├── prompter/
│ └── customized/
│ └── mobile_agent_prompter.py # Prompt construction
├── module/
│ └── sessions/
│ └── mobile_session.py # Session management
└── client/
└── mcp/
└── http_servers/
└── mobile_mcp_server.py # MCP server implementation
```
## Next Steps
To understand MobileAgent's complete architecture:
1. [State Machine](state.md) - Learn about the 3-state FSM
2. [Processing Strategy](strategy.md) - Understand the 4-phase pipeline
3. [MCP Commands](commands.md) - Explore mobile UI interaction commands
4. [As Galaxy Device](as_galaxy_device.md) - Configure for multi-device workflows
For deployment and configuration, see the Quick Start Guide (coming soon).
---
### Documents Docs Server Overview (`documents/docs/server/overview.md`)
# Agent Server Overview
The **Agent Server** is the central orchestration engine that transforms UFO into a distributed multi-agent system, enabling seamless task coordination across heterogeneous devices through persistent WebSocket connections and robust state management.
New to the Agent Server? Start with the [Quick Start Guide](./quick_start.md) to get up and running in minutes.
## What is the Agent Server?
The Agent Server is a **FastAPI-based asynchronous WebSocket server** that serves as the communication hub for UFO's distributed architecture. It bridges constellation orchestrators, device agents, and external systems through a unified protocol interface.
### Core Responsibilities
| Capability | Description | Key Benefit |
|------------|-------------|-------------|
| **🔌 Connection Management** | Tracks device & constellation client lifecycles | Real-time device availability awareness |
| **🎯 Task Orchestration** | Coordinates execution across distributed devices | Centralized workflow control |
| **💾 State Management** | Maintains session lifecycles & execution contexts | Stateful multi-turn task execution |
| **🌐 Dual API Interface** | WebSocket (AIP) + HTTP (REST) endpoints | Flexible integration options |
| **🛡️ Resilience** | Handles disconnections, timeouts, failures gracefully | Production-grade reliability |
**Why Use the Agent Server?**
- **Centralized Control**: Single point of orchestration for multi-device workflows
- **Protocol Abstraction**: Clients communicate via [AIP](../aip/overview.md), hiding network complexity
- **Async by Design**: Non-blocking execution enables high concurrency
- **Platform Agnostic**: Supports Windows, Linux, macOS (in development)
The Agent Server is part of UFO's distributed **server-client architecture**, where it handles orchestration and state management while [Agent Clients](../client/overview.md) handle command execution. See [Server-Client Architecture](../infrastructure/agents/server_client_architecture.md) for the complete design rationale and communication patterns.
---
## Architecture
The server follows a clean separation of concerns with distinct layers for web service, connection management, and protocol handling.
### Architectural Overview
**Component Interaction Diagram:**
```mermaid
graph TB
subgraph "Web Layer"
FastAPI[FastAPI App]
HTTP[HTTP API]
WS[WebSocket /ws]
end
subgraph "Service Layer"
WSM[Client Manager]
SM[Session Manager]
WSH[WebSocket Handler]
end
subgraph "Clients"
DC[Device Clients]
CC[Constellation Clients]
end
FastAPI --> HTTP
FastAPI --> WS
HTTP --> SM
HTTP --> WSM
WS --> WSH
WSH --> WSM
WSH --> SM
DC -->|WebSocket| WS
CC -->|WebSocket| WS
style FastAPI fill:#e1f5ff
style WSM fill:#fff4e1
style SM fill:#f0ffe1
style WSH fill:#ffe1f5
```
This layered design ensures each component has a single, well-defined responsibility. The managers maintain state while the handler implements protocol logic.
### Core Components
| Component | Responsibility | Key Operations |
|-----------|---------------|----------------|
| **FastAPI Application** | Web service layer | ✅ HTTP endpoint routing ✅ WebSocket connection acceptance ✅ Request/response handling ✅ CORS and middleware |
| **Client Connection Manager** | Connection registry | ✅ Client identity tracking ✅ Session ↔ client mapping ✅ Device info caching ✅ Connection lifecycle hooks |
| **Session Manager** | Execution lifecycle | ✅ Platform-specific session creation ✅ Background async task execution ✅ Result callback delivery ✅ Session cancellation |
| **WebSocket Handler** | Protocol implementation | ✅ AIP message parsing/routing ✅ Client registration ✅ Heartbeat monitoring ✅ Task/command dispatch |
**Component Documentation:**
- [Session Manager](./session_manager.md) - Session lifecycle and background execution
- [Client Connection Manager](./client_connection_manager.md) - Connection registry and client tracking
- [WebSocket Handler](./websocket_handler.md) - AIP protocol message handling
- [HTTP API](./api.md) - REST endpoint specifications
---
## Key Capabilities
### 1. Multi-Client Coordination
The server supports two distinct client types with different roles in the distributed architecture.
**Client Type Comparison:**
| Aspect | Device Client | Constellation Client |
|--------|---------------|---------------------|
| **Role** | Task executor | Task orchestrator |
| **Connection** | Long-lived WebSocket | Long-lived WebSocket |
| **Registration** | `ClientType.DEVICE` | `ClientType.CONSTELLATION` |
| **Capabilities** | Local execution, telemetry | Multi-device coordination |
| **Target Field** | Not required | Required for routing |
| **Example** | Windows agent, Linux agent | ConstellationClient orchestrator |
**Device Clients**
- Execute tasks locally on Windows/Linux machines
- Report hardware specs and real-time status
- Respond to commands via MCP tool servers
- Stream execution logs back to server
See [Agent Client Overview](../client/overview.md) for detailed client architecture.
**Constellation Clients**
- Orchestrate multi-device workflows from a central point
- Dispatch tasks to specific target devices via `target_id`
- Coordinate complex cross-device DAG execution
- Aggregate results from multiple devices
Both client types connect to `/ws` and register using the `REGISTER` message. The server differentiates behavior based on `client_type` field. For the complete server-client architecture and design rationale, see [Server-Client Architecture](../infrastructure/agents/server_client_architecture.md).
See [Quick Start](./quick_start.md) for registration examples.
---
### 2. Session Lifecycle Management
Unlike stateless HTTP servers, the Agent Server maintains **session state** throughout task execution, enabling multi-turn interactions and result callbacks.
**Session Lifecycle State Machine:**
```mermaid
stateDiagram-v2
[*] --> Created: create_session()
Created --> Running: Start execution
Running --> Completed: Success
Running --> Failed: Error
Running --> Cancelled: Disconnect
Completed --> [*]
Failed --> [*]
Cancelled --> [*]
note right of Running
Async background execution
Non-blocking server
end note
```
**Lifecycle Stages:**
| Stage | Trigger | Session Manager Action | Server State |
|-------|---------|----------------------|--------------|
| **Created** | HTTP dispatch or AIP `TASK` | Platform-specific session instantiation | Session ID generated |
| **Running** | Background task start | Async execution without blocking | Awaiting results |
| **Completed** | `TASK_END` (success) | Callback delivery to client | Results cached |
| **Failed** | `TASK_END` (error) | Error callback delivery | Error logged |
| **Cancelled** | Client disconnect | Cancel async task, cleanup | Session removed |
!!!warning "Platform-Specific Sessions"
The SessionManager creates different session types based on the target platform:
- **Windows**: `WindowsSession` with UI automation support
- **Linux**: `LinuxSession` with bash automation
- Auto-detected or overridden via `--platform` flag
**Session Manager Responsibilities:**
- ✅ **Platform abstraction**: Hides Windows/Linux differences
- ✅ **Background execution**: Non-blocking async task execution
- ✅ **Callback routing**: Delivers results via WebSocket
- ✅ **Resource cleanup**: Cancels tasks on disconnect
- ✅ **Result caching**: Stores results for HTTP retrieval
---
### 3. Resilient Communication
The server implements the [Agent Interaction Protocol (AIP)](../aip/overview.md), providing structured, type-safe communication with automatic failure handling.
**Protocol Features:**
| Feature | Implementation | Benefit |
|---------|----------------|---------|
| **Structured Messages** | Pydantic models with validation | Type safety, automatic serialization |
| **Connection Health** | Heartbeat every 20-30s | Early failure detection |
| **Error Recovery** | Exponential backoff reconnection | Transient fault tolerance |
| **State Tracking** | Session client mapping | Proper cleanup on disconnect |
| **Message Correlation** | `request_id`, `prev_response_id` chains | Request-response tracing |
**Disconnection Handling Flow:**
```mermaid
sequenceDiagram
participant Client
participant Server
participant SM as Session Manager
Client-xServer: Connection lost
Server->>SM: Cancel sessions
SM->>SM: Cleanup resources
Server->>Server: Remove from registry
Note over Server: Client can reconnect with same client_id
```
!!!danger "Important: Session Cancellation on Disconnect"
When a client disconnects (device or constellation), **all associated sessions are immediately cancelled** to prevent orphaned tasks and resource leaks.
---
### 4. Dual API Interface
The server provides two API styles to support different integration patterns: real-time WebSocket for agents and simple HTTP for external systems.
**WebSocket API (AIP-based)**
Purpose: Real-time bidirectional communication with agent clients
| Message Type | Direction | Purpose |
|--------------|-----------|---------|
| `REGISTER` | Client Server | Initial capability advertisement |
| `TASK` | Server Client | Task assignment with commands |
| `COMMAND` | Server Client | Individual command execution |
| `COMMAND_RESULTS` | Client Server | Execution results |
| `TASK_END` | Bidirectional | Task completion notification |
| `HEARTBEAT` | Bidirectional | Connection keepalive |
| `DEVICE_INFO_REQUEST/RESPONSE` | Bidirectional | Telemetry exchange |
| `ERROR` | Bidirectional | Error condition reporting |
!!!example "WebSocket Connection"
```python
import websockets
async with websockets.connect("ws://localhost:5000/ws") as ws:
# Register as device client
await ws.send(json.dumps({
"message_type": "REGISTER",
"client_id": "windows_agent_001",
"client_type": "device",
"metadata": {"platform": "windows", "gpu": "NVIDIA RTX 3080"}
}))
```
**HTTP REST API**
Purpose: Task dispatch and monitoring from external systems (HTTP clients, CI/CD, etc.)
| Endpoint | Method | Purpose | Authentication |
|----------|--------|---------|----------------|
| `/api/dispatch` | POST | Dispatch task to device | Optional (if configured) |
| `/api/task_result/{task_name}` | GET | Retrieve task results | Optional |
| `/api/clients` | GET | List connected clients | Optional |
| `/api/health` | GET | Server health check | None |
!!!example "HTTP Task Dispatch"
```bash
# Dispatch task to device
curl -X POST http://localhost:5000/api/dispatch \
-H "Content-Type: application/json" \
-d '{
"client_id": "my_windows_device",
"request": "Open Notepad and type Hello World",
"task_name": "test_task_001"
}'
# Response: {"status": "dispatched", "session_id": "session_abc123", "task_name": "test_task_001"}
# Retrieve results
curl http://localhost:5000/api/task_result/test_task_001
```
See [HTTP API Reference](./api.md) for complete endpoint documentation.
---
## Workflow Examples
### Complete Task Dispatch Flow
**End-to-End HTTP WebSocket Device Execution:**
```mermaid
sequenceDiagram
participant EXT as External System
participant HTTP as HTTP API
participant SM as Session Manager
participant WSH as WebSocket Handler
participant DC as Device Client
EXT->>HTTP: POST /api/dispatch {client_id, request, task_name}
HTTP->>SM: create_session()
SM->>SM: Create platform session
SM-->>HTTP: session_id
HTTP-->>EXT: 200 {session_id, task_name}
SM->>WSH: send_task(session_id, task)
WSH->>DC: TASK message (AIP)
DC-->>WSH: ACK
rect rgb(240, 255, 240)
Note over DC: Background Execution
DC->>DC: Execute via MCP tools
DC->>DC: Generate results
end
DC->>WSH: COMMAND_RESULTS
WSH->>SM: on_result_callback()
SM->>SM: Cache results
DC->>WSH: TASK_END (COMPLETED)
WSH->>SM: on_task_end()
EXT->>HTTP: GET /task_result/{session_id}
HTTP->>SM: get_results()
SM-->>HTTP: results
HTTP-->>EXT: 200 {results}
```
The green box highlights async execution on the device side, which doesn't block the server.
### Multi-Device Constellation Workflow
**Constellation Client Coordinating Multiple Devices:**
```mermaid
sequenceDiagram
participant CC as Constellation Client
participant Server as Agent Server
participant D1 as Device 1 (GPU)
participant D2 as Device 2 (CPU)
CC->>Server: REGISTER (constellation)
Server-->>CC: HEARTBEAT (OK)
Note over CC: Plan multi-device DAG
CC->>Server: TASK (target: device_1) Subtask 1: Image processing
Server->>D1: TASK (forward)
CC->>Server: TASK (target: device_2) Subtask 2: Data extraction
Server->>D2: TASK (forward)
par Parallel Execution
D1->>D1: Process image on GPU
D2->>D2: Extract data from DB
end
D1->>Server: COMMAND_RESULTS
Server->>CC: COMMAND_RESULTS (from device_1)
D2->>Server: COMMAND_RESULTS
Server->>CC: COMMAND_RESULTS (from device_2)
Note over CC: Combine results, Update DAG
D1->>Server: TASK_END
D2->>Server: TASK_END
Server->>CC: TASK_END (both tasks)
```
The server acts as a message router, forwarding tasks to target devices and routing results back to the constellation orchestrator. See [Constellation Documentation](../galaxy/overview.md) for more details on multi-device orchestration.
---
## Platform Support
The server automatically detects client platforms and creates appropriate session implementations.
**Supported Platforms:**
| Platform | Session Type | Capabilities | Status |
|----------|--------------|--------------|--------|
| **Windows** | `WindowsSession` | UI automation (UIA) COM API integration Native app control Screenshot capture | Full support |
| **Linux** | `LinuxSession` | Bash automation GUI tools (xdotool) Package management Process control | Full support |
| **macOS** | (Planned) | AppleScript UI automation Native app control | 🚧 In development |
**Platform Auto-Detection:**
The server automatically detects the client's platform during registration. You can override this globally with the `--platform` flag when needed for testing or specific deployment scenarios.
```bash
python -m ufo.server.app --platform windows # Force Windows sessions
python -m ufo.server.app --platform linux # Force Linux sessions
python -m ufo.server.app # Auto-detect (default)
```
!!!warning "When to Use Platform Override"
Use `--platform` override when:
- Testing cross-platform sessions without actual devices
- Running server in container different from target platform
- Debugging platform-specific session behavior
For more details on platform-specific implementations, see [Windows Agent](../linux/overview.md) and [Linux Agent](../linux/overview.md).
---
## Configuration
The server runs out-of-the-box with sensible defaults. Advanced configuration inherits from UFO's central config system.
### Command-Line Arguments
```bash
python -m ufo.server.app [OPTIONS]
```
**Available Options:**
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `--port` | int | 5000 | Server listening port |
| `--host` | str | `0.0.0.0` | Bind address (use `127.0.0.1` for localhost only) |
| `--platform` | str | auto | Force platform (`windows`, `linux`) |
| `--log-level` | str | `INFO` | Logging level (`DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`, `OFF`) |
| `--local` | flag | False | Restrict to local connections only |
!!!example "Configuration Examples"
```bash
# Development: Local-only with debug logging
python -m ufo.server.app --local --log-level DEBUG --port 8000
# Production: External access, info logging
python -m ufo.server.app --host 0.0.0.0 --port 5000 --log-level INFO
# Testing: Force Linux sessions
python -m ufo.server.app --platform linux --port 9000
```
### UFO Configuration Inheritance
The server uses UFO's central configuration from `config_dev.yaml`:
| Config Section | Inherited Settings |
|----------------|-------------------|
| **Agent Strategies** | HostAgent, AppAgent, EvaluationAgent configurations |
| **LLM Models** | Model endpoints, API keys, temperature settings |
| **Automators** | UI automation, COM API, web automation configs |
| **Logging** | Log file paths, rotation, format |
| **Prompts** | Agent system prompts, example templates |
See [Configuration Guide](../configuration/system/overview.md) for comprehensive config documentation.
---
## Monitoring & Operations
### Health Monitoring
Monitor server status and performance using HTTP endpoints.
**Health Check Endpoints:**
```bash
# Server health and uptime
curl http://localhost:5000/api/health
# Response:
# {
# "status": "healthy",
# "online_clients": [...]
# }
# Connected clients list
curl http://localhost:5000/api/clients
# Response:
# {
# "online_clients": ["windows_001", "linux_002", ...]
# }
```
For comprehensive monitoring strategies including performance metrics collection, log aggregation patterns, alert configuration, and dashboard setup, see [Monitoring Guide](./monitoring.md).
### Error Handling
The server handles common failure scenarios gracefully to maintain system stability.
**Disconnection Handling Matrix:**
| Scenario | Server Detection | Automatic Action | Client Impact |
|----------|-----------------|------------------|---------------|
| **Device Disconnect** | Heartbeat timeout / WebSocket close | Cancel device sessions, notify constellation | Task fails, constellation retries |
| **Constellation Disconnect** | Heartbeat timeout / WebSocket close | Continue device execution, skip callbacks | Device completes but results not delivered |
| **Task Execution Failure** | `TASK_END` with error status | Log error, store in results | Client receives error via callback/HTTP |
| **Network Partition** | Heartbeat timeout | Mark disconnected, enable reconnection | Client reconnects with same ID |
| **Server Crash** | N/A | Clients detect via heartbeat | Clients reconnect to new instance |
!!!note "Reconnection Support"
Clients can reconnect with the same `client_id`. The server will re-register the client and restore heartbeat monitoring, but **will not restore previous sessions** (sessions are ephemeral).
---
## Best Practices
### Development Environment
Optimize your development workflow with these recommended practices.
**Recommended Development Configuration:**
```bash
# Isolate to localhost, enable detailed logging
python -m ufo.server.app \
--host 127.0.0.1 \
--port 5000 \
--local \
--log-level DEBUG
```
**Development Checklist:**
- Use `--local` flag to prevent external access
- Enable `DEBUG` logging for detailed traces
- Monitor logs in separate terminal: `tail -f logs/ufo_server.log`
- Test with single device before adding multiple clients
- Use HTTP API for quick task dispatch testing
- Verify heartbeat monitoring with client disconnection
!!!example "Development Testing Pattern"
```bash
# Terminal 1: Start server with debug logging
python -m ufo.server.app --local --log-level DEBUG
# Terminal 2: Connect device client
python -m ufo.client.client --ws --ws-server ws://127.0.0.1:5000/ws
# Terminal 3: Dispatch test task
curl -X POST http://127.0.0.1:5000/api/dispatch \
-H "Content-Type: application/json" \
-d '{"client_id": "windowsagent", "request": "Open Notepad", "task_name": "test_001"}'
```
### Production Deployment
The default configuration is **not production-ready**. Implement these security and reliability measures.
**Production Architecture:**
```mermaid
graph LR
Internet[Internet]
LB[Load Balancer nginx/HAProxy]
SSL[SSL/TLS Termination]
subgraph "UFO Server Cluster"
S1[Server Instance 1 :5000]
S2[Server Instance 2 :5001]
S3[Server Instance 3 :5002]
end
Monitor[Monitoring Prometheus/Grafana]
PM[Process Manager systemd/PM2]
Internet --> LB
LB --> SSL
SSL --> S1
SSL --> S2
SSL --> S3
PM -.Manages.-> S1
PM -.Manages.-> S2
PM -.Manages.-> S3
S1 -.Metrics.-> Monitor
S2 -.Metrics.-> Monitor
S3 -.Metrics.-> Monitor
style LB fill:#ffe1f5
style SSL fill:#fff4e1
style Monitor fill:#f0ffe1
```
**Production Checklist:**
| Category | Recommendation | Rationale |
|----------|---------------|-----------|
| **Reverse Proxy** | nginx, Apache, or cloud load balancer | SSL termination, rate limiting, DDoS protection |
| **SSL/TLS** | Enable WSS (WebSocket Secure) | Encrypt client-server communication |
| **Authentication** | Add auth middleware to FastAPI | Prevent unauthorized access |
| **Process Management** | systemd (Linux), PM2 (Node.js), Docker | Auto-restart on crash, resource limits |
| **Monitoring** | `/api/health` polling, metrics export | Detect issues proactively |
| **Logging** | Structured logging, log aggregation (ELK) | Centralized debugging and audit trails |
| **Resource Limits** | Set max connections, memory limits | Prevent resource exhaustion |
**Example Nginx Configuration:**
```nginx
upstream ufo_server {
server localhost:5000;
}
server {
listen 443 ssl;
server_name ufo-server.example.com;
ssl_certificate /path/to/cert.pem;
ssl_certificate_key /path/to/key.pem;
# WebSocket endpoint
location /ws {
proxy_pass http://ufo_server;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
proxy_read_timeout 3600s;
}
# HTTP API
location /api/ {
proxy_pass http://ufo_server;
proxy_set_header Host $host;
}
}
```
### Scaling Strategies
The server can scale horizontally for high-load deployments, but requires careful session management.
**Scaling Patterns:**
| Pattern | Description | Use Case | Considerations |
|---------|-------------|----------|----------------|
| **Vertical** | Increase CPU/RAM on single instance | < 100 concurrent clients | Simplest, no session distribution |
| **Horizontal (Sticky Sessions)** | Multiple instances with session affinity | 100-1000 clients | Load balancer routes same client to same instance |
| **Horizontal (Shared State)** | Multiple instances with Redis | > 1000 clients | Requires session state externalization |
!!!warning "Current Limitation"
The current implementation stores session state in-memory. For horizontal scaling, use **sticky sessions** (client affinity) in your load balancer to route clients to consistent server instances. **Future**: Shared state backend (Redis) for true stateless horizontal scaling.
---
## Troubleshooting
### Common Issues
**Issue: Clients Can't Connect**
```bash
# Symptom: Connection refused
Error: WebSocket connection to 'ws://localhost:5000/ws' failed
# Diagnosis:
1. Check server is running: curl http://localhost:5000/api/health
2. Verify port: netstat -an | grep 5000
3. Check firewall: sudo ufw status
# Solution:
# Start server with correct host binding
python -m ufo.server.app --host 0.0.0.0 --port 5000
```
**Issue: Sessions Not Executing**
```bash
# Symptom: Task dispatched but no results
# Diagnosis:
1. Check server logs for errors
2. Verify client is connected: curl http://localhost:5000/api/clients
3. Check target_id matches client_id
# Solution:
# Ensure client_id in request matches registered client
curl -X POST http://localhost:5000/api/dispatch \
-d '{"client_id": "correct_client_id", "request": "test", "task_name": "test_001"}'
```
**Issue: Memory Leak / High Memory Usage**
```bash
# Symptom: Server memory grows over time
# Diagnosis:
1. Check session cleanup in logs
2. Monitor /api/health for session count
3. Profile with memory_profiler
# Solution:
# Ensure clients send TASK_END to complete sessions
# Restart server periodically (systemd handles this)
# Implement session timeout (future feature)
```
### Debug Mode
!!!example "Enable Maximum Verbosity"
```bash
# Ultra-verbose debugging
python -m ufo.server.app \
--log-level DEBUG \
--local \
--port 5000 2>&1 | tee debug.log
# Watch logs in real-time
tail -f debug.log | grep -E "(ERROR|WARNING|Session|WebSocket)"
```
---
## Documentation Map
Explore related documentation to deepen your understanding of the Agent Server ecosystem.
### Getting Started
| Document | Purpose |
|----------|---------|
| [Quick Start](./quick_start.md) | Get server running in < 5 minutes |
| [Client Registration](./quick_start.md) | How clients connect to server |
### Architecture & Components
| Document | Purpose |
|----------|---------|
| [Session Manager](./session_manager.md) | Task execution lifecycle deep-dive |
| [Client Connection Manager](./client_connection_manager.md) | Connection registry internals |
| [WebSocket Handler](./websocket_handler.md) | AIP protocol message handling |
| [HTTP API](./api.md) | REST endpoint specifications |
### Operations
| Document | Purpose |
|----------|---------|
| [Monitoring](./monitoring.md) | Health checks, metrics, alerting |
### Related Documentation
| Document | Purpose |
|----------|---------|
| [AIP Protocol](../aip/overview.md) | Communication protocol specification |
| [Agent Architecture](../infrastructure/agents/overview.md) | Agent design and FSM framework |
| [Server-Client Architecture](../infrastructure/agents/server_client_architecture.md) | Distributed architecture rationale |
| [Client Overview](../client/overview.md) | Device client architecture |
| [MCP Integration](../mcp/overview.md) | Model Context Protocol tool servers |
---
## Next Steps
Follow this recommended sequence to master the Agent Server:
**1. Run the Server** (5 minutes)
- Follow the [Quick Start Guide](./quick_start.md)
- Verify server responds to `/api/health`
**2. Connect a Client** (10 minutes)
- Use [Device Client](../client/quick_start.md)
- Verify registration in server logs
- Check `/api/clients` endpoint
**3. Dispatch Tasks** (15 minutes)
- Use [HTTP API](./api.md) to send tasks
- Retrieve results via `/api/task_result`
- Observe WebSocket message flow in logs
**4. Understand Architecture** (30 minutes)
- Read [Session Manager](./session_manager.md) internals
- Study [WebSocket Handler](./websocket_handler.md) protocol implementation
- Review [AIP Protocol](../aip/overview.md) message types
**5. Deploy to Production** (varies)
- Set up reverse proxy (nginx)
- Configure SSL/TLS
- Implement monitoring
- Test failover scenarios
---
### Documents Docs Ufo2 Overview (`documents/docs/ufo2/overview.md`)
# UFO² — Windows AgentOS
[](https://arxiv.org/abs/2504.14603)
[](https://opensource.org/licenses/MIT)
[](https://github.com/microsoft/UFO)
[](https://www.youtube.com/watch?v=QT_OhygMVXU)
**UFO²** is a Windows AgentOS that reimagines desktop automation as a first-class operating system abstraction. Unlike traditional Computer-Using Agents (CUAs) that rely on screenshots and simulated inputs, UFO² deeply integrates with Windows OS through UI Automation APIs, application-specific introspection, and hybrid GUI–API execution—enabling robust, efficient, and non-disruptive automation across 20+ real-world applications.
---
## What is UFO²?
UFO² addresses fundamental limitations of existing desktop automation solutions:
**Traditional RPA (UiPath, Power Automate):**
❌ Fragile scripts that break with UI changes
❌ Requires extensive manual maintenance
❌ Limited adaptability to dynamic environments
**Current CUAs (Claude, Operator):**
❌ Visual-only inputs with high cognitive overhead
❌ Miss native OS APIs and application internals
❌ Lock users out during automation (poor UX)
**UFO² AgentOS:**
✅ **Deep OS Integration** — Windows UIA, Win32, WinCOM APIs
✅ **Hybrid GUI–API Actions** — Native APIs + fallback GUI automation
✅ **Continuous Knowledge Learning** — RAG-enhanced from docs & execution history
✅ **Picture-in-Picture Desktop** — Parallel automation without user disruption
✅ **10%+ better success rate** than state-of-the-art CUAs
Figure 1: Comparison between (a) traditional CUAs that rely on screenshots and simulated inputs, and (b) UFO² AgentOS that deeply integrates with OS APIs, application internals, and hybrid GUI–API execution.
---
## Core Architecture
UFO² implements a **hierarchical multi-agent system** optimized for Windows desktop automation:
Figure 2: UFO² system architecture featuring the two-tier agent hierarchy (HostAgent + AppAgents), hybrid control detection pipeline, continuous knowledge substrate integration, and unified GUI–API action layer coordinated through MCP servers.
### Two-Tier Agent Hierarchy
| Agent Type | Role | Key Capabilities |
|------------|------|------------------|
| **HostAgent** | Desktop Orchestrator | Task decomposition • Application selection • Cross-app coordination • AppAgent lifecycle management |
| **AppAgent** | Application Executor | UI element interaction • Hybrid GUI–API execution • Application-specific automation • Result reporting |
**Design Philosophy:**
- **HostAgent** handles **WHAT** (which application) and **WHEN** (task sequencing)
- **AppAgent** handles **HOW** (UI/API interaction) and **WHERE** (control targeting)
- **Blackboard** facilitates inter-agent communication without tight coupling
- **State Machines** ensure deterministic execution flow and error recovery
!!!info "Learn More"
- [**HostAgent Documentation**](host_agent/overview.md) — 7-state FSM, desktop orchestration, AppAgent lifecycle
- [**AppAgent Documentation**](app_agent/overview.md) — 6-state FSM, UI automation, hybrid action execution
- [**Agent Architecture**](../infrastructure/agents/overview.md) — Three-layer design principles
---
## Key Innovations
### 1. Deep OS Integration 🔧
UFO² embeds directly into Windows OS infrastructure:
- **UI Automation (UIA):** Introspects accessibility trees for standard controls
- **Win32 APIs:** Low-level window management and process control
- **WinCOM:** Interacts with Office applications (Excel, Word, Outlook)
- **Hybrid Detection:** Fuses UIA metadata + visual grounding for non-standard UI elements
!!!tip "Hybrid Control Detection"
Combines Windows UIA APIs with vision models ([OmniParser](https://arxiv.org/abs/2408.00203)) to detect both standard and custom UI controls—bridging structured accessibility trees and pixel-level perception.
📖 [Control Detection Guide](core_features/control_detection/overview.md)
### 2. Unified GUI–API Action Layer ⚡
Traditional CUAs simulate mouse/keyboard only. UFO² chooses the best execution method:
**GUI Actions** (fallback):
`click`, `type`, `select`, `scroll` → Reliable for any application
**Native APIs** (preferred):
- Excel: `xlwings` for direct cell/chart manipulation
- Outlook: `win32com` for email operations
- PowerPoint: `python-pptx` for slide editing
→ **51% fewer LLM calls** via speculative multi-action execution
**Model Context Protocol (MCP) Servers:**
Extensible framework for adding application-specific APIs without modifying agent code.
!!!info "Learn More"
📖 [Hybrid Actions Guide](core_features/hybrid_actions.md) • [MCP Integration](../mcp/overview.md)
### 3. Continuous Knowledge Substrate 📚
UFO² learns from three knowledge sources without model retraining:
| Source | Content | Integration Method |
|--------|---------|-------------------|
| **Help Documents** | Official app documentation, API references | Vectorized retrieval (RAG) |
| **Bing Search** | Real-time web knowledge for latest features | Dynamic query expansion |
| **Execution History** | Past successful/failed action sequences | Experience replay & pattern mining |
**Result:** Agents improve autonomously by retrieving relevant context at execution time.
!!!info "Knowledge Integration"
📖 [Knowledge Substrate Overview](core_features/knowledge_substrate/overview.md)
📖 [Learning from Help Documents](core_features/knowledge_substrate/learning_from_help_document.md)
📖 [Experience Learning](core_features/knowledge_substrate/experience_learning.md)
### 4. Speculative Multi-Action Execution 🚀
Reduce LLM latency by predicting and validating action sequences:
**Traditional Approach:**
1 LLM call → 1 action → observe → repeat → **High latency**
**UFO² Speculative Execution:**
1 LLM call → predict N actions → validate with UI state → execute all → **51% fewer queries**
**Validation Mechanism:**
Lightweight control-state checks ensure predicted actions remain valid before execution.
!!!example "Efficiency Gain"
**Task:** "Fill form fields A1–A10 with sequential numbers"
- **Traditional CUA:** 10 LLM calls (1 per field) → ~30 seconds
- **UFO² Speculative:** 1 LLM call predicts all 10 actions → ~8 seconds
📖 [Multi-Action Execution Guide](core_features/multi_action.md)
### 5. Picture-in-Picture Desktop 🖼️
**Problem:** Existing CUAs lock users out during automation (poor UX).
**UFO² Solution:** Nested virtual desktop via Windows Remote Desktop loopback:
- **User Desktop:** Continue working normally
- **Agent Desktop (PiP):** Automation runs in parallel sandboxed environment
- **Zero Interference:** User and agent don't compete for mouse/keyboard
**Implementation:**
Built on Windows native remote desktop infrastructure—secure, isolated, non-disruptive.
!!!success "User Experience"
Users can continue email, browsing, or coding while UFO² automates Excel reports in the background PiP desktop.
---
## System Components
### Processing Pipeline
Both HostAgent and AppAgent execute a **4-phase processing cycle**:
| Phase | Purpose | HostAgent Strategy | AppAgent Strategy |
|-------|---------|-------------------|------------------|
| **1. Data Collection** | Gather environment state | Desktop screenshot, app list | App screenshot, UI tree, control annotations |
| **2. LLM Interaction** | Decide next action | Select application, plan subtask | Select control, plan action sequence |
| **3. Action Execution** | Execute commands | Launch app, create AppAgent | Execute GUI/API actions |
| **4. Memory Update** | Record execution | Save orchestration step | Save interaction step, update blackboard |
!!!info "Processing Details"
📖 [Strategy Layer](../infrastructure/agents/design/processor.md) — Processing framework and dependency chain
📖 [State Layer](../infrastructure/agents/design/state.md) — FSM design principles
### Command System
Commands are dispatched through **MCP (Model Context Protocol)** servers:
**HostAgent Commands:**
- **Desktop Capture:** `capture_desktop_screenshot`
- **Window Management:** `get_desktop_app_info`, `get_app_window`
- **Process Control:** `launch_application`, `close_application`
**AppAgent Commands:**
- **Screenshot:** `capture_screenshot`, `annotate_screenshot`
- **UI Inspection:** `get_control_info`, `get_ui_tree`
- **UI Interaction:** `click`, `set_edit_text`, `wheel_mouse_input`
- **Control Selection:** `select_control_by_index`, `select_control_by_name`
!!!info "Command Architecture"
📖 [Command Layer](../infrastructure/agents/design/command.md) — MCP integration and command dispatch
📖 [MCP Servers](../mcp/overview.md) — Server architecture and custom server creation
---
## Configuration
UFO² integrates with a centralized YAML-based configuration system:
```yaml
# config/ufo/host_agent_config.yaml
host_agent:
visual_mode: true # Enable screenshot-based reasoning
max_subtasks: 10 # Maximum subtasks per session
llm_config:
model: "gpt-4o"
temperature: 0.0
# config/ufo/app_agent_config.yaml
app_agent:
visual_mode: true # Enable UI screenshot analysis
control_backend: "uia" # UI Automation (uia) or Win32 (win32)
max_steps: 20 # Maximum steps per subtask
```
!!!tip "Complete Configuration Guide"
For detailed configuration options, model setup, and advanced customization:
📖 **[Configuration & Setup](../configuration/system/overview.md)** — Complete system configuration reference
📖 **[Model Setup](../configuration/models/overview.md)** — LLM provider configuration (OpenAI, Azure, Gemini, Claude, etc.)
📖 **[MCP Configuration](../configuration/system/mcp_reference.md)** — MCP server and extension configuration
---
## Quick Start
### Basic Usage
UFO² is designed to be run from the command line:
**Interactive Mode:**
```powershell
# Start UFO² in interactive mode
python -m ufo --task
```
**Example:**
```powershell
python -m ufo --task excel_demo
```
This will prompt you to enter your request interactively:
```
Welcome to use UFO🛸, A UI-focused Agent for Windows OS Interaction.
Please enter your request to be completed🛸: Create a chart from Sheet1 data in Excel
```
**Direct Request Mode:**
```powershell
# Execute with a specific request directly
python -m ufo --task -r ""
```
**Example:**
```powershell
python -m ufo --task excel_demo -r "Open Excel and create a chart from Sheet1 data"
```
!!!tip "Complete Setup Guide"
For detailed installation, configuration, and advanced usage options, see the **[Quick Start Guide](../getting_started/quick_start_ufo2.md)**.
### What Happens Under the Hood
1. **Session** creates **HostAgent** with user request
2. **HostAgent** captures desktop, selects "Microsoft Excel", launches app
3. **HostAgent** creates **AppAgent** for Excel, delegates subtask
4. **AppAgent** captures Excel UI, identifies chart insertion control
5. **AppAgent** executes hybrid action (API if available, GUI fallback)
6. **AppAgent** reports completion to **HostAgent**
7. **HostAgent** verifies task, returns success to **Session**
!!!tip "Next Steps"
📖 [Getting Started Guide](../getting_started/quick_start_ufo2.md)
📖 [Creating Your AppAgent](../tutorials/creating_app_agent/overview.md)
---
## Documentation Navigation
### Core Concepts
- [**HostAgent**](host_agent/overview.md) — Desktop orchestrator with 7-state FSM
- [**AppAgent**](app_agent/overview.md) — Application executor with 6-state FSM
- [**Agent Types**](../infrastructure/agents/agent_types.md) — Platform-specific implementations
- [**Evaluation Agent**](evaluation/evaluation_agent.md) — Automated testing and benchmarking
### Advanced Features
- [**Hybrid Actions**](core_features/hybrid_actions.md) — GUI–API execution layer
- [**Control Detection**](core_features/control_detection/overview.md) — UIA + visual grounding
- [**Knowledge Substrate**](core_features/knowledge_substrate/overview.md) — RAG-enhanced learning
- [**Multi-Action Execution**](core_features/multi_action.md) — Speculative action planning
- [**Follower Mode**](advanced_usage/follower_mode.md) — Human-in-the-loop execution
- [**Batch Mode**](advanced_usage/batch_mode.md) — Bulk task processing
### System Architecture
- [**Device Agent Overview**](../infrastructure/agents/overview.md) — Three-layer architecture
- [**State Layer**](../infrastructure/agents/design/state.md) — FSM design principles
- [**Strategy Layer**](../infrastructure/agents/design/processor.md) — Processing framework
- [**Command Layer**](../infrastructure/agents/design/command.md) — MCP integration
### Development
- [**Creating AppAgent**](../tutorials/creating_app_agent/overview.md) — Custom agent development
- [**MCP Servers**](../mcp/overview.md) — Building custom MCP servers
- [**Configuration**](../configuration/system/overview.md) — System configuration reference
- [**Prompts**](prompts/overview.md) — Prompt engineering guide
### Benchmarking & Logs
- [**Benchmark Overview**](evaluation/benchmark/overview.md) — WindowsAgentArena, OSWorld
- [**Performance Logs**](evaluation/logs/overview.md) — Execution logs and debugging
---
## Research Impact
UFO² demonstrates that **system-level integration** and **architectural design** matter more than model size alone:
!!!success "Key Findings"
- **10%+ improvement** over Claude/Operator on WindowsAgentArena
- **51% fewer LLM calls** via speculative multi-action execution
- **Robust to UI changes** through hybrid UIA + visual detection
- **Continuous learning** without model retraining via RAG
- **Non-disruptive UX** via Picture-in-Picture desktop
**Research Paper:**
📄 [UFO²: A Grounded OS Agent for Windows](https://arxiv.org/abs/2504.14603)
---
## Get Started
Ready to explore UFO²? Choose your path:
!!!info "Learning Paths"
**🚀 New Users:** Start with [Quick Start Guide](../getting_started/quick_start_ufo2.md)
**🔧 Developers:** Read [Creating AppAgent](../tutorials/creating_app_agent/overview.md)
**🏗️ System Architects:** Study [Device Agent Architecture](../infrastructure/agents/overview.md)
**📊 Researchers:** Check [Benchmark Results](evaluation/benchmark/overview.md)
**Next:** [HostAgent Deep Dive](host_agent/overview.md) → Understand desktop orchestration
---
## 🌐 Media Coverage
Check out our official deep dive of UFO on [this Youtube Video](https://www.youtube.com/watch?v=QT_OhygMVXU).
UFO sightings have garnered attention from various media outlets, including:
- [微软正式开源UFO²，Windows桌面迈入「AgentOS 时代」](https://www.jiqizhixin.com/articles/2025-05-06-13)
- [Microsoft's UFO abducts traditional user interfaces for a smarter Windows experience](https://the-decoder.com/microsofts-ufo-abducts-traditional-user-interfaces-for-a-smarter-windows-experience/)
- [🚀 UFO & GPT-4-V: Sit back and relax, mientras GPT lo hace todo🌌](https://www.linkedin.com/posts/gutierrezfrancois_ai-ufo-microsoft-activity-7176819900399652865-pLoo?utm_source=share&utm_medium=member_desktop)
- [The AI PC - The Future of Computers? - Microsoft UFO](https://www.youtube.com/watch?v=1k4LcffCq3E)
- [下一代Windows系统曝光：基于GPT-4V，Agent跨应用调度，代号UFO](https://baijiahao.baidu.com/s?id=1790938358152188625&wfr=spider&for=pc)
- [下一代智能版 Windows 要来了？微软推出首个 Windows Agent，命名为 UFO！](https://blog.csdn.net/csdnnews/article/details/136161570)
- [Microsoft発のオープンソース版「UFO」登場！ Windowsを自動操縦するAIエージェントを試す](https://internet.watch.impress.co.jp/docs/column/shimizu/1570581.html)
---
## 📚 Citation
If you build on this work, please cite the AgentOS framework:
**UFO² – The Desktop AgentOS (2025)**
```bibtex
@article{zhang2025ufo2,
title = {{UFO2: The Desktop AgentOS}},
author = {Zhang, Chaoyun and Huang, He and Ni, Chiming and Mu, Jian and Qin, Si and He, Shilin and Wang, Lu and Yang, Fangkai and Zhao, Pu and Du, Chao and Li, Liqun and Kang, Yu and Jiang, Zhao and Zheng, Suzhen and Wang, Rujia and Qian, Jiaxu and Ma, Minghua and Lou, Jian-Guang and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei},
journal = {arXiv preprint arXiv:2504.14603},
year = {2025}
}
```
**UFO – A UI‑Focused Agent for Windows OS Interaction (2024)**
```bibtex
@article{zhang2024ufo,
title = {{UFO: A UI-Focused Agent for Windows OS Interaction}},
author = {Zhang, Chaoyun and Li, Liqun and He, Shilin and Zhang, Xu and Qiao, Bo and Qin, Si and Ma, Minghua and Kang, Yu and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei and Zhang, Qi},
journal = {arXiv preprint arXiv:2402.07939},
year = {2024}
}
```
---
## 🎨 Related Projects
- **TaskWeaver** — a code‑first LLM agent for data analytics:
- **LLM‑Brained GUI Agents: A Survey**: • [GitHub](https://github.com/vyokky/LLM-Brained-GUI-Agents-Survey) • [Interactive site](https://vyokky.github.io/LLM-Brained-GUI-Agents-Survey/)
---
## ❓Get Help
- ❔GitHub Issues (preferred)
- For other communications, please contact [ufo-agent@microsoft.com](mailto:ufo-agent@microsoft.com)
---
### Documents Docs Configuration Models Overview (`documents/docs/configuration/models/overview.md`)
# Supported Models
UFO supports a wide variety of LLM models and APIs. You can configure different models for `HOST_AGENT`, `APP_AGENT`, `BACKUP_AGENT`, and `EVALUATION_AGENT` in the `config/ufo/agents.yaml` file to optimize for performance, cost, or specific capabilities.
## Available Model Integrations
| Provider | Documentation | Visual Support | Authentication |
| --- | --- | --- | --- |
| **OpenAI** | [OpenAI API](./openai.md) | ✅ | API Key |
| **Azure OpenAI (AOAI)** | [Azure OpenAI API](./azure_openai.md) | ✅ | API Key / Azure AD |
| **Google Gemini** | [Gemini API](./gemini.md) | ✅ | API Key |
| **Anthropic Claude** | [Claude API](./claude.md) | ✅ | API Key |
| **Qwen (Alibaba)** | [Qwen API](./qwen.md) | ✅ | API Key |
| **DeepSeek** | [DeepSeek API](./deepseek.md) | ❌ | API Key |
| **Ollama** | [Ollama API](./ollama.md) | ⚠️ Limited | Local |
| **OpenAI Operator** | [Operator (CUA)](./operator.md) | ✅ | Azure AD |
| **Custom Models** | [Custom API](./custom_model.md) | Depends | Varies |
## Model Selection Guide
### By Use Case
**For Production Deployments:**
- **Primary**: OpenAI GPT-4o or Azure OpenAI (enterprise features)
- **Cost-optimized**: GPT-4o-mini for APP_AGENT, GPT-4o for HOST_AGENT
- **Privacy-sensitive**: Ollama (local models)
**For Development & Testing:**
- **Fast iteration**: Gemini 2.0 Flash (high speed, low cost)
- **Local testing**: Ollama with llama2 or similar
- **Budget-friendly**: DeepSeek or Qwen models
**For Specialized Tasks:**
- **Computer control**: OpenAI Operator (CUA model)
- **Code generation**: DeepSeek-Coder or Claude
- **Long context**: Gemini 1.5 Pro (large context window)
### By Capability
**Vision Support (Screenshot Understanding):**
- ✅ OpenAI GPT-4o, GPT-4-turbo
- ✅ Azure OpenAI (vision-enabled deployments)
- ✅ Google Gemini (all 1.5+ models)
- ✅ Claude 3+ (all variants)
- ✅ Qwen-VL models
- ⚠️ Ollama (llava models only)
- ❌ DeepSeek (text-only)
**JSON Schema Support:**
- ✅ OpenAI / Azure OpenAI
- ✅ Google Gemini
- ⚠️ Limited: Claude, Qwen, Ollama
## Configuration Architecture
Each model is implemented as a separate class in the `ufo/llm` directory, inheriting from the `BaseService` class in `ufo/llm/base.py`. All models implement the `chat_completion` method to maintain a consistent interface.
**Key Configuration Files:**
- **`config/ufo/agents.yaml`**: Primary agent configuration (HOST, APP, BACKUP, EVALUATION, OPERATOR)
- **`config/ufo/system.yaml`**: System-wide LLM parameters (MAX_TOKENS, TEMPERATURE, etc.)
- **`config/ufo/prices.yaml`**: Cost tracking for different models
## Multi-Provider Setup
You can mix and match providers for different agents to optimize cost and performance:
```yaml
# Use OpenAI for planning
HOST_AGENT:
API_TYPE: "openai"
API_MODEL: "gpt-4o"
# Use Azure OpenAI for execution (cost control)
APP_AGENT:
API_TYPE: "aoai"
API_MODEL: "gpt-4o-mini"
# Use Claude for evaluation
EVALUATION_AGENT:
API_TYPE: "claude"
API_MODEL: "claude-3-5-sonnet-20241022"
```
## Getting Started
1. Choose your LLM provider from the table above
2. Follow the provider-specific documentation to obtain API keys
3. Configure `config/ufo/agents.yaml` with your credentials
4. Refer to the [Quick Start Guide](../../getting_started/quick_start_ufo2.md) to begin
**For detailed configuration options:**
- [Agent Configuration Guide](../system/agents_config.md) - Complete configuration reference
- [System Configuration](../system/system_config.md) - LLM parameters and behavior
- [Quick Start Guide](../../getting_started/quick_start_ufo2.md) - Step-by-step setup
---
### Documents Docs Configuration System Overview (`documents/docs/configuration/system/overview.md`)
# Configuration Architecture
UFO² features a modern, modular configuration system designed for flexibility, maintainability, and backward compatibility. This guide explains the overall architecture and design principles.
## Design Philosophy
The configuration system follows professional software engineering best practices:
### Separation of Concerns
Configuration files are organized by domain rather than monolithic structure:
- **Agent configurations** (`agents.yaml`) - LLM settings for different agents → [Agent Config Guide](./agents_config.md)
- **System configurations** (`system.yaml`) - Execution and runtime settings → [System Config Guide](./system_config.md)
- **RAG configurations** (`rag.yaml`) - Knowledge retrieval settings → [RAG Config Guide](./rag_config.md)
- **MCP configurations** (`mcp.yaml`) - Model Context Protocol servers → [MCP Config Guide](./mcp_reference.md)
- **Pricing configurations** (`prices.yaml`) - Cost tracking for different models → [Pricing Config Guide](./prices_config.md)
- **Third-party configurations** (`third_party.yaml`) - External agent integration (LinuxAgent, HardwareAgent) → [Third-Party Config Guide](./third_party_config.md)
### Type Safety + Flexibility
Hybrid approach combining:
- **Fixed typed fields** - IDE autocomplete, type checking, and IntelliSense
- **Dynamic YAML fields** - Add new settings without code changes
**Example:**
```python
# Type-safe access (recommended)
config = get_ufo_config()
max_step = config.system.max_step # IDE autocomplete!
api_model = config.app_agent.api_model
# Dynamic access (for custom fields)
custom_value = config.CUSTOM_FEATURE_FLAG
new_setting = config["NEW_YAML_KEY"]
# Backward compatible (legacy code still works)
max_step_old = config["MAX_STEP"]
```
### Backward Compatibility
Zero breaking changes - existing code continues to work:
- Old configuration paths still supported (`ufo/config/`)
- Old access patterns still work (`config["MAX_STEP"]`)
- Automatic migration warnings guide users to new structure
Your existing code will continue to work without any modifications. The system automatically falls back to legacy paths and access patterns. See the [Migration Guide](./migration.md) for details on upgrading to the new structure.
### Auto-Discovery
No manual file registration needed:
- All `*.yaml` files in `config/ufo/` are automatically loaded
- Files are merged intelligently with deep merging
- Environment-specific overrides (`*_dev.yaml`, `*_test.yaml`) supported
## Directory Structure
```
UFO/
├── config/ ← New Configuration Root (Recommended)
│ ├── ufo/ ← UFO² Configurations
│ │ ├── agents.yaml # LLM agent settings
│ │ ├── agents.yaml.template # Template for setup
│ │ ├── system.yaml # System and runtime settings
│ │ ├── rag.yaml # RAG knowledge settings
│ │ ├── mcp.yaml # MCP server configurations
│ │ ├── prices.yaml # Model pricing
│ │ └── third_party.yaml # Third-party agents (optional)
│ │
│ ├── galaxy/ ← Galaxy Configurations
│ │ ├── agent.yaml # Constellation agent settings
│ │ ├── agent.yaml.template # Template for setup
│ │ ├── constellation.yaml # Constellation runtime settings
│ │ └── devices.yaml # Device/client configurations
│ │
│ ├── config_loader.py # Modern config loader
│ └── config_schemas.py # Type definitions
│
└── ufo/config/ ← Legacy Path (Still Supported)
└── config.yaml # Old monolithic config
```
---
## Galaxy Configuration Files
The Galaxy constellation system has its own set of configuration files in `config/galaxy/`:
| File | Purpose | Template | Documentation |
|------|---------|----------|---------------|
| **constellation.yaml** | Constellation runtime settings (heartbeat, concurrency, step limits) | No | [Galaxy Constellation Config](./galaxy_constellation.md) |
| **devices.yaml** | Device agent definitions (device_id, server_url, capabilities, metadata) | No | [Galaxy Devices Config](./galaxy_devices.md) |
| **agent.yaml** | Constellation agent LLM configuration (API settings, prompts) | **Yes** (.template) | [Galaxy Agent Config](./galaxy_agent.md) |
### Galaxy Configuration Structure
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Galaxy Configuration Loading
```python
# Load Galaxy configurations
from config.config_loader import get_galaxy_config
# Load Galaxy configuration (includes agent and constellation settings)
galaxy_config = get_galaxy_config()
# Access agent configuration (LLM settings)
agent_config = galaxy_config.agent.constellation_agent
# Access constellation runtime settings
constellation_settings = galaxy_config.constellation
# Or use raw dict access for backward compatibility
constellation_id = galaxy_config["CONSTELLATION_ID"]
```
**Galaxy vs UFO Configuration:**
- **UFO Configurations** (`config/ufo/`) - Single-agent automation settings
- **Galaxy Configurations** (`config/galaxy/`) - Multi-device constellation settings
- Both systems can coexist in the same project
---
## Configuration Loading Process
### Priority Chain
The configuration system uses a clear priority chain (highest to lowest):
1. **New modular configs** - `config/{module}/*.yaml`
2. **Legacy monolithic config** - `{module}/config/config.yaml`
3. **Environment variables** - Runtime overrides
When the same setting exists in multiple locations, the **new modular config** takes precedence over legacy configs. Values are merged with later sources overriding earlier ones.
### Loading Algorithm
```python
def load_config():
# Step 1: Start with environment variables (lowest priority)
config_data = dict(os.environ)
# Step 2: Load legacy config if exists (middle priority)
if exists("ufo/config/config.yaml"):
legacy_data = load_yaml("ufo/config/config.yaml")
merge(config_data, legacy_data)
# Step 3: Load new modular configs (highest priority)
for yaml_file in discover("config/ufo/*.yaml"):
new_data = load_yaml(yaml_file)
merge(config_data, new_data)
# Step 4: Create typed config object
return UFOConfig.from_dict(config_data)
```
### Deep Merging
Configuration files are merged recursively, allowing you to split configurations across multiple files without duplication:
```yaml
# config/ufo/agents.yaml
HOST_AGENT:
API_TYPE: "openai"
API_MODEL: "gpt-4o"
# config/ufo/custom.yaml (added later)
HOST_AGENT:
TEMPERATURE: 0.5 # Added to HOST_AGENT
# Result: HOST_AGENT has all three fields
```
Fields from later files are added to (not replacing) earlier configurations.
## File Organization Patterns
### Split by Domain (Current Approach)
```
config/ufo/
├── agents.yaml # All agent LLM configs
├── system.yaml # All system settings
├── rag.yaml # All RAG settings
├── mcp.yaml # All MCP servers
└── prices.yaml # Model pricing
```
**Advantages:** Easy to find related settings, clear separation of concerns, good for documentation.
### Alternative: Split by Agent
```
config/ufo/
├── host_agent.yaml # HOST_AGENT config
├── app_agent.yaml # APP_AGENT config
├── system.yaml # Shared system config
└── rag.yaml # Shared RAG config
```
**Advantages:** Agent-specific settings isolated, easy to customize per agent, good for multi-agent scenarios.
Both patterns work! The loader auto-discovers and merges all YAML files.
## Environment-Specific Overrides
Support for development, testing, and production environments:
```bash
# Base configuration
config/ufo/agents.yaml # All environments
# Environment-specific overrides
config/ufo/agents_dev.yaml # Development only
config/ufo/agents_test.yaml # Testing only
config/ufo/agents_prod.yaml # Production only
```
**Activation**:
```bash
# Set environment
export UFO_ENV=dev # Linux/Mac
$env:UFO_ENV = "dev" # Windows PowerShell
# Configuration loads:
# 1. agents.yaml (base)
# 2. agents_dev.yaml (overrides)
```
## Type System
### Fixed Types (Recommended)
Provides IDE autocomplete and type safety:
```python
@dataclass
class SystemConfig:
max_step: int = 50
max_retry: int = 20
temperature: float = 0.0
# ...
# Usage - IDE knows the types!
config.system.max_step # int
config.system.temperature # float
```
### Dynamic Types (Flexible)
For custom or experimental settings. Learn more about adding custom fields in the [Extending Configuration guide](./extending.md).
**Example:**
```python
# In YAML
MY_CUSTOM_FEATURE: True
NEW_EXPERIMENTAL_SETTING: "value"
# In code - dynamic access
if config.MY_CUSTOM_FEATURE:
setting = config.NEW_EXPERIMENTAL_SETTING
```
### Hybrid Approach
Best of both worlds:
```python
class SystemConfig:
# Fixed fields
max_step: int = 50
# Dynamic extras
_extras: Dict[str, Any]
def __getattr__(self, name):
# Try extras for unknown fields
return self._extras.get(name)
```
## Migration Warnings
The system provides clear warnings when using legacy paths:
```
⚠️ LEGACY CONFIG PATH DETECTED: UFO
Using legacy config: ufo/config/
Please migrate to: config/ufo/
Quick migration:
mkdir -p config/ufo
cp ufo/config/*.yaml config/ufo/
Or use migration tool:
python -m ufo.tools.migrate_config
```
These warnings appear once per session and guide you to migrate to the new structure.
## Best Practices
**Recommended Practices:**
- **Use modular files** - Split by domain or agent
- **Use typed access** - `config.system.max_step` over `config["MAX_STEP"]`
- **Add templates** - Provide `.template` files for sensitive data
- **Document custom fields** - Add comments in YAML
- **Use environment overrides** - For dev/test/prod differences
**Anti-Patterns to Avoid:**
- **Mix old and new** - Migrate fully to new structure
- **Put secrets in YAML** - Use environment variables instead
- **Duplicate settings** - Leverage deep merging
- **Break backward compat** - Keep `config["OLD_KEY"]` working
- **Hardcode paths** - Use config system
## Configuration Lifecycle
```mermaid
graph LR
A[Application Start] --> B[Load Environment Vars]
B --> C[Check for Legacy Config]
C --> D[Load New Modular Configs]
D --> E[Deep Merge All Sources]
E --> F[Apply Transformations]
F --> G[Create Typed Config Object]
G --> H[Cache for Reuse]
H --> I[Application Running]
```
## Next Steps
### UFO Configuration Guides
- **[Agent Configuration](./agents_config.md)** - LLM and API settings for all agents
- **[System Configuration](./system_config.md)** - Runtime and execution settings
- **[RAG Configuration](./rag_config.md)** - Knowledge retrieval and learning settings
- **[MCP Configuration](./mcp_reference.md)** - Model Context Protocol servers
- **[Pricing Configuration](./prices_config.md)** - LLM cost tracking
- **[Third-Party Configuration](./third_party_config.md)** - External agent integration (LinuxAgent, HardwareAgent)
- **[Migration Guide](./migration.md)** - How to migrate from old to new config
- **[Extending Configuration](./extending.md)** - How to add new configuration options
### Galaxy Configuration Guides
- **[Galaxy Constellation Configuration](./galaxy_constellation.md)** - Runtime settings for constellation orchestrator
- **[Galaxy Devices Configuration](./galaxy_devices.md)** - Device definitions and capabilities
- **[Galaxy Agent Configuration](./galaxy_agent.md)** - LLM configuration for constellation agent
## API Reference
For detailed API documentation of configuration classes and methods, see:
- `config.config_loader.ConfigLoader` - Configuration loading and caching
- `config.config_schemas.UFOConfig` - UFO configuration schema
- `config.config_schemas.GalaxyConfig` - Galaxy configuration schema
- `config.config_loader.get_ufo_config()` - Get UFO configuration instance
- `config.config_loader.get_galaxy_config()` - Get Galaxy configuration instance
---
### Documents Docs Galaxy Agent Registration Overview (`documents/docs/galaxy/agent_registration/overview.md`)
# 🌟 Agent Registration & Profiling - Overview
**Agent Registration** is the cornerstone of the AIP (Agent Interaction Protocol) initialization process. It enables dynamic discovery, capability advertisement, and intelligent task allocation across distributed constellation agents.
---
## 📋 Introduction
*An overview of the Constellation Agent architecture showing the registration and profiling system.*
At the core of AIP's initialization process is the **ConstellationClient** (implemented as `ConstellationDeviceManager`), which maintains a global registry of active agents. Any device agent service that exposes a WebSocket endpoint and implements the AIP task dispatch and result-return protocol can be seamlessly integrated into UFO, providing remarkable **extensibility**.
The multi-source profiling pipeline enables **transparent capability discovery** and **safe adaptation** to environmental drift without direct administrator intervention.
For a complete understanding of the constellation system, see:
- [Constellation Overview](../constellation/overview.md) - Multi-device coordination architecture
- [Constellation Agent Overview](../constellation_agent/overview.md) - Agent behavior and patterns
- [AIP Protocol Overview](../../aip/overview.md) - Message protocol details
---
## 🎯 Core Concepts
### Agent Registry
The agent registry is a centralized store that tracks all active constellation agents. Each registered agent is represented by an **AgentProfile** that consolidates comprehensive information about the agent's capabilities, system resources, and operational status.
| Component | Responsibility | Location |
|-----------|---------------|----------|
| **ConstellationDeviceManager** | Central coordinator for device management | `galaxy/client/device_manager.py` |
| **DeviceRegistry** | Device registration and information storage | `galaxy/client/components/device_registry.py` |
| **AgentProfile** | Multi-source agent metadata representation | `galaxy/client/components/types.py` |
| **ClientConnectionManager** | Server-side client connection tracking | `ufo/server/services/client_connection_manager.py` |
### Multi-Source Profiling
Each **AgentProfile** consolidates information from **three distinct sources**, creating a comprehensive and dynamically updated view of each agent.
```mermaid
graph TB
subgraph Sources
UC[User Config devices.yaml]
SM[AIP Registration Service Manifest]
CT[Device Telemetry DeviceInfoProvider]
end
UC -->|device_id, capabilities metadata| AP[AgentProfile]
SM -->|platform, client_type registration_time| AP
CT -->|system_info supported_features| AP
AP --> CR[ConstellationDeviceManager]
CR --> TA[Intelligent Task Routing]
style UC fill:#e1f5ff
style SM fill:#fff4e1
style CT fill:#e8f5e9
style AP fill:#f3e5f5
```
**Source Details:**
| Source | Provider | Information Type | Update Frequency |
|--------|----------|------------------|------------------|
| **1. User Configuration** | Administrator (devices.yaml + constellation.yaml) | Endpoint identity, user preferences, capabilities | Static (config load) |
| **2. Service Manifest** | Device Agent Service (AIP) | Client type, platform, registration metadata | On registration |
| **3. Client Telemetry** | Device Client (DeviceInfoProvider) | Hardware specs, OS info, network status | On connection + periodic updates |
**Note:** While constellation.yaml contains runtime settings like heartbeat intervals, the device-specific configuration is in devices.yaml.
---
## 🔄 Registration Flow
*Agent registration flow: multi-source AgentProfile construction and registration.*
### Registration Process Overview
The registration process follows a well-defined sequence that ensures comprehensive profiling and validation:
```mermaid
sequenceDiagram
participant Admin as Administrator
participant CDM as ConstellationDeviceManager
participant Server as UFO Server
participant DIP as DeviceInfoProvider
Note over Admin,DIP: Phase 1: User Configuration
Admin->>CDM: register_device(device_id, capabilities)
CDM->>CDM: Create AgentProfile
Note over Admin,DIP: Phase 2: WebSocket Connection
CDM->>Server: connect_device()
Server-->>CDM: Connection established
Note over Admin,DIP: Phase 3: Service Registration
CDM->>Server: REGISTER message
Server-->>CDM: Registration confirmed
Note over Admin,DIP: Phase 4: Telemetry Collection
CDM->>Server: request_device_info()
Server->>DIP: collect_system_info()
DIP-->>Server: system_info
Server-->>CDM: system_info
CDM->>CDM: Merge into AgentProfile
Note over Admin,DIP: Phase 5: Ready for Tasks
CDM->>CDM: Set device to IDLE
```
**Registration Phases:**
| Phase | Description | Components Involved | Result |
|-------|-------------|---------------------|--------|
| **1. User Configuration** | Administrator registers device with endpoint and capabilities | ConstellationDeviceManager, DeviceRegistry | AgentProfile created with user-specified data |
| **2. WebSocket Connection** | Establish persistent connection to device agent server | WebSocketConnectionManager | Active WebSocket channel |
| **3. Service Registration** | AIP registration protocol exchange with capability advertisement | RegistrationProtocol, UFOWebSocketHandler | Client type and platform recorded |
| **4. Telemetry Collection** | Retrieve runtime system information from device | DeviceInfoProvider, DeviceInfoProtocol | Hardware, OS, and feature data merged |
| **5. Activation** | Set device to IDLE state, ready for task assignment | DeviceRegistry | Agent ready for constellation tasks |
Devices can be registered with `auto_connect=True` to automatically establish connection, or `auto_connect=False` to require manual connection via `connect_device()`.
---
## 📊 AgentProfile Structure
The **AgentProfile** is the primary data structure representing a registered constellation agent. For detailed information about the AgentProfile and its lifecycle operations, see [Agent Profile Documentation](./agent_profile.md).
### Core Fields
The **AgentProfile** is the primary data structure representing a registered constellation agent:
```python
@dataclass
class AgentProfile:
"""Device information and capabilities"""
# Identity
device_id: str # Unique device identifier
server_url: str # WebSocket endpoint URL
# Platform & Capabilities
os: Optional[str] = None # Operating system (windows, linux, darwin)
capabilities: List[str] # Advertised capabilities/features
metadata: Dict[str, Any] # Additional metadata
# Operational Status
status: DeviceStatus # Current connection/operational status
last_heartbeat: Optional[datetime] # Last heartbeat timestamp
# Connection Management
connection_attempts: int = 0 # Connection retry counter
max_retries: int = 5 # Maximum retry attempts
# Task Execution
current_task_id: Optional[str] = None # Currently executing task ID
```
**Field Categories:**
| Category | Fields | Purpose |
|----------|--------|---------|
| **Identity** | `device_id`, `server_url` | Unique identification and endpoint location |
| **Platform** | `os`, `capabilities`, `metadata` | System type and advertised features |
| **Status** | `status`, `last_heartbeat` | Real-time operational state tracking |
| **Resilience** | `connection_attempts`, `max_retries` | Connection retry management |
| **Execution** | `current_task_id` | Task assignment tracking |
### Metadata Structure
The `metadata` field is a flexible dictionary that aggregates information from all three sources:
```python
metadata = {
# From User Configuration (Source 1)
"location": "home_office",
"performance": "high",
"description": "Primary development laptop",
"operation_engineer_email": "admin@example.com",
# From Service Manifest (Source 2)
"platform": "windows",
"registration_time": "2025-11-06T10:30:00Z",
# From Client Telemetry (Source 3)
"system_info": {
"platform": "windows",
"os_version": "10.0.22631",
"cpu_count": 16,
"memory_total_gb": 32.0,
"hostname": "DESKTOP-DEV01",
"ip_address": "192.168.1.100",
"platform_type": "computer",
"schema_version": "1.0"
},
"custom_metadata": {
"datacenter": "us-west-2",
"tier": "production"
}
}
```
For a complete example, see the [Agent Profile Documentation](./agent_profile.md#example-profiles).
---
## 🔄 Agent Lifecycle States
*Lifecycle state transitions of the Constellation Agent.*
The agent lifecycle is managed through a state machine that tracks connection, registration, and task execution states. For more details on agent behavior and state management, see [Constellation Agent State Management](../constellation_agent/state.md).
### State Definitions
```python
class DeviceStatus(Enum):
"""Device connection status"""
DISCONNECTED = "disconnected" # Not connected to server
CONNECTING = "connecting" # Attempting to establish connection
CONNECTED = "connected" # Connected, initializing
REGISTERING = "registering" # Performing registration handshake
IDLE = "idle" # Connected and ready for tasks
BUSY = "busy" # Executing a task
FAILED = "failed" # Connection/execution failed
```
### State Transition Diagram
```mermaid
stateDiagram-v2
[*] --> DISCONNECTED: Initial State
DISCONNECTED --> CONNECTING: register_device() / connect_device()
CONNECTING --> CONNECTED: WebSocket established
CONNECTING --> FAILED: Connection error
CONNECTED --> REGISTERING: Send REGISTER message
REGISTERING --> IDLE: Registration confirmed + system info collected
REGISTERING --> FAILED: Registration rejected
IDLE --> BUSY: assign_task_to_device()
BUSY --> IDLE: Task completed
BUSY --> FAILED: Task failed / device disconnected
FAILED --> CONNECTING: Automatic reconnection
IDLE --> DISCONNECTED: disconnect_device() / connection lost
BUSY --> DISCONNECTED: disconnect_device() / connection lost
DISCONNECTED --> [*]: shutdown()
```
**Transition Events:**
| From State | To State | Trigger | Action |
|------------|----------|---------|--------|
| DISCONNECTED | CONNECTING | `connect_device()` | Initiate WebSocket connection |
| CONNECTING | CONNECTED | WebSocket handshake complete | Update status |
| CONNECTED | REGISTERING | Send REGISTER message | AIP registration protocol |
| REGISTERING | IDLE | Registration confirmed | Collect system info, ready for tasks |
| IDLE | BUSY | `assign_task_to_device()` | Execute task |
| BUSY | IDLE | Task completes | Clear current_task_id |
| Any | DISCONNECTED | Connection lost | Cleanup, schedule reconnection |
| FAILED | CONNECTING | Retry timer | Attempt reconnection (if under max_retries) |
**Important:** When a device disconnects or enters FAILED state, the system automatically schedules reconnection attempts up to `max_retries` times with `reconnect_delay` interval.
---
## 🛠️ Key Components
### 1. ConstellationDeviceManager
**File:** `galaxy/client/device_manager.py`
The central coordinator for all device management operations in the constellation system.
**Responsibilities:**
- Device registration and lifecycle management
- Connection establishment and monitoring
- Task assignment and execution coordination
- Automatic reconnection handling
**Key Methods:**
```python
class ConstellationDeviceManager:
async def register_device(
device_id: str,
server_url: str,
os: str,
capabilities: List[str],
metadata: Dict[str, Any],
auto_connect: bool = True
) -> bool
async def connect_device(device_id: str) -> bool
async def assign_task_to_device(
task_id: str,
device_id: str,
task_description: str,
task_data: Dict[str, Any]
) -> ExecutionResult
def get_device_info(device_id: str) -> Optional[AgentProfile]
```
See [Device Registry Documentation](./device_registry.md) for DeviceRegistry details.
### 2. DeviceRegistry
**File:** `galaxy/client/components/device_registry.py`
Manages device registration and information storage with a focus on data management.
**Responsibilities:**
- Store and retrieve AgentProfile instances
- Update device status and metadata
- Track connection attempts and heartbeats
- Merge multi-source information
**Key Methods:**
```python
class DeviceRegistry:
def register_device(...) -> AgentProfile
def update_device_status(device_id: str, status: DeviceStatus)
def update_device_system_info(device_id: str, system_info: Dict)
def set_device_busy(device_id: str, task_id: str)
def set_device_idle(device_id: str)
```
### 3. RegistrationProtocol (AIP)
**File:** `aip/protocol/registration.py`
Handles AIP registration message exchange for both device and constellation clients.
**Responsibilities:**
- Device agent registration
- Constellation client registration
- Capability advertisement
- Registration validation and confirmation
**Key Methods:**
```python
class RegistrationProtocol(AIPProtocol):
async def register_as_device(
device_id: str,
metadata: Dict[str, Any],
platform: str
) -> bool
async def register_as_constellation(
constellation_id: str,
target_device: str,
metadata: Dict[str, Any]
) -> bool
async def send_registration_confirmation()
async def send_registration_error(error: str)
```
See [AIP Protocol Documentation](../../aip/overview.md) for protocol details.
### 4. DeviceInfoProvider
**File:** `ufo/client/device_info_provider.py`
Collects device system information (telemetry source).
**Responsibilities:**
- Auto-detect platform, OS, and hardware
- Collect CPU, memory, network information
- Detect supported features based on platform
- Provide DeviceSystemInfo structure
**Key Methods:**
```python
class DeviceInfoProvider:
@staticmethod
def collect_system_info(
client_id: str,
custom_metadata: Optional[Dict]
) -> DeviceSystemInfo
```
See [Device Info Provider Documentation](../../client/device_info.md) for telemetry details.
### 5. ClientConnectionManager (Server)
**File:** `ufo/server/services/client_connection_manager.py`
Server-side client connection tracking and management. For detailed information about the server-side implementation, see [Client Connection Manager Documentation](../../server/client_connection_manager.md).
**Responsibilities:**
- Track connected clients (devices and constellations)
- Store device system information received during registration
- Manage session-to-client mappings
- Merge server configuration with client telemetry
**Key Methods:**
```python
class ClientConnectionManager:
def add_client(
client_id: str,
platform: str,
ws: WebSocket,
client_type: ClientType,
metadata: Dict
)
def get_device_system_info(device_id: str) -> Optional[Dict]
```
---
## 📝 Configuration
Agent registration uses two configuration files:
**1. `config/galaxy/devices.yaml`** - Device definitions:
- Device endpoints and identities
- User-specified capabilities and metadata
- Connection parameters (max retries, auto-connect)
**2. `config/galaxy/constellation.yaml`** - Runtime settings:
- Constellation identification and logging
- Heartbeat interval and reconnection delay
- Task concurrency and step limits
See [Galaxy Devices Configuration](../../configuration/system/galaxy_devices.md) and [Galaxy Constellation Configuration](../../configuration/system/galaxy_constellation.md) for details.
**Example Device Configuration (devices.yaml):**
```yaml
# Device Configuration - YAML Format
# Runtime settings are configured in constellation.yaml
devices:
- device_id: "windowsagent"
server_url: "ws://localhost:5005/ws"
os: "windows"
capabilities: ["web_browsing", "office_applications"]
metadata:
location: "office_desktop"
performance: "high"
max_retries: 5
auto_connect: true
```
For complete configuration schema, examples, and best practices, see:
👉 **[Galaxy Devices Configuration Guide](../../configuration/system/galaxy_devices.md)**
---
## 🚀 Usage Example
### Basic Registration
```python
from galaxy.client.device_manager import ConstellationDeviceManager
# Initialize manager
manager = ConstellationDeviceManager(
task_name="test_constellation",
heartbeat_interval=30.0,
reconnect_delay=5.0
)
# Register and connect device
success = await manager.register_device(
device_id="windows_workstation",
server_url="ws://localhost:5005/ws",
os="windows",
capabilities=["gui", "browser", "office"],
metadata={
"location": "home_office",
"performance": "medium"
},
auto_connect=True # Automatically connect after registration
)
if success:
print("✅ Device registered and connected")
# Get device profile
profile = manager.get_device_info("windows_workstation")
print(f"Device: {profile.device_id}")
print(f"Status: {profile.status.value}")
print(f"Capabilities: {profile.capabilities}")
print(f"System Info: {profile.metadata.get('system_info')}")
```
### Task Assignment
```python
# Assign task to registered device
result = await manager.assign_task_to_device(
task_id="task_001",
device_id="windows_workstation",
task_description="Open Excel and create a report",
task_data={"file_path": "C:\\Reports\\monthly.xlsx"},
timeout=300.0
)
print(f"Task Status: {result.status}")
print(f"Result: {result.result}")
```
For more details on task assignment and execution, see:
- [Registration Flow Documentation](./registration_flow.md) - Detailed examples
- [Constellation Task Distribution](../constellation/overview.md) - Task routing strategies
---
## 🔗 Cross-References
### Related Documentation
| Topic | Document | Description |
|-------|----------|-------------|
| **Device Info Collection** | [Device Info Provider](../../client/device_info.md) | Client-side telemetry collection |
| **AIP Protocol** | [AIP Overview](../../aip/overview.md) | Agent Interaction Protocol fundamentals |
| **AIP Messages** | [AIP Messages](../../aip/messages.md) | Message structure and types |
| **Agent Profile** | [Agent Profile](./agent_profile.md) | Detailed AgentProfile structure |
| **Registration Flow** | [Registration Flow](./registration_flow.md) | Step-by-step registration process |
| **Galaxy Devices Config** | [Galaxy Devices Configuration](../../configuration/system/galaxy_devices.md) | YAML configuration reference |
| **Device Registry** | [Device Registry](./device_registry.md) | Registry component details |
| **Constellation System** | [Constellation Overview](../constellation/overview.md) | Multi-device coordination |
| **Client Connection Manager** | [Server Connection Manager](../../server/client_connection_manager.md) | Server-side connection tracking |
### Architecture Diagrams
- **Constellation Agent Overview**: `documents/docs/img/constellation_agent.png`
- **Agent Registration Flow**: `documents/docs/img/agent_registry.png`
- **Agent Lifecycle States**: `documents/docs/img/agent_state.png`
---
## 💡 Key Benefits
The multi-source profiling approach provides several advantages:
**1. Improved Task Allocation Accuracy**
- Administrators specify high-level capabilities
- Service manifests advertise supported tools
- Telemetry provides real-time hardware status
**2. Transparent Capability Discovery**
- No manual system info entry required
- Automatic feature detection based on platform
- Dynamic updates without configuration changes
**3. Safe Adaptation to Environmental Drift**
- System changes (upgrades, hardware additions) automatically reflected
- No administrator intervention needed for routine updates
- Consistent metadata across distributed agents
**4. Reliable Scheduling Decisions**
- Fresh and accurate information for task routing
- Hardware-aware task assignment (CPU/memory requirements)
- Platform-specific capability matching
---
## 🎯 Next Steps
1. **Understand AgentProfile in Detail**: Read [Agent Profile Documentation](./agent_profile.md)
2. **Learn Registration Process**: Follow [Registration Flow](./registration_flow.md)
3. **Configure Your Devices**: See [Galaxy Devices Configuration](../../configuration/system/galaxy_devices.md)
4. **Explore Device Registry**: Check [Device Registry](./device_registry.md)
5. **Study AIP Protocol**: Read [AIP Documentation](../../aip/overview.md)
---
## 📚 Additional Resources
- **Source Code**: `galaxy/client/device_manager.py`
- **AIP Protocol**: `aip/protocol/registration.py`
- **Device Info**: `ufo/client/device_info_provider.py`
- **Configuration**: `config/galaxy/devices.yaml`
**Best Practice:** Always configure devices with meaningful metadata and capabilities to enable intelligent task routing. The system will automatically enhance this information with telemetry data.
---
### Documents Docs Galaxy Client Overview (`documents/docs/galaxy/client/overview.md`)
# Galaxy Client Overview
Galaxy Client is the client-side layer responsible for multi-device coordination in the UFO³ framework. At its core is **ConstellationClient**, which manages device registration, connection, and task assignment. **GalaxyClient** provides a lightweight wrapper offering convenient session management interfaces.
## Related Documentation
- [ConstellationClient](./constellation_client.md) - Core device coordination component
- [DeviceManager](./device_manager.md) - Low-level connection management
- [Components](./components.md) - Modular component architecture
- [AIP Integration](./aip_integration.md) - Communication protocol integration
- [GalaxyClient](./galaxy_client.md) - Session wrapper API
- [Configuration](../../configuration/system/galaxy_constellation.md) - Device configuration guide
## The Complete Path: From User Request to Device Execution
To understand Galaxy Client, we first need to see the entire system workflow. When a user submits a task request, the system processes it through several layers:
### 1. User Interaction Layer (Optional)
Users can interact with the Galaxy system in two ways:
**Interactive Mode**: Users input natural language requests through a command-line interface (CLI), which are received and processed by GalaxyClient. This mode is primarily used for rapid prototyping and manual testing.
**Programmatic Mode**: Developers directly call the Python API of ConstellationClient or GalaxyClient, integrating Galaxy into their applications. This is the recommended approach for production environments.
### 2. Session Management Layer (GalaxyClient)
GalaxyClient's role is to manage the lifecycle of task sessions. It doesn't handle specific device operations but instead:
- Initializes and holds a ConstellationClient instance
- Creates a GalaxySession for each user request
- Passes requests to ConstellationAgent for DAG planning (task decomposition)
- Coordinates TaskConstellationOrchestrator to execute the DAG
- Collects and aggregates execution results
**GalaxyClient is optional**. If your application doesn't need session management, you can use ConstellationClient directly.
### 3. Device Coordination Layer (ConstellationClient)
ConstellationClient is the heart of Galaxy Client. It is responsible for:
**Device Management**: Registering devices (each device has a unique ID, server URL, capability list, etc.), connecting to devices (via WebSocket), disconnecting devices, and monitoring device health status.
**Task Assignment**: Receiving task requests from upper layers (TaskConstellationOrchestrator), selecting appropriate devices based on capabilities, sending tasks to devices via the AIP protocol, and waiting for and collecting task execution results.
ConstellationClient doesn't concern itself with how tasks are decomposed (that's ConstellationAgent's responsibility) or how DAGs are executed (that's TaskConstellationOrchestrator's responsibility). It focuses on "device-level matters."
### 4. Connection Management Layer (DeviceManager)
DeviceManager is the core internal component of ConstellationClient, responsible for all low-level connection management:
**WebSocket Connection Establishment**: Establishes WebSocket connections with Agent Server, sends AIP REGISTER messages to register device identity, and requests device system information (DEVICE_INFO_REQUEST).
**Connection Monitoring**: Sends HEARTBEAT messages every 20-30 seconds to check if devices are online. If a timeout occurs with no response, it triggers disconnection handling and automatically attempts reconnection (up to max_retries times).
**Message Routing**: Starts a background message processing loop, receives messages returned by devices (TASK_END, COMMAND_RESULTS, etc.), and dispatches messages to appropriate handlers.
**Task Queuing**: If a device is busy executing another task, new tasks are queued and automatically dequeued when the device becomes idle.
### 5. Protocol Layer (AIP)
All communication with devices goes through the [Agent Interaction Protocol (AIP)](../../aip/overview.md). AIP is a WebSocket-based messaging protocol that defines standard message types and interaction flows. Main message types used by Galaxy Client include:
- `REGISTER`: Register device identity with Agent Server
- `DEVICE_INFO_REQUEST/RESPONSE`: Request and return device system information
- `TASK`: Assign task to device
- `TASK_END`: Device reports task completion
- `HEARTBEAT/HEARTBEAT_ACK`: Heartbeat health check
- `COMMAND_RESULTS`: Device reports intermediate execution results
- `ERROR`: Error reporting
For detailed AIP explanation, see [AIP Integration](./aip_integration.md).
## Component Responsibilities
Having understood the overall flow, let's examine the specific responsibilities of each component:
### ConstellationClient: The Device Coordination Facade
ConstellationClient implements the Facade pattern. It provides simple device management APIs externally while delegating actual work to DeviceManager internally.
**What it does:**
```python
# Register device
await client.register_device(
device_id="windows_pc",
server_url="ws://192.168.1.100:5000/ws",
os="windows",
capabilities=["office", "web", "email"]
)
# Connect device
success = await client.connect_device("windows_pc")
# Assign task
result = await client.assign_task_to_device(
device_id="windows_pc",
task_request=TaskRequest(...)
)
# Query status
status = client.get_device_status("windows_pc")
```
**What it doesn't do:**
- DAG planning (handled by ConstellationAgent)
- DAG execution (handled by TaskConstellationOrchestrator)
- Session management (handled by GalaxySession)
See [ConstellationClient documentation](./constellation_client.md) for detailed API reference.
### DeviceManager: The Connection Management Engine
DeviceManager is the "engine" of ConstellationClient. It uses 5 modular components to accomplish connection management:
**DeviceRegistry**: Stores AgentProfiles for all registered devices (including device ID, URL, status, capabilities, metadata, etc.). This component maintains the single source of truth for device state. When a device connects, disconnects, or changes status, DeviceRegistry is updated. Other components query DeviceRegistry to make decisions.
**WebSocketConnectionManager**: Manages WebSocket connection lifecycle (connect, disconnect, send messages). This component handles the low-level WebSocket operations, including establishing connections, handling connection errors, and sending AIP messages. It maintains a mapping from device_id to WebSocket objects.
**HeartbeatManager**: Background heartbeat loop that periodically sends HEARTBEAT to check device health. This runs as an independent asyncio task for each connected device. If a device fails to respond within the timeout period (2 × heartbeat_interval), HeartbeatManager triggers the disconnection handler, allowing the system to detect and respond to connection failures quickly.
**MessageProcessor**: Background message processing loop that receives and routes AIP messages. This component runs a continuous loop for each device, receiving messages from the WebSocket and dispatching them to appropriate handlers. For example, TASK_END messages are used to complete task futures, COMMAND_RESULTS are logged for progress tracking, and ERROR messages trigger error handling.
**TaskQueueManager**: Manages task queue for each device, queuing tasks when device is busy. When a task is assigned to a busy device, it's placed in that device's queue. When the device completes its current task and becomes IDLE, TaskQueueManager automatically dequeues the next task and executes it. This ensures tasks are never lost even when devices are overloaded.
This modular design ensures each component has a single responsibility, making testing and maintenance easier. See [DeviceManager documentation](./device_manager.md) for details.
### GalaxyClient: Session Management Wrapper
GalaxyClient provides a higher-level abstraction on top of ConstellationClient:
```python
client = GalaxyClient()
await client.initialize() # Initialize ConstellationClient and connect devices
# Process user request (internally creates GalaxySession, calls ConstellationAgent for DAG planning)
result = await client.process_request("Open Excel and create a sales chart")
await client.shutdown() # Cleanup resources
```
GalaxyClient's main value lies in:
- Simplifying initialization flow (automatically loads device info from config)
- Providing session management (creates independent GalaxySession for each request)
- Integrating display components (Rich console output, progress bars, etc.)
- Supporting interactive mode (command-line interface)
If your application already has its own session management logic, you can skip GalaxyClient and use ConstellationClient directly. See [GalaxyClient documentation](./galaxy_client.md) for detailed API.
## Typical Workflow Example
Let's walk through a complete example, from user request to device execution:
### Scenario: Processing a Multi-Device Task
Suppose a user submits: "Download sales.xlsx from email, analyze it in Excel on Windows, then generate a report PDF on Linux".
**Step 1: Initialize GalaxyClient**
```python
client = GalaxyClient()
await client.initialize()
```
What happens inside `initialize()`:
1. GalaxyClient loads device information from config file (`device_info.yaml`)
2. Creates ConstellationClient instance and passes configuration
3. ConstellationClient calls `device_manager.register_device()` to register each device
4. If `auto_connect: true` is configured, automatically calls `device_manager.connect_device()`
5. DeviceManager executes connection flow for each device (detailed below)
**Step 2: Device Connection Flow (Inside DeviceManager)**
For each device (e.g., "windows_pc" and "linux_server"), DeviceManager executes:
```mermaid
sequenceDiagram
participant DM as DeviceManager
participant WS as WebSocketConnectionManager
participant Server as Agent Server
participant Device as Device Agent
Note over DM,Device: 1. Establish WebSocket Connection
DM->>WS: connect_to_device(device_info)
WS->>Server: WebSocket handshake
Server-->>WS: Connection established
Note over DM,Device: 2. Register Device Identity (AIP REGISTER)
WS->>Server: REGISTER message
Server->>Device: Forward registration
Device-->>Server: Service manifest (available MCP servers)
Server-->>WS: REGISTER_CONFIRMATION
Note over DM,Device: 3. Request Device System Info
WS->>Server: DEVICE_INFO_REQUEST
Server->>Device: Request system info
Device-->>Server: System info (CPU, memory, OS, etc.)
Server-->>WS: DEVICE_INFO_RESPONSE
WS->>DM: Update AgentProfile
Note over DM,Device: 4. Start Background Services
DM->>DM: Start MessageProcessor (message handling loop)
DM->>DM: Start HeartbeatManager (heartbeat loop)
DM->>DM: Set device status to IDLE
```
This sequence diagram shows the connection establishment process. First, a WebSocket connection is established with the Agent Server. Then, the device registers its identity through the AIP REGISTER message, allowing the server to know which device is connecting and what capabilities it offers. Next, the client requests detailed system information from the device to populate the AgentProfile with actual hardware and software details. Finally, background services are started to maintain the connection and handle incoming messages.
**Step 3: User Request Processing**
```python
result = await client.process_request("Download sales.xlsx...")
```
Inside `process_request()`:
1. GalaxyClient creates a GalaxySession
2. GalaxySession calls ConstellationAgent for task planning
3. ConstellationAgent (LLM-powered) decomposes task into DAG:
- Task 1: Download sales.xlsx from email (requires "email" capability)
- Task 2: Analyze in Excel (requires "office" capability, depends on Task 1)
- Task 3: Generate PDF on Linux (requires "pdf_generation" capability, depends on Task 2)
4. TaskConstellationOrchestrator executes DAG:
- Based on capability matching, Task 1 assigned to device with "email" capability
- Task 2 assigned to "windows_pc" (has "office" capability)
- Task 3 assigned to "linux_server" (has "pdf_generation" capability)
The DAG structure ensures tasks execute in the correct order respecting dependencies, while allowing independent tasks to run in parallel across different devices.
**Step 4: Task Assignment and Execution (ConstellationClient/DeviceManager)**
For each task, ConstellationClient calls:
```python
result = await client.assign_task_to_device(
device_id="windows_pc",
task_request=TaskRequest(
task_id="task_2",
request="Analyze sales.xlsx in Excel",
...
)
)
```
Inside `assign_task_to_device()`:
1. DeviceManager checks device status (via DeviceRegistry)
2. If device is IDLE, execute task immediately
3. If device is BUSY, task enters queue (TaskQueueManager)
4. WebSocketConnectionManager sends TASK message to device via AIP
5. MessageProcessor waits in background for device to return COMMAND_RESULTS and TASK_END
6. When task completes, DeviceManager changes device status back to IDLE
7. If there are queued tasks, automatically dequeue and execute next task
The queuing mechanism ensures no tasks are lost when devices are busy, and tasks are executed in order as devices become available.
**Step 5: Connection Monitoring (Continuous Background Process)**
Throughout task execution, HeartbeatManager continuously monitors each device:
- Sends HEARTBEAT message every 20-30 seconds
- If device responds, updates `last_heartbeat` timestamp
- If timeout with no response (2 × heartbeat_interval), triggers disconnection handling:
- Stops MessageProcessor and HeartbeatManager
- Sets device status to DISCONNECTED
- If device was executing a task, marks task as failed
- Attempts automatic reconnection (up to max_retries times)
This continuous monitoring ensures the system quickly detects and responds to connection failures, maintaining reliable communication with devices.
**Step 6: Result Collection and Return**
After all tasks complete:
1. TaskConstellationOrchestrator aggregates all task results
2. GalaxySession generates session results (including execution time, rounds, DAG statistics)
3. GalaxyClient returns results to user
4. Results are automatically saved to log directory
The complete execution trace is preserved in logs for debugging and analysis.
## Relationships with Other System Components
Galaxy Client is not an isolated system—it closely collaborates with other UFO³ components:
### Depends on Agent Server for Message Routing
Galaxy Client doesn't connect directly to devices but routes through [Agent Server](../../server/overview.md). Agent Server's role is to:
**Maintain Device Registry**: Tracks which devices are online and their connection details. When a device connects, Agent Server registers it in the central registry.
**Route Messages**: Forwards TASK messages from Galaxy Client to the correct device based on device_id. The server acts as a message broker, decoupling clients from devices.
**Broadcast Device Status**: Notifies clients when devices come online or go offline, enabling clients to maintain accurate device availability information.
**Load Balancing**: If multiple clients connect to the same device, Agent Server can distribute load and prevent conflicts.
### Used by ConstellationAgent for Task Planning
When GalaxyClient receives a user request, it calls [ConstellationAgent](../constellation_agent/overview.md) to decompose the request into a DAG (Directed Acyclic Graph). ConstellationAgent is LLM-powered and can:
**Understand Natural Language**: Parses user requests to identify subtasks and their relationships. For example, "Download file and then analyze it" is recognized as two sequential tasks.
**Identify Task Dependencies**: Determines which tasks must complete before others can start, constructing a proper dependency graph.
**Suggest Device Assignments**: Based on device capabilities, recommends which device should execute each task. If a task requires "office" capability, it's assigned to devices that advertise this capability.
**Dynamically Adjust DAG**: If issues arise during execution (e.g., a device fails), ConstellationAgent can replan and modify the DAG to adapt to the new situation.
For more details, see [ConstellationAgent Documentation](../constellation_agent/overview.md).
### Coordinates with TaskConstellationOrchestrator for DAG Execution
Once ConstellationAgent creates the DAG, [TaskConstellationOrchestrator](../constellation_orchestrator/overview.md) executes it across devices. The orchestrator:
- **Respects Dependencies**: Ensures tasks execute in the correct order based on the DAG structure
- **Selects Devices**: Chooses appropriate devices based on capability matching
- **Parallel Execution**: Runs independent tasks concurrently across different devices
- **Handles Failures**: Manages task failures and triggers replanning if needed
For more details, see [TaskConstellationOrchestrator Documentation](../constellation_orchestrator/overview.md).
### Collaborates with Device Agents for Task Execution
The actual task execution happens on [Device Agents](../../client/overview.md) running on each device (such as UFO² Desktop Agent, Linux Agent, etc.). Device Agents are responsible for:
**Receiving Tasks**: Accepts tasks from Agent Server and parses task requirements. Each task specifies what action to perform and what parameters to use.
**Invoking MCP Servers**: Calls local MCP servers to perform specific operations (such as opening Excel, running commands, etc.). MCP servers provide the actual execution capabilities.
**Reporting Progress**: Sends intermediate execution results through COMMAND_RESULTS messages, allowing clients to track progress in real-time.
**Handling Errors**: Deals with local errors and exceptions, reporting them back to the client through ERROR messages for proper error handling.
### Unified Communication through AIP Protocol
All cross-component communication follows the [AIP protocol](../../aip/overview.md). AIP provides:
**Standardized Message Formats**: Uses Pydantic models to define message structure, ensuring type safety and validation at both ends of communication.
**Type-Safe Message Validation**: Automatically validates message fields using Pydantic, catching errors early before they propagate through the system.
**Request-Response Correlation**: Uses request_id/response_id fields to match requests with their responses, enabling proper async handling.
**Error Handling Mechanism**: Defines standard ERROR message types for reporting and handling failures consistently across all components.
## Configuration and Deployment
### Device Configuration
Device information is defined through configuration files. See [Galaxy Configuration](../../configuration/system/galaxy_constellation.md) for complete configuration options.
A typical configuration example:
```yaml
# config/galaxy/constellation.yaml
task_name: "production_constellation"
heartbeat_interval: 30.0 # Heartbeat interval (seconds)
reconnect_delay: 5.0 # Reconnection delay (seconds)
max_concurrent_tasks: 5 # Max concurrent tasks per device
devices:
- device_id: "windows_pc"
server_url: "ws://192.168.1.100:5000/ws"
os: "windows"
capabilities: ["office", "email", "web"]
auto_connect: true
max_retries: 5 # Maximum reconnection attempts
- device_id: "linux_server"
server_url: "ws://192.168.1.101:5000/ws"
os: "linux"
capabilities: ["database", "api", "pdf_generation"]
auto_connect: true
max_retries: 10
```
Configuration fields explained:
- **task_name**: Unique identifier for this constellation, used in logs and debugging
- **heartbeat_interval**: How often to check device health (recommended: 20-30 seconds)
- **reconnect_delay**: Wait time between reconnection attempts (recommended: 3-5 seconds)
- **max_concurrent_tasks**: Maximum tasks a device can execute simultaneously
- **capabilities**: List of capabilities each device provides, used for task assignment
- **auto_connect**: Whether to automatically connect when client initializes
- **max_retries**: Maximum reconnection attempts before giving up
### Development vs Production Environment
**Development Recommendations:**
- Use interactive mode for quick testing: `python -m galaxy --interactive`
- Enable DEBUG log level for detailed information
- Single-device configuration to simplify debugging
- Use local Agent Server (`ws://127.0.0.1:5000/ws`)
- Lower heartbeat_interval (e.g., 10 seconds) for faster failure detection
**Production Recommendations:**
- Use WSS (secure WebSocket) instead of WS for encrypted communication
- Configure reasonable heartbeat_interval (20-30 seconds) to balance responsiveness and network overhead
- Set appropriate max_retries (5-10 attempts) based on network reliability
- Enable automatic reconnection (`auto_connect: true`) for resilience
- Monitor device health status via `get_device_status()` API and set up alerts
- Configure log rotation and archiving to prevent disk space issues
- Use connection pooling if connecting to many devices
- Implement circuit breaker pattern for failing devices
## Detailed Component Documentation
- [ConstellationClient API Reference](./constellation_client.md) - Complete device coordination API
- [DeviceManager Internals](./device_manager.md) - Detailed connection management mechanisms
- [Components Module](./components.md) - Detailed explanation of 5 core components
- [AIP Integration](./aip_integration.md) - How to use the communication protocol
- [GalaxyClient Session Wrapper](./galaxy_client.md) - Session management API
## Summary
Galaxy Client provides the core multi-device coordination capabilities in UFO³. Through layered design, it simplifies complex distributed system management into clear APIs:
- **ConstellationClient** is the core of device management, handling device registration, connection, and task assignment
- **DeviceManager** is the underlying engine, processing WebSocket, heartbeat, message routing, and task queuing
- **GalaxyClient** is an optional session wrapper, providing more convenient high-level APIs
If you're new to Galaxy Client, we recommend reading the documentation in this order:
1. This Overview (understand overall architecture and workflow)
2. [ConstellationClient](./constellation_client.md) (learn core API)
3. [Components](./components.md) (understand modular components)
4. [DeviceManager](./device_manager.md) (dive deep into connection management)
5. [AIP Integration](./aip_integration.md) (master communication protocol)
If you need to get started quickly, jump directly to [GalaxyClient](./galaxy_client.md) example code.
---
### Documents Docs Galaxy Constellation Overview (`documents/docs/galaxy/constellation/overview.md`)
# Task Constellation — Overview
Example of a Task Constellation illustrating both sequential and parallel dependencies
---
## 🌌 Introduction
The **Task Constellation** is the central abstraction in Galaxy that captures the concurrent and asynchronous structure of distributed task execution. It provides a formal, directed acyclic graph (DAG) representation of complex workflows, enabling consistent scheduling, fault-tolerant orchestration, and runtime dynamism across heterogeneous devices.
At its core, a Task Constellation decomposes complex user requests into interdependent subtasks connected through explicit dependency edges. This formalism not only enables correct distributed execution but also supports runtime adaptation—allowing new tasks or dependencies to be introduced as the workflow evolves.
For information on how Task Constellations are orchestrated and scheduled, see the [Constellation Orchestrator](../constellation_orchestrator/overview.md) documentation. To understand how agents interact with constellations, refer to the [Constellation Agent](../constellation_agent/overview.md) guide.
---
## 🎯 Core Components
The Task Constellation framework consists of four primary components:
| Component | Purpose | Key Features |
|-----------|---------|--------------|
| **[TaskStar](task_star.md)** | Atomic execution unit | Self-contained task with description, device assignment, execution state, dependencies |
| **[TaskStarLine](task_star_line.md)** | Dependency relationship | Directed edge with conditional logic, success-only, completion-only, or unconditional execution |
| **[TaskConstellation](task_constellation.md)** | DAG orchestrator | Complete workflow graph with validation, scheduling, and dynamic modification |
| **[ConstellationEditor](constellation_editor.md)** | Interactive editor | Command pattern-based interface with undo/redo for safe constellation manipulation |
---
## 📐 Formal Model
### Mathematical Foundation
A Task Constellation $\mathcal{C}$ is formally defined as a directed acyclic graph (DAG):
$$
\mathcal{C} = (\mathcal{T}, \mathcal{E})
$$
where:
- $\mathcal{T}$ is the set of all **TaskStars** (task nodes)
- $\mathcal{E}$ is the set of **TaskStarLines** (dependency edges)
### TaskStar Representation
Each TaskStar $t_i \in \mathcal{T}$ encapsulates a complete task specification:
$$
t_i = (\text{name}_ i, \text{description}_ i, \text{target\_device\_id}_ i, \text{tips}_ i, \text{status}_ i, \text{dependencies}_ i)
$$
**Components:**
- **name**: Short name for the task
- **description**: Natural-language specification sent to the device agent
- **target_device_id**: ID of the device agent responsible for execution
- **tips**: List of guidance hints to help the device agent complete the task
- **status**: Current execution state (pending, running, completed, failed, cancelled, waiting_dependency)
- **dependencies**: Set of prerequisite task IDs that must complete first
### TaskStarLine Representation
Each TaskStarLine $e_{i \rightarrow j} \in \mathcal{E}$ represents a dependency from task $t_i$ to task $t_j$.
**Dependency Types:**
| Type | Behavior |
|------|----------|
| **Unconditional** | $t_j$ always waits for $t_i$ to complete |
| **Success-only** | $t_j$ proceeds only if $t_i$ succeeds |
| **Completion-only** | $t_j$ proceeds when $t_i$ completes (regardless of success/failure) |
| **Conditional** | $t_j$ proceeds based on a user-defined or runtime condition |
---
## ✨ Key Advantages
### 1. Explicit Task Ordering
Task dependencies are explicitly captured in the DAG structure, ensuring correctness across distributed execution without ambiguity.
### 2. Natural Parallelism
The DAG topology naturally exposes parallelizable tasks, enabling efficient concurrent execution across heterogeneous devices.
### 3. Runtime Dynamism
Unlike static DAG schedulers, Task Constellations are **mutable objects**. Tasks and dependency edges can be:
- **Added**: Introduce new subtasks or diagnostic tasks
- **Removed**: Prune completed or redundant nodes
- **Modified**: Rewire dependencies, update conditions, change device assignments
This enables adaptive execution without restarting the entire workflow.
### 4. Formal Guarantees
The DAG representation provides formal properties:
- **Acyclicity**: No circular dependencies
- **Causal consistency**: Execution respects logical ordering
- **Safe concurrency**: Parallel execution without race conditions
---
## 🔄 Lifecycle States
The Task Constellation progresses through several states during its lifecycle:
```mermaid
stateDiagram-v2
[*] --> CREATED: Initialize
CREATED --> READY: Add tasks & dependencies
READY --> EXECUTING: Start execution
EXECUTING --> EXECUTING: Tasks running
EXECUTING --> COMPLETED: All tasks succeed
EXECUTING --> FAILED: All tasks fail
EXECUTING --> PARTIALLY_FAILED: Some succeed, some fail
COMPLETED --> [*]
FAILED --> [*]
PARTIALLY_FAILED --> [*]
```
| State | Description |
|-------|-------------|
| **CREATED** | Constellation initialized, no tasks added |
| **READY** | Tasks and dependencies configured, ready to execute |
| **EXECUTING** | At least one task is running or completed |
| **COMPLETED** | All tasks completed successfully |
| **FAILED** | All tasks failed |
| **PARTIALLY_FAILED** | Some tasks succeeded, some failed |
---
## 📊 DAG Metrics
### Parallelism Analysis
The Task Constellation provides several metrics to analyze workflow parallelism:
#### Critical Path Length ($L$)
The longest serial dependency chain in the constellation:
$$
L = \max_{p \in \text{paths}} |p|
$$
where $|p|$ is the length of path $p$ from any root to any leaf node.
#### Total Work ($W$)
Sum of all task execution durations:
$$
W = \sum_{t_i \in \mathcal{T}} \text{duration}(t_i)
$$
#### Parallelism Ratio ($P$)
Measure of achievable parallelism:
$$
P = \frac{W}{L}
$$
- $P = 1$: Completely serial execution
- $P > 1$: Parallel execution possible
- Higher $P$ indicates more parallelism
#### Maximum Width
Maximum number of tasks that can execute concurrently:
$$
\text{MaxWidth} = \max_{\text{level}} |\text{tasks at level}|
$$
!!!info "Calculation Modes"
The constellation supports two calculation modes:
- **Node Count Mode**: Uses task counts when execution is incomplete
- **Actual Time Mode**: Uses real execution durations when all tasks are terminal
---
## 🛠️ Core Operations
### DAG Construction
```python
from galaxy.constellation import TaskConstellation, TaskStar, TaskStarLine
# Create constellation
constellation = TaskConstellation(name="my_workflow")
# Add tasks
task_a = TaskStar(name="task_a", description="Checkout code on laptop")
task_b = TaskStar(name="task_b", description="Build on GPU server")
task_c = TaskStar(name="task_c", description="Deploy to staging")
constellation.add_task(task_a)
constellation.add_task(task_b)
constellation.add_task(task_c)
# Add dependencies
dep_ab = TaskStarLine.create_success_only(
from_task_id=task_a.task_id,
to_task_id=task_b.task_id,
description="Build depends on successful checkout"
)
dep_bc = TaskStarLine.create_unconditional(
from_task_id=task_b.task_id,
to_task_id=task_c.task_id,
description="Deploy after build"
)
constellation.add_dependency(dep_ab)
constellation.add_dependency(dep_bc)
```
### DAG Validation
```python
# Validate structure
is_valid, errors = constellation.validate_dag()
if not is_valid:
print(f"Validation errors: {errors}")
# Check for cycles
has_cycles = constellation.has_cycle()
# Get topological order
order = constellation.get_topological_order()
print(f"Execution order: {order}")
```
### Parallelism Analysis
```python
# Get parallelism metrics
metrics = constellation.get_parallelism_metrics()
print(f"Critical Path Length: {metrics['critical_path_length']}")
print(f"Total Work: {metrics['total_work']}")
print(f"Parallelism Ratio: {metrics['parallelism_ratio']}")
print(f"Critical Path: {metrics['critical_path_tasks']}")
# Get maximum width
max_width = constellation.get_max_width()
print(f"Maximum concurrent tasks: {max_width}")
```
---
## 🔧 Dynamic Modification
### Safe Editing with ConstellationEditor
```python
from galaxy.constellation.editor import ConstellationEditor
# Create editor with undo/redo support
editor = ConstellationEditor(constellation)
# Add a new diagnostic task
diagnostic_task = editor.create_and_add_task(
task_id="diag_1",
description="Check server health",
name="Server Health Check"
)
# Add conditional dependency
editor.create_and_add_dependency(
from_task_id=task_b.task_id,
to_task_id=diagnostic_task.task_id,
dependency_type="CONDITIONAL",
condition_description="Run diagnostic if build fails"
)
# Undo if needed
if something_wrong:
editor.undo()
# Get modifiable components
modifiable_tasks = constellation.get_modifiable_tasks()
modifiable_deps = constellation.get_modifiable_dependencies()
```
!!!warning "Modification Safety"
Tasks and dependencies can only be modified if they are in `PENDING` or `WAITING_DEPENDENCY` status. Running or completed tasks cannot be modified to ensure execution consistency.
---
## 📈 Example Workflows
### Sequential Workflow
```mermaid
graph LR
A[Task A] --> B[Task B]
B --> C[Task C]
```
- **Parallelism Ratio**: 1.0 (completely serial)
- **Maximum Width**: 1
### Parallel Workflow
```mermaid
graph LR
A[Task A] --> B[Task B]
A --> C[Task C]
B --> D[Task D]
C --> D
```
- **Parallelism Ratio**: 2.0 (B and C can run in parallel)
- **Maximum Width**: 2
### Complex Workflow
```mermaid
graph LR
A[Task A] --> B[Task B]
A --> C[Task C]
B --> D[Task D]
C --> E[Task E]
D --> F[Task F]
E --> F
```
- **Parallelism Ratio**: ~1.67
- **Maximum Width**: 3 (B, C, E can run concurrently after A completes)
---
## 🎨 Visualization
The Task Constellation provides multiple visualization modes for monitoring and debugging:
### Overview Mode
High-level constellation structure with task counts and state
### Topology Mode
DAG graph showing task relationships and dependencies
### Details Mode
Detailed task information including execution times and status
### Execution Mode
Real-time execution flow with progress tracking
```python
# Display constellation
constellation.display_dag(mode="overview") # or "topology", "details", "execution"
```
For interactive web-based visualization, check out the [Galaxy WebUI](../webui.md).
---
## 📚 Component Documentation
Explore detailed documentation for each component:
- **[TaskStar](task_star.md)** — Atomic execution units representing individual tasks in the constellation
- **[TaskStarLine](task_star_line.md)** — Dependency relationships connecting tasks with conditional logic
- **[TaskConstellation](task_constellation.md)** — Complete DAG orchestrator managing workflow execution and coordination
- **[ConstellationEditor](constellation_editor.md)** — Interactive editor with command pattern and undo/redo capabilities
### Related Documentation
- **[Constellation Orchestrator](../constellation_orchestrator/overview.md)** — Learn how constellations are scheduled and executed across devices
- **[Constellation Agent](../constellation_agent/overview.md)** — Understand how agents plan and manage constellation lifecycles
- **[Evaluation & Metrics](../evaluation/performance_metrics.md)** — Monitor constellation performance and analyze execution patterns
---
## 🔬 Research Background
The Task Constellation model is grounded in formal DAG theory and distributed systems research. Key properties include:
- **Acyclicity guarantees** through Kahn's algorithm for topological sorting
- **Topological ordering** for consistent execution
- **Critical path analysis** for performance optimization
- **Dynamic graph evolution** without compromising consistency
For more on Galaxy's architecture and design principles, see the [Galaxy Overview](../overview.md).
---
## 💡 Best Practices
!!!tip "Designing Effective Constellations"
1. **Keep tasks atomic**: Each TaskStar should represent a single, well-defined operation
2. **Minimize dependencies**: Reduce unnecessary dependencies to maximize parallelism
3. **Use appropriate dependency types**: Choose conditional dependencies for error handling
4. **Validate early**: Run `validate_dag()` before execution
5. **Monitor metrics**: Track parallelism ratio to optimize workflow design
**Common Patterns:**
- **Fan-out**: One task spawns multiple independent parallel tasks
- **Fan-in**: Multiple parallel tasks converge to a single task
- **Pipeline**: Sequential stages with parallel tasks within each stage
- **Conditional branching**: Use conditional dependencies for error handling paths
---
## 🚀 Next Steps
- Learn about **[TaskStar](task_star.md)** — Atomic task execution units
- Explore **[TaskStarLine](task_star_line.md)** — Dependency relationships
- Master **[TaskConstellation](task_constellation.md)** — DAG orchestration
- Try **[ConstellationEditor](constellation_editor.md)** — Interactive editing
---
### Documents Docs Galaxy Constellation Agent Overview (`documents/docs/galaxy/constellation_agent/overview.md`)
# Constellation Agent — The Centralized Constellation Weaver
The **Constellation Agent** serves as the central intelligence of UFO³ Galaxy, acting as both a planner and replanner. It interprets user intent, constructs executable Task Constellations, and dynamically steers their evolution across heterogeneous devices. By bridging high-level natural-language goals and concrete multi-agent execution, the Constellation Agent provides unified orchestration through a feedback-driven control loop.
For an overview of the Galaxy system architecture, see [Galaxy Overview](../overview.md).
## 🌟 Introduction
**Figure:** An overview of the Constellation Agent showing the dual-mode control cycle between creation and editing phases.
The Constellation Agent extends the abstract [Task Constellation](../constellation/overview.md) model into runtime execution. Residing within the **ConstellationClient** (see [Galaxy Client](../client/overview.md)), it transforms user requests into structured DAG workflows and continuously refines them as distributed agents provide feedback.
Unlike traditional static DAG schedulers, the Constellation Agent operates as a **dynamic orchestrator** powered by an LLM-driven architecture and governed by a finite-state machine (FSM). This design enables it to alternate between two complementary operating modes:
- **Creation Mode**: Synthesizes initial Task Constellations from user instructions
- **Editing Mode**: Incrementally refines constellations based on runtime feedback
This feedback-driven control loop achieves tight coupling between symbolic reasoning and distributed execution, maintaining global consistency while adapting to changing device conditions.
## 🎯 Core Responsibilities
The Constellation Agent orchestrates distributed workflows through structured feedback loops, alternating between creation and editing phases with explicit operational boundaries. For details on task execution, see [Constellation Orchestrator](../constellation_orchestrator/overview.md).
### Primary Functions
| Function | Description | Mode |
|----------|-------------|------|
| **Request Interpretation** | Parse user goals and context into actionable requirements | Creation |
| **DAG Synthesis** | Decompose requests into structured Task Constellations with dependencies | Creation |
| **Device Assignment** | Map tasks to appropriate devices based on AgentProfile capabilities | Creation |
| **Runtime Monitoring** | Track task completion events and constellation state | Editing |
| **Dynamic Adaptation** | Add, remove, or modify tasks/dependencies based on feedback | Editing |
| **Consistency Maintenance** | Ensure DAG validity and execution correctness throughout lifecycle | Both |
## 🏗️ Architecture
### Dual-Mode Control System
The Constellation Agent implements a **dual-mode control pattern** that separates planning from replanning:
```mermaid
graph LR
A[User Request] --> B[Creation Mode]
B --> C[Initial Constellation]
C --> D[Orchestrator]
D --> E[Task Execution]
E --> F{Event Queue}
F -->|Task Completed| G[Editing Mode]
G --> H[Updated Constellation]
H --> D
F -->|All Complete| I[Finish]
style B fill:#e1f5ff
style G fill:#fff4e1
style I fill:#e8f5e9
```
### Component Integration
```mermaid
graph TB
subgraph "Constellation Agent"
FSM[Finite State Machine]
Prompter[Prompter]
Processor[Agent Processor]
end
subgraph "MCP Layer"
Dispatcher[Command Dispatcher]
MCP[MCP Server Manager]
Editor[Constellation Editor MCP]
end
subgraph "Execution Layer"
Orchestrator[Task Orchestrator]
EventBus[Event Bus]
end
FSM --> Prompter
Prompter --> Processor
Processor --> Dispatcher
Dispatcher --> MCP
MCP --> Editor
Editor --> Orchestrator
Orchestrator --> EventBus
EventBus -->|Task Events| FSM
style FSM fill:#e1f5ff
style MCP fill:#fff4e1
style Orchestrator fill:#e8f5e9
```
## 🔄 Creation Mode
In creation mode, the Constellation Agent receives a user request and generates the initial Task Constellation.
### Inputs
| Input | Type | Description |
|-------|------|-------------|
| **User Request** | `str` | Natural language goal or structured command |
| **AgentProfile Registry** | `Dict[str, AgentProfile]` | Available device agents with capabilities and metadata |
| **Demonstration Examples** | `List[Example]` | In-context learning examples for task decomposition |
### Processing Flow
```mermaid
sequenceDiagram
participant User
participant Agent as Constellation Agent
participant Prompter
participant LLM
participant Dispatcher as Command Dispatcher
participant MCP as MCP Server Manager
participant Editor as Constellation Editor MCP
participant Orchestrator
User->>Agent: Submit Request
Agent->>Prompter: Format Creation Prompt
Prompter->>LLM: Send Prompt + Examples
LLM->>Agent: Return Constellation JSON
Agent->>Dispatcher: Execute build_constellation
Dispatcher->>MCP: Route Command
MCP->>Editor: Call build_constellation
Editor->>MCP: Return Built Constellation
MCP->>Dispatcher: Return Result
Dispatcher->>Agent: Constellation Ready
Agent->>Orchestrator: Start Execution
Orchestrator-->>Agent: Constellation Started
Agent->>User: Display Initial Plan
```
### Outputs
| Output | Type | Description |
|--------|------|-------------|
| **Task Constellation** | `TaskConstellation` | Structured DAG with tasks and dependencies |
| **Observation** | `str` | Analysis of input context and device profiles |
| **Thought** | `str` | Reasoning trace explaining decomposition logic |
| **State** | `ConstellationAgentStatus` | Next FSM state (typically `CONTINUE`) |
| **Result** | `Any` | Summary for user or error message |
**Example: Creation Mode Response**
**User Request:** "Download dataset on laptop, preprocess on server, train model on GPU"
**Generated Constellation:**
- Task 1: `fetch_data` → Device: laptop
- Task 2: `preprocess` → Device: linux_server (depends on Task 1)
- Task 3: `train_model` → Device: gpu_server (depends on Task 2)
**Thought:** "Decomposed into 3 sequential tasks based on computational requirements. Laptop handles download, server preprocesses data, GPU server trains model."
## ✏️ Editing Mode
During execution, the Constellation Agent enters editing mode to process task completion events and adapt the constellation.
### Inputs
| Input | Type | Description |
|-------|------|-------------|
| **Original Request** | `str` | The initial user request for context |
| **AgentProfile Registry** | `Dict[str, AgentProfile]` | Current device availability |
| **Current Constellation** | `TaskConstellation` | Serialized constellation snapshot |
| **Task Events** | `List[TaskEvent]` | Completion/failure events from orchestrator |
| **Demonstration Examples** | `List[Example]` | In-context learning examples for editing |
### Processing Flow
```mermaid
sequenceDiagram
participant Orchestrator
participant EventBus
participant Agent as Constellation Agent
participant Prompter
participant LLM
participant Dispatcher as Command Dispatcher
participant MCP as MCP Server Manager
participant Editor as Constellation Editor MCP
Orchestrator->>EventBus: Task Completed Event
EventBus->>Agent: Queue Event
Agent->>Agent: Collect Pending Events
Agent->>Dispatcher: Sync Constellation State
Dispatcher->>MCP: build_constellation (sync)
MCP->>Editor: Update State
Agent->>Prompter: Format Editing Prompt
Prompter->>LLM: Send Current State + Events
LLM->>Agent: Return Modification Actions
Agent->>Dispatcher: Execute Modification Commands
Dispatcher->>MCP: Route Commands
MCP->>Editor: Apply Modifications
Editor->>MCP: Return Updated Constellation
MCP->>Dispatcher: Return Results
Dispatcher->>Agent: Constellation Updated
Agent->>EventBus: Publish Modified Event
Agent->>Orchestrator: Continue Execution
```
### Editing Operations
The agent can perform the following modifications through the MCP-based Constellation Editor:
| Operation | Use Case | Example |
|-----------|----------|---------|
| **Add Task** | Introduce follow-up or diagnostic tasks | Add health check after training fails |
| **Remove Task** | Prune redundant or obsolete tasks | Remove preprocessing if data is pre-processed |
| **Update Task** | Modify description, device, or tips | Switch training to different GPU |
| **Add Dependency** | Establish new task relationships | Make validation depend on training |
| **Remove Dependency** | Decouple independent tasks | Remove unnecessary sequential constraint |
| **Update Dependency** | Change conditional logic | Update success criteria for task trigger |
> **Note:** Only tasks in `PENDING` or `WAITING_DEPENDENCY` status can be modified. Running or completed tasks are **read-only** to ensure execution consistency.
### Outputs
| Output | Type | Description |
|--------|------|-------------|
| **Updated Constellation** | `TaskConstellation` | Modified DAG with new tasks/dependencies |
| **Thought** | `str` | Reasoning explaining modifications or no-op |
| **State** | `ConstellationAgentStatus` | Next FSM state (`CONTINUE`, `FINISH`, or `FAIL`) |
| **Result** | `Any` | Summary of changes or completion status |
## 🔁 Finite-State Machine Lifecycle
**Figure:** Lifecycle state transitions of the Constellation Agent FSM.
The Constellation Agent's behavior is governed by a **4-state finite-state machine**:
| State | Description | Triggers |
|-------|-------------|----------|
| **START** | Initialize constellation, begin orchestration | Agent instantiation, restart after completion |
| **CONTINUE** | Monitor events, process feedback, update constellation | Task completion/failure events |
| **FINISH** | Successful termination, aggregate results | All tasks completed successfully |
| **FAIL** | Terminal error state, abort execution | Irrecoverable errors, validation failures |
### State Transition Rules
```mermaid
stateDiagram-v2
[*] --> START: Initialize Agent
START --> CONTINUE: Constellation Created
START --> FAIL: Creation Failed
CONTINUE --> CONTINUE: Process Events
CONTINUE --> FINISH: All Tasks Complete
CONTINUE --> FAIL: Critical Error
CONTINUE --> START: New Constellation Needed
FINISH --> [*]
FAIL --> [*]
note right of START
Creation Mode:
- Generate initial constellation
- Validate DAG structure
- Start orchestration
end note
note right of CONTINUE
Editing Mode:
- Wait for task events
- Process completion feedback
- Apply modifications
end note
```
For detailed state machine documentation, see [State Machine Details](state.md).
## 🛠️ MCP-Based Constellation Editor
The Constellation Agent interacts with the **Constellation Editor** through the **Model Context Protocol (MCP)** layer. The architecture uses:
- **MCP Server Manager**: Routes commands to appropriate MCP servers
- **Command Dispatcher**: Provides a unified interface for executing MCP commands
- **Constellation Editor MCP Server**: Implements the actual constellation manipulation operations
This MCP-based architecture provides:
- **Protocol Standardization**: Consistent interface across all agent types
- **Loose Coupling**: Agent logic decoupled from editor implementation
- **Extensibility**: Easy to add new operations or alternative editors
- **Tool Discovery**: Dynamic tool listing via `list_tools` command
### Core MCP Operations
The Constellation Editor MCP Server exposes the following operations:
| Operation | Purpose | Inputs | Output |
|------|---------|--------|--------|
| `build_constellation` | Batch-create constellation from config | Configuration dict, clear flag | Built constellation |
| `add_task` | Add atomic task node | Task ID, name, description, device, tips | Updated constellation |
| `remove_task` | Remove task and dependencies | Task ID | Updated constellation |
| `update_task` | Modify task fields | Task ID + updated fields | Updated constellation |
| `add_dependency` | Create dependency edge | From/to task IDs, type, condition | Updated constellation |
| `remove_dependency` | Delete dependency | Dependency ID | Updated constellation |
| `update_dependency` | Update dependency logic | Dependency ID, condition | Updated constellation |
All operations are:
- **Idempotent**: Safe to retry without side effects
- **Atomic**: Single operation per command
- **Validated**: Ensures DAG consistency after each modification
- **Auditable**: All changes are logged and traceable
For complete MCP command specifications and examples, see [Command Reference](command.md). For details on the underlying Task Constellation structure, see [Task Constellation Overview](../constellation/overview.md).
## 📋 Processing Pipeline
The Constellation Agent follows a **4-phase processing pipeline** for both creation and editing modes:
### Phase 1: Context Provision
```python
# Load available MCP tools from Constellation Editor
await agent.context_provision(context=context)
# Queries MCP server for available operations via list_tools
# Formats tools into LLM-compatible prompt
```
### Phase 2: LLM Interaction
```python
# Construct prompt based on mode
prompt = agent.message_constructor(
request=user_request,
device_info=agent_profiles,
constellation=current_constellation
)
# Get LLM response
response = await llm.query(prompt)
# Returns: ConstellationAgentResponse with thought, status, actions
```
### Phase 3: Action Execution
```python
# Execute MCP commands via Command Dispatcher
for command in response.actions:
result = await context.command_dispatcher.execute_commands([command])
# Validate constellation
is_valid, errors = constellation.validate_dag()
```
### Phase 4: Memory Update
```python
# Update global context
context.set(ContextNames.CONSTELLATION, updated_constellation)
context.set(ContextNames.ROUND_RESULT, results)
# Log to memory
memory.add_round_log(
step=step,
weaving_mode=mode,
request=request,
constellation=constellation,
response=response
)
```
## 🎭 Prompter Architecture
The Constellation Agent uses the **Factory Pattern** to create appropriate prompters for different weaving modes (creation and editing).
### Prompter Hierarchy
```mermaid
classDiagram
class BaseConstellationPrompter {
<>
+format_agent_profile()
+format_constellation()
+user_content_construction()
+system_prompt_construction()
}
class ConstellationCreationPrompter {
+user_prompt_construction()
+examples_prompt_helper()
}
class ConstellationEditingPrompter {
+user_prompt_construction()
+examples_prompt_helper()
}
class ConstellationPrompterFactory {
+create_prompter(mode)
+get_supported_modes()
}
BaseConstellationPrompter <|-- ConstellationCreationPrompter
BaseConstellationPrompter <|-- ConstellationEditingPrompter
ConstellationPrompterFactory --> BaseConstellationPrompter
```
### Factory Pattern Benefits
| Benefit | Description |
|---------|-------------|
| **Mode Isolation** | Creation and editing prompts remain independent |
| **Extensibility** | New modes can be added without modifying existing code |
| **Type Safety** | Compile-time checking for prompter selection |
| **Testability** | Each prompter can be unit tested independently |
For complete prompter architecture documentation, see [Prompter Details](strategy.md).
## 💡 Key Design Benefits
### 1. Unified Reasoning and Control
High-level task synthesis and low-level execution coordination are decoupled yet tightly synchronized through the Task Constellation abstraction. The agent focuses on semantic reasoning while the orchestrator handles distributed execution.
### 2. Dynamic Adaptability
The editable constellation enables:
- **Failure Recovery**: Add diagnostic tasks after failures
- **Resource Reallocation**: Switch tasks to available devices
- **Opportunistic Execution**: Insert new tasks as conditions permit
### 3. End-to-End Observability
Complete lineage tracking of:
- **State Transitions**: FSM state changes logged with timestamps
- **Modifications**: All edits tracked with before/after snapshots
- **Events**: Task completion events queued and processed
- **Reasoning Traces**: LLM thought processes captured in memory
### 4. Safe Modification Guarantees
The FSM + MCP Server architecture ensures:
- **Acyclicity**: DAG validation prevents circular dependencies
- **Consistency**: Only modifiable tasks can be edited
- **Atomicity**: Each MCP operation is atomic and idempotent
- **Auditability**: Full modification history maintained
## 🔍 Example Workflow
### User Request
```
"Download MNIST dataset on laptop, train CNN on GPU server,
evaluate on test server, deploy to production if accuracy > 95%"
```
### Creation Mode Output
```json
{
"thought": "Decomposed into 4 tasks: (1) download on laptop, (2) train on GPU, (3) evaluate on test server, (4) conditional deploy based on accuracy",
"status": "CONTINUE",
"constellation": {
"tasks": [
{"task_id": "task_001", "name": "download_mnist", "device": "laptop"},
{"task_id": "task_002", "name": "train_cnn", "device": "gpu_server"},
{"task_id": "task_003", "name": "evaluate", "device": "test_server"},
{"task_id": "task_004", "name": "deploy", "device": "prod_server"}
],
"dependencies": [
{"from": "task_001", "to": "task_002", "type": "SUCCESS_ONLY"},
{"from": "task_002", "to": "task_003", "type": "SUCCESS_ONLY"},
{"from": "task_003", "to": "task_004", "type": "CONDITIONAL",
"condition": "accuracy > 0.95"}
]
}
}
```
### Editing Mode Event
```
Task task_003 (evaluate) completed with result: {"accuracy": 0.92}
```
### Editing Mode Output
```json
{
"thought": "Evaluation accuracy (92%) did not meet deployment threshold (95%). Adding retraining task with adjusted hyperparameters. Removing original deployment task.",
"status": "CONTINUE",
"actions": [
{"tool": "add_task", "parameters": {
"task_id": "task_005",
"name": "retrain_with_tuning",
"device": "gpu_server",
"description": "Retrain with learning rate decay and data augmentation"
}},
{"tool": "add_dependency", "parameters": {
"from": "task_003", "to": "task_005", "type": "SUCCESS_ONLY"
}},
{"tool": "remove_task", "parameters": {"task_id": "task_004"}}
]
}
```
## 📊 Performance Characteristics
### Creation Complexity
- **Time**: $O(n \cdot m)$ where $n$ is task count, $m$ is LLM inference time
- **Space**: $O(n + e)$ for $n$ tasks and $e$ edges
- **Validation**: $O(n + e)$ for DAG cycle detection (DFS)
### Editing Complexity
- **Event Processing**: $O(k)$ for $k$ queued events (batched)
- **Modification**: $O(1)$ per MCP command (constant time)
- **Re-validation**: $O(n + e)$ for modified constellation
### Scalability
| Metric | Typical | Maximum Tested |
|--------|---------|----------------|
| Tasks per Constellation | 5-20 | 100+ |
| Dependencies per Constellation | 4-30 | 200+ |
| Editing Events per Session | 1-10 | 50+ |
| LLM Response Time | 2-5s | 15s |
## 🔗 Related Components
- **[Task Constellation](../constellation/overview.md)** — Abstract DAG model
- **[TaskStar](../constellation/task_star.md)** — Atomic execution units
- **[TaskStarLine](../constellation/task_star_line.md)** — Dependency relationships
- **[Constellation Orchestrator](../constellation_orchestrator/overview.md)** — Distributed executor
- **[State Machine](state.md)** — FSM lifecycle details
- **[Prompter Details](strategy.md)** — Prompter architecture
- **[Command Reference](command.md)** — Editor operation specifications
## 🎯 Summary
The Constellation Agent serves as the **central weaver** of distributed intelligence in UFO³ Galaxy. Through its dual-mode control loop, finite-state machine governance, and MCP-based constellation manipulation, it transforms abstract user goals into live, evolving constellations—maintaining both rigor and adaptability across the complete lifecycle of multi-device orchestration.
**Key Capabilities:**
- **Semantic Decomposition**: Natural language → structured DAG
- **Dynamic Adaptation**: Runtime graph evolution based on feedback
- **MCP Integration**: Protocol-based tool invocation for extensibility
- **Formal Guarantees**: DAG validity + safe concurrent modification
- **Complete Observability**: Full lineage tracking and reasoning traces
- **Modular Design**: Clean separation between reasoning and execution
---
### Documents Docs Galaxy Constellation Orchestrator Overview (`documents/docs/galaxy/constellation_orchestrator/overview.md`)
# Constellation Orchestrator Overview
## Introduction
The **Constellation Orchestrator** is the execution engine at the heart of UFO's multi-device orchestration system. While the Constellation Agent handles reasoning and task graph evolution, the orchestrator transforms these declarative plans into concrete execution across heterogeneous devices.
Unlike traditional DAG schedulers that execute static task graphs, the Constellation Orchestrator operates as a **living execution fabric** where tasks evolve concurrently, react to runtime signals, and adapt to new decisions from the reasoning agent in real-time.
*The Constellation Orchestrator bridges TaskConstellation and execution, enabling asynchronous, adaptive task orchestration across devices.*
## Key Capabilities
The orchestrator achieves three critical goals that traditional schedulers struggle to balance:
| Capability | Description | Benefit |
|------------|-------------|---------|
| **Asynchronous Parallelism** | Execute independent tasks concurrently across heterogeneous devices | Maximize device utilization and minimize idle time |
| **Safety & Consistency** | Maintain correctness under concurrent DAG updates from LLM reasoning | Prevent race conditions and invalid execution states |
| **Runtime Adaptivity** | React to feedback from both devices and LLM reasoning dynamically | Enable intelligent re-planning and error recovery |
## Architecture Overview
The Constellation Orchestrator is built on five fundamental design pillars:
```mermaid
graph TB
A[Event-Driven Coordination] --> B[Orchestrator Core]
C[Asynchronous Scheduling] --> B
D[Safe Assignment Locking] --> B
E[Consistency Enforcement] --> B
F[Batched Editing] --> B
B --> G[Device Execution]
B --> H[Constellation Evolution]
style B fill:#4a90e2,stroke:#333,stroke-width:3px,color:#fff
style A fill:#7cb342,stroke:#333,stroke-width:2px
style C fill:#7cb342,stroke:#333,stroke-width:2px
style D fill:#7cb342,stroke:#333,stroke-width:2px
style E fill:#7cb342,stroke:#333,stroke-width:2px
style F fill:#7cb342,stroke:#333,stroke-width:2px
```
### 1. Event-Driven Coordination
The orchestrator operates as a fully event-driven system using an observer pattern and internal event bus. Instead of polling or global checkpoints, it reacts immediately to four primary event types:
- `TASK_STARTED` - Task assigned and execution begins
- `TASK_COMPLETED` - Task finishes successfully
- `TASK_FAILED` - Task execution fails
- `CONSTELLATION_MODIFIED` - DAG structure updated by agent
This design provides **high responsiveness** and eliminates synchronization overhead.
[Learn more →](event_driven_coordination.md)
### 2. Asynchronous Scheduling
A continuous scheduling loop monitors the evolving TaskConstellation, identifies ready tasks (dependencies satisfied), and dispatches them concurrently to available devices. Critically, **task execution and constellation editing proceed in parallel**, overlapping computation with orchestration.
This enables **maximum parallelism** and **minimal latency** in cross-device workflows.
[Learn more →](asynchronous_scheduling.md)
### 3. Safe Assignment Locking
To prevent race conditions when LLM-driven edits overlap with task execution, the orchestrator employs a safe assignment lock protocol. During edit cycles, new task assignments are suspended while modifications are applied atomically and synchronized with runtime progress.
This guarantees **atomicity** and **prevents conflicts** between execution and modification.
[Learn more →](safe_assignment_locking.md)
### 4. Consistency Enforcement
The orchestrator enforces three runtime invariants to preserve correctness even under partial or invalid LLM updates:
- **I1 (Single Assignment)**: Each task has at most one active device assignment
- **I2 (Acyclic Consistency)**: Edits preserve DAG acyclicity (no cycles)
- **I3 (Valid Update)**: Only PENDING tasks and their dependents can be modified
These invariants ensure **structural integrity** and **semantic validity** of the constellation.
[Learn more →](consistency_guarantees.md)
### 5. Batched Constellation Editing
To balance responsiveness with efficiency, the orchestrator batches multiple task completion events and applies their resulting modifications atomically. This amortizes LLM invocation overhead while preserving atomicity and consistency.
This achieves both **efficiency** and **adaptivity** without excessive micro-edits.
[Learn more →](batched_editing.md)
## System Components
The orchestrator consists of two primary components working in tandem:
### TaskConstellationOrchestrator
The main execution orchestrator focused on flow control and coordination. It manages:
- Event-driven task lifecycle (start, complete, fail)
- Asynchronous scheduling loop
- Safe assignment locking protocol
- Integration with modification synchronizer
[API Reference →](api_reference.md)
### ConstellationManager
Handles device assignment, resource management, and constellation lifecycle. It provides:
- Multiple assignment strategies (round-robin, capability-match, load-balance)
- Device validation and status tracking
- Constellation registration and metadata management
[Learn more →](constellation_manager.md)
## Execution Flow
The orchestration workflow follows this sequence:
```mermaid
sequenceDiagram
participant U as User Request
participant O as Orchestrator
participant C as Constellation
participant S as Synchronizer
participant D as Devices
participant A as Agent
U->>O: orchestrate_constellation()
O->>C: Validate DAG
O->>C: Assign devices
loop Execution Loop
O->>C: Get ready tasks
O->>D: Dispatch tasks (async)
D-->>O: Task completed event
O->>S: Wait for modifications
S->>A: Trigger editing
A->>C: Update constellation
A-->>S: Modification complete
S->>O: Sync constellation state
end
O->>U: Return results
```
The orchestrator treats task execution as an **open-world process** - continuously evolving, reacting, and converging toward user intent rather than executing a fixed plan.
## Design Highlights
### Asynchronous by Default
Every operation runs asynchronously using Python's `asyncio`, enabling:
- Concurrent task execution across devices
- Non-blocking event handling
- Parallel constellation editing
### LLM-Aware Orchestration
Unlike traditional schedulers, the orchestrator is designed for **reasoning-aware execution**:
- Expects and handles dynamic graph modifications
- Synchronizes LLM reasoning with runtime execution
- Validates and enforces safety under AI-driven changes
### Production-Ready Safeguards
- Timeout protection for modifications (default: 600s)
- Automatic validation before every execution cycle
- Device assignment verification
- Cycle detection on every edit
- Comprehensive error handling and logging
## Performance Characteristics
| Metric | Description | Implementation |
|--------|-------------|----------------|
| **Latency** | Time from task ready to execution start | Minimized via event-driven dispatch |
| **Throughput** | Tasks completed per unit time | Maximized via async parallelism |
| **Overhead** | Orchestration cost per task | Reduced via batched editing |
| **Scalability** | Performance with increasing tasks/devices | Linear with async coordination |
## Getting Started
### Basic Usage
```python
from galaxy.constellation import TaskConstellationOrchestrator
from galaxy.client.device_manager import ConstellationDeviceManager
# Create orchestrator
device_manager = ConstellationDeviceManager()
orchestrator = TaskConstellationOrchestrator(device_manager)
# Orchestrate constellation
results = await orchestrator.orchestrate_constellation(
constellation=my_constellation,
assignment_strategy="capability_match"
)
```
### With Modification Synchronizer
```python
from galaxy.session.observers.constellation_sync_observer import (
ConstellationModificationSynchronizer
)
# Create synchronizer
synchronizer = ConstellationModificationSynchronizer(orchestrator)
orchestrator.set_modification_synchronizer(synchronizer)
# Subscribe to events
event_bus.subscribe(synchronizer)
# Now orchestrator will wait for LLM edits to complete
```
## Related Documentation
- **[Event-Driven Coordination](event_driven_coordination.md)** - Event system and observer pattern
- **[Asynchronous Scheduling](asynchronous_scheduling.md)** - Concurrent task execution
- **[Safe Assignment Locking](safe_assignment_locking.md)** - Race condition prevention
- **[Consistency Guarantees](consistency_guarantees.md)** - Runtime invariants
- **[Batched Editing](batched_editing.md)** - Efficient constellation updates
- **[Constellation Manager](constellation_manager.md)** - Device and resource management
- **[API Reference](api_reference.md)** - Complete API documentation
## Further Reading
- [TaskConstellation Documentation](../constellation/overview.md) - Understand the DAG structure
- [Constellation Agent](../constellation_agent/overview.md) - LLM-based reasoning
- [Device Manager](../client/device_manager.md) - Device communication layer
---
!!!tip "Next Steps"
To understand how events drive orchestration, continue to [Event-Driven Coordination](event_driven_coordination.md).
---
### Documents Docs Galaxy Observer Overview (`documents/docs/galaxy/observer/overview.md`)
# Observer System — Overview
The **Observer System** in UFO Galaxy implements an event-driven architecture that enables real-time monitoring, visualization, and coordination of constellation execution. It provides a decoupled, extensible mechanism for components to react to system events without tight coupling.
The system implements the classic **Observer Pattern** (also known as Publish-Subscribe), enabling loose coupling between event producers and consumers. This allows the system to be extended with new observers without modifying existing code.
---
## 🎯 Purpose and Design Goals
The observer system serves several critical functions in the Galaxy framework:
1. **Real-time Monitoring** — Track task execution, constellation lifecycle, and system events
2. **Visualization** — Provide live updates for DAG topology and execution progress
3. **Metrics Collection** — Gather performance statistics and execution data
4. **Synchronization** — Coordinate between agent modifications and orchestrator execution
5. **Agent Output Handling** — Display agent responses and actions in real-time
---
## 🏗️ Architecture Overview
The observer system consists of three main layers:
```mermaid
graph TB
subgraph "Event Publishers"
A1[Orchestrator]
A2[Agent]
A3[Device Manager]
end
subgraph "Event Bus Layer"
B[EventBus Singleton]
end
subgraph "Observer Layer"
C1[ConstellationProgressObserver]
C2[SessionMetricsObserver]
C3[DAGVisualizationObserver]
C4[ConstellationModificationSynchronizer]
C5[AgentOutputObserver]
end
subgraph "Handler Layer"
D1[TaskVisualizationHandler]
D2[ConstellationVisualizationHandler]
end
A1 -->|publish events| B
A2 -->|publish events| B
A3 -->|publish events| B
B -->|notify| C1
B -->|notify| C2
B -->|notify| C3
B -->|notify| C4
B -->|notify| C5
C3 -->|delegate| D1
C3 -->|delegate| D2
style B fill:#4a90e2,stroke:#333,stroke-width:3px,color:#fff
style C1 fill:#66bb6a,stroke:#333,stroke-width:2px
style C2 fill:#66bb6a,stroke:#333,stroke-width:2px
style C3 fill:#66bb6a,stroke:#333,stroke-width:2px
style C4 fill:#ffa726,stroke:#333,stroke-width:2px
style C5 fill:#66bb6a,stroke:#333,stroke-width:2px
```
**Architecture Layers:**
| Layer | Component | Responsibility |
|-------|-----------|----------------|
| **Event Publishers** | Orchestrator, Agent, Device Manager | Generate events during system operation |
| **Event Bus** | `EventBus` singleton | Central message broker, manages subscriptions and routing |
| **Observers** | 5 specialized observers | React to specific event types and perform actions |
| **Handlers** | Task & Constellation handlers | Delegate visualization logic for specific components |
---
## 📊 Core Components
### Event System Core
The foundation of the observer system consists of:
| Component | Location | Description |
|-----------|----------|-------------|
| **EventBus** | `galaxy/core/events.py` | Central message broker managing subscriptions |
| **EventType** | `galaxy/core/events.py` | Enumeration of all system event types |
| **Event Classes** | `galaxy/core/events.py` | Base (`Event`) and specialized (`TaskEvent`, `ConstellationEvent`, `AgentEvent`, `DeviceEvent`) event data structures |
| **Interfaces** | `galaxy/core/events.py` | `IEventObserver`, `IEventPublisher` contracts |
For detailed documentation of the event system core components, see the **[Event System Core](event_system.md)** documentation.
### Observer Implementations
Five specialized observers handle different aspects of system monitoring:
| Observer | File Location | Primary Role | Key Features |
|----------|---------------|--------------|--------------|
| **ConstellationProgressObserver** | `galaxy/session/observers/base_observer.py` | Task progress tracking | Queues completion events for agent, coordinates task lifecycle |
| **SessionMetricsObserver** | `galaxy/session/observers/base_observer.py` | Performance metrics | Collects timing, success rates, modification statistics |
| **DAGVisualizationObserver** | `galaxy/session/observers/dag_visualization_observer.py` | Real-time visualization | Displays constellation topology and execution flow |
| **ConstellationModificationSynchronizer** | `galaxy/session/observers/constellation_sync_observer.py` | Modification coordination | Prevents race conditions between agent and orchestrator |
| **AgentOutputObserver** | `galaxy/session/observers/agent_output_observer.py` | Agent interaction display | Shows agent responses and actions in real-time |
---
## 🔄 Event Flow
The following diagram illustrates how events flow through the system:
```mermaid
sequenceDiagram
participant O as Orchestrator
participant EB as EventBus
participant CPO as ProgressObserver
participant SMO as MetricsObserver
participant DVO as VisualizationObserver
participant A as Agent
O->>EB: publish(TASK_STARTED)
EB->>CPO: on_event(event)
EB->>SMO: on_event(event)
EB->>DVO: on_event(event)
Note over DVO: Display task start
Note over SMO: Increment task count
O->>EB: publish(TASK_COMPLETED)
EB->>CPO: on_event(event)
EB->>SMO: on_event(event)
EB->>DVO: on_event(event)
CPO->>A: add_task_completion_event()
Note over A: Process result, modify constellation
A->>EB: publish(CONSTELLATION_MODIFIED)
EB->>SMO: on_event(event)
EB->>DVO: on_event(event)
Note over DVO: Display updated DAG
Note over SMO: Track modification
```
The event flow demonstrates how a single action (task completion) triggers multiple observers, each performing their specialized function without interfering with others.
---
## 📋 Event Types
The system defines the following event types:
### Task Events
Track individual task execution lifecycle:
| Event Type | Trigger | Data Includes |
|------------|---------|---------------|
| `TASK_STARTED` | Task begins execution | task_id, status, constellation_id |
| `TASK_COMPLETED` | Task finishes successfully | task_id, result, execution_time, newly_ready_tasks |
| `TASK_FAILED` | Task encounters error | task_id, error, retry_info |
### Constellation Events
Monitor constellation-level operations:
| Event Type | Trigger | Data Includes |
|------------|---------|---------------|
| `CONSTELLATION_STARTED` | Constellation begins processing | constellation, initial_statistics, processing_time |
| `CONSTELLATION_COMPLETED` | All tasks finished | constellation, final_statistics, execution_time |
| `CONSTELLATION_FAILED` | Constellation encounters error | constellation, error |
| `CONSTELLATION_MODIFIED` | Structure changed by agent | old_constellation, new_constellation, on_task_id, modification_type, changes |
### Agent Events
Display agent interactions:
| Event Type | Trigger | Data Includes |
|------------|---------|---------------|
| `AGENT_RESPONSE` | Agent generates response | agent_name, agent_type, response_data |
| `AGENT_ACTION` | Agent executes action | agent_name, action_type, actions |
### Device Events
Monitor device status (used by client):
| Event Type | Trigger | Data Includes |
|------------|---------|---------------|
| `DEVICE_CONNECTED` | Device joins pool | device_id, device_status, device_info |
| `DEVICE_DISCONNECTED` | Device leaves pool | device_id, device_status |
| `DEVICE_STATUS_CHANGED` | Device state changes | device_id, device_status, all_devices |
---
## 🚀 Usage Example
Here's a complete example showing how observers are initialized and used in a Galaxy session:
```
/* Detailed source-code truncated for AI context efficiency. */
```
---
## 🔑 Key Benefits
### 1. Decoupling
Events decouple components — publishers don't need to know about observers:
- **Orchestrator** publishes task events without knowing who's listening
- **Agent** publishes modification events without coordinating with orchestrator
- **New observers** can be added without changing existing code
### 2. Extensibility
Add custom observers for new functionality:
```python
from galaxy.core.events import IEventObserver, Event, EventType
class CustomMetricsObserver(IEventObserver):
"""Custom observer for domain-specific metrics."""
def __init__(self):
self.custom_metrics = {}
async def on_event(self, event: Event) -> None:
if event.event_type == EventType.TASK_COMPLETED:
# Collect custom metrics
task_type = event.data.get("task_type")
if task_type not in self.custom_metrics:
self.custom_metrics[task_type] = []
self.custom_metrics[task_type].append({
"duration": event.data.get("execution_time"),
"result": event.result
})
# Subscribe to specific events
event_bus = get_event_bus()
custom_observer = CustomMetricsObserver()
event_bus.subscribe(custom_observer, {EventType.TASK_COMPLETED})
```
### 3. Concurrent Execution
All observers are notified concurrently using `asyncio.gather()`:
- No observer blocks another
- Exceptions in one observer don't affect others
- Efficient parallel processing
### 4. Type-Safe Event Handling
Specialized event classes provide type safety:
```python
async def on_event(self, event: Event) -> None:
if isinstance(event, TaskEvent):
# TaskEvent-specific handling
task_id = event.task_id # Type-safe access
status = event.status
elif isinstance(event, ConstellationEvent):
# ConstellationEvent-specific handling
constellation_id = event.constellation_id
state = event.constellation_state
```
---
## 📚 Component Documentation
Explore detailed documentation for each observer:
- **[Session Metrics Observer](metrics_observer.md)** — Performance metrics and statistics collection
- **[Event System Core](event_system.md)** — Event bus, event types, and interfaces
!!! note "Additional Observers"
Documentation for `ConstellationProgressObserver`, `DAGVisualizationObserver`, `ConstellationModificationSynchronizer`, and `AgentOutputObserver` is available in their source code files. These observers handle task progress tracking, real-time visualization, modification synchronization, and agent output display respectively.
---
## 🔗 Related Documentation
- **[Constellation Orchestrator](../constellation_orchestrator/overview.md)** — Event publishers for task execution
- **[Constellation Agent](../constellation_agent/overview.md)** — Event publishers for agent operations
- **[Performance Metrics](../evaluation/performance_metrics.md)** — How metrics are collected and analyzed
- **[Event-Driven Coordination](../constellation_orchestrator/event_driven_coordination.md)** — Deep dive into event system architecture
---
## 💡 Best Practices
### Observer Lifecycle Management
Properly manage observer subscriptions to prevent memory leaks:
```python
# Subscribe observers
observers = [progress_observer, metrics_observer, viz_observer]
for observer in observers:
event_bus.subscribe(observer)
try:
# Execute constellation
await orchestrator.execute_constellation(constellation)
finally:
# Clean up observers
for observer in observers:
event_bus.unsubscribe(observer)
```
### Event-Specific Subscription
Subscribe only to relevant events for efficiency:
```python
# Instead of subscribing to all events
event_bus.subscribe(observer) # Receives ALL events
# Subscribe to specific event types
event_bus.subscribe(observer, {
EventType.TASK_COMPLETED,
EventType.TASK_FAILED,
EventType.CONSTELLATION_MODIFIED
})
```
### Error Handling in Observers
Always handle exceptions gracefully:
```python
async def on_event(self, event: Event) -> None:
try:
# Process event
await self._handle_event(event)
except Exception as e:
self.logger.error(f"Error processing event: {e}")
# Don't re-raise - let other observers continue
```
---
## 🎓 Summary
The Observer System provides a robust, event-driven foundation for monitoring and coordinating Galaxy's constellation execution:
- **Event Bus** acts as central message broker
- **5 specialized observers** handle different aspects of monitoring
- **Loose coupling** enables extensibility and maintainability
- **Concurrent execution** ensures efficient event processing
- **Type-safe events** provide clear contracts and error prevention
For implementation details of specific observers, refer to the individual component documentation pages linked above.
---
### Documents Docs Infrastructure Agents Overview (`documents/docs/infrastructure/agents/overview.md`)
# Device Agent Architecture
Device Agents are the execution engines of UFO3's multi-device orchestration system. Each device agent operates as an autonomous, intelligent controller that translates high-level user intentions into low-level system commands. The architecture is designed for **extensibility**, **safety**, and **scalability** across heterogeneous computing environments.
## Overview
UFO3 orchestrates tasks across multiple devices through a network of **Device Agents**. Originally designed as a Windows automation framework (UFO2), the architecture has evolved to support diverse platforms including Linux, macOS, and embedded systems. This document describes the abstract design principles and interfaces that enable this multi-platform capability.
**Key Capabilities:**
- **Multi-Platform**: Windows agents (HostAgent, AppAgent), Linux agent, extensible to macOS and embedded systems
- **Safe Execution**: Server-client separation isolates reasoning from system-level operations
- **Scalable Architecture**: Hierarchical agent coordination supports complex cross-device workflows
- **LLM-Driven Reasoning**: Dynamic decision-making using large language models
- **Modular Design**: Three-layer architecture (State, Strategy, Command) enables customization
---
## Three-Layer Architecture
Device agents implement a **three-layer framework** that separates concerns, promotes modularity, and enables extensibility:
```mermaid
graph TB
subgraph "Device Agent Architecture"
subgraph "Level-1: State Layer (FSM)"
S1[AgentState]
S2[State Machine]
S3[State Transitions]
S1 --> S2 --> S3
end
subgraph "Level-2: Strategy Layer (Execution Logic)"
P1[ProcessorTemplate Strategy Orchestrator]
P2[DATA_COLLECTION Strategies]
P3[LLM_INTERACTION Strategies]
P4[ACTION_EXECUTION Strategies]
P5[MEMORY_UPDATE Strategies]
P1 -->|manages & executes| P2
P2 --> P3 --> P4 --> P5
end
subgraph "Level-3: Command Layer (System Interface)"
C1[CommandDispatcher]
C2[MCP Tools]
C3[Atomic Commands]
C1 --> C2 --> C3
end
S3 -->|delegates to| P1
P5 -->|executes via| C1
end
LLM[Large Language Model]
P3 -.->|reasoning| LLM
LLM -.->|decisions| P4
```
### Layer Responsibilities
| Layer | Level | Responsibility | Key Components | Extensibility |
|-------|-------|----------------|----------------|---------------|
| **State** | Level-1 | Finite State Machine governing agent lifecycle | `AgentState`, `AgentStateManager`, `AgentStatus` | Register new states via `@AgentStateManager.register` |
| **Strategy** | Level-2 | Execution logic layer: processor manages sequence of modular strategies | `ProcessorTemplate`, `ProcessingStrategy`, `ProcessingPhase`, `Middleware` | Compose custom strategies via `ComposedStrategy`, add middleware |
| **Command** | Level-3 | Atomic system operations mapped to MCP tools | `BasicCommandDispatcher`, `Command`, MCP integration | Add new tools via client-side MCP server registration |
**Design Rationale:**
The three-layer separation ensures:
- **State Layer (Level-1)**: Controls *when* and *what* to execute (state transitions, agent handoff)
- **Strategy Layer (Level-2)**: Defines *how* to execute (processor orchestrates modular strategies)
- **Command Layer (Level-3)**: Performs *actual* execution (deterministic system operations)
This separation allows replacing individual layers without affecting others.
---
## Level-1: State Layer (FSM)
The **State Layer** implements a Finite State Machine (FSM) that governs the agent's execution lifecycle. Each state encapsulates:
- A **processor** (strategy execution logic)
- **Transition rules** (to next state)
- **Agent handoff logic** (for multi-agent workflows)
```mermaid
stateDiagram-v2
[*] --> CONTINUE
CONTINUE --> CONTINUE: Success
CONTINUE --> PENDING: Wait for external event
CONTINUE --> CONFIRM: User confirmation needed
CONTINUE --> SCREENSHOT: Capture observation
CONTINUE --> FINISH: Task complete
CONTINUE --> FAIL: Error occurred
CONTINUE --> ERROR: Critical failure
PENDING --> CONTINUE: Event received
CONFIRM --> CONTINUE: User confirmed
CONFIRM --> FAIL: User rejected
SCREENSHOT --> CONTINUE: Screenshot captured
FINISH --> [*]
FAIL --> [*]
ERROR --> [*]
```
### AgentStatus Enum
```python
class AgentStatus(Enum):
"""Agent status enumeration"""
ERROR = "ERROR" # Critical error occurred
FINISH = "FINISH" # Task completed successfully
CONTINUE = "CONTINUE" # Normal execution
FAIL = "FAIL" # Task failed
PENDING = "PENDING" # Waiting for external event
CONFIRM = "CONFIRM" # Awaiting user confirmation
SCREENSHOT = "SCREENSHOT" # Screenshot capture needed
```
**State Registration:**
New states can be registered dynamically using the `@AgentStateManager.register` decorator:
```python
@AgentStateManager.register
class CustomState(AgentState):
async def handle(self, agent, context):
# Custom state logic
pass
def next_state(self, agent):
return AgentStateManager.get_state("CONTINUE")
```
**See [State Layer Documentation](design/state.md) for complete details.**
---
## Level-2: Strategy Layer (Execution Logic)
The **Strategy Layer** implements the execution logic within each state. Each state encapsulates a **processor** that manages a sequence of **strategies** to implement step-level workflow. This layer consists of two key components:
### Processor: Strategy Orchestrator
The **ProcessorTemplate** orchestrates the execution of strategies:
- **Registers Strategies**: Configures which strategies execute in each phase
- **Manages Middleware**: Wraps strategy execution with logging, metrics, error handling
- **Validates Dependencies**: Ensures strategies have required data before execution
- **Controls Execution**: Sequences strategies through fixed workflow phases
```mermaid
graph TB
State[AgentState] -->|encapsulates| Processor[ProcessorTemplate Strategy Orchestrator]
Processor -->|1. Register| Strategies[ProcessingStrategies]
Processor -->|2. Wrap| Middleware[Middleware Chain]
Processor -->|3. Validate| Dependencies[Strategy Dependencies]
Processor -->|4. Execute| Workflow[Workflow Phases]
Workflow --> DC[DATA_COLLECTION]
Workflow --> LLM[LLM_INTERACTION]
Workflow --> AE[ACTION_EXECUTION]
Workflow --> MU[MEMORY_UPDATE]
DC & LLM & AE & MU -.->|implements| Strategies
```
**Processor and Strategy Relationship:**
- **Processor**: Framework that manages the sequence of strategies
- **Strategy**: Modular, reusable execution units
Together they form **Level-2: Strategy Layer**, which handles:
- Data collection and environment inspection
- Prompt construction and LLM reasoning
- Action planning and tool invocation
- Memory updates and context synchronization
### Strategy: Modular Execution Units
**ProcessingStrategies** are modular execution units with a unified `execute()` interface:
```mermaid
graph LR
A[DATA_COLLECTION] --> B[LLM_INTERACTION]
B --> C[ACTION_EXECUTION]
C --> D[MEMORY_UPDATE]
A1[Screenshots UI Info System Status] --> A
B1[Prompt Construction LLM Call Response Parsing] --> B
C1[Command Dispatch MCP Execution Result Handling] --> C
D1[Memory Items Blackboard Update Context Sync] --> D
```
### Four Core Strategy Types
| Strategy Type | ProcessingPhase | Purpose | Examples |
|---------------|-----------------|---------|----------|
| **DATA_COLLECTION** | `data_collection` | Gather contextual information | Screenshot capture, UI tree extraction, system info |
| **LLM_INTERACTION** | `llm_interaction` | Construct prompts, interact with LLM, parse responses | Prompt building, LLM reasoning, JSON parsing |
| **ACTION_EXECUTION** | `action_execution` | Execute commands from LLM/toolkits | Click, type, scroll, API calls |
| **MEMORY_UPDATE** | `memory_update` | Update short-term/long-term memory | Add memory items, update blackboard, sync context |
**Strategy Layer Configuration Example:**
Each state configures its processor with strategies and middleware:
```python
class AppAgentProcessor(ProcessorTemplate):
def _setup_strategies(self):
# Register strategies for each phase
self.strategies[ProcessingPhase.DATA_COLLECTION] = ComposedStrategy([
AppScreenshotCaptureStrategy(),
AppControlInfoStrategy()
])
self.strategies[ProcessingPhase.LLM_INTERACTION] = AppLLMInteractionStrategy()
self.strategies[ProcessingPhase.ACTION_EXECUTION] = AppActionExecutionStrategy()
self.strategies[ProcessingPhase.MEMORY_UPDATE] = AppMemoryUpdateStrategy()
def _setup_middleware(self):
# Add middleware for logging, metrics, error handling
self.middleware_chain = [
LoggingMiddleware(),
PerformanceMetricsMiddleware(),
ErrorHandlingMiddleware()
]
```
**See [Processor Documentation](design/processor.md) and [Strategy Documentation](design/strategy.md) for complete details.**
---
## Level-3: Command Layer (System Interface)
The **Command Layer** provides atomic, deterministic system operations. Each command maps to an **MCP tool** that executes on the device client.
```mermaid
sequenceDiagram
participant Agent as Device Agent (Server)
participant Dispatcher as CommandDispatcher
participant Protocol as AIP Protocol
participant Client as Device Client
participant MCP as MCP Tool
Agent->>Dispatcher: execute_commands([command1, command2])
Dispatcher->>Protocol: Send ServerMessage (COMMAND)
Protocol->>Client: WebSocket (AIP)
Client->>MCP: Route to MCP server
MCP->>MCP: Execute tool function
MCP->>Client: Return result
Client->>Protocol: Send ClientMessage (RESULT)
Protocol->>Dispatcher: Receive results
Dispatcher->>Agent: Return List[Result]
```
### Command Structure
```python
@dataclass
class Command:
"""Atomic command to be executed on device client"""
tool_name: str # MCP tool name (e.g., "click_element")
parameters: Dict[str, Any] # Tool arguments
tool_type: str # "data_collection" or "action"
call_id: str # Unique identifier
```
!!! warning "Deterministic Execution"
Commands are designed to be:
- **Atomic**: Single, indivisible operation
- **Deterministic**: Same inputs → same outputs
- **Auditable**: Full command history logged
- **Reversible**: Where possible, support undo operations
**Extensibility:**
New commands can be added by:
1. Registering MCP tool on device client
2. LLM dynamically selects tool from available MCP registry
3. No server-side code changes required
**See [Command Layer Documentation](design/command.md) for complete details.**
---
## Server-Client Architecture
Device agents use a **server-client separation** to balance safety, scalability, and functionality:
```mermaid
graph TB
subgraph "Server Side (UFO3 Orchestrator)"
Server[Device Agent Server]
State[State Machine]
Processor[Strategy Processor]
LLM[LLM Service]
Memory[Memory & Context]
Server --> State
Server --> Processor
Server --> Memory
Processor -.-> LLM
end
subgraph "Communication Layer"
AIP[AIP Protocol WebSocket]
end
subgraph "Client Side (Device)"
Client[Device Client]
Dispatcher[Command Dispatcher]
MCP[MCP Server Manager]
Tools[MCP Tools]
OS[Operating System]
Client --> Dispatcher
Dispatcher --> MCP
MCP --> Tools
Tools --> OS
end
Server <-->|Commands/Results| AIP
AIP <-->|Commands/Results| Client
```
### Separation of Concerns
| Component | Location | Responsibilities | Security Boundary |
|-----------|----------|------------------|-------------------|
| **Agent Server** | Orchestrator | State management, reasoning, planning, memory | Untrusted (LLM-driven decisions) |
| **Device Client** | Device | Command execution, MCP tool calls, resource access | Trusted (validated operations) |
| **AIP Protocol** | Communication | Message serialization, WebSocket transport, error handling | Secure channel (authentication, encryption) |
**Why Server-Client Separation?**
**Safety**: Isolates potentially unsafe LLM-generated decisions from direct system access. Clients validate all commands before execution.
**Scalability**: Single orchestrator server manages multiple device clients. Reduces per-device resource requirements.
**Flexibility**: Device clients can run on resource-constrained devices (embedded systems, mobile) while heavy reasoning occurs on server.
**See [Server-Client Architecture](server_client_architecture.md) for complete details.**
---
## Supported Device Platforms
UFO3 currently supports **Windows** and **Linux** device agents, with architecture designed for extensibility to other platforms.
### Windows Agents
```mermaid
graph TB
subgraph "Windows Device (Two-Tier Hierarchy)"
Host[HostAgent Application Selection]
App1[AppAgent Word]
App2[AppAgent Excel]
App3[AppAgent Browser]
Host -->|delegates| App1
Host -->|delegates| App2
Host -->|delegates| App3
end
User[User Request] --> Host
```
**HostAgent** (Application-Level Coordinator):
- Selects appropriate application(s) for user request
- Decomposes tasks into application-specific subtasks
- Coordinates multiple AppAgents
- Manages application switching and data transfer
**AppAgent** (Application-Level Executor):
- Controls specific Windows application (Word, Excel, browser, etc.)
- Uses UI Automation for control element discovery
- Executes application-specific actions (type, click, scroll)
- Maintains application context and memory
!!! example "Windows Agent Example"
**User Request**: "Create a chart from sales.xlsx and insert into report.docx"
1. **HostAgent** decomposes:
- Open Excel → Create chart → Copy chart
- Open Word → Paste chart
2. **AppAgent (Excel)**: Opens `sales.xlsx`, creates chart, copies to clipboard
3. **AppAgent (Word)**: Opens `report.docx`, pastes chart at cursor
### Linux Agent
```mermaid
graph TB
subgraph "Linux Device (Single-Tier Architecture)"
Linux[LinuxAgent Direct System Control]
Shell[Shell Commands]
Files[File Operations]
Apps[Application Launch]
Linux --> Shell
Linux --> Files
Linux --> Apps
end
User[User Request] --> Linux
```
**LinuxAgent** (System-Level Executor):
- Direct shell command execution
- File system operations
- Application launch and management
- Single-tier architecture (no application-level hierarchy)
!!! info "Architecture Difference"
**Windows** uses two-tier hierarchy (HostAgent → AppAgent) due to:
- UI Automation framework's application-centric model
- Distinct application contexts requiring specialized agents
**Linux** uses single-tier architecture because:
- Shell provides unified interface to all system operations
- Application control occurs through same command-line interface
### Platform Comparison
| Feature | Windows (UFO2) | Linux | macOS (Future) | Embedded (Future) |
|---------|----------------|-------|----------------|-------------------|
| **Agent Hierarchy** | Two-tier (Host → App) | Single-tier | TBD | Single-tier |
| **UI Control** | UI Automation | X11/Wayland | Accessibility API | Platform-specific |
| **Command Interface** | MCP tools (Win32 API) | MCP tools (Shell) | MCP tools (AppleScript) | MCP tools (Custom) |
| **Observation** | Screenshot + UI tree | Screenshot + Shell output | Screenshot + UI tree | Sensor data |
| **State Management** | Shared FSM | Shared FSM | Shared FSM | Shared FSM |
| **Strategy Layer** | Processor framework | Processor framework | Processor framework | Processor framework |
| **Current Status** | ✅ Production | ✅ Production | 🔜 Planned | 🔜 Planned |
**Extensibility Path:**
Adding a new platform requires:
1. **Implement Agent Class**: Extend `BasicAgent` (inherit State layer, Processor framework)
2. **Create Processor**: Subclass `ProcessorTemplate`, implement platform-specific strategies
3. **Define MCP Tools**: Register platform-specific MCP tools on device client
4. **Register Agent**: Use `@AgentRegistry.register` decorator
No changes to core State layer, Processor framework, or AIP protocol required.
**See [Agent Types Documentation](agent_types.md) for complete implementation details.**
---
## Agent Lifecycle
A typical device agent execution follows this lifecycle:
```mermaid
sequenceDiagram
participant User
participant Orchestrator
participant Agent
participant State
participant Processor
participant LLM
participant Dispatcher
participant Client
User->>Orchestrator: Submit task
Orchestrator->>Agent: Initialize agent (CONTINUE state)
loop Until FINISH/FAIL/ERROR
Agent->>State: handle(agent, context)
State->>Processor: execute strategies
Processor->>Processor: DATA_COLLECTION
Note over Processor: Screenshot, UI info
Processor->>LLM: LLM_INTERACTION
LLM-->>Processor: Action decision
Processor->>Dispatcher: ACTION_EXECUTION
Dispatcher->>Client: Execute commands
Client-->>Dispatcher: Results
Dispatcher-->>Processor: Results
Processor->>Processor: MEMORY_UPDATE
Note over Processor: Update memory, blackboard
State->>State: next_state(agent)
State->>Agent: Update agent status
end
Agent->>Orchestrator: Task complete/failed
Orchestrator->>User: Return result
```
### Execution Phases
1. **Initialization**: Agent created with default state (`CONTINUE`), processor, memory
2. **State Handling**: Current state's `handle()` method invoked with agent and context
3. **Strategy Execution**: Processor runs strategies in sequence (DATA_COLLECTION → LLM_INTERACTION → ACTION_EXECUTION → MEMORY_UPDATE)
4. **State Transition**: State's `next_state()` determines next FSM state
5. **Repeat/Terminate**: Loop continues until terminal state (`FINISH`, `FAIL`, `ERROR`)
!!! tip "Multi-Agent Handoff"
For multi-agent scenarios (e.g., Windows HostAgent → AppAgent), states implement `next_agent()`:
```python
def next_agent(self, agent: BasicAgent) -> BasicAgent:
# HostAgent delegates to AppAgent
if agent.status == "DELEGATE_TO_APP":
return agent.create_app_agent(...)
return agent
```
---
## Memory and Context Management
Device agents maintain two types of memory:
### Short-Term Memory (Agent Memory)
**Purpose**: Track agent's execution history within a session
**Implementation**: `Memory` class with `MemoryItem` entries
```python
class Memory:
"""Agent's short-term memory"""
_content: List[MemoryItem]
def add_memory_item(self, memory_item: MemoryItem):
"""Add new memory entry"""
self._content.append(memory_item)
```
**Content**: Actions taken, observations made, results received
**Lifetime**: Single session (cleared between tasks)
### Long-Term Memory (Blackboard)
**Purpose**: Share information across agents and sessions
**Implementation**: `Blackboard` class with multiple memory types
```python
class Blackboard:
"""Multi-agent shared memory"""
_questions: Memory # Q&A history
_requests: Memory # Request history
_trajectories: Memory # Action trajectories
_screenshots: Memory # Visual observations
```
**Content**: Common knowledge, successful action patterns, user preferences
**Lifetime**: Persistent across sessions (can be saved/loaded)
**Blackboard Usage Example:**
**Scenario**: HostAgent delegates to AppAgent (Excel)
1. HostAgent adds to blackboard:
- Request: "Create sales chart"
- Context: Previous analysis results
2. AppAgent reads from blackboard:
- Retrieves request and context
- Adds action trajectories as executed
- Adds screenshot after chart creation
3. HostAgent reads updated blackboard:
- Verifies chart creation
- Continues to next step (insert to Word)
**See [Memory System Documentation](design/memory.md) for complete details.**
---
## Integration with UFO3 Components
Device agents integrate with other UFO3 components:
```mermaid
graph TB
subgraph "UFO3 Architecture"
Session[Session/Round Manager]
Context[Global Context]
Agent[Device Agent]
Dispatcher[Command Dispatcher]
AIP[AIP Protocol]
Client[Device Client]
MCP[MCP Servers]
end
Session -->|manages lifecycle| Agent
Session -->|provides| Context
Agent -->|reads/writes| Context
Agent -->|sends commands| Dispatcher
Dispatcher -->|uses| AIP
AIP <-->|WebSocket| Client
Client -->|calls| MCP
```
### Integration Points
| Component | Relationship | Description |
|-----------|--------------|-------------|
| **Session Manager** | Parent | Creates agents, manages agent lifecycle, coordinates multi-agent workflows |
| **Round Manager** | Sibling | Manages round-based execution, tracks round state, synchronizes with agent steps |
| **Global Context** | Shared State | Agent reads request/config, writes results/status, shares data across components |
| **Command Dispatcher** | Execution Interface | Agent sends commands, dispatcher routes to client, returns results |
| **AIP Protocol** | Communication | Serializes commands/results, manages WebSocket, handles errors/timeouts |
| **Device Client** | Executor | Receives commands, invokes MCP tools, returns results |
| **MCP Servers** | Tool Registry | Provides available tools, executes tool functions, returns structured results |
**See [Session Documentation](../modules/session.md), [Context Documentation](../modules/context.md), and [AIP Protocol](../../aip/overview.md) for integration details.**
---
## Design Patterns
Device agent architecture leverages several design patterns:
### 1. State Pattern (FSM Layer)
**Purpose**: Encapsulate state-specific behavior, enable dynamic state transitions
**Implementation**: `AgentState` abstract class, concrete state classes
```python
class AgentState(ABC):
@abstractmethod
async def handle(self, agent, context):
"""Execute state-specific logic"""
pass
@abstractmethod
def next_state(self, agent):
"""Determine next state"""
pass
```
### 2. Strategy Pattern (Strategy Layer)
**Purpose**: Define family of algorithms (strategies), make them interchangeable
**Implementation**: `ProcessingStrategy` protocol, concrete strategy classes
```python
class ProcessingStrategy(Protocol):
async def execute(self, agent, context) -> ProcessingResult:
"""Execute strategy logic"""
pass
```
### 3. Template Method Pattern (Processor Framework)
**Purpose**: Define skeleton of algorithm, let subclasses override specific steps
**Implementation**: `ProcessorTemplate` abstract class
```python
class ProcessorTemplate(ABC):
@abstractmethod
def _setup_strategies(self):
"""Subclass defines which strategies to use"""
pass
async def process(self, agent, context):
"""Template method - runs strategies in sequence"""
for phase, strategy in self.strategies.items():
result = await strategy.execute(agent, context)
# Handle result, update context
```
### 4. Singleton Pattern (State Manager)
**Purpose**: Ensure single instance of state registry
**Implementation**: `AgentStateManager` with metaclass
```python
class AgentStateManager(ABC, metaclass=SingletonABCMeta):
_state_mapping: Dict[str, Type[AgentState]] = {}
def get_state(self, status: str) -> AgentState:
"""Lazy load and return state instance"""
pass
```
### 5. Registry Pattern (Agent Registration)
**Purpose**: Register agent types, enable dynamic agent creation
**Implementation**: `AgentRegistry` decorator
```python
@AgentRegistry.register(agent_name="appagent", processor_cls=AppAgentProcessor)
class AppAgent(BasicAgent):
pass
```
### 6. Blackboard Pattern (Multi-Agent Coordination)
**Purpose**: Share data across multiple agents
**Implementation**: `Blackboard` class
```python
class Blackboard:
_questions: Memory
_requests: Memory
_trajectories: Memory
_screenshots: Memory
```
---
## Best Practices
### State Design
- Keep states **focused**: Each state should have single, clear responsibility
- Use **rule-based transitions** for deterministic flows, **LLM-driven transitions** for adaptive behavior
- Implement **error states** for graceful degradation
- Document **state invariants** and **transition conditions**
### Strategy Design
- Keep strategies **atomic**: Each strategy should perform one cohesive task
- Declare **dependencies explicitly** using `get_dependencies()`
- Use **ComposedStrategy** to combine multiple strategies within a phase
- Implement **fail-fast** for critical errors, **continue-on-error** for optional operations
### Command Design
- Keep commands **atomic**: Single, indivisible operation
- Design commands to be **idempotent** where possible
- Validate **arguments** on client side before execution
- Return **structured results** with success/failure status
### Memory Management
- Use **short-term memory** for agent-specific execution history
- Use **blackboard** for multi-agent coordination and persistent knowledge
- **Clear memory** between sessions to avoid context pollution
- Implement **memory pruning** for long-running sessions
!!! warning "Security Considerations"
- **Validate all commands** on client side before execution
- **Sanitize LLM outputs** before converting to commands
- **Limit command scope** via MCP tool permissions
- **Audit all actions** for compliance and debugging
- **Isolate agents** to prevent unauthorized cross-agent access
---
## Related Documentation
**Deep Dive Into Layers:**
- [State Layer Documentation](design/state.md): FSM, AgentState, transitions, state registration
- [Processor and Strategy Documentation](design/processor.md): ProcessorTemplate, strategies, dependency management
- [Command Layer Documentation](design/command.md): CommandDispatcher, MCP integration, atomic commands
**Supporting Systems:**
- [Memory System Documentation](design/memory.md): Memory, MemoryItem, Blackboard patterns
- [Agent Types Documentation](agent_types.md): Windows agents, Linux agent, platform-specific implementations
**Integration Points:**
- [Server-Client Architecture](server_client_architecture.md): Server and client separation, communication patterns
- [Server Architecture](../../server/overview.md): Agent server, WebSocket manager, orchestration
- [Client Architecture](../../client/overview.md): Device client, MCP servers, command execution
- [AIP Protocol](../../aip/overview.md): Agent Interaction Protocol for server-client communication
- [MCP Integration](../../mcp/overview.md): Model Context Protocol for tool execution
---
## Summary
**Key Takeaways:**
✅ **Three-Layer Architecture**: State (FSM) → Strategy (Execution Logic) → Command (System Interface)
✅ **Server-Client Separation**: Safe isolation of reasoning (server) from execution (client)
✅ **Multi-Platform Support**: Windows (two-tier), Linux (single-tier), extensible to macOS and embedded
✅ **LLM-Driven Reasoning**: Dynamic decision-making with structured command output
✅ **Modular & Extensible**: Register new states, compose strategies, add MCP tools without core changes
✅ **Memory Systems**: Short-term (agent memory) and long-term (blackboard) for coordination
✅ **Design Patterns**: State, Strategy, Template Method, Singleton, Registry, Blackboard
The Device Agent architecture provides a **robust, extensible foundation** for multi-device automation. By separating concerns across three layers and isolating reasoning from execution, UFO3 achieves both **safety** and **flexibility** for orchestrating complex cross-device workflows.
---
## Reference
Below is the reference for the `BasicAgent` class. All device agents inherit from `BasicAgent` and implement platform-specific processors and states:
::: agents.agent.basic.BasicAgent
---
### Documents Docs Infrastructure Modules Overview (`documents/docs/infrastructure/modules/overview.md`)
# Module System Overview
The **Module System** is the core execution engine of UFO, orchestrating the complete lifecycle of user interactions from initial request to final completion. It manages sessions, rounds, context state, and command dispatch across both Windows and Linux platforms.
**Quick Navigation:**
- New to modules? Start with [Session](./session.md) and [Round](./round.md) basics
- Understanding state? See [Context](./context.md) management
- Command execution? Check [Dispatcher](./dispatcher.md) patterns
---
## Architecture Overview
The module system implements a **hierarchical execution model** with clear separation of concerns:
```mermaid
graph TB
subgraph "User Interaction Layer"
UI[Interactor User I/O]
end
subgraph "Session Management Layer"
SF[SessionFactory Creates sessions]
SP[SessionPool Manages multiple sessions]
S[Session Conversation lifecycle]
end
subgraph "Execution Layer"
R[Round Single request handler]
C[Context Shared state]
end
subgraph "Command Layer"
D[Dispatcher Command routing]
LCD[LocalCommandDispatcher]
WSD[WebSocketCommandDispatcher]
end
subgraph "Platform Layer"
WS[WindowsBaseSession]
LS[LinuxBaseSession]
SS[ServiceSession]
end
UI -.Request.-> SF
SF --> SP
SP --> S
S --> R
R --> C
R --> D
D --> LCD
D --> WSD
S -.inherits.-> WS
S -.inherits.-> LS
S -.inherits.-> SS
style UI fill:#e1f5ff
style SF fill:#fff4e1
style SP fill:#f0ffe1
style S fill:#ffe1f5
style R fill:#e1ffe1
style C fill:#ffe1e1
style D fill:#f5e1ff
```
---
## Core Components
### 1. Session Management
A **Session** represents a complete conversation between the user and UFO, potentially spanning multiple requests and rounds.
**Session Hierarchy:**
```mermaid
classDiagram
class BaseSession {
<>
+task: str
+context: Context
+rounds: Dict[int, BaseRound]
+run()
+create_new_round()
+is_finished()
}
class WindowsBaseSession {
+host_agent: HostAgent
+_init_agents()
}
class LinuxBaseSession {
+agent: LinuxAgent
+_init_agents()
}
class Session {
+mode: str
+next_request()
}
class ServiceSession {
+task_protocol: TaskExecutionProtocol
+_init_context()
}
class LinuxSession {
+next_request()
}
class FollowerSession {
+plan_reader: PlanReader
}
BaseSession <|-- WindowsBaseSession
BaseSession <|-- LinuxBaseSession
WindowsBaseSession <|-- Session
WindowsBaseSession <|-- ServiceSession
LinuxBaseSession <|-- LinuxSession
Session <|-- FollowerSession
```
**Session Types:**
| Session Type | Platform | Use Case | Communication |
|--------------|----------|----------|---------------|
| **Session** | Windows | Interactive mode | Local |
| **ServiceSession** | Windows | Server-controlled | WebSocket (AIP) |
| **LinuxSession** | Linux | Interactive mode | Local |
| **LinuxServiceSession** | Linux | Server-controlled | WebSocket (AIP) |
| **FollowerSession** | Windows | Plan execution | Local |
| **FromFileSession** | Windows | Batch processing | Local |
| **OpenAIOperatorSession** | Windows | Operator mode | Local |
!!!example "Session Creation"
```python
from ufo.module.session_pool import SessionFactory
# Create interactive Windows session
factory = SessionFactory()
sessions = factory.create_session(
task="email_task",
mode="normal",
plan="",
request="Open Outlook and send an email"
)
# Create Linux service session
linux_session = factory.create_service_session(
task="data_task",
should_evaluate=True,
id="session_001",
request="Process CSV files",
platform_override="linux"
)
```
---
### 2. Round Execution
A **Round** handles a single user request by orchestrating agents through a state machine, executing actions until completion.
**Round Lifecycle:**
```mermaid
stateDiagram-v2
[*] --> Created: Initialize Round
Created --> AgentHandle: agent.handle(context)
AgentHandle --> StateTransition: Determine next state
StateTransition --> AgentSwitch: Switch agent if needed
AgentSwitch --> SubtaskCheck: Check if subtask ends
SubtaskCheck --> CaptureSnapshot: Subtask complete
SubtaskCheck --> AgentHandle: Continue
CaptureSnapshot --> AgentHandle: Next subtask
AgentHandle --> RoundComplete: is_finished() = True
RoundComplete --> Evaluation: should_evaluate = True
RoundComplete --> [*]: should_evaluate = False
Evaluation --> [*]
note right of AgentHandle
Agent processes current state
Updates context
Executes actions
end note
note right of StateTransition
State pattern determines:
- Next state
- Next agent
- Round completion
end note
```
**Key Round Operations:**
| Operation | Purpose | Trigger |
|-----------|---------|---------|
| `agent.handle(context)` | Process current state | Each iteration |
| `state.next_state(agent)` | Determine next state | After handle |
| `state.next_agent(agent)` | Switch agent if needed | After state transition |
| `capture_last_snapshot()` | Save UI state | Subtask/Round end |
| `evaluation()` | Assess completion | Round end (if enabled) |
!!!warning "Round Termination Conditions"
A round finishes when:
- `state.is_round_end()` returns `True`
- Session step exceeds `ufo_config.system.max_step`
- Agent enters ERROR state
---
### 3. Context State Management
**Context** is a type-safe key-value store that maintains state across all rounds in a session.
**Context Architecture:**
```mermaid
graph LR
subgraph "Context Storage"
CN[ContextNames Enum]
CV[Context Values Dict]
end
subgraph "Tracked Data"
ID[Session/Round IDs]
ST[Steps & Costs]
LOG[Loggers]
APP[Application State]
CMD[Command Dispatcher]
end
subgraph "Access Patterns"
GET[context.get(key)]
SET[context.set(key, value)]
UPD[context.update_dict(key, dict)]
end
CN -.defines.-> CV
CV --> ID
CV --> ST
CV --> LOG
CV --> APP
CV --> CMD
GET -.reads.-> CV
SET -.writes.-> CV
UPD -.merges.-> CV
style CN fill:#e1f5ff
style CV fill:#fff4e1
style GET fill:#f0ffe1
style SET fill:#ffe1f5
style UPD fill:#f5e1ff
```
**Context Categories:**
| Category | Context Names | Type | Purpose |
|----------|---------------|------|---------|
| **Identifiers** | `ID`, `CURRENT_ROUND_ID` | `int` | Session/round tracking |
| **Execution State** | `SESSION_STEP`, `ROUND_STEP` | `int/dict` | Progress tracking |
| **Cost Tracking** | `SESSION_COST`, `ROUND_COST` | `float/dict` | LLM API costs |
| **Requests** | `REQUEST`, `SUBTASK`, `PREVIOUS_SUBTASKS` | `str/list` | Task information |
| **Application** | `APPLICATION_WINDOW`, `APPLICATION_PROCESS_NAME` | `UIAWrapper/str` | UI automation |
| **Logging** | `LOGGER`, `REQUEST_LOGGER`, `EVALUATION_LOGGER` | `FileWriter` | Log outputs |
| **Communication** | `HOST_MESSAGE`, `CONTROL_REANNOTATION` | `list` | Agent messages |
| **Infrastructure** | `command_dispatcher` | `BasicCommandDispatcher` | Command execution |
!!!example "Context Usage Patterns"
```python
from ufo.module.context import Context, ContextNames
# Initialize context
context = Context()
# Set values
context.set(ContextNames.REQUEST, "Open Notepad")
context.set(ContextNames.SESSION_STEP, 0)
# Get values
request = context.get(ContextNames.REQUEST) # "Open Notepad"
step = context.get(ContextNames.SESSION_STEP) # 0
# Update dictionaries (for round-specific tracking)
round_costs = {1: 0.05, 2: 0.03}
context.update_dict(ContextNames.ROUND_COST, round_costs)
# Auto-sync current round values
current_cost = context.current_round_cost # Auto-synced
```
---
### 4. Command Dispatching
**Dispatchers** route commands to execution environments (local MCP tools or remote WebSocket clients) and handle result delivery.
**Dispatcher Architecture:**
```mermaid
graph TB
subgraph "Agent Layer"
AG[Agent generates commands]
end
subgraph "Dispatcher Layer"
BD[BasicCommandDispatcher Abstract base]
LCD[LocalCommandDispatcher MCP tools]
WSD[WebSocketCommandDispatcher AIP protocol]
end
subgraph "Execution Layer"
CR[CommandRouter]
CM[ComputerManager]
MCP[MCP Servers]
WS[WebSocket Client]
end
subgraph "Result Handling"
RES[Results: List~Result~]
ERR[Error Results]
end
AG --> BD
BD -.implements.-> LCD
BD -.implements.-> WSD
LCD --> CR
CR --> CM
CM --> MCP
WSD --> WS
MCP --> RES
WS --> RES
LCD --> ERR
WSD --> ERR
style AG fill:#e1f5ff
style BD fill:#fff4e1
style LCD fill:#f0ffe1
style WSD fill:#ffe1f5
style RES fill:#e1ffe1
```
**Dispatcher Comparison:**
| Dispatcher | Use Case | Communication | Error Handling | Timeout |
|------------|----------|---------------|----------------|---------|
| **LocalCommandDispatcher** | Interactive sessions | Direct MCP calls | Generates error Results | 6000s |
| **WebSocketCommandDispatcher** | Service sessions | AIP protocol messages | Generates error Results | 6000s |
!!!example "Command Dispatch Flow"
```python
from aip.messages import Command
# Create commands
commands = [
Command(
tool_name="click_element",
parameters={"control_label": "1", "button": "left"},
tool_type="windows"
)
]
# Execute via dispatcher (attached to context)
results = await context.command_dispatcher.execute_commands(
commands=commands,
timeout=30.0
)
# Process results
for result in results:
if result.status == ResultStatus.SUCCESS:
print(f"Action succeeded: {result.result}")
else:
print(f"Action failed: {result.error}")
```
---
### 5. User Interaction
**Interactor** provides rich CLI experiences for user input with styled prompts, panels, and confirmations.
**Interaction Flows:**
```mermaid
sequenceDiagram
participant U as User
participant I as Interactor
participant S as Session
participant R as Round
U->>I: Start UFO
I->>I: first_request()
I-->>U: 🛸 Welcome Panel
U->>I: "Open Notepad"
I->>S: Initial request
S->>R: Create Round 1
R->>R: Execute...
R-->>S: Round complete
S->>I: new_request()
I-->>U: 🛸 Next Request Panel
U->>I: "Type hello"
I->>S: Next request
S->>R: Create Round 2
R->>R: Execute...
R-->>S: Round complete
S->>I: new_request()
I-->>U: 🛸 Next Request Panel
U->>I: "N"
I-->>U: 👋 Goodbye Panel
I->>S: complete=True
S->>S: Terminate
S->>I: experience_asker()
I-->>U: 💾 Save Experience Panel
U->>I: Yes
I->>S: Save experience
```
**Interactor Functions:**
| Function | Purpose | Returns | Example UI |
|----------|---------|---------|-----------|
| `first_request()` | Initial request prompt | `str` | 🛸 Welcome Panel with examples |
| `new_request()` | Subsequent requests | `Tuple[str, bool]` | 🛸 Next Request Panel |
| `experience_asker()` | Save experience prompt | `bool` | 💾 Learning & Memory Panel |
| `question_asker()` | Collect information | `str` | 🤔 Numbered Question Panel |
| `sensitive_step_asker()` | Security confirmation | `bool` | 🔒 Security Check Panel |
!!!example "Styled User Prompts"
```python
from ufo.module import interactor
# First interaction with rich welcome
request = interactor.first_request()
# Shows:
# ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
# ┃ 🛸 UFO Assistant ┃
# ┃ 🚀 Welcome to UFO - Your AI Assistant ┃
# ┃ ...examples... ┃
# ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
# Get next request
request, complete = interactor.new_request()
if complete:
print("User exited")
# Ask for permission on sensitive actions
proceed = interactor.sensitive_step_asker(
action="Delete file",
control_text="important.docx"
)
if not proceed:
print("Action cancelled by user")
```
---
### 6. Session Factory & Pool
**SessionFactory** creates platform-specific sessions based on mode and configuration, while **SessionPool** manages batch execution.
**Factory Creation Logic:**
```mermaid
graph TB
START[SessionFactory.create_session]
PLATFORM{Platform?}
MODE{Mode?}
WNORMAL[Windows Session]
WSERVICE[Windows ServiceSession]
WFOLLOWER[Windows FollowerSession]
WBATCH[Windows FromFileSession]
WOPERATOR[Windows OpenAIOperatorSession]
LNORMAL[Linux Session]
LSERVICE[Linux ServiceSession]
START --> PLATFORM
PLATFORM -->|Windows| MODE
PLATFORM -->|Linux| MODE
MODE -->|normal| WNORMAL
MODE -->|service| WSERVICE
MODE -->|follower| WFOLLOWER
MODE -->|batch_normal| WBATCH
MODE -->|operator| WOPERATOR
MODE -->|normal Linux| LNORMAL
MODE -->|service Linux| LSERVICE
style START fill:#e1f5ff
style PLATFORM fill:#fff4e1
style MODE fill:#f0ffe1
style WNORMAL fill:#ffe1f5
style LNORMAL fill:#ffe1f5
```
**Session Modes:**
| Mode | Platform | Description | Input | Evaluation |
|------|----------|-------------|-------|------------|
| **normal** | Both | Interactive single-task | User input | Optional |
| **service** | Both | WebSocket-controlled | Remote request | Optional |
| **follower** | Windows | Replay recorded plan | Plan JSON file | Optional |
| **batch_normal** | Windows | Multiple tasks from files | JSON folder | Per-task |
| **operator** | Windows | OpenAI Operator API | User input | Optional |
| **normal_operator** | Both | Interactive with operator | User input | Optional |
!!!example "SessionFactory Usage"
```python
from ufo.module.session_pool import SessionFactory, SessionPool
factory = SessionFactory()
# Interactive Windows session
sessions = factory.create_session(
task="task1",
mode="normal",
plan="",
request="Open calculator"
)
# Batch Windows sessions from folder
batch_sessions = factory.create_session(
task="batch_task",
mode="batch_normal",
plan="./plans/", # Folder with multiple .json files
request=""
)
# Run all sessions
pool = SessionPool(batch_sessions)
await pool.run_all()
```
---
## Cross-Platform Support
The module system provides a unified API while allowing platform-specific behavior through inheritance.
**Platform Differences:**
| Aspect | Windows | Linux |
|--------|---------|-------|
| **Agent Architecture** | HostAgent → AppAgent (two-tier) | LinuxAgent (single-tier) |
| **HostAgent** | ✅ Used for planning | ❌ Not used |
| **Session Base** | `WindowsBaseSession` | `LinuxBaseSession` |
| **UI Automation** | UIA (pywinauto) | Custom automation |
| **Service Mode** | `ServiceSession` | `LinuxServiceSession` |
| **Evaluation** | ✅ Full support | ⚠️ Limited |
| **Markdown Logs** | ✅ Supported | ⚠️ Planned |
!!!example "Platform Detection"
```python
import platform
# Auto-detect platform
current_platform = platform.system().lower() # 'windows' or 'linux'
# Override platform
sessions = factory.create_session(
task="cross_platform_task",
mode="normal",
plan="",
request="List files",
platform_override="linux" # Force Linux session
)
```
---
## Execution Flow
Understanding how components interact during a complete user request:
```
/* Detailed source-code truncated for AI context efficiency. */
```
---
## File Structure
```
ufo/module/
├── __init__.py
├── basic.py # BaseSession, BaseRound, FileWriter
├── context.py # Context, ContextNames
├── dispatcher.py # Command dispatchers
├── interactor.py # User interaction functions
├── session_pool.py # SessionFactory, SessionPool
└── sessions/
├── __init__.py
├── platform_session.py # WindowsBaseSession, LinuxBaseSession
├── session.py # Session, FollowerSession, FromFileSession
├── service_session.py # ServiceSession
├── linux_session.py # LinuxSession, LinuxServiceSession
└── plan_reader.py # PlanReader for follower mode
```
---
## Key Design Patterns
### 1. State Pattern
Agents use the State pattern to manage transitions and determine control flow.
```python
# Agent state determines:
next_state = agent.state.next_state(agent)
next_agent = agent.state.next_agent(agent)
is_done = agent.state.is_round_end()
```
### 2. Factory Pattern
SessionFactory creates appropriate session types based on platform and mode.
### 3. Command Pattern
Commands encapsulate actions with parameters, enabling async execution and result tracking.
### 4. Observer Pattern
Context changes notify dependent components (implicit through shared state).
---
## Best Practices
!!!tip "Session Management"
- ✅ Always initialize context before creating rounds
- ✅ Use `SessionFactory` for session creation (handles platform differences)
- ✅ Attach command dispatcher to context early
- ✅ Call `context._sync_round_values()` before accessing round-specific data
- ❌ Don't access round context before round initialization
!!!tip "Round Execution"
- ✅ Let the state machine control agent transitions
- ✅ Capture snapshots at subtask boundaries
- ✅ Check `is_finished()` before each iteration
- ❌ Don't bypass state transitions
- ❌ Don't manually manipulate agent states
!!!tip "Context Usage"
- ✅ Use `ContextNames` enum for type-safe access
- ✅ Update dictionaries with `update_dict()` for merging
- ✅ Use properties (`current_round_cost`) for auto-synced values
- ❌ Don't directly access `_context` dictionary
- ❌ Don't store non-serializable objects without marking them
!!!tip "Command Dispatch"
- ✅ Always await `execute_commands()` (async)
- ✅ Handle timeout exceptions gracefully
- ✅ Check `ResultStatus` before using results
- ❌ Don't ignore error results
- ❌ Don't assume commands succeed
---
## Configuration
Key configuration options from `ufo_config`:
| Setting | Location | Default | Purpose |
|---------|----------|---------|---------|
| `max_step` | `system.max_step` | 50 | Max steps per session |
| `max_round` | `system.max_round` | 10 | Max rounds per session |
| `eva_session` | `system.eva_session` | `True` | Evaluate session |
| `eva_round` | `system.eva_round` | `False` | Evaluate each round |
| `save_experience` | `system.save_experience` | `"ask"` | When to save experience |
| `log_to_markdown` | `system.log_to_markdown` | `True` | Generate markdown logs |
| `save_ui_tree` | `system.save_ui_tree` | `True` | Save UI tree snapshots |
---
## Documentation Index
| Document | Description |
|----------|-------------|
| [Session](./session.md) | Session lifecycle and management |
| [Round](./round.md) | Round execution and orchestration |
| [Context](./context.md) | State management and context names |
| [Dispatcher](./dispatcher.md) | Command routing and execution |
| [Session Pool](./session_pool.md) | Factory and batch management |
| [Platform Sessions](./platform_sessions.md) | Windows/Linux implementations |
---
## Next Steps
**Learning Path:**
1. **Understand Sessions**: Read [Session](./session.md) to grasp the conversation model
2. **Learn Rounds**: Study [Round](./round.md) to understand action execution
3. **Master Context**: Review [Context](./context.md) for state management
4. **Explore Dispatch**: Check [Dispatcher](./dispatcher.md) for command execution
5. **Platform Specifics**: See [Platform Sessions](./platform_sessions.md) for Windows/Linux differences
---
### Documents Docs Tutorials Creating App Agent Overview (`documents/docs/tutorials/creating_app_agent/overview.md`)
# Enhancing AppAgent Capabilities
UFO² provides a flexible framework for application developers and users to enhance `AppAgent` capabilities for specific applications. AppAgent enhancement is about **augmenting** the existing AppAgent's capabilities through:
- **Knowledge** (help documents, demonstrations) to guide decision-making
- **Native API tools** (via MCP servers) for efficient automation
- **Application-specific context** for better understanding
## Enhancement Components
The `AppAgent` can be enhanced through three complementary approaches:
| Component | Description | Tutorial | Implementation Guide |
| --- | --- | --- | --- |
| **[Help Documents](./help_document_provision.md)** | Provide application-specific guidance and instructions to help the agent understand tasks and workflows | [Provision Guide](./help_document_provision.md) | [Learning from Help Documents](../../ufo2/core_features/knowledge_substrate/learning_from_help_document.md) |
| **[User Demonstrations](./demonstration_provision.md)** | Supply recorded user interactions to teach the agent how to perform specific tasks through examples | [Provision Guide](./demonstration_provision.md) | [Learning from Demonstrations](../../ufo2/core_features/knowledge_substrate/learning_from_demonstration.md) |
| **[Native API Tools](./warpping_app_native_api.md)** | Create custom MCP action servers that wrap application COM APIs or other native interfaces for efficient automation | [Wrapping Guide](./warpping_app_native_api.md) | [Creating MCP Servers](../creating_mcp_servers.md) |
## Enhancement Workflow
```mermaid
graph TB
Enhancement[AppAgent Enhancement Workflow]
Enhancement --> KnowledgeLayer[Knowledge Layer RAG-based]
Enhancement --> ToolLayer[Tool Layer MCP Servers]
KnowledgeLayer --> HelpDocs[Help Documents]
KnowledgeLayer --> DemoTraj[User Demonstrations]
ToolLayer --> UITools[UI Automation Tools]
ToolLayer --> APITools[Native API Tools]
HelpDocs --> EnhancedAgent[Enhanced AppAgent]
DemoTraj --> EnhancedAgent
UITools --> EnhancedAgent
APITools --> EnhancedAgent
style Enhancement fill:#e1f5ff,stroke:#01579b,stroke-width:3px
style KnowledgeLayer fill:#fff3e0,stroke:#e65100,stroke-width:2px
style ToolLayer fill:#f3e5f5,stroke:#4a148c,stroke-width:2px
style HelpDocs fill:#fffde7,stroke:#f57f17,stroke-width:2px
style DemoTraj fill:#fffde7,stroke:#f57f17,stroke-width:2px
style UITools fill:#fce4ec,stroke:#880e4f,stroke-width:2px
style APITools fill:#fce4ec,stroke:#880e4f,stroke-width:2px
style EnhancedAgent fill:#e8f5e9,stroke:#1b5e20,stroke-width:3px
```
## When to Use Each Component?
### Help Documents
**Use when:**
- You have official documentation, tutorials, or guides for your application
- Tasks require domain-specific knowledge or procedures
- You want the agent to understand application concepts and terminology
**Example:** Providing Excel formula documentation to help the agent use advanced Excel functions correctly.
### User Demonstrations
**Use when:**
- You can demonstrate the task yourself
- The task involves a specific sequence of UI interactions
- Visual/procedural knowledge is easier to show than describe
**Example:** Recording how to create a pivot table in Excel to teach the agent the exact steps.
### Native API Tools
**Use when:**
- Your application exposes COM APIs, REST APIs, or other programmable interfaces
- GUI automation is slow or unreliable for certain operations
- You need deterministic, high-performance automation
**Example:** Creating an MCP server that wraps Excel's COM API for inserting tables, formatting cells, etc.
## Enhancement Strategy
!!!tip "Hybrid Approach for Best Results"
Combine all three components for maximum effectiveness:
1. **Knowledge Foundation**: Provide help documents for conceptual understanding
2. **Procedural Learning**: Add demonstrations for complex workflows
3. **Efficient Execution**: Implement native API tools for performance-critical operations
The AppAgent will:
- Use knowledge to **understand** what to do
- Reference demonstrations to **learn** how to do it
- Leverage API tools when available for **efficient** execution
- Fall back to UI automation when needed
## Getting Started
Follow the tutorials in order to enhance your AppAgent:
1. **[Provide Help Documents](./help_document_provision.md)** - Start with knowledge
2. **[Add User Demonstrations](./demonstration_provision.md)** - Teach by example
3. **[Wrap Native APIs](./warpping_app_native_api.md)** - Enable efficient automation
## Related Documentation
- [AppAgent Overview](../../ufo2/app_agent/overview.md) - Understanding AppAgent architecture
- [Knowledge Substrate](../../ufo2/core_features/knowledge_substrate/overview.md) - How knowledge enhancement works
- [Creating MCP Servers](../creating_mcp_servers.md) - Building custom automation tools
- [MCP Configuration](../../mcp/configuration.md) - Registering MCP servers with AppAgent
- [Hybrid GUI–API Actions](../../ufo2/core_features/hybrid_actions.md) - Understanding dual-mode automation
---
### Documents Docs Tutorials Creating Device Agent Index (`documents/docs/tutorials/creating_device_agent/index.md`)
# Creating Device Agents - Tutorial Series
This tutorial series teaches you how to create new device agents for UFO³, using **LinuxAgent** as a reference implementation.
## 📚 Tutorial Structure
### [Part 0: Overview](overview.md)
**Introduction to device agents and architecture overview**
- Understanding device agents vs third-party agents
- Server-client architecture
- LinuxAgent as reference implementation
- Tutorial roadmap
**Time**: 15 minutes | **Difficulty**: ⭐
---
### [Part 1: Core Components](core_components.md)
**Building server-side components**
- Agent Class implementation
- Processor and strategy orchestration
- State Manager and FSM
- Processing Strategies (LLM, Action)
- Prompter for LLM interaction
**Time**: 45 minutes | **Difficulty**: ⭐⭐⭐
---
### [Part 2: MCP Server Development](mcp_server.md)
**Creating platform-specific MCP servers** *(Placeholder - Under Development)*
- MCP server architecture
- Defining MCP tools
- Command execution logic
- Error handling and validation
**Time**: 30 minutes | **Difficulty**: ⭐⭐
---
### [Part 3: Client Setup](client_setup.md)
**Setting up the device client** *(Placeholder - Under Development)*
- Client initialization and configuration
- MCP server manager integration
- WebSocket connection setup
- Platform detection
**Time**: 20 minutes | **Difficulty**: ⭐⭐
---
### [Part 4: Configuration & Deployment](configuration.md)
**Configuring and deploying your agent** *(Placeholder - Under Development)*
- `third_party.yaml` configuration
- `devices.yaml` device registration
- Prompt template creation
- Deployment steps
- Galaxy integration
**Time**: 25 minutes | **Difficulty**: ⭐⭐
---
### [Part 5: Testing & Debugging](testing.md)
**Testing and debugging your implementation** *(Placeholder - Under Development)*
- Unit testing strategies
- Integration testing
- Debugging techniques
- Common issues and solutions
**Time**: 30 minutes | **Difficulty**: ⭐⭐⭐
---
### [Part 6: Complete Example: MobileAgent](example_mobile_agent.md)
**Hands-on walkthrough creating MobileAgent** *(Placeholder - Under Development)*
- Step-by-step implementation
- Android/iOS platform specifics
- UI Automator integration
- Complete working example
**Time**: 60 minutes | **Difficulty**: ⭐⭐⭐⭐
---
## Quick Navigation
| I Want To... | Go To |
|--------------|-------|
| Understand device agent architecture | [Overview](overview.md#understanding-device-agents) |
| Study LinuxAgent implementation | [Overview](overview.md#linuxagent-reference-implementation) |
| Create Agent Class | [Core Components - Step 1](core_components.md#step-1-agent-class) |
| Build Processor | [Core Components - Step 2](core_components.md#step-2-processor) |
| Implement State Machine | [Core Components - Step 3](core_components.md#step-3-state-manager) |
| Write Processing Strategies | [Core Components - Step 4](core_components.md#step-4-processing-strategies) |
| Create Prompter | [Core Components - Step 5](core_components.md#step-5-prompter) |
| Build MCP Server | [MCP Server](mcp_server.md) *(placeholder)* |
| Setup Client | [Client Setup](client_setup.md) *(placeholder)* |
| Configure & Deploy | [Configuration](configuration.md) *(placeholder)* |
| Test & Debug | [Testing](testing.md) *(placeholder)* |
| Complete Example | [MobileAgent Example](example_mobile_agent.md) *(placeholder)* |
---
## Prerequisites
Before starting, ensure you have:
- ✅ Python 3.10+
- ✅ UFO³ repository cloned
- ✅ Basic understanding of async programming
- ✅ Familiarity with [Agent Architecture](../../infrastructure/agents/overview.md)
---
## Learning Path
```mermaid
graph LR
A[Overview ✅ Complete] --> B[Core Components ✅ Complete]
B --> C[MCP Server 📝 Placeholder]
C --> D[Client Setup 📝 Placeholder]
D --> E[Configuration 📝 Placeholder]
E --> F[Testing 📝 Placeholder]
F --> G[Complete Example 📝 Placeholder]
style A fill:#c8e6c9
style B fill:#c8e6c9
style C fill:#fff3e0
style D fill:#fff3e0
style E fill:#fff3e0
style F fill:#fff3e0
style G fill:#fff3e0
```
**Recommended Path**:
1. ✅ **Completed**: [Overview](overview.md) - Understand architecture
2. ✅ **Completed**: [Core Components](core_components.md) - Build server-side components
3. 📝 **Placeholder**: [MCP Server](mcp_server.md) - Create device commands
4. 📝 **Placeholder**: [Client Setup](client_setup.md) - Setup device client
5. 📝 **Placeholder**: [Configuration](configuration.md) - Configure and deploy
6. 📝 **Placeholder**: [Testing](testing.md) - Test and debug
7. 📝 **Placeholder**: [Complete Example](example_mobile_agent.md) - Full MobileAgent implementation
---
## Additional Resources
- **[Agent Architecture Overview](../../infrastructure/agents/overview.md)** - Three-layer architecture
- **[Agent Types](../../infrastructure/agents/agent_types.md)** - Platform-specific implementations
- **[Linux Quick Start](../../getting_started/quick_start_linux.md)** - Deploy LinuxAgent
- **[Creating Third-Party Agents](../creating_third_party_agents.md)** - Related tutorial
- **[MCP Overview](../../mcp/overview.md)** - Model Context Protocol
- **[Server Overview](../../server/overview.md)** - Server architecture
- **[Client Overview](../../client/overview.md)** - Client architecture
---
## Getting Help
If you encounter issues:
1. 📖 Review the [FAQ](../../faq.md)
2. 🐛 Check [troubleshooting guides](core_components.md#testing-your-implementation)
3. 💬 Ask in GitHub Discussions
4. 🐞 Report bugs on GitHub Issues
---
## Contributing
Found an issue or want to improve these tutorials?
- 📝 Submit a PR with improvements
- 💡 Suggest new topics
- 🔍 Report errors or unclear sections
---
**Ready to start?** → [Begin with Overview](overview.md)
---
### Documents Docs Tutorials Creating Device Agent Overview (`documents/docs/tutorials/creating_device_agent/overview.md`)
# Creating a New Device Agent - Complete Tutorial
This comprehensive tutorial teaches you how to create a new device agent (like `MobileAgent`, `AndroidAgent`, or `iOSAgent`) and integrate it with UFO³'s multi-device orchestration system. We'll use **LinuxAgent** as our primary reference implementation.
---
## 📋 Table of Contents
1. [Introduction](#introduction)
2. [Prerequisites](#prerequisites)
3. [Understanding Device Agents](#understanding-device-agents)
4. [LinuxAgent: Reference Implementation](#linuxagent-reference-implementation)
5. [Architecture Overview](#architecture-overview)
6. [Tutorial Roadmap](#tutorial-roadmap)
---
## Introduction
### What is a Device Agent?
A **Device Agent** is a specialized AI agent that controls and automates tasks on a specific type of device or platform. Unlike traditional third-party agents that extend specific functionality, device agents represent entire computing platforms with their own:
- **Execution Environment**: Device-specific OS, runtime, and APIs
- **Control Mechanism**: UI automation, CLI commands, or platform APIs
- **Communication Protocol**: Client-server architecture via WebSocket
- **MCP Integration**: Device-specific MCP servers for command execution
### Device Agent vs Third-Party Agent
| Aspect | Device Agent | Third-Party Agent |
|--------|--------------|-------------------|
| **Scope** | Full platform control (Windows, Linux, Mobile) | Specific functionality (Hardware, Web) |
| **Architecture** | Client-Server separation | Runs on orchestrator server |
| **Communication** | WebSocket + AIP Protocol | Direct method calls |
| **MCP Servers** | Platform-specific MCP servers | Shares MCP servers |
| **Examples** | WindowsAgent, LinuxAgent, MobileAgent | HardwareAgent, WebAgent |
| **Deployment** | Separate client process on device | Part of orchestrator |
### When to Create a Device Agent
Create a **Device Agent** when you need to:
- Control an entirely new platform (mobile, IoT, embedded)
- Execute tasks on remote or distributed devices
- Integrate with Galaxy multi-device orchestration
- Isolate execution for security or scalability
Create a **Third-Party Agent** when you need to:
- Extend existing platform with new capabilities
- Add specialized tools or APIs
- Run alongside existing agents
---
## Prerequisites
Before starting this tutorial, ensure you have:
### Knowledge Requirements
- ✅ **Python 3.10+**: Intermediate Python programming skills
- ✅ **Async Programming**: Understanding of `async`/`await` patterns
- ✅ **UFO³ Basics**: Familiarity with [Agent Architecture](../../infrastructure/agents/overview.md)
- ✅ **MCP Protocol**: Understanding of [Model Context Protocol](../../mcp/overview.md)
- ✅ **WebSocket**: Basic knowledge of WebSocket communication
### Recommended Reading
| Priority | Topic | Link | Time |
|----------|-------|------|------|
| 🥇 | **Agent Architecture Overview** | [Infrastructure/Agents](../../infrastructure/agents/overview.md) | 20 min |
| 🥇 | **LinuxAgent Quick Start** | [Quick Start: Linux](../../getting_started/quick_start_linux.md) | 15 min |
| 🥈 | **Server-Client Architecture** | [Server Overview](../../server/overview.md), [Client Overview](../../client/overview.md) | 30 min |
| 🥈 | **MCP Integration** | [MCP Overview](../../mcp/overview.md) | 20 min |
| 🥉 | **AIP Protocol** | [AIP Protocol](../../aip/overview.md) | 15 min |
### Development Environment
```bash
# Clone UFO³ repository
git clone https://github.com/microsoft/UFO.git
cd UFO
# Install dependencies
pip install -r requirements.txt
# Verify installation
python -c "import ufo; print('UFO³ installed successfully')"
```
---
## Understanding Device Agents
### Three-Layer Architecture
All device agents in UFO³ follow a **unified three-layer architecture**:
```mermaid
graph TB
subgraph "Device Agent Architecture"
subgraph "Level-1: State Layer (FSM)"
S1[AgentState]
S2[State Machine]
S3[State Transitions]
S1 --> S2 --> S3
end
subgraph "Level-2: Strategy Layer (Execution Logic)"
P1[ProcessorTemplate]
P2[DATA_COLLECTION]
P3[LLM_INTERACTION]
P4[ACTION_EXECUTION]
P5[MEMORY_UPDATE]
P1 --> P2 --> P3 --> P4 --> P5
end
subgraph "Level-3: Command Layer (System Interface)"
C1[CommandDispatcher]
C2[MCP Tools]
C3[Device Commands]
C1 --> C2 --> C3
end
S3 -->|delegates to| P1
P5 -->|executes via| C1
end
style S1 fill:#e1f5ff
style P1 fill:#fff3e0
style C1 fill:#f3e5f5
```
**Key Layers**:
1. **State Layer (Level-1)**: Finite State Machine controlling agent lifecycle
2. **Strategy Layer (Level-2)**: Processing pipeline with modular strategies
3. **Command Layer (Level-3)**: Atomic system operations via MCP
For detailed architecture, see [Agent Architecture Documentation](../../infrastructure/agents/overview.md).
---
### Server-Client Separation
Device agents use a **server-client architecture** for security and scalability:
```mermaid
graph LR
subgraph "Server Side (Orchestrator)"
Server[Device Agent Server]
State[State Machine]
Processor[Strategy Processor]
LLM[LLM Service]
Server --> State
Server --> Processor
Processor -.-> LLM
end
subgraph "Communication"
AIP[AIP Protocol WebSocket]
end
subgraph "Client Side (Device)"
Client[Device Client]
MCP[MCP Server Manager]
Tools[Platform Tools]
OS[Device OS]
Client --> MCP
MCP --> Tools
Tools --> OS
end
Server <-->|Commands/Results| AIP
AIP <-->|Commands/Results| Client
style Server fill:#e1f5ff
style Client fill:#c8e6c9
style AIP fill:#fff3e0
```
**Separation Benefits**:
| Component | Location | Responsibilities | Security |
|-----------|----------|------------------|----------|
| **Agent Server** | Orchestrator | Reasoning, planning, state management | Untrusted (LLM-driven) |
| **Device Client** | Target Device | Command execution, resource access | Trusted (validated operations) |
| **AIP Protocol** | Network | Message transport, serialization | Encrypted channel |
**Separation Benefits**:
- **Security**: Isolates LLM reasoning from system-level execution
- **Scalability**: Single orchestrator manages multiple devices
- **Flexibility**: Clients run on resource-constrained devices (mobile, IoT)
- **Safety**: Client validates all commands before execution
---
## LinuxAgent: Reference Implementation
### Why LinuxAgent as Reference?
**LinuxAgent** is the ideal reference for creating new device agents because:
- ✅ **Simple Architecture**: Single-tier agent (no HostAgent delegation)
- ✅ **Clear Separation**: Clean server-client boundary
- ✅ **Well-Documented**: Comprehensive code and documentation
- ✅ **Production-Ready**: Battle-tested in real deployments
- ✅ **Minimal Complexity**: Focuses on core device agent patterns
### LinuxAgent Components
```mermaid
graph TB
subgraph "Server Side (ufo/agents/)"
LA[LinuxAgent Class customized_agent.py]
LAP[LinuxAgentProcessor customized_agent_processor.py]
LAS[LinuxAgent Strategies linux_agent_strategy.py]
LAST[LinuxAgent States linux_agent_state.py]
LA --> LAP
LAP --> LAS
LA --> LAST
end
subgraph "Client Side (ufo/client/)"
Client[UFO Client client.py]
MCP[MCP Server Manager mcp_server_manager.py]
LinuxMCP[Linux MCP Server linux_mcp_server.py]
Client --> MCP
MCP --> LinuxMCP
end
subgraph "Configuration"
Config[third_party.yaml]
Devices[devices.yaml]
Prompts[Prompt Templates]
end
LA -.reads.-> Config
Client -.reads.-> Devices
LA -.uses.-> Prompts
style LA fill:#c8e6c9
style LAP fill:#c8e6c9
style LAS fill:#c8e6c9
style LAST fill:#c8e6c9
style Client fill:#e1f5ff
style MCP fill:#e1f5ff
style LinuxMCP fill:#e1f5ff
```
**File Locations**:
| Component | File Path | Purpose |
|-----------|-----------|---------|
| **Agent Class** | `ufo/agents/agent/customized_agent.py` | LinuxAgent definition |
| **Processor** | `ufo/agents/processors/customized/customized_agent_processor.py` | LinuxAgentProcessor |
| **Strategies** | `ufo/agents/processors/strategies/linux_agent_strategy.py` | LLM & Action strategies |
| **States** | `ufo/agents/states/linux_agent_state.py` | State machine states |
| **Prompter** | `ufo/prompter/customized/linux_agent_prompter.py` | Prompt construction |
| **Client** | `ufo/client/client.py` | Device client entry point |
| **MCP Server** | `ufo/client/mcp/http_servers/linux_mcp_server.py` | Command execution |
---
### LinuxAgent Architecture Diagram
```mermaid
sequenceDiagram
participant User
participant Server as LinuxAgent Server
participant AIP as AIP Protocol
participant Client as Linux Client
participant MCP as Linux MCP Server
participant Shell as Bash Shell
User->>Server: User Request: "List files in /tmp"
Server->>Server: State: ContinueLinuxAgentState
Server->>Server: Processor: LinuxAgentProcessor
Server->>Server: Strategy: LLM_INTERACTION
Note over Server: Construct prompt, call LLM
Server->>Server: LLM Response: execute_command("ls -la /tmp")
Server->>Server: Strategy: ACTION_EXECUTION
Server->>AIP: COMMAND: execute_command
AIP->>Client: WebSocket: COMMAND
Client->>MCP: Call MCP Tool: execute_command
MCP->>Shell: Execute: ls -la /tmp
Shell-->>MCP: stdout, stderr, exit_code
MCP-->>Client: Result
Client->>AIP: WebSocket: RESULT
AIP->>Server: RESULT
Server->>Server: Strategy: MEMORY_UPDATE
Server->>Server: Update memory & blackboard
Server->>Server: State Transition: FINISH
Server->>User: Task Complete
```
**Key Execution Flow**:
1. **User Request** → LinuxAgent Server receives request
2. **State Machine** → Activates `ContinueLinuxAgentState`
3. **Processor** → Executes `LinuxAgentProcessor` strategies
4. **LLM Interaction** → Generates shell command
5. **Action Execution** → Sends command via AIP to client
6. **MCP Execution** → Client executes via Linux MCP Server
7. **Result Handling** → Server receives result, updates memory
8. **State Transition** → Moves to `FINISH` state
---
## Architecture Overview
### Complete Device Agent Architecture
When creating a new device agent (e.g., `MobileAgent`), you'll implement these components:
```mermaid
graph TB
subgraph "1. Agent Definition"
A1[Agent Class MobileAgent]
A2[Processor MobileAgentProcessor]
A3[State Manager MobileAgentStateManager]
end
subgraph "2. Processing Strategies"
S1[DATA_COLLECTION Screenshot, UI Tree]
S2[LLM_INTERACTION Prompt Construction]
S3[ACTION_EXECUTION Command Dispatch]
S4[MEMORY_UPDATE Context Update]
end
subgraph "3. MCP Server"
M1[MCP Server mobile_mcp_server.py]
M2[MCP Tools tap, swipe, type, etc.]
end
subgraph "4. Configuration"
C1[third_party.yaml Agent Config]
C2[devices.yaml Device Registry]
C3[Prompt Templates LLM Prompts]
end
subgraph "5. Client"
CL1[Device Client client.py]
CL2[MCP Manager mcp_server_manager.py]
end
A1 --> A2
A2 --> S1 & S2 & S3 & S4
S3 --> M1
M1 --> M2
A1 -.reads.-> C1
CL1 --> CL2
CL2 --> M1
CL1 -.reads.-> C2
A2 -.uses.-> C3
style A1 fill:#c8e6c9
style A2 fill:#c8e6c9
style A3 fill:#c8e6c9
style M1 fill:#e1f5ff
style CL1 fill:#e1f5ff
```
**Implementation Checklist**:
- [ ] **Agent Class**: Define `MobileAgent` inheriting from `CustomizedAgent`
- [ ] **Processor**: Create `MobileAgentProcessor` with custom strategies
- [ ] **State Manager**: Implement `MobileAgentStateManager` and states
- [ ] **Strategies**: Build platform-specific LLM and action strategies
- [ ] **MCP Server**: Develop MCP server with platform tools
- [ ] **Prompter**: Create custom prompter for mobile context
- [ ] **Client Setup**: Configure client to run on mobile device
- [ ] **Configuration**: Add agent config to `third_party.yaml`
- [ ] **Device Registry**: Register device in `devices.yaml`
- [ ] **Prompt Templates**: Write LLM prompt templates
---
## Tutorial Roadmap
This tutorial is split into **6 detailed guides**:
### 📘 Part 1: [Core Components](core_components.md)
Learn to implement the **server-side components**:
- Agent Class definition
- Processor and strategies
- State Manager and states
- Prompter for LLM interaction
**Time**: 45 minutes
**Difficulty**: ⭐⭐⭐
---
### 📘 Part 2: [MCP Server Development](mcp_server.md)
Create a **platform-specific MCP server**:
- MCP server architecture
- Defining MCP tools
- Command execution logic
- Error handling and validation
**Time**: 30 minutes
**Difficulty**: ⭐⭐
---
### 📘 Part 3: [Client Configuration](client_setup.md)
Set up the **device client**:
- Client initialization
- MCP server manager integration
- WebSocket connection setup
- Platform detection
**Time**: 20 minutes
**Difficulty**: ⭐⭐
---
### 📘 Part 4: [Configuration & Deployment](configuration.md)
Configure and deploy your agent:
- `third_party.yaml` configuration
- `devices.yaml` device registration
- Prompt template creation
- Galaxy integration
**Time**: 25 minutes
**Difficulty**: ⭐⭐
---
### 📘 Part 5: [Testing & Debugging](testing.md)
Test and debug your implementation:
- Unit testing strategies
- Integration testing
- Debugging techniques
- Common issues and solutions
**Time**: 30 minutes
**Difficulty**: ⭐⭐⭐
---
### 📘 Part 6: [Complete Example: MobileAgent](example_mobile_agent.md)
**Hands-on walkthrough** creating `MobileAgent`:
- Step-by-step implementation
- Android/iOS platform specifics
- UI Automator integration
- Complete working example
**Time**: 60 minutes
**Difficulty**: ⭐⭐⭐⭐
---
## Quick Start Guide
For experienced developers, here's a **minimal implementation checklist**:
### 1️⃣ Create Agent Class
```python
# ufo/agents/agent/customized_agent.py
@AgentRegistry.register(
agent_name="MobileAgent",
third_party=True,
processor_cls=MobileAgentProcessor
)
class MobileAgent(CustomizedAgent):
def __init__(self, name, main_prompt, example_prompt):
super().__init__(name, main_prompt, example_prompt,
process_name=None, app_root_name=None, is_visual=None)
self._blackboard = Blackboard()
self.set_state(self.default_state)
self._context_provision_executed = False
@property
def default_state(self):
return ContinueMobileAgentState()
```
### 2️⃣ Create Processor
```python
# ufo/agents/processors/customized/customized_agent_processor.py
class MobileAgentProcessor(CustomizedProcessor):
def _setup_strategies(self):
# Compose multiple data collection strategies
self.strategies[ProcessingPhase.DATA_COLLECTION] = ComposedStrategy(
strategies=[
MobileScreenshotCaptureStrategy(fail_fast=True),
MobileAppsCollectionStrategy(fail_fast=False),
MobileControlsCollectionStrategy(fail_fast=False),
],
name="MobileDataCollectionStrategy",
fail_fast=True,
)
self.strategies[ProcessingPhase.LLM_INTERACTION] = (
MobileLLMInteractionStrategy(fail_fast=True)
)
self.strategies[ProcessingPhase.ACTION_EXECUTION] = (
MobileActionExecutionStrategy(fail_fast=False)
)
self.strategies[ProcessingPhase.MEMORY_UPDATE] = (
AppMemoryUpdateStrategy(fail_fast=False)
)
```
### 3️⃣ Create MCP Server
```python
# ufo/client/mcp/http_servers/mobile_mcp_server.py
def create_mobile_mcp_server(host="localhost", port=8020):
mcp = FastMCP("Mobile MCP Server", stateless_http=False,
json_response=True, host=host, port=port)
@mcp.tool()
async def tap_element(x: int, y: int) -> dict:
# Execute tap via ADB or platform API
pass
mcp.run(transport="streamable-http")
```
### 4️⃣ Configure Agent
```yaml
# config/ufo/third_party.yaml
ENABLED_THIRD_PARTY_AGENTS: ["MobileAgent"]
THIRD_PARTY_AGENT_CONFIG:
MobileAgent:
VISUAL_MODE: True
AGENT_NAME: "MobileAgent"
APPAGENT_PROMPT: "ufo/prompts/third_party/mobile_agent.yaml"
APPAGENT_EXAMPLE_PROMPT: "ufo/prompts/third_party/mobile_agent_example.yaml"
INTRODUCTION: "MobileAgent controls Android/iOS devices..."
```
### 5️⃣ Register Device
```yaml
# config/galaxy/devices.yaml
devices:
- device_id: "mobile_agent_1"
server_url: "ws://localhost:5010/ws"
os: "android"
capabilities: ["ui_automation", "app_testing"]
metadata:
device_model: "Pixel 6"
android_version: "13"
max_retries: 5
```
### 6️⃣ Start Server & Client
```bash
# Terminal 1: Start Agent Server
python -m ufo.server.app --port 5010
# Terminal 2: Start Device Client
python -m ufo.client.client \
--ws --ws-server ws://localhost:5010/ws \
--client-id mobile_agent_1 \
--platform android
# Terminal 3: Start MCP Server (on device or accessible endpoint)
python -m ufo.client.mcp.http_servers.mobile_mcp_server --port 8020
```
---
## Next Steps
**Ready to Build Your Device Agent?**
Start with Part 1: [Core Components →](core_components.md)
Or jump to a specific topic:
- [MCP Server Development](mcp_server.md)
- [Configuration & Deployment](configuration.md)
- [Complete Example: MobileAgent](example_mobile_agent.md)
---
## Related Documentation
- **[Agent Architecture](../../infrastructure/agents/overview.md)** - Three-layer architecture deep dive
- **[Linux Agent Quick Start](../../getting_started/quick_start_linux.md)** - LinuxAgent deployment guide
- **[Server Overview](../../server/overview.md)** - Server-side orchestration
- **[Client Overview](../../client/overview.md)** - Client-side execution
- **[MCP Overview](../../mcp/overview.md)** - Model Context Protocol
- **[AIP Protocol](../../aip/overview.md)** - Agent Interaction Protocol
- **[Creating Third-Party Agents](../creating_third_party_agents.md)** - Third-party agent tutorial
---
## Summary
**Key Takeaways**:
- **Device Agents** control entire platforms (Windows, Linux, Mobile)
- **Server-Client Architecture** separates reasoning from execution
- **Three-Layer Design** provides modular, extensible framework
- **LinuxAgent** is the best reference implementation
- **6-Part Tutorial** covers all aspects of device agent creation
- **MCP Integration** enables platform-specific command execution
- **Galaxy Integration** supports multi-device orchestration
**Ready to build your first device agent? Let's get started!** 🚀
---
### Documents Docs Ufo2 App Agent Overview (`documents/docs/ufo2/app_agent/overview.md`)
# AppAgent: Application Execution Agent
**AppAgent** is the core execution runtime in UFO, responsible for carrying out individual subtasks within a specific Windows application. Each AppAgent functions as an isolated, application-specialized worker process launched and orchestrated by the central HostAgent.
---
## What is AppAgent?
AppAgent Architecture: Application-specialized worker process for subtask execution
**AppAgent** operates as a **child agent** under the HostAgent's orchestration:
- **Isolated Runtime**: Each AppAgent is dedicated to a single Windows application
- **Subtask Executor**: Executes specific subtasks delegated by HostAgent
- **Application Expert**: Tailored with deep knowledge of the target app's API surface, control semantics, and domain logic
- **Hybrid Execution**: Leverages both GUI automation and API-based actions through MCP commands
Unlike monolithic Computer-Using Agents (CUAs) that treat all GUI contexts uniformly, each AppAgent is tailored to a single application and operates with specialized knowledge of its interface and capabilities.
---
## Core Responsibilities
```mermaid
graph TB
subgraph "AppAgent Core Responsibilities"
SR[Sense: Capture Application State]
RE[Reason: Analyze Next Action]
EX[Execute: GUI or API Action]
RP[Report: Write Results to Blackboard]
end
SR --> RE
RE --> EX
EX --> RP
RP --> SR
style SR fill:#e3f2fd
style RE fill:#fff3e0
style EX fill:#f1f8e9
style RP fill:#fce4ec
```
| Responsibility | Description | Example |
|---------------|-------------|---------|
| **State Sensing** | Capture application UI, detect controls, understand current state | Screenshot Word window → Detect 50 controls → Annotate UI elements |
| **Reasoning** | Analyze state and determine next action using LLM | "Table visible with Export button [12] → Click to export data" |
| **Action Execution** | Execute GUI clicks or API calls via MCP commands | `click_input(control_id=12)` or `execute_word_command("export_table")` |
| **Result Reporting** | Write execution results to shared Blackboard | Write extracted data to `subtask_result_1` for HostAgent |
---
## ReAct-Style Control Loop
Upon receiving a subtask and execution context from the HostAgent, the AppAgent initializes a **ReAct-style control loop** where it iteratively:
1. **Observes** the current application state (screenshot + control detection)
2. **Thinks** about the next step (LLM reasoning)
3. **Acts** by executing either a GUI or API-based action (MCP commands)
```mermaid
sequenceDiagram
participant HostAgent
participant AppAgent
participant Application
participant Blackboard
HostAgent->>AppAgent: Delegate subtask "Extract table from Word"
loop ReAct Loop
AppAgent->>Application: Observe (screenshot + controls)
Application-->>AppAgent: UI state
AppAgent->>AppAgent: Think (LLM reasoning)
AppAgent->>Application: Act (click/API call)
Application-->>AppAgent: Action result
end
AppAgent->>Blackboard: Write result
AppAgent->>HostAgent: Return control
```
The MCP command system enables **reliable control** over dynamic and complex UIs by favoring structured API commands whenever available, while retaining fallback to GUI-based interaction commands when necessary.
---
## Execution Architecture
### Finite State Machine
AppAgent uses a finite state machine with 7 states to control its execution flow:
- **CONTINUE**: Continue processing the current subtask
- **FINISH**: Successfully complete the subtask
- **ERROR**: Encounter an unrecoverable error
- **FAIL**: Fail to complete the subtask
- **PENDING**: Wait for user input or clarification
- **CONFIRM**: Request user confirmation for sensitive actions
- **SCREENSHOT**: Capture and re-annotate the application screenshot
**State Details**: See [State Machine Documentation](state.md) for complete state definitions and transitions.
### 4-Phase Processing Pipeline
Each execution round follows a 4-phase pipeline:
```mermaid
graph LR
DC[Phase 1: DATA_COLLECTION Screenshot + Controls] --> LLM[Phase 2: LLM_INTERACTION Reasoning]
LLM --> AE[Phase 3: ACTION_EXECUTION GUI/API Action]
AE --> MU[Phase 4: MEMORY_UPDATE Record Action]
style DC fill:#e1f5ff
style LLM fill:#fff4e6
style AE fill:#e8f5e9
style MU fill:#fce4ec
```
**Strategy Details**: See [Processing Strategy Documentation](strategy.md) for complete pipeline implementation.
---
## Hybrid GUI–API Execution
AppAgent executes actions through the **MCP (Model-Context Protocol) command system**, which provides a unified interface for both GUI automation and native API calls:
```python
# GUI-based command (fallback)
command = Command(
tool_name="click_input",
parameters={"control_id": "12", "button": "left"}
)
await command_dispatcher.execute_commands([command])
# API-based command (preferred when available)
command = Command(
tool_name="word_export_table",
parameters={"format": "csv", "path": "output.csv"}
)
await command_dispatcher.execute_commands([command])
```
**Implementation**: See [Hybrid Actions](../core_features/hybrid_actions.md) for details on the MCP command system.
---
## Knowledge Enhancement
AppAgent is enhanced with **Retrieval Augmented Generation (RAG)** from heterogeneous sources:
| Knowledge Source | Purpose | Configuration |
|-----------------|---------|---------------|
| **Help Documents** | Application-specific documentation | [Learning from Help Documents](../core_features/knowledge_substrate/learning_from_help_document.md) |
| **Bing Search** | Latest information and updates | [Learning from Bing Search](../core_features/knowledge_substrate/learning_from_bing_search.md) |
| **Self-Demonstrations** | Successful action trajectories | [Experience Learning](../core_features/knowledge_substrate/experience_learning.md) |
| **Human Demonstrations** | Expert-provided workflows | [Learning from Demonstrations](../core_features/knowledge_substrate/learning_from_demonstration.md) |
**Knowledge Substrate Overview**: See [Knowledge Substrate](../core_features/knowledge_substrate/overview.md) for the complete RAG architecture.
---
## Command System
AppAgent executes actions through the **MCP (Model-Context Protocol)** command system:
**Application-Level Commands**:
- `capture_window_screenshot` - Capture application window
- `get_control_info` - Detect UI controls via UIA/OmniParser
- `click_input` - Click on UI control
- `set_edit_text` - Type text into input field
- `annotation` - Annotate screenshot with control labels
**Command Details**: See [Command System Documentation](commands.md) for complete command reference.
---
## Control Detection Backends
AppAgent supports multiple control detection backends for comprehensive UI understanding:
**UIA (UI Automation):**
Native Windows UI Automation API for standard controls
- ✅ Fast and accurate
- ✅ Works with most Windows applications
- ❌ May miss custom controls
**OmniParser (Visual Detection):**
Vision-based grounding model for visual elements
- ✅ Detects icons, images, custom controls
- ✅ Works with web content
- ❌ Requires external service
**Hybrid (UIA + OmniParser):**
Best of both worlds - maximum coverage
- ✅ Native controls + visual elements
- ✅ Comprehensive UI understanding
**Control Detection Details**: See [Control Detection Overview](../core_features/control_detection/overview.md).
---
## Input and Output
### AppAgent Input
| Input | Description | Source |
|-------|-------------|--------|
| **User Request** | Original user request in natural language | HostAgent |
| **Sub-Task** | Specific subtask to execute | HostAgent delegation |
| **Application Context** | Target app name, window info | HostAgent |
| **Control Information** | Detected UI controls with labels | Data collection phase |
| **Screenshots** | Clean, annotated, previous step images | Data collection phase |
| **Blackboard** | Shared memory for inter-agent communication | Global context |
| **Retrieved Knowledge** | Help docs, demos, search results | RAG system |
### AppAgent Output
| Output | Description | Consumer |
|--------|-------------|----------|
| **Observation** | Current UI state description | LLM context |
| **Thought** | Reasoning about next action | Execution log |
| **ControlLabel** | Selected control to interact with | Action executor |
| **Function** | MCP command to execute (click_input, set_edit_text, etc.) | Command dispatcher |
| **Args** | Command parameters | Command dispatcher |
| **Status** | Agent state (CONTINUE, FINISH, etc.) | State machine |
| **Blackboard Update** | Execution results | HostAgent |
**Example Output**:
```json
{
"Observation": "Word document with table, Export button at [12]",
"Thought": "Click Export to extract table data",
"ControlLabel": "12",
"Function": "click_input",
"Args": {"button": "left"},
"Status": "CONTINUE"
}
```
---
## Related Documentation
**Detailed Documentation:**
- **[State Machine](state.md)**: Complete FSM with state definitions and transitions
- **[Processing Strategy](strategy.md)**: 4-phase pipeline implementation details
- **[Command System](commands.md)**: Application-level MCP commands reference
**Core Features:**
- **[Hybrid Actions](../core_features/hybrid_actions.md)**: MCP command system for GUI–API execution
- **[Control Detection](../core_features/control_detection/overview.md)**: UIA and visual detection
- **[Knowledge Substrate](../core_features/knowledge_substrate/overview.md)**: RAG system overview
**Tutorials:**
- **[Creating AppAgent](../../tutorials/creating_app_agent/overview.md)**: Step-by-step guide
- **[Help Document Provision](../../tutorials/creating_app_agent/help_document_provision.md)**: Add help docs
- **[Demonstration Provision](../../tutorials/creating_app_agent/demonstration_provision.md)**: Add demos
- **[Wrapping App-Native API](../../tutorials/creating_app_agent/warpping_app_native_api.md)**: Integrate APIs
---
## API Reference
:::agents.agent.app_agent.AppAgent
---
## Summary
**AppAgent Key Characteristics:**
✅ **Application-Specialized Worker**: Dedicated to single Windows application
✅ **ReAct Control Loop**: Iterative observe → think → act execution
✅ **Hybrid Execution**: GUI automation + API calls via MCP commands
✅ **7-State FSM**: Robust state management for execution control
✅ **4-Phase Pipeline**: Structured data collection → reasoning → action → memory
✅ **Knowledge-Enhanced**: RAG from docs, demos, and search
✅ **Orchestrated by HostAgent**: Child agent in hierarchical architecture
**Next Steps:**
1. **Deep Dive**: Read [State Machine](state.md) and [Processing Strategy](strategy.md) for implementation details
2. **Learn Features**: Explore [Core Features](../core_features/hybrid_actions.md) for advanced capabilities
3. **Hands-On Tutorial**: Follow [Creating AppAgent](../../tutorials/creating_app_agent/overview.md) guide
---
### Documents Docs Ufo2 Dataflow Overview (`documents/docs/ufo2/dataflow/overview.md`)
# Introduction
This repository contains the implementation of the **Data Collection** process for training the **Large Action Models** (LAMs) in the paper of [Large Action Models: From Inception to Implementation](https://arxiv.org/abs/2412.10047). The **Data Collection** process is designed to streamline task processing, ensuring that all necessary steps are seamlessly integrated from initialization to execution. This module is part of the [**UFO**](https://arxiv.org/abs/2402.07939) project.
# Dataflow
Dataflow uses UFO to implement `instantiation`, `execution`, and `dataflow` for a given task, with options for batch processing and single processing.
1. **[Instantiation](./instantiation.md)**: Instantiation refers to the process of setting up and preparing a task for execution. This step typically involves `choosing template`, `prefill` and `filter`.
2. **[Execution](./execution.md)**: Execution is the actual process of running the task. This step involves carrying out the actions or operations specified by the `Instantiation`. And after execution, an evaluate agent will evaluate the quality of the whole execution process.
3. **Dataflow**: Dataflow is the overarching process that combines **instantiation** and **execution** into a single pipeline. It provides an end-to-end solution for processing tasks, ensuring that all necessary steps (from initialization to execution) are seamlessly integrated.
You can use `instantiation` and `execution` independently if you only need to perform one specific part of the process. When both steps are required for a task, the `dataflow` process streamlines them, allowing you to execute tasks from start to finish in a single pipeline.
The overall processing of dataflow is as below. Given a task-plan data, the LLMwill instantiatie the task-action data, including choosing template, prefill, filter.
## How To Use
### 1. Install Packages
You should install the necessary packages in the UFO root folder:
```bash
pip install -r requirements.txt
```
### 2. Configure the LLMs
Before running dataflow, you need to provide your LLM configurations **individually for PrefillAgent and FilterAgent**. You can create your own config file `dataflow/config/config.yaml`, by copying the `dataflow/config/config.yaml.template` and editing config for **PREFILL_AGENT** and **FILTER_AGENT** as follows:
#### OpenAI
```bash
VISUAL_MODE: True, # Whether to use the visual mode
API_TYPE: "openai" , # The API type, "openai" for the OpenAI API.
API_BASE: "https://api.openai.com/v1/chat/completions", # The the OpenAI API endpoint.
API_KEY: "sk-", # The OpenAI API key, begin with sk-
API_VERSION: "2024-02-15-preview", # "2024-02-15-preview" by default
API_MODEL: "gpt-4-vision-preview", # The only OpenAI model
```
#### Azure OpenAI (AOAI)
```bash
VISUAL_MODE: True, # Whether to use the visual mode
API_TYPE: "aoai" , # The API type, "aoai" for the Azure OpenAI.
API_BASE: "YOUR_ENDPOINT", # The AOAI API address. Format: https://{your-resource-name}.openai.azure.com
API_KEY: "YOUR_KEY", # The aoai API key
API_VERSION: "2024-02-15-preview", # "2024-02-15-preview" by default
API_MODEL: "gpt-4-vision-preview", # The only OpenAI model
API_DEPLOYMENT_ID: "YOUR_AOAI_DEPLOYMENT", # The deployment id for the AOAI API
```
You can also non-visial model (e.g., GPT-4) for each agent, by setting `VISUAL_MODE: False` and proper `API_MODEL` (openai) and `API_DEPLOYMENT_ID` (aoai).
#### Non-Visual Model Configuration
You can utilize non-visual models (e.g., GPT-4) for each agent by configuring the following settings in the `config.yaml` file:
- ``VISUAL_MODE: False # To enable non-visual mode.``
- Specify the appropriate `API_MODEL` (OpenAI) and `API_DEPLOYMENT_ID` (AOAI) for each agent.
Ensure you configure these settings accurately to leverage non-visual models effectively.
#### Other Configurations
`config_dev.yaml` specifies the paths of relevant files and contains default settings. The match strategy for the window match and control filter supports options: `'contains'`, `'fuzzy'`, and `'regex'`, allowing flexible matching strategy for users. The `MAX_STEPS` is the max step for the execute_flow, which can be set by users.
!!!note
The specific implementation and invocation method of the matching strategy can refer to [windows_app_env](./windows_app_env.md).
!!!note
**BE CAREFUL!** If you are using GitHub or other open-source tools, do not expose your `config.yaml` online, as it contains your private keys.
### 3. Prepare Files
Certain files need to be prepared before running the task.
#### 3.1. Tasks as JSON
The tasks that need to be instantiated should be organized in a folder of JSON files, with the default folder path set to dataflow `/tasks `. This path can be changed in the `dataflow/config/config.yaml` file, or you can specify it in the terminal, as mentioned in **4. Start Running**. For example, a task stored in `dataflow/tasks/prefill/` may look like this:
```json
{
// The app you want to use
"app": "word",
// A unique ID to distinguish different tasks
"unique_id": "1",
// The task and steps to be instantiated
"task": "Type 'hello' and set the font type to Arial",
"refined_steps": [
"Type 'hello'",
"Set the font to Arial"
]
}
```
#### 3.2. Templates and Descriptions
You should place an app file as a reference for instantiation in a folder named after the app.
For example, if you have `template1.docx` for Word, it should be located at `dataflow/templates/word/template1.docx`.
Additionally, for each app folder, there should be a `description.json` file located at `dataflow/templates/word/description.json`, which describes each template file in detail. It may look like this:
```json
{
"template1.docx": "A document with a rectangle shape",
"template2.docx": "A document with a line of text"
}
```
If a `description.json` file is not present, one template file will be selected at random.
#### 3.3. Final Structure
Ensure the following files are in place:
- [X] JSON files to be instantiated
- [X] Templates as references for instantiation
- [X] Description file in JSON format
The structure of the files can be:
```txt
dataflow/
|
├── tasks
│ └── prefill
│ ├── bulleted.json
│ ├── delete.json
│ ├── draw.json
│ ├── macro.json
│ └── rotate.json
├── templates
│ └── word
│ ├── description.json
│ ├── template1.docx
│ ├── template2.docx
│ ├── template3.docx
│ ├── template4.docx
│ ├── template5.docx
│ ├── template6.docx
│ └── template7.docx
└── ...
```
### 4. Start Running
After finishing the previous steps, you can use the following commands in the command line. We provide single / batch process, for which you need to give the single file path / folder path. Determine the type of path provided by the user and automatically decide whether to process a single task or batch tasks.
Also, you can choose to use `instantiation` / `execution` sections individually, or use them as a whole section, which is named as `dataflow`.
The default task hub is set to be `"TASKS_HUB"` in `dataflow/config_dev.yaml`.
* Dataflow Task:
```bash
python -m dataflow -dataflow --task_path path_to_task_file
```
* Instantiation Task:
```bash
python -m dataflow -instantiation --task_path path_to_task_file
```
* Execution Task:
```bash
python -m dataflow -execution --task_path path_to_task_file
```
!!! note
1. Users should be careful to save the original files while using this project; otherwise, the files will be closed when the app is shut down.
2. After starting the project, users should not close the app window while the program is taking screenshots.
## Workflow
### Instantiation
There are three key steps in the instantiation process:
1. `Choose a template` file according to the specified app and instruction.
2. `Prefill` the task using the current screenshot.
3. `Filter` the established task.
Given the initial task, the dataflow first choose a template (`Phase 1`), the prefill the initial task based on word envrionment to obtain task-action data (`Phase 2`). Finnally, it will filter the established task to evaluate the quality of task-action data. (`Phase 3`)
!!! note
The more detailed code design documentation for instantiation can be found in [instantiation](./instantiation.md).
### Execution
The instantiated plans will be executed by a execute task. After execution, evalution agent will evaluation the quality of the entire execution process.
!!! note
The more detailed code design documentation for execution can be found in [execution](./execution.md).
## Result
The results will be saved in the `results\` directory under `instantiation`, `execution`, and `dataflow`, and will be further stored in subdirectories based on the execution outcomes.
!!! note
The more detailed information of result can be found in [result](./result.md).
## Quick Start
We prepare two cases to show the dataflow, which can be found in `dataflow\tasks\prefill`. So after installing required packages, you can type the following command in the command line:
```
python -m dataflow -dataflow
```
And you can see the hints showing in the terminal, which means the dataflow is working.
### Structure of related files
After the two tasks are finished, the task and output files would appear as follows:
```bash
UFO/
├── dataflow/
│ └── results/
│ ├── saved_document/ # Directory for saved documents
│ │ ├── bulleted.docx # Result of the "bulleted" task
│ │ └── rotate.docx # Result of the "rotate" task
│ ├── dataflow/ # Dataflow results directory
│ │ ├── execution_pass/ # Successfully executed tasks
│ │ │ ├── bulleted.json # Execution result for the "bulleted" task
│ │ │ ├── rotate.json # Execution result for the "rotate" task
│ │ │ └── ...
└── ...
```
The specific results can be referenced in the [result](./result.md) in JSON format along with example data.
### Log files
The corresponding logs can be found in the directories `logs/bulleted` and `logs/rotate`, as shown below. Detailed logs for each workflow are recorded, capturing every step of the execution process.
# Reference
### AppEnum
::: data_flow_controller.AppEnum
### TaskObject
::: data_flow_controller.TaskObject
### DataFlowController
::: data_flow_controller.DataFlowController
---
### Documents Docs Ufo2 Host Agent Overview (`documents/docs/ufo2/host_agent/overview.md`)
# HostAgent: Desktop Orchestrator
**HostAgent** serves as the centralized control plane of UFO². It interprets user-specified goals, decomposes them into structured subtasks, instantiates and dispatches AppAgent modules, and coordinates their progress across the system. HostAgent provides system-level services for introspection, planning, application lifecycle management, and multi-agent synchronization.
---
## Architecture Overview
Operating atop the native Windows substrate, HostAgent monitors active applications, issues shell commands to spawn new processes as needed, and manages the creation and teardown of application-specific AppAgent instances. All coordination occurs through a persistent state machine, which governs the transitions across execution phases.
Figure: HostAgent architecture showing the finite state machine, processing pipeline, and interactions with AppAgents through the Blackboard pattern.
---
## Core Responsibilities
### Task Decomposition
Given a user's natural language input, HostAgent identifies the underlying task goal and decomposes it into a dependency-ordered subtask graph.
**Example:** User request "Extract data from Word and create an Excel chart" becomes:
1. Extract table from Word document
2. Create chart in Excel with extracted data
Figure: HostAgent decomposes user requests into sequential subtasks, assigns each to the appropriate application, and orchestrates AppAgents to complete them in dependency order.
### Application Lifecycle Management
For each subtask, HostAgent inspects system process metadata (via UIA APIs) to determine whether the target application is running. If not, it launches the program and registers it with the runtime.
### AppAgent Instantiation
HostAgent spawns the corresponding AppAgent for each active application, providing it with task context, memory references, and relevant toolchains (e.g., APIs, documentation).
### Task Scheduling and Control
The global execution plan is serialized into a finite state machine (FSM), allowing HostAgent to enforce execution order, detect failures, and resolve dependencies across agents. See **[State Machine Details](state.md)** for the FSM architecture.
### Shared State Communication
HostAgent reads from and writes to a global blackboard, enabling inter-agent communication and system-level observability for debugging and replay.
---
## Key Characteristics
- **Scope**: Desktop-level orchestrator (system-wide, not application-specific)
- **Lifecycle**: Single instance per session, persists throughout task execution
- **Hierarchy**: Parent agent that manages multiple child AppAgents
- **Communication**: Owns and coordinates the shared Blackboard
- **Control**: 7-state finite state machine with 4-phase processing pipeline
---
## Execution Workflow
```mermaid
sequenceDiagram
participant User
participant HostAgent
participant Blackboard
participant AppAgent1
participant AppAgent2
User->>HostAgent: "Extract Word table, create Excel chart"
HostAgent->>HostAgent: Decompose into subtasks
HostAgent->>Blackboard: Write subtask 1
HostAgent->>AppAgent1: Create/Get Word AppAgent
AppAgent1->>AppAgent1: Execute Word task
AppAgent1->>Blackboard: Write result 1
AppAgent1-->>HostAgent: Return FINISH
HostAgent->>Blackboard: Read result 1
HostAgent->>Blackboard: Write subtask 2
HostAgent->>AppAgent2: Create/Get Excel AppAgent
AppAgent2->>Blackboard: Read result 1
AppAgent2->>AppAgent2: Execute Excel task
AppAgent2->>Blackboard: Write result 2
AppAgent2-->>HostAgent: Return FINISH
HostAgent->>HostAgent: Verify completion
HostAgent-->>User: Task completed
```
---
## Deep Dive Topics
- **[State Machine](state.md)**: 7-state FSM architecture and transitions
- **[Processing Strategy](strategy.md)**: 4-phase processing pipeline
- **[Command System](commands.md)**: Desktop-level MCP commands
---
## Input and Output
### HostAgent Input
| Input | Description | Type |
|-------|-------------|------|
| User Request | Natural language task description | String |
| Application Information | Active application metadata | List of Dicts |
| Desktop Screenshots | Visual context of desktop state | Image |
| Previous Sub-Tasks | Completed subtask history | List of Dicts |
| Previous Plan | Planned future subtasks | List of Strings |
| Blackboard | Shared memory space | Dictionary |
### HostAgent Output
| Output | Description | Type |
|--------|-------------|------|
| Observation | Desktop screenshot analysis | String |
| Thought | Reasoning process | String |
| Current Sub-Task | Active subtask description | String |
| Message | Information for AppAgent | String |
| ControlLabel | Selected application index | String |
| ControlText | Selected application name | String |
| Plan | Future subtask sequence | List of Strings |
| Status | Agent state (CONTINUE/ASSIGN/FINISH/etc.) | String |
| Comment | User-facing information | String |
| Questions | Clarification requests | List of Strings |
| Bash | System command to execute | String |
**Example Output:**
```json
{
"Observation": "Desktop shows Microsoft Word with document open containing a table",
"Thought": "User wants to extract data from Word first",
"Current Sub-Task": "Extract the table data from the document",
"Message": "Starting data extraction from Word document",
"ControlLabel": "0",
"ControlText": "Microsoft Word - Document1",
"Plan": ["Extract table from Word", "Create chart in Excel"],
"Status": "ASSIGN",
"Comment": "Delegating table extraction to Word AppAgent",
"Questions": [],
"Bash": ""
}
```
---
## Related Documentation
**Architecture & Design:**
- **[Windows Agent Overview](../overview.md)**: Module architecture and hierarchy
- **[AppAgent](../app_agent/overview.md)**: Application automation agent
- **[Blackboard](../../infrastructure/agents/design/blackboard.md)**: Inter-agent communication
- **[Memory System](../../infrastructure/agents/design/memory.md)**: Execution history
**Configuration:**
- **[Configuration System Overview](../../configuration/system/overview.md)**: System configuration structure
- **[Agents Configuration](../../configuration/system/agents_config.md)**: LLM and agent settings
- **[System Configuration](../../configuration/system/system_config.md)**: Runtime and execution settings
- **[MCP Reference](../../configuration/system/mcp_reference.md)**: MCP server configuration
**System Integration:**
- **[Session Management](../../infrastructure/modules/session.md)**: Session lifecycle
- **[Round Management](../../infrastructure/modules/round.md)**: Execution rounds
---
## API Reference
:::agents.agent.host_agent.HostAgent
---
## Summary
HostAgent is the desktop-level orchestrator that:
- Decomposes tasks and coordinates AppAgents
- Operates at system level, not application level
- Uses a 7-state FSM: CONTINUE → ASSIGN → AppAgent → CONTINUE → FINISH
- Executes a 4-phase pipeline: DATA_COLLECTION → LLM → ACTION → MEMORY
- Creates, caches, and reuses AppAgent instances
- Provides shared Blackboard memory for all agents
- Maintains single instance per session managing multiple AppAgents
**Next Steps:**
1. Read [State Machine](state.md) for FSM details
2. Read [Processing Strategy](strategy.md) for pipeline architecture
3. Read [Command System](commands.md) for available desktop operations
4. Read [AppAgent](../app_agent/overview.md) for application-level execution
---
### Documents Docs Ufo2 Prompts Overview (`documents/docs/ufo2/prompts/overview.md`)
# Prompts
All prompts used in UFO are stored in the `ufo/prompts` directory. The folder structure is as follows:
```
📦prompts
┣ 📂demonstration # Prompts for summarizing human demonstrations
┣ 📂evaluation # Prompts for the EvaluationAgent
┣ 📂examples # Demonstration examples for in-context learning
┣ 📂nonvisual # Examples for non-visual LLMs
┗ 📂visual # Examples for visual LLMs
┣ 📂experience # Prompts for summarizing agent self-experience
┣ 📂share # Shared prompt templates
┗ 📂base # Basic version of shared prompts
┣ 📜api.yaml # Basic API prompt
┣ 📜app_agent.yaml # Basic AppAgent prompt template
┗ 📜host_agent.yaml # Basic HostAgent prompt template
┗ 📂third_party # Third-party integration prompts (e.g., Linux agents)
```
Visual LLMs can process screenshots while non-visual LLMs rely on text-only control information.
## Agent Prompts
Agent prompts are constructed from the following components:
| Component | Description | Source |
| --- | --- | --- |
| **Basic Template** | Base template with system and user roles | YAML files in `share/base/` |
| **API Documentation** | Skills and APIs available to the agent | Dynamically generated from MCP tools |
| **Examples** | In-context learning demonstrations | YAML files in `examples/visual/` or `examples/nonvisual/` |
You can find the base templates in the `share/base` directory.
## How Prompts Are Constructed
The agent's `Prompter` class is responsible for:
1. **Loading** YAML templates from the file system
2. **Formatting** API documentation from available tools
3. **Selecting** appropriate examples based on model type (visual/nonvisual)
4. **Combining** all components into a structured message list for the LLM
5. **Injecting** runtime context (observations, screenshots, retrieved knowledge)
Each agent type has its own specialized Prompter:
- **HostAgentPrompter**: Desktop-level orchestration with third-party agent support
- **AppAgentPrompter**: Application-level interactions with multi-action capabilities
- **EvaluationAgentPrompter**: Task evaluation and success assessment
For comprehensive details about the Prompter class architecture, template loading, and prompt construction workflow, see the [Prompter documentation](../../infrastructure/agents/design/prompter.md).
---
### Documents Docs Ufo2 Core Features Control Detection Overview (`documents/docs/ufo2/core_features/control_detection/overview.md`)
# Control Detection
We support different control detection methods to detect controls in the application to accommodate both standard (UIA) and custom controls (Visual).
## Detection Methods
| Method | Description | Use Case |
|--------|-------------|----------|
| [**UIA**](./uia_detection.md) | Uses Windows UI Automation framework to detect standard controls. Provides APIs to access and manipulate UI elements in Windows applications. | Standard Windows applications with native controls |
| [**Visual (OmniParser)**](./visual_detection.md) | Uses OmniParser vision-based detection to identify custom controls through computer vision techniques based on visual appearance. | Applications with custom controls, icons, or visual elements not accessible via UIA |
| [**Hybrid**](./hybrid_detection.md) | Combines both UIA and OmniParser detection methods. Merges results from both approaches, removing duplicates based on IoU overlap. | Maximum coverage for applications with both standard and custom controls |
## Configuration
Configure the control detection method by setting the `CONTROL_BACKEND` parameter in `config/ufo/system.yaml`:
```yaml
# Use UIA only (default, recommended)
CONTROL_BACKEND: ["uia"]
# Use OmniParser only
CONTROL_BACKEND: ["omniparser"]
# Use hybrid mode (UIA + OmniParser)
CONTROL_BACKEND: ["uia", "omniparser"]
```
See [System Configuration](../../../configuration/system/system_config.md#control-backend) for detailed configuration options.
---
### Documents Docs Ufo2 Core Features Knowledge Substrate Overview (`documents/docs/ufo2/core_features/knowledge_substrate/overview.md`)
# Knowledge Substrate
UFO provides versatile mechanisms to enhance the AppAgent's capabilities through RAG (Retrieval-Augmented Generation) and other knowledge retrieval techniques. These mechanisms improve the AppAgent's task understanding, plan quality, and interaction efficiency with applications.
## Supported Knowledge Sources
UFO currently supports the following knowledge retrieval methods:
| Knowledge Source | Description |
|------------------|-------------|
| [Help Documents](./learning_from_help_document.md) | Retrieve knowledge from offline help documentation indexed for specific applications. |
| [Bing Search](./learning_from_bing_search.md) | Search online information via Bing to obtain up-to-date knowledge. |
| [Self-Experience](./experience_learning.md) | Learn from the agent's own successful task execution history. |
| [User Demonstrations](./learning_from_demonstration.md) | Learn from action trajectories demonstrated by users. |
## Context Provision
UFO provides knowledge to the AppAgent through the `context_provision` method defined in the `AppAgent` class:
```
/* Detailed source-code truncated for AI context efficiency. */
```
The `context_provision` method loads various knowledge retrievers based on the configuration settings in `config.yaml`:
- **Offline document retriever**: Loads indexed help documentation for the target application
- **Online search retriever**: Creates a Bing search indexer when a search request is provided
- **Experience retriever**: Loads the agent's historical successful experiences
- **Demonstration retriever**: Loads user-demonstrated action trajectories
- **MCP context**: Loads Model Context Protocol tool information for the current application
## Retriever API Reference
UFO employs the `Retriever` class located in `ufo/rag/retriever.py` to retrieve knowledge from various sources. For detailed API documentation, see:
:::rag.retriever.Retriever
---
### Documents Docs Ufo2 Evaluation Benchmark Overview (`documents/docs/ufo2/evaluation/benchmark/overview.md`)
# Benchmark Overview
UFO² is rigorously benchmarked on two publicly‑available live‑task suites:
| Benchmark | Scope |
|-----------|-------|
| [**Windows Agent Arena (WAA)**](./windows_agent_arena.md) | 154 real Windows tasks across 15 applications (Office, Edge, File Explorer, VS Code, …) |
| [**OSWorld (Windows)**](./osworld.md) | 49 cross‑application tasks that mix Office 365, browser and system utilities |
The integration of these benchmarks into UFO² is in separate repositories. Please follow the above documents for more details.
!!!note
we have revised the verification scripts of some cases to ensure the correctness of the results.
---
### Documents Docs Ufo2 Evaluation Logs Overview (`documents/docs/ufo2/evaluation/logs/overview.md`)
# UFO Logs
UFO generates comprehensive logs for debugging, analysis, and evaluation. Understanding these logs is essential for diagnosing issues and improving agent performance.
## Log Types
| Log Type | Description | Location |
| --- | --- | --- |
| [Request Log](./request_logs.md) | LLM prompt requests at each step | `logs/{task_name}/request.log` |
| [Step Log](./step_logs.md) | Agent responses and execution details | `logs/{task_name}/response.log` |
| [Evaluation Log](./evaluation_logs.md) | Task evaluation results | `logs/{task_name}/evaluation.log` |
| [Screenshots](./screenshots_logs.md) | UI screenshots and visual captures | `logs/{task_name}/` |
| [UI Tree](./ui_tree_logs.md) | Application UI structure data | `logs/{task_name}/ui_tree/` |
All logs are stored in the `logs/{task_name}` directory, where `{task_name}` is auto-generated based on timestamp.
---
### Documents Docs Choose Path (`documents/docs/choose_path.md`)
# Choosing Your Path: UFO² or UFO³ Galaxy?
Not sure which UFO framework to use? This guide will help you make the right choice based on your specific needs.
---
## 🗺️ Quick Decision Tree
Use this interactive flowchart to find the best solution for your use case:
```mermaid
graph TD
Start[What are you trying to automate?] --> Q1{Involves multiple devices/platforms?}
Q1 -->|Yes| Q2{Need parallel execution across devices?}
Q1 -->|No| Q3{Complex multi-app workflow on Windows?}
Q2 -->|Yes| Galaxy[✨ Use UFO³ Galaxy]
Q2 -->|No, sequential| Q4{Can tasks run independently?}
Q4 -->|Yes, independent| UFO2_Multi[Use UFO² on each device separately]
Q4 -->|No, dependencies| Galaxy
Q3 -->|Yes| UFO2[🪟 Use UFO²]
Q3 -->|No, simple task| UFO2
Q3 -->|Might scale later| Hybrid[Use UFO² now, Galaxy-ready setup]
Galaxy --> GalaxyDoc[📖 See Galaxy Quick Start]
UFO2 --> UFO2Doc[📖 See UFO² Quick Start]
UFO2_Multi --> UFO2Doc
Hybrid --> MigrationDoc[📖 See Migration Guide]
style Galaxy fill:#fff9c4
style UFO2 fill:#c8e6c9
style UFO2_Multi fill:#c8e6c9
style Hybrid fill:#e1bee7
click GalaxyDoc "./getting_started/quick_start_galaxy.md"
click UFO2Doc "./getting_started/quick_start_ufo2.md"
click MigrationDoc "./getting_started/migration_ufo2_to_galaxy.md"
```
---
## 📊 Quick Comparison Matrix
| Dimension | UFO² Desktop AgentOS | UFO³ Galaxy |
|-----------|---------------------|-------------|
| **Target Scope** | Single Windows desktop | Multiple devices (Windows/Linux/macOS) |
| **Best For** | Simple local automation | Complex cross-device workflows |
| **Setup Complexity** | ⭐ Simple | ⭐⭐⭐ Moderate (requires device pool) |
| **Learning Curve** | ⭐⭐ Easy | ⭐⭐⭐⭐ Advanced |
| **Execution Model** | Sequential multi-app | Parallel DAG orchestration |
| **Network Required** | ❌ No | ✅ Yes (WebSocket between devices) |
| **Parallelism** | Within single device | Across multiple devices |
| **Fault Tolerance** | Retry on same device | Retry + task migration |
| **Typical Latency** | 10-30s (local) | 20-60s (includes orchestration) |
| **Ideal Task Count** | 1-5 steps | 5-20+ steps with dependencies |
**Quick Rule of Thumb:**
- **1 device + simple workflow** → UFO²
- **2+ devices OR complex dependencies** → Galaxy
- **Not sure?** → Start with UFO², migrate later ([Migration Guide](./getting_started/migration_ufo2_to_galaxy.md))
---
## 🎯 Scenario-Based Recommendations
### Scenario 1: Desktop Productivity Automation
**Task:** "Create a weekly report: extract data from Excel, generate charts in PowerPoint, send via Outlook"
**Recommendation:** ✅ **UFO²**
**Why:**
- All applications on one Windows desktop
- Sequential workflow (Excel → PowerPoint → Outlook)
- No cross-device dependencies
**Learn More:** [UFO² Overview](./ufo2/overview.md)
---
### Scenario 2: Development Workflow Automation
**Task:** "Clone repo on my laptop, build Docker image on GPU server, run tests on CI cluster, open results on my desktop"
**Recommendation:** ✅ **UFO³ Galaxy**
**Why:**
- Spans 3+ devices (laptop, GPU server, CI cluster, desktop)
- Sequential dependencies (clone → build → test → display)
- Requires device coordination and data transfer
**Learn More:** [Galaxy Overview](./galaxy/overview.md)
---
### Scenario 3: Batch Data Processing
**Task:** "Process 100 files: fetch from cloud, clean data, run ML model, save results"
**Recommendation:** **Depends on setup**
| Setup | Recommendation | Why |
|-------|---------------|-----|
| **Single powerful workstation** | ✅ UFO² | All processing on one machine, simpler |
| **Distributed cluster** | ✅ Galaxy | Parallel processing across nodes, faster |
| **Mix (local + cloud GPU)** | ✅ Galaxy | Heterogeneous resources |
**Learn More:**
- [UFO² for Single Device](./getting_started/quick_start_ufo2.md)
- [Galaxy for Distributed](./getting_started/quick_start_galaxy.md)
---
### Scenario 4: Cross-Platform Testing
**Task:** "Test web app on Windows Chrome, Linux Firefox, and macOS Safari"
**Recommendation:** ✅ **UFO³ Galaxy**
**Why:**
- Requires 3 different OS platforms
- Parallel execution saves time
- Centralized result aggregation
**Learn More:** [Galaxy Multi-Platform Support](./galaxy/overview.md#cross-device-collaboration)
---
### Scenario 5: File Management & Organization
**Task:** "Organize Downloads folder by file type, compress old files, upload to cloud"
**Recommendation:** ✅ **UFO²**
**Why:**
- Single-device local file operations
- No network dependencies
- Simple sequential workflow
**Learn More:** [UFO² Quick Start](./getting_started/quick_start_ufo2.md)
---
### Scenario 6: Multi-Stage Data Pipeline
**Task:** "Collect logs from 5 Linux servers, aggregate on central server, analyze, generate dashboard on Windows"
**Recommendation:** ✅ **UFO³ Galaxy**
**Why:**
- Multiple source devices (5 Linux servers)
- Parallel log collection (5x faster than sequential)
- Cross-platform (Linux → Windows)
- Complex dependency graph
**Learn More:** [Galaxy Task Constellation](./galaxy/constellation/overview.md)
---
### Scenario 7: Learning Agent Development
**Task:** "I'm new to agent development and want to learn by building simple automation"
**Recommendation:** ✅ **UFO²**
**Why:**
- Simpler architecture (easier to understand)
- Faster feedback loop (local execution)
- Comprehensive documentation and examples
- Can upgrade to Galaxy later
**Learn More:** [UFO² Quick Start](./getting_started/quick_start_ufo2.md)
---
### Scenario 8: Enterprise Workflow Integration
**Task:** "Integrate with existing CI/CD pipeline across dev laptops, build servers, and test farms"
**Recommendation:** ✅ **UFO³ Galaxy**
**Why:**
- Enterprise-scale device coordination
- Fault tolerance with automatic recovery
- Formal safety guarantees for correctness
- Supports heterogeneous infrastructure
**Learn More:** [Galaxy Architecture](./galaxy/overview.md#architecture)
---
## 🔀 Hybrid Approaches
You don't have to choose just one! Here are common hybrid patterns:
### Pattern 1: UFO² as Galaxy Device
**Setup:** Run UFO² as a Galaxy device (requires both server and client)
```bash
# Terminal 1: Start UFO² Server on Windows desktop
python -m ufo.server.app --port 5000
# Terminal 2: Start UFO² Client (connect to server)
python -m ufo.client.client --ws --ws-server ws://localhost:5000/ws --client-id my_windows_device --platform windows
```
**Benefits:**
- Keep UFO² for local Windows expertise
- Gain Galaxy's cross-device orchestration
- Best of both worlds
**Learn More:** [UFO² as Galaxy Device](./ufo2/as_galaxy_device.md)
---
### Pattern 2: Gradual Migration
**Strategy:** Start with UFO² for immediate needs, prepare for Galaxy expansion
**Phase 1:** Use UFO² standalone
```bash
python -m ufo --task "Your current task"
```
**Phase 2:** Make UFO² Galaxy-compatible
```yaml
# config/galaxy/devices.yaml (prepare in advance)
devices:
- device_id: "my_windows"
server_url: "ws://localhost:5000/ws" # Where UFO client connects to UFO server
os: "windows"
capabilities: ["office", "web"]
```
**Phase 3:** Start UFO device agent and connect to Galaxy
```bash
# Terminal 1: Start UFO Server on your Windows machine
python -m ufo.server.app --port 5000
# Terminal 2: Start UFO Client (connects to UFO server above)
python -m ufo.client.client --ws --ws-server ws://localhost:5000/ws --client-id my_windows --platform windows
# Terminal 3: Start Galaxy (on control machine, can be same or different)
python -m galaxy --request "Cross-device workflow"
```
**Learn More:** [Migration Guide](./getting_started/migration_ufo2_to_galaxy.md)
---
### Pattern 3: Domain-Specific Split
**Strategy:** Use different frameworks for different workflow types
| Workflow Type | Framework | Example |
|--------------|-----------|---------|
| **Daily desktop tasks** | UFO² | Email processing, document creation |
| **Development workflows** | Galaxy | Code build → test → deploy |
| **Data processing** | Galaxy (if distributed) | Multi-node ML training |
| **Quick automation** | UFO² | One-off tasks |
**Learn More:** [When to Use Which](./getting_started/migration_ufo2_to_galaxy.md#when-to-use-which)
---
## 🚫 Common Misconceptions
### Misconception 1: "Galaxy is always better because it's newer"
**Reality:** UFO² is better for simple single-device tasks due to:
- Lower latency (no network overhead)
- Simpler setup and debugging
- Battle-tested stability
**Use Galaxy only when you actually need multi-device orchestration.**
---
### Misconception 2: "I need to rewrite everything to migrate to Galaxy"
**Reality:** UFO² can run as a Galaxy device with minimal changes:
```bash
# Terminal 1: Start UFO Server
python -m ufo.server.app --port 5000
# Terminal 2: Start UFO Client in WebSocket mode
python -m ufo.client.client --ws --ws-server ws://localhost:5000/ws --client-id my_device --platform windows
```
**Learn More:** [Migration Guide](./getting_started/migration_ufo2_to_galaxy.md#option-2-convert-ufo2-instance-to-galaxy-device)
---
### Misconception 3: "Galaxy can't run on a single device"
**Reality:** Galaxy works perfectly on one device if you need:
- DAG-based workflow planning
- Advanced monitoring and trajectory reports
- Preparation for future multi-device expansion
```yaml
# Single-device Galaxy setup
devices:
- device_id: "localhost"
server_url: "ws://localhost:5005/ws"
```
---
### Misconception 4: "UFO² is deprecated in favor of Galaxy"
**Reality:** UFO² is actively maintained and recommended for single-device use:
- More efficient for local tasks
- Simpler for beginners
- Core component when used as Galaxy device
**Both frameworks are complementary, not competing.**
---
## 🎓 Learning Paths
### For Beginners
**Week 1-2: Start with UFO²**
1. [UFO² Quick Start](./getting_started/quick_start_ufo2.md)
2. Build simple automation (file management, email, etc.)
3. Understand HostAgent/AppAgent architecture
**Week 3-4: Explore Advanced UFO²**
4. [Hybrid GUI-API Actions](./ufo2/core_features/hybrid_actions.md)
5. [MCP Server Integration](./mcp/overview.md)
6. [Customization & Learning](./ufo2/advanced_usage/customization.md)
**Week 5+: Graduate to Galaxy (if needed)**
7. [Migration Guide](./getting_started/migration_ufo2_to_galaxy.md)
8. [Galaxy Quick Start](./getting_started/quick_start_galaxy.md)
9. Build cross-device workflows
---
### For Experienced Developers
**Direct to Galaxy** if you already know you need multi-device:
1. [Galaxy Quick Start](./getting_started/quick_start_galaxy.md)
2. [Task Constellation Concepts](./galaxy/constellation/overview.md)
3. [ConstellationAgent Deep Dive](./galaxy/constellation_agent/overview.md)
4. [Performance Monitoring](./galaxy/evaluation/performance_metrics.md)
---
## 📋 Decision Checklist
Still unsure? Answer these questions:
**Q1: Does your workflow involve 2+ physical devices?**
- ✅ Yes → **Galaxy**
- ❌ No → Continue to Q2
**Q2: Do you need parallel execution across different machines?**
- ✅ Yes → **Galaxy**
- ❌ No → Continue to Q3
**Q3: Does your workflow have complex dependencies (DAG structure)?**
- ✅ Yes, complex DAG → **Galaxy**
- ❌ No, simple sequence → Continue to Q4
**Q4: Are you comfortable with distributed systems concepts?**
- ✅ Yes → **Galaxy** (if any of Q1-Q3 is yes)
- ❌ No → **UFO²** (learn basics first)
**Q5: Do you need cross-platform support (Windows + Linux)?**
- ✅ Yes → **Galaxy**
- ❌ No, Windows only → **UFO²**
---
**Result:**
- **3+ "Galaxy" answers** → Use Galaxy ([Quick Start](./getting_started/quick_start_galaxy.md))
- **Mostly "UFO²" answers** → Use UFO² ([Quick Start](./getting_started/quick_start_ufo2.md))
- **Mixed answers** → Start with UFO², keep Galaxy option open ([Migration Guide](./getting_started/migration_ufo2_to_galaxy.md))
---
## 🔗 Next Steps
### If you chose UFO²:
1. 📖 [UFO² Quick Start Guide](./getting_started/quick_start_ufo2.md)
2. 🎯 [UFO² Overview & Architecture](./ufo2/overview.md)
3. 🛠️ [Configuration Guide](./configuration/system/overview.md)
### If you chose Galaxy:
1. 📖 [Galaxy Quick Start Guide](./getting_started/quick_start_galaxy.md)
2. 🎯 [Galaxy Overview & Architecture](./galaxy/overview.md)
3. 🌟 [Task Constellation Concepts](./galaxy/constellation/overview.md)
### If you're still exploring:
1. 📊 [Detailed Comparison](./getting_started/migration_ufo2_to_galaxy.md#when-to-use-which)
2. 🎬 [Demo Video](https://www.youtube.com/watch?v=QT_OhygMVXU)
3. 📄 [Research Paper](https://arxiv.org/abs/2504.14603)
---
## 💡 Pro Tips
!!! tip "Start Simple"
When in doubt, start with **UFO²**. It's easier to scale up to Galaxy later than to debug a complex Galaxy setup when you don't need it.
!!! tip "Hybrid is Valid"
Don't feel locked into one choice. You can use **UFO² for local tasks** and **Galaxy for cross-device workflows** simultaneously.
!!! tip "Test Before Committing"
Try both for a simple workflow to see which feels more natural for your use case:
```bash
# UFO² test
python -m ufo --task "Create test report"
# Galaxy test
python -m galaxy --request "Create test report"
```
!!! warning "Network Requirements"
Galaxy requires **stable network connectivity** between devices. If your environment has network restrictions, UFO² might be more reliable.
---
## 🤝 Getting Help
- **Documentation:** [https://microsoft.github.io/UFO/](https://microsoft.github.io/UFO/)
- **GitHub Issues:** [https://github.com/microsoft/UFO/issues](https://github.com/microsoft/UFO/issues)
- **Discussions:** [https://github.com/microsoft/UFO/discussions](https://github.com/microsoft/UFO/discussions)
Still have questions? Check the [Migration FAQ](./getting_started/migration_ufo2_to_galaxy.md#getting-help) or open a discussion on GitHub!
---
### Documents Docs Faq (`documents/docs/faq.md`)
# Frequently Asked Questions (FAQ)
Quick answers to common questions about UFO³ Galaxy, UFO², Linux Agents, and general troubleshooting.
---
## 🎯 General Questions
### Q: What is UFO³?
**A:** UFO³ is the third iteration of the UFO project, encompassing three major frameworks:
- **UFO²** - Desktop AgentOS for Windows automation
- **UFO³ Galaxy** - Multi-device orchestration framework
- **Linux Agent** - Server and CLI automation for Linux
### Q: Why is it called UFO?
**A:** UFO stands for **U**I **Fo**cused agent. The name was given to the first version of the project and has been retained through all iterations (UFO v1, UFO², UFO³) as the project evolved from a simple UI-focused agent to a comprehensive multi-device orchestration framework.
### Q: Which version should I use?
**A:** Choose based on your needs:
| Use Case | Recommended Version |
|----------|-------------------|
| Windows desktop automation only | [UFO²](getting_started/quick_start_ufo2.md) |
| Cross-device workflows (Windows + Linux) | [UFO³ Galaxy](getting_started/quick_start_galaxy.md) |
| Linux server management only | [Linux Agent](getting_started/quick_start_linux.md) |
| Multi-device orchestration | [UFO³ Galaxy](getting_started/quick_start_galaxy.md) |
### Q: What's the difference between UFO² and UFO³ Galaxy?
**A: UFO²** is for single Windows desktop automation with:
- Deep Windows OS integration (UIA, Win32, COM)
- Office application automation
- GUI + API hybrid execution
**UFO³ Galaxy** orchestrates multiple devices with:
- Cross-platform support (Windows + Linux)
- Distributed task execution
- Device capability-based routing
- Constellation-based DAG orchestration
See [Migration Guide](getting_started/migration_ufo2_to_galaxy.md) for details.
### Q: Can I use UFO on Linux or macOS?
**A:** Yes and No:
- **✅ Linux:** Supported via Linux Agent for server/CLI automation
- **❌ macOS:** Not currently supported (Windows and Linux only)
- **Windows:** Full UFO² desktop automation support
---
## 🔧 Installation & Setup
### Q: Which Python version do I need?
**A:** Python **3.10 or higher** is required for all UFO³ components.
```bash
# Check your Python version
python --version
```
### Q: What models does UFO support?
**A:** UFO³ supports multiple LLM providers:
- **OpenAI** - GPT-4o, GPT-4, GPT-3.5
- **Azure OpenAI** - All Azure-hosted models
- **Google Gemini** - Gemini Pro, Gemini Flash
- **Anthropic Claude** - Claude 3.5, Claude 3
- **Qwen** - Local or API deployment
- **DeepSeek** - DeepSeek models
- **Ollama** - Local model hosting
- And more...
See [Model Configuration Guide](configuration/models/overview.md) for the complete list and setup instructions.
### Q: Can I use non-vision models in UFO?
**A:** Yes! You can disable visual mode:
```yaml
# config/ufo/system.yaml
VISUAL_MODE: false
```
However, UFO² is designed for vision models. Non-vision models may have reduced performance for GUI automation tasks.
### Q: Can I host my own LLM endpoint?
**A:** Yes! UFO³ supports custom endpoints:
```yaml
# config/ufo/agents.yaml
HOST_AGENT:
API_TYPE: "openai" # Or compatible API
API_BASE: "http://your-endpoint.com/v1/chat/completions"
API_KEY: "your-key"
API_MODEL: "your-model-name"
```
See [Model Configuration](configuration/models/overview.md) for details.
### Q: Do I need API keys for all agents?
**A:** No, only for LLM-powered agents:
| Component | Requires API Key | Purpose |
|-----------|-----------------|---------|
| **ConstellationAgent** (Galaxy) | ✅ Yes | Orchestration reasoning |
| **HostAgent** (UFO²) | ✅ Yes | Task planning |
| **AppAgent** (UFO²) | ✅ Yes | Action execution |
| **LinuxAgent** | ✅ Yes | Command planning |
| **Device Server** | ❌ No | Message routing only |
| **MCP Servers** | ❌ No | Tool provider only |
---
## ⚙️ Configuration
### Q: Where are configuration files located?
**A:** UFO³ uses a modular configuration system in `config/`:
```
config/
├── ufo/ # UFO² configuration
│ ├── agents.yaml # LLM and agent settings
│ ├── system.yaml # Runtime settings
│ ├── rag.yaml # Knowledge retrieval
│ └── mcp.yaml # MCP server configuration
└── galaxy/ # Galaxy configuration
├── agent.yaml # ConstellationAgent LLM
├── devices.yaml # Device pool
└── constellation.yaml # Runtime settings
```
### Q: Can I still use the old `ufo/config/config.yaml`?
**A:** Yes, for backward compatibility, but we recommend migrating to the new modular system:
```bash
# Check current configuration
python -m ufo.tools.validate_config ufo --show-config
# Migrate from legacy to new
python -m ufo.tools.migrate_config
```
See [Configuration Migration Guide](configuration/system/migration.md) for details.
### Q: How do I protect my API keys?
**A:** Best practices for API key security:
1. **Never commit `.yaml` files with keys** - Use `.template` files
```bash
# Good pattern
config/ufo/agents.yaml.template # Commit this (with placeholders)
config/ufo/agents.yaml # DON'T commit (has real keys)
```
2. **Use environment variables** for sensitive data:
```yaml
# In agents.yaml
HOST_AGENT:
API_KEY: ${OPENAI_API_KEY} # Reads from environment
```
3. **Add to `.gitignore`**:
```
config/**/agents.yaml
config/**/agent.yaml
!**/*.template
```
---
## 🌌 UFO³ Galaxy Questions
### Q: What's the minimum number of devices for Galaxy?
**A:** Galaxy requires **at least 1 device agent** (Windows or Linux) to be useful, but you can start with just one device and add more later.
```yaml
# Minimal Galaxy setup (1 device)
devices:
- device_id: "my_windows_pc"
server_url: "ws://localhost:5000/ws"
os: "windows"
```
### Q: Can Galaxy mix Windows and Linux devices?
**A:** Yes! Galaxy can orchestrate heterogeneous devices:
```yaml
devices:
- device_id: "windows_desktop"
os: "windows"
capabilities: ["office", "excel", "outlook"]
- device_id: "linux_server"
os: "linux"
capabilities: ["server", "database", "log_analysis"]
```
Galaxy automatically routes tasks based on device capabilities.
### Q: Do all devices need to be on the same network?
**A:** No, devices can be distributed across networks using SSH tunneling:
- **Same network:** Direct WebSocket connections
- **Different networks:** Use SSH tunnels (reverse/forward)
- **Cloud + local:** SSH tunnels with public gateways
See [Linux Quick Start - SSH Tunneling](getting_started/quick_start_linux.md#network-connectivity-ssh-tunneling) for examples.
### Q: How does Galaxy decide which device to use?
**A:** Galaxy uses **capability-based routing**:
1. Analyzes the task requirements
2. Matches against device `capabilities` in `devices.yaml`
3. Considers device `metadata` (OS, performance, etc.)
4. Selects the best-fit device(s)
Example:
```yaml
# Task: "Analyze error logs on the production server"
# → Galaxy routes to device with:
capabilities:
- "log_analysis"
- "server_management"
os: "linux"
```
---
## 🐧 Linux Agent Questions
### Q: Does the Linux Agent require a GUI?
**A:** No! The Linux Agent is designed for headless servers:
- Executes CLI commands via MCP
- No X11/desktop environment needed
- Works over SSH
- Perfect for remote servers
### Q: Can I run multiple Linux Agents on one machine?
**A:** Yes, using different ports and client IDs:
```bash
# Agent 1
python -m ufo.server.app --port 5001
python -m ufo.client.client --ws --client-id linux_1 --platform linux
# Agent 2 (same machine)
python -m ufo.server.app --port 5002
python -m ufo.client.client --ws --client-id linux_2 --platform linux
```
### Q: What's the MCP service for?
**A:** The MCP (Model Context Protocol) service provides the **actual command execution tools** for the Linux Agent:
```
Linux Agent (LLM reasoning)
↓
MCP Service (tool provider)
↓
Bash commands (actual execution)
```
Without MCP, the Linux Agent can't execute commands - it can only plan them.
---
## 🪟 UFO² Questions
### Q: Does UFO² work on Windows 10?
**A:** Yes! UFO² supports:
- ✅ Windows 11 (recommended)
- ✅ Windows 10 (fully supported)
- ❌ Windows 8.1 or earlier (not tested)
### Q: Can UFO² automate Office apps?
**A:** Yes! UFO² has enhanced Office support through:
- **MCP Office servers** - Direct API access to Excel, Word, Outlook, PowerPoint
- **GUI automation** - Fallback for unsupported operations
- **Hybrid execution** - Automatically chooses API or GUI
Enable MCP in `config/ufo/mcp.yaml` for better Office automation.
### Q: Does UFO² interrupt my work?
**A:** UFO² can run automation tasks on your current desktop. For non-disruptive operation, you can run it on a separate machine or virtual desktop environment.
> **Note:** Picture-in-Picture mode is planned for future releases.
### Q: Can I use UFO² without MCP?
**A:** UFO² requires MCP (Model Context Protocol) servers for tool execution. MCP provides the interface between the LLM agents and system operations (Windows APIs, Office automation, etc.). Without MCP, UFO² cannot perform actions.
---
## 🐛 Common Issues & Troubleshooting
### Issue: "Configuration file not found"
**Error:**
```
FileNotFoundError: config/ufo/agents.yaml not found
```
**Solution:**
```bash
# Copy template files
cp config/ufo/agents.yaml.template config/ufo/agents.yaml
# Edit with your API keys
notepad config/ufo/agents.yaml # Windows
nano config/ufo/agents.yaml # Linux
```
### Issue: "API Authentication Error"
**Error:**
```
openai.AuthenticationError: Invalid API key
```
**Solutions:**
1. **Check API key format:**
```yaml
API_KEY: "sk-..." # OpenAI starts with sk-
API_KEY: "..." # Azure uses deployment key
```
2. **Verify API_TYPE matches your provider:**
```yaml
API_TYPE: "openai" # For OpenAI
API_TYPE: "aoai" # For Azure OpenAI
```
3. **Check for extra spaces/quotes** in YAML
4. **For Azure:** Verify `API_DEPLOYMENT_ID` is set
### Issue: "Connection aborted / Remote end closed connection"
**Error:**
```
Error making API request: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
```
**Solutions:**
- Check network connection (VPN, proxy, firewall)
- Verify LLM endpoint is accessible: `curl https://api.openai.com/v1/models`
- Check endpoint status (Azure, OpenAI, etc.)
- Try increasing timeout in config
- Verify API base URL is correct
### Issue: "Device not connecting to Galaxy"
**Error:**
```
ERROR - [WS] Failed to connect to ws://localhost:5000/ws
Connection refused
```
**Checklist:**
- [ ] Is the server running? (`curl http://localhost:5000/api/health`)
- [ ] Port number correct? (Server: `--port 5000`, Client: `ws://...:5000/ws`)
- [ ] Platform flag set? (`--platform windows` or `--platform linux`)
- [ ] Firewall blocking? (Allow port 5000)
- [ ] SSH tunnel established? (If using remote devices)
### Issue: "device_id mismatch in Galaxy"
**Error:**
```
ERROR - Device 'linux_agent_1' not found in configuration
```
**Cause:** Mismatch between `devices.yaml` and client command
**Solution:** Ensure exact match:
| Location | Field | Example |
|----------|-------|---------|
| `devices.yaml` | `device_id:` | `"linux_agent_1"` |
| Client command | `--client-id` | `linux_agent_1` |
**Critical:** IDs must match **exactly** (case-sensitive, no typos).
### Issue: "MCP service not responding (Linux)"
**Error:**
```
ERROR - Cannot connect to MCP server at http://127.0.0.1:8010
```
**Solutions:**
1. **Check if MCP service is running:**
```bash
curl http://localhost:8010/health
ps aux | grep linux_mcp_server
```
2. **Restart MCP service:**
```bash
pkill -f linux_mcp_server
python -m ufo.client.mcp.http_servers.linux_mcp_server
```
3. **Check port conflict:**
```bash
lsof -i :8010
# If port taken, use different port:
python -m ufo.client.mcp.http_servers.linux_mcp_server --port 8011
```
### Issue: "Tasks failing after X steps"
**Cause:** `MAX_STEP` limit reached
**Solution:** Increase step limit in `config/ufo/system.yaml`:
```yaml
# Default is 50
MAX_STEP: 100 # For complex tasks
# Or disable limit (not recommended)
MAX_STEP: -1
```
### Issue: "Too many LLM calls / high cost"
**Solutions:**
1. **Enable action sequences** (bundles actions):
```yaml
# config/ufo/system.yaml
ACTION_SEQUENCE: true
```
2. **Use vision-capable models for GUI tasks:**
```yaml
# config/ufo/agents.yaml
APP_AGENT:
API_MODEL: "gpt-4o" # Use vision models for GUI automation
```
> **Note:** Non-vision models like gpt-3.5-turbo cannot process screenshots and should not be used for GUI automation tasks.
3. **Enable experience learning** (reuse patterns):
```yaml
# config/ufo/rag.yaml
RAG_EXPERIENCE: true
```
### Issue: "Why is the latency high?"
**A:** Latency depends on several factors:
- **LLM response time** - GPT-4o typically takes 10-30 seconds per step
- **Network speed** - API calls to OpenAI/Azure endpoints
- **Endpoint workload** - Provider server load
- **Visual mode** - Image processing adds overhead
**To reduce latency:**
- Use faster models (gpt-3.5-turbo vs gpt-4o)
- Enable action sequences to batch operations
- Use local models (Ollama) if acceptable
- Disable visual mode if not needed
### Issue: "Can I use non-English requests?"
**A:** Yes! Most modern LLMs support multiple languages:
- GPT-4o, GPT-4: Excellent multilingual support
- Gemini: Good multilingual support
- Qwen: Excellent for Chinese
- Claude: Good multilingual support
Performance may vary by language and model. Test with your specific language and model combination.
---
## 📚 Where to Find More Help
### Documentation
| Topic | Link |
|-------|------|
| **Getting Started** | [UFO² Quick Start](getting_started/quick_start_ufo2.md), [Galaxy Quick Start](getting_started/quick_start_galaxy.md), [Linux Quick Start](getting_started/quick_start_linux.md) |
| **Configuration** | [Configuration Overview](configuration/system/overview.md) |
| **Troubleshooting** | Quick start guides have detailed troubleshooting sections |
| **Architecture** | [Project Structure](project_directory_structure.md) |
| **More Guidance** | [User & Developer Guide](getting_started/more_guidance.md) |
### Community & Support
- **GitHub Discussions:** [https://github.com/microsoft/UFO/discussions](https://github.com/microsoft/UFO/discussions)
- **GitHub Issues:** [https://github.com/microsoft/UFO/issues](https://github.com/microsoft/UFO/issues)
- **Email:** ufo-agent@microsoft.com
### Debugging Tips
1. **Enable debug logging:**
```yaml
# config/ufo/system.yaml
LOG_LEVEL: "DEBUG"
```
2. **Check log files:**
```
logs//
├── request.log # Request logs
├── response.log # Response logs
├── action_step*.png # Screenshots at each step
└── action_step*_annotated.png # Annotated screenshots
```
3. **Validate configuration:**
```bash
python -m ufo.tools.validate_config ufo --show-config
python -m ufo.tools.validate_config galaxy --show-config
```
4. **Test LLM connectivity:**
```python
# Test your API key
from openai import OpenAI
client = OpenAI(api_key="your-key")
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}]
)
print(response.choices[0].message.content)
```
---
> **💡 Still have questions?** Check the [More Guidance](getting_started/more_guidance.md) page for additional resources, or reach out to the community!
---
### Documents Docs Project Directory Structure (`documents/docs/project_directory_structure.md`)
# Project Directory Structure
This repository implements **UFO³**, a multi-tier AgentOS architecture spanning from single-device automation (UFO²) to cross-device orchestration (Galaxy). This document provides an overview of the directory structure to help you understand the codebase organization.
> **New to UFO³?** Start with the [Documentation Home](index.md) for an introduction and [Quick Start Guide](getting_started/quick_start_galaxy.md) to get up and running.
**Architecture Overview:**
- **🌌 Galaxy**: Multi-device DAG-based orchestration framework that coordinates agents across different platforms
- **🎯 UFO²**: Single-device Windows desktop agent system that can serve as Galaxy's sub-agent
- **🔌 AIP**: Agent Integration Protocol for cross-device communication
- **⚙️ Modular Configuration**: Type-safe configs in `config/galaxy/` and `config/ufo/`
---
## 📦 Root Directory Structure
```
UFO/
├── galaxy/ # 🌌 Multi-device orchestration framework
├── ufo/ # 🎯 Desktop AgentOS (can be Galaxy sub-agent)
├── config/ # ⚙️ Modular configuration system
├── aip/ # 🔌 Agent Integration Protocol
├── documents/ # 📖 MkDocs documentation site
├── vectordb/ # 🗄️ Vector database for RAG
├── learner/ # 📚 Help document indexing tools
├── record_processor/ # 🎥 Human demonstration parser
├── dataflow/ # 📊 Data collection pipeline
├── model_worker/ # 🤖 Custom LLM deployment tools
├── logs/ # 📝 Execution logs (auto-generated)
├── scripts/ # 🛠️ Utility scripts
├── tests/ # 🧪 Unit and integration tests
└── requirements.txt # 📦 Python dependencies
```
---
## 🌌 Galaxy Framework (`galaxy/`)
The cross-device orchestration framework that transforms natural language requests into executable DAG workflows distributed across heterogeneous devices.
### Directory Structure
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Key Components
| Component | Description | Documentation |
|-----------|-------------|---------------|
| **ConstellationAgent** | AI-powered agent that generates and modifies task DAGs | [Galaxy Overview](galaxy/overview.md) |
| **TaskConstellation** | DAG container with validation and state management | [Constellation](galaxy/constellation/overview.md) |
| **TaskOrchestrator** | Event-driven execution coordinator | [Constellation Orchestrator](galaxy/constellation_orchestrator/overview.md) |
| **DeviceManager** | Multi-device coordination and assignment | [Device Manager](galaxy/client/device_manager.md) |
| **Visualization** | Rich console DAG monitoring | [Galaxy Overview](galaxy/overview.md) |
**Galaxy Documentation:**
- [Galaxy Overview](galaxy/overview.md) - Architecture and concepts
- [Quick Start](getting_started/quick_start_galaxy.md) - Get started with Galaxy
- [Constellation Agent](galaxy/constellation_agent/overview.md) - AI-powered task planning
- [Constellation Orchestrator](galaxy/constellation_orchestrator/overview.md) - Event-driven coordination
- [Device Manager](galaxy/client/device_manager.md) - Multi-device management
---
## 🎯 UFO² Desktop AgentOS (`ufo/`)
Single-device desktop automation system implementing a two-tier agent architecture (HostAgent + AppAgent) with hybrid GUI-API automation.
### Directory Structure
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Key Components
| Component | Description | Documentation |
|-----------|-------------|---------------|
| **HostAgent** | Desktop-level orchestration with 7-state FSM | [HostAgent Overview](ufo2/host_agent/overview.md) |
| **AppAgent** | Application-level execution with 6-state FSM | [AppAgent Overview](ufo2/app_agent/overview.md) |
| **MCP System** | Extensible command execution framework | [MCP Overview](mcp/overview.md) |
| **Automator** | Hybrid GUI-API automation with fallback | [Core Features](ufo2/core_features/hybrid_actions.md) |
| **RAG** | Knowledge retrieval from multiple sources | [Knowledge Substrate](ufo2/core_features/knowledge_substrate/overview.md) |
**UFO² Documentation:**
- [UFO² Overview](ufo2/overview.md) - Architecture and concepts
- [Quick Start](getting_started/quick_start_ufo2.md) - Get started with UFO²
- [HostAgent States](ufo2/host_agent/state.md) - Desktop orchestration states
- [AppAgent States](ufo2/app_agent/state.md) - Application execution states
- [As Galaxy Device](ufo2/as_galaxy_device.md) - Using UFO² as Galaxy sub-agent
- [Creating Custom Agents](tutorials/creating_app_agent/overview.md) - Build your own application agents
---
## 🔌 Agent Integration Protocol (`aip/`)
Standardized message passing protocol for cross-device communication between Galaxy and UFO² agents.
```
aip/
├── messages.py # Message types (Command, Result, Event, Error)
├── protocol/ # Protocol definitions
├── transport/ # Transport layers (HTTP, WebSocket, MQTT)
├── endpoints/ # API endpoints
├── extensions/ # Protocol extensions
└── resilience/ # Retry and error handling
```
**Purpose**: Enables Galaxy to coordinate UFO² agents running on different devices and platforms through standardized messaging over HTTP/WebSocket.
**Documentation**: See [AIP Overview](aip/overview.md) for protocol details and [Message Types](aip/messages.md) for message specifications.
---
## 🐧 Linux Agent
Lightweight CLI-based agent for Linux devices that integrates with Galaxy as a third-party device agent.
**Key Features**:
- **CLI Execution**: Execute shell commands on Linux systems
- **Galaxy Integration**: Register as device in Galaxy's multi-device orchestration
- **Simple Architecture**: Minimal dependencies, easy deployment
- **Cross-Platform Tasks**: Enable Windows + Linux workflows in Galaxy
**Configuration**: Configured in `config/ufo/third_party.yaml` under `THIRD_PARTY_AGENT_CONFIG.LinuxAgent`
**Linux Agent Documentation:**
- [Linux Agent Overview](linux/overview.md) - Architecture and capabilities
- [Quick Start](getting_started/quick_start_linux.md) - Setup and deployment
- [As Galaxy Device](linux/as_galaxy_device.md) - Integration with Galaxy
---
## 📱 Mobile Agent
Android device automation agent that enables UI automation, app control, and mobile-specific operations through ADB integration.
**Key Features**:
- **UI Automation**: Touch, swipe, and text input via ADB
- **Visual Context**: Screenshot capture and UI hierarchy analysis
- **App Management**: Launch apps, navigate between applications
- **Galaxy Integration**: Serve as mobile device in cross-platform workflows
- **Platform Support**: Android devices (physical and emulators)
**Configuration**: Configured in `config/ufo/third_party.yaml` under `THIRD_PARTY_AGENT_CONFIG.MobileAgent`
**Mobile Agent Documentation:**
- [Mobile Agent Overview](mobile/overview.md) - Architecture and capabilities
- [Quick Start](getting_started/quick_start_mobile.md) - Setup and deployment
- [As Galaxy Device](mobile/as_galaxy_device.md) - Integration with Galaxy
---
## ⚙️ Configuration (`config/`)
Modular configuration system with type-safe schemas and auto-discovery.
```
config/
├── galaxy/ # Galaxy configuration
│ ├── agent.yaml.template # ConstellationAgent LLM settings template
│ ├── agent.yaml # ConstellationAgent LLM settings (active)
│ ├── constellation.yaml # Constellation orchestration settings
│ ├── devices.yaml # Multi-device registry
│ └── dag_templates/ # Pre-built DAG templates (future)
│
├── ufo/ # UFO² configuration
│ ├── agents.yaml.template # Agent LLM configs template
│ ├── agents.yaml # Agent LLM configs (active)
│ ├── system.yaml # System settings
│ ├── rag.yaml # RAG settings
│ ├── mcp.yaml # MCP server configs
│ ├── third_party.yaml # Third-party agent configs (LinuxAgent, etc.)
│ └── prices.yaml # API pricing data
│
├── config_loader.py # Auto-discovery config loader
└── config_schemas.py # Pydantic validation schemas
```
**Configuration Files:**
- Template files (`.yaml.template`) should be copied to `.yaml` and edited
- Active config files (`.yaml`) contain API keys and should NOT be committed
- **Galaxy**: Uses `config/galaxy/agent.yaml` for ConstellationAgent LLM settings
- **UFO²**: Uses `config/ufo/agents.yaml` for HostAgent/AppAgent LLM settings
- **Third-Party**: Configure LinuxAgent and HardwareAgent in `config/ufo/third_party.yaml`
- Use `python -m ufo.tools.convert_config` to migrate from legacy configs
**Configuration Documentation:**
- [Configuration Overview](configuration/system/overview.md) - System architecture
- [Agents Configuration](configuration/system/agents_config.md) - LLM and agent settings
- [System Configuration](configuration/system/system_config.md) - Runtime and execution settings
- [RAG Configuration](configuration/system/rag_config.md) - Knowledge retrieval
- [Third-Party Configuration](configuration/system/third_party_config.md) - LinuxAgent and external agents
- [MCP Configuration](configuration/system/mcp_reference.md) - MCP server setup
- [Model Configuration](configuration/models/overview.md) - LLM provider setup
---
## 📖 Documentation (`documents/`)
MkDocs documentation site with comprehensive guides and API references.
```
documents/
├── docs/ # Markdown documentation source
│ ├── getting_started/ # Installation and quick starts
│ ├── galaxy/ # Galaxy framework docs
│ ├── ufo2/ # UFO² architecture docs
│ ├── linux/ # Linux agent documentation
│ ├── mcp/ # MCP server documentation
│ ├── aip/ # Agent Interaction Protocol docs
│ ├── configuration/ # Configuration guides
│ ├── infrastructure/ # Core infrastructure (agents, modules)
│ ├── server/ # Server-client architecture docs
│ ├── client/ # Client components docs
│ ├── tutorials/ # Step-by-step tutorials
│ ├── modules/ # Module-specific docs
│ └── about/ # Project information
│
├── mkdocs.yml # MkDocs configuration
└── site/ # Generated static site
```
**Documentation Sections**:
| Section | Description |
|---------|-------------|
| **Getting Started** | Installation, quick starts, migration guides |
| **Galaxy** | Multi-device orchestration, DAG workflows, device management |
| **UFO²** | Desktop agents, automation features, benchmarks |
| **Linux** | Linux agent integration, CLI executor for Galaxy |
| **MCP** | Server documentation, custom server development |
| **AIP** | Agent Interaction Protocol, message types, transport layers |
| **Configuration** | System settings, model configs, deployment |
| **Infrastructure** | Core components, agent design, server-client architecture |
| **Tutorials** | Creating agents, custom automators, advanced usage |
---
## 🗄️ Supporting Modules
### VectorDB (`vectordb/`)
Vector database storage for RAG knowledge sources (help documents, execution traces, user demonstrations). See [RAG Configuration](configuration/system/rag_config.md) for setup details.
### Learner (`learner/`)
Tools for indexing help documents into vector database for RAG retrieval. Integrates with the [Knowledge Substrate](ufo2/core_features/knowledge_substrate/overview.md) feature.
### Record Processor (`record_processor/`)
Parses human demonstrations from Windows Step Recorder for learning from user actions.
### Dataflow (`dataflow/`)
Data collection pipeline for Large Action Model (LAM) training. See the [Dataflow](ufo2/dataflow/overview.md) documentation for workflow details.
### Model Worker (`model_worker/`)
Custom LLM deployment tools for running local models. See [Model Configuration](configuration/models/overview.md) for supported providers.
### Logs (`logs/`)
Auto-generated execution logs organized by task and timestamp, including screenshots, UI trees, and agent actions.
---
## 🎯 Galaxy vs UFO² vs Linux Agent vs Mobile Agent: When to Use What?
| Aspect | Galaxy | UFO² | Linux Agent | Mobile Agent |
|--------|--------|------|-------------|--------------|
| **Scope** | Multi-device orchestration | Single-device Windows automation | Single-device Linux CLI | Single-device Android automation |
| **Use Cases** | Cross-platform workflows, distributed tasks | Desktop automation, Office tasks | Server management, CLI operations | Mobile app testing, UI automation |
| **Architecture** | DAG-based task workflows | Two-tier state machines | Simple CLI executor | UI automation via ADB |
| **Platform** | Orchestrator (platform-agnostic) | Windows | Linux | Android |
| **Complexity** | Complex multi-step workflows | Simple to moderate tasks | Simple command execution | UI interaction and app control |
| **Best For** | Cross-device collaboration | Windows desktop tasks | Linux server operations | Mobile app automation |
| **Integration** | Orchestrates all agents | Can be Galaxy device | Can be Galaxy device | Can be Galaxy device |
**Choosing the Right Framework:**
- **Use Galaxy** when: Tasks span multiple devices/platforms, complex workflows with dependencies
- **Use UFO² Standalone** when: Single-device Windows automation, rapid prototyping
- **Use Linux Agent** when: Linux server/CLI operations needed in Galaxy workflows
- **Use Mobile Agent** when: Android device automation, mobile app testing, UI interactions
- **Best Practice**: Galaxy orchestrates UFO² (Windows) + Linux Agent (Linux) + Mobile Agent (Android) for comprehensive cross-platform tasks
---
## 🚀 Quick Start
### Galaxy Multi-Device Orchestration
```bash
# Interactive mode
python -m galaxy --interactive
# Single request
python -m galaxy --request "Your cross-device task"
```
**Documentation**: [Galaxy Quick Start](getting_started/quick_start_galaxy.md)
### UFO² Desktop Automation
```bash
# Interactive mode
python -m ufo --task
# With custom config
python -m ufo --task --config_path config/ufo/
```
**Documentation**: [UFO² Quick Start](getting_started/quick_start_ufo2.md)
---
## 📚 Key Documentation Links
### Getting Started
- [Installation & Setup](getting_started/quick_start_galaxy.md)
- [Galaxy Quick Start](getting_started/quick_start_galaxy.md)
- [UFO² Quick Start](getting_started/quick_start_ufo2.md)
- [Linux Agent Quick Start](getting_started/quick_start_linux.md)
- [Mobile Agent Quick Start](getting_started/quick_start_mobile.md)
- [Migration Guide](getting_started/migration_ufo2_to_galaxy.md)
### Galaxy Framework
- [Galaxy Overview](galaxy/overview.md)
- [Constellation Agent](galaxy/constellation_agent/overview.md)
- [Constellation Orchestrator](galaxy/constellation_orchestrator/overview.md)
- [Task Constellation](galaxy/constellation/overview.md)
- [Device Manager](galaxy/client/device_manager.md)
### UFO² Desktop AgentOS
- [UFO² Overview](ufo2/overview.md)
- [HostAgent](ufo2/host_agent/overview.md)
- [AppAgent](ufo2/app_agent/overview.md)
- [Core Features](ufo2/core_features/hybrid_actions.md)
- [As Galaxy Device](ufo2/as_galaxy_device.md)
### Linux Agent
- [Linux Agent Overview](linux/overview.md)
- [As Galaxy Device](linux/as_galaxy_device.md)
### Mobile Agent
- [Mobile Agent Overview](mobile/overview.md)
- [As Galaxy Device](mobile/as_galaxy_device.md)
### MCP System
- [MCP Overview](mcp/overview.md)
- [Local Servers](mcp/local_servers.md)
- [Creating MCP Servers](tutorials/creating_mcp_servers.md)
### Agent Integration Protocol
- [AIP Overview](aip/overview.md)
- [Message Types](aip/messages.md)
- [Transport Layers](aip/transport.md)
### Configuration
- [Configuration Overview](configuration/system/overview.md)
- [Agents Configuration](configuration/system/agents_config.md)
- [System Configuration](configuration/system/system_config.md)
- [Model Configuration](configuration/models/overview.md)
- [MCP Configuration](configuration/system/mcp_reference.md)
---
## 🏗️ Architecture Principles
UFO³ follows **SOLID principles** and established software engineering patterns:
- **Single Responsibility**: Each component has a focused purpose
- **Open/Closed**: Extensible through interfaces and plugins
- **Interface Segregation**: Focused interfaces for different capabilities
- **Dependency Inversion**: Dependency injection for loose coupling
- **Event-Driven**: Observer pattern for real-time monitoring
- **State Machines**: Well-defined states and transitions for agents
- **Command Pattern**: Encapsulated DAG editing with undo/redo
---
## 📝 Additional Resources
- **[GitHub Repository](https://github.com/microsoft/UFO)** - Source code and issues
- **[Research Paper](https://arxiv.org/abs/2504.14603)** - UFO³ technical details
- **[Documentation Site](https://microsoft.github.io/UFO/)** - Full documentation
- **[Video Demo](https://www.youtube.com/watch?v=QT_OhygMVXU)** - YouTube demonstration
---
**Next Steps:**
1. Start with [Galaxy Quick Start](getting_started/quick_start_galaxy.md) for multi-device orchestration
2. Or explore [UFO² Quick Start](getting_started/quick_start_ufo2.md) for single-device automation
3. Check [FAQ](faq.md) for common questions
4. Join our community and contribute!
---
### Documents Docs About CODE OF CONDUCT (`documents/docs/about/CODE_OF_CONDUCT.md`)
# Microsoft Open Source Code of Conduct
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
Resources:
- [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/)
- [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
- Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns
---
### Documents Docs About CONTRIBUTING (`documents/docs/about/CONTRIBUTING.md`)
# Contributing
This project welcomes contributions and suggestions. Most contributions require you to
agree to a Contributor License Agreement (CLA) declaring that you have the right to,
and actually do, grant us the rights to use your contribution. For details, visit
https://cla.microsoft.com.
When you submit a pull request, a CLA-bot will automatically determine whether you need
to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the
instructions provided by the bot. You will only need to do this once across all repositories using our CLA.
!!! note
You should sunmit your pull request to the `pre-release` branch, not the `main` branch.
This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
or contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
---
### Documents Docs About DISCLAIMER (`documents/docs/about/DISCLAIMER.md`)
# Disclaimer: Code Execution and Data Handling Notice
By choosing to run the provided code, you acknowledge and agree to the following terms and conditions regarding the functionality and data handling practices:
## 1. Code Functionality:
The code you are about to execute has the capability to capture screenshots of your working desktop environment and active applications. These screenshots will be processed and sent to the GPT model for inference.
## 2. Data Privacy and Storage:
It is crucial to note that Microsoft, the provider of this code, explicitly states that it does not collect or save any of the transmitted data. The captured screenshots are processed in real-time for the purpose of inference, and no permanent storage or record of this data is retained by Microsoft.
## 3. User Responsibility:
By running the code, you understand and accept the responsibility for the content and nature of the data present on your desktop during the execution period. It is your responsibility to ensure that no sensitive or confidential information is visible or captured during this process.
## 4. Security Measures:
Microsoft has implemented security measures to safeguard the action execution. However, it is recommended that you run the code in a secure and controlled environment to minimize potential risks. Ensure that you are running the latest security updates on your system.
## 5. Consent for Inference:
You explicitly provide consent for the GPT model to analyze the captured screenshots for the purpose of generating relevant outputs. This consent is inherent in the act of executing the code.
## 6. No Guarantee of Accuracy:
The outputs generated by the GPT model are based on patterns learned during training and may not always be accurate or contextually relevant. Microsoft does not guarantee the accuracy or suitability of the inferences made by the model.
## 7. Indemnification:
Users agree to defend, indemnify, and hold Microsoft harmless from and against all damages, costs, and attorneys' fees in connection with any claims arising from the use of this Repo.
## 8. Reporting Infringements:
If anyone believes that this Repo infringes on their rights, please notify the project owner via the provided project owner email. Microsoft will investigate and take appropriate actions as necessary.
## 9. Modifications to the Disclaimer:
Microsoft reserves the right to update or modify this disclaimer at any time without prior notice. It is your responsibility to review the disclaimer periodically for any changes.
By proceeding to execute the code, you acknowledge that you have read, understood, and agreed to the terms outlined in this disclaimer. If you do not agree with these terms, refrain from running the provided code.
---
### Documents Docs About SUPPORT (`documents/docs/about/SUPPORT.md`)
# Support
## How to file issues and get help
This project uses GitHub Issues to track bugs and feature requests. Please search the existing
issues before filing new issues to avoid duplicates. For new issues, file your bug or
feature request as a new Issue.
You may use [GitHub Issues](https://github.com/microsoft/UFO/issues) to raise questions, bug reports, and feature requests.
For help and questions about using this project, please please contact [ufo-agent@microsoft.com](mailto:ufo-agent@microsoft.com).
## Microsoft Support Policy
Support for this **PROJECT or PRODUCT** is limited to the resources listed above.
---
### Documents Docs Aip Endpoints (`documents/docs/aip/endpoints.md`)
# AIP Endpoints
Endpoints combine protocol, transport, and resilience components to provide production-ready AIP communication for servers, clients, and orchestrators.
## Endpoint Types at a Glance
| Endpoint Type | Role | Used By | Key Features |
|---------------|------|---------|--------------|
| **DeviceServerEndpoint** | Server | Device Agent Service | ✅ Multiplexed connections ✅ Session management ✅ Task dispatching ✅ Result aggregation |
| **DeviceClientEndpoint** | Client | Device Agent Client | ✅ Auto-reconnection ✅ Heartbeat management ✅ Command execution ✅ Telemetry reporting |
| **ConstellationEndpoint** | Orchestrator | ConstellationClient | ✅ Multi-device coordination ✅ Task distribution ✅ Device info querying ✅ Connection pooling |
---
## Endpoint Architecture
**Endpoint Inheritance Hierarchy:**
AIP provides three specialized endpoint implementations that all inherit common functionality from a shared base class:
```mermaid
graph TB
Base[AIPEndpoint Base]
Base --> Server[DeviceServerEndpoint Server-Side]
Base --> Client[DeviceClientEndpoint Client-Side]
Base --> Constellation[ConstellationEndpoint Orchestrator]
Base -.->|Protocol| P[Message Handling]
Base -.->|Resilience| R[Reconnection + Heartbeat]
Base -.->|Sessions| S[State Tracking]
style Base fill:#e1f5ff
style Server fill:#fff4e1
style Client fill:#f0ffe1
style Constellation fill:#ffe1f5
```
The dashed arrows indicate capabilities that the base class provides to all subclasses. This inheritance design ensures consistent behavior across all endpoint types while allowing specialization for server, client, and orchestrator roles.
**Base Endpoint Components:**
All endpoints inherit from `AIPEndpoint`, which provides:
- **Protocol**: Message serialization and handling
- **Reconnection Strategy**: Automatic reconnection with backoff
- **Timeout Manager**: Operation timeout management
- **Session Handlers**: Per-session state tracking
## Base Endpoint: AIPEndpoint
### Common Methods
| Method | Purpose | Example Usage |
|--------|---------|---------------|
| `start()` | Start endpoint | `await endpoint.start()` |
| `stop()` | Stop endpoint | `await endpoint.stop()` |
| `is_connected()` | Check connection | `if endpoint.is_connected(): ...` |
| `send_with_timeout()` | Send with timeout | `await endpoint.send_with_timeout(msg, 30.0)` |
| `receive_with_timeout()` | Receive with timeout | `msg = await endpoint.receive_with_timeout(ServerMessage, 60.0)` |
**Basic Usage Pattern:**
```python
from aip.endpoints.base import AIPEndpoint
# Start endpoint
await endpoint.start()
# Check connection
if endpoint.is_connected():
await endpoint.handle_message(msg)
# Send with timeout
await endpoint.send_with_timeout(msg, timeout=30.0)
# Clean shutdown
await endpoint.stop()
```
---
## DeviceServerEndpoint
Wraps UFO's server-side WebSocket handler with AIP protocol support for managing multiple device connections simultaneously.
### Configuration
```python
from aip.endpoints import DeviceServerEndpoint
endpoint = DeviceServerEndpoint(
ws_manager=ws_manager, # WebSocket connection manager
session_manager=session_manager, # Session state manager
local=False # Local vs remote deployment
)
```
### Integration with FastAPI
```python
from fastapi import FastAPI, WebSocket
from aip.endpoints import DeviceServerEndpoint
app = FastAPI()
endpoint = DeviceServerEndpoint(ws_manager, session_manager)
@app.websocket("/ws")
async def websocket_route(websocket: WebSocket):
await endpoint.handle_websocket(websocket)
```
### Key Features
| Feature | Description | Benefit |
|---------|-------------|---------|
| **Multiplexed Connections** | Handle multiple clients simultaneously | Scale to many devices |
| **Session Management** | Track active sessions per device | Maintain conversation context |
| **Task Dispatching** | Route tasks to appropriate clients | Targeted execution |
| **Result Aggregation** | Collect and format execution results | Unified response handling |
| **Auto Task Cancellation** | Cancel tasks on disconnect | Prevent orphaned tasks |
**Backward Compatibility:**
The Device Server Endpoint maintains full compatibility with UFO's existing WebSocket handler.
### Task Cancellation on Disconnection
```python
# Automatically called when device disconnects
await endpoint.cancel_device_tasks(
device_id="device_001",
reason="device_disconnected"
)
```
---
## DeviceClientEndpoint
Wraps UFO's client-side WebSocket client with AIP protocol support, automatic reconnection, and heartbeat management.
### Configuration
```python
from aip.endpoints import DeviceClientEndpoint
endpoint = DeviceClientEndpoint(
ws_url="ws://localhost:8000/ws",
ufo_client=ufo_client,
max_retries=3,
timeout=120.0
)
```
### Automatic Features
| Feature | Default Behavior | Configuration |
|---------|------------------|---------------|
| **Heartbeat** | Starts on connection | 20s interval (fixed) |
| **Reconnection** | Exponential backoff | `max_retries=3`, `initial_backoff=2.0` |
| **Message Routing** | Auto-routes to UFO client | Handled internally |
| **Connection Management** | Auto-connect on start | Transparent to user |
**Lifecycle Management Example:**
```python
# Start and connect
await endpoint.start()
# Handle messages automatically
# (routed to underlying UFO client)
# Stop heartbeat and close
await endpoint.stop()
```
### Reconnection Strategy
```python
from aip.resilience import ReconnectionStrategy
reconnection_strategy = ReconnectionStrategy(
max_retries=3,
initial_backoff=2.0,
max_backoff=60.0
)
endpoint = DeviceClientEndpoint(
ws_url=url,
ufo_client=client,
reconnection_strategy=reconnection_strategy
)
```
---
## ConstellationEndpoint
Enables the ConstellationClient to communicate with multiple devices simultaneously, managing connections, tasks, and queries.
### Configuration
```python
from aip.endpoints import ConstellationEndpoint
endpoint = ConstellationEndpoint(
task_name="multi_device_task",
message_processor=processor # Optional custom processor
)
```
### Multi-Device Operations
| Operation | Method | Description |
|-----------|--------|-------------|
| **Connect** | `connect_to_device()` | Establish connection to device |
| **Send Task** | `send_task_to_device()` | Dispatch task to specific device |
| **Query Info** | `request_device_info()` | Get device telemetry |
| **Check Status** | `is_device_connected()` | Verify connection health |
| **Disconnect** | `disconnect_device()` | Close device connection |
| **Disconnect All** | `stop()` | Shutdown all connections |
### Connecting to Devices
```python
# Connect using AgentProfile
connection = await endpoint.connect_to_device(
device_info=agent_profile, # AgentProfile object
message_processor=processor
)
```
Learn more about [AgentProfile configuration](../galaxy/client/device_manager.md) in the Galaxy documentation.
### Sending Tasks
```python
# Dispatch task to specific device
result = await endpoint.send_task_to_device(
device_id="device_001",
task_request={
"request": "Open Notepad",
"task_name": "open_notepad",
"session_id": "session_123"
}
)
```
### Querying Device Info
```python
# Request telemetry update
device_info = await endpoint.request_device_info("device_001")
if device_info:
print(f"OS: {device_info['os']}")
print(f"CPU: {device_info['cpu']}")
print(f"GPU: {device_info.get('gpu', 'N/A')}")
```
### Connection Management
**Managing Multiple Devices:**
```python
# Check connection before sending
if endpoint.is_device_connected("device_001"):
await endpoint.send_task_to_device(...)
# Disconnect specific device
await endpoint.disconnect_device("device_001")
# Disconnect all devices
await endpoint.stop()
```
### Disconnection Handling
```python
# Automatically triggered on device disconnect
await endpoint.on_device_disconnected("device_001")
# Cancels pending tasks
await endpoint.cancel_device_tasks(
device_id="device_001",
reason="device_disconnected"
)
# Attempts reconnection (if enabled)
success = await endpoint.reconnect_device("device_001")
```
---
## Endpoint Lifecycle Patterns
### Server Lifecycle
**Server Endpoint State Transitions:**
This state diagram shows the lifecycle of a server endpoint from initialization through connection handling to shutdown:
```mermaid
stateDiagram-v2
[*] --> Initialize: Create endpoint
Initialize --> Started: start()
Started --> Listening: Accept connections
Listening --> Handling: Handle WebSocket
Handling --> Listening: Connection closed
Listening --> Stopped: stop()
Stopped --> [*]
```
The `Listening → Handling` loop represents the server accepting multiple client connections. Each connection is handled independently while the server remains in the listening state.
**Server Lifecycle Code:**
```python
# 1. Initialize
endpoint = DeviceServerEndpoint(client_manager, session_manager)
# 2. Start
await endpoint.start()
# 3. Handle connections
@app.websocket("/ws")
async def handle_ws(websocket: WebSocket):
await endpoint.handle_websocket(websocket)
# 4. Stop (on shutdown)
await endpoint.stop()
```
### Client Lifecycle
**Client Endpoint State Transitions with Auto-Reconnection:**
This diagram shows the client lifecycle including automatic reconnection attempts when the connection is lost:
```mermaid
stateDiagram-v2
[*] --> Initialize: Create endpoint
Initialize --> Connecting: start()
Connecting --> Connected: Connection established
Connected --> Heartbeat: Auto-start heartbeat
Heartbeat --> Handling: Handle messages
Handling --> Heartbeat: Continue
Heartbeat --> Reconnecting: Connection lost
Reconnecting --> Connected: Reconnect successful
Reconnecting --> Stopped: Max retries
Connected --> Stopped: stop()
Stopped --> [*]
```
The `Heartbeat → Handling` loop represents normal operation with periodic heartbeats. The `Reconnecting → Connected` transition shows automatic recovery from network failures.
**Client Lifecycle Code:**
```python
# 1. Initialize
endpoint = DeviceClientEndpoint(ws_url, ufo_client)
# 2. Connect
await endpoint.start()
# 3. Handle messages (automatic)
# UFO client receives and processes messages
# 4. Disconnect
await endpoint.stop()
```
### Constellation Lifecycle
**Constellation Lifecycle Code:**
```python
# 1. Initialize
endpoint = ConstellationEndpoint(task_name)
# 2. Start
await endpoint.start()
# 3. Connect to devices
await endpoint.connect_to_device(device_info1)
await endpoint.connect_to_device(device_info2)
# 4. Send tasks
await endpoint.send_task_to_device(device_id, task_request)
# 5. Cleanup (disconnects all devices)
await endpoint.stop()
```
---
## Resilience Features
!!!warning "Built-In Resilience"
All endpoints include automatic reconnection, timeout management, and heartbeat monitoring for production reliability.
### Resilience Configuration
| Component | Configuration | Purpose |
|-----------|---------------|---------|
| **Reconnection** | `ReconnectionStrategy` | Auto-reconnect with backoff |
| **Timeout** | `TimeoutManager` | Enforce operation timeouts |
| **Heartbeat** | `HeartbeatManager` | Monitor connection health |
**Configuring Resilience:**
```python
from aip.resilience import ReconnectionStrategy, ReconnectionPolicy
strategy = ReconnectionStrategy(
max_retries=5,
initial_backoff=1.0,
max_backoff=60.0,
backoff_multiplier=2.0,
policy=ReconnectionPolicy.EXPONENTIAL_BACKOFF
)
endpoint = DeviceClientEndpoint(
ws_url=url,
ufo_client=client,
reconnection_strategy=strategy
)
```
### Timeout Operations
```python
# Send with custom timeout
await endpoint.send_with_timeout(msg, timeout=30.0)
# Receive with custom timeout
msg = await endpoint.receive_with_timeout(ServerMessage, timeout=60.0)
```
[→ See detailed resilience documentation](./resilience.md)
---
## Error Handling Patterns
### Connection Errors
```python
try:
await endpoint.start()
except ConnectionError as e:
logger.error(f"Failed to connect: {e}")
# Reconnection handled automatically if enabled
```
### Task Execution Errors
```python
try:
result = await endpoint.send_task_to_device(device_id, task)
except TimeoutError:
logger.error("Task execution timeout")
except Exception as e:
logger.error(f"Task failed: {e}")
```
### Custom Disconnection Handling
```python
class CustomEndpoint(DeviceClientEndpoint):
async def on_device_disconnected(self, device_id: str) -> None:
logger.warning(f"Device {device_id} disconnected")
# Custom cleanup logic
await self.custom_cleanup(device_id)
# Call parent implementation
await super().on_device_disconnected(device_id)
```
---
## Best Practices
**Endpoint Selection:**
| Use Case | Endpoint Type |
|----------|---------------|
| Device agent server | `DeviceServerEndpoint` |
| Device agent client | `DeviceClientEndpoint` |
| Multi-device orchestrator | `ConstellationEndpoint` |
!!!warning "Configuration Guidelines"
- **Set appropriate timeouts** based on deployment environment
- **Configure reconnection** based on network reliability
- **Monitor connection health** with `is_connected()` checks
- **Implement custom handlers** for application-specific cleanup
!!!success "Resource Management"
- **Always call `stop()`** during shutdown to prevent leaks
- **Use message processors** for custom message handling
- **Handle disconnections** with `on_device_disconnected` overrides
**Custom Message Processor:**
```python
class MyProcessor:
async def process_message(self, msg):
# Custom processing
logger.info(f"Processing: {msg.type}")
# ...
endpoint = ConstellationEndpoint(
task_name="task",
message_processor=MyProcessor()
)
```
---
## Quick Reference
### Import Endpoints
```python
from aip.endpoints import (
AIPEndpoint, # Base class
DeviceServerEndpoint, # Server-side
DeviceClientEndpoint, # Client-side
ConstellationEndpoint, # Orchestrator-side
)
```
### Related Documentation
- [Protocol Reference](./protocols.md) - Protocol implementations used by endpoints
- [Transport Layer](./transport.md) - Transport configuration and options
- [Resilience](./resilience.md) - Reconnection and heartbeat management
- [Messages](./messages.md) - Message types and validation
- [Overview](./overview.md) - System architecture and design
- [Galaxy Client](../galaxy/client/overview.md) - Multi-device orchestration with ConstellationClient
- [UFO Server](../server/websocket_handler.md) - WebSocket server implementation
- [UFO Client](../client/websocket_client.md) - WebSocket client implementation
---
### Documents Docs Aip Messages (`documents/docs/aip/messages.md`)
# AIP Message Reference
AIP uses **Pydantic-based messages** for automatic validation, serialization, and type safety. All messages transmit as JSON over WebSocket.
## Message Overview
### Bidirectional Communication
**Message Flow Overview:**
This diagram illustrates all message types and their directions in the AIP protocol, showing how clients and servers communicate bidirectionally:
```mermaid
graph LR
Client[Device Client]
Server[Device Service]
Client -->|REGISTER| Server
Client -->|COMMAND_RESULTS| Server
Client -->|TASK_END| Server
Client -->|HEARTBEAT| Server
Server -->|TASK| Client
Server -->|COMMAND| Client
Server -->|HEARTBEAT| Client
Server -->|TASK_END| Client
Client <-->|DEVICE_INFO| Server
Client <-->|ERROR| Server
style Client fill:#f0ffe1
style Server fill:#fff4e1
```
Unidirectional arrows indicate request-response patterns, while bidirectional arrows (`<-->`) indicate messages that can be initiated by either party. Note that both `HEARTBEAT` and `TASK_END` can flow in both directions depending on the scenario.
### Message Types Quick Reference
| Direction | Message Type | Purpose | Key Fields |
|-----------|--------------|---------|------------|
| **Client → Server** | | | |
| | `REGISTER` | Initial capability advertisement | `client_id`, `metadata` |
| | `COMMAND_RESULTS` | Return command execution results | `action_results`, `prev_response_id` |
| | `TASK_END` | Notify task completion | `status`, `session_id` |
| | `HEARTBEAT` | Keepalive signal | `client_id` |
| **Server → Client** | | | |
| | `TASK` | Task assignment | `user_request`, `task_name` |
| | `COMMAND` | Command execution request | `actions`, `response_id` |
| | `HEARTBEAT` | Keepalive acknowledgment | `response_id` |
| | `TASK_END` | Task completion notification | `status`, `result` |
| **Bidirectional** | | | |
| | `DEVICE_INFO_REQUEST` | Request device telemetry | `request_id` |
| | `DEVICE_INFO_RESPONSE` | Device information | Device specs |
| | `ERROR` | Error condition | `error` message |
---
---
## Core Data Structures
These Pydantic models form the building blocks for all AIP messages.
### Essential Types Summary
| Type | Purpose | Key Fields | Usage |
|------|---------|------------|-------|
| **Rect** | UI element coordinates | `x`, `y`, `width`, `height` | UI automation |
| **ControlInfo** | UI control metadata | `annotation_id`, `name`, `rectangle` | Control discovery |
| **WindowInfo** | Window metadata | `process_id`, `is_active` (extends ControlInfo) | Window management |
| **MCPToolInfo** | Tool definition | `tool_key`, `namespace`, `input_schema` | Capability advertisement |
| **Command** | Execution request | `tool_name`, `parameters`, `call_id` | Action dispatch |
| **Result** | Execution outcome | `status`, `result`, `error` | Result reporting |
### Rect (Rectangle)
Represents UI element bounding box.
```python
rect = Rect(x=100, y=200, width=300, height=150)
```
| Field | Type | Description |
|-------|------|-------------|
| `x` | int | X-coordinate (top-left) |
| `y` | int | Y-coordinate (top-left) |
| `width` | int | Width in pixels |
| `height` | int | Height in pixels |
### ControlInfo
UI control element metadata.
**ControlInfo Example:**
```python
control = ControlInfo(
annotation_id="ctrl_001",
name="Submit Button",
class_name="Button",
rectangle=Rect(x=100, y=200, width=80, height=30),
is_enabled=True,
is_visible=True
)
```
**Complete Field List:**
| Field | Type | Description |
|-------|------|-------------|
| `annotation_id` | str? | Unique annotation identifier |
| `name` | str? | Control name |
| `title` | str? | Control title |
| `handle` | int? | Windows handle (HWND) |
| `class_name` | str? | UI class name |
| `rectangle` | Rect? | Bounding rectangle |
| `control_type` | str? | Type (Button, TextBox, etc.) |
| `automation_id` | str? | UI Automation ID |
| `is_enabled` | bool? | Enabled state |
| `is_visible` | bool? | Visibility state |
| `source` | str? | Data source identifier |
| `text_content` | str? | Text content |
### WindowInfo
Window metadata (extends ControlInfo).
**Additional Fields:**
| Field | Type | Description |
|-------|------|-------------|
| `process_id` | int? | Process ID (PID) |
| `process_name` | str? | Process name (e.g., "notepad.exe") |
| `is_minimized` | bool? | Minimized state |
| `is_maximized` | bool? | Maximized state |
| `is_active` | bool? | Has focus |
### MCPToolInfo
MCP tool capability definition.
**Tool Advertisement:**
Device agents use `MCPToolInfo` to advertise their capabilities during registration.
```python
tool_info = MCPToolInfo(
tool_key="ui_automation.click_button",
tool_name="click_button",
namespace="ui_automation",
tool_type="action",
description="Click a button by its ID",
input_schema={
"type": "object",
"properties": {
"button_id": {"type": "string"}
}
}
)
```
| Field | Type | Description |
|-------|------|-------------|
| `tool_key` | str | Unique key (`namespace.tool_name`) |
| `tool_name` | str | Tool name |
| `namespace` | str | MCP namespace |
| `tool_type` | str | `"action"` or `"data_collection"` |
| `description` | str? | Tool description |
| `input_schema` | dict? | JSON schema for inputs |
| `output_schema` | dict? | JSON schema for outputs |
| `meta` | dict? | Metadata |
| `annotations` | dict? | Additional annotations |
Learn more about [MCP tools and capabilities](../mcp/overview.md).
---
## Command and Result Structures
### Command
Execution request sent to device agents.
**Command Structure:**
```python
cmd = Command(
tool_name="click_element",
parameters={"control_id": "btn_submit"},
tool_type="action",
call_id="cmd_12345"
)
```
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `tool_name` | str | ✅ | Name of tool to execute |
| `parameters` | dict | | Tool parameters |
| `tool_type` | str | ✅ | `"data_collection"` or `"action"` |
| `call_id` | str | | Unique identifier for correlation |
**Call ID Correlation:**
Use `call_id` to match commands with their results in the `Result` object.
### ResultStatus
Execution outcome enumeration.
| Status | Meaning | When to Use |
|--------|---------|-------------|
| `SUCCESS` | ✅ Completed successfully | Command executed without errors |
| `FAILURE` | ❌ Failed with error | Execution encountered an error |
| `SKIPPED` | ⏭️ Skipped execution | Conditional execution, not run |
| `NONE` | ⚪ No status | Initial/unknown state |
### Result
Command execution outcome.
!!!warning "Always Check Status"
Check `status` before accessing `result`. If `FAILURE`, use `error` field for diagnostics.
```python
# Success result
result = Result(
status=ResultStatus.SUCCESS,
result={"element_found": True, "clicked": True},
namespace="ui_automation",
call_id="cmd_12345"
)
# Failure result
result = Result(
status=ResultStatus.FAILURE,
error="Element not found: btn_submit",
namespace="ui_automation",
call_id="cmd_12345"
)
```
| Field | Type | Description |
|-------|------|-------------|
| `status` | ResultStatus | Execution status |
| `error` | str? | Error message (if FAILURE) |
| `result` | Any | Result payload (type varies by tool) |
| `namespace` | str? | Namespace of executed tool |
| `call_id` | str? | Matches Command.call_id |
---
## Status Enumerations
### TaskStatus
Task lifecycle states.
**State Transitions:**
**Task Lifecycle State Machine:**
This diagram shows the possible state transitions during task execution, including the multi-turn loop and terminal states:
```mermaid
stateDiagram-v2
[*] --> CONTINUE: Task starts
CONTINUE --> CONTINUE: Multi-step
CONTINUE --> COMPLETED: Success
CONTINUE --> FAILED: Error
OK --> OK: Heartbeat
ERROR --> [*]: Terminal
```
The `CONTINUE → CONTINUE` self-loop represents multi-turn execution where tasks request additional commands before completion. `COMPLETED` and `FAILED` are terminal success/failure states.
| Status | Meaning | Usage |
|--------|---------|-------|
| `CONTINUE` | 🔄 Task ongoing | Multi-turn execution, more steps needed |
| `COMPLETED` | ✅ Task done | Successful completion |
| `FAILED` | ❌ Task failed | Error encountered |
| `OK` | ✓ Acknowledgment | Heartbeat, health check passed |
| `ERROR` | ⚠️ Protocol error | Protocol-level error |
**Multi-Turn Execution:**
`CONTINUE` enables agents to request additional commands before marking a task as complete, supporting complex multi-step workflows.
---
## Client Types
### ClientType
Identifies the type of client connecting to the server.
| Type | Role | Characteristics |
|------|------|----------------|
| `DEVICE` | Device agent executor | • Executes tasks locally • Reports telemetry • Single-device focus |
| `CONSTELLATION` | Multi-device orchestrator | • Manages multiple devices • Coordinates tasks • Requires `target_id` |
**Registration by Type:**
```python
# Device client
device_msg = ClientMessage(
type=ClientMessageType.REGISTER,
client_type=ClientType.DEVICE,
client_id="device_001"
)
# Constellation client
constellation_msg = ClientMessage(
type=ClientMessageType.REGISTER,
client_type=ClientType.CONSTELLATION,
client_id="orchestrator_001",
target_id="device_001" # Target device
)
```
---
## ClientMessage (Client → Server)
Devices and constellation clients use `ClientMessage` to communicate with the server.
### Message Types
| Type | Purpose | Required Fields |
|------|---------|----------------|
| **REGISTER** | Initial registration | `client_id`, `client_type` |
| **HEARTBEAT** | Keepalive | `client_id`, `status=OK` |
| **TASK** | Request task execution | `request`, `client_id` |
| **TASK_END** | Notify completion | `session_id`, `status` |
| **COMMAND_RESULTS** | Return results | `action_results`, `prev_response_id` |
| **DEVICE_INFO_REQUEST** | Request telemetry | `request_id` |
| **DEVICE_INFO_RESPONSE** | Provide telemetry | Device data |
| **ERROR** | Report error | `error` |
### Common Fields
| Field | Type | Description |
|-------|------|-------------|
| `type` | ClientMessageType | Message type |
| `status` | TaskStatus | Current task status |
| `client_type` | ClientType | DEVICE or CONSTELLATION |
| `session_id` | str? | Session identifier |
| `task_name` | str? | Human-readable task name |
| `client_id` | str? | Unique client identifier |
| `target_id` | str? | Target device (for constellation) |
| `request` | str? | Request text (for TASK) |
| `action_results` | List[Result]? | Command results |
| `timestamp` | str? | ISO 8601 timestamp |
| `request_id` | str? | Unique request identifier |
| `prev_response_id` | str? | Previous response ID |
| `error` | str? | Error message |
| `metadata` | dict? | Additional metadata |
### Example: REGISTER
```python
register_msg = ClientMessage(
type=ClientMessageType.REGISTER,
client_type=ClientType.DEVICE,
client_id="windows_agent_001",
status=TaskStatus.OK,
timestamp="2024-11-04T10:30:00Z",
metadata={
"platform": "windows",
"os_version": "Windows 11",
"capabilities": ["ui_automation", "file_operations"]
}
)
```
### Example: COMMAND_RESULTS
```python
results_msg = ClientMessage(
type=ClientMessageType.COMMAND_RESULTS,
client_id="windows_agent_001",
session_id="session_123",
prev_response_id="resp_456", # Links to server's COMMAND message
status=TaskStatus.CONTINUE,
action_results=[
Result(status=ResultStatus.SUCCESS, result={"clicked": True}),
Result(status=ResultStatus.SUCCESS, result={"text_entered": True})
],
timestamp="2024-11-04T10:31:00Z",
request_id="req_789"
)
```
---
## ServerMessage (Server → Client)
Device services use `ServerMessage` to assign tasks and send commands to clients.
### Message Types
| Type | Purpose | Required Fields |
|------|---------|----------------|
| **TASK** | Assign task | `user_request`, `task_name`, `session_id` |
| **COMMAND** | Execute commands | `actions`, `response_id`, `session_id` |
| **TASK_END** | Notify completion | `status`, `session_id` |
| **HEARTBEAT** | Keepalive ack | `response_id` |
| **DEVICE_INFO_REQUEST** | Request telemetry | `request_id` |
| **DEVICE_INFO_RESPONSE** | Telemetry data | Device info |
| **ERROR** | Error notification | `error` |
### Common Fields
| Field | Type | Description |
|-------|------|-------------|
| `type` | ServerMessageType | Message type |
| `status` | TaskStatus | Current task status |
| `user_request` | str? | Original user request |
| `agent_name` | str? | Agent handling task |
| `process_name` | str? | Process for execution context |
| `root_name` | str? | Root application name |
| `actions` | List[Command]? | Commands to execute |
| `messages` | List[str]? | Log messages |
| `error` | str? | Error description |
| `session_id` | str? | Session identifier |
| `task_name` | str? | Task name |
| `timestamp` | str? | ISO 8601 timestamp |
| `response_id` | str? | Response identifier |
| `result` | Any? | Result payload |
### Example: TASK Assignment
```python
task_msg = ServerMessage(
type=ServerMessageType.TASK,
status=TaskStatus.CONTINUE,
user_request="Open Notepad and create a new file",
task_name="create_notepad_file",
session_id="session_123",
response_id="resp_001",
agent_name="AppAgent",
process_name="notepad.exe",
timestamp="2024-11-04T10:30:00Z"
)
```
### Example: COMMAND Execution
```python
command_msg = ServerMessage(
type=ServerMessageType.COMMAND,
status=TaskStatus.CONTINUE,
session_id="session_123",
response_id="resp_456",
actions=[
Command(
tool_name="launch_application",
parameters={"app_name": "notepad"},
tool_type="action",
call_id="cmd_001"
),
Command(
tool_name="type_text",
parameters={"text": "Hello World"},
tool_type="action",
call_id="cmd_002"
)
],
timestamp="2024-11-04T10:30:30Z"
)
```
### Example: TASK_END
```python
task_end_msg = ServerMessage(
type=ServerMessageType.TASK_END,
status=TaskStatus.COMPLETED,
session_id="session_123",
response_id="resp_999",
result={
"file_created": True,
"path": "C:\\Users\\user\\document.txt"
},
timestamp="2024-11-04T10:35:00Z"
)
```
---
## Message Validation
!!!warning "Built-In Validation"
AIP provides `MessageValidator` class for ensuring message integrity. Always validate messages before processing to prevent protocol errors.
### Validation Methods
| Method | Purpose | Requirements |
|--------|---------|-------------|
| `validate_registration()` | Check registration | `type=REGISTER`, `client_id` present |
| `validate_task_request()` | Check task request | `type=TASK`, `request` and `client_id` present |
| `validate_command_results()` | Check results | `type=COMMAND_RESULTS`, `prev_response_id` present |
| `validate_server_message()` | Check server msg | `type` and `status` present |
**Validation Usage:**
```python
from aip.messages import MessageValidator
# Validate registration
if MessageValidator.validate_registration(client_message):
await process_registration(client_message)
# Validate task request
if MessageValidator.validate_task_request(client_message):
await dispatch_task(client_message)
# Validate command results
if MessageValidator.validate_command_results(client_message):
await process_results(client_message)
```
---
## Message Correlation
AIP uses identifier chains to maintain conversation context across multiple message exchanges.
### Correlation Pattern
**Message Identifier Chaining:**
This sequence diagram demonstrates how messages are linked together using correlation IDs to maintain conversation context:
```mermaid
sequenceDiagram
participant C as Client
participant S as Server
C->>S: request_id: "req_001"
S->>C: response_id: "resp_001"
C->>S: request_id: "req_002" prev_response_id: "resp_001"
S->>C: response_id: "resp_002"
```
Each new request includes `prev_response_id` pointing to the previous server response, forming a traceable conversation chain. This pattern enables audit trails, debugging, and request-response correlation in multi-turn conversations.
### Correlation Fields
| Field | Purpose | Example |
|-------|---------|---------|
| `request_id` | Unique request identifier | `"req_abc123"` |
| `response_id` | Unique response identifier | `"resp_def456"` |
| `prev_response_id` | Links to previous response | `"resp_def456"` |
| `session_id` | Groups related messages | `"session_xyz"` |
| `call_id` | Correlates commands/results | `"cmd_001"` |
### Session Tracking
**Session-Based Grouping:**
All messages within a task execution share the same `session_id` for traceability.
```python
# All messages use same session_id
SESSION_ID = "session_abc123"
task_msg.session_id = SESSION_ID
command_msg.session_id = SESSION_ID
results_msg.session_id = SESSION_ID
task_end_msg.session_id = SESSION_ID
```
---
## Best Practices
!!!success "Message Construction"
**Timestamps**: Always use ISO 8601 format
```python
from datetime import datetime, timezone
timestamp = datetime.now(timezone.utc).isoformat()
```
**Unique IDs**: Generate UUIDs for correlation
```python
import uuid
request_id = str(uuid.uuid4())
```
!!!warning "Error Handling"
- Check `Result.status` before accessing result data
- Always provide meaningful error messages
- Use `ResultStatus.FAILURE` with descriptive `error` field
**Extensibility:**
- Use `metadata` field for custom data without breaking protocol
- Leverage Pydantic's validation for type safety
- Always correlate messages with `prev_response_id`
---
## Quick Reference
### Import Messages
```python
from aip.messages import (
ClientMessage,
ServerMessage,
ClientMessageType,
ServerMessageType,
ClientType,
TaskStatus,
Command,
Result,
ResultStatus,
MessageValidator,
)
```
### Related Documentation
- [Protocol Guide](./protocols.md) - How protocols construct and use messages
- [Endpoints](./endpoints.md) - How endpoints handle messages
- [Overview](./overview.md) - High-level message flow in system architecture
- [Transport Layer](./transport.md) - WebSocket transport for message delivery
- [Resilience](./resilience.md) - Message retry and timeout handling
- [MCP Integration](../mcp/overview.md) - How MCP tools integrate with AIP messages
---
### Documents Docs Aip Protocols (`documents/docs/aip/protocols.md`)
# AIP Protocol Reference
## Protocol Stack Overview
AIP uses a three-layer architecture where specialized protocols handle domain-specific concerns, the core protocol manages message processing, and the transport layer provides network communication.
```mermaid
graph TB
subgraph "Specialized Protocols"
RP[RegistrationProtocol]
TEP[TaskExecutionProtocol]
CP[CommandProtocol]
HP[HeartbeatProtocol]
DIP[DeviceInfoProtocol]
end
subgraph "Core Protocol"
AIP["AIPProtocol Message serialization Middleware pipeline Message routing"]
end
subgraph "Transport Layer"
WS[WebSocket]
HTTP3[HTTP/3 Future]
GRPC[gRPC Future]
end
RP --> AIP
TEP --> AIP
CP --> AIP
HP --> AIP
DIP --> AIP
AIP --> WS
AIP -.-> HTTP3
AIP -.-> GRPC
style AIP fill:#e1f5ff
style WS fill:#f0ffe1
```
This layered design enables clean separation of concerns: specialized protocols implement domain logic, the core protocol handles serialization and routing, and the transport layer abstracts network details. Dashed arrows indicate future transport options.
### Protocol Comparison
| Protocol | Purpose | Key Messages | Use When |
|----------|---------|--------------|----------|
| **RegistrationProtocol** | Agent capability advertisement | `REGISTER`, `HEARTBEAT(OK)` | Device joins constellation |
| **TaskExecutionProtocol** | Task lifecycle management | `TASK`, `COMMAND`, `TASK_END` | Executing multi-step tasks |
| **CommandProtocol** | Command validation | Validation utilities | Before sending/receiving commands |
| **HeartbeatProtocol** | Connection health monitoring | `HEARTBEAT` | Periodic keepalive |
| **DeviceInfoProtocol** | Telemetry exchange | `DEVICE_INFO_REQUEST/RESPONSE` | Querying device state |
---
## Core Protocol: AIPProtocol
`AIPProtocol` provides transport-agnostic message handling with middleware support and automatic serialization.
### Quick Start
```python
from aip.protocol import AIPProtocol
from aip.transport import WebSocketTransport
transport = WebSocketTransport()
protocol = AIPProtocol(transport)
```
### Core Operations
| Operation | Method | Description |
|-----------|--------|-------------|
| **Send** | `send_message(msg)` | Serialize and send Pydantic message |
| **Receive** | `receive_message(MsgType)` | Receive and deserialize to type |
| **Dispatch** | `dispatch_message(msg)` | Route to registered handler |
| **Error** | `send_error(error, id)` | Send error notification |
| **Status** | `is_connected()` | Check connection state |
### Middleware Pipeline
Add middleware for logging, authentication, metrics, or custom transformations.
```python
from aip.protocol.base import ProtocolMiddleware
class LoggingMiddleware(ProtocolMiddleware):
async def process_outgoing(self, msg):
logger.info(f"→ {msg.type}")
return msg
async def process_incoming(self, msg):
logger.info(f"← {msg.type}")
return msg
protocol.add_middleware(LoggingMiddleware())
```
**Execution Order:**
- **Outgoing**: First added → First executed
- **Incoming**: Last added → First executed (reverse)
### Message Handler Registration
```python
async def handle_task(msg):
logger.info(f"Handling task: {msg.task_name}")
# Process task...
protocol.register_handler("task", handle_task)
# Auto-dispatch to handler
await protocol.dispatch_message(server_msg)
```
[→ See transport configuration](./transport.md)
---
## RegistrationProtocol {#registration-protocol}
Handles initial registration and capability advertisement when agents join the constellation.
### Registration Flow
The following diagram shows the two-way handshake for device registration, including validation and acknowledgment:
```mermaid
sequenceDiagram
participant C as Client
participant S as Server
C->>S: REGISTER (device_id, metadata, capabilities)
S->>S: Validate registration and Store AgentProfile
alt Success
S->>C: HEARTBEAT (OK)
else Failure
S->>C: ERROR (reason)
end
```
Upon successful registration, the server stores the `AgentProfile` and responds with a `HEARTBEAT` acknowledgment. Failed registrations (e.g., duplicate device_id) return an `ERROR` message with diagnostic details.
### Device Registration
**Client-Side Registration:**
```python
from aip.protocol import RegistrationProtocol
reg_protocol = RegistrationProtocol(transport)
success = await reg_protocol.register_as_device(
device_id="windows_agent_001",
metadata={
"platform": "windows",
"os_version": "Windows 11",
"cpu": "Intel i7",
"ram_gb": 16,
"capabilities": ["ui_automation", "file_operations"]
},
platform="windows"
)
```
**Auto-Added Fields:**
- `timestamp`: Registration time (ISO 8601)
- `client_type`: Set to `ClientType.DEVICE`
[→ See ClientType and ClientMessage in Message Reference](./messages.md)
### Constellation Registration
**Orchestrator Registration:**
```python
success = await reg_protocol.register_as_constellation(
constellation_id="orchestrator_001",
target_device="windows_agent_001", # Required
metadata={
"orchestrator_version": "2.0.0",
"max_concurrent_tasks": 10
}
)
```
!!!warning "Target Device Required"
Constellation clients **must** specify `target_device` to indicate which device they coordinate.
### Server-Side Handlers
| Method | Purpose | When to Use |
|--------|---------|-------------|
| `send_registration_confirmation()` | Acknowledge successful registration | After validating and storing profile |
| `send_registration_error()` | Report registration failure | Invalid ID, duplicate, or validation error |
---
## TaskExecutionProtocol {#task-execution-protocol}
Manages the complete task lifecycle: assignment → command execution → result reporting → completion.
### Task Lifecycle
This state diagram shows the complete task execution lifecycle, including the multi-turn command loop where agents can request additional commands before completion:
```mermaid
stateDiagram-v2
[*] --> TaskAssigned: TASK
TaskAssigned --> CommandSent: COMMAND
CommandSent --> ResultsReceived: COMMAND_RESULTS
ResultsReceived --> CommandSent: CONTINUE
ResultsReceived --> TaskCompleted: COMPLETED/FAILED
TaskCompleted --> [*]: TASK_END
note right of ResultsReceived
Multi-turn: Agent can request
more commands before completion
end note
```
The `CONTINUE` loop (ResultsReceived → CommandSent) enables iterative task refinement where the agent can execute commands, evaluate results, and request follow-up commands before declaring completion.
### Client → Server: Task Request
```python
from aip.protocol import TaskExecutionProtocol
task_protocol = TaskExecutionProtocol(transport)
await task_protocol.send_task_request(
request="Open Notepad and create test.txt",
task_name="create_notepad_file",
session_id="session_123",
client_id="windows_agent_001",
client_type=ClientType.DEVICE,
metadata={"priority": "high"}
)
```
### Server → Client: Task Assignment
```python
await task_protocol.send_task_assignment(
user_request="Open Notepad and create a file",
task_name="create_notepad_file",
session_id="session_123",
response_id="resp_001",
agent_name="AppAgent",
process_name="notepad.exe"
)
```
### Server → Client: Command Dispatch
Send multiple commands in one message to reduce network overhead.
**Method 1: Using ServerMessage**
```python
from aip.messages import ServerMessage, Command, TaskStatus
server_msg = ServerMessage(
type=ServerMessageType.COMMAND,
status=TaskStatus.CONTINUE,
session_id="session_123",
response_id="resp_002",
actions=[
Command(tool_name="launch_application",
parameters={"app_name": "notepad"},
tool_type="action", call_id="cmd_001"),
Command(tool_name="type_text",
parameters={"text": "Hello"},
tool_type="action", call_id="cmd_002")
]
)
await task_protocol.send_command(server_msg)
```
**Method 2: Using send_commands**
```python
await task_protocol.send_commands(
actions=[Command(...)],
session_id="session_123",
response_id="resp_003",
status=TaskStatus.CONTINUE,
agent_name="AppAgent"
)
```
### Client → Server: Command Results
```python
from aip.messages import Result, ResultStatus
await task_protocol.send_command_results(
action_results=[
Result(status=ResultStatus.SUCCESS,
result={"app_launched": True},
call_id="cmd_001"),
Result(status=ResultStatus.SUCCESS,
result={"text_entered": True},
call_id="cmd_002")
],
session_id="session_123",
client_id="windows_agent_001",
prev_response_id="resp_002", # Links to COMMAND message
status=TaskStatus.CONTINUE
)
```
[→ See Result and ResultStatus definitions in Message Reference](./messages.md)
### Task Completion
**Server → Client: Success**
```python
await task_protocol.send_task_end(
session_id="session_123",
status=TaskStatus.COMPLETED,
result={
"file_created": True,
"path": "C:\\Users\\user\\test.txt"
},
response_id="resp_999"
)
```
**Server → Client: Failure**
```python
await task_protocol.send_task_end(
session_id="session_123",
status=TaskStatus.FAILED,
error="Notepad failed to launch: Access denied",
response_id="resp_999"
)
```
### Complete Task Flow
This comprehensive sequence diagram shows the complete flow from task request to completion, including the multi-turn command loop where the agent iteratively executes commands and requests follow-up actions:
```mermaid
sequenceDiagram
participant CC as ConstellationClient
participant CA as ConstellationAgent
participant DS as DeviceService
participant DC as DeviceClient
CC->>CA: TASK request
CA->>DS: TASK assignment
DS->>DC: TASK (forward)
loop Multi-turn execution
DC->>DS: Request COMMAND
DS->>CA: Forward request
CA->>CA: Plan next action
CA->>DS: COMMAND
DS->>DC: COMMAND (forward)
DC->>DC: Execute
DC->>DS: COMMAND_RESULTS
DS->>CA: COMMAND_RESULTS
end
CA->>DS: TASK_END
DS->>DC: TASK_END (forward)
CC->>CC: Update TaskConstellation
```
The loop in the middle represents iterative task execution where the agent can perform multiple command cycles before determining the task is complete. Each cycle involves planning, execution, and result evaluation.
---
## CommandProtocol
Provides validation utilities for commands and results before transmission.
### Validation Methods
| Method | Validates | Returns |
|--------|-----------|---------|
| `validate_command(cmd)` | Single command structure | `bool` |
| `validate_commands(cmds)` | List of commands | `bool` |
| `validate_result(result)` | Single result structure | `bool` |
| `validate_results(results)` | List of results | `bool` |
### Usage Pattern
```python
from aip.protocol import CommandProtocol
cmd_protocol = CommandProtocol(transport)
# Validate before sending
cmd = Command(tool_name="click", parameters={"id": "btn"}, tool_type="action")
if cmd_protocol.validate_command(cmd):
await task_protocol.send_commands([cmd], ...)
else:
logger.error("Invalid command structure")
# Validate results before transmission
results = [Result(...), Result(...)]
if cmd_protocol.validate_results(results):
await task_protocol.send_command_results(results, ...)
```
!!!warning "Validation Best Practice"
Always validate commands and results before transmission to catch protocol errors early and prevent runtime failures.
---
## HeartbeatProtocol {#heartbeat-protocol}
Periodic keepalive messages detect broken connections and network issues.
### Heartbeat Flow
The heartbeat protocol uses a simple ping-pong pattern to verify connection health at regular intervals:
```mermaid
sequenceDiagram
participant C as Client
participant S as Server
loop Every 20-30s
C->>S: HEARTBEAT (client_id)
S->>S: Update last_seen timestamp
S->>C: HEARTBEAT (OK)
end
Note over C,S: If no response → Connection dead
```
If the server fails to receive a heartbeat within the timeout window, it marks the connection as dead and triggers disconnection handling. This prevents silent connection failures from going undetected.
### Client-Side Heartbeat
```python
from aip.protocol import HeartbeatProtocol
heartbeat_protocol = HeartbeatProtocol(transport)
await heartbeat_protocol.send_heartbeat(
client_id="windows_agent_001",
metadata={"custom_info": "value"} # Optional
)
```
### Server-Side Response
```python
await heartbeat_protocol.send_heartbeat_ack(
response_id="resp_hb_001"
)
```
!!!tip "Automatic Management"
The `HeartbeatManager` automates heartbeat sending—you rarely need to call these methods directly.
[→ See HeartbeatManager](./resilience.md#heartbeat-manager)
---
## DeviceInfoProtocol
Request and report device hardware/software information for informed scheduling.
### Info Request Flow
The server can request fresh device information at any time to make informed scheduling decisions:
```mermaid
sequenceDiagram
participant S as Server
participant C as Client
S->>C: DEVICE_INFO_REQUEST
C->>C: Collect telemetry (OS, CPU, GPU, RAM, etc.)
C->>S: DEVICE_INFO_RESPONSE (device specs)
```
This pull-based telemetry model allows the orchestrator to query device capabilities on-demand (e.g., before assigning a GPU-intensive task) rather than relying on stale registration data.
### Constellation → Server: Request Info
```python
from aip.protocol import DeviceInfoProtocol
info_protocol = DeviceInfoProtocol(transport)
await info_protocol.request_device_info(
constellation_id="orchestrator_001",
target_device="windows_agent_001",
request_id="req_info_001"
)
```
### Server → Client: Provide Info
The server responds with device information (or an error if collection failed):
```python
device_info = {
"os": "Windows 11",
"cpu": "Intel i7-12700K",
"ram_gb": 32,
"gpu": "NVIDIA RTX 3080",
"disk_free_gb": 500,
"active_processes": 145,
"network_status": "connected"
}
await info_protocol.send_device_info_response(
device_info=device_info,
request_id="req_info_001",
error=None # Set to error message string if info collection failed
)
```
### Use Cases
!!!success "Device-Aware Task Scheduling"
- **GPU-aware scheduling**: Check GPU availability before assigning vision tasks
- **Load balancing**: Distribute tasks based on CPU/RAM usage
- **Health monitoring**: Track device status over time
---
## Protocol Patterns
### Multi-Turn Conversations
Use `prev_response_id` to maintain conversation context across multiple exchanges.
This diagram shows how messages are chained together using `prev_response_id` to maintain conversation context:
```mermaid
graph LR
A["Server: COMMAND response_id=001"] --> B["Client: RESULTS prev_response_id=001 request_id=002"]
B --> C["Server: COMMAND response_id=003"]
C --> D["Client: RESULTS prev_response_id=003 request_id=004"]
```
Each response references the previous message's `response_id` in its `prev_response_id` field, forming a traceable conversation chain. This enables debugging, audit trails, and request-response correlation.
```python
# Turn 1: Server sends command
await protocol.send_message(ServerMessage(
type=ServerMessageType.COMMAND,
response_id="resp_001",
...
))
# Turn 2: Client sends results
await protocol.send_message(ClientMessage(
type=ClientMessageType.COMMAND_RESULTS,
request_id="req_001",
prev_response_id="resp_001", # Links to previous
...
))
```
### Session-Based Communication
All messages in a task share the same `session_id` for traceability.
```python
SESSION_ID = "session_abc123"
# All use same session_id
task_msg.session_id = SESSION_ID
command_msg.session_id = SESSION_ID
results_msg.session_id = SESSION_ID
task_end_msg.session_id = SESSION_ID
```
### Error Recovery
**Protocol-Level Errors (Connection Issues):**
```python
try:
await protocol.send_message(msg)
except ConnectionError:
await reconnect()
except IOError as e:
logger.error(f"I/O error: {e}")
```
**Application-Level Errors (Task Failures):**
```python
# Send error through protocol
await protocol.send_error(
error_msg="Invalid command: tool_name missing",
response_id=msg.response_id
)
```
---
## Best Practices
### Protocol Selection
Use specialized protocols instead of manually constructing messages with `AIPProtocol`.
| Task | Protocol |
|------|----------|
| Agent registration | `RegistrationProtocol` |
| Task execution | `TaskExecutionProtocol` |
| Command validation | `CommandProtocol` |
| Keepalive | `HeartbeatProtocol` |
| Device telemetry | `DeviceInfoProtocol` |
### Validation
- Always validate commands/results before transmission
- Use `MessageValidator` for message integrity checks
- Catch validation errors early
### Session Management
- **Always set `session_id`** for task-related messages
- Use **correlation IDs** (`prev_response_id`) for multi-turn conversations
- **Generate unique IDs** with `uuid.uuid4()`
### Error Handling
- **Distinguish** protocol errors (connection) from application errors (task failure)
- **Propagate errors** explicitly through error messages
- **Leverage middleware** for cross-cutting concerns (logging, metrics, auth)
!!!danger "Resource Cleanup"
Always close protocols when done to release transport resources.
---
## Quick Reference
### Import Protocols
```python
from aip.protocol import (
AIPProtocol,
RegistrationProtocol,
TaskExecutionProtocol,
CommandProtocol,
HeartbeatProtocol,
DeviceInfoProtocol,
)
```
### Related Documentation
- [Message Reference](./messages.md) - Message types and structures
- [Transport Layer](./transport.md) - WebSocket implementation
- [Endpoints](./endpoints.md) - Protocol usage in endpoints
- [Resilience](./resilience.md) - Connection management and recovery
- [Overview](./overview.md) - System architecture
---
### Documents Docs Aip Resilience (`documents/docs/aip/resilience.md`)
# AIP Resilience
AIP's resilience layer ensures stable communication and consistent orchestration across distributed agent constellations through automatic reconnection, heartbeat monitoring, and timeout management.
## Resilience Components
| Component | Purpose | Key Features |
|-----------|---------|--------------|
| **ReconnectionStrategy** | Auto-reconnect on disconnect | Exponential backoff, max retries, policies |
| **HeartbeatManager** | Connection health monitoring | Periodic keepalive, failure detection |
| **TimeoutManager** | Operation timeout enforcement | Configurable timeouts, async cancellation |
| **ConnectionProtocol** | State management | Bidirectional fault handling, task cleanup |
---
## Resilient Connection Protocol
The Resilient Connection Protocol governs how connection disruptions are detected, handled, and recovered between ConstellationClient and Device Agents.
### Connection State Diagram
This state diagram shows how devices transition between connection states and the internal sub-states during disconnection recovery:
```mermaid
stateDiagram-v2
[*] --> CONNECTED: Initial connection
CONNECTED --> DISCONNECTED: Connection lost
DISCONNECTED --> CONNECTED: Reconnect succeeds
DISCONNECTED --> [*]: Max retries / Manual removal
state DISCONNECTED {
[*] --> DetectFailure
DetectFailure --> CancelTasks
CancelTasks --> NotifyOrchestrator
NotifyOrchestrator --> AttemptReconnect
AttemptReconnect --> [*]: Success
}
note right of DISCONNECTED
• Invisible to scheduler
• Tasks marked FAILED
• Auto-reconnect triggered
end note
```
The nested states within `DISCONNECTED` show the cleanup and recovery sequence: detect the failure, cancel running tasks, notify the orchestrator, then attempt reconnection with exponential backoff.
### Device Disconnection Workflow
!!!danger "Impact on Running Tasks"
All tasks running on a disconnected device are **immediately marked as FAILED** to maintain TaskConstellation consistency.
| Phase | Action | Trigger |
|-------|--------|---------|
| **1. Detection** | Connection failure detected | WebSocket close, heartbeat timeout, network error |
| **2. State Transition** | `CONNECTED` → `DISCONNECTED` | Agent excluded from scheduler |
| **3. Task Failure** | Mark tasks as `TASK_FAILED` | Propagate to ConstellationAgent |
| **4. Auto-Reconnect** | Background routine triggered | Exponential backoff |
| **5. Recovery** | `DISCONNECTED` → `CONNECTED` | Resume scheduling |
**Task Cancellation:**
```python
# Automatically called on disconnection
await device_server.cancel_device_tasks(client_id, reason="device_disconnected")
```
### ConstellationClient Disconnection
When ConstellationClient disconnects, Device Agent Servers proactively clean up to prevent orphaned tasks.
This sequence diagram shows the proactive cleanup sequence when the orchestrator disconnects, ensuring all running tasks are properly aborted:
```mermaid
sequenceDiagram
participant CC as ConstellationClient
participant DAS as Device Agent Server
participant Tasks as Running Tasks
CC-xDAS: Connection lost
DAS->>DAS: Detect termination signal
DAS->>Tasks: Abort all tasks for client
Tasks->>Tasks: Cleanup resources
DAS->>DAS: Maintain consistency
Note over DAS: Prevents: • Resource leaks • Orphaned tasks • Inconsistent states
```
The `x` marker on the connection arrow indicates an abnormal termination. The server immediately detects this and cascades the cleanup signal to all associated tasks, preventing resource leaks.
**Guarantees:**
- ✅ No orphaned tasks or zombie processes
- ✅ End-to-end consistency across client-server boundary
- ✅ Automatic resource cleanup
- ✅ Synchronized task state reflection
---
## ReconnectionStrategy
Manages reconnection attempts with configurable backoff policies to handle transient network failures.
### Configuration
```python
from aip.resilience import ReconnectionStrategy, ReconnectionPolicy
strategy = ReconnectionStrategy(
max_retries=5, # Maximum attempts
initial_backoff=1.0, # Initial delay (seconds)
max_backoff=60.0, # Maximum delay (seconds)
backoff_multiplier=2.0, # Exponential multiplier
policy=ReconnectionPolicy.EXPONENTIAL_BACKOFF
)
```
[→ See how ReconnectionStrategy is used in endpoints](./endpoints.md)
### Backoff Policies
Select the policy that matches your deployment environment's network characteristics.
| Policy | Backoff Pattern | Best For | Example Sequence |
|--------|----------------|----------|------------------|
| **EXPONENTIAL_BACKOFF** | Doubles each attempt | Internet, unreliable networks | 1s → 2s → 4s → 8s → 16s |
| **LINEAR_BACKOFF** | Linear increase | Local networks, testing | 1s → 2s → 3s → 4s → 5s |
| **IMMEDIATE** | No delay | ⚠️ Testing only | 0s → 0s → 0s → 0s → 0s |
| **NONE** | No reconnection | Manual control | Disabled |
!!!danger "IMMEDIATE Policy Warning"
`IMMEDIATE` policy can overwhelm servers with rapid retry attempts. **Use only for testing.**
### Reconnection Workflow
This flowchart shows the complete reconnection logic from failure detection through recovery or permanent failure:
```mermaid
graph TD
A[Connection Lost] --> B[Cancel Pending Tasks]
B --> C[Notify Upper Layers]
C --> D{Retry Count < Max?}
D -->|Yes| E[Calculate Backoff]
E --> F[Wait Backoff Duration]
F --> G[Attempt Reconnect]
G --> H{Success?}
H -->|Yes| I[Restore Session]
H -->|No| J[Increment Retry Count]
J --> D
D -->|No| K[Max Retries Reached]
K --> L[Permanent Failure]
I --> M[Resume Operations]
style I fill:#d4edda
style L fill:#f8d7da
```
The loop between "Attempt Reconnect" and "Increment Retry Count" continues until either reconnection succeeds (green path) or max retries are exhausted (red path). Backoff duration increases with each failed attempt.
### Reconnection Example
```python
async def handle_disconnection(
endpoint: AIPEndpoint,
device_id: str,
on_reconnect: Optional[Callable] = None
):
# Step 1: Cancel pending tasks
await strategy._cancel_pending_tasks(endpoint, device_id)
# Step 2: Notify upper layers
await strategy._notify_disconnection(endpoint, device_id)
# Step 3: Attempt reconnection
reconnected = await strategy.attempt_reconnection(endpoint, device_id)
# Step 4: Call reconnection callback
if reconnected and on_reconnect:
await on_reconnect()
```
### Custom Reconnection Callback
```python
async def on_reconnected():
logger.info("Device reconnected, resuming tasks")
await restore_task_queue()
await sync_device_state()
await strategy.handle_disconnection(
endpoint=endpoint,
device_id="device_001",
on_reconnect=on_reconnected
)
```
---
## HeartbeatManager {#heartbeat-manager}
Sends periodic keepalive messages to detect broken connections before they cause failures.
### Configuration
```python
from aip.resilience import HeartbeatManager
from aip.protocol import HeartbeatProtocol
heartbeat_protocol = HeartbeatProtocol(transport)
heartbeat_manager = HeartbeatManager(
protocol=heartbeat_protocol,
default_interval=30.0 # 30 seconds
)
```
[→ See HeartbeatProtocol reference](./protocols.md#heartbeat-protocol)
### Lifecycle Management
| Operation | Method | Description |
|-----------|--------|-------------|
| **Start** | `start_heartbeat(client_id, interval)` | Begin periodic heartbeat for client |
| **Stop** | `stop_heartbeat(client_id)` | Stop heartbeat for specific client |
| **Stop All** | `stop_all()` | Stop all active heartbeats |
| **Check Status** | `is_running(client_id)` | Verify if heartbeat is active |
| **Get Interval** | `get_interval(client_id)` | Retrieve current interval |
### Usage Example
```python
# Start heartbeat for a client
await heartbeat_manager.start_heartbeat(
client_id="device_001",
interval=20.0 # Override default
)
# Check if running
if heartbeat_manager.is_running("device_001"):
logger.info("Heartbeat active")
# Stop for specific client
await heartbeat_manager.stop_heartbeat("device_001")
# Stop all heartbeats (cleanup)
await heartbeat_manager.stop_all()
```
### Heartbeat Loop Internals
The heartbeat manager automatically sends periodic heartbeats. If the protocol is not connected, it logs a warning and continues the loop:
```python
async def _heartbeat_loop(client_id: str, interval: float):
"""Internal heartbeat loop (automatic)"""
try:
while True:
await asyncio.sleep(interval)
if protocol.is_connected():
try:
await protocol.send_heartbeat(client_id)
except Exception as e:
logger.error(f"Error sending heartbeat: {e}")
# Continue loop, connection manager handles disconnection
else:
logger.warning("Protocol not connected, skipping heartbeat")
except asyncio.CancelledError:
logger.debug("Heartbeat loop cancelled")
```
### Failure Detection
When the transport layer fails to send a heartbeat (connection closed), errors are logged but the loop continues running. The connection manager is responsible for detecting the disconnection through transport-level errors and triggering the reconnection strategy.
This sequence diagram shows how heartbeat errors are handled:
```mermaid
sequenceDiagram
participant HM as HeartbeatManager
participant P as Protocol
participant T as Transport
loop Every interval
HM->>P: send_heartbeat()
P->>T: Send via WebSocket
alt Connection alive
T-->>P: Success
P-->>HM: Continue
else Connection dead
T-xP: ConnectionError
P-xHM: Error (caught)
HM->>HM: Log error, continue loop
Note over HM: Connection manager handles disconnection at transport level
end
end
```
The `x` markers indicate error paths. When the transport layer fails to send a heartbeat, the error is caught and logged. The heartbeat loop continues, while the connection manager detects the disconnection at the transport level and initiates recovery.
### Interval Guidelines
| Environment | Recommended Interval | Rationale |
|-------------|---------------------|-----------|
| **Local network** | 10-20s | Quick failure detection, low latency |
| **Internet** | 30-60s | Balance overhead vs detection speed |
| **Mobile/Unreliable** | 60-120s | Reduce battery/bandwidth usage |
| **Critical systems** | 5-10s | Fastest failure detection |
---
## TimeoutManager
Prevents operations from hanging indefinitely by enforcing configurable timeouts with automatic cancellation.
### Configuration
```python
from aip.resilience import TimeoutManager
timeout_manager = TimeoutManager(
default_timeout=120.0 # 120 seconds
)
```
[→ See how timeouts are used in protocol operations](./protocols.md)
### Usage Patterns
**Default Timeout:**
```python
result = await timeout_manager.with_timeout(
protocol.send_message(msg),
operation_name="send_message"
)
```
**Custom Timeout:**
```python
result = await timeout_manager.with_timeout(
protocol.receive_message(ServerMessage),
timeout=60.0,
operation_name="receive_message"
)
```
### Error Handling
```python
from asyncio import TimeoutError
try:
result = await timeout_manager.with_timeout(
long_running_operation(),
timeout=30.0
)
except TimeoutError:
logger.error("Operation timed out after 30 seconds")
# Handle timeout: retry, fail task, notify user
```
### Recommended Timeouts
| Operation | Timeout | Rationale |
|-----------|---------|-----------|
| **Registration** | 10-30s | Simple message exchange |
| **Task Dispatch** | 30-60s | May involve scheduling logic |
| **Command Execution** | 60-300s | Depends on command complexity |
| **Heartbeat** | 5-10s | Fast failure detection needed |
| **Disconnection** | 5-15s | Clean shutdown |
| **Device Info Query** | 15-30s | Telemetry collection |
---
## Integration with Endpoints
Endpoints automatically integrate all resilience components—no manual wiring needed.
### Example: DeviceClientEndpoint
```python
from aip.endpoints import DeviceClientEndpoint
endpoint = DeviceClientEndpoint(
ws_url="ws://localhost:8000/ws",
ufo_client=client,
max_retries=3, # Reconnection retries
timeout=120.0 # Connection timeout
)
# Resilience handled automatically on start
await endpoint.start()
```
**Note**: The endpoint creates its own `ReconnectionStrategy` internally with the specified `max_retries`.
### Built-In Features
| Feature | Behavior | Configuration |
|---------|----------|---------------|
| **Auto-Reconnection** | Triggered on disconnect | Via `ReconnectionStrategy` |
| **Heartbeat** | Starts on connection | Managed by `HeartbeatManager` |
| **Timeout Enforcement** | Applied to all operations | Via `TimeoutManager` |
| **Task Cancellation** | Auto-cancel on disconnect | Built-in to endpoint |
[→ See endpoint documentation](./endpoints.md)
[→ See WebSocket transport details](./transport.md)
---
## Best Practices by Environment
### Local Network (Low Latency, High Reliability)
```python
strategy = ReconnectionStrategy(
max_retries=3,
initial_backoff=1.0,
max_backoff=10.0,
policy=ReconnectionPolicy.LINEAR_BACKOFF
)
heartbeat_interval = 20.0 # Quick detection
timeout_default = 60.0
```
### Internet (Variable Latency, Moderate Reliability)
```python
strategy = ReconnectionStrategy(
max_retries=5,
initial_backoff=2.0,
max_backoff=60.0,
policy=ReconnectionPolicy.EXPONENTIAL_BACKOFF
)
heartbeat_interval = 30.0 # Balance overhead and detection
timeout_default = 120.0
```
### Unreliable Network (High Latency, Low Reliability)
```python
strategy = ReconnectionStrategy(
max_retries=10,
initial_backoff=5.0,
max_backoff=300.0, # Up to 5 minutes
policy=ReconnectionPolicy.EXPONENTIAL_BACKOFF
)
heartbeat_interval = 60.0 # Reduce overhead
timeout_default = 180.0
```
---
## Error Scenarios
### Scenario 1: Transient Network Failure
**Problem**: Network glitch disconnects client for 3 seconds.
**Resolution**:
1. ✅ Disconnection detected via heartbeat timeout
2. ✅ Automatic reconnection triggered (1st attempt after 2s)
3. ✅ Connection restored successfully
4. ✅ Heartbeat resumes
5. ✅ Tasks continue
### Scenario 2: Prolonged Outage
**Problem**: Device offline for 10 minutes.
**Resolution**:
1. ❌ Initial disconnection detected
2. ⏳ Multiple reconnection attempts (exponential backoff: 2s, 4s, 8s, 16s, 32s)
3. ❌ All attempts fail (max retries reached)
4. ⚠️ Tasks marked as FAILED
5. 📢 ConstellationAgent notified
6. ♻️ Tasks reassigned to other devices
### Scenario 3: Server Restart
**Problem**: Server restarts, causing all clients to disconnect at once.
**Resolution**:
1. ⚠️ All clients detect disconnection
2. ⏳ Each client begins reconnection (with jitter to avoid thundering herd)
3. ✅ Server restarts and accepts connections
4. ✅ Clients reconnect and re-register
5. ✅ Task execution resumes
### Scenario 4: Heartbeat Timeout
**Problem**: Heartbeat not received within timeout period.
**Resolution**:
1. ⏰ HeartbeatManager detects missing pong
2. ⚠️ Connection marked as potentially dead
3. 🔄 Disconnection handling triggered
4. ⏳ Reconnection attempted
5. ✅ If successful, heartbeat resumes
---
## Monitoring and Observability
### Enable Resilience Logging
```python
import logging
# Enable detailed resilience logs
logging.getLogger("aip.resilience").setLevel(logging.INFO)
```
### Custom Event Handlers
```python
class CustomEndpoint(DeviceClientEndpoint):
async def on_device_disconnected(self, device_id: str) -> None:
# Custom cleanup
await self.cleanup_resources(device_id)
logger.warning(f"Device {device_id} disconnected")
# Call parent implementation
await super().on_device_disconnected(device_id)
async def reconnect_device(self, device_id: str) -> bool:
# Custom reconnection logic
success = await self.custom_reconnect(device_id)
if success:
await self.restore_state(device_id)
logger.info(f"Device {device_id} reconnected")
return success
```
### Graceful Degradation
```python
if not await strategy.attempt_reconnection(endpoint, device_id):
logger.error(f"Failed to reconnect {device_id} after max retries")
# Graceful degradation
await notify_operator(f"Device {device_id} offline")
await reassign_tasks_to_other_devices(device_id)
await update_monitoring_dashboard(device_id, "offline")
```
---
## Testing Resilience
Test resilience by simulating network failures and verifying recovery.
```python
# Simulate disconnection
await transport.close()
# Verify reconnection
assert await endpoint.reconnect_device(device_id)
# Verify heartbeat resumes
await asyncio.sleep(1)
assert heartbeat_manager.is_running(device_id)
# Verify task state
assert all(task.status == TaskStatus.FAILED for task in orphaned_tasks)
```
---
## Quick Reference
### Import Resilience Components
```python
from aip.resilience import (
ReconnectionStrategy,
ReconnectionPolicy,
HeartbeatManager,
TimeoutManager,
)
```
### Related Documentation
- [Endpoints](./endpoints.md) - How endpoints use resilience
- [Transport Layer](./transport.md) - Transport-level connection management
- [Protocol Reference](./protocols.md) - Protocol-level error handling
- [Overview](./overview.md) - System architecture and design
---
### Documents Docs Aip Transport (`documents/docs/aip/transport.md`)
# AIP Transport Layer
The transport layer provides a pluggable abstraction for AIP's network communication, decoupling protocol logic from underlying network implementations through a unified Transport interface.
## Transport Architecture
AIP uses a transport abstraction pattern that allows different network protocols to be swapped without changing higher-level protocol logic. The current implementation focuses on WebSocket, with future support planned for HTTP/3 and gRPC:
```mermaid
graph TD
subgraph "Transport Abstraction"
TI[Transport Interface]
TI --> |implements| WST[WebSocketTransport]
TI --> |future| H3T[HTTP/3 Transport]
TI --> |future| GRPC[gRPC Transport]
end
subgraph "WebSocket Transport"
WST --> |client-side| WSC[websockets library]
WST --> |server-side| FAPI[FastAPI WebSocket]
WST --> |adapter| ADP[Unified Adapter]
end
subgraph "Protocol Layer"
PROTO[AIP Protocols]
PROTO --> |uses| TI
end
style WST fill:#d4edda
style TI fill:#d1ecf1
```
The unified adapter bridges client and server WebSocket libraries, providing a consistent interface regardless of which side of the connection you're on. This design pattern enables protocol code to be transport-agnostic.
---
## Transport Interface
All transport implementations must implement the `Transport` interface for interoperability.
### Core Operations
| Method | Purpose | Return Type |
|--------|---------|-------------|
| `connect(url, **kwargs)` | Establish connection to remote endpoint | `None` |
| `send(data)` | Send raw bytes | `None` |
| `receive()` | Receive raw bytes | `bytes` |
| `close()` | Close connection gracefully | `None` |
| `wait_closed()` | Wait for connection to fully close | `None` |
| `is_connected` (property) | Check connection status | `bool` |
### Interface Definition
```python
from aip.transport import Transport
class Transport(ABC):
@abstractmethod
async def connect(self, url: str, **kwargs) -> None:
"""Connect to remote endpoint"""
@abstractmethod
async def send(self, data: bytes) -> None:
"""Send data"""
@abstractmethod
async def receive(self) -> bytes:
"""Receive data"""
@abstractmethod
async def close(self) -> None:
"""Close connection"""
@abstractmethod
async def wait_closed(self) -> None:
"""Wait for connection to fully close"""
@property
@abstractmethod
def is_connected(self) -> bool:
"""Check connection status"""
```
---
## WebSocket Transport
`WebSocketTransport` provides persistent, full-duplex, bidirectional communication over WebSocket protocol (RFC 6455).
### Quick Start
**Client-Side:**
```python
from aip.transport import WebSocketTransport
# Create and configure
transport = WebSocketTransport(
ping_interval=30.0,
ping_timeout=180.0,
close_timeout=10.0,
max_size=100 * 1024 * 1024 # 100MB
)
# Connect
await transport.connect("ws://localhost:8000/ws")
# Communicate
await transport.send(b"Hello Server")
data = await transport.receive()
# Cleanup
await transport.close()
```
**Server-Side (FastAPI):**
```python
from fastapi import WebSocket
from aip.transport import WebSocketTransport
async def websocket_endpoint(websocket: WebSocket):
await websocket.accept()
# Wrap existing WebSocket
transport = WebSocketTransport(websocket=websocket)
# Use unified interface
data = await transport.receive()
await transport.send(b"Response")
```
**Note**: WebSocketTransport automatically detects whether it's wrapping a FastAPI WebSocket or a client connection and selects the appropriate adapter.
[→ See how endpoints use WebSocketTransport](./endpoints.md)
### Configuration Parameters
🔧 Configuration Options (Click to expand)
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| **ping_interval** | `float` | `30.0` | Time between ping messages (seconds). Keepalive mechanism. |
| **ping_timeout** | `float` | `180.0` | Max wait for pong response (seconds). Connection marked dead if exceeded. |
| **close_timeout** | `float` | `10.0` | Timeout for graceful close handshake (seconds). |
| **max_size** | `int` | `104857600` | Max message size in bytes (100MB). Messages exceeding this are rejected. |
**Usage Guidelines:**
!!!warning "max_size for Large Payloads"
Set `max_size` based on application needs. Large screenshots, models, or binary data may require higher limits. Consider compression for payloads approaching this limit.
### Pinned Connections and Redirects
`connect(url, pinned_addresses=...)` accepts a non-empty sequence of approved IP addresses. The transport connects to the first address, preserves the original hostname for TLS verification and SNI on `wss://` connections, and checks the connected peer against the approved addresses after a successful handshake. Callers must obtain these addresses from their URL validation policy; passing an address to the transport does not itself authorize that destination.
For pinned connections, HTTP redirects are rejected **before another TCP connection is opened**. This applies to 301, 302, 303, 307, and 308 responses with a `Location` header, including cross-origin, same-origin, and relative redirects. The transport enters `TransportState.ERROR` and raises `ConnectionError` with a message containing:
```text
WebSocket redirects are not allowed for pinned connections
```
The project's `websockets==12.0` legacy client normally follows redirects during the handshake. A per-connection redirect guard prevents that behavior for pinned connections. Post-handshake peer validation alone is insufficient: a redirected server could receive the upgrade request and reject the handshake before peer validation ever runs.
!!!warning "Configure the Final Device URL"
Galaxy's device connection manager supplies pinned addresses, so device endpoints must accept the WebSocket upgrade without a redirect. Configure the final URL directly and ensure it passes the URL validation policy. Do not remove pinning to work around a redirect rejection. See [Server URL and Redirects](../configuration/system/galaxy_devices.md#server-url-and-redirects).
Direct transport calls that omit `pinned_addresses` or pass `None` retain the library's existing redirect behavior. An empty address sequence is rejected. This policy does not change server-side transports that wrap an already accepted WebSocket.
### Connection States
WebSocket connections transition through multiple states during their lifecycle. This diagram shows all possible states and transitions:
```mermaid
stateDiagram-v2
[*] --> DISCONNECTED
DISCONNECTED --> CONNECTING: connect()
CONNECTING --> CONNECTED: Success
CONNECTING --> ERROR: Failure
CONNECTED --> DISCONNECTING: close()
DISCONNECTING --> DISCONNECTED: Complete
CONNECTED --> ERROR: Network failure
ERROR --> DISCONNECTED: Reset
note right of CONNECTED
• is_connected = True
• send/receive active
• Ping/pong running
end note
```
Only the `CONNECTED` state allows data transmission. The `ERROR` state is a terminal state that requires reset before attempting reconnection.
**State Definitions:**
| State | Meaning | Actions Allowed |
|-------|---------|-----------------|
| `DISCONNECTED` | No active connection | `connect()` |
| `CONNECTING` | Connection in progress | Wait for result |
| `CONNECTED` | Active connection | `send()`, `receive()`, `close()` |
| `DISCONNECTING` | Closing in progress | Wait for completion |
| `ERROR` | Error occurred | Investigate, reset |
**Check State:**
```python
from aip.transport import TransportState
if transport.state == TransportState.CONNECTED:
await transport.send(data)
else:
logger.warning("Transport not connected")
```
### Ping/Pong Keepalive
WebSocket automatically sends ping frames at `ping_interval` to detect broken connections.
This sequence diagram shows the automatic ping/pong mechanism for detecting broken connections:
```mermaid
sequenceDiagram
participant C as Client
participant S as Server
loop Every ping_interval
C->>S: ping frame
S->>C: pong frame
Note over C: Connection healthy
end
C->>S: ping frame
S-xC: No response
Note over C: Timeout after ping_timeout
C->>C: Mark connection dead
C->>C: Close connection
```
The `x` marker indicates a failed pong response. After `ping_timeout` expires without receiving a pong, the connection is automatically marked dead and closed, triggering reconnection logic.
**Timeout Behavior:**
- ✅ **Pong received within `ping_timeout`**: Connection healthy, continue
- ❌ **No pong within `ping_timeout`**: Connection marked dead, automatic close triggered
### Error Handling
!!!danger "Always Handle ConnectionError"
Connection failures can occur at any time due to network issues. Wrap send/receive in try-except blocks.
**Connection Errors:**
```python
try:
await transport.connect("ws://localhost:8000/ws")
except ConnectionError as e:
logger.error(f"Failed to connect: {e}")
await handle_connection_failure()
```
**Send/Receive Errors:**
```python
try:
await transport.send(data)
response = await transport.receive()
except ConnectionError:
logger.warning("Connection closed during operation")
await reconnect()
except IOError as e:
logger.error(f"I/O error: {e}")
await handle_io_error(e)
```
**Graceful Shutdown:**
```python
try:
# Close with timeout
await transport.close()
# Wait for complete shutdown
await transport.wait_closed()
except Exception as e:
logger.error(f"Error during shutdown: {e}")
```
**Note**: The transport sends a WebSocket close frame and waits for the peer's close frame within `close_timeout` before terminating the connection.
### Adapter Pattern
AIP uses adapters to provide a unified interface across different WebSocket libraries without exposing implementation details.
**Supported WebSocket Implementations:**
| Implementation | Use Case | Adapter |
|----------------|----------|---------|
| **websockets library** | Client-side connections | `WebSocketsLibAdapter` |
| **FastAPI WebSocket** | Server-side endpoints | `FastAPIWebSocketAdapter` |
**Automatic Detection:**
```python
# Server-side: Automatically uses FastAPIWebSocketAdapter
transport = WebSocketTransport(websocket=fastapi_websocket)
# Client-side: Automatically uses WebSocketsLibAdapter
transport = WebSocketTransport()
await transport.connect("ws://server:8000/ws")
```
**Benefits:**
- ✅ Protocol-level code remains unchanged across client/server
- ✅ API differences abstracted by adapters
- ✅ Easy to add new WebSocket implementations
- ✅ Testability through adapter mocking
---
## Message Encoding
AIP uses UTF-8 encoded JSON for all messages, leveraging Pydantic for serialization/deserialization.
### Encoding Flow
This diagram shows the transformation steps from Pydantic model to network bytes:
```mermaid
graph LR
A[Pydantic Model] -->|model_dump_json| B[JSON String]
B -->|encode utf-8| C[bytes]
C -->|transport.send| D[Network]
style A fill:#d4edda
style D fill:#d1ecf1
```
Pydantic handles type validation and JSON serialization, UTF-8 encoding converts to bytes, then the transport layer sends over the network. Decoding follows the reverse path.
**Send Example:**
```python
from aip.messages import ClientMessage
# 1. Create Pydantic model
msg = ClientMessage(
message_type="TASK_RESULT",
task_id="task_123",
result={"status": "success"}
)
# 2. Serialize to JSON string
json_str = msg.model_dump_json()
# 3. Encode to bytes
bytes_data = json_str.encode('utf-8')
# 4. Send via transport
await transport.send(bytes_data)
```
### Decoding Flow
```mermaid
graph LR
A[Network] -->|transport.receive| B[bytes]
B -->|decode utf-8| C[JSON String]
C -->|model_validate_json| D[Pydantic Model]
style A fill:#d1ecf1
style D fill:#d4edda
```
**Receive Example:**
```python
from aip.messages import ServerMessage
# 1. Receive bytes
bytes_data = await transport.receive()
# 2. Decode to JSON string
json_str = bytes_data.decode('utf-8')
# 3. Deserialize to Pydantic model
msg = ServerMessage.model_validate_json(json_str)
# 4. Use typed data
print(f"Task ID: {msg.task_id}")
```
---
## Performance Optimization
### Performance Comparison
| Scenario | Recommended Configuration | Rationale |
|----------|---------------------------|-----------|
| **Large Messages** | `max_size=500MB`, compression | Screenshots, binary data |
| **High Throughput** | Batch messages, `ping_interval=60s` | Reduce overhead per message |
| **Low Latency** | Dedicated connections, `ping_interval=10s` | Fast failure detection |
| **Mobile Networks** | `ping_interval=60s`, compression | Reduce battery/bandwidth usage |
### Optimization Strategies
**Large Messages Strategy:**
For messages approaching `max_size`:
**Option 1: Compression**
```python
import gzip
compressed = gzip.compress(large_data)
await transport.send(compressed)
```
**Option 2: Chunking**
```python
chunk_size = 1024 * 1024 # 1MB chunks
for i in range(0, len(large_data), chunk_size):
chunk = large_data[i:i+chunk_size]
await transport.send(chunk)
```
**Option 3: Streaming Protocol**
Consider implementing a custom streaming protocol for very large payloads.
[→ See message encoding details in Protocol Reference](./protocols.md)
**High Throughput Strategy:**
For high message rates:
**Batch Messages:**
```python
batch = [msg1, msg2, msg3, msg4]
batch_json = json.dumps([msg.model_dump() for msg in batch])
await transport.send(batch_json.encode('utf-8'))
```
**Reduce Ping Frequency:**
```python
transport = WebSocketTransport(
ping_interval=60.0 # Less overhead
)
```
**Low Latency Strategy:**
For real-time applications:
**Fast Failure Detection:**
```python
transport = WebSocketTransport(
ping_interval=10.0, # Quick detection
ping_timeout=30.0
)
```
**Dedicated Connections:**
```python
# One transport per device (no sharing)
device_transports = {
device_id: WebSocketTransport()
for device_id in devices
}
```
---
## Transport Extensions
!!!warning "Future Implementations"
AIP's architecture supports multiple transport implementations. The following are planned but not yet implemented.
### HTTP/3 Transport (Planned)
**Benefits:**
- ✅ Multiplexing without head-of-line blocking (QUIC protocol)
- ✅ 0-RTT connection resumption (faster reconnection)
- ✅ Better mobile network performance (connection migration)
- ✅ Built-in encryption (TLS 1.3)
**Use Cases:**
- High-latency networks (satellite, mobile)
- Frequent reconnections (mobile roaming)
- Multiple concurrent streams per connection
### gRPC Transport (Planned)
**Benefits:**
- ✅ Strong typing with Protocol Buffers
- ✅ Built-in load balancing
- ✅ Bidirectional streaming RPCs
- ✅ Code generation for multiple languages
**Use Cases:**
- Cross-language interoperability
- Microservices communication
- Performance-critical paths
### Custom Transport Implementation
Implement custom transports for specialized protocols:
```python
from aip.transport.base import Transport
class CustomTransport(Transport):
async def connect(self, url: str, **kwargs) -> None:
# Custom connection logic
self._connection = await custom_protocol.connect(url)
async def send(self, data: bytes) -> None:
await self._connection.write(data)
async def receive(self) -> bytes:
return await self._connection.read()
async def close(self) -> None:
await self._connection.shutdown()
@property
def is_connected(self) -> bool:
return self._connection is not None and self._connection.is_open
```
**Integration:**
Custom transports can be used directly with protocols:
```python
from aip.protocol import AIPProtocol
# Use custom transport with protocol
transport = CustomTransport()
await transport.connect("custom://server:port")
protocol = AIPProtocol(transport)
await protocol.send_message(message)
```
[→ See Transport interface specification above](#transport-interface)
[→ See Protocol usage examples](./protocols.md)
---
## Best Practices
### Environment-Specific Configuration
Adapt transport settings to your deployment environment's characteristics.
| Environment | ping_interval | ping_timeout | max_size | close_timeout |
|-------------|--------------|--------------|----------|---------------|
| **Local Network** | 10-20s | 30-60s | 100MB | 5s |
| **Internet** | 30-60s | 120-180s | 100MB | 10s |
| **Unreliable Network** | 60-120s | 180-300s | 50MB | 15s |
| **Mobile** | 60s | 180s | 10MB | 10s |
**Local Network Example:**
```python
transport = WebSocketTransport(
ping_interval=15.0, # Quick failure detection
ping_timeout=45.0,
close_timeout=5.0
)
```
**Internet Example:**
```python
transport = WebSocketTransport(
ping_interval=30.0, # Balance overhead and detection
ping_timeout=180.0,
close_timeout=10.0
)
```
**Mobile Network Example:**
```python
transport = WebSocketTransport(
ping_interval=60.0, # Reduce battery usage
ping_timeout=180.0,
max_size=10 * 1024 * 1024 # 10MB for mobile
)
```
### Connection Health Monitoring
Always verify connection status before critical operations:
```python
# Check before sending
if not transport.is_connected:
logger.warning("Transport not connected, attempting reconnection")
await reconnect_transport()
# Proceed with send
await transport.send(data)
```
### Resilience Integration
Transport alone provides low-level communication. Combine with resilience components for production readiness:
```python
from aip.resilience import ReconnectionStrategy
strategy = ReconnectionStrategy(max_retries=5)
try:
await transport.send(data)
except ConnectionError:
# Trigger reconnection
await strategy.handle_disconnection(endpoint, device_id)
```
[→ See Resilience documentation](./resilience.md)
[→ See HeartbeatManager for connection health monitoring](./resilience.md#heartbeat-manager)
### Logging and Observability
```python
import logging
# Enable transport debug logs
logging.getLogger("aip.transport").setLevel(logging.DEBUG)
# Custom transport event logging
class LoggedTransport(WebSocketTransport):
async def send(self, data: bytes) -> None:
logger.debug(f"Sending {len(data)} bytes")
await super().send(data)
async def receive(self) -> bytes:
data = await super().receive()
logger.debug(f"Received {len(data)} bytes")
return data
```
### Resource Cleanup
!!!danger "Prevent Resource Leaks"
Always close transports to prevent socket/memory leaks:
**Context Manager Pattern (Recommended):**
```python
async with WebSocketTransport() as transport:
await transport.connect("ws://localhost:8000/ws")
await transport.send(data)
# Automatic cleanup on exit
```
**Try-Finally Pattern:**
```python
transport = WebSocketTransport()
try:
await transport.connect("ws://localhost:8000/ws")
await transport.send(data)
finally:
await transport.close()
```
---
## Quick Reference
### Import Transport Components
```python
from aip.transport import (
Transport, # Abstract base class
WebSocketTransport, # WebSocket implementation
TransportState, # Connection states enum
)
```
### Common Patterns
| Pattern | Code |
|---------|------|
| **Create transport** | `transport = WebSocketTransport()` |
| **Connect** | `await transport.connect("ws://host:port/path")` |
| **Send** | `await transport.send(data.encode('utf-8'))` |
| **Receive** | `data = await transport.receive()` |
| **Check status** | `if transport.is_connected: ...` |
| **Close** | `await transport.close()` |
### Related Documentation
- [Protocol Reference](./protocols.md) - How protocols use transports
- [Resilience](./resilience.md) - Connection management and reconnection
- [Endpoints](./endpoints.md) - Transport usage in endpoints
- [Messages](./messages.md) - Message encoding/decoding
---
### Documents Docs Client Computer (`documents/docs/client/computer.md`)
# Computer
The **Computer** class is the core execution layer of the UFO client. It manages MCP (Model Context Protocol) tool execution, maintains tool registries, and provides thread-isolated execution for reliability. Each Computer instance represents a distinct execution context with its own namespace and resource management.
## Architecture Overview
The Computer layer provides the execution engine for MCP tools with three main components:
```mermaid
graph TB
CommandRouter["CommandRouter Command Routing"]
ComputerManager["ComputerManager Instance Management"]
Computer["Computer Core Execution Layer"]
MCPServerManager["MCP Server Manager Process Isolation"]
CommandRouter -->|Routes To| ComputerManager
ComputerManager -->|Creates & Manages| Computer
Computer -->|Data Collection| DataServers["Data Collection Servers screenshot, ui_detection, etc."]
Computer -->|Actions| ActionServers["Action Servers gui_automation, file_operations, etc."]
Computer -->|Uses| ToolsRegistry["Tools Registry tool_type::tool_name → MCPToolCall"]
Computer -->|Provides| MetaTools["Meta Tools list_tools built-in introspection"]
Computer -->|Delegates To| MCPServerManager
```
**Computer** manages MCP tool execution with thread isolation and timeout control (6000-second timeout, 10-worker thread pool).
**ComputerManager** handles multiple Computer instances with namespace-based routing.
**CommandRouter** routes and executes commands across Computer instances with early-exit support.
### Key Responsibilities
- **Tool Registration**: Register tools from multiple MCP servers with namespace isolation
- **Command Routing**: Convert high-level commands to MCP tool calls
- **Execution Management**: Execute tools in isolated thread pools with timeout protection
- **Meta Tools**: Provide introspection capabilities (e.g., `list_tools`)
## Table of Contents
## Core Components
### 1. Computer Class
The `Computer` class manages a single logical computer with its own set of MCP servers and tools.
#### Key Attributes
| Attribute | Type | Description |
|-----------|------|-------------|
| `_name` | `str` | Unique identifier for the computer instance |
| `_process_name` | `str` | Associated process name for MCP server isolation |
| `_data_collection_servers` | `Dict[str, BaseMCPServer]` | Servers for data collection (screenshot, UI detection, etc.) |
| `_action_servers` | `Dict[str, BaseMCPServer]` | Servers for actions (GUI automation, file operations, etc.) |
| `_tools_registry` | `Dict[str, MCPToolCall]` | Registry of all available tools (key: `tool_type::tool_name`) |
| `_meta_tools` | `Dict[str, Callable]` | Built-in introspection tools |
| `_executor` | `ThreadPoolExecutor` | Thread pool for isolated tool execution (10 workers) |
| `_tool_timeout` | `int` | Tool execution timeout (6000 seconds = 100 minutes) |
#### Tool Namespaces
Computer supports two types of tool namespaces:
- **`data_collection`**: Tools for gathering information (non-destructive operations)
- **`action`**: Tools for performing actions (state-changing operations)
```python
# Tool key format: "tool_type::tool_name"
"data_collection::screenshot" # Take screenshot
"data_collection::ui_detection" # Detect UI elements
"action::click" # Click UI element
"action::type_text" # Type text
```
> **Note:** Different namespaces allow the same tool name to exist in both data collection and action contexts. For example, both `data_collection::get_file_info` and `action::get_file_info` can coexist.
### 2. ComputerManager Class
The `ComputerManager` creates and manages multiple `Computer` instances based on agent configurations.
#### Computer Instance Key
Each computer instance is identified by a unique key:
```python
key = f"{agent_name}::{process_name}::{root_name}"
```
**Example:**
```python
"host_agent::chrome::default" # Default chrome computer for host_agent
"host_agent::vscode::custom_config" # Custom VSCode computer for host_agent
```
#### Configuration Structure
```yaml
mcp:
host_agent:
default:
data_collection:
- namespace: "screenshot"
server_type: "local"
module: "ufo.client.mcp.local_servers.screenshot"
reset: false
- namespace: "ui_detection"
server_type: "local"
module: "ufo.client.mcp.local_servers.ui_detection"
reset: false
action:
- namespace: "gui_automation"
server_type: "local"
module: "ufo.client.mcp.local_servers.gui_automation"
reset: false
```
**Configuration Requirements**
- Each agent must have at least a `default` root configuration
- If `root_name` is not found, the manager falls back to `default`
- Missing configurations will raise a `ValueError`
### 3. CommandRouter Class
The `CommandRouter` executes commands on the appropriate `Computer` instance by routing through the `ComputerManager`.
#### Execution Flow
```mermaid
graph LR
Command --> CommandRouter
CommandRouter --> ComputerManager
ComputerManager -->|get_or_create| Computer
Computer -->|command2tool| ToolCall[MCPToolCall]
ToolCall -->|run_actions| Result[MCP Tool Result]
```
## Initialization
### Computer Initialization
```python
from ufo.client.computer import Computer
from ufo.client.mcp.mcp_server_manager import MCPServerManager
# Create MCP server manager
mcp_manager = MCPServerManager()
# Initialize computer
computer = Computer(
name="my_computer",
process_name="my_process",
mcp_server_manager=mcp_manager,
data_collection_servers_config=[
{
"namespace": "screenshot",
"server_type": "local",
"module": "ufo.client.mcp.local_servers.screenshot"
}
],
action_servers_config=[
{
"namespace": "gui_automation",
"server_type": "local",
"module": "ufo.client.mcp.local_servers.gui_automation"
}
]
)
# Async initialization (required)
await computer.async_init()
```
> **⚠️ Important:** You **must** call `await computer.async_init()` after creating a `Computer` instance. This registers all MCP servers and their tools asynchronously.
### ComputerManager Initialization
```python
from ufo.client.computer import ComputerManager
# Load configuration
with open("config.yaml") as f:
configs = yaml.safe_load(f)
# Create manager
manager = ComputerManager(
configs=configs,
mcp_server_manager=mcp_manager
)
# Get or create computer instance
computer = await manager.get_or_create(
agent_name="host_agent",
process_name="chrome",
root_name="default"
)
```
## Tool Execution
### Basic Tool Execution
```python
from aip.messages import MCPToolCall
# Create tool call
tool_call = MCPToolCall(
tool_key="data_collection::screenshot",
tool_name="screenshot",
parameters={"region": "full_screen"}
)
# Execute tool
results = await computer.run_actions([tool_call])
# Check result
if results[0].is_error:
print(f"Error: {results[0].content}")
else:
print(f"Success: {results[0].data}")
```
### Command to Tool Conversion
The `command2tool()` method converts high-level `Command` objects to `MCPToolCall` objects:
```python
from aip.messages import Command
# Create command
command = Command(
tool_name="screenshot",
tool_type="data_collection",
parameters={"region": "active_window"}
)
# Convert to tool call
tool_call = computer.command2tool(command)
# Execute
results = await computer.run_actions([tool_call])
```
If `tool_type` is not specified in the command, the `command2tool()` method will automatically detect whether the tool is registered as `data_collection` or `action`.
### Batch Tool Execution
```python
# Execute multiple tools sequentially
tool_calls = [
MCPToolCall(tool_key="data_collection::screenshot", tool_name="screenshot"),
MCPToolCall(tool_key="data_collection::ui_detection", tool_name="detect_ui"),
MCPToolCall(tool_key="action::click", tool_name="click", parameters={"x": 100, "y": 200})
]
results = await computer.run_actions(tool_calls)
for i, result in enumerate(results):
print(f"Tool {i}: {'Success' if not result.is_error else 'Failed'}")
```
## Thread Isolation & Timeout
### Why Thread Isolation?
MCP tools may contain **blocking operations** (e.g., `time.sleep()`, synchronous I/O) that can block the event loop and cause WebSocket disconnections. To prevent this:
1. Each tool call runs in a **separate thread** with its own event loop
2. The thread pool has **10 concurrent workers**
3. Each tool call has a **timeout of 6000 seconds** (100 minutes)
### Implementation Details
```python
def _call_tool_in_thread():
"""Execute MCP tool call in isolated thread with its own event loop."""
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
async def _do_call():
async with Client(server) as client:
return await client.call_tool(
name=tool_name,
arguments=params,
raise_on_error=False
)
return loop.run_until_complete(_do_call())
finally:
loop.close()
# Execute in thread pool with timeout
result = await asyncio.wait_for(
loop.run_in_executor(self._executor, _call_tool_in_thread),
timeout=self._tool_timeout
)
```
If a tool execution exceeds 6000 seconds, it will be cancelled and return a timeout error:
```python
CallToolResult(
is_error=True,
content=[TextContent(text="Tool execution timed out after 6000s")]
)
```
## Meta Tools
Meta tools are **built-in introspection tools** that provide information about the computer's capabilities.
### Registering Meta Tools
Use the `@Computer.meta_tool()` decorator to register a method as a meta tool:
```python
class Computer:
@meta_tool("list_tools")
async def list_tools(
self,
tool_type: Optional[str] = None,
namespace: Optional[str] = None,
remove_meta: bool = True
) -> CallToolResult:
"""List all available tools."""
# Implementation...
```
### Using Meta Tools
```python
# List all action tools
tool_call = MCPToolCall(
tool_key="action::list_tools",
tool_name="list_tools",
parameters={"tool_type": "action"}
)
result = await computer.run_actions([tool_call])
tools = result[0].data # List of available action tools
```
**Example:**
```python
# List all tools in "screenshot" namespace
result = await computer.run_actions([
MCPToolCall(
tool_key="data_collection::list_tools",
tool_name="list_tools",
parameters={"namespace": "screenshot", "remove_meta": True}
)
])
# Returns: [{"tool_name": "take_screenshot", "description": "...", ...}]
```
## Dynamic Server Management
### Adding a Server
```python
from ufo.client.mcp.mcp_server_manager import BaseMCPServer
# Create new MCP server
new_server = mcp_manager.create_or_get_server(
mcp_config={
"namespace": "custom_tools",
"server_type": "local",
"module": "my_custom_mcp_server"
},
reset=False,
process_name="my_process"
)
# Add to computer
await computer.add_server(
namespace="custom_tools",
mcp_server=new_server,
tool_type="action"
)
```
### Removing a Server
```python
# Remove server and all its tools
await computer.delete_server(
namespace="custom_tools",
tool_type="action"
)
```
**Use cases for dynamic server management:**
- Add specialized tools for specific tasks
- Remove servers to reduce memory footprint
- Hot-reload MCP servers during development
## Command Routing
The `CommandRouter` orchestrates command execution across multiple computers.
### Basic Usage
```python
from ufo.client.computer import CommandRouter
from aip.messages import Command, Result
# Create router
router = CommandRouter(computer_manager=manager)
# Execute commands
commands = [
Command(tool_name="screenshot", tool_type="data_collection"),
Command(tool_name="click", tool_type="action", parameters={"x": 100, "y": 200})
]
results = await router.execute(
agent_name="host_agent",
process_name="chrome",
root_name="default",
commands=commands,
early_exit=True # Stop on first error
)
for result in results:
print(f"Status: {result.status}")
print(f"Data: {result.data}")
```
### Error Handling
```python
# early_exit=True: Stop on first error
results = await router.execute(
agent_name="host_agent",
process_name="chrome",
root_name="default",
commands=commands,
early_exit=True
)
# early_exit=False: Execute all commands even if some fail
results = await router.execute(
agent_name="host_agent",
process_name="chrome",
root_name="default",
commands=commands,
early_exit=False
)
```
> **⚠️ Warning:** When `early_exit=True`, if a command fails, subsequent commands will **not** be executed, and their results will be set to `ResultStatus.SKIPPED`.
## Tool Registry
The tools registry maintains a mapping of all available tools.
### Tool Key Format
```python
tool_key = f"{tool_type}::{tool_name}"
# Examples:
"data_collection::screenshot"
"action::click"
"data_collection::list_tools" # Meta tool
```
### Accessing Tools
```python
# Get tool info
tool_info = computer._tools_registry.get("action::click")
# Tool info contains:
print(tool_info.tool_name) # "click"
print(tool_info.tool_type) # "action"
print(tool_info.namespace) # e.g., "gui_automation"
print(tool_info.description) # Tool description
print(tool_info.input_schema) # JSON schema for input parameters
print(tool_info.mcp_server) # Reference to MCP server
```
## Best Practices
### Configuration
1. **Use namespaces wisely**: Group related tools under meaningful namespaces
2. **Separate concerns**: Use `data_collection` for read-only operations, `action` for state changes
3. **Configure timeouts**: Adjust `_tool_timeout` for long-running operations
4. **Use default root**: Always provide a `default` root configuration as fallback
### Performance Optimization
1. **Register servers in parallel**: The `async_init()` method already does this via `asyncio.gather()`
2. **Reuse Computer instances**: Let `ComputerManager` cache instances rather than creating new ones
3. **Limit concurrent tools**: The thread pool has 10 workers; excessive parallel tools may queue
4. **Reset servers carefully**: Setting `reset=True` in server config will restart the MCP server process
### Common Pitfalls
> **⚠️ Important:** Avoid these common mistakes:
> - **Forgetting `async_init()`**: Always call after creating a `Computer` instance
> - **Tool key collisions**: Ensure tool names are unique within each `tool_type`
> - **Timeout too short**: Some operations (e.g., file downloads) may need longer timeouts
> - **Blocking in meta tools**: Meta tools should be fast; avoid I/O operations
## Error Handling
### Tool Execution Errors
```python
try:
results = await computer.run_actions([tool_call])
if results[0].is_error:
error_message = results[0].content[0].text
print(f"Tool error: {error_message}")
except ValueError as e:
print(f"Tool not registered: {e}")
except asyncio.TimeoutError:
print("Tool execution timed out")
except Exception as e:
print(f"Unexpected error: {e}")
```
### Configuration Errors
```python
try:
computer = await manager.get_or_create(
agent_name="host_agent",
process_name="chrome",
root_name="invalid_root"
)
except ValueError as e:
print(f"Configuration error: {e}")
# Fallback to default
computer = await manager.get_or_create(
agent_name="host_agent",
process_name="chrome",
root_name="default"
)
```
## Integration Points
### With UFO Client
The `Computer` is created and managed by the `UFOClient`:
```python
# In UFOClient
self.command_router = CommandRouter(computer_manager)
# Execute commands from server
results = await self.command_router.execute(
agent_name=self.agent_name,
process_name=self.process_name,
root_name=self.root_name,
commands=command_list
)
```
### With MCP Server Manager
The `Computer` relies on `MCPServerManager` for server lifecycle management:
```python
# Create or get existing MCP server
mcp_server = self.mcp_server_manager.create_or_get_server(
mcp_config=server_config,
reset=False,
process_name=self._process_name
)
```
See [MCP Integration](mcp_integration.md) for more details on MCP server management.
## Related Documentation
- [UFO Client Overview](overview.md) - High-level client architecture
- [UFO Client](ufo_client.md) - Command execution orchestration
- [Computer Manager](computer_manager.md) - Multi-computer instance management
- [MCP Integration](mcp_integration.md) - MCP server details
- [AIP Messages](../aip/messages.md) - Command and Result message formats
---
### Documents Docs Client Computer Manager (`documents/docs/client/computer_manager.md`)
# Computer Manager & Computer
The **Computer Manager** orchestrates multiple **Computer** instances, each representing an isolated execution namespace with dedicated MCP servers and tools. This enables context-specific tool routing and fine-grained control over data collection vs. action execution.
---
## Overview
The Computer layer consists of two components working together:
- **ComputerManager**: High-level orchestrator managing multiple Computer instances
- **Computer**: Individual execution namespace with its own MCP servers and tool registry
### Computer Manager Responsibilities
| Capability | Description | Implementation |
|------------|-------------|----------------|
| **Multi-Computer Management** | Create and manage multiple Computer instances | Per-process, per-agent namespaces |
| **Namespace Isolation** | Separate tool namespaces for different contexts | Independent MCP servers per Computer |
| **Command Routing** | Route commands to appropriate Computer instances | CommandRouter resolves by agent/process/root |
| **MCP Server Configuration** | Configure data collection and action servers | Config-driven server initialization |
| **Lifecycle Management** | Initialize, reset, and tear down Computers | Async initialization, cascading reset |
### Computer (Instance) Responsibilities
| Capability | Description | Implementation |
|------------|-------------|----------------|
| **Tool Registry** | Maintain registry of available MCP tools | `_tools_registry` dict |
| **Tool Execution** | Execute MCP tool calls with timeout protection | Thread pool isolation (max 10 workers) |
| **Server Management** | Manage data collection and action MCP servers | Separate namespaces |
| **Meta Tools** | Provide built-in tools (list_tools, etc.) | Decorated meta tool methods |
| **Async Initialization** | Initialize MCP servers asynchronously | `async_init()` |
**Architectural Relationship:**
```mermaid
graph TB
subgraph "Computer Manager Layer"
CM[Computer Manager]
CR[Command Router]
end
subgraph "Computer Instances"
C1[Computer: default]
C2[Computer: notepad.exe]
C3[Computer: explorer.exe]
end
subgraph "Computer 1 Components"
C1 --> DC1[Data Collection Servers]
C1 --> AS1[Action Servers]
C1 --> TR1[Tool Registry]
C1 --> MT1[Meta Tools]
end
CM -->|manages| C1
CM -->|manages| C2
CM -->|manages| C3
CR -->|routes to| C1
CR -->|routes to| C2
CR -->|routes to| C3
style CM fill:#ffe0b2
style C1 fill:#bbdefb
style C2 fill:#bbdefb
style C3 fill:#bbdefb
```
---
## 🏗️ Computer Manager Architecture
### Computer Instance Management
```mermaid
graph LR
subgraph "ComputerManager"
Config[UFO Config]
Registry[Computer Registry]
end
subgraph "Computers"
Default[default_agent]
Proc1[notepad.exe]
Proc2[explorer.exe]
end
Config -->|creates| Default
Config -->|creates| Proc1
Config -->|creates| Proc2
Registry -->|tracks| Default
Registry -->|tracks| Proc1
Registry -->|tracks| Proc2
style Config fill:#fff3e0
style Registry fill:#e1f5fe
```
**Computer Namespaces:**
| Namespace Type | Purpose | Example |
|----------------|---------|---------|
| **Data Collection** | Gathering information, non-invasive queries | Screenshots, UI element detection, app state |
| **Action** | Performing actions, invasive operations | GUI automation, file operations, app control |
Data collection tools are designed for non-invasive information gathering, while action tools have full control for state-changing operations.
---
## Computer Manager Architecture
## 🖥️ Computer (Instance) Architecture
### Internal Structure
```mermaid
graph TB
subgraph "Computer Instance"
Init[Initialization]
Servers[MCP Servers]
Registry[Tool Registry]
Execution[Tool Execution]
end
subgraph "MCP Servers"
Servers --> DC[Data Collection Servers]
Servers --> AS[Action Servers]
end
subgraph "Tool Registry"
Registry --> TR[_tools_registry Dict]
TR -->|key: action::click| T1[MCPToolCall]
TR -->|key: data_collection::screenshot| T2[MCPToolCall]
TR -->|key: action::list_tools| T3[Meta Tool]
end
subgraph "Execution Engine"
Execution --> TP[Thread Pool Executor]
Execution --> TO[Timeout Protection]
TP -->|max 10 workers| Threads[Isolated Threads]
end
Init --> Servers
Servers --> Registry
Registry --> Execution
style Init fill:#c8e6c9
style Servers fill:#bbdefb
style Registry fill:#fff9c4
style Execution fill:#ffccbc
```
**Key Attributes:**
| Attribute | Type | Purpose |
|-----------|------|---------|
| `_name` | `str` | Computer name (identifier) |
| `_process_name` | `str` | Associated process (e.g., "notepad.exe") |
| `_data_collection_servers` | `Dict[str, BaseMCPServer]` | Namespace → MCP server mapping (data collection) |
| `_action_servers` | `Dict[str, BaseMCPServer]` | Namespace → MCP server mapping (actions) |
| `_tools_registry` | `Dict[str, MCPToolCall]` | Tool key → tool info mapping |
| `_meta_tools` | `Dict[str, Callable]` | Built-in meta tools |
| `_executor` | `ThreadPoolExecutor` | Thread pool for tool execution (10 workers) |
| `_tool_timeout` | `int` | Tool execution timeout: **6000 seconds (100 minutes)** |
> **Note:** The tool execution timeout is 6000 seconds (100 minutes), allowing for very long-running operations while preventing indefinite hangs.
---
## Initialization
### Computer Manager Initialization
**Creating Computer Manager:**
```python
from ufo.client.computer import ComputerManager
from ufo.client.mcp.mcp_server_manager import MCPServerManager
from config.config_loader import get_ufo_config
# 1. Get UFO configuration
ufo_config = get_ufo_config()
# 2. Initialize MCP server manager
mcp_server_manager = MCPServerManager()
# 3. Create computer manager
computer_manager = ComputerManager(
ufo_config.to_dict(),
mcp_server_manager
)
```
### Computer Instance Initialization
**Computer Async Initialization:**
```python
computer = Computer(
name="default_agent",
process_name="explorer.exe",
mcp_server_manager=mcp_server_manager,
data_collection_servers_config=[...],
action_servers_config=[...]
)
# Async initialization (required)
await computer.async_init()
```
**Initialization Flow:**
```mermaid
sequenceDiagram
participant Code
participant Computer
participant MCP as MCP Server Manager
participant Servers
Code->>Computer: __init__(name, process, configs)
Computer->>Computer: Create thread pool executor
Computer->>Computer: Register meta tools
Code->>Computer: async_init()
Computer->>Computer: _init_data_collection_servers()
Computer->>MCP: create_or_get_server(config)
MCP-->>Computer: BaseMCPServer
Computer->>Computer: _init_action_servers()
Computer->>MCP: create_or_get_server(config)
MCP-->>Computer: BaseMCPServer
par Register Data Collection Servers
Computer->>Servers: register_mcp_servers(data_collection)
and Register Action Servers
Computer->>Servers: register_mcp_servers(action)
end
Servers-->>Computer: Tools registered
```
**Configuration Example:**
```yaml
data_collection_servers:
- namespace: screenshot_collector
type: local
module: ufo.client.mcp.local_servers.screenshot_server
reset: false
- namespace: ui_collector
type: local
module: ufo.client.mcp.local_servers.ui_server
reset: false
action_servers:
- namespace: gui_automator
type: local
module: ufo.client.mcp.local_servers.automation_server
reset: false
```
---
## 🔀 Command Routing
### CommandRouter
The CommandRouter resolves which Computer instance should handle each command based on agent/process/root context.
**Routing Signature:**
```python
async def execute(
self,
agent_name: str,
process_name: str,
root_name: str,
commands: List[Command]
) -> List[Result]
```
**Routing Logic:**
```mermaid
graph TD
Start[Command List]
Start --> Resolve[Resolve Computer Instance]
Resolve -->|agent_name, process_name, root_name| Computer[Get/Create Computer]
Computer --> Loop[For Each Command]
Loop --> Parse[Parse Command to MCPToolCall]
Parse --> Lookup[Lookup Tool in Registry]
Lookup -->|Found| Execute[Execute Tool]
Lookup -->|Not Found| Error[Return Error Result]
Execute --> Timeout[Tool Execution with Timeout]
Timeout -->|Success| Result[Return Result]
Timeout -->|Timeout| TimeoutError[Timeout Error Result]
Timeout -->|Exception| ExecError[Execution Error Result]
Result --> Collect[Collect Results]
Error --> Collect
TimeoutError --> Collect
ExecError --> Collect
Collect --> Return[Return List[Result]]
style Start fill:#e1f5fe
style Computer fill:#bbdefb
style Execute fill:#c8e6c9
style Collect fill:#fff9c4
```
---
## 🔧 Tool Execution
### Tool Execution Pipeline
MCP tools are executed in isolated threads to prevent blocking operations (like `time.sleep`) from blocking the main event loop and causing WebSocket disconnections.
**Execution Flow:**
```mermaid
sequenceDiagram
participant Computer
participant TP as Thread Pool
participant Thread
participant Loop as New Event Loop
participant MCP as MCP Server
Computer->>Computer: _run_action(tool_call)
Computer->>Computer: Lookup tool in registry
alt Meta Tool
Computer->>Computer: Execute meta tool directly
Computer-->>Computer: Result
else MCP Tool
Computer->>TP: Submit _call_tool_in_thread()
TP->>Thread: Execute in thread
Thread->>Loop: Create new event loop
Loop->>MCP: client.call_tool(name, params)
alt Success (within timeout)
MCP-->>Loop: Result
Loop-->>Thread: Result
Thread-->>TP: Result
TP-->>Computer: CallToolResult
else Timeout (> 6000s)
Note over Computer,MCP: Tool execution timeout
Computer-->>Computer: TimeoutError Result
else Exception
Note over Computer,MCP: Tool execution failed
Computer-->>Computer: Error Result
end
end
```
**Thread Pool Configuration:**
| Parameter | Value | Purpose |
|-----------|-------|---------|
| `max_workers` | **10** | Maximum concurrent tool executions |
| `thread_name_prefix` | `"mcp_tool_"` | Thread naming for debugging |
| Timeout | **6000 seconds (100 minutes)** | Per-tool execution timeout |
**Code Implementation:**
```python
def _call_tool_in_thread():
"""
Execute MCP tool call in an isolated thread with its own event loop.
This prevents blocking operations in MCP tools from blocking the main event loop.
"""
# Create a new event loop for this thread
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
try:
async def _do_call():
async with Client(server) as client:
return await client.call_tool(
name=tool_name, arguments=params, raise_on_error=False
)
return loop.run_until_complete(_do_call())
finally:
loop.close()
# Execute in thread pool with timeout protection
result = await asyncio.wait_for(
loop.run_in_executor(self._executor, _call_tool_in_thread),
timeout=self._tool_timeout
)
```
---
## 🛠️ Tool Registry
### Tool Registration
Tools are discovered from MCP servers during initialization and registered with unique keys.
**Tool Key Format:**
```
::
Examples:
- action::click
- action::type_text
- data_collection::screenshot
- data_collection::get_ui_elements
```
**Registration Process:**
```python
async def register_one_mcp_server(
self, namespace: str, tool_type: str, mcp_server: BaseMCPServer
) -> None:
async with Client(mcp_server.server) as client:
tools = await client.list_tools()
for tool in tools:
tool_key = self.make_tool_key(tool_type, tool.name)
self._register_tool(
tool_key=tool_key,
tool_name=tool.name,
title=tool.title,
namespace=namespace,
tool_type=tool_type,
description=tool.description,
input_schema=tool.inputSchema,
output_schema=tool.outputSchema,
mcp_server=mcp_server
)
```
**MCPToolCall Structure:**
| Field | Type | Description |
|-------|------|-------------|
| `tool_key` | `str` | Unique key (e.g., "action::click") |
| `tool_name` | `str` | Tool name (e.g., "click") |
| `title` | `str` | Display title |
| `namespace` | `str` | Server namespace |
| `tool_type` | `str` | "action" or "data_collection" |
| `description` | `str` | Tool description |
| `input_schema` | `Dict` | Input parameters schema |
| `output_schema` | `Dict` | Output schema |
| `mcp_server` | `BaseMCPServer` | Reference to server |
---
## Meta Tools
Meta tools are built-in methods decorated with `@meta_tool` that provide computer-level operations.
**Example: list_tools Meta Tool**
```python
@Computer.meta_tool("list_tools")
async def list_tools(
self,
tool_type: Optional[str] = None,
namespace: Optional[str] = None,
remove_meta: bool = True
) -> CallToolResult:
"""
Get available tools of a specific type.
"""
tools = []
for tool in self._tools_registry.values():
if ((tool_type is None or tool.tool_type == tool_type)
and (namespace is None or tool.namespace == namespace)
and (not remove_meta or tool.tool_name not in self._meta_tools)):
tools.append(tool.tool_info.model_dump())
return CallToolResult(
content=[TextContent(type="text", text=json.dumps(tools))]
)
```
**Meta Tool Registration:**
```python
# In __init__:
for attr in dir(self):
method = getattr(self, attr)
if callable(method) and hasattr(method, "_meta_tool_name"):
name = getattr(method, "_meta_tool_name")
self._meta_tools[name] = method
```
---
## 🔄 Lifecycle Management
### Reset
```python
# Computer Manager reset (cascades to all computers)
computer_manager.reset()
# Computer instance reset
computer.reset()
```
**Reset Operations:**
| Component | Reset Action |
|-----------|--------------|
| Computer Manager | Reset all Computer instances |
| Computer | Clear tool registry, reset MCP servers |
| MCP Servers | Reset server state |
---
## Best Practices
### Monitor Tool Execution Times
```python
import time
start = time.time()
result = await computer._run_action(tool_call)
duration = time.time() - start
if duration > 300: # 5 minutes
logger.warning(f"Slow tool: {tool_call.tool_name} took {duration}s")
```
### Handle Timeouts Gracefully
```python
# 100-minute timeout is generous but not infinite
# Design tools to complete within reasonable time
```
### Use Namespace Isolation
```python
# Separate data collection from actions
data_tools = await computer.list_tools(tool_type="data_collection")
action_tools = await computer.list_tools(tool_type="action")
```
---
## 🚀 Next Steps
👉 [Device Info Provider](./device_info.md) - System profiling
👉 [MCP Integration](./mcp_integration.md) - MCP server details
👉 [UFO Client](./ufo_client.md) - Execution orchestration
👉 [Quick Start](./quick_start.md) - Get started with client
👉 [Configuration](../configuration/system/overview.md) - UFO configuration
---
### Documents Docs Client Device Info (`documents/docs/client/device_info.md`)
# 📱 Device Info Provider
The **Device Info Provider** collects comprehensive system information from client devices during registration, enabling intelligent task assignment and device selection in constellation (multi-device) scenarios.
Device information is proactively collected during client registration and pushed to the server, reducing latency and enabling immediate task routing decisions.
---
## 📋 Overview
**Core Capabilities:**
| Capability | Description | Use Case |
|------------|-------------|----------|
| **System Detection** | Auto-detect OS, version, architecture | Platform-specific task routing |
| **Hardware Profiling** | CPU count, memory capacity | Resource-aware task assignment |
| **Network Discovery** | Hostname, IP address | Network topology mapping |
| **Feature Detection** | GUI, CLI, browser, office apps | Capability-based device selection |
| **Extensibility** | Custom metadata support | Environment-specific configuration |
**Supported Platforms:**
| Platform | Status | Features Detected |
|----------|--------|-------------------|
| **Windows** | ✅ Full Support | GUI, CLI, browser, file system, office, Windows apps |
| **Linux** | ✅ Full Support | GUI, CLI, browser, file system, office, Linux apps |
| **macOS** | ✅ Full Support | GUI, CLI, browser, file system, office |
| **Mobile** | 🔮 Planned | Touch, mobile apps, sensors |
| **IoT** | 🔮 Planned | Sensors, actuators, limited resources |
---
## 🏗️ Architecture
### DeviceSystemInfo Dataclass
The device info structure captures essential information to minimize registration overhead:
```mermaid
classDiagram
class DeviceSystemInfo {
+string device_id
+string platform
+string os_version
+int cpu_count
+float memory_total_gb
+string hostname
+string ip_address
+List~string~ supported_features
+string platform_type
+string schema_version
+Dict custom_metadata
+to_dict() Dict
}
class DeviceInfoProvider {
+collect_system_info() DeviceSystemInfo
-_get_platform() string
-_get_os_version() string
-_get_cpu_count() int
-_get_memory_total_gb() float
-_get_hostname() string
-_get_ip_address() string
-_detect_features() List~string~
-_get_platform_type() string
}
DeviceInfoProvider ..> DeviceSystemInfo : creates
```
**Field Reference:**
| Field | Type | Description | Example |
|-------|------|-------------|---------|
| `device_id` | `str` | Unique client identifier | `"device_windows_001"` |
| `platform` | `str` | OS platform (lowercase) | `"windows"`, `"linux"`, `"darwin"` |
| `os_version` | `str` | OS version string | `"10.0.19045"` (Windows 10) |
| `cpu_count` | `int` | Number of CPU cores | `8` |
| `memory_total_gb` | `float` | Total RAM in GB (rounded to 2 decimals) | `16.0` |
| `hostname` | `str` | Network hostname | `"DESKTOP-ABC123"` |
| `ip_address` | `str` | Local IP address | `"192.168.1.100"` |
| `supported_features` | `List[str]` | Detected capabilities | `["gui", "cli", "browser", "office"]` |
| `platform_type` | `str` | Device category | `"computer"`, `"mobile"`, `"web"`, `"iot"` |
| `schema_version` | `str` | Schema version for compatibility | `"1.0"` |
| `custom_metadata` | `Dict` | User-defined metadata | `{"environment": "production"}` |
---
## 🔍 Collection Process
### Automatic Collection
```python
from ufo.client.device_info_provider import DeviceInfoProvider
# Collect system information
system_info = DeviceInfoProvider.collect_system_info(
client_id="device_windows_001",
custom_metadata=None # Or load from config
)
# Result: DeviceSystemInfo object
print(system_info.platform) # "windows"
print(system_info.cpu_count) # 8
print(system_info.memory_total_gb) # 16.0
print(system_info.supported_features) # ["gui", "cli", "browser", ...]
# Convert to dict for transmission
device_dict = system_info.to_dict()
```
**Collection Flow:**
```mermaid
sequenceDiagram
participant Client
participant DIP as Device Info Provider
participant OS as Operating System
Client->>DIP: collect_system_info(client_id, custom_metadata)
par Collect Basic Info
DIP->>OS: platform.system()
OS-->>DIP: "Windows"
DIP->>OS: platform.version()
OS-->>DIP: "10.0.19045"
and Collect Hardware Info
DIP->>OS: os.cpu_count()
OS-->>DIP: 8
DIP->>OS: psutil.virtual_memory()
OS-->>DIP: 16GB
and Collect Network Info
DIP->>OS: socket.gethostname()
OS-->>DIP: "DESKTOP-ABC123"
DIP->>OS: socket.getsockname()
OS-->>DIP: "192.168.1.100"
end
DIP->>DIP: _detect_features()
DIP->>DIP: _get_platform_type()
DIP-->>Client: DeviceSystemInfo
```
---
## 🎯 Feature Detection
### Platform-Specific Features
Features are automatically detected based on the platform to enable capability-based device selection.
**Windows Features:**
```python
features = [
"gui", # Graphical user interface
"cli", # Command line interface
"browser", # Web browser support
"file_system", # File system operations
"office", # Office applications (Word, Excel, etc.)
"windows_apps" # Windows-specific applications
]
```
**Linux Features:**
```python
features = [
"gui", # Graphical user interface (X11/Wayland)
"cli", # Bash/shell
"browser", # Firefox, Chrome, etc.
"file_system", # Linux file system
"office", # LibreOffice, etc.
"linux_apps" # Linux-specific applications
]
```
**macOS Features:**
```python
features = [
"gui", # macOS GUI
"cli", # Terminal
"browser", # Safari, Chrome, etc.
"file_system", # macOS file system
"office" # Office for Mac
]
```
**Feature Detection Logic:**
| Platform | Detected Features | Rationale |
|----------|-------------------|-----------|
| `windows`, `linux`, `darwin` | GUI, CLI, browser, file_system, office | Desktop/laptop computers have full capabilities |
| `android`, `ios` (future) | Touch, mobile apps, camera | Mobile-specific features |
| Custom | User-defined | Extensible via custom_metadata |
---
## 💡 Usage Examples
### Basic Collection
```python
from ufo.client.device_info_provider import DeviceInfoProvider
# Collect with auto-detection
info = DeviceInfoProvider.collect_system_info(
client_id="device_001",
custom_metadata=None
)
print(f"Platform: {info.platform}")
print(f"CPU Cores: {info.cpu_count}")
print(f"Memory: {info.memory_total_gb} GB")
print(f"Features: {', '.join(info.supported_features)}")
```
### With Custom Metadata
```python
# Add environment-specific metadata
custom_meta = {
"environment": "production",
"datacenter": "us-east-1",
"role": "automation_worker",
"team": "qa"
}
info = DeviceInfoProvider.collect_system_info(
client_id="device_prod_001",
custom_metadata=custom_meta
)
# Custom metadata is preserved
print(info.custom_metadata["environment"]) # "production"
```
### JSON Serialization
```python
# Convert to dictionary for transmission
device_dict = info.to_dict()
# Serialize to JSON
import json
json_str = json.dumps(device_dict, indent=2)
# Example output:
# {
# "device_id": "device_001",
# "platform": "windows",
# "os_version": "10.0.19045",
# "cpu_count": 8,
# "memory_total_gb": 16.0,
# "hostname": "DESKTOP-ABC123",
# "ip_address": "192.168.1.100",
# "supported_features": ["gui", "cli", "browser", "file_system", "office", "windows_apps"],
# "platform_type": "computer",
# "schema_version": "1.0",
# "custom_metadata": {}
# }
```
---
## ⚠️ Error Handling
### Graceful Degradation
If any detection method fails, the provider returns minimal info instead of crashing.
**Error Handling Strategy:**
```python
try:
# Attempt full collection
return DeviceSystemInfo(...)
except Exception as e:
logger.error(f"Error collecting system info: {e}", exc_info=True)
# Return minimal info on error
return DeviceSystemInfo(
device_id=client_id,
platform="unknown",
os_version="unknown",
cpu_count=0,
memory_total_gb=0.0,
hostname="unknown",
ip_address="unknown",
supported_features=[],
platform_type="unknown",
custom_metadata=custom_metadata or {}
)
```
**Individual Method Failures:**
| Method | Failure Behavior | Fallback Value |
|--------|------------------|----------------|
| `_get_platform()` | Catch exception | `"unknown"` |
| `_get_os_version()` | Catch exception | `"unknown"` |
| `_get_cpu_count()` | Catch exception | `0` |
| `_get_memory_total_gb()` | psutil not installed or exception | `0.0` |
| `_get_hostname()` | Catch exception | `"unknown"` |
| `_get_ip_address()` | Primary method fails | Try hostname resolution, then `"unknown"` |
---
## 🔧 Memory Detection Details
### psutil Dependency
!!!warning "Optional Dependency"
Memory detection requires `psutil`. If not installed, memory will be reported as `0.0`.
**Installation:**
```bash
pip install psutil
```
**Detection Code:**
```python
@staticmethod
def _get_memory_total_gb() -> float:
"""Get total memory in GB"""
try:
import psutil
total_memory = psutil.virtual_memory().total
return round(total_memory / (1024**3), 2) # Convert to GB, round to 2 decimals
except ImportError:
logger.warning("psutil not installed, memory info unavailable")
return 0.0
except Exception:
return 0.0
```
---
## 🌐 IP Address Detection
### Multi-Method Approach
!!!tip "Robust IP Detection"
IP detection uses a two-stage approach for reliability.
**Primary Method (Socket Connection):**
```python
# Connect to external address (doesn't actually send data)
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(("8.8.8.8", 80)) # Google DNS
ip = s.getsockname()[0]
s.close()
```
**Fallback Method (Hostname Resolution):**
```python
# If primary fails, resolve via hostname
ip = socket.gethostbyname(socket.gethostname())
```
**Final Fallback:**
```python
# If all methods fail
return "unknown"
```
---
## 🚀 Integration Points
### WebSocket Client Registration
The WebSocket client uses the Device Info Provider during registration:
```python
# In websocket client's register_client()
from ufo.client.device_info_provider import DeviceInfoProvider
system_info = DeviceInfoProvider.collect_system_info(
self.ufo_client.client_id,
custom_metadata=None
)
metadata = {
"system_info": system_info.to_dict(),
"registration_time": datetime.now(timezone.utc).isoformat()
}
await self.registration_protocol.register_as_device(
device_id=self.ufo_client.client_id,
metadata=metadata,
platform=self.ufo_client.platform
)
```
See [WebSocket Client](./websocket_client.md) for complete registration flow details.
### Agent Server
The server receives device info during registration and stores it in the agent profile:
```python
# Server-side AgentProfile integration
device_info = registration_data["metadata"]["system_info"]
agent_profile.add_device(device_id, device_info)
```
See [Server Quick Start](../server/quick_start.md) for server-side processing details.
---
## ✅ Best Practices
**1. Add Custom Metadata for Environment Tracking**
```python
custom_meta = {
"environment": os.getenv("ENVIRONMENT", "development"),
"version": "1.0.0",
"deployment_region": "us-west-2",
"cost_center": "engineering"
}
system_info = DeviceInfoProvider.collect_system_info(
client_id="device_001",
custom_metadata=custom_meta
)
```
**2. Install psutil for Accurate Memory Detection**
```bash
pip install psutil
```
**3. Use Descriptive Client IDs**
```python
# Include environment and location in client_id
client_id = f"device_{platform}_{env}_{location}_{instance_id}"
# Example: "device_windows_prod_us-west_001"
```
**4. Log Collection Results**
```python
system_info = DeviceInfoProvider.collect_system_info(...)
logger.info(
f"Collected device info: "
f"platform={system_info.platform}, "
f"cpu={system_info.cpu_count}, "
f"memory={system_info.memory_total_gb}GB, "
f"features={system_info.supported_features}"
)
```
**5. Validate Before Sending**
```python
system_info = DeviceInfoProvider.collect_system_info(...)
# Validate essential fields
assert system_info.device_id, "Device ID required"
assert system_info.platform != "unknown", "Platform detection failed"
assert system_info.cpu_count > 0, "CPU detection failed"
```
---
## 🚀 Next Steps
- [WebSocket Client](./websocket_client.md) - See how device info is used in registration
- [Quick Start](./quick_start.md) - Connect your device to the server
- [MCP Integration](./mcp_integration.md) - Understand client tool capabilities
- [Server Quick Start](../server/quick_start.md) - Learn server-side registration processing
---
### Documents Docs Client Mcp Integration (`documents/docs/client/mcp_integration.md`)
# 🔌 MCP Integration
**MCP (Model Context Protocol)** provides the tool execution layer in UFO clients, enabling agents to collect system state and execute actions through a standardized interface. This page provides a **client-focused overview** of how MCP integrates into the client architecture.
**Related Documentation:**
- [MCP Overview](../mcp/overview.md) - Core MCP concepts and architecture
- [Configuration Guide](../mcp/configuration.md) - Server configuration details
- [Data Collection Servers](../mcp/data_collection.md) - Observation tools
- [Action Servers](../mcp/action.md) - Execution tools
- [Creating MCP Servers](../tutorials/creating_mcp_servers.md) - Build custom tools
---
## 🏗️ MCP in Client Architecture
### Role in the Client Stack
```mermaid
graph TB
Server[Agent Server via WebSocket]
Client[UFO Client Session Orchestration]
Router[Command Router Command Execution]
Computer[Computer MCP Tool Manager]
MCPMgr[MCP Server Manager Server Lifecycle]
DataServers[Data Collection Servers UICollector, etc.]
ActionServers[Action Servers UIExecutor, CommandLineExecutor]
Server -->|AIP Commands| Client
Client -->|Execute Actions| Router
Router -->|Route to Computer| Computer
Computer -->|Manage Servers| MCPMgr
Computer -->|Register & Execute| DataServers
Computer -->|Register & Execute| ActionServers
style Computer fill:#e1f5ff
style MCPMgr fill:#fff4e6
style DataServers fill:#e8f5e9
style ActionServers fill:#fff3e0
```
**Key Components:**
| Component | Location | Responsibility |
|-----------|----------|----------------|
| **Computer** | `ufo.client.computer.Computer` | Manages MCP servers, routes tool calls, executes in thread pool |
| **MCP Server Manager** | `ufo.client.mcp.mcp_server_manager.MCPServerManager` | Creates/manages server instances (local/http/stdio) |
| **Command Router** | `ufo.client.computer.CommandRouter` | Routes commands to appropriate Computer instances |
| **Data Collection Servers** | Various MCP servers | Tools for gathering system state (read-only) |
| **Action Servers** | Various MCP servers | Tools for performing state changes |
---
## 🔄 Client-MCP Integration Flow
### End-to-End Execution
```mermaid
sequenceDiagram
participant Server as Agent Server
participant Client as UFO Client
participant Router as Command Router
participant Computer as Computer
participant MCP as MCP Server
Server->>Client: AIP Command (tool_name, parameters)
Client->>Router: execute_actions(commands)
Router->>Computer: command2tool()
Computer->>Computer: Convert to MCPToolCall
Router->>Computer: run_actions([tool_call])
Computer->>MCP: call_tool(tool_name, parameters)
MCP-->>Computer: CallToolResult
Computer-->>Router: Results
Router-->>Client: List[Result]
Client-->>Server: AIP Result message
```
**Execution Stages:**
| Stage | Component | Description |
|-------|-----------|-------------|
| **1. Command Reception** | UFO Client | Receives AIP Command from server |
| **2. Command Routing** | Command Router | Routes to appropriate Computer instance |
| **3. Command Conversion** | Computer | AIP Command → MCPToolCall |
| **4. Tool Execution** | Computer | Executes tool via MCP Server |
| **5. Result Return** | UFO Client | Packages result for server |
---
## 💻 Computer: The MCP Manager
### Computer Class Overview
The `Computer` class is the **client-side MCP manager**, handling server registration, tool discovery, and execution.
**Core Responsibilities:**
```python
from ufo.client.computer import Computer
from ufo.client.mcp.mcp_server_manager import MCPServerManager
# Initialize Computer with MCP servers
computer = Computer(
name="notepad_computer",
process_name="notepad.exe",
mcp_server_manager=mcp_manager,
data_collection_servers_config=[
{"namespace": "UICollector", "type": "local", "reset": False}
],
action_servers_config=[
{"namespace": "HostUIExecutor", "type": "local", "reset": False}
]
)
# Async initialization registers all tools
await computer.async_init()
```
**Initialization Sequence:**
| Step | Action | Result |
|------|--------|--------|
| 1. Create MCP Server Manager | Initialize server lifecycle manager | Ready to create servers |
| 2. Initialize data_collection servers | Register observation tools | UICollector ready |
| 3. Initialize action servers | Register execution tools | HostUIExecutor, CommandLineExecutor ready |
| 4. Register MCP servers | Query each server for tools | Tool registry populated |
See [Computer](./computer.md) for detailed class documentation.
---
## 🛠️ Two Server Types
### Data Collection vs Action
Understanding the difference between server types is essential for proper MCP usage:
**Comparison:**
| Aspect | Data Collection Servers | Action Servers |
|--------|------------------------|----------------|
| **Purpose** | Observe system state | Modify system state |
| **Examples** | `take_screenshot`, `detect_ui_elements` | `click`, `type_text`, `run_command` |
| **Invocation** | LLM-selected tools | LLM-selected tools |
| **Side Effects** | ❌ None (read-only) | ✅ Yes (state changes) |
| **Namespace** | `"data_collection"` | `"action"` |
| **Tool Key Format** | `data_collection::tool_name` | `action::tool_name` |
**Data Collection Example:**
```python
# Example: Take screenshot for UI analysis
result = await computer.run_actions([
computer.command2tool(Command(
tool_name="take_screenshot",
tool_type="data_collection",
parameters={"region": "active_window"}
))
])
```
**Action Example:**
```python
# Example: Click a button
result = await computer.run_actions([
computer.command2tool(Command(
tool_name="click",
tool_type="action",
parameters={
"control_text": "Save",
"control_type": "Button"
}
))
])
```
See [MCP Overview - Server Types](../mcp/overview.md#1-two-server-types) for detailed comparison.
---
## 📋 Server Configuration
### Configuration File
MCP servers are configured in `config/ufo/mcp.yaml`:
```yaml
HostAgent:
default:
data_collection:
- namespace: UICollector # Server namespace
type: local # local, http, or stdio
reset: false # Reset on each step?
action:
- namespace: HostUIExecutor # Server namespace
type: local
reset: false
- namespace: CommandLineExecutor # Multiple servers allowed
type: local
reset: false
```
**Configuration Parameters:**
| Parameter | Type | Description | Example |
|-----------|------|-------------|---------|
| `namespace` | `str` | Server identifier (must match registered name) | `"UICollector"` |
| `type` | `str` | Deployment type: `local`, `http`, `stdio` | `"local"` |
| `reset` | `bool` | Reset server state on each step | `false` |
!!!tip "📖 Full Configuration Guide"
See [MCP Configuration](../mcp/configuration.md) for advanced configuration including:
- HTTP server endpoints
- Stdio server commands
- Custom server parameters
- Environment-specific configs
---
## 🔧 Tool Registry & Execution
### Tool Discovery
The Computer automatically discovers and registers tools from all configured MCP servers during initialization:
**Automatic Registration:**
```python
# During computer.async_init()
async def register_mcp_servers(self, servers, tool_type):
"""Register tools from all MCP servers"""
for namespace, server in servers.items():
# Connect to MCP server
async with Client(server.server) as client:
# List available tools
tools = await client.list_tools()
# Register each tool with unique key
for tool in tools:
tool_key = self.make_tool_key(tool_type, tool.name)
self._tools_registry[tool_key] = MCPToolCall(
tool_key=tool_key,
tool_name=tool.name,
title=tool.title,
namespace=namespace,
tool_type=tool_type,
description=tool.description,
input_schema=tool.inputSchema,
output_schema=tool.outputSchema,
mcp_server=server
)
```
**Tool Registry Structure:**
| Field | Type | Description |
|-------|------|-------------|
| `tool_key` | `str` | Unique key: `"tool_type::tool_name"` |
| `tool_name` | `str` | Tool name (e.g., `"take_screenshot"`) |
| `title` | `str` | Display title |
| `namespace` | `str` | Server namespace (e.g., `"UICollector"`) |
| `tool_type` | `str` | `"data_collection"` or `"action"` |
| `description` | `str` | Tool description |
| `input_schema` | `dict` | JSON schema for parameters |
| `output_schema` | `dict` | JSON schema for results |
| `mcp_server` | `BaseMCPServer` | Server instance |
### Tool Execution
Tools execute in isolated threads with timeout protection (default: 6000 seconds = 100 minutes per tool):
```python
# Thread pool configuration
self._executor = concurrent.futures.ThreadPoolExecutor(
max_workers=10,
thread_name_prefix="mcp_tool_"
)
self._tool_timeout = 6000 # 100 minutes
```
See [Computer](./computer.md) for execution details.
---
## 🚀 Integration Examples
### Basic Usage
```python
from ufo.client.computer import ComputerManager, CommandRouter
from ufo.client.mcp.mcp_server_manager import MCPServerManager
from aip.messages import Command
# Create MCP server manager
mcp_server_manager = MCPServerManager()
# Create computer manager (manages Computer instances)
computer_manager = ComputerManager(config, mcp_server_manager)
# Create command router
command_router = CommandRouter(computer_manager)
# Execute action through MCP
command = Command(
tool_name="click",
tool_type="action",
parameters={
"control_text": "Save",
"control_type": "Button"
}
)
# Router creates Computer instance and executes
results = await command_router.execute(
agent_name="HostAgent",
process_name="notepad.exe",
root_name="default",
commands=[command]
)
```
### Custom MCP Server
```python
from fastmcp import FastMCP
# Define custom MCP server
mcp = FastMCP("CustomTools")
@mcp.tool()
async def custom_action(param: str) -> str:
"""Execute custom action"""
return f"Executed: {param}"
# Register in config/ufo/mcp.yaml:
# action:
# - namespace: CustomTools
# type: local
# reset: false
```
**For step-by-step instructions:**
- [Creating MCP Servers](../tutorials/creating_mcp_servers.md) - Build your own MCP tools
---
## 🔗 Integration Points
### With Other Client Components
**UFO Client:**
- Receives AIP Commands from server
- Delegates to Command Router
- Returns AIP Results
**Command Router:**
- Routes commands to appropriate Computer instance (by agent/process/root name)
- Manages command execution with early-exit support
**Computer:**
- **MCP entry point**: Manages all MCP servers
- Executes tools via MCP Server Manager
- Maintains tool registry
**MCP Server Manager:**
- Creates and manages MCP server instances
- Supports local, HTTP, and stdio deployment types
See [UFO Client](./ufo_client.md) and [Computer](./computer.md) for integration details.
---
## 📚 Related Documentation
### Client Components
| Component | Description | Link |
|-----------|-------------|------|
| **Computer** | Core MCP execution layer | [Computer](./computer.md) |
| **UFO Client** | Session orchestration | [UFO Client](./ufo_client.md) |
| **WebSocket Client** | Server communication | [WebSocket Client](./websocket_client.md) |
### MCP Deep Dive
| Topic | Description | Link |
|-------|-------------|------|
| **MCP Overview** | Architecture, concepts, deployment models | [Overview](../mcp/overview.md) |
| **Data Collection** | Observation tools (UI, screenshots, system) | [Data Collection](../mcp/data_collection.md) |
| **Action Servers** | Execution tools (click, type, run) | [Action](../mcp/action.md) |
| **Configuration** | YAML configuration guide | [Configuration](../mcp/configuration.md) |
| **Local Servers** | Built-in in-process servers | [Local Servers](../mcp/local_servers.md) |
| **Remote Servers** | HTTP/Stdio deployment | [Remote Servers](../mcp/remote_servers.md) |
| **Creating MCP Servers** | Build your own tools | [Creating MCP Servers](../tutorials/creating_mcp_servers.md) |
---
## 🎯 Key Takeaways
**MCP in Client - Summary**
**1. Computer is the MCP Manager**
- Manages all MCP server instances
- Routes tool calls to appropriate servers
- Executes in thread pool for isolation
**2. Two Server Types**
- **Data Collection**: Read-only, observation tools
- **Action**: State-changing, execution tools
**3. Configuration-Driven**
- Servers configured in `config/ufo/mcp.yaml`
- Supports local, HTTP, and stdio deployment
**4. Automatic Registration**
- Tools auto-discovered during initialization
- Tool registry built from server metadata
**5. Detailed Docs Available**
- Full MCP section at [MCP Overview](../mcp/overview.md)
- Custom server guides, examples, troubleshooting
---
## 🚀 Next Steps
- [MCP Overview](../mcp/overview.md) - Understand MCP architecture in depth
- [Computer](./computer.md) - See how MCP servers are managed
- [Creating MCP Servers](../tutorials/creating_mcp_servers.md) - Build your own MCP tools
---
### Documents Docs Client Quick Start (`documents/docs/client/quick_start.md`)
# ⚡ Quick Start
Get your device connected to the UFO Agent Server and start executing tasks in minutes. No complex setup—just run a single command.
---
## 📋 Prerequisites
Before connecting a client device, ensure these requirements are met:
| Requirement | Version/Details | Verification Command |
|-------------|-----------------|----------------------|
| **Python** | 3.10 or higher | `python --version` |
| **UFO Installation** | Latest version with dependencies | `python -c "import ufo; print('✅ Installed')"` |
| **Running Server** | Agent server accessible on network | `curl http://server:5000/api/health` |
| **Network Access** | Client can reach server WebSocket endpoint | Test connectivity to server |
!!! tip "Server First!"
**Always start the Agent Server before connecting clients.** The server must be running and accessible for clients to register successfully.
👉 [Server Quick Start Guide](../server/quick_start.md)
**Verify Server Status:**
**Windows:**
```powershell
# Test HTTP API
Invoke-WebRequest -Uri http://localhost:5000/api/health
# Test WebSocket (requires wscat)
wscat -c ws://localhost:5000/ws
```
**Linux/macOS:**
```bash
# Test HTTP API
curl http://localhost:5000/api/health
# Test WebSocket (requires wscat)
wscat -c ws://localhost:5000/ws
```
---
## 🚀 Starting a Device Client
### Minimal Command (Local Server)
Connect to a server running on the same machine with default settings:
```bash
python -m ufo.client.client --ws --client-id my_device
```
**What This Does:**
| Parameter | Default Value | Purpose |
|-----------|---------------|---------|
| `--ws` | N/A (flag) | **Enable WebSocket mode** (required) |
| `--client-id` | `my_device` | Unique identifier for this device |
| `--ws-server` | `ws://localhost:5000/ws` | Connect to local server |
| `--platform` | Auto-detected | Detected from `platform.system()` |
| `--max-retries` | `5` | Connection retry attempts |
### Connect to Remote Server
Connect to a server running on a different machine in your network:
```bash
python -m ufo.client.client \
--ws \
--ws-server ws://192.168.1.100:5000/ws \
--client-id device_windows_001
```
**Network Requirements:**
- ✅ Client can ping the server: `ping 192.168.1.100`
- ✅ Port **5000** is accessible (firewall allows)
- ✅ Server is running and listening on correct port
### Override Platform Detection
!!! tip "When to Override"
Normally, the client auto-detects the platform (`windows` or `linux`). Override when:
- Running in container/VM with mismatched OS
- Testing cross-platform behavior
- Platform detection fails
```bash
python -m ufo.client.client \
--ws \
--ws-server ws://127.0.0.1:5000/ws \
--client-id my_linux_device \
--platform linux
```
### Complete Command (All Options)
Production-ready configuration with all available options:
```bash
python -m ufo.client.client \
--ws \
--ws-server ws://192.168.1.100:5000/ws \
--client-id device_windows_prod_01 \
--platform windows \
--max-retries 10 \
--log-level WARNING
```
**Enhancements:**
- 🔁 **10 retries**: Resilient to temporary network issues
- 📋 **WARNING logging**: Default level (less verbose than INFO)
- 🏷️ **Descriptive ID**: `device_windows_prod_01` clearly identifies environment
---
## 📝 Connection Parameters Reference
All available command-line options for the UFO client.
### Required Parameters
| Parameter | Description | Example |
|-----------|-------------|---------|
| `--ws` | **Enable WebSocket mode** (flag, no value) | `--ws` |
### Connection Parameters
| Parameter | Type | Default | Description | Example |
|-----------|------|---------|-------------|---------|
| `--ws-server` | `str` | `ws://localhost:5000/ws` | WebSocket server URL | `--ws-server ws://192.168.1.10:5000/ws` |
| `--max-retries` | `int` | `5` | Maximum connection retry attempts | `--max-retries 10` |
### Device Parameters
| Parameter | Type | Default | Description | Example |
|-----------|------|---------|-------------|---------|
| `--client-id` | `str` | `client_001` | **Unique device identifier** | `--client-id device_win_prod_01` |
| `--platform` | `str` | Auto-detect | Platform override: `windows` or `linux` | `--platform linux` |
### Logging Parameters
| Parameter | Type | Default | Description | Example |
|-----------|------|---------|-------------|---------|
| `--log-level` | `str` | `WARNING` | Logging verbosity: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`, `OFF` | `--log-level DEBUG` |
!!! warning "Unique Client IDs - Critical!"
**Each device MUST have a unique `--client-id`.** Duplicate IDs will cause:
- ❌ Connection conflicts (devices disconnecting each other)
- ❌ Task routing failures (tasks sent to wrong device)
- ❌ Session corruption (server state confusion)
**Best Practice:** Use descriptive IDs:
```
✅ device_windows_prod_datacenter1_rack3
✅ device_linux_staging_jenkins_worker2
❌ client_001
❌ device1
```
---
## ✅ Successful Connection
### Client Logs
When the client connects successfully, you'll see this sequence:
```log
INFO - Platform detected/specified: windows
INFO - UFO Client initialized for platform: windows
INFO - [WS] Connecting to ws://127.0.0.1:5000/ws (attempt 1/5)
INFO - [WS] [AIP] Collected device info: platform=windows, cpu=8, memory=16.0GB
INFO - [WS] [AIP] Attempting to register as device_windows_001
INFO - [WS] [AIP] ✅ Successfully registered as device_windows_001
INFO - [WS] Heartbeat loop started (interval: 30s)
```
**Registration Flow:**
```mermaid
sequenceDiagram
participant C as Client
participant S as Server
C->>C: Load Config & Initialize MCP
C->>S: WebSocket Connect
S-->>C: Connection Ack
C->>C: Collect Device Info
C->>S: REGISTRATION (id, platform, capabilities)
S->>S: Validate & Store
S-->>C: REGISTRATION_ACK
loop Every 30s
C->>S: HEARTBEAT
S-->>C: HEARTBEAT_ACK
end
Note over C,S: Ready for Commands
```
### Server Logs
On the server side, you'll see:
```log
INFO - [WS] ✅ Registered device client: device_windows_001
INFO - [WS] Device device_windows_001 capabilities: {
"platform": "windows",
"cpu_count": 8,
"memory_gb": 16.0,
"mcp_servers": ["system_info", "gui_automation"]
}
```
---
## 🔍 Verify Connection
### Check Connected Clients (HTTP API)
From the server machine or any network-accessible machine:
**cURL:**
```bash
curl http://localhost:5000/api/clients
```
**PowerShell:**
```powershell
Invoke-RestMethod -Uri http://localhost:5000/api/clients | ConvertTo-Json
```
**Python:**
```python
import requests
response = requests.get("http://localhost:5000/api/clients")
print(response.json())
```
**Expected Response:**
```json
{
"clients": [
{
"client_id": "device_windows_001",
"type": "device",
"platform": "windows",
"connected_at": 1730736000.0,
"uptime_seconds": 45,
"capabilities": {
"cpu_count": 8,
"memory_gb": 16.0,
"mcp_servers": ["system_info", "gui_automation"]
}
}
],
"total": 1
}
```
**Client Status Indicators:**
| Field | Description | Example |
|-------|-------------|---------|
| `client_id` | Unique device identifier | `device_windows_001` |
| `type` | Client type (always `"device"`) | `device` |
| `platform` | Operating system | `windows`, `linux` |
| `connected_at` | Unix timestamp of connection | `1730736000.0` |
| `uptime_seconds` | Seconds since connection | `45` |
| `capabilities` | Device hardware/software profile | CPU, memory, MCP servers |
### Monitor Heartbeats
The client sends **heartbeat messages every 30 seconds** to prove it's still alive.
**Client Logs (DEBUG level):**
```log
DEBUG - [WS] [AIP] Heartbeat sent
DEBUG - [WS] [AIP] Heartbeat acknowledged
```
**Server Logs (DEBUG level):**
```log
DEBUG - [WS] Heartbeat received from device_windows_001
DEBUG - [WS] Heartbeat acknowledged for device_windows_001
```
---
## 🎯 Running Your First Task
Once the client is connected, dispatch a simple task from the server to verify end-to-end functionality.
### Dispatch Task via HTTP API
**cURL:**
```bash
curl -X POST http://localhost:5000/api/dispatch \
-H "Content-Type: application/json" \
-d '{
"client_id": "device_windows_001",
"request": "Open Notepad and type Hello from UFO"
}'
```
**PowerShell:**
```powershell
$body = @{
client_id = "device_windows_001"
request = "Open Notepad and type Hello from UFO"
} | ConvertTo-Json
Invoke-RestMethod -Uri http://localhost:5000/api/dispatch `
-Method POST `
-ContentType "application/json" `
-Body $body
```
**Python:**
```python
import requests
response = requests.post(
"http://localhost:5000/api/dispatch",
json={
"client_id": "device_windows_001",
"request": "Open Notepad and type Hello from UFO"
}
)
print(response.json())
```
### Server Response
```json
{
"status": "success",
"session_id": "session_20251104_143022_abc123",
"message": "Task dispatched to device_windows_001",
"client_id": "device_windows_001"
}
```
**Response Fields:**
| Field | Description |
|-------|-------------|
| `status` | `"success"` or `"error"` |
| `session_id` | Unique session identifier for tracking |
| `message` | Human-readable status message |
| `client_id` | Target device that received the task |
### Client Execution Logs
```log
INFO - [WS] Starting task: Open Notepad and type Hello from UFO
INFO - [WS] [AIP] Sent task request with platform: windows
INFO - Executing 3 actions in total
INFO - [WS] [AIP] Sent client result for prev_response_id: resp_abc123
INFO - [WS] Task session_20251104_143022_abc123 completed
```
**Execution Flow:**
```mermaid
sequenceDiagram
participant API as HTTP API
participant Server
participant Client
participant App as Notepad
API->>Server: POST /dispatch
Server->>Server: Create Session
Server-->>API: {session_id, status}
Server->>Client: COMMAND
Client->>App: Launch & Type
App-->>Client: Done
Client->>Server: COMMAND_RESULTS
```
---
## ⚠️ Common Issues
### 1. Connection Refused
**Symptom:**
```log
ERROR - [WS] Unexpected error: [Errno 10061] Connect call failed
ERROR - [WS] Max retries reached. Exiting.
```
**Root Causes:**
| Cause | Verification | Solution |
|-------|--------------|----------|
| Server not running | `curl http://localhost:5000/api/health` | Start server first |
| Wrong port | Check server startup logs | Use correct port (`--ws-server ws://...`) |
| Firewall blocking | `telnet server 5000` | Allow port 5000 in firewall |
| Server using `--local` flag | Check server CLI args | Connect from localhost only |
**Solutions:**
**Verify Server:**
```bash
# Check if server is running
curl http://localhost:5000/api/health
# Expected response:
# {"status": "healthy", "uptime_seconds": 123}
```
**Check Firewall:**
```bash
# Windows: Check if port is listening
netstat -an | findstr ":5000"
# Linux: Check if port is listening
netstat -tuln | grep :5000
```
**Fix Connection:**
```bash
# Ensure server and client match:
# Server: --port 5000
# Client: --ws-server ws://localhost:5000/ws
```
### 2. Registration Failed
**Symptom:**
```log
ERROR - [WS] [AIP] ❌ Failed to register as device_windows_001
RuntimeError: Registration failed for device_windows_001
```
**Root Causes:**
| Cause | Explanation | Solution |
|-------|-------------|----------|
| Duplicate client ID | Another device using same ID | Use unique `--client-id` |
| Server rejecting connection | Server validation error | Check server logs for details |
| Network interruption | Connection dropped during registration | Retry connection |
| Device info collection error | Failed to gather system info | Check MCP server initialization |
**Solutions:**
**Check Duplicate IDs:**
```bash
# List all connected clients
curl http://localhost:5000/api/clients | grep client_id
# If your ID appears, choose a different one
python -m ufo.client.client --ws --client-id NEW_UNIQUE_ID
```
**Check Server Logs:**
```bash
# Server logs show detailed rejection reasons
# Example: "Client ID already exists"
# Example: "Platform mismatch"
```
### 3. Platform Detection Issues
**Symptom:**
```log
WARNING - Platform not detected correctly
WARNING - Defaulting to platform: unknown
```
**Solution:**
Explicitly set the platform:
```bash
python -m ufo.client.client \
--ws \
--ws-server ws://127.0.0.1:5000/ws \
--client-id my_device \
--platform windows # or 'linux'
```
**Platform Values:**
| Value | OS | Auto-Detection |
|-------|----|-|
| `windows` | Windows 10/11, Server 2016+ | `platform.system() == "Windows"` |
| `linux` | Ubuntu, Debian, RHEL, etc. | `platform.system() == "Linux"` |
### 4. Heartbeat Timeout
**Symptom:**
```log
ERROR - [WS] Connection closed: ConnectionClosedError
INFO - [WS] Reconnecting... (attempt 2/5)
```
**Root Causes:**
| Cause | Description | Solution |
|-------|-------------|----------|
| Network instability | Wi-Fi dropouts, packet loss | Use wired connection |
| Server crashed | Server process terminated | Restart server |
| Proxy interference | Corporate proxy blocking WebSocket | Configure proxy bypass |
| Firewall timeout | Idle connection timeout | Reduce heartbeat interval |
**Solutions:**
**Increase Retries:**
```bash
# For unreliable networks
python -m ufo.client.client \
--ws \
--ws-server ws://server:5000/ws \
--client-id my_device \
--max-retries 20
```
**Check Network:**
```bash
# Test sustained connection
ping -t server # Windows
ping server # Linux (Ctrl+C to stop)
```
**Verify Server:**
```bash
# Check if server is still running
curl http://server:5000/api/health
```
---
## 🌐 Multiple Devices
Connect multiple devices to the same server for **fleet management** and **task distribution**.
### Example Configuration
**Device 1 (Windows Desktop):**
```bash
python -m ufo.client.client \
--ws \
--ws-server ws://192.168.1.100:5000/ws \
--client-id device_windows_desktop_001
```
**Device 2 (Linux Server):**
```bash
python -m ufo.client.client \
--ws \
--ws-server ws://192.168.1.100:5000/ws \
--client-id device_linux_server_001 \
--platform linux
```
**Device 3 (Windows Laptop):**
```bash
python -m ufo.client.client \
--ws \
--ws-server ws://192.168.1.100:5000/ws \
--client-id device_windows_laptop_002
```
### Verify All Connected
```bash
curl http://192.168.1.100:5000/api/clients
```
**Expected Response:**
```json
{
"clients": [
{
"client_id": "device_windows_desktop_001",
"type": "device",
"platform": "windows",
"uptime_seconds": 120
},
{
"client_id": "device_linux_server_001",
"type": "device",
"platform": "linux",
"uptime_seconds": 95
},
{
"client_id": "device_windows_laptop_002",
"type": "device",
"platform": "windows",
"uptime_seconds": 45
}
],
"total": 3
}
```
**Client ID Naming Convention:**
```
device____
Examples:
- device_windows_prod_datacenter1_001
- device_linux_staging_cloud_aws_002
- device_windows_dev_office_laptop_john
```
---
## 🔧 Running as Background Service
!!! tip "Production Deployment"
For production use, run the client as a **system service** that starts automatically and restarts on failure.
### Linux (systemd)
Create `/etc/systemd/system/ufo-client.service`:
```ini
[Unit]
Description=UFO Device Client - Execution Agent
Documentation=https://github.com/microsoft/UFO
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=ufouser
Group=ufouser
WorkingDirectory=/home/ufouser/UFO2
# Environment variables (if needed)
Environment="PYTHONUNBUFFERED=1"
# Main command
ExecStart=/usr/bin/python3 -m ufo.client.client \
--ws \
--ws-server ws://192.168.1.100:5000/ws \
--client-id device_linux_prod_01 \
--platform linux \
--log-level INFO
# Restart policy
Restart=always
RestartSec=10
StartLimitBurst=5
StartLimitIntervalSec=300
# Resource limits (optional)
LimitNOFILE=65536
MemoryLimit=2G
# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=ufo-client
[Install]
WantedBy=multi-user.target
```
**Enable and Start:**
```bash
# Reload systemd configuration
sudo systemctl daemon-reload
# Enable service (start on boot)
sudo systemctl enable ufo-client
# Start service now
sudo systemctl start ufo-client
# Check status
sudo systemctl status ufo-client
# View logs
sudo journalctl -u ufo-client -f
```
**Service Management:**
| Command | Purpose |
|---------|---------|
| `systemctl start ufo-client` | Start the service |
| `systemctl stop ufo-client` | Stop the service |
| `systemctl restart ufo-client` | Restart the service |
| `systemctl status ufo-client` | Check service status |
| `journalctl -u ufo-client -f` | Follow logs in real-time |
| `systemctl disable ufo-client` | Disable auto-start |
### Windows (NSSM)
**NSSM** (Non-Sucking Service Manager) wraps any application as a Windows service.
**1. Download NSSM:**
Download from [nssm.cc](https://nssm.cc/download)
**2. Install Service:**
```powershell
# Install as service
nssm install UFOClient "C:\Python310\python.exe" `
"-m" "ufo.client.client" `
"--ws" `
"--ws-server" "ws://192.168.1.100:5000/ws" `
"--client-id" "device_windows_prod_01" `
"--log-level" "INFO"
# Set working directory
nssm set UFOClient AppDirectory "C:\UFO2"
# Set restart policy
nssm set UFOClient AppExit Default Restart
nssm set UFOClient AppRestartDelay 10000
# Set logging
nssm set UFOClient AppStdout "C:\UFO2\logs\client-stdout.log"
nssm set UFOClient AppStderr "C:\UFO2\logs\client-stderr.log"
```
**3. Manage Service:**
```powershell
# Start service
nssm start UFOClient
# Check status
nssm status UFOClient
# Stop service
nssm stop UFOClient
# Remove service
nssm remove UFOClient confirm
```
**Alternative: Windows Task Scheduler**
```powershell
# Create scheduled task to run on startup
$action = New-ScheduledTaskAction -Execute "python.exe" `
-Argument "-m ufo.client.client --ws --ws-server ws://server:5000/ws --client-id device_win_01"
$trigger = New-ScheduledTaskTrigger -AtStartup
$settings = New-ScheduledTaskSettingsSet -RestartCount 3 -RestartInterval (New-TimeSpan -Minutes 1)
Register-ScheduledTask -TaskName "UFOClient" `
-Action $action `
-Trigger $trigger `
-Settings $settings `
-User "System" `
-RunLevel Highest
```
### PM2 (Cross-Platform)
**PM2** is a cross-platform process manager with built-in load balancing, monitoring, and auto-restart.
**1. Install PM2:**
```bash
npm install pm2
```
**2. Create Ecosystem File (`ecosystem.config.js`):**
```javascript
module.exports = {
apps: [{
name: "ufo-client",
script: "python",
args: [
"-m", "ufo.client.client",
"--ws",
"--ws-server", "ws://192.168.1.100:5000/ws",
"--client-id", "device_prod_01",
"--log-level", "INFO"
],
cwd: "/home/user/UFO2",
interpreter: "none",
autorestart: true,
watch: false,
max_restarts: 10,
min_uptime: "10s",
restart_delay: 5000,
env: {
PYTHONUNBUFFERED: "1"
}
}]
};
```
**3. Start with PM2:**
```bash
# Start from ecosystem file
pm2 start ecosystem.config.js
# Or start directly
pm2 start "python -m ufo.client.client --ws --ws-server ws://192.168.1.100:5000/ws --client-id device_001" \
--name ufo-client
# Save PM2 configuration
pm2 save
# Enable startup script (auto-start on boot)
pm2 startup
# Follow the instructions printed by the command
# Monitor
pm2 monit
# View logs
pm2 logs ufo-client
```
**PM2 Management:**
| Command | Purpose |
|---------|---------|
| `pm2 list` | List all processes |
| `pm2 start ufo-client` | Start process |
| `pm2 stop ufo-client` | Stop process |
| `pm2 restart ufo-client` | Restart process |
| `pm2 delete ufo-client` | Remove process |
| `pm2 logs ufo-client` | View logs |
| `pm2 monit` | Real-time monitoring dashboard |
---
## 🏭 Production Deployment Best Practices
Follow these best practices for reliable production deployments.
### 1. Descriptive Client IDs
```bash
# ❌ Bad: Generic, non-unique
--client-id client_001
--client-id device1
# ✅ Good: Descriptive, environment, location
--client-id production_windows_datacenter1_rack3_slot1
--client-id staging_linux_cloud_aws_us-east-1_worker2
--client-id dev_windows_office_john_laptop
```
**ID Structure:**
```
___
- environment: production, staging, development, test
- platform: windows, linux
- location: datacenter1, cloud_aws, office
- identifier: unique number or name
```
### 2. Structured Logging
**File Logging:**
```bash
# Redirect to log file with rotation
python -m ufo.client.client \
--ws \
--ws-server ws://server:5000/ws \
--client-id device_prod_01 \
--log-level INFO \
> /var/log/ufo-client.log 2>&1
```
**Systemd Journal:**
```bash
# Already configured in systemd service
# View logs:
journalctl -u ufo-client -f --since "1 hour ago"
```
**Syslog:**
```bash
# Configure Python logging to send to syslog
# Add to config_dev.yaml:
# logging:
# handlers:
# syslog:
# class: logging.handlers.SysLogHandler
# address: /dev/log
```
### 3. Automatic Restart on Failure
**Service Configuration:**
| Platform | Mechanism | Restart Delay | Max Restarts |
|----------|-----------|---------------|--------------|
| Linux | systemd | 10 seconds | Unlimited (with rate limiting) |
| Windows | NSSM | 10 seconds | Unlimited |
| Cross-platform | PM2 | 5 seconds | 10 attempts, then manual |
### 4. Health Monitoring
**Monitoring Script:**
```bash
#!/bin/bash
# check-ufo-client.sh
CLIENT_ID="device_prod_01"
SERVER_URL="http://192.168.1.100:5000"
# Check if client is connected
response=$(curl -s "${SERVER_URL}/api/clients" | grep -c "${CLIENT_ID}")
if [ "$response" -eq "0" ]; then
echo "ALERT: Client ${CLIENT_ID} is not connected!"
# Send alert (email, Slack, PagerDuty, etc.)
exit 1
else
echo "OK: Client ${CLIENT_ID} is connected"
exit 0
fi
```
**Run via cron:**
```cron
# Check every 5 minutes
*/5 * * * * /usr/local/bin/check-ufo-client.sh
```
### 5. Secure Communication
!!! danger "Production Security"
**Never expose clients to the internet without these security measures:**
**Use WSS (WebSocket Secure):**
```bash
# Production: Encrypted WebSocket
--ws-server wss://ufo-server.company.com/ws
# Development only: Unencrypted
--ws-server ws://localhost:5000/ws
```
**Server-Side TLS Configuration:**
```bash
# Server with TLS
python -m ufo.server.app \
--port 5000 \
--ssl-cert /path/to/cert.pem \
--ssl-key /path/to/key.pem
```
**Network Security:**
| Measure | Implementation |
|---------|----------------|
| **Firewall Rules** | Allow only server IP on port 5000 |
| **VPN/Private Network** | Run server on internal network only |
| **Authentication** | Implement client authentication (future feature) |
| **Certificate Validation** | Verify server TLS certificates |
---
## 🔧 Troubleshooting Commands
Use these commands to diagnose connection and execution issues.
### Test Server Connectivity
**HTTP Health Check:**
```bash
curl http://localhost:5000/api/health
# Expected response:
# {"status": "healthy", "uptime_seconds": 3456}
```
**WebSocket Test:**
```bash
# Install wscat
npm install wscat
# Test WebSocket connection
wscat -c ws://localhost:5000/ws
# You should see connection established
# Send a test message (will likely be rejected, but connection works)
```
**Network Connectivity:**
```bash
# Test if server is reachable
ping 192.168.1.100
# Test if port is open
telnet 192.168.1.100 5000 # Windows/Linux
nc -zv 192.168.1.100 5000 # Linux/macOS
```
### Check Connected Clients
```bash
# List all connected clients
curl http://localhost:5000/api/clients | python -m json.tool
# Check specific client
curl http://localhost:5000/api/clients | grep "device_windows_001"
```
### Monitor Client Logs
**Increase Verbosity:**
```bash
# Enable DEBUG logging
python -m ufo.client.client \
--ws \
--client-id my_device \
--log-level DEBUG
```
**Filter Logs:**
```bash
# Only show errors
python -m ufo.client.client --ws --client-id my_device 2>&1 | grep ERROR
# Only show connection events
python -m ufo.client.client --ws --client-id my_device 2>&1 | grep -E "Connect|Register"
```
### Test Task Dispatch
```bash
# Dispatch simple test task
curl -X POST http://localhost:5000/api/dispatch \
-H "Content-Type: application/json" \
-d '{
"client_id": "device_windows_001",
"request": "List all files in the current directory"
}'
```
---
## 🚀 Next Steps
!!! tip "Continue Learning"
Now that your client is connected and running tasks:
**1. Understand Registration Flow**
Learn how clients register with the server and exchange device profiles:
👉 [UFO Client Overview](./overview.md)
**2. Explore Device Information**
Deep dive into what device information is collected and how it's used for task assignment:
👉 [Device Info Provider](./device_info.md)
**3. Master WebSocket Communication**
Understand the AIP protocol and WebSocket message flow:
👉 [WebSocket Client](./websocket_client.md)
**4. Configure MCP Servers**
Learn how to add custom tools and configure MCP servers:
👉 [MCP Integration](../mcp/overview.md)
**5. Study the AIP Protocol**
Deep dive into message types, flow control, and error handling:
👉 [AIP Protocol](../aip/overview.md)
**6. Production Deployment**
Best practices for running clients in production environments:
👉 [Configuration Guide](../configuration/system/overview.md)
---
### Documents Docs Client Ufo Client (`documents/docs/client/ufo_client.md`)
# 🎯 UFO Client
The **UFO Client** is the execution engine that receives commands from the server, routes them to appropriate tools via the CommandRouter, and aggregates results. It focuses on stateless command execution, delegating all decision-making to the server.
## 📋 Overview
The UFO Client bridges network communication and local tool execution.
**Key Capabilities:**
| Capability | Description | Implementation |
|------------|-------------|----------------|
| **Command Execution** | Processes server commands deterministically | `execute_step()`, `execute_actions()` |
| **Session Management** | Tracks session state and metadata | Session ID, agent/process/root names |
| **Result Aggregation** | Collects and structures tool execution results | Returns `List[Result]` |
| **Thread Safety** | Ensures safe concurrent execution | `asyncio.Lock` (`task_lock`) |
| **State Management** | Maintains agent, process, and root names | Property setters with validation |
| **Manager Coordination** | Orchestrates ComputerManager and MCPServerManager | `reset()` cascades to all managers |
The UFO Client follows a stateless execution philosophy:
- Executes commands sent by the server
- Routes commands to the appropriate tools
- Returns execution results
- Does **not** decide which commands to run
- Does **not** interpret user requests
- Does **not** store long-term state
**Architectural Position:**
```mermaid
graph LR
subgraph Server["Server Side (Orchestration)"]
SRV[Agent Server]
LLM[LLM Reasoning]
end
subgraph Network["Network Layer"]
WSC[WebSocket Client]
end
subgraph Client["Client Side (Execution)"]
UFC[UFO Client]
CR[Command Router]
Tools[MCP Tools]
end
SRV -->|Commands| WSC
WSC -->|execute_step| UFC
UFC -->|execute| CR
CR -->|tool calls| Tools
Tools -->|results| CR
CR -->|results| UFC
UFC -->|results| WSC
WSC -->|results| SRV
LLM -->|planning| SRV
style SRV fill:#ffe0b2
style UFC fill:#bbdefb
style Tools fill:#c8e6c9
```
## 🏗️ Architecture
The UFO Client has a minimal API surface—just initialization, execution, and reset.
### Component Structure
```mermaid
graph TB
subgraph "UFOClient"
State[Session State]
Execution[Execution Methods]
Dependencies[Manager Dependencies]
end
subgraph "Session State"
State1[session_id]
State2[agent_name]
State3[process_name]
State4[root_name]
State5[task_lock]
end
subgraph "Execution Methods"
Exec1[execute_step]
Exec2[execute_actions]
Exec3[reset]
end
subgraph "Dependencies"
Dep1[CommandRouter]
Dep2[ComputerManager]
Dep3[MCPServerManager]
end
State --> State1
State --> State2
State --> State3
State --> State4
State --> State5
Execution --> Exec1
Execution --> Exec2
Execution --> Exec3
Dependencies --> Dep1
Dependencies --> Dep2
Dependencies --> Dep3
Exec1 --> Exec2
Exec2 --> Dep1
Exec3 --> Dep2
Exec3 --> Dep3
style State fill:#e3f2fd
style Execution fill:#f1f8e9
style Dependencies fill:#fff3e0
```
**Class Attributes:**
| Attribute | Type | Purpose |
|-----------|------|---------|
| `mcp_server_manager` | `MCPServerManager` | Manages MCP server lifecycle |
| `computer_manager` | `ComputerManager` | Manages computer instances (tool namespaces) |
| `command_router` | `CommandRouter` | Routes commands to appropriate computers |
| `task_lock` | `asyncio.Lock` | Ensures thread-safe execution |
| `client_id` | `str` | Unique identifier for this client (default: `"client_001"`) |
| `platform` | `str` | Platform type (`"windows"` or `"linux"`) - auto-detected if not provided |
| `session_id` | `Optional[str]` | Current session identifier |
| `agent_name` | `Optional[str]` | Active agent (e.g., `"HostAgent"`, `"AppAgent"`) |
| `process_name` | `Optional[str]` | Process context (e.g., `"notepad.exe"`) |
| `root_name` | `Optional[str]` | Root operation name |
## 🚀 Initialization
Creating a UFO Client requires two manager instances: MCPServerManager and ComputerManager.
```python
from ufo.client.ufo_client import UFOClient
from ufo.client.computer import ComputerManager
from ufo.client.mcp.mcp_server_manager import MCPServerManager
# 1. Initialize MCP Server Manager
mcp_server_manager = MCPServerManager()
mcp_server_manager.create_servers_from_config() # Load from config_dev.yaml
# 2. Initialize Computer Manager
computer_manager = ComputerManager(
ufo_config.to_dict(),
mcp_server_manager
)
# 3. Create UFO Client
client = UFOClient(
mcp_server_manager=mcp_server_manager,
computer_manager=computer_manager,
client_id="device_windows_001",
platform="windows"
)
```
**Constructor Parameters:**
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `mcp_server_manager` | `MCPServerManager` | ✅ Yes | - | MCP server lifecycle manager |
| `computer_manager` | `ComputerManager` | ✅ Yes | - | Computer instance manager |
| `client_id` | `str` | No | `"client_001"` | Unique client identifier |
| `platform` | `str` | No | Auto-detected | Platform type: `"windows"` or `"linux"` |
**Initialization Side Effects:**
1. Creates `CommandRouter` instance (delegates to ComputerManager)
2. Initializes `task_lock` (`asyncio.Lock()`)
3. Sets session state to `None` (session_id, agent_name, process_name, root_name)
## 📊 Session State Management
The UFO Client maintains contextual metadata for the current execution session.
### Session ID
**Purpose:** Unique identifier for the current task session
```python
# Set session ID (typically set by server)
client.session_id = "session_20251104_143022_abc123"
# Get session ID
current_session = client.session_id # "session_20251104_143022_abc123"
# Clear session ID
client.reset() # Sets session_id to None
```
**Validation:**
```python
# ✅ Valid
client.session_id = "session_123"
client.session_id = None
# ❌ Invalid - raises ValueError
client.session_id = 12345 # Not a string
```
### Agent Name
**Purpose:** Identifies the active agent (HostAgent, AppAgent, etc.)
```python
# Set agent name (from server message)
client.agent_name = "HostAgent"
# Get agent name
agent = client.agent_name # "HostAgent"
```
**Common Agent Names:**
| Agent Name | Purpose |
|------------|---------|
| `HostAgent` | OS-level operations (start apps, manage files) |
| `AppAgent` | Application-specific operations (UI automation) |
| `FollowerAgent` | Follow predefined workflows |
### Process Name
**Purpose:** Identifies the process context
```python
# Set process name (from server message)
client.process_name = "notepad.exe"
# Get process name
process = client.process_name # "notepad.exe"
```
**Usage:** Helps route commands to the correct application context
### Root Name
**Purpose:** Identifies the root operation name
```python
# Set root name (from server message)
client.root_name = "open_application"
# Get root name
root = client.root_name # "open_application"
```
**Property Validation:**
All properties validate their inputs:
```python
try:
client.agent_name = 123 # Not a string
except ValueError as e:
print(e) # "Agent name must be a string or None."
```
**Validation Table:**
| Property | Valid Types | Raises on Invalid |
|----------|-------------|-------------------|
| `session_id` | `str`, `None` | `ValueError` |
| `agent_name` | `str`, `None` | `ValueError` |
| `process_name` | `str`, `None` | `ValueError` |
| `root_name` | `str`, `None` | `ValueError` |
## ⚙️ Command Execution
### Execute Step (Main Entry Point)
`execute_step()` processes one complete server message, extracting metadata and executing all commands.
**Signature:**
```python
async def execute_step(self, response: ServerMessage) -> List[Result]:
"""
Perform a single step execution.
:param response: The ServerMessage instance to process.
:return: A list of Result instances.
"""
```
**Execution Flow:**
```mermaid
sequenceDiagram
participant WSC as WebSocket Client
participant UFC as UFO Client
participant CR as Command Router
participant Tools
WSC->>UFC: execute_step(ServerMessage)
Note over UFC: 1. Extract Metadata
UFC->>UFC: self.agent_name = response.agent_name
UFC->>UFC: self.process_name = response.process_name
UFC->>UFC: self.root_name = response.root_name
Note over UFC: 2. Execute Actions
UFC->>UFC: execute_actions(response.actions)
UFC->>CR: command_router.execute( agent_name, process_name, root_name, commands)
CR->>Tools: Route commands to tools
Tools-->>CR: Results
CR-->>UFC: List[Result]
UFC-->>WSC: List[Result]
```
**Implementation:**
```python
async def execute_step(self, response: ServerMessage) -> List[Result]:
"""Perform a single step execution."""
# Extract metadata from server response
self.agent_name = response.agent_name
self.process_name = response.process_name
self.root_name = response.root_name
# Execute actions
action_results = await self.execute_actions(response.actions)
return action_results
```
**Example Usage:**
```python
from aip.messages import ServerMessage
# Receive server message
server_response = ServerMessage.model_validate_json(msg)
# Execute step
action_results = await client.execute_step(server_response)
# action_results is List[Result]
for result in action_results:
print(f"Action: {result.action}, Status: {result.status}")
```
### Execute Actions
`execute_actions()` executes a list of commands via the CommandRouter.
**Signature:**
```python
async def execute_actions(self, commands: Optional[List[Command]]) -> List[Result]:
"""
Execute the actions provided by the server.
:param commands: List of actions to execute.
:returns: Results of the executed actions.
"""
```
**Implementation:**
```python
async def execute_actions(self, commands: Optional[List[Command]]) -> List[Result]:
"""Execute the actions provided by the server."""
action_results = []
if commands:
self.logger.info(f"Executing {len(commands)} actions in total")
# Delegate to CommandRouter
action_results = await self.command_router.execute(
agent_name=self.agent_name,
process_name=self.process_name,
root_name=self.root_name,
commands=commands
)
return action_results
```
**Example:**
```python
from aip.messages import Command
commands = [
Command(
action="click",
parameters={"control_label": "Start", "x": 10, "y": 10}
),
Command(
action="type_text",
parameters={"text": "notepad"}
),
Command(
action="press_key",
parameters={"key": "enter"}
)
]
# Execute all commands
results = await client.execute_actions(commands)
# results contains Result object for each command
```
**Command Execution Table:**
| Step | Action | Component |
|------|--------|-----------|
| 1 | Receive commands | UFO Client |
| 2 | Log command count | UFO Client |
| 3 | Call CommandRouter | UFO Client |
| 4 | Route to Computer | CommandRouter |
| 5 | Execute via MCP | Computer |
| 6 | Collect results | CommandRouter |
| 7 | Return results | UFO Client |
See [Computer Manager](./computer_manager.md) for command routing details.
## 🔄 State Reset
!!!warning "Critical for Multi-Task Execution"
Always reset state between tasks to prevent data leakage between sessions.
**Signature:**
```python
def reset(self):
"""Reset session state and dependent managers."""
```
**Implementation:**
```python
def reset(self):
"""Reset session state and dependent managers."""
# Clear session state
self._session_id = None
self._agent_name = None
self._process_name = None
self._root_name = None
# Reset managers
self.computer_manager.reset()
self.mcp_server_manager.reset()
self.logger.info("Client state has been reset.")
```
**Reset Cascade:**
```mermaid
graph TD
Reset[client.reset]
Reset --> S1[session_id = None]
Reset --> S2[agent_name = None]
Reset --> S3[process_name = None]
Reset --> S4[root_name = None]
Reset --> M1[computer_manager.reset]
Reset --> M2[mcp_server_manager.reset]
M1 --> C1[Clear computer instances]
M2 --> M3[Reset MCP servers]
style Reset fill:#ffcdd2
style M1 fill:#fff9c4
style M2 fill:#fff9c4
```
**When to Reset:**
| Scenario | Why Reset |
|----------|-----------|
| **Before starting new task** | Clear previous task state |
| **On task completion** | Prepare for next task |
| **On task failure** | Clean up failed state |
| **On server disconnection** | Reset to known good state |
**Note:** The WebSocket client automatically calls `reset()` before starting new tasks:
```python
async with self.ufo_client.task_lock:
self.ufo_client.reset() # Automatic
await self.task_protocol.send_task_request(...)
```
## 🔒 Thread Safety
The UFO Client uses `asyncio.Lock` to prevent concurrent state modifications.
**Lock Implementation:**
```python
# In UFOClient.__init__
self.task_lock = asyncio.Lock()
```
**Usage in WebSocket Client:**
```python
# In WebSocket client
async with client.task_lock:
client.reset()
await client.execute_step(server_response)
```
**Protected Operations:**
| Operation | Protected By | Reason |
|-----------|--------------|--------|
| Session state modifications | `task_lock` | Prevent race conditions |
| Command execution | `task_lock` | Ensure one task at a time |
| State reset | `task_lock` | Atomic reset operation |
!!!warning "Single Task Execution"
The lock ensures only **one task executes at a time**. Attempting concurrent execution will block until the lock is released.
## 📋 Complete Execution Pipeline
```mermaid
sequenceDiagram
participant Server
participant WSC as WebSocket Client
participant UFC as UFO Client
participant CR as Command Router
participant CM as Computer Manager
participant Comp as Computer
participant Tool as MCP Tool
Note over Server,Tool: Full Execution Pipeline
Server->>WSC: COMMAND message
WSC->>UFC: execute_step(ServerMessage)
Note over UFC: Extract Metadata
UFC->>UFC: agent_name = "HostAgent"
UFC->>UFC: process_name = "explorer.exe"
UFC->>UFC: root_name = "navigate"
Note over UFC: Execute Actions
UFC->>CR: execute(agent, process, root, commands)
CR->>CM: Route commands
CM->>Comp: Get computer instance
Comp->>Tool: Execute tool
Tool-->>Comp: Result
Comp-->>CM: Result
CM-->>CR: List[Result]
CR-->>UFC: List[Result]
UFC-->>WSC: List[Result]
WSC->>Server: COMMAND_RESULTS (via AIP)
```
## ⚠️ Error Handling
### Command Execution Errors
Individual command failures are captured in `Result` objects, not thrown as exceptions.
**Error Result Structure:**
```python
from aip.messages import Result, ResultStatus
error_result = Result(
action="click",
status=ResultStatus.ERROR,
error_message="Control not found",
observation="Failed to locate control with label 'Start'"
)
```
**Handling Execution Errors:**
```python
try:
results = await client.execute_actions(commands)
# Check each result
for result in results:
if result.status == ResultStatus.ERROR:
logger.error(f"Action {result.action} failed: {result.error_message}")
else:
logger.info(f"Action {result.action} succeeded")
except Exception as e:
# Unexpected error (not tool failure)
logger.error(f"Command execution failed: {e}", exc_info=True)
```
### Property Validation Errors
```python
try:
client.session_id = 12345 # Invalid type
except ValueError as e:
logger.error(f"Invalid session ID: {e}")
# ValueError: Session ID must be a string or None.
```
**Error Handling Table:**
| Error Type | Raised By | Handling |
|------------|-----------|----------|
| Tool execution error | MCP tools | Captured in `Result.error_message` |
| Property validation error | Property setters | `ValueError` exception |
| Unexpected errors | Any component | Logged, may propagate |
## 📝 Logging
The UFO Client logs all major events for debugging and monitoring.
**Log Examples:**
**Initialization:**
```log
INFO - UFO Client initialized for platform: windows
```
**Session State Changes:**
```log
INFO - Session ID set to: session_20251104_143022_abc123
INFO - Agent name set to: HostAgent
INFO - Process name set to: notepad.exe
INFO - Root name set to: open_application
```
**Execution:**
```log
INFO - Executing 5 actions in total
```
**Reset:**
```log
INFO - Client state has been reset.
```
**Log Level Recommendations:**
| Environment | Level | Rationale |
|-------------|-------|-----------|
| Development | `DEBUG` | See all operations |
| Staging | `INFO` | Track execution flow |
| Production | `INFO` | Monitor without spam |
| Troubleshooting | `DEBUG` | Diagnose issues |
## 💡 Usage Example
### Complete Workflow
This example shows how to use the UFO Client in a typical workflow.
```python
import asyncio
from ufo.client.ufo_client import UFOClient
from aip.messages import ServerMessage, Command, ServerMessageType, TaskStatus
async def main():
# 1. Initialize client
client = UFOClient(
mcp_server_manager=mcp_manager,
computer_manager=computer_manager,
client_id="device_windows_001",
platform="windows"
)
# 2. Simulate server message
server_msg = ServerMessage(
type=ServerMessageType.COMMAND,
session_id="session_123",
response_id="resp_456",
agent_name="HostAgent",
process_name="explorer.exe",
root_name="navigate_folder",
actions=[
Command(action="click", parameters={"label": "File"}),
Command(action="click", parameters={"label": "New Folder"})
],
status=TaskStatus.PROCESSING
)
# 3. Execute step
async with client.task_lock: # Thread-safe execution
results = await client.execute_step(server_msg)
# 4. Process results
for result in results:
print(f"Action: {result.action}")
print(f"Status: {result.status}")
print(f"Observation: {result.observation}")
if result.status == ResultStatus.ERROR:
print(f"Error: {result.error_message}")
# 5. Reset for next task
client.reset()
asyncio.run(main())
```
## ✅ Best Practices
### Development Best Practices
**1. Always Reset Between Tasks**
```python
async with client.task_lock:
client.reset() # Clear previous state
await client.execute_step(new_server_response)
```
**2. Use Property Setters (Not Direct Assignment)**
```python
# ✅ Good - validates input
client.session_id = "session_123"
# ❌ Bad - bypasses validation
client._session_id = "session_123"
```
**3. Log Execution Progress**
```python
self.logger.info(f"Executing {len(commands)} actions for {self.agent_name}")
```
**4. Handle Errors Gracefully**
```python
try:
results = await client.execute_actions(commands)
except Exception as e:
self.logger.error(f"Execution failed: {e}", exc_info=True)
# Error is also captured in results
```
### Production Best Practices
**1. Use Thread Locks Consistently**
```python
# Always use task_lock for state operations
async with client.task_lock:
client.reset()
results = await client.execute_step(msg)
```
**2. Monitor Execution Times**
```python
import time
start = time.time()
results = await client.execute_actions(commands)
duration = time.time() - start
if duration > 60: # Alert if > 1 minute
logger.warning(f"Slow execution: {duration}s for {len(commands)} commands")
```
**3. Validate Results**
```python
# Check for failures
failed_actions = [r for r in results if r.status == ResultStatus.ERROR]
if failed_actions:
logger.error(f"{len(failed_actions)} actions failed")
# Report to monitoring system
```
## 🔗 Integration Points
### WebSocket Client Integration
The WebSocket client uses UFO Client for all command execution.
**Integration:**
```python
# In WebSocket client
action_results = await self.ufo_client.execute_step(server_response)
```
See [WebSocket Client](./websocket_client.md) for communication details.
### Command Router Integration
The UFO Client delegates all execution to the CommandRouter.
**Integration:**
```python
action_results = await self.command_router.execute(
agent_name=self.agent_name,
process_name=self.process_name,
root_name=self.root_name,
commands=commands
)
```
See [Computer Manager](./computer_manager.md) for routing details.
### Computer Manager Integration
The Computer Manager maintains computer instances for tool execution.
**Integration:**
```python
# Reset cascades to computer manager
self.computer_manager.reset()
```
See [Computer Manager](./computer_manager.md) for management details.
### MCP Server Manager Integration
The MCP Server Manager handles MCP server creation and cleanup.
**Integration:**
```python
# Reset cascades to MCP server manager
self.mcp_server_manager.reset()
```
See [MCP Integration](./mcp_integration.md) for MCP details.
## 🚀 Next Steps
**Continue Learning**
1. **Understand Network Communication** - Learn how the WebSocket client uses UFO Client: [WebSocket Client](./websocket_client.md)
2. **Explore Command Routing** - See how commands are routed to the right tools: [Computer Manager](./computer_manager.md)
3. **Study Device Profiling** - Understand device information collection: [Device Info Provider](./device_info.md)
4. **Learn About MCP Integration** - Deep dive into MCP server management: [MCP Integration](./mcp_integration.md)
5. **Master AIP Messages** - Understand message structures: [AIP Messages](../aip/messages.md)
---
### Documents Docs Client Websocket Client (`documents/docs/client/websocket_client.md`)
# 🔌 WebSocket Client
The **WebSocket Client** implements the **AIP (Agent Interaction Protocol)** for reliable, bidirectional communication between device clients and the Agent Server. It provides the low-level communication infrastructure for UFO device clients.
## 📋 Overview
The WebSocket client handles all network communication aspects, allowing the UFO Client to focus on task execution.
**Key Responsibilities:**
| Capability | Description | Implementation |
|------------|-------------|----------------|
| **Connection Management** | Persistent WebSocket connection with automatic retry | Exponential backoff, configurable max retries |
| **AIP Protocol Implementation** | Structured message handling via Registration, Heartbeat, Task Execution | Three protocol handlers |
| **Device Registration** | Automatic registration with device profile on connect | Push model (proactive info collection) |
| **Heartbeat Monitoring** | Regular keepalive messages for connection health | Configurable interval (default: 30s) |
| **Message Routing** | Dispatch incoming messages to appropriate handlers | Type-based routing |
| **Error Handling** | Graceful error recovery and reporting | Retry logic, error propagation via AIP |
**Message Flow Overview:**
```mermaid
graph LR
subgraph "Client Side"
WSC[WebSocket Client]
AIP[AIP Protocols]
UFC[UFO Client]
end
subgraph "Network"
WS[WebSocket Connection]
end
subgraph "Server Side"
Server[Agent Server]
end
WSC <-->|AIP Messages| AIP
AIP <-->|WebSocket| WS
WS <-->|TCP/IP| Server
WSC -->|Delegate Execution| UFC
style WSC fill:#bbdefb
style AIP fill:#c8e6c9
style Server fill:#ffe0b2
```
## 🏗️ Architecture
The WebSocket client is organized into distinct layers for connection management, protocol handling, and message routing.
### Component Structure
```mermaid
graph TB
subgraph "UFOWebSocketClient"
CM[Connection Management Layer]
PH[Protocol Handler Layer]
MR[Message Routing Layer]
end
subgraph "Connection Management"
CM1[connect_and_listen]
CM2[Retry Logic]
CM3[State Tracking]
end
subgraph "AIP Protocols"
PH1[RegistrationProtocol]
PH2[HeartbeatProtocol]
PH3[TaskExecutionProtocol]
end
subgraph "Message Handlers"
MR1[recv_loop]
MR2[handle_message]
MR3[handle_commands]
MR4[handle_task_end]
end
CM --> CM1
CM --> CM2
CM --> CM3
PH --> PH1
PH --> PH2
PH --> PH3
MR --> MR1
MR --> MR2
MR --> MR3
MR --> MR4
CM1 --> PH
PH --> MR
style CM fill:#e3f2fd
style PH fill:#f1f8e9
style MR fill:#fff3e0
```
### Class Structure
| Component | Type | Purpose |
|-----------|------|---------|
| **UFOWebSocketClient** | Main Class | Orchestrates all WebSocket communication |
| **WebSocketTransport** | AIP Component | Low-level WebSocket send/receive |
| **RegistrationProtocol** | AIP Protocol | Client registration messages |
| **HeartbeatProtocol** | AIP Protocol | Connection keepalive messages |
| **TaskExecutionProtocol** | AIP Protocol | Task request/result messages |
---
## 🔄 Connection Lifecycle
### Initialization & Connection Flow
```mermaid
sequenceDiagram
participant Main as Client Main
participant WSC as WebSocket Client
participant WS as WebSocket
participant Server
Note over Main: 1. Initialization
Main->>WSC: Create UFOWebSocketClient(ws_url, ufo_client)
WSC->>WSC: Initialize attributes (max_retries=3, timeout=120)
Note over WSC,Server: 2. Connection Attempt
WSC->>WS: websockets.connect(ws_url)
WS->>Server: TCP Handshake
Server-->>WS: WebSocket Upgrade
WS-->>WSC: Connection Established
Note over WSC,Server: 3. AIP Protocol Initialization
WSC->>WSC: Create WebSocketTransport(ws)
WSC->>WSC: Create RegistrationProtocol(transport)
WSC->>WSC: Create HeartbeatProtocol(transport)
WSC->>WSC: Create TaskExecutionProtocol(transport)
Note over WSC,Server: 4. Device Registration
WSC->>WSC: Collect Device Info
WSC->>Server: REGISTRATION (via AIP)
Server-->>WSC: REGISTRATION_ACK
WSC->>WSC: Set connected_event
Note over WSC,Server: 5. Message Handling
par Receive Loop
loop Continuous
Server->>WSC: Server Messages
WSC->>WSC: Route to Handlers
end
and Heartbeat Loop
loop Every 30s
WSC->>Server: HEARTBEAT
Server-->>WSC: HEARTBEAT_ACK
end
end
```
### Initialization Code
Creating a WebSocket client:
```python
from ufo.client.websocket import UFOWebSocketClient
from ufo.client.ufo_client import UFOClient
# Create UFO client (execution engine)
ufo_client = UFOClient(
mcp_server_manager=mcp_manager,
computer_manager=computer_manager,
client_id="device_windows_001",
platform="windows"
)
# Create WebSocket client (communication layer)
ws_client = UFOWebSocketClient(
ws_url="ws://localhost:5000/ws",
ufo_client=ufo_client,
max_retries=3, # Default: 3 attempts
timeout=120 # Heartbeat interval in seconds (default: 120)
)
# Connect and start listening (blocking call)
await ws_client.connect_and_listen()
```
**Constructor Parameters:**
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `ws_url` | `str` | Required | WebSocket server URL (e.g., `ws://localhost:5000/ws`) |
| `ufo_client` | `UFOClient` | Required | UFO client instance for command execution |
| `max_retries` | `int` | `3` | Maximum connection retry attempts |
| `timeout` | `float` | `120` | Heartbeat interval in seconds (passed to `heartbeat_loop()`) |
**Note:** The `timeout` parameter is passed to `heartbeat_loop(interval)` to control heartbeat frequency. While `heartbeat_loop()` has a default of 30s in its signature, the client constructor uses 120s which is passed when calling the method.
### Connection Establishment Details
The client uses specific WebSocket parameters optimized for long-running task execution:
**WebSocket Connection Parameters:**
```python
async with websockets.connect(
self.ws_url,
ping_interval=20, # Send WebSocket ping every 20 seconds
ping_timeout=180, # Wait up to 3 minutes for pong response
close_timeout=10, # 10 second close handshake timeout
max_size=100 * 1024 * 1024 # 100MB max message size
) as ws:
# Connection established
```
**Parameter Rationale:**
| Parameter | Value | Reason |
|-----------|-------|--------|
| `ping_interval` | **20 seconds** | Frequent keepalive to detect connection loss quickly |
| `ping_timeout` | **180 seconds** | Tolerates long-running operations (e.g., complex tasks) |
| `close_timeout` | **10 seconds** | Quick cleanup on intentional disconnect |
| `max_size` | **100 MB** | Supports large screenshots, logs, file transfers |
**Note:** The 180-second `ping_timeout` ensures the connection stays alive during lengthy tool executions (up to 100 minutes per tool).
## 📝 Registration Flow
### Device Information Collection
UFO uses a **push model** for device information: clients proactively send their profile during registration, rather than waiting for the server to request it. This reduces latency for constellation (multi-client) scenarios.
**Device Info Collection:**
```python
from ufo.client.device_info_provider import DeviceInfoProvider
# Collect comprehensive system information
system_info = DeviceInfoProvider.collect_system_info(
client_id=self.ufo_client.client_id,
custom_metadata=None # Server adds custom metadata if configured
)
# System info includes:
# - platform (windows/linux/darwin)
# - os_version
# - cpu_count
# - memory_total_gb
# - hostname
# - ip_address
# - supported_features
# - platform_type
```
**Metadata Structure:**
```python
metadata = {
"system_info": {
"platform": "windows",
"os_version": "Windows-10-10.0.19045",
"cpu_count": 8,
"memory_total_gb": 16.0,
"hostname": "DESKTOP-ABC123",
"ip_address": "192.168.1.100",
# ... additional fields
},
"registration_time": "2025-11-05T14:30:00.123Z"
}
```
See [Device Info Provider](./device_info.md) for complete field descriptions.
### Registration Message Exchange
```mermaid
sequenceDiagram
participant Client
participant AIP as AIP Registration Protocol
participant Server
Note over Client: Collect Device Info
Client->>Client: DeviceInfoProvider.collect_system_info()
Note over Client,Server: Registration Request
Client->>AIP: register_as_device( device_id, metadata, platform)
AIP->>Server: REGISTRATION {device_id, metadata, platform}
Note over Server: Validate & Store
Server->>Server: Check for duplicate ID
Server->>Server: Store device info
Server->>Server: Add to client registry
Note over Client,Server: Registration Response
Server-->>AIP: REGISTRATION_ACK {success: true}
AIP-->>Client: success = True
Client->>Client: Set connected_event
Client->>Client: Log success
```
**Registration Code:**
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Registration Outcomes
**Success Scenario:**
```log
INFO - [WS] \[AIP] Collected device info: platform=windows, cpu=8, memory=16.0GB
INFO - [WS] \[AIP] Attempting to register as device_windows_001
INFO - [WS] \[AIP] ✅ Successfully registered as device_windows_001
```
- `connected_event` is set (allows task requests)
- Client enters message handling loops
**Failure Scenario:**
```log
ERROR - [WS] \[AIP] ❌ Failed to register as device_windows_001
RuntimeError: Registration failed for device_windows_001
```
- Connection is closed
- Retry logic engages (exponential backoff)
**Common Failure Causes:**
| Cause | Server Behavior | Client Action |
|-------|----------------|---------------|
| Duplicate client ID | Reject registration | Change client ID, retry |
| Server capacity limit | Reject registration | Wait and retry later |
| Network interruption | Timeout | Automatic retry with backoff |
| Invalid platform | Reject registration | Fix platform parameter |
---
## 💓 Heartbeat Mechanism
Heartbeats prove the client is still alive and responsive, allowing the server to detect disconnected clients quickly.
### Heartbeat Loop Implementation
**Default Configuration:**
| Parameter | Value | Configurable |
|-----------|-------|--------------|
| **Interval** | 30 seconds | ✅ Yes (function parameter) |
| **Protocol** | AIP HeartbeatProtocol | No |
| **Error Handling** | Break loop on failure | No |
**Heartbeat Code:**
```python
async def heartbeat_loop(self, interval: float = 30) -> None:
"""
Send periodic heartbeat messages using AIP HeartbeatProtocol.
:param interval: Interval between heartbeats in seconds (default: 30)
"""
while True:
await asyncio.sleep(interval)
try:
await self.heartbeat_protocol.send_heartbeat(
self.ufo_client.client_id
)
self.logger.debug("[WS] \[AIP] Heartbeat sent")
except (ConnectionError, IOError) as e:
self.logger.debug(
f"[WS] \[AIP] Heartbeat failed (connection closed): {e}"
)
break # Exit loop if connection is closed
```
**Customizing Heartbeat Interval:**
Adjust the interval when calling the heartbeat loop:
```python
# In handle_messages():
await asyncio.gather(
self.recv_loop(),
self.heartbeat_loop(interval=60) # Custom 60-second interval
)
```
### Heartbeat Message Structure
**Client → Server (Heartbeat):**
```json
{
"type": "HEARTBEAT",
"client_id": "device_windows_001",
"timestamp": "2025-11-05T14:30:22.123Z"
}
```
**Server → Client (Heartbeat Ack - Optional):**
```json
{
"type": "HEARTBEAT",
"timestamp": "2025-11-05T14:30:22.456Z"
}
```
### Heartbeat State Diagram
```mermaid
stateDiagram-v2
[*] --> Sleeping
Sleeping --> SendingHeartbeat: After interval (30s)
SendingHeartbeat --> Success: Sent successfully
SendingHeartbeat --> Failed: Connection error
Success --> Sleeping: Continue loop
Failed --> [*]: Exit loop
note right of Sleeping
Wait for interval duration
(default: 30 seconds)
end note
note right of Failed
Connection closed
recv_loop will also exit
Outer retry logic activates
end note
```
---
## 📨 Message Handling
### Message Router
All incoming messages are validated against the AIP schema and routed based on their `type` field.
**Message Dispatcher Code:**
```python
async def handle_message(self, msg: str):
"""Dispatch messages based on their type."""
try:
# Parse and validate message
data = ServerMessage.model_validate_json(msg)
msg_type = data.type
self.logger.info(f"[WS] Received message: {data}")
# Route by type
if msg_type == ServerMessageType.TASK:
await self.start_task(data.user_request, data.task_name)
elif msg_type == ServerMessageType.HEARTBEAT:
self.logger.info("[WS] Heartbeat received")
elif msg_type == ServerMessageType.TASK_END:
await self.handle_task_end(data)
elif msg_type == ServerMessageType.ERROR:
self.logger.error(f"[WS] Server error: {data.error}")
elif msg_type == ServerMessageType.COMMAND:
await self.handle_commands(data)
else:
self.logger.warning(f"[WS] Unknown message type: {msg_type}")
except Exception as e:
self.logger.error(f"[WS] Error handling message: {e}", exc_info=True)
```
**Message Type Routing:**
| Server Message Type | Handler Method | Purpose |
|---------------------|----------------|---------|
| `TASK` | `start_task()` | Begin new task execution |
| `COMMAND` | `handle_commands()` | Execute specific commands |
| `TASK_END` | `handle_task_end()` | Process task completion |
| `HEARTBEAT` | Log only | Acknowledge keepalive |
| `ERROR` | Log error | Handle server-side errors |
| Unknown | Log warning | Ignore unrecognized types |
### Task Start Handler
!!!warning "Single Task Execution"
The client executes **only one task at a time**. New task requests are ignored if a task is currently running.
**Task Start Flow:**
```mermaid
sequenceDiagram
participant Server
participant WSC as WebSocket Client
participant UFC as UFO Client
participant Task as Task Coroutine
Server->>WSC: TASK message {user_request, task_name}
alt Current Task Running
WSC->>WSC: Check current_task.done()
WSC->>Server: ⚠️ Ignore (log warning)
else No Task Running
WSC->>Task: Create task_loop() coroutine
Task->>UFC: Reset session state
Task->>Task: Build metadata (platform)
Task->>Server: TASK_REQUEST (via AIP)
Server-->>Task: Acknowledgment
Task->>WSC: Task coroutine running
end
```
**Task Start Code:**
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Command Execution Handler
The server sends specific commands (tool calls) to execute, and the client returns results.
**Command Execution Flow:**
```python
async def handle_commands(self, server_response: ServerMessage):
"""
Handle commands received from server.
Uses AIP TaskExecutionProtocol to send results back.
"""
response_id = server_response.response_id
task_status = server_response.status
self.session_id = server_response.session_id
# Execute commands via UFO Client
action_results = await self.ufo_client.execute_step(server_response)
# Send results via AIP
await self.task_protocol.send_task_result(
session_id=self.session_id,
prev_response_id=response_id,
action_results=action_results,
status=task_status,
client_id=self.ufo_client.client_id
)
self.logger.info(
f"[WS] \[AIP] Sent client result for prev_response_id: {response_id}"
)
# Check for task completion
if task_status in [TaskStatus.COMPLETED, TaskStatus.FAILED]:
await self.handle_task_end(server_response)
```
**Execution Steps:**
1. **Extract Metadata**: Get `response_id`, `task_status`, `session_id`
2. **Execute Commands**: Delegate to `ufo_client.execute_step()`
3. **Send Results**: Use `TaskExecutionProtocol.send_task_result()`
4. **Check Completion**: Handle task end if status is terminal
### Task Completion Handler
```python
async def handle_task_end(self, server_response: ServerMessage):
"""Handle task end messages from server."""
if server_response.status == TaskStatus.COMPLETED:
self.logger.info(
f"[WS] Task {self.session_id} completed, result: {server_response.result}"
)
elif server_response.status == TaskStatus.FAILED:
self.logger.info(
f"[WS] Task {self.session_id} failed, with error: {server_response.error}"
)
else:
self.logger.warning(
f"[WS] Unknown task status for {self.session_id}: {server_response.status}"
)
```
---
## ⚠️ Error Handling
### Connection Error Recovery
The client automatically retries failed connections using exponential backoff to avoid overwhelming the server.
**Retry Logic:**
```python
async def connect_and_listen(self):
"""Connect with automatic retry."""
while self.retry_count < self.max_retries:
try:
async with websockets.connect(...) as ws:
# Initialize protocols
self.transport = WebSocketTransport(ws)
self.registration_protocol = RegistrationProtocol(self.transport)
self.heartbeat_protocol = HeartbeatProtocol(self.transport)
self.task_protocol = TaskExecutionProtocol(self.transport)
await self.register_client()
self.retry_count = 0 # Reset on successful connection
await self.handle_messages()
except (websockets.ConnectionClosedError, websockets.ConnectionClosedOK) as e:
self.logger.error(f"[WS] Connection closed: {e}")
self.retry_count += 1
await self._maybe_retry()
except Exception as e:
self.logger.error(f"[WS] Unexpected error: {e}", exc_info=True)
self.retry_count += 1
await self._maybe_retry()
self.logger.error("[WS] Max retries reached. Exiting.")
```
**Exponential Backoff:**
```python
async def _maybe_retry(self):
"""Exponential backoff before retry."""
if self.retry_count < self.max_retries:
wait_time = 2 ** self.retry_count # 2s, 4s, 8s, 16s...
self.logger.info(f"[WS] Retrying in {wait_time}s...")
await asyncio.sleep(wait_time)
```
**Retry Schedule:**
| Attempt | Wait Time | Cumulative Wait |
|---------|-----------|-----------------|
| 1st retry | 2 seconds | 2s |
| 2nd retry | 4 seconds | 6s |
| 3rd retry | 8 seconds | 14s |
| **Max retries reached** | Exit | - |
**Default Max Retries = 3**
Based on source code: `max_retries: int = 3` in constructor. Increase for unreliable networks:
```python
ws_client = UFOWebSocketClient(
ws_url="ws://...",
ufo_client=ufo_client,
max_retries=10 # More resilient
)
```
### Message Parsing Errors
**Graceful Error Handling:**
```python
try:
data = ServerMessage.model_validate_json(msg)
# Process message...
except Exception as e:
self.logger.error(f"[WS] Error handling message: {e}", exc_info=True)
# Message is dropped, client continues listening
```
Message parsing errors don't crash the client—the error is logged and the receive loop continues.
### Registration Error Handling
**Fallback to Minimal Metadata:**
```python
try:
system_info = DeviceInfoProvider.collect_system_info(...)
metadata = {"system_info": system_info.to_dict()}
except Exception as e:
self.logger.error(f"[WS] \[AIP] Error collecting device info: {e}")
# Continue with minimal metadata
metadata = {
"registration_time": datetime.datetime.now(datetime.timezone.utc).isoformat()
}
```
If device info collection fails, registration still proceeds with minimal metadata (timestamp only).
---
## 🔌 AIP Protocol Integration
The WebSocket client uses three specialized AIP protocols for different communication patterns.
### 1. Registration Protocol
**Purpose:** Client registration and device profile exchange
```python
from aip.protocol.registration import RegistrationProtocol
self.registration_protocol = RegistrationProtocol(self.transport)
# Register as device
success = await self.registration_protocol.register_as_device(
device_id="device_windows_001",
metadata={"system_info": {...}},
platform="windows"
)
```
**Key Methods:**
| Method | Parameters | Returns | Purpose |
|--------|------------|---------|---------|
| `register_as_device()` | `device_id`, `metadata`, `platform` | `bool` | Register client as device |
See [AIP Registration Protocol](../aip/protocols.md#registration-protocol) for message format details.
### 2. Heartbeat Protocol
**Purpose:** Connection keepalive and health monitoring
```python
from aip.protocol.heartbeat import HeartbeatProtocol
self.heartbeat_protocol = HeartbeatProtocol(self.transport)
# Send heartbeat
await self.heartbeat_protocol.send_heartbeat("device_windows_001")
```
**Key Methods:**
| Method | Parameters | Returns | Purpose |
|--------|------------|---------|---------|
| `send_heartbeat()` | `client_id` | `None` | Send keepalive message |
See [AIP Heartbeat Protocol](../aip/protocols.md#heartbeat-protocol) for message format details.
### 3. Task Execution Protocol
**Purpose:** Task request and result exchange
```python
from aip.protocol.task_execution import TaskExecutionProtocol
self.task_protocol = TaskExecutionProtocol(self.transport)
# Send task request
await self.task_protocol.send_task_request(
request="Open Notepad",
task_name="task_001",
session_id=None,
client_id="device_windows_001",
metadata={"platform": "windows"}
)
# Send task result
await self.task_protocol.send_task_result(
session_id="session_123",
prev_response_id="resp_456",
action_results=[...],
status=TaskStatus.COMPLETED,
client_id="device_windows_001"
)
```
**Key Methods:**
| Method | Parameters | Returns | Purpose |
|--------|------------|---------|---------|
| `send_task_request()` | `request`, `task_name`, `session_id`, `client_id`, `metadata` | `None` | Request task execution |
| `send_task_result()` | `session_id`, `prev_response_id`, `action_results`, `status`, `client_id` | `None` | Return execution results |
See [AIP Task Execution Protocol](../aip/protocols.md#task-execution-protocol) for message format details.
---
## 🔍 Connection State Management
### State Checking
Use `is_connected()` to check if the client is ready to send messages.
**Implementation:**
```python
def is_connected(self) -> bool:
"""Check if WebSocket is connected and registered."""
return (
self.connected_event.is_set() # Registration succeeded
and self._ws is not None # WebSocket exists
and not self._ws.closed # WebSocket is open
)
```
**Usage Example:**
```python
if ws_client.is_connected():
await ws_client.start_task("Open Calculator", "task_calc")
else:
logger.error("Not connected to server - cannot send task")
```
### Connected Event
The `connected_event` is an `asyncio.Event` that signals successful registration.
**Usage Pattern:**
```python
# Wait for connection before sending requests
await ws_client.connected_event.wait()
# Now safe to send task requests
await ws_client.start_task("Open Notepad", "task_notepad")
```
**Event Lifecycle:**
| State | Event Status | Meaning |
|-------|--------------|---------|
| Initial | Not set | Client not connected |
| Connecting | Not set | WebSocket connecting, registering |
| Registered | **Set** | ✅ Ready to send/receive messages |
| Disconnected | Cleared | Connection lost, will retry |
## ✅ Best Practices
### Development Best Practices
**1. Enable DEBUG Logging**
```python
import logging
logging.basicConfig(level=logging.DEBUG)
```
**Output:**
```log
DEBUG - [WS] [AIP] Heartbeat sent
DEBUG - [WS] [AIP] Heartbeat failed (connection closed): ...
INFO - [WS] Received message: ServerMessage(type='COMMAND', ...)
```
**2. Test Connection Before Full Integration**
```python
# Test just connection and registration
ws_client = UFOWebSocketClient(ws_url, ufo_client)
await ws_client.connect_and_listen() # Should register successfully
```
**3. Handle Connection Loss Gracefully**
```python
try:
await ws_client.connect_and_listen()
except Exception as e:
logger.error(f"WebSocket client error: {e}")
# Implement recovery (e.g., alert, restart)
```
### Production Best Practices
**1. Use Appropriate Retry Limits**
For production networks with occasional instability:
```python
ws_client = UFOWebSocketClient(
ws_url="wss://production-server.com/ws",
ufo_client=ufo_client,
max_retries=10 # More retries for resilience
)
```
**2. Monitor Connection Health**
Log heartbeat success/failure for alerting:
```python
# In heartbeat_loop (add custom monitoring):
try:
await self.heartbeat_protocol.send_heartbeat(...)
self.logger.debug("[WS] ✅ Heartbeat sent successfully")
# Update metrics: heartbeat_success_count++
except Exception as e:
self.logger.error(f"[WS] ❌ Heartbeat failed: {e}")
# Trigger alert: connection_health_alert()
```
**3. Use Secure WebSocket (WSS)**
```python
# Production: Encrypted WebSocket
ws_client = UFOWebSocketClient(
ws_url="wss://ufo-server.company.com/ws", # WSS, not WS
ufo_client=ufo_client
)
```
**4. Clean State on Reconnection**
The client automatically resets state:
```python
async with self.ufo_client.task_lock:
self.ufo_client.reset() # Clears session state
# Send new task request
```
### Error Handling Best Practices
!!!warning "Defensive Programming"
**1. Expect Transient Failures**
```python
# Increase retries for unreliable networks
max_retries=10
# Monitor retry count in logs
self.logger.info(f"[WS] Retry {self.retry_count}/{self.max_retries}")
```
**2. Validate Messages Before Processing**
```python
# Already handled by Pydantic in source code:
data = ServerMessage.model_validate_json(msg) # Raises on invalid
```
**3. Report Errors via AIP**
```python
# Send structured error messages back to server
error_msg = ClientMessage(
type=ClientMessageType.ERROR,
error=str(e),
client_id=self.ufo_client.client_id,
timestamp=datetime.datetime.now(datetime.timezone.utc).isoformat()
)
await self.transport.send(error_msg.model_dump_json().encode())
```
---
## 🔗 Integration Points
### UFO Client Integration
The WebSocket client delegates all command execution to the UFO Client.
**Execution Flow:**
```python
# WebSocket client receives command
action_results = await self.ufo_client.execute_step(server_response)
```
**Integration:**
| WebSocket Client Role | UFO Client Role |
|----------------------|-----------------|
| Receive commands from server | Execute commands via MCP tools |
| Parse server messages | Manage computer/tool registry |
| Send results back | Collect execution results |
| Handle connection errors | Handle execution errors |
See [UFO Client](./ufo_client.md) for execution details.
### Device Info Provider Integration
Device information is collected once during registration.
**Integration:**
```python
from ufo.client.device_info_provider import DeviceInfoProvider
system_info = DeviceInfoProvider.collect_system_info(
client_id=self.ufo_client.client_id,
custom_metadata=None
)
```
See [Device Info Provider](./device_info.md) for profiling details.
### AIP Transport Integration
All messages go through the WebSocket transport layer.
**Transport Creation:**
```python
from aip.transport.websocket import WebSocketTransport
self.transport = WebSocketTransport(ws)
```
**Transport Usage:**
- **Protocols use transport** for sending messages
- **Direct transport access** for error messages
See [AIP Transport Layer](../aip/transport.md) for transport details.
## 🚀 Next Steps
**Continue Learning**
1. **Connect Your Client** - Follow the step-by-step guide: [Quick Start Guide](./quick_start.md)
2. **Understand Command Execution** - Learn how the UFO Client executes commands: [UFO Client Documentation](./ufo_client.md)
3. **Explore Device Profiling** - See what device information is collected: [Device Info Provider](./device_info.md)
4. **Master the AIP Protocol** - Deep dive into message formats: [AIP Protocol Guide](../aip/protocols.md)
5. **Study Server-Side Registration** - Understand how the server handles registration: [Server Overview](../server/overview.md)
---
### Documents Docs Galaxy Webui (`documents/docs/galaxy/webui.md`)
# Galaxy WebUI
The **Galaxy WebUI** is a modern, interactive web interface for the UFO³ Galaxy Framework. It provides real-time visualization of task constellations, device status, agent interactions, and execution flow through an elegant, space-themed interface.
Galaxy WebUI - Interactive constellation visualization and real-time monitoring
---
## 🌟 Overview
The Galaxy WebUI transforms the command-line Galaxy experience into a rich, visual interface where you can:
- **🗣️ Chat with Galaxy**: Submit natural language requests through an intuitive chat interface
- **📊 Visualize Constellations**: Watch task constellations form and execute as interactive DAG graphs
- **🎯 Monitor Execution**: Track task status, device assignments, and real-time progress
- **🔄 See Agent Reasoning**: Observe agent thoughts, plans, and decision-making processes
- **🖥️ Manage Devices**: View, monitor, and **add new devices** through the UI
- **➕ Add Device Agents**: Register new device agents dynamically without restarting
- **📡 Stream Events**: Follow the event log to understand system behavior in real-time
---
## 🚀 Quick Start
### Starting the WebUI
Configure a strong, pre-generated API key from your secret manager before
starting the backend. This masked PowerShell prompt keeps the key out of console
output and shell history:
```powershell
$key = Read-Host "Galaxy WebUI API key" -AsSecureString
$env:GALAXY_WEBUI_API_KEY = [System.Net.NetworkCredential]::new("", $key).Password
# Launch Galaxy with WebUI
python -m galaxy --webui
```
The WebUI will automatically:
1. Start the backend server on `http://127.0.0.1:8000` (or next available port)
2. Load the configured API key without logging or printing it
3. Open your default browser to the interface without embedding the key in HTML
4. Require the configured key for authenticated HTTP and WebSocket requests
Startup fails with setup instructions if no key is configured. Clients must use
the configured key in the `X-API-Key` header or WebSocket `token` parameter.
!!!tip "Custom Session Name"
```powershell
python -m galaxy --webui --session-name "data_pipeline_demo"
```
### First Request
1. **Enter your request** in the chat input at the bottom
2. **Press Enter** or click Send
3. **Watch the constellation form** in the DAG visualization panel
4. **Monitor task execution** as devices process their assigned tasks
5. **See results** displayed in the chat window
---
## 🏗️ Architecture
### Design Principles
The Galaxy WebUI backend follows **software engineering best practices**:
**Separation of Concerns:**
- **Models Layer**: Pydantic models ensure type safety and validation
- **Services Layer**: Business logic isolated from presentation
- **Handlers Layer**: WebSocket message processing logic
- **Routers Layer**: HTTP endpoint definitions
**Dependency Injection:**
- `AppState` class provides centralized state management
- `get_app_state()` dependency injection function
- Replaces global variables with type-safe properties
**Type Safety:**
- Pydantic models for all API requests/responses
- Enums for constants (`WebSocketMessageType`, `RequestStatus`)
- `TYPE_CHECKING` pattern for forward references
- Comprehensive type annotations throughout
**Modularity:**
- Clear module boundaries
- Easy to test individual components
- Simple to extend with new features
- Better code organization and maintainability
### System Architecture
The Galaxy WebUI follows a modern client-server architecture with real-time event streaming:
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Component Overview
#### Backend Components
The Galaxy WebUI backend follows a **modular architecture** with clear separation of concerns:
| Component | File/Directory | Responsibility |
|-----------|----------------|----------------|
| **FastAPI Server** | `galaxy/webui/server.py` | Application initialization, middleware, router registration, lifespan management |
| **Models** | `galaxy/webui/models/` | Pydantic models for requests/responses, enums for type safety |
| **Services** | `galaxy/webui/services/` | Business logic layer (config, device, galaxy operations) |
| **Handlers** | `galaxy/webui/handlers/` | WebSocket message processing and routing |
| **Routers** | `galaxy/webui/routers/` | FastAPI endpoint definitions organized by feature |
| **Dependencies** | `galaxy/webui/dependencies.py` | Dependency injection for state management (AppState) |
| **WebSocket Observer** | `galaxy/webui/websocket_observer.py` | Event subscription and broadcasting to WebSocket clients |
| **Event Serializer** | Built into observer | Converts Python objects to JSON-compatible format |
**Detailed Backend Structure:**
```
galaxy/webui/
├── server.py # Main FastAPI application
├── dependencies.py # AppState and dependency injection
├── websocket_observer.py # EventSerializer + WebSocketObserver
├── models/
│ ├── __init__.py # Export all models
│ ├── enums.py # WebSocketMessageType, RequestStatus enums
│ ├── requests.py # Pydantic request models
│ └── responses.py # Pydantic response models
├── services/
│ ├── __init__.py
│ ├── config_service.py # Configuration management
│ ├── device_service.py # Device operations and snapshots
│ └── galaxy_service.py # Galaxy client interactions
├── handlers/
│ ├── __init__.py
│ └── websocket_handlers.py # WebSocket message handler
├── routers/
│ ├── __init__.py
│ ├── health.py # Health check endpoint
│ ├── devices.py # Device management endpoints
│ └── websocket.py # WebSocket endpoint
└── templates/
└── index.html # Fallback HTML page
```
**Architecture Benefits:**
✅ **Maintainability**: Each module has a single, clear responsibility
✅ **Testability**: Services and handlers can be unit tested independently
✅ **Type Safety**: Pydantic models validate all inputs/outputs
✅ **Extensibility**: Easy to add new endpoints, message types, or services
✅ **Readability**: Clear module boundaries improve code comprehension
✅ **Reusability**: Services can be shared across multiple endpoints
#### Frontend Components
| Component | Location | Purpose |
|-----------|----------|---------|
| **App** | `src/App.tsx` | Main layout, connection status, theme management |
| **ChatWindow** | `src/components/chat/ChatWindow.tsx` | Message display and input interface |
| **DagPreview** | `src/components/constellation/DagPreview.tsx` | Interactive constellation graph visualization |
| **DevicePanel** | `src/components/devices/DevicePanel.tsx` | Device status cards, search, and add button |
| **DeviceCard** | `src/components/devices/DeviceCard.tsx` | Individual device status display |
| **AddDeviceModal** | `src/components/devices/AddDeviceModal.tsx` | Modal dialog for adding new devices |
| **RightPanel** | `src/components/layout/RightPanel.tsx` | Tabbed panel for constellation, tasks, details |
| **EventLog** | `src/components/EventLog.tsx` | Real-time event stream display |
| **GalaxyStore** | `src/store/galaxyStore.ts` | Zustand state management |
| **WebSocket Client** | `src/services/websocket.ts` | WebSocket connection with auto-reconnect |
---
## 🔌 Communication Protocol
### HTTP API Endpoints
#### Health Check
```http
GET /health
X-API-Key:
```
**Response:**
```json
{
"status": "healthy",
"connections": 3,
"events_sent": 1247
}
```
#### Add Device
```http
POST /api/devices
Content-Type: application/json
X-API-Key:
```
**Request Body:**
```json
{
"device_id": "windows-laptop-1",
"server_url": "ws://192.168.1.100:8080",
"os": "Windows",
"capabilities": ["excel", "outlook", "browser"],
"metadata": {
"region": "us-west-2",
"owner": "data-team"
},
"auto_connect": true,
"max_retries": 5
}
```
!!!warning "Device URLs Must Not Redirect"
Supply the final WebSocket endpoint in `server_url`. Galaxy pins the validated destination and rejects HTTP 301, 302, 303, 307, and 308 redirects before connecting to the redirect target, including same-origin and relative redirects. See [Server URL and Redirects](../configuration/system/galaxy_devices.md#server-url-and-redirects) for configuration guidance.
When `auto_connect` is enabled, the connection attempt runs in the background. A successful registration response does not guarantee that the WebSocket connection succeeded. If connection logs contain `WebSocket redirects are not allowed for pinned connections`, configure the final URL directly; retrying the redirecting URL will not resolve the failure.
**Success Response (200):**
```json
{
"status": "success",
"message": "Device 'windows-laptop-1' added successfully",
"device": {
"device_id": "windows-laptop-1",
"server_url": "ws://192.168.1.100:8080",
"os": "Windows",
"capabilities": ["excel", "outlook", "browser"],
"auto_connect": true,
"max_retries": 5,
"metadata": {
"region": "us-west-2",
"owner": "data-team"
}
}
}
```
**Error Responses:**
- **404 Not Found**: `devices.yaml` configuration file not found
```json
{
"detail": "devices.yaml not found"
}
```
- **409 Conflict**: Device ID already exists
```json
{
"detail": "Device ID 'windows-laptop-1' already exists"
}
```
- **500 Internal Server Error**: Failed to add device
```json
{
"detail": "Failed to add device: "
}
```
### WebSocket Connection
The WebUI maintains a persistent WebSocket connection to the Galaxy backend for bidirectional real-time communication.
**Connection URL:** `ws://localhost:8000/ws?token=`
!!!warning "Authentication Required"
The WebSocket endpoint requires a valid `token` query parameter matching the server's API key. Connections without a valid token are rejected with code **1008** (Policy Violation).
### Message Types
#### Client → Server
**1. User Request**
```json
{
"type": "request",
"text": "Extract sales data and create an Excel report",
"timestamp": 1234567890
}
```
**2. Session Reset**
```json
{
"type": "reset",
"timestamp": 1234567890
}
```
**3. Ping (Keepalive)**
```json
{
"type": "ping",
"timestamp": 1234567890
}
```
#### Server → Client
**1. Welcome Message**
```json
{
"type": "welcome",
"message": "Connected to Galaxy Web UI",
"timestamp": 1234567890
}
```
**2. Device Snapshot (on connect)**
```json
{
"event_type": "device_snapshot",
"source_id": "webui.server",
"timestamp": 1234567890,
"data": {
"event_name": "device_snapshot",
"device_count": 2
},
"all_devices": {
"windows_device_1": {
"device_id": "windows_device_1",
"status": "connected",
"os": "windows",
"capabilities": ["desktop_automation", "excel"],
"metadata": {},
"last_heartbeat": "2025-11-09T10:30:00",
"current_task_id": null
}
}
}
```
**3. Galaxy Events**
All Galaxy events are forwarded to the WebUI in real-time:
```json
{
"event_type": "agent_response",
"source_id": "ConstellationAgent",
"timestamp": 1234567890,
"agent_name": "ConstellationAgent",
"agent_type": "constellation",
"output_type": "response",
"output_data": {
"thought": "I need to decompose this task...",
"plan": ["Analyze requirements", "Create DAG", "Assign devices"],
"response": "Creating constellation with 3 tasks"
}
}
```
```json
{
"event_type": "constellation_created",
"source_id": "TaskConstellation",
"timestamp": 1234567890,
"constellation_id": "constellation_123",
"constellation_state": "planning",
"data": {
"constellation": {
"constellation_id": "constellation_123",
"name": "Sales Report Pipeline",
"state": "planning",
"tasks": {
"task_1": {
"task_id": "task_1",
"name": "Extract Data",
"status": "pending",
"target_device_id": "linux_device_1"
}
},
"dependencies": {
"task_2": ["task_1"]
}
}
}
}
```
```json
{
"event_type": "task_status_changed",
"source_id": "TaskOrchestrator",
"timestamp": 1234567890,
"task_id": "task_1",
"status": "running",
"result": null,
"error": null
}
```
```json
{
"event_type": "device_status_changed",
"source_id": "DeviceManager",
"timestamp": 1234567890,
"device_id": "windows_device_1",
"device_status": "busy",
"device_info": {
"current_task_id": "task_2"
}
}
```
---
## 🎨 User Interface
### Main Layout
The WebUI uses a three-panel layout:
```mermaid
graph LR
subgraph UI["Galaxy WebUI Layout"]
subgraph Header["🌌 Header Bar"]
H1[Galaxy Logo]
H2[Connection Status]
H3[Settings]
end
subgraph Left["📱 Left Panel: Devices"]
L1[Device Card 1 Windows 🟢 Connected]
L2[Device Card 2 Linux 🔵 Busy]
L3[Device Card 3 macOS 🟢 Idle]
end
subgraph Center["💬 Center Panel: Chat"]
C1[Message History User/Agent/Actions]
C2[Action Trees Collapsible]
C3[Input Box Type request...]
end
subgraph Right["📊 Right Panel: Tabs"]
R1[🌟 Constellation DAG Graph]
R2[📋 Tasks Task List]
R3[📝 Details Selected Info]
end
Header -.-> Left
Header -.-> Center
Header -.-> Right
Left -.-> Center
Center -.-> Right
end
style Header fill:#1a1a2e,stroke:#00d4ff,stroke-width:2px,color:#fff
style Left fill:#0f1419,stroke:#10b981,stroke-width:2px,color:#fff
style Center fill:#16213e,stroke:#7b2cbf,stroke-width:2px,color:#fff
style Right fill:#1a1a2e,stroke:#ff006e,stroke-width:2px,color:#fff
```
### Key Features
#### 🗣️ Chat Interface
**Location:** Center panel
**Features:**
- Natural language input for requests
- Message history with agent responses
- Collapsible action trees showing execution details
- Thought, plan, and response display
- Status indicators (pending, running, completed, failed)
- Markdown rendering for rich text
- Code block syntax highlighting
**Message Types:**
- **User Messages**: Your requests to Galaxy
- **Agent Responses**: ConstellationAgent thoughts, plans, and responses
- **Action Messages**: Individual constellation operations (add_task, build_constellation, etc.)
- **System Messages**: Status updates and notifications
#### 📊 DAG Visualization
**Location:** Right panel → Constellation tab
**Features:**
- Interactive node-and-edge graph
- Real-time task status updates
- Color-coded status indicators:
- 🔵 Pending: Gray
- 🟡 Running: Blue (animated)
- 🟢 Completed: Green
- 🔴 Failed: Red
- ⚫ Skipped: Orange
- Dependency edges showing task relationships
- Pan and zoom controls
- Automatic layout optimization
- Node click to view task details
**Interaction:**
- **Click node**: Select task and show details
- **Pan**: Click and drag background
- **Zoom**: Mouse wheel or pinch gesture
- **Fit view**: Click fit-to-screen button
#### 🖥️ Device Management
**Location:** Left sidebar
**Features:**
- Device status cards with real-time updates
- Color-coded status indicators:
- 🟢 Connected/Idle: Green
- 🔵 Busy: Blue
- 🟡 Connecting: Yellow
- 🔴 Disconnected/Failed: Red
- Capability badges
- Current task assignment
- Last heartbeat timestamp
- Connection metrics
- Click to view device details
- **➕ Add Device Button**: Manually add new devices through UI
**Device Information:**
- OS type and version
- Server URL
- Installed applications
- Performance tier
- Custom metadata
**Adding a New Device:**
Click the **"+"** button in the Device Panel header to open the Add Device Modal:
Add Device Modal - Register new device agents through the UI
1. **Basic Information:**
- **Device ID**: Unique identifier for the device (required)
- **Server URL**: WebSocket endpoint URL (must start with `ws://` or `wss://`)
- **Operating System**: Select from Windows, Linux, macOS, or enter custom OS
2. **Capabilities:**
- Add capabilities one by one (e.g., `excel`, `outlook`, `browser`)
- Remove capabilities by clicking the ✕ icon
- At least one capability is required
3. **Advanced Options:**
- **Auto-connect**: Automatically connect to device after registration (default: enabled)
- **Max Retries**: Maximum connection retry attempts (default: 5)
4. **Metadata (Optional):**
- Add custom key-value pairs for additional device information
- Examples: `region: us-east-1`, `tier: premium`, `owner: team-a`
**API Endpoint:**
```http
POST /api/devices
Content-Type: application/json
{
"device_id": "my-device-1",
"server_url": "ws://192.168.1.100:8080",
"os": "Windows",
"capabilities": ["excel", "outlook", "powerpoint"],
"metadata": {
"region": "us-east-1",
"tier": "standard"
},
"auto_connect": true,
"max_retries": 5
}
```
**Response:**
```json
{
"status": "success",
"message": "Device 'my-device-1' added successfully",
"device": {
"device_id": "my-device-1",
"server_url": "ws://192.168.1.100:8080",
"os": "Windows",
"capabilities": ["excel", "outlook", "powerpoint"],
"auto_connect": true,
"max_retries": 5,
"metadata": {
"region": "us-east-1",
"tier": "standard"
}
}
}
```
**Device Registration Process:**
When a device is added through the UI:
1. **Validation**: Form data is validated (required fields, URL format, duplicate device_id)
2. **Configuration**: Device is saved to `config/galaxy/devices.yaml`
3. **Registration**: Device is registered with the Galaxy Device Manager
4. **Connection**: If `auto_connect` is enabled, connection is initiated automatically
5. **Event Broadcast**: Device status updates are broadcast to all WebSocket clients
6. **UI Update**: Device card appears in the Device Panel with real-time status
#### 📋 Task Details
**Location:** Right panel → Tasks tab / Details tab
**Features:**
- Task name and description
- Current status with icon
- Assigned device
- Dependencies and dependents
- Input and output data
- Execution results
- Error messages (if failed)
- Execution timeline
- Retry information
#### 📡 Event Log
**Location:** Right panel (optional view)
**Features:**
- Real-time event stream
- Event type filtering
- Timestamp display
- JSON payload viewer
- Search and filter
- Auto-scroll option
- Export to JSON
---
## 🎨 Theme and Styling
### Design System
The Galaxy WebUI uses a **space-themed design** with a dark color palette and vibrant accents.
#### Color Palette
```typescript
// Primary Colors
galaxy-dark: #0a0e27 // Deep space background
galaxy-blue: #00d4ff // Cyan accent (primary actions)
galaxy-purple: #7b2cbf // Purple accent (secondary)
galaxy-pink: #ff006e // Pink accent (tertiary)
// Status Colors
emerald: #10b981 // Success/Completed
cyan: #06b6d4 // Running/Active
amber: #f59e0b // Warning/Pending
rose: #f43f5e // Error/Failed
slate: #64748b // Neutral/Disabled
```
#### Visual Effects
- **Starfield Background**: Animated particle system with depth layers
- **Glassmorphism**: Frosted glass panels with backdrop blur
- **Glow Effects**: Neon-style glows on interactive elements
- **Smooth Animations**: Framer Motion for transitions
- **Gradient Accents**: Multi-color gradients on headers and buttons
#### Accessibility
- **High Contrast Mode**: Toggle for improved readability
- **Keyboard Navigation**: Full keyboard support
- **Screen Reader**: ARIA labels and semantic HTML
- **Focus Indicators**: Clear focus rings on interactive elements
---
## 🔧 Configuration
### Server Configuration
The WebUI server is configured through command-line arguments:
```powershell
python -m galaxy --webui [OPTIONS]
```
**Options:**
| Flag | Description | Default |
|------|-------------|---------|
| `--webui` | Enable WebUI mode | `False` |
| `--session-name` | Session display name | `"Galaxy Session"` |
| `--log-level` | Logging level | `INFO` |
| `--port` | Server port (if implemented) | `8000` |
**API Key Authentication:**
Configure `GALAXY_WEBUI_API_KEY` before starting the server, or pass
`start_server(api_key=...)` programmatically. The explicit argument takes
precedence over the environment variable. Startup fails if no key is configured;
the server does not generate keys or include them in startup output. Protected HTTP endpoints require an
`X-API-Key` header, and the WebSocket endpoint requires a `token` query parameter.
The API key is never embedded in HTML or injected into `window.__GALAXY_API_KEY__`.
**Environment Variables:**
| Variable | Description | Default |
|----------|-------------|---------|
| `GALAXY_WEBUI_API_KEY` | Backend API key; overridden by `start_server(api_key=...)` | Required unless supplied programmatically |
| `GALAXY_CORS_ORIGINS` | Comma-separated list of allowed CORS origins | `http://localhost:8000,http://127.0.0.1:8000` |
### Frontend Configuration
**Development Mode:**
```bash
cd galaxy/webui/frontend
npm run dev
```
Access at: `http://localhost:5173` (Vite dev server with HMR)
**Environment Variables:**
```bash
# .env.development
VITE_WS_URL=ws://localhost:8000/ws
VITE_API_URL=http://localhost:8000
```
**Build Configuration:**
```bash
cd galaxy/webui/frontend
npm run build
```
Builds production-ready frontend to `galaxy/webui/frontend/dist/`
---
## 🔍 Event Handling
### Event Flow
```mermaid
flowchart TD
A[Galaxy Core Event] --> B[Event Bus publish]
B --> C[WebSocketObserver on_event]
C --> D[EventSerializer serialize_event]
D --> D1[Type-specific field extraction]
D --> D2[Recursive value serialization]
D2 --> D3[Python → JSON]
D3 --> E[WebSocket Broadcast to all clients]
E --> F[Frontend Clients receive message]
F --> G[Store Update Zustand]
G --> H[UI Re-render React Components]
style A fill:#0a0e27,stroke:#ff006e,stroke-width:2px,color:#fff
style C fill:#16213e,stroke:#7b2cbf,stroke-width:2px,color:#fff
style D fill:#1a1a2e,stroke:#f59e0b,stroke-width:2px,color:#fff
style E fill:#1a1a2e,stroke:#00d4ff,stroke-width:2px,color:#fff
style G fill:#0f1419,stroke:#10b981,stroke-width:2px,color:#fff
style H fill:#1a1a2e,stroke:#f59e0b,stroke-width:2px,color:#fff
```
### Event Serialization
The `EventSerializer` class handles conversion of complex Python objects to JSON-compatible format:
**Features:**
- **Type Handler Registry**: Pre-registered handlers for Galaxy-specific types (TaskStarLine, TaskConstellation)
- **Type Caching**: Cached imports to avoid repeated import attempts
- **Recursive Serialization**: Handles nested structures (dicts, lists, dataclasses, Pydantic models)
- **Polymorphic Event Handling**: Different serialization logic for TaskEvent, ConstellationEvent, AgentEvent, DeviceEvent
- **Fallback Strategies**: Multiple serialization attempts with graceful fallback to string representation
**Serialization Chain:**
1. Handle primitives (str, int, float, bool, None)
2. Handle datetime objects → ISO format
3. Handle collections (dict, list, tuple) → recursive serialization
4. Check registered type handlers (TaskStarLine, TaskConstellation)
5. Try dataclass serialization (`asdict()`)
6. Try Pydantic model serialization (`model_dump()`)
7. Try generic `to_dict()` method
8. Fallback to `str()` representation
### Event Types
The WebUI subscribes to all Galaxy event types:
| Event Type | Source | Description |
|------------|--------|-------------|
| `agent_request` | ConstellationAgent | User request received |
| `agent_response` | ConstellationAgent | Agent thought/plan/response |
| `constellation_created` | TaskConstellation | New constellation formed |
| `constellation_updated` | TaskConstellation | Constellation modified |
| `constellation_completed` | TaskConstellation | All tasks finished |
| `task_created` | TaskOrchestrator | New task added |
| `task_assigned` | TaskOrchestrator | Task assigned to device |
| `task_started` | TaskOrchestrator | Task execution started |
| `task_status_changed` | TaskOrchestrator | Task status updated |
| `task_completed` | TaskOrchestrator | Task finished successfully |
| `task_failed` | TaskOrchestrator | Task encountered error |
| `device_connected` | DeviceManager | Device came online |
| `device_disconnected` | DeviceManager | Device went offline |
| `device_status_changed` | DeviceManager | Device status updated |
| `device_heartbeat` | DeviceManager | Device health check |
### State Management
The frontend uses **Zustand** for centralized state management:
```typescript
// Store Structure
interface GalaxyStore {
// Connection
connectionStatus: ConnectionStatus;
connected: boolean;
// Session
session: {
id: string | null;
displayName: string;
startedAt: number | null;
};
// Data
messages: Message[];
constellations: Record;
tasks: Record;
devices: Record;
notifications: NotificationItem[];
// UI State
ui: {
activeConstellationId: string | null;
activeTaskId: string | null;
activeDeviceId: string | null;
rightPanelTab: 'constellation' | 'tasks' | 'details';
showDeviceDrawer: boolean;
};
}
```
---
## 📱 Responsive Design
The WebUI is designed to work on various screen sizes:
### Desktop (1920px+)
- Three-panel layout with full sidebar
- Large DAG visualization
- Expanded device cards
### Laptop (1280px - 1919px)
- Standard three-panel layout
- Medium DAG visualization
- Compact device cards
### Tablet (768px - 1279px)
- Collapsible sidebar
- Simplified DAG view
- Stacked layout option
### Mobile (< 768px)
- Single-panel navigation
- Tab-based interface
- Touch-optimized controls
!!!warning "Recommended Resolution"
For the best experience, use a desktop or laptop with at least **1280px width**. The DAG visualization requires adequate screen space for clear readability.
---
## 🐛 Troubleshooting
### Connection Issues
**Problem:** WebSocket connection fails
**Solutions:**
1. **Verify backend is running:**
```powershell
# Check health endpoint
curl http://localhost:8000/health
```
2. **Check firewall settings:**
- Allow incoming connections on port 8000
- Check corporate firewall/proxy settings
3. **Verify WebSocket URL:**
- Browser console should show: `WebSocket connection established`
- Check for CORS errors in console
4. **Try different port:**
```powershell
python -m galaxy --webui --port 8080
```
### Frontend Not Loading
**Problem:** Blank page or "Server is running" placeholder
**Solutions:**
1. **Build the frontend:**
```bash
cd galaxy/webui/frontend
npm install
npm run build
```
2. **Check build output:**
- Verify `galaxy/webui/frontend/dist/` exists
- Check for TypeScript errors: `npm run build`
3. **Clear browser cache:**
- Hard refresh: `Ctrl+Shift+R` (Windows) or `Cmd+Shift+R` (Mac)
- Clear site data in DevTools
### Events Not Appearing
**Problem:** No events shown in UI, DAG not updating
**Solutions:**
1. **Check event system:**
- Look for "WebSocket observer registered" in backend logs
- Verify connection count: `curl http://localhost:8000/health`
2. **Check browser console:**
- Look for WebSocket message logs
- Check for JavaScript errors
3. **Enable debug mode:**
```powershell
python -m galaxy --webui --log-level DEBUG
```
### Performance Issues
**Problem:** UI slow or unresponsive
**Solutions:**
1. **Limit event log size:**
- Event log keeps last 200 events
- Messages limited to 500
2. **Reduce DAG complexity:**
- Large constellations (>50 tasks) may be slow
- Consider viewport culling for very large graphs
3. **Check browser performance:**
- Close unnecessary tabs
- Use Chrome/Edge for best performance
- Disable browser extensions
### Device Addition Issues
**Problem:** Cannot add device through UI
**Solutions:**
1. **Check `devices.yaml` exists:**
```powershell
# Verify configuration file
Test-Path config/galaxy/devices.yaml
```
2. **Verify device ID uniqueness:**
- Device ID must be unique across all devices
- Check existing devices in the Device Panel
3. **Validate server URL format:**
- Must start with `ws://` or `wss://`
- Example: `ws://192.168.1.100:8080` or `wss://device.example.com`
- Ensure device server is actually running at that URL
4. **Check backend logs:**
```powershell
# Look for error messages
python -m galaxy --webui --log-level DEBUG
```
**Problem:** Device added but not connecting
**Solutions:**
1. **Verify device server is running:**
- Check that the device agent is running at the specified URL
- Test connection: `curl ws://your-device-url/`
2. **Check firewall/network:**
- Ensure WebSocket port is open
- Verify no proxy/firewall blocking connection
3. **Check device logs:**
- Look at the device agent logs for connection errors
- Verify device can reach the Galaxy server
4. **Manual connection:**
- If `auto_connect` failed, devices will retry automatically
- Check `connection_attempts` in device details
- Increase `max_retries` if needed
**Problem:** Validation errors when adding device
**Common Validation Issues:**
| Error | Cause | Solution |
|-------|-------|----------|
| "Device ID is required" | Empty device_id field | Provide a unique identifier |
| "Device ID already exists" | Duplicate device_id | Choose a different ID |
| "Server URL is required" | Empty server_url | Provide WebSocket URL |
| "Invalid WebSocket URL" | Wrong URL format | Use `ws://` or `wss://` prefix |
| "OS is required" | No OS selected | Select or enter OS type |
| "At least one capability required" | No capabilities added | Add at least one capability |
---
## 🧪 Development
### Prerequisites
- **Node.js** >= 18
- **npm** >= 9
- **Python** >= 3.10
- **Galaxy** installed and configured
### Development Setup
```bash
# 1. Install frontend dependencies
cd galaxy/webui/frontend
npm install
# 2. Start development server
npm run dev
# 3. In another terminal, start Galaxy backend
cd ../../..
python -m galaxy --webui
```
**Development URL:** `http://localhost:5173`
### Project Structure
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Building for Production
```bash
cd galaxy/webui/frontend
npm run build
```
Output: `galaxy/webui/frontend/dist/`
### Code Quality
**Frontend:**
```bash
# Lint
npm run lint
# Type check
npm run type-check
# Format
npm run format
```
**Backend:**
The modular architecture improves testability. Example unit tests:
```python
# tests/webui/test_event_serializer.py
import pytest
from galaxy.webui.websocket_observer import EventSerializer
from galaxy.core.events import TaskEvent
def test_serialize_task_event():
"""Test serialization of TaskEvent."""
serializer = EventSerializer()
event = TaskEvent(
event_type=EventType.TASK_STARTED,
source_id="test",
timestamp=1234567890,
task_id="task_1",
status="running",
result=None,
error=None
)
result = serializer.serialize_event(event)
assert result["event_type"] == "task_started"
assert result["task_id"] == "task_1"
assert result["status"] == "running"
def test_serialize_nested_dict():
"""Test recursive serialization of nested structures."""
serializer = EventSerializer()
data = {
"level1": {
"level2": {
"value": 42
}
}
}
result = serializer.serialize_value(data)
assert result["level1"]["level2"]["value"] == 42
```
```python
# tests/webui/test_services.py
import pytest
from galaxy.webui.services.device_service import DeviceService
from galaxy.webui.dependencies import AppState
def test_build_device_snapshot():
"""Test device snapshot building."""
app_state = AppState()
# Setup mock galaxy_client with devices
service = DeviceService(app_state)
snapshot = service.build_device_snapshot()
assert "device_count" in snapshot
assert "all_devices" in snapshot
```
```python
# tests/webui/test_handlers.py
import pytest
from unittest.mock import AsyncMock, MagicMock
from galaxy.webui.handlers.websocket_handlers import WebSocketMessageHandler
from galaxy.webui.models.enums import WebSocketMessageType
@pytest.mark.asyncio
async def test_handle_ping():
"""Test ping message handling."""
websocket = AsyncMock()
app_state = MagicMock()
handler = WebSocketMessageHandler(websocket, app_state)
response = await handler.handle_message({
"type": WebSocketMessageType.PING,
"timestamp": 1234567890
})
assert response["type"] == "pong"
```
---
## 🚀 Advanced Usage
### Extending the Backend
The modular architecture makes it easy to extend the Galaxy WebUI backend:
#### Adding a New API Endpoint
**1. Define Pydantic models:**
```python
# galaxy/webui/models/requests.py
from pydantic import BaseModel, Field
class TaskQueryRequest(BaseModel):
"""Request to query task status."""
task_id: str = Field(..., description="The task ID to query")
include_history: bool = Field(default=False)
```
```python
# galaxy/webui/models/responses.py
from pydantic import BaseModel
class TaskQueryResponse(BaseModel):
"""Response with task details."""
task_id: str
status: str
result: dict | None = None
```
**2. Create a service method:**
```python
# galaxy/webui/services/task_service.py
from typing import Dict, Any
from galaxy.webui.dependencies import AppState
class TaskService:
"""Service for task-related operations."""
def __init__(self, app_state: AppState):
self.app_state = app_state
def get_task_details(self, task_id: str, include_history: bool) -> Dict[str, Any]:
"""Get details for a specific task."""
galaxy_session = self.app_state.galaxy_session
if not galaxy_session:
raise ValueError("No active Galaxy session")
# Your business logic here
task = galaxy_session.get_task(task_id)
return {
"task_id": task.task_id,
"status": task.status.value,
"result": task.result if include_history else None
}
```
**3. Add a router endpoint:**
```python
# galaxy/webui/routers/tasks.py
from fastapi import APIRouter, Depends
from galaxy.webui.dependencies import get_app_state
from galaxy.webui.models.requests import TaskQueryRequest
from galaxy.webui.models.responses import TaskQueryResponse
from galaxy.webui.services.task_service import TaskService
router = APIRouter(prefix="/api/tasks", tags=["tasks"])
@router.post("/query", response_model=TaskQueryResponse)
async def query_task(
request: TaskQueryRequest,
app_state = Depends(get_app_state)
):
"""Query task status and details."""
service = TaskService(app_state)
result = service.get_task_details(request.task_id, request.include_history)
return TaskQueryResponse(**result)
```
**4. Register the router:**
```python
# galaxy/webui/server.py
from galaxy.webui.routers import tasks_router
app.include_router(tasks_router)
```
#### Adding a New WebSocket Message Type
**1. Add enum value:**
```python
# galaxy/webui/models/enums.py
class WebSocketMessageType(str, Enum):
"""Types of messages exchanged via WebSocket."""
# ... existing types ...
CUSTOM_ACTION = "custom_action"
```
**2. Add request model:**
```python
# galaxy/webui/models/requests.py
class CustomActionMessage(BaseModel):
"""Custom action message."""
action_name: str
parameters: Dict[str, Any] = Field(default_factory=dict)
```
**3. Add handler method:**
```python
# galaxy/webui/handlers/websocket_handlers.py
async def _handle_custom_action(self, data: Dict[str, Any]) -> Dict[str, Any]:
"""Handle custom action messages."""
message = CustomActionMessage(**data)
# Your logic here
result = await self.service.perform_custom_action(
message.action_name,
message.parameters
)
return {
"type": "custom_action_completed",
"result": result
}
```
**4. Register handler:**
```python
# galaxy/webui/handlers/websocket_handlers.py
def __init__(self, websocket: WebSocket, app_state: AppState):
# ... existing code ...
self._handlers[WebSocketMessageType.CUSTOM_ACTION] = self._handle_custom_action
```
#### Customizing Event Serialization
Add custom serialization for new types:
```python
# galaxy/webui/websocket_observer.py
class EventSerializer:
def _register_handlers(self) -> None:
"""Register type-specific serialization handlers."""
# ... existing handlers ...
# Add custom type handler
try:
from your_module import CustomType
self._cached_types["CustomType"] = CustomType
self._type_handlers[CustomType] = self._serialize_custom_type
except ImportError:
self._cached_types["CustomType"] = None
def _serialize_custom_type(self, value: Any) -> Dict[str, Any]:
"""Serialize a CustomType object."""
try:
return {
"id": value.id,
"data": self.serialize_value(value.data),
"metadata": value.get_metadata()
}
except Exception as e:
self.logger.warning(f"Failed to serialize CustomType: {e}")
return str(value)
```
### Custom Event Handlers
You can extend the WebUI with custom event handlers:
```typescript
// src/services/customHandlers.ts
import { GalaxyEvent } from './websocket';
export function handleCustomEvent(event: GalaxyEvent) {
if (event.event_type === 'custom_event') {
// Your custom logic
console.log('Custom event:', event);
}
}
```
### Programmatic Device Management
Add devices programmatically using the API:
```typescript
// Add a device via API
async function addDevice(deviceConfig: {
device_id: string;
server_url: string;
os: string;
capabilities: string[];
metadata?: Record;
auto_connect?: boolean;
max_retries?: number;
}) {
const response = await fetch('http://localhost:8000/api/devices', {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify(deviceConfig),
});
if (!response.ok) {
const error = await response.json();
throw new Error(error.detail || 'Failed to add device');
}
return await response.json();
}
// Usage example
try {
const result = await addDevice({
device_id: 'production-server-1',
server_url: 'wss://prod-device.company.com',
os: 'Linux',
capabilities: ['docker', 'kubernetes', 'python'],
metadata: {
region: 'us-east-1',
environment: 'production',
tier: 'premium',
},
auto_connect: true,
max_retries: 10,
});
console.log('Device added:', result.device);
} catch (error) {
console.error('Failed to add device:', error);
}
```
**Batch Device Addition:**
```python
# Python script to add multiple devices
import requests
import json
devices = [
{
"device_id": "win-desktop-1",
"server_url": "ws://192.168.1.10:8080",
"os": "Windows",
"capabilities": ["office", "excel", "outlook"],
},
{
"device_id": "linux-server-1",
"server_url": "ws://192.168.1.20:8080",
"os": "Linux",
"capabilities": ["python", "docker", "git"],
},
{
"device_id": "mac-laptop-1",
"server_url": "ws://192.168.1.30:8080",
"os": "macOS",
"capabilities": ["safari", "xcode", "python"],
}
]
for device in devices:
response = requests.post(
"http://localhost:8000/api/devices",
json=device,
headers={"Content-Type": "application/json"}
)
if response.status_code == 200:
result = response.json()
print(f"✅ Added: {result['device']['device_id']}")
else:
error = response.json()
print(f"❌ Failed: {device['device_id']} - {error.get('detail')}")
```
**Checking Device Status:**
After adding devices, monitor their connection status through WebSocket events:
```typescript
// Listen for device connection events
websocket.onmessage = (event) => {
const data = JSON.parse(event.data);
if (data.event_type === 'device_status_changed') {
console.log(`Device ${data.device_id} status: ${data.device_status}`);
if (data.device_status === 'connected') {
console.log('✅ Device connected successfully');
} else if (data.device_status === 'failed') {
console.log('❌ Device connection failed');
}
}
};
```
### Custom Components
Add custom visualization components:
```tsx
// src/components/custom/MyVisualization.tsx
import React from 'react';
import { useGalaxyStore } from '../../store/galaxyStore';
export const MyVisualization: React.FC = () => {
const constellation = useGalaxyStore(s =>
s.constellations[s.ui.activeConstellationId || '']
);
return (
{/* Your custom visualization */}
);
};
```
### Theming
Create custom themes by extending Tailwind configuration:
```javascript
// tailwind.config.js
module.exports = {
theme: {
extend: {
colors: {
'custom-primary': '#your-color',
'custom-secondary': '#your-color',
},
},
},
};
```
---
## 📊 Monitoring and Analytics
### Health Check
**Endpoint:** `GET /health` (requires `X-API-Key` header)
```json
{
"status": "healthy",
"connections": 3,
"events_sent": 1247
}
```
### Metrics
The WebUI tracks:
- Active WebSocket connections
- Total events broadcasted
- Device online/offline status
- Task execution statistics
- Session duration
### Logging
**Backend Logs:**
```
INFO - WebSocket connection established from ('127.0.0.1', 54321)
INFO - Broadcasting event #42: agent_response to 2 clients
INFO - WebSocket client disconnected. Total connections: 1
```
**Frontend Console:**
```javascript
🌌 Connected to Galaxy WebSocket
📨 Raw WebSocket message received
📦 Parsed event data: {event_type: 'constellation_created', ...}
```
---
## 🔒 Security Considerations
### Authentication
The Galaxy WebUI uses **API key authentication** for all endpoints:
- **HTTP endpoints** (`/health`, `/api/devices`): Require the `X-API-Key` header.
- **WebSocket endpoint** (`/ws`): Requires a `token` query parameter.
- Set `GALAXY_WEBUI_API_KEY` before startup, or supply the `api_key` parameter of
`start_server()` (which takes precedence).
- Startup fails if no key is configured. The server does not generate keys or
include them in startup logs/output, and never embeds them in HTML responses.
### CORS
CORS origins are restricted by default to `http://localhost:8000` and `http://127.0.0.1:8000`. To allow additional origins, set the `GALAXY_CORS_ORIGINS` environment variable:
```bash
# Allow additional origins (comma-separated)
set GALAXY_CORS_ORIGINS=http://localhost:8000,http://127.0.0.1:8000,https://your-domain.com
```
### Server Binding
The WebUI server binds to `127.0.0.1` (localhost only) by default. This prevents the server from being accessed from other machines on the network. To allow external access, explicitly pass a different host to `start_server()`.
### Production Deployment
When deploying to production:
1. **Use HTTPS/WSS:**
```python
# Use secure WebSocket
wss://your-domain.com/ws?token=
```
2. **Configure CORS:**
```bash
# Set allowed origins via environment variable
set GALAXY_CORS_ORIGINS=https://your-domain.com
```
3. **Secure the API Key:**
- Provision a strong, pre-generated API key through `GALAXY_WEBUI_API_KEY` or `start_server(api_key=...)`, preferably using a secret manager
- Do not expose the API key in client-side code accessible to untrusted users
- Do not place the key in frontend `VITE_` variables or commit it to source control
- Rotate any key previously captured in shared server logs or console output
4. **Rate Limiting:**
- Limit request frequency
- Throttle WebSocket messages
- Prevent DoS attacks
---
## 📚 Additional Resources
### Documentation
- [FastAPI WebSocket Documentation](https://fastapi.tiangolo.com/advanced/websockets/)
- [React Documentation](https://react.dev/)
- [ReactFlow Documentation](https://reactflow.dev/)
- [Zustand Documentation](https://github.com/pmndrs/zustand)
- [Tailwind CSS Documentation](https://tailwindcss.com/)
- [Vite Documentation](https://vitejs.dev/)
### Galaxy Framework
- [Galaxy Overview](overview.md)
- [Constellation Agent](constellation_agent/overview.md)
- [Task Orchestrator](constellation_orchestrator/overview.md)
- [Device Manager](client/device_manager.md)
### Community
- [GitHub Issues](https://github.com/microsoft/UFO/issues)
- [GitHub Discussions](https://github.com/microsoft/UFO/discussions)
- [Email Support](mailto:ufo-agent@microsoft.com)
---
## 🎯 Next Steps
Now that you understand the Galaxy WebUI:
1. **[Quick Start Guide](../getting_started/quick_start_galaxy.md)** - Set up your first Galaxy session
2. **[Constellation Agent](constellation_agent/overview.md)** - Learn about task decomposition
3. **[Task Orchestrator](constellation_orchestrator/overview.md)** - Understand task execution
4. **[Device Manager](client/device_manager.md)** - Configure and monitor devices
Happy orchestrating with Galaxy WebUI! 🌌✨
---
### Documents Docs Getting Started Migration Ufo2 To Galaxy (`documents/docs/getting_started/migration_ufo2_to_galaxy.md`)
# Migration Guide: UFO² to UFO³ Galaxy
This guide helps you understand the evolution from **UFO²** (Desktop AgentOS) to **UFO³ Galaxy** (Multi-Device AgentOS), and provides practical steps for migrating your workflows to leverage Galaxy's cross-device orchestration capabilities.
---
## 🌟 Understanding the UFO Evolution
### The UFO Journey
The UFO project has evolved through three major iterations, each addressing increasingly complex automation challenges:
```mermaid
graph LR
A[UFO v1 2024-02] -->|Desktop Agent| B[UFO² 2025-04]
B -->|Multi-Device| C[UFO³ Galaxy 2025-11]
style A fill:#e3f2fd
style B fill:#c8e6c9
style C fill:#fff9c4
```
#### **UFO (v1.0)** — The Beginning
📅 *Released: February 2024*
- **Vision**: Screenshot-based Windows automation
- **Architecture**: Multi-agent (HostAgent + AppAgents)
- **Approach**: GPT-4V + pure GUI automation (click/type)
- **Scope**: Single Windows desktop, cross-app workflows
- **Limitation**: No deep OS integration
**Key Innovation:** First LLM-powered multi-agent GUI automation framework
---
#### **UFO² (v2.0)** — Desktop AgentOS
📅 *Released: April 2025*
📄 *Paper:* [UFO²: A Windows Agent for Seamless OS Interaction](https://arxiv.org/abs/2504.14603)
- **Vision**: Deep OS integration for robust automation
- **Architecture**: Two-tier hierarchy (HostAgent + AppAgents)
- **Innovations**:
- ✅ **Hybrid GUI–API execution** (51% fewer LLM calls)
- ✅ **Windows UIA + Win32 + WinCOM APIs**
- ✅ **Continuous knowledge learning** from docs & experience
- ✅ **Picture-in-Picture desktop** (non-disruptive automation)
- ✅ **MCP server integration** for tool augmentation
- **Scope**: Single Windows desktop
- **Success**: 10%+ better than state-of-the-art CUAs
**Key Innovation:** First agent to deeply integrate with Windows OS internals
---
#### **UFO³ Galaxy** — Multi-Device AgentOS
📅 *Released: November 2025*
📄 *Paper:* UFO³: Weaving the Digital Agent Galaxy *(Coming Soon)*
- **Vision**: Cross-device orchestration at scale
- **Architecture**: Constellation-based distributed DAG orchestration
- **Innovations**:
- ✅ **Task Constellation** (dynamic DAG decomposition)
- ✅ **Asynchronous parallel execution** across devices
- ✅ **Event-driven coordination** with formal safety guarantees
- ✅ **Dual-mode DAG evolution** (creation + editing)
- ✅ **Agent Interaction Protocol** (persistent WebSocket)
- ✅ **Heterogeneous device support** (Windows, Linux, macOS)
- **Scope**: Multi-device workflows across platforms
- **Capability**: Orchestrate 10+ devices simultaneously
**Key Innovation:** First LLM-powered multi-device orchestration framework with provable correctness
---
### Architecture Evolution
#### UFO v1 Architecture
**Multi-Agent (GUI-Only)**
```
User Request
↓
HostAgent
↓
AppAgent 1, 2, 3...
↓
Windows Apps (GUI)
```
**Capabilities:**
- Multi-app workflows
- Pure screenshot + click/type
- No API integration
- Single device
#### UFO² Architecture
**Two-Tier Hierarchy (Hybrid)**
```
User Request
↓
HostAgent
↓
AppAgent 1, 2, 3...
↓
Windows Apps (GUI + API)
```
**Capabilities:**
- Multi-app workflows
- Desktop orchestration
- Hybrid GUI–API execution
- Deep OS integration
- Single device
#### UFO³ Galaxy Architecture
**Constellation Model (Distributed)**
```
User Request
↓
ConstellationAgent
↓
Task Constellation (DAG)
↓
Device 1, 2, 3... (UFO² instances)
↓
Cross-Platform Apps
```
**Capabilities:**
- Multi-device workflows
- Parallel execution
- Dynamic adaptation
- Heterogeneous platforms
---
## 🎯 When to Use Which?
### Use **UFO²** (Desktop AgentOS) When:
✅ You're automating tasks on a **single Windows desktop**
✅ You need **deep Windows integration** (Office, File Explorer, etc.)
✅ You want **fast, simple execution** without network overhead
✅ You're learning agent automation basics
✅ Your workflow is entirely **local** (no cross-device dependencies)
**Examples:**
- "Create a PowerPoint presentation from this Excel data"
- "Organize my Downloads folder by file type"
- "Send emails to all contacts in this spreadsheet"
---
### Use **UFO³ Galaxy** When:
✅ Your workflow spans **multiple devices** (Windows, Linux, servers)
✅ You need **parallel task execution** for performance
✅ You have **complex dependencies** between subtasks
✅ You want **dynamic workflow adaptation** based on results
✅ You need **fault tolerance** and automatic recovery
✅ You're orchestrating **heterogeneous systems** (desktop + server + cloud)
**Examples:**
- "Clone repo on my laptop, build Docker image on GPU server, deploy to staging, run tests on CI cluster"
- "Fetch data from cloud storage, preprocess on Linux workstation, train model on A100 node, visualize on my Windows machine"
- "Collect logs from all Linux servers, analyze for errors, generate report on Windows"
---
### Can You Use Both?
**Yes!** UFO² can run as a **device agent** in the Galaxy:
```
Galaxy (Orchestrator)
├── Windows Device (UFO² instance)
├── Linux Device (UFO² instance)
└── Server Device (UFO² instance)
```
This is the **recommended hybrid approach** for complex workflows.
---
## 🔄 Key Concept Mapping
Understanding how UFO² concepts map to Galaxy:
| UFO² Concept | Galaxy Equivalent | Relationship |
|--------------|-------------------|--------------|
| **HostAgent** | **ConstellationAgent** | Global orchestrator (but across devices) |
| **AppAgent** | **Device Agent (HostAgent)** | Local executor on each device |
| **Session** | **GalaxySession** | Workflow execution context |
| **Round** | **Constellation Round** | Orchestration iteration |
| **Action** | **TaskStar** | Executable unit (but on specific device) |
| **Blackboard** | **Task Results** | Inter-task communication |
| **Config File** | `config/ufo/` → `config/galaxy/` | Configuration location |
| **Execution Mode** | `python -m ufo.server.app --port ` | Device runs as WebSocket server |
### Architecture Translation
**UFO² (Single Device):**
```python
# UFO² executes locally
python -m ufo --task "Create report from data.xlsx"
# HostAgent coordinates AppAgents on one desktop
HostAgent
├── ExcelAgent (data.xlsx)
├── WordAgent (report.docx)
└── OutlookAgent (send email)
```
**Galaxy (Multi-Device):**
```python
# Galaxy orchestrates across devices
python -m galaxy --request "Create report from data on Server, generate PDF on Windows"
# ConstellationAgent creates DAG, assigns to devices
ConstellationAgent
└── TaskConstellation (DAG)
├── TaskStar-1: Fetch data → Linux Server
├── TaskStar-2: Process → GPU Workstation
└── TaskStar-3: Generate PDF → Windows Desktop
```
---
## ⚙️ Configuration Migration
### Step 1: Preserve UFO² Configuration
**Keep your existing UFO² config** — you'll use it for device agents:
```
config/ufo/
├── agents.yaml # LLM config for device agents
├── app_agent.yaml # AppAgent settings
├── host_agent.yaml # HostAgent settings
└── ...
```
**No changes needed** — each Galaxy device will use its own UFO² config.
---
### Step 2: Create Galaxy Configuration
Galaxy adds **new orchestration-level config**:
#### A. ConstellationAgent LLM Config
```bash
# Copy template
copy config\galaxy\agent.yaml.template config\galaxy\agent.yaml
```
Edit `config/galaxy/agent.yaml`:
```yaml
# ConstellationAgent LLM (orchestrator)
CONSTELLATION_AGENT:
API_TYPE: "openai" # or "azure", "qwen", etc.
API_BASE: "https://api.openai.com/v1"
API_KEY: "sk-your-api-key-here"
API_MODEL: "gpt-4o"
API_VERSION: null
# Optional: Use different model for orchestration
# Recommended: Use GPT-4o or Claude for complex DAG reasoning
```
---
#### B. Device Pool Configuration
**New in Galaxy:** Define all available devices
```bash
# Create device registry
notepad config\galaxy\devices.yaml
```
```yaml
devices:
# Your Windows desktop (existing UFO² instance)
- device_id: "my_windows_desktop"
server_url: "ws://localhost:5005/ws"
os: "windows"
capabilities:
- "office_applications" # Excel, Word, PowerPoint
- "web_browsing"
- "file_management"
metadata:
location: "local"
os: "windows"
performance: "high"
auto_connect: true
max_retries: 5
# Linux workstation
- device_id: "linux_workstation"
server_url: "ws://192.168.1.100:5001/ws"
os: "linux"
capabilities:
- "python"
- "docker"
- "server"
metadata:
location: "office"
os: "ubuntu_22.04"
performance: "high"
gpu: "nvidia_a100"
auto_connect: true
# GPU server
- device_id: "gpu_server"
server_url: "ws://192.168.1.200:5002/ws"
os: "linux"
capabilities:
- "machine_learning"
- "cuda"
- "docker"
metadata:
os: "centos_7"
gpu: "nvidia_v100"
performance: "ultra"
```
**Capability Matching:** ConstellationAgent uses these capabilities to assign tasks intelligently.
---
#### C. Constellation Runtime Config
```bash
notepad config\galaxy\constellation.yaml
```
```yaml
# Constellation Orchestration Settings
CONSTELLATION_ID: "my_constellation"
HEARTBEAT_INTERVAL: 30.0 # Device health check (seconds)
RECONNECT_DELAY: 5.0 # Auto-reconnect delay
MAX_CONCURRENT_TASKS: 6 # Parallel task limit
MAX_STEP: 15 # Max orchestration rounds
# Device Configuration
DEVICE_INFO: "config/galaxy/devices.yaml"
# Logging
LOG_TO_MARKDOWN: true # Generate trajectory reports
```
---
## 🚀 Migration Steps
### Option 1: Keep UFO² for Local, Add Galaxy for Multi-Device
**Best for:** Gradual adoption, maintaining existing workflows
1. **Continue using UFO² for single-device tasks**
```bash
python -m ufo --task "Your local task"
```
2. **Use Galaxy only when you need multi-device orchestration**
```bash
python -m galaxy --request "Your cross-device task"
```
3. **No migration required** — both coexist independently
---
### Option 2: Convert UFO² Instance to Galaxy Device
**Best for:** Leveraging Galaxy's orchestration for all workflows
#### Step 1: Start UFO² as Agent Server
**On each device** (Windows, Linux, etc.), run UFO² server:
```bash
# Windows Desktop
python -m ufo.server.app --port 5005
# Linux Workstation
python -m ufo.server.app --port 5001
# GPU Server
python -m ufo.server.app --port 5002
```
**What this does:**
- Starts WebSocket server on the device
- Listens for task assignments from Galaxy
- Uses existing UFO² agents (HostAgent/AppAgent) for local execution
- Reports results back to ConstellationClient
---
#### Step 2: Configure Galaxy Client
Create `config/galaxy/devices.yaml` with all your devices (see Configuration section above).
---
#### Step 3: Launch Galaxy Client
```bash
# Interactive mode
python -m galaxy --interactive
# Direct request
python -m galaxy --request "Clone repo on laptop, build on server, test on Windows"
```
**What happens:**
1. ConstellationAgent decomposes request into DAG
2. TaskStars assigned to devices based on capabilities
3. Devices execute tasks using their local UFO² agents
4. Results aggregated and presented to user
---
### Option 3: Programmatic Migration
**Best for:** Custom workflows, CI/CD integration
#### UFO² API (Before):
```python
from ufo.module.session_pool import SessionFactory, SessionPool
import asyncio
async def main():
# Create UFO² session on local device
sessions = SessionFactory().create_session(
task="my_task",
mode="normal",
plan="",
request="Create a presentation from data.xlsx"
)
# Run session
pool = SessionPool(sessions)
await pool.run_all()
asyncio.run(main())
```
#### Galaxy API (After):
```python
from galaxy import GalaxyClient
import asyncio
async def main():
# Galaxy session coordinating multiple devices
client = GalaxyClient(session_name="my_workflow")
await client.initialize()
result = await client.process_request(
"Clone repo on laptop, build on server, test on Windows"
)
print(f"Workflow completed: {result}")
await client.shutdown()
asyncio.run(main())
```
**Key Differences:**
- Both are **async** (UFO² v2.0+ uses asyncio)
- UFO²: Uses `SessionFactory` + `SessionPool` pattern
- Galaxy: Uses `GalaxyClient` for multi-device orchestration
- Galaxy returns **constellation results** (multi-device)
- Galaxy requires **device registration** first
---
## 📊 Feature Comparison
### Preserved UFO² Features in Galaxy
When running UFO² as a Galaxy device, you **keep all UFO² capabilities**:
| UFO² Feature | Available in Galaxy Device? | Notes |
|--------------|----------------------------|-------|
| ✅ Hybrid GUI–API execution | ✅ Yes | Each device uses its native UFO² agent |
| ✅ Windows UIA/Win32/COM | ✅ Yes | Full OS integration preserved |
| ✅ MCP server integration | ✅ Yes | Devices can use custom MCP servers |
| ✅ Continuous learning | ✅ Yes | Each device maintains its own RAG |
| ✅ Picture-in-Picture | ✅ Yes | Non-disruptive execution on each device |
| ✅ AppAgent specialization | ✅ Yes | HostAgent manages local AppAgents |
---
### New Galaxy-Only Features
| Feature | Description | Benefit |
|---------|-------------|---------|
| **Task Constellation** | DAG-based task decomposition | Complex workflow planning |
| **Parallel Execution** | Asynchronous multi-device tasks | 3-5x faster for parallelizable work |
| **Dynamic Adaptation** | Runtime DAG modification | Self-healing workflows |
| **Device Assignment** | Capability-based task placement | Optimal resource utilization |
| **Cross-Platform** | Windows + Linux + macOS support | Heterogeneous orchestration |
| **Event-Driven Coordination** | Observer pattern for task events | Reactive workflow control |
| **Formal Safety Guarantees** | I1-I3 invariants | Provably correct concurrent execution |
---
## 🛠️ Practical Examples
### Example 1: Simple Local Task
**UFO² (Before):**
```bash
python -m ufo --task "Create a presentation from data.xlsx"
```
**Galaxy (After) — Option A: Keep UFO²**
```bash
# No change needed — continue using UFO² for local tasks
python -m ufo --task "Create a presentation from data.xlsx"
```
**Galaxy (After) — Option B: Use Galaxy**
```bash
# Galaxy will assign to local Windows device automatically
python -m galaxy --request "Create a presentation from data.xlsx on my desktop"
```
**When to use which?**
- Use UFO² if you only have one Windows desktop (simpler)
- Use Galaxy if you want logging/monitoring features
---
### Example 2: Cross-Device Workflow
**UFO² (Before):**
```bash
# ❌ Not possible — UFO² is single-device only
# You'd need to manually:
# 1. SSH to server
# 2. Run build command
# 3. Copy results back
# 4. Open locally
```
**Galaxy (After):**
```bash
python -m galaxy --request \
"Clone https://github.com/myrepo on laptop, \
build Docker image on gpu_server, \
deploy to staging server, \
open logs on my Windows desktop"
```
**Galaxy automatically:**
1. Creates 4-task DAG
2. Assigns tasks to capable devices
3. Executes in parallel where possible
4. Streams results back
---
### Example 3: Data Pipeline
**UFO² (Before):**
```python
# UFO² requires manual orchestration across multiple steps
from ufo.module.session_pool import SessionFactory, SessionPool
import asyncio
async def main():
# Step 1: Fetch data (local)
sessions_1 = SessionFactory().create_session(
task="fetch_data",
mode="normal",
plan="",
request="Download dataset from cloud storage"
)
pool_1 = SessionPool(sessions_1)
await pool_1.run_all()
# Step 2: Manually transfer to server
# scp data.csv user@server:/data/
# Step 3: SSH and run processing
# ssh server "python process.py"
# Step 4: Manually copy results back
# scp server:/output/results.csv .
# Step 5: Visualize locally
sessions_2 = SessionFactory().create_session(
task="visualize",
mode="normal",
plan="",
request="Create charts from results.csv"
)
pool_2 = SessionPool(sessions_2)
await pool_2.run_all()
asyncio.run(main())
```
**Galaxy (After):**
```python
import asyncio
from galaxy import GalaxyClient
async def main():
client = GalaxyClient(session_name="data_pipeline")
await client.initialize()
# Single request — Galaxy handles orchestration
await client.process_request(
"Fetch dataset from cloud to laptop, "
"preprocess on linux_workstation, "
"train model on gpu_server, "
"visualize results on my Windows desktop"
)
await client.shutdown()
asyncio.run(main())
```
**Galaxy automatically:**
- Creates dependency chain
- Transfers data between devices
- Executes pipeline stages in order
- Handles failures with retries
---
## 🎓 Learning Path
### For UFO² Users
1. **Week 1: Understand Concepts**
- Read [Galaxy Overview](../galaxy/overview.md)
- Understand Task Constellation and DAG model
- Compare with UFO² two-tier hierarchy
2. **Week 2: Hands-On**
- Set up one Windows device as Galaxy agent
- Run simple multi-step workflow
- Compare logs: UFO² vs Galaxy
3. **Week 3: Multi-Device**
- Add Linux device to pool
- Create cross-platform workflow
- Monitor with trajectory reports
4. **Week 4: Advanced**
- Build custom device capabilities
- Integrate MCP servers across devices
- Optimize task assignment logic
---
## 📚 Related Documentation
### Migration Resources
- **[Galaxy Quick Start](./quick_start_galaxy.md)** — Step-by-step Galaxy setup
- **[UFO² Quick Start](./quick_start_ufo2.md)** — UFO² reference
- **[Device Configuration](../configuration/system/galaxy_devices.md)** — Device pool setup
- **[Agent Registration](../galaxy/agent_registration/overview.md)** — How devices join Galaxy
### Architecture Deep Dives
- **[Galaxy Overview](../galaxy/overview.md)** — Constellation architecture
- **[UFO² Overview](../ufo2/overview.md)** — Desktop AgentOS design
- **[Constellation Agent](../galaxy/constellation_agent/overview.md)** — DAG orchestration
- **[Task Constellation](../galaxy/constellation/overview.md)** — DAG structure
### Operational Guides
- **[Trajectory Report](../galaxy/evaluation/trajectory_report.md)** — Execution logs
- **[Performance Metrics](../galaxy/evaluation/performance_metrics.md)** — Monitoring
- **[AIP Protocol](../aip/overview.md)** — Device communication
---
## 🤝 Getting Help
### Common Questions
**Q: Can I still use UFO² after migrating to Galaxy?**
A: Yes! They coexist. Use UFO² for simple local tasks, Galaxy for multi-device workflows.
**Q: Do I need to rewrite my custom agents?**
A: No. Existing UFO² agents work as-is when running as Galaxy devices.
**Q: Is Galaxy production-ready?**
A: Galaxy is in active development. UFO² is more mature for mission-critical single-device workflows.
**Q: Can I mix Windows and Linux devices?**
A: Yes! That's Galaxy's key feature. Each device uses its native UFO² implementation.
**Q: How do I debug failed cross-device workflows?**
A: Check `logs/galaxy//output.md` for step-by-step execution details and DAG visualizations.
---
## 🚦 Migration Checklist
Use this checklist to track your migration progress:
- [ ] **Understand UFO evolution** (v1 → UFO² → Galaxy)
- [ ] **Decide migration strategy** (hybrid vs full Galaxy)
- [ ] **Preserve UFO² config** (`config/ufo/` untouched)
- [ ] **Create Galaxy config** (`config/galaxy/agent.yaml`, `devices.yaml`)
- [ ] **Start devices as servers** (each device runs `python -m ufo.server.app --port `)
- [ ] **Test single-device workflow** (verify connectivity)
- [ ] **Test multi-device workflow** (cross-platform task)
- [ ] **Review trajectory reports** (`logs/galaxy/*/output.md`)
- [ ] **Compare performance** (UFO² vs Galaxy for your use cases)
- [ ] **Update automation scripts** (if using programmatic API)
- [ ] **Train team** (share this guide!)
---
**🎉 Congratulations!** You're now ready to leverage the full power of UFO³ Galaxy's multi-device orchestration while preserving your existing UFO² workflows.
For questions or issues, please open an issue on [GitHub](https://github.com/microsoft/UFO) or check the [documentation](https://microsoft.github.io/UFO/).
---
### Documents Docs Getting Started More Guidance (`documents/docs/getting_started/more_guidance.md`)
# More Guidance
This page provides additional guidance and resources for different user types and use cases.
---
## 🎯 For End Users
If you want to use UFO³ to automate your tasks on Windows, Linux, or across multiple devices, here's your learning path:
### 1. Getting Started (5-10 minutes)
Choose your path based on your needs:
| Your Goal | Start Here | Time |
|-----------|-----------|------|
| **Automate Windows desktop tasks** | [UFO² Quick Start](quick_start_ufo2.md) | 5 min |
| **Manage Linux servers** | [Linux Quick Start](quick_start_linux.md) | 10 min |
| **Orchestrate multiple devices** | [Galaxy Quick Start](quick_start_galaxy.md) | 10 min |
### 2. Configure Your Environment (10-20 minutes)
After installation, customize UFO³ to your needs:
**Essential Configuration:**
- **[Agent Configuration](../configuration/system/agents_config.md)** - Set up LLM API keys (OpenAI, Azure, Gemini, Claude, etc.)
- **[System Configuration](../configuration/system/system_config.md)** - Adjust runtime settings (step limits, timeouts, logging)
**Optional Enhancements:**
- **[RAG Configuration](../configuration/system/rag_config.md)** - Add external knowledge sources:
- Offline help documents
- Bing search integration
- Experience learning from past tasks
- User demonstrations
- **[MCP Configuration](../configuration/system/mcp_reference.md)** - Enable tool servers for:
- Better Office automation
- Linux command execution
- Custom tool integration
> **💡 Configuration Tip:** Start with default settings and adjust only what you need. See [Configuration Overview](../configuration/system/overview.md) for the big picture.
### 3. Learn Core Features (20-30 minutes)
**For UFO² Users (Windows Desktop Automation):**
| Feature | Documentation | What It Does |
|---------|---------------|--------------|
| **Hybrid GUI-API Execution** | [Hybrid Actions](../ufo2/core_features/hybrid_actions.md) | Combines UI automation with native API calls for faster, more reliable execution |
| **Knowledge Substrate** | [Knowledge Overview](../ufo2/core_features/knowledge_substrate/overview.md) | Augments agents with external knowledge (docs, search, experience) |
| **MCP Integration** | [MCP Overview](../mcp/overview.md) | Extends capabilities with custom tools and Office APIs |
**For Galaxy Users (Multi-Device Orchestration):**
| Feature | Documentation | What It Does |
|---------|---------------|--------------|
| **Task Constellation** | [Constellation Overview](../galaxy/constellation_orchestrator/overview.md) | Decomposes tasks into parallel DAGs across devices |
| **Device Capabilities** | [Galaxy Devices Config](../configuration/system/galaxy_devices.md) | Routes tasks based on device capabilities and metadata |
| **Asynchronous Execution** | [Constellation Overview](../galaxy/constellation/overview.md) | Executes subtasks in parallel for faster completion |
| **Agent Interaction Protocol** | [AIP Overview](../aip/overview.md) | Enables persistent WebSocket communication between devices |
### 4. Troubleshooting & Support
**When Things Go Wrong:**
1. **Check the [FAQ](../faq.md)** - Common issues and solutions
2. **Review logs** - Located in `logs//`:
```
logs/my-task-2025-11-11/
├── request.log # Request logs
├── response.log # Response logs
├── action_step*.png # Screenshots at each step
└── action_step*_annotated.png # Annotated screenshots
```
3. **Validate configuration:**
```bash
python -m ufo.tools.validate_config ufo --show-config
```
4. **Enable debug logging:**
```yaml
# config/ufo/system.yaml
LOG_LEVEL: "DEBUG"
```
**Get Help:**
- **[GitHub Discussions](https://github.com/microsoft/UFO/discussions)** - Ask questions, share tips
- **[GitHub Issues](https://github.com/microsoft/UFO/issues)** - Report bugs, request features
- **Email:** ufo-agent@microsoft.com
---
## 👨💻 For Developers
If you want to contribute to UFO³ or build extensions, here's your development guide:
### 1. Understand the Architecture (30-60 minutes)
**Start with the big picture:**
- **[Project Structure](../project_directory_structure.md)** - Codebase organization and component roles
- **[Configuration Architecture](../configuration/system/overview.md)** - New modular config system design
**Deep dive into core components:**
| Component | Documentation | What to Learn |
|-----------|---------------|---------------|
| **Session** | [Session Module](../infrastructure/modules/session.md) | Task lifecycle management, state tracking |
| **Round** | [Round Module](../infrastructure/modules/round.md) | Single agent reasoning cycle |
| **HostAgent** | [HostAgent](../ufo2/host_agent/overview.md) | High-level task planning and app selection |
| **AppAgent** | [AppAgent](../ufo2/app_agent/overview.md) | Low-level action execution |
| **ConstellationAgent** | [ConstellationAgent](../galaxy/constellation_agent/overview.md) | Multi-device task orchestration |
### 2. Set Up Development Environment (15-30 minutes)
**Installation:**
```bash
# Clone the repository
git clone https://github.com/microsoft/UFO.git
cd UFO
# Create development environment
conda create -n ufo-dev python=3.10
conda activate ufo-dev
# Install dependencies (including dev tools)
pip install -r requirements.txt
pip install pytest pytest-cov black flake8 # Testing & linting
```
**Configuration:**
```bash
# Create config files from templates
cp config/ufo/agents.yaml.template config/ufo/agents.yaml
cp config/galaxy/agent.yaml.template config/galaxy/agent.yaml
# Edit with your development API keys
# (Consider using lower-cost models for testing)
```
### 3. Explore the Codebase (1-2 hours)
**Key Directories:**
```
UFO/
├── ufo/ # Core UFO² implementation
│ ├── agents/ # HostAgent, AppAgent
│ ├── automator/ # UI automation engines
│ ├── prompter/ # Prompt management
│ └── module/ # Core modules (Session, Round)
├── galaxy/ # Galaxy orchestration framework
│ ├── agents/ # ConstellationAgent
│ ├── constellation/ # DAG orchestration
│ └── core/ # Core Galaxy infrastructure
├── aip/ # Agent Interaction Protocol
│ ├── protocol/ # Message definitions
│ └── transport/ # WebSocket transport
├── ufo/client/ # Device agents (Windows, Linux)
│ ├── client.py # Generic client
│ └── mcp/ # MCP integration
├── ufo/server/ # Device agent server
│ └── app.py # FastAPI server
└── config/ # Configuration system
├── ufo/ # UFO² configs
└── galaxy/ # Galaxy configs
```
**Entry Points:**
- **UFO² Main:** `ufo/__main__.py`
- **Galaxy Main:** `galaxy/__main__.py`
- **Server:** `ufo/server/app.py`
- **Client:** `ufo/client/client.py`
### 4. Development Workflows
#### Adding a New Feature
1. **Identify the component** to modify (Agent, Module, Automator, etc.)
2. **Read existing code** in that component
3. **Check related tests** in `tests/` directory
4. **Implement your feature** following existing patterns
5. **Add tests** for your feature
6. **Update documentation** if needed
#### Extending Configuration
See **[Extending Configuration](../configuration/system/extending.md)** for:
- Adding custom fields
- Creating new config modules
- Environment-specific overrides
- Plugin configuration patterns
#### Creating Custom MCP Servers
See **[Creating MCP Servers Tutorial](../tutorials/creating_mcp_servers.md)** for:
- MCP server architecture
- Tool definition and registration
- HTTP vs. local vs. stdio servers
- Integration with UFO³
### 5. Testing & Debugging
**Run Tests:**
```bash
# Run all tests
pytest
# Run specific test file
pytest tests/config/test_config_system.py
# Run with coverage
pytest --cov=ufo --cov-report=html
```
**Debug Logging:**
```python
# Add debug logs to your code
import logging
logger = logging.getLogger(__name__)
logger.debug("Debug message with context: %s", variable)
logger.info("Informational message")
logger.warning("Warning message")
logger.error("Error message")
```
**Interactive Debugging:**
```python
# Add breakpoint in code
import pdb; pdb.set_trace()
# Or use VS Code debugger with launch.json
```
### 6. Code Style & Best Practices
**Formatting:**
```bash
# Auto-format with black
black ufo/ galaxy/
# Check style with flake8
flake8 ufo/ galaxy/
```
**Best Practices:**
- ✅ Use type hints: `def process(data: Dict[str, Any]) -> Optional[str]:`
- ✅ Write docstrings for public functions
- ✅ Follow existing code patterns
- ✅ Add comments for complex logic
- ✅ Keep functions focused and modular
- ✅ Handle errors gracefully
- ✅ Write tests for new features
**Configuration Best Practices:**
- ✅ Use typed config access: `config.system.max_step`
- ✅ Provide `.template` files for sensitive configs
- ✅ Document custom fields in YAML comments
- ✅ Use environment variables for secrets: `${OPENAI_API_KEY}`
- ✅ Validate configurations early: `ConfigValidator.validate()`
### 7. Contributing Guidelines
**Before Submitting a PR:**
1. **Test your changes** thoroughly
2. **Update documentation** if needed
3. **Follow code style** (black + flake8)
4. **Write clear commit messages**
5. **Reference related issues** in PR description
**PR Template:**
```markdown
## Description
Brief description of changes
## Type of Change
- [ ] Bug fix
- [ ] New feature
- [ ] Documentation update
- [ ] Refactoring
## Testing
- [ ] Added tests for new functionality
- [ ] All tests pass locally
- [ ] Manual testing completed
## Checklist
- [ ] Code follows project style
- [ ] Documentation updated
- [ ] No breaking changes (or documented)
```
### 8. Advanced Topics
**For Deep Customization:**
- **[Prompt Engineering](../ufo2/prompts/overview.md)** - Customize agent prompts
- **[State Management](../galaxy/constellation/overview.md)** - Constellation state machine internals
- **[Protocol Extensions](../aip/messages.md)** - Extend AIP message types
- **[Custom Automators](../ufo2/core_features/control_detection/overview.md)** - Implement new automation backends
---
## 🎓 Learning Paths
### Path 1: Basic User → Power User
1. ✅ Complete quick start for your platform
2. ✅ Run 5-10 simple automation tasks
3. ✅ Configure RAG for your organization's docs
4. ✅ Enable MCP for better Office automation
5. ✅ Set up experience learning for common tasks
6. ✅ Create custom device configurations (Galaxy)
**Time Investment:** 2-4 hours
**Outcome:** Efficient automation of daily tasks
### Path 2: Power User → Developer
1. ✅ Understand project structure and architecture
2. ✅ Read Session and Round module code
3. ✅ Create a custom MCP server
4. ✅ Add custom metadata to device configs
5. ✅ Contribute documentation improvements
6. ✅ Submit your first bug fix PR
**Time Investment:** 10-20 hours
**Outcome:** Ability to extend and customize UFO³
### Path 3: Developer → Core Contributor
1. ✅ Deep dive into agent implementations
2. ✅ Understand Galaxy orchestration internals
3. ✅ Study AIP protocol and transport layer
4. ✅ Implement a new agent capability
5. ✅ Add support for a new LLM provider
6. ✅ Contribute major features or refactorings
**Time Investment:** 40+ hours
**Outcome:** Core contributor to UFO³ project
---
## 📚 Additional Resources
### Documentation Hubs
| Topic | Link | Description |
|-------|------|-------------|
| **Getting Started** | [Getting Started Index](../index.md#getting-started) | All quick start guides |
| **Configuration** | [Configuration Overview](../configuration/system/overview.md) | Complete config system documentation |
| **Architecture** | [Galaxy Overview](../galaxy/overview.md), [UFO² Overview](../ufo2/overview.md) | System architecture and design |
| **API Reference** | [Agent APIs](../infrastructure/agents/overview.md) | Agent interfaces and APIs |
| **Tutorials** | [Creating Device Agents](../tutorials/creating_device_agent/index.md) | Step-by-step guides |
### Community Resources
- **[GitHub Repository](https://github.com/microsoft/UFO)** - Source code and releases
- **[GitHub Discussions](https://github.com/microsoft/UFO/discussions)** - Q&A and community
- **[GitHub Issues](https://github.com/microsoft/UFO/issues)** - Bug reports and features
- **[Project Website](https://microsoft.github.io/UFO/)** - Official website
### Research Papers
- **UFO v1** (Feb 2024): [A UI-Focused Agent for Windows OS Interaction](https://arxiv.org/abs/2402.07939)
- **UFO² v2** (Apr 2025): [A Windows Agent for Seamless OS Interaction](https://arxiv.org/abs/2504.14603)
- **UFO³ Galaxy** (Nov 2025): UFO³: Weaving the Digital Agent Galaxy *(Coming Soon)*
---
## 🆘 Need More Help?
- **Can't find what you're looking for?** Check the [FAQ](../faq.md)
- **Still stuck?** Ask on [GitHub Discussions](https://github.com/microsoft/UFO/discussions)
- **Found a bug?** Open an issue on [GitHub Issues](https://github.com/microsoft/UFO/issues)
- **Want to contribute?** Read the [Contributing Guidelines](https://github.com/microsoft/UFO/blob/main/CONTRIBUTING.md)
**Happy automating!** 🚀
---
### Documents Docs Getting Started Quick Start Galaxy (`documents/docs/getting_started/quick_start_galaxy.md`)
# Quick Start Guide - UFO³ Galaxy
Welcome to **UFO³ Galaxy** – the Multi-Device AgentOS! This guide will help you orchestrate complex cross-platform workflows across multiple devices in just a few steps.
**What is UFO³ Galaxy?**
UFO³ Galaxy is a multi-tier orchestration framework that coordinates distributed agents across Windows and Linux devices. It enables complex workflows that span multiple machines, combining desktop automation, server operations, and heterogeneous device capabilities into unified task execution.
---
## 🛠️ Step 1: Installation
### Requirements
- **Python** >= 3.10
- **Windows OS** >= 10 (for Windows agents)
- **Linux** (for Linux agents)
- **Git** (for cloning the repository)
- **Network connectivity** between all devices
### Installation Steps
```powershell
# [Optional] Create conda environment
conda create -n ufo python=3.10
conda activate ufo
# Clone the repository
git clone https://github.com/microsoft/UFO.git
cd UFO
# Install dependencies
pip install -r requirements.txt
```
> **💡 Tip:** If you want to use Qwen as your LLM, uncomment the related libraries in `requirements.txt` before installing.
---
## ⚙️ Step 2: Configure ConstellationAgent LLM
UFO³ Galaxy uses a **ConstellationAgent** that orchestrates all device agents. You need to configure its LLM settings.
### Configure Constellation Agent
```powershell
# Copy template to create constellation agent config
copy config\galaxy\agent.yaml.template config\galaxy\agent.yaml
notepad config\galaxy\agent.yaml # Edit your LLM API credentials
```
**Configuration File Location:**
```
config/galaxy/
├── agent.yaml.template # Template - COPY THIS
├── agent.yaml # Your config with API keys (DO NOT commit)
└── devices.yaml # Device pool configuration (Step 4)
```
### LLM Configuration Examples
#### Azure OpenAI Configuration
**Edit `config/galaxy/agent.yaml`:**
```yaml
CONSTELLATION_AGENT:
REASONING_MODEL: false
API_TYPE: "aoai"
API_BASE: "https://YOUR_RESOURCE.openai.azure.com"
API_KEY: "YOUR_AOAI_KEY"
API_VERSION: "2024-02-15-preview"
API_MODEL: "gpt-4o"
API_DEPLOYMENT_ID: "YOUR_DEPLOYMENT_ID"
```
> **ℹ️ More LLM Options:** Galaxy supports various LLM providers including Qwen, Gemini, Claude, DeepSeek, and more. See the [Model Configuration Guide](../configuration/models/overview.md) for complete details.
---
# Prompt configurations (use defaults)
CONSTELLATION_CREATION_PROMPT: "galaxy/prompts/constellation/share/constellation_creation.yaml"
CONSTELLATION_EDITING_PROMPT: "galaxy/prompts/constellation/share/constellation_editing.yaml"
CONSTELLATION_CREATION_EXAMPLE_PROMPT: "galaxy/prompts/constellation/examples/constellation_creation_example.yaml"
CONSTELLATION_EDITING_EXAMPLE_PROMPT: "galaxy/prompts/constellation/examples/constellation_editing_example.yaml"
```
#### OpenAI Configuration
```yaml
CONSTELLATION_AGENT:
REASONING_MODEL: false
API_TYPE: "openai"
API_BASE: "https://api.openai.com/v1/chat/completions"
API_KEY: "sk-YOUR_KEY_HERE"
API_VERSION: "2025-02-01-preview"
API_MODEL: "gpt-4o"
# Prompt configurations (use defaults)
CONSTELLATION_CREATION_PROMPT: "galaxy/prompts/constellation/share/constellation_creation.yaml"
CONSTELLATION_EDITING_PROMPT: "galaxy/prompts/constellation/share/constellation_editing.yaml"
CONSTELLATION_CREATION_EXAMPLE_PROMPT: "galaxy/prompts/constellation/examples/constellation_creation_example.yaml"
CONSTELLATION_EDITING_EXAMPLE_PROMPT: "galaxy/prompts/constellation/examples/constellation_editing_example.yaml"
```
/* Detailed source-code truncated for AI context efficiency. */
```powershell
# Terminal 1: Start UFO² Server
python -m ufo.server.app --port 5000
# Terminal 2: Start UFO² Client (connect to server)
python -m ufo.client.client `
--ws `
--ws-server ws://localhost:5000/ws `
--client-id windows_device_1 `
--platform windows
```
> **💡 Important:** Always include `--platform windows` for Windows devices and `--platform linux` for Linux devices!
### Example: Quick Linux Device Setup
**On your Linux machine:**
```bash
# Terminal 1: Start Device Agent Server
python -m ufo.server.app --port 5001
# Terminal 2: Start Linux Client (connect to server)
python -m ufo.client.client \
--ws \
--ws-server ws://localhost:5001/ws \
--client-id linux_device_1 \
--platform linux
# Terminal 3: Start HTTP MCP Server (for Linux tools)
python -m ufo.client.mcp.http_servers.linux_mcp_server
```
> **💡 Note:** For detailed Mobile Agent setup with ADB and Android device configuration, see [Mobile Quick Start](quick_start_mobile.md).
---
## 🔌 Step 4: Configure Device Pool
After starting your device agents, register them in Galaxy's device pool configuration.
### Option 1: Add Devices via Configuration File
### Edit Device Configuration
```powershell
notepad config\galaxy\devices.yaml
```
### Example Device Pool Configuration
```yaml
# Device Configuration for Galaxy
# Each device agent must be registered here
devices:
# Windows Device (UFO²)
- device_id: "windows_device_1" # Must match --client-id
server_url: "ws://localhost:5000/ws" # Must match server WebSocket URL
os: "windows"
capabilities:
- "desktop_automation"
- "office_applications"
- "excel"
- "word"
- "outlook"
- "email"
- "web_browsing"
metadata:
os: "windows"
version: "11"
performance: "high"
installed_apps:
- "Microsoft Excel"
- "Microsoft Word"
- "Microsoft Outlook"
- "Google Chrome"
description: "Primary Windows desktop for office automation"
auto_connect: true
max_retries: 5
# Linux Device
- device_id: "linux_device_1" # Must match --client-id
server_url: "ws://localhost:5001/ws" # Must match server WebSocket URL
os: "linux"
capabilities:
- "server_management"
- "log_analysis"
- "file_operations"
- "database_operations"
metadata:
os: "linux"
performance: "medium"
logs_file_path: "/var/log/myapp/app.log"
dev_path: "/home/user/projects/"
warning_log_pattern: "WARN"
error_log_pattern: "ERROR|FATAL"
description: "Development server for backend operations"
auto_connect: true
max_retries: 5
# Mobile Device (Android)
- device_id: "mobile_phone_1" # Must match --client-id
server_url: "ws://localhost:5001/ws" # Must match server WebSocket URL
os: "mobile"
capabilities:
- "mobile"
- "android"
- "ui_automation"
- "messaging"
- "camera"
- "location"
metadata:
os: "mobile"
device_type: "phone"
android_version: "13"
screen_size: "1080x2400"
installed_apps:
- "com.android.chrome"
- "com.google.android.apps.maps"
- "com.whatsapp"
description: "Android phone for mobile automation and testing"
auto_connect: true
max_retries: 5
```
> **⚠️ Critical:** IDs and URLs must match exactly:
>
> - `device_id` must exactly match the `--client-id` flag
> - `server_url` must exactly match the server WebSocket URL
> - Otherwise, Galaxy cannot control the device!
**Complete Configuration Guide:** For detailed information about all configuration options, capabilities, and metadata, see [Galaxy Devices Configuration](../configuration/system/galaxy_devices.md).
### Option 2: Add Devices via WebUI (When Using --webui Mode)
If you start Galaxy with the `--webui` flag (see Step 5), you can add new device agents directly through the web interface without editing configuration files.
**Steps to Add Device via WebUI:**
1. **Launch Galaxy with WebUI** (as shown in Step 5):
```powershell
python -m galaxy --webui
```
/* Detailed source-code truncated for AI context efficiency. */
```powershell
# Assume you are in the cloned UFO folder
python -m galaxy --webui
```
This will start the Galaxy server with WebUI and automatically open your browser to the interactive interface:
🎨 Galaxy WebUI - Interactive constellation visualization and chat interface
**WebUI Features:**
- 🗣️ **Chat Interface**: Submit requests and interact with ConstellationAgent in real-time
- 📊 **Live DAG Visualization**: Watch task constellation formation and execution
- 🎯 **Task Status Tracking**: Monitor each TaskStar's progress and completion
- 🔄 **Dynamic Updates**: See constellation evolution as tasks complete
- 📱 **Responsive Design**: Works on desktop and tablet devices
**Default URL:** `http://127.0.0.1:8000` (automatically finds next available port if 8000 is occupied)
!!!tip "API Key"
An API key is generated automatically on startup and printed to the console. The built-in frontend receives the key automatically. If you need to call the API from external tools, use the printed key in the `X-API-Key` header for HTTP requests or as the `token` query parameter for WebSocket connections.
---
### 💬 Launch Galaxy - Interactive Terminal Mode
Start Galaxy in interactive mode where you can enter requests dynamically:
```powershell
# Assume you are in the cloned UFO folder
python -m galaxy --interactive
```
**Expected Output:**
```
🌌 Welcome to UFO³ Galaxy Framework
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Multi-Device AI Orchestration System
📡 Initializing Galaxy...
✅ ConstellationAgent initialized
✅ Connected to device: windows_device_1 (windows)
✅ Connected to device: linux_device_1 (linux)
🌟 Galaxy Ready - 2 devices online
Please enter your request 🛸:
```
---
### ⚡ Launch Galaxy - Direct Request Mode
Invoke Galaxy with a specific request directly:
```powershell
python -m galaxy --request "Your task description here"
```
**Example:**
```powershell
python -m galaxy --request "Generate a sales report from the database and create an Excel dashboard"
```
---
### 🎬 Launch Galaxy - Demo Mode
Run Galaxy in demo mode to see example workflows:
```powershell
python -m galaxy --demo
```
---
## 🎯 Step 6: Try Your First Multi-Device Workflow
### Example 1: Simple Cross-Platform Task
**User Request:**
> "Check the server logs for errors and email me a summary"
**Galaxy orchestrates:**
1. **Linux Device**: Analyze server logs for error patterns
2. **Windows Device**: Open Outlook, create email with log summary
3. **Windows Device**: Send email
**How to run:**
```powershell
python -m galaxy --request "Check the server logs for errors and email me a summary"
```
### Example 2: Data Processing Pipeline
**User Request:**
> "Export sales data from the database, create an Excel report with charts, and email it to the team"
**Galaxy orchestrates:**
1. **Linux Device**: Query database, export CSV
2. **Windows Device**: Open Excel, import CSV, create charts
3. **Windows Device**: Open Outlook, attach Excel file, send email
**How to run:**
```powershell
python -m galaxy --request "Export sales data from the database, create an Excel report with charts, and email it to the team"
```
### Example 3: Multi-Server Monitoring
**User Request:**
> "Check all servers for disk usage and alert if any are above 80%"
**Galaxy orchestrates:**
1. **Linux Device 1**: Check disk usage on server 1
2. **Linux Device 2**: Check disk usage on server 2
3. **Galaxy**: Aggregate results, check thresholds
4. **Windows Device**: Send alert email if needed
---
## 📔 Step 7: Understanding Device Routing
Galaxy uses **capability-based routing** to intelligently assign tasks to appropriate devices.
### How Galaxy Selects Devices
| Factor | Description | Example |
|--------|-------------|---------|
| **Capabilities** | Matches task requirements | `"excel"` → Windows device with Excel |
| **OS Requirement** | Platform-specific tasks | Linux commands → Linux device |
| **Metadata** | Device-specific context | Email task → device with Outlook |
| **Status** | Online and healthy devices only | Skips offline devices |
### Example Task Decomposition
**User Request:**
> "Prepare monthly reports and distribute to team"
**Galaxy Decomposition:**
```yaml
Subtask 1:
Description: "Extract monthly data from database"
Target Device: linux_device_1
Reason: Has "database_operations" capability
Subtask 2:
Description: "Create Excel report with visualizations"
Target Device: windows_device_1
Reason: Has "excel" capability
Subtask 3:
Description: "Email reports to distribution list"
Target Device: windows_device_1
Reason: Has "email" and "outlook" capabilities
```
---
## 🔄 Step 8: Execution Logs
Galaxy automatically saves execution logs, task graphs, and device traces for debugging and analysis.
**Log Location:**
```
./logs//
```
**Log Contents:**
| File/Folder | Description |
|-------------|-------------|
| `constellation/` | DAG visualization and task decomposition |
| `device_logs/` | Individual device execution logs |
| `screenshots/` | Screenshots from Windows devices (if enabled) |
| `task_results/` | Task execution results |
| `request_response.log` | Complete LLM request/response logs |
> **Analyzing Logs:** Use the logs to debug task routing, identify bottlenecks, replay execution flow, and analyze orchestration decisions.
---
## 🔧 Advanced Configuration
### Custom Session Name
```powershell
python -m galaxy --request "Your task" --session-name "my_project"
```
### Custom Output Directory
```powershell
python -m galaxy --request "Your task" --output-dir "./custom_results"
```
### Debug Mode
```powershell
python -m galaxy --interactive --log-level DEBUG
```
### Limit Maximum Rounds
```powershell
python -m galaxy --interactive --max-rounds 20
```
---
## ❓ Troubleshooting
### Issue 1: Device Not Appearing in Galaxy
**Error:** Device not found in configuration
```log
ERROR - Device 'windows_device_1' not found in configuration
```
**Solutions:**
1. Verify `devices.yaml` configuration:
```powershell
notepad config\galaxy\devices.yaml
```
2. Check device ID matches:
- In `devices.yaml`: `device_id: "windows_device_1"`
- In client command: `--client-id windows_device_1`
3. Check server URL matches:
- In `devices.yaml`: `server_url: "ws://localhost:5000/ws"`
- In client command: `--ws-server ws://localhost:5000/ws`
### Issue 2: Device Agent Not Connecting
**Error:** Connection refused
```log
ERROR - [WS] Failed to connect to ws://localhost:5000/ws
Connection refused
```
**Solutions:**
1. Verify server is running:
```powershell
curl http://localhost:5000/api/health
```
2. Check port number is correct:
- Server: `--port 5000`
- Client: `ws://localhost:5000/ws`
3. Ensure platform flag is set:
```powershell
# For Windows devices
--platform windows
# For Linux devices
--platform linux
```
### Issue 3: Galaxy Cannot Find Constellation Agent Config
**Error:** Configuration file not found
```log
ERROR - Cannot find config/galaxy/agent.yaml
```
**Solution:**
```powershell
# Copy template to create configuration file
copy config\galaxy\agent.yaml.template config\galaxy\agent.yaml
# Edit with your LLM credentials
notepad config\galaxy\agent.yaml
```
### Issue 4: Task Not Routed to Expected Device
**Issue:** Wrong device selected for task
**Diagnosis:** Check device capabilities in `devices.yaml`:
```yaml
capabilities:
- "desktop_automation"
- "office_applications"
- "excel" # Required for Excel tasks
- "outlook" # Required for email tasks
```
**Solution:** Add appropriate capabilities to your device configuration.
---
## 📚 Additional Resources
### Core Documentation
**Architecture & Concepts:**
- [Galaxy Overview](../galaxy/overview.md) - System architecture and design principles
- [Constellation Orchestrator](../galaxy/constellation_orchestrator/overview.md) - Task orchestration and DAG management
- [Agent Interaction Protocol (AIP)](../aip/overview.md) - Communication substrate
### Device Agent Setup
**Device Agent Guides:**
- [UFO² as Galaxy Device](../ufo2/as_galaxy_device.md) - Complete Windows device setup
- [Linux as Galaxy Device](../linux/as_galaxy_device.md) - Complete Linux device setup
- [Mobile as Galaxy Device](../mobile/as_galaxy_device.md) - Complete Android device setup
- [UFO² Overview](../ufo2/overview.md) - Windows desktop automation capabilities
- [Linux Agent Overview](../linux/overview.md) - Linux server automation capabilities
- [Mobile Agent Overview](../mobile/overview.md) - Android mobile automation capabilities
### Configuration
**Configuration Guides:**
- [Galaxy Devices Configuration](../configuration/system/galaxy_devices.md) - Complete device pool configuration
- [Galaxy Constellation Configuration](../configuration/system/galaxy_constellation.md) - Runtime settings
- [Agents Configuration](../configuration/system/agents_config.md) - LLM settings for all agents
- [Model Configuration](../configuration/models/overview.md) - Supported LLM providers
### Advanced Features
**Advanced Topics:**
- [Task Constellation](../galaxy/constellation/task_constellation.md) - DAG-based task planning
- [Constellation Orchestrator](../galaxy/constellation_orchestrator/overview.md) - Multi-device orchestration
- [Device Registry](../galaxy/agent_registration/device_registry.md) - Device management
- [Agent Profiles](../galaxy/agent_registration/agent_profile.md) - Multi-source profiling
---
## ❓ Getting Help
- 📖 **Documentation**: [https://microsoft.github.io/UFO/](https://microsoft.github.io/UFO/)
- 🐛 **GitHub Issues**: [https://github.com/microsoft/UFO/issues](https://github.com/microsoft/UFO/issues) (preferred)
- 📧 **Email**: [ufo-agent@microsoft.com](mailto:ufo-agent@microsoft.com)
---
## 🎯 Next Steps
Now that Galaxy is set up, explore these guides to unlock its full potential:
1. **[Add More Devices](../configuration/system/galaxy_devices.md)** - Expand your device pool
2. **[Configure Capabilities](../configuration/system/galaxy_devices.md)** - Optimize task routing
3. **[Constellation Agent](../galaxy/constellation_agent/overview.md)** - Deep dive into orchestration agent
4. **[Advanced Orchestration](../galaxy/constellation_orchestrator/overview.md)** - Deep dive into DAG planning
Happy orchestrating with UFO³ Galaxy! 🌌🚀
---
### Documents Docs Getting Started Quick Start Linux (`documents/docs/getting_started/quick_start_linux.md`)
# ⚡ Quick Start: Linux Agent
Get your Linux device running as a UFO³ device agent in 5 minutes. This guide walks you through server/client configuration and MCP service initialization.
---
## 📋 Prerequisites
Before you begin, ensure you have:
- **Python 3.10+** installed on both server and client machines
- **UFO repository** cloned
- **Network connectivity** between server and client machines
- **Linux machine** for task execution (client)
- **Terminal access** (bash, ssh, etc.)
- **LLM configured** in `config/ufo/agents.yaml` (same as AppAgent)
| Component | Minimum Version | Verification Command |
|-----------|----------------|---------------------|
| Python | 3.10 | `python3 --version` |
| Git | 2.0+ | `git --version` |
| Network | N/A | `ping ` |
| LLM API Key | N/A | Check `config/ufo/agents.yaml` |
> **⚠️ LLM Configuration Required:** The Linux Agent shares the same LLM configuration with the AppAgent. Before starting, ensure you have configured your LLM provider (OpenAI, Azure OpenAI, Gemini, Claude, etc.) and added your API keys to `config/ufo/agents.yaml`. See [Model Setup Guide](../configuration/models/overview.md) for detailed instructions.
---
## 📦 Step 1: Install Dependencies
Install all dependencies from the requirements file:
```bash
pip install -r requirements.txt
```
**Verify installation:**
```bash
python3 -c "import ufo; print('✅ UFO² installed successfully')"
```
> **Tip:** For production deployments, use a virtual environment to isolate dependencies:
>
> ```bash
> python3 -m venv venv
> source venv/bin/activate # Linux/macOS
> pip install -r requirements.txt
> ```
---
## 🖥️ Step 2: Start Device Agent Server
**Server Component:** The Device Agent Server is the central hub that manages connections from client devices and dispatches tasks. It can run on any machine (Linux, Windows, or remote server).
### Server Machine Setup
You can run the server on:
- ✅ Same machine as the client (localhost setup for testing)
- ✅ Different machine on the same network
- ✅ Remote server (requires proper network routing/SSH tunneling)
### Basic Server Startup
On the server machine, run:
```bash
python -m ufo.server.app --port 5001
```
**Expected Output:**
```console
2024-11-06 10:30:22 - ufo.server.app - INFO - Starting UFO Server on 0.0.0.0:5001
INFO: Started server process [12345]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:5001 (Press CTRL+C to quit)
```
Once you see "Uvicorn running", the server is ready at `ws://0.0.0.0:5001/ws`.
### Server Configuration Options
| Argument | Default | Description | Example |
|----------|---------|-------------|---------|
| `--port` | `5000` | Server listening port | `--port 5001` |
| `--host` | `0.0.0.0` | Bind address (0.0.0.0 = all interfaces) | `--host 127.0.0.1` |
| `--log-level` | `INFO` | Logging verbosity | `--log-level DEBUG` |
**Custom Server Configuration:**
**Custom Port:**
```bash
python -m ufo.server.app --port 8080
```
**Specific IP Binding:**
```bash
python -m ufo.server.app --host 192.168.1.100 --port 5001
```
**Debug Mode:**
```bash
python -m ufo.server.app --port 5001 --log-level DEBUG
```
### Verify Server is Running
```bash
# Test server health endpoint
curl http://localhost:5001/api/health
```
**Expected Response:**
```json
{
"status": "healthy",
"online_clients": []
}
```
> **Documentation Reference:** For detailed server configuration and advanced features, see [Server Quick Start Guide](../server/quick_start.md).
---
## 🐧 Step 3: Start Device Agent Client (Linux Machine)
**Client Component:** The Device Agent Client runs on the Linux machine where you want to execute tasks. It connects to the server via WebSocket and receives task commands.
### Basic Client Startup
On the Linux machine where you want to execute tasks:
```bash
python -m ufo.client.client \
--ws \
--ws-server ws://172.23.48.1:5001/ws \
--client-id linux_agent_1 \
--platform linux
```
### Client Parameters Explained
| Parameter | Required | Description | Example |
|-----------|----------|-------------|---------|
| `--ws` | ✅ Yes | Enable WebSocket mode | `--ws` |
| `--ws-server` | ✅ Yes | Server WebSocket URL | `ws://172.23.48.1:5001/ws` |
| `--client-id` | ✅ Yes | **Unique** device identifier | `linux_agent_1` |
| `--platform` | ✅ Yes (Linux) | Platform type (must be `linux` for Linux Agent) | `--platform linux` |
> **⚠️ Critical Requirements:**
>
> 1. `--client-id` must be globally unique - No two devices can share the same ID
> 2. `--platform linux` is mandatory - Without this flag, the Linux Agent won't work correctly
> 3. Server address must be correct - Replace `172.23.48.1:5001` with your actual server IP and port
### Understanding the WebSocket URL
The `--ws-server` parameter format is:
```
ws://:/ws
```
Examples:
| Scenario | WebSocket URL | Description |
|----------|---------------|-------------|
| **Localhost** | `ws://localhost:5001/ws` | Server and client on same machine |
| **Same Network** | `ws://192.168.1.100:5001/ws` | Server on local network |
| **Remote Server** | `ws://203.0.113.50:5001/ws` | Server on internet (public IP) |
| **SSH Tunnel** | `ws://localhost:5001/ws` | After SSH reverse tunnel setup |
### Connection Success Indicators
**Client Logs:**
```log
INFO - Platform detected/specified: linux
INFO - UFO Client initialized for platform: linux
INFO - [WS] Connecting to ws://172.23.48.1:5001/ws (attempt 1/5)
INFO - [WS] [AIP] Successfully registered as linux_agent_1
INFO - [WS] Heartbeat loop started (interval: 30s)
```
**Server Logs:**
```log
INFO - [WS] ✅ Registered device client: linux_agent_1
INFO - [WS] Device linux_agent_1 platform: linux
```
Client is connected and ready to receive tasks when you see "Successfully registered"!
### Verify Connection
```bash
# Check connected clients on server
curl http://172.23.48.1:5001/api/clients
```
**Expected Response:**
```json
{
"clients": [
{
"client_id": "linux_agent_1",
"type": "device",
"platform": "linux",
"connected_at": 1730899822.0,
"uptime_seconds": 45
}
]
}
```
> **Documentation Reference:** For detailed client configuration, see [Client Quick Start Guide](../client/quick_start.md).
---
## 🔌 Step 4: Start MCP Service (Linux Machine)
**MCP Service Component:** The MCP (Model Context Protocol) Service provides the execution layer for CLI commands. It must be running on the same Linux machine as the client to handle command execution requests.
### Start the MCP Server
On the Linux machine (same machine as the client):
```bash
python -m ufo.client.mcp.http_servers.linux_mcp_server
```
**Expected Output:**
```console
INFO: Started server process [23456]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:8010 (Press CTRL+C to quit)
```
The MCP service is now ready to execute CLI commands at `http://127.0.0.1:8010`.
### What is the MCP Service?
The **Linux MCP Server** provides two main functionalities:
| Command | Purpose | Example Use Case |
|---------|---------|------------------|
| `EXEC_CLI` | Execute shell commands | `ls -la`, `grep pattern file.txt`, `ps aux` |
| `SYS_INFO` | Retrieve system information | CPU usage, memory stats, disk space |
**Architecture:**
```mermaid
sequenceDiagram
participant Agent as Linux Agent
participant MCP as Linux MCP Server
participant Shell as Bash Shell
Agent->>MCP: EXEC_CLI {command: "ls -la"}
MCP->>Shell: Execute command
Shell-->>MCP: stdout, stderr, exit_code
MCP-->>Agent: {result, output}
```
### MCP Service Configuration
The MCP server typically runs on `localhost:8010` by default. The client automatically connects to it when configured properly.
> **⚠️ MCP Service Must Be Running:** If the MCP service is not running, the Linux Agent cannot execute commands and will fail with:
> ```
> ERROR: Cannot connect to MCP server at http://127.0.0.1:8010
> ```
**Documentation Reference:** For detailed MCP command specifications, see [MCP Overview](../mcp/overview.md), [Linux MCP Commands](../linux/commands.md), and [BashExecutor Server](../mcp/servers/bash_executor.md).
---
## 🎯 Step 5: Dispatch Tasks via HTTP API
Once the server, client, and MCP service are all running, you can dispatch tasks to the Linux agent through the server's HTTP API.
### API Endpoint
```
POST http://:/api/dispatch
```
### Request Format
```json
{
"client_id": "linux_agent_1",
"request": "Your natural language task description",
"task_name": "optional_task_identifier"
}
```
### Example: Simple File Listing
**Using cURL:**
```bash
curl -X POST http://172.23.48.1:5001/api/dispatch \
-H "Content-Type: application/json" \
-d '{
"client_id": "linux_agent_1",
"request": "List all files in the /tmp directory",
"task_name": "list_tmp_files"
}'
```
**Using Python:**
```python
import requests
response = requests.post(
"http://172.23.48.1:5001/api/dispatch",
json={
"client_id": "linux_agent_1",
"request": "List all files in the /tmp directory",
"task_name": "list_tmp_files"
}
)
print(response.json())
```
**Using HTTPie:**
```bash
http POST http://172.23.48.1:5001/api/dispatch \
client_id=linux_agent_1 \
request="List all files in the /tmp directory" \
task_name=list_tmp_files
```
**Successful Response:**
```json
{
"status": "dispatched",
"task_name": "list_tmp_files",
"client_id": "linux_agent_1",
"session_id": "550e8400-e29b-41d4-a716-446655440000"
}
```
### Example: System Information Query
```bash
curl -X POST http://172.23.48.1:5001/api/dispatch \
-H "Content-Type: application/json" \
-d '{
"client_id": "linux_agent_1",
"request": "Show disk usage for all mounted filesystems",
"task_name": "check_disk_usage"
}'
```
### Example: Log File Analysis
```bash
curl -X POST http://172.23.48.1:5001/api/dispatch \
-H "Content-Type: application/json" \
-d '{
"client_id": "linux_agent_1",
"request": "Find all ERROR or FATAL entries in /var/log/app.log from the last hour",
"task_name": "analyze_error_logs"
}'
```
### Task Execution Flow
```mermaid
sequenceDiagram
participant API as HTTP Client
participant Server as Agent Server
participant Client as Linux Client
participant MCP as MCP Service
participant Shell as Bash
Note over API,Server: 1. Task Submission
API->>Server: POST /api/dispatch {client_id, request}
Server->>Server: Generate session_id
Server-->>API: {status: dispatched, session_id}
Note over Server,Client: 2. Task Assignment
Server->>Client: TASK_ASSIGNMENT (via WebSocket)
Client->>Client: Parse request Plan actions
Note over Client,MCP: 3. Command Execution
Client->>MCP: EXEC_CLI {command: "ls -la /tmp"}
MCP->>Shell: Execute command
Shell-->>MCP: stdout, stderr, exit_code
MCP-->>Client: {result, output}
Note over Client,Server: 4. Result Reporting
Client->>Server: TASK_RESULT {status, result}
```
### Request Parameters
| Field | Required | Type | Description | Example |
|-------|----------|------|-------------|---------|
| `client_id` | ✅ Yes | string | Target Linux agent ID (must match `--client-id`) | `"linux_agent_1"` |
| `request` | ✅ Yes | string | Natural language task description | `"List files in /var/log"` |
| `task_name` | ❌ Optional | string | Unique task identifier (auto-generated if omitted) | `"task_001"` |
> **⚠️ Client Must Be Online:** If the `client_id` is not connected, you'll receive:
> ```json
> {
> "detail": "Client not online"
> }
> ```
>
> Verify the client is connected:
> ```bash
> curl http://172.23.48.1:5001/api/clients
> ```
---
## 🌉 Network Connectivity & SSH Tunneling
When the server and client are on different networks or behind firewalls, you may need SSH tunneling to establish connectivity.
### Scenario 1: Same Network (No Tunnel Needed)
**Setup:**
- Server: `192.168.1.100:5001`
- Client: `192.168.1.50` (same LAN)
**Client Command:**
```bash
python -m ufo.client.client \
--ws \
--ws-server ws://192.168.1.100:5001/ws \
--client-id linux_agent_1 \
--platform linux
```
**No additional configuration needed** ✅
### Scenario 2: Client Behind Firewall (Reverse SSH Tunnel)
**Problem:**
- Server: `203.0.113.50:5001` (public IP, accessible)
- Client: `192.168.1.50` (private network, behind NAT/firewall)
- **Client cannot directly reach server**
**Solution: SSH Reverse Tunnel**
On the **client machine**, create an SSH reverse tunnel:
```bash
ssh -N -R 5001:localhost:5001 user@203.0.113.50
```
**Parameters:**
- `-N`: No remote command execution (tunnel only)
- `-R 5001:localhost:5001`: Forward remote port 5001 to local port 5001
- `user@203.0.113.50`: SSH server address (where the UFO server runs)
**What This Does:**
```mermaid
graph LR
Client[Client Machine 192.168.1.50]
SSH[SSH Tunnel]
Server[Server Machine 203.0.113.50]
Client -->|SSH Reverse Tunnel| SSH
SSH -->|Port 5001| Server
style Client fill:#e1f5ff
style Server fill:#ffe1e1
style SSH fill:#fffacd
```
**After tunnel is established:**
```bash
# Client can now connect to localhost:5001
python -m ufo.client.client \
--ws \
--ws-server ws://localhost:5001/ws \
--client-id linux_agent_1 \
--platform linux
```
### Scenario 3: Server Behind Firewall (Forward SSH Tunnel)
**Problem:**
- Server: `192.168.1.100:5001` (private network)
- Client: `203.0.113.75` (public network)
- **Client cannot directly reach server**
**Solution: SSH Forward Tunnel**
On the **client machine**, create an SSH forward tunnel to the server's network:
```bash
ssh -N -L 5001:192.168.1.100:5001 gateway-user@vpn.company.com
```
**Parameters:**
- `-N`: No remote command execution
- `-L 5001:192.168.1.100:5001`: Forward local port 5001 to remote 192.168.1.100:5001
- `gateway-user@vpn.company.com`: SSH gateway that can access the server
**After tunnel is established:**
```bash
# Client connects to localhost, which forwards to server
python -m ufo.client.client \
--ws \
--ws-server ws://localhost:5001/ws \
--client-id linux_agent_1 \
--platform linux
```
### Example: Complex Tunnel Setup
**Situation:**
- Server IP: `10.0.0.50:5001` (corporate network)
- Client IP: `192.168.1.75` (home network)
- SSH Gateway: `vpn.company.com` (accessible from internet)
**Step 1: Create SSH Tunnel**
```bash
# On client machine
ssh -N -L 5001:10.0.0.50:5001 myuser@vpn.company.com
```
**Step 2: Start Client (in another terminal)**
```bash
python -m ufo.client.client \
--ws \
--ws-server ws://localhost:5001/ws \
--client-id linux_agent_home_1 \
--platform linux
```
### SSH Tunnel Best Practices
For production use, add these flags to your SSH tunnel:
```bash
ssh -N \
-L 5001:server:5001 \
-o ServerAliveInterval=60 \
-o ServerAliveCountMax=3 \
-o ExitOnForwardFailure=yes \
user@gateway
```
**Flags explained:**
- `ServerAliveInterval=60`: Send keep-alive every 60 seconds
- `ServerAliveCountMax=3`: Disconnect after 3 failed keep-alives
- `ExitOnForwardFailure=yes`: Exit if port forwarding fails
### Persistent SSH Tunnel with Autossh
For production, use `autossh` to automatically restart the tunnel if it fails:
```bash
# Install autossh
sudo apt-get install autossh # Debian/Ubuntu
# Start persistent tunnel
autossh -M 0 \
-N \
-L 5001:server:5001 \
-o ServerAliveInterval=60 \
-o ServerAliveCountMax=3 \
user@gateway
```
> **ℹ️ Network Configuration:** For more network configuration details, see [Server Quick Start - Troubleshooting](../server/quick_start.md#common-issues-troubleshooting).
---
## 🌌 Step 6: Configure as UFO³ Galaxy Device
To use the Linux Agent as a managed device within the **UFO³ Galaxy** multi-tier framework, you need to register it in the `devices.yaml` configuration file.
### Device Configuration File
The Galaxy configuration is located at:
```
config/galaxy/devices.yaml
```
### Add Linux Agent Configuration
Edit `config/galaxy/devices.yaml` and add your Linux agent under the `devices` section:
```yaml
devices:
- device_id: "linux_agent_1"
server_url: "ws://172.23.48.1:5001/ws"
os: "linux"
capabilities:
- "server"
- "log_analysis"
- "file_operations"
metadata:
os: "linux"
performance: "medium"
logs_file_path: "/var/log/myapp/app.log"
dev_path: "/home/user/development/"
warning_log_pattern: "WARN"
error_log_pattern: "ERROR|FATAL"
auto_connect: true
max_retries: 5
```
### Configuration Fields Explained
| Field | Required | Type | Description | Example |
|-------|----------|------|-------------|---------|
| `device_id` | ✅ Yes | string | **Must match client `--client-id`** | `"linux_agent_1"` |
| `server_url` | ✅ Yes | string | **Must match server WebSocket URL** | `"ws://172.23.48.1:5001/ws"` |
| `os` | ✅ Yes | string | Operating system | `"linux"` |
| `capabilities` | ❌ Optional | list | Device capabilities (for task routing) | `["server", "log_analysis"]` |
| `metadata` | ❌ Optional | dict | Custom metadata for task context | See below |
| `auto_connect` | ❌ Optional | boolean | Auto-connect on Galaxy startup | `true` |
| `max_retries` | ❌ Optional | integer | Connection retry attempts | `5` |
### Metadata Fields (Custom)
The `metadata` section can contain any custom fields relevant to your Linux agent:
| Field | Purpose | Example |
|-------|---------|---------|
| `logs_file_path` | Path to application logs | `"/var/log/app.log"` |
| `dev_path` | Development directory | `"/home/user/dev/"` |
| `warning_log_pattern` | Regex pattern for warnings | `"WARN"` |
| `error_log_pattern` | Regex pattern for errors | `"ERROR\|FATAL"` |
| `performance` | Performance tier | `"high"`, `"medium"`, `"low"` |
| `description` | Human-readable description | `"Production database server"` |
### Multiple Linux Agents Example
```yaml
devices:
- device_id: "linux_agent_1"
server_url: "ws://172.23.48.1:5001/ws"
os: "linux"
capabilities:
- "web_server"
metadata:
logs_file_path: "/var/log/nginx/access.log"
dev_path: "/var/www/html/"
warning_log_pattern: "WARN"
error_log_pattern: "ERROR|FATAL"
auto_connect: true
max_retries: 5
- device_id: "linux_agent_2"
server_url: "ws://172.23.48.2:5002/ws"
os: "linux"
capabilities:
- "database_server"
metadata:
logs_file_path: "/var/log/postgresql/postgresql.log"
dev_path: "/var/lib/postgresql/"
warning_log_pattern: "WARNING"
error_log_pattern: "ERROR|FATAL|PANIC"
auto_connect: true
max_retries: 5
- device_id: "linux_agent_3"
server_url: "ws://172.23.48.3:5003/ws"
os: "linux"
capabilities:
- "monitoring"
metadata:
logs_file_path: "/var/log/prometheus/prometheus.log"
dev_path: "/opt/prometheus/"
warning_log_pattern: "level=warn"
error_log_pattern: "level=error"
auto_connect: true
max_retries: 5
```
### Critical Requirements
> **⚠️ Configuration Validation - These fields MUST match exactly:**
>
> 1. **`device_id` in YAML** ↔ **`--client-id` in client command**
> ```yaml
> device_id: "linux_agent_1" # In devices.yaml
> ```
> ```bash
> --client-id linux_agent_1 # In client command
> ```
>
> 2. **`server_url` in YAML** ↔ **`--ws-server` in client command**
> ```yaml
> server_url: "ws://172.23.48.1:5001/ws" # In devices.yaml
> ```
> ```bash
> --ws-server ws://172.23.48.1:5001/ws # In client command
> ```
>
> **If these don't match, Galaxy cannot control the device!**
### Using Galaxy to Control Linux Agents
Once configured, you can launch Galaxy and it will automatically manage the Linux agents:
```bash
python -m galaxy --interactive
```
**Galaxy will:**
1. ✅ Automatically load device configuration from `config/galaxy/devices.yaml`
2. ✅ Connect to all configured devices
3. ✅ Orchestrate multi-device tasks
4. ✅ Route tasks based on capabilities
5. ✅ Monitor device health
> **ℹ️ Galaxy Documentation:** For detailed Galaxy configuration and usage, see:
>
> - [Galaxy Overview](../galaxy/overview.md)
> - [Galaxy Quick Start](quick_start_galaxy.md)
> - [Constellation Orchestrator](../galaxy/constellation_orchestrator/overview.md)
---
## 🐛 Common Issues & Troubleshooting
### Issue 1: Client Cannot Connect to Server
**Error: Connection Refused**
Symptoms:
```log
ERROR - [WS] Failed to connect to ws://172.23.48.1:5001/ws
Connection refused
```
**Diagnosis Checklist:**
- [ ] Is the server running? (`curl http://172.23.48.1:5001/api/health`)
- [ ] Is the port correct? (Check server startup logs)
- [ ] Can client reach server IP? (`ping 172.23.48.1`)
- [ ] Is firewall blocking port 5001?
- [ ] Is SSH tunnel established (if needed)?
**Solutions:**
Verify Server:
```bash
# On server machine
curl http://localhost:5001/api/health
# From client machine
curl http://172.23.48.1:5001/api/health
```
Check Network:
```bash
# Test connectivity
ping 172.23.48.1
# Test port accessibility
nc -zv 172.23.48.1 5001
telnet 172.23.48.1 5001
```
Check Firewall:
```bash
# On server machine (Ubuntu/Debian)
sudo ufw status
sudo ufw allow 5001/tcp
# On server machine (RHEL/CentOS)
sudo firewall-cmd --list-ports
sudo firewall-cmd --add-port=5001/tcp --permanent
sudo firewall-cmd --reload
```
### Issue 2: MCP Service Not Responding
**Error: Cannot Execute Commands**
Symptoms:
```log
ERROR - Cannot connect to MCP server at http://127.0.0.1:8010
ERROR - Command execution failed
```
**Diagnosis:**
- [ ] Is the MCP service running?
- [ ] Is it running on the correct port?
- [ ] Are there any startup errors in MCP logs?
**Solutions:**
Verify MCP Service:
```bash
# Check if MCP service is running
curl http://localhost:8010/health
# Or check process
ps aux | grep linux_mcp_server
```
Restart MCP Service:
```bash
# Kill existing process (if hung)
pkill -f linux_mcp_server
# Start fresh
python -m ufo.client.mcp.http_servers.linux_mcp_server
```
Check Port Conflict:
```bash
# See if something else is using port 8010
lsof -i :8010
netstat -tuln | grep 8010
# If port is taken, start MCP on different port
python -m ufo.client.mcp.http_servers.linux_mcp_server --port 8011
```
### Issue 3: Missing `--platform linux` Flag
**Error: Incorrect Agent Type**
Symptoms:
- Client connects but cannot execute Linux commands
- Server logs show wrong platform type
- Tasks fail with "unsupported operation" errors
**Cause:** Forgot to add `--platform linux` flag when starting the client.
**Solution:**
```bash
# Wrong (missing platform)
python -m ufo.client.client --ws --client-id linux_agent_1
# Correct
python -m ufo.client.client \
--ws \
--client-id linux_agent_1 \
--platform linux
```
### Issue 4: Duplicate Client ID
**Error: Registration Failed**
Symptoms:
```log
ERROR - [WS] Registration failed: client_id already exists
ERROR - Another device is using ID 'linux_agent_1'
```
**Cause:** Multiple clients trying to use the same `client_id`.
**Solutions:**
1. **Use unique client IDs:**
```bash
# Device 1
--client-id linux_agent_1
# Device 2
--client-id linux_agent_2
# Device 3
--client-id linux_agent_3
```
2. **Check currently connected clients:**
```bash
curl http://172.23.48.1:5001/api/clients
```
### Issue 5: Galaxy Cannot Find Device
**Error: Device Not Configured**
Symptoms:
```log
ERROR - Device 'linux_agent_1' not found in configuration
WARNING - Cannot dispatch task to unknown device
```
**Cause:** Mismatch between `devices.yaml` configuration and actual client setup.
**Diagnosis:**
Check that these match **exactly**:
| Location | Field | Example |
|----------|-------|---------|
| `devices.yaml` | `device_id` | `"linux_agent_1"` |
| Client command | `--client-id` | `linux_agent_1` |
| `devices.yaml` | `server_url` | `"ws://172.23.48.1:5001/ws"` |
| Client command | `--ws-server` | `ws://172.23.48.1:5001/ws` |
**Solution:** Update `devices.yaml` to match your client configuration, or vice versa.
### Issue 6: SSH Tunnel Keeps Disconnecting
**Error: Tunnel Connection Lost**
Symptoms:
- Client disconnects after a few minutes
- SSH tunnel closes unexpectedly
- "Connection reset by peer" errors
**Solutions:**
Use ServerAliveInterval:
```bash
ssh -N \
-L 5001:server:5001 \
-o ServerAliveInterval=60 \
-o ServerAliveCountMax=3 \
user@gateway
```
Use Autossh:
```bash
autossh -M 0 \
-N \
-L 5001:server:5001 \
-o ServerAliveInterval=60 \
user@gateway
```
Run in Screen/Tmux:
```bash
# Start screen session
screen -S ssh-tunnel
# Run SSH tunnel
ssh -N -L 5001:server:5001 user@gateway
# Detach: Ctrl+A, then D
# Reattach: screen -r ssh-tunnel
```
---
## 📚 Next Steps
You've successfully set up a Linux Agent! Explore these topics to deepen your understanding:
### Immediate Next Steps
| Priority | Topic | Time | Link |
|----------|-------|------|------|
| 🥇 | **Linux Agent Architecture** | 10 min | [Overview](../linux/overview.md) |
| 🥈 | **State Machine & Processing** | 15 min | [State Machine](../linux/state.md) |
| 🥉 | **MCP Commands Reference** | 10 min | [Commands](../linux/commands.md) |
### Advanced Topics
| Topic | Description | Link |
|-------|-------------|------|
| **Processing Strategy** | 3-phase pipeline (LLM, Action, Memory) | [Strategy](../linux/strategy.md) |
| **Galaxy Integration** | Multi-device orchestration | [Galaxy Overview](../galaxy/overview.md) |
| **MCP Protocol** | Deep dive into command execution | [MCP Overview](../mcp/overview.md) |
| **Server Architecture** | Understanding the server internals | [Server Overview](../server/overview.md) |
### Production Deployment
| Best Practice | Description | Link |
|---------------|-------------|------|
| **Systemd Service** | Run client as Linux service | [Client Guide](../client/quick_start.md#running-as-background-service) |
| **Log Management** | Structured logging and rotation | [Server Monitoring](../server/monitoring.md) |
| **Security Hardening** | SSL/TLS, authentication, firewalls | [Server Guide](../server/quick_start.md#production-deployment) |
---
## ✅ Summary
## ✅ What You've Accomplished
Congratulations! You've successfully:
✅ Switched to the `linux-client` branch
✅ Installed all dependencies
✅ Started the Device Agent Server
✅ Connected a Linux Device Agent Client
✅ Launched the MCP service for command execution
✅ Dispatched tasks via HTTP API
✅ (Optional) Configured SSH tunneling for remote access
✅ (Optional) Registered the device in Galaxy configuration
**Your Linux Agent is Ready**
You can now:
- 🎯 Execute CLI commands on Linux machines remotely
- 📊 Analyze log files across multiple servers
- 🔧 Manage development environments
- 🌌 Integrate with UFO³ Galaxy for multi-device workflows
**Start exploring and automating your Linux infrastructure!** 🚀
---
### Documents Docs Getting Started Quick Start Mobile (`documents/docs/getting_started/quick_start_mobile.md`)
# ⚡ Quick Start: Mobile Agent
Get your Android device running as a UFO³ device agent in 10 minutes. This guide walks you through ADB setup, server/client configuration, and MCP service initialization for Android automation.
> **📚 Documentation Navigation:**
>
> - **Architecture & Concepts:** [Mobile Agent Overview](../mobile/overview.md)
> - **State Management:** [State Machine](../mobile/state.md)
> - **Processing Pipeline:** [Processing Strategy](../mobile/strategy.md)
> - **Available Commands:** [MCP Commands Reference](../mobile/commands.md)
> - **Galaxy Integration:** [As Galaxy Device](../mobile/as_galaxy_device.md)
---
## 📋 Prerequisites
Before you begin, ensure you have:
- **Python 3.10+** installed on your computer
- **UFO repository** cloned from [GitHub](https://github.com/microsoft/UFO)
- **Android device** (physical device or emulator) with Android 5.0+ (API 21+)
- **ADB (Android Debug Bridge)** installed and accessible
- **USB debugging enabled** on your Android device (for physical devices)
- **Network connectivity** between server and client machines
- **LLM configured** in `config/ufo/agents.yaml` (see [Model Configuration](../configuration/models/overview.md))
| Component | Minimum Version | Verification Command |
|-----------|----------------|---------------------|
| Python | 3.10 | `python --version` |
| Android OS | 5.0 (API 21) | Check device settings |
| ADB | Latest | `adb --version` |
| LLM API Key | N/A | Check `config/ufo/agents.yaml` |
> **⚠️ LLM Configuration Required:** The Mobile Agent shares the same LLM configuration with the AppAgent. Before starting, ensure you have configured your LLM provider (OpenAI, Azure OpenAI, Gemini, Claude, etc.) and added your API keys to `config/ufo/agents.yaml`. See [Model Setup Guide](../configuration/models/overview.md) for detailed instructions.
---
## 📱 Step 0: Android Device Setup
You can use either a **physical Android device** or an **Android emulator**. Choose the setup method that fits your needs.
### Option A: Physical Android Device Setup
#### 1. Enable Developer Options
On your Android device:
1. Open **Settings** → **About phone**
2. Tap **Build number** 7 times
3. You'll see "You are now a developer!"
#### 2. Enable USB Debugging
1. Go to **Settings** → **System** → **Developer options**
2. Turn on **USB debugging**
3. (Optional) Turn on **Stay awake** (device won't sleep while charging)
#### 3. Connect Device to Computer
**Via USB Cable:**
```bash
# Connect device via USB
# On device, allow USB debugging when prompted
# Verify connection
adb devices
```
**Expected Output:**
```
List of devices attached
XXXXXXXXXXXXXX device
```
**Via Wireless ADB (Android 11+):**
```bash
# On device: Settings → Developer options → Wireless debugging
# Get IP address and port (e.g., 192.168.1.100:5555)
# On computer: Connect to device
adb connect 192.168.1.100:5555
# Verify connection
adb devices
```
**Expected Output:**
```
List of devices attached
192.168.1.100:5555 device
```
### Option B: Android Emulator Setup
#### Option B1: Using Android Studio Emulator (Recommended)
**Step 1: Install Android Studio**
Download from: https://developer.android.com/studio
**Windows:**
```powershell
# Download Android Studio installer
# Run: android-studio-xxx.exe
# Follow installation wizard
```
**macOS:**
```bash
# Download Android Studio DMG
# Drag to Applications folder
# Open Android Studio
```
**Linux:**
```bash
# Download Android Studio tarball
tar -xzf android-studio-*.tar.gz
cd android-studio/bin
./studio.sh
```
**Step 2: Install Android SDK Components**
1. Open Android Studio
2. Go to **Tools** → **SDK Manager**
3. Install:
- ✅ Android SDK Platform (API 33 or higher)
- ✅ Android SDK Platform-Tools
- ✅ Android SDK Build-Tools
- ✅ Android Emulator
**Step 3: Create Virtual Device**
1. In Android Studio, click **Device Manager** (phone icon)
2. Click **Create Device**
3. Select hardware:
- **Phone** category
- Choose **Pixel 6** or **Pixel 7** (recommended)
- Click **Next**
4. Select system image:
- Choose **Release Name**: **Tiramisu** (Android 13, API 33) or newer
- Click **Download** if not installed
- Click **Next**
5. Configure AVD:
- **AVD Name**: `Pixel_6_API_33` (or your choice)
- **Startup orientation**: Portrait
- **Graphics**: Automatic or Hardware
- Click **Finish**
**Step 4: Start Emulator**
**From Android Studio:**
1. Open **Device Manager**
2. Click ▶️ (Play button) next to your AVD
**From Command Line:**
```bash
# List available emulators
emulator -list-avds
# Start emulator
emulator -avd Pixel_6_API_33 &
```
**Step 5: Verify ADB Connection**
```bash
# Wait for emulator to fully boot (~1-2 minutes)
adb devices
```
**Expected Output:**
```
List of devices attached
emulator-5554 device
```
#### Option B2: Using Genymotion (Alternative)
**Step 1: Install Genymotion**
Download from: https://www.genymotion.com/download/
```bash
# Free personal edition available
# Requires VirtualBox (auto-installed)
```
**Step 2: Create Virtual Device**
1. Open Genymotion
2. Click **+** (Add new device)
3. Sign in with Genymotion account (free)
4. Select device:
- **Google Pixel 6** or similar
- **Android 13.0** or newer
5. Click **Install**
6. Click **Start**
**Step 3: Verify ADB Connection**
```bash
adb devices
```
**Expected Output:**
```
List of devices attached
192.168.56.101:5555 device
```
### Verify Device is Ready
Run this test to ensure device is accessible:
```bash
# Get device model
adb shell getprop ro.product.model
# Get Android version
adb shell getprop ro.build.version.release
# Test screenshot capability
adb shell screencap -p /sdcard/test.png
adb pull /sdcard/test.png .
```
If all commands succeed, your device is ready! ✅
---
## 🔧 Step 1: Install ADB (Android Debug Bridge)
ADB is essential for communicating with Android devices. Choose your platform:
### Windows
**Option 1: Install via Android Studio (Recommended)**
ADB is included with Android Studio (see Step 0 Option B1).
After installation, add to PATH:
```powershell
# Add Android SDK platform-tools to PATH
# Default location:
$env:PATH += ";C:\Users\\AppData\Local\Android\Sdk\platform-tools"
# Test
adb --version
```
**Option 2: Standalone ADB Installation**
```powershell
# Download platform-tools
# https://developer.android.com/studio/releases/platform-tools
# Extract to C:\adb
# Add to PATH:
$env:PATH += ";C:\adb"
# Test
adb --version
```
**Make PATH Permanent (Optional):**
1. Open **System Properties** → **Environment Variables**
2. Under **User variables**, edit **Path**
3. Add: `C:\Users\\AppData\Local\Android\Sdk\platform-tools`
4. Click **OK**
### macOS
**Option 1: Via Homebrew (Recommended)**
```bash
# Install Homebrew (if not installed)
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
# Install ADB
brew install android-platform-tools
# Verify
adb --version
```
**Option 2: Via Android Studio**
ADB is included with Android Studio. Add to PATH:
```bash
# Add to ~/.zshrc or ~/.bash_profile
export PATH="$PATH:$HOME/Library/Android/sdk/platform-tools"
# Reload
source ~/.zshrc
# Test
adb --version
```
### Linux
**Ubuntu/Debian:**
```bash
sudo apt update
sudo apt install -y adb
# Verify
adb --version
```
**Fedora/RHEL:**
```bash
sudo dnf install android-tools
# Verify
adb --version
```
**Arch Linux:**
```bash
sudo pacman -S android-tools
# Verify
adb --version
```
### Verify ADB Installation
```bash
adb version
```
**Expected Output:**
```
Android Debug Bridge version 1.0.41
Version 34.0.5-10900879
```
---
## 📦 Step 2: Install Python Dependencies
Install all UFO dependencies:
```bash
cd /path/to/UFO
pip install -r requirements.txt
```
**Verify installation:**
```bash
python -c "import ufo; print('✅ UFO installed successfully')"
```
> **Tip:** For production deployments, use a virtual environment:
>
> ```bash
> python -m venv venv
>
> # Windows
> venv\Scripts\activate
>
> # macOS/Linux
> source venv/bin/activate
>
> pip install -r requirements.txt
> ```
---
## 🖥️ Step 3: Start Device Agent Server
**Server Component:** The Device Agent Server manages connections from Android devices and dispatches tasks.
### Basic Server Startup
On your computer (where Python is installed):
```bash
python -m ufo.server.app --port 5001 --platform mobile
```
**Expected Output:**
```console
INFO - Starting UFO Server on 0.0.0.0:5001
INFO - Platform: mobile
INFO - Log level: WARNING
INFO: Started server process [12345]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:5001 (Press CTRL+C to quit)
```
Once you see "Uvicorn running", the server is ready at `ws://0.0.0.0:5001/ws`.
### Server Configuration Options
| Argument | Default | Description | Example |
|----------|---------|-------------|---------|
| `--port` | `5000` | Server listening port | `--port 5001` |
| `--host` | `0.0.0.0` | Bind address | `--host 127.0.0.1` |
| `--platform` | Auto | Platform override | `--platform mobile` |
| `--log-level` | `WARNING` | Logging verbosity | `--log-level DEBUG` |
**Custom Configuration Examples:**
```bash
# Different port
python -m ufo.server.app --port 8080 --platform mobile
# Localhost only
python -m ufo.server.app --host 127.0.0.1 --port 5001 --platform mobile
# Debug mode
python -m ufo.server.app --port 5001 --platform mobile --log-level DEBUG
```
### Verify Server is Running
```bash
curl http://localhost:5001/api/health
```
**Expected Response (when no clients connected):**
```json
{
"status": "healthy",
"online_clients": []
}
```
> **💡 Tip:** The `online_clients` list will be empty until you start and connect the Mobile Client in Step 5.
---
## 🔌 Step 4: Start MCP Services (Android Machine)
**MCP Service Component:** Two MCP servers provide Android device interaction capabilities. They must be running before starting the client.
> **💡 Learn More:** For detailed documentation on all available MCP commands and their usage, see the [MCP Commands Reference](../mobile/commands.md).
### Understanding the Two MCP Servers
MobileAgent uses **two separate MCP servers** for different responsibilities:
| Server | Port | Purpose | Tools |
|--------|------|---------|-------|
| **Data Collection** | 8020 | Screenshot, UI tree, device info, apps list | 5 read-only tools |
| **Action** | 8021 | Touch actions, typing, app launching | 8 control tools |
### Start Both MCP Servers
**Recommended: Start Both Servers Together**
On the machine with ADB access to your Android device:
```bash
python -m ufo.client.mcp.http_servers.mobile_mcp_server \
--host localhost \
--data-port 8020 \
--action-port 8021 \
--server both
```
**Expected Output:**
```console
====================================================================
UFO Mobile MCP Servers (Android)
Android device control via ADB and Model Context Protocol
====================================================================
Using ADB: adb
Checking ADB connection...
List of devices attached
emulator-5554 device
✅ Found 1 connected device(s)
====================================================================
🚀 Starting both servers on localhost (shared state)
- Data Collection Server: localhost:8020
- Action Server: localhost:8021
Note: Both servers share the same MobileServerState for caching
✅ Starting both servers in same process (shared MobileServerState)
- Data Collection Server: localhost:8020
- Action Server: localhost:8021
======================================================================
Both servers share MobileServerState cache. Press Ctrl+C to stop.
======================================================================
✅ Data Collection Server thread started
✅ Action Server thread started
======================================================================
Both servers are running. Press Ctrl+C to stop.
======================================================================
```
**Alternative: Start Servers Separately**
If needed, you can start each server in separate terminals:
**Terminal 1: Data Collection Server**
```bash
python -m ufo.client.mcp.http_servers.mobile_mcp_server \
--host localhost \
--data-port 8020 \
--server data
```
**Terminal 2: Action Server**
```bash
python -m ufo.client.mcp.http_servers.mobile_mcp_server \
--host localhost \
--action-port 8021 \
--server action
```
> **⚠️ Important:** When running servers separately, they won't share cached state, which may impact performance. Running both together is recommended.
### MCP Server Configuration Options
| Argument | Default | Description | Example |
|----------|---------|-------------|---------|
| `--host` | `localhost` | Server host | `--host 127.0.0.1` |
| `--data-port` | `8020` | Data collection server port | `--data-port 8020` |
| `--action-port` | `8021` | Action server port | `--action-port 8021` |
| `--server` | `both` | Which server(s) to start | `--server both` |
| `--adb-path` | `adb` | Path to ADB executable | `--adb-path /path/to/adb` |
### Verify MCP Servers are Running
**Check Data Collection Server:**
```bash
curl http://localhost:8020/health
```
**Check Action Server:**
```bash
curl http://localhost:8021/health
```
Both should return a health status response indicating the server is operational.
### What if ADB is not in PATH?
If ADB is not in your system PATH, specify the full path:
**Windows:**
```bash
python -m ufo.client.mcp.http_servers.mobile_mcp_server \
--adb-path "C:\Users\YourUsername\AppData\Local\Android\Sdk\platform-tools\adb.exe" \
--server both
```
**macOS:**
```bash
python -m ufo.client.mcp.http_servers.mobile_mcp_server \
--adb-path "$HOME/Library/Android/sdk/platform-tools/adb" \
--server both
```
**Linux:**
```bash
python -m ufo.client.mcp.http_servers.mobile_mcp_server \
--adb-path /usr/bin/adb \
--server both
```
---
## 📱 Step 5: Start Device Agent Client
**Client Component:** The Device Agent Client connects your Android device to the server and executes mobile automation tasks.
### Basic Client Startup
On your computer (same machine as MCP servers):
```bash
python -m ufo.client.client \
--ws \
--ws-server ws://localhost:5001/ws \
--client-id mobile_phone_1 \
--platform mobile
```
### Client Parameters Explained
| Parameter | Required | Description | Example |
|-----------|----------|-------------|---------|
| `--ws` | ✅ Yes | Enable WebSocket mode | `--ws` |
| `--ws-server` | ✅ Yes | Server WebSocket URL | `ws://localhost:5001/ws` |
| `--client-id` | ✅ Yes | **Unique** device identifier | `mobile_phone_1` |
| `--platform` | ✅ Yes | Platform type (must be `mobile`) | `--platform mobile` |
> **⚠️ Critical Requirements:**
>
> 1. `--client-id` must be globally unique - No two devices can share the same ID
> 2. `--platform mobile` is mandatory - Without this flag, the Mobile Agent won't work correctly
> 3. Server address must be correct - Use actual server IP if not on localhost
### Understanding the WebSocket URL
The `--ws-server` parameter format is:
```
ws://:/ws
```
Examples:
| Scenario | WebSocket URL | Description |
|----------|---------------|-------------|
| **Same Machine** | `ws://localhost:5001/ws` | Server and client on same computer |
| **Same Network** | `ws://192.168.1.100:5001/ws` | Server on local network |
| **Remote Server** | `ws://203.0.113.50:5001/ws` | Server on internet (public IP) |
### Connection Success Indicators
**Client Logs:**
```log
INFO - Platform detected/specified: mobile
INFO - UFO Client initialized for platform: mobile
INFO - [WS] Connecting to ws://localhost:5001/ws (attempt 1/5)
INFO - [WS] [AIP] Successfully registered as mobile_phone_1
INFO - [WS] Heartbeat loop started (interval: 30s)
```
**Server Logs:**
```log
INFO - [WS] ✅ Registered device client: mobile_phone_1
INFO - [WS] Device mobile_phone_1 platform: mobile
```
Client is connected and ready to receive tasks when you see "Successfully registered"! ✅
### Verify Connection
```bash
# Check connected clients on server
curl http://localhost:5001/api/clients
```
**Expected Response:**
```json
{
"online_clients": ["mobile_phone_1"]
}
```
> **Note:** The response shows only client IDs. For detailed information about each client, check the server logs.
---
## 🎯 Step 6: Dispatch Tasks via HTTP API
Once the server, client, and MCP services are all running, you can dispatch tasks to your Android device through the server's HTTP API.
### API Endpoint
```
POST http://:/api/dispatch
```
### Request Format
```json
{
"client_id": "mobile_phone_1",
"request": "Your natural language task description",
"task_name": "optional_task_identifier"
}
```
### Example 1: Launch an App
**Using cURL:**
```bash
curl -X POST http://localhost:5001/api/dispatch \
-H "Content-Type: application/json" \
-d '{
"client_id": "mobile_phone_1",
"request": "Open Google Chrome browser",
"task_name": "launch_chrome"
}'
```
**Using Python:**
```python
import requests
response = requests.post(
"http://localhost:5001/api/dispatch",
json={
"client_id": "mobile_phone_1",
"request": "Open Google Chrome browser",
"task_name": "launch_chrome"
}
)
print(response.json())
```
**Successful Response:**
```json
{
"status": "dispatched",
"task_name": "launch_chrome",
"client_id": "mobile_phone_1",
"session_id": "550e8400-e29b-41d4-a716-446655440000"
}
```
### Example 2: Search on Maps
```bash
curl -X POST http://localhost:5001/api/dispatch \
-H "Content-Type: application/json" \
-d '{
"client_id": "mobile_phone_1",
"request": "Open Google Maps and search for coffee shops nearby",
"task_name": "search_coffee"
}'
```
### Example 3: Type and Submit Text
```bash
curl -X POST http://localhost:5001/api/dispatch \
-H "Content-Type: application/json" \
-d '{
"client_id": "mobile_phone_1",
"request": "Open Chrome, search for weather forecast, and show me the results",
"task_name": "check_weather"
}'
```
### Example 4: Take Screenshot
```bash
curl -X POST http://localhost:5001/api/dispatch \
-H "Content-Type: application/json" \
-d '{
"client_id": "mobile_phone_1",
"request": "Take a screenshot of the current screen",
"task_name": "capture_screen"
}'
```
### Task Execution Flow
```mermaid
sequenceDiagram
participant API as HTTP Client
participant Server as Agent Server
participant Client as Mobile Client
participant MCP as MCP Services
participant Device as Android Device
Note over API,Server: 1. Task Submission
API->>Server: POST /api/dispatch {client_id, request}
Server->>Server: Generate session_id
Server-->>API: {status: dispatched, session_id}
Note over Server,Client: 2. Task Assignment
Server->>Client: TASK_ASSIGNMENT (via WebSocket)
Client->>Client: Initialize Mobile Agent
Note over Client,MCP: 3. Data Collection
Client->>MCP: Capture screenshot
Client->>MCP: Get installed apps
Client->>MCP: Get UI controls
MCP->>Device: ADB commands
Device-->>MCP: Screenshot + Apps + Controls
MCP-->>Client: Visual context
Note over Client: 4. LLM Decision
Client->>Client: Construct prompt with screenshots
Client->>Client: Get action from LLM
Note over Client,MCP: 5. Action Execution
Client->>MCP: Execute mobile action (tap, swipe, launch_app, etc.)
MCP->>Device: ADB input commands
Device-->>MCP: Action result
MCP-->>Client: Success/Failure
Note over Client,Server: 6. Result Reporting
Client->>Server: TASK_RESULT {status, screenshots, actions}
Server-->>API: Task completed
```
### Request Parameters
| Field | Required | Type | Description | Example |
|-------|----------|------|-------------|---------|
| `client_id` | ✅ Yes | string | Target mobile device ID (must match `--client-id`) | `"mobile_phone_1"` |
| `request` | ✅ Yes | string | Natural language task description | `"Open Chrome"` |
| `task_name` | ❌ Optional | string | Unique task identifier (auto-generated if omitted) | `"task_001"` |
> **⚠️ Client Must Be Online:** If the `client_id` is not connected, you'll receive:
> ```json
> {
> "detail": "Client not online"
> }
> ```
>
> Verify the client is connected:
> ```bash
> curl http://localhost:5001/api/clients
> ```
---
## 🌌 Step 7: Configure as UFO³ Galaxy Device
To use the Mobile Agent as a managed device within the **UFO³ Galaxy** multi-tier framework, you need to register it in the `devices.yaml` configuration file.
> **📖 Detailed Guide:** For comprehensive information on using Mobile Agent in Galaxy, including multi-device workflows and advanced configuration, see [Using Mobile Agent as Galaxy Device](../mobile/as_galaxy_device.md).
### Device Configuration File
The Galaxy configuration is located at:
```
config/galaxy/devices.yaml
```
### Add Mobile Agent Configuration
Edit `config/galaxy/devices.yaml` and add your Mobile agent:
```yaml
devices:
- device_id: "mobile_phone_1"
server_url: "ws://localhost:5001/ws"
os: "mobile"
capabilities:
- "mobile"
- "android"
- "messaging"
- "maps"
- "camera"
metadata:
os: "mobile"
device_type: "phone"
android_version: "13"
screen_size: "1080x2400"
installed_apps:
- "com.android.chrome"
- "com.google.android.apps.maps"
- "com.whatsapp"
description: "Android phone for mobile automation"
auto_connect: true
max_retries: 5
```
### Configuration Fields Explained
| Field | Required | Type | Description | Example |
|-------|----------|------|-------------|---------|
| `device_id` | ✅ Yes | string | **Must match client `--client-id`** | `"mobile_phone_1"` |
| `server_url` | ✅ Yes | string | **Must match server WebSocket URL** | `"ws://localhost:5001/ws"` |
| `os` | ✅ Yes | string | Operating system | `"mobile"` |
| `capabilities` | ❌ Optional | list | Device capabilities | `["mobile", "android"]` |
| `metadata` | ❌ Optional | dict | Custom metadata | See below |
| `auto_connect` | ❌ Optional | boolean | Auto-connect on Galaxy startup | `true` |
| `max_retries` | ❌ Optional | integer | Connection retry attempts | `5` |
### Metadata Fields (Custom)
The `metadata` section provides context to the LLM:
| Field | Purpose | Example |
|-------|---------|---------|
| `device_type` | Phone, tablet, emulator | `"phone"` |
| `android_version` | OS version | `"13"` |
| `screen_size` | Resolution | `"1080x2400"` |
| `installed_apps` | Available apps | `["com.android.chrome", ...]` |
| `description` | Human-readable description | `"Personal phone"` |
### Multiple Mobile Devices Example
```yaml
devices:
# Personal Phone
- device_id: "mobile_phone_personal"
server_url: "ws://192.168.1.100:5001/ws"
os: "mobile"
capabilities:
- "mobile"
- "android"
- "messaging"
- "whatsapp"
- "maps"
metadata:
os: "mobile"
device_type: "phone"
android_version: "13"
installed_apps:
- "com.whatsapp"
- "com.google.android.apps.maps"
description: "Personal Android phone"
auto_connect: true
max_retries: 5
# Work Phone
- device_id: "mobile_phone_work"
server_url: "ws://192.168.1.101:5002/ws"
os: "mobile"
capabilities:
- "mobile"
- "android"
- "email"
- "teams"
metadata:
os: "mobile"
device_type: "phone"
android_version: "12"
installed_apps:
- "com.microsoft.office.outlook"
- "com.microsoft.teams"
description: "Work Android phone"
auto_connect: true
max_retries: 5
# Tablet
- device_id: "mobile_tablet_home"
server_url: "ws://192.168.1.102:5003/ws"
os: "mobile"
capabilities:
- "mobile"
- "android"
- "tablet"
- "media"
metadata:
os: "mobile"
device_type: "tablet"
android_version: "13"
screen_size: "2560x1600"
installed_apps:
- "com.netflix.mediaclient"
description: "Home tablet for media"
auto_connect: true
max_retries: 5
```
### Critical Requirements
> **⚠️ Configuration Validation - These fields MUST match exactly:**
>
> 1. **`device_id` in YAML** ↔ **`--client-id` in client command**
> 2. **`server_url` in YAML** ↔ **`--ws-server` in client command**
>
> **If these don't match, Galaxy cannot control the device!**
### Using Galaxy to Control Mobile Agents
Once configured, launch Galaxy:
```bash
python -m galaxy --interactive
```
**Galaxy will:**
1. ✅ Load device configuration from `config/galaxy/devices.yaml`
2. ✅ Connect to all configured Android devices
3. ✅ Orchestrate multi-device tasks
4. ✅ Route tasks based on capabilities
> **ℹ️ Galaxy Documentation:** For detailed Galaxy usage, see:
>
> - [Galaxy Overview](../galaxy/overview.md)
> - [Galaxy Quick Start](quick_start_galaxy.md)
> - [Mobile Agent as Galaxy Device](../mobile/as_galaxy_device.md)
---
## 🔍 Understanding Mobile Agent Internals
Now that you have Mobile Agent running, you may want to understand how it works under the hood:
### State Machine
Mobile Agent uses a **3-state finite state machine** to manage task execution:
- **CONTINUE** - Active execution, processing user requests
- **FINISH** - Task completed successfully
- **FAIL** - Unrecoverable error occurred
Learn more: [State Machine Documentation](../mobile/state.md)
### Processing Pipeline
During the CONTINUE state, Mobile Agent executes a **4-phase pipeline**:
1. **Data Collection** - Capture screenshots, get apps, collect UI controls
2. **LLM Interaction** - Send visual context to LLM for decision making
3. **Action Execution** - Execute mobile actions (tap, swipe, type, etc.)
4. **Memory Update** - Record actions and results for context
Learn more: [Processing Strategy Documentation](../mobile/strategy.md)
### Available Commands
Mobile Agent uses **13 MCP commands** across two servers:
- **Data Collection Server (8020)**: 5 read-only commands
- **Action Server (8021)**: 8 control commands
Learn more: [MCP Commands Reference](../mobile/commands.md)
---
## 🐛 Common Issues & Troubleshooting
### Issue 1: ADB Device Not Found
**Error: No Devices Detected**
Symptoms:
```bash
$ adb devices
List of devices attached
# Empty list
```
**Solutions:**
**For Physical Devices:**
1. **Check USB connection:**
- Use a different USB cable (some cables are charge-only)
- Try a different USB port on your computer
- Ensure USB debugging is enabled on device
2. **Authorize computer on device:**
- Disconnect and reconnect USB
- On device, tap "Allow USB debugging" when prompted
- Check "Always allow from this computer"
3. **Restart ADB server:**
```bash
adb kill-server
adb start-server
adb devices
```
4. **Check USB driver (Windows):**
- Install Google USB Driver via Android Studio SDK Manager
- Or install device-specific driver from manufacturer
**For Emulators:**
1. **Wait for emulator to fully boot** (can take 1-2 minutes)
2. **Restart emulator:**
- Close emulator completely
- Start emulator again from Android Studio or command line
3. **Check emulator is running:**
```bash
emulator -list-avds
emulator -avd Pixel_6_API_33
```
### Issue 2: MCP Server Cannot Connect to Device
**Error: ADB Connection Failed**
Symptoms:
```log
ERROR - Failed to execute ADB command
ERROR - Device not accessible
```
**Solutions:**
1. **Verify ADB connection first:**
```bash
adb devices
```
Device should show "device" status (not "offline" or "unauthorized")
2. **Test ADB commands manually:**
```bash
adb shell getprop ro.product.model
adb shell screencap -p /sdcard/test.png
```
3. **Restart MCP servers with debug output:**
```bash
# Kill existing servers
pkill -f mobile_mcp_server
# Start with explicit ADB path
python -m ufo.client.mcp.http_servers.mobile_mcp_server \
--adb-path $(which adb) \
--server both
```
4. **Check device permissions:**
- Ensure USB debugging is still authorized
- Revoke and re-grant USB debugging authorization on device
### Issue 3: Client Cannot Connect to Server
**Error: Connection Refused or Failed**
Symptoms:
```log
ERROR - [WS] Failed to connect to ws://localhost:5001/ws
Connection refused
```
**Solutions:**
1. **Verify server is running:**
```bash
curl http://localhost:5001/api/health
```
Should return:
```json
{
"status": "healthy",
"online_clients": []
}
```
2. **Check server address:**
- If server and client are on different machines, use server's IP address
- Replace `localhost` with actual IP address (e.g., `ws://192.168.1.100:5001/ws`)
- Ensure the port number matches the server's `--port` argument
3. **Check firewall settings:**
```bash
# Windows: Allow port 5001
netsh advfirewall firewall add rule name="UFO Server" dir=in action=allow protocol=TCP localport=5001
# macOS: System Preferences → Security & Privacy → Firewall → Firewall Options
# Linux (Ubuntu):
sudo ufw allow 5001/tcp
```
### Issue 4: Missing `--platform mobile` Flag
**Error: Incorrect Agent Type**
Symptoms:
- Client connects but cannot execute mobile commands
- Server logs show wrong platform type
- Tasks fail with "unsupported operation" errors
**Solution:**
Always include `--platform mobile` when starting the client:
```bash
# Wrong (missing platform)
python -m ufo.client.client --ws --client-id mobile_phone_1
# Correct
python -m ufo.client.client \
--ws \
--client-id mobile_phone_1 \
--platform mobile
```
### Issue 5: Screenshot Capture Fails
**Error: Cannot Capture Screenshot**
Symptoms:
```log
ERROR - Failed to capture screenshot
ERROR - screencap command failed
```
**Solutions:**
1. **Test screenshot manually:**
```bash
adb shell screencap -p /sdcard/test.png
adb pull /sdcard/test.png .
```
2. **Check device storage:**
```bash
adb shell df -h /sdcard
```
Ensure sufficient space on device
3. **Check permissions:**
```bash
adb shell ls -l /sdcard
```
4. **Try alternative screenshot method:**
```bash
adb exec-out screencap -p > screenshot.png
```
### Issue 6: UI Controls Not Found
**Error: Control Information Missing**
Symptoms:
```log
WARNING - Failed to get UI controls
WARNING - UI tree dump failed
```
**Solutions:**
1. **Test UI dump manually:**
```bash
adb shell uiautomator dump /sdcard/window_dump.xml
adb shell cat /sdcard/window_dump.xml
```
2. **Enable accessibility services:**
- Some apps require accessibility services for UI automation
- Settings → Accessibility → Enable required services
3. **Update Android WebView:**
- Old WebView versions may cause UI dump issues
- Update via Play Store: Android System WebView
4. **Restart device:**
```bash
adb reboot
# Wait for device to restart
adb wait-for-device
```
### Issue 7: Emulator Too Slow
**Error: Performance Issues**
Symptoms:
- Emulator lags or freezes
- Actions take very long to execute
- Timeouts occur frequently
**Solutions:**
1. **Enable Hardware Acceleration:**
- **Windows:** Ensure Hyper-V or Intel HAXM is enabled
- **macOS:** Hypervisor.framework is used automatically
- **Linux:** Install KVM
2. **Allocate More Resources:**
- In Android Studio AVD Manager, edit AVD
- Increase RAM to 2048 MB or higher
- Increase VM heap to 512 MB
- Set Graphics to "Hardware - GLES 2.0"
3. **Use x86_64 System Image:**
- Faster than ARM images
- Download x86_64 image in SDK Manager
4. **Reduce Screen Resolution:**
- Edit AVD settings
- Choose lower resolution (e.g., 720x1280 instead of 1080x2400)
### Issue 8: Multiple Devices Connected
**Error: More Than One Device**
Symptoms:
```bash
$ adb devices
List of devices attached
emulator-5554 device
192.168.1.100:5555 device
```
**Solutions:**
1. **Specify device for ADB:**
```bash
# Use emulator
export ANDROID_SERIAL=emulator-5554
# Use physical device
export ANDROID_SERIAL=192.168.1.100:5555
```
2. **Disconnect other devices:**
```bash
# Disconnect wireless device
adb disconnect 192.168.1.100:5555
```
3. **Run separate MCP servers:**
```bash
# Server for emulator
ANDROID_SERIAL=emulator-5554 python -m ufo.client.mcp.http_servers.mobile_mcp_server --data-port 8020 --action-port 8021 --server both
# Server for physical device
ANDROID_SERIAL=192.168.1.100:5555 python -m ufo.client.mcp.http_servers.mobile_mcp_server --data-port 8022 --action-port 8023 --server both
```
---
## 📚 Next Steps
You've successfully set up a Mobile Agent! Explore these topics to deepen your understanding:
### Immediate Next Steps
| Priority | Topic | Time | Link |
|----------|-------|------|------|
| 🥇 | **Mobile Agent Architecture** | 10 min | [Overview](../mobile/overview.md) |
| 🥈 | **State Machine & Processing** | 15 min | [State Machine](../mobile/state.md) |
| 🥉 | **MCP Commands Reference** | 15 min | [Commands](../mobile/commands.md) |
### Advanced Topics
| Topic | Description | Link |
|-------|-------------|------|
| **Processing Strategy** | 4-phase pipeline (Data, LLM, Action, Memory) | [Strategy](../mobile/strategy.md) |
| **Galaxy Integration** | Multi-device orchestration with UFO³ | [As Galaxy Device](../mobile/as_galaxy_device.md) |
| **MCP Protocol Details** | Deep dive into mobile interaction protocol | [Commands](../mobile/commands.md) |
### Production Deployment
| Best Practice | Description |
|---------------|-------------|
| **Persistent ADB** | Keep ADB connection stable for physical devices |
| **Emulator Management** | Automate emulator lifecycle (start/stop/reset) |
| **Screenshot Storage** | Configure log paths and cleanup policies in `config/ufo/system.yaml` |
| **Security** | Use secure WebSocket (wss://) for remote deployments |
> **💡 Learn More:** For comprehensive understanding of the Mobile Agent architecture and processing flow, see the [Mobile Agent Overview](../mobile/overview.md).
---
## ✅ Summary
Congratulations! You've successfully:
✅ Set up Android device (physical or emulator)
✅ Installed ADB (Android Debug Bridge)
✅ Installed Python dependencies
✅ Started the Device Agent Server
✅ Launched MCP services (data collection + action)
✅ Connected Mobile Device Agent Client
✅ Dispatched mobile automation tasks via HTTP API
✅ (Optional) Configured device in Galaxy
**Your Mobile Agent is Ready**
You can now:
- 📱 Automate Android apps remotely
- 🖼️ Capture and analyze screenshots
- 🎯 Interact with UI controls precisely
- 🌌 Integrate with UFO³ Galaxy for cross-platform workflows
**Start exploring mobile automation!** 🚀
---
## 💡 Pro Tips
### Quick Start Command Summary
**Start everything in order:**
```bash
# Terminal 1: Start server
python -m ufo.server.app --port 5001 --platform mobile
# Terminal 2: Start MCP services
python -m ufo.client.mcp.http_servers.mobile_mcp_server --server both
# Terminal 3: Start client
python -m ufo.client.client --ws --ws-server ws://localhost:5001/ws --client-id mobile_phone_1 --platform mobile
# Terminal 4: Dispatch task
curl -X POST http://localhost:5001/api/dispatch \
-H "Content-Type: application/json" \
-d '{"client_id": "mobile_phone_1", "request": "Open Chrome browser"}'
```
### Development Shortcuts
**Create shell scripts for common operations:**
**Windows (PowerShell):**
```powershell
# start-mobile-agent.ps1
Start-Process powershell -ArgumentList "-NoExit", "-Command", "python -m ufo.server.app --port 5001 --platform mobile"
Start-Sleep 2
Start-Process powershell -ArgumentList "-NoExit", "-Command", "python -m ufo.client.mcp.http_servers.mobile_mcp_server --server both"
Start-Sleep 2
Start-Process powershell -ArgumentList "-NoExit", "-Command", "python -m ufo.client.client --ws --ws-server ws://localhost:5001/ws --client-id mobile_phone_1 --platform mobile"
```
**macOS/Linux (Bash):**
```bash
#!/bin/bash
# start-mobile-agent.sh
# Start server in background
python -m ufo.server.app --port 5001 --platform mobile &
sleep 2
# Start MCP services in background
python -m ufo.client.mcp.http_servers.mobile_mcp_server --server both &
sleep 2
# Start client in foreground
python -m ufo.client.client --ws --ws-server ws://localhost:5001/ws --client-id mobile_phone_1 --platform mobile
```
Make executable:
```bash
chmod +x start-mobile-agent.sh
./start-mobile-agent.sh
```
### Testing Your Setup
**Quick test to verify everything works:**
```bash
# Test 1: Check ADB
adb devices
# Should show your device
# Test 2: Check Server
curl http://localhost:5001/api/health
# Should return {"status": "healthy"}
# Test 3: Check MCP
curl http://localhost:8020/health
curl http://localhost:8021/health
# Should return health status
# Test 4: Check Client
curl http://localhost:5001/api/clients
# Should show mobile_phone_1
# Test 5: Dispatch simple task
curl -X POST http://localhost:5001/api/dispatch \
-H "Content-Type: application/json" \
-d '{"client_id": "mobile_phone_1", "request": "Take a screenshot"}'
# Should return dispatched status
```
**Happy Mobile Automation! 🎉**
---
### Documents Docs Getting Started Quick Start Ufo2 (`documents/docs/getting_started/quick_start_ufo2.md`)
# Quick Start Guide
Welcome to **UFO²** – the Desktop AgentOS! This guide will help you get started with UFO² in just a few minutes.
**What is UFO²?**
UFO² is a Desktop AgentOS that turns natural-language requests into automatic, reliable, multi-application workflows on Windows. It goes beyond UI-focused automation by combining GUI actions with native API calls for faster and more robust execution.
---
## 🛠️ Step 1: Installation
### Requirements
- **Python** >= 3.10
- **Windows OS** >= 10
- **Git** (for cloning the repository)
### Installation Steps
```powershell
# [Optional] Create conda environment
conda create -n ufo python=3.10
conda activate ufo
# Clone the repository
git clone https://github.com/microsoft/UFO.git
cd UFO
# Install dependencies
pip install -r requirements.txt
```
> **💡 Tip:** If you want to use Qwen as your LLM, uncomment the related libraries in `requirements.txt` before installing.
---
---
## ⚙️ Step 2: Configure LLMs
> **📢 New Configuration System (Recommended)**
> UFO² now uses a **new modular config system** located in `config/ufo/` with auto-discovery and type validation. While the legacy `ufo/config/config.yaml` is still supported for backward compatibility, we strongly recommend migrating to the new system for better maintainability.
### Option 1: New Config System (Recommended)
The new config files are organized in `config/ufo/` with separate YAML files for different components:
```powershell
# Copy template to create your agent config file (contains API keys)
copy config\ufo\agents.yaml.template config\ufo\agents.yaml
notepad config\ufo\agents.yaml # Edit your LLM API credentials
```
**Directory Structure:**
```
config/ufo/
├── agents.yaml.template # Template: Agent configs (HOST_AGENT, APP_AGENT) - COPY & EDIT THIS
├── agents.yaml # Your agent configs with API keys (DO NOT commit to git)
├── rag.yaml # RAG and knowledge settings (default values, edit if needed)
├── system.yaml # System settings (default values, edit if needed)
├── mcp.yaml # MCP integration settings (default values, edit if needed)
└── ... # Other modular configs with defaults
```
> **Configuration Files:** `agents.yaml` contains sensitive information (API keys) and must be configured. Other config files have default values and only need editing for customization.
**Migration Benefits:**
- ✅ **Type Safety**: Automatic validation with Pydantic schemas
- ✅ **Auto-Discovery**: No manual config loading needed
- ✅ **Modular**: Separate concerns into individual files
- ✅ **IDE Support**: Better autocomplete and error detection
### Option 2: Legacy Config (Backward Compatible)
For existing users, the old config path still works:
```powershell
copy ufo\config\config.yaml.template ufo\config\config.yaml
notepad ufo\config\config.yaml # Paste your key & endpoint
```
> **Config Precedence:** If both old and new configs exist, the new config in `config/ufo/` takes precedence. A warning will be displayed during startup.
---
### LLM Configuration Examples
#### OpenAI Configuration
**New Config (`config/ufo/agents.yaml`):**
```yaml
HOST_AGENT:
VISUAL_MODE: true
API_TYPE: "openai"
API_BASE: "https://api.openai.com/v1/chat/completions"
API_KEY: "sk-YOUR_KEY_HERE" # Replace with your actual API key
API_VERSION: "2025-02-01-preview"
API_MODEL: "gpt-4o"
APP_AGENT:
VISUAL_MODE: true
API_TYPE: "openai"
API_BASE: "https://api.openai.com/v1/chat/completions"
API_KEY: "sk-YOUR_KEY_HERE" # Replace with your actual API key
API_VERSION: "2025-02-01-preview"
API_MODEL: "gpt-4o"
```
**Legacy Config (`ufo/config/config.yaml`):**
```yaml
HOST_AGENT:
VISUAL_MODE: True
API_TYPE: "openai"
API_BASE: "https://api.openai.com/v1/chat/completions"
API_KEY: "sk-YOUR_KEY_HERE"
API_VERSION: "2024-02-15-preview"
API_MODEL: "gpt-4o"
```
#### Azure OpenAI (AOAI) Configuration
**New Config (`config/ufo/agents.yaml`):**
```yaml
HOST_AGENT:
VISUAL_MODE: true
API_TYPE: "aoai"
API_BASE: "https://YOUR_RESOURCE.openai.azure.com"
API_KEY: "YOUR_AOAI_KEY"
API_VERSION: "2024-02-15-preview"
API_MODEL: "gpt-4o"
API_DEPLOYMENT_ID: "YOUR_DEPLOYMENT_ID"
APP_AGENT:
VISUAL_MODE: true
API_TYPE: "aoai"
API_BASE: "https://YOUR_RESOURCE.openai.azure.com"
API_KEY: "YOUR_AOAI_KEY"
API_VERSION: "2024-02-15-preview"
API_MODEL: "gpt-4o"
API_DEPLOYMENT_ID: "YOUR_DEPLOYMENT_ID"
```
> **ℹ️ More LLM Options:** UFO² supports various LLM providers including Qwen, Gemini, Claude, DeepSeek, and more. See the [Model Configuration Guide](../configuration/models/overview.md) for complete details.
---
---
## 📔 Step 3: Additional Settings (Optional)
### RAG Configuration
Enhance UFO's capabilities with external knowledge through Retrieval Augmented Generation (RAG):
**For New Config**: Edit `config/ufo/rag.yaml` (already exists with default values)
**For Legacy Config**: Edit `ufo/config/config.yaml`
**Available RAG Options:**
| Feature | Documentation | Description |
|---------|--------------|-------------|
| **Offline Help Documents** | [Learning from Help Documents](../ufo2/core_features/knowledge_substrate/learning_from_help_document.md) | Retrieve information from offline help documentation |
| **Online Bing Search** | [Learning from Bing Search](../ufo2/core_features/knowledge_substrate/learning_from_bing_search.md) | Utilize up-to-date online search results |
| **Self-Experience** | [Experience Learning](../ufo2/core_features/knowledge_substrate/experience_learning.md) | Save task trajectories into memory for future reference |
| **User Demonstrations** | [Learning from Demonstrations](../ufo2/core_features/knowledge_substrate/learning_from_demonstration.md) | Learn from user-provided demonstrations |
**Example RAG Config (`config/ufo/rag.yaml`):**
```yaml
# Enable Bing search
RAG_ONLINE_SEARCH: true
BING_API_KEY: "YOUR_BING_API_KEY" # Get from https://www.microsoft.com/en-us/bing/apis
# Enable experience learning
RAG_EXPERIENCE: true
```
> **ℹ️ RAG Resources:** See [Knowledge Substrate Overview](../ufo2/core_features/knowledge_substrate/overview.md) for complete RAG configuration and best practices.
---
---
## 🎉 Step 4: Start UFO²
### Interactive Mode
Start UFO² in interactive mode where you can enter requests dynamically:
```powershell
# Assume you are in the cloned UFO folder
python -m ufo --task
```
**Expected Output:**
```
Welcome to use UFO🛸, A UI-focused Agent for Windows OS Interaction.
_ _ _____ ___
| | | || ___| / _ \
| | | || |_ | | | |
| |_| || _| | |_| |
\___/ |_| \___/
Please enter your request to be completed🛸:
```
### Direct Request Mode
Invoke UFO² with a specific task and request directly:
```powershell
python -m ufo --task -r ""
```
**Example:**
```powershell
python -m ufo --task email_demo -r "Send an email to john@example.com with subject 'Meeting Reminder'"
```
---
---
## 🎥 Step 5: Execution Logs
UFO² automatically saves execution logs, screenshots, and traces for debugging and analysis.
**Log Location:**
```
./logs//
```
**Log Contents:**
| File/Folder | Description |
|-------------|-------------|
| `screenshots/` | Screenshots captured during execution |
| `action_*.json` | Agent actions and responses |
| `ui_trees/` | UI control tree snapshots (if enabled) |
| `request_response.log` | Complete LLM request/response logs |
> **Analyzing Logs:** Use the logs to debug agent behavior, replay execution flow, and analyze agent decision-making patterns.
> **Privacy Notice:** Screenshots may contain sensitive or confidential information. Ensure no private data is visible during execution. See [DISCLAIMER.md](https://github.com/microsoft/UFO/blob/main/DISCLAIMER.md) for details.
---
## 🔄 Migrating from Legacy Config
If you're upgrading from an older version that used `ufo/config/config.yaml`, UFO² provides an **automated conversion tool**.
### Automatic Conversion (Recommended)
```powershell
# Interactive conversion with automatic backup
python -m ufo.tools.convert_config
# Preview changes first (dry run)
python -m ufo.tools.convert_config --dry-run
# Force conversion without confirmation
python -m ufo.tools.convert_config --force
```
**What the tool does:**
- ✅ Splits monolithic `config.yaml` into modular files
- ✅ Converts flow-style YAML (with braces) to block-style YAML
- ✅ Maps legacy file names to new structure
- ✅ Preserves all configuration values
- ✅ Creates timestamped backup for rollback
- ✅ Validates output files
**Conversion Mapping:**
| Legacy File | → | New File(s) | Transformation |
|-------------|---|-------------|----------------|
| `config.yaml` (monolithic) | → | `agents.yaml` + `rag.yaml` + `system.yaml` | Smart field splitting |
| `agent_mcp.yaml` | → | `mcp.yaml` | Rename + format conversion |
| `config_prices.yaml` | → | `prices.yaml` | Rename + format conversion |
> **Migration Guide:** For detailed migration instructions, rollback procedures, and troubleshooting, see the [Configuration Migration Guide](../configuration/system/migration.md).
---
## 📚 Additional Resources
### Core Documentation
**Architecture & Concepts:**
- [UFO² Overview](../ufo2/overview.md) - System architecture and design principles
- [HostAgent](../ufo2/host_agent/overview.md) - Desktop-level coordination agent
- [AppAgent](../ufo2/app_agent/overview.md) - Application-level execution agent
### Configuration
**Configuration Guides:**
- [Configuration Overview](../configuration/system/overview.md) - Configuration system architecture
- [Agents Configuration](../configuration/system/agents_config.md) - LLM and agent settings
- [System Configuration](../configuration/system/system_config.md) - Runtime and execution settings
- [MCP Configuration](../configuration/system/mcp_reference.md) - MCP server settings
- [Model Configuration](../configuration/models/overview.md) - Supported LLM providers
### Advanced Features
**Advanced Topics:**
- [Hybrid Actions](../ufo2/core_features/hybrid_actions.md) - GUI + API automation
- [Control Detection](../ufo2/core_features/control_detection/overview.md) - UIA + Vision detection
- [Knowledge Substrate](../ufo2/core_features/knowledge_substrate/overview.md) - RAG and learning
- [Multi-Action Execution](../ufo2/core_features/multi_action.md) - Speculative action batching
### Evaluation & Benchmarks
**Benchmarking:**
- [Benchmark Overview](../ufo2/evaluation/benchmark/overview.md) - Evaluation framework and datasets
- [Windows Agent Arena](../ufo2/evaluation/benchmark/windows_agent_arena.md) - 154 real Windows tasks
- [OSWorld](../ufo2/evaluation/benchmark/osworld.md) - Cross-application benchmarks
---
## ❓ Getting Help
- 📖 **Documentation**: [https://microsoft.github.io/UFO/](https://microsoft.github.io/UFO/)
- 🐛 **GitHub Issues**: [https://github.com/microsoft/UFO/issues](https://github.com/microsoft/UFO/issues) (preferred)
- 📧 **Email**: [ufo-agent@microsoft.com](mailto:ufo-agent@microsoft.com)
---
## 🎯 Next Steps
Now that UFO² is set up, explore these guides to unlock its full potential:
1. **[Configuration Customization](../configuration/system/overview.md)** - Fine-tune UFO² behavior
2. **[Knowledge Substrate Setup](../ufo2/core_features/knowledge_substrate/overview.md)** - Enable RAG capabilities
3. **[Creating Custom Agents](../tutorials/creating_app_agent/overview.md)** - Build specialized agents
4. **[MCP Integration](../mcp/overview.md)** - Extend with custom MCP servers
Happy automating with UFO²! 🛸
---
### Documents Docs Linux As Galaxy Device (`documents/docs/linux/as_galaxy_device.md`)
# Using Linux Agent as Galaxy Device
Configure Linux Agent as a sub-agent in UFO's Galaxy framework to enable cross-platform, multi-device task orchestration. Galaxy can coordinate Linux agents alongside Windows devices to execute complex workflows spanning multiple systems.
## Overview
The **Galaxy framework** provides multi-tier orchestration capabilities, allowing you to manage multiple device agents (Windows, Linux, etc.) from a central ConstellationAgent. When configured as a Galaxy device, LinuxAgent becomes a **sub-agent** that can:
- Execute Linux-specific subtasks assigned by Galaxy
- Participate in cross-platform workflows (e.g., Windows + Linux collaboration)
- Report execution status back to the orchestrator
- Be dynamically selected based on capabilities and metadata
For detailed information about LinuxAgent's design and capabilities, see [Linux Agent Overview](overview.md).
## Galaxy Architecture with Linux Agent
```mermaid
graph TB
User[User Request]
Galaxy[Galaxy ConstellationAgent Orchestrator]
subgraph "Device Pool"
Win1[Windows Device 1 HostAgent]
Win2[Windows Device 2 HostAgent]
Linux1[Linux Agent 1 CLI Executor]
Linux2[Linux Agent 2 CLI Executor]
Linux3[Linux Agent 3 CLI Executor]
end
User -->|Complex Task| Galaxy
Galaxy -->|Windows Subtask| Win1
Galaxy -->|Windows Subtask| Win2
Galaxy -->|Linux Subtask| Linux1
Galaxy -->|Linux Subtask| Linux2
Galaxy -->|Linux Subtask| Linux3
style Galaxy fill:#ffe1e1
style Linux1 fill:#e1f5ff
style Linux2 fill:#e1f5ff
style Linux3 fill:#e1f5ff
```
Galaxy orchestrates task decomposition, device selection based on capabilities, parallel execution, and result aggregation across all devices.
## Configuration Guide
### Step 1: Configure Device in `devices.yaml`
Add your Linux agent(s) to the device list in `config/galaxy/devices.yaml`:
**Example Configuration:**
```yaml
devices:
- device_id: "linux_agent_1"
server_url: "ws://172.23.48.1:5001/ws"
os: "linux"
capabilities:
- "server"
- "log_analysis"
- "file_operations"
- "database_management"
metadata:
os: "linux"
performance: "high"
logs_file_path: "/var/log/myapp/app.log"
dev_path: "/home/user/development/"
warning_log_pattern: "WARN"
error_log_pattern: "ERROR|FATAL"
description: "Production web server"
auto_connect: true
max_retries: 5
```
### Step 2: Understanding Configuration Fields
| Field | Required | Type | Description |
|-------|----------|------|-------------|
| `device_id` | ✅ Yes | string | **Unique identifier** - must match client `--client-id` |
| `server_url` | ✅ Yes | string | WebSocket URL - must match server endpoint |
| `os` | ✅ Yes | string | Operating system - set to `"linux"` |
| `capabilities` | ❌ Optional | list | Skills/capabilities for task routing |
| `metadata` | ❌ Optional | dict | Custom context for LLM-based task execution |
| `auto_connect` | ❌ Optional | boolean | Auto-connect on Galaxy startup (default: `true`) |
| `max_retries` | ❌ Optional | integer | Connection retry attempts (default: `5`) |
### Step 3: Capabilities-Based Task Routing
Galaxy uses the `capabilities` field to intelligently route subtasks to appropriate devices. Define capabilities based on server roles, task types, installed software, or data access requirements.
**Example Capability Configurations:**
**Web Server:**
```yaml
capabilities:
- "web_server"
- "nginx"
- "ssl_management"
- "log_analysis"
```
**Database Server:**
```yaml
capabilities:
- "database_server"
- "postgresql"
- "backup_management"
- "query_optimization"
```
**CI/CD Server:**
```yaml
capabilities:
- "ci_cd"
- "docker"
- "kubernetes"
- "deployment"
```
**Monitoring Server:**
```yaml
capabilities:
- "monitoring"
- "prometheus"
- "grafana"
- "alerting"
```
### Step 4: Metadata for Contextual Execution
The `metadata` field provides contextual information that the LLM uses when generating commands for the Linux agent.
**Metadata Examples:**
**Web Server Metadata:**
```yaml
metadata:
os: "linux"
logs_file_path: "/var/log/nginx/access.log"
error_log_path: "/var/log/nginx/error.log"
web_root: "/var/www/html"
ssl_cert_path: "/etc/letsencrypt/live/example.com/"
warning_log_pattern: "WARN"
error_log_pattern: "ERROR|FATAL"
performance: "high"
description: "Production nginx web server"
```
**Database Server Metadata:**
```yaml
metadata:
os: "linux"
logs_file_path: "/var/log/postgresql/postgresql.log"
data_path: "/var/lib/postgresql/14/main"
backup_path: "/mnt/backups/postgresql"
warning_log_pattern: "WARNING"
error_log_pattern: "ERROR|FATAL|PANIC"
performance: "high"
description: "Production PostgreSQL 14 database"
```
**Development Server Metadata:**
```yaml
metadata:
os: "linux"
dev_path: "/home/developer/projects"
logs_file_path: "/var/log/app/dev.log"
git_repo_path: "/home/developer/repos"
warning_log_pattern: "WARN"
error_log_pattern: "ERROR"
performance: "medium"
description: "Development and testing environment"
```
**How Metadata is Used:**
The LLM receives metadata in the system prompt, enabling context-aware command generation. For example, with the web server metadata above, when the user requests "Find all 500 errors in the last hour", the LLM can generate the appropriate command using the correct log path.
## Multi-Device Configuration Example
**Complete Galaxy Setup:**
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Starting Galaxy with Linux Agents
### Prerequisites
Ensure all components are running before starting Galaxy:
1. Device Agent Servers running on all machines
2. Device Agent Clients connected to their respective servers
3. MCP Services running on all Linux agents
4. LLM configured in `config/ufo/agents.yaml` (for UFO) or `config/galaxy/agent.yaml` (for Galaxy)
### Launch Sequence
**Step 1: Start all Device Agent Servers**
```bash
# On web server machine (192.168.1.101)
python -m ufo.server.app --port 5001
# On database server machine (192.168.1.102)
python -m ufo.server.app --port 5002
# On monitoring server machine (192.168.1.103)
python -m ufo.server.app --port 5003
```
**Step 2: Start all Linux Clients**
```bash
# On web server
python -m ufo.client.client \
--ws \
--ws-server ws://192.168.1.101:5001/ws \
--client-id linux_web_server \
--platform linux
# On database server
python -m ufo.client.client \
--ws \
--ws-server ws://192.168.1.102:5002/ws \
--client-id linux_db_server \
--platform linux
# On monitoring server
python -m ufo.client.client \
--ws \
--ws-server ws://192.168.1.103:5003/ws \
--client-id linux_monitoring \
--platform linux
```
**Step 3: Start all MCP Services**
```bash
# On each Linux machine
python -m ufo.client.mcp.http_servers.linux_mcp_server
```
**Step 4: Launch Galaxy**
```bash
# On your control machine (interactive mode)
python -m galaxy --interactive
```
**Or launch with a specific request:**
```bash
python -m galaxy "Your task description here"
```
Galaxy will automatically connect to all configured devices and display the orchestration interface.
## Example Multi-Device Workflows
### Workflow 1: Cross-Platform Data Processing
**User Request:**
> "Generate a sales report in Excel from the database, then email it to the team"
**Galaxy Orchestration:**
```mermaid
sequenceDiagram
participant User
participant Galaxy
participant LinuxDB as Linux DB Server
participant WinDesktop as Windows Desktop
User->>Galaxy: Request sales report
Galaxy->>Galaxy: Decompose task
Note over Galaxy,LinuxDB: Subtask 1: Extract data
Galaxy->>LinuxDB: "Export sales data from PostgreSQL to CSV"
LinuxDB->>LinuxDB: Execute SQL query
LinuxDB->>LinuxDB: Generate CSV file
LinuxDB-->>Galaxy: CSV file location
Note over Galaxy,WinDesktop: Subtask 2: Create Excel report
Galaxy->>WinDesktop: "Create Excel report from CSV"
WinDesktop->>WinDesktop: Open Excel
WinDesktop->>WinDesktop: Import CSV
WinDesktop->>WinDesktop: Format report
WinDesktop-->>Galaxy: Excel file created
Note over Galaxy,WinDesktop: Subtask 3: Send email
Galaxy->>WinDesktop: "Email report to team"
WinDesktop->>WinDesktop: Open Outlook
WinDesktop->>WinDesktop: Attach file
WinDesktop->>WinDesktop: Send email
WinDesktop-->>Galaxy: Email sent
Galaxy-->>User: Task completed
```
### Workflow 2: Multi-Server Log Analysis
**User Request:**
> "Check all servers for error patterns in the last hour and summarize findings"
**Galaxy Orchestration:**
1. **Linux Web Server**: Analyze nginx logs for HTTP 500 errors
2. **Linux DB Server**: Check PostgreSQL logs for query failures
3. **Linux Monitoring**: Review Prometheus alerts
4. **Galaxy**: Aggregate results and generate summary report
### Workflow 3: Deployment Pipeline
**User Request:**
> "Deploy the new application version to production"
**Galaxy Orchestration:**
1. **Linux CI/CD Server**: Build Docker image from Git repository
2. **Linux Web Server**: Stop current service, pull new image, restart
3. **Linux DB Server**: Run database migrations
4. **Linux Monitoring**: Verify health checks and metrics
5. **Windows Desktop**: Send deployment notification email
---
## Task Assignment Behavior
### How Galaxy Routes Tasks to Linux Agents
Galaxy's ConstellationAgent uses several factors to select the appropriate device for each subtask:
| Factor | Description | Example |
|--------|-------------|---------|
| **Capabilities** | Match subtask requirements to device capabilities | `"database_server"` → DB server agent |
| **OS Requirement** | Platform-specific tasks routed to correct OS | Linux commands → Linux agents |
| **Metadata Context** | Use device-specific paths and configurations | Log analysis → agent with correct log path |
| **Device Status** | Only assign to online, healthy devices | Skip offline or failing devices |
| **Load Balancing** | Distribute tasks across similar devices | Round-robin across web servers |
### Example Task Decomposition
**User Request:**
> "Monitor system health across all servers and alert if any issues found"
**Galaxy Decomposition:**
```yaml
Task 1:
Description: "Check web server health"
Target: linux_web_server
Reason: Has "web_server" capability
Task 2:
Description: "Check database health"
Target: linux_db_server
Reason: Has "database_server" capability
Task 3:
Description: "Review monitoring alerts"
Target: linux_monitoring
Reason: Has "monitoring" capability
Task 4:
Description: "Aggregate results and send alert email"
Target: windows_desktop_1
Reason: Has "email" capability
```
## Critical Configuration Requirements
!!!danger "Configuration Validation"
Ensure these match exactly or Galaxy cannot control the device:
- **Device ID**: `device_id` in `devices.yaml` must match `--client-id` in client command
- **Server URL**: `server_url` in `devices.yaml` must match `--ws-server` in client command
- **Platform**: Must include `--platform linux` in client command
## Monitoring & Debugging
### Verify Device Registration
**Check Galaxy device pool:**
```bash
# List all connected devices
curl http://:5000/api/devices
```
**Expected response:**
```json
{
"devices": [
{
"device_id": "linux_web_server",
"os": "linux",
"status": "online",
"capabilities": ["web_server", "nginx", "log_analysis"]
},
{
"device_id": "linux_db_server",
"os": "linux",
"status": "online",
"capabilities": ["database_server", "postgresql"]
}
]
}
```
### View Task Assignments
Galaxy logs show task routing decisions:
```log
INFO - [Galaxy] Task decomposition: 3 subtasks created
INFO - [Galaxy] Subtask 1 → linux_web_server (capability match: web_server)
INFO - [Galaxy] Subtask 2 → linux_db_server (capability match: database_server)
INFO - [Galaxy] Subtask 3 → windows_desktop_1 (capability match: email)
```
### Troubleshooting Device Connection
**Issue**: Linux agent not appearing in Galaxy device pool
**Diagnosis:**
1. Check if client is connected to server:
```bash
curl http://192.168.1.101:5001/api/clients
```
2. Verify `devices.yaml` configuration matches client parameters
3. Check Galaxy logs for connection errors
4. Ensure `auto_connect: true` in `devices.yaml`
## Related Documentation
- [Linux Agent Overview](overview.md) - Architecture and design principles
- [Quick Start Guide](../getting_started/quick_start_linux.md) - Step-by-step setup
- [Galaxy Overview](../galaxy/overview.md) - Multi-device orchestration framework
- [Galaxy Quick Start](../getting_started/quick_start_galaxy.md) - Galaxy deployment guide
- [Constellation Orchestrator](../galaxy/constellation_orchestrator/overview.md) - Task orchestration
- [Galaxy Devices Configuration](../configuration/system/galaxy_devices.md) - Complete device configuration reference
## Summary
Using Linux Agent as a Galaxy device enables multi-device orchestration with capability-based routing, metadata context for LLM-aware command generation, parallel execution, and seamless cross-platform workflows between Linux and Windows agents.
---
### Documents Docs Linux Commands (`documents/docs/linux/commands.md`)
# LinuxAgent MCP Commands
LinuxAgent interacts with Linux systems through MCP (Model Context Protocol) tools provided by the Linux MCP Server. These tools provide atomic building blocks for CLI task execution, isolating system-specific operations within the MCP server layer.
## Command Architecture
### MCP Server Integration
LinuxAgent commands are executed through the MCP server infrastructure:
```mermaid
graph LR
A[LinuxAgent] --> B[Command Dispatcher]
B --> C[MCP Server]
C --> D[Linux Shell]
D --> E[stdout/stderr]
E --> C
C --> B
B --> A
```
### Command Dispatcher
The command dispatcher routes commands to the appropriate MCP server:
```python
from aip.messages import Command
# Create command
command = Command(
tool_name="execute_command",
parameters={"command": "df -h", "timeout": 30},
tool_type="action"
)
# Execute command via dispatcher
results = await command_dispatcher.execute_commands([command])
execution_result = results[0].result
```
## Primary MCP Tools
### 1. execute_command - Execute Shell Commands
**Purpose**: Execute arbitrary shell commands and capture structured results.
#### Tool Specification
```python
tool_name = "execute_command"
parameters = {
"command": "df -h", # Shell command to execute
"timeout": 30, # Execution timeout (seconds, default: 30)
"cwd": "/home/user" # Optional working directory
}
```
#### Execution Flow
```mermaid
sequenceDiagram
participant Agent
participant Dispatcher
participant MCP
participant Shell
Agent->>Dispatcher: execute_command: df -h
Dispatcher->>MCP: Forward command
MCP->>Shell: Execute: df -h
Shell->>Shell: Run command
Shell-->>MCP: stdout + stderr + exit_code
MCP->>MCP: Structure result
MCP-->>Dispatcher: Execution result
Dispatcher-->>Agent: Structured result
```
#### Result Structure
```python
{
"success": True, # Boolean indicating success
"exit_code": 0, # Process exit code
"stdout": "Filesystem Size Used Avail Use% Mounted on\n/dev/sda1 100G 50G 46G 52% /\n",
"stderr": "" # Standard error output
}
```
#### Common Use Cases
| Use Case | Command Example | Description |
|----------|----------------|-------------|
| **File Operations** | `ls -la /home/user` | List directory contents |
| **Text Processing** | `grep "error" /var/log/syslog` | Search log files |
| **System Monitoring** | `top -bn1` | Check system processes |
| **Disk Management** | `df -h` | Check disk space |
| **Network Operations** | `ping -c 4 example.com` | Test network connectivity |
| **Archive Creation** | `tar -czf backup.tar.gz /data` | Create compressed archives |
| **Package Management** | `apt list --installed` | List installed packages |
#### Error Handling
**Exit Code Interpretation**:
- **0**: Success
- **1-125**: Command-specific errors
- **126**: Command not executable
- **127**: Command not found
- **128+n**: Terminated by signal n
**Example Error Result**:
```python
{
"success": False,
"error": "Command not found: invalid_cmd"
}
```
#### Security Considerations
!!!warning "Command Safety"
The MCP server blocks dangerous commands including:
- `rm -rf /` - Recursive root deletion
- Fork bombs - `:(){ :|:& };:`
- `mkfs` - Filesystem formatting
- `dd if=/dev/zero` - Device overwriting
- `shutdown`, `reboot` - System shutdown
Commands execute with user permissions, no automatic privilege escalation. Timeout protection prevents hung processes.
### 2. get_system_info - Collect System Information
**Purpose**: Gather basic Linux system information using standard commands.
#### Tool Specification
```python
tool_name = "get_system_info"
parameters = {} # No parameters required
```
#### Information Collected
The tool executes these commands and returns their output:
| Info Type | Command | Data Returned |
|-----------|---------|---------------|
| **uname** | `uname -a` | System and kernel information |
| **uptime** | `uptime` | System uptime and load averages |
| **memory** | `free -h` | Memory usage statistics (human-readable) |
| **disk** | `df -h` | Disk space for all mounted filesystems |
#### Execution Flow
```mermaid
sequenceDiagram
participant Agent
participant Dispatcher
participant MCP
participant System
Agent->>Dispatcher: get_system_info
Dispatcher->>MCP: Forward request
MCP->>System: Execute uname, uptime, free, df
System-->>MCP: Command outputs
MCP->>MCP: Aggregate results
MCP-->>Dispatcher: Structured info
Dispatcher-->>Agent: System information
```
#### Result Example
```python
{
"uname": "Linux hostname 5.15.0-91-generic #101-Ubuntu SMP x86_64 GNU/Linux",
"uptime": " 14:23:45 up 5 days, 3:12, 2 users, load average: 0.52, 0.58, 0.59",
"memory": " total used free shared buff/cache available\nMem: 15Gi 8.2Gi 1.5Gi 256Mi 5.8Gi 7.0Gi\nSwap: 8.0Gi 512Mi 7.5Gi",
"disk": "Filesystem Size Used Avail Use% Mounted on\n/dev/sda1 100G 50G 46G 52% /\n/dev/sdb1 500G 200G 276G 42% /data"
}
```
## Command Execution Pipeline
### Atomic Building Blocks
The MCP tools `execute_command` and `get_system_info` serve as atomic operations:
```mermaid
graph TD
A[User Request] --> B[LLM Reasoning]
B --> C{Select Tool}
C -->|Execute CLI| D[execute_command]
C -->|Get System Info| E[get_system_info]
D --> F[Capture Result]
E --> F
F --> G[Update Memory]
G --> H{Task Complete?}
H -->|No| B
H -->|Yes| I[FINISH]
```
### Isolation of System Operations
By isolating system operations in the MCP server layer, the architecture achieves clear separation: the Agent layer focuses on LLM reasoning and workflow orchestration, while the MCP layer handles system-specific command execution. This provides testability (commands can be mocked) and portability (MCP servers can be deployed remotely).
## Command Composition
### Sequential Execution
LinuxAgent executes commands sequentially, building on previous results:
```python
# Round 1: Check disk space
{
"action": {
"tool": "execute_command",
"arguments": {"command": "df -h /data"}
}
}
# Result: 276GB available
# Round 2: Create backup (informed by Round 1 result)
{
"action": {
"tool": "execute_command",
"arguments": {
"command": "tar -czf /data/backup.tar.gz /home/user"
}
}
}
```
### Conditional Execution
LLM can make decisions based on command results:
```python
# Round 1: Check if file exists
{
"action": {
"tool": "execute_command",
"arguments": {"command": "ls /data/backup.tar.gz"}
}
}
# Result: exit_code=2 (file not found)
# Round 2: File doesn't exist, create it
{
"thought": "Backup doesn't exist, creating new one",
"action": {
"tool": "execute_command",
"arguments": {"command": "tar -czf /data/backup.tar.gz /home/user"}
}
}
```
### Error Recovery
Commands can be retried or alternatives attempted:
```python
# Round 1: Try privileged command
{
"action": {
"tool": "execute_command",
"arguments": {"command": "systemctl restart nginx"}
}
}
# Result: success=False, error="Permission denied"
# Round 2: Switch to user-level alternative
{
"thought": "Don't have sudo access, using alternative approach",
"action": {
"tool": "execute_command",
"arguments": {"command": "kill -HUP $(cat /var/run/nginx.pid)"}
}
}
```
---
## Command Implementation
### MCP Server Location
The MCP server implementation for Linux commands can be found in:
```
ufo/client/mcp/http_servers/
└── linux_mcp_server.py
```
### Example Implementation Skeleton
```
/* Detailed source-code truncated for AI context efficiency. */
```
---
## Best Practices
### Tool Usage
- Use `get_system_info` for quick system overview
- Use `execute_command` for custom or complex operations
- Check `success` field and `exit_code` to detect errors
- Parse `stdout` for structured data when possible
- Set timeouts appropriately to prevent hung processes
### Security
!!!warning "Security Best Practices"
The MCP server has built-in protections, but be cautious:
- Dangerous commands are automatically blocked
- Commands execute with user permissions only
- Avoid sudo when possible (requires user interaction)
- Sanitize outputs before logging (may contain sensitive data)
### Error Handling
- Check `success` field before considering command successful
- Parse `stderr` for error messages
- Implement retries for transient errors
- Provide alternatives when primary approach fails
## Comparison with Other Agent Commands
| Agent | Command Types | Execution Layer | Result Format |
|-------|--------------|-----------------|---------------|
| **LinuxAgent** | CLI + SysInfo | MCP server | success/exit_code/stdout/stderr |
| **AppAgent** | UI + API | Automator + MCP | UI state + API responses |
| **HostAgent** | Desktop + Shell | Automator + MCP | Desktop state + results |
LinuxAgent's command set is intentionally minimal and focused:
- **execute_command**: General-purpose command execution
- **get_system_info**: Standardized system information
This simplicity reflects the CLI environment's text-based, command-driven nature.
## Next Steps
- [State Machine](state.md) - Understand how command execution fits into the FSM
- [Processing Strategy](strategy.md) - See how commands are integrated into the 3-phase pipeline
- [Overview](overview.md) - Return to LinuxAgent architecture overview
- [MCP Overview](../mcp/overview.md) - MCP server implementation details
---
### Documents Docs Linux State (`documents/docs/linux/state.md`)
# LinuxAgent State Machine
LinuxAgent uses a **3-state finite state machine (FSM)** to manage CLI task execution flow. The minimal state set captures essential execution progression while maintaining simplicity and predictability. States transition based on LLM decisions and command execution results.
## State Machine Architecture
### State Enumeration
```python
class LinuxAgentStatus(Enum):
"""Store the status of the linux agent"""
CONTINUE = "CONTINUE" # Task is ongoing, requires further commands
FINISH = "FINISH" # Task completed successfully
FAIL = "FAIL" # Task cannot proceed, unrecoverable error
```
### State Management
LinuxAgent states are managed by `LinuxAgentStateManager`, which implements the agent state registry pattern:
```python
class LinuxAgentStateManager(AgentStateManager):
"""Manages the states of the linux agent"""
_state_mapping: Dict[str, Type[LinuxAgentState]] = {}
@property
def none_state(self) -> AgentState:
return NoneLinuxAgentState()
```
All LinuxAgent states are registered using the `@LinuxAgentStateManager.register` decorator, enabling dynamic state lookup by name.
## State Transition Diagram
Figure: Lifecycle state transitions of the LinuxAgent. The agent starts in CONTINUE state, executes CLI commands iteratively, and transitions to FINISH upon completion or FAIL upon encountering unrecoverable errors.
## State Definitions
### 1. CONTINUE State
**Purpose**: Active execution state where LinuxAgent processes the user request and executes CLI commands.
```python
@LinuxAgentStateManager.register
class ContinueLinuxAgentState(LinuxAgentState):
"""The class for the continue linux agent state"""
async def handle(self, agent: "LinuxAgent", context: Optional["Context"] = None):
"""Execute the 3-phase processing pipeline"""
await agent.process(context)
def is_round_end(self) -> bool:
return False # Round continues
def is_subtask_end(self) -> bool:
return False # Subtask continues
@classmethod
def name(cls) -> str:
return LinuxAgentStatus.CONTINUE.value
```
| Property | Value |
|----------|-------|
| **Type** | Active |
| **Processor Executed** | ✓ Yes (3 phases) |
| **Round Ends** | No |
| **Subtask Ends** | No |
| **Duration** | Single round |
| **Next States** | CONTINUE, FINISH, FAIL |
**Behavior**:
1. Constructs prompts with previous execution results
2. Gets next CLI command from LLM
3. Executes command via MCP server
4. Updates memory with execution results
5. Determines next state based on LLM response
**State Transition Logic**:
- **CONTINUE → CONTINUE**: Task requires more commands to complete
- **CONTINUE → FINISH**: LLM determines task is complete
- **CONTINUE → FAIL**: Unrecoverable error encountered (e.g., permission denied, resource unavailable)
### 2. FINISH State
**Purpose**: Terminal state indicating successful task completion.
```python
@LinuxAgentStateManager.register
class FinishLinuxAgentState(LinuxAgentState):
"""The class for the finish linux agent state"""
def next_agent(self, agent: "LinuxAgent") -> "LinuxAgent":
return agent
def next_state(self, agent: "LinuxAgent") -> LinuxAgentState:
return FinishLinuxAgentState() # Remains in FINISH
def is_subtask_end(self) -> bool:
return True # Subtask completed
def is_round_end(self) -> bool:
return True # Round ends
@classmethod
def name(cls) -> str:
return LinuxAgentStatus.FINISH.value
```
| Property | Value |
|----------|-------|
| **Type** | Terminal |
| **Processor Executed** | ✗ No |
| **Round Ends** | Yes |
| **Subtask Ends** | Yes |
| **Duration** | Permanent |
| **Next States** | FINISH (no transition) |
**Behavior**:
- Signals task completion to session manager
- No further processing occurs
- Agent instance can be terminated
FINISH state is reached when all required CLI commands have been executed successfully, the LLM determines the user request has been fulfilled, and no errors or exceptions occurred during execution.
### 3. FAIL State
**Purpose**: Terminal state indicating task failure due to unrecoverable errors.
```python
@LinuxAgentStateManager.register
class FailLinuxAgentState(LinuxAgentState):
"""The class for the fail linux agent state"""
def next_agent(self, agent: "LinuxAgent") -> "LinuxAgent":
return agent
def next_state(self, agent: "LinuxAgent") -> LinuxAgentState:
return FinishLinuxAgentState() # Transitions to FINISH for cleanup
def is_round_end(self) -> bool:
return True # Round ends
def is_subtask_end(self) -> bool:
return True # Subtask failed
@classmethod
def name(cls) -> str:
return LinuxAgentStatus.FAIL.value
```
| Property | Value |
|----------|-------|
| **Type** | Terminal (Error) |
| **Processor Executed** | ✗ No |
| **Round Ends** | Yes |
| **Subtask Ends** | Yes |
| **Duration** | Transitions to FINISH |
| **Next States** | FINISH |
**Behavior**:
- Logs failure reason and context
- Transitions to FINISH state for cleanup
- Session manager receives failure status
!!!error "Failure Conditions"
FAIL state is reached when insufficient privileges prevent command execution, required system resources are not accessible (disk full, network unreachable), repeated command syntax errors occur, the LLM explicitly indicates task cannot be completed, or task requirements exceed current system capabilities.
**Error Recovery**:
While FAIL is a terminal state, the error information is logged for debugging:
```python
# Example error logging in FAIL state
agent.logger.error(f"Task failed: {error_message}")
agent.logger.debug(f"Last command: {last_command}")
agent.logger.debug(f"Command output: {stderr}")
```
## State Transition Rules
### Transition Decision Logic
State transitions are determined by the LLM's response in the **CONTINUE** state:
```python
# LLM returns status in response
parsed_response = {
"action": {
"command": "df -h",
"status": "CONTINUE" # or "FINISH" or "FAIL"
},
"thought": "Need to check disk space first"
}
# Agent updates its status based on LLM decision
agent.status = parsed_response["action"]["status"]
next_state = LinuxAgentStateManager().get_state(agent.status)
```
### Transition Matrix
| Current State | Condition | Next State | Trigger |
|---------------|-----------|------------|---------|
| **CONTINUE** | LLM returns CONTINUE | CONTINUE | More commands needed |
| **CONTINUE** | LLM returns FINISH | FINISH | Task completed |
| **CONTINUE** | LLM returns FAIL | FAIL | Unrecoverable error |
| **CONTINUE** | Exception raised | FAIL | System error |
| **FINISH** | Any | FINISH | No transition |
| **FAIL** | Any | FINISH | Cleanup transition |
## State-Specific Processing
### CONTINUE State Processing Pipeline
When in CONTINUE state, LinuxAgent executes the full 3-phase pipeline:
```mermaid
graph TD
A[CONTINUE State] --> B[Phase 1: LLM Interaction]
B --> C[Phase 2: Action Execution]
C --> D[Phase 3: Memory Update]
D --> E{Check Status}
E -->|CONTINUE| A
E -->|FINISH| F[FINISH State]
E -->|FAIL| G[FAIL State]
```
### Terminal States (FINISH / FAIL)
Terminal states perform no processing:
- **FINISH**: Clean termination, results available in memory
- **FAIL**: Error termination, error details logged
## Deterministic Control Flow
The 3-state design ensures deterministic, traceable execution with predictable behavior (every execution path is well-defined), debuggability (state transitions are logged and traceable), testability (finite state space simplifies testing), and maintainability (simple state set reduces complexity).
## Comparison with Other Agents
| Agent | States | Complexity | Use Case |
|-------|--------|------------|----------|
| **LinuxAgent** | 3 | Minimal | CLI task execution |
| **AppAgent** | 6 | Moderate | Windows app automation |
| **HostAgent** | 7 | High | Desktop orchestration |
LinuxAgent's minimal 3-state design reflects its focused scope: execute CLI commands to fulfill user requests. The simplified state machine eliminates unnecessary complexity while maintaining robust error handling and completion detection.
## Implementation Details
The state machine implementation can be found in:
```
ufo/agents/states/linux_agent_state.py
```
Key classes:
- `LinuxAgentStatus`: State enumeration
- `LinuxAgentStateManager`: State registry and lookup
- `LinuxAgentState`: Abstract base class
- `ContinueLinuxAgentState`: Active execution state
- `FinishLinuxAgentState`: Successful completion state
- `FailLinuxAgentState`: Error termination state
- `NoneLinuxAgentState`: Initial/undefined state
## Next Steps
- [Processing Strategy](strategy.md) - Understand the 3-phase processing pipeline executed in CONTINUE state
- [MCP Commands](commands.md) - Explore CLI command execution and system information retrieval
- [Overview](overview.md) - Return to LinuxAgent architecture overview
---
### Documents Docs Linux Strategy (`documents/docs/linux/strategy.md`)
# LinuxAgent Processing Strategy
LinuxAgent executes a **3-phase processing pipeline** in the **CONTINUE** state. Each phase handles a specific aspect of CLI task execution: LLM decision making, action execution, and memory recording. This streamlined design separates prompt construction and LLM reasoning from command execution and state updates, enhancing modularity and traceability.
## Strategy Assembly
Processing strategies are assembled and orchestrated by the `LinuxAgentProcessor` class defined in `ufo/agents/processors/customized/customized_agent_processor.py`. The processor coordinates the 3-phase pipeline execution.
### LinuxAgentProcessor Overview
The `LinuxAgentProcessor` extends `CustomizedProcessor` and manages the Linux-specific workflow:
```python
class LinuxAgentProcessor(CustomizedProcessor):
"""
Processor for Linux MCP Agent.
Manages CLI command execution workflow with:
- LLM-based command generation
- MCP-based command execution
- Memory-based result tracking
"""
def _setup_strategies(self) -> None:
"""Setup the 3-phase processing pipeline"""
# Phase 1: LLM Interaction (critical - fail_fast=True)
self.strategies[ProcessingPhase.LLM_INTERACTION] = (
LinuxLLMInteractionStrategy(fail_fast=True)
)
# Phase 2: Action Execution (graceful - fail_fast=False)
self.strategies[ProcessingPhase.ACTION_EXECUTION] = (
LinuxActionExecutionStrategy(fail_fast=False)
)
# Phase 3: Memory Update (graceful - fail_fast=False)
self.strategies[ProcessingPhase.MEMORY_UPDATE] = (
AppMemoryUpdateStrategy(fail_fast=False)
)
```
### Strategy Registration
| Phase | Strategy Class | fail_fast | Rationale |
|-------|---------------|-----------|-----------|
| **LLM_INTERACTION** | `LinuxLLMInteractionStrategy` | ✓ True | LLM failure requires immediate recovery |
| **ACTION_EXECUTION** | `LinuxActionExecutionStrategy` | ✗ False | Command failures can be handled gracefully |
| **MEMORY_UPDATE** | `AppMemoryUpdateStrategy` | ✗ False | Memory failures shouldn't block execution |
**Fail-Fast vs Graceful:**
- **fail_fast=True**: Critical phases where errors should immediately transition to FAIL state
- **fail_fast=False**: Non-critical phases where errors can be logged and execution continues
## Three-Phase Pipeline
### Pipeline Execution Flow
```mermaid
graph LR
A[CONTINUE State] --> B[Phase 1: LLM Interaction]
B --> C[Phase 2: Action Execution]
C --> D[Phase 3: Memory Update]
D --> E[Determine Next State]
E --> F{Status?}
F -->|CONTINUE| A
F -->|FINISH| G[FINISH State]
F -->|FAIL| H[FAIL State]
```
## Phase 1: LLM Interaction Strategy
**Purpose**: Construct prompts with execution context and obtain next CLI command from LLM.
### Strategy Implementation
```python
@depends_on("request")
@provides("parsed_response", "response_text", "llm_cost",
"prompt_message", "action", "thought", "comment")
class LinuxLLMInteractionStrategy(AppLLMInteractionStrategy):
"""
Strategy for LLM interaction with Linux Agent specific prompting.
Handles:
- Context-aware prompt construction with previous results
- LLM interaction with retry logic
- Response parsing and validation
"""
async def execute(self, agent: "LinuxAgent",
context: ProcessingContext) -> ProcessingResult:
"""Execute LLM interaction for Linux Agent"""
```
### Phase 1 Workflow
```mermaid
sequenceDiagram
participant Strategy
participant Agent
participant Prompter
participant LLM
Strategy->>Agent: Get previous plan
Strategy->>Agent: Get blackboard context
Agent-->>Strategy: Previous execution results
Strategy->>Prompter: Construct prompt
Prompter->>Prompter: Build system message
Prompter->>Prompter: Build user message with context
Prompter-->>Strategy: Complete prompt
Strategy->>LLM: Send prompt
LLM-->>Strategy: CLI command + status
Strategy->>Strategy: Parse response
Strategy->>Strategy: Validate command
Strategy-->>Agent: Parsed response + cost
```
### Prompt Construction
The strategy constructs comprehensive prompts using:
1. **System Message**: Agent role and capabilities
2. **User Request**: Original task description
3. **Previous Results**: Command outputs from prior executions
4. **Blackboard Context**: Shared state from other agents (if any)
5. **Last Success Actions**: Previously successful commands
```python
prompt_message = agent.message_constructor(
dynamic_examples=[], # Few-shot examples (optional)
dynamic_knowledge="", # Retrieved knowledge (optional)
plan=plan, # Previous execution plan
request=request, # User request
blackboard_prompt=blackboard_prompt, # Shared context
last_success_actions=last_success_actions # Successful commands
)
```
### LLM Response Format
The LLM returns a structured response:
```json
{
"thought": "Need to check disk space before creating backup",
"action": {
"tool": "execute_command",
"arguments": {
"command": "df -h"
},
"status": "CONTINUE"
},
"comment": "Checking available disk space"
}
```
### Proactive Information Gathering
LinuxAgent proactively requests system information when needed, eliminating unnecessary overhead and increasing responsiveness.
### Error Handling
```python
try:
response_text, llm_cost = await self._get_llm_response(
agent, prompt_message
)
parsed_response = self._parse_app_response(agent, response_text)
return ProcessingResult(
success=True,
data={
"parsed_response": parsed_response,
"response_text": response_text,
"llm_cost": llm_cost,
...
}
)
except Exception as e:
self.logger.error(f"LLM interaction failed: {str(e)}")
return self.handle_error(e, ProcessingPhase.LLM_INTERACTION, context)
```
---
## Phase 2: Action Execution Strategy
**Purpose**: Execute CLI commands returned by LLM and capture structured results.
### Strategy Implementation
```python
class LinuxActionExecutionStrategy(AppActionExecutionStrategy):
"""
Strategy for executing actions in Linux Agent.
Handles:
- CLI command execution via MCP server
- Result capturing (stdout, stderr, exit code)
- Error handling and retry logic
"""
async def execute(self, agent: "LinuxAgent",
context: ProcessingContext) -> ProcessingResult:
"""Execute Linux Agent actions"""
```
### Phase 2 Workflow
```mermaid
sequenceDiagram
participant Strategy
participant MCP
participant Linux
Strategy->>Strategy: Extract command from LLM response
Strategy->>MCP: execute_command: df -h
MCP->>Linux: Execute shell command
Linux-->>MCP: stdout + stderr + exit_code
MCP-->>Strategy: Execution result
Strategy->>Strategy: Create action info
Strategy->>Strategy: Format for memory
Strategy-->>Agent: Execution results
```
### Command Execution
The strategy dispatches commands to the MCP server:
```python
# Extract parsed LLM response
parsed_response: AppAgentResponse = context.get_local("parsed_response")
command_dispatcher = context.global_context.command_dispatcher
# Execute the command via MCP
execution_results = await self._execute_app_action(
command_dispatcher,
parsed_response.action
)
```
### Result Capture
Execution results are structured for downstream processing:
```python
{
"success": True,
"exit_code": 0,
"stdout": "Filesystem Size Used Avail Use% Mounted on\n/dev/sda1 100G 50G 46G 52% /",
"stderr": ""
}
```
### Action Info Creation
Results are formatted into `ActionCommandInfo` objects:
```python
actions = self._create_action_info(
parsed_response.action,
execution_results,
)
action_info = ListActionCommandInfo(actions)
action_info.color_print() # Pretty print to console
```
### Error Handling
```python
try:
execution_results = await self._execute_app_action(...)
return ProcessingResult(
success=True,
data={
"execution_result": execution_results,
"action_info": action_info,
"control_log": control_log,
"status": status
}
)
except Exception as e:
self.logger.error(f"Action execution failed: {traceback.format_exc()}")
return self.handle_error(e, ProcessingPhase.ACTION_EXECUTION, context)
```
---
## Phase 3: Memory Update Strategy
**Purpose**: Persist execution results and commands into agent memory for future reference.
### Strategy Implementation
LinuxAgent reuses the `AppMemoryUpdateStrategy` from the app agent framework:
```python
self.strategies[ProcessingPhase.MEMORY_UPDATE] = AppMemoryUpdateStrategy(
fail_fast=False # Memory failures shouldn't stop process
)
```
### Phase 3 Workflow
```mermaid
sequenceDiagram
participant Strategy
participant Memory
participant Context
Strategy->>Context: Get execution results
Strategy->>Context: Get LLM response
Strategy->>Memory: Create memory item
Memory->>Memory: Store command
Memory->>Memory: Store stdout/stderr
Memory->>Memory: Store timestamp
Strategy->>Context: Update round result
Strategy-->>Agent: Memory updated
```
### Memory Structure
Each execution round is stored as a memory item:
```python
{
"round": 1,
"request": "Check disk space and create backup",
"thought": "Need to check disk space first",
"action": {
"command": "EXEC_CLI",
"parameters": {"command": "df -h"}
},
"result": {
"stdout": "Filesystem Size Used...",
"stderr": "",
"exit_code": 0
},
"status": "CONTINUE",
"timestamp": "2025-11-06T10:30:45"
}
```
### Iterative Refinement
Memory enables iterative refinement:
1. **Round 1**: Check disk space → Result: 50G available
2. **Round 2**: Create backup (knowing 50G is available)
3. **Round 3**: Verify backup creation
Each round builds on previous results stored in memory.
### Error Recovery
Memory also stores errors for recovery:
```python
{
"round": 2,
"action": {"tool": "execute_command", "arguments": {"command": "invalid_cmd"}},
"result": {
"success": False,
"error": "Command not found: invalid_cmd"
},
"status": "FAIL"
}
```
## Middleware Stack
LinuxAgent uses specialized middleware for logging:
```python
def _setup_middleware(self) -> None:
"""Setup middleware pipeline for Linux Agent"""
self.middleware_chain = [LinuxLoggingMiddleware()]
```
### LinuxLoggingMiddleware
Provides enhanced logging specific to Linux operations:
```python
class LinuxLoggingMiddleware(AppAgentLoggingMiddleware):
"""Specialized logging middleware for Linux Agent"""
def starting_message(self, context: ProcessingContext) -> str:
request = context.get_local("request")
return f"Completing the user request [{request}] on Linux."
```
**Logged Information**:
- User request
- Each CLI command executed
- Command outputs (stdout/stderr)
- Execution timestamps
- State transitions
- LLM costs
---
## Context Finalization
After processing, the processor updates global context:
```python
def _finalize_processing_context(self, processing_context: ProcessingContext):
"""Finalize processing context by updating ContextNames fields"""
super()._finalize_processing_context(processing_context)
try:
result = processing_context.get_local("result")
if result:
self.global_context.set(ContextNames.ROUND_RESULT, result)
except Exception as e:
self.logger.warning(f"Failed to update context: {e}")
```
This makes execution results available to:
- Subsequent rounds (iterative execution)
- Other agents (if part of multi-agent workflow)
- Session manager (for monitoring and logging)
---
## Strategy Dependency Graph
The three phases have clear dependencies:
```mermaid
graph TD
A[request] --> B[Phase 1: LLM Interaction]
B --> C[parsed_response]
B --> D[llm_cost]
B --> E[prompt_message]
C --> F[Phase 2: Action Execution]
F --> G[execution_result]
F --> H[action_info]
C --> I[Phase 3: Memory Update]
G --> I
H --> I
I --> J[Memory Updated]
J --> K[Next Round or Terminal State]
```
Dependencies are declared using decorators:
```python
@depends_on("request")
@provides("parsed_response", "response_text", "llm_cost", ...)
class LinuxLLMInteractionStrategy(AppLLMInteractionStrategy):
...
```
---
## Modular Design Benefits
The 3-phase strategy design provides:
!!!success "Modularity Benefits"
- **Separation of Concerns**: LLM reasoning, command execution, and memory are isolated
- **Testability**: Each phase can be tested independently
- **Extensibility**: New strategies can be added without modifying existing code
- **Reusability**: Memory strategy is shared with AppAgent
- **Maintainability**: Clear boundaries between decision-making and execution
- **Traceability**: Each phase logs its operations independently
---
## Comparison with Other Agents
| Agent | Phases | Data Collection | LLM | Action | Memory |
|-------|--------|----------------|-----|--------|--------|
| **LinuxAgent** | 3 | ✗ None | ✓ CLI commands | ✓ MCP execute_command | ✓ Results |
| **AppAgent** | 4 | ✓ Screenshots + UI | ✓ UI actions | ✓ GUI + API | ✓ Results |
| **HostAgent** | 4 | ✓ Desktop snapshot | ✓ App selection | ✓ Orchestration | ✓ Results |
LinuxAgent omits the **DATA_COLLECTION** phase because there's no GUI to capture (CLI-based), system info is obtained on-demand via MCP tools, and previous execution results provide necessary context. This reflects the proactive information gathering principle.
## Implementation Location
The strategy implementations can be found in:
```
ufo/agents/processors/
├── customized/
│ └── customized_agent_processor.py # LinuxAgentProcessor
└── strategies/
└── linux_agent_strategy.py # Linux-specific strategies
```
Key classes:
- `LinuxAgentProcessor`: Strategy orchestrator
- `LinuxLLMInteractionStrategy`: Prompt construction and LLM interaction
- `LinuxActionExecutionStrategy`: CLI command execution
- `LinuxLoggingMiddleware`: Enhanced logging
## Next Steps
- [MCP Commands](commands.md) - Explore the CLI execution commands used by LinuxAgent
- [State Machine](state.md) - Understand the 3-state FSM that controls strategy execution
- [Overview](overview.md) - Return to LinuxAgent architecture overview
---
### Documents Docs Mcp Action (`documents/docs/mcp/action.md`)
# Action Servers
## Overview
**Action Servers** provide tools that modify system state by executing actions. These servers enable agents to interact with the environment, automate tasks, and implement decisions.
**Action servers are the only servers whose tools can be selected by the LLM agent.** At each step, the agent chooses which action tool to execute based on the task and current context.
- **LLM Decision**: Agent actively selects from available action tools
- **Dynamic Selection**: Different action chosen at each step based on needs
- **Tool Visibility**: All action tools are presented to the LLM in the prompt
**[Data Collection Servers](./data_collection.md) are NOT LLM-selectable** - they are automatically invoked by the framework.
### How Tool Metadata Becomes LLM Instructions
**Every action tool's implementation directly affects what the LLM sees and understands.** The UFO² framework automatically extracts:
- **`Annotated` type hints**: Parameter types, constraints, and descriptions
- **Docstrings**: Tool purpose, parameter explanations, return value descriptions
- **Function signatures**: Parameter names, defaults, required vs. optional
These are automatically assembled into structured tool instructions that appear in the LLM's prompt. The LLM uses these instructions to understand what each tool does, select the appropriate tool for each step, and call the tool with correct parameters.
**Therefore, developers MUST write clear, comprehensive metadata.** For examples:
- See [AppUIExecutor documentation](servers/app_ui_executor.md) for well-documented UI automation tools
- See [WordCOMExecutor documentation](servers/word_com_executor.md) for COM API tool examples
- See [Creating Custom MCP Servers Tutorial](../tutorials/creating_mcp_servers.md) for step-by-step guide on writing tool metadata
```mermaid
graph TB
LLM["LLM Agent Decision (Selects Action Tool)"]
Agent["Agent Decision 'Click OK Button'"]
MCP["MCP Server Action Server"]
subgraph Tools["Available Action Tools"]
Click["click()"]
Type["type_text()"]
Insert["insert_table()"]
Shell["run_shell()"]
end
System["System Modified ✅ Side Effects"]
LLM --> Agent
Agent --> MCP
MCP --> Tools
Tools --> System
style LLM fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style Agent fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style MCP fill:#e8f5e9,stroke:#388e3c,stroke-width:2px
style Tools fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style System fill:#ffebee,stroke:#c62828,stroke-width:2px
```
**Side Effects:**
- **✅ Modifies State**: Can change system, files, UI
- **⚠️ Not Idempotent**: Same action may have different results
- **🔒 Use with Caution**: Always verify before executing
- **📝 Audit Trail**: Log all actions for debugging
- **🤖 LLM-Controlled**: Agent decides when and which action to execute
## Tool Type Identifier
All action tools use the tool type:
```python
tool_type = "action"
```
Tool keys follow the format:
```python
tool_key = "action::{tool_name}"
# Examples:
"action::click"
"action::type_text"
"action::run_shell"
```
## Built-in Action Servers
UFO² provides several built-in action servers for different automation scenarios. Below is a summary - click each server name for detailed documentation including all tools, parameters, and usage examples.
### UI Automation Servers
| Server | Agent | Description | Documentation |
|--------|-------|-------------|---------------|
| **[HostUIExecutor](servers/host_ui_executor.md)** | HostAgent | Window selection and desktop-level UI automation | [Full Details →](servers/host_ui_executor.md) |
| **[AppUIExecutor](servers/app_ui_executor.md)** | AppAgent | Application-level UI automation (clicks, typing, scrolling) | [Full Details →](servers/app_ui_executor.md) |
### Command Execution Servers
| Server | Platform | Description | Documentation |
|--------|----------|-------------|---------------|
| **[CommandLineExecutor](servers/command_line_executor.md)** | Windows | Launch applications via direct execution (no shell) | [Full Details →](servers/command_line_executor.md) |
| **[BashExecutor](servers/bash_executor.md)** | Linux | Execute Linux commands via HTTP server | [Full Details →](servers/bash_executor.md) |
### Office Automation Servers (COM API)
| Server | Application | Description | Documentation |
|--------|-------------|-------------|---------------|
| **[WordCOMExecutor](servers/word_com_executor.md)** | Microsoft Word | Word document automation (insert table, format text, etc.) | [Full Details →](servers/word_com_executor.md) |
| **[ExcelCOMExecutor](servers/excel_com_executor.md)** | Microsoft Excel | Excel automation (insert data, create charts, etc.) | [Full Details →](servers/excel_com_executor.md) |
| **[PowerPointCOMExecutor](servers/ppt_com_executor.md)** | Microsoft PowerPoint | PowerPoint automation (slides, formatting, etc.) | [Full Details →](servers/ppt_com_executor.md) |
### Specialized Servers
| Server | Purpose | Description | Documentation |
|--------|---------|-------------|---------------|
| **[PDFReaderExecutor](servers/pdf_reader_executor.md)** | PDF Processing | Extract text from PDFs with human simulation | [Full Details →](servers/pdf_reader_executor.md) |
| **[ConstellationEditor](servers/constellation_editor.md)** | Multi-Device | Create and manage multi-device task workflows | [Full Details →](servers/constellation_editor.md) |
| **[HardwareExecutor](servers/hardware_executor.md)** | Hardware Control | Control Arduino, robot arms, test fixtures, mobile devices | [Full Details →](servers/hardware_executor.md) |
**Quick Reference:** Each server documentation page includes:
- 📋 **Complete tool reference** with all parameters and return values
- 💡 **Code examples** showing actual usage patterns
- ⚙️ **Configuration examples** for different scenarios
- ✅ **Best practices** with do's and don'ts
- 🎯 **Use cases** with complete workflows
## Configuration Examples
Action servers are configured in `config/ufo/mcp.yaml`. Each server's documentation provides detailed configuration examples.
### Basic Configuration
```yaml
HostAgent:
default:
action:
- namespace: HostUIExecutor
type: local
reset: false
- namespace: CommandLineExecutor
type: local
reset: false
```
### App-Specific Configuration
```yaml
AppAgent:
# Default configuration for all apps
default:
action:
- namespace: AppUIExecutor
type: local
reset: false
# Word-specific configuration
WINWORD.EXE:
action:
- namespace: AppUIExecutor
type: local
reset: false
- namespace: WordCOMExecutor
type: local
reset: true # Reset when switching documents
# Excel-specific configuration
EXCEL.EXE:
action:
- namespace: AppUIExecutor
type: local
reset: false
- namespace: ExcelCOMExecutor
type: local
reset: true # Reset when switching workbooks
```
### Multi-Platform Configuration
```yaml
# Windows agent
HostAgent:
default:
action:
- namespace: HostUIExecutor
type: local
- namespace: CommandLineExecutor
type: local
# Linux agent
LinuxAgent:
default:
action:
- namespace: BashExecutor
type: http
host: "192.168.1.100"
port: 8010
path: "/mcp"
```
For complete configuration details, see:
- [MCP Configuration Guide](configuration.md) - Complete configuration reference
- Individual server documentation for server-specific configuration options
## Best Practices
### General Principles
#### 1. Verify Before Acting
Always observe before executing actions:
```python
# ✅ Good: Verify target exists
control_info = await computer.run_actions([
MCPToolCall(tool_key="data_collection::get_control_info", ...)
])
if control_info[0].data and control_info[0].data["is_enabled"]:
await computer.run_actions([
MCPToolCall(tool_key="action::click", ...)
])
```
#### 2. Handle Action Failures
Actions can fail for many reasons - always implement error handling and retries.
#### 3. Validate Inputs
Never execute unsanitized commands, especially with `run_shell` and similar tools.
#### 4. Wait for Action Completion
Some actions need time to complete - add appropriate delays after launching applications or triggering UI changes.
For detailed best practices including code examples, error handling patterns, and common pitfalls, see the individual server documentation:
- [HostUIExecutor Best Practices](servers/host_ui_executor.md)
- [AppUIExecutor Best Practices](servers/app_ui_executor.md)
- [CommandLineExecutor Best Practices](servers/command_line_executor.md)
- [WordCOMExecutor Best Practices](servers/word_com_executor.md)
- [ExcelCOMExecutor Best Practices](servers/excel_com_executor.md)
- [PowerPointCOMExecutor Best Practices](servers/ppt_com_executor.md)
- [PDFReaderExecutor Best Practices](servers/pdf_reader_executor.md)
- [ConstellationEditor Best Practices](servers/constellation_editor.md)
- [HardwareExecutor Best Practices](servers/hardware_executor.md)
- [BashExecutor Best Practices](servers/bash_executor.md)
## Common Use Cases
For complete use case examples with detailed workflows, see the individual server documentation:
### UI Automation
- **Form Filling**: [AppUIExecutor](servers/app_ui_executor.md)
- **Window Management**: [HostUIExecutor](servers/host_ui_executor.md)
### Document Automation
- **Word Processing**: [WordCOMExecutor](servers/word_com_executor.md)
- **Excel Data Processing**: [ExcelCOMExecutor](servers/excel_com_executor.md)
- **PowerPoint Generation**: [PowerPointCOMExecutor](servers/ppt_com_executor.md)
- **PDF Extraction**: [PDFReaderExecutor](servers/pdf_reader_executor.md)
### System Automation
- **Application Launching**: [CommandLineExecutor](servers/command_line_executor.md)
- **Linux Command Execution**: [BashExecutor](servers/bash_executor.md)
### Multi-Device Workflows
- **Task Distribution**: [ConstellationEditor](servers/constellation_editor.md)
- **Hardware Control**: [HardwareExecutor](servers/hardware_executor.md)
## Error Handling
Action servers implement robust error handling with timeouts and retries. For detailed error handling patterns specific to each server, see:
- [HostUIExecutor](servers/host_ui_executor.md)
- [AppUIExecutor](servers/app_ui_executor.md)
- [CommandLineExecutor](servers/command_line_executor.md)
- [BashExecutor](servers/bash_executor.md)
- And other server-specific documentation
### General Timeout Handling
Actions are executed with a timeout (default: 6000 seconds):
```python
try:
result = await computer.run_actions([
MCPToolCall(tool_key="action::run_shell", ...)
])
except asyncio.TimeoutError:
logger.error("Action timed out after 6000 seconds")
# Cleanup or retry logic...
```
### General Retry Pattern
```python
async def retry_action(action: MCPToolCall, max_retries: int = 3):
"""Retry an action with exponential backoff."""
for attempt in range(max_retries):
try:
result = await computer.run_actions([action])
if not result[0].is_error:
return result[0]
logger.warning(f"Attempt {attempt + 1} failed: {result[0].content}")
if attempt < max_retries - 1:
await asyncio.sleep(2 ** attempt) # Exponential backoff
except Exception as e:
logger.error(f"Exception on attempt {attempt + 1}: {e}")
if attempt < max_retries - 1:
await asyncio.sleep(2 ** attempt)
raise ValueError(f"Action failed after {max_retries} attempts")
```
## Integration with Data Collection
Actions should be paired with data collection for verification:
```python
# Pattern: Observe → Act → Verify
# 1. Observe: Capture initial state
before_screenshot = await computer.run_actions([
MCPToolCall(tool_key="data_collection::take_screenshot", ...)
])
# 2. Act: Execute action
action_result = await computer.run_actions([
MCPToolCall(tool_key="action::click", ...)
])
# 3. Verify: Check result
await asyncio.sleep(1) # Wait for UI update
after_screenshot = await computer.run_actions([
MCPToolCall(tool_key="data_collection::take_screenshot", ...)
])
```
For more details on agent execution patterns:
- [HostAgent Commands](../ufo2/host_agent/commands.md) - HostAgent command patterns
- [AppAgent Commands](../ufo2/app_agent/commands.md) - AppAgent action patterns
- [Agent Overview](../ufo2/overview.md) - UFO² agent architecture
For more details on data collection:
- [Data Collection Servers](data_collection.md) - Observation tools
- [UICollector Documentation](servers/ui_collector.md) - Complete data collection reference
## Related Documentation
- [Data Collection Servers](data_collection.md) - Observation tools
- [Configuration Guide](configuration.md) - Configure action servers
- [Local Servers](local_servers.md) - Built-in action servers overview
- [Remote Servers](remote_servers.md) - HTTP deployment for actions
- [Computer](../client/computer.md) - Action execution layer
- [MCP Overview](overview.md) - High-level MCP architecture
**Safety Reminder:** Action servers can **modify system state**. Always:
1. ✅ **Validate inputs** before execution
2. ✅ **Verify targets** exist and are accessible
3. ✅ **Log all actions** for audit trail
4. ✅ **Handle failures** gracefully with retries
5. ✅ **Test in safe environment** before production use
---
### Documents Docs Mcp Configuration (`documents/docs/mcp/configuration.md`)
# MCP Configuration Guide
## Overview
MCP configuration in UFO² uses a **hierarchical YAML structure** that maps agents to their MCP servers. The configuration file is located at:
```
config/ufo/mcp.yaml
```
For complete field documentation, see [MCP Reference](../configuration/system/mcp_reference.md).
## Configuration Structure
```yaml
AgentName: # Name of the agent
SubType: # Sub-type (e.g., "default", "WINWORD.EXE")
data_collection: # Data collection servers
- namespace: ... # Server namespace
type: ... # Server type (local/http/stdio)
... # Additional server config
action: # Action servers
- namespace: ...
type: ...
...
```
### Hierarchy Levels
1. **Agent Name** - Top-level agent identifier (e.g., `HostAgent`, `AppAgent`)
2. **Sub-Type** - Context-specific configuration (e.g., `default`, `WINWORD.EXE`)
3. **Tool Type** - `data_collection` or `action`
4. **Server List** - Array of MCP server configurations
```
AgentName
└─ SubType
├─ data_collection
│ ├─ Server 1
│ ├─ Server 2
│ └─ ...
└─ action
├─ Server 1
├─ Server 2
└─ ...
```
**Default Sub-Type:** Always define a `default` sub-type as a fallback configuration. If a specific sub-type is not found, the agent will use `default`.
## Server Configuration Fields
### Common Fields
All MCP servers share these fields:
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `namespace` | `string` | ✅ Yes | Unique identifier for the server |
| `type` | `string` | ✅ Yes | Server type: `local`, `http`, or `stdio` |
| `reset` | `boolean` | ❌ No | Whether to reset server state (default: `false`) |
| `start_args` | `array` | ❌ No | Arguments passed to server initialization |
### Local Server Fields
For `type: local`:
```yaml
- namespace: UICollector
type: local
start_args: []
reset: false
```
| Field | Description |
|-------|-------------|
| `start_args` | Arguments passed to server factory function |
Local servers are retrieved from the `MCPRegistry` and run in-process.
### HTTP Server Fields
For `type: http`:
```yaml
- namespace: HardwareCollector
type: http
host: "localhost"
port: 8006
path: "/mcp"
reset: false
```
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `host` | `string` | ✅ Yes | Server hostname or IP |
| `port` | `integer` | ✅ Yes | Server port number |
| `path` | `string` | ✅ Yes | URL path to MCP endpoint |
HTTP servers run on remote machines and are accessed via REST API.
### Stdio Server Fields
For `type: stdio`:
```yaml
- namespace: CustomProcessor
type: stdio
command: "python"
start_args: ["-m", "custom_mcp_server"]
env: {"API_KEY": "secret"}
cwd: "/path/to/server"
reset: false
```
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `command` | `string` | ✅ Yes | Executable command |
| `start_args` | `array` | ❌ No | Command-line arguments |
| `env` | `object` | ❌ No | Environment variables |
| `cwd` | `string` | ❌ No | Working directory |
Stdio servers run as child processes and communicate via stdin/stdout.
## Agent Configurations
### HostAgent
System-level agent for OS-wide automation:
```yaml
HostAgent:
default:
data_collection:
- namespace: UICollector
type: local
start_args: []
reset: false
action:
- namespace: HostUIExecutor
type: local
start_args: []
reset: false
- namespace: CommandLineExecutor
type: local
start_args: []
reset: false
```
**Tools Available**:
- **Data Collection**: UI detection, screenshots
- **Actions**: System-wide clicks, window management, CLI execution
### AppAgent
Application-specific agent:
#### Default Configuration
```yaml
AppAgent:
default:
data_collection:
- namespace: UICollector
type: local
start_args: []
reset: false
action:
- namespace: AppUIExecutor
type: local
start_args: []
reset: false
- namespace: CommandLineExecutor
type: local
start_args: []
reset: false
```
#### Word-Specific Configuration
```yaml
AppAgent:
WINWORD.EXE:
data_collection:
- namespace: UICollector
type: local
start_args: []
reset: false
action:
- namespace: AppUIExecutor
type: local
start_args: []
reset: false
- namespace: WordCOMExecutor
type: local
start_args: []
reset: true # Reset COM state when switching documents
```
**Tools Available**:
- **Data Collection**: Same as default
- **Actions**: App UI automation + Word COM API (insert_table, select_text, etc.)
**Reset Flag:** Set `reset: true` for stateful tools (like COM executors) to prevent state leakage between contexts (e.g., different documents).
#### Excel-Specific Configuration
```yaml
AppAgent:
EXCEL.EXE:
data_collection:
- namespace: UICollector
type: local
reset: false
action:
- namespace: AppUIExecutor
type: local
reset: false
- namespace: ExcelCOMExecutor
type: local
reset: true
```
#### PowerPoint-Specific Configuration
```yaml
AppAgent:
POWERPNT.EXE:
data_collection:
- namespace: UICollector
type: local
reset: false
action:
- namespace: AppUIExecutor
type: local
reset: false
- namespace: PowerPointCOMExecutor
type: local
reset: true
```
#### File Explorer Configuration
```yaml
AppAgent:
explorer.exe:
data_collection:
- namespace: UICollector
type: local
reset: false
action:
- namespace: AppUIExecutor
type: local
reset: false
- namespace: PDFReaderExecutor
type: local
reset: true
```
### ConstellationAgent
Multi-device coordination agent:
```yaml
ConstellationAgent:
default:
action:
- namespace: ConstellationEditor
type: local
start_args: []
reset: false
```
**Tools Available**:
- **Actions**: Create tasks, assign devices, check task status
### HardwareAgent
Remote hardware monitoring agent:
```yaml
HardwareAgent:
default:
data_collection:
- namespace: HardwareCollector
type: http
host: "localhost"
port: 8006
path: "/mcp"
reset: false
action:
- namespace: HardwareExecutor
type: http
host: "localhost"
port: 8006
path: "/mcp"
reset: false
```
**Tools Available**:
- **Data Collection**: CPU info, memory info, disk info
- **Actions**: Hardware control commands
**Remote Deployment:** For remote servers, ensure the HTTP MCP server is running on the target machine. See [Remote Servers](remote_servers.md) for deployment guide.
### LinuxAgent
Linux system agent:
```yaml
LinuxAgent:
default:
action:
- namespace: BashExecutor
type: http
host: "localhost"
port: 8010
path: "/mcp"
reset: false
```
**Tools Available**:
- **Actions**: Bash command execution
## Configuration Examples
### Example 1: Local-Only Agent
```yaml
SimpleAgent:
default:
data_collection:
- namespace: UICollector
type: local
reset: false
action:
- namespace: SimpleExecutor
type: local
reset: false
```
### Example 2: Hybrid Agent (Local + Remote)
```yaml
HybridAgent:
default:
data_collection:
# Local UI detection
- namespace: UICollector
type: local
reset: false
# Remote hardware monitoring
- namespace: HardwareCollector
type: http
host: "192.168.1.100"
port: 8006
path: "/mcp"
reset: false
action:
# Local UI automation
- namespace: UIExecutor
type: local
reset: false
# Remote command execution
- namespace: RemoteExecutor
type: http
host: "192.168.1.100"
port: 8007
path: "/mcp"
reset: false
```
### Example 3: Multi-Context Agent
```yaml
MultiContextAgent:
# Default configuration
default:
data_collection:
- namespace: BasicCollector
type: local
action:
- namespace: BasicExecutor
type: local
# Specialized for Chrome
chrome.exe:
data_collection:
- namespace: BasicCollector
type: local
- namespace: WebCollector
type: local
action:
- namespace: BasicExecutor
type: local
- namespace: BrowserExecutor
type: local
reset: true
# Specialized for VS Code
Code.exe:
data_collection:
- namespace: BasicCollector
type: local
- namespace: IDECollector
type: local
action:
- namespace: BasicExecutor
type: local
- namespace: CodeExecutor
type: local
reset: true
```
## Best Practices
### 1. Use Descriptive Namespaces
```yaml
# ✅ Good: Clear and descriptive
namespace: WindowsUICollector
namespace: ExcelCOMExecutor
namespace: LinuxBashExecutor
# ❌ Bad: Generic and unclear
namespace: Collector1
namespace: Server
namespace: Tools
```
### 2. Group Related Servers
```yaml
# ✅ Good: Logical grouping
HostAgent:
default:
data_collection:
- namespace: UICollector # All UI-related
- namespace: ScreenshotTaker
action:
- namespace: UIExecutor # All UI actions
- namespace: WindowManager
# ❌ Bad: Mixed purposes
HostAgent:
default:
data_collection:
- namespace: UICollector
- namespace: HardwareMonitor # Different purpose
```
### 3. Reset Stateful Servers
```yaml
# ✅ Good: Reset COM servers
WordCOMExecutor:
type: local
reset: true # Prevents state leakage
# ❌ Bad: Not resetting can cause issues
WordCOMExecutor:
type: local
reset: false # May retain state from previous document
```
### 4. Validate Remote Server Availability
```yaml
# When using remote servers, ensure they're accessible
HardwareCollector:
type: http
host: "192.168.1.100" # ✅ Verify this host is reachable
port: 8006 # ✅ Verify port is open
path: "/mcp" # ✅ Verify endpoint exists
```
### 5. Use Environment Variables for Secrets
```yaml
# ✅ Good: Use environment variables
- namespace: SecureAPI
type: http
host: "${API_HOST}"
port: "${API_PORT}"
auth:
token: "${API_TOKEN}"
# ❌ Bad: Hardcoded secrets
- namespace: SecureAPI
type: http
host: "api.example.com"
auth:
token: "secret123" # Don't commit this!
```
## Loading Configuration
### From File
```python
import yaml
from pathlib import Path
# Load MCP configuration
config_path = Path("config/ufo/mcp.yaml")
with open(config_path) as f:
mcp_config = yaml.safe_load(f)
# Access agent configuration
host_agent_config = mcp_config["HostAgent"]["default"]
```
### Programmatically
```python
from ufo.config import get_config
# Get full configuration
configs = get_config()
# Access MCP section
mcp_config = configs.get("mcp", {})
# Get specific agent
host_agent = mcp_config.get("HostAgent", {}).get("default", {})
```
## Validation
### Schema Validation
UFO² validates MCP configuration on load:
```python
from ufo.config.config_schemas import MCPConfigSchema
# Validate configuration
try:
MCPConfigSchema.validate(mcp_config)
print("✅ Configuration is valid")
except ValidationError as e:
print(f"❌ Configuration error: {e}")
```
### Common Validation Errors
| Error | Cause | Solution |
|-------|-------|----------|
| `Missing required field: namespace` | Server missing namespace | Add `namespace` field |
| `Invalid server type: unknown` | Unsupported type | Use `local`, `http`, or `stdio` |
| `Missing host for http server` | HTTP server without host | Add `host` and `port` |
| `Duplicate namespace` | Same namespace used twice | Use unique namespaces |
## Debugging Configuration
### Enable Debug Logging
```python
import logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger("ufo.client.mcp")
# Will show server creation and registration
# DEBUG: Creating MCP server 'UICollector' of type local
# DEBUG: Registered MCP server 'UICollector' with 15 tools
```
### Check Loaded Servers
```python
from ufo.client.mcp.mcp_server_manager import MCPServerManager
# List all registered servers
servers = MCPServerManager._servers_mapping
for namespace, server in servers.items():
print(f"Server: {namespace}, Type: {type(server).__name__}")
```
### Test Server Connectivity
```python
async def test_server(config):
"""Test if MCP server is accessible."""
try:
server = MCPServerManager.create_mcp_server(config)
print(f"✅ Server '{config['namespace']}' is accessible")
# List tools
if hasattr(server, 'server'):
from fastmcp.client import Client
async with Client(server.server) as client:
tools = await client.list_tools()
print(f" Tools: {[tool.name for tool in tools]}")
except Exception as e:
print(f"❌ Server '{config['namespace']}' failed: {e}")
```
## Migration Guide
### From Old Configuration Format
If you're migrating from an older UFO configuration:
**Old Format** (config.yaml):
```yaml
MCP_SERVERS:
- name: ui_collector
module: ufo.mcp.ui_server
```
**New Format** (mcp.yaml):
```yaml
HostAgent:
default:
data_collection:
- namespace: UICollector
type: local
```
For detailed migration instructions, see [Configuration Migration Guide](../configuration/system/migration.md).
## Related Documentation
- [MCP Overview](overview.md) - High-level MCP architecture
- [Data Collection Servers](data_collection.md) - Data collection configuration
- [Action Servers](action.md) - Action server configuration
- [Local Servers](local_servers.md) - Built-in local MCP servers
- [Remote Servers](remote_servers.md) - HTTP and Stdio deployment
- [Creating Custom MCP Servers Tutorial](../tutorials/creating_mcp_servers.md) - Build your own servers
- [MCP Reference](../configuration/system/mcp_reference.md) - Complete field reference
- [Configuration Guide](../configuration/system/overview.md) - General configuration guide
- [HostAgent Overview](../ufo2/host_agent/overview.md) - HostAgent configuration examples
- [AppAgent Overview](../ufo2/app_agent/overview.md) - AppAgent configuration examples
**Configuration Philosophy:**
MCP configuration follows the **convention over configuration** principle:
- **Sensible defaults** - Minimal configuration required
- **Explicit when needed** - Full control when customization is necessary
- **Type-safe** - Validated on load to catch errors early
- **Hierarchical** - Inherit from defaults, override as needed
---
### Documents Docs Mcp Data Collection (`documents/docs/mcp/data_collection.md`)
# Data Collection Servers
## Overview
**Data Collection Servers** provide read-only tools that observe and retrieve system state without modifying it. These servers are essential for agents to understand the current environment before taking actions.
**Data Collection servers are automatically invoked by the UFO² framework** to gather context and build observation prompts for the LLM. The LLM agent **does not select these tools** - they run in the background to provide system state information.
- **Framework-Driven**: Automatically called to collect screenshots, UI controls, system info
- **Observation Purpose**: Build the prompt that the LLM uses for decision-making
- **Not in Tool List**: These tools are NOT presented to the LLM as selectable actions
**Only [Action Servers](./action.md) are LLM-selectable.**
```mermaid
graph TB
Framework["UFO² Framework (Automatic Invocation)"]
AgentStep["Agent Step Observation & Prompt Build"]
MCP["MCP Server UICollector"]
subgraph Tools["Data Collection Tools"]
Screenshot["take_screenshot()"]
WindowList["get_window_list()"]
ControlInfo["get_control_info()"]
end
SystemState["System State → LLM Context"]
Framework --> AgentStep
Framework --> MCP
MCP --> Tools
Tools --> SystemState
SystemState --> AgentStep
style Framework fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
style AgentStep fill:#fff3e0,stroke:#f57c00,stroke-width:2px
style MCP fill:#e8f5e9,stroke:#388e3c,stroke-width:2px
style Tools fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px
style SystemState fill:#fce4ec,stroke:#c2185b,stroke-width:2px
```
**Characteristics:**
- **❌ No Side Effects**: Cannot modify system state
- **✅ Safe to Retry**: Can be called multiple times without risk
- **✅ Idempotent**: Same input always produces same output
- **📊 Observation Only**: Provides information for decision-making
- **🤖 Framework-Invoked**: Not selectable by LLM agent
## Tool Type Identifier
All data collection tools use the tool type:
```python
tool_type = "data_collection"
```
Tool keys follow the format:
```python
tool_key = "data_collection::{tool_name}"
# Examples:
"data_collection::take_screenshot"
"data_collection::get_window_list"
"data_collection::get_control_info"
```
## Built-in Data Collection Servers
### UICollector
**Purpose**: Collect UI element information and screenshots
**Namespace**: `UICollector`
**Platform**: Windows (using pywinauto)
**Tools**: 8 tools for UI observation including screenshots, window lists, control info, and annotations
For complete documentation including all tool details, parameters, return types, and usage examples, see:
**[→ UICollector Full Documentation](servers/ui_collector.md)**
#### Quick Example
```python
from aip.messages import Command
# Take a screenshot of the active window
screenshot_cmd = Command(
tool_name="take_screenshot",
tool_type="data_collection",
parameters={
"region": "active_window",
"save_path": "screenshots/current.png"
}
)
# Get list of all windows
windows_cmd = Command(
tool_name="get_window_list",
tool_type="data_collection",
parameters={}
)
```
For detailed tool specifications, advanced usage patterns, and best practices, see the [UICollector documentation](servers/ui_collector.md).
## Configuration Examples
Data collection servers are configured in `config/ufo/mcp.yaml`. For detailed configuration options, see the [UICollector documentation](servers/ui_collector.md#configuration).
### Basic Configuration
```yaml
HostAgent:
default:
data_collection:
- namespace: UICollector
type: local
start_args: []
reset: false
```
### Multi-Server Configuration
```yaml
HostAgent:
default:
data_collection:
- namespace: UICollector
type: local
reset: false
```
### App-Specific Configuration
```yaml
AppAgent:
WINWORD.EXE:
data_collection:
- namespace: UICollector
type: local
reset: false # Don't reset when switching between documents
EXCEL.EXE:
data_collection:
- namespace: UICollector
type: local
reset: true # Reset when switching between spreadsheets
```
## Best Practices
For detailed best practices with complete code examples, see the [UICollector documentation](servers/ui_collector.md).
### General Guidelines
#### 1. Call Before Action
Always collect data before executing actions to make informed decisions.
#### 2. Cache Results
Data collection results can be cached when state hasn't changed to improve performance.
#### 3. Handle Failures Gracefully
Data collection can fail if windows close or controls disappear - implement proper error handling.
#### 4. Minimize Screenshot Calls
Screenshots are expensive operations - take one screenshot and analyze it multiple times rather than taking multiple screenshots.
5. **Use Appropriate Regions**
Choose the smallest region that contains needed information (e.g., active window vs. full screen).
See the [UICollector documentation](servers/ui_collector.md) for detailed examples and anti-patterns.
## Common Use Cases
For complete use case examples with detailed code, see the [UICollector documentation](servers/ui_collector.md).
### UI Element Detection
Discover windows and controls for automation targeting.
### Screen Monitoring
Monitor screen changes for event-driven automation.
### System Health Check
Check system resources before executing heavy tasks.
See the [UICollector documentation](servers/ui_collector.md) for complete workflow examples.
## Error Handling
For detailed error handling patterns, see the [UICollector documentation](servers/ui_collector.md).
### Common Errors
| Error | Cause | Solution |
|-------|-------|----------|
| `WindowNotFoundError` | Target window closed | Check window existence first |
| `ControlNotFoundError` | Control not accessible | Use alternative identification method |
| `ScreenshotFailedError` | Graphics driver issue | Retry with different region |
| `TimeoutError` | Operation took too long | Increase timeout or simplify query |
See the [UICollector documentation](servers/ui_collector.md) for complete error recovery examples.
## Performance Considerations
For detailed performance optimization techniques, see the [UICollector documentation](servers/ui_collector.md).
### Key Optimizations
- **Screenshot Optimization**: Use region parameters to capture only needed areas
- **Parallel Data Collection**: Collect independent data in parallel when possible
- **Caching**: Cache results when state hasn't changed
See the [UICollector documentation](servers/ui_collector.md) for complete examples.
## Integration with Agents
Data collection servers are typically used in the **observation phase** of agent execution. See the [UICollector documentation](servers/ui_collector.md) for complete integration patterns.
For more details on agent architecture and execution flow:
- [HostAgent Overview](../ufo2/host_agent/overview.md) - HostAgent architecture and workflow
- [AppAgent Overview](../ufo2/app_agent/overview.md) - AppAgent architecture and workflow
- [Agent Overview](../ufo2/overview.md) - UFO² agent system architecture
```python
# Agent execution loop
while not task_complete:
# 1. Observe: Collect current state
screenshot = await data_collection_server.take_screenshot()
# 2. Reason: Agent decides next action
next_action = agent.plan(screenshot)
# 3. Act: Execute action
result = await action_server.execute(next_action)
# 4. Verify: Check action result
new_screenshot = await data_collection_server.take_screenshot()
```
## Related Documentation
- **[UICollector Full Documentation](servers/ui_collector.md)** - Complete tool reference with all parameters and examples
- [Action Servers](action.md) - State-changing execution tools
- [Configuration Guide](configuration.md) - How to configure data collection servers
- [Local Servers](local_servers.md) - Built-in local MCP servers
- [Remote Servers](remote_servers.md) - HTTP deployment for data collection
- [Computer](../client/computer.md) - Tool execution layer
- [MCP Overview](overview.md) - High-level MCP architecture
**Key Takeaways:**
- Data collection servers are **read-only** and **safe to retry**
- Always **observe before acting** to make informed decisions
- **Cache results** when state hasn't changed to improve performance
- Handle **errors gracefully** with retries and fallback logic
- Use **appropriate regions** and **parallel collection** for performance
- See the **[UICollector documentation](servers/ui_collector.md)** for complete details
---
### Documents Docs Mcp Local Servers (`documents/docs/mcp/local_servers.md`)
# Local MCP Servers
Local MCP servers run in-process with the UFO² agent, providing fast and efficient access to tools without network overhead. They are the most common server type for built-in functionality.
**For remote HTTP servers** (BashExecutor, HardwareExecutor, MobileExecutor), see [Remote Servers](./remote_servers.md).
## Overview
UFO² includes several built-in local MCP servers organized by functionality. This page provides a quick reference - click each server name for complete documentation.
| Server | Type | Description | Full Documentation |
|--------|------|-------------|-------------------|
| **UICollector** | Data Collection | Windows UI observation | **[→ Full Docs](servers/ui_collector.md)** |
| **HostUIExecutor** | Action | Desktop-level UI automation | **[→ Full Docs](servers/host_ui_executor.md)** |
| **AppUIExecutor** | Action | Application-level UI automation | **[→ Full Docs](servers/app_ui_executor.md)** |
| **CommandLineExecutor** | Action | Application launching (no shell) | **[→ Full Docs](servers/command_line_executor.md)** |
| **WordCOMExecutor** | Action | Microsoft Word COM API | **[→ Full Docs](servers/word_com_executor.md)** |
| **ExcelCOMExecutor** | Action | Microsoft Excel COM API | **[→ Full Docs](servers/excel_com_executor.md)** |
| **PowerPointCOMExecutor** | Action | Microsoft PowerPoint COM API | **[→ Full Docs](servers/ppt_com_executor.md)** |
| **PDFReaderExecutor** | Action | PDF text extraction | **[→ Full Docs](servers/pdf_reader_executor.md)** |
| **ConstellationEditor** | Action | Multi-device task orchestration | **[→ Full Docs](servers/constellation_editor.md)** |
---
## Server Summaries
### UICollector
**Type**: Data Collection (read-only, automatically invoked)
**Platform**: Windows
**Tools**: 8 tools for screenshots, window lists, control info, and annotations
**[→ See complete UICollector documentation](servers/ui_collector.md)** for all tool details, parameters, return values, and examples.
---
### HostUIExecutor
**Type**: Action (LLM-selectable, state-modifying)
**Platform**: Windows
**Agent**: HostAgent
**Tool**: `select_application_window` - Window selection and focus management
**[→ See complete HostUIExecutor documentation](servers/host_ui_executor.md)** for tool specifications and workflow examples.
---
### AppUIExecutor
**Type**: Action (LLM-selectable, GUI automation)
**Platform**: Windows
**Agent**: AppAgent
**Tools**: 9 tools for clicks, typing, scrolling, and UI interaction
**[→ See complete AppUIExecutor documentation](servers/app_ui_executor.md)** for all automation tools and usage patterns.
---
### CommandLineExecutor
**Type**: Action (LLM-selectable, application launching)
**Platform**: Cross-platform
**Agent**: HostAgent, AppAgent
**Tool**: `run_shell` - Launch applications (executes with `shell=False` to prevent shell injection)
**[→ See complete CommandLineExecutor documentation](servers/command_line_executor.md)** for security guidelines and examples.
---
### WordCOMExecutor
**Type**: Action (LLM-selectable, Word COM API)
**Platform**: Windows
**Agent**: AppAgent (WINWORD.EXE only)
**Tools**: 6 tools for Word document automation
**[→ See complete WordCOMExecutor documentation](servers/word_com_executor.md)** for all Word automation tools.
---
### ExcelCOMExecutor
**Type**: Action (LLM-selectable, Excel COM API)
**Platform**: Windows
**Agent**: AppAgent (EXCEL.EXE only)
**Tools**: 6 tools for Excel automation
**[→ See complete ExcelCOMExecutor documentation](servers/excel_com_executor.md)** for all Excel manipulation tools.
---
### PowerPointCOMExecutor
**Type**: Action (LLM-selectable, PowerPoint COM API)
**Platform**: Windows
**Agent**: AppAgent (POWERPNT.EXE only)
**Tools**: 2 tools for PowerPoint automation
**[→ See complete PowerPointCOMExecutor documentation](servers/ppt_com_executor.md)** for PowerPoint tools and examples.
---
### PDFReaderExecutor
**Type**: Action (LLM-selectable, PDF text extraction)
**Platform**: Windows
**Agent**: AppAgent (explorer.exe)
**Tools**: 3 tools for PDF text extraction with human simulation
**[→ See complete PDFReaderExecutor documentation](servers/pdf_reader_executor.md)** for PDF extraction tools and workflows.
---
### ConstellationEditor
**Type**: Action (LLM-selectable, multi-device orchestration)
**Platform**: Cross-platform
**Agent**: ConstellationAgent
**Tools**: 7 tools for task and dependency management
**[→ See complete ConstellationEditor documentation](servers/constellation_editor.md)** for multi-device workflow tools.
---
## Configuration
All local servers are configured in `config/ufo/mcp.yaml`. For detailed configuration options, see:
- [MCP Configuration Guide](./configuration.md) - Complete configuration reference
- Individual server documentation for server-specific configuration
**Example configuration:**
```yaml
AppAgent:
WINWORD.EXE:
data_collection:
- namespace: UICollector
type: local
reset: false
action:
- namespace: AppUIExecutor # GUI automation
type: local
reset: false
- namespace: WordCOMExecutor # API automation
type: local
reset: true # Reset when switching documents
- namespace: CommandLineExecutor
type: local
reset: false
```
## See Also
- [MCP Overview](./overview.md) - MCP architecture and concepts
- [Data Collection Servers](./data_collection.md) - Data collection overview
- [Action Servers](./action.md) - Action server overview
- [MCP Configuration](./configuration.md) - Configuration guide
- [Remote Servers](./remote_servers.md) - HTTP/Stdio deployment
- [Creating Custom MCP Servers Tutorial](../tutorials/creating_mcp_servers.md) - Learn to build your own servers
- [HostAgent Overview](../ufo2/host_agent/overview.md) - HostAgent architecture
- [AppAgent Overview](../ufo2/app_agent/overview.md) - AppAgent architecture
- [Hybrid Actions](../ufo2/core_features/hybrid_actions.md) - GUI + API dual-mode automation
---
### Documents Docs Mcp Remote Servers (`documents/docs/mcp/remote_servers.md`)
# Remote MCP Servers
Remote MCP servers run as separate processes or on different machines, communicating with UFO² over HTTP or stdio. This enables **cross-platform automation**, process isolation, and distributed workflows.
**Cross-Platform Automation:** Remote servers enable **Windows UFO² agents to control Linux systems, mobile devices, and hardware** through HTTP MCP servers running on those platforms.
## Deployment Models
### HTTP Servers
HTTP MCP servers run as standalone HTTP services, accessible via REST-like endpoints.
**Advantages:**
- Cross-platform communication (Windows ↔ Linux, Windows ↔ Hardware)
- Language-agnostic (server can be in Python, Go, Rust, etc.)
- Network-accessible (local or remote deployment)
- Stateless design (each request is independent)
**Use Cases:**
- Linux command execution from Windows
- Hardware device control (Arduino, robot arms, test fixtures)
- Mobile device automation (Android, iOS via robot arm)
- Distributed multi-machine workflows
### Stdio Servers
Stdio MCP servers run as child processes, communicating via stdin/stdout.
**Advantages:**
- Process isolation (sandboxed execution)
- Clean resource management (process lifetime)
- Standard protocol (works with any language)
**Use Cases:**
- Custom Python/Node.js tools running in separate environments
- Third-party MCP servers
- Sandboxed execution for security
---
## Built-in Remote Servers
### HardwareExecutor
**Type**: Action (HTTP deployment)
**Purpose**: Control hardware devices (Arduino HID, BB-8 test fixture, robot arm, mobile devices)
**Deployment**: HTTP server on hardware controller machine
**Agent**: HardwareAgent
**Tools**: 30+ hardware control tools
**[→ See complete HardwareExecutor documentation](servers/hardware_executor.md)** for all hardware control tools, deployment instructions, and usage examples.
---
### BashExecutor
**Type**: Action (HTTP deployment)
**Purpose**: Execute shell commands on Linux systems
**Deployment**: HTTP server on Linux machine
**Agent**: LinuxAgent
**Tools**: 2 tools for command execution and system info
**[→ See complete BashExecutor documentation](servers/bash_executor.md)** for Linux command execution, security guidelines, and systemd setup.
---
### MobileExecutor
**Type**: Action + Data Collection (HTTP deployment, dual-server)
**Purpose**: Android device automation via ADB
**Deployment**: HTTP servers on machine with ADB access
**Agent**: MobileAgent
**Ports**: 8020 (data collection), 8021 (action)
**Tools**: 13+ tools for Android automation
**Architecture**: Runs as **two HTTP servers** that share a singleton state manager for coordinated operations:
- **Mobile Data Collection Server** (port 8020): Screenshots, UI tree, device info, app list, controls
- **Mobile Action Server** (port 8021): Tap, swipe, type, launch apps, press keys, control clicks
Both servers require a bearer credential from the `UFO_MCP_API_KEY` environment variable. They fail to start when the variable is not set.
**[→ See complete MobileExecutor documentation](servers/mobile_executor.md)** for all Android automation tools, dual-server architecture, deployment instructions, and usage examples.
---
## Configuration Reference
### HTTP Server Fields
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `namespace` | String | ✅ Yes | Unique server identifier |
| `type` | String | ✅ Yes | Must be `"http"` |
| `host` | String | ✅ Yes | Server hostname or IP |
| `port` | Integer | ✅ Yes | Server port number |
| `path` | String | ✅ Yes | HTTP endpoint path |
| `auth` | String | ❌ No | Bearer credential for authenticated servers. Use an environment-variable placeholder, not a literal secret. Required by MobileExecutor. |
| `reset` | Boolean | ❌ No | Reset on context switch (default: `false`) |
### Stdio Server Fields
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `namespace` | String | ✅ Yes | Unique server identifier |
| `type` | String | ✅ Yes | Must be `"stdio"` |
| `command` | String | ✅ Yes | Executable command |
| `start_args` | List[String] | ❌ No | Command-line arguments |
| `env` | Dict | ❌ No | Environment variables |
| `cwd` | String | ❌ No | Working directory |
| `reset` | Boolean | ❌ No | Reset on context switch (default: `false`) |
---
## Example Configurations
### HTTP: Hardware Control
```yaml
HardwareAgent:
default:
action:
- namespace: HardwareExecutor
type: http
host: "192.168.1.100"
port: 8006
path: "/mcp"
```
**Server Start:**
```bash
python -m ufo.client.mcp.http_servers.hardware_mcp_server --host 0.0.0.0 --port 8006
```
See the [HardwareExecutor documentation](servers/hardware_executor.md) for complete deployment instructions.
### HTTP: Linux Command Execution
```yaml
LinuxAgent:
default:
action:
- namespace: BashExecutor
type: http
host: "192.168.1.50"
port: 8010
path: "/mcp"
```
**Server Start:**
```bash
python -m ufo.client.mcp.http_servers.linux_mcp_server --host 0.0.0.0 --port 8010
```
See the [BashExecutor documentation](servers/bash_executor.md) for systemd service setup.
### HTTP: Android Device Automation
```yaml
MobileAgent:
default:
data_collection:
- namespace: MobileDataCollector
type: http
host: "192.168.1.60" # Android automation server
port: 8020
path: "/mcp"
auth: "${UFO_MCP_API_KEY}"
action:
- namespace: MobileActionExecutor
type: http
host: "192.168.1.60"
port: 8021
path: "/mcp"
auth: "${UFO_MCP_API_KEY}"
```
**Server Start:**
```bash
# Generate a high-entropy key, then set the same value on the authorized client.
export UFO_MCP_API_KEY="$(python -c 'import secrets; print(secrets.token_urlsafe(32))')"
# Start both servers (recommended - they share state)
python -m ufo.client.mcp.http_servers.mobile_mcp_server --server both --host 0.0.0.0 --data-port 8020 --action-port 8021
```
PowerShell equivalent:
```powershell
$env:UFO_MCP_API_KEY = python -c "import secrets; print(secrets.token_urlsafe(32))"
```
Set `UFO_MCP_API_KEY` to the same value in the UFO client process so the YAML placeholders resolve. An unresolved placeholder is rejected before the client connects.
Bearer authentication does not encrypt traffic. For non-loopback deployments, use TLS through a reverse proxy, an authenticated reverse proxy, or a trusted private tunnel. Keep firewall rules as an additional reachability control.
See the [MobileExecutor documentation](servers/mobile_executor.md) for complete deployment instructions and ADB setup.
### Stdio: Custom Python Server
```yaml
CustomAgent:
default:
action:
- namespace: CustomProcessor
type: stdio
command: "python"
start_args: ["-m", "custom_mcp_server"]
env:
API_KEY: "secret_key"
cwd: "/path/to/server"
```
---
## Best Practices
**Recommended Practices:**
- ✅ **Use HTTP for cross-platform automation**
- ✅ **Use stdio for process isolation**
- ✅ **Validate remote server connectivity** before deployment
- ✅ **Set appropriate timeouts** for long-running commands
- ✅ **Use environment variables** for sensitive credentials
- ✅ **Protect bearer credentials in transit** with TLS or a trusted private tunnel
**Anti-Patterns to Avoid:**
- ❌ **Don't expose HTTP servers directly to the public internet**
- ❌ **Don't hardcode credentials** in configuration files
- ❌ **Don't forget to start remote servers** before client connection
---
## See Also
- [MCP Overview](./overview.md) - MCP architecture and deployment models
- [Local Servers](./local_servers.md) - In-process servers
- [MCP Configuration](./configuration.md) - Complete configuration reference
- [Action Servers](./action.md) - Action execution overview
- **[Creating Custom MCP Servers Tutorial](../tutorials/creating_mcp_servers.md)** - Step-by-step guide for HTTP/Stdio servers
- [HardwareExecutor](servers/hardware_executor.md) - Complete hardware control reference
- [BashExecutor](servers/bash_executor.md) - Complete Linux command reference
---
### Documents Docs Mobile As Galaxy Device (`documents/docs/mobile/as_galaxy_device.md`)
# Using Mobile Agent as Galaxy Device
Configure Mobile Agent as a sub-agent in UFO's Galaxy framework to enable cross-platform, multi-device task orchestration. Galaxy can coordinate Mobile agents alongside Windows and Linux devices to execute complex workflows spanning multiple systems and platforms.
> **📖 Prerequisites:**
>
> Before configuring Mobile Agent in Galaxy, ensure you have:
>
> - Completed the [Mobile Agent Quick Start Guide](../getting_started/quick_start_mobile.md) - Learn how to set up server, MCP services, and client
> - Read the [Mobile Agent Overview](overview.md) - Understand Mobile Agent's design and capabilities
> - Reviewed the [Galaxy Overview](../galaxy/overview.md) - Understand multi-device orchestration
## Overview
The **Galaxy framework** provides multi-tier orchestration capabilities, allowing you to manage multiple device agents (Windows, Linux, Android, etc.) from a central ConstellationAgent. When configured as a Galaxy device, MobileAgent becomes a **sub-agent** that can:
- Execute Android-specific subtasks assigned by Galaxy
- Participate in cross-platform workflows (e.g., Windows + Android + Linux collaboration)
- Report execution status back to the orchestrator
- Be dynamically selected based on capabilities and metadata
For detailed information about MobileAgent's design and capabilities, see [Mobile Agent Overview](overview.md).
## Galaxy Architecture with Mobile Agent
```mermaid
graph TB
User[User Request]
Galaxy[Galaxy ConstellationAgent Orchestrator]
subgraph "Device Pool"
Win1[Windows Device 1 HostAgent]
Linux1[Linux Agent 1 CLI Executor]
Mobile1[Mobile Agent 1 Android Phone]
Mobile2[Mobile Agent 2 Android Tablet]
Mobile3[Mobile Agent 3 Android Emulator]
end
User -->|Complex Task| Galaxy
Galaxy -->|Windows Subtask| Win1
Galaxy -->|Linux Subtask| Linux1
Galaxy -->|Mobile Subtask| Mobile1
Galaxy -->|Mobile Subtask| Mobile2
Galaxy -->|Mobile Subtask| Mobile3
style Galaxy fill:#ffe1e1
style Mobile1 fill:#c8e6c9
style Mobile2 fill:#c8e6c9
style Mobile3 fill:#c8e6c9
```
Galaxy orchestrates:
- **Task decomposition** - Break complex requests into platform-specific subtasks
- **Device selection** - Choose appropriate devices based on capabilities
- **Parallel execution** - Execute subtasks concurrently across devices
- **Result aggregation** - Combine results from all devices
---
## Configuration Guide
### Step 1: Configure Device in `devices.yaml`
Add your Mobile agent(s) to the device list in `config/galaxy/devices.yaml`:
#### Example Configuration
```yaml
devices:
- device_id: "mobile_phone_1"
server_url: "ws://192.168.1.100:5001/ws"
os: "mobile"
capabilities:
- "mobile"
- "android"
- "messaging"
- "camera"
- "location"
metadata:
os: "mobile"
device_type: "phone"
android_version: "13"
screen_size: "1080x2400"
installed_apps:
- "com.google.android.apps.maps"
- "com.whatsapp"
- "com.android.chrome"
description: "Personal Android phone for mobile tasks"
auto_connect: true
max_retries: 5
```
### Step 2: Understanding Configuration Fields
| Field | Required | Type | Description |
|-------|----------|------|-------------|
| `device_id` | ✅ Yes | string | **Unique identifier** - must match client `--client-id` |
| `server_url` | ✅ Yes | string | WebSocket URL - must match server endpoint |
| `os` | ✅ Yes | string | Operating system - set to `"mobile"` |
| `capabilities` | ❌ Optional | list | Skills/capabilities for task routing |
| `metadata` | ❌ Optional | dict | Custom context for LLM-based task execution |
| `auto_connect` | ❌ Optional | boolean | Auto-connect on Galaxy startup (default: `true`) |
| `max_retries` | ❌ Optional | integer | Connection retry attempts (default: `5`) |
### Step 3: Capabilities-Based Task Routing
Galaxy uses the `capabilities` field to intelligently route subtasks to appropriate devices. Define capabilities based on device features, installed apps, or task types.
#### Example Capability Configurations
**Personal Phone:**
```yaml
capabilities:
- "mobile"
- "android"
- "messaging"
- "whatsapp"
- "maps"
- "camera"
- "location"
```
**Work Phone:**
```yaml
capabilities:
- "mobile"
- "android"
- "email"
- "calendar"
- "office_apps"
- "vpn"
```
**Testing Emulator:**
```yaml
capabilities:
- "mobile"
- "android"
- "testing"
- "automation"
- "screenshots"
```
**Tablet:**
```yaml
capabilities:
- "mobile"
- "android"
- "tablet"
- "large_screen"
- "media"
- "reading"
```
### Step 4: Metadata for Contextual Execution
The `metadata` field provides contextual information that the LLM uses when generating actions for the Mobile agent.
#### Metadata Examples
**Personal Phone Metadata:**
```yaml
metadata:
os: "mobile"
device_type: "phone"
android_version: "13"
sdk_version: "33"
screen_size: "1080x2400"
screen_density: "420"
installed_apps:
- "com.google.android.apps.maps"
- "com.whatsapp"
- "com.android.chrome"
- "com.spotify.music"
contacts:
- "John Doe"
- "Jane Smith"
description: "Personal Android phone with social and navigation apps"
```
**Work Device Metadata:**
```yaml
metadata:
os: "mobile"
device_type: "phone"
android_version: "12"
screen_size: "1080x2340"
installed_apps:
- "com.microsoft.office.outlook"
- "com.microsoft.teams"
- "com.slack"
vpn_configured: true
email_accounts:
- "work@company.com"
description: "Work phone with corporate apps and VPN"
```
**Testing Emulator Metadata:**
```yaml
metadata:
os: "mobile"
device_type: "emulator"
android_version: "14"
sdk_version: "34"
screen_size: "1080x1920"
installed_apps:
- "com.example.testapp"
adb_over_network: true
description: "Android emulator for app testing"
```
#### How Metadata is Used
The LLM receives metadata in the system prompt, enabling context-aware action generation:
- **App Availability**: LLM knows which apps can be launched
- **Screen Size**: Informs swipe distances and touch coordinates
- **Android Version**: Affects available features and UI patterns
- **Device Type**: Phone vs tablet affects UI layout
- **Custom Fields**: Any additional context you provide
**Example**: With the personal phone metadata above, when the user requests "Navigate to restaurant", the LLM knows Maps is installed and can generate `launch_app(package_name="com.google.android.apps.maps")`.
---
## Multi-Device Configuration Example
### Complete Galaxy Setup
```
/* Detailed source-code truncated for AI context efficiency. */
```
---
## Starting Galaxy with Mobile Agents
### Prerequisites
Ensure all components are running before starting Galaxy:
1. ✅ Device Agent Servers running on all machines
2. ✅ Device Agent Clients connected to their respective servers
3. ✅ MCP Services running (both data collection and action servers)
4. ✅ ADB accessible and Android devices connected
5. ✅ USB debugging enabled on all Android devices
6. ✅ LLM configured in `config/ufo/agents.yaml` or `config/galaxy/agent.yaml`
### Launch Sequence
#### Step 1: Prepare Android Devices
```bash
# Check ADB connection to all devices
adb devices
# Expected output:
# List of devices attached
# 192.168.1.103:5555 device
# 192.168.1.104:5555 device
# emulator-5554 device
```
**For Physical Devices:**
1. Enable USB debugging in Developer Options
2. Connect via USB or wireless ADB
3. Accept ADB debugging prompt on device
**For Emulators:**
1. Start Android emulator
2. ADB connects automatically
#### Step 2: Start Device Agent Servers
```bash
# On machine hosting personal phone agent (192.168.1.103)
python -m ufo.server.app --port 5002 --platform mobile
# On machine hosting work phone agent (192.168.1.104)
python -m ufo.server.app --port 5003 --platform mobile
# On machine hosting tablet agent (192.168.1.105)
python -m ufo.server.app --port 5004 --platform mobile
```
#### Step 3: Start MCP Servers for Each Device
```bash
# On machine hosting personal phone
python -m ufo.client.mcp.http_servers.mobile_mcp_server \
--host localhost \
--data-port 8020 \
--action-port 8021 \
--server both
# On machine hosting work phone
python -m ufo.client.mcp.http_servers.mobile_mcp_server \
--host localhost \
--data-port 8022 \
--action-port 8023 \
--server both
# On machine hosting tablet
python -m ufo.client.mcp.http_servers.mobile_mcp_server \
--host localhost \
--data-port 8024 \
--action-port 8025 \
--server both
```
#### Step 4: Start Mobile Clients
```bash
# Personal phone client
python -m ufo.client.client \
--ws \
--ws-server ws://192.168.1.103:5002/ws \
--client-id mobile_phone_personal \
--platform mobile
# Work phone client
python -m ufo.client.client \
--ws \
--ws-server ws://192.168.1.104:5003/ws \
--client-id mobile_phone_work \
--platform mobile
# Tablet client
python -m ufo.client.client \
--ws \
--ws-server ws://192.168.1.105:5004/ws \
--client-id mobile_tablet_home \
--platform mobile
```
#### Step 5: Launch Galaxy
```bash
# On your control machine (interactive mode)
python -m galaxy --interactive
```
**Or launch with a specific request:**
```bash
python -m galaxy "Your cross-device task description here"
```
Galaxy will automatically connect to all configured devices and display the orchestration interface.
---
## Example Multi-Device Workflows
### Workflow 1: Cross-Platform Productivity
**User Request:**
> "Get my meeting notes from email on work phone, summarize them on desktop, and send summary to team via WhatsApp on personal phone"
**Galaxy Orchestration:**
```mermaid
sequenceDiagram
participant User
participant Galaxy
participant WorkPhone as Work Phone (Android)
participant Desktop as Windows Desktop
participant PersonalPhone as Personal Phone (Android)
User->>Galaxy: Request meeting workflow
Galaxy->>Galaxy: Decompose task
Note over Galaxy,WorkPhone: Subtask 1: Get notes from email
Galaxy->>WorkPhone: "Open Outlook and find meeting notes"
WorkPhone->>WorkPhone: Launch Outlook app
WorkPhone->>WorkPhone: Navigate to inbox
WorkPhone->>WorkPhone: Find meeting email
WorkPhone->>WorkPhone: Extract notes text
WorkPhone-->>Galaxy: Notes content
Note over Galaxy,Desktop: Subtask 2: Summarize on desktop
Galaxy->>Desktop: "Summarize meeting notes"
Desktop->>Desktop: Open Word
Desktop->>Desktop: Paste notes
Desktop->>Desktop: Generate summary
Desktop-->>Galaxy: Summary document
Note over Galaxy,PersonalPhone: Subtask 3: Send via WhatsApp
Galaxy->>PersonalPhone: "Send summary to team on WhatsApp"
PersonalPhone->>PersonalPhone: Launch WhatsApp
PersonalPhone->>PersonalPhone: Select team group
PersonalPhone->>PersonalPhone: Type summary message
PersonalPhone->>PersonalPhone: Send message
PersonalPhone-->>Galaxy: Message sent
Galaxy-->>User: Workflow completed
```
### Workflow 2: Mobile Testing Across Devices
**User Request:**
> "Test the new app on phone, tablet, and emulator, capture screenshots of each screen"
**Galaxy Orchestration:**
1. **Mobile Phone**: Install app, navigate through screens, capture screenshots
2. **Mobile Tablet**: Install app (tablet layout), navigate screens, capture screenshots
3. **Mobile Emulator**: Install app, run automated test suite, capture screenshots
4. **Windows Desktop**: Aggregate screenshots, generate test report
### Workflow 3: Location-Based Multi-Device Task
**User Request:**
> "Find nearest coffee shops on phone, book table using tablet, add calendar event on work phone"
**Galaxy Orchestration:**
1. **Personal Phone**: Launch Maps, search "coffee shops near me", get results
2. **Tablet**: Open booking app, select coffee shop, book table
3. **Work Phone**: Open Calendar, create event with location and time
4. **Galaxy**: Aggregate confirmations and notify user
---
## Task Assignment Behavior
### How Galaxy Routes Tasks to Mobile Agents
Galaxy's ConstellationAgent uses several factors to select the appropriate mobile device for each subtask:
| Factor | Description | Example |
|--------|-------------|---------|
| **Capabilities** | Match subtask requirements to device capabilities | `"messaging"` → Personal phone |
| **OS Requirement** | Platform-specific tasks routed to correct OS | Mobile tasks → Mobile agents |
| **Metadata Context** | Use device-specific apps and configurations | WhatsApp task → Device with WhatsApp |
| **Device Type** | Phone vs tablet for different UI requirements | Media viewing → Tablet |
| **Device Status** | Only assign to online, healthy devices | Skip offline or failing devices |
| **Load Balancing** | Distribute tasks across similar devices | Round-robin across phones |
### Example Task Decomposition
**User Request:**
> "Check messages on personal phone, review calendar on work phone, and play video on tablet"
**Galaxy Decomposition:**
```yaml
Task 1:
Description: "Check messages on WhatsApp"
Target: mobile_phone_personal
Reason: Has "whatsapp" capability and personal messaging apps
Task 2:
Description: "Review today's calendar events"
Target: mobile_phone_work
Reason: Has "calendar" capability and work email/calendar
Task 3:
Description: "Play video on YouTube"
Target: mobile_tablet_home
Reason: Has "media" capability and larger screen suitable for video
```
---
## Critical Configuration Requirements
!!!danger "Configuration Validation"
Ensure these match exactly or Galaxy cannot control the device:
- **Device ID**: `device_id` in `devices.yaml` must match `--client-id` in client command
- **Server URL**: `server_url` in `devices.yaml` must match `--ws-server` in client command
- **Platform**: Must include `--platform mobile` in client command
- **ADB Access**: Android device must be accessible via ADB
- **MCP Servers**: Both data collection and action servers must be running
---
## Monitoring & Debugging
### Verify Device Registration
**Check Galaxy device pool:**
```bash
curl http://:5000/api/devices
```
**Expected response:**
```json
{
"devices": [
{
"device_id": "mobile_phone_personal",
"os": "mobile",
"status": "online",
"capabilities": ["mobile", "android", "messaging", "whatsapp", "maps"]
},
{
"device_id": "mobile_phone_work",
"os": "mobile",
"status": "online",
"capabilities": ["mobile", "android", "email", "calendar", "teams"]
}
]
}
```
### View Task Assignments
Galaxy logs show task routing decisions:
```log
INFO - [Galaxy] Task decomposition: 3 subtasks created
INFO - [Galaxy] Subtask 1 → mobile_phone_personal (capability match: messaging)
INFO - [Galaxy] Subtask 2 → mobile_phone_work (capability match: calendar)
INFO - [Galaxy] Subtask 3 → mobile_tablet_home (capability match: media)
```
### Troubleshooting Device Connection
**Issue**: Mobile agent not appearing in Galaxy device pool
**Diagnosis:**
1. **Check ADB connection:**
```bash
adb devices
```
2. **Verify client connection:**
```bash
curl http://192.168.1.103:5002/api/clients
```
3. **Check `devices.yaml` configuration** matches client parameters
4. **Review Galaxy logs** for connection errors
5. **Ensure `auto_connect: true`** in `devices.yaml`
6. **Check MCP servers** are running:
```bash
curl http://localhost:8020/health # Data collection server
curl http://localhost:8021/health # Action server
```
---
## Mobile-Specific Considerations
### Screenshot Capture for Galaxy
Mobile agents automatically capture screenshots during execution, which Galaxy can:
- Display in orchestration UI
- Include in execution reports
- Use for debugging failed tasks
- Share with other agents for context
### Touch Coordinates Across Devices
Different Android devices have different screen sizes and densities. Galaxy handles this by:
- Using control IDs instead of absolute coordinates
- Having each mobile agent handle device-specific coordinate calculations
- Storing device resolution in metadata for reference
### App Availability
Galaxy can query `installed_apps` from metadata to:
- Route tasks to devices with required apps
- Skip devices missing necessary apps
- Suggest app installation when needed
---
## Related Documentation
- [Mobile Agent Overview](overview.md) - Architecture and design principles
- [Mobile Agent Commands](commands.md) - MCP tools for device interaction
- [Galaxy Overview](../galaxy/overview.md) - Multi-device orchestration framework
- [Galaxy Quick Start](../getting_started/quick_start_galaxy.md) - Galaxy deployment guide
- [Constellation Orchestrator](../galaxy/constellation_orchestrator/overview.md) - Task orchestration
- [Galaxy Devices Configuration](../configuration/system/galaxy_devices.md) - Complete device configuration reference
---
## Summary
Using Mobile Agent as a Galaxy device enables sophisticated multi-device orchestration:
- **Cross-Platform Workflows**: Seamlessly combine Android, Windows, and Linux tasks
- **Capability-Based Routing**: Galaxy selects the right device for each subtask
- **Visual Context**: Screenshots provide rich execution tracing
- **Parallel Execution**: Multiple mobile devices work concurrently
- **Metadata-Aware**: LLM uses device-specific context (installed apps, screen size, etc.)
- **Robust Caching**: Efficient ADB usage through smart caching strategies
With Mobile Agent in Galaxy, you can orchestrate complex workflows spanning mobile apps, desktop applications, and server systems from a single unified interface.
---
### Documents Docs Mobile Commands (`documents/docs/mobile/commands.md`)
# MobileAgent MCP Commands
MobileAgent interacts with Android devices through MCP (Model Context Protocol) tools provided by two specialized MCP servers. These tools provide atomic building blocks for mobile task execution, isolating device-specific operations within the MCP server layer.
> **📖 Related Documentation:**
>
> - [Mobile Agent Overview](overview.md) - Architecture and core responsibilities
> - [State Machine](state.md) - FSM states and transitions
> - [Processing Strategy](strategy.md) - How commands are orchestrated in the 4-phase pipeline
> - [Quick Start Guide](../getting_started/quick_start_mobile.md) - Set up MCP servers for your device
## Command Architecture
### Dual MCP Server Design
MobileAgent uses two separate MCP servers for different responsibilities:
```mermaid
graph LR
A[MobileAgent] --> B[Command Dispatcher]
B --> C[Data Collection Server Port 8020]
B --> D[Action Server Port 8021]
C --> E[ADB Commands screencap, uiautomator, pm list]
D --> F[ADB Commands input tap/swipe/text, monkey]
E --> G[Android Device]
F --> G
C -.Shared State.-> H[MobileServerState Singleton]
D -.Shared State.-> H
```
**Why Two Servers?**
- **Separation of Concerns**: Data retrieval vs. device control
- **Performance**: Data collection can cache aggressively, actions invalidate caches
- **Security**: Different tools can have different permission levels
- **Scalability**: Servers can run on different hosts if needed
**Shared State**: Both servers share a singleton `MobileServerState` for:
- Caching control information (5 seconds TTL)
- Caching installed apps (5 minutes TTL)
- Caching UI tree (5 seconds TTL)
- Coordinating cache invalidation after actions
### Command Dispatcher
The command dispatcher routes commands to the appropriate MCP server:
```python
from aip.messages import Command
# Create data collection command
command = Command(
tool_name="capture_screenshot",
parameters={},
tool_type="data_collection"
)
# Execute command via dispatcher
results = await command_dispatcher.execute_commands([command])
screenshot_url = results[0].result
```
---
## Data Collection Server Tools (Port 8020)
The Data Collection Server provides read-only tools for gathering device information.
### 1. capture_screenshot - Capture Device Screenshot
**Purpose**: Take screenshot from Android device and return as base64-encoded image.
#### Tool Specification
```python
tool_name = "capture_screenshot"
parameters = {} # No parameters required
```
#### Execution Flow
```mermaid
sequenceDiagram
participant Agent
participant MCP
participant ADB
participant Device
Agent->>MCP: capture_screenshot()
MCP->>ADB: screencap -p /sdcard/screen_temp.png
ADB->>Device: Execute screenshot
Device-->>ADB: Screenshot saved
ADB->>Device: pull /sdcard/screen_temp.png
Device-->>ADB: PNG file
MCP->>MCP: Encode to base64
MCP->>ADB: rm /sdcard/screen_temp.png
MCP-->>Agent: data:image/png;base64,...
```
#### Result Format
```python
# Direct base64 data URI string (not a dict)
"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA..."
```
#### Use Cases
| Use Case | Description |
|----------|-------------|
| **UI Analysis** | Understand current screen state |
| **Visual Context** | Provide screenshots to LLM for decision making |
| **Debugging** | Capture UI state at each step |
| **Annotation Base** | Base image for control labeling |
#### Error Handling
```python
# Failures return as exceptions
try:
screenshot_url = await capture_screenshot()
except Exception as e:
# "Failed to capture screenshot on device"
# "Failed to pull screenshot from device"
pass
```
---
### 2. get_ui_tree - Get UI Hierarchy XML
**Purpose**: Retrieve the complete UI hierarchy in XML format for detailed UI structure analysis.
#### Tool Specification
```python
tool_name = "get_ui_tree"
parameters = {} # No parameters required
```
#### Execution Flow
```mermaid
sequenceDiagram
participant Agent
participant MCP
participant ADB
participant Device
Agent->>MCP: get_ui_tree()
MCP->>ADB: uiautomator dump /sdcard/window_dump.xml
ADB->>Device: Dump UI hierarchy
Device-->>ADB: XML created
ADB->>Device: cat /sdcard/window_dump.xml
Device-->>ADB: XML content
ADB-->>MCP: XML string
MCP->>MCP: Cache UI tree (5s TTL)
MCP-->>Agent: UI tree dictionary
```
#### Result Format
```python
{
"success": True,
"ui_tree": """
...
""",
"format": "xml"
}
```
#### Use Cases
- Advanced UI analysis requiring full hierarchy
- Custom control parsing logic
- Debugging UI structure
- Extracting accessibility information
---
### 3. get_device_info - Get Device Information
**Purpose**: Gather comprehensive device information including model, Android version, screen size, and battery status.
#### Tool Specification
```python
tool_name = "get_device_info"
parameters = {} # No parameters required
```
#### Information Collected
| Info Type | ADB Command | Data Returned |
|-----------|-------------|---------------|
| **Model** | `getprop ro.product.model` | Device model name |
| **Android Version** | `getprop ro.build.version.release` | Android version (e.g., "13") |
| **SDK Version** | `getprop ro.build.version.sdk` | API level (e.g., "33") |
| **Screen Size** | `wm size` | Resolution (e.g., "Physical size: 1080x2400") |
| **Screen Density** | `wm density` | DPI (e.g., "Physical density: 420") |
| **Battery Level** | `dumpsys battery` | Battery percentage |
| **Battery Status** | `dumpsys battery` | Charging status |
#### Result Format
```python
{
"success": True,
"device_info": {
"model": "Pixel 6",
"android_version": "13",
"sdk_version": "33",
"screen_size": "Physical size: 1080x2400",
"screen_density": "Physical density: 420",
"battery_level": "85",
"battery_status": "2" # 2 = Charging, 3 = Discharging
},
"from_cache": False # True if returned from cache
}
```
**Caching**: Device info is cached for 60 seconds as it changes infrequently.
---
### 4. get_mobile_app_target_info - List Installed Apps
**Purpose**: Retrieve list of installed applications as TargetInfo objects.
#### Tool Specification
```python
tool_name = "get_mobile_app_target_info"
parameters = {
"filter": "", # Filter pattern (optional)
"include_system_apps": False, # Include system apps (default: False)
"force_refresh": False # Bypass cache (default: False)
}
```
#### Execution Flow
```mermaid
sequenceDiagram
participant Agent
participant MCP
participant Cache
participant ADB
participant Device
Agent->>MCP: get_mobile_app_target_info(include_system_apps=False)
alt Cache Hit (not forced refresh)
MCP->>Cache: Check cache (5min TTL)
Cache-->>MCP: Cached app list
MCP-->>Agent: Apps from cache
else Cache Miss
MCP->>ADB: pm list packages -3
ADB->>Device: List user-installed packages
Device-->>ADB: Package list
ADB-->>MCP: Packages
MCP->>MCP: Parse to TargetInfo objects
MCP->>Cache: Update cache
MCP-->>Agent: App list
end
```
#### Result Format
```python
[
{
"id": "1",
"name": "com.android.chrome",
"package": "com.android.chrome"
},
{
"id": "2",
"name": "com.google.android.apps.maps",
"package": "com.google.android.apps.maps"
},
{
"id": "3",
"name": "com.whatsapp",
"package": "com.whatsapp"
}
]
```
**Notes**:
- `id`: Sequential number for LLM reference
- `name`: Package name (display name not available via simple ADB)
- `package`: Full package identifier
**Caching**: Apps list is cached for 5 minutes to reduce overhead.
---
### 5. get_app_window_controls_target_info - Get UI Controls
**Purpose**: Extract UI controls from current screen with IDs for precise interaction.
#### Tool Specification
```python
tool_name = "get_app_window_controls_target_info"
parameters = {
"force_refresh": False # Bypass cache (default: False)
}
```
#### Execution Flow
```mermaid
sequenceDiagram
participant Agent
participant MCP
participant Cache
participant ADB
participant Device
Agent->>MCP: get_app_window_controls_target_info()
alt Cache Hit (not forced refresh)
MCP->>Cache: Check cache (5s TTL)
Cache-->>MCP: Cached controls
MCP-->>Agent: Controls from cache
else Cache Miss
MCP->>ADB: uiautomator dump /sdcard/window_dump.xml
ADB->>Device: Dump UI
Device-->>ADB: XML file
ADB->>Device: cat /sdcard/window_dump.xml
Device-->>ADB: XML content
ADB-->>MCP: UI hierarchy
MCP->>MCP: Parse XML
MCP->>MCP: Filter meaningful controls
MCP->>MCP: Validate rectangles
MCP->>MCP: Assign sequential IDs
MCP->>Cache: Update cache
MCP-->>Agent: Controls list
end
```
#### Control Selection Criteria
Controls are included if they meet any of these criteria:
- `clickable="true"` - Can be tapped
- `long-clickable="true"` - Supports long-press
- `scrollable="true"` - Can be scrolled
- `checkable="true"` - Checkbox or toggle
- Has `text` or `content-desc` - Has label
- Type includes "Edit", "Button" - Input or action element
#### Rectangle Validation
Controls with invalid rectangles are filtered out:
```python
# Bounds format: [left, top, right, bottom]
# Valid rectangle must have:
# - right > left (positive width)
# - bottom > top (positive height)
# - All coordinates > 0
if right <= left or bottom <= top or right == 0 or bottom == 0:
skip_control() # Invalid rectangle
```
#### Result Format
```python
[
{
"id": "1",
"name": "Search",
"type": "EditText",
"rect": [48, 96, 912, 192] # [left, top, right, bottom] in pixels
},
{
"id": "2",
"name": "Search",
"type": "ImageButton",
"rect": [912, 96, 1032, 192]
},
{
"id": "3",
"name": "Maps",
"type": "TextView",
"rect": [0, 216, 1080, 360]
}
]
```
**Caching**: Controls are cached for 5 seconds but **automatically invalidated** after any action (UI likely changed).
---
## Action Server Tools (Port 8021)
The Action Server provides tools for device control and manipulation.
### 6. tap - Tap at Coordinates
**Purpose**: Perform tap/click action at specified screen coordinates.
#### Tool Specification
```python
tool_name = "tap"
parameters = {
"x": 480, # X coordinate (pixels from left)
"y": 240 # Y coordinate (pixels from top)
}
```
#### Execution Flow
```mermaid
sequenceDiagram
participant Agent
participant MCP
participant ADB
participant Device
Agent->>MCP: tap(x=480, y=240)
MCP->>ADB: input tap 480 240
ADB->>Device: Inject tap event
Device-->>ADB: Success
ADB-->>MCP: Success
MCP->>MCP: Invalidate controls cache
MCP-->>Agent: Result
```
#### Result Format
```python
{
"success": True,
"action": "tap(480, 240)",
"output": "",
"error": ""
}
```
**Cache Invalidation**: Automatically invalidates control cache after tap (UI likely changed).
---
### 7. swipe - Swipe Gesture
**Purpose**: Perform swipe gesture from start to end coordinates.
#### Tool Specification
```python
tool_name = "swipe"
parameters = {
"start_x": 500,
"start_y": 1500,
"end_x": 500,
"end_y": 500,
"duration": 300 # milliseconds (default: 300)
}
```
#### Common Use Cases
| Use Case | Start | End | Description |
|----------|-------|-----|-------------|
| **Scroll Up** | (500, 1500) | (500, 500) | Swipe from bottom to top |
| **Scroll Down** | (500, 500) | (500, 1500) | Swipe from top to bottom |
| **Scroll Left** | (900, 600) | (100, 600) | Swipe from right to left |
| **Scroll Right** | (100, 600) | (900, 600) | Swipe from left to right |
#### Result Format
```python
{
"success": True,
"action": "swipe(500,1500)->(500,500) in 300ms",
"output": "",
"error": ""
}
```
**Cache Invalidation**: Automatically invalidates control cache after swipe.
---
### 8. type_text - Type Text into Control
**Purpose**: Type text into a specific input field control.
#### Tool Specification
```python
tool_name = "type_text"
parameters = {
"text": "hello world",
"control_id": "5", # REQUIRED: Control ID from get_app_window_controls_target_info
"control_name": "Search", # REQUIRED: Control name (must match)
"clear_current_text": False # Clear existing text first (default: False)
}
```
#### Execution Flow
```mermaid
sequenceDiagram
participant Agent
participant MCP
participant Cache
participant ADB
participant Device
Agent->>MCP: type_text(text="hello", control_id="5", control_name="Search")
MCP->>Cache: Get control by ID
Cache-->>MCP: Control with rect
MCP->>MCP: Calculate center position
MCP->>ADB: input tap x y (focus control)
ADB->>Device: Tap input field
alt clear_current_text=True
MCP->>ADB: input keyevent KEYCODE_DEL (x50)
ADB->>Device: Delete existing text
end
MCP->>MCP: Escape text (spaces -> %s)
MCP->>ADB: input text hello%sworld
ADB->>Device: Type text
Device-->>ADB: Success
MCP->>MCP: Invalidate controls cache
MCP-->>Agent: Result
```
#### Important Notes
!!!warning "Control ID Requirement"
The `control_id` parameter is **REQUIRED**. You must:
1. Call `get_app_window_controls_target_info` first
2. Identify the input field control
3. Use its `id` and `name` in `type_text`
The tool will:
- Verify the control exists in cache
- Click the control to focus it
- Then type the text
**Text Escaping**: Spaces are automatically converted to `%s` for Android input shell compatibility.
#### Result Format
```python
{
"success": True,
"action": "type_text(text='hello world', control_id='5', control_name='Search')",
"message": "Clicked control 'Search' at (480, 144) | Typed text: 'hello world'",
"control_info": {
"id": "5",
"name": "Search",
"type": "EditText"
}
}
```
---
### 9. launch_app - Launch Application
**Purpose**: Launch an application by package name or app ID.
#### Tool Specification
```python
tool_name = "launch_app"
parameters = {
"package_name": "com.google.android.apps.maps", # Package name
"id": "2" # Optional: App ID from get_mobile_app_target_info
}
```
#### Usage Modes
**Mode 1: Launch by package name**
```python
launch_app(package_name="com.android.settings")
```
**Mode 2: Launch from cached app list**
```python
# First call get_mobile_app_target_info to cache apps
# Then use app ID from the list
launch_app(package_name="com.android.settings", id="5")
```
**Mode 3: Launch by app name (fuzzy search)**
```python
# If package_name doesn't contain ".", search by name
launch_app(package_name="Maps") # Finds "com.google.android.apps.maps"
```
#### Execution Flow
```mermaid
sequenceDiagram
participant Agent
participant MCP
participant ADB
participant Device
Agent->>MCP: launch_app(package_name="com.google.android.apps.maps")
alt ID provided
MCP->>MCP: Verify ID in cache
MCP->>MCP: Get package from cache
else Name only (no dots)
MCP->>ADB: pm list packages
MCP->>MCP: Search for matching package
end
MCP->>ADB: monkey -p com.google.android.apps.maps -c android.intent.category.LAUNCHER 1
ADB->>Device: Launch app
Device-->>ADB: App started
ADB-->>MCP: Success
MCP-->>Agent: Result
```
#### Result Format
```python
{
"success": True,
"message": "Launched com.google.android.apps.maps",
"package_name": "com.google.android.apps.maps",
"output": "Events injected: 1",
"error": "",
"app_info": { # If ID was provided
"id": "2",
"name": "com.google.android.apps.maps",
"package": "com.google.android.apps.maps"
}
}
```
---
### 10. press_key - Press Hardware/Software Key
**Purpose**: Press a hardware or software key for navigation and system actions.
#### Tool Specification
```python
tool_name = "press_key"
parameters = {
"key_code": "KEYCODE_BACK" # Key code name
}
```
#### Common Key Codes
| Key Code | Description | Use Case |
|----------|-------------|----------|
| `KEYCODE_HOME` | Home button | Return to home screen |
| `KEYCODE_BACK` | Back button | Navigate back |
| `KEYCODE_MENU` | Menu button | Open options menu |
| `KEYCODE_ENTER` | Enter key | Submit form |
| `KEYCODE_DEL` | Delete key | Delete character |
| `KEYCODE_APP_SWITCH` | Recent apps | Switch between apps |
| `KEYCODE_POWER` | Power button | Lock screen |
| `KEYCODE_VOLUME_UP` | Volume up | Increase volume |
| `KEYCODE_VOLUME_DOWN` | Volume down | Decrease volume |
#### Result Format
```python
{
"success": True,
"action": "press_key(KEYCODE_BACK)",
"output": "",
"error": ""
}
```
---
### 11. click_control - Click Control by ID
**Purpose**: Click a UI control by its ID from the cached control list.
#### Tool Specification
```python
tool_name = "click_control"
parameters = {
"control_id": "5", # REQUIRED: Control ID from get_app_window_controls_target_info
"control_name": "Search Button" # REQUIRED: Control name (must match)
}
```
#### Execution Flow
```mermaid
sequenceDiagram
participant Agent
participant MCP
participant Cache
participant ADB
participant Device
Agent->>MCP: click_control(control_id="5", control_name="Search")
MCP->>Cache: Get control by ID "5"
Cache-->>MCP: Control with rect [48,96,912,192]
MCP->>MCP: Verify name matches
MCP->>MCP: Calculate center: x=(48+912)/2, y=(96+192)/2
MCP->>ADB: input tap 480 144
ADB->>Device: Tap at (480, 144)
Device-->>ADB: Success
MCP->>MCP: Invalidate controls cache
MCP-->>Agent: Result
```
#### Result Format
```python
{
"success": True,
"action": "click_control(id=5, name=Search)",
"message": "Clicked control 'Search' at (480, 144)",
"control_info": {
"id": "5",
"name": "Search",
"type": "EditText",
"rect": [48, 96, 912, 192]
}
}
```
**Name Verification**: If the provided `control_name` doesn't match the cached control's name, a warning is returned but the action still executes using the ID.
---
### 12. wait - Wait/Sleep
**Purpose**: Wait for a specified duration.
#### Tool Specification
```python
tool_name = "wait"
parameters = {
"seconds": 1.0 # Duration in seconds (can be decimal)
}
```
#### Use Cases
- Wait for app to load
- Wait for animation to complete
- Wait for UI transition
- Pace actions for stability
#### Examples
```python
wait(seconds=1.0) # Wait 1 second
wait(seconds=0.5) # Wait 500ms
wait(seconds=2.5) # Wait 2.5 seconds
```
#### Result Format
```python
{
"success": True,
"action": "wait(1.0s)",
"message": "Waited for 1.0 seconds"
}
```
**Limits**:
- Minimum: 0 seconds
- Maximum: 60 seconds
---
### 13. invalidate_cache - Manual Cache Invalidation
**Purpose**: Manually invalidate cached data to force refresh on next query.
#### Tool Specification
```python
tool_name = "invalidate_cache"
parameters = {
"cache_type": "all" # "controls", "apps", "ui_tree", "device_info", or "all"
}
```
#### Cache Types
| Cache Type | Description | Auto-Invalidated |
|------------|-------------|------------------|
| `controls` | UI controls list | ✓ After actions |
| `apps` | Installed apps list | ✗ Never |
| `ui_tree` | UI hierarchy XML | ✗ Never |
| `device_info` | Device information | ✗ Never |
| `all` | All caches | Varies |
#### Result Format
```python
{
"success": True,
"message": "Controls cache invalidated"
}
```
**Use Cases**:
- Manually refresh apps list after installing/uninstalling
- Force UI tree refresh after significant screen change
- Debug caching issues
---
## Command Execution Pipeline
### Atomic Building Blocks
The MCP tools serve as atomic operations for mobile task execution:
```mermaid
graph TD
A[User Request] --> B[Data Collection Phase]
B --> B1[capture_screenshot]
B --> B2[get_mobile_app_target_info]
B --> B3[get_app_window_controls_target_info]
B1 --> C[LLM Reasoning]
B2 --> C
B3 --> C
C --> D{Select Action}
D -->|Launch| E[launch_app]
D -->|Type| F[type_text]
D -->|Click| G[click_control]
D -->|Swipe| H[swipe]
D -->|Navigate| I[press_key]
D -->|Wait| J[wait]
E --> K[Capture Result]
F --> K
G --> K
H --> K
I --> K
J --> K
K --> L[Update Memory]
L --> M{Task Complete?}
M -->|No| B
M -->|Yes| N[FINISH]
```
### Command Composition
MobileAgent executes commands sequentially, building on previous results:
```python
# Round 1: Capture UI and launch app
{
"action": {
"function": "launch_app",
"arguments": {"package_name": "com.google.android.apps.maps", "id": "2"}
}
}
# Result: Maps launched
# Round 2: Capture new UI, identify search field
{
"action": {
"function": "click_control",
"arguments": {"control_id": "5", "control_name": "Search"}
}
}
# Result: Search field focused
# Round 3: Type query
{
"action": {
"function": "type_text",
"arguments": {
"text": "restaurants",
"control_id": "5",
"control_name": "Search"
}
}
}
# Result: Text entered
```
---
## Best Practices
### Data Collection Tools
- Use `get_app_window_controls_target_info` before every action to get fresh control IDs
- Cache is your friend: don't force refresh unless necessary
- Annotated screenshots help LLM identify controls precisely
### Action Tools
!!!success "Action Best Practices"
- **Always** call `get_app_window_controls_target_info` before `click_control` or `type_text`
- Use control IDs instead of coordinates for robustness
- Add `wait` after actions that trigger UI changes (app launch, navigation)
- Check `success` field in results before considering action successful
- Use `press_key(KEYCODE_BACK)` for navigation instead of screen taps when possible
### Caching
- Controls cache: 5 seconds TTL, invalidated after actions
- Apps cache: 5 minutes TTL, manually invalidate if apps change
- Device info cache: 60 seconds TTL, useful for metadata
### Error Handling
```python
# Always check success field
result = await click_control(control_id="5", control_name="Search")
if not result["success"]:
# Handle error: control not found, device disconnected, etc.
pass
```
---
## Implementation Location
The MCP server implementation can be found in:
```
ufo/client/mcp/http_servers/
└── mobile_mcp_server.py
```
Key components:
- `MobileServerState`: Singleton state manager for caching
- `create_mobile_data_collection_server()`: Data collection server (port 8020)
- `create_mobile_action_server()`: Action server (port 8021)
---
## Comparison with Other Agent Commands
| Agent | Command Types | Execution Layer | Visual Context | Result Format |
|-------|--------------|-----------------|----------------|---------------|
| **MobileAgent** | UI + Apps + Touch | MCP (ADB) | ✓ Screenshots + Controls | success/message/control_info |
| **LinuxAgent** | CLI + SysInfo | MCP (SSH) | ✗ Text-only | success/exit_code/stdout/stderr |
| **AppAgent** | UI + API | Automator + MCP | ✓ Screenshots + Controls | UI state + API responses |
MobileAgent's command set reflects the mobile environment:
- **Touch-based**: tap, swipe instead of click, drag
- **Visual**: Screenshots are essential for UI understanding
- **App-centric**: Focus on app launching and switching
- **Control-based**: Precise control IDs instead of coordinates
- **Cached**: Aggressive caching to reduce ADB overhead
---
## Next Steps
- [State Machine](state.md) - Understand how command execution fits into the FSM
- [Processing Strategy](strategy.md) - See how commands are integrated into the 4-phase pipeline
- [Overview](overview.md) - Return to MobileAgent architecture overview
- [As Galaxy Device](as_galaxy_device.md) - Configure MobileAgent for multi-device workflows
---
### Documents Docs Mobile State (`documents/docs/mobile/state.md`)
# MobileAgent State Machine
MobileAgent uses a **3-state finite state machine (FSM)** to manage Android device task execution flow. The minimal state set captures essential execution progression while maintaining simplicity and predictability. States transition based on LLM decisions and action execution results.
> **📖 Related Documentation:**
>
> - [Mobile Agent Overview](overview.md) - Architecture and core responsibilities
> - [Processing Strategy](strategy.md) - 4-phase pipeline execution in CONTINUE state
> - [MCP Commands](commands.md) - Available mobile interaction commands
> - [Quick Start Guide](../getting_started/quick_start_mobile.md) - Set up your first Mobile Agent
## State Machine Architecture
### State Enumeration
```python
class MobileAgentStatus(Enum):
"""Store the status of the mobile agent"""
CONTINUE = "CONTINUE" # Task is ongoing, requires further actions
FINISH = "FINISH" # Task completed successfully
FAIL = "FAIL" # Task cannot proceed, unrecoverable error
```
### State Management
MobileAgent states are managed by `MobileAgentStateManager`, which implements the agent state registry pattern:
```python
class MobileAgentStateManager(AgentStateManager):
"""Manages the states of the mobile agent"""
_state_mapping: Dict[str, Type[MobileAgentState]] = {}
@property
def none_state(self) -> AgentState:
return NoneMobileAgentState()
```
All MobileAgent states are registered using the `@MobileAgentStateManager.register` decorator, enabling dynamic state lookup by name.
## State Transition Diagram
```mermaid
stateDiagram-v2
[*] --> CONTINUE: Start Task
CONTINUE --> CONTINUE: More Actions Needed (LLM returns CONTINUE)
CONTINUE --> FINISH: Task Complete (LLM returns FINISH)
CONTINUE --> FAIL: Unrecoverable Error (LLM returns FAIL or Exception)
FINISH --> [*]: Session Ends
FAIL --> FINISH: Cleanup
FINISH --> [*]: Session Ends
note right of CONTINUE
Active execution state:
- Capture screenshots
- Collect UI controls
- Get LLM decision
- Execute actions
- Update memory
end note
note right of FINISH
Terminal state:
- Task completed successfully
- Results available in memory
- Agent can be terminated
end note
note right of FAIL
Error terminal state:
- Unrecoverable error occurred
- Error details logged
- Transitions to FINISH for cleanup
end note
```
## State Definitions
### 1. CONTINUE State
**Purpose**: Active execution state where MobileAgent processes the user request and executes mobile actions.
```python
@MobileAgentStateManager.register
class ContinueMobileAgentState(MobileAgentState):
"""The class for the continue mobile agent state"""
async def handle(self, agent: "MobileAgent", context: Optional["Context"] = None):
"""Execute the 4-phase processing pipeline"""
await agent.process(context)
def is_round_end(self) -> bool:
return False # Round continues
def is_subtask_end(self) -> bool:
return False # Subtask continues
@classmethod
def name(cls) -> str:
return MobileAgentStatus.CONTINUE.value
```
| Property | Value |
|----------|-------|
| **Type** | Active |
| **Processor Executed** | ✓ Yes (4 phases) |
| **Round Ends** | No |
| **Subtask Ends** | No |
| **Duration** | Single round |
| **Next States** | CONTINUE, FINISH, FAIL |
**Behavior**:
1. **Data Collection Phase**:
- Captures device screenshot
- Retrieves installed apps list
- Collects current screen UI controls
- Creates annotated screenshot with control IDs
2. **LLM Interaction Phase**:
- Constructs prompts with screenshots and control information
- Gets next action from LLM
- Parses and validates response
3. **Action Execution Phase**:
- Executes mobile actions (tap, swipe, type, launch app, etc.)
- Captures execution results
4. **Memory Update Phase**:
- Updates memory with screenshots and action results
- Stores control information for next round
5. **State Determination**:
- Analyzes LLM response for next state
**State Transition Logic**:
- **CONTINUE → CONTINUE**: Task requires more actions to complete (e.g., need to navigate through multiple screens)
- **CONTINUE → FINISH**: LLM determines task is complete (e.g., successfully filled form and submitted)
- **CONTINUE → FAIL**: Unrecoverable error encountered (e.g., required app not installed, control not found after multiple attempts)
### 2. FINISH State
**Purpose**: Terminal state indicating successful task completion.
```python
@MobileAgentStateManager.register
class FinishMobileAgentState(MobileAgentState):
"""The class for the finish mobile agent state"""
def next_agent(self, agent: "MobileAgent") -> "MobileAgent":
return agent
def next_state(self, agent: "MobileAgent") -> MobileAgentState:
return FinishMobileAgentState() # Remains in FINISH
def is_subtask_end(self) -> bool:
return True # Subtask completed
def is_round_end(self) -> bool:
return True # Round ends
@classmethod
def name(cls) -> str:
return MobileAgentStatus.FINISH.value
```
| Property | Value |
|----------|-------|
| **Type** | Terminal |
| **Processor Executed** | ✗ No |
| **Round Ends** | Yes |
| **Subtask Ends** | Yes |
| **Duration** | Permanent |
| **Next States** | FINISH (no transition) |
**Behavior**:
- Signals task completion to session manager
- No further processing occurs
- Agent instance can be terminated
- Screenshots and action history available in memory
**FINISH state is reached when**:
- All required mobile actions have been executed successfully
- The LLM determines the user request has been fulfilled
- Target UI state has been achieved (e.g., form submitted, information displayed)
- No errors or exceptions occurred during execution
### 3. FAIL State
**Purpose**: Terminal state indicating task failure due to unrecoverable errors.
```python
@MobileAgentStateManager.register
class FailMobileAgentState(MobileAgentState):
"""The class for the fail mobile agent state"""
def next_agent(self, agent: "MobileAgent") -> "MobileAgent":
return agent
def next_state(self, agent: "MobileAgent") -> MobileAgentState:
return FinishMobileAgentState() # Transitions to FINISH for cleanup
def is_round_end(self) -> bool:
return True # Round ends
def is_subtask_end(self) -> bool:
return True # Subtask failed
@classmethod
def name(cls) -> str:
return MobileAgentStatus.FAIL.value
```
| Property | Value |
|----------|-------|
| **Type** | Terminal (Error) |
| **Processor Executed** | ✗ No |
| **Round Ends** | Yes |
| **Subtask Ends** | Yes |
| **Duration** | Transitions to FINISH |
| **Next States** | FINISH |
**Behavior**:
- Logs failure reason and context
- Captures final screenshot for debugging
- Transitions to FINISH state for cleanup
- Session manager receives failure status
!!!error "Failure Conditions"
FAIL state is reached when:
- **App Unavailable**: Required app is not installed or cannot be launched
- **Control Not Found**: Target UI control cannot be located after multiple attempts
- **Device Disconnected**: ADB connection lost during execution
- **Permission Denied**: Required permissions not granted on device
- **Timeout**: Actions take too long to complete
- **LLM Explicit Failure**: LLM explicitly indicates task cannot be completed
- **Repeated Action Failures**: Multiple consecutive actions fail
**Error Recovery**:
While FAIL is a terminal state, the error information is logged for debugging:
```python
# Example error logging in FAIL state
agent.logger.error(f"Mobile task failed: {error_message}")
agent.logger.debug(f"Last action: {last_action}")
agent.logger.debug(f"Current screenshot saved to: {screenshot_path}")
agent.logger.debug(f"UI controls at failure: {current_controls}")
```
## State Transition Rules
### Transition Decision Logic
State transitions are determined by the LLM's response in the **CONTINUE** state:
```python
# LLM returns status in response
parsed_response = {
"action": {
"function": "click_control",
"arguments": {"control_id": "5", "control_name": "Search"},
"status": "CONTINUE" # or "FINISH" or "FAIL"
},
"thought": "Need to click the search button to proceed"
}
# Agent updates its status based on LLM decision
agent.status = parsed_response["action"]["status"]
next_state = MobileAgentStateManager().get_state(agent.status)
```
### Transition Matrix
| Current State | Condition | Next State | Trigger |
|---------------|-----------|------------|---------|
| **CONTINUE** | LLM returns CONTINUE | CONTINUE | More actions needed (e.g., navigating multiple screens) |
| **CONTINUE** | LLM returns FINISH | FINISH | Task completed (e.g., information found and displayed) |
| **CONTINUE** | LLM returns FAIL | FAIL | Unrecoverable error (e.g., required control not available) |
| **CONTINUE** | Exception raised | FAIL | System error (e.g., ADB disconnected) |
| **FINISH** | Any | FINISH | No transition |
| **FAIL** | Any | FINISH | Cleanup transition |
## State-Specific Processing
### CONTINUE State Processing Pipeline
When in CONTINUE state, MobileAgent executes the full 4-phase pipeline:
```mermaid
graph TD
A[CONTINUE State] --> B[Phase 1: Data Collection]
B --> B1[Capture Screenshot]
B1 --> B2[Get Installed Apps]
B2 --> B3[Get Current Controls]
B3 --> B4[Create Annotated Screenshot]
B4 --> C[Phase 2: LLM Interaction]
C --> C1[Construct Prompt with Visual Context]
C1 --> C2[Send to LLM]
C2 --> C3[Parse Response]
C3 --> D[Phase 3: Action Execution]
D --> D1[Execute Mobile Action]
D1 --> D2[Capture Result]
D2 --> E[Phase 4: Memory Update]
E --> E1[Store Screenshot]
E1 --> E2[Store Action Result]
E2 --> E3[Update Control Cache]
E3 --> F{Check Status}
F -->|CONTINUE| A
F -->|FINISH| G[FINISH State]
F -->|FAIL| H[FAIL State]
```
### Terminal States (FINISH / FAIL)
Terminal states perform no processing:
- **FINISH**: Clean termination, results and screenshots available in memory
- **FAIL**: Error termination, error details and final screenshot logged
## Deterministic Control Flow
The 3-state design ensures deterministic, traceable execution:
- **Predictable Behavior**: Every execution path is well-defined
- **Debuggability**: State transitions are logged with screenshots for visual debugging
- **Testability**: Finite state space simplifies testing
- **Maintainability**: Simple state set reduces complexity
- **Visual Traceability**: Screenshots at each state provide visual execution history
## Comparison with Other Agents
| Agent | States | Complexity | Visual | Use Case |
|-------|--------|------------|--------|----------|
| **MobileAgent** | 3 | Minimal | ✓ Screenshots | Android mobile automation |
| **LinuxAgent** | 3 | Minimal | ✗ Text-only | Linux CLI task execution |
| **AppAgent** | 6 | Moderate | ✓ Screenshots | Windows app automation |
| **HostAgent** | 7 | High | ✓ Screenshots | Desktop orchestration |
MobileAgent's minimal 3-state design reflects its focused scope: execute mobile UI actions to fulfill user requests. The simplified state machine eliminates unnecessary complexity while maintaining robust error handling and completion detection, similar to LinuxAgent but with visual context support.
## Mobile-Specific Considerations
### Screenshot-Based State Tracking
Unlike LinuxAgent (text-based) or AppAgent (Windows UI API), MobileAgent relies heavily on screenshots for state understanding:
- Each CONTINUE round starts with a fresh screenshot
- Annotated screenshots show control IDs for precise interaction
- Screenshots are saved to memory for debugging and analysis
- Visual context helps LLM understand current UI state
### Control Caching
MobileAgent caches control information to minimize ADB overhead:
- Controls are cached for 5 seconds
- Cache is invalidated after each action (UI likely changed)
- Control dictionary enables quick lookup by ID
- Reduces repeated UI tree parsing
### Touch-Based Interaction
State transitions in MobileAgent are triggered by touch actions rather than keyboard commands:
- **Tap**: Primary interaction method
- **Swipe**: For scrolling and gestures
- **Type**: Text input (requires focused control)
- **Long-press**: For context menus (planned)
## Implementation Details
The state machine implementation can be found in:
```
ufo/agents/states/mobile_agent_state.py
```
Key classes:
- `MobileAgentStatus`: State enumeration (CONTINUE, FINISH, FAIL)
- `MobileAgentStateManager`: State registry and lookup
- `MobileAgentState`: Abstract base class
- `ContinueMobileAgentState`: Active execution state with 4-phase pipeline
- `FinishMobileAgentState`: Successful completion state
- `FailMobileAgentState`: Error termination state
- `NoneMobileAgentState`: Initial/undefined state
## Next Steps
- [Processing Strategy](strategy.md) - Understand the 4-phase processing pipeline executed in CONTINUE state
- [MCP Commands](commands.md) - Explore mobile UI interaction and app management commands
- [Overview](overview.md) - Return to MobileAgent architecture overview
---
### Documents Docs Mobile Strategy (`documents/docs/mobile/strategy.md`)
# MobileAgent Processing Strategy
MobileAgent executes a **4-phase processing pipeline** in the **CONTINUE** state. Each phase handles a specific aspect of mobile task execution: data collection (screenshots and controls), LLM decision making, action execution, and memory recording. This design separates visual context gathering from prompt construction, LLM reasoning, mobile action execution, and state updates, enhancing modularity and traceability.
> **📖 Related Documentation:**
>
> - [Mobile Agent Overview](overview.md) - Architecture and core responsibilities
> - [State Machine](state.md) - FSM states (this strategy runs in CONTINUE state)
> - [MCP Commands](commands.md) - Available commands used in each phase
> - [Quick Start Guide](../getting_started/quick_start_mobile.md) - Set up your first Mobile Agent
## Strategy Assembly
Processing strategies are assembled and orchestrated by the `MobileAgentProcessor` class defined in `ufo/agents/processors/customized/customized_agent_processor.py`. The processor coordinates the 4-phase pipeline execution.
### MobileAgentProcessor Overview
The `MobileAgentProcessor` extends `CustomizedProcessor` and manages the Mobile-specific workflow:
```python
class MobileAgentProcessor(CustomizedProcessor):
"""
Processor for Mobile Android MCP Agent.
Handles data collection, LLM interaction, and action execution for Android devices.
"""
def _setup_strategies(self) -> None:
"""Setup processing strategies for Mobile Agent."""
# Phase 1: Data Collection (composed strategy - fail_fast=True)
self.strategies[ProcessingPhase.DATA_COLLECTION] = ComposedStrategy(
strategies=[
MobileScreenshotCaptureStrategy(fail_fast=True),
MobileAppsCollectionStrategy(fail_fast=False),
MobileControlsCollectionStrategy(fail_fast=False),
],
name="MobileDataCollectionStrategy",
fail_fast=True,
)
# Phase 2: LLM Interaction (critical - fail_fast=True)
self.strategies[ProcessingPhase.LLM_INTERACTION] = (
MobileLLMInteractionStrategy(fail_fast=True)
)
# Phase 3: Action Execution (graceful - fail_fast=False)
self.strategies[ProcessingPhase.ACTION_EXECUTION] = (
MobileActionExecutionStrategy(fail_fast=False)
)
# Phase 4: Memory Update (graceful - fail_fast=False)
self.strategies[ProcessingPhase.MEMORY_UPDATE] = (
AppMemoryUpdateStrategy(fail_fast=False)
)
```
### Strategy Registration
| Phase | Strategy Class | fail_fast | Rationale |
|-------|---------------|-----------|-----------|
| **DATA_COLLECTION** | `ComposedStrategy` (3 sub-strategies) | ✓ True | Visual context is critical for mobile interaction |
| **LLM_INTERACTION** | `MobileLLMInteractionStrategy` | ✓ True | LLM failure requires immediate recovery |
| **ACTION_EXECUTION** | `MobileActionExecutionStrategy` | ✗ False | Action failures can be handled gracefully |
| **MEMORY_UPDATE** | `AppMemoryUpdateStrategy` | ✗ False | Memory failures shouldn't block execution |
**Fail-Fast vs Graceful:**
- **fail_fast=True**: Critical phases where errors should immediately transition to FAIL state
- **fail_fast=False**: Non-critical phases where errors can be logged and execution continues
## Four-Phase Pipeline
### Pipeline Execution Flow
```mermaid
graph LR
A[CONTINUE State] --> B[Phase 1: Data Collection]
B --> C[Phase 2: LLM Interaction]
C --> D[Phase 3: Action Execution]
D --> E[Phase 4: Memory Update]
E --> F[Determine Next State]
F --> G{Status?}
G -->|CONTINUE| A
G -->|FINISH| H[FINISH State]
G -->|FAIL| I[FAIL State]
```
## Phase 1: Data Collection Strategy (Composed)
**Purpose**: Gather comprehensive visual and structural information about the current mobile UI state.
Phase 1 is a **composed strategy** consisting of three sub-strategies executed sequentially:
1. **Screenshot Capture**: Take device screenshot
2. **Apps Collection**: List installed applications
3. **Controls Collection**: Extract UI hierarchy and annotate controls
### Sub-Strategy 1.1: Screenshot Capture
```python
@depends_on("log_path", "session_step")
@provides(
"clean_screenshot_path",
"clean_screenshot_url",
"annotated_screenshot_url", # Initially None, set by Controls Collection
"screenshot_saved_time",
)
class MobileScreenshotCaptureStrategy(BaseProcessingStrategy):
"""
Strategy for capturing Android device screenshots.
"""
```
#### Workflow
```mermaid
sequenceDiagram
participant Strategy
participant MCP
participant ADB
participant Device
Strategy->>MCP: capture_screenshot command
MCP->>ADB: screencap -p /sdcard/screen_temp.png
ADB->>Device: Execute screenshot
Device-->>ADB: Screenshot saved
ADB->>Device: Pull screenshot
Device-->>ADB: PNG file
ADB-->>MCP: PNG data
MCP->>MCP: Encode to base64
MCP-->>Strategy: data:image/png;base64,...
Strategy->>Strategy: Save to log_path
Strategy-->>Agent: Screenshot URL + path
```
#### Output
```python
{
"clean_screenshot_path": "logs/.../action_step1.png",
"clean_screenshot_url": "data:image/png;base64,iVBORw0KGgoAAAANS...",
"annotated_screenshot_url": None, # Set by Controls Collection
"screenshot_saved_time": 0.234 # seconds
}
```
### Sub-Strategy 1.2: Apps Collection
```python
@depends_on("clean_screenshot_url")
@provides("installed_apps", "apps_collection_time")
class MobileAppsCollectionStrategy(BaseProcessingStrategy):
"""
Strategy for collecting installed apps information from Android device.
"""
```
#### Workflow
```mermaid
sequenceDiagram
participant Strategy
participant MCP
participant ADB
participant Device
Strategy->>MCP: get_mobile_app_target_info
MCP->>MCP: Check cache (5min TTL)
alt Cache Hit
MCP-->>Strategy: Cached app list
else Cache Miss
MCP->>ADB: pm list packages -3
ADB->>Device: List user apps
Device-->>ADB: Package list
ADB-->>MCP: Packages
MCP->>MCP: Parse to TargetInfo
MCP->>MCP: Update cache
MCP-->>Strategy: App list
end
Strategy-->>Agent: Installed apps
```
#### Output Format
```python
{
"installed_apps": [
{
"id": "1",
"name": "com.android.chrome",
"package": "com.android.chrome"
},
{
"id": "2",
"name": "com.google.android.apps.maps",
"package": "com.google.android.apps.maps"
},
...
],
"apps_collection_time": 0.156 # seconds
}
```
**Caching**: Apps list is cached for 5 minutes to reduce ADB overhead, as installed apps rarely change during a session.
### Sub-Strategy 1.3: Controls Collection
```python
@depends_on("clean_screenshot_url")
@provides(
"current_controls",
"controls_collection_time",
"annotated_screenshot_url",
"annotated_screenshot_path",
"annotation_dict",
)
class MobileControlsCollectionStrategy(BaseProcessingStrategy):
"""
Strategy for collecting current screen controls information from Android device.
Creates annotated screenshots with control labels.
"""
```
#### Workflow
```mermaid
sequenceDiagram
participant Strategy
participant MCP
participant ADB
participant Device
participant Photographer
Strategy->>MCP: get_app_window_controls_target_info
MCP->>MCP: Check cache (5s TTL)
alt Cache Hit
MCP-->>Strategy: Cached controls
else Cache Miss
MCP->>ADB: uiautomator dump /sdcard/window_dump.xml
ADB->>Device: Dump UI hierarchy
Device-->>ADB: XML file
ADB->>Device: cat /sdcard/window_dump.xml
Device-->>ADB: XML content
ADB-->>MCP: UI hierarchy XML
MCP->>MCP: Parse XML
MCP->>MCP: Extract clickable controls
MCP->>MCP: Validate rectangles
MCP->>MCP: Assign IDs
MCP->>MCP: Update cache
MCP-->>Strategy: Controls list
end
Strategy->>Strategy: Convert to TargetInfo
Strategy->>Photographer: Create annotated screenshot
Photographer->>Photographer: Draw control IDs on screenshot
Photographer-->>Strategy: Annotated image
Strategy-->>Agent: Controls + Annotated screenshot
```
#### UI Hierarchy Parsing
The strategy parses Android UI XML to extract meaningful controls:
```xml
```
**Control Selection Criteria**:
- `clickable="true"` - Can be tapped
- `long-clickable="true"` - Supports long-press
- `scrollable="true"` - Can be scrolled
- `checkable="true"` - Checkbox or toggle
- Has `text` or `content-desc` - Has label
- Type includes "Edit", "Button" - Input or action element
**Rectangle Validation**:
Controls with invalid rectangles are filtered out:
```python
# Bounds format: [left, top, right, bottom]
if right <= left or bottom <= top:
# Invalid: width or height is zero/negative
skip_control()
```
#### Output Format
```python
{
"current_controls": [
{
"id": "1",
"name": "Search",
"type": "EditText",
"rect": [48, 96, 912, 192] # [left, top, right, bottom]
},
{
"id": "2",
"name": "Search",
"type": "ImageButton",
"rect": [912, 96, 1032, 192]
},
...
],
"annotated_screenshot_url": "data:image/png;base64,...",
"annotated_screenshot_path": "logs/.../action_step1_annotated.png",
"annotation_dict": {
"1": {"id": "1", "name": "Search", "type": "EditText", ...},
"2": {"id": "2", "name": "Search", "type": "ImageButton", ...},
...
},
"controls_collection_time": 0.345 # seconds
}
```
**Caching**: Controls are cached for 5 seconds, but the cache is invalidated after every action (UI likely changed).
### Composed Strategy Execution
The three sub-strategies are executed sequentially in a single composed strategy:
```python
ComposedStrategy(
strategies=[
MobileScreenshotCaptureStrategy(fail_fast=True),
MobileAppsCollectionStrategy(fail_fast=False),
MobileControlsCollectionStrategy(fail_fast=False),
],
name="MobileDataCollectionStrategy",
fail_fast=True, # Overall failure if screenshot capture fails
)
```
**Execution Order**:
1. Screenshot Capture (critical)
2. Apps Collection (optional, continues on failure)
3. Controls Collection (optional, continues on failure)
---
## Phase 2: LLM Interaction Strategy
**Purpose**: Construct mobile-specific prompts with visual context and obtain next action from LLM.
### Strategy Implementation
```python
@depends_on("installed_apps", "current_controls", "clean_screenshot_url")
@provides(
"parsed_response",
"response_text",
"llm_cost",
"prompt_message",
"action",
"thought",
"comment",
)
class MobileLLMInteractionStrategy(AppLLMInteractionStrategy):
"""
Strategy for LLM interaction with Mobile Agent specific prompting.
"""
```
### Phase 2 Workflow
```mermaid
sequenceDiagram
participant Strategy
participant Agent
participant Prompter
participant LLM
Strategy->>Agent: Get previous plan
Strategy->>Agent: Get blackboard context
Agent-->>Strategy: Previous execution results
Strategy->>Prompter: Construct mobile prompt
Prompter->>Prompter: Build system message (APIs + examples)
Prompter->>Prompter: Add screenshot images
Prompter->>Prompter: Add annotated screenshot
Prompter->>Prompter: Add text prompt with context
Prompter-->>Strategy: Complete multimodal prompt
Strategy->>LLM: Send prompt
LLM-->>Strategy: Mobile action + status
Strategy->>Strategy: Parse response
Strategy->>Strategy: Validate action
Strategy-->>Agent: Parsed response + cost
```
### Prompt Construction
The strategy constructs comprehensive multimodal prompts:
```python
prompt_message = agent.message_constructor(
dynamic_examples=[], # Few-shot examples (optional)
dynamic_knowledge="", # Retrieved knowledge (optional)
plan=plan, # Previous execution plan
request=request, # User request
installed_apps=installed_apps, # Available apps
current_controls=current_controls, # UI controls with IDs
screenshot_url=clean_screenshot_url, # Clean screenshot
annotated_screenshot_url=annotated_screenshot_url, # With control IDs
blackboard_prompt=blackboard_prompt, # Shared context
last_success_actions=last_success_actions # Successful actions
)
```
### Multimodal Content Structure
The prompt includes both visual and textual elements:
```python
user_content = [
# 1. Clean screenshot (for visual understanding)
{
"type": "image_url",
"image_url": {"url": "data:image/png;base64,iVBORw0KGgo..."}
},
# 2. Annotated screenshot (for control identification)
{
"type": "image_url",
"image_url": {"url": "data:image/png;base64,iVBORw0KGgo..."}
},
# 3. Text prompt with context
{
"type": "text",
"text": """
[Previous Plan]: [...]
[User Request]: Search for restaurants on Maps
[Installed Apps]: [
{"id": "1", "name": "com.google.android.apps.maps", ...},
...
]
[Current Screen Controls]: [
{"id": "1", "name": "Search", "type": "EditText", ...},
{"id": "2", "name": "Search", "type": "ImageButton", ...},
...
]
[Last Success Actions]: [...]
"""
}
]
```
### LLM Response Format
The LLM returns a structured mobile action:
```json
{
"thought": "I need to launch Google Maps app first",
"action": {
"function": "launch_app",
"arguments": {
"package_name": "com.google.android.apps.maps",
"id": "1"
},
"status": "CONTINUE"
},
"comment": "Launching Maps to search for restaurants"
}
```
### Mobile-Specific Features
**Visual Context Priority**: LLM sees both clean and annotated screenshots, enabling better UI understanding than text-only descriptions.
**Control ID References**: Annotated screenshot shows control IDs, allowing LLM to precisely reference UI elements in actions.
**App Awareness**: LLM knows which apps are installed, enabling intelligent app selection and launching.
**Touch-Based Actions**: LLM generates mobile-specific actions (tap, swipe, type) instead of desktop actions (click, drag, keyboard).
---
## Phase 3: Action Execution Strategy
**Purpose**: Execute mobile actions returned by LLM and capture structured results.
### Strategy Implementation
```python
class MobileActionExecutionStrategy(AppActionExecutionStrategy):
"""
Strategy for executing actions in Mobile Agent.
"""
```
### Phase 3 Workflow
```mermaid
sequenceDiagram
participant Strategy
participant MCP
participant ADB
participant Device
Strategy->>Strategy: Extract action from LLM response
alt launch_app
Strategy->>MCP: launch_app(package_name)
MCP->>ADB: monkey -p package_name
else click_control
Strategy->>MCP: click_control(control_id, control_name)
MCP->>MCP: Get control from cache
MCP->>MCP: Calculate center position
MCP->>ADB: input tap x y
else type_text
Strategy->>MCP: type_text(text, control_id, ...)
MCP->>ADB: input tap (focus control)
MCP->>ADB: input text (type)
else swipe
Strategy->>MCP: swipe(start_x, start_y, end_x, end_y)
MCP->>ADB: input swipe ...
else tap
Strategy->>MCP: tap(x, y)
MCP->>ADB: input tap x y
else press_key
Strategy->>MCP: press_key(key_code)
MCP->>ADB: input keyevent KEY_CODE
else wait
Strategy->>Strategy: asyncio.sleep(seconds)
end
ADB->>Device: Execute command
Device-->>ADB: Result
ADB-->>MCP: Success/Failure
MCP->>MCP: Invalidate controls cache
MCP-->>Strategy: Execution result
Strategy->>Strategy: Create action info
Strategy->>Strategy: Format for memory
Strategy-->>Agent: Execution results
```
### Action Execution Flow
```python
# Extract parsed LLM response
parsed_response: AppAgentResponse = context.get_local("parsed_response")
command_dispatcher = context.global_context.command_dispatcher
# Execute the action via MCP
execution_results = await self._execute_app_action(
command_dispatcher,
parsed_response.action
)
```
### Result Capture
Execution results are structured for downstream processing:
```python
{
"success": True,
"action": "click_control(id=5, name=Search)",
"message": "Clicked control 'Search' at (480, 144)",
"control_info": {
"id": "5",
"name": "Search",
"type": "EditText",
"rect": [48, 96, 912, 192]
}
}
```
### Action Info Creation
Results are formatted into `ActionCommandInfo` objects:
```python
actions = self._create_action_info(
parsed_response.action,
execution_results,
)
action_info = ListActionCommandInfo(actions)
action_info.color_print() # Pretty print to console
```
### Cache Invalidation
After each action, control caches are invalidated:
```python
# Mobile MCP server automatically invalidates caches after actions
# This ensures next round gets fresh UI state
mobile_state.invalidate_controls()
```
---
## Phase 4: Memory Update Strategy
**Purpose**: Persist execution results, screenshots, and control information into agent memory for future reference.
### Strategy Implementation
MobileAgent reuses the `AppMemoryUpdateStrategy` from the app agent framework:
```python
self.strategies[ProcessingPhase.MEMORY_UPDATE] = AppMemoryUpdateStrategy(
fail_fast=False # Memory failures shouldn't stop process
)
```
### Phase 4 Workflow
```mermaid
sequenceDiagram
participant Strategy
participant Memory
participant Context
Strategy->>Context: Get execution results
Strategy->>Context: Get LLM response
Strategy->>Context: Get screenshots
Strategy->>Memory: Create memory item
Memory->>Memory: Store screenshots (clean + annotated)
Memory->>Memory: Store action details
Memory->>Memory: Store control information
Memory->>Memory: Store timestamp
Strategy->>Context: Update round result
Strategy-->>Agent: Memory updated
```
### Memory Structure
Each execution round is stored as a memory item:
```python
{
"round": 1,
"request": "Search for restaurants on Maps",
"thought": "I need to launch Google Maps app first",
"action": {
"function": "launch_app",
"arguments": {
"package_name": "com.google.android.apps.maps",
"id": "1"
}
},
"result": {
"success": True,
"message": "Launched com.google.android.apps.maps"
},
"screenshots": {
"clean": "logs/.../action_step1.png",
"annotated": "logs/.../action_step1_annotated.png"
},
"controls": [
{"id": "1", "name": "Search", "type": "EditText", ...},
...
],
"status": "CONTINUE",
"timestamp": "2025-11-14T10:30:45"
}
```
### Iterative Refinement
Memory enables iterative refinement across rounds:
1. **Round 1**: Launch Maps app → Maps opened
2. **Round 2**: Click search field (using control ID from Round 1 screenshot)
3. **Round 3**: Type "restaurants" → Text entered
4. **Round 4**: Click search button → Results displayed
Each round builds on previous results and screenshots stored in memory.
### Visual Debugging
Memory stores screenshots for each round, enabling visual debugging:
- **Clean Screenshots**: Show actual device UI
- **Annotated Screenshots**: Show control IDs used by LLM
- **Action Sequence**: Visual trace of entire task execution
---
## Middleware Stack
MobileAgent uses specialized middleware for logging:
```python
def _setup_middleware(self) -> None:
"""Setup middleware pipeline for Mobile Agent"""
self.middleware_chain = [MobileLoggingMiddleware()]
```
### MobileLoggingMiddleware
Provides enhanced logging specific to Mobile operations:
```python
class MobileLoggingMiddleware(AppAgentLoggingMiddleware):
"""Specialized logging middleware for Mobile Agent"""
def starting_message(self, context: ProcessingContext) -> str:
request = context.get("request") or "Unknown Request"
return f"Completing the user request: [bold cyan]{request}[/bold cyan] on Mobile."
```
**Logged Information**:
- User request
- Screenshots captured (with paths)
- Apps collected
- Controls identified (with IDs)
- Each mobile action executed
- Action results
- State transitions
- LLM costs
- Timing information
---
## Context Finalization
After processing, the processor updates global context:
```python
def _finalize_processing_context(self, processing_context: ProcessingContext):
"""Finalize processing context by updating ContextNames fields"""
super()._finalize_processing_context(processing_context)
try:
result = processing_context.get_local("result")
if result:
self.global_context.set(ContextNames.ROUND_RESULT, result)
except Exception as e:
self.logger.warning(f"Failed to update context: {e}")
```
This makes execution results available to:
- Subsequent rounds (iterative execution)
- Other agents (if part of multi-agent workflow)
- Session manager (for monitoring and logging)
---
## Strategy Dependency Graph
The four phases have clear dependencies:
```mermaid
graph TD
A[log_path + session_step] --> B[Phase 1.1: Screenshot Capture]
B --> C[clean_screenshot_url]
C --> D[Phase 1.2: Apps Collection]
D --> E[installed_apps]
C --> F[Phase 1.3: Controls Collection]
F --> G[current_controls]
F --> H[annotated_screenshot_url]
F --> I[annotation_dict]
E --> J[Phase 2: LLM Interaction]
G --> J
C --> J
H --> J
J --> K[parsed_response]
J --> L[llm_cost]
K --> M[Phase 3: Action Execution]
I --> M
M --> N[execution_result]
M --> O[action_info]
K --> P[Phase 4: Memory Update]
N --> P
O --> P
C --> P
H --> P
P --> Q[Memory Updated]
Q --> R[Next Round or Terminal State]
```
---
## Modular Design Benefits
The 4-phase strategy design provides:
!!!success "Modularity Benefits"
- **Separation of Concerns**: Data collection, LLM reasoning, action execution, and memory are isolated
- **Visual Context**: Screenshots provide rich UI understanding beyond text descriptions
- **Testability**: Each phase can be tested independently with mocked data
- **Extensibility**: New data collection strategies can be added (e.g., accessibility info)
- **Reusability**: Memory strategy is shared with AppAgent
- **Maintainability**: Clear boundaries between perception, decision, and action
- **Traceability**: Each phase logs its operations independently with visual artifacts
- **Performance**: Caching strategies reduce ADB overhead
---
## Comparison with Other Agents
| Agent | Phases | Data Collection | Visual | LLM | Action | Memory |
|-------|--------|----------------|--------|-----|--------|--------|
| **MobileAgent** | 4 | ✓ Screenshots + Controls + Apps | ✓ Multimodal | ✓ Mobile actions | ✓ Touch/swipe | ✓ Results + Screenshots |
| **LinuxAgent** | 3 | ✗ On-demand | ✗ Text-only | ✓ CLI commands | ✓ Shell | ✓ Results |
| **AppAgent** | 4 | ✓ Screenshots + UI | ✓ Multimodal | ✓ UI actions | ✓ GUI + API | ✓ Results + Screenshots |
| **HostAgent** | 4 | ✓ Desktop snapshot | ✓ Multimodal | ✓ App selection | ✓ Orchestration | ✓ Results |
MobileAgent's 4-phase pipeline includes **DATA_COLLECTION** phase because:
- Mobile UI requires visual context (screenshots)
- Control identification needs UI hierarchy parsing
- Touch targets need precise coordinates
- Apps list informs available actions
- Annotation creates visual correspondence between LLM and execution
This reflects the visual, touch-based nature of mobile interaction.
---
## Implementation Location
The strategy implementations can be found in:
```
ufo/agents/processors/
├── customized/
│ └── customized_agent_processor.py # MobileAgentProcessor
└── strategies/
└── mobile_agent_strategy.py # Mobile-specific strategies
```
Key classes:
- `MobileAgentProcessor`: Strategy orchestrator
- `MobileScreenshotCaptureStrategy`: Screenshot capture via ADB
- `MobileAppsCollectionStrategy`: Installed apps collection
- `MobileControlsCollectionStrategy`: UI controls extraction and annotation
- `MobileLLMInteractionStrategy`: Multimodal prompt construction and LLM interaction
- `MobileActionExecutionStrategy`: Mobile action execution
- `MobileLoggingMiddleware`: Enhanced logging
---
## Next Steps
- [MCP Commands](commands.md) - Explore the mobile UI interaction and app management commands
- [State Machine](state.md) - Understand the 3-state FSM that controls strategy execution
- [Overview](overview.md) - Return to MobileAgent architecture overview
---
### Documents Docs Server Api (`documents/docs/server/api.md`)
# HTTP API Reference
The UFO Server provides a RESTful HTTP API for external systems to dispatch tasks, monitor client connections, retrieve results, and perform health checks. All endpoints are prefixed with `/api`.
## 🎯 Overview
```mermaid
graph LR
subgraph "External Systems"
Web[Web App]
Script[Python Script]
Tool[Automation Tool]
end
subgraph "UFO Server HTTP API"
Dispatch[POST /api/dispatch]
Clients[GET /api/clients]
Result[GET /api/task_result]
Health[GET /api/health]
end
subgraph "Server Core"
WSM[Client Connection Manager]
SM[Session Manager]
WH[WebSocket Handler]
end
Web --> Dispatch
Script --> Clients
Tool --> Result
Tool --> Health
Dispatch --> WSM
Dispatch --> SM
Clients --> WSM
Result --> SM
Health --> WSM
Health --> SM
WSM --> WH
SM --> WH
style Dispatch fill:#bbdefb
style Clients fill:#c8e6c9
style Result fill:#fff9c4
style Health fill:#ffcdd2
```
**Core Capabilities:**
| Capability | Endpoint | Description |
|------------|----------|-------------|
| **Task Dispatch** | `POST /api/dispatch` | Send tasks to connected devices via HTTP |
| **Client Monitoring** | `GET /api/clients` | Query connected devices and constellations |
| **Result Retrieval** | `GET /api/task_result/{task_name}` | Fetch task execution results |
| **Health Checks** | `GET /api/health` | Monitor server status and uptime |
**Why Use the HTTP API?**
- **External Integration**: Trigger UFO tasks from web apps, scripts, or CI/CD pipelines
- **Stateless**: No WebSocket connection required
- **RESTful**: Standard HTTP methods and JSON payloads
- **Monitoring**: Health checks for load balancers and monitoring systems
---
## 📡 Endpoints
### POST /api/dispatch
Send a task to a connected device without establishing a WebSocket connection. Ideal for external systems, web apps, and automation scripts.
#### Request Format
**Corrected Request Body** (based on actual source code):
```json
{
"client_id": "device_windows_001",
"request": "Open Chrome and navigate to github.com",
"task_name": "github_navigation_task"
}
```
**Request Schema:**
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `client_id` | `string` | ✅ **Yes** | - | Target client identifier (device or constellation) |
| `request` | `string` | ✅ **Yes** | - | Natural language task description (user request) |
| `task_name` | `string` | ⚠️ No | Auto-generated UUID | Human-readable task identifier |
**Important:** The correct parameter names (verified from source code) are:
- `client_id` (not `device_id`)
- `request` (not `task`)
- `task_name` (optional identifier)
#### Success Response (200)
```json
{
"status": "dispatched",
"task_name": "github_navigation_task",
"client_id": "device_windows_001",
"session_id": "d4e5f6a7-b8c9-1234-5678-9abcdef01234"
}
```
**Response Schema:**
| Field | Type | Description |
|-------|------|-------------|
| `status` | `string` | Always `"dispatched"` on success |
| `task_name` | `string` | Task identifier (from request or auto-generated) |
| `client_id` | `string` | Target client ID |
| `session_id` | `string` | UUID for tracking task execution (use with `/api/task_result`) |
#### Error Responses
**Client Not Online (404):**
```json
{
"detail": "Client not online"
}
```
**Cause:** Target client is not connected to the server.
**Solution:** Check `/api/clients` to see available clients.
**Empty Client ID (400):**
```json
{
"detail": "Empty client ID"
}
```
**Cause:** `client_id` field is missing or empty.
**Solution:** Provide a valid `client_id` in the request body.
**Empty Task Content (400):**
```json
{
"detail": "Empty task content"
}
```
**Cause:** `request` field is missing or empty.
**Solution:** Provide a non-empty task description in the `request` field.
#### Implementation Details
**Source Code** (verified from `ufo/server/services/api.py`):
```
/* Detailed source-code truncated for AI context efficiency. */
```
**Tip:** Use the returned `session_id` to track results via `GET /api/task_result/{task_name}`.
#### Sequence Diagram
```mermaid
sequenceDiagram
participant Client as External Client
participant API as HTTP API
participant WSM as Client Connection Manager
participant WS as Client WebSocket
Client->>API: POST /api/dispatch {client_id, request, task_name}
Note over API: Validate request (client_id, request not empty)
API->>WSM: get_client(client_id)
WSM-->>API: WebSocket connection
alt Client Not Online
WSM-->>API: None
API-->>Client: 404: Client not online
end
Note over API: Generate session_id Generate response_id
API->>WS: send_task_assignment() (via AIP TaskExecutionProtocol)
Note over WS: Task queued for execution
API-->>Client: 200: {status: "dispatched", session_id, task_name}
Note over Client: Poll /api/task_result/{task_name} to get result
```
---
### GET /api/clients
Query all currently connected clients (devices and constellations) to determine which targets are available for task dispatch.
#### Request
```http
GET /api/clients
```
**No parameters required.**
#### Success Response (200)
```json
{
"online_clients": [
"device_windows_001",
"device_linux_002",
"constellation_orchestrator_001"
]
}
```
**Response Schema:**
| Field | Type | Description |
|-------|------|-------------|
| `online_clients` | `array` | List of all connected client IDs |
**Source Code:**
```python
@router.get("/api/clients")
async def list_clients():
return {"online_clients": client_manager.list_clients()}
```
#### Usage Patterns
**Source Code:**
```python
@router.get("/api/clients")
async def list_clients():
return {"online_clients": client_manager.list_clients()}
```
#### Usage Patterns
**Check Device Availability:**
```python
import requests
response = requests.get("http://localhost:5000/api/clients")
clients = response.json()["online_clients"]
target_device = "device_windows_001"
if target_device in clients:
print(f"✅ {target_device} is online")
# Dispatch task
else:
print(f"❌ {target_device} is offline")
```
**Filter by Client Type:**
```python
# Note: Current API doesn't return client types
# You must know your client naming convention
# Example: devices start with "device_", constellations with "constellation_"
clients = response.json()["online_clients"]
devices = [c for c in clients if c.startswith("device_")]
constellations = [c for c in clients if c.startswith("constellation_")]
print(f"Devices online: {len(devices)}")
print(f"Constellations online: {len(constellations)}")
```
**Monitor Client Count:**
```python
import time
while True:
response = requests.get("http://localhost:5000/api/clients")
clients = response.json()["online_clients"]
print(f"[{time.strftime('%H:%M:%S')}] Clients online: {len(clients)}")
time.sleep(10) # Check every 10 seconds
```
---
### GET /api/task_result/{task_name}
Poll this endpoint to get the result of a dispatched task. Use the `task_name` returned from `/api/dispatch`.
#### Request
```http
GET /api/task_result/github_navigation_task
```
**Path Parameters:**
| Parameter | Type | Description |
|-----------|------|-------------|
| `task_name` | `string` | Task identifier (from `/api/dispatch` response) |
#### Response States
**Pending (200):**
Task is still running:
```json
{
"status": "pending"
}
```
**Action:** Continue polling until status changes to `"done"`.
**Completed (200):**
Task has finished:
```json
{
"status": "done",
"result": {
"action": "Opened Chrome and navigated to github.com",
"screenshot": "base64_encoded_image_data",
"control_label": "Address bar",
"control_text": "github.com"
}
}
```
**Action:** Process the result. The `result` structure depends on the task type and device implementation.
**Not Found (Implicit):**
If `task_name` doesn't exist in session manager:
```json
{
"status": "pending"
}
```
**Note:** Current implementation returns `{"status": "pending"}` for non-existent tasks (not a 404 error).
#### Implementation Details
**Source Code:**
```python
@router.get("/api/task_result/{task_name}")
async def get_task_result(task_name: str):
# Query session manager for result
result = session_manager.get_result_by_task(task_name)
if not result:
return {"status": "pending"}
return {"status": "done", "result": result}
```
**Note on Result Retention:**
Results are stored in memory and may be cleared after:
- Server restart
- Session cleanup (if implemented)
- Memory limits reached
**Recommendation:** Poll frequently and persist results on the client side.
#### Polling Pattern
**Recommended Polling Implementation:**
```python
import requests
import time
def wait_for_result(task_name: str, timeout: int = 300, interval: int = 2) -> dict:
"""
Poll for task result with timeout.
Args:
task_name: Task identifier
timeout: Maximum wait time in seconds (default: 5 minutes)
interval: Poll interval in seconds (default: 2 seconds)
Returns:
Task result dictionary
Raises:
TimeoutError: If task doesn't complete within timeout
"""
start_time = time.time()
while True:
elapsed = time.time() - start_time
if elapsed > timeout:
raise TimeoutError(
f"Task '{task_name}' did not complete within {timeout}s"
)
response = requests.get(
f"http://localhost:5000/api/task_result/{task_name}"
)
data = response.json()
if data["status"] == "done":
print(f"✅ Task completed in {elapsed:.1f}s")
return data["result"]
print(f"⏳ Waiting for task... ({elapsed:.0f}s)")
time.sleep(interval)
# Usage
try:
result = wait_for_result("github_navigation_task", timeout=60)
print("Result:", result)
except TimeoutError as e:
print(f"❌ {e}")
```
---
### GET /api/health
Use this endpoint for monitoring systems, load balancers, and Kubernetes liveness/readiness probes.
#### Request
```http
GET /api/health
```
**No parameters required.**
#### Success Response (200)
```json
{
"status": "healthy",
"online_clients": [
"device_windows_001",
"device_linux_002",
"constellation_orchestrator_001"
]
}
```
**Response Schema:**
| Field | Type | Description |
|-------|------|-------------|
| `status` | `string` | Always `"healthy"` if server is responding |
| `online_clients` | `array` | List of connected client IDs |
#### Implementation Details
**Source Code:**
```python
@router.get("/api/health")
async def health_check():
return {
"status": "healthy",
"online_clients": client_manager.list_clients()
}
```
#### Integration Examples
**Kubernetes Liveness Probe:**
```yaml
apiVersion: v1
kind: Pod
metadata:
name: ufo-server
spec:
containers:
- name: ufo-server
image: ufo-server:latest
ports:
- containerPort: 5000
livenessProbe:
httpGet:
path: /api/health
port: 5000
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
httpGet:
path: /api/health
port: 5000
initialDelaySeconds: 5
periodSeconds: 10
```
**Monitoring Script:**
```python
import requests
import time
def monitor_server_health(url="http://localhost:5000/api/health"):
"""Continuous health monitoring."""
consecutive_failures = 0
while True:
try:
response = requests.get(url, timeout=5)
if response.status_code == 200:
data = response.json()
client_count = len(data.get("online_clients", []))
print(
f"✅ Server healthy - {client_count} clients connected"
)
consecutive_failures = 0
else:
consecutive_failures += 1
print(
f"⚠️ Server returned {response.status_code} "
f"(failures: {consecutive_failures})"
)
except requests.RequestException as e:
consecutive_failures += 1
print(
f"❌ Server unreachable: {e} "
f"(failures: {consecutive_failures})"
)
if consecutive_failures >= 3:
# Trigger alert (email, Slack, PagerDuty, etc.)
send_alert(f"Server down for {consecutive_failures} checks")
time.sleep(30)
```
**nginx Health Check:**
```nginx
upstream ufo_backend {
server localhost:5000;
# Health check (requires nginx plus or third-party module)
check interval=10000 rise=2 fall=3 timeout=5000 type=http;
check_http_send "GET /api/health HTTP/1.0\r\n\r\n";
check_http_expect_alive http_2xx http_3xx;
}
```
---
## 💻 Usage Examples
### Python (requests)
**Complete Task Dispatch Workflow:**
```
/* Detailed source-code truncated for AI context efficiency. */
```
### cURL
**Command-Line HTTP Requests:**
**Dispatch Task:**
```bash
curl -X POST http://localhost:5000/api/dispatch \
-H "Content-Type: application/json" \
-d '{
"client_id": "device_windows_001",
"request": "Open Calculator",
"task_name": "open_calculator"
}'
# Response:
# {
# "status": "dispatched",
# "task_name": "open_calculator",
# "client_id": "device_windows_001",
# "session_id": "a1b2c3d4-..."
# }
```
**Get Clients:**
```bash
curl http://localhost:5000/api/clients
# Response:
# {
# "online_clients": [
# "device_windows_001",
# "device_linux_002"
# ]
# }
```
**Get Task Result:**
```bash
curl http://localhost:5000/api/task_result/open_calculator
# Response (pending):
# {"status": "pending"}
# Response (complete):
# {
# "status": "done",
# "result": {"action": "Opened Calculator", ...}
# }
```
**Health Check:**
```bash
curl http://localhost:5000/api/health
# Response:
# {
# "status": "healthy",
# "online_clients": ["device_windows_001", ...]
# }
```
### JavaScript (fetch)
**Browser/Node.js Integration:**
```
/* Detailed source-code truncated for AI context efficiency. */
```
---
## ⚠️ Error Handling
### Standard Error Format
All API errors follow FastAPI's standard format:
```json
{
"detail": "Error message description"
}
```
### HTTP Status Codes
| Code | Meaning | When It Occurs | How to Handle |
|------|---------|----------------|---------------|
| **200** | OK | Request succeeded | Process response data |
| **400** | Bad Request | Missing/empty `client_id` or `request` | Check request parameters |
| **404** | Not Found | Client not online | Check `/api/clients` first |
| **422** | Unprocessable Entity | Invalid JSON schema | Validate request body |
| **500** | Internal Server Error | Unexpected server error | Retry or contact admin |
### Error Handling Patterns
**Robust Error Handling:**
```
/* Detailed source-code truncated for AI context efficiency. */
```
---
## 💡 Best Practices
### 1. Validate Client Availability
Always verify the target client is online before dispatching tasks.
```python
def is_client_online(client_id: str) -> bool:
"""Check if a client is currently connected."""
response = requests.get("http://localhost:5000/api/clients")
clients = response.json()["online_clients"]
return client_id in clients
# Usage
if is_client_online("device_windows_001"):
# Dispatch task
pass
else:
print("Device is offline")
```
### 2. Implement Exponential Backoff
Use exponential backoff to reduce server load when polling for results.
```python
import time
def poll_with_backoff(task_name: str, max_wait: int = 300):
"""Poll for result with exponential backoff."""
interval = 1 # Start with 1 second
max_interval = 30 # Cap at 30 seconds
waited = 0
while waited < max_wait:
response = requests.get(
f"http://localhost:5000/api/task_result/{task_name}"
)
data = response.json()
if data["status"] == "done":
return data["result"]
time.sleep(interval)
waited += interval
# Exponential backoff: 1s 2s 4s 8s 16s 30s (capped)
interval = min(interval * 2, max_interval)
raise TimeoutError(f"Task did not complete in {max_wait}s")
```
### 3. Use Health Checks for Monitoring
Integrate health checks into your monitoring infrastructure.
```python
import requests
import logging
def check_server_health() -> bool:
"""
Check server health for monitoring.
Returns:
True if healthy, False otherwise
"""
try:
response = requests.get(
"http://localhost:5000/api/health",
timeout=5
)
if response.status_code == 200:
data = response.json()
logging.info(
f"Server healthy - {len(data.get('online_clients', []))} clients"
)
return True
else:
logging.warning(f"Server returned {response.status_code}")
return False
except requests.RequestException as e:
logging.error(f"Health check failed: {e}")
return False
```
### 4. Handle Timeouts Gracefully
Set appropriate timeouts - different tasks have different execution times.
```python
def dispatch_with_timeout(
client_id: str,
request: str,
task_name: str,
result_timeout: int = 60
):
"""Dispatch task and wait for result with custom timeout."""
# Dispatch (short timeout for HTTP request)
dispatch_response = requests.post(
"http://localhost:5000/api/dispatch",
json={"client_id": client_id, "request": request, "task_name": task_name},
timeout=10 # 10 seconds for dispatch
)
task_name = dispatch_response.json()["task_name"]
# Wait for result (longer timeout for task execution)
start_time = time.time()
while time.time() - start_time < result_timeout:
result_response = requests.get(
f"http://localhost:5000/api/task_result/{task_name}",
timeout=5 # 5 seconds per poll
)
data = result_response.json()
if data["status"] == "done":
return data["result"]
time.sleep(2)
raise TimeoutError(
f"Task '{task_name}' did not complete within {result_timeout}s"
)
```
### 5. Log All API Interactions
**Production Logging:**
```python
import logging
import requests
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s'
)
def dispatch_with_logging(client_id: str, request: str, task_name: str):
"""Dispatch task with detailed logging."""
logging.info(
f"Dispatching task: client_id={client_id}, "
f"task_name={task_name}, request='{request}'"
)
try:
response = requests.post(
"http://localhost:5000/api/dispatch",
json={
"client_id": client_id,
"request": request,
"task_name": task_name
}
)
response.raise_for_status()
data = response.json()
logging.info(
f"Task dispatched successfully: session_id={data['session_id']}"
)
return data
except requests.HTTPError as e:
logging.error(
f"Dispatch failed: {e.response.status_code} - "
f"{e.response.json().get('detail')}"
)
raise
except Exception as e:
logging.error(f"Unexpected error during dispatch: {e}")
raise
```
### 6. Cache Client List
Reduce API calls by caching the client list if you're dispatching multiple tasks.
```python
from datetime import datetime, timedelta
class ClientCache:
def __init__(self, ttl_seconds=10):
self.ttl = timedelta(seconds=ttl_seconds)
self.cache = None
self.last_fetch = None
def get_clients(self):
"""Get clients with caching."""
now = datetime.now()
# Return cache if still valid
if self.cache and self.last_fetch and (now - self.last_fetch) < self.ttl:
return self.cache
# Fetch new data
response = requests.get("http://localhost:5000/api/clients")
self.cache = response.json()["online_clients"]
self.last_fetch = now
return self.cache
# Usage
cache = ClientCache(ttl_seconds=30)
for task in tasks:
clients = cache.get_clients() # Uses cache if within TTL
if task["client_id"] in clients:
dispatch_task(task)
```
---
## 🔗 Integration Points
### API Router Architecture
```mermaid
graph TB
subgraph "HTTP API Layer"
Router[FastAPI Router]
Dispatch[POST /api/dispatch]
Clients[GET /api/clients]
Result[GET /api/task_result]
Health[GET /api/health]
end
subgraph "Service Layer"
WSM[Client Connection Manager]
SM[Session Manager]
end
subgraph "Protocol Layer"
AIP[AIP TaskExecutionProtocol]
WS[WebSocket Transport]
end
Router --> Dispatch
Router --> Clients
Router --> Result
Router --> Health
Dispatch --> WSM
Dispatch --> AIP
Clients --> WSM
Result --> SM
Health --> WSM
AIP --> WS
WSM --> WS
style Dispatch fill:#bbdefb
style Clients fill:#c8e6c9
style Result fill:#fff9c4
style Health fill:#ffcdd2
```
### With Client Connection Manager
**API ClientConnectionManager:**
- `get_client(client_id)`: Get WebSocket connection for task dispatch
- `list_clients()`: List all online clients
**Example:**
```python
# In POST /api/dispatch
ws = client_manager.get_client(client_id)
if not ws:
raise HTTPException(status_code=404, detail="Client not online")
# In GET /api/clients
clients = client_manager.list_clients()
return {"online_clients": clients}
```
### With Session Manager
**API SessionManager:**
- `get_result_by_task(task_name)`: Retrieve task result by task name
**Example:**
```python
# In GET /api/task_result/{task_name}
result = session_manager.get_result_by_task(task_name)
if not result:
return {"status": "pending"}
return {"status": "done", "result": result}
```
### With AIP Protocol
**API uses AIP for task dispatch:**
```python
# Create AIP protocol instance
transport = WebSocketTransport(ws)
task_protocol = TaskExecutionProtocol(transport)
# Send task via AIP
await task_protocol.send_task_assignment(
user_request=user_request,
task_name=task_name,
session_id=session_id,
response_id=response_id,
)
```
---
## 📚 Complete API Reference
### Endpoints Summary
| Method | Endpoint | Description | Auth Required |
|--------|----------|-------------|---------------|
| `POST` | `/api/dispatch` | Dispatch task to client | No |
| `GET` | `/api/clients` | List online clients | No |
| `GET` | `/api/task_result/{task_name}` | Get task result | No |
| `GET` | `/api/health` | Health check | No |
**Note on Authentication:**
The current API implementation does **not** include authentication. For production deployments, consider adding:
- API keys
- OAuth2/JWT tokens
- Rate limiting
- IP whitelisting
### Request/Response Models
#### Dispatch Request
```python
{
"client_id": str, # Required
"request": str, # Required
"task_name": str # Optional (auto-generated if not provided)
}
```
#### Dispatch Response
```python
{
"status": "dispatched",
"task_name": str,
"client_id": str,
"session_id": str # UUID
}
```
#### Clients Response
```python
{
"online_clients": List[str]
}
```
#### Task Result Response
```python
# Pending
{
"status": "pending"
}
# Complete
{
"status": "done",
"result": Dict[str, Any] # Structure depends on task type
}
```
#### Health Response
```python
{
"status": "healthy",
"online_clients": List[str]
}
```
---
## 🎓 Summary
The HTTP API provides a **stateless, RESTful interface** for external systems to interact with the UFO server without maintaining WebSocket connections.
**Key Characteristics:**
| Aspect | Details |
|--------|---------|
| **Protocol** | HTTP/1.1, REST, JSON |
| **Port** | 5000 (default, configurable) |
| **Authentication** | None (add for production) |
| **State** | Stateless (uses Client Connection Manager for client state) |
| **Task Dispatch** | Via AIP TaskExecutionProtocol |
| **Result Retrieval** | Polling-based (no push notifications) |
**Use Cases:**
1. **Web Applications**: Trigger UFO tasks from web frontends
2. **Automation Scripts**: Integrate UFO into CI/CD pipelines
3. **External Tools**: Connect third-party systems to UFO
4. **Monitoring**: Health checks for infrastructure monitoring
**Architecture Position:**
```mermaid
graph TD
subgraph "External World"
E1[Web App]
E2[Python Script]
E3[Automation Tool]
end
subgraph "UFO Server"
API[HTTP API]
WSM[Client Connection Manager]
SM[Session Manager]
WH[WebSocket Handler]
end
subgraph "Clients"
D1[Device 1]
D2[Device 2]
C1[Constellation]
end
E1 -->|HTTP POST/GET| API
E2 -->|HTTP POST/GET| API
E3 -->|HTTP POST/GET| API
API --> WSM
API --> SM
WSM --> WH
SM --> WH
WH <-->|WebSocket| D1
WH <-->|WebSocket| D2
WH <-->|WebSocket| C1
style API fill:#bbdefb
style WSM fill:#c8e6c9
style SM fill:#fff9c4
```
**For More Information:**
- [Server Overview](./overview.md) - UFO server architecture and components
- [Client Connection Manager](./client_connection_manager.md) - Client registry and connection management
- [Session Manager](./session_manager.md) - Task execution and result tracking
- [Quick Start](./quick_start.md) - Get started with UFO server
---
### Documents Docs Server Client Connection Manager (`documents/docs/server/client_connection_manager.md`)
# Client Connection Manager
The **ClientConnectionManager** is the central registry for all connected clients, maintaining connection state, session mappings, device information, and providing efficient lookup mechanisms for client routing and management.
For more context on how this component fits into the server architecture, see the [Server Overview](overview.md).
---
## 🎯 Overview
The Client Connection Manager serves as the "address book" and "session tracker" for the entire server:
| Responsibility | Description | Benefit |
|----------------|-------------|---------|
| **Client Registry** | Store all connected device and constellation clients | Fast O(1) client lookup by ID |
| **Session Tracking** | Map sessions to their constellation orchestrators | Enable proper cleanup on disconnection |
| **Device Mapping** | Track which device is executing which session | Route task results correctly |
| **Connection State** | Monitor which clients are online | Validate before dispatching tasks |
| **System Info Caching** | Store device capabilities and configuration | Optimize constellation decision-making |
| **Statistics** | Provide connection metrics | Monitoring and capacity planning |
### Architecture Position
```mermaid
graph TB
subgraph "Clients"
D1[Device 1]
D2[Device 2]
C1[Constellation 1]
end
subgraph "Server - ClientConnectionManager"
WSM[Client Connection Manager]
subgraph "Storage"
CR[Client Registry online_clients]
CS[Constellation Sessions _constellation_sessions]
DS[Device Sessions _device_sessions]
SI[System Info Cache system_info]
end
end
subgraph "Server Components"
WH[WebSocket Handler]
SM[Session Manager]
API[API Router]
end
D1 -->|"add_client()"| WSM
D2 -->|"add_client()"| WSM
C1 -->|"add_client()"| WSM
WSM --> CR
WSM --> CS
WSM --> DS
WSM --> SI
WH -->|"get_client()"| WSM
WH -->|"is_device_connected()"| WSM
SM -->|"get_device_sessions()"| WSM
API -->|"list_clients()"| WSM
style WSM fill:#ffecb3
style CR fill:#c8e6c9
style CS fill:#bbdefb
style DS fill:#f8bbd0
```
---
## 📦 Core Data Structures
### ClientInfo Dataclass
Each connected client is represented by a `ClientInfo` dataclass that stores all relevant connection details:
```python
@dataclass
class ClientInfo:
"""Information about a connected client."""
websocket: WebSocket # Active WebSocket connection
client_type: ClientType # DEVICE or CONSTELLATION
connected_at: datetime # Connection timestamp
metadata: Dict = None # Additional client metadata
platform: str = "windows" # OS platform (windows/linux)
system_info: Dict = None # Device system information (for devices only)
# AIP protocol instances for this client
transport: Optional[WebSocketTransport] = None # AIP WebSocket transport
task_protocol: Optional[TaskExecutionProtocol] = None # AIP task protocol
```
**Field Descriptions:**
| Field | Type | Purpose | Example |
|-------|------|---------|---------|
| `websocket` | `WebSocket` | FastAPI WebSocket connection object | `` |
| `client_type` | `ClientType` | Whether DEVICE or CONSTELLATION | `ClientType.DEVICE` |
| `connected_at` | `datetime` | When client registered | `2024-11-04 14:30:22` |
| `metadata` | `Dict` | Custom metadata from registration message | `{"hostname": "WIN-001"}` |
| `platform` | `str` | Operating system | `"windows"`, `"linux"` |
| `system_info` | `Dict` | Device capabilities and system specs | See System Info Structure below |
| `transport` | `Optional[WebSocketTransport]` | AIP WebSocket transport layer | `` |
| `task_protocol` | `Optional[TaskExecutionProtocol]` | AIP task execution protocol handler | `` |
**System Info Structure Example:**
```json
{
"os": "Windows",
"os_version": "11 Pro 22H2",
"processor": "Intel Core i7-1185G7",
"memory_total": 17014632448,
"memory_available": 8459743232,
"screen_resolution": "1920x1080",
"installed_applications": ["Chrome", "Excel", "Notepad++"],
"supported_features": ["ui_automation", "web_browsing", "file_ops"],
"custom_metadata": {
"tags": ["production", "office"],
"tier": "high_performance"
}
}
```
---
## 👥 Client Registry Management
The client registry (`online_clients`) is the authoritative source of truth for all connected clients.
### Adding Clients
```python
def add_client(
self,
client_id: str,
platform: str,
ws: WebSocket,
client_type: ClientType = ClientType.DEVICE,
metadata: Dict = None,
transport: Optional[WebSocketTransport] = None,
task_protocol: Optional[TaskExecutionProtocol] = None
):
"""Register a new client connection."""
with self.lock: # Thread-safe access
# Extract system info if provided (device clients only)
system_info = None
if metadata and "system_info" in metadata and client_type == ClientType.DEVICE:
system_info = metadata.get("system_info")
# Merge with server-configured metadata if available
server_config = self._device_configs.get(client_id, {})
if server_config:
system_info = self._merge_device_info(system_info, server_config)
logger.info(f"Merged server config for device {client_id}")
# Create ClientInfo and add to registry
self.online_clients[client_id] = ClientInfo(
websocket=ws,
platform=platform,
client_type=client_type,
connected_at=datetime.now(),
metadata=metadata or {},
system_info=system_info,
transport=transport,
task_protocol=task_protocol
)
```
**Example - Adding a Device Client:**
```python
client_manager.add_client(
client_id="device_windows_001",
platform="windows",
ws=websocket,
client_type=ClientType.DEVICE,
metadata={
"hostname": "WIN-OFFICE-01",
"system_info": {
"os": "Windows",
"screen_resolution": "1920x1080",
"installed_applications": ["Chrome", "Excel"]
}
},
transport=websocket_transport,
task_protocol=task_execution_protocol
)
```
**Example - Adding a Constellation Client:**
```python
client_manager.add_client(
client_id="constellation_orchestrator_001",
platform="linux", # Platform of the constellation server
ws=websocket,
client_type=ClientType.CONSTELLATION,
metadata={
"orchestrator_version": "2.0.0",
"max_concurrent_tasks": 10
},
transport=websocket_transport,
task_protocol=task_execution_protocol
)
```
**Thread Safety:**
```python
with self.lock: # threading.Lock ensures atomic operations
self.online_clients[client_id] = client_info
```
!!! warning "Client ID Uniqueness"
If a client reconnects with the same `client_id`, the new connection **overwrites** the old entry. This effectively disconnects the old WebSocket. Use unique IDs to prevent collisions.
### Retrieving Clients
The ClientConnectionManager provides several methods to lookup clients based on different criteria:
**Get WebSocket Connection:**
```python
def get_client(self, client_id: str) -> WebSocket | None:
"""Get WebSocket connection for a client."""
with self.lock:
client_info = self.online_clients.get(client_id)
return client_info.websocket if client_info else None
```
**Usage:**
```python
target_ws = client_manager.get_client("device_windows_001")
if target_ws:
await target_ws.send_text(message)
```
**Get Full Client Info:**
```python
def get_client_info(self, client_id: str) -> ClientInfo | None:
"""Get complete information about a client."""
with self.lock:
return self.online_clients.get(client_id)
```
**Usage:**
```python
client_info = client_manager.get_client_info("device_windows_001")
if client_info:
print(f"Platform: {client_info.platform}")
print(f"Connected at: {client_info.connected_at}")
print(f"Type: {client_info.client_type}")
```
**Get Client Type:**
```python
def get_client_type(self, client_id: str) -> ClientType | None:
"""Get the type of a client."""
with self.lock:
client_info = self.online_clients.get(client_id)
return client_info.client_type if client_info else None
```
**Usage:**
```python
client_type = client_manager.get_client_type("client_001")
if client_type == ClientType.DEVICE:
# Handle device-specific logic
elif client_type == ClientType.CONSTELLATION:
# Handle constellation-specific logic
```
**List All Clients:**
```python
def list_clients(self) -> List[str]:
"""List all online client IDs."""
with self.lock:
return list(self.online_clients.keys())
```
**Usage:**
```python
online_ids = client_manager.list_clients()
print(f"Currently online: {len(online_ids)} clients")
```
**List by Type:**
```python
def list_clients_by_type(self, client_type: ClientType) -> List[str]:
"""List all online clients of a specific type."""
with self.lock:
return [
client_id
for client_id, client_info in self.online_clients.items()
if client_info.client_type == client_type
]
```
**Usage:**
```python
devices = client_manager.list_clients_by_type(ClientType.DEVICE)
constellations = client_manager.list_clients_by_type(ClientType.CONSTELLATION)
print(f"Devices online: {len(devices)}")
print(f"Constellations online: {len(constellations)}")
```
### Removing Clients
```python
def remove_client(self, client_id: str):
"""Remove a client from the registry."""
with self.lock:
self.online_clients.pop(client_id, None)
logger.info(f"[ClientConnectionManager] Removed client: {client_id}")
```
!!!danger "Cleanup Required"
When removing a client, you should **also** clean up:
- Session mappings (`_constellation_sessions`, `_device_sessions`)
- Cached system info (automatically removed via ClientInfo deletion)
- Active sessions (via SessionManager.cancel_task())
See client disconnect cleanup pattern below.
```
---
## 🔍 Connection State Checking
Always check if the target device is connected before attempting to dispatch tasks. This prevents errors and improves user experience.
### Device Connection Validation
```python
def is_device_connected(self, device_id: str) -> bool:
"""Check if a device client is currently connected."""
with self.lock:
client_info = self.online_clients.get(device_id)
if not client_info:
return False
# Verify it's a DEVICE client (not constellation)
return client_info.client_type == ClientType.DEVICE
```
**Example - Validate Before Task Dispatch:**
```python
# In WebSocket Handler - constellation requesting task on device
target_device_id = data.target_id
if not client_manager.is_device_connected(target_device_id):
error_msg = f"Target device '{target_device_id}' is not connected"
await send_error(error_msg)
raise ValueError(error_msg)
# Safe to dispatch
target_ws = client_manager.get_client(target_device_id)
await dispatch_task(target_ws, task_request)
```
!!! warning "Type Check is Critical"
The method returns `False` if the client exists but is **not a device** (e.g., it's a constellation). This prevents accidentally dispatching device tasks to constellation clients.
### Generic Online Status Check
```python
# Not shown in source but implied
def is_online(self, client_id: str) -> bool:
"""Check if any client (device or constellation) is currently online."""
with self.lock:
return client_id in self.online_clients
```
**Comparison:**
| Method | Checks | Returns True When |
|--------|--------|-------------------|
| `is_device_connected(device_id)` | Client exists **AND** is DEVICE type | Device client is online |
| `is_online(client_id)` | Client exists (any type) | Any client is online |
---
## 📋 Session Mapping
The ClientConnectionManager tracks sessions from **two perspectives**:
1. **Constellation → Sessions**: Which sessions did a constellation initiate?
2. **Device → Sessions**: Which sessions is a device currently executing?
This dual tracking enables proper cleanup when either constellation or device disconnects.
```mermaid
graph TB
subgraph "Constellation Perspective"
C[Constellation_001]
CS[_constellation_sessions]
CS --> S1[session_abc]
CS --> S2[session_def]
CS --> S3[session_ghi]
end
subgraph "Device Perspective"
D[Device_windows_001]
DS[_device_sessions]
DS --> S1
DS --> S4[session_jkl]
end
subgraph "Disconnection Cleanup"
DC{Constellation Disconnects}
DD{Device Disconnects}
end
DC -->|Cancel| S1
DC -->|Cancel| S2
DC -->|Cancel| S3
DD -->|Cancel| S1
DD -->|Cancel| S4
style C fill:#bbdefb
style D fill:#c8e6c9
style S1 fill:#ffcdd2
```
### Constellation Session Mapping
Constellation clients initiate tasks on remote devices. Track these sessions to enable cleanup when the orchestrator disconnects.
**Add Constellation Session:**
```python
def add_constellation_session(self, client_id: str, session_id: str):
"""Map a session to its constellation orchestrator."""
with self.lock:
if client_id not in self._constellation_sessions:
self._constellation_sessions[client_id] = []
self._constellation_sessions[client_id].append(session_id)
```
**Get Constellation Sessions:**
```python
def get_constellation_sessions(self, client_id: str) -> List[str]:
"""Get all sessions initiated by a constellation client."""
with self.lock:
return self._constellation_sessions.get(client_id, []).copy()
# .copy() prevents external modification of internal list
```
**Remove Constellation Sessions:**
```python
def remove_constellation_sessions(self, client_id: str) -> List[str]:
"""Remove and return all sessions for a constellation."""
with self.lock:
return self._constellation_sessions.pop(client_id, [])
# Returns removed sessions for cleanup
```
**Example - Constellation Disconnect Cleanup:**
```python
# In WebSocket Handler - when constellation disconnects
constellation_id = "constellation_001"
# Get all sessions this constellation initiated
session_ids = client_manager.get_constellation_sessions(constellation_id)
logger.info(
f"Constellation {constellation_id} disconnected, "
f"cancelling {len(session_ids)} sessions"
)
# Cancel each session
for session_id in session_ids:
await session_manager.cancel_task(
session_id,
reason="constellation_disconnected" # Don't send callback
)
# Remove mappings
client_manager.remove_constellation_sessions(constellation_id)
```
### Device Session Mapping
Device clients execute tasks sent by constellations (or themselves). Track these sessions to enable cleanup when the device disconnects.
**Add Device Session:**
```python
def add_device_session(self, device_id: str, session_id: str):
"""Map a session to the device executing it."""
with self.lock:
if device_id not in self._device_sessions:
self._device_sessions[device_id] = []
self._device_sessions[device_id].append(session_id)
```
**Get Device Sessions:**
```python
def get_device_sessions(self, device_id: str) -> List[str]:
"""Get all sessions running on a specific device."""
with self.lock:
return self._device_sessions.get(device_id, []).copy()
```
**Remove Device Sessions:**
```python
def remove_device_sessions(self, device_id: str) -> List[str]:
"""Remove and return all sessions for a device."""
with self.lock:
return self._device_sessions.pop(device_id, [])
```
!!!example "Device Disconnect Cleanup"
```python
# In WebSocket Handler - when device disconnects
device_id = "device_windows_001"
# Get all sessions running on this device
session_ids = client_manager.get_device_sessions(device_id)
logger.info(
f"Device {device_id} disconnected, "
f"cancelling {len(session_ids)} sessions"
)
# Cancel each session
for session_id in session_ids:
await session_manager.cancel_task(
session_id,
reason="device_disconnected" # Send callback to constellation
)
# Remove mappings
client_manager.remove_device_sessions(device_id)
```
### Session Mapping Lifecycle
```mermaid
sequenceDiagram
participant C as Constellation
participant WH as WebSocket Handler
participant WSM as ClientConnectionManager
participant D as Device
Note over C,D: Task Dispatch
C->>WH: TASK request (target_id=device_001)
WH->>WH: Generate session_id="session_abc"
Note over WH,WSM: Map Session to Both Clients
WH->>WSM: add_constellation_session("constellation_001", "session_abc")
WH->>WSM: add_device_session("device_001", "session_abc")
Note over WSM: Session Mappings
WSM->>WSM: _constellation_sessions["constellation_001"] = ["session_abc"]
WSM->>WSM: _device_sessions["device_001"] = ["session_abc"]
Note over WH,D: Task Execution
WH->>D: TASK_ASSIGNMENT (session_abc)
D->>D: Execute task
Note over D,WH: Result Delivery
D->>WH: TASK_END (session_abc)
WH->>C: TASK_END (session_abc)
Note over WH,WSM: Cleanup (not shown in actual code)
Note right of WH: Sessions remain in mappings until client disconnects!
```
!!!warning "Sessions Persist Until Cleanup"
Session mappings are **not automatically removed** when tasks complete. They persist until:
1. The constellation disconnects (removes all its sessions)
2. The device disconnects (removes all its sessions)
3. Manual cleanup (future feature)
**Implication:** Over time, `_constellation_sessions` and `_device_sessions` can grow large. Consider implementing periodic cleanup for completed sessions.
### Dual Mapping Example
!!!example "Single Session, Dual Mapping"
When a constellation dispatches a task to a device:
```python
constellation_id = "constellation_orchestrator_001"
device_id = "device_windows_001"
session_id = "session_abc123"
# Session is mapped to BOTH the constellation and the device
client_manager.add_constellation_session(constellation_id, session_id)
client_manager.add_device_session(device_id, session_id)
# Later retrieval
constellation_sessions = client_manager.get_constellation_sessions(constellation_id)
# Returns: ["session_abc123", ...]
device_sessions = client_manager.get_device_sessions(device_id)
# Returns: ["session_abc123", ...]
```
**Why dual mapping?**
- If **constellation disconnects**: Cancel all its sessions (notify devices)
- If **device disconnects**: Cancel all sessions on that device (notify constellations)
---
## 💻 System Information Management
The ClientConnectionManager caches device system information to enable intelligent task routing by constellations without repeatedly querying devices.
### System Info Storage
**Stored Automatically During Registration:**
```python
def add_client(self, client_id, platform, ws, client_type, metadata):
"""Add client and extract system info if provided."""
system_info = None
if metadata and "system_info" in metadata and client_type == ClientType.DEVICE:
system_info = metadata.get("system_info")
# Merge with server configuration if available
server_config = self._device_configs.get(client_id, {})
if server_config:
system_info = self._merge_device_info(system_info, server_config)
self.online_clients[client_id] = ClientInfo(
websocket=ws,
platform=platform,
client_type=client_type,
system_info=system_info, # Cached here
...
)
```
### Retrieving System Information
**Get Single Device Info:**
```python
def get_device_system_info(self, device_id: str) -> Optional[Dict[str, Any]]:
"""Get device system information by device ID."""
with self.lock:
client_info = self.online_clients.get(device_id)
if client_info and client_info.client_type == ClientType.DEVICE:
return client_info.system_info
return None
```
**Usage:**
```python
device_info = client_manager.get_device_system_info("device_windows_001")
if device_info:
screen_res = device_info.get("screen_resolution")
apps = device_info.get("installed_applications", [])
print(f"Screen: {screen_res}")
print(f"Apps: {len(apps)} installed")
```
**Get All Devices Info:**
```python
def get_all_devices_info(self) -> Dict[str, Dict[str, Any]]:
"""Get system information for all connected devices."""
with self.lock:
return {
device_id: client_info.system_info
for device_id, client_info in self.online_clients.items()
if client_info.client_type == ClientType.DEVICE
and client_info.system_info
}
```
**Usage:**
```python
all_devices = client_manager.get_all_devices_info()
for device_id, info in all_devices.items():
print(f"{device_id}: {info.get('os')} - {info.get('screen_resolution')}")
# Example output:
# device_windows_001: Windows - 1920x1080
# device_linux_001: Linux - 2560x1440
```
### Server Configuration Merging
The ClientConnectionManager supports loading device-specific configuration from YAML/JSON files and **merging** them with auto-detected system info.
**Device Configuration File (`device_config.yaml`):**
```yaml
devices:
device_windows_001:
tags: ["production", "office", "high_priority"]
tier: "high_performance"
additional_features: ["excel_automation", "pdf_generation"]
max_concurrent_tasks: 5
device_linux_001:
tags: ["development", "testing"]
tier: "standard"
additional_features: ["docker_support"]
```
**Loading Configuration:**
```python
# Initialize ClientConnectionManager with config file
client_manager = ClientConnectionManager(device_config_path="config/device_config.yaml")
# Configuration is automatically loaded during __init__
```
**Merge Process:**
```python
def _merge_device_info(
self,
system_info: Dict[str, Any],
server_config: Dict[str, Any]
) -> Dict[str, Any]:
"""Merge auto-detected system info with server configuration."""
merged = {**system_info} # Start with auto-detected info
# Add all server config to custom_metadata
if "custom_metadata" not in merged:
merged["custom_metadata"] = {}
merged["custom_metadata"].update(server_config)
# Special handling: merge capabilities
if "supported_features" in system_info and "additional_features" in server_config:
merged["supported_features"] = list(
set(system_info["supported_features"] + server_config["additional_features"])
)
# Add server tags
if "tags" in server_config:
merged["tags"] = server_config["tags"]
return merged
```
**Result:**
```json
{
"os": "Windows",
"screen_resolution": "1920x1080",
"supported_features": [
"ui_automation",
"web_browsing",
"file_ops",
"excel_automation",
"pdf_generation"
],
"tags": ["production", "office", "high_priority"],
"custom_metadata": {
"tier": "high_performance",
"max_concurrent_tasks": 5,
"tags": ["production", "office", "high_priority"],
"additional_features": ["excel_automation", "pdf_generation"]
}
}
```
**Why Merge Configuration?**
- **Auto-detected info**: Always accurate (OS, memory, screen resolution)
- **Server config**: Administrative metadata (tags, tier, priorities)
- **Combined**: Rich device profile for intelligent task routing
---
## 📊 Client Statistics and Monitoring
The `get_stats()` method provides basic metrics for monitoring connected clients.
### Get Statistics
```python
def get_stats(self) -> Dict[str, int]:
"""Get statistics about connected clients."""
with self.lock:
device_count = sum(
1
for info in self.online_clients.values()
if info.client_type == ClientType.DEVICE
)
constellation_count = sum(
1
for info in self.online_clients.values()
if info.client_type == ClientType.CONSTELLATION
)
return {
"total": len(self.online_clients),
"device_clients": device_count,
"constellation_clients": constellation_count
}
```
**Example Usage:**
```python
# Get current statistics
stats = client_manager.get_stats()
print(f"📊 Server Statistics:")
print(f" Total Clients: {stats['total']}")
print(f" Devices: {stats['device_clients']}")
print(f" Constellations: {stats['constellation_clients']}")
# Output:
# 📊 Server Statistics:
# Total Clients: 5
# Devices: 3
# Constellations: 2
```
### Filtering and Querying
**Filter by Platform:**
```python
def get_devices_by_platform(self, platform: str) -> List[str]:
"""Get all device IDs for a specific platform."""
with self.lock:
return [
device_id
for device_id, client_info in self.online_clients.items()
if client_info.client_type == ClientType.DEVICE
and client_info.platform == platform
]
# Usage
windows_devices = client_manager.get_devices_by_platform("Windows")
linux_devices = client_manager.get_devices_by_platform("Linux")
```
**Filter by Connection Time:**
```python
from datetime import datetime, timedelta
def get_recently_connected(self, minutes: int = 5) -> List[str]:
"""Get clients connected in the last N minutes."""
cutoff_time = datetime.now() - timedelta(minutes=minutes)
with self.lock:
return [
client_id
for client_id, client_info in self.online_clients.items()
if client_info.connected_at >= cutoff_time
]
# Usage
recent_clients = client_manager.get_recently_connected(minutes=10)
```
**Filter by Capability:**
```python
def find_devices_with_capability(self, capability: str) -> List[str]:
"""Find devices that support a specific capability."""
with self.lock:
matches = []
for device_id, client_info in self.online_clients.items():
if client_info.client_type != ClientType.DEVICE:
continue
if not client_info.system_info:
continue
features = client_info.system_info.get("supported_features", [])
if capability in features:
matches.append(device_id)
return matches
# Usage
excel_devices = client_manager.find_devices_with_capability("excel_automation")
docker_devices = client_manager.find_devices_with_capability("docker_support")
```
---
## 🎯 Usage Patterns
### Safe Task Dispatch
```python
async def dispatch_task_to_device(
client_manager: ClientConnectionManager,
constellation_id: str,
target_device_id: str,
task_request: dict,
session_id: str
):
"""Dispatch task with comprehensive validation."""
# Step 1: Validate constellation is connected
if not client_manager.is_online(constellation_id):
raise ValueError(f"Constellation {constellation_id} not connected")
# Step 2: Validate target device is connected
if not client_manager.is_device_connected(target_device_id):
raise ValueError(f"Device {target_device_id} not connected")
# Step 3: Get device WebSocket
device_ws = client_manager.get_client(target_device_id)
if not device_ws:
raise ValueError(f"Could not get WebSocket for device {target_device_id}")
# Step 4: Track session mappings
client_manager.add_constellation_session(constellation_id, session_id)
client_manager.add_device_session(target_device_id, session_id)
# Step 5: Send task
await device_ws.send_json({
"type": "TASK_ASSIGNMENT",
"session_id": session_id,
"request": task_request
})
logger.info(
f"Task {session_id} dispatched: "
f"{constellation_id} → {target_device_id}"
)
```
### Graceful Client Disconnect Handling
```python
async def handle_client_disconnect(
client_manager: ClientConnectionManager,
session_manager: SessionManager,
client_id: str,
client_type: ClientType
):
"""Handle client disconnect with full cleanup."""
logger.info(f"Client disconnected: {client_id} ({client_type})")
# Step 1: Get all related sessions
if client_type == ClientType.CONSTELLATION:
session_ids = client_manager.get_constellation_sessions(client_id)
cancel_reason = "constellation_disconnected"
else: # DEVICE
session_ids = client_manager.get_device_sessions(client_id)
cancel_reason = "device_disconnected"
# Step 2: Cancel all sessions
for session_id in session_ids:
try:
await session_manager.cancel_task(session_id, reason=cancel_reason)
logger.info(f"Cancelled session {session_id}")
except Exception as e:
logger.error(f"Failed to cancel {session_id}: {e}")
# Step 3: Remove session mappings
if client_type == ClientType.CONSTELLATION:
client_manager.remove_constellation_sessions(client_id)
else:
client_manager.remove_device_sessions(client_id)
# Step 4: Remove client from registry
client_manager.remove_client(client_id)
logger.info(
f"Cleanup complete: {client_id}, "
f"cancelled {len(session_ids)} sessions"
)
```
### Intelligent Device Selection
```python
def select_optimal_device(
client_manager: ClientConnectionManager,
required_platform: str = None,
required_capabilities: List[str] = None,
preferred_tags: List[str] = None
) -> Optional[str]:
"""Select the best available device for a task."""
with client_manager.lock:
candidates = []
for device_id, client_info in client_manager.online_clients.items():
# Filter by type
if client_info.client_type != ClientType.DEVICE:
continue
# Filter by platform
if required_platform and client_info.platform != required_platform:
continue
# Filter by capabilities
if required_capabilities and client_info.system_info:
features = client_info.system_info.get("supported_features", [])
if not all(cap in features for cap in required_capabilities):
continue
# Calculate score based on preferred tags
score = 0
if preferred_tags and client_info.system_info:
tags = client_info.system_info.get("tags", [])
score = len(set(tags) & set(preferred_tags))
candidates.append((device_id, score))
if not candidates:
return None
# Return device with highest score (or first if all score 0)
candidates.sort(key=lambda x: x[1], reverse=True)
return candidates[0][0]
# Usage
device_id = select_optimal_device(
client_manager,
required_platform="Windows",
required_capabilities=["excel_automation"],
preferred_tags=["production", "high_priority"]
)
if device_id:
print(f"Selected device: {device_id}")
else:
print("No suitable device available")
```
### Session Cleanup After Task Completion
**Note:** Current implementation does **not automatically remove** session mappings when tasks complete. Consider implementing this pattern:
```python
async def handle_task_completion(
client_manager: ClientConnectionManager,
session_id: str,
constellation_id: str,
device_id: str
):
"""Clean up session mappings after task completes."""
# Task has completed (or failed)
# Option 1: Remove from both mappings
# (Requires adding remove_session method to ClientConnectionManager)
# client_manager.remove_session(session_id)
# Option 2: Leave mappings until disconnect
# (Current behavior - sessions accumulate)
logger.info(f"Task {session_id} completed, mappings retained")
```
---
## 💡 Best Practices
### Thread Safety
The ClientConnectionManager is accessed by multiple WebSocket handlers concurrently. **Always** acquire the lock before modifying shared state.
```python
# WRONG - No thread safety
def bad_example(self):
if "device_001" in self.online_clients:
client = self.online_clients["device_001"]
# Another thread might remove the client here!
return client.websocket
# CORRECT - Thread-safe
def good_example(self):
with self.lock:
if "device_001" in self.online_clients:
client = self.online_clients["device_001"]
return client.websocket
return None
```
### Validate Before Dispatch
Always check if the target device is connected before attempting to send messages.
```python
# CORRECT - Validation first
if client_manager.is_device_connected(target_device_id):
device_ws = client_manager.get_client(target_device_id)
await device_ws.send_json(task_data)
else:
logger.error(f"Device {target_device_id} not connected")
# Handle error appropriately
```
### Cleanup on Disconnect
When a client disconnects, clean up **all** related resources:
**Checklist:**
- [x] Cancel all related sessions
- [x] Remove session mappings (constellation/device)
- [x] Remove client from online registry
- [x] Remove device info cache (if applicable)
- [x] Notify affected parties
### Cache Device Information
Balance freshness and performance:
- **Cache during registration**: Fast lookups for task routing
- **Update on REQUEST_DEVICE_LIST**: Keep cache fresh
- **Don't cache sensitive data**: Only cache non-sensitive system info
```python
# During registration - cache system info
client_manager.add_client(
device_id,
platform="Windows",
ws=websocket,
client_type=ClientType.DEVICE,
metadata={"system_info": system_info} # Cached automatically
)
# Later - fast lookup without querying device
device_info = client_manager.get_device_system_info(device_id)
```
### Handle Edge Cases
**Case 1: Client re-connects with same ID**
```python
# Old connection still in registry
if client_manager.is_online(client_id):
logger.warning(f"Client {client_id} already connected, removing old connection")
client_manager.remove_client(client_id)
# Now add new connection
client_manager.add_client(client_id, platform, ws, client_type, metadata)
```
**Case 2: Session mapped to disconnected clients**
```python
# Before dispatching
if not client_manager.is_device_connected(device_id):
# Device disconnected, session mapping might still exist
# This is expected - cleanup happens on disconnect
raise ValueError(f"Device {device_id} no longer connected")
```
**Case 3: Constellation and device both disconnect**
```python
# Session will be cancelled twice (once for each disconnect)
# Ensure cancel_task is idempotent:
async def cancel_task(self, session_id, reason):
if session_id not in self.sessions:
logger.debug(f"Session {session_id} already cancelled")
return # Idempotent
# Proceed with cancellation
```
### Monitor Session Accumulation
**Note:** Session mappings are **not automatically removed** after task completion. Over time, this can cause memory growth.
**Mitigation strategies:**
**Periodic Cleanup:**
```python
async def cleanup_completed_sessions(client_manager, session_manager):
"""Remove mappings for completed sessions."""
all_sessions = set()
all_sessions.update(
session_id
for sessions in client_manager._constellation_sessions.values()
for session_id in sessions
)
all_sessions.update(
session_id
for sessions in client_manager._device_sessions.values()
for session_id in sessions
)
for session_id in all_sessions:
session = session_manager.get_session(session_id)
if session and session.state in [SessionState.COMPLETED, SessionState.FAILED]:
# Remove from ClientConnectionManager
# (Requires implementing remove_session method)
pass
```
**Cleanup on Completion:**
```python
# In task completion handler
async def on_task_complete(session_id, constellation_id, device_id):
# Remove specific session from mappings
client_manager.remove_session_from_constellation(constellation_id, session_id)
client_manager.remove_session_from_device(device_id, session_id)
```
---
## 🔗 Integration Points
### With WebSocket Handler
```mermaid
sequenceDiagram
participant WH as WebSocket Handler
participant WSM as ClientConnectionManager
participant SM as Session Manager
Note over WH,WSM: Client Registration
WH->>WSM: add_client(id, platform, ws, type, metadata)
WSM-->>WH: Client added
Note over WH,WSM: Task Dispatch
WH->>WSM: is_device_connected(device_id)
WSM-->>WH: True
WH->>WSM: get_client(device_id)
WSM-->>WH: WebSocket
WH->>WSM: add_constellation_session(const_id, session_id)
WH->>WSM: add_device_session(device_id, session_id)
Note over WH,SM: Task Execution
WH->>SM: execute_task_async(...)
Note over WH,WSM: Client Disconnect
WH->>WSM: get_constellation_sessions(client_id)
WSM-->>WH: [session_ids...]
WH->>SM: cancel_task(session_id, reason)
WH->>WSM: remove_constellation_sessions(client_id)
WH->>WSM: remove_client(client_id)
```
**ClientConnectionManager provides:**
- Client registration and lookup
- Connection state validation
- Session tracking for cleanup
**WebSocket Handler provides:**
- WebSocket lifecycle management
- Protocol message handling
- Disconnect notifications
### With Session Manager
**ClientConnectionManager provides:**
- Client connectivity status Session Manager checks before execution
- Session mappings Session Manager uses for cleanup
**Session Manager provides:**
- Session state ClientConnectionManager can query to cleanup completed sessions (future)
- Cancellation results ClientConnectionManager uses to notify clients
### With API Router
```python
# In HTTP API endpoints
from fastapi import APIRouter, HTTPException
@router.get("/devices")
async def list_devices():
"""List all connected devices."""
stats = client_manager.get_stats()
return {
"devices": stats["devices"]["ids"],
"count": stats["devices"]["count"],
"by_platform": stats["devices"]["platforms"]
}
@router.get("/devices/{device_id}")
async def get_device_info(device_id: str):
"""Get device system information."""
if not client_manager.is_device_connected(device_id):
raise HTTPException(status_code=404, detail="Device not connected")
system_info = client_manager.get_device_system_info(device_id)
return {"device_id": device_id, "system_info": system_info}
@router.get("/stats")
async def get_server_stats():
"""Get server statistics."""
return client_manager.get_stats()
```
/* Detailed source-code truncated for AI context efficiency. */
```python
@dataclass
class ClientInfo:
websocket: WebSocket # WebSocket connection
client_type: ClientType # DEVICE or CONSTELLATION
connected_at: datetime # Connection timestamp
metadata: Optional[Dict] # Registration metadata
platform: Optional[str] # "Windows", "Linux", "Darwin"
system_info: Optional[Dict] # Device capabilities and configuration
```
**ClientType:**
```python
class ClientType(Enum):
DEVICE = "device" # Execution client
CONSTELLATION = "constellation" # Orchestrator client
```
/* Detailed source-code truncated for AI context efficiency. */
```mermaid
graph TD
subgraph "ClientConnectionManager Core"
R[Client Registry]
S[Session Mapping]
D[Device Info Cache]
T[Thread Safety]
end
WH[WebSocket Handler] -->|Register/Lookup| R
WH -->|Track Sessions| S
WH -->|Cache System Info| D
SM[Session Manager] -->|Validate Connection| R
SM -->|Query Sessions| S
API[API Router] -->|List Devices| R
API -->|Get Stats| R
API -->|Device Info| D
R -.->|Thread Lock| T
S -.->|Thread Lock| T
D -.->|Thread Lock| T
style R fill:#bbdefb
style S fill:#c8e6c9
style D fill:#fff9c4
style T fill:#ffcdd2
```
**For More Information:**
- [Server Overview](./overview.md) - UFO server architecture and components
- [WebSocket Handler](./websocket_handler.md) - AIP protocol message handling
- [Session Manager](./session_manager.md) - Session lifecycle and background execution
- [Quick Start](./quick_start.md) - Get started with UFO server
---
### Documents Docs Server Monitoring (`documents/docs/server/monitoring.md`)
# Monitoring and Observability
Monitor the health, performance, and reliability of your UFO Server deployment with comprehensive observability tools, metrics, and alerting strategies.
!!! tip "Before You Begin"
Make sure you have the UFO Server running. See the [Quick Start Guide](./quick_start.md) for setup instructions.
## 🎯 Overview
```mermaid
graph TB
subgraph "Monitoring Layers"
Health[Health Checks]
Metrics[Performance Metrics]
Logs[Logs & Analysis]
Alerts[Alerting]
end
subgraph "UFO Server"
API[HTTP API]
WS[WebSocket]
end
subgraph "Tools"
K8s[Kubernetes]
Prom[Prometheus]
Slack[Notifications]
end
Health --> API
Metrics --> WS
Logs --> WS
Health --> K8s
Metrics --> Prom
Alerts --> Slack
style Health fill:#bbdefb
style Metrics fill:#c8e6c9
style Logs fill:#fff9c4
style Alerts fill:#ffcdd2
```
**Monitoring Capabilities:**
| Layer | Purpose | Tools |
|-------|---------|-------|
| **Health Checks** | Service availability and uptime | `/api/health`, Kubernetes probes |
| **Performance Metrics** | Latency, throughput, resource usage | Prometheus, custom dashboards |
| **Logging** | Event tracking, debugging, auditing | Python logging, log aggregation |
| **Alerting** | Proactive issue detection | Slack, Email, PagerDuty |
**Why Monitor?**
- **Detect Issues Early**: Catch problems before users notice
- **Performance Optimization**: Identify bottlenecks and inefficiencies
- **Capacity Planning**: Track growth and resource utilization
- **Debugging**: Trace errors and understand system behavior
- **SLA Compliance**: Ensure service level objectives are met
---
## 🏥 Health Checks
### HTTP Health Endpoint
The `/api/health` endpoint provides real-time server status without authentication. For detailed API specifications, see the [HTTP API Reference](./api.md).
#### Endpoint Details
```http
GET /api/health
```
**Response (200 OK):**
```json
{
"status": "healthy",
"online_clients": [
"device_windows_001",
"device_linux_002",
"constellation_orchestrator_001"
]
}
```
**Response Schema:**
| Field | Type | Description |
|-------|------|-------------|
| `status` | `string` | Always `"healthy"` if server is responding |
| `online_clients` | `array` | List of connected client IDs |
**Quick Test:**
```bash
# Test health endpoint
curl http://localhost:5000/api/health
# With jq for formatted output
curl -s http://localhost:5000/api/health | jq .
```
### Automated Health Monitoring
#### Kubernetes Liveness and Readiness Probes
Example production Kubernetes configuration:
```yaml
apiVersion: v1
kind: Pod
metadata:
name: ufo-server
labels:
app: ufo-server
spec:
containers:
- name: ufo-server
image: ufo-server:latest
ports:
- containerPort: 5000
name: http
protocol: TCP
# Liveness probe - restart container if failing
livenessProbe:
httpGet:
path: /api/health
port: 5000
scheme: HTTP
initialDelaySeconds: 30 # Wait 30s after startup
periodSeconds: 10 # Check every 10s
timeoutSeconds: 5 # 5s timeout per check
successThreshold: 1 # 1 success = healthy
failureThreshold: 3 # 3 failures = restart
# Readiness probe - remove from service if failing
readinessProbe:
httpGet:
path: /api/health
port: 5000
scheme: HTTP
initialDelaySeconds: 10 # Wait 10s after startup
periodSeconds: 5 # Check every 5s
timeoutSeconds: 3 # 3s timeout
successThreshold: 1 # 1 success = ready
failureThreshold: 2 # 2 failures = not ready
# Resource limits
resources:
requests:
memory: "256Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "500m"
```
**Probe Configuration Guide:**
| Parameter | Liveness | Readiness | Purpose |
|-----------|----------|-----------|---------|
| `initialDelaySeconds` | 30 | 10 | Time before first check (allow startup) |
| `periodSeconds` | 10 | 5 | How often to check |
| `timeoutSeconds` | 5 | 3 | Max time for response |
| `failureThreshold` | 3 | 2 | Failures before action (restart/unready) |
#### Uptime Monitoring Script
Example continuous health monitoring script:
```
/* Detailed source-code truncated for AI context efficiency. */
```
#### Docker Healthcheck
Docker Compose health configuration:
```yaml
version: '3.8'
services:
ufo-server:
image: ufo-server:latest
ports:
- "5000:5000"
# Docker health check
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:5000/api/health"]
interval: 30s
timeout: 5s
retries: 3
start_period: 40s
restart: unless-stopped
environment:
- LOG_LEVEL=INFO
volumes:
- ./logs:/app/logs
- ./config:/app/config
```
---
## 📊 Performance Metrics
### Request Latency Monitoring
Track API response times to detect performance degradation.
#### Latency Measurement
```
/* Detailed source-code truncated for AI context efficiency. */
```
**Sample Output:**
```
📊 Latency Report:
================================================================================
health:
Count: 100
Min: 2.34 ms
Max: 45.67 ms
Mean: 8.12 ms
Median: 6.89 ms
P95: 15.23 ms
P99: 32.45 ms
clients:
Count: 100
Min: 3.12 ms
Max: 52.34 ms
Mean: 10.45 ms
Median: 9.12 ms
P95: 18.90 ms
P99: 38.67 ms
```
### Task Throughput Monitoring
Track task completion rate to detect bottlenecks.
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Connection Stability Metrics
!!! warning "Monitor Client Connection Reliability"
Track disconnection rates to identify network or client issues. For more on client management, see the [Client Connection Manager](./client_connection_manager.md) documentation.
```
/* Detailed source-code truncated for AI context efficiency. */
```
---
## 📝 Logging and Analysis
### Log Configuration
Production logging setup:
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Log Event Categories
**Key Events to Log:**
**Connection Events:**
```python
# These log messages are generated by the WebSocket Handler
# See: WebSocket Handler documentation for connection lifecycle details
logger.info(f"[WS] ✅ Registered {client_type} client: {client_id}",
extra={"client_id": client_id, "client_type": client_type})
logger.info(f"[WS] 🔌 Client disconnected: {client_id}",
extra={"client_id": client_id})
```
**Task Events:**
```python
# These log messages are generated by the Session Manager
# See: Session Manager documentation for task lifecycle details
logger.info(f"[Session] Created session {session_id} for task: {task_name}",
extra={"session_id": session_id, "task_name": task_name})
logger.info(f"[Session] Task completed: {session_id}",
extra={"session_id": session_id, "duration_seconds": duration})
logger.warning(f"[Session] Task cancelled: {session_id} (reason: {reason})",
extra={"session_id": session_id, "cancel_reason": reason})
```
**Error Events:**
```python
logger.error(f"[WS] ❌ Failed to send result for session {session_id}: {error}",
extra={"session_id": session_id}, exc_info=True)
logger.error(f"[Session] Task execution failed: {session_id}",
extra={"session_id": session_id}, exc_info=True)
```
### Log Analysis Scripts
Parse and analyze JSON logs:
```
/* Detailed source-code truncated for AI context efficiency. */
```
---
## 🚨 Alerting Systems
### Alert Conditions
!!! danger "Critical Conditions to Monitor"
Track these critical conditions to maintain server reliability.
**1. No Connected Devices**
```python
def check_device_availability():
"""Alert if no devices are connected."""
response = requests.get("http://localhost:5000/api/clients")
clients = response.json()["online_clients"]
devices = [c for c in clients if c.startswith("device_")]
if len(devices) == 0:
send_alert(
severity="critical",
title="No Devices Connected",
message="UFO server has no connected devices. Task dispatch unavailable."
)
return False
elif len(devices) < 3:
send_alert(
severity="warning",
title="Low Device Count",
message=f"Only {len(devices)} devices connected (expected 3+)."
)
return True
```
**2. High Error Rate**
```python
def check_error_rate(log_file="logs/ufo_server.log", threshold=0.1):
"""Alert if error rate exceeds threshold."""
import json
total = 0
errors = 0
with open(log_file, 'r') as f:
for line in f:
try:
log = json.loads(line)
total += 1
if log.get("level") in ["ERROR", "CRITICAL"]:
errors += 1
except:
continue
error_rate = errors / total if total > 0 else 0
if error_rate > threshold:
send_alert(
severity="warning",
title=f"High Error Rate: {error_rate * 100:.1f}%",
message=f"{errors} errors out of {total} log entries"
)
return False
return True
```
**3. Slow Response Times**
```python
def check_latency(threshold_ms=1000):
"""Alert if health endpoint is slow."""
start = time.time()
try:
response = requests.get("http://localhost:5000/api/health", timeout=5)
latency_ms = (time.time() - start) * 1000
if latency_ms > threshold_ms:
send_alert(
severity="warning",
title=f"Slow Response Time: {latency_ms:.0f}ms",
message=f"/api/health responded in {latency_ms:.0f}ms (threshold: {threshold_ms}ms)"
)
return False
return True
except Exception as e:
send_alert(
severity="critical",
title="Health Check Failed",
message=f"Cannot reach health endpoint: {e}"
)
return False
```
**4. Session Failure Rate**
```python
def check_session_failure_rate(threshold=0.2):
"""Alert if too many sessions are failing."""
# Requires session tracking in logs
import json
completed = 0
failed = 0
with open("logs/ufo_server.log", 'r') as f:
for line in f:
try:
log = json.loads(line)
message = log.get("message", "")
if "Task completed" in message:
completed += 1
elif "Task failed" in message or "Task cancelled" in message:
failed += 1
except:
continue
total = completed + failed
failure_rate = failed / total if total > 0 else 0
if failure_rate > threshold:
send_alert(
severity="warning",
title=f"High Task Failure Rate: {failure_rate * 100:.1f}%",
message=f"{failed} failed out of {total} tasks"
)
return False
return True
```
### Alert Delivery Methods
**Email Alerts:**
```python
import smtplib
from email.message import EmailMessage
def send_email_alert(title, message, severity="info"):
"""Send email alert via SMTP."""
# Email configuration
smtp_host = "smtp.gmail.com"
smtp_port = 587
sender = "alerts@example.com"
receiver = "admin@example.com"
password = "your_app_password"
# Create message
msg = EmailMessage()
msg['Subject'] = f"[{severity.upper()}] UFO Server - {title}"
msg['From'] = sender
msg['To'] = receiver
# Email body
body = f"""
UFO Server Alert
Severity: {severity.upper()}
Title: {title}
Message:
{message}
Timestamp: {datetime.now().isoformat()}
--
UFO Server Monitoring System
"""
msg.set_content(body)
try:
with smtplib.SMTP(smtp_host, smtp_port) as server:
server.starttls()
server.login(sender, password)
server.send_message(msg)
logging.info(f"Email alert sent: {title}")
except Exception as e:
logging.error(f"Failed to send email alert: {e}")
```
**Slack Alerts:**
```python
import requests
def send_slack_alert(title, message, severity="info"):
"""Send alert to Slack via webhook."""
webhook_url = "https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
# Color coding by severity
colors = {
"critical": "#ff0000",
"error": "#ff6600",
"warning": "#ffcc00",
"info": "#00ccff"
}
# Slack message payload
payload = {
"attachments": [{
"color": colors.get(severity, "#cccccc"),
"title": f"🚨 UFO Server Alert - {title}",
"text": message,
"fields": [
{
"title": "Severity",
"value": severity.upper(),
"short": True
},
{
"title": "Timestamp",
"value": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
"short": True
}
],
"footer": "UFO Server Monitoring"
}]
}
try:
response = requests.post(webhook_url, json=payload, timeout=5)
response.raise_for_status()
logging.info(f"Slack alert sent: {title}")
except Exception as e:
logging.error(f"Failed to send Slack alert: {e}")
```
**PagerDuty Integration:**
```python
import requests
def send_pagerduty_alert(title, message, severity="error"):
"""Send alert to PagerDuty."""
routing_key = "YOUR_PAGERDUTY_ROUTING_KEY"
# Map severity to PagerDuty severity
pd_severity_map = {
"critical": "critical",
"error": "error",
"warning": "warning",
"info": "info"
}
payload = {
"routing_key": routing_key,
"event_action": "trigger",
"payload": {
"summary": title,
"source": "ufo-server",
"severity": pd_severity_map.get(severity, "error"),
"custom_details": {
"message": message,
"timestamp": datetime.now().isoformat()
}
}
}
try:
response = requests.post(
"https://events.pagerduty.com/v2/enqueue",
json=payload,
timeout=5
)
response.raise_for_status()
logging.info(f"PagerDuty alert sent: {title}")
except Exception as e:
logging.error(f"Failed to send PagerDuty alert: {e}")
```
### Unified Alert Function
```python
def send_alert(title: str, message: str, severity: str = "info",
channels: list = ["slack", "email"]):
"""
Send alert to multiple channels.
Args:
title: Alert title
message: Alert message
severity: One of "critical", "error", "warning", "info"
channels: List of channels to send to
"""
for channel in channels:
try:
if channel == "slack":
send_slack_alert(title, message, severity)
elif channel == "email":
send_email_alert(title, message, severity)
elif channel == "pagerduty":
send_pagerduty_alert(title, message, severity)
else:
logging.warning(f"Unknown alert channel: {channel}")
except Exception as e:
logging.error(f"Failed to send alert via {channel}: {e}")
# Usage
send_alert(
title="Server Healthy",
message="All systems operational",
severity="info",
channels=["slack"]
)
send_alert(
title="No Devices Connected",
message="Critical: UFO server has no connected devices",
severity="critical",
channels=["slack", "email", "pagerduty"]
)
```
---
## Best Practices
### 1. Monitoring Strategy
**Layered Monitoring Approach:**
| Layer | Purpose | Frequency |
|-------|---------|-----------|
| **Health Checks** | Service availability | Every 30-60 seconds |
| **Performance Metrics** | Response times, throughput | Continuous |
| **Error Logs** | Debugging and diagnostics | Real-time |
| **Alerts** | Critical issue notification | Event-driven |
### 2. Alert Thresholds
!!! warning "Avoid Alert Fatigue"
Set reasonable thresholds to prevent excessive alerting:
- **No devices for > 5 minutes**: Critical
- **Error rate > 10%**: Warning
- **Response time > 2 seconds**: Warning
- **Session failure rate > 20%**: Warning
- **3 consecutive health check failures**: Critical
### 3. Log Retention
**Log Retention Policy:**
| Log Type | Retention | Storage |
|----------|-----------|---------|
| **Detailed logs** | 7 days | Local SSD |
| **Summary logs** | 30 days | Local disk |
| **Monthly summaries** | 1 year | Archive storage |
| **Error logs** | 90 days | Separate file |
### 4. Performance Baselines
**Establish Baselines:**
Track normal operating metrics to detect anomalies:
```python
BASELINE_METRICS = {
"health_latency_ms": 10, # Typical: 5-15ms
"clients_latency_ms": 15, # Typical: 10-20ms
"active_clients": 5, # Expected: 3-10
"tasks_per_minute": 2, # Expected: 1-5
"error_rate": 0.02, # Expected: < 5%
}
# Alert if deviation > 50%
if actual_latency > BASELINE_METRICS["health_latency_ms"] * 1.5:
send_alert("Performance degradation detected")
```
### 5. Multi-Channel Alerting
!!!example "Route Alerts by Severity"
```python
ALERT_ROUTING = {
"critical": ["slack", "email", "pagerduty"],
"error": ["slack", "email"],
"warning": ["slack"],
"info": ["slack"]
}
def send_alert(title, message, severity="info"):
channels = ALERT_ROUTING.get(severity, ["slack"])
# Send to appropriate channels...
```
---
## 🎓 Summary
Production monitoring requires a **layered approach** combining health checks, performance metrics, structured logging, and proactive alerting.
**Monitoring Stack:**
```mermaid
graph LR
subgraph "Collect"
HC[Health Checks]
PM[Metrics]
LOG[Logs]
end
subgraph "Store & Analyze"
Files[Log Files]
Dash[Dashboard]
end
subgraph "Alert"
Rules[Alert Rules]
Notify[Notifications]
end
HC --> Dash
PM --> Dash
LOG --> Files
Files --> Rules
Dash --> Rules
Rules --> Notify
style HC fill:#bbdefb
style PM fill:#c8e6c9
style LOG fill:#fff9c4
style Rules fill:#ffcdd2
```
**Key Takeaways:**
1. **Health Checks**: Use `/api/health` for liveness/readiness probes
2. **Metrics**: Track latency, throughput, and stability continuously
3. **Logging**: Use structured JSON logs for machine-readable analysis
4. **Alerting**: Set up multi-channel alerts with appropriate thresholds
5. **Dashboards**: Build real-time dashboards for visibility
**For More Information:**
- [HTTP API](./api.md) - Health endpoint details
- [Client Connection Manager](./client_connection_manager.md) - Client statistics
- [Session Manager](./session_manager.md) - Task tracking
- [Quick Start](./quick_start.md) - Get started with UFO server
## Next Steps
- [Quick Start](./quick_start.md) - Get the server running
- [HTTP API](./api.md) - API endpoint reference
- [WebSocket Handler](./websocket_handler.md) - Connection management
- [Session Manager](./session_manager.md) - Task execution tracking
---
### Documents Docs Server Quick Start (`documents/docs/server/quick_start.md`)
# Quick Start
This hands-on guide walks you through starting the UFO Agent Server, connecting clients, and dispatching your first task. Perfect for first-time users.
---
## 📋 Prerequisites
Before you begin, ensure you have:
- **Python 3.10+** installed
- **UFO dependencies** installed (`pip install -r requirements.txt`)
- **Network connectivity** for WebSocket connections
- **Terminal access** (PowerShell, bash, or equivalent)
| Component | Minimum Version | Recommended |
|-----------|----------------|-------------|
| Python | 3.10 | 3.11+ |
| FastAPI | 0.104+ | Latest |
| Uvicorn | 0.24+ | Latest |
| UFO | - | Latest commit |
---
## 🚀 Starting the Server
### Basic Startup
Start the server with default settings (port **5000**):
```bash
python -m ufo.server.app
```
**Expected Output:**
```console
2024-11-04 14:30:22 - ufo.server.app - INFO - Starting UFO Server on 0.0.0.0:5000
2024-11-04 14:30:22 - ufo.server.app - INFO - Platform: auto-detected
2024-11-04 14:30:22 - ufo.server.app - INFO - Log level: WARNING
INFO: Started server process [12345]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)
```
Once you see "Uvicorn running", the server is ready to accept WebSocket connections at `ws://0.0.0.0:5000/ws`.
### Configuration Options
| Argument | Type | Default | Description | Example |
|----------|------|---------|-------------|---------|
| `--port` | int | `5000` | Server listening port | `--port 8080` |
| `--host` | str | `0.0.0.0` | Bind address (0.0.0.0 = all interfaces) | `--host 127.0.0.1` |
| `--platform` | str | `auto` | Platform override (`windows`, `linux`) | `--platform windows` |
| `--log-level` | str | `WARNING` | Logging verbosity | `--log-level DEBUG` |
| `--local` | flag | `False` | Restrict to localhost connections only | `--local` |
**Common Startup Configurations:**
**Development (Local Only):**
```bash
python -m ufo.server.app --local --log-level DEBUG
```
- Accepts connections only from `localhost`
- Verbose debug logging
- Default port 5000
**Custom Port:**
```bash
python -m ufo.server.app --port 8080
```
- Useful if port 5000 is already in use
- Accessible from network
**Production (Linux):**
```bash
python -m ufo.server.app --port 5000 --platform linux --log-level WARNING
```
- Explicit platform specification
- Reduced logging for performance
- Production-ready configuration
**Multi-Interface Binding:**
```bash
python -m ufo.server.app --host 192.168.1.100 --port 5000
```
- Binds to specific network interface
- Useful for multi-homed servers
---
## 🖥️ Connecting Device Clients
A Device Client is an agent running on a physical or virtual machine that can execute tasks. Each device connects via WebSocket and registers with a unique `client_id`.
Once the server is running, connect device agents using the command line:
### Platform-Specific Commands
**Windows Device:**
```bash
python -m ufo.client.client --ws --ws-server ws://127.0.0.1:5000/ws --client-id my_windows_device
```
**Linux Device:**
```bash
python -m ufo.client.client --ws --ws-server ws://127.0.0.1:5000/ws --client-id my_linux_device --platform linux
```
When a client connects successfully, the server logs will display:
```console
INFO: [WS] 📱 Device client my_windows_device connected
```
### Client Connection Parameters
| Parameter | Required | Type | Description | Example |
|-----------|----------|------|-------------|---------|
| `--ws` | Yes | flag | Enable WebSocket mode (vs. local mode) | `--ws` |
| `--ws-server` | Yes | URL | Server WebSocket endpoint | `ws://127.0.0.1:5000/ws` |
| `--client-id` | Yes | string | Unique device identifier (must be unique across all clients) | `device_win_001` |
| `--platform` | ⚠️ Optional | string | Platform type: `windows`, `linux` | `--platform windows` |
!!!warning "Important: Client ID Uniqueness"
Each `client_id` must be globally unique. If a client connects with an existing ID, the old connection will be terminated.
!!!tip "Platform Auto-Detection"
If you don't specify `--platform`, the client will auto-detect the operating system. However, **explicit specification is recommended** for clarity.
### Registration Protocol Flow
```mermaid
sequenceDiagram
participant C as Device Client
participant S as Agent Server
C->>S: WebSocket CONNECT /ws
S-->>C: Connection accepted
C->>S: REGISTER {client_id, platform}
S->>S: Validate & register
S-->>C: REGISTER_CONFIRM
Note over C: Client Ready
```
The registration process uses the **Agent Interaction Protocol (AIP)** for structured communication. See [AIP Documentation](../aip/overview.md) for details.
---
## 🌌 Connecting Constellation Clients
A Constellation Client is an orchestrator that coordinates multi-device tasks. It connects to the server and can dispatch work across multiple registered device clients.
### Basic Constellation Connection
```bash
python -m galaxy.constellation.constellation --ws --ws-server ws://127.0.0.1:5000/ws --target-id my_windows_device
```
### Constellation Parameters
| Parameter | Required | Description | Example |
|-----------|----------|-------------|---------|
| `--ws` | Yes | Enable WebSocket mode | `--ws` |
| `--ws-server` | Yes | Server WebSocket URL | `ws://127.0.0.1:5000/ws` |
| `--target-id` | ⚠️ Optional | Initial target device ID for tasks | `my_windows_device` |
!!!danger "Important: Target Device Must Be Online"
If you specify `--target-id`, that device **must already be connected** to the server. Otherwise, registration will fail with: `Target device 'my_windows_device' is not connected`
A constellation can dynamically dispatch tasks to different devices, not just the `target-id`. For more on multi-device orchestration, see [Constellation Documentation](../galaxy/overview.md).
---
## Verifying the Setup
### Method 1: Check Connected Clients
Use the HTTP API to verify connections:
```bash
curl http://localhost:5000/api/clients
```
**Expected Response:**
```json
{
"online_clients": ["my_windows_device", "my_linux_device"]
}
```
If you see your `client_id` in the list, the device is successfully connected and ready to receive tasks.
### Method 2: Health Check
```bash
curl http://localhost:5000/api/health
```
**Expected Response:**
```json
{
"status": "healthy",
"online_clients": ["my_windows_device"]
}
```
The `/api/health` endpoint is useful for health checks in production monitoring systems.
---
## 🎯 Dispatching Your First Task
The easiest way to send a task to a connected device is through the HTTP `/api/dispatch` endpoint.
### Basic Task Dispatch
Use the HTTP API to dispatch a task to a connected device:
```bash
curl -X POST http://localhost:5000/api/dispatch \
-H "Content-Type: application/json" \
-d '{
"client_id": "my_windows_device",
"request": "Open Notepad and type Hello World",
"task_name": "test_task_001"
}'
```
**Request Body Parameters:**
| Field | Required | Type | Description | Example |
|-------|----------|------|-------------|---------|
| `client_id` | Yes | string | Target device identifier | `"my_windows_device"` |
| `request` | Yes | string | Natural language task description | `"Open Notepad"` |
| `task_name` | ⚠️ Optional | string | Unique task identifier (auto-generated if omitted) | `"task_001"` |
**Successful Response:**
```json
{
"status": "dispatched",
"task_name": "test_task_001",
"client_id": "my_windows_device",
"session_id": "3f4a2b1c-9d8e-4f3a-b2c1-9a8b7c6d5e4f"
}
```
The `status: "dispatched"` indicates the task was successfully sent to the device. The device will begin executing immediately.
!!!warning "Client Must Be Online"
If the target `client_id` is not connected, you'll receive `{"detail": "Client not online"}`. Use `/api/clients` to verify the device is connected first.
### Task Execution Flow
```mermaid
sequenceDiagram
participant API as HTTP Client
participant S as Server
participant D as Device
API->>S: POST /api/dispatch
S->>D: TASK (AIP)
D->>D: Execute task
D->>S: TASK_RESULT
API->>S: GET /task_result
S->>API: Results
```
For detailed API specifications, see [HTTP API Reference](./api.md).
### Checking Task Results
Use the task name to retrieve results:
```bash
curl http://localhost:5000/api/task_result/test_task_001
```
**While Task is Running:**
```json
{
"status": "pending"
}
```
**When Task Completes:**
```json
{
"status": "done",
"result": {
"action_taken": "Opened Notepad and typed 'Hello World'",
"screenshot": "base64_encoded_image...",
"observation": "Task completed successfully"
}
}
```
!!!tip "Polling Best Practice"
For long-running tasks, poll every 2-5 seconds. Most simple tasks complete within 10-30 seconds.
### Advanced Task Dispatch
**Complex Multi-Step Task:**
```bash
curl -X POST http://localhost:5000/api/dispatch \
-H "Content-Type: application/json" \
-d '{
"client_id": "my_windows_device",
"request": "Open Excel, create a new worksheet, and enter sales data for Q4 2024",
"task_name": "excel_q4_report"
}'
```
**Web Automation Task:**
```bash
curl -X POST http://localhost:5000/api/dispatch \
-H "Content-Type: application/json" \
-d '{
"client_id": "my_windows_device",
"request": "Open Chrome, navigate to GitHub.com, and search for UFO framework",
"task_name": "github_search"
}'
```
**File Management Task:**
```bash
curl -X POST http://localhost:5000/api/dispatch \
-H "Content-Type: application/json" \
-d '{
"client_id": "my_linux_device",
"request": "Create a folder named test_data and copy all .txt files from Documents",
"task_name": "file_organization"
}'
```
---
## 🐛 Common Issues & Troubleshooting
### Issue 1: Port Already in Use
**Symptoms:**
```console
ERROR: [Errno 98] Address already in use
```
**Cause:** Another process is already using port 5000.
**Solutions:**
**Use Different Port:**
```bash
python -m ufo.server.app --port 8080
```
**Find & Kill Process (Linux/Mac):**
```bash
# Find process using port 5000
lsof -i :5000
# Kill the process
kill -9
```
**Find & Kill Process (Windows):**
```powershell
# Find process using port 5000
netstat -ano | findstr :5000
# Kill the process
taskkill /PID /F
```
### Issue 2: Connection Refused
**Symptoms:**
```console
[WS] Failed to connect to ws://127.0.0.1:5000/ws
Connection refused
```
**Diagnosis Checklist:**
- Is the server actually running? Check for "Uvicorn running" message
- Does the port match in both server and client commands?
- Are you using `--local` mode? If yes, clients must connect from `localhost`
- Is there a firewall blocking the connection?
**Solutions:**
1. Verify server is running:
```bash
curl http://localhost:5000/api/health
```
2. Check server logs for startup errors
3. If using `--local` mode, ensure client uses `127.0.0.1`
4. If connecting from another machine, remove `--local` flag
### Issue 3: Device Not Connected Error
**Symptoms:**
When dispatching a task:
```json
{
"detail": "Client not online"
}
```
**Diagnosis:**
1. List all connected clients:
```bash
curl http://localhost:5000/api/clients
```
2. Check the `client_id` matches exactly (case-sensitive!)
**Solutions:**
- Verify the device client is running and successfully registered
- Check server logs for `📱 Device client connected`
- Ensure no typos in `client_id` when dispatching
- If the device disconnected, restart the client connection
### Issue 4: Empty Task Content Error
**Symptoms:**
```json
{
"detail": "Empty task content"
}
```
**Cause:** The `request` field in `/api/dispatch` is missing or empty.
**Solution:** Always include the `request` field with a task description.
### Issue 5: Firewall Blocking Connections
**Symptoms:** Clients on other machines cannot connect, but `curl localhost:5000/api/health` works on server machine.
**Diagnosis:**
1. **Check server is listening on all interfaces:**
```bash
# Should show 0.0.0.0:5000 (not 127.0.0.1:5000)
netstat -tuln | grep 5000
```
2. **Test from remote machine:**
```bash
curl http://:5000/api/health
```
**Solutions:**
**Windows Firewall:**
```powershell
# Allow incoming connections on port 5000
New-NetFirewallRule -DisplayName "UFO Server" `
-Direction Inbound `
-Protocol TCP `
-LocalPort 5000 `
-Action Allow
```
**Linux (ufw):**
```bash
sudo ufw allow 5000/tcp
sudo ufw reload
```
**Linux (firewalld):**
```bash
sudo firewall-cmd --zone=public --add-port=5000/tcp --permanent
sudo firewall-cmd --reload
```
### Issue 6: Target Device Not Connected (Constellation)
**Symptoms:**
```console
Target device 'my_windows_device' is not connected
```
**Solution:**
1. Connect the device client first
2. Wait for registration confirmation (check server logs)
3. Then connect constellation
!!!tip "Debug Mode"
For maximum verbosity, start the server with: `python -m ufo.server.app --log-level DEBUG`
---
## 📚 Next Steps
Now that you have the server running and can dispatch tasks, explore these topics:
### Immediate Next Steps
| Step | Topic | Time | Description |
|------|-------|------|-------------|
| 1️⃣ | [Server Architecture](./overview.md) | 10 min | Understand the three-tier architecture and component interactions |
| 2️⃣ | [HTTP API Reference](./api.md) | 15 min | Explore all available API endpoints for integration |
| 3️⃣ | [Client Setup Guide](../client/quick_start.md) | 10 min | Learn advanced client configuration options |
| 4️⃣ | [AIP Protocol](../aip/overview.md) | 20 min | Deep dive into the Agent Interaction Protocol |
### Advanced Topics
| Topic | Relevance | Link |
|-------|-----------|------|
| **Session Management** | Understanding task lifecycle and state | [Session Manager](./session_manager.md) |
| **WebSocket Handler** | Low-level connection handling | [WebSocket Handler](./websocket_handler.md) |
| **Monitoring & Operations** | Production deployment best practices | [Monitoring](./monitoring.md) |
| **Constellation Mode** | Multi-device orchestration | Coming Soon |
---
## 🚀 Production Deployment
!!!warning "Production Readiness Checklist"
Before deploying to production, ensure you address these critical areas:
### 1. Process Management
!!!example "Systemd Service (Linux)"
Create `/etc/systemd/system/ufo-server.service`:
```ini
[Unit]
Description=UFO Agent Server
After=network.target
[Service]
Type=simple
User=ufo
WorkingDirectory=/opt/ufo
Environment="PATH=/opt/ufo/venv/bin"
ExecStart=/opt/ufo/venv/bin/python -m ufo.server.app --port 5000 --log-level INFO
Restart=always
RestartSec=10
StandardOutput=journal
StandardError=journal
[Install]
WantedBy=multi-user.target
```
**Enable and start:**
```bash
sudo systemctl daemon-reload
sudo systemctl enable ufo-server
sudo systemctl start ufo-server
sudo systemctl status ufo-server
```
**PM2 Process Manager (Cross-Platform):**
```bash
# Install PM2
npm install pm2
# Start server with PM2
pm2 start "python -m ufo.server.app --port 5000" --name ufo-server
# Setup auto-restart on system boot
pm2 startup
pm2 save
# Monitor
pm2 logs ufo-server
pm2 monit
```
For complete production deployment guidance including SSL/TLS, security hardening, and scaling strategies, see [Server Overview - Production Deployment](./overview.md#production-deployment).
---
## 🎓 What You Learned
You've successfully:
- Started the UFO Agent Server with custom configurations
- Connected device and constellation clients via WebSocket
- Dispatched tasks using the HTTP API
- Verified connections and monitored health
- Troubleshot common issues
- Learned production deployment best practices
Continue your journey with:
- **Architecture Deep Dive**: [Server Overview](./overview.md)
- **API Exploration**: [HTTP API Reference](./api.md)
- **Client Development**: [Client Documentation](../client/overview.md)
- **Multi-Device Coordination**: [Constellation Overview](../galaxy/overview.md)
---
### Documents Docs Server Session Manager (`documents/docs/server/session_manager.md`)
# Session Manager
The **SessionManager** orchestrates agent session lifecycles, coordinates background task execution, and maintains execution state across the server. It serves as the "execution engine" that powers UFO's autonomous task capabilities.
For context on how this component fits into the server architecture, see the [Server Overview](overview.md).
---
## 🎯 Overview
The SessionManager is a critical server component that bridges task dispatch and actual execution:
| Capability | Description | Benefit |
|------------|-------------|---------|
| **Platform-Agnostic Creation** | Automatically creates Windows/Linux sessions | No manual platform handling needed |
| **Background Execution** | Tasks run without blocking WebSocket event loop | Maintains connection health during long tasks |
| **State Tracking** | Monitors session lifecycle (created → running → completed/failed) | Enables task monitoring & result retrieval |
| **Graceful Cancellation** | Handles disconnections with context-aware cleanup | Prevents orphaned tasks & resource leaks |
| **Concurrent Management** | Multiple sessions can run simultaneously | Supports multi-device orchestration |
### Architecture Position
```mermaid
graph TB
subgraph "Agent Server"
WH[WebSocket Handler]
SM[Session Manager]
SF[Session Factory]
subgraph "Sessions"
WS[Windows Service Session]
LS[Linux Service Session]
LOC[Local Session]
end
end
Client[Device Client] -->|WebSocket| WH
WH -->|"execute_task_async()"| SM
SM -->|"create session"| SF
SF -->|"platform=windows"| WS
SF -->|"platform=linux"| LS
SF -->|"local=true"| LOC
WS -->|"execute commands"| Client
LS -->|"execute commands"| Client
SM -->|"callback(result)"| WH
WH -->|"TASK_END message"| Client
style SM fill:#ffecb3
style SF fill:#c8e6c9
style WH fill:#bbdefb
```
**Why Background Execution Matters:**
Without background execution, a long-running task (e.g., 5-minute workflow) would **block the WebSocket event loop**, preventing:
- Heartbeat messages from being sent/received
- Ping/pong frames from maintaining the connection
- Other clients' tasks from being dispatched
Background execution solves this by using Python's `asyncio.create_task()` to run sessions concurrently.
---
## 🏗 Core Functionality
### Session Creation
The SessionManager uses the **SessionFactory** pattern to create platform-specific session implementations. This abstraction layer automatically selects the correct session type based on platform and mode.
**Creating a Session:**
```python
session = session_manager.get_or_create_session(
session_id="session_abc123",
task_name="create_file",
request="Open Notepad and create a file",
task_protocol=task_protocol, # AIP TaskExecutionProtocol instance
platform_override="windows" # or "linux" or None (auto-detect)
)
```
**Session Types:**
| Session Type | Use Case | Platform | Dispatcher | MCP Tools |
|--------------|----------|----------|------------|-----------|
| **ServiceSession (Windows)** | Remote Windows device | Windows | AIP protocol-based | Windows MCP servers |
| **LinuxServiceSession** | Remote Linux device | Linux | AIP protocol-based | Linux MCP servers |
| **Local Session** | Local testing/debugging | Any | Direct execution | Local MCP servers |
**Platform Detection:**
If `platform_override=None`, the SessionManager uses Python's `platform.system()` to auto-detect:
- `"Windows"` → ServiceSession (Windows)
- `"Linux"` → LinuxServiceSession
- `"Darwin"` (macOS) → Currently uses LinuxServiceSession
**Session Factory Logic Flow:**
```mermaid
graph LR
A[get_or_create_session] --> B{Session exists?}
B -->|Yes| C[Return existing]
B -->|No| D{local mode?}
D -->|Yes| E[Create Local Session]
D -->|No| F{Platform?}
F -->|windows| G[ServiceSession]
F -->|linux| H[LinuxServiceSession]
E --> I[Store in sessions dict]
G --> I
H --> I
I --> J[Return session]
style D fill:#ffe0b2
style F fill:#ffe0b2
style I fill:#c8e6c9
```
### Background Execution
The **critical innovation** of the SessionManager is background task execution using `asyncio.create_task()`. This prevents long-running sessions from blocking the WebSocket event loop.
**Execute Task Asynchronously:**
```python
await session_manager.execute_task_async(
session_id=session_id,
task_name=task_name,
request=user_request,
task_protocol=task_protocol, # AIP TaskExecutionProtocol instance
platform_override="windows",
callback=result_callback # Called when task completes
)
```
**Benefits of Background Execution:**
| Benefit | Description | Impact |
|---------|-------------|--------|
| **WebSocket Health** | Ping/pong continues uninterrupted | Prevents connection timeouts (30-60s) |
| **Heartbeat Flow** | Heartbeat messages can be sent/received | Maintains connection liveness |
| **Concurrent Sessions** | Multiple sessions run simultaneously | Supports multi-device orchestration |
| **Event Loop Availability** | Handler can process other messages | Responsive to new connections/dispatches |
| **Graceful Cancellation** | Tasks can be cancelled mid-execution | Clean disconnection handling |
**Background Execution Flow:**
```mermaid
sequenceDiagram
participant WH as WebSocket Handler
participant SM as Session Manager
participant BT as Background Task
participant S as Session
participant CB as Callback
Note over WH,SM: 1️⃣ Task Dispatch
WH->>SM: execute_task_async(session_id, request, callback)
SM->>SM: get_or_create_session()
SM->>BT: asyncio.create_task(_run_session_background)
SM-->>WH: Return immediately (non-blocking!)
Note over WH: Event loop free for other tasks
WH->>WH: Can process heartbeats, ping/pong, new tasks
Note over BT,S: 2️⃣ Background Execution
BT->>S: await session.run()
S->>S: LLM reasoning Action selection Execution
Note over S: Long-running task (30s - 5min)
S-->>BT: Execution complete
Note over BT,CB: 3️⃣ Result Callback
BT->>BT: Build ServerMessage (TASK_END)
BT->>SM: set_results(session_id)
BT->>CB: await callback(session_id, result_message)
CB->>WH: Send result via WebSocket
Note over BT: 4️⃣ Cleanup
BT->>SM: Remove from _running_tasks dict
```
**Thread Safety:**
The SessionManager uses `threading.Lock` for thread-safe access to shared dictionaries:
```python
with self.lock:
self.sessions[session_id] = session
```
This prevents race conditions in multi-threaded environments (though FastAPI primarily uses async/await).
### Callback Mechanism
When a task completes (successfully, with errors, or via cancellation), the SessionManager invokes a registered callback function. This decouples task execution from result delivery.
**Registering a Callback:**
```python
async def send_result_to_client(session_id: str, result_msg: ServerMessage):
"""Called when task completes."""
await websocket.send_text(result_msg.model_dump_json())
logger.info(f"Sent TASK_END for {session_id}")
await session_manager.execute_task_async(
session_id="abc123",
task_name="open_notepad",
request="Open Notepad",
task_protocol=task_protocol,
callback=send_result_to_client # Register callback
)
```
**Callback Execution Flow:**
```mermaid
stateDiagram-v2
[*] --> TaskRunning: Background task starts
TaskRunning --> ResultsCollected: session.run() completes
ResultsCollected --> StatusDetermined: Check session.is_finished() / is_error()
StatusDetermined --> MessageBuilt: Create ServerMessage(TASK_END)
MessageBuilt --> ResultsPersisted: set_results(session_id)
ResultsPersisted --> CallbackInvoked: await callback(session_id, message)
CallbackInvoked --> [*]: Cleanup _running_tasks
TaskRunning --> TaskCancelled: asyncio.CancelledError
TaskCancelled --> CancellationHandled: Check cancellation_reason
CancellationHandled --> MessageBuilt: Create failure message
TaskRunning --> ErrorOccurred: Exception raised
ErrorOccurred --> ErrorLogged: Log traceback
ErrorLogged --> MessageBuilt: Create error message
```
**ServerMessage Structure:**
```python
result_message = ServerMessage(
type=ServerMessageType.TASK_END,
status=TaskStatus.COMPLETED, # or FAILED
session_id="abc123",
error=None, # or error message if failed
result=session.results, # Dict[str, Any]
timestamp="2024-11-04T14:30:22.123456+00:00",
response_id="uuid-v4"
)
```
| Field | Type | Description | Example |
|-------|------|-------------|---------|
| `type` | ServerMessageType | Always `TASK_END` for completion | `ServerMessageType.TASK_END` |
| `status` | TaskStatus | `COMPLETED`, `FAILED`, or `CANCELLED` | `TaskStatus.COMPLETED` |
| `session_id` | str | Session identifier | `"abc123"` |
| `error` | Optional[str] | Error message if task failed | `"Device disconnected"` |
| `result` | Dict[str, Any] | Task execution results | `{"action": "opened notepad", "screenshot": "..."}` |
| `timestamp` | str | ISO 8601 timestamp (UTC) | `"2024-11-04T14:30:22Z"` |
| `response_id` | str | Unique response UUID | `"3f4a2b1c-9d8e-4f3a-b2c1-..."` |
**Callback Error Handling:**
If the callback raises an exception, the SessionManager **logs the error but doesn't fail the session**:
```python
try:
await callback(session_id, result_message)
except Exception as e:
self.logger.error(f"Callback error: {e}")
# Session results are still persisted!
```
This prevents callback bugs from breaking task execution.
### Task Cancellation
The SessionManager supports **graceful task cancellation** with different behaviors based on **why** the cancellation occurred. This is critical for handling client disconnections properly.
**Cancel a Running Task:**
```python
await session_manager.cancel_task(
session_id="session_abc123",
reason="device_disconnected" # or "constellation_disconnected"
)
```
**Cancellation Reasons:**
| Reason | Scenario | Callback Behavior | Use Case |
|--------|----------|-------------------|----------|
| `constellation_disconnected` | Constellation client lost connection | **No callback** (client is gone) | Task requester disconnected, no one to notify |
| `device_disconnected` | Target device lost connection | **Send callback** to constellation | Notify orchestrator to reassign task |
| `user_requested` | Manual cancellation via API | **Send callback** to requester | Explicit cancellation command |
**Cancellation Flow:**
```mermaid
sequenceDiagram
participant C as Client (Constellation)
participant WH as WebSocket Handler
participant SM as Session Manager
participant BT as Background Task
participant D as Device
Note over C,BT: Scenario 1: Device Disconnects During Task
C->>WH: Task dispatched to device
WH->>SM: execute_task_async(session_id, callback)
SM->>BT: Background task running
BT->>D: Executing actions
Note over D: Device disconnects
D--xWH: WebSocket closed
WH->>SM: cancel_task(session_id, reason="device_disconnected")
SM->>BT: task.cancel()
BT->>BT: Catch asyncio.CancelledError
BT->>BT: Build failure message
BT->>WH: callback(session_id, failure_msg)
WH->>C: TASK_END (status=FAILED, error="Device disconnected")
Note over C,SM: Scenario 2: Constellation Disconnects During Task
C->>WH: Task dispatched
WH->>SM: execute_task_async()
SM->>BT: Background task running
Note over C: Constellation disconnects
C--xWH: WebSocket closed
WH->>SM: cancel_task(session_id, reason="constellation_disconnected")
SM->>BT: task.cancel()
BT->>BT: Catch asyncio.CancelledError
BT->>BT: Skip callback (client gone)
BT->>SM: Remove session
```
**Cancellation Implementation Details:**
```python
async def cancel_task(self, session_id: str, reason: str) -> bool:
"""Cancel a running background task."""
task = self._running_tasks.get(session_id)
if task and not task.done():
# Store reason for use in _run_session_background
self._cancellation_reasons[session_id] = reason
# Request cancellation
task.cancel()
# Wait for graceful shutdown (max 2 seconds)
try:
await asyncio.wait_for(task, timeout=2.0)
except (asyncio.CancelledError, asyncio.TimeoutError):
pass # Expected
# Cleanup
self._running_tasks.pop(session_id, None)
self._cancellation_reasons.pop(session_id, None)
self.remove_session(session_id)
return True
return False
```
**Important Notes:**
- **Cancellation is asynchronous**: The background task receives an `asyncio.CancelledError` at the next `await` point. If the session is executing synchronous code (e.g., LLM inference), cancellation won't take effect until that operation completes.
- **Grace Period**: The SessionManager waits up to **2 seconds** for graceful cancellation before giving up.
**Best Practice:**
When a client disconnects, the WebSocket Handler should:
1. Identify all active sessions for that client
2. Call `cancel_task()` with the appropriate `reason`
3. Clean up client registration in ClientConnectionManager
This prevents orphaned sessions from consuming resources.
---
## 🔄 Session Lifecycle
Sessions follow a predictable lifecycle from initial dispatch through execution to final cleanup. Understanding this flow is essential for debugging and monitoring.```mermaid
stateDiagram-v2
[*] --> Created: get_or_create_session()
Created --> Stored: Add to sessions dict
Stored --> BackgroundTask: execute_task_async()
BackgroundTask --> Running: await session.run()
Running --> Completed: session.is_finished() == True
Running --> Failed: session.is_error() == True
Running --> Cancelled: asyncio.CancelledError
Running --> Exception: Exception raised
Completed --> ResultsCollected: Gather session.results
Failed --> ResultsCollected: Include error details
Cancelled --> ResultsCollected: Include cancellation reason
Exception --> ResultsCollected: Include exception message
ResultsCollected --> ResultsPersisted: set_results(session_id)
ResultsPersisted --> CallbackInvoked: await callback(session_id, message)
CallbackInvoked --> Cleanup: remove_session(session_id)
Cleanup --> [*]
```
### Lifecycle Stages
| Stage | Description | Key Operations | Duration |
|-------|-------------|----------------|----------|
| **1. Creation** | Session object instantiated | `get_or_create_session()` | < 100ms |
| **2. Registration** | Stored in sessions dict with ID | `sessions[session_id] = session` | < 10ms |
| **3. Background Dispatch** | Task created with `asyncio.create_task()` | `_running_tasks[session_id] = task` | < 50ms |
| **4. Execution** | Session runs (LLM + actions) | `await session.run()` | 10s - 5min |
| **5. Result Collection** | Gather results and determine status | `session.results`, `session.is_finished()` | < 100ms |
| **6. Persistence** | Save results to results dict | `set_results(session_id)` | < 10ms |
| **7. Callback** | Notify registered callback | `await callback(session_id, msg)` | 50-500ms |
| **8. Cleanup** | Remove from active sessions | `remove_session(session_id)` | < 10ms |
**Complete Lifecycle Example:**
```python
# Stage 1-2: Creation
session = session_manager.get_or_create_session(
session_id="abc123",
task_name="demo_task",
request="Open Notepad",
task_protocol=task_protocol,
platform_override="windows"
)
# Stage 3: Background Dispatch
await session_manager.execute_task_async(
session_id="abc123",
task_name="demo_task",
request="Open Notepad",
task_protocol=task_protocol,
platform_override="windows",
callback=send_result_callback
)
# Returns immediately! Task runs in background
# Stage 4: Execution (happens in background)
# session.run() executes:
# - LLM reasoning
# - Action selection
# - Command execution via device
# - Result observation
# Stage 5-6: Results (automatic)
# Session completes, results collected and persisted
# Stage 7: Callback (automatic)
# await callback("abc123", ServerMessage(...))
# Stage 8: Cleanup (manual or automatic)
session_manager.remove_session("abc123")
```
**Session Persistence:**
Sessions remain in the `sessions` dict until explicitly removed via `remove_session()`. This allows:
- **Result retrieval** via `/api/task_result/{task_name}`
- **Session inspection** for debugging
- **Reconnection scenarios** (future feature)
However, this means **sessions consume memory** until cleaned up. Implement periodic cleanup for production deployments.
---
## 💾 State Management
The SessionManager maintains three separate dictionaries for different aspects of session state:
### 1. Active Sessions Storage
```python
self.sessions: Dict[str, BaseSession] = {}
```
| Purpose | Structure | Lifecycle | Thread Safety |
|---------|-----------|-----------|---------------|
| Store active session objects | `{session_id: BaseSession}` | Until `remove_session()` called | `threading.Lock` |
**Session Storage Operations:**
```python
# Store session
with self.lock:
self.sessions[session_id] = session
# Retrieve session
with self.lock:
session = self.sessions.get(session_id)
# Remove session
with self.lock:
self.sessions.pop(session_id, None)
```
**Benefits:**
- Fast O(1) lookup by session ID
- Thread-safe with lock
- Supports session reuse (future reconnections)
**Considerations:**
- ⚠️ Memory grows with active sessions
- ⚠️ Manual cleanup required (`remove_session()`)
- ⚠️ No automatic expiration
### 2. Result Caching
```python
self.results: Dict[str, Dict[str, Any]] = {}
```
| Purpose | Structure | When Populated | Retrieval Methods |
|---------|-----------|----------------|-------------------|
| Cache completed task results | `{session_id: results_dict}` | After task completion via `set_results()` | `get_result()`, `get_result_by_task()` |
**Result Storage & Retrieval:**
```python
# Persist results after completion
def set_results(self, session_id: str):
with self.lock:
if session_id in self.sessions:
self.results[session_id] = self.sessions[session_id].results
# Retrieve by session ID
result = session_manager.get_result("abc123")
# Returns: {"action": "opened notepad", "screenshot": "base64..."}
# Retrieve by task name
result = session_manager.get_result_by_task("demo_task")
```
**Result Structure Example:**
```json
{
"action_taken": "Opened Notepad and typed 'Hello World'",
"screenshot": "base64_encoded_screenshot_data...",
"observation": "Notepad window is visible with text 'Hello World'",
"success": true,
"metadata": {
"steps_taken": 3,
"execution_time_seconds": 12.5
}
}
```
### 3. Task Name Mapping
```python
self.session_id_dict: Dict[str, str] = {}
```
| Purpose | Structure | Use Case |
|---------|-----------|----------|
| Map task names to session IDs | `{task_name: session_id}` | Allow result retrieval by task name instead of session ID |
**Task Name Mapping:**
```python
# Created during session creation
self.session_id_dict[task_name] = session_id
# Usage: Get result by task name
def get_result_by_task(self, task_name: str):
with self.lock:
session_id = self.session_id_dict.get(task_name)
if session_id:
return self.get_result(session_id)
```
**Why This Matters:**
The HTTP API endpoint `/api/task_result/{task_name}` allows clients to check results using the **task name** they provided, without needing to track session IDs:
```bash
# Client only needs to remember task name
curl http://localhost:5000/api/task_result/demo_task
# Instead of tracking session ID
curl http://localhost:5000/api/task_result/abc123
```
### 4. Running Tasks Tracking
```python
self._running_tasks: Dict[str, asyncio.Task] = {}
```
| Purpose | Structure | Use Case |
|---------|-----------|----------|
| Track active background tasks for cancellation | `{session_id: asyncio.Task}` | Enable graceful task cancellation when clients disconnect |
**Running Task Management:**
```python
# Register background task
task = asyncio.create_task(self._run_session_background(...))
self._running_tasks[session_id] = task
# Cancel running task
task = self._running_tasks.get(session_id)
if task and not task.done():
task.cancel()
await asyncio.wait_for(task, timeout=2.0)
# Cleanup after completion
self._running_tasks.pop(session_id, None)
```
### 5. Cancellation Reasons Tracking
```python
self._cancellation_reasons: Dict[str, str] = {}
```
| Purpose | Structure | Lifecycle |
|---------|-----------|-----------|
| Store why each task was cancelled | `{session_id: reason}` | From `cancel_task()` to `_run_session_background()` cleanup |
**Cancellation Reason Flow:**
```python
# Store reason when cancelling
async def cancel_task(self, session_id: str, reason: str):
self._cancellation_reasons[session_id] = reason
task.cancel()
# Retrieve reason during cancellation handling
async def _run_session_background(...):
try:
await session.run()
except asyncio.CancelledError:
reason = self._cancellation_reasons.get(session_id, "unknown")
if reason == "device_disconnected":
# Send callback to constellation
elif reason == "constellation_disconnected":
# Skip callback
```
---
### Thread Safety
The SessionManager uses `threading.Lock` for thread-safe access to shared dictionaries:
```python
def __init__(self):
self.lock = threading.Lock()
def get_or_create_session(self, ...):
with self.lock:
if session_id not in self.sessions:
self.sessions[session_id] = session
return self.sessions[session_id]
```
**Why this matters:** Although FastAPI primarily uses async/await (single-threaded event loop), the lock protects against:
- **Thread pool executors** for sync operations
- **Background tasks** accessing shared state
- **Future multi-threading** in FastAPI/Uvicorn
**Performance Consideration:**
Lock contention is minimal because:
- Lock is held only for **dictionary operations** (O(1) operations)
- Session execution happens **outside the lock** (async background tasks)
- Most operations are **read-heavy** (get_result) which are fast
---
## 🖥 Platform Support
The SessionManager supports both Windows and Linux platforms through the **SessionFactory** abstraction layer. Platform-specific implementations handle OS-specific UI automation and tool execution.
### Platform Detection
```mermaid
graph TD
A[get_or_create_session] --> B{platform_override specified?}
B -->|Yes| C[Use specified platform]
B -->|No| D[Auto-detect via platform.system]
D --> E{OS Detected}
E -->|"Windows"| F[platform = 'windows']
E -->|"Linux"| G[platform = 'linux']
E -->|"Darwin" macOS| H[platform = 'linux' ⚠️ Treated as Linux]
C --> I[SessionFactory.create_service_session]
F --> I
G --> I
H --> I
I --> J{Platform?}
J -->|windows| K[ServiceSession]
J -->|linux| L[LinuxServiceSession]
style H fill:#ffe0b2
style K fill:#c8e6c9
style L fill:#bbdefb
```
**Platform Detection Code:**
```python
def __init__(self, platform_override: Optional[str] = None):
self.platform = platform_override or platform.system().lower()
# platform.system() returns: "Windows", "Linux", or "Darwin"
self.logger.info(f"SessionManager initialized for platform: {self.platform}")
```
### Platform-Specific Sessions
| Platform | Session Class | UI Automation | MCP Tools | Status |
|----------|---------------|---------------|-----------|--------|
| **Windows** | `ServiceSession` | Win32 API, UI Automation | Windows MCP servers (filesystem, browser, etc.) | Fully Supported |
| **Linux** | `LinuxServiceSession` | X11/Wayland, AT-SPI | Linux MCP servers | Fully Supported |
| **macOS (Darwin)** | `LinuxServiceSession` | Currently treated as Linux | Linux MCP servers | ⚠️ Experimental |
**Windows Session Creation:**
```python
# Explicit Windows platform
session = session_manager.get_or_create_session(
session_id="win_session_001",
task_name="windows_task",
request="Open File Explorer and navigate to Downloads",
task_protocol=task_protocol,
platform_override="windows"
)
# Creates ServiceSession
```
**Linux Session Creation:**
```python
# Explicit Linux platform
session = session_manager.get_or_create_session(
session_id="linux_session_001",
task_name="linux_task",
request="Open Nautilus and create a new folder",
task_protocol=task_protocol,
platform_override="linux"
)
# Creates LinuxServiceSession
```
**Auto-Detection:**
```python
# Let SessionManager detect platform
session = session_manager.get_or_create_session(
session_id="auto_session_001",
task_name="auto_task",
request="Open text editor",
task_protocol=task_protocol,
platform_override=None # Auto-detect
)
# Uses platform.system() to determine session type
```
**macOS Limitations:**
macOS (Darwin) is currently treated as Linux, which may result in:
- Incorrect UI automation commands
- Missing macOS-specific tool integrations
- ⚠️ Limited functionality
**Recommendation:** Use explicit `platform_override="linux"` for Linux-like behavior, or wait for dedicated macOS session implementation.
---
## 🐛 Error Handling
The SessionManager implements comprehensive error handling to prevent task failures from breaking the server.
### Error Categories
| Error Type | Handler | Behavior | Example |
|------------|---------|----------|---------|
| **Session Execution Errors** | `try/except in _run_session_background` | Status = FAILED, error message in results | LLM API timeout, invalid action |
| **Callback Errors** | `try/except around callback invocation` | Log error, continue execution | WebSocket closed before callback |
| **Cancellation** | `asyncio.CancelledError handler` | Check reason, conditional callback | Client disconnected mid-task |
| **Unknown State** | Status check after `session.run()` | Status = FAILED, error = "unknown state" | Session neither finished nor errored |
### Session Execution Error Handling
```python
async def _run_session_background(...):
try:
await session.run() # May raise exceptions
# Determine status
if session.is_error():
status = TaskStatus.FAILED
session.results = session.results or {"failure": "session ended with an error"}
elif session.is_finished():
status = TaskStatus.COMPLETED
else:
status = TaskStatus.FAILED
error = "Session ended in unknown state"
except asyncio.CancelledError:
# Handle cancellation (see Cancellation section)
...
except Exception as e:
# Catch all other exceptions
import traceback
traceback.print_exc()
self.logger.error(f"Error in session {session_id}: {e}")
status = TaskStatus.FAILED
error = str(e)
```
**Error Result Structure:**
When a session fails, the result includes error details:
```json
{
"status": "FAILED",
"error": "LLM API timeout after 60 seconds",
"session_id": "abc123",
"result": {
"failure": "session ended with an error",
"last_action": "open_notepad",
"traceback": "Traceback (most recent call last)..."
}
}
```
### Callback Error Handling
```python
try:
await callback(session_id, result_message)
except Exception as e:
import traceback
self.logger.error(
f"Callback error for session {session_id}: {e}\n{traceback.format_exc()}"
)
# Session results are STILL persisted!
# Client may not receive notification
```
**Callback Failures Don't Fail Sessions:**
If the callback raises an exception (e.g., WebSocket already closed), the SessionManager:
- **Logs the error** for debugging
- **Persists the results** in `self.results`
- **Completes cleanup** (removes from `_running_tasks`)
- **Does NOT re-raise** the exception
**Implication:** Results can be retrieved via `/api/task_result/{task_name}` even if WebSocket notification failed.
### Unknown State Handling
```python
if session.is_error():
status = TaskStatus.FAILED
elif session.is_finished():
status = TaskStatus.COMPLETED
else:
# Unknown state - neither finished nor errored
status = TaskStatus.FAILED
error = "Session ended in unknown state"
self.logger.warning(f"Session {session_id} ended in unknown state")
```
**Edge Case - Session Hangs:**
If `session.run()` completes but the session is neither `is_finished()` nor `is_error()`, this indicates:
- Possible bug in session state management
- Incomplete session implementation
- Unexpected session interruption
The SessionManager marks this as **FAILED** to prevent silent failures.
---
## 💡 Best Practices
Follow these best practices to ensure reliable, scalable session management:
### 1. Configure Appropriate Timeouts
Session timeouts should match task complexity:
| Task Type | Timeout | Reason |
|-----------|---------|--------|
| **Simple UI Actions** | 60-120s | Open app, click button, type text |
| **Medium Workflows** | 120-300s | Multi-step automation (3-5 steps) |
| **Complex Tasks** | 300-600s | Complex workflows requiring LLM reasoning |
| **Batch Operations** | 600-1800s | Processing multiple files, data entry |
```python
# Configure in UFO config
ufo_config.system.timeout = 300 # 5 minutes for medium tasks
```
### 2. Monitor Session Count
Sessions consume memory. Implement limits to prevent resource exhaustion:
```python
MAX_CONCURRENT_SESSIONS = 100 # Adjust based on server resources
async def execute_task_safe(session_manager, ...):
active_count = len(session_manager.sessions)
if active_count >= MAX_CONCURRENT_SESSIONS:
# Option 1: Reject new sessions
raise HTTPException(
status_code=503,
detail=f"Server at capacity ({active_count} active sessions)"
)
# Option 2: Cancel oldest sessions
oldest_session_id = min(
session_manager.sessions.keys(),
key=lambda s: session_manager.sessions[s].created_at
)
await session_manager.cancel_task(
oldest_session_id,
reason="capacity_limit"
)
# Proceed with new session
await session_manager.execute_task_async(...)
```
### 3. Clean Up Completed Sessions
⚠️ **Memory Leak Prevention:**
Sessions persist in `sessions` dict until explicitly removed. Implement cleanup:
```python
# Option 1: Cleanup immediately after result retrieval
result = session_manager.get_result(session_id)
if result:
session_manager.remove_session(session_id)
# Option 2: Periodic cleanup task
import asyncio
async def cleanup_old_sessions(session_manager, max_age_seconds=3600):
"""Remove sessions older than max_age_seconds."""
while True:
await asyncio.sleep(300) # Check every 5 minutes
current_time = time.time()
with session_manager.lock:
to_remove = []
for session_id, session in session_manager.sessions.items():
age = current_time - session.created_at
if age > max_age_seconds and session_id not in session_manager._running_tasks:
to_remove.append(session_id)
for session_id in to_remove:
session_manager.remove_session(session_id)
logger.info(f"Cleaned up old session: {session_id}")
# Start cleanup task on server startup
asyncio.create_task(cleanup_old_sessions(session_manager))
```
### 4. Handle Cancellation Gracefully
Different cancellation reasons require different responses:
```python
async def handle_client_disconnect(client_id, client_type, session_manager, client_manager):
"""Handle disconnection based on client type."""
if client_type == ClientType.CONSTELLATION:
# Constellation disconnected - cancel all its tasks
session_ids = client_manager.get_constellation_sessions(client_id)
for session_id in session_ids:
await session_manager.cancel_task(
session_id,
reason="constellation_disconnected" # Don't send callback
)
elif client_type == ClientType.DEVICE:
# Device disconnected - notify constellations to reassign
session_ids = client_manager.get_device_sessions(client_id)
for session_id in session_ids:
await session_manager.cancel_task(
session_id,
reason="device_disconnected" # Send callback to constellation
)
# Clean up client registration
client_manager.remove_client(client_id)
```
### 5. Log Session Lifecycle Events
Log key lifecycle events for debugging and monitoring: ```python
# Session creation
self.logger.info(f"Created {platform} session: {session_id} (type: {session_type})")
# Background task start
self.logger.info(f"🚀 Started background task {session_id}")
# Execution timing
elapsed = loop.time() - start_time
self.logger.info(f"⏱️ Session {session_id} execution took {elapsed:.2f}s")
# Status determination
self.logger.info(f"Session {session_id} finished successfully")
self.logger.warning(f"⚠️ Session {session_id} ended with error")
# Cancellation
self.logger.warning(f"🛑 Session {session_id} was cancelled (reason: {reason})")
# Cleanup
self.logger.info(f"Session {session_id} completed with status {status}")
```
### 6. Implement Result Expiration
Prevent `results` dict from growing indefinitely:
```python
from collections import OrderedDict
import time
class SessionManagerWithExpiration(SessionManager):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
# Store (result, timestamp) tuples
self.results: Dict[str, Tuple[Dict, float]] = {}
self.result_ttl = 3600 # 1 hour
def set_results(self, session_id: str):
with self.lock:
if session_id in self.sessions:
self.results[session_id] = (
self.sessions[session_id].results,
time.time()
)
def get_result(self, session_id: str):
with self.lock:
if session_id in self.results:
result, timestamp = self.results[session_id]
# Check expiration
if time.time() - timestamp > self.result_ttl:
self.results.pop(session_id)
return None
return result
return None
```
### 7. Monitor Background Tasks
Monitor background tasks for unexpectedly long execution:
```python
import asyncio
async def monitor_long_running_tasks(session_manager, threshold=600):
"""Alert on tasks running longer than threshold seconds."""
while True:
await asyncio.sleep(60) # Check every minute
current_time = asyncio.get_event_loop().time()
for session_id, task in session_manager._running_tasks.items():
# Calculate task age (approximation)
session = session_manager.sessions.get(session_id)
if session and hasattr(session, 'start_time'):
age = current_time - session.start_time
if age > threshold:
logger.warning(
f"⚠️ Long-running task detected: {session_id} "
f"(running for {age:.1f}s)"
)
```
---
## 🔗 Integration with Server Components
The SessionManager doesn't operate in isolation—it's deeply integrated with other server components.
### Integration Architecture
```mermaid
graph TB
subgraph "External"
HTTP[HTTP API Client]
WS_C[WebSocket Client]
end
subgraph "Server Components"
API[API Router /api/dispatch]
WH[WebSocket Handler]
WSM[Client Connection Manager]
SM[Session Manager]
SF[Session Factory]
end
subgraph "Sessions"
WIN[Windows Session]
LIN[Linux Session]
end
HTTP -->|POST /api/dispatch| API
WS_C -->|WebSocket /ws| WH
API -->|execute_task_async| SM
WH -->|execute_task_async| SM
SM -->|create session| SF
SF -->|windows| WIN
SF -->|linux| LIN
SM -->|add_constellation_session| WSM
SM -->|add_device_session| WSM
SM -->|callback| WH
WH -->|TASK_END message| WS_C
style SM fill:#ffecb3
style SF fill:#c8e6c9
style WSM fill:#bbdefb
```
### 1. WebSocket Handler Integration
The WebSocket Handler creates sessions with callbacks to send results back to clients:
```python
# In WebSocket Handler
async def handle_task_dispatch(self, session_id, request, client_id):
"""Handle incoming task from constellation."""
# Define callback to send results back
async def send_result(sid: str, msg: ServerMessage):
await self.websocket.send_text(msg.model_dump_json())
logger.info(f"Sent TASK_END for {sid}")
# Execute task with callback
await self.session_manager.execute_task_async(
session_id=session_id,
task_name=f"task_{session_id[:8]}",
request=request,
task_protocol=self.task_protocol, # AIP protocol instance
platform_override=None, # Auto-detect
callback=send_result # Register callback
)
```
For more details, see the [WebSocket Handler Documentation](websocket_handler.md).
### 2. Client Connection Manager Integration
The Client Connection Manager tracks which clients own which sessions:
```python
# Track constellation sessions
client_manager.add_constellation_session(
constellation_id="constellation_001",
session_id="session_abc123"
)
# Track device sessions
client_manager.add_device_session(
device_id="device_windows_001",
session_id="session_abc123"
)
# Retrieve all sessions for a client
session_ids = client_manager.get_constellation_sessions("constellation_001")
# On disconnect, cancel all client sessions
for session_id in session_ids:
await session_manager.cancel_task(session_id, reason="client_disconnected")
```
For more details, see the [Client Connection Manager Documentation](client_connection_manager.md).
### 3. HTTP API Integration
The API router uses SessionManager to retrieve results:
```python
# In API router (ufo/server/services/api.py)
@router.post("/api/dispatch")
async def dispatch_task_api(data: Dict[str, Any]):
client_id = data.get("client_id")
user_request = data.get("request")
task_name = data.get("task_name", str(uuid4()))
# Get client protocol
task_protocol = client_manager.get_task_protocol(client_id)
if not task_protocol:
raise HTTPException(status_code=404, detail="Client not online")
session_id = str(uuid4())
# Use AIP protocol to send task
# ... send TASK_ASSIGNMENT via protocol ...
return {
"status": "dispatched",
"task_name": task_name,
"client_id": client_id,
"session_id": session_id
}
@router.get("/api/task_result/{task_name}")
async def get_task_result(task_name: str):
# Use SessionManager to retrieve results
result = session_manager.get_result_by_task(task_name)
if not result:
return {"status": "pending"}
return {"status": "done", "result": result}
```
---
## 📖 API Reference
Complete SessionManager API reference:### Initialization
```python
from ufo.server.services.session_manager import SessionManager
# Initialize with platform override
manager = SessionManager(platform_override="windows")
# Initialize with auto-detection
manager = SessionManager(platform_override=None)
```
**Parameters:**
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `platform_override` | `Optional[str]` | `None` | Platform type (`"windows"`, `"linux"`, or `None` for auto-detect) |
---
### get_or_create_session()
```python
session = manager.get_or_create_session(
session_id="abc123",
task_name="demo_task",
request="Open Notepad",
task_protocol=task_protocol,
platform_override="windows",
local=False
)
```
**Parameters:**
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `session_id` | `str` | Yes | - | Unique session identifier |
| `task_name` | `Optional[str]` | No | `"test_task"` | Human-readable task name |
| `request` | `Optional[str]` | No | `None` | User request text |
| `task_protocol` | `Optional[TaskExecutionProtocol]` | No | `None` | AIP TaskExecutionProtocol instance |
| `platform_override` | `Optional[str]` | No | `None` | Platform type override |
| `local` | `bool` | No | `False` | Whether to create local session (for testing) |
**Returns:** `BaseSession` - Platform-specific session instance
---
### execute_task_async()
```python
session_id = await manager.execute_task_async(
session_id="abc123",
task_name="demo_task",
request="Open Notepad",
task_protocol=task_protocol,
platform_override="windows",
callback=my_callback
)
```
**Parameters:**
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `session_id` | `str` | Yes | Session identifier |
| `task_name` | `str` | Yes | Task name |
| `request` | `str` | Yes | User request text |
| `task_protocol` | `Optional[TaskExecutionProtocol]` | No | AIP TaskExecutionProtocol instance |
| `platform_override` | `str` | Yes | Platform type |
| `callback` | `Optional[Callable]` | No | Async function called on completion |
**Callback Signature:**
```python
async def callback(session_id: str, result_message: ServerMessage) -> None:
...
```
**Returns:** `str` - The session ID (same as input)
---
### cancel_task()
```python
success = await manager.cancel_task(
session_id="abc123",
reason="device_disconnected"
)
```
**Parameters:**
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `session_id` | `str` | Yes | - | Session to cancel |
| `reason` | `str` | No | `"constellation_disconnected"` | Cancellation reason |
**Valid Reasons:**
- `"constellation_disconnected"` - Don't send callback
- `"device_disconnected"` - Send callback to constellation
- `"user_requested"` - Manual cancellation
**Returns:** `bool` - `True` if task was found and cancelled, `False` otherwise
---
### get_result()
```python
result = manager.get_result("abc123")
```
**Parameters:**
| Parameter | Type | Description |
|-----------|------|-------------|
| `session_id` | `str` | Session identifier |
**Returns:** `Optional[Dict[str, Any]]` - Session results dict, or `None` if not found
---
### get_result_by_task()
```python
result = manager.get_result_by_task("demo_task")
```
**Parameters:**
| Parameter | Type | Description |
|-----------|------|-------------|
| `task_name` | `str` | Task name |
**Returns:** `Optional[Dict[str, Any]]` - Session results dict, or `None` if not found
---
### set_results()
```python
manager.set_results("abc123")
```
**Parameters:**
| Parameter | Type | Description |
|-----------|------|-------------|
| `session_id` | `str` | Session identifier |
**Returns:** `None`
**Purpose:** Persist session results to `results` dict for later retrieval
---
### remove_session()
```python
manager.remove_session("abc123")
```
**Parameters:**
| Parameter | Type | Description |
|-----------|------|-------------|
| `session_id` | `str` | Session to remove |
**Returns:** `None`
**Purpose:** Remove session from active sessions dict (cleanup)
---
## 📚 Related Documentation
Explore related components to understand the full server architecture:
| Component | Purpose | Link |
|-----------|---------|------|
| **Server Overview** | High-level architecture and capabilities | [Overview](./overview.md) |
| **Quick Start** | Start server and dispatch first task | [Quick Start](./quick_start.md) |
| **WebSocket Handler** | Message handling and protocol implementation | [WebSocket Handler](./websocket_handler.md) |
| **Client Connection Manager** | Connection management and client tracking | [Client Connection Manager](./client_connection_manager.md) |
| **HTTP API** | RESTful API endpoints | [API Reference](./api.md) |
| **Session Factory** | Session creation patterns | [Session Pool](../infrastructure/modules/session_pool.md) |
| **AIP Protocol** | Agent Interaction Protocol details | [AIP Overview](../aip/overview.md) |
---
## 🎓 Key Takeaways
After reading this guide, you should understand:
- **Background execution** prevents WebSocket blocking during long tasks
- **SessionFactory** creates platform-specific sessions (Windows/Linux)
- **Callbacks** decouple task execution from result delivery
- **Cancellation reasons** enable context-aware disconnection handling
- **Thread safety** protects shared state in concurrent environments
- **State management** uses five separate dicts (sessions, results, task_names, running_tasks, cancellation_reasons)
- **Best practices** prevent resource exhaustion and memory leaks
**Next Steps:**
- Explore [WebSocket Handler](./websocket_handler.md) to see how sessions are triggered
- Learn about [AIP Protocol](../aip/overview.md) for task assignment message format
- Review [Client Connection Manager](./client_connection_manager.md) for session-to-client mapping
---
### Documents Docs Server Websocket Handler (`documents/docs/server/websocket_handler.md`)
# WebSocket Handler
The **UFOWebSocketHandler** is the central nervous system of the server, implementing the Agent Interaction Protocol (AIP) to manage structured, reliable communication between the server and all connected clients.
For context on how this component fits into the server architecture, see the [Server Overview](overview.md).
---
## 🎯 Overview
The WebSocket Handler acts as the protocol orchestrator, managing all aspects of client communication:
| Responsibility | Description | Protocol Used |
|----------------|-------------|---------------|
| **Client Registration** | Validate and register new device/constellation connections | AIP Registration Protocol |
| **Task Dispatch** | Route task requests to appropriate devices | AIP Task Execution Protocol |
| **Heartbeat Monitoring** | Maintain connection health via periodic pings | AIP Heartbeat Protocol |
| **Device Info Exchange** | Query and share device capabilities | AIP Device Info Protocol |
| **Command Results** | Relay execution results from devices to requesters | AIP Message Transport |
| **Error Handling** | Gracefully handle communication failures | AIP Error Protocol |
| **Connection Lifecycle** | Manage registration → active → cleanup flow | WebSocket + AIP |
### Architecture Position
```mermaid
graph TB
subgraph "Clients"
DC[Device Clients]
CC[Constellation Clients]
end
subgraph "Server - WebSocket Handler"
WH[UFOWebSocketHandler]
subgraph "AIP Protocols"
REG[Registration Protocol]
HB[Heartbeat Protocol]
DI[Device Info Protocol]
TE[Task Execution Protocol]
end
subgraph "Message Router"
MH[handle_message]
end
end
subgraph "Server Components"
WSM[Client Connection Manager]
SM[Session Manager]
end
DC -->|WebSocket /ws| WH
CC -->|WebSocket /ws| WH
WH --> REG
WH --> HB
WH --> DI
WH --> TE
WH --> MH
MH -->|"handle_task_request"| TE
MH -->|"handle_heartbeat"| HB
MH -->|"handle_device_info_request"| DI
MH -->|"handle_command_result"| SM
WH -->|"add_client / get_client"| WSM
WH -->|"execute_task_async"| SM
style WH fill:#ffecb3
style MH fill:#bbdefb
style SM fill:#c8e6c9
style WSM fill:#f8bbd0
```
---
## 🔌 AIP Protocol Integration
The handler uses **four specialized AIP protocols**, each handling a specific aspect of communication. This separation of concerns makes the code maintainable and testable. For detailed protocol specifications, see the [AIP Protocol Documentation](../aip/overview.md).
```python
def __init__(self, client_manager, session_manager, local=False):
# Initialize per-connection protocols
self.transport = None
self.registration_protocol = None
self.heartbeat_protocol = None
self.device_info_protocol = None
self.task_protocol = None
```
| Protocol | Purpose | Key Methods | Message Types |
|----------|---------|-------------|---------------|
| **Registration Protocol** | Client identity and validation | `send_registration_confirmation()`, `send_registration_error()` | `REGISTER`, `REGISTER_CONFIRM` |
| **Heartbeat Protocol** | Connection health monitoring | `send_heartbeat_ack()` | `HEARTBEAT`, `HEARTBEAT_ACK` |
| **Device Info Protocol** | Capability exchange | `send_device_info_response()`, `send_device_info_request()` | `DEVICE_INFO_REQUEST`, `DEVICE_INFO_RESPONSE` |
| **Task Execution Protocol** | Task lifecycle management | `send_task_assignment()`, `send_ack()`, `send_error()` | `TASK`, `TASK_ASSIGNMENT`, `TASK_END` |
**Protocol Initialization per Connection:**
```python
async def connect(self, websocket: WebSocket) -> str:
await websocket.accept()
# Initialize AIP protocols for this connection
self.transport = WebSocketTransport(websocket)
self.registration_protocol = RegistrationProtocol(self.transport)
self.heartbeat_protocol = HeartbeatProtocol(self.transport)
self.device_info_protocol = DeviceInfoProtocol(self.transport)
self.task_protocol = TaskExecutionProtocol(self.transport)
# ... registration flow ...
```
**Per-Connection Protocol Instances:**
Each WebSocket connection gets its **own set of protocol instances**, ensuring message routing and state management are isolated between clients.
---
## 📝 Client Registration
Registration is the critical first step when a client connects. The handler validates client identity, checks permissions, and establishes the communication session.
### Registration Flow
```mermaid
sequenceDiagram
participant C as Client (Device/Constellation)
participant WS as WebSocket Handler
participant RP as Registration Protocol
participant WSM as Client Connection Manager
Note over C,WS: 1️⃣ Connection Establishment
C->>WS: WebSocket CONNECT /ws
WS->>WS: await websocket.accept()
WS->>WS: Initialize AIP protocols
Note over WS,RP: 2️⃣ Registration Message
WS->>C: (AIP Transport ready)
C->>RP: REGISTER {client_id, client_type, platform, metadata}
RP->>RP: Parse & validate JSON
Note over RP,WS: 3️⃣ Validation
RP->>WS: ClientMessage object
WS->>WS: Validate client_id exists
WS->>WS: Validate message type = REGISTER
alt Client Type = Constellation
WS->>WSM: is_device_connected(target_id)?
alt Target device offline
WSM-->>WS: False
WS->>RP: send_registration_error()
RP->>C: ERROR: "Target device not connected"
WS->>C: WebSocket close()
else Target device online
WSM-->>WS: True
WS->>WSM: add_client(client_id, ...)
end
else Client Type = Device
WS->>WSM: add_client(client_id, platform, ...)
end
Note over WS,C: 4️⃣ Confirmation
WS->>RP: send_registration_confirmation()
RP->>C: REGISTER_CONFIRM {status: "success"}
Note over C: Client registered, ready for tasks
C->>C: Start message listening loop
```
### Registration Steps (Code Walkthrough)
**Step 1: WebSocket Connection Accepted**
```python
async def connect(self, websocket: WebSocket) -> str:
# Accept WebSocket connection
await websocket.accept()
# Initialize AIP protocols for this connection
self.transport = WebSocketTransport(websocket)
self.registration_protocol = RegistrationProtocol(self.transport)
self.heartbeat_protocol = HeartbeatProtocol(self.transport)
self.device_info_protocol = DeviceInfoProtocol(self.transport)
self.task_protocol = TaskExecutionProtocol(self.transport)
```
**Step 2: Receive Registration Message**
```python
async def _parse_registration_message(self) -> ClientMessage:
"""Parse and validate registration message using AIP Transport."""
self.logger.info("[WS] [AIP] Waiting for registration message...")
# Receive via AIP Transport
reg_data = await self.transport.receive()
if isinstance(reg_data, bytes):
reg_data = reg_data.decode("utf-8")
# Parse using Pydantic model
reg_info = ClientMessage.model_validate_json(reg_data)
self.logger.info(
f"[WS] [AIP] Received registration from {reg_info.client_id}, "
f"type={reg_info.client_type}"
)
return reg_info
```
**Expected Registration Message:**
```json
{
"type": "REGISTER",
"client_id": "device_windows_001",
"client_type": "DEVICE",
"platform": "windows",
"metadata": {
"hostname": "DESKTOP-ABC123",
"os_version": "Windows 11 Pro",
"screen_resolution": "1920x1080"
}
}
```
**Step 3: Validation**
```python
# Basic validation
if not reg_info.client_id:
raise ValueError("Client ID is required for WebSocket registration")
if reg_info.type != ClientMessageType.REGISTER:
raise ValueError("First message must be a registration message")
# Constellation-specific validation
if client_type == ClientType.CONSTELLATION:
await self._validate_constellation_client(reg_info)
```
**Constellation Validation:**
```python
async def _validate_constellation_client(self, reg_info: ClientMessage) -> None:
"""Validate constellation's claimed target_id."""
claimed_device_id = reg_info.target_id
if not claimed_device_id:
return # No device_id to validate
# Check if target device is connected
if not self.client_manager.is_device_connected(claimed_device_id):
error_msg = f"Target device '{claimed_device_id}' is not connected"
self.logger.warning(f"[WS] Constellation registration failed: {error_msg}")
# Send error via AIP protocol
await self._send_error_response(error_msg)
await self.transport.close()
raise ValueError(error_msg)
```
**Step 4: Register Client in [ClientConnectionManager](./client_connection_manager.md)**
```python
client_type = reg_info.client_type
platform = reg_info.metadata.get("platform", "windows") if reg_info.metadata else "windows"
# Register in Client Connection Manager
self.client_manager.add_client(
client_id,
platform,
websocket,
client_type,
reg_info.metadata,
transport=self.transport,
task_protocol=self.task_protocol,
)
```
**Step 5: Send Confirmation**
```python
async def _send_registration_confirmation(self) -> None:
"""Send successful registration confirmation using AIP RegistrationProtocol."""
self.logger.info("[WS] [AIP] Sending registration confirmation...")
await self.registration_protocol.send_registration_confirmation()
self.logger.info("[WS] [AIP] Registration confirmation sent")
```
**Confirmation Message:**
```json
{
"type": "REGISTER_CONFIRM",
"status": "success",
"timestamp": "2024-11-04T14:30:22.123456+00:00",
"response_id": "uuid-v4"
}
```
**Step 6: Log Success**
```python
def _log_client_connection(self, client_id: str, client_type: ClientType) -> None:
"""Log successful client connection with appropriate emoji."""
if client_type == ClientType.DEVICE:
self.logger.info(f"[WS] Registered device client: {client_id}")
elif client_type == ClientType.CONSTELLATION:
self.logger.info(f"[WS] 🌟 Registered constellation client: {client_id}")
```
### Validation Rules
| Validation | Check | Error Message | Action |
|------------|-------|---------------|--------|
| **Client ID Presence** | `client_id` field exists and not empty | `"Client ID is required"` | Reject connection |
| **Message Type** | First message type == `REGISTER` | `"First message must be a registration message"` | Reject connection |
| **Target Device (Constellation)** | If `target_id` specified, device must be online | `"Target device '' is not connected"` | Send error + close |
| **Client ID Uniqueness** | No existing client with same ID | Handled by ClientConnectionManager | Disconnect old connection |
**Constellation Dependency:**
Constellations **must** specify a valid `target_id` that refers to an already-connected device. If the device is offline or doesn't exist, registration fails immediately.
**Workaround:** Connect devices first, then constellations.
**Security Consideration:**
The current implementation does **not** authenticate clients. Any client can register with any `client_id`. For production deployments:
- Implement authentication tokens in `metadata`
- Validate client certificates (TLS client auth)
- Use API keys or OAuth tokens
- Whitelist allowed `client_id` patterns
---
## 📨 Message Handling
After registration, the handler enters a message loop, routing incoming client messages to specialized handlers based on message type.
### Message Dispatcher
```mermaid
graph TB
WS[WebSocket receive_text] --> Parse[Parse ClientMessage JSON]
Parse --> Router{Message Type?}
Router -->|TASK| HT[handle_task_request]
Router -->|COMMAND_RESULTS| HC[handle_command_result]
Router -->|HEARTBEAT| HH[handle_heartbeat]
Router -->|ERROR| HE[handle_error]
Router -->|DEVICE_INFO_REQUEST| HD[handle_device_info_request]
Router -->|DEVICE_INFO_RESPONSE| HDR[handle_device_info_response]
Router -->|Unknown| HU[handle_unknown]
HT --> SM[Session Manager]
HC --> CD[Command Dispatcher]
HH --> HP[Heartbeat Protocol]
HE --> Log[Error Logging]
HD --> DIP[Device Info Protocol]
style Router fill:#ffe0b2
style SM fill:#c8e6c9
style HP fill:#bbdefb
```
**Dispatcher Implementation:**
```python
async def handle_message(self, msg: str, websocket: WebSocket) -> None:
"""Dispatch incoming messages to specific handlers."""
try:
# Parse message using Pydantic model
data = ClientMessage.model_validate_json(msg)
client_id = data.client_id
client_type = data.client_type
msg_type = data.type
# Route to appropriate handler
if msg_type == ClientMessageType.TASK:
await self.handle_task_request(data, websocket)
elif msg_type == ClientMessageType.COMMAND_RESULTS:
await self.handle_command_result(data)
elif msg_type == ClientMessageType.HEARTBEAT:
await self.handle_heartbeat(data, websocket)
elif msg_type == ClientMessageType.ERROR:
await self.handle_error(data, websocket)
elif msg_type == ClientMessageType.DEVICE_INFO_REQUEST:
await self.handle_device_info_request(data, websocket)
elif msg_type == ClientMessageType.DEVICE_INFO_RESPONSE:
await self.handle_device_info_response(data, websocket)
else:
await self.handle_unknown(data, websocket)
except Exception as e:
self.logger.error(f"Error handling message: {e}")
try:
await self.task_protocol.send_error(str(e))
except (ConnectionError, IOError):
pass # Connection already closed
```
**Message Type Handlers:**
| Handler | Triggered By | Purpose | Response |
|---------|-------------|---------|----------|
| `handle_task_request` | `TASK` | Client requests task execution | `TASK_ASSIGNMENT` device |
| `handle_command_result` | `COMMAND_RESULTS` | Device reports command execution result | Unblock command dispatcher |
| `handle_heartbeat` | `HEARTBEAT` | Connection health ping | `HEARTBEAT_ACK` |
| `handle_error` | `ERROR` | Client reports error | Log + send error acknowledgment |
| `handle_device_info_request` | `DEVICE_INFO_REQUEST` | Constellation queries device capabilities | `DEVICE_INFO_RESPONSE` |
| `handle_device_info_response` | `DEVICE_INFO_RESPONSE` | Device provides info (pull model) | Store in ClientConnectionManager |
| `handle_unknown` | Any other type | Unknown/unsupported message | Log warning + send error |
---
### Task Request Handling
The handler supports task requests from **both device clients** (self-execution) and **constellation clients** (orchestrated execution on target devices).
**Task Request Flow:**
```mermaid
sequenceDiagram
participant C as Constellation
participant WH as WebSocket Handler
participant WSM as Client Connection Manager
participant SM as Session Manager
participant D as Device
Note over C,WH: 1️⃣ Task Request
C->>WH: TASK {request, target_id, session_id}
WH->>WH: Validate target_id
Note over WH,WSM: 2️⃣ Resolve Target Device
WH->>WSM: get_client(target_id)
WSM-->>WH: Device WebSocket
WH->>WSM: get_client_info(target_id)
WSM-->>WH: {platform: "windows"}
Note over WH,SM: 3️⃣ Create Session
WH->>SM: execute_task_async( session_id, request, target_ws, platform, callback )
SM-->>WH: session_id (non-blocking!)
Note over WH,C: 4️⃣ Immediate Acknowledgment
WH->>C: ACK {session_id, status: "dispatched"}
Note over SM,D: 5️⃣ Background Execution
SM->>D: TASK_ASSIGNMENT {request, session_id}
D->>D: Execute task (LLM + actions)
Note over D,WH: 6️⃣ Result Callback
D-->>SM: Task complete
SM->>WH: callback(session_id, result_msg)
WH->>C: TASK_END {status, result}
```
**Device Client Self-Execution:**
When a **device** requests a task for itself:
```python
async def handle_task_request(self, data: ClientMessage, websocket: WebSocket):
client_id = data.client_id
client_type = data.client_type
if client_type == ClientType.DEVICE:
# Device executing task on itself
target_ws = websocket # Use requesting client's WebSocket
platform = self.client_manager.get_client_info(client_id).platform
target_device_id = client_id
# ...
```
**Constellation Orchestrated Execution:**
When a **constellation** dispatches a task to a target device:
```python
async def handle_task_request(self, data: ClientMessage, websocket: WebSocket):
client_id = data.client_id
client_type = data.client_type
if client_type == ClientType.CONSTELLATION:
# Constellation dispatching to target device
target_device_id = data.target_id
target_ws = self.client_manager.get_client(target_device_id)
platform = self.client_manager.get_client_info(target_device_id).platform
# Validate target device exists
if not target_ws:
raise ValueError(f"Target device '{target_device_id}' not connected")
# Track session mappings
session_id = data.session_id or str(uuid.uuid4())
self.client_manager.add_constellation_session(client_id, session_id)
self.client_manager.add_device_session(target_device_id, session_id)
# ...
```
**Background Task Execution:**
```python
# Define callback for result delivery
async def send_result(sid: str, result_msg: ServerMessage):
"""Send result back to requester when task completes."""
# Send to constellation client
if client_type == ClientType.CONSTELLATION:
if websocket.client_state == WebSocketState.CONNECTED:
await websocket.send_text(result_msg.model_dump_json())
# Also send to target device (optional)
if target_ws and target_ws.client_state == WebSocketState.CONNECTED:
await target_ws.send_text(result_msg.model_dump_json())
else:
# Send to device client
if websocket.client_state == WebSocketState.CONNECTED:
await websocket.send_text(result_msg.model_dump_json())
# Execute in background via SessionManager
await self.session_manager.execute_task_async(
session_id=session_id,
task_name=task_name,
request=data.request,
websocket=target_ws, # Device WebSocket for command dispatcher
platform_override=platform,
callback=send_result # Called when task completes
)
# Send immediate acknowledgment (non-blocking)
await self.task_protocol.send_ack(session_id=session_id)
```
**Why Immediate ACK?**
The handler sends an **immediate ACK** after dispatching the task to the [SessionManager](./session_manager.md). This confirms:
- Task was received and validated
- Session was created successfully
- Task is now executing in background
The actual task result is delivered later via the `send_result` callback.
**Session Tracking:**
| Client Type | Session Tracking | Purpose |
|-------------|------------------|---------|
| **Device** | `client_manager.add_device_session(device_id, session_id)` | Track which device is executing the session |
| **Constellation** | `client_manager.add_constellation_session(constellation_id, session_id)` | Track which constellation requested the session |
| Both | Session Manager stores session `BaseSession` object | Execute and manage task lifecycle |
---
### Command Result Handling
When a device executes a command (e.g., "click button", "type text"), it sends results back to the server for processing by the session's command dispatcher.
**Command Result Flow:**
```mermaid
sequenceDiagram
participant S as Session (on server)
participant CD as Command Dispatcher
participant D as Device
participant WH as WebSocket Handler
Note over S,D: Session is running on server
S->>CD: execute_command("open_notepad")
CD->>D: Send command via WebSocket (response_id="cmd_123")
CD->>CD: await response (blocking)
Note over D: Device executes command
D->>D: Open Notepad application
D->>D: Take screenshot
Note over D,WH: Send result back
D->>WH: COMMAND_RESULTS {response_id="cmd_123", result, screenshot}
WH->>WH: handle_command_result()
WH->>CD: set_result(response_id, data)
Note over CD: Unblocks await!
CD-->>S: Return command result
S->>S: Continue session execution
```
**Handler Implementation:**
```python
async def handle_command_result(self, data: ClientMessage):
"""
Handle command execution results from devices.
Unblocks the command dispatcher waiting for this response.
"""
response_id = data.prev_response_id # ID of the command request
session_id = data.session_id
self.logger.debug(
f"[WS] Received command result for response_id={response_id}, "
f"session_id={session_id}"
)
# Get session's command dispatcher
session = self.session_manager.get_or_create_session(session_id)
command_dispatcher = session.context.command_dispatcher
# Set result (unblocks waiting dispatcher)
await command_dispatcher.set_result(response_id, data)
self.logger.debug(
f"[WS] Command result set for response_id={response_id}"
)
```
**Critical for Session Execution:**
Without proper command result handling, sessions would **hang indefinitely** waiting for device responses. The `set_result()` call is what unblocks the `await` in the command dispatcher.
---
### Heartbeat Handling
Heartbeats are lightweight ping/pong messages that ensure the WebSocket connection is alive and healthy.
```python
async def handle_heartbeat(self, data: ClientMessage, websocket: WebSocket) -> None:
"""Handle heartbeat messages using AIP HeartbeatProtocol."""
self.logger.debug(f"[WS] [AIP] Heartbeat from {data.client_id}")
try:
# Send acknowledgment via AIP protocol
await self.heartbeat_protocol.send_heartbeat_ack()
self.logger.debug(f"[WS] [AIP] Heartbeat response sent to {data.client_id}")
except (ConnectionError, IOError) as e:
# Connection closed - log but don't fail
self.logger.debug(f"[WS] [AIP] Could not send heartbeat ack: {e}")
```
**Heartbeat Message:**
```json
{
"type": "HEARTBEAT",
"client_id": "device_windows_001",
"timestamp": "2024-11-04T14:30:22.123456+00:00"
}
```
**Heartbeat ACK:**
```json
{
"type": "HEARTBEAT_ACK",
"timestamp": "2024-11-04T14:30:22.234567+00:00",
"response_id": "uuid-v4"
}
```
**Heartbeat Best Practices:**
- **Frequency:** Clients should send heartbeats every **30-60 seconds**
- **Timeout:** Server should consider connection dead after **2-3 missed heartbeats**
- **Lightweight:** Heartbeat messages are small and processed quickly
- **Non-blocking:** Heartbeat handling doesn't block task execution
---
### Device Info Handling
Constellations can query device capabilities (screen resolution, installed apps, OS version) to make intelligent task routing decisions.
**Device Info Request Flow:**
```mermaid
sequenceDiagram
participant C as Constellation
participant WH as WebSocket Handler
participant WSM as Client Connection Manager
participant DIP as Device Info Protocol
participant D as Device
Note over C,WH: 1️⃣ Request Device Info
C->>WH: DEVICE_INFO_REQUEST {target_id, request_id}
Note over WH,WSM: 2️⃣ Resolve Device
WH->>WSM: get_client(target_id)
WSM-->>WH: Device WebSocket
Note over WH,D: 3️⃣ Forward Request
WH->>DIP: send_device_info_request()
DIP->>D: DEVICE_INFO_REQUEST
Note over D: 4️⃣ Collect Info
D->>D: Gather system info (screen, OS, apps)
Note over D,WH: 5️⃣ Response
D->>DIP: DEVICE_INFO_RESPONSE {screen_res, os_version, ...}
DIP->>WH: Parse response
Note over WH,C: 6️⃣ Forward to Constellation
WH->>C: DEVICE_INFO_RESPONSE {device_info, request_id}
```
```python
async def handle_device_info_request(
self, data: ClientMessage, websocket: WebSocket
) -> None:
"""Handle device info requests from constellations."""
device_id = data.target_id
request_id = data.request_id
self.logger.info(
f"[WS] Constellation {data.client_id} requesting info for device {device_id}"
)
# Get device info (may involve querying the device)
device_info = await self.get_device_info(device_id)
# Send via AIP protocol
await self.device_info_protocol.send_device_info_response(
device_info=device_info,
request_id=request_id
)
```
**Device Info Structure:**
```json
{
"device_id": "device_windows_001",
"platform": "windows",
"os_version": "Windows 11 Pro 22H2",
"screen_resolution": "1920x1080",
"installed_applications": ["Chrome", "Excel", "Notepad", "..."],
"capabilities": ["ui_automation", "file_operations", "web_browsing"],
"cpu_cores": 8,
"memory_gb": 16
}
```
---
## 🔌 Client Disconnection
**Critical Cleanup Process:**
When a client disconnects (gracefully or abruptly), the handler must clean up sessions, remove registry entries, and prevent resource leaks.
### Disconnection Detection
```python
async def handler(self, websocket: WebSocket) -> None:
"""FastAPI WebSocket entry point."""
client_id = None
try:
# Registration
client_id = await self.connect(websocket)
# Message loop
while True:
msg = await websocket.receive_text()
asyncio.create_task(self.handle_message(msg, websocket))
except WebSocketDisconnect as e:
# Normal disconnection
self.logger.warning(
f"[WS] {client_id} disconnected code={e.code}, reason={e.reason}"
)
if client_id:
await self.disconnect(client_id)
except Exception as e:
# Unexpected error
self.logger.error(f"[WS] Error with client {client_id}: {e}")
if client_id:
await self.disconnect(client_id)
```
### Cleanup Process
```mermaid
graph TD
A[Client Disconnects] --> B{Get Client Info}
B --> C{Client Type?}
C -->|Device| D[Get Device Sessions]
C -->|Constellation| E[Get Constellation Sessions]
D --> F[Cancel Each Session reason='device_disconnected']
E --> G[Cancel Each Session reason='constellation_disconnected']
F --> H[Remove Device Session Mappings]
G --> I[Remove Constellation Session Mappings]
H --> J[Remove Client from ClientConnectionManager]
I --> J
J --> K[Log Disconnection]
K --> L[Cleanup Complete]
style F fill:#ffcdd2
style G fill:#ffcdd2
style J fill:#c8e6c9
```
**Device Client Cleanup:**
```python
async def disconnect(self, client_id: str) -> None:
"""Handle client disconnection and cleanup."""
client_info = self.client_manager.get_client_info(client_id)
if client_info and client_info.client_type == ClientType.DEVICE:
# Get all sessions running on this device
session_ids = self.client_manager.get_device_sessions(client_id)
if session_ids:
self.logger.info(
f"[WS] 📱 Device {client_id} disconnected, "
f"cancelling {len(session_ids)} active session(s)"
)
# Cancel all sessions
for session_id in session_ids:
try:
await self.session_manager.cancel_task(
session_id,
reason="device_disconnected" # Send callback to constellation
)
except Exception as e:
self.logger.error(f"Error cancelling session {session_id}: {e}")
# Clean up mappings
self.client_manager.remove_device_sessions(client_id)
```
**Constellation Client Cleanup:**
```python
if client_info and client_info.client_type == ClientType.CONSTELLATION:
# Get all sessions initiated by constellation
session_ids = self.client_manager.get_constellation_sessions(client_id)
if session_ids:
self.logger.info(
f"[WS] 🌟 Constellation {client_id} disconnected, "
f"cancelling {len(session_ids)} active session(s)"
)
# Cancel all associated sessions
for session_id in session_ids:
try:
await self.session_manager.cancel_task(
session_id,
reason="constellation_disconnected" # Don't send callback
)
except Exception as e:
self.logger.error(f"Error cancelling session {session_id}: {e}")
# Clean up mappings
self.client_manager.remove_constellation_sessions(client_id)
```
**Final Registry Cleanup:**
```python
# Remove client from registry
self.client_manager.remove_client(client_id)
self.logger.info(f"[WS] {client_id} disconnected")
```
### Cancellation Behavior Comparison
| Scenario | Cancellation Reason | Callback Sent? | Why? |
|----------|---------------------|----------------|------|
| **Device Disconnects** | `device_disconnected` | Yes Constellation | Notify orchestrator to reassign task |
| **Constellation Disconnects** | `constellation_disconnected` | No | Requester is gone, no one to notify |
**Proper Cleanup is Critical:**
Failing to clean up disconnected clients leads to:
- **Orphaned sessions** consuming server memory
- **Stale WebSocket references** causing errors
- **Registry pollution** with non-existent clients
- **Resource leaks** (file handles, memory)
---
## 🚨 Error Handling
The handler implements comprehensive error handling to prevent failures from cascading and breaking the entire server.
### Error Categories
| Error Type | Handler Location | Recovery Strategy |
|------------|------------------|-------------------|
| **Connection Errors** | `send_*` methods | Log and skip (connection already closed) |
| **Message Parsing Errors** | `handle_message` | Send error response via AIP |
| **Task Execution Errors** | `handle_task_request` | Log + send error via task protocol |
| **Validation Errors** | `_validate_*` methods | Send error + close connection |
| **Callback Errors** | Session Manager | Log but don't fail session |
### Connection Error Handling
```python
async def handle_heartbeat(self, data: ClientMessage, websocket: WebSocket):
try:
await self.heartbeat_protocol.send_heartbeat_ack()
except (ConnectionError, IOError) as e:
# Connection closed - log but don't fail
self.logger.debug(f"Could not send heartbeat ack: {e}")
# Don't raise - connection is already closed
```
**Why Catch and Ignore?**
When a connection is abruptly closed, attempts to send messages will raise `ConnectionError`. Since the client is already gone, there's no point in propagating the error—just log it and continue cleanup.
### Message Parsing Errors
```python
async def handle_message(self, msg: str, websocket: WebSocket):
try:
data = ClientMessage.model_validate_json(msg)
# ... route to handlers ...
except Exception as e:
import traceback
traceback.print_exc()
self.logger.error(f"Error handling message: {e}")
# Try to send error response
try:
await self.task_protocol.send_error(str(e))
except (ConnectionError, IOError) as send_error:
self.logger.debug(f"Could not send error response: {send_error}")
```
**Error Message Format:**
```json
{
"type": "ERROR",
"error": "Invalid message format: missing required field 'client_id'",
"timestamp": "2024-11-04T14:30:22.123456+00:00",
"response_id": "uuid-v4"
}
```
### Task Execution Errors
```python
async def handle_task_request(self, data: ClientMessage, websocket: WebSocket):
try:
# Validate target device
if client_type == ClientType.CONSTELLATION:
target_ws = self.client_manager.get_client(target_device_id)
if not target_ws:
raise ValueError(f"Target device '{target_device_id}' not connected")
# Execute task
await self.session_manager.execute_task_async(...)
except Exception as e:
self.logger.error(f"Error handling task: {e}")
await self.task_protocol.send_error(str(e))
```
### Validation Errors with Connection Closure
```python
async def _validate_constellation_client(self, reg_info: ClientMessage) -> None:
"""Validate constellation's target device."""
claimed_device_id = reg_info.target_id
if not self.client_manager.is_device_connected(claimed_device_id):
error_msg = f"Target device '{claimed_device_id}' is not connected"
self.logger.warning(f"Constellation registration failed: {error_msg}")
# Send error via AIP protocol
await self._send_error_response(error_msg)
# Close connection immediately
await self.transport.close()
# Raise to prevent further processing
raise ValueError(error_msg)
```
**When to Close Connections:**
Close connections immediately for:
- **Invalid registration** (missing client_id, wrong message type)
- **Authorization failures** (target device not connected for constellations)
- **Protocol violations** (sending TASK before REGISTER)
For other errors, log and send error messages but **keep connection alive**.
---
## Best Practices
### 1. Validate Early and Thoroughly
```python
# Good: Validate immediately after parsing
async def handle_task_request(self, data: ClientMessage, websocket: WebSocket):
if not data.request:
raise ValueError("Task request cannot be empty")
if not data.client_id:
raise ValueError("Client ID required")
if data.client_type == ClientType.CONSTELLATION and not data.target_id:
raise ValueError("Constellation must specify target_id")
# ... proceed with validated data ...
```
### 2. Always Check Connection State Before Sending
```python
from starlette.websockets import WebSocketState
# Good: Check state before sending
async def send_result(sid: str, result_msg: ServerMessage):
if websocket.client_state == WebSocketState.CONNECTED:
await websocket.send_text(result_msg.model_dump_json())
else:
self.logger.debug(f"Cannot send result, connection closed for {sid}")
```
**WebSocket States:**
| State | Description | Can Send? |
|-------|-------------|-----------|
| `CONNECTING` | Handshake in progress | No |
| `CONNECTED` | Active connection | Yes |
| `DISCONNECTED` | Connection closed | No |
### 3. Handle Cancellation Gracefully with Context
```python
# Good: Different reasons need different handling
async def disconnect(self, client_id: str):
client_info = self.client_manager.get_client_info(client_id)
if client_info.client_type == ClientType.CONSTELLATION:
reason = "constellation_disconnected" # Don't send callback
else:
reason = "device_disconnected" # Send callback to constellation
for session_id in session_ids:
await self.session_manager.cancel_task(session_id, reason=reason)
```
### 4. Use Structured Logging with Context
```python
# Good: Include client type and context
if client_type == ClientType.CONSTELLATION:
self.logger.info(
f"[WS] 🌟 Constellation {client_id} requesting task on {target_id}"
)
else:
self.logger.debug(
f"[WS] 📱 Received device message from {client_id}, type: {data.type}"
)
```
**Logging Levels:**
- `DEBUG`: Heartbeats, message routing, low-level protocol details
- `INFO`: Registration, disconnection, task dispatch, major lifecycle events
- `WARNING`: Validation failures, connection issues, recoverable errors
- `ERROR`: Unexpected exceptions, critical failures
### 5. Implement Async Message Handling
```python
# Good: Process messages in background tasks
async def handler(self, websocket: WebSocket):
while True:
msg = await websocket.receive_text()
asyncio.create_task(self.handle_message(msg, websocket))
# Loop continues immediately, doesn't wait for handler to finish
```
**Why `asyncio.create_task`?**
Without `create_task`, the handler would process messages **sequentially**, blocking new messages while handling the current one. This is problematic for:
- Long-running task dispatches
- Command result processing
- Device info queries
Background tasks allow **concurrent message processing** while keeping the receive loop responsive.
---
## 📚 Related Documentation
Explore related components to understand the full server architecture:
| Component | Purpose | Link |
|-----------|---------|------|
| **Server Overview** | High-level architecture and capabilities | [Overview](./overview.md) |
| **Quick Start** | Start server and dispatch first task | [Quick Start](./quick_start.md) |
| **Session Manager** | Session lifecycle and background execution | [Session Manager](./session_manager.md) |
| **Client Connection Manager** | Connection registry and session tracking | [Client Connection Manager](./client_connection_manager.md) |
| **HTTP API** | RESTful API endpoints | [API Reference](./api.md) |
| **AIP Protocol** | Agent Interaction Protocol details | [AIP Overview](../aip/overview.md) |
---
## 🎓 What You Learned
After reading this guide, you should understand:
- **AIP Protocol Integration** - Four specialized protocols handle different communication aspects
- **Registration Flow** - Validation → Registration → Confirmation
- **Message Routing** - Central dispatcher routes messages to specialized handlers
- **Dual Client Support** - Devices (self-execution) vs. Constellations (orchestration)
- **Background Task Dispatch** - Immediate ACK + async execution
- **Command Result Handling** - Unblocks command dispatcher waiting for device responses
- **Heartbeat Monitoring** - Lightweight connection health checks
- **Disconnection Cleanup** - Context-aware session cancellation and registry cleanup
- **Error Handling** - Graceful degradation without cascading failures
**Next Steps:**
- Explore [Session Manager](./session_manager.md) to understand background execution internals
- Learn about [Client Connection Manager](./client_connection_manager.md) for client registry management
- Review [AIP Protocol Documentation](../aip/overview.md) for message format specifications
---
### Documents Docs Tutorials Creating Mcp Servers (`documents/docs/tutorials/creating_mcp_servers.md`)
# Creating Custom MCP Servers - Complete Tutorial
This tutorial teaches you how to create, register, and deploy custom MCP servers for UFO² agents. You'll learn to build **local**, **HTTP**, and **stdio** MCP servers, and how to register them with different agents.
**Prerequisites**: Basic Python knowledge, familiarity with [MCP Overview](../mcp/overview.md) and [MCP Configuration](../mcp/configuration.md). Review [Built-in Local Servers](../mcp/local_servers.md) as examples.
---
## Table of Contents
1. [Overview](#overview)
2. [Local MCP Servers](#local-mcp-servers)
3. [HTTP MCP Servers](#http-mcp-servers)
4. [Stdio MCP Servers](#stdio-mcp-servers)
5. [Registering Servers with Agents](#registering-servers-with-agents)
6. [Best Practices](#best-practices)
7. [Troubleshooting](#troubleshooting)
---
## Overview
### MCP Server Types
UFO² supports three deployment models:
| Type | Deployment | Use Case | Complexity |
|------|------------|----------|------------|
| **Local** | In-process with agent | Fast, built-in tools | ⭐ Simple |
| **HTTP** | Standalone HTTP server | Cross-platform, remote control | ⭐⭐ Moderate |
| **Stdio** | Child process (stdin/stdout) | Process isolation, third-party tools | ⭐⭐⭐ Advanced |
### Server Categories
All MCP servers fall into two categories:
| Category | Purpose | LLM Selectable? | Auto-Invoked? |
|----------|---------|-----------------|---------------|
| **Data Collection** | Read-only observation | ❌ No | ✅ Yes |
| **Action** | State-changing execution | ✅ Yes | ❌ No |
**Tool Selection:**
- **Data Collection tools**: Automatically invoked by the framework to build observation prompts
- **Action tools**: LLM agent actively selects which tool to execute at each step
**Important**: Write clear docstrings and type annotations - they become LLM instructions!
---
## Local MCP Servers
Local servers run **in-process** with the UFO² agent, providing the fastest tool access.
### Step 1: Create Your Server
Create a Python file in `ufo/client/mcp/local_servers/` (or your custom location):
```
/* Detailed source-code truncated for AI context efficiency. */
```
!!!warning "Critical Design Rules"
1. **Use `@MCPRegistry.register_factory_decorator("Namespace")`** to register the factory
2. **Factory function must return a `FastMCP` instance**
3. **Use `@mcp.tool()` decorator** for each tool
4. **Write detailed docstrings** - they become LLM instructions
5. **Use `Annotated[Type, Field(description="...")]`** for all parameters and returns
6. **Namespace must be unique** across all servers
### Step 2: Import the Server
Add your server to `ufo/client/mcp/local_servers/__init__.py`:
```python
# File: ufo/client/mcp/local_servers/__init__.py
from .my_custom_server import create_my_custom_server
# ... other imports
__all__ = [
"create_my_custom_server",
# ... other exports
]
```
### Step 3: Configure in mcp.yaml
Add your server to the appropriate agent in `config/ufo/mcp.yaml`:
```yaml
# For action server (LLM-selectable)
CustomAgent:
default:
action:
- namespace: MyCustomExecutor
type: local
reset: false
# For data collection server (auto-invoked)
CustomAgent:
default:
data_collection:
- namespace: MyCustomCollector
type: local
reset: false
```
### Step 4: Test Your Server
Test locally before integration:
```python
# File: test_my_server.py
import asyncio
from fastmcp.client import Client
from ufo.client.mcp.local_servers.my_custom_server import create_my_custom_server
async def test_server():
"""Test the custom MCP server."""
server = create_my_custom_server()
async with Client(server) as client:
# List available tools
tools = await client.list_tools()
print(f"Available tools: {[t.name for t in tools]}")
# Test greet_user tool
result = await client.call_tool(
"greet_user",
arguments={"name": "Alice", "formal": True}
)
print(f"Greeting: {result.data}")
# Test calculate_sum tool
result = await client.call_tool(
"calculate_sum",
arguments={"numbers": [1, 2, 3, 4, 5]}
)
print(f"Sum: {result.data}")
if __name__ == "__main__":
asyncio.run(test_server())
```
### Example: Application-Specific Server
Here's a real-world example - a server for Chrome browser automation. For more details on wrapping application native APIs, see [Wrapping App Native API](creating_app_agent/warpping_app_native_api.md).
```
/* Detailed source-code truncated for AI context efficiency. */
```
**Configuration:**
```yaml
AppAgent:
chrome.exe:
data_collection:
- namespace: UICollector
type: local
action:
- namespace: AppUIExecutor # Generic UI automation
type: local
- namespace: ChromeExecutor # Chrome-specific tools
type: local
reset: true # Reset when switching tabs/windows
```
---
## HTTP MCP Servers
HTTP servers run as **standalone services**, enabling cross-platform automation and distributed workflows.
### Step 1: Create HTTP Server
Create a standalone Python script:
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Step 2: Start the HTTP Server
Run the server as a standalone process:
```bash
# Start on localhost
python -m ufo.client.mcp.http_servers.my_http_server --host localhost --port 8020
# Start on all interfaces (for remote access)
python -m ufo.client.mcp.http_servers.my_http_server --host 0.0.0.0 --port 8020
```
**For production, run as a background service:**
**Linux/macOS:**
```bash
nohup python -m ufo.client.mcp.http_servers.my_http_server --host 0.0.0.0 --port 8020 &
```
**Windows:**
```powershell
Start-Process python -ArgumentList "-m", "ufo.client.mcp.http_servers.my_http_server", "--host", "0.0.0.0", "--port", "8020" -WindowStyle Hidden
```
### Step 3: Configure HTTP Server in mcp.yaml
```yaml
RemoteAgent:
default:
action:
- namespace: MyHTTPExecutor
type: http
host: "localhost" # Or remote IP: "192.168.1.100"
port: 8020
path: "/mcp"
reset: false
```
### Step 4: Test HTTP Server
Test connectivity before integration:
```python
# File: test_http_server.py
import asyncio
from fastmcp.client import Client
async def test_http_server():
"""Test the HTTP MCP server."""
server_url = "http://localhost:8020/mcp"
async with Client(server_url) as client:
# List tools
tools = await client.list_tools()
print(f"Available tools: {[t.name for t in tools]}")
# Test process_data
result = await client.call_tool(
"process_data",
arguments={"data": "Hello World", "transform": "reverse"}
)
print(f"Process result: {result.data}")
# Test get_server_info
result = await client.call_tool("get_server_info", arguments={})
print(f"Server info: {result.data}")
if __name__ == "__main__":
asyncio.run(test_http_server())
```
### Example: Cross-Platform Linux Executor
Real-world example - controlling Linux systems from Windows:
```
/* Detailed source-code truncated for AI context efficiency. */
```
**Deploy on Linux:**
```bash
# Start server on Linux machine
python -m ufo.client.mcp.http_servers.linux_executor --host 0.0.0.0 --port 8010
```
**Configure on Windows UFO²:**
```yaml
LinuxAgent:
default:
action:
- namespace: LinuxExecutor
type: http
host: "192.168.1.50" # Linux machine IP
port: 8010
path: "/mcp"
```
**Cross-Platform Workflow**: Now your Windows UFO² agent can execute Linux commands remotely! The LLM will select `execute_command` or `get_system_info` as needed.
---
## Stdio MCP Servers
Stdio servers run as **child processes**, communicating via stdin/stdout. They provide process isolation and work with any language.
### Step 1: Create Stdio Server
Create a standalone script that reads JSON-RPC from stdin and writes to stdout:
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Step 2: Configure Stdio Server in mcp.yaml
```yaml
CustomAgent:
default:
action:
- namespace: CustomStdioExecutor
type: stdio
command: "python"
start_args: ["custom_stdio_server.py"]
env:
API_KEY: "secret_key"
LOG_LEVEL: "INFO"
cwd: "/path/to/server/directory"
reset: false
```
!!!warning "Stdio Limitations"
- **More complex** than local/HTTP servers
- Requires implementing **JSON-RPC protocol** manually
- Better suited for **third-party MCP servers** than custom tools
- For custom Python tools, **prefer local or HTTP servers**
### Example: Third-Party Node.js Server
Stdio is ideal for integrating existing MCP servers written in other languages:
```yaml
CustomAgent:
default:
action:
- namespace: NodeJSTools
type: stdio
command: "node"
start_args: ["./node_mcp_server/index.js"]
env:
NODE_ENV: "production"
cwd: "/path/to/node_mcp_server"
```
---
## Registering Servers with Agents
### Agent-Specific Registration
Different agents can use different MCP server configurations:
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Multi-Server Agent Configuration
Agents can register **multiple servers** of the same category:
```yaml
HybridAgent:
default:
# Multiple data collection sources
data_collection:
- namespace: UICollector
type: local
- namespace: HardwareCollector # Remote hardware monitoring
type: http
host: "192.168.1.50"
port: 8006
path: "/mcp"
- namespace: SystemMetrics # Custom metrics
type: local
# Multiple action executors (LLM chooses best tool)
action:
- namespace: AppUIExecutor # GUI automation
type: local
- namespace: WordCOMExecutor # API automation
type: local
reset: true
- namespace: LinuxExecutor # Remote Linux control
type: http
host: "192.168.1.100"
port: 8010
path: "/mcp"
- namespace: CustomExecutor # Custom actions
type: local
```
**How it works:**
1. **Data collection tools**: All servers are invoked automatically to build observation
2. **Action tools**: LLM sees tools from ALL action servers and selects the best one
**Example LLM decision:**
```
Task: "Create a Word document with sales data from the Linux database"
Step 1: Get data from Linux
→ LLM selects: LinuxExecutor::execute_command(
command="mysql -e 'SELECT * FROM sales'"
)
Step 2: Create Word document
→ LLM selects: WordCOMExecutor::insert_table(rows=10, columns=3)
Step 3: Format the table
→ LLM selects: WordCOMExecutor::select_table(number=1)
→ AppUIExecutor::click_input(name="Table Design")
```
### Configuration Hierarchy
Agent configurations follow this **inheritance hierarchy**:
```
AgentName
├─ default (fallback configuration)
│ ├─ data_collection
│ └─ action
└─ SubType (e.g., "WINWORD.EXE")
├─ data_collection
└─ action
```
**Lookup logic:**
1. Check for `AgentName.SubType`
2. If not found, use `AgentName.default`
3. If neither exists, raise error
**Example:**
```yaml
AppAgent:
# Fallback for all apps
default:
action:
- namespace: AppUIExecutor
type: local
# Overrides default for Word
WINWORD.EXE:
action:
- namespace: AppUIExecutor
type: local
- namespace: WordCOMExecutor
type: local
```
---
## Best Practices
### 1. Write Comprehensive Docstrings
Your docstrings are **directly converted to LLM prompts**. The LLM uses them to understand:
- **What** the tool does
- **When** to use it
- **How** to use it correctly
**Bad Example:**
```python
@mcp.tool()
def process(data: str) -> str:
"""Process data.""" # ❌ Too vague
return data.upper()
```
**Good Example:**
```python
@mcp.tool()
def process_text_to_uppercase(
text: Annotated[str, Field(description="The input text to convert.")],
) -> Annotated[str, Field(description="The text converted to uppercase.")]:
"""
Convert text to uppercase letters.
Use this tool when you need to standardize text formatting or make text
more prominent. Works with all Unicode characters.
Examples:
- "hello world" → "HELLO WORLD"
- "Café" → "CAFÉ"
""" # ✅ Clear, detailed, with examples
return text.upper()
```
### 2. Use Descriptive Parameter Names
```python
# ❌ Bad: Unclear parameter names
@mcp.tool()
def func(a: str, b: int, c: bool) -> str:
...
# ✅ Good: Self-documenting parameter names
@mcp.tool()
def send_email(
recipient_address: str,
message_body: str,
use_html_format: bool = False,
) -> str:
...
```
### 3. Provide Default Values
```python
@mcp.tool()
def search_files(
query: Annotated[str, Field(description="Search query.")],
case_sensitive: Annotated[
bool, Field(description="Case-sensitive search?")
] = False, # ✅ Sensible default
max_results: Annotated[
int, Field(description="Maximum results to return.")
] = 10, # ✅ Sensible default
) -> list[str]:
"""Search for files matching the query."""
...
```
### 4. Handle Errors Gracefully
```python
@mcp.tool()
def divide_numbers(
dividend: Annotated[float, Field(description="Number to divide.")],
divisor: Annotated[float, Field(description="Number to divide by.")],
) -> Annotated[dict, Field(description="Division result or error.")]:
"""
Divide two numbers and return the result.
Returns an error if divisor is zero.
"""
try:
if divisor == 0:
return {
"success": False,
"error": "Cannot divide by zero.",
}
result = dividend / divisor
return {
"success": True,
"result": result,
}
except Exception as e:
return {
"success": False,
"error": f"Division failed: {str(e)}",
}
```
### 5. Use Reset for Stateful Servers
```yaml
# ✅ Good: Reset COM servers when switching contexts
AppAgent:
WINWORD.EXE:
action:
- namespace: WordCOMExecutor
type: local
reset: true # Prevents state leakage between documents
# ❌ Bad: Not resetting can cause issues
AppAgent:
WINWORD.EXE:
action:
- namespace: WordCOMExecutor
type: local
reset: false # May retain state from previous document
```
### 6. Validate Remote Server Connectivity
Before deploying, test connectivity:
```python
import asyncio
from fastmcp.client import Client
async def validate_server(url: str):
"""Validate HTTP server is accessible."""
try:
async with Client(url) as client:
tools = await client.list_tools()
print(f"✅ Server {url} is accessible")
print(f" Tools: {[t.name for t in tools]}")
return True
except Exception as e:
print(f"❌ Server {url} is NOT accessible: {e}")
return False
# Test before adding to mcp.yaml
asyncio.run(validate_server("http://192.168.1.100:8020/mcp"))
```
### 7. Use Environment Variables for Secrets
```yaml
# ❌ Bad: Hardcoded secrets
CustomAgent:
default:
action:
- namespace: APIExecutor
type: http
host: "api.example.com"
port: 443
auth_token: "sk-1234567890" # Don't commit this!
# ✅ Good: Use environment variables
CustomAgent:
default:
action:
- namespace: APIExecutor
type: http
host: "${API_HOST}"
port: "${API_PORT}"
auth_token: "${API_TOKEN}"
```
Set environment variables before running UFO²:
```bash
export API_HOST="api.example.com"
export API_PORT="443"
export API_TOKEN="sk-1234567890"
```
---
## Troubleshooting
### Common Issues
#### 1. "No MCP server found for name 'MyServer'"
**Cause**: Server not registered in MCPRegistry.
**Solution**:
```python
# Ensure you're using the decorator
@MCPRegistry.register_factory_decorator("MyServer")
def create_my_server(*args, **kwargs) -> FastMCP:
...
# Or manually register
MCPRegistry.register_factory("MyServer", create_my_server)
```
#### 2. "Connection refused" for HTTP Server
**Cause**: HTTP server not running or wrong host/port.
**Solution**:
```bash
# Verify server is running
curl http://localhost:8020/mcp
# Check firewall rules
# Windows:
netsh advfirewall firewall add rule name="MCP Server" dir=in action=allow protocol=TCP localport=8020
# Linux:
sudo ufw allow 8020/tcp
```
#### 3. Tools Not Appearing in LLM Prompt
**Cause**: Server registered in wrong category (data_collection vs action).
**Solution**:
```yaml
# For LLM-selectable tools, use 'action'
CustomAgent:
default:
action: # ✅ Correct for LLM-selectable tools
- namespace: MyExecutor
type: local
# For auto-invoked observation, use 'data_collection'
CustomAgent:
default:
data_collection: # ✅ Correct for automatic observation
- namespace: MyCollector
type: local
```
#### 4. Server State Leaking Between Contexts
**Cause**: `reset: false` for stateful servers.
**Solution**:
```yaml
# Set reset: true for stateful servers
AppAgent:
WINWORD.EXE:
action:
- namespace: WordCOMExecutor
type: local
reset: true # ✅ Reset COM state when switching documents
```
#### 5. Timeout Errors for Long-Running Tools
**Cause**: Default timeout is 6000 seconds (100 minutes).
**Solution**:
```python
# In Computer class, adjust timeout
self._tool_timeout = 12000 # 200 minutes
```
### Debugging Tips
#### Enable Debug Logging
```python
import logging
logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger("ufo.client.mcp")
```
#### Check Registered Servers
```python
from ufo.client.mcp.mcp_server_manager import MCPServerManager
# List all registered servers
for namespace, server in MCPServerManager._servers_mapping.items():
print(f"Server: {namespace}, Type: {type(server).__name__}")
```
#### Test Server in Isolation
```python
# Test local server
from ufo.client.mcp.local_servers.my_custom_server import create_my_custom_server
import asyncio
from fastmcp.client import Client
async def test():
server = create_my_custom_server()
async with Client(server) as client:
tools = await client.list_tools()
print(f"Tools: {[t.name for t in tools]}")
asyncio.run(test())
```
---
## Next Steps
Now that you've learned to create MCP servers, explore these related topics:
1. **Review Built-in Servers**: See [Local Servers](../mcp/local_servers.md) for production examples
2. **Explore HTTP Deployment**: Read [Remote Servers](../mcp/remote_servers.md) for cross-platform automation
3. **Understand Agent Configuration**: Study [MCP Configuration](../mcp/configuration.md) for advanced setups
4. **Learn about Computer Class**: Review [Computer](../client/computer.md) to understand the MCP client integration
5. **Create Your First Agent**: Follow [Creating App Agent](creating_app_agent/overview.md) to build custom agents
---
## Related Documentation
- [MCP Overview](../mcp/overview.md) - MCP architecture and concepts
- [MCP Configuration](../mcp/configuration.md) - Complete configuration reference
- [Local Servers](../mcp/local_servers.md) - Built-in local servers
- [Remote Servers](../mcp/remote_servers.md) - HTTP/Stdio deployment
- [Data Collection Servers](../mcp/data_collection.md) - Observation tools
- [Action Servers](../mcp/action.md) - Execution tools
- [MCP Reference](../configuration/system/mcp_reference.md) - Quick reference guide
---
## Best Practices Summary
- ✅ **Write clear docstrings** - they become LLM instructions
- ✅ **Use descriptive names** - for tools, parameters, and namespaces
- ✅ **Handle errors gracefully** - return structured error messages
- ✅ **Test in isolation** - before integrating with agents
- ✅ **Use `reset: true`** - for stateful servers (COM, API clients)
- ✅ **Validate connectivity** - for HTTP/Stdio servers before deployment
---
### Documents Docs Tutorials Creating Third Party Agents (`documents/docs/tutorials/creating_third_party_agents.md`)
# Creating Custom Third-Party Agents - Complete Tutorial
This tutorial teaches you how to create, register, and deploy custom third-party agents that extend UFO²'s capabilities beyond Windows GUI automation. You'll learn the complete process using **HardwareAgent** as a reference implementation.
**Prerequisites**: Basic Python knowledge, familiarity with UFO² agent architecture, [Agent Configuration](../configuration/system/agents_config.md), and [Third-Party Configuration](../configuration/system/third_party_config.md).
---
## Table of Contents
1. [Overview](#overview)
2. [Understanding Third-Party Agents](#understanding-third-party-agents)
3. [Step-by-Step Implementation](#step-by-step-implementation)
4. [Complete Example: HardwareAgent](#complete-example-hardwareagent)
5. [Registering with HostAgent](#registering-with-hostagent)
6. [Configuration and Deployment](#configuration-and-deployment)
7. [Best Practices](#best-practices)
8. [Troubleshooting](#troubleshooting)
---
## Overview
### What are Third-Party Agents?
Third-party agents are specialized agents that extend UFO²'s capabilities to handle tasks beyond standard Windows GUI automation. They work alongside the core agents (HostAgent and AppAgent) to provide domain-specific functionality.
**Key Characteristics**:
- ✅ Independent agent implementation with custom logic
- ✅ Registered and managed by HostAgent
- ✅ Selectable as execution targets by the LLM
- ✅ Can use MCP servers and custom tools
- ✅ Configurable via YAML files
**Common Use Cases**:
- 🔧 **Hardware Control**: Physical device manipulation (HardwareAgent)
- 🐧 **Linux CLI**: Server and CLI command execution (LinuxAgent)
- 🌐 **Web Automation**: Browser-based tasks without GUI
- 📡 **IoT Integration**: Smart device control
- 🤖 **Robotic Process Automation**: Custom automation workflows
---
## Understanding Third-Party Agents
### Architecture Overview
Third-party agents integrate with UFO² through a well-defined architecture:
```mermaid
graph TB
HostAgent["HostAgent - Orchestrates all agents - Registers third-party agents as selectable targets - Routes tasks to appropriate agents"]
AppAgent["AppAgent (GUI tasks)"]
HardwareAgent["HardwareAgent (Hardware)"]
YourAgent["YourAgent (Custom)"]
Strategies["Processing Strategies - LLM Interaction - Action Execution - Memory Updates"]
HostAgent --> AppAgent
HostAgent --> HardwareAgent
HostAgent --> YourAgent
AppAgent --> Strategies
HardwareAgent --> Strategies
YourAgent --> Strategies
style HostAgent fill:#e1f5ff,stroke:#0288d1,stroke-width:2px
style AppAgent fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px
style HardwareAgent fill:#fff3e0,stroke:#ff9800,stroke-width:2px
style YourAgent fill:#e8f5e9,stroke:#4caf50,stroke-width:2px
style Strategies fill:#fce4ec,stroke:#e91e63,stroke-width:2px
```
### Agent Registry System
UFO² uses a registry pattern to dynamically load and manage agents:
```python
@AgentRegistry.register(
agent_name="YourAgent", # Unique identifier
third_party=True, # Mark as third-party
processor_cls=YourProcessor # Processing logic
)
class YourAgent(CustomizedAgent):
"""Your custom agent implementation."""
pass
```
**How it works**:
1. **Registration**: `@AgentRegistry.register()` decorator registers your agent class
2. **Filtering**: Registry checks if agent is in `ENABLED_THIRD_PARTY_AGENTS` config
3. **Instantiation**: HostAgent creates instances when needed
4. **Target Selection**: LLM can select your agent as an execution target
---
## Step-by-Step Implementation
### Step 1: Create Agent Class
Create your agent class by inheriting from `CustomizedAgent`:
```
/* Detailed source-code truncated for AI context efficiency. */
```
**Key Points**:
- ✅ **Inherit from `CustomizedAgent`**: Provides base functionality
- ✅ **Use `@AgentRegistry.register()`**: Enables dynamic loading
- ✅ **Set `third_party=True`**: Triggers configuration filtering
- ✅ **Specify `processor_cls`**: Links to your processing logic
---
### Step 2: Create Processor Class
Create a processor that defines how your agent processes tasks. For detailed information about processors and strategies, see [Agent Architecture](../infrastructure/agents/overview.md).
```
/* Detailed source-code truncated for AI context efficiency. */
```
**Strategy Setup Guidelines**:
| Phase | Purpose | fail_fast | Strategy Options |
|-------|---------|-----------|------------------|
| **DATA_COLLECTION** | Capture observations | `True` | Screenshot, sensor data, API calls |
| **LLM_INTERACTION** | Get LLM decision | `True` | Custom prompts, function calling |
| **ACTION_EXECUTION** | Execute action | `False` | Custom tools, API calls, commands |
| **MEMORY_UPDATE** | Save history | `False` | Standard or custom memory logic |
---
### Step 3: Create Custom Strategies (Optional)
If you need custom processing logic, create strategy classes:
```
/* Detailed source-code truncated for AI context efficiency. */
```
**When to Create Custom Strategies**:
- ✅ Need domain-specific action execution (e.g., hardware APIs)
- ✅ Special LLM interaction patterns (e.g., multi-turn dialogs)
- ✅ Custom data collection (e.g., sensor readings, external APIs)
- ❌ Standard GUI automation (use default strategies)
---
### Step 4: Create Prompt Templates
Create YAML prompt templates to guide your agent's LLM interactions:
```yaml
# File: ufo/prompts/third_party/your_agent.yaml
system: |
You are YourAgent, a specialized AI agent that handles [specific domain tasks].
Your capabilities include:
- [Capability 1]: Description
- [Capability 2]: Description
- [Capability 3]: Description
You have access to the following tools:
{apis}
Guidelines:
1. Analyze the user's request carefully
2. Select the most appropriate tool for the task
3. Provide clear reasoning for your decisions
4. Handle errors gracefully
Available actions:
- your_action_1: Description and usage
- your_action_2: Description and usage
- finish: Complete the task
user: |
## Previous Actions
{previous_actions}
## Current Task
User Request: {request}
## Available Tools
{tool_list}
## Instructions
Based on the above information:
1. Analyze what needs to be done
2. Select the appropriate action
3. Provide the action parameters
Respond with:
- Thought: Your reasoning
- Action: The action to take
- Arguments: Parameters for the action
```
```yaml
# File: ufo/prompts/third_party/your_agent_example.yaml
example_1: |
User Request: [Example request]
Thought: [Agent's reasoning]
Action: your_action_1
Arguments:
param1: value1
param2: value2
example_2: |
User Request: [Another example]
Thought: [Agent's reasoning]
Action: finish
Arguments:
summary: Task completed successfully
```
**Prompt Design Best Practices**:
- ✅ **Clear role definition**: Explain what your agent does
- ✅ **Tool descriptions**: List available actions with usage
- ✅ **Examples**: Provide concrete examples of interactions
- ✅ **Error handling**: Include guidance for error scenarios
- ✅ **Output format**: Specify expected response structure
---
## Complete Example: HardwareAgent
Let's examine the complete implementation of **HardwareAgent** as a reference:
### Agent Class
```python
# File: ufo/agents/agent/customized_agent.py
@AgentRegistry.register(
agent_name="HardwareAgent",
third_party=True,
processor_cls=HardwareAgentProcessor
)
class HardwareAgent(CustomizedAgent):
"""
HardwareAgent is a specialized agent that interacts with hardware components.
It extends CustomizedAgent to provide additional functionality specific to hardware.
Use cases:
- Robotic arm control for keyboard/mouse input
- USB device plug/unplug automation
- Physical hardware testing
- Sensor data collection
"""
pass # Inherits all functionality from CustomizedAgent
```
**Why so simple?**
- ✅ **Inheritance**: Gets all functionality from `CustomizedAgent`
- ✅ **Composition**: Custom logic goes in the Processor
- ✅ **Separation of Concerns**: Agent defines "what", Processor defines "how"
---
### Processor Class
```python
# File: ufo/agents/processors/customized/customized_agent_processor.py
class HardwareAgentProcessor(CustomizedProcessor):
"""
Processor for Hardware Agent.
Handles hardware-specific processing logic including:
- Visual mode for screenshot understanding
- Custom action execution for hardware APIs
- Hardware-specific error handling
"""
pass # Uses default strategy setup from CustomizedProcessor
```
**Default Strategy Setup**:
```python
# From CustomizedProcessor._setup_strategies()
def _setup_strategies(self) -> None:
# Data collection with screenshots
self.strategies[ProcessingPhase.DATA_COLLECTION] = (
CustomizedScreenshotCaptureStrategy(fail_fast=True)
)
# LLM interaction with custom prompts
self.strategies[ProcessingPhase.LLM_INTERACTION] = (
CustomizedLLMInteractionStrategy(fail_fast=True)
)
# Action execution using standard tools
self.strategies[ProcessingPhase.ACTION_EXECUTION] = (
AppActionExecutionStrategy(fail_fast=False)
)
# Memory updates
self.strategies[ProcessingPhase.MEMORY_UPDATE] = (
AppMemoryUpdateStrategy(fail_fast=False)
)
```
---
### Configuration
```yaml
# File: config/ufo/third_party.yaml
ENABLED_THIRD_PARTY_AGENTS: ["HardwareAgent"]
THIRD_PARTY_AGENT_CONFIG:
HardwareAgent:
# Enable visual mode for screenshot understanding
VISUAL_MODE: True
# Agent identifier (must match @AgentRegistry.register name)
AGENT_NAME: "HardwareAgent"
# Prompt templates
APPAGENT_PROMPT: "ufo/prompts/share/base/app_agent.yaml"
APPAGENT_EXAMPLE_PROMPT: "ufo/prompts/examples/visual/app_agent_example.yaml"
API_PROMPT: "ufo/prompts/third_party/hardware_agent_api.yaml"
# Description for LLM context
INTRODUCTION: "The HardwareAgent is used to manipulate hardware components of the computer without using GUI, such as robotic arms for keyboard input and mouse control, plug and unplug devices such as USB drives, and other hardware-related tasks."
```
**Configuration Fields**:
| Field | Required | Description |
|-------|----------|-------------|
| `VISUAL_MODE` | Optional | Enable screenshot-based reasoning |
| `AGENT_NAME` | **Required** | Must match registry name exactly |
| `APPAGENT_PROMPT` | **Required** | Main prompt template path |
| `APPAGENT_EXAMPLE_PROMPT` | **Required** | Example prompt template path |
| `API_PROMPT` | Optional | API/tool description prompt |
| `INTRODUCTION` | **Required** | Agent description for LLM |
---
## Registering with HostAgent
### How HostAgent Discovers Third-Party Agents
The registration process is automatic through the Agent Registry system:
```
/* Detailed source-code truncated for AI context efficiency. */
```
**Target Registry Flow**:
```
1. HostAgent starts processing
↓
2. _register_applications_and_agents() called
↓
3. _register_third_party_agents() called
↓
4. Read ENABLED_THIRD_PARTY_AGENTS from config
↓
5. Create TargetInfo for each agent
↓
6. Register in TargetRegistry
↓
7. LLM can now select third-party agents as targets
```
### LLM Target Selection
When HostAgent presents targets to the LLM:
```json
{
"available_targets": [
{"id": "1", "name": "Microsoft Word", "kind": "APPLICATION"},
{"id": "2", "name": "Google Chrome", "kind": "APPLICATION"},
{"id": "3", "name": "HardwareAgent", "kind": "THIRD_PARTY_AGENT"},
{"id": "4", "name": "LinuxAgent", "kind": "THIRD_PARTY_AGENT"}
]
}
```
The LLM selects a target based on the task:
```json
{
"thought": "Need to control physical hardware for USB operations",
"selected_target": "3", // HardwareAgent
"action": "delegate_to_agent"
}
```
### Agent Instantiation
When LLM selects your agent, HostAgent creates an instance:
```python
# File: ufo/agents/agent/host_agent.py
@staticmethod
def create_agent(agent_type: str, *args, **kwargs) -> BasicAgent:
"""
Create an agent based on the given type.
"""
if agent_type == "host":
return HostAgent(*args, **kwargs)
elif agent_type == "app":
return AppAgent(*args, **kwargs)
elif agent_type in AgentRegistry.list_agents():
# Third-party agents are retrieved from registry
return AgentRegistry.get(agent_type)(*args, **kwargs)
else:
raise ValueError("Invalid agent type: {}".format(agent_type))
```
**Instantiation Flow**:
```
1. LLM selects "HardwareAgent"
↓
2. HostAgent calls create_agent("HardwareAgent")
↓
3. AgentRegistry.get("HardwareAgent") retrieves class
↓
4. Class instantiated with config parameters
↓
5. Agent executes task
↓
6. Results returned to HostAgent
```
---
## Configuration and Deployment
### Step 1: Enable Your Agent
Edit `config/ufo/third_party.yaml`:
```yaml
ENABLED_THIRD_PARTY_AGENTS: ["YourAgent"]
THIRD_PARTY_AGENT_CONFIG:
YourAgent:
VISUAL_MODE: False # Set True if using screenshots
AGENT_NAME: "YourAgent"
APPAGENT_PROMPT: "ufo/prompts/third_party/your_agent.yaml"
APPAGENT_EXAMPLE_PROMPT: "ufo/prompts/third_party/your_agent_example.yaml"
INTRODUCTION: "YourAgent handles [specific tasks] by [method]. Use this agent when you need to [use case]."
```
**Configuration Checklist**:
- ✅ Add agent name to `ENABLED_THIRD_PARTY_AGENTS`
- ✅ Create config block under `THIRD_PARTY_AGENT_CONFIG`
- ✅ Set `AGENT_NAME` to match registry name
- ✅ Provide paths to prompt templates
- ✅ Write clear `INTRODUCTION` for LLM context
---
### Step 2: Add Prompt Templates
Create your prompt files:
```
ufo/prompts/third_party/
├── your_agent.yaml # Main prompt template
└── your_agent_example.yaml # Example interactions
```
**Template Requirements**:
- ✅ Define agent role and capabilities
- ✅ List available actions/tools
- ✅ Provide clear output format
- ✅ Include error handling guidance
- ✅ Add concrete examples
---
### Step 3: Test Configuration
Test that your agent loads correctly:
```python
# test_your_agent.py
from config.config_loader import get_ufo_config
from ufo.agents.agent.basic import AgentRegistry
def test_agent_registration():
"""Test that YourAgent is registered correctly."""
config = get_ufo_config()
# Check if agent is enabled
assert "YourAgent" in config.system.enabled_third_party_agents
print("✅ Agent is enabled in config")
# Check if agent is registered
registered_agents = AgentRegistry.list_agents()
assert "YourAgent" in registered_agents
print("✅ Agent is registered in AgentRegistry")
# Test agent instantiation
agent_cls = AgentRegistry.get("YourAgent")
agent_config = config.system.third_party_agent_config["YourAgent"]
agent = agent_cls(
name="test_agent",
main_prompt=agent_config["APPAGENT_PROMPT"],
example_prompt=agent_config["APPAGENT_EXAMPLE_PROMPT"],
)
print(f"✅ Agent instantiated: {agent}")
# Check processor
assert hasattr(agent, "_processor_cls")
print(f"✅ Processor registered: {agent._processor_cls}")
if __name__ == "__main__":
test_agent_registration()
```
Run test:
```powershell
python test_your_agent.py
```
---
### Step 4: Integration Testing
Test your agent in a full UFO² session:
```python
# integration_test.py
from ufo.agents.agent.host_agent import HostAgent
from config.config_loader import get_ufo_config
def test_agent_selection():
"""Test that HostAgent can discover and select YourAgent."""
config = get_ufo_config()
# Create HostAgent
host_agent = HostAgent(
name="host",
is_visual=True,
main_prompt="ufo/prompts/share/base/host_agent.yaml",
example_prompt="ufo/prompts/examples/visual/host_agent_example.yaml",
api_prompt="ufo/prompts/share/base/api.yaml",
)
# Verify third-party agents are in target registry
# (This happens during HostAgent processing)
print("✅ HostAgent created successfully")
print(f"Enabled third-party agents: {config.system.enabled_third_party_agents}")
if __name__ == "__main__":
test_agent_selection()
```
---
## Best Practices
### Code Organization
```
ufo/
├── agents/
│ ├── agent/
│ │ └── customized_agent.py # Agent classes
│ └── processors/
│ ├── customized/
│ │ └── customized_agent_processor.py # Processors
│ └── strategies/
│ └── your_agent_strategy.py # Custom strategies
├── prompts/
│ └── third_party/
│ ├── your_agent.yaml # Main prompt
│ └── your_agent_example.yaml # Examples
config/
└── ufo/
└── third_party.yaml # Configuration
```
**Organization Guidelines**:
- ✅ **Agent classes** → `ufo/agents/agent/customized_agent.py`
- ✅ **Processors** → `ufo/agents/processors/customized/`
- ✅ **Custom strategies** → `ufo/agents/processors/strategies/`
- ✅ **Prompts** → `ufo/prompts/third_party/`
- ✅ **Configuration** → `config/ufo/third_party.yaml`
---
### Naming Conventions
| Component | Naming Pattern | Example |
|-----------|----------------|---------|
| Agent Class | `{Name}Agent` | `HardwareAgent`, `WebAgent` |
| Processor Class | `{Name}AgentProcessor` | `HardwareAgentProcessor` |
| Strategy Class | `{Name}{Phase}Strategy` | `HardwareActionExecutionStrategy` |
| Registry Name | Same as class (no suffix) | `"HardwareAgent"` |
| Config Key | Same as registry name | `HardwareAgent:` |
---
### Error Handling
Implement robust error handling in your strategies:
```python
async def execute(self, agent, context) -> ProcessingResult:
try:
# Main execution logic
result = await self._do_work(agent, context)
return ProcessingResult(
success=True,
data=result,
error=None
)
except SpecificError as e:
# Handle expected errors gracefully
self.logger.warning(f"Expected error: {e}")
return ProcessingResult(
success=False,
data={"partial_result": "..."},
error=f"Recoverable error: {str(e)}"
)
except Exception as e:
# Log unexpected errors
self.logger.error(f"Unexpected error: {e}", exc_info=True)
if self.fail_fast:
raise # Re-raise if configured to fail fast
return ProcessingResult(
success=False,
data={},
error=f"Fatal error: {str(e)}"
)
```
**Error Handling Guidelines**:
- ✅ Use `ProcessingResult` to communicate outcomes
- ✅ Log errors at appropriate levels (warning/error)
- ✅ Respect `fail_fast` setting
- ✅ Provide actionable error messages
- ✅ Return partial results when possible
---
### Logging
Use structured logging throughout your agent:
```python
import logging
class YourAgentProcessor(CustomizedProcessor):
def __init__(self, agent, global_context):
super().__init__(agent, global_context)
self.logger = logging.getLogger(__name__)
async def execute(self, agent, context):
# Info: Normal operation flow
self.logger.info(f"Processing task: {context.get_local('task')}")
# Debug: Detailed debugging info
self.logger.debug(f"Context state: {context.get_all_local()}")
# Warning: Recoverable issues
self.logger.warning(f"Retrying action after failure")
# Error: Serious problems
self.logger.error(f"Action failed: {error}", exc_info=True)
```
**Logging Best Practices**:
- ✅ Use `self.logger` from base class
- ✅ Log at appropriate levels (debug/info/warning/error)
- ✅ Include context in log messages
- ✅ Use `exc_info=True` for exceptions
- ✅ Avoid logging sensitive data
---
### Testing
Create comprehensive tests for your agent:
```python
# tests/test_your_agent.py
import pytest
from ufo.agents.agent.customized_agent import YourAgent
from ufo.agents.processors.customized.customized_agent_processor import (
YourAgentProcessor
)
class TestYourAgent:
@pytest.fixture
def agent(self):
"""Create test agent instance."""
return YourAgent(
name="test_agent",
main_prompt="ufo/prompts/third_party/your_agent.yaml",
example_prompt="ufo/prompts/third_party/your_agent_example.yaml",
)
def test_agent_initialization(self, agent):
"""Test agent initializes correctly."""
assert agent.name == "test_agent"
assert agent.prompter is not None
def test_processor_registration(self, agent):
"""Test processor is registered."""
assert hasattr(agent, "_processor_cls")
assert agent._processor_cls == YourAgentProcessor
@pytest.mark.asyncio
async def test_action_execution(self, agent, mock_context):
"""Test action execution logic."""
processor = YourAgentProcessor(agent, mock_context)
result = await processor.execute_phase(
ProcessingPhase.ACTION_EXECUTION,
agent,
mock_context
)
assert result.success == True
```
**Test Coverage Checklist**:
- ✅ Agent initialization
- ✅ Processor registration
- ✅ Strategy execution
- ✅ Error handling
- ✅ Configuration loading
- ✅ Integration with HostAgent
---
## Troubleshooting
### Issue 1: Agent Not Registered
!!!bug "Error Message"
```
ValueError: No agent class registered under 'YourAgent'
```
**Diagnosis**: Agent is not enabled in configuration or decorator is missing.
**Solutions**:
1. Check configuration:
```yaml
# config/ufo/third_party.yaml
ENABLED_THIRD_PARTY_AGENTS: ["YourAgent"] # ← Must include your agent
```
2. Verify decorator:
```python
@AgentRegistry.register(
agent_name="YourAgent", # ← Must match config
third_party=True, # ← Must be True
processor_cls=YourAgentProcessor
)
class YourAgent(CustomizedAgent):
pass
```
3. Check import:
```python
# Ensure your agent module is imported
# In ufo/agents/agent/__init__.py or customized_agent.py
from ufo.agents.agent.customized_agent import YourAgent
```
---
### Issue 2: Prompt Files Not Found
!!!bug "Error Message"
```
FileNotFoundError: ufo/prompts/third_party/your_agent.yaml
```
**Diagnosis**: Prompt template files don't exist or paths are incorrect.
**Solutions**:
1. Create prompt files:
```powershell
# Create directory if needed
New-Item -ItemType Directory -Force -Path "ufo\prompts\third_party"
# Create prompt files
New-Item -ItemType File -Path "ufo\prompts\third_party\your_agent.yaml"
New-Item -ItemType File -Path "ufo\prompts\third_party\your_agent_example.yaml"
```
2. Verify paths in configuration:
```yaml
THIRD_PARTY_AGENT_CONFIG:
YourAgent:
APPAGENT_PROMPT: "ufo/prompts/third_party/your_agent.yaml"
APPAGENT_EXAMPLE_PROMPT: "ufo/prompts/third_party/your_agent_example.yaml"
```
3. Check file permissions:
```powershell
# Verify files are readable
Test-Path "ufo\prompts\third_party\your_agent.yaml"
```
---
### Issue 3: Agent Not Appearing in Target List
!!!bug "Symptom"
HostAgent doesn't show your third-party agent as a selectable target.
**Diagnosis**: Agent is registered but not appearing in TargetRegistry.
**Solutions**:
1. Check enabled agents:
```python
from config.config_loader import get_ufo_config
config = get_ufo_config()
print(config.system.enabled_third_party_agents)
# Should include "YourAgent"
```
2. Verify TargetKind:
```python
# In your registration code
TargetInfo(
kind=TargetKind.THIRD_PARTY_AGENT.value, # ← Correct kind
name=agent_name,
)
```
3. Check HostAgent logs:
```
[INFO] Registered 2 third-party agents
```
4. Test target registry directly:
```python
from ufo.agents.processors.schemas.target import TargetRegistry, TargetKind
registry = TargetRegistry()
targets = registry.get_by_kind(TargetKind.THIRD_PARTY_AGENT)
print(targets) # Should include your agent
```
---
### Issue 4: Processor Not Executing
!!!bug "Symptom"
Agent instantiates but processor strategies don't execute.
**Diagnosis**: Processor class not properly linked or strategies not set up.
**Solutions**:
1. Verify processor_cls in decorator:
```python
@AgentRegistry.register(
agent_name="YourAgent",
third_party=True,
processor_cls=YourAgentProcessor # ← Must be specified
)
```
2. Check processor initialization:
```python
class YourAgentProcessor(CustomizedProcessor):
def __init__(self, agent, global_context):
super().__init__(agent, global_context) # ← Must call super
# Your custom init
```
3. Verify strategy setup:
```python
def _setup_strategies(self) -> None:
# Must populate self.strategies dict
self.strategies[ProcessingPhase.LLM_INTERACTION] = ...
```
4. Check processor is created:
```python
# In your test
assert hasattr(agent, "_processor_cls")
processor = agent._processor_cls(agent, global_context)
assert processor is not None
```
---
### Issue 5: LLM Not Selecting Your Agent
!!!bug "Symptom"
Agent is registered but LLM never selects it.
**Diagnosis**: Agent description unclear or not suitable for user requests.
**Solutions**:
1. Improve `INTRODUCTION`:
```yaml
INTRODUCTION: "Use YourAgent when you need to [clear use case]. It provides [specific capabilities] through [method]. Examples: [concrete examples]."
```
2. Add clear examples in prompt:
```yaml
# your_agent_example.yaml
example_1: |
User: [Clear example request]
Agent: [Clear example response]
```
3. Test with explicit requests:
```python
# Test with request that clearly needs your agent
user_request = "Use YourAgent to [specific task]"
```
4. Check HostAgent prompt includes your agent:
```
Available targets:
- YourAgent: [Your INTRODUCTION text should appear here]
```
---
## Advanced Topics
### Multi-MCP Integration
Integrate multiple MCP servers with your agent:
```yaml
# config/ufo/agent_mcp.yaml
YourAgent:
mcp_servers:
hardware_control:
type: "local"
module: "your_package.hardware_mcp"
config:
device_port: "/dev/ttyUSB0"
data_collection:
type: "http"
url: "http://localhost:8080/mcp"
config:
api_key: "${SENSOR_API_KEY}"
```
See [Creating Custom MCP Servers](./creating_mcp_servers.md) for details.
---
### State Management
Maintain agent state across invocations:
```python
from ufo.agents.memory.blackboard import Blackboard
class YourAgent(CustomizedAgent):
def __init__(self, name, main_prompt, example_prompt):
super().__init__(name, main_prompt, example_prompt)
# Use blackboard for persistent state
self._blackboard = Blackboard()
@property
def blackboard(self) -> Blackboard:
return self._blackboard
def save_state(self, key: str, value: Any):
"""Save state to blackboard."""
self.blackboard.add_entry(key, value)
def load_state(self, key: str) -> Any:
"""Load state from blackboard."""
return self.blackboard.get_entry(key)
```
---
### Custom Prompter
Create a custom prompter for specialized LLM interactions:
```python
from ufo.prompter.app_prompter import AppPrompter
class YourAgentPrompter(AppPrompter):
"""Custom prompter for YourAgent."""
def user_content_construction(
self,
prev_plan: List[str],
user_request: str,
retrieved_docs: str,
last_success_actions: List[Dict],
**kwargs
) -> List[Dict[str, str]]:
"""
Construct custom user message content.
"""
# Add custom context
custom_context = self._build_custom_context(**kwargs)
# Call parent method
base_content = super().user_content_construction(
prev_plan=prev_plan,
user_request=user_request,
retrieved_docs=retrieved_docs,
last_success_actions=last_success_actions
)
# Insert custom content
base_content.insert(0, {
"type": "text",
"text": custom_context
})
return base_content
```
Use custom prompter in your agent:
```python
class YourAgent(CustomizedAgent):
def get_prompter(self, is_visual, main_prompt, example_prompt):
return YourAgentPrompter(main_prompt, example_prompt)
```
---
## Related Documentation
- **[Third-Party Agent Configuration](../configuration/system/third_party_config.md)** - Configuration reference
- **[Agent Configuration](../configuration/system/agents_config.md)** - Core agent LLM settings
- **[Creating Custom MCP Servers](./creating_mcp_servers.md)** - MCP server development for custom tools
- **[Agent Architecture](../infrastructure/agents/overview.md)** - Understanding agent design patterns
- **[HostAgent Strategy](../ufo2/host_agent/strategy.md)** - Learn how HostAgent orchestrates third-party agents
- **[AppAgent Strategy](../ufo2/app_agent/strategy.md)** - Processing strategies reference
---
## Summary
**Key Takeaways:**
✅ **Third-party agents extend UFO²** with specialized capabilities
✅ **Use `@AgentRegistry.register()`** to register your agent
✅ **Create processor classes** to define processing logic
✅ **Configure in third_party.yaml** to enable your agent
✅ **HostAgent automatically discovers** enabled third-party agents
✅ **LLM selects agents** based on task requirements
✅ **Follow HardwareAgent** as a reference implementation
**Build powerful third-party agents to extend UFO²!** 🚀
---
## Quick Reference
### Minimal Agent Implementation
```python
# 1. Agent class
@AgentRegistry.register(
agent_name="MyAgent", third_party=True, processor_cls=MyProcessor
)
class MyAgent(CustomizedAgent):
pass
# 2. Processor class
class MyProcessor(CustomizedProcessor):
pass # Use default strategies
# 3. Configuration
# config/ufo/third_party.yaml
ENABLED_THIRD_PARTY_AGENTS: ["MyAgent"]
THIRD_PARTY_AGENT_CONFIG:
MyAgent:
AGENT_NAME: "MyAgent"
APPAGENT_PROMPT: "ufo/prompts/third_party/my_agent.yaml"
APPAGENT_EXAMPLE_PROMPT: "ufo/prompts/third_party/my_agent_example.yaml"
INTRODUCTION: "MyAgent handles [tasks]."
# 4. Prompt templates
# Create: ufo/prompts/third_party/my_agent.yaml
# Create: ufo/prompts/third_party/my_agent_example.yaml
```
**That's all you need to get started!** 🎉
---
### Documents Docs Ufo2 As Galaxy Device (`documents/docs/ufo2/as_galaxy_device.md`)
# UFO² as UFO³ Galaxy Device
Integrate **UFO² (Windows Desktop Automation Agent)** into the **UFO³ Galaxy framework** as a managed sub-agent device. This enables Galaxy to orchestrate complex cross-platform workflows combining Windows desktop automation with Linux server operations and other heterogeneous devices.
## Overview
UFO² can function as a **device agent** within the UFO³ Galaxy multi-tier orchestration framework. When configured as a Galaxy device, UFO² operates in **server-client mode**, allowing the Galaxy ConstellationAgent to:
- Dispatch Windows automation subtasks to UFO² devices
- Coordinate cross-platform workflows (Windows desktop + Linux servers)
- Leverage UFO²'s HostAgent and AppAgent capabilities at scale
- Manage multiple Windows devices from a unified control plane
- Dynamically select devices based on capabilities and installed applications
UFO² integration follows the **server-client architecture** pattern where the UFO² Server manages task orchestration and state machines, the UFO² Client executes Windows automation commands via MCP tools, and the Galaxy ConstellationAgent acts as the top-level orchestrator. Communication is enabled through the Agent Interaction Protocol (AIP). For detailed architecture information, see [Server-Client Architecture](../infrastructure/agents/server_client_architecture.md).
## Galaxy Integration Architecture
```mermaid
graph TB
User[User Request]
Galaxy[Galaxy ConstellationAgent Top-Level Orchestrator]
subgraph "Device Pool"
subgraph "Windows Devices (UFO²)"
Win1[UFO² Device 1 Office Desktop]
Win2[UFO² Device 2 Dev Workstation]
Win3[UFO² Device 3 Test Machine]
end
subgraph "Linux Devices"
Linux1[Linux Agent 1 Web Server]
Linux2[Linux Agent 2 Database Server]
end
subgraph "Other Devices"
Mobile1[Mobile Device]
Cloud1[Cloud Service]
end
end
User -->|Complex Cross-Platform Task| Galaxy
Galaxy -->|Windows Automation Subtask| Win1
Galaxy -->|Desktop Application Task| Win2
Galaxy -->|Testing Task| Win3
Galaxy -->|Server Management Task| Linux1
Galaxy -->|Database Query Task| Linux2
Galaxy -->|Mobile Automation| Mobile1
Galaxy -->|API Integration| Cloud1
style Galaxy fill:#ffe1e1
style Win1 fill:#e1f5ff
style Win2 fill:#e1f5ff
style Win3 fill:#e1f5ff
style Linux1 fill:#f0ffe1
style Linux2 fill:#f0ffe1
```
**Example Multi-Device Workflow:**
> **User Request:** "Generate a sales report from the database, create an Excel dashboard, and email it to the team"
**Galaxy orchestrates:**
1. **Linux DB Server**: Extract sales data from PostgreSQL → CSV export
2. **UFO² Desktop**: Open Excel, import CSV, create visualizations and pivot tables
3. **UFO² Desktop**: Open Outlook, compose email with Excel attachment
4. **UFO² Desktop**: Send email to distribution list
## Prerequisites
Before configuring UFO² as a Galaxy device, ensure you have:
| Component | Requirement | Verification |
|-----------|-------------|--------------|
| **UFO Repository** | Cloned and up-to-date | `git pull origin main` |
| **Python** | 3.10+ installed | `python --version` |
| **Dependencies** | All packages installed | `pip install -r requirements.txt` |
| **LLM Configuration** | API keys configured | Check `config/ufo/agents.yaml` |
| **Network** | Server-client connectivity | `ping ` |
| **Windows Machine** | UFO² will run here | Windows 10/11 |
### Configure Agent Configuration
**Before proceeding with Galaxy integration**, you must configure your agent settings in `config/ufo/agents.yaml`:
1. Copy the template file:
```powershell
Copy-Item config\ufo\agents.yaml.template config\ufo\agents.yaml
```
2. Configure your LLM provider (OpenAI, Azure OpenAI, etc.) and add API keys
Without proper agent configuration, UFO² cannot function as a Galaxy device. See [Agents Configuration Guide](../configuration/system/agents_config.md) for detailed setup instructions.
## Server-Client Mode Setup
UFO² **must** operate in **server-client mode** when integrated into Galaxy. This architecture separates orchestration (server) from execution (client), enabling Galaxy to manage multiple UFO² devices efficiently. Unlike standalone UFO² usage (local mode), Galaxy integration requires running UFO² in distributed server-client mode to ensure Galaxy can communicate with UFO² via Agent Interaction Protocol (AIP), multiple UFO² clients can be managed by a single server, task state is managed server-side for reliability, and clients remain stateless execution endpoints.
## Step 1: Start UFO² Server
The **UFO² Server** handles task orchestration, state management, and LLM-driven decision-making. It communicates with Galaxy and dispatches commands to UFO² clients.
### Basic Server Startup
Launch UFO² Server on the machine that will host the server (can be any Windows/Linux machine):
```powershell
python -m ufo.server.app --port 5000
```
**Expected Output:**
```console
2025-11-06 10:30:22 - ufo.server.app - INFO - Starting UFO Server on 0.0.0.0:5000
INFO: Started server process [12345]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)
```
Once you see "Uvicorn running", the server is ready at `ws://0.0.0.0:5000/ws`.
### Server Configuration Options
| Argument | Default | Description | Example |
|----------|---------|-------------|---------|
| `--port` | `5000` | Server listening port | `--port 5000` |
| `--host` | `0.0.0.0` | Bind address (0.0.0.0 = all interfaces) | `--host 192.168.1.100` |
| `--log-level` | `WARNING` | Logging verbosity | `--log-level DEBUG` |
| `--local` | `False` | Run server in local mode | `--local` |
**Examples:**
Specific port:
```powershell
python -m ufo.server.app --port 5000
```
Specific IP binding:
```powershell
python -m ufo.server.app --host 192.168.1.100 --port 5000
```
Debug mode:
```powershell
python -m ufo.server.app --port 5000 --log-level DEBUG
```
### Verify Server Health
```powershell
# Test server health endpoint
curl http://localhost:5000/api/health
```
**Expected Response:**
```json
{
"status": "healthy",
"online_clients": []
}
```
## Step 2: Start UFO² Client (Windows Machine)
The **UFO² Client** runs on the Windows machine where you want to perform desktop automation. It connects to the UFO² server via WebSocket and executes automation commands through MCP tools.
### Basic Client Startup
Connect UFO² Client to Server on the **Windows machine** where you want to run desktop automation:
```powershell
python -m ufo.client.client `
--ws `
--ws-server ws://192.168.1.100:5000/ws `
--client-id ufo2_desktop_1 `
--platform windows
```
**Note:** In PowerShell, use backtick `` ` `` for line continuation. In Command Prompt, use `^`.
### Client Parameters Explained
| Parameter | Required | Description | Example |
|-----------|----------|-------------|---------|
| `--ws` | ✅ Yes | Enable WebSocket mode | `--ws` |
| `--ws-server` | ✅ Yes | Server WebSocket URL | `ws://192.168.1.100:5000/ws` |
| `--client-id` | ✅ Yes | **Unique** device identifier | `ufo2_desktop_1` |
| `--platform` | ✅ Yes | Platform type (must be `windows` for UFO²) | `--platform windows` |
**Important:**
- `--client-id` must be globally unique - No two devices can share the same ID
- `--platform windows` is mandatory - Without this flag, UFO² won't work correctly
- Server address must be correct - Replace `192.168.1.100:5000` with your actual server IP and port
### Understanding the WebSocket URL
The `--ws-server` parameter format is:
```
ws://:/ws
```
Examples:
| Scenario | WebSocket URL | Description |
|----------|---------------|-------------|
| **Localhost** | `ws://localhost:5000/ws` | Server and client on same machine |
| **Same Network** | `ws://192.168.1.100:5000/ws` | Server on local network |
| **Remote Server** | `ws://203.0.113.50:5000/ws` | Server on internet (public IP) |
### Connection Success Indicators
**Client Logs:**
```log
INFO - Platform detected/specified: windows
INFO - UFO Client initialized for platform: windows
INFO - [WS] Connecting to ws://192.168.1.100:5000/ws (attempt 1/5)
INFO - [WS] [AIP] Successfully registered as ufo2_desktop_1
INFO - [WS] Heartbeat loop started (interval: 30s)
```
**Server Logs:**
```log
INFO - [WS] ✅ Registered device client: ufo2_desktop_1
INFO - [WS] Device ufo2_desktop_1 platform: windows
```
When you see "Successfully registered", the UFO² client is connected and ready to receive tasks.
### Verify Connection
```powershell
# Check connected clients on server
curl http://192.168.1.100:5000/api/clients
```
**Expected Response:**
```json
{
"clients": [
{
"client_id": "ufo2_desktop_1",
"type": "device",
"platform": "windows",
"connected_at": 1730899822.0,
"uptime_seconds": 45
}
]
}
```
## Step 3: Configure MCP Services
UFO² relies on **MCP (Model Context Protocol) servers** to provide Windows automation capabilities. Unlike Linux agents that may require separate HTTP MCP servers, UFO² MCP servers are primarily **local** and start automatically with the client.
UFO² uses **local MCP servers** that run in-process with the client:
- **UI Automation MCP**: Click, type, screenshot, control detection
- **File Operations MCP**: Read, write, copy, delete files
- **Application Control MCP**: Launch apps, switch windows, close processes
These are **automatically initialized** when the UFO² client starts.
### Default MCP Configuration
By default, UFO² client automatically starts all necessary **local MCP servers**. No additional configuration is required for standard Windows automation.
When you start the UFO² client, it automatically initializes UI automation tools, registers file operation handlers, configures application control interfaces, and sets up screenshot and OCR capabilities.
### Optional: HTTP MCP Server (Advanced)
For specialized scenarios requiring **remote MCP access** (e.g., hardware automation via external tools), you can optionally start HTTP-based MCP servers. However, note that there is no `windows_mcp_server.py` in the codebase. Available HTTP MCP servers are:
- `hardware_mcp_server.py` - For hardware-level operations
- `linux_mcp_server.py` - For Linux-specific operations
Start an HTTP MCP server if needed:
```powershell
python -m ufo.client.mcp.http_servers.hardware_mcp_server
```
**Note:** For standard Galaxy integration with UFO², local MCP servers are sufficient and HTTP MCP servers are not required.
## Step 4: Configure as Galaxy Device
To integrate UFO² into the Galaxy framework, register it in the Galaxy device configuration file.
### Device Configuration File
The Galaxy device pool is configured in `config/galaxy/devices.yaml`.
### Add UFO² Device Configuration
Edit `config/galaxy/devices.yaml` and add your UFO² device(s) under the `devices` section:
```yaml
devices:
- device_id: "ufo2_desktop_1"
server_url: "ws://192.168.1.100:5000/ws"
os: "windows"
capabilities:
- "desktop_automation"
- "office_applications"
- "web_browsing"
- "email"
- "file_management"
metadata:
os: "windows"
version: "11"
performance: "high"
installed_apps:
- "Microsoft Excel"
- "Microsoft Word"
- "Microsoft PowerPoint"
- "Microsoft Outlook"
- "Google Chrome"
- "Adobe Acrobat"
description: "Primary office workstation for document automation"
auto_connect: true
max_retries: 5
```
### Configuration Fields Explained
| Field | Required | Type | Description | Example |
|-------|----------|------|-------------|---------|
| `device_id` | ✅ Yes | string | **Must match client `--client-id`** | `"ufo2_desktop_1"` |
| `server_url` | ✅ Yes | string | **Must match server WebSocket URL** | `"ws://192.168.1.100:5000/ws"` |
| `os` | ✅ Yes | string | Operating system | `"windows"` |
| `capabilities` | ❌ Optional | list | Device capabilities (for task routing) | `["desktop_automation", "office"]` |
| `metadata` | ❌ Optional | dict | Custom metadata for task context | See below |
| `auto_connect` | ❌ Optional | boolean | Auto-connect on Galaxy startup | `true` |
| `max_retries` | ❌ Optional | integer | Connection retry attempts | `5` |
### Capabilities-Based Task Routing
Galaxy uses the `capabilities` field to intelligently route subtasks to appropriate UFO² devices. Define capabilities based on application categories (e.g., `"office_applications"`, `"web_browsing"`), task types (e.g., `"desktop_automation"`, `"data_entry"`), specific software (e.g., `"excel"`, `"outlook"`), and user workflows (e.g., `"email"`, `"reporting"`).
**Example capability configurations:**
**Office Workstation:**
```yaml
capabilities:
- "desktop_automation"
- "office_applications"
- "excel"
- "word"
- "powerpoint"
- "outlook"
- "email"
- "reporting"
```
**Web Development Machine:**
```yaml
capabilities:
- "desktop_automation"
- "web_browsing"
- "chrome"
- "visual_studio_code"
- "git"
- "development"
```
**Testing Workstation:**
```yaml
capabilities:
- "desktop_automation"
- "ui_testing"
- "web_browsing"
- "screenshot_comparison"
- "quality_assurance"
```
**Media Production:**
```yaml
capabilities:
- "desktop_automation"
- "media_editing"
- "photoshop"
- "premiere"
- "video_processing"
- "image_manipulation"
```
The `metadata` field provides **contextual information** that the LLM can use when generating automation commands.
**Metadata Examples:**
**Office Workstation Metadata:**
```yaml
metadata:
os: "windows"
version: "11"
performance: "high"
installed_apps:
- "Microsoft Excel"
- "Microsoft Word"
- "Microsoft Outlook"
- "Adobe Acrobat Reader"
default_paths:
documents: "C:\\Users\\user\\Documents"
downloads: "C:\\Users\\user\\Downloads"
desktop: "C:\\Users\\user\\Desktop"
email_account: "user@company.com"
description: "Primary office workstation"
```
**Development Workstation Metadata:**
```yaml
metadata:
os: "windows"
version: "11"
performance: "high"
installed_apps:
- "Visual Studio Code"
- "Google Chrome"
- "Git"
- "Node.js"
- "Python"
default_paths:
projects: "C:\\Users\\dev\\Projects"
repos: "C:\\Users\\dev\\Repos"
git_username: "developer"
description: "Development environment"
```
**Testing Workstation Metadata:**
```yaml
metadata:
os: "windows"
version: "10"
performance: "medium"
installed_apps:
- "Google Chrome"
- "Microsoft Edge"
- "Firefox"
- "Selenium"
test_data_path: "C:\\TestData"
screenshot_path: "C:\\Screenshots"
description: "Automated testing environment"
```
**How Metadata is Used:**
The LLM receives metadata in the system prompt, enabling context-aware automation:
```
System Context:
- Device: ufo2_desktop_1
- OS: Windows 11
- Installed Apps: Microsoft Excel, Microsoft Word, Microsoft Outlook
- Documents Path: C:\Users\user\Documents
User Request: "Create a new Excel spreadsheet and save it as Q4_Report.xlsx"
UFO² Output:
1. Launch Microsoft Excel
2. Create new workbook
3. Save as C:\Users\user\Documents\Q4_Report.xlsx
```
## Step 5: Multiple UFO² Devices Configuration
Galaxy can manage **multiple UFO² devices** simultaneously, enabling parallel Windows automation across different machines.
**Multi-Device Galaxy Configuration Example:**
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Step 6: Launch Galaxy with UFO² Devices
Once all components are configured, launch Galaxy to begin orchestrating multi-device workflows.
### Prerequisites Checklist
Ensure all components are running **before** starting Galaxy:
1. ✅ **UFO² Server(s)** running on configured ports
2. ✅ **UFO² Client(s)** connected to their respective servers
3. ✅ **MCP Services** initialized (automatic with UFO² client)
4. ✅ **LLM configured** in `config/ufo/agents.yaml`
5. ✅ **Network connectivity** between all components
### Launch Sequence
**Step 1: Start all UFO² Servers**
```powershell
# On first Windows machine (192.168.1.100)
python -m ufo.server.app --port 5000
# On second Windows machine (192.168.1.101)
python -m ufo.server.app --port 5001
# On third Windows machine (192.168.1.102)
python -m ufo.server.app --port 5002
```
**Step 2: Start all UFO² Clients**
```powershell
# On first Windows desktop
python -m ufo.client.client `
--ws `
--ws-server ws://192.168.1.100:5000/ws `
--client-id ufo2_office_1 `
--platform windows
# On second Windows desktop
python -m ufo.client.client `
--ws `
--ws-server ws://192.168.1.101:5001/ws `
--client-id ufo2_office_2 `
--platform windows
# On development workstation
python -m ufo.client.client `
--ws `
--ws-server ws://192.168.1.102:5002/ws `
--client-id ufo2_dev_1 `
--platform windows
```
**Step 3: Launch Galaxy**
```powershell
# On your control machine (interactive mode)
python -m galaxy --interactive
```
**Or launch with a specific request:**
```powershell
python -m galaxy "Your task description here"
```
Galaxy will automatically connect to all configured UFO² devices (based on `config/galaxy/devices.yaml`) and display the orchestration interface.
## Example Multi-Device Workflows
### Workflow 1: Cross-Platform Report Generation
**User Request:**
> "Generate a weekly sales report: extract data from PostgreSQL, create Excel dashboard, and email to management"
**Galaxy Orchestration:**
```mermaid
sequenceDiagram
participant User
participant Galaxy
participant LinuxDB as Linux DB Server
participant UFO2 as UFO² Desktop
User->>Galaxy: Request sales report
Galaxy->>Galaxy: Decompose task
Note over Galaxy,LinuxDB: Subtask 1: Extract data
Galaxy->>LinuxDB: "Export sales data from PostgreSQL to CSV"
LinuxDB->>LinuxDB: Execute SQL query
LinuxDB->>LinuxDB: Generate CSV file
LinuxDB-->>Galaxy: CSV file location
Note over Galaxy,UFO2: Subtask 2: Create Excel report
Galaxy->>UFO2: "Create Excel dashboard from CSV"
UFO2->>UFO2: Open Excel
UFO2->>UFO2: Import CSV data
UFO2->>UFO2: Create pivot tables
UFO2->>UFO2: Add charts and formatting
UFO2-->>Galaxy: Excel file created
Note over Galaxy,UFO2: Subtask 3: Send email
Galaxy->>UFO2: "Email report to management"
UFO2->>UFO2: Open Outlook
UFO2->>UFO2: Compose email with attachment
UFO2->>UFO2: Send email
UFO2-->>Galaxy: Email sent
Galaxy-->>User: Task completed
```
### Workflow 2: Parallel Document Processing
**User Request:**
> "Process all invoices in the shared folder: convert PDFs to Excel, categorize by vendor, and summarize totals"
**Galaxy Orchestration:**
1. **UFO² Desktop 1**: Process invoices A-M (parallel batch 1)
2. **UFO² Desktop 2**: Process invoices N-Z (parallel batch 2)
3. **UFO² Desktop 1**: Consolidate results into master Excel file
4. **UFO² Desktop 1**: Generate summary report
5. **UFO² Desktop 1**: Send notification email
### Workflow 3: Development Workflow Automation
**User Request:**
> "Pull latest code, run tests, and create deployment package"
**Galaxy Orchestration:**
1. **UFO² Dev Workstation**: Open VS Code, pull from Git repository
2. **UFO² Dev Workstation**: Run automated tests, capture results
3. **Linux Build Server**: Build deployment package
4. **UFO² Dev Workstation**: Open browser, upload to staging server
5. **UFO² Desktop**: Send deployment notification email
---
## Task Assignment Behavior
### How Galaxy Routes Tasks to UFO² Devices
Galaxy's ConstellationAgent uses several factors to select the appropriate UFO² device for each subtask:
| Factor | Description | Example |
|--------|-------------|---------|
| **Capabilities** | Match subtask requirements to device capabilities | `"excel"` → Office workstation |
| **OS Requirement** | Platform-specific tasks routed to correct OS | Windows automation → UFO² devices |
| **Metadata Context** | Use device-specific apps and configurations | Email task → device with Outlook |
| **Device Status** | Only assign to online, healthy devices | Skip offline or failing devices |
| **Load Balancing** | Distribute tasks across similar devices | Round-robin across office desktops |
### Example Task Decomposition
**User Request:**
> "Prepare quarterly financial reports and distribute to stakeholders"
**Galaxy Decomposition:**
```yaml
Task 1:
Description: "Extract financial data from database"
Target: linux_db_server
Reason: Has "database_server" capability
Task 2:
Description: "Create Excel financial dashboard"
Target: ufo2_office_1
Reason: Has "excel" capability, device is idle
Task 3:
Description: "Generate PowerPoint presentation"
Target: ufo2_office_2
Reason: Has "powerpoint" capability
Task 4:
Description: "Email reports to stakeholders"
Target: ufo2_office_1
Reason: Has "outlook" and "email" capabilities
```
## Critical Configuration Requirements
!!!danger "Configuration Validation Checklist"
Ensure these match **exactly** or Galaxy cannot control the UFO² device:
**Device ID Match:**
- In `devices.yaml`: `device_id: "ufo2_desktop_1"`
- In client command: `--client-id ufo2_desktop_1`
**Server URL Match:**
- In `devices.yaml`: `server_url: "ws://192.168.1.100:5000/ws"`
- In client command: `--ws-server ws://192.168.1.100:5000/ws`
**Platform Specification:**
- Must include `--platform windows` for UFO² devices
## Monitoring & Debugging
### Verify Device Registration
Check if clients are connected to UFO² server:
```powershell
curl http://192.168.1.100:5000/api/clients
```
**Expected response:**
```json
{
"online_clients": [
{
"client_id": "ufo2_office_1",
"type": "device",
"platform": "windows",
"connected_at": 1730899822.0,
"uptime_seconds": 45
},
{
"client_id": "ufo2_office_2",
"type": "device",
"platform": "windows",
"connected_at": 1730899850.0,
"uptime_seconds": 17
}
]
}
```
### View Task Assignments
Galaxy logs show task routing decisions:
```log
INFO - [Galaxy] Task decomposition: 3 subtasks created
INFO - [Galaxy] Subtask 1 → linux_db_server (capability match: database_server)
INFO - [Galaxy] Subtask 2 → ufo2_office_1 (capability match: excel)
INFO - [Galaxy] Subtask 3 → ufo2_office_1 (capability match: email)
```
### Troubleshooting Device Connection
**Issue**: UFO² device not appearing in Galaxy device pool
**Diagnosis:**
1. Check if client is connected to server:
```powershell
curl http://192.168.1.100:5000/api/clients
```
2. Verify `devices.yaml` configuration matches client parameters
3. Check Galaxy logs for connection errors
4. Ensure `auto_connect: true` in `devices.yaml`
5. Verify UFO² server is running and accessible
## Common Issues & Troubleshooting
### Issue 1: UFO² Client Cannot Connect to Server
!!!bug "Error: Connection Refused"
**Symptoms:**
```log
ERROR - [WS] Failed to connect to ws://192.168.1.100:5000/ws
Connection refused
```
**Diagnosis Checklist:**
- [ ] Is the UFO² server running? (`curl http://192.168.1.100:5000/api/health`)
- [ ] Is the port correct? (Check server startup logs)
- [ ] Can client reach server IP? (`ping 192.168.1.100`)
- [ ] Is Windows Firewall blocking port 5000?
- [ ] Is the WebSocket URL correct? (should start with `ws://`)
**Solutions:**
**Verify Server:**
```powershell
# On server machine
curl http://localhost:5000/api/health
# From client machine
curl http://192.168.1.100:5000/api/health
```
**Check Network:**
```powershell
# Test connectivity
ping 192.168.1.100
# Test port accessibility (requires telnet client)
Test-NetConnection -ComputerName 192.168.1.100 -Port 5000
```
**Check Windows Firewall:**
```powershell
# Allow port through firewall
New-NetFirewallRule -DisplayName "UFO Server" `
-Direction Inbound `
-LocalPort 5000 `
-Protocol TCP `
-Action Allow
```
### Issue 2: Missing `--platform windows` Flag
!!!bug "Error: Incorrect Agent Type"
**Symptoms:**
- Client connects but cannot execute Windows automation
- Server logs show wrong platform type
- Tasks fail with "unsupported operation" errors
**Cause:**
Forgot to add `--platform windows` flag when starting the client.
**Solution:**
```powershell
# Wrong (missing platform)
python -m ufo.client.client --ws --client-id ufo2_desktop_1
# Correct
python -m ufo.client.client `
--ws `
--client-id ufo2_desktop_1 `
--platform windows
```
### Issue 3: Duplicate Client ID
!!!bug "Error: Registration Failed"
**Symptoms:**
```log
ERROR - [WS] Registration failed: client_id already exists
ERROR - Another device is using ID 'ufo2_desktop_1'
```
**Cause:**
Multiple UFO² clients trying to use the same `client_id`.
**Solutions:**
1. **Use unique client IDs:**
```powershell
# Device 1
--client-id ufo2_desktop_1
# Device 2
--client-id ufo2_desktop_2
# Device 3
--client-id ufo2_dev_1
```
2. **Check currently connected clients:**
```powershell
curl http://192.168.1.100:5000/api/clients
```
### Issue 4: Galaxy Cannot Find UFO² Device
!!!bug "Error: Device Not Configured"
**Symptoms:**
```log
ERROR - Device 'ufo2_desktop_1' not found in configuration
WARNING - Cannot dispatch task to unknown device
```
**Cause:**
Mismatch between `devices.yaml` configuration and actual client setup.
**Diagnosis:**
Check that these match **exactly**:
| Location | Field | Example |
|----------|-------|---------|
| `devices.yaml` | `device_id` | `"ufo2_desktop_1"` |
| Client command | `--client-id` | `ufo2_desktop_1` |
| `devices.yaml` | `server_url` | `"ws://192.168.1.100:5000/ws"` |
| Client command | `--ws-server` | `ws://192.168.1.100:5000/ws` |
**Solution:**
Update `devices.yaml` to match your client configuration, or vice versa.
### Issue 5: MCP Tools Not Available
!!!bug "Error: Tool Execution Failed"
**Symptoms:**
```log
ERROR - MCP tool 'click' not found
ERROR - Cannot execute Windows automation command
```
**Diagnosis:**
- [ ] Is UFO² client running properly?
- [ ] Are local MCP servers initialized?
- [ ] Check client startup logs for MCP initialization errors
**Solution:**
Restart UFO² client and verify MCP initialization:
```powershell
python -m ufo.client.client `
--ws `
--ws-server ws://192.168.1.100:5000/ws `
--client-id ufo2_desktop_1 `
--platform windows
```
Look for:
```log
INFO - MCP servers initialized: ui_automation, file_operations, app_control
INFO - UFO Client ready with 15 available tools
```
---
## Comparison with Standalone UFO²
| Aspect | Standalone UFO² | UFO² as Galaxy Device |
|--------|----------------|----------------------|
| **Architecture** | Local mode (single process) | Server-client mode (distributed) |
| **Control** | Direct user interaction | Galaxy orchestration |
| **Multi-Device** | Single device only | Multiple UFO² devices |
| **Cross-Platform** | Windows only | Windows + Linux + others |
| **Task Distribution** | Manual | Automatic (capabilities-based) |
| **Scalability** | Limited to one machine | Scales to device pool |
| **Use Case** | Individual automation tasks | Enterprise multi-tier workflows |
| **Configuration** | Simple (no server/client setup) | Requires server-client + Galaxy config |
**When to use Standalone UFO²:**
- Simple, single-device Windows automation
- Development and testing
- Personal productivity tasks
- No need for cross-platform workflows
**When to use UFO² as Galaxy Device:**
- Enterprise-scale automation
- Multi-device orchestration
- Cross-platform workflows (Windows + Linux)
- Centralized management and monitoring
- Parallel task execution across multiple machines
## Related Documentation
- **[UFO² Overview](overview.md)** - Architecture and core concepts
- **[HostAgent](host_agent/overview.md)** - Desktop-level automation
- **[AppAgent](app_agent/overview.md)** - Application-specific automation
- **[Galaxy Overview](../galaxy/overview.md)** - Multi-tier orchestration framework
- **[Server-Client Architecture](../infrastructure/agents/server_client_architecture.md)** - Distributed agent design
- **[Linux as Galaxy Device](../linux/as_galaxy_device.md)** - Linux agent integration (similar pattern)
- **[Quick Start Linux](../getting_started/quick_start_linux.md)** - Similar server-client setup for Linux
## Summary
Integrating UFO² into UFO³ Galaxy enables:
- **Multi-tier orchestration** - Galaxy coordinates UFO² + Linux + other devices
- **Cross-platform workflows** - Seamlessly combine Windows desktop + Linux servers
- **Capability-based routing** - Intelligent task assignment to appropriate devices
- **Scalable automation** - Manage multiple UFO² devices from unified control plane
- **Enterprise-ready** - Centralized monitoring, fault isolation, load balancing
- **Server-client architecture** - Separation of orchestration and execution
- **Local MCP services** - Automatic initialization, no manual setup required
**Next Steps:**
1. Start with a single UFO² device to verify the setup
2. Add more UFO² devices as needed for parallel execution
3. Integrate Linux agents for cross-platform workflows
4. Define custom capabilities for your specific use cases
5. Monitor Galaxy logs to understand task routing decisions
---
### Documents Docs Configuration Models Azure Openai (`documents/docs/configuration/models/azure_openai.md`)
# Azure OpenAI (AOAI)
## Step 1: Create Azure OpenAI Resource
To use the Azure OpenAI API, create an account on the [Azure OpenAI website](https://azure.microsoft.com/en-us/products/ai-services/openai-service). After creating an account, deploy a model and obtain your API key and endpoint.
## Step 2: Configure Agent Settings
Configure the `HOST_AGENT` and `APP_AGENT` in the `config/ufo/agents.yaml` file to use the Azure OpenAI API.
If the file doesn't exist, copy it from the template:
```powershell
Copy-Item config\ufo\agents.yaml.template config\ufo\agents.yaml
```
Edit `config/ufo/agents.yaml` with your Azure OpenAI configuration:
### Option 1: API Key Authentication (Recommended for Development)
```yaml
HOST_AGENT:
VISUAL_MODE: True # Enable visual mode to understand screenshots
REASONING_MODEL: False # Set to True for o-series models
API_TYPE: "aoai" # Use Azure OpenAI API
API_BASE: "https://YOUR_RESOURCE.openai.azure.com" # Your Azure endpoint
API_KEY: "YOUR_AOAI_KEY" # Your Azure OpenAI API key
API_VERSION: "2024-02-15-preview" # API version
API_MODEL: "gpt-4o" # Model name
API_DEPLOYMENT_ID: "YOUR_DEPLOYMENT_ID" # Your deployment name
APP_AGENT:
VISUAL_MODE: True
REASONING_MODEL: False
API_TYPE: "aoai"
API_BASE: "https://YOUR_RESOURCE.openai.azure.com"
API_KEY: "YOUR_AOAI_KEY"
API_VERSION: "2024-02-15-preview"
API_MODEL: "gpt-4o-mini" # Use gpt-4o-mini for cost efficiency
API_DEPLOYMENT_ID: "YOUR_DEPLOYMENT_ID"
```
### Option 2: Azure AD Authentication (Recommended for Production)
For Azure Active Directory authentication, use `API_TYPE: "azure_ad"`:
```yaml
HOST_AGENT:
VISUAL_MODE: True
REASONING_MODEL: False
API_TYPE: "azure_ad" # Use Azure AD authentication
API_BASE: "https://YOUR_RESOURCE.openai.azure.com" # Your Azure endpoint
API_VERSION: "2024-02-15-preview"
API_MODEL: "gpt-4o"
API_DEPLOYMENT_ID: "YOUR_DEPLOYMENT_ID"
# Azure AD Configuration
AAD_TENANT_ID: "YOUR_TENANT_ID" # Your Azure tenant ID
AAD_API_SCOPE: "YOUR_SCOPE" # Your API scope
AAD_API_SCOPE_BASE: "YOUR_SCOPE_BASE" # Scope base (without api:// prefix)
APP_AGENT:
VISUAL_MODE: True
REASONING_MODEL: False
API_TYPE: "azure_ad"
API_BASE: "https://YOUR_RESOURCE.openai.azure.com"
API_VERSION: "2024-02-15-preview"
API_MODEL: "gpt-4o-mini"
API_DEPLOYMENT_ID: "YOUR_DEPLOYMENT_ID"
AAD_TENANT_ID: "YOUR_TENANT_ID"
AAD_API_SCOPE: "YOUR_SCOPE"
AAD_API_SCOPE_BASE: "YOUR_SCOPE_BASE"
```
**Configuration Fields:**
- **`VISUAL_MODE`**: Set to `True` to enable vision capabilities. Ensure your deployment supports visual inputs
- **`API_TYPE`**: Use `"aoai"` for API key auth or `"azure_ad"` for Azure AD auth
- **`API_BASE`**: Your Azure OpenAI endpoint URL (format: `https://{resource-name}.openai.azure.com`)
- **`API_KEY`**: Your Azure OpenAI API key (not needed for Azure AD auth)
- **`API_VERSION`**: Azure API version (e.g., `"2024-02-15-preview"`)
- **`API_MODEL`**: Model identifier (e.g., `gpt-4o`, `gpt-4o-mini`)
- **`API_DEPLOYMENT_ID`**: Your Azure deployment name (required for AOAI)
- **`AAD_TENANT_ID`**: Azure tenant ID (required for Azure AD auth)
- **`AAD_API_SCOPE`**: Azure AD API scope (required for Azure AD auth)
- **`AAD_API_SCOPE_BASE`**: Scope base without `api://` prefix (required for Azure AD auth)
**For detailed configuration options, see:**
- [Agent Configuration Guide](../system/agents_config.md) - Complete agent settings reference
- [Model Configuration Overview](overview.md) - Compare different LLM providers
- [OpenAI](openai.md) - Standard OpenAI API setup
## Step 3: Start Using UFO
After configuration, you can start using UFO with the Azure OpenAI API. Refer to the [Quick Start Guide](../../getting_started/quick_start_ufo2.md) for detailed instructions on running your first tasks.
---
### Documents Docs Configuration Models Claude (`documents/docs/configuration/models/claude.md`)
# Anthropic Claude
## Step 1: Obtain API Key
To use the Claude API, create an account on the [Anthropic Console](https://console.anthropic.com/) and access your API key from the API keys section.
## Step 2: Install Dependencies
Install the required Anthropic Python package:
```bash
pip install -U anthropic==0.37.1
```
## Step 3: Configure Agent Settings
Configure the `HOST_AGENT` and `APP_AGENT` in the `config/ufo/agents.yaml` file to use the Claude API.
If the file doesn't exist, copy it from the template:
```powershell
Copy-Item config\ufo\agents.yaml.template config\ufo\agents.yaml
```
Edit `config/ufo/agents.yaml` with your Claude configuration:
```yaml
HOST_AGENT:
VISUAL_MODE: True # Enable visual mode to understand screenshots
API_TYPE: "claude" # Use Claude API
API_BASE: "https://api.anthropic.com" # Claude API endpoint
API_KEY: "YOUR_CLAUDE_API_KEY" # Your Claude API key
API_MODEL: "claude-3-5-sonnet-20241022" # Model name
API_VERSION: "2023-06-01" # API version
APP_AGENT:
VISUAL_MODE: True
API_TYPE: "claude"
API_BASE: "https://api.anthropic.com"
API_KEY: "YOUR_CLAUDE_API_KEY"
API_MODEL: "claude-3-5-sonnet-20241022"
API_VERSION: "2023-06-01"
```
**Configuration Fields:**
- **`VISUAL_MODE`**: Set to `True` to enable vision capabilities. Most Claude 3+ models support visual inputs (see [Claude models](https://www.anthropic.com/pricing#anthropic-api))
- **`API_TYPE`**: Use `"claude"` for Claude API (case-sensitive in code: lowercase)
- **`API_BASE`**: Claude API endpoint - `https://api.anthropic.com`
- **`API_KEY`**: Your Anthropic API key from the console
- **`API_MODEL`**: Model identifier (e.g., `claude-3-5-sonnet-20241022`, `claude-3-opus-20240229`)
- **`API_VERSION`**: API version identifier
**Available Models:**
- **Claude 3.5 Sonnet**: `claude-3-5-sonnet-20241022` - Best balance of intelligence and speed
- **Claude 3 Opus**: `claude-3-opus-20240229` - Most capable model
- **Claude 3 Sonnet**: `claude-3-sonnet-20240229` - Balanced performance
- **Claude 3 Haiku**: `claude-3-haiku-20240307` - Fast and cost-effective
**For detailed configuration options, see:**
- [Agent Configuration Guide](../system/agents_config.md) - Complete agent settings reference
- [Model Configuration Overview](overview.md) - Compare different LLM providers
- [Anthropic Documentation](https://docs.anthropic.com/) - Official Claude API docs
## Step 4: Start Using UFO
After configuration, you can start using UFO with the Claude API. Refer to the [Quick Start Guide](../../getting_started/quick_start_ufo2.md) for detailed instructions on running your first tasks.
---
### Documents Docs Configuration Models Custom Model (`documents/docs/configuration/models/custom_model.md`)
# Customized LLM Models
UFO supports and welcomes the integration of custom LLM models. If you have a custom LLM model that you would like to use with UFO, follow the steps below to configure it.
## Step 1: Create and Serve Your Model
Create a custom LLM model and serve it on your local or remote environment. Ensure your model has an accessible API endpoint.
## Step 2: Implement Model Service Class
Create a Python script under the `ufo/llm` directory and implement your own LLM model class by inheriting the `BaseService` class from `ufo/llm/base.py`.
**Reference Example:** See `PlaceHolderService` in `ufo/llm/placeholder.py` as a template.
You must implement the `chat_completion` method:
```python
def chat_completion(
self,
messages: List[Dict[str, str]],
n: int = 1,
temperature: Optional[float] = None,
max_tokens: Optional[int] = None,
top_p: Optional[float] = None,
**kwargs: Any,
) -> Tuple[List[str], Optional[float]]:
"""
Generates completions for a given list of messages.
Args:
messages: The list of messages to generate completions for.
n: The number of completions to generate for each message.
temperature: Controls the randomness (higher = more random).
max_tokens: The maximum number of tokens in completions.
top_p: Controls diversity (higher = more diverse).
**kwargs: Additional keyword arguments.
Returns:
Tuple[List[str], Optional[float]]:
- List of generated completions for each message
- Cost of the API call (None if not applicable)
Raises:
Exception: If an error occurs while making the API request.
"""
# Your implementation here
pass
```
**Key Implementation Points:**
- Handle message formatting according to your model's API
- Process visual inputs if `VISUAL_MODE` is enabled
- Implement retry logic for failed requests
- Calculate and return cost if applicable
## Step 3: Configure Agent Settings
Configure the `HOST_AGENT` and `APP_AGENT` in the `config/ufo/agents.yaml` file to use your custom model.
If the file doesn't exist, copy it from the template:
```powershell
Copy-Item config\ufo\agents.yaml.template config\ufo\agents.yaml
```
Edit `config/ufo/agents.yaml` with your custom model configuration:
```yaml
HOST_AGENT:
VISUAL_MODE: True # Set based on your model's capabilities
API_TYPE: "custom_model" # Use custom model type
API_BASE: "http://your-endpoint:port" # Your model's API endpoint
API_KEY: "YOUR_API_KEY" # Your API key (if required)
API_MODEL: "your-model-name" # Your model identifier
APP_AGENT:
VISUAL_MODE: True
API_TYPE: "custom_model"
API_BASE: "http://your-endpoint:port"
API_KEY: "YOUR_API_KEY"
API_MODEL: "your-model-name"
```
**Configuration Fields:**
- **`VISUAL_MODE`**: Set to `True` if your model supports visual inputs
- **`API_TYPE`**: Use `"custom_model"` for custom implementations
- **`API_BASE`**: Your custom model's API endpoint URL
- **`API_KEY`**: Authentication key (if your model requires it)
- **`API_MODEL`**: Model identifier or name
**For detailed configuration options, see:**
- [Agent Configuration Guide](../system/agents_config.md) - Complete agent settings reference
- [Model Configuration Overview](overview.md) - Compare different LLM providers
## Step 4: Register Your Model
Update the model factory in `ufo/llm/__init__.py` to include your custom model class:
```python
from ufo.llm.your_model import YourModelService
# Add to the model factory mapping
MODEL_FACTORY = {
# ... existing models ...
"custom_model": YourModelService,
}
```
## Step 5: Start Using UFO
After configuration, you can start using UFO with your custom model. Refer to the [Quick Start Guide](../../getting_started/quick_start_ufo2.md) for detailed instructions on running your first tasks.
**Testing Your Integration:**
1. Test with simple requests first
2. Verify visual mode works (if applicable)
3. Check error handling and retry logic
4. Monitor response quality and latency
---
### Documents Docs Configuration Models Deepseek (`documents/docs/configuration/models/deepseek.md`)
# DeepSeek Model
## Step 1: Obtain API Key
DeepSeek is developed by DeepSeek AI. To use DeepSeek models, go to [DeepSeek Platform](https://www.deepseek.com/), register an account, and obtain your API key from the API management console.
## Step 2: Configure Agent Settings
Configure the `HOST_AGENT` and `APP_AGENT` in the `config/ufo/agents.yaml` file to use the DeepSeek model.
If the file doesn't exist, copy it from the template:
```powershell
Copy-Item config\ufo\agents.yaml.template config\ufo\agents.yaml
```
Edit `config/ufo/agents.yaml` with your DeepSeek configuration:
```yaml
HOST_AGENT:
VISUAL_MODE: False # DeepSeek models typically don't support visual inputs
API_TYPE: "deepseek" # Use DeepSeek API
API_KEY: "YOUR_DEEPSEEK_API_KEY" # Your DeepSeek API key
API_MODEL: "deepseek-chat" # Model name
APP_AGENT:
VISUAL_MODE: False
API_TYPE: "deepseek"
API_KEY: "YOUR_DEEPSEEK_API_KEY"
API_MODEL: "deepseek-chat"
```
**Configuration Fields:**
- **`VISUAL_MODE`**: Set to `False` - Most DeepSeek models don't support visual inputs
- **`API_TYPE`**: Use `"deepseek"` for DeepSeek API (case-sensitive in code: lowercase)
- **`API_KEY`**: Your DeepSeek API key
- **`API_MODEL`**: Model identifier (e.g., `deepseek-chat`, `deepseek-coder`)
**Available Models:**
- **DeepSeek-Chat**: `deepseek-chat` - General conversation model
- **DeepSeek-Coder**: `deepseek-coder` - Code-specialized model
**For detailed configuration options, see:**
- [Agent Configuration Guide](../system/agents_config.md) - Complete agent settings reference
- [Model Configuration Overview](overview.md) - Compare different LLM providers
## Step 3: Start Using UFO
After configuration, you can start using UFO with the DeepSeek model. Refer to the [Quick Start Guide](../../getting_started/quick_start_ufo2.md) for detailed instructions on running your first tasks.
**Note:** Since DeepSeek models don't support visual mode, UFO will operate in text-only mode, which may limit some UI automation capabilities that rely on screenshot understanding.
---
### Documents Docs Configuration Models Gemini (`documents/docs/configuration/models/gemini.md`)
# Google Gemini
## Step 1: Obtain API Key
To use the Google Gemini API, create an account on [Google AI Studio](https://ai.google.dev/) and generate your API key from the API keys section.
## Step 2: Install Dependencies
Install the required Google GenAI Python package:
```bash
pip install -U google-genai==1.12.1
```
## Step 3: Configure Agent Settings
Configure the `HOST_AGENT` and `APP_AGENT` in the `config/ufo/agents.yaml` file to use the Google Gemini API.
If the file doesn't exist, copy it from the template:
```powershell
Copy-Item config\ufo\agents.yaml.template config\ufo\agents.yaml
```
Edit `config/ufo/agents.yaml` with your Gemini configuration:
```yaml
HOST_AGENT:
VISUAL_MODE: True # Enable visual mode to understand screenshots
JSON_SCHEMA: True # Enable JSON schema for structured responses
API_TYPE: "gemini" # Use Gemini API
API_BASE: "https://generativelanguage.googleapis.com" # Gemini API endpoint
API_KEY: "YOUR_GEMINI_API_KEY" # Your Gemini API key
API_MODEL: "gemini-2.0-flash-exp" # Model name
API_VERSION: "v1beta" # API version
APP_AGENT:
VISUAL_MODE: True
JSON_SCHEMA: True
API_TYPE: "gemini"
API_BASE: "https://generativelanguage.googleapis.com"
API_KEY: "YOUR_GEMINI_API_KEY"
API_MODEL: "gemini-2.0-flash-exp"
API_VERSION: "v1beta"
```
**Configuration Fields:**
- **`VISUAL_MODE`**: Set to `True` to enable vision capabilities. Most Gemini models support visual inputs (see [Gemini models](https://ai.google.dev/gemini-api/docs/models/gemini))
- **`JSON_SCHEMA`**: Set to `True` to enable structured JSON output formatting
- **`API_TYPE`**: Use `"gemini"` for Google Gemini API (case-sensitive in code: lowercase)
- **`API_BASE`**: Gemini API endpoint - `https://generativelanguage.googleapis.com`
- **`API_KEY`**: Your Google AI API key
- **`API_MODEL`**: Model identifier (e.g., `gemini-2.0-flash-exp`, `gemini-1.5-pro`)
- **`API_VERSION`**: API version (typically `v1beta`)
**Available Models:**
- **Gemini 2.0 Flash**: `gemini-2.0-flash-exp` - Latest experimental model with multimodal capabilities
- **Gemini 1.5 Pro**: `gemini-1.5-pro` - Advanced reasoning and long context
- **Gemini 1.5 Flash**: `gemini-1.5-flash` - Fast and efficient
**Rate Limits:**
If you encounter `429 Resource has been exhausted` errors, you've hit the rate limit of your Gemini API quota. Consider:
- Reducing request frequency
- Upgrading your API tier
- Using exponential backoff for retries
**For detailed configuration options, see:**
- [Agent Configuration Guide](../system/agents_config.md) - Complete agent settings reference
- [Model Configuration Overview](overview.md) - Compare different LLM providers
- [Gemini API Documentation](https://ai.google.dev/gemini-api) - Official Gemini API docs
## Step 4: Start Using UFO
After configuration, you can start using UFO with the Gemini API. Refer to the [Quick Start Guide](../../getting_started/quick_start_ufo2.md) for detailed instructions on running your first tasks.
---
### Documents Docs Configuration Models Ollama (`documents/docs/configuration/models/ollama.md`)
# Ollama
## Step 1: Install and Start Ollama
Go to [Ollama](https://github.com/jmorganca/ollama) and follow the installation instructions for your platform.
**For Linux & WSL2:**
```bash
# Install Ollama
curl https://ollama.ai/install.sh | sh
# Start the Ollama server
ollama serve
```
**For Windows/Mac:** Download and install from the [Ollama website](https://ollama.ai/).
## Step 2: Pull and Test a Model
Open a new terminal and pull a model:
```bash
# Pull a model (e.g., llama2)
ollama pull llama2
# Test the model
ollama run llama2
```
By default, Ollama starts a server at `http://localhost:11434`, which will be used as the API base in your configuration.
## Step 3: Configure Agent Settings
Configure the `HOST_AGENT` and `APP_AGENT` in the `config/ufo/agents.yaml` file to use Ollama.
If the file doesn't exist, copy it from the template:
```powershell
Copy-Item config\ufo\agents.yaml.template config\ufo\agents.yaml
```
Edit `config/ufo/agents.yaml` with your Ollama configuration:
```yaml
HOST_AGENT:
VISUAL_MODE: True # Enable if model supports vision (e.g., llava)
API_TYPE: "ollama" # Use Ollama API
API_BASE: "http://localhost:11434" # Ollama server endpoint
API_KEY: "ollama" # Placeholder (not used but required)
API_MODEL: "llama2" # Model name (must match pulled model)
APP_AGENT:
VISUAL_MODE: True
API_TYPE: "ollama"
API_BASE: "http://localhost:11434"
API_KEY: "ollama"
API_MODEL: "llama2"
```
**Configuration Fields:**
- **`VISUAL_MODE`**: Set to `True` only for vision-capable models like `llava`
- **`API_TYPE`**: Use `"ollama"` for Ollama API (case-sensitive in code: lowercase)
- **`API_BASE`**: Ollama server URL (default: `http://localhost:11434`)
- **`API_KEY`**: Placeholder value (not used but required in config)
- **`API_MODEL`**: Model name matching your pulled model
**Important: Increase Context Length**
UFO requires at least 20,000 tokens to function properly. Ollama's default context length is 2048 tokens, which is insufficient. You must create a custom model with increased context:
1. Create a `Modelfile`:
```text
FROM llama2
PARAMETER num_ctx 32768
```
2. Build the custom model:
```bash
ollama create llama2-max-ctx -f Modelfile
```
3. Use the custom model in your config:
```yaml
API_MODEL: "llama2-max-ctx"
```
For more details, see [Ollama's Modelfile documentation](https://github.com/ollama/ollama/blob/main/docs/modelfile.md).
**For detailed configuration options, see:**
- [Agent Configuration Guide](../system/agents_config.md) - Complete agent settings reference
- [Model Configuration Overview](overview.md) - Compare different LLM providers
## Step 4: Start Using UFO
After configuration, you can start using UFO with Ollama. Refer to the [Quick Start Guide](../../getting_started/quick_start_ufo2.md) for detailed instructions on running your first tasks.
---
### Documents Docs Configuration Models Openai (`documents/docs/configuration/models/openai.md`)
# OpenAI
## Step 1: Obtain API Key
To use the OpenAI API, create an account on the [OpenAI website](https://platform.openai.com/signup). After creating an account, you can access your API key from the [API keys page](https://platform.openai.com/account/api-keys).
## Step 2: Configure Agent Settings
After obtaining the API key, configure the `HOST_AGENT` and `APP_AGENT` in the `config/ufo/agents.yaml` file to use the OpenAI API.
If the file doesn't exist, copy it from the template:
```powershell
Copy-Item config\ufo\agents.yaml.template config\ufo\agents.yaml
```
Edit `config/ufo/agents.yaml` with your OpenAI configuration:
```yaml
HOST_AGENT:
VISUAL_MODE: True # Enable visual mode to understand screenshots
REASONING_MODEL: False # Set to True for o-series models (o1, o3, o3-mini)
API_TYPE: "openai" # Use OpenAI API
API_BASE: "https://api.openai.com/v1" # OpenAI API endpoint
API_KEY: "sk-YOUR_KEY_HERE" # Your OpenAI API key (starts with sk-)
API_VERSION: "2025-02-01-preview" # API version
API_MODEL: "gpt-4o" # Model name (gpt-4o, gpt-4o-mini, etc.)
APP_AGENT:
VISUAL_MODE: True
REASONING_MODEL: False
API_TYPE: "openai"
API_BASE: "https://api.openai.com/v1"
API_KEY: "sk-YOUR_KEY_HERE"
API_VERSION: "2025-02-01-preview"
API_MODEL: "gpt-4o-mini" # Use gpt-4o-mini for cost efficiency
```
**Configuration Fields:**
- **`VISUAL_MODE`**: Set to `True` to enable vision capabilities. Ensure your selected model supports visual inputs (see [OpenAI models](https://platform.openai.com/docs/models))
- **`REASONING_MODEL`**: Set to `True` when using o-series models (o1, o3, o3-mini) which have different behavior
- **`API_TYPE`**: Use `"openai"` for OpenAI API
- **`API_BASE`**: OpenAI API base URL - `https://api.openai.com/v1`
- **`API_KEY`**: Your OpenAI API key from the API keys page
- **`API_VERSION`**: API version identifier
- **`API_MODEL`**: Model identifier (e.g., `gpt-4o`, `gpt-4o-mini`, `gpt-4-turbo`)
**For detailed configuration options, see:**
- [Agent Configuration Guide](../system/agents_config.md) - Complete agent settings reference
- [Model Configuration Overview](overview.md) - Compare different LLM providers
- [Azure OpenAI](azure_openai.md) - Alternative Azure-hosted OpenAI setup
## Step 3: Start Using UFO
After configuration, you can start using UFO with the OpenAI API. Refer to the [Quick Start Guide](../../getting_started/quick_start_ufo2.md) for detailed instructions on running your first tasks.
---
### Documents Docs Configuration Models Operator (`documents/docs/configuration/models/operator.md`)
# OpenAI CUA (Operator)
The [Operator](https://openai.com/index/computer-using-agent/) is a specialized agentic model tailored for Computer-Using Agents (CUA). It's currently available via the Azure OpenAI API (AOAI) using the [Response API](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/responses?tabs=python-secure).
## Step 1: Create Azure OpenAI Resource
To use the Operator model, create an account on the [Azure OpenAI website](https://azure.microsoft.com/en-us/products/ai-services/openai-service). After creating an account, deploy the Operator model and access your API key.
## Step 2: Configure Operator Agent
Configure the `OPERATOR` in the `config/ufo/agents.yaml` file to use the Azure OpenAI Operator model.
If the file doesn't exist, copy it from the template:
```powershell
Copy-Item config\ufo\agents.yaml.template config\ufo\agents.yaml
```
Edit `config/ufo/agents.yaml` with your Operator configuration:
```yaml
OPERATOR:
SCALER: [1024, 768] # Visual input resolution [width, height]
API_TYPE: "azure_ad" # Use Azure AD authentication
API_MODEL: "computer-use-preview-20250311" # Operator model name
API_VERSION: "2025-03-01-preview" # API version for Operator
API_BASE: "https://YOUR_RESOURCE.openai.azure.com" # Your Azure endpoint
# Azure AD Authentication (required)
AAD_TENANT_ID: "YOUR_TENANT_ID" # Your Azure tenant ID
AAD_API_SCOPE: "YOUR_SCOPE" # Your API scope
AAD_API_SCOPE_BASE: "YOUR_SCOPE_BASE" # Scope base (without api:// prefix)
```
**Configuration Fields:**
- **`SCALER`**: Resolution for visual input `[width, height]` (recommended: `[1024, 768]`)
- **`API_TYPE`**: Use `"azure_ad"` for Azure AD authentication (or `"aoai"` for API key auth)
- **`API_MODEL`**: Operator model identifier (e.g., `computer-use-preview-20250311`)
- **`API_VERSION`**: API version for Operator (e.g., `2025-03-01-preview`)
- **`API_BASE`**: Your Azure OpenAI endpoint URL
- **`AAD_TENANT_ID`**: Azure tenant ID (required for Azure AD auth)
- **`AAD_API_SCOPE`**: Azure AD API scope (required for Azure AD auth)
- **`AAD_API_SCOPE_BASE`**: Scope base without `api://` prefix (required for Azure AD auth)
**For API Key Authentication (Development):**
If you prefer API key authentication instead of Azure AD:
```yaml
OPERATOR:
SCALER: [1024, 768]
API_TYPE: "aoai" # Use API key authentication
API_MODEL: "computer-use-preview-20250311"
API_VERSION: "2025-03-01-preview"
API_BASE: "https://YOUR_RESOURCE.openai.azure.com"
API_KEY: "YOUR_AOAI_KEY" # Your Azure OpenAI API key
API_DEPLOYMENT_ID: "YOUR_DEPLOYMENT_ID" # Your deployment name
```
## Step 3: Run Operator in UFO
UFO supports running Operator in two modes:
1. **Standalone Agent**: Run Operator as a single agent
2. **As AppAgent**: Call Operator as a separate `AppAgent` from the `HostAgent`
Operator uses a specialized visual-only workflow different from other models and currently does not support the standard `AppAgent` workflow.
**For detailed usage instructions, see:**
- [Operator as AppAgent](../../ufo2/advanced_usage/operator_as_app_agent.md) - How to integrate Operator into UFO workflows
- [Agent Configuration Guide](../system/agents_config.md) - Complete agent settings reference
- [Azure OpenAI](azure_openai.md) - General Azure OpenAI setup
**Important Notes:**
- Operator is a visual-only model optimized for computer control tasks
- It uses a different workflow from standard text-based models
- Best suited for direct UI manipulation and visual understanding tasks
- Requires Azure OpenAI deployment (not available via standard OpenAI API)
---
### Documents Docs Configuration Models Qwen (`documents/docs/configuration/models/qwen.md`)
# Qwen Model
## Step 1: Obtain API Key
Qwen (Tongyi Qianwen) is developed by Alibaba DAMO Academy. To use Qwen models, go to [DashScope](https://dashscope.aliyun.com/), register an account, and obtain your API key. Detailed instructions are available in the [DashScope documentation](https://help.aliyun.com/zh/dashscope/developer-reference/activate-dashscope-and-create-an-api-key) (Chinese).
## Step 2: Configure Agent Settings
Configure the `HOST_AGENT` and `APP_AGENT` in the `config/ufo/agents.yaml` file to use the Qwen model.
If the file doesn't exist, copy it from the template:
```powershell
Copy-Item config\ufo\agents.yaml.template config\ufo\agents.yaml
```
Edit `config/ufo/agents.yaml` with your Qwen configuration:
```yaml
HOST_AGENT:
VISUAL_MODE: True # Enable visual mode for vision-capable models
API_TYPE: "qwen" # Use Qwen API
API_KEY: "YOUR_QWEN_API_KEY" # Your DashScope API key
API_MODEL: "qwen-vl-max" # Model name (e.g., qwen-vl-max, qwen-max)
APP_AGENT:
VISUAL_MODE: True
API_TYPE: "qwen"
API_KEY: "YOUR_QWEN_API_KEY"
API_MODEL: "qwen-vl-max"
```
**Configuration Fields:**
- **`VISUAL_MODE`**: Set to `True` for vision-capable models (qwen-vl-*). Set to `False` for text-only models
- **`API_TYPE`**: Use `"qwen"` for Qwen API (case-sensitive in code: lowercase)
- **`API_KEY`**: Your DashScope API key
- **`API_MODEL`**: Model identifier (see [Qwen model list](https://help.aliyun.com/zh/dashscope/developer-reference/model-square/))
**Available Models:**
- **Qwen-VL-Max**: `qwen-vl-max` - Vision and language model
- **Qwen-Max**: `qwen-max` - Text-only advanced model
- **Qwen-Plus**: `qwen-plus` - Balanced performance model
**For detailed configuration options, see:**
- [Agent Configuration Guide](../system/agents_config.md) - Complete agent settings reference
- [Model Configuration Overview](overview.md) - Compare different LLM providers
## Step 3: Start Using UFO
After configuration, you can start using UFO with the Qwen model. Refer to the [Quick Start Guide](../../getting_started/quick_start_ufo2.md) for detailed instructions on running your first tasks.
---
### Documents Docs Configuration System Agents Config (`documents/docs/configuration/system/agents_config.md`)
# Agent Configuration (agents.yaml)
Configure all LLM models and agent-specific settings for UFO². Each agent type can use different models and API configurations for optimal performance.
## Overview
The `agents.yaml` file defines LLM settings for all agents in UFO². This is the **most important configuration file** as it contains your API keys and model selections.
**File Location**: `config/ufo/agents.yaml`
**Initial Setup Required:**
1. **Copy the template file**:
```powershell
Copy-Item config\ufo\agents.yaml.template config\ufo\agents.yaml
```
2. **Edit `config/ufo/agents.yaml`** with your API keys and settings
3. **Never commit `agents.yaml`** to version control (it contains secrets)
## Quick Start
### Step 1: Create Configuration File
```powershell
# Copy template to create your configuration
Copy-Item config\ufo\agents.yaml.template config\ufo\agents.yaml
```
### Step 2: Configure Your LLM Provider
Choose your LLM provider and edit `config/ufo/agents.yaml`:
**OpenAI:**
```yaml
HOST_AGENT:
VISUAL_MODE: True
API_TYPE: "openai"
API_BASE: "https://api.openai.com/v1/chat/completions"
API_KEY: "sk-YOUR_OPENAI_KEY_HERE"
API_MODEL: "gpt-4o"
API_VERSION: "2025-02-01-preview"
APP_AGENT:
VISUAL_MODE: True
API_TYPE: "openai"
API_BASE: "https://api.openai.com/v1/chat/completions"
API_KEY: "sk-YOUR_OPENAI_KEY_HERE"
API_MODEL: "gpt-4o-mini"
API_VERSION: "2025-02-01-preview"
```
**Azure OpenAI:**
```yaml
HOST_AGENT:
VISUAL_MODE: True
API_TYPE: "aoai"
API_BASE: "https://YOUR_RESOURCE.openai.azure.com"
API_KEY: "YOUR_AOAI_KEY"
API_MODEL: "gpt-4o"
API_VERSION: "2024-02-15-preview"
API_DEPLOYMENT_ID: "gpt-4o-deployment"
APP_AGENT:
VISUAL_MODE: True
API_TYPE: "aoai"
API_BASE: "https://YOUR_RESOURCE.openai.azure.com"
API_KEY: "YOUR_AOAI_KEY"
API_MODEL: "gpt-4o-mini"
API_VERSION: "2024-02-15-preview"
API_DEPLOYMENT_ID: "gpt-4o-mini-deployment"
```
**Google Gemini:**
```yaml
HOST_AGENT:
VISUAL_MODE: True
API_TYPE: "gemini"
API_BASE: "https://generativelanguage.googleapis.com"
API_KEY: "YOUR_GEMINI_API_KEY"
API_MODEL: "gemini-2.0-flash-exp"
API_VERSION: "v1beta"
```
**Anthropic Claude:**
```yaml
HOST_AGENT:
VISUAL_MODE: True
API_TYPE: "claude"
API_BASE: "https://api.anthropic.com"
API_KEY: "YOUR_CLAUDE_API_KEY"
API_MODEL: "claude-3-5-sonnet-20241022"
API_VERSION: "2023-06-01"
```
### Step 3: Verify Configuration
```python
from config.config_loader import get_ufo_config
config = get_ufo_config()
print(f"HOST_AGENT model: {config.host_agent.api_model}")
print(f"APP_AGENT model: {config.app_agent.api_model}")
```
---
## Agent Types
UFO² uses different agents for different purposes. Each can be configured with different models.
| Agent | Purpose | Recommended Model | Frequency |
|-------|---------|-------------------|-----------|
| **HOST_AGENT** | Task planning, app coordination | GPT-4o, GPT-4 | Low (planning) |
| **APP_AGENT** | Action execution, UI interaction | GPT-4o-mini, GPT-4o | High (every action) |
| **BACKUP_AGENT** | Fallback when others fail | GPT-4-vision-preview | Rare (errors) |
| **EVALUATION_AGENT** | Task completion evaluation | GPT-4o | Low (end of task) |
| **OPERATOR** | CUA-based automation | computer-use-preview | Optional |
**Cost Optimization Tips:**
- Use **GPT-4o** for HOST_AGENT (complex planning)
- Use **GPT-4o-mini** for APP_AGENT (frequent actions, 60% cheaper)
- Same model can be used for BACKUP_AGENT and EVALUATION_AGENT
## Configuration Fields
### Common Fields (All Agents)
These fields are available for `HOST_AGENT`, `APP_AGENT`, `BACKUP_AGENT`, `EVALUATION_AGENT`, and `OPERATOR`.
#### Core Settings
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `VISUAL_MODE` | Boolean | ❌ | `True` | Enable vision capabilities (screenshot understanding) |
| `REASONING_MODEL` | Boolean | ❌ | `False` | Whether model is a reasoning model (o1, o3, o3-mini) |
| `API_TYPE` | String | ✅ | `"openai"` | LLM provider type |
| `API_BASE` | String | ✅ | varies | API endpoint URL |
| `API_KEY` | String | ✅ | `""` | API authentication key |
| `API_MODEL` | String | ✅ | varies | Model identifier |
| `API_VERSION` | String | ❌ | `"2025-02-01-preview"` | API version |
**Legend:** ✅ = Required (must be set), ❌ = Optional (has default value)
#### API_TYPE Options
| API_TYPE | Provider | Example API_BASE |
|----------|----------|------------------|
| `"openai"` | OpenAI | `https://api.openai.com/v1/chat/completions` |
| `"aoai"` | Azure OpenAI | `https://YOUR_RESOURCE.openai.azure.com` |
| `"azure_ad"` | Azure OpenAI (AD auth) | `https://YOUR_RESOURCE.openai.azure.com` |
| `"gemini"` | Google Gemini | `https://generativelanguage.googleapis.com` |
| `"claude"` | Anthropic Claude | `https://api.anthropic.com` |
| `"qwen"` | Alibaba Qwen | varies |
| `"ollama"` | Ollama (local) | `http://localhost:11434` |
#### Azure OpenAI Additional Fields
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `API_DEPLOYMENT_ID` | String | ✅ (for AOAI) | Azure deployment name |
**Example**:
```yaml
HOST_AGENT:
API_TYPE: "aoai"
API_BASE: "https://myresource.openai.azure.com"
API_KEY: "abc123..."
API_MODEL: "gpt-4o"
API_DEPLOYMENT_ID: "gpt-4o-deployment-name"
```
#### Azure AD Authentication Fields
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `AAD_TENANT_ID` | String | ✅ (for azure_ad) | Azure AD tenant ID |
| `AAD_API_SCOPE` | String | ✅ (for azure_ad) | Azure AD API scope |
| `AAD_API_SCOPE_BASE` | String | ✅ (for azure_ad) | Scope base URL |
**Example**:
```yaml
HOST_AGENT:
API_TYPE: "azure_ad"
API_BASE: "https://myresource.openai.azure.com"
AAD_TENANT_ID: "your-tenant-id"
AAD_API_SCOPE: "your-scope"
AAD_API_SCOPE_BASE: "API://your-scope-base"
API_MODEL: "gpt-4o"
API_DEPLOYMENT_ID: "gpt-4o-deployment"
```
#### Prompt Configuration
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `PROMPT` | String | ❌ | Path to main prompt template |
| `EXAMPLE_PROMPT` | String | ❌ | Path to example prompt template |
| `API_PROMPT` | String | ❌ | Path to API usage prompt (APP_AGENT only) |
**Default Prompt Paths:**
```yaml
HOST_AGENT:
PROMPT: "ufo/prompts/share/base/host_agent.yaml"
EXAMPLE_PROMPT: "ufo/prompts/examples/{mode}/host_agent_example.yaml"
APP_AGENT:
PROMPT: "ufo/prompts/share/base/app_agent.yaml"
EXAMPLE_PROMPT: "ufo/prompts/examples/{mode}/app_agent_example.yaml"
API_PROMPT: "ufo/prompts/share/base/api.yaml"
```
You can customize prompts by creating your own YAML files and updating these paths. See the [Customization Guide](../../ufo2/advanced_usage/customization.md) for details.
#### OPERATOR-Specific Fields
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `SCALER` | List[int] | ❌ | Screen dimensions for visual input `[width, height]`, default: `[1024, 768]` |
**Example:**
```yaml
OPERATOR:
SCALER: [1920, 1080] # Full HD resolution
API_MODEL: "computer-use-preview-20250311"
# ... other settings
```
## Complete Configuration Example
Here's a complete `agents.yaml` with all agent types configured:
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Multi-Provider Configuration
You can use different providers for different agents:
```yaml
# Use OpenAI for planning
HOST_AGENT:
API_TYPE: "openai"
API_BASE: "https://api.openai.com/v1/chat/completions"
API_KEY: "sk-YOUR_OPENAI_KEY"
API_MODEL: "gpt-4o"
# Use Azure OpenAI for actions (cost control)
APP_AGENT:
API_TYPE: "aoai"
API_BASE: "https://mycompany.openai.azure.com"
API_KEY: "YOUR_AZURE_KEY"
API_MODEL: "gpt-4o-mini"
API_DEPLOYMENT_ID: "gpt-4o-mini-deploy"
# Use Claude for evaluation
EVALUATION_AGENT:
API_TYPE: "claude"
API_BASE: "https://api.anthropic.com"
API_KEY: "YOUR_CLAUDE_KEY"
API_MODEL: "claude-3-5-sonnet-20241022"
```
## Model Recommendations
### For HOST_AGENT (Planning)
| Model | Provider | Pros | Cons |
|-------|----------|------|------|
| **gpt-4o** | OpenAI | Best overall, fast, multimodal | $$ |
| **gpt-4-turbo** | OpenAI | Good quality, cheaper than GPT-4 | Slower |
| **claude-3-5-sonnet** | Anthropic | Excellent reasoning | No vision API yet |
| **gemini-2.0-flash** | Google | Fast, cheap, multimodal | New, less tested |
### For APP_AGENT (Execution)
| Model | Provider | Pros | Cons |
|-------|----------|------|------|
| **gpt-4o-mini** | OpenAI | 60% cheaper, fast, good quality | Slightly less capable |
| **gpt-4o** | OpenAI | Best quality | More expensive |
| **gemini-1.5-flash** | Google | Very cheap, fast | Less accurate |
### For OPERATOR (CUA Mode)
| Model | Provider | Notes |
|-------|----------|-------|
| **computer-use-preview-20250311** | OpenAI | Supported model for Operator mode (Computer Use Agent) |
## Reasoning Models
For models like OpenAI o1, o3, o3-mini, set `REASONING_MODEL: True`:
```yaml
HOST_AGENT:
REASONING_MODEL: True # Enable for o1/o3/o3-mini
API_TYPE: "openai"
API_MODEL: "o3-mini"
# ... other settings
```
**Note:** Reasoning models have different behavior including no streaming responses, different token limits, and may have different pricing.
## Environment Variables
Instead of hardcoding API keys, you can use environment variables:
```yaml
HOST_AGENT:
API_KEY: "${OPENAI_API_KEY}" # Reads from environment variable
APP_AGENT:
API_KEY: "${AZURE_OPENAI_KEY}"
```
**Setting environment variables**:
**Windows (PowerShell):**
```powershell
$env:OPENAI_API_KEY = "sk-your-key"
$env:AZURE_OPENAI_KEY = "your-azure-key"
```
**Windows (Persistent):**
```powershell
[System.Environment]::SetEnvironmentVariable('OPENAI_API_KEY', 'sk-your-key', 'User')
```
**Linux/macOS:**
```bash
export OPENAI_API_KEY="sk-your-key"
export AZURE_OPENAI_KEY="your-azure-key"
```
## Programmatic Access
```python
from config.config_loader import get_ufo_config
config = get_ufo_config()
# Access HOST_AGENT settings
host_model = config.host_agent.api_model
host_type = config.host_agent.api_type
host_visual = config.host_agent.visual_mode
# Access APP_AGENT settings
app_model = config.app_agent.api_model
app_key = config.app_agent.api_key
# Check if agent is configured
if config.host_agent.api_key:
print("HOST_AGENT is configured")
else:
print("Warning: HOST_AGENT API key not set")
```
## Troubleshooting
### Issue 1: "agents.yaml not found"
**Error Message:**
```
FileNotFoundError: config/ufo/agents.yaml not found
```
**Solution:** Copy the template file
```powershell
Copy-Item config\ufo\agents.yaml.template config\ufo\agents.yaml
```
### Issue 2: API Authentication Errors
**Error Message:**
```
openai.AuthenticationError: Invalid API key
```
**Solutions:**
1. Verify API key is correct
2. Check for extra spaces or quotes
3. Ensure API_TYPE matches your provider
4. For Azure, verify API_DEPLOYMENT_ID is set
### Issue 3: Model Not Found
**Error Message:**
```
openai.NotFoundError: The model 'gpt-4o' does not exist
```
**Solutions:**
1. Verify model name is correct (check provider's documentation)
2. For Azure, ensure deployment exists and API_DEPLOYMENT_ID matches
3. Check if you have access to the model
### Issue 4: Rate Limits
**Error Message:**
```
openai.RateLimitError: Rate limit exceeded
```
**Solutions:**
1. Add delays between requests (configure in `system.yaml`)
2. Upgrade your API plan
3. Use different API keys for different agents
## Security Best Practices
**API Key Security Guidelines:**
1. ✅ **Never commit `agents.yaml` to Git**
- Add to `.gitignore`
- Only commit `agents.yaml.template`
2. ✅ **Use environment variables** for production
```yaml
API_KEY: "${OPENAI_API_KEY}"
```
3. ✅ **Rotate keys regularly**
4. ✅ **Use separate keys** for dev/prod environments
5. ✅ **Restrict key permissions** (e.g., read-only for evaluation agents)
## Related Documentation
- **[Third-Party Agent Configuration](third_party_config.md)** - Configure external agents like LinuxAgent and HardwareAgent
- **[Creating Custom Third-Party Agents](../../tutorials/creating_third_party_agents.md)** - Build your own specialized agents
- **[System Configuration](system_config.md)** - Runtime and execution settings
- **[MCP Configuration](mcp_reference.md)** - Tool server configuration
- **[RAG Configuration](rag_config.md)** - Knowledge retrieval settings
- **[Model Setup Guide](../models/overview.md)** - Provider-specific setup
- **[Migration Guide](migration.md)** - Migrating from legacy config
## Summary
**Key Takeaways:**
✅ **Copy template first**: `Copy-Item config\ufo\agents.yaml.template config\ufo\agents.yaml`
✅ **Add your API keys**: Edit `agents.yaml` with your credentials
✅ **Choose models wisely**: GPT-4o for planning, GPT-4o-mini for actions
✅ **Never commit secrets**: Keep `agents.yaml` out of version control
✅ **Use environment variables**: For production deployments
**Your agents are now ready to work!** 🚀
---
### Documents Docs Configuration System Extending (`documents/docs/configuration/system/extending.md`)
# Extending Configuration
This guide shows you how to add custom configuration options to UFO2.
**Three Ways to Extend:**
1. **Simple YAML files** - Quick custom settings in existing files
2. **New configuration files** - Organize new features separately
3. **Typed configuration schemas** - Full type safety with Python dataclasses
## Method 1: Adding Fields to Existing Files
For simple customizations, add fields directly to existing configuration files.
```yaml
# config/ufo/system.yaml
MAX_STEP: 20
SLEEP_TIME: 1.0
# Your custom fields
CUSTOM_TIMEOUT: 300
DEBUG_MODE: true
FEATURE_FLAGS:
enable_telemetry: false
use_experimental_api: true
```
### Accessing Custom Fields
```python
from config.config_loader import get_ufo_config
config = get_ufo_config()
# Access custom fields dynamically
timeout = config.system.CUSTOM_TIMEOUT # 300
debug = config.system.DEBUG_MODE # True
use_experimental = config.system.FEATURE_FLAGS['use_experimental_api'] # True
```
Custom fields are automatically discovered and loaded - no code modifications needed!
---
## Method 2: Creating New Configuration Files
For larger features, create dedicated configuration files.
```yaml
# config/ufo/analytics.yaml
ANALYTICS:
enabled: true
backend: "influxdb"
endpoint: "http://localhost:8086"
database: "ufo_metrics"
retention: "30d"
metrics:
- name: "task_duration"
type: "histogram"
- name: "success_rate"
type: "counter"
```
### Automatic Discovery
The config loader automatically discovers and loads all YAML files in `config/ufo/`:
```python
# No registration needed!
config = get_ufo_config()
# Your new file is automatically loaded
analytics_enabled = config.ANALYTICS['enabled']
metrics = config.ANALYTICS['metrics']
```
---
## Method 3: Typed Configuration Schemas
For production features requiring type safety and validation, define typed schemas.
```python
# config/config_schemas.py
from dataclasses import dataclass, field
from typing import List, Literal
@dataclass
class MetricConfig:
"""Configuration for a single metric."""
name: str
type: Literal["counter", "histogram", "gauge"]
tags: List[str] = field(default_factory=list)
@dataclass
class AnalyticsConfig:
"""Analytics system configuration."""
# Required fields
enabled: bool
backend: Literal["influxdb", "prometheus", "datadog"]
endpoint: str
# Optional fields with defaults
database: str = "ufo_metrics"
retention: str = "30d"
batch_size: int = 100
flush_interval: float = 10.0
# Nested configuration
metrics: List[MetricConfig] = field(default_factory=list)
def __post_init__(self):
"""Validate configuration after initialization."""
if self.enabled and not self.endpoint:
raise ValueError("endpoint required when analytics enabled")
if self.batch_size <= 0:
raise ValueError("batch_size must be positive")
```
### Step 2: Integrate into UFOConfig
```python
# config/config_schemas.py
from dataclasses import dataclass
@dataclass
class UFOConfig:
"""Main UFO configuration."""
host_agent: AgentConfig
app_agent: AgentConfig
system: SystemConfig
rag: RAGConfig
analytics: AnalyticsConfig # Add your new config
# ... rest of implementation
```
### Step 3: Use Typed Configuration
```python
from config.config_loader import get_ufo_config
config = get_ufo_config()
# Type-safe access with IDE autocomplete
if config.analytics.enabled:
for metric in config.analytics.metrics:
print(f"Metric: {metric.name}, Type: {metric.type}")
# Validation happens automatically
batch_size = config.analytics.batch_size # Guaranteed > 0
```
---
## Common Patterns
### Environment-Specific Overrides
```yaml
# config/ufo/system.yaml (base)
LOG_LEVEL: "INFO"
DEBUG_MODE: false
CACHE_SIZE: 1000
# config/ufo/system.dev.yaml (development override)
LOG_LEVEL: "DEBUG"
DEBUG_MODE: true
PROFILING_ENABLED: true
# config/ufo/system.prod.yaml (production override)
LOG_LEVEL: "WARNING"
CACHE_SIZE: 10000
MONITORING_ENABLED: true
```
### Feature Flags
```yaml
# config/ufo/features.yaml
FEATURES:
experimental_actions: false
multi_device_mode: true
advanced_logging: false
# Per-agent feature flags
agent_features:
host_agent:
use_vision_model: true
parallel_processing: false
app_agent:
speculative_execution: true
action_batching: true
```
### Plugin Configuration
```yaml
# config/ufo/plugins.yaml
PLUGINS:
enabled: true
auto_discover: true
load_order:
- "core"
- "analytics"
- "custom"
plugins:
analytics:
enabled: true
config_file: "config/plugins/analytics.yaml"
custom_processor:
enabled: false
class: "plugins.custom.MyProcessor"
priority: 100
```
---
## Best Practices
**DO - Recommended Practices**
- ✅ **Group related settings** in dedicated files
- ✅ **Use typed schemas** for production features
- ✅ **Provide sensible defaults** for all optional fields
- ✅ **Add validation** in `__post_init__` methods
- ✅ **Document all fields** with docstrings
- ✅ **Use environment overrides** for deployment-specific settings
- ✅ **Version your config schemas** when making breaking changes
- ✅ **Test configuration loading** in CI/CD pipelines
**DON'T - Anti-Patterns**
- ❌ **Don't hardcode secrets** - use environment variables
- ❌ **Don't duplicate settings** across multiple files
- ❌ **Don't use dynamic field names** - breaks type safety
- ❌ **Don't skip validation** - catch errors early
- ❌ **Don't mix concerns** - keep configs focused
- ❌ **Don't ignore warnings** from config loader
- ❌ **Don't commit sensitive data** - use .env files
---
## Security Considerations
!!!warning "Secrets Management"
Never commit sensitive data to configuration files:
```yaml
# ? BAD - Hardcoded secrets
DATABASE:
password: "my-secret-password"
api_key: "sk-1234567890"
# ? GOOD - Environment variable references
DATABASE:
password: "${DB_PASSWORD}"
api_key: "${API_KEY}"
```
### "Environment Variables"
Use environment variables for secrets:
```python
import os
from config.config_loader import get_ufo_config
config = get_ufo_config()
# Resolve environment variables
db_password = os.getenv('DB_PASSWORD')
api_key = os.getenv('API_KEY')
```
---
## Testing Your Configuration
```python
import pytest
from config.config_loader import ConfigLoader
from config.ufo.schemas.analytics_config import AnalyticsConfig
def test_analytics_config_defaults():
"""Test analytics configuration defaults."""
config_data = {
'enabled': True,
'backend': 'influxdb',
'endpoint': 'http://localhost:8086'
}
analytics = AnalyticsConfig(**config_data)
assert analytics.enabled is True
assert analytics.database == 'ufo_metrics' # Default
assert analytics.batch_size == 100 # Default
def test_analytics_config_validation():
"""Test analytics configuration validation."""
with pytest.raises(ValueError, match="endpoint required"):
AnalyticsConfig(enabled=True, backend='influxdb', endpoint='')
with pytest.raises(ValueError, match="batch_size must be positive"):
AnalyticsConfig(
enabled=True,
backend='influxdb',
endpoint='http://localhost',
batch_size=-1
)
def test_config_loading():
"""Test full configuration loading."""
loader = ConfigLoader()
config = loader.load_ufo_config('config/ufo')
# Verify custom configuration loaded
assert hasattr(config, 'analytics')
assert config.analytics.enabled in [True, False]
```
---
## Next Steps
- **[Agents Configuration](./agents_config.md)** - LLM and agent settings
- **[System Configuration](./system_config.md)** - Runtime and execution settings
- **[RAG Configuration](./rag_config.md)** - Knowledge retrieval settings
- **[Migration Guide](./migration.md)** - Migrate from legacy configuration
- **[Configuration Overview](./overview.md)** - Understand configuration system design
---
### Documents Docs Configuration System Galaxy Agent (`documents/docs/configuration/system/galaxy_agent.md`)
# Galaxy Constellation Agent Configuration
**agent.yaml** configures the **Constellation Agent** - the AI agent responsible for creating constellations (task decomposition) and editing them based on execution results.
---
## Overview
The **agent.yaml** configuration file provides **LLM and API settings** for the Constellation Agent. This agent is responsible for:
- **Constellation Creation**: Breaking down user requests into device-specific tasks
- **Constellation Editing**: Adjusting task plans based on execution results
- **Device Selection**: Choosing appropriate devices for each sub-task
- **Task Orchestration**: Coordinating multi-device workflows
**Configuration Separation:**
- **agent.yaml** - LLM configuration for constellation agent (this document)
- **constellation.yaml** - Runtime settings for orchestrator ([Galaxy Constellation Configuration](./galaxy_constellation.md))
- **devices.yaml** - Device definitions ([Galaxy Devices Configuration](./galaxy_devices.md))
**Agent Role in System:**
```mermaid
graph TB
A[User Request] -->|Natural Language| B[Constellation Agent]
B -->|Uses LLM Config| C[agent.yaml]
B -->|Creates/Edits| D[Constellation Plan]
D -->|Tasks| E[Device Agent 1]
D -->|Tasks| F[Device Agent 2]
D -->|Tasks| G[Device Agent N]
style B fill:#e1f5ff
style C fill:#ffe1e1
style D fill:#fff4e1
```
---
## File Location
**Standard Location:**
```
UFO2/
├── config/
│ └── galaxy/
│ ├── agent.yaml # ← Constellation agent config (copy from template)
│ ├── agent.yaml.template # ← Template for initial setup
│ ├── constellation.yaml # ← Runtime settings
│ └── devices.yaml # ← Device definitions
```
!!!warning "Setup Required"
1. Copy `agent.yaml.template` to `agent.yaml`
2. Fill in your API credentials (API_KEY, AAD_TENANT_ID, etc.)
3. Never commit `agent.yaml` with real credentials to version control
**Loading in Code:**
```python
from config.config_loader import get_galaxy_config
# Load Galaxy configuration (includes agent settings)
config = get_galaxy_config()
# Access constellation agent settings
agent_config = config.constellation_agent
reasoning_model = agent_config.reasoning_model
api_type = agent_config.api_type
api_model = agent_config.api_model
```
---
## Configuration Schema
### Complete Schema
```yaml
# Galaxy Constellation Agent Configuration
CONSTELLATION_AGENT:
# Reasoning
REASONING_MODEL: bool # Enable reasoning/chain-of-thought
# API Connection
API_TYPE: string # API provider type
API_BASE: string # API base URL
API_KEY: string # API authentication key
API_VERSION: string # API version
API_MODEL: string # Model name/deployment
# Azure AD Authentication (for azure_ad API_TYPE)
AAD_TENANT_ID: string # Azure AD tenant ID
AAD_API_SCOPE: string # API scope name
AAD_API_SCOPE_BASE: string # API scope base GUID
# Prompt Configuration
CONSTELLATION_CREATION_PROMPT: string # Path to creation prompt template
CONSTELLATION_EDITING_PROMPT: string # Path to editing prompt template
CONSTELLATION_CREATION_EXAMPLE_PROMPT: string # Path to creation examples
CONSTELLATION_EDITING_EXAMPLE_PROMPT: string # Path to editing examples
```
---
## Configuration Fields
### Reasoning Capabilities
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `REASONING_MODEL` | `bool` | No | `False` | Enable chain-of-thought reasoning for complex planning |
**Example:**
```yaml
CONSTELLATION_AGENT:
REASONING_MODEL: False # Standard LLM response (faster)
```
!!!tip "Reasoning Model"
Set `REASONING_MODEL: True` for:
- Complex multi-device workflows
- Tasks requiring step-by-step planning
- Debugging constellation failures
**Trade-off:** Slower response time, higher token cost
---
### API Connection Settings
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `API_TYPE` | `string` | Yes | - | API provider: `"openai"`, `"azure"`, `"azure_ad"`, `"aoai"` |
| `API_BASE` | `string` | Yes* | - | API base URL (required for Azure) |
| `API_KEY` | `string` | Yes* | - | API authentication key (required for non-AAD auth) |
| `API_VERSION` | `string` | Yes* | - | API version (required for Azure) |
| `API_MODEL` | `string` | Yes | - | Model name or deployment name |
**Supported API Types:**
| API_TYPE | Provider | Authentication | Example API_BASE |
|----------|----------|----------------|------------------|
| `openai` | OpenAI | API Key | Not required (uses default) |
| `azure` | Azure OpenAI | API Key | `https://your-resource.openai.azure.com/` |
| `azure_ad` | Azure OpenAI | Azure AD (AAD) | `https://your-resource.azure-api.net/` |
| `aoai` | Azure OpenAI (alias) | API Key | `https://your-resource.openai.azure.com/` |
---
#### Example 1: OpenAI Configuration
```yaml
CONSTELLATION_AGENT:
API_TYPE: "openai"
API_KEY: "sk-proj-..." # Your OpenAI API key
API_MODEL: "gpt-4o" # OpenAI model name
API_VERSION: "2024-02-01" # Optional for OpenAI
```
---
#### Example 2: Azure OpenAI (API Key Auth)
```yaml
CONSTELLATION_AGENT:
API_TYPE: "azure"
API_BASE: "https://my-resource.openai.azure.com/"
API_KEY: "abc123..." # Azure OpenAI API key
API_VERSION: "2025-02-01-preview"
API_MODEL: "gpt-4o-deployment" # Your deployment name
```
---
#### Example 3: Azure OpenAI (Azure AD Auth)
```yaml
CONSTELLATION_AGENT:
API_TYPE: "azure_ad"
API_BASE: "https://cloudgpt-openai.azure-api.net/"
API_VERSION: "2025-02-01-preview"
API_MODEL: "gpt-5-chat-20251003"
# Azure AD Configuration
AAD_TENANT_ID: "72f988bf-86f1-41af-91ab-2d7cd011db47"
AAD_API_SCOPE: "openai"
AAD_API_SCOPE_BASE: "feb7b661-cac7-44a8-8dc1-163b63c23df2"
```
!!!warning "Azure AD Authentication"
When using `API_TYPE: "azure_ad"`:
- No `API_KEY` needed (uses Azure AD token)
- Requires `AAD_TENANT_ID`, `AAD_API_SCOPE`, `AAD_API_SCOPE_BASE`
- User must be authenticated with `az login` or have proper AAD credentials
---
### Azure AD Fields (azure_ad API_TYPE only)
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `AAD_TENANT_ID` | `string` | Yes* | Azure AD tenant GUID |
| `AAD_API_SCOPE` | `string` | Yes* | API scope identifier (e.g., "openai") |
| `AAD_API_SCOPE_BASE` | `string` | Yes* | API scope base GUID |
*Required only when `API_TYPE: "azure_ad"`
---
### Prompt Configuration Paths
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `CONSTELLATION_CREATION_PROMPT` | `string` | Yes | - | Path to constellation creation prompt template |
| `CONSTELLATION_EDITING_PROMPT` | `string` | Yes | - | Path to constellation editing prompt template |
| `CONSTELLATION_CREATION_EXAMPLE_PROMPT` | `string` | Yes | - | Path to creation examples (few-shot learning) |
| `CONSTELLATION_EDITING_EXAMPLE_PROMPT` | `string` | Yes | - | Path to editing examples (few-shot learning) |
**Default Prompt Paths:**
```yaml
CONSTELLATION_AGENT:
CONSTELLATION_CREATION_PROMPT: "galaxy/prompts/constellation/share/constellation_creation.yaml"
CONSTELLATION_EDITING_PROMPT: "galaxy/prompts/constellation/share/constellation_editing.yaml"
CONSTELLATION_CREATION_EXAMPLE_PROMPT: "galaxy/prompts/constellation/examples/constellation_creation_example.yaml"
CONSTELLATION_EDITING_EXAMPLE_PROMPT: "galaxy/prompts/constellation/examples/constellation_editing_example.yaml"
```
!!!tip "Custom Prompts"
You can customize prompts for your use case:
```yaml
CONSTELLATION_CREATION_PROMPT: "custom_prompts/my_constellation_creation.yaml"
```
---
## Complete Examples
### Example 1: Production (Azure AD)
```yaml
# Galaxy Constellation Agent Configuration - Production
# Uses Azure OpenAI with Azure AD authentication
CONSTELLATION_AGENT:
# Capabilities
REASONING_MODEL: False
# Azure OpenAI (Azure AD Auth)
API_TYPE: "azure_ad"
API_BASE: "https://cloudgpt-openai.azure-api.net/"
API_VERSION: "2025-02-01-preview"
API_MODEL: "gpt-5-chat-20251003"
# Azure AD Configuration
AAD_TENANT_ID: "72f988bf-86f1-41af-91ab-2d7cd011db47"
AAD_API_SCOPE: "openai"
AAD_API_SCOPE_BASE: "feb7b661-cac7-44a8-8dc1-163b63c23df2"
# Prompt Configurations
CONSTELLATION_CREATION_PROMPT: "galaxy/prompts/constellation/share/constellation_creation.yaml"
CONSTELLATION_EDITING_PROMPT: "galaxy/prompts/constellation/share/constellation_editing.yaml"
CONSTELLATION_CREATION_EXAMPLE_PROMPT: "galaxy/prompts/constellation/examples/constellation_creation_example.yaml"
CONSTELLATION_EDITING_EXAMPLE_PROMPT: "galaxy/prompts/constellation/examples/constellation_editing_example.yaml"
```
---
### Example 2: Development (OpenAI)
```yaml
# Galaxy Constellation Agent Configuration - Development
# Uses OpenAI API for quick testing
CONSTELLATION_AGENT:
# Capabilities
REASONING_MODEL: True # Enable for debugging
# OpenAI API
API_TYPE: "openai"
API_KEY: "sk-proj-..." # Your OpenAI API key (DO NOT COMMIT!)
API_MODEL: "gpt-4o"
API_VERSION: "2024-02-01"
# Prompt Configurations (default paths)
CONSTELLATION_CREATION_PROMPT: "galaxy/prompts/constellation/share/constellation_creation.yaml"
CONSTELLATION_EDITING_PROMPT: "galaxy/prompts/constellation/share/constellation_editing.yaml"
CONSTELLATION_CREATION_EXAMPLE_PROMPT: "galaxy/prompts/constellation/examples/constellation_creation_example.yaml"
CONSTELLATION_EDITING_EXAMPLE_PROMPT: "galaxy/prompts/constellation/examples/constellation_editing_example.yaml"
```
---
### Example 3: Azure OpenAI (API Key)
```yaml
# Galaxy Constellation Agent Configuration - Azure (API Key Auth)
# Uses Azure OpenAI with API key authentication
CONSTELLATION_AGENT:
# Capabilities
REASONING_MODEL: False
# Azure OpenAI (API Key Auth)
API_TYPE: "azure"
API_BASE: "https://my-openai-resource.openai.azure.com/"
API_KEY: "abc123..." # Azure OpenAI API key (DO NOT COMMIT!)
API_VERSION: "2025-02-01-preview"
API_MODEL: "gpt-4o-deployment-name"
# Prompt Configurations
CONSTELLATION_CREATION_PROMPT: "galaxy/prompts/constellation/share/constellation_creation.yaml"
CONSTELLATION_EDITING_PROMPT: "galaxy/prompts/constellation/share/constellation_editing.yaml"
CONSTELLATION_CREATION_EXAMPLE_PROMPT: "galaxy/prompts/constellation/examples/constellation_creation_example.yaml"
CONSTELLATION_EDITING_EXAMPLE_PROMPT: "galaxy/prompts/constellation/examples/constellation_editing_example.yaml"
```
---
## Security Best Practices
!!!danger "Never Commit Credentials"
**DO NOT commit `agent.yaml` with real credentials to version control!**
✅ **Recommended Workflow:**
```bash
# 1. Copy template
cp config/galaxy/agent.yaml.template config/galaxy/agent.yaml
# 2. Edit agent.yaml with your credentials
# (This file is .gitignored)
# 3. Commit only the template
git add config/galaxy/agent.yaml.template
git commit -m "Update agent template"
```
**Use Environment Variables for Sensitive Data:**
```yaml
# In agent.yaml
CONSTELLATION_AGENT:
API_KEY: ${GALAXY_API_KEY} # Read from environment variable
```
```bash
# In your shell
export GALAXY_API_KEY="sk-proj-..."
```
---
## Integration with Other Configurations
The agent configuration works together with other Galaxy configs:
**agent.yaml** (LLM config) + **constellation.yaml** (runtime) + **devices.yaml** (devices) → **Complete Galaxy System**
### Complete Initialization Example
```
/* Detailed source-code truncated for AI context efficiency. */
```
---
## Best Practices
**Configuration Best Practices:**
1. **Use Templates for Team Collaboration**
```bash
# Share template, not credentials
config/galaxy/agent.yaml.template # ✅ Commit this
config/galaxy/agent.yaml # ❌ Never commit this
```
2. **Test with OpenAI, Deploy with Azure**
```yaml
# Development: OpenAI (fast iteration)
API_TYPE: "openai"
# Production: Azure (enterprise features)
API_TYPE: "azure_ad"
```
3. **Use Reasoning Mode Selectively**
```yaml
# For complex workflows
REASONING_MODEL: True
# For simple tasks
REASONING_MODEL: False # Faster
```
---
## Related Documentation
| Topic | Document | Description |
|-------|----------|-------------|
| **Constellation Runtime** | [Galaxy Constellation Configuration](./galaxy_constellation.md) | Runtime settings for orchestrator |
| **Device Configuration** | [Galaxy Devices Configuration](./galaxy_devices.md) | Device definitions |
| **System Configuration** | [Configuration Overview](./overview.md) | Overall configuration architecture |
---
## Next Steps
1. **Copy Template**: `cp agent.yaml.template agent.yaml`
2. **Configure Credentials**: Fill in API_KEY or AAD settings
3. **Configure Runtime**: See [Galaxy Constellation Configuration](./galaxy_constellation.md)
4. **Configure Devices**: See [Galaxy Devices Configuration](./galaxy_devices.md)
5. **Test Constellation**: Run Galaxy orchestrator
---
## Source Code References
- **ConstellationAgent**: `galaxy/agents/constellation_agent.py`
- **Configuration Loading**: `config/config_loader.py`
- **Configuration Schemas**: `config/config_schemas.py`
- **Prompt Templates**: `galaxy/prompts/constellation/`
---
### Documents Docs Configuration System Galaxy Constellation (`documents/docs/configuration/system/galaxy_constellation.md`)
# Galaxy Constellation Runtime Configuration
**constellation.yaml** defines constellation-wide runtime settings that control how the Galaxy orchestrator manages devices, tasks, and logging across the entire constellation system.
---
## Overview
The **constellation.yaml** configuration file provides **constellation-level runtime settings** that apply to the entire Galaxy system. These settings control:
- Constellation identification and logging
- Heartbeat and connection management
- Task concurrency and step limits
- Device configuration file path
**Configuration Separation:**
- **constellation.yaml** - Runtime settings for the constellation orchestrator (this document)
- **devices.yaml** - Individual device definitions ([Galaxy Devices Configuration](./galaxy_devices.md))
- **agent.yaml** - LLM configuration for constellation agent ([Galaxy Agent Configuration](./galaxy_agent.md))
**Configuration Relationship:**
```mermaid
graph TB
A[constellation.yaml] -->|Runtime Settings| B[ConstellationDeviceManager]
C[devices.yaml] -->|Device Definitions| B
D[agent.yaml] -->|LLM Config| E[ConstellationAgent]
B -->|Orchestrates| F[Device Agents]
E -->|Plans Tasks| B
style A fill:#e1f5ff
style C fill:#fff4e1
style D fill:#ffe1e1
```
---
## File Location
**Standard Location:**
```
UFO2/
├── config/
│ └── galaxy/
│ ├── constellation.yaml # ← Runtime settings (this file)
│ ├── devices.yaml # ← Device definitions
│ └── agent.yaml.template # ← Agent LLM configuration template
```
**Loading in Code:**
```python
import yaml
from galaxy.client.device_manager import ConstellationDeviceManager
# Load constellation configuration
with open("config/galaxy/constellation.yaml", "r", encoding="utf-8") as f:
config = yaml.safe_load(f)
# Initialize ConstellationDeviceManager with runtime settings
manager = ConstellationDeviceManager(
task_name=config["CONSTELLATION_ID"],
heartbeat_interval=config["HEARTBEAT_INTERVAL"],
reconnect_delay=config["RECONNECT_DELAY"]
)
# Load device configuration from specified path
device_config_path = config["DEVICE_INFO"]
with open(device_config_path, "r", encoding="utf-8") as f:
devices_config = yaml.safe_load(f)
# Register devices
for device in devices_config["devices"]:
await manager.register_device(**device)
```
---
## Configuration Schema
### Complete Schema
```yaml
# Galaxy Constellation Configuration
# Runtime settings for constellation system
# Constellation Identity & Logging
CONSTELLATION_ID: string # Unique constellation identifier
LOG_TO_MARKDOWN: bool # Save trajectory logs to markdown
# Connection & Health Management
HEARTBEAT_INTERVAL: float # Heartbeat check interval (seconds)
RECONNECT_DELAY: float # Reconnection delay (seconds)
# Task & Execution Limits
MAX_CONCURRENT_TASKS: int # Maximum concurrent tasks
MAX_STEP: int # Maximum steps per session
# Device Configuration Reference
DEVICE_INFO: string # Path to devices.yaml file
```
---
## Configuration Fields
### Constellation Identity & Logging
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `CONSTELLATION_ID` | `string` | Yes | - | Unique identifier for this constellation instance |
| `LOG_TO_MARKDOWN` | `bool` | No | `true` | Whether to save trajectory logs in markdown format |
**Example:**
```yaml
CONSTELLATION_ID: "production_constellation"
LOG_TO_MARKDOWN: true
```
**Constellation ID Best Practices:**
Use descriptive names that indicate environment and purpose:
- `production_main` - Main production constellation
- `dev_testing` - Development testing constellation
- `qa_regression` - QA regression testing constellation
---
### Connection & Health Management
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `HEARTBEAT_INTERVAL` | `float` | No | `30.0` | Interval (in seconds) between heartbeat checks for connected devices |
| `RECONNECT_DELAY` | `float` | No | `5.0` | Delay (in seconds) before attempting to reconnect a failed device |
**Example:**
```yaml
HEARTBEAT_INTERVAL: 30.0 # Check device health every 30 seconds
RECONNECT_DELAY: 5.0 # Wait 5 seconds before reconnecting
```
!!!info "Heartbeat Mechanism"
The heartbeat system monitors device agent connections:
- Every `HEARTBEAT_INTERVAL` seconds, the constellation checks if devices are responsive
- If a device fails to respond, it is marked as `FAILED`
- After `RECONNECT_DELAY` seconds, automatic reconnection is attempted
- Reconnection continues until `max_retries` is reached (configured per-device in devices.yaml)
**Tuning Guidelines:**
| Environment | HEARTBEAT_INTERVAL | RECONNECT_DELAY | Rationale |
|-------------|-------------------|-----------------|-----------|
| **Production** | 10.0 - 30.0 | 5.0 - 10.0 | Balance responsiveness with network overhead |
| **Development** | 30.0 - 60.0 | 3.0 - 5.0 | Reduce noise during debugging |
| **Testing** | 5.0 - 10.0 | 2.0 - 3.0 | Faster failure detection for tests |
---
### Task & Execution Limits
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `MAX_CONCURRENT_TASKS` | `int` | No | `6` | Maximum number of tasks that can run concurrently across all devices |
| `MAX_STEP` | `int` | No | `15` | Maximum number of steps allowed per session before termination |
**Example:**
```yaml
MAX_CONCURRENT_TASKS: 6 # Allow 6 tasks to run simultaneously
MAX_STEP: 15 # Limit sessions to 15 steps
```
!!!warning "Concurrency Considerations"
- **MAX_CONCURRENT_TASKS** controls task queue parallelism across the entire constellation
- Each device can handle 1 task at a time (per device, not global)
- Example: 6 devices + MAX_CONCURRENT_TASKS=6 → All devices can be busy simultaneously
- Example: 10 devices + MAX_CONCURRENT_TASKS=4 → Only 4 devices busy at once, 6 idle
**Task Concurrency Calculation:**
```python
# Effective concurrency
effective_concurrency = min(
num_registered_devices,
MAX_CONCURRENT_TASKS
)
# Example 1: 3 devices, MAX_CONCURRENT_TASKS=6
# → effective_concurrency = 3 (device-limited)
# Example 2: 10 devices, MAX_CONCURRENT_TASKS=4
# → effective_concurrency = 4 (config-limited)
```
**MAX_STEP Guidelines:**
| Use Case | MAX_STEP | Rationale |
|----------|----------|-----------|
| **Simple Automation** | 5 - 10 | Quick tasks (open app, click button) |
| **Complex Workflows** | 15 - 30 | Multi-step processes (data entry, reporting) |
| **Unrestricted** | 100+ | Research, exploratory tasks |
---
### Device Configuration Reference
| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `DEVICE_INFO` | `string` | Yes | - | Relative or absolute path to `devices.yaml` configuration file |
**Example:**
```yaml
DEVICE_INFO: "config/galaxy/devices.yaml"
```
**Path Resolution:**
- **Relative paths** are resolved from the UFO2 project root
- **Absolute paths** are supported for external configuration files
- The loader validates that the file exists and is readable
**Example Paths:**
```yaml
# Relative path (recommended)
DEVICE_INFO: "config/galaxy/devices.yaml"
# Absolute path
DEVICE_INFO: "/etc/ufo/galaxy/devices.yaml"
# Different config for testing
DEVICE_INFO: "config/galaxy/devices_test.yaml"
```
---
## Complete Examples
### Example 1: Production Configuration
```yaml
# Galaxy Constellation Configuration - Production
# High reliability, moderate concurrency
# Identity & Logging
CONSTELLATION_ID: "production_main"
LOG_TO_MARKDOWN: true
# Connection & Health
HEARTBEAT_INTERVAL: 15.0 # Fast failure detection
RECONNECT_DELAY: 10.0 # Give devices time to recover
# Task Limits
MAX_CONCURRENT_TASKS: 10 # High concurrency for production load
MAX_STEP: 20 # Allow complex workflows
# Device Configuration
DEVICE_INFO: "config/galaxy/devices.yaml"
```
**Use Case:** Production constellation managing office automation across 10+ devices.
---
### Example 2: Development Configuration
```yaml
# Galaxy Constellation Configuration - Development
# Relaxed settings for testing and debugging
# Identity & Logging
CONSTELLATION_ID: "dev_testing"
LOG_TO_MARKDOWN: true
# Connection & Health
HEARTBEAT_INTERVAL: 60.0 # Reduce noise during debugging
RECONNECT_DELAY: 5.0 # Fast reconnects for quick iteration
# Task Limits
MAX_CONCURRENT_TASKS: 3 # Limit concurrency for easier debugging
MAX_STEP: 50 # Allow exploration and experimentation
# Device Configuration
DEVICE_INFO: "config/galaxy/devices_dev.yaml"
```
**Use Case:** Development environment with 2-3 test devices for feature development.
---
### Example 3: Testing/CI Configuration
```yaml
# Galaxy Constellation Configuration - CI/CD
# Fast failure detection, limited concurrency
# Identity & Logging
CONSTELLATION_ID: "ci_regression"
LOG_TO_MARKDOWN: true
# Connection & Health
HEARTBEAT_INTERVAL: 5.0 # Very fast detection for CI
RECONNECT_DELAY: 2.0 # Quick retries in CI environment
# Task Limits
MAX_CONCURRENT_TASKS: 4 # Parallel test execution
MAX_STEP: 15 # Strict limits for regression tests
# Device Configuration
DEVICE_INFO: "config/galaxy/devices_ci.yaml"
```
**Use Case:** Automated testing in CI/CD pipeline with controlled test devices.
---
## Integration with Device Configuration
The constellation configuration works together with device configuration:
**constellation.yaml (runtime)** + **devices.yaml (device definitions)** → **Complete Constellation System**
### Loading Workflow
```mermaid
sequenceDiagram
participant App as Application
participant Config as constellation.yaml
participant DevConfig as devices.yaml
participant Manager as ConstellationDeviceManager
App->>Config: Load constellation.yaml
Config-->>App: Runtime settings
App->>Manager: Initialize with runtime settings
Note over Manager: CONSTELLATION_ID, HEARTBEAT_INTERVAL, etc.
App->>Config: Read DEVICE_INFO path
Config-->>App: "config/galaxy/devices.yaml"
App->>DevConfig: Load devices.yaml from path
DevConfig-->>App: Device definitions
App->>Manager: Register devices
Manager->>Manager: Apply runtime settings to all devices
```
### Example: Complete Initialization
```python
import yaml
from galaxy.client.device_manager import ConstellationDeviceManager
# 1. Load constellation runtime settings
with open("config/galaxy/constellation.yaml", "r", encoding="utf-8") as f:
constellation_config = yaml.safe_load(f)
# 2. Initialize manager with runtime settings
manager = ConstellationDeviceManager(
task_name=constellation_config["CONSTELLATION_ID"],
heartbeat_interval=constellation_config["HEARTBEAT_INTERVAL"],
reconnect_delay=constellation_config["RECONNECT_DELAY"]
)
# 3. Load device configuration from path specified in constellation.yaml
device_config_path = constellation_config["DEVICE_INFO"]
with open(device_config_path, "r", encoding="utf-8") as f:
devices_config = yaml.safe_load(f)
# 4. Register all devices
for device in devices_config["devices"]:
await manager.register_device(
device_id=device["device_id"],
server_url=device["server_url"],
os=device.get("os"),
capabilities=device.get("capabilities", []),
metadata=device.get("metadata", {}),
max_retries=device.get("max_retries", 5),
auto_connect=device.get("auto_connect", True)
)
print(f"✅ Constellation '{constellation_config['CONSTELLATION_ID']}' initialized")
print(f" Devices registered: {len(devices_config['devices'])}")
print(f" Max concurrent tasks: {constellation_config['MAX_CONCURRENT_TASKS']}")
```
---
## Best Practices
**Configuration Best Practices:**
1. **Use Environment-Specific Configurations**
```bash
config/galaxy/
├── constellation.yaml # Base production config
├── constellation_dev.yaml # Development overrides
├── constellation_test.yaml # Testing overrides
```
2. **Tune Heartbeat for Your Network**
```yaml
# Local network - fast heartbeats
HEARTBEAT_INTERVAL: 10.0
# WAN/Internet - slower heartbeats
HEARTBEAT_INTERVAL: 60.0
```
3. **Match Concurrency to Use Case**
```yaml
# High-throughput automation
MAX_CONCURRENT_TASKS: 20
# Resource-constrained environment
MAX_CONCURRENT_TASKS: 3
```
4. **Set Reasonable Step Limits**
```yaml
# Prevent runaway sessions
MAX_STEP: 30
# For debugging (see all steps)
MAX_STEP: 100
```
---
## Related Documentation
| Topic | Document | Description |
|-------|----------|-------------|
| **Device Configuration** | [Galaxy Devices Configuration](./galaxy_devices.md) | Device definitions and capabilities |
| **Agent Configuration** | [Galaxy Agent Configuration](./galaxy_agent.md) | LLM settings for constellation agent |
| **Agent Registration** | [Agent Registration Overview](../../galaxy/agent_registration/overview.md) | Registration process and architecture |
| **System Configuration** | [Configuration Overview](./overview.md) | Overall configuration architecture |
---
## Next Steps
1. **Configure Devices**: See [Galaxy Devices Configuration](./galaxy_devices.md)
2. **Configure Agent**: See [Galaxy Agent Configuration](./galaxy_agent.md)
3. **Understand Registration**: Read [Agent Registration Overview](../../galaxy/agent_registration/overview.md)
4. **Run Constellation**: Check Galaxy orchestrator documentation
---
## Source Code References
- **ConstellationDeviceManager**: `galaxy/client/device_manager.py`
- **Configuration Loading**: `config/config_loader.py`
- **Configuration Schemas**: `config/config_schemas.py`
---
### Documents Docs Configuration System Galaxy Devices (`documents/docs/configuration/system/galaxy_devices.md`)
# Galaxy Devices Configuration
Device configuration in **devices.yaml** defines the constellation's device agents, providing device identity, capabilities, metadata, and connection parameters for each agent in the constellation.
---
## Overview
The **devices.yaml** configuration file defines the **devices array** for the Galaxy constellation system. It provides:
- Device identity and endpoint information
- User-specified capabilities
- Custom metadata and preferences
- Connection and retry parameters
**Constellation vs Device Configuration:**
- **devices.yaml** - Defines individual device agents (this document)
- **constellation.yaml** - Defines constellation-wide runtime settings
- See [Galaxy Constellation Configuration](./galaxy_constellation.md) for runtime settings
**Configuration Flow:**
```mermaid
graph LR
A[devices.yaml] -->|Load| B[ConstellationDeviceManager]
B -->|Parse| C[Device Entries]
C -->|For Each Device| D[DeviceRegistry.register_device]
D -->|Create| E[AgentProfile v1]
E -->|If auto_connect| F[Connection Process]
F -->|Merge| G[Complete AgentProfile]
style A fill:#e1f5ff
style E fill:#fff4e1
style G fill:#c8e6c9
```
---
## 📁 File Location
**Standard Location:**
```
UFO2/
├── config/
└── galaxy/
├── devices.yaml # 📄 Device definitions (this file)
├── constellation.yaml # ⚙️ Runtime settings
└── agent.yaml.template # 🤖 Agent LLM configuration template
```
**Loading in Code:**
```python
from galaxy.client.device_manager import ConstellationDeviceManager
import yaml
# Load device configuration
with open("config/galaxy/devices.yaml", "r", encoding="utf-8") as f:
devices_config = yaml.safe_load(f)
# Load constellation configuration
with open("config/galaxy/constellation.yaml", "r", encoding="utf-8") as f:
constellation_config = yaml.safe_load(f)
# Initialize manager with constellation settings
manager = ConstellationDeviceManager(
task_name=constellation_config.get("CONSTELLATION_ID", "default"),
heartbeat_interval=constellation_config.get("HEARTBEAT_INTERVAL", 30.0),
reconnect_delay=constellation_config.get("RECONNECT_DELAY", 5.0)
)
# Register devices from devices.yaml
for device_config in devices_config["devices"]:
await manager.register_device(
device_id=device_config["device_id"],
server_url=device_config["server_url"],
os=device_config.get("os"),
capabilities=device_config.get("capabilities", []),
metadata=device_config.get("metadata", {}),
max_retries=device_config.get("max_retries", 5),
auto_connect=device_config.get("auto_connect", True)
)
```
---
## 📝 Configuration Schema
### File Structure
```yaml
# Device Configuration - YAML Format
# Defines devices for the constellation
# Runtime settings are configured in constellation.yaml
devices: # List of device configurations
- device_id: string # Unique device identifier
server_url: string # WebSocket URL of device agent
os: string # Operating system
capabilities: list[string] # Device capabilities
metadata: dict # Custom metadata
max_retries: int # Connection retry limit
auto_connect: bool # Auto-connect on registration
```
---
### Device Configuration Fields
#### Required Fields
| Field | Type | Description | Example |
|-------|------|-------------|---------|
| `device_id` | `string` | **Unique device identifier** | `"windowsagent"`, `"linux_server_01"` |
| `server_url` | `string` | **WebSocket endpoint URL** | `"ws://localhost:5005/ws"` |
!!!danger "Required Fields"
`device_id` and `server_url` are **required** for every device. Registration will fail without them.
#### Server URL and Redirects
Set `server_url` to the final `ws://` or `wss://` endpoint that accepts the WebSocket upgrade directly. Galaxy pins outbound device connections to validated IP addresses and rejects HTTP redirects before opening a connection to a redirect destination.
!!!warning "Redirecting Endpoints Must Be Updated"
HTTP 301, 302, 303, 307, and 308 responses are rejected, including relative and same-origin redirects. The connection fails with an error containing `WebSocket redirects are not allowed for pinned connections`.
If an endpoint redirects, update its configured `server_url` to the final endpoint, including the correct scheme, port, and path. For a proxy that redirects `ws://` to `wss://`, configure the `wss://` URL directly. The final URL must still satisfy Galaxy's URL validation policy; allowing the original endpoint does not authorize a redirect destination.
This restriction prevents redirect-based server-side request forgery (SSRF), where an initially allowed endpoint redirects the handshake to an unvalidated internal address. Increasing `max_retries` does not resolve a redirect rejection.
See [Pinned Connections and Redirects](../../aip/transport.md#pinned-connections-and-redirects) for transport behavior.
#### Optional Fields
| Field | Type | Default | Description | Example |
|-------|------|---------|-------------|---------|
| `os` | `string` | `None` | Operating system type | `"windows"`, `"linux"`, `"darwin"` |
| `capabilities` | `list[string]` | `[]` | Device capabilities | `["web_browsing", "office"]` |
| `metadata` | `dict` | `{}` | Custom metadata | See [Metadata Fields](#metadata-fields) |
| `max_retries` | `int` | `5` | Maximum connection retries | `3`, `10` |
| `auto_connect` | `bool` | `true` | Auto-connect after registration | `true`, `false` |
!!!danger "Required Fields"
`device_id` and `server_url` are **required** for every device. Registration will fail without them.
---
---
### Metadata Fields
The `metadata` dictionary is **completely flexible** and can contain any custom fields. However, some common patterns are recommended:
**Recommended Metadata Fields:**
| Field | Type | Description | Example |
|-------|------|-------------|---------|
| `location` | `string` | Physical location | `"office_desktop"`, `"datacenter_rack_a42"` |
| `performance` | `string` | Performance tier | `"low"`, `"medium"`, `"high"`, `"very_high"` |
| `description` | `string` | Human-readable description | `"Primary Windows workstation"` |
| `tags` | `list[string]` | Custom tags | `["production", "gpu", "critical"]` |
| `operation_engineer_email` | `string` | Contact email | `"admin@example.com"` |
| `operation_engineer_name` | `string` | Contact name | `"John Doe"` |
**Custom Fields (Application-Specific):**
```yaml
metadata:
# File paths
logs_file_path: "/var/log/application.log"
dev_path: "/home/deploy/projects/"
app_log_file: "log_detailed.xlsx"
# Excel logging
sheet_name_for_writing_log_in_excel: "report"
# Email configuration
sender_name: "Automation Bot"
# Log patterns
warning_log_pattern: "WARN"
error_log_pattern: "ERROR or FATAL"
# GPU information
gpu_type: "NVIDIA RTX 4090"
gpu_count: 2
gpu_memory_gb: 48
```
---
## 📚 Complete Example
### Example 1: Multi-Device Constellation
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Example 2: Development Environment
```yaml
# Device Configuration - YAML Format
# Runtime settings are configured in constellation.yaml
devices:
- device_id: "dev_windows"
server_url: "ws://localhost:5005/ws"
os: "windows"
capabilities:
- "web_browsing"
- "office_applications"
metadata:
location: "developer_laptop"
performance: "medium"
description: "Development Windows machine"
environment: "development"
max_retries: 3
auto_connect: true
- device_id: "dev_linux"
server_url: "ws://localhost:5001/ws"
os: "linux"
capabilities:
- "cli"
- "file_system"
metadata:
location: "developer_laptop"
performance: "medium"
description: "Development Linux VM"
environment: "development"
max_retries: 3
auto_connect: false # Manual connection for debugging
```
---
## 🔄 Multi-Source Metadata Merging
The `metadata` field in configuration is **Source 1** in the multi-source profiling architecture. It will be merged with:
- **Source 2**: Service-level manifest (registration data)
- **Source 3**: Client telemetry (DeviceInfoProvider)
### Merging Process
```mermaid
graph TB
subgraph "Source 1: User Config"
UC[metadata in devices.yaml]
UC --> |location, performance, tags| Final
end
subgraph "Source 2: Service Manifest"
SM[AIP Registration]
SM --> |platform, registration_time| Final
end
subgraph "Source 3: Client Telemetry"
CT[DeviceInfoProvider]
CT --> |system_info object| Final
end
Final[Complete metadata in AgentProfile]
style UC fill:#e1f5ff
style SM fill:#fff4e1
style CT fill:#e8f5e9
style Final fill:#f3e5f5
```
**Before Merging (User Config Only):**
```yaml
metadata:
location: "office_desktop"
performance: "high"
description: "Primary Windows workstation"
```
**After Merging (All Sources):**
```python
metadata = {
# Source 1: User Config
"location": "office_desktop",
"performance": "high",
"description": "Primary Windows workstation",
# Source 2: Service Manifest
"platform": "windows",
"registration_time": "2025-11-06T10:30:00Z",
# Source 3: Client Telemetry
"system_info": {
"platform": "windows",
"os_version": "10.0.22631",
"cpu_count": 16,
"memory_total_gb": 32.0,
"hostname": "DESKTOP-DEV01",
"ip_address": "192.168.1.100",
"platform_type": "computer",
"schema_version": "1.0"
}
}
```
See [AgentProfile Documentation](../../galaxy/agent_registration/agent_profile.md#multi-source-construction) for merging details.
---
## 🎯 Use Cases and Patterns
### Pattern 1: Office Automation
```yaml
devices:
- device_id: "office_pc"
server_url: "ws://localhost:5005/ws"
os: "windows"
capabilities:
- "web_browsing"
- "office_applications"
- "email_sending"
- "file_management"
metadata:
location: "office_desktop"
performance: "medium"
description: "Office PC for daily automation tasks"
operation_engineer_email: "it@company.com"
sender_name: "Office Automation"
app_log_file: "office_automation.xlsx"
```
**Task Assignment:**
```python
# Find device with office capabilities
devices = manager.get_all_devices(connected=True)
for device_id, profile in devices.items():
if "office_applications" in profile.capabilities:
await manager.assign_task_to_device(
task_id="create_report",
device_id=device_id,
task_description="Create monthly report in Excel",
task_data={"template": "monthly_template.xlsx"}
)
```
### Pattern 2: Server Monitoring
```yaml
devices:
- device_id: "prod_server_01"
server_url: "ws://10.0.1.50:5001/ws"
os: "linux"
capabilities:
- "server_management"
- "log_monitoring"
metadata:
location: "datacenter_us_west"
performance: "high"
logs_file_path: "/var/log/app.log"
warning_log_pattern: "WARN"
error_log_pattern: "ERROR|FATAL"
```
**Task Assignment:**
```python
# Monitor server logs
await manager.assign_task_to_device(
task_id="monitor_logs",
device_id="prod_server_01",
task_description="Check logs for errors",
task_data={
"log_file": profile.metadata["logs_file_path"],
"error_pattern": profile.metadata["error_log_pattern"]
}
)
```
### Pattern 3: GPU Computation
```yaml
devices:
- device_id: "gpu_node_01"
server_url: "ws://192.168.1.100:5005/ws"
os: "linux"
capabilities:
- "gpu_computation"
- "model_training"
- "data_processing"
metadata:
location: "ml_lab_rack_01"
performance: "very_high"
gpu_type: "NVIDIA A100"
gpu_count: 4
gpu_memory_gb: 320 # 4 × 80GB
cpu_count: 96
memory_total_gb: 1024
```
**Task Assignment:**
```python
# Select GPU device based on metadata
devices = manager.get_all_devices(connected=True)
for device_id, profile in devices.items():
metadata = profile.metadata
if (
"gpu_computation" in profile.capabilities
and metadata.get("gpu_count", 0) >= 4
and metadata.get("gpu_memory_gb", 0) >= 300
):
await manager.assign_task_to_device(
task_id="train_model",
device_id=device_id,
task_description="Train large language model",
task_data={"model": "llama-70b", "dataset": "training_data.json"}
)
```
---
## ⚠️ Validation and Best Practices
### Required Field Validation
```python
def validate_device_config(device: dict) -> bool:
"""Validate device configuration."""
# Required fields
if "device_id" not in device:
logger.error("Missing required field: device_id")
return False
if "server_url" not in device:
logger.error("Missing required field: server_url")
return False
# Validate server_url format
if not device["server_url"].startswith("ws://") and \
not device["server_url"].startswith("wss://"):
logger.error(f"Invalid server_url: {device['server_url']}")
return False
return True
```
### Best Practices
!!!tip "Configuration Best Practices"
**1. Use Meaningful device_id**
```yaml
# ✅ Good: Descriptive and unique
device_id: "windows_office_pc_01"
device_id: "linux_prod_server_us_west_01"
device_id: "gpu_ml_workstation_lab_a"
# ❌ Bad: Generic or ambiguous
device_id: "device1"
device_id: "test"
device_id: "agent"
```
**2. Specify Granular Capabilities**
```yaml
# ✅ Good: Specific capabilities
capabilities:
- "web_browsing_chrome"
- "office_excel_automation"
- "email_outlook"
# ❌ Bad: Vague capabilities
capabilities:
- "office"
- "internet"
```
**3. Include Rich Metadata**
```yaml
# ✅ Good: Comprehensive metadata
metadata:
location: "datacenter_us_west_rack_a42"
performance: "very_high"
description: "Production GPU server for ML training"
tags: ["production", "ml", "gpu", "critical"]
operation_engineer_email: "ml-ops@company.com"
gpu_type: "NVIDIA A100"
gpu_count: 4
# ❌ Bad: Minimal metadata
metadata:
location: "server room"
```
**4. Set Appropriate max_retries**
```yaml
# Critical production devices
max_retries: 10
# Development/test devices
max_retries: 3
```
**5. Use auto_connect Wisely**
```yaml
# Production: auto-connect
auto_connect: true
# Development/debugging: manual connect
auto_connect: false
```
---
## 🔧 Loading and Parsing
### Basic Loading
```python
import yaml
with open("config/galaxy/devices.yaml", "r", encoding="utf-8") as f:
config = yaml.safe_load(f)
# Access constellation-level settings
constellation_id = config.get("constellation_id", "default")
heartbeat_interval = config.get("heartbeat_interval", 30.0)
# Access devices
devices = config.get("devices", [])
```
### Loading with Validation
```python
import yaml
from typing import Dict, List, Any
def load_and_validate_config(config_path: str) -> Dict[str, Any]:
"""Load and validate devices configuration."""
with open(config_path, "r", encoding="utf-8") as f:
config = yaml.safe_load(f)
# Validate top-level structure
if "devices" not in config:
raise ValueError("Configuration must contain 'devices' list")
if not isinstance(config["devices"], list):
raise ValueError("'devices' must be a list")
# Validate each device
for i, device in enumerate(config["devices"]):
if "device_id" not in device:
raise ValueError(f"Device {i}: Missing 'device_id'")
if "server_url" not in device:
raise ValueError(f"Device {i}: Missing 'server_url'")
# Validate URL format
if not device["server_url"].startswith(("ws://", "wss://")):
raise ValueError(
f"Device {device['device_id']}: Invalid server_url format"
)
return config
```
### Registration from Config
```python
async def register_devices_from_config(
manager: ConstellationDeviceManager,
config_path: str
) -> List[str]:
"""Register all devices from configuration file."""
config = load_and_validate_config(config_path)
registered = []
failed = []
for device_config in config["devices"]:
try:
success = await manager.register_device(
device_id=device_config["device_id"],
server_url=device_config["server_url"],
os=device_config.get("os"),
capabilities=device_config.get("capabilities", []),
metadata=device_config.get("metadata", {}),
max_retries=device_config.get("max_retries", 5),
auto_connect=device_config.get("auto_connect", True)
)
if success:
registered.append(device_config["device_id"])
else:
failed.append(device_config["device_id"])
except Exception as e:
logger.error(
f"Failed to register {device_config['device_id']}: {e}"
)
failed.append(device_config["device_id"])
logger.info(f"Registered: {len(registered)} devices")
if failed:
logger.warning(f"Failed: {len(failed)} devices - {failed}")
return registered
```
---
## 🔗 Related Documentation
| Topic | Document | Description |
|-------|----------|-------------|
| **Overview** | [Agent Registration Overview](./overview.md) | Registration architecture |
| **AgentProfile** | [AgentProfile](../../galaxy/agent_registration/agent_profile.md) | Profile structure and merging |
| **Registration Flow** | [Registration Flow](../../galaxy/agent_registration/registration_flow.md) | Registration process |
| **Device Registry** | [Device Registry](../../galaxy/agent_registration/device_registry.md) | Registry component |
| **Device Info** | [Device Info Provider](../../client/device_info.md) | Telemetry (Source 3) |
---
## 💡 Tips and Tricks
!!!tip "Advanced Configuration Tips"
**Use YAML Anchors for Reusable Metadata**
```yaml
# Define reusable metadata templates
_metadata_templates:
production_server: &prod_server
environment: "production"
tags: ["production", "critical"]
max_retries: 10
dev_server: &dev_server
environment: "development"
tags: ["development", "testing"]
max_retries: 3
devices:
- device_id: "prod_server_01"
server_url: "ws://10.0.1.50:5001/ws"
metadata:
<<: *prod_server # Merge production template
location: "datacenter_us_west"
- device_id: "dev_server_01"
server_url: "ws://localhost:5001/ws"
metadata:
<<: *dev_server # Merge dev template
location: "developer_laptop"
```
**Environment Variable Substitution**
```yaml
# Use environment variables for sensitive data
devices:
- device_id: "prod_server"
server_url: "${SERVER_URL}" # From environment
metadata:
api_key: "${API_KEY}"
```
---
## 🚀 Next Steps
1. **Create Your Configuration**: Copy example and customize
2. **Validate Configuration**: Use validation function
3. **Register Devices**: Load config and register
4. **Monitor Status**: Check device status after registration
---
## 📚 Source Code References
- **Example Config**: `config/galaxy/devices.yaml`
- **Loading Logic**: `galaxy/client/device_manager.py`
- **DeviceRegistry**: `galaxy/client/components/device_registry.py`
- **AgentProfile**: `galaxy/client/components/types.py`
---
### Documents Docs Configuration System Mcp Reference (`documents/docs/configuration/system/mcp_reference.md`)
# MCP Configuration Reference
This document provides a quick reference for MCP (Model Context Protocol) server configuration in UFO².
For comprehensive MCP configuration guide with examples, best practices, and detailed explanations, see:
- **[MCP Configuration Guide](../../mcp/configuration.md)** - Complete configuration documentation
- [MCP Overview](../../mcp/overview.md) - Architecture and concepts
- [Data Collection Servers](../../mcp/data_collection.md) - Observation tools
- [Action Servers](../../mcp/action.md) - Execution tools
## Quick Reference
**Configuration File**: `config/ufo/mcp.yaml`
### Structure
```yaml
AgentName: # e.g., "HostAgent", "AppAgent"
SubType: # "default" or app name (e.g., "WINWORD.EXE")
data_collection: # Data collection servers (read-only)
- namespace: ...
type: ... # "local", "http", or "stdio"
action: # Action servers (state-changing)
- namespace: ...
type: ...
```
### Server Types
| Type | Description | Use Case |
|------|-------------|----------|
| `local` | In-process server | Fast, built-in tools |
| `http` | Remote HTTP server | Cross-machine, language-agnostic |
| `stdio` | Child process via stdin/stdout | Process isolation |
### Common Fields
| Field | Type | Required | Description |
|-------|------|----------|-------------|
| `namespace` | String | ✅ Yes | Unique server identifier |
| `type` | String | ✅ Yes | Server type: `local`, `http`, or `stdio` |
| `reset` | Boolean | ❌ No | Reset on context switch (default: `false`) |
### Local Server Example
```yaml
HostAgent:
default:
data_collection:
- namespace: UICollector
type: local
start_args: []
reset: false
action:
- namespace: HostUIExecutor
type: local
reset: false
```
### HTTP Server Example
```yaml
HardwareAgent:
default:
data_collection:
- namespace: HardwareCollector
type: http
host: "localhost"
port: 8006
path: "/mcp"
reset: false
```
### Stdio Server Example
```yaml
CustomAgent:
default:
action:
- namespace: CustomProcessor
type: stdio
command: "python"
start_args: ["-m", "custom_mcp_server"]
env: {"API_KEY": "secret"}
cwd: "/path/to/server"
```
## Built-in Agent Configurations
### HostAgent (System-Level)
- **Data Collection**: UICollector
- **Actions**: HostUIExecutor, CommandLineExecutor
### AppAgent (Application-Level)
**Default**: UICollector, AppUIExecutor, CommandLineExecutor
**App-Specific**:
- **WINWORD.EXE**: + WordCOMExecutor
- **EXCEL.EXE**: + ExcelCOMExecutor
- **POWERPNT.EXE**: + PowerPointCOMExecutor
- **explorer.exe**: + PDFReaderExecutor
### ConstellationAgent
- **Actions**: ConstellationEditor
### HardwareAgent
- **Data Collection**: HardwareCollector (HTTP)
- **Actions**: HardwareExecutor (HTTP)
### LinuxAgent
- **Actions**: BashExecutor (HTTP)
## Reset Behavior
!!!tip "When to Use `reset: true`"
- **COM executors** (Word, Excel, PowerPoint) - Prevents state leakage between documents
- **Stateful tools** - Requires clean state per task
**Default: `false`** - Server persists across context switches
## Access in Code
```python
from config.config_loader import get_ufo_config
config = get_ufo_config()
mcp_config = config.MCP
# Get agent-specific config
host_agent = mcp_config.get("HostAgent", {})
app_agent = mcp_config.get("AppAgent", {})
# Get sub-type config
word_config = app_agent.get("WINWORD.EXE", app_agent.get("default", {}))
```
## Complete Documentation
For detailed configuration guide including:
- Complete field reference for all server types
- Agent-specific configuration examples
- Best practices and anti-patterns
- Configuration validation
- Debugging and troubleshooting
- Migration guide
See **[MCP Configuration Guide](../../mcp/configuration.md)**
!!!tip "Creating Custom MCP Servers"
Want to create your own MCP servers? See the **[Creating Custom MCP Servers Tutorial](../../tutorials/creating_mcp_servers.md)** for step-by-step instructions on building local, HTTP, and stdio servers.
## Related Documentation
- [MCP Overview](../../mcp/overview.md) - MCP architecture
- [Data Collection Servers](../../mcp/data_collection.md) - Read-only tools
- [Action Servers](../../mcp/action.md) - State-changing tools
- [Local Servers](../../mcp/local_servers.md) - Built-in servers
- [Remote Servers](../../mcp/remote_servers.md) - HTTP/Stdio deployment
- **[Creating Custom MCP Servers Tutorial](../../tutorials/creating_mcp_servers.md)** - Build your own servers
- [Configuration Overview](./overview.md) - General configuration system
- [System Configuration](./system_config.md) - MCP-related system settings
---
### Documents Docs Configuration System Migration (`documents/docs/configuration/system/migration.md`)
# Configuration Migration Guide
This guide helps you migrate from the legacy configuration system (`ufo/config/config.yaml`) to the new modular configuration system (`config/ufo/`).
**Migration Overview:** Migrating to the new configuration system is **optional but recommended**. Your existing configuration will continue to work, but the new system offers better organization, type safety, and IDE support.
## Why Migrate?
The new configuration system offers several advantages:
| Feature | Legacy (`ufo/config/`) | New (`config/ufo/`) |
|---------|----------------------|-------------------|
| **Structure** | Single monolithic YAML | Modular domain-specific files |
| **Type Safety** | Dict access only | Typed + dynamic access |
| **IDE Support** | No autocomplete | Full IntelliSense |
| **Scalability** | Hard to maintain | Easy to extend |
| **Documentation** | External docs | Self-documenting structure |
| **Environment Support** | Manual | Built-in dev/test/prod |
## Migration Methods
### Option 1: Automatic Migration (Recommended)
Use the built-in migration tool:
**Automatic Migration Tool**:
```bash
# From UFO2 root directory
python -m ufo.tools.migrate_config
# Or with options
python -m ufo.tools.migrate_config --backup --validate
```
**What it does**:
1. ✅ Reads your legacy `ufo/config/config.yaml`
2. ✅ Splits into modular files by domain
3. ✅ Creates backup of original file
4. ✅ Validates the new configuration
5. ✅ Provides migration report
!!!warning "Backup Reminder"
Always backup your configuration before migration! The tool creates a backup automatically, but it's good practice to keep your own copy.
### Option 2: Manual Migration
Step-by-step manual migration process.
#### Step 1: Create Directory Structure
```bash
# Create new config directories
mkdir -p config/ufo
mkdir -p config/galaxy # If using Galaxy
```
#### Step 2: Copy Templates
```bash
# Copy template files
cp config/ufo/agents.yaml.template config/ufo/agents.yaml
cp config/galaxy/agent.yaml.template config/galaxy/agent.yaml # If using Galaxy
```
#### Step 3: Split Configuration
Split your `ufo/config/config.yaml` into modular files:
**Legacy config.yaml**:
```yaml
# ufo/config/config.yaml (OLD - Monolithic)
HOST_AGENT:
API_TYPE: "openai"
API_KEY: "sk-..."
API_MODEL: "gpt-4o"
APP_AGENT:
API_TYPE: "openai"
API_KEY: "sk-..."
API_MODEL: "gpt-4o"
MAX_STEP: 50
MAX_RETRY: 20
TEMPERATURE: 0.0
RAG_OFFLINE_DOCS: False
RAG_EXPERIENCE: True
```
**New modular structure**:
`config/ufo/agents.yaml`:
```yaml
# Agent LLM configurations
HOST_AGENT:
API_TYPE: "openai"
API_KEY: "sk-..."
API_MODEL: "gpt-4o"
APP_AGENT:
API_TYPE: "openai"
API_KEY: "sk-..."
API_MODEL: "gpt-4o"
```
`config/ufo/system.yaml`:
```yaml
# System and runtime configurations
MAX_STEP: 50
MAX_RETRY: 20
TEMPERATURE: 0.0
```
`config/ufo/rag.yaml`:
```yaml
# RAG knowledge configurations
RAG_OFFLINE_DOCS: False
RAG_EXPERIENCE: True
```
#### Step 4: Verify Configuration
**Verification Script**:
```python
# Test your new configuration
from config.config_loader import get_ufo_config
config = get_ufo_config()
# Verify values loaded correctly
print(f"Max step: {config.system.max_step}")
print(f"Host agent model: {config.host_agent.api_model}")
print(f"RAG experience: {config.rag.experience}")
```
#### Step 5: Update Code (Optional)
Modernize configuration access patterns:
```python
# OLD (still works but deprecated)
config = Config()
max_step = config["MAX_STEP"]
api_model = config["HOST_AGENT"]["API_MODEL"]
# NEW (recommended)
config = get_ufo_config()
max_step = config.system.max_step # Type-safe!
api_model = config.host_agent.api_model # IDE autocomplete!
```
#### Step 6: Clean Up Legacy Config
!!!danger "Remove Legacy Config Only After Verification"
Only remove the legacy config after thoroughly testing that the new configuration works correctly!
```bash
# Backup legacy config
cp ufo/config/config.yaml ufo/config/config.yaml.backup
# Remove legacy config (after verifying new config works)
rm ufo/config/config.yaml
```
## Field Mapping Reference
### Agent Configurations
| Legacy Location | New Location | Notes |
|----------------|--------------|-------|
| `HOST_AGENT.*` | `config/ufo/agents.yaml` → `HOST_AGENT.*` | Same structure |
| `APP_AGENT.*` | `config/ufo/agents.yaml` → `APP_AGENT.*` | Same structure |
| `BACKUP_AGENT.*` | `config/ufo/agents.yaml` → `BACKUP_AGENT.*` | Same structure |
| `EVALUATION_AGENT.*` | `config/ufo/agents.yaml` → `EVALUATION_AGENT.*` | Same structure |
| `OPERATOR.*` | `config/ufo/agents.yaml` → `OPERATOR.*` | New in UFO² |
### System Configurations
| Legacy Field | New Location | New Access Pattern |
|-------------|--------------|-------------------|
| `MAX_STEP` | `config/ufo/system.yaml` | `config.system.max_step` |
| `MAX_RETRY` | `config/ufo/system.yaml` | `config.system.max_retry` |
| `TEMPERATURE` | `config/ufo/system.yaml` | `config.system.temperature` |
| `CONTROL_BACKEND` | `config/ufo/system.yaml` | `config.system.control_backend` |
| `ACTION_SEQUENCE` | `config/ufo/system.yaml` | `config.system.action_sequence` |
### RAG Configurations
| Legacy Field | New Location | New Access Pattern |
|-------------|--------------|-------------------|
| `RAG_OFFLINE_DOCS` | `config/ufo/rag.yaml` | `config.rag.offline_docs` |
| `RAG_EXPERIENCE` | `config/ufo/rag.yaml` | `config.rag.experience` |
| `RAG_DEMONSTRATION` | `config/ufo/rag.yaml` | `config.rag.demonstration` |
| `BING_API_KEY` | `config/ufo/rag.yaml` | `config.rag.BING_API_KEY` |
### MCP Configurations
| Legacy Field | New Location | Notes |
|-------------|--------------|-------|
| `USE_MCP` | `config/ufo/system.yaml` | Keep in system config |
| `MCP_SERVERS_CONFIG` | `config/ufo/system.yaml` | Points to `config/ufo/mcp.yaml` |
| MCP server definitions | `config/ufo/mcp.yaml` | New dedicated file |
## Common Migration Scenarios
### Scenario 1: Different Models for Different Agents
**Legacy approach** (duplicated config):
```yaml
# ufo/config/config.yaml
HOST_AGENT:
API_MODEL: "gpt-4o"
# ... other settings
APP_AGENT:
API_MODEL: "gpt-4o-mini" # Different model
# ... other settings
```
**New approach** (clear separation):
```yaml
# config/ufo/agents.yaml
HOST_AGENT:
API_MODEL: "gpt-4o"
APP_AGENT:
API_MODEL: "gpt-4o-mini"
```
### Scenario 2: Environment-Specific Settings
**Legacy approach** (manual switching):
```yaml
# ufo/config/config.yaml
# Manually comment/uncomment for different environments
# MAX_STEP: 10 # Development
MAX_STEP: 50 # Production
```
**New approach** (automatic environment support):
```yaml
# config/ufo/system.yaml (base)
MAX_STEP: 50
# config/ufo/system_dev.yaml (development override)
MAX_STEP: 10
LOG_LEVEL: "DEBUG"
```
```bash
# Set environment
export UFO_ENV=dev # Automatically uses system_dev.yaml overrides
```
### Scenario 3: Custom Experimental Features
**Legacy approach** (modify code):
```python
# Had to modify Config class
class Config:
def __init__(self):
self.MY_CUSTOM_FEATURE = True # Added to code
```
**New approach** (just add to YAML):
```yaml
# config/ufo/custom.yaml (new file)
MY_CUSTOM_FEATURE: True
EXPERIMENTAL_SETTING: "value"
```
```python
# Automatically available
config = get_ufo_config()
if config.MY_CUSTOM_FEATURE:
value = config.EXPERIMENTAL_SETTING
```
## Validation After Migration
### 1. Test Configuration Loading
```python
from config.config_loader import get_ufo_config
# Load configuration
config = get_ufo_config()
# Verify critical settings
assert config.system.max_step > 0
assert config.host_agent.api_key != ""
assert config.app_agent.api_model != ""
print("✅ Configuration loaded successfully!")
```
### 2. Test Backward Compatibility
```python
# Old access patterns should still work
config = get_ufo_config()
# Dict-style access (legacy)
max_step_old = config["MAX_STEP"]
host_agent_old = config["HOST_AGENT"]
# Verify they match new access
assert max_step_old == config.system.max_step
assert host_agent_old["API_MODEL"] == config.host_agent.api_model
print("✅ Backward compatibility verified!")
```
### 3. Run Application Tests
```bash
# Test with simple task
python -m ufo --task "Open Notepad"
# Check logs for configuration warnings
# Should not see "LEGACY CONFIG PATH DETECTED" after migration
```
## Troubleshooting
### Issue: "No configuration found"
**Cause**: Configuration files not in expected locations
!!!bug "Solution"
Verify file locations and permissions
```bash
# Verify file locations
ls config/ufo/agents.yaml
ls config/ufo/system.yaml
# Check file permissions
chmod 644 config/ufo/*.yaml
```
### Issue: "Configuration conflicts detected"
**Cause**: Both legacy and new configs exist
!!!warning "Conflict Resolution"
Choose one of these options to resolve conflicts
```bash
# Option 1: Remove legacy config (after backup)
mv ufo/config/config.yaml ufo/config/config.yaml.backup
# Option 2: Disable automatic fallback (in code)
config = get_ufo_config() # Will warn but use new path
```
### Issue: "Missing required fields"
**Cause**: Required fields not present in new configuration
!!!failure "Required Fields Missing"
Ensure all required agent fields are present
```yaml
# config/ufo/agents.yaml
# Ensure all required agent fields present:
HOST_AGENT:
API_TYPE: "openai" # Required
API_BASE: "..." # Required
API_KEY: "..." # Required
API_MODEL: "..." # Required
```
```
### Issue: "Type errors in code"
**Cause**: Using old dict-style access with new typed config
**Solution**:
```python
# OLD (can cause type issues)
config["HOST_AGENT"]["API_MODEL"]
# NEW (type-safe)
config.host_agent.api_model
# Or keep old style for now
config["HOST_AGENT"]["API_MODEL"] # Still works!
```
## Migration Checklist
- [ ] Backup legacy configuration
- [ ] Create `config/ufo/` directory
- [ ] Copy and customize template files
- [ ] Split monolithic config into modular files
- [ ] Test configuration loading
- [ ] Verify backward compatibility
- [ ] Update code to use new access patterns (optional)
- [ ] Run application tests
- [ ] Remove legacy configuration (after verification)
- [ ] Update documentation/README
- [ ] Commit changes to version control
## Rollback Procedure
If migration causes issues:
!!!danger "Emergency Rollback"
Your application will immediately fall back to the legacy configuration without any code changes.
```bash
# 1. Restore legacy config from backup
cp ufo/config/config.yaml.backup ufo/config/config.yaml
# 2. Remove new config files
rm -rf config/ufo/*.yaml
# 3. Restart application
# Old configuration will be used automatically
```
## Getting Help
If you encounter issues during migration:
1. **Check the logs** for detailed error messages
2. **Review configuration guides** ([Agents Config](./agents_config.md), [System Config](./system_config.md), [RAG Config](./rag_config.md)) for correct field names
3. **Consult [Configuration Overview](./overview.md)** for system design
4. **Open an issue** on GitHub with:
- Your legacy config (redacted sensitive data)
- Error messages
- Steps you've tried
## Next Steps
After successful migration:
- **[Agents Configuration](./agents_config.md)** - Configure LLM and agent settings
- **[System Configuration](./system_config.md)** - Configure runtime and execution settings
- **[RAG Configuration](./rag_config.md)** - Configure knowledge retrieval
- **[Extending Configuration](./extending.md)** - Learn how to add custom settings
---
### Documents Docs Configuration System Prices Config (`documents/docs/configuration/system/prices_config.md`)
# Pricing Configuration (prices.yaml)
Configure token pricing for different LLM models to track and estimate API costs during UFO² execution.
---
## Overview
The `prices.yaml` file defines the cost per 1,000 tokens for different LLM models. UFO² uses this information to calculate and report the estimated cost of task executions.
**File Location**: `config/ufo/prices.yaml`
!!!warning "Pricing May Be Outdated"
The pricing information in this file **may not be current**. LLM providers frequently update their pricing.
- Always verify current pricing on provider websites
- This file will be updated periodically
- Use these values as estimates only
---
## Quick Start
### View Current Pricing
```yaml
# config/ufo/prices.yaml
gpt-4o:
prompt: 0.0025
completion: 0.01
gpt-4o-mini:
prompt: 0.00015
completion: 0.0006
gpt-4-turbo:
prompt: 0.01
completion: 0.03
```
### Add Your Model
```yaml
# Add pricing for your custom model
my-custom-model:
prompt: 0.001 # USD per 1K prompt tokens
completion: 0.003 # USD per 1K completion tokens
```
---
## Configuration Format
### Structure
Each model has two pricing fields:
```yaml
model-name:
prompt:
completion:
```
| Field | Type | Unit | Description |
|-------|------|------|-------------|
| `prompt` | Float | USD/1K tokens | Cost per 1,000 input (prompt) tokens |
| `completion` | Float | USD/1K tokens | Cost per 1,000 output (completion) tokens |
---
## Common Models (As of Template)
!!!info "Verify Current Pricing"
These prices are from the template and **may be outdated**. Always check provider websites for current pricing:
- [OpenAI Pricing](https://openai.com/pricing)
- [Azure OpenAI Pricing](https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/)
- [Anthropic Pricing](https://www.anthropic.com/pricing)
- [Google AI Pricing](https://ai.google.dev/pricing)
### OpenAI Models
| Model | Prompt ($/1K) | Completion ($/1K) | Notes |
|-------|---------------|-------------------|-------|
| `gpt-4o` | $0.0025 | $0.01 | Latest GPT-4 optimized |
| `gpt-4o-mini` | $0.00015 | $0.0006 | Cheaper alternative |
| `gpt-4-turbo` | $0.01 | $0.03 | GPT-4 Turbo |
| `gpt-4-vision-preview` | $0.01 | $0.03 | GPT-4 with vision |
| `gpt-3.5-turbo` | $0.0005 | $0.0015 | GPT-3.5 |
### Example Configuration
```yaml
# OpenAI Models
gpt-4o:
prompt: 0.0025
completion: 0.01
gpt-4o-mini:
prompt: 0.00015
completion: 0.0006
gpt-4-turbo:
prompt: 0.01
completion: 0.03
gpt-4-vision-preview:
prompt: 0.01
completion: 0.03
gpt-3.5-turbo:
prompt: 0.0005
completion: 0.0015
# Claude Models (example)
claude-3-5-sonnet-20241022:
prompt: 0.003
completion: 0.015
# Gemini Models (example)
gemini-2.0-flash-exp:
prompt: 0.0
completion: 0.0
```
---
## Cost Tracking
UFO² automatically tracks costs when pricing information is available.
### During Execution
```python
# UFO² automatically calculates costs
Session logs show:
- Total prompt tokens used
- Total completion tokens used
- Estimated cost (based on prices.yaml)
```
### View Cost Summary
After task execution, check logs:
```
logs//cost_summary.json
```
**Example output**:
```json
{
"total_cost_usd": 0.15,
"prompt_tokens": 5000,
"completion_tokens": 2000,
"model": "gpt-4o"
}
```
---
## Updating Pricing
### Step 1: Check Current Pricing
Visit your LLM provider's pricing page:
- **OpenAI**: https://openai.com/pricing
- **Azure OpenAI**: https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/
- **Anthropic**: https://www.anthropic.com/pricing
- **Google**: https://ai.google.dev/pricing
### Step 2: Update prices.yaml
```yaml
# Update with current pricing
gpt-4o:
prompt: 0.0025 # Update if changed
completion: 0.01
```
### Step 3: Add New Models
```yaml
# Add newly released models
gpt-5:
prompt: 0.005
completion: 0.02
```
---
## Programmatic Access
```python
from config.config_loader import get_ufo_config
config = get_ufo_config()
# Get pricing for a specific model
model_name = "gpt-4o"
if model_name in config.prices:
prompt_cost = config.prices[model_name]["prompt"]
completion_cost = config.prices[model_name]["completion"]
print(f"{model_name}:")
print(f" Prompt: ${prompt_cost}/1K tokens")
print(f" Completion: ${completion_cost}/1K tokens")
else:
print(f"No pricing info for {model_name}")
```
---
## Cost Estimation Example
```python
# Example: Estimate cost for a task
prompt_tokens = 10000 # 10K prompt tokens
completion_tokens = 5000 # 5K completion tokens
model = "gpt-4o"
# Get pricing
prompt_cost_per_1k = 0.0025
completion_cost_per_1k = 0.01
# Calculate
total_cost = (
(prompt_tokens / 1000) * prompt_cost_per_1k +
(completion_tokens / 1000) * completion_cost_per_1k
)
print(f"Estimated cost: ${total_cost:.4f}")
# Output: Estimated cost: $0.0750
```
---
## Notes
!!!info "Important Notes"
- ✅ Pricing is for **cost estimation only**, not billing
- ✅ Actual costs may vary based on your provider contract
- ✅ Different Azure regions may have different pricing
- ✅ Some models have tiered pricing based on volume
- ✅ Prices change frequently - update regularly
---
## Related Documentation
- **[Agent Configuration](agents_config.md)** - LLM model selection
- **[System Configuration](system_config.md)** - Token limits and usage
---
## Summary
!!!success "Key Takeaways"
✅ **prices.yaml tracks LLM costs** - Estimates API spending
✅ **Pricing may be outdated** - Always verify current rates
✅ **Update regularly** - Providers change pricing frequently
✅ **Add new models** - Include pricing for any custom models
✅ **Cost tracking is automatic** - UFO² calculates costs during execution
**Keep pricing updated for accurate cost tracking!** 💰
---
### Documents Docs Configuration System Rag Config (`documents/docs/configuration/system/rag_config.md`)
# RAG Configuration (rag.yaml)
Configure Retrieval-Augmented Generation (RAG) to enhance UFO² with external knowledge sources, online search, experience learning, and demonstration-based learning.
---
## Overview
The `rag.yaml` file configures knowledge retrieval systems that augment UFO²'s capabilities beyond its base LLM knowledge. RAG helps UFO² make better decisions by providing:
- **Offline Documentation**: Application manuals and documentation
- **Online Search**: Real-time web search via Bing
- **Experience Learning**: Learn from past successful executions
- **Demonstration Learning**: Learn from user demonstrations
**File Location**: `config/ufo/rag.yaml`
**Optional Configuration:** RAG features are **optional**. UFO² works without them, but they can significantly improve performance on complex or domain-specific tasks.
---
## Quick Start
### Disable All RAG (Default)
```yaml
# Minimal configuration - no external knowledge
RAG_OFFLINE_DOCS: False
RAG_ONLINE_SEARCH: False
RAG_EXPERIENCE: False
RAG_DEMONSTRATION: False
```
### Enable Online Search Only
```yaml
# Most useful for general tasks
RAG_OFFLINE_DOCS: False
RAG_ONLINE_SEARCH: True
BING_API_KEY: "YOUR_BING_API_KEY_HERE"
RAG_ONLINE_SEARCH_TOPK: 5
RAG_ONLINE_RETRIEVED_TOPK: 5
RAG_EXPERIENCE: False
RAG_DEMONSTRATION: False
```
### Enable Experience Learning
```yaml
# Learn from past executions
RAG_OFFLINE_DOCS: False
RAG_ONLINE_SEARCH: False
RAG_EXPERIENCE: True
RAG_EXPERIENCE_RETRIEVED_TOPK: 5
RAG_DEMONSTRATION: False
```
### Enable All Features
```yaml
# Maximum knowledge augmentation
RAG_OFFLINE_DOCS: True
RAG_OFFLINE_DOCS_RETRIEVED_TOPK: 1
RAG_ONLINE_SEARCH: True
BING_API_KEY: "YOUR_BING_API_KEY_HERE"
RAG_ONLINE_SEARCH_TOPK: 5
RAG_ONLINE_RETRIEVED_TOPK: 5
RAG_EXPERIENCE: True
RAG_EXPERIENCE_RETRIEVED_TOPK: 5
RAG_DEMONSTRATION: True
RAG_DEMONSTRATION_RETRIEVED_TOPK: 5
```
---
## RAG Components
### 1. Offline Documentation
Retrieve relevant documentation from local knowledge bases (app manuals, guides, API docs).
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `RAG_OFFLINE_DOCS` | Boolean | `False` | Enable offline documentation retrieval |
| `RAG_OFFLINE_DOCS_RETRIEVED_TOPK` | Integer | `1` | Number of documents to retrieve |
**Example**:
```yaml
RAG_OFFLINE_DOCS: True
RAG_OFFLINE_DOCS_RETRIEVED_TOPK: 1
```
!!!info "Use Case"
- Application-specific tasks (Excel formulas, Word formatting)
- Domain-specific workflows (accounting, design)
- Requires pre-indexed documentation
**Setup**:
1. Place documentation in `vectordb/docs/`
2. Index documents: `python -m learner`
3. Enable in `rag.yaml`
---
### 2. Online Search
Search the web in real-time using Bing Search API.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `RAG_ONLINE_SEARCH` | Boolean | `False` | Enable online Bing search |
| `BING_API_KEY` | String | `""` | Bing Search API key |
| `RAG_ONLINE_SEARCH_TOPK` | Integer | `5` | Number of search results to fetch |
| `RAG_ONLINE_RETRIEVED_TOPK` | Integer | `5` | Number of results to include in prompt |
**Example**:
```yaml
RAG_ONLINE_SEARCH: True
BING_API_KEY: "abc123xyz..."
RAG_ONLINE_SEARCH_TOPK: 5
RAG_ONLINE_RETRIEVED_TOPK: 5
```
!!!tip "Getting Bing API Key"
1. Go to [Azure Portal](https://portal.azure.com)
2. Create a "Bing Search v7" resource
3. Copy the API key from "Keys and Endpoint"
4. Add to `rag.yaml`: `BING_API_KEY: "your-key"`
**Use Cases**:
- Tasks requiring current information
- Unfamiliar applications or features
- Troubleshooting specific error messages
- Finding how-to guides dynamically
**Example Query Flow**:
```
User Request: "Create a pivot table in Excel"
↓
Bing Search: "how to create pivot table in Excel"
↓
Retrieved: Top 5 results about pivot tables
↓
LLM receives context from search results
↓
Better action decisions
```
---
### 3. Experience Learning
Learn from UFO²'s own past successful task executions.
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `RAG_EXPERIENCE` | Boolean | `False` | Enable experience learning |
| `RAG_EXPERIENCE_RETRIEVED_TOPK` | Integer | `5` | Number of past experiences to retrieve |
| `EXPERIENCE_SAVED_PATH` | String | Auto-generated | Path to experience database |
| `EXPERIENCE_PROMPT` | String | Auto-generated | Experience prompt template |
**Example**:
```yaml
RAG_EXPERIENCE: True
RAG_EXPERIENCE_RETRIEVED_TOPK: 5
```
!!!info "How It Works"
1. UFO² completes a task successfully
2. Task steps are saved to experience database
3. For future similar tasks, relevant past experiences are retrieved
4. LLM learns from successful patterns
**Use Cases**:
- Repetitive tasks with slight variations
- Learning organizational-specific workflows
- Improving over time on common tasks
**Example**:
```
First Time: "Create a monthly sales report"
→ Task succeeds, 15 steps recorded
Second Time: "Create a quarterly sales report"
→ Retrieves "monthly report" experience
→ Adapts the pattern, faster execution
```
**Default Paths**:
```yaml
# Auto-generated if not specified
EXPERIENCE_SAVED_PATH: "vectordb/experience"
EXPERIENCE_PROMPT: "ufo/prompts/share/experience/experience.yaml"
```
---
### 4. Demonstration Learning
Learn from user demonstrations (you show UFO² how to do a task).
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `RAG_DEMONSTRATION` | Boolean | `False` | Enable demonstration learning |
| `RAG_DEMONSTRATION_RETRIEVED_TOPK` | Integer | `5` | Number of demonstrations to retrieve |
| `DEMONSTRATION_SAVED_PATH` | String | Auto-generated | Path to demonstration database |
| `DEMONSTRATION_PROMPT` | String | Auto-generated | Demonstration prompt template |
**Example**:
```yaml
RAG_DEMONSTRATION: True
RAG_DEMONSTRATION_RETRIEVED_TOPK: 5
```
!!!info "How It Works"
1. User demonstrates a task (UFO² records it)
2. Demonstration is saved with annotations
3. For similar future tasks, demonstrations are retrieved
4. LLM mimics the demonstrated behavior
**Use Cases**:
- Complex, domain-specific workflows
- Organizational-specific procedures
- Tasks with many edge cases
**Workflow**:
```
1. Record Demonstration:
python -m ufo --mode demonstration
→ Perform task manually
→ UFO² records your actions
2. Save Demonstration:
→ Stored in vectordb/demonstration/
3. Future Task:
"Do the same report formatting"
→ Retrieves your demonstration
→ Replicates your steps
```
**Default Paths**:
```yaml
# Auto-generated if not specified
DEMONSTRATION_SAVED_PATH: "vectordb/demonstration"
DEMONSTRATION_PROMPT: "ufo/prompts/share/demonstration/demonstration.yaml"
```
---
## Complete Configuration Examples
### Minimal (No RAG)
```yaml
# config/ufo/rag.yaml
RAG_OFFLINE_DOCS: False
RAG_ONLINE_SEARCH: False
RAG_EXPERIENCE: False
RAG_DEMONSTRATION: False
```
### Online Search Only
```yaml
RAG_OFFLINE_DOCS: False
RAG_ONLINE_SEARCH: True
BING_API_KEY: "your-bing-api-key-here"
RAG_ONLINE_SEARCH_TOPK: 5
RAG_ONLINE_RETRIEVED_TOPK: 5
RAG_EXPERIENCE: False
RAG_DEMONSTRATION: False
```
### Experience Learning Only
```yaml
RAG_OFFLINE_DOCS: False
RAG_ONLINE_SEARCH: False
RAG_EXPERIENCE: True
RAG_EXPERIENCE_RETRIEVED_TOPK: 5
EXPERIENCE_SAVED_PATH: "vectordb/experience"
EXPERIENCE_PROMPT: "ufo/prompts/share/experience/experience.yaml"
RAG_DEMONSTRATION: False
```
### Full RAG Setup
```yaml
# Offline docs
RAG_OFFLINE_DOCS: True
RAG_OFFLINE_DOCS_RETRIEVED_TOPK: 1
# Online search
RAG_ONLINE_SEARCH: True
BING_API_KEY: "your-bing-api-key"
RAG_ONLINE_SEARCH_TOPK: 5
RAG_ONLINE_RETRIEVED_TOPK: 5
# Experience
RAG_EXPERIENCE: True
RAG_EXPERIENCE_RETRIEVED_TOPK: 5
EXPERIENCE_SAVED_PATH: "vectordb/experience"
EXPERIENCE_PROMPT: "ufo/prompts/share/experience/experience.yaml"
# Demonstration
RAG_DEMONSTRATION: True
RAG_DEMONSTRATION_RETRIEVED_TOPK: 5
DEMONSTRATION_SAVED_PATH: "vectordb/demonstration"
DEMONSTRATION_PROMPT: "ufo/prompts/share/demonstration/demonstration.yaml"
```
---
## Setting Up Each RAG Component
### Setup: Offline Documentation
**Step 1**: Prepare documentation
```powershell
# Place docs in vectordb/docs/
New-Item -ItemType Directory -Path "vectordb\docs\excel" -Force
Copy-Item "C:\path\to\excel_guide.pdf" "vectordb\docs\excel\"
```
**Step 2**: Index documents
```powershell
python -m learner --index-docs
```
**Step 3**: Enable in config
```yaml
RAG_OFFLINE_DOCS: True
RAG_OFFLINE_DOCS_RETRIEVED_TOPK: 1
```
---
### Setup: Online Search
**Step 1**: Get Bing API key
1. Go to [Azure Portal](https://portal.azure.com)
2. Create resource → Search for "Bing Search v7"
3. Create the resource
4. Go to "Keys and Endpoint"
5. Copy Key 1
**Step 2**: Add to config
```yaml
RAG_ONLINE_SEARCH: True
BING_API_KEY: "your-copied-key-here"
RAG_ONLINE_SEARCH_TOPK: 5
RAG_ONLINE_RETRIEVED_TOPK: 5
```
**Step 3**: Test
```python
from config.config_loader import get_ufo_config
config = get_ufo_config()
print(f"Bing search enabled: {config.rag.online_search}")
print(f"API key set: {bool(config.rag.BING_API_KEY)}")
```
---
### Setup: Experience Learning
**Step 1**: Enable in config
```yaml
RAG_EXPERIENCE: True
RAG_EXPERIENCE_RETRIEVED_TOPK: 5
```
**Step 2**: Run tasks normally
```powershell
python -m ufo --request "Create a sales report"
```
**Step 3**: Successful tasks are auto-saved
```
Experience saved to: vectordb/experience/
```
**Step 4**: Future tasks retrieve experiences
```powershell
# Similar task will use past experience
python -m ufo --request "Create a quarterly report"
```
---
### Setup: Demonstration Learning
**Step 1**: Record demonstration
```powershell
python -m ufo --mode demonstration --task "format_monthly_report"
```
**Step 2**: Perform task manually
- UFO² records your every action
- Add annotations/comments
**Step 3**: Save demonstration
```
Demonstration saved to: vectordb/demonstration/
```
**Step 4**: Enable in config
```yaml
RAG_DEMONSTRATION: True
RAG_DEMONSTRATION_RETRIEVED_TOPK: 5
```
**Step 5**: Use demonstrations
```powershell
python -m ufo --request "Format the report like I showed you"
```
---
## Programmatic Access
```python
from config.config_loader import get_ufo_config
config = get_ufo_config()
# Check RAG settings
if config.rag.online_search:
print(f"Online search enabled")
print(f"Top K: {config.rag.online_search_topk}")
if config.rag.experience:
print(f"Experience learning enabled")
print(f"Experience path: {config.rag.EXPERIENCE_SAVED_PATH}")
if config.rag.offline_docs:
print(f"Offline docs enabled")
# Access specific fields
bing_key = config.rag.BING_API_KEY
exp_topk = config.rag.experience_retrieved_topk
```
---
## Performance Considerations
### Impact on Speed
| RAG Type | Speed Impact | When to Use |
|----------|--------------|-------------|
| **Offline Docs** | Low | Always (if indexed) |
| **Online Search** | Medium | For unfamiliar tasks |
| **Experience** | Low | Always (improves over time) |
| **Demonstration** | Low | For specific workflows |
### Impact on Cost
| RAG Type | Cost Impact | Notes |
|----------|-------------|-------|
| **Offline Docs** | None | One-time indexing cost |
| **Online Search** | Low | Bing API: ~$3/1000 queries |
| **Experience** | None | Free storage |
| **Demonstration** | None | Free storage |
!!!tip "Recommended Configuration"
For most users:
```yaml
RAG_ONLINE_SEARCH: True # Useful for general tasks
RAG_EXPERIENCE: True # Improves over time
RAG_OFFLINE_DOCS: False # Unless you have specific docs
RAG_DEMONSTRATION: False # Unless training specific workflows
```
---
## Troubleshooting
### Issue 1: Bing Search Not Working
!!!bug "Error Message"
```
BingSearchError: Invalid API key
```
**Solutions**:
1. Verify API key is correct
2. Check key has not expired
3. Ensure Bing Search v7 resource is active
4. Check Azure subscription is active
---
### Issue 2: Experience Not Retrieved
!!!bug "Symptom"
UFO² doesn't seem to learn from past tasks
**Solutions**:
1. Check experience database exists:
```powershell
Test-Path "vectordb\experience"
```
2. Verify tasks completed successfully
3. Check similarity threshold (may be too strict)
4. Increase `RAG_EXPERIENCE_RETRIEVED_TOPK`
---
### Issue 3: Offline Docs Not Indexed
!!!bug "Error Message"
```
No offline documents found
```
**Solutions**:
1. Run indexing:
```powershell
python -m learner --index-docs
```
2. Check documents are in `vectordb/docs/`
3. Verify supported formats (PDF, TXT, MD)
---
### Issue 4: Too Much Context
!!!bug "Symptom"
Token limits exceeded, slow responses
**Solution**: Reduce Top-K values
```yaml
RAG_ONLINE_RETRIEVED_TOPK: 3 # Instead of 5
RAG_EXPERIENCE_RETRIEVED_TOPK: 3
RAG_DEMONSTRATION_RETRIEVED_TOPK: 3
```
---
## Best Practices
### When to Enable Each Component
| Scenario | Recommended RAG |
|----------|----------------|
| **General automation** | Online Search |
| **Repetitive tasks** | Experience Learning |
| **Domain-specific workflows** | Offline Docs + Demonstration |
| **Learning over time** | Experience |
| **New to UFO²** | Online Search only |
| **Production deployment** | Experience + Offline Docs |
### Top-K Selection
| Field | Recommended Range | Notes |
|-------|-------------------|-------|
| `RAG_ONLINE_SEARCH_TOPK` | 3-10 | More = better context, slower |
| `RAG_ONLINE_RETRIEVED_TOPK` | 3-5 | Balance quality vs tokens |
| `RAG_EXPERIENCE_RETRIEVED_TOPK` | 3-5 | Most relevant experiences |
| `RAG_DEMONSTRATION_RETRIEVED_TOPK` | 1-3 | Usually need few examples |
| `RAG_OFFLINE_DOCS_RETRIEVED_TOPK` | 1-2 | Docs are usually long |
---
## Environment Variables
Store API keys securely:
```yaml
# Use environment variable instead of hardcoded key
BING_API_KEY: "${BING_API_KEY}"
```
**Set environment variable**:
**Windows PowerShell:**
```powershell
$env:BING_API_KEY = "your-key-here"
```
**Windows (Persistent):**
```powershell
[System.Environment]::SetEnvironmentVariable('BING_API_KEY', 'your-key', 'User')
```
---
## Related Documentation
- **[Agent Configuration](agents_config.md)** - LLM settings
- **[System Configuration](system_config.md)** - Runtime settings
---
## Summary
!!!success "Key Takeaways"
✅ **RAG is optional** - UFO² works without it
✅ **Online Search** - Most useful for general tasks (needs Bing API key)
✅ **Experience** - Free, improves over time automatically
✅ **Offline Docs** - Great for domain-specific knowledge
✅ **Demonstration** - Best for complex, specific workflows
✅ **Start simple** - Enable Online Search first, add others as needed
**Enhance UFO² with knowledge retrieval!** 🧠
---
### Documents Docs Configuration System System Config (`documents/docs/configuration/system/system_config.md`)
# System Configuration (system.yaml)
Configure UFO²'s runtime behavior, execution limits, control backends, logging, and operational parameters. This file controls how UFO² interacts with the Windows environment.
## Overview
The `system.yaml` file defines runtime settings that control UFO²'s behavior during task execution. Unlike `agents.yaml` (which configures LLMs), this file configures **how** UFO² operates on Windows.
**File Location**: `config/ufo/system.yaml`
**Note:** Unlike `agents.yaml`, the `system.yaml` file is **already present** in the repository with sensible defaults. You can use it as-is or customize it for your needs.
## Quick Configuration
### Default Configuration (Works Out of Box)
```yaml
# Most users can use default settings
MAX_STEP: 50
MAX_ROUND: 1
CONTROL_BACKEND: ["uia"]
USE_MCP: True
PRINT_LOG: False
```
### Recommended for Development
```yaml
# More verbose logging for debugging
MAX_STEP: 50
MAX_ROUND: 1
PRINT_LOG: True
LOG_LEVEL: "DEBUG"
CONTROL_BACKEND: ["uia"]
```
### Recommended for Production
```yaml
# Optimized for reliability
MAX_STEP: 100
MAX_ROUND: 3
CONTROL_BACKEND: ["uia"]
USE_MCP: True
SAFE_GUARD: True
LOG_TO_MARKDOWN: True
```
## Configuration Categories
The `system.yaml` file is organized into logical sections:
| Category | Purpose | Key Fields |
|----------|---------|------------|
| **[LLM Parameters](#llm-parameters)** | API call settings | `MAX_TOKENS`, `TEMPERATURE`, `TIMEOUT` |
| **[Execution Limits](#execution-limits)** | Task boundaries | `MAX_STEP`, `MAX_ROUND`, `SLEEP_TIME` |
| **[Control Backend](#control-backend)** | UI detection methods | `CONTROL_BACKEND`, `IOU_THRESHOLD` |
| **[Action Configuration](#action-configuration)** | Interaction behavior | `CLICK_API`, `INPUT_TEXT_API`, `MAXIMIZE_WINDOW` |
| **[Logging](#logging)** | Output and debugging | `PRINT_LOG`, `LOG_LEVEL`, `LOG_XML` |
| **[MCP Settings](#mcp-settings)** | Tool server integration | `USE_MCP`, `MCP_SERVERS_CONFIG` |
| **[Safety](#safety)** | Security controls | `SAFE_GUARD`, `CONTROL_LIST` |
| **[Control Filtering](#control-filtering)** | UI element filtering | `CONTROL_FILTER_TYPE`, `CONTROL_FILTER_TOP_K` |
## LLM Parameters
These settings control how UFO² communicates with LLM APIs.
### Fields
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `MAX_TOKENS` | Integer | `2000` | Maximum tokens for LLM response |
| `MAX_RETRY` | Integer | `20` | Maximum retries for failed API calls |
| `TEMPERATURE` | Float | `0.0` | Sampling temperature (0.0 = deterministic, 1.0 = creative) |
| `TOP_P` | Float | `0.0` | Nucleus sampling threshold |
| `TIMEOUT` | Integer | `60` | API call timeout (seconds) |
### Example
```yaml
# Conservative settings (recommended)
MAX_TOKENS: 2000
MAX_RETRY: 20
TEMPERATURE: 0.0 # Deterministic
TOP_P: 0.0
TIMEOUT: 60
# Creative settings (experimental)
# MAX_TOKENS: 4000
# TEMPERATURE: 0.7 # More creative
# TOP_P: 0.9
```
**When to Adjust:**
- **Increase MAX_TOKENS** if responses are getting cut off
- **Increase TEMPERATURE** if you want more varied responses (not recommended)
- **Keep at 0.0** for consistent, repeatable automation
- **Increase TIMEOUT** for slow API connections
## Execution Limits
Control how long and how many attempts UFO² makes for tasks.
### Fields
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `MAX_STEP` | Integer | `50` | Maximum steps per task |
| `MAX_ROUND` | Integer | `1` | Maximum rounds per task (retries from start) |
| `SLEEP_TIME` | Integer | `1` | Wait time between steps (seconds) |
| `RECTANGLE_TIME` | Integer | `1` | Duration to show visual highlights (seconds) |
### Example
```yaml
# Default settings
MAX_STEP: 50
MAX_ROUND: 1
SLEEP_TIME: 1
RECTANGLE_TIME: 1
# For complex tasks
# MAX_STEP: 100
# MAX_ROUND: 3
# For faster execution (risky)
# SLEEP_TIME: 0
```
**Note on Step vs Round:**
- **STEP**: Individual action (click, type, etc.)
- **ROUND**: Complete task attempt from start
Example: If `MAX_ROUND: 3`, UFO² will retry the entire task up to 3 times if it fails.
## Control Backend
Configure how UFO² detects and interacts with UI elements.
### Fields
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `CONTROL_BACKEND` | List[String] | `["uia"]` | UI detection backends to use |
| `IOU_THRESHOLD_FOR_MERGE` | Float | `0.1` | IoU threshold for merging overlapping controls |
### Available Backends
| Backend | Description | Pros | Cons |
|---------|-------------|------|------|
| `"uia"` | UI Automation | Fast, reliable, Windows native | May miss some controls |
| `"omniparser"` | Vision-based | Finds visual-only elements | Requires GPU, slow |
**Note:** `win32` backend is no longer supported.
### Example
```yaml
# Recommended: Use UIA (default)
CONTROL_BACKEND: ["uia"]
IOU_THRESHOLD_FOR_MERGE: 0.1
# With vision-based parsing (slow)
# CONTROL_BACKEND: ["uia", "omniparser"]
```
**Best Practice:** Use `["uia"]` as the default backend. Add `"omniparser"` only if you need vision-based control detection.
## Action Configuration
Configure how UFO² performs actions on UI elements.
### Core Action Settings
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `ACTION_SEQUENCE` | Boolean | `False` | Enable multi-action sequences in one step |
| `SHOW_VISUAL_OUTLINE_ON_SCREEN` | Boolean | `False` | Show visual highlights during execution |
| `MAXIMIZE_WINDOW` | Boolean | `False` | Maximize application windows before actions |
| `JSON_PARSING_RETRY` | Integer | `3` | Retries for parsing LLM JSON responses |
### Click Settings
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `CLICK_API` | String | `"click_input"` | Click method to use |
| `AFTER_CLICK_WAIT` | Integer | `0` | Wait time after clicking (seconds) |
### Input Settings
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `INPUT_TEXT_API` | String | `"type_keys"` | Text input method |
| `INPUT_TEXT_ENTER` | Boolean | `False` | Press Enter after typing |
| `INPUT_TEXT_INTER_KEY_PAUSE` | Float | `0.05` | Pause between keystrokes (seconds) |
### Example
```yaml
# Recommended settings
ACTION_SEQUENCE: True # Enable multi-action for speed
SHOW_VISUAL_OUTLINE_ON_SCREEN: False
MAXIMIZE_WINDOW: False
JSON_PARSING_RETRY: 3
CLICK_API: "click_input"
AFTER_CLICK_WAIT: 0
INPUT_TEXT_API: "type_keys"
INPUT_TEXT_ENTER: False
INPUT_TEXT_INTER_KEY_PAUSE: 0.05
# For visual debugging
# SHOW_VISUAL_OUTLINE_ON_SCREEN: True
# If clicks are too fast
# AFTER_CLICK_WAIT: 1
# For automation that needs Enter key
# INPUT_TEXT_ENTER: True
```
!!!info "Input Methods"
- **`type_keys`**: Simulates keyboard (slower, more realistic)
- **`set_text`**: Direct text insertion (faster, may not trigger events)
---
## Logging
Control UFO²'s logging output and debugging information.
### Fields
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `PRINT_LOG` | Boolean | `False` | Print logs to console |
| `LOG_LEVEL` | String | `"DEBUG"` | Logging verbosity level |
| `LOG_TO_MARKDOWN` | Boolean | `True` | Save logs as Markdown files |
| `LOG_XML` | Boolean | `False` | Log UI tree XML at each step |
| `CONCAT_SCREENSHOT` | Boolean | `False` | Concatenate control screenshots |
| `INCLUDE_LAST_SCREENSHOT` | Boolean | `True` | Include previous screenshot in context |
| `SCREENSHOT_TO_MEMORY` | Boolean | `True` | Load screenshots into memory |
| `REQUEST_TIMEOUT` | Integer | `250` | Request timeout for vision models |
### Log Levels
| Level | Usage | When to Use |
|-------|-------|-------------|
| `"DEBUG"` | Detailed debugging info | Development, troubleshooting |
| `"INFO"` | General information | Normal operation |
| `"WARNING"` | Warning messages | Production |
| `"ERROR"` | Errors only | Production (minimal logs) |
### Example
```yaml
# Development settings
PRINT_LOG: True
LOG_LEVEL: "DEBUG"
LOG_TO_MARKDOWN: True
LOG_XML: True # Useful for debugging UI detection
# Production settings
# PRINT_LOG: False
# LOG_LEVEL: "WARNING"
# LOG_TO_MARKDOWN: True
# LOG_XML: False
# Memory optimization
# SCREENSHOT_TO_MEMORY: False
```
!!!tip "Log Files Location"
Logs are saved to `logs//` directory.
---
## MCP Settings
Configure Model Context Protocol (MCP) tool servers.
### Fields
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `USE_MCP` | Boolean | `True` | Enable MCP tool integration |
| `MCP_SERVERS_CONFIG` | String | `"config/ufo/mcp.yaml"` | Path to MCP servers config |
| `MCP_PREFERRED_APPS` | List[String] | `[]` | Apps that prefer MCP over UI automation |
| `MCP_FALLBACK_TO_UI` | Boolean | `True` | Fall back to UI if MCP fails |
| `MCP_INSTRUCTIONS_PATH` | String | `"ufo/config/mcp_instructions"` | MCP instruction templates path |
| `MCP_TOOL_TIMEOUT` | Integer | `30` | MCP tool execution timeout (seconds) |
| `MCP_LOG_EXECUTION` | Boolean | `False` | Log detailed MCP execution |
### Example
```yaml
# Recommended settings
USE_MCP: True
MCP_SERVERS_CONFIG: "config/ufo/mcp.yaml"
MCP_FALLBACK_TO_UI: True
MCP_TOOL_TIMEOUT: 30
MCP_LOG_EXECUTION: False
# Prefer MCP for VS Code and Terminal
MCP_PREFERRED_APPS:
- "Code.exe"
- "WindowsTerminal.exe"
# Debugging MCP issues
# MCP_LOG_EXECUTION: True
# MCP_TOOL_TIMEOUT: 60
```
!!!info "What is MCP?"
MCP (Model Context Protocol) provides programmatic APIs for applications, offering more reliable automation than UI-based control.
See [MCP Configuration](mcp_reference.md) for details.
---
## Safety
Security and safety controls to prevent dangerous operations.
### Fields
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `SAFE_GUARD` | Boolean | `False` | Enable safety checks |
| `CONTROL_LIST` | List[String] | See below | Allowed UI control types |
### Default CONTROL_LIST
```yaml
CONTROL_LIST:
- "Button"
- "Edit"
- "TabItem"
- "Document"
- "ListItem"
- "MenuItem"
- "ScrollBar"
- "TreeItem"
- "Hyperlink"
- "ComboBox"
- "RadioButton"
- "Spinner"
- "CheckBox"
- "Group"
- "Text"
```
### Example
```yaml
# Enable safety for production
SAFE_GUARD: True
CONTROL_LIST:
- "Button"
- "Edit"
- "TabItem"
# Add only safe control types
# Disable for full automation (risky)
# SAFE_GUARD: False
```
!!!danger "Safety Warning"
When `SAFE_GUARD: True`, UFO² will only interact with control types in `CONTROL_LIST`. This prevents accidental dangerous operations but may limit functionality.
---
## Control Filtering
Advanced UI element filtering using semantic and icon similarity.
### Fields
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `CONTROL_FILTER_TYPE` | List[String] | `[]` | Filter types to enable |
| `CONTROL_FILTER_TOP_K_PLAN` | Integer | `2` | Top K plans to consider |
| `CONTROL_FILTER_TOP_K_SEMANTIC` | Integer | `15` | Top K controls by text similarity |
| `CONTROL_FILTER_TOP_K_ICON` | Integer | `15` | Top K controls by icon similarity |
| `CONTROL_FILTER_MODEL_SEMANTIC_NAME` | String | `"all-MiniLM-L6-v2"` | Semantic embedding model |
| `CONTROL_FILTER_MODEL_ICON_NAME` | String | `"clip-ViT-B-32"` | Icon embedding model |
### Filter Types
| Type | Description | Use Case |
|------|-------------|----------|
| `"TEXT"` | Text-based filtering | Filter by control labels |
| `"SEMANTIC"` | Semantic similarity | Find similar controls by meaning |
| `"ICON"` | Icon similarity | Find controls by icon appearance |
### Example
```yaml
# Disable filtering (use all controls)
CONTROL_FILTER_TYPE: []
# Enable semantic filtering (recommended)
CONTROL_FILTER_TYPE: ["SEMANTIC"]
CONTROL_FILTER_TOP_K_SEMANTIC: 15
CONTROL_FILTER_MODEL_SEMANTIC_NAME: "all-MiniLM-L6-v2"
# Enable all filtering (most selective)
# CONTROL_FILTER_TYPE: ["TEXT", "SEMANTIC", "ICON"]
# CONTROL_FILTER_TOP_K_SEMANTIC: 20
# CONTROL_FILTER_TOP_K_ICON: 20
```
!!!warning "Performance Impact"
- Filtering reduces the number of controls sent to LLM (faster, cheaper)
- But may filter out the target control (less reliable)
- Start without filtering, add if you have too many controls
---
## API Usage Configuration
Configure native API usage for Office applications.
### Fields
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `USE_APIS` | Boolean | `True` | Enable COM API usage for Office applications |
| `API_PROMPT` | String | `"ufo/prompts/share/base/api.yaml"` | API prompt template |
| `APP_API_PROMPT_ADDRESS` | Dict | See below | App-specific API prompts |
### Default APP_API_PROMPT_ADDRESS
```yaml
APP_API_PROMPT_ADDRESS:
"WINWORD.EXE": "ufo/prompts/apps/word/api.yaml"
"EXCEL.EXE": "ufo/prompts/apps/excel/api.yaml"
"msedge.exe": "ufo/prompts/apps/web/api.yaml"
"chrome.exe": "ufo/prompts/apps/web/api.yaml"
"POWERPNT.EXE": "ufo/prompts/apps/powerpoint/api.yaml"
```
### Example
```yaml
# Enable API usage (recommended for Office)
USE_APIS: True
API_PROMPT: "ufo/prompts/share/base/api.yaml"
APP_API_PROMPT_ADDRESS:
"WINWORD.EXE": "ufo/prompts/apps/word/api.yaml"
"EXCEL.EXE": "ufo/prompts/apps/excel/api.yaml"
# Disable for pure UI automation
# USE_APIS: False
```
!!!tip "When to Use APIs"
COM APIs are faster and more reliable for Office applications. Keep `USE_APIS: True` for best results with Word, Excel, PowerPoint.
---
## Complete Example Configuration
Here's a complete, production-ready `system.yaml`:
```
/* Detailed source-code truncated for AI context efficiency. */
```
---
## Programmatic Access
```python
from config.config_loader import get_ufo_config
config = get_ufo_config()
# Access system settings
max_step = config.system.max_step
log_level = config.system.log_level
control_backends = config.system.control_backend
# Check MCP settings
if config.system.use_mcp:
mcp_config_path = config.system.mcp_servers_config
print(f"MCP enabled, config: {mcp_config_path}")
# Modify at runtime (not recommended)
# config.system.max_step = 200
```
---
## Troubleshooting
### Issue 1: Tasks Failing After X Steps
!!!bug "Error Message"
```
Task stopped: Maximum steps (50) reached
```
**Solution**: Increase `MAX_STEP`
```yaml
MAX_STEP: 100 # or higher
```
### Issue 2: Controls Not Detected
**Symptom:** UFO² can't find UI elements
**Solutions:**
1. Try enabling omniparser for vision-based detection:
```yaml
CONTROL_BACKEND: ["uia", "omniparser"]
```
2. Disable filtering:
```yaml
CONTROL_FILTER_TYPE: []
```
### Issue 3: Actions Too Fast
**Symptom:** Actions execute before UI is ready
**Solution:** Add delays
```yaml
SLEEP_TIME: 2
AFTER_CLICK_WAIT: 1
```
### Issue 4: Logs Too Verbose
**Symptom:** Too much console output
**Solution:** Reduce logging
```yaml
PRINT_LOG: False
LOG_LEVEL: "WARNING"
```
---
## Performance Tuning
### For Speed
```yaml
MAX_STEP: 50
SLEEP_TIME: 0
CONTROL_BACKEND: ["uia"]
CONTROL_FILTER_TYPE: ["SEMANTIC"] # Reduce LLM input
ACTION_SEQUENCE: True # Multi-action in one step
```
### For Reliability
```yaml
MAX_STEP: 100
MAX_ROUND: 3
SLEEP_TIME: 2
AFTER_CLICK_WAIT: 1
CONTROL_BACKEND: ["uia"]
CONTROL_FILTER_TYPE: [] # Don't filter out controls
```
### For Debugging
```yaml
PRINT_LOG: True
LOG_LEVEL: "DEBUG"
LOG_XML: True
SHOW_VISUAL_OUTLINE_ON_SCREEN: True
MCP_LOG_EXECUTION: True
```
---
## Related Documentation
- **[Agent Configuration](agents_config.md)** - LLM and API settings
- **[MCP Configuration](mcp_reference.md)** - Tool server configuration
- **[RAG Configuration](rag_config.md)** - Knowledge retrieval
## Summary
**Key Takeaways:**
✅ **Default settings work** - Start with defaults, adjust as needed
✅ **Increase MAX_STEP** for complex tasks
✅ **Use ["uia"]** for control detection
✅ **Enable ACTION_SEQUENCE** for faster execution
✅ **Adjust logging** based on dev vs production
✅ **Enable MCP** for better Office automation
**Fine-tune system settings for optimal performance!** ⚙️
---
### Documents Docs Configuration System Third Party Config (`documents/docs/configuration/system/third_party_config.md`)
# Third-Party Agent Configuration (third_party.yaml)
Configure third-party agents that extend UFO²'s capabilities beyond Windows GUI automation, such as LinuxAgent for CLI operations and HardwareAgent for physical device control.
---
## Overview
The `third_party.yaml` file configures external agents that integrate with UFO² to provide specialized capabilities. These agents work alongside the standard HostAgent and AppAgent to handle tasks that require non-GUI interactions.
**File Location**: `config/ufo/third_party.yaml`
**Advanced Feature:** Third-party agent configuration is an **advanced optional feature**. Most users only need the core agents (HostAgent, AppAgent). Configure third-party agents only when you need specialized capabilities.
!!!tip "Creating Custom Third-Party Agents"
Want to build your own third-party agent? See the **[Creating Custom Third-Party Agents Tutorial](../../tutorials/creating_third_party_agents.md)** for a complete step-by-step guide using HardwareAgent as an example.
---
## Quick Start
### Default Configuration
```yaml
# Enable third-party agents
ENABLED_THIRD_PARTY_AGENTS: ["LinuxAgent"]
THIRD_PARTY_AGENT_CONFIG:
LinuxAgent:
AGENT_NAME: "LinuxAgent"
APPAGENT_PROMPT: "ufo/prompts/third_party/linux_agent.yaml"
APPAGENT_EXAMPLE_PROMPT: "ufo/prompts/third_party/linux_agent_example.yaml"
INTRODUCTION: "For Linux Use Only."
```
### Disable All Third-Party Agents
```yaml
# Disable all third-party agents
ENABLED_THIRD_PARTY_AGENTS: []
```
---
## Available Third-Party Agents
### LinuxAgent
**Purpose**: Execute Linux CLI commands and server operations.
!!!info "UFO³ Integration"
LinuxAgent is used by **UFO³ Galaxy** to orchestrate Linux devices as sub-agents in multi-device workflows. When Galaxy routes a task to a Linux device, it uses LinuxAgent to execute commands via CLI.
**Configuration**:
```yaml
THIRD_PARTY_AGENT_CONFIG:
LinuxAgent:
AGENT_NAME: "LinuxAgent"
APPAGENT_PROMPT: "ufo/prompts/third_party/linux_agent.yaml"
APPAGENT_EXAMPLE_PROMPT: "ufo/prompts/third_party/linux_agent_example.yaml"
INTRODUCTION: "For Linux Use Only."
```
**Fields**:
| Field | Type | Description |
|-------|------|-------------|
| `AGENT_NAME` | String | Agent identifier (must be "LinuxAgent") |
| `APPAGENT_PROMPT` | String | Path to main prompt template |
| `APPAGENT_EXAMPLE_PROMPT` | String | Path to example prompt template |
| `INTRODUCTION` | String | Agent description for LLM context |
**When to Enable**:
- ✅ Using UFO³ Galaxy with Linux devices
- ✅ Need to execute Linux CLI commands
- ✅ Managing Linux servers from Windows
- ✅ Cross-platform automation workflows
**Related Documentation**:
- [Linux Agent as Galaxy Device](../../linux/as_galaxy_device.md)
- [Linux Agent Quick Start](../../getting_started/quick_start_linux.md)
---
### HardwareAgent
**Purpose**: Control physical hardware components (robotic arms, USB devices, etc.).
!!!warning "Experimental Feature"
HardwareAgent is an experimental feature for controlling physical hardware. Requires specialized hardware setup and is not commonly used.
**Configuration**:
```yaml
THIRD_PARTY_AGENT_CONFIG:
HardwareAgent:
VISUAL_MODE: True
AGENT_NAME: "HardwareAgent"
APPAGENT_PROMPT: "ufo/prompts/share/base/app_agent.yaml"
APPAGENT_EXAMPLE_PROMPT: "ufo/prompts/examples/visual/app_agent_example.yaml"
API_PROMPT: "ufo/prompts/third_party/hardware_agent_api.yaml"
INTRODUCTION: "The HardwareAgent is used to manipulate hardware components of the computer without using GUI, such as robotic arms for keyboard input and mouse control, plug and unplug devices such as USB drives, and other hardware-related tasks."
```
**Fields**:
| Field | Type | Description |
|-------|------|-------------|
| `VISUAL_MODE` | Boolean | Enable visual mode (screenshot understanding) |
| `AGENT_NAME` | String | Agent identifier (must be "HardwareAgent") |
| `APPAGENT_PROMPT` | String | Path to main prompt template |
| `APPAGENT_EXAMPLE_PROMPT` | String | Path to example prompt template |
| `API_PROMPT` | String | Path to hardware API prompt template |
| `INTRODUCTION` | String | Agent description for LLM context |
**When to Enable**:
- ✅ Using robotic arms for physical input
- ✅ Automated USB device management
- ✅ Physical hardware testing/automation
- ✅ Research projects with hardware control
**Related Documentation**:
- [Creating Custom Third-Party Agents](../../tutorials/creating_third_party_agents.md) - Tutorial using HardwareAgent as example
---
## Configuration Fields
### ENABLED_THIRD_PARTY_AGENTS
**Type**: `List[String]`
**Default**: `[]` (no third-party agents enabled)
List of third-party agent names to enable. Only agents listed here will be loaded and available.
**Options**:
- `"LinuxAgent"` - Linux CLI execution
- `"HardwareAgent"` - Physical hardware control
**Examples**:
```yaml
# Enable LinuxAgent only (recommended for UFO³)
ENABLED_THIRD_PARTY_AGENTS: ["LinuxAgent"]
# Enable both agents
ENABLED_THIRD_PARTY_AGENTS: ["LinuxAgent", "HardwareAgent"]
# Disable all third-party agents
ENABLED_THIRD_PARTY_AGENTS: []
```
### THIRD_PARTY_AGENT_CONFIG
**Type**: `Dict[String, Dict]`
Configuration dictionary for each third-party agent. Each agent has its own configuration block.
**Structure**:
```yaml
THIRD_PARTY_AGENT_CONFIG:
AgentName:
AGENT_NAME: "AgentName"
# Agent-specific fields...
```
---
## Complete Configuration Example
### For UFO³ Galaxy (Recommended)
```yaml
# Enable LinuxAgent for UFO³ Galaxy multi-device orchestration
ENABLED_THIRD_PARTY_AGENTS: ["LinuxAgent"]
THIRD_PARTY_AGENT_CONFIG:
LinuxAgent:
AGENT_NAME: "LinuxAgent"
APPAGENT_PROMPT: "ufo/prompts/third_party/linux_agent.yaml"
APPAGENT_EXAMPLE_PROMPT: "ufo/prompts/third_party/linux_agent_example.yaml"
INTRODUCTION: "For Linux Use Only."
```
### With Hardware Support
```yaml
# Enable both Linux and Hardware agents
ENABLED_THIRD_PARTY_AGENTS: ["LinuxAgent", "HardwareAgent"]
THIRD_PARTY_AGENT_CONFIG:
LinuxAgent:
AGENT_NAME: "LinuxAgent"
APPAGENT_PROMPT: "ufo/prompts/third_party/linux_agent.yaml"
APPAGENT_EXAMPLE_PROMPT: "ufo/prompts/third_party/linux_agent_example.yaml"
INTRODUCTION: "For Linux Use Only."
HardwareAgent:
VISUAL_MODE: True
AGENT_NAME: "HardwareAgent"
APPAGENT_PROMPT: "ufo/prompts/share/base/app_agent.yaml"
APPAGENT_EXAMPLE_PROMPT: "ufo/prompts/examples/visual/app_agent_example.yaml"
API_PROMPT: "ufo/prompts/third_party/hardware_agent_api.yaml"
INTRODUCTION: "The HardwareAgent is used to manipulate hardware components of the computer without using GUI, such as robotic arms for keyboard input and mouse control, plug and unplug devices such as USB drives, and other hardware-related tasks."
```
### Minimal (No Third-Party Agents)
```yaml
# Disable all third-party agents (default for standalone UFO²)
ENABLED_THIRD_PARTY_AGENTS: []
```
---
## UFO³ Galaxy Integration
When using UFO³ Galaxy for multi-device orchestration, LinuxAgent must be enabled to support Linux devices.
### Setup for Galaxy
**Step 1**: Enable LinuxAgent in `config/ufo/third_party.yaml`
```yaml
ENABLED_THIRD_PARTY_AGENTS: ["LinuxAgent"]
THIRD_PARTY_AGENT_CONFIG:
LinuxAgent:
AGENT_NAME: "LinuxAgent"
APPAGENT_PROMPT: "ufo/prompts/third_party/linux_agent.yaml"
APPAGENT_EXAMPLE_PROMPT: "ufo/prompts/third_party/linux_agent_example.yaml"
INTRODUCTION: "For Linux Use Only."
```
**Step 2**: Configure Linux devices in `config/galaxy/devices.yaml`
```yaml
devices:
- device_id: "linux_server_1"
server_url: "ws://192.168.1.100:5001/ws"
os: "linux"
capabilities:
- "server"
- "cli"
- "database"
```
**Step 3**: Start Linux Agent components
See [Linux Agent as Galaxy Device](../../linux/as_galaxy_device.md) for complete setup.
---
## Programmatic Access
```python
from config.config_loader import get_ufo_config
config = get_ufo_config()
# Check which third-party agents are enabled
enabled_agents = config.ENABLED_THIRD_PARTY_AGENTS
print(f"Enabled third-party agents: {enabled_agents}")
# Access agent configuration
if "LinuxAgent" in enabled_agents:
linux_config = config.THIRD_PARTY_AGENT_CONFIG["LinuxAgent"]
print(f"LinuxAgent prompt: {linux_config['APPAGENT_PROMPT']}")
# Check if specific agent is enabled
linux_enabled = "LinuxAgent" in config.ENABLED_THIRD_PARTY_AGENTS
print(f"LinuxAgent enabled: {linux_enabled}")
```
---
## Adding Custom Third-Party Agents
You can add your own third-party agents by following the patterns described below. For a complete tutorial, see **[Creating Custom Third-Party Agents](../../tutorials/creating_third_party_agents.md)**.
### Quick Overview
### Step 1: Create Agent Implementation
```python
# ufo/agents/third_party/my_agent.py
class MyCustomAgent:
def __init__(self, config):
self.config = config
# Initialize your agent
```
### Step 2: Add Configuration
```yaml
ENABLED_THIRD_PARTY_AGENTS: ["MyCustomAgent"]
THIRD_PARTY_AGENT_CONFIG:
MyCustomAgent:
AGENT_NAME: "MyCustomAgent"
APPAGENT_PROMPT: "ufo/prompts/third_party/my_agent.yaml"
APPAGENT_EXAMPLE_PROMPT: "ufo/prompts/third_party/my_agent_example.yaml"
INTRODUCTION: "Custom agent description."
# Your custom fields
CUSTOM_FIELD: "value"
```
### Step 3: Register Agent
Add your agent to the third-party agent registry in UFO²'s agent loader.
---
## Troubleshooting
### Issue 1: LinuxAgent Not Working
!!!bug "Error Message"
```
LinuxAgent not found or not enabled
```
**Solution**: Check configuration
```yaml
# Verify LinuxAgent is in enabled list
ENABLED_THIRD_PARTY_AGENTS: ["LinuxAgent"]
```
### Issue 2: Prompt Files Not Found
!!!bug "Error Message"
```
FileNotFoundError: ufo/prompts/third_party/linux_agent.yaml
```
**Solution**: Verify prompt files exist
```powershell
# Check if prompt files exist
Test-Path "ufo\prompts\third_party\linux_agent.yaml"
Test-Path "ufo\prompts\third_party\linux_agent_example.yaml"
```
### Issue 3: Agent Configuration Not Loaded
!!!bug "Symptom"
Third-party agent configuration changes not taking effect
**Solution**: Restart UFO² application
```powershell
# Configuration is loaded at startup
# Restart UFO² to apply changes
```
---
## Best Practices
!!!tip "Recommendations"
- ✅ **Enable only what you need** - Don't enable agents you're not using
- ✅ **For UFO³ Galaxy** - Always enable LinuxAgent when using Linux devices
- ✅ **Keep prompts up to date** - Ensure prompt files exist and are current
- ✅ **Document custom agents** - Add clear introduction text for LLM context
- ✅ **Test configurations** - Verify agents load correctly after configuration changes
!!!danger "Warnings"
- ❌ **Don't enable HardwareAgent** without proper hardware setup
- ❌ **Don't modify AGENT_NAME** - Must match the agent class name
- ❌ **Don't delete prompt files** - Agents will fail to initialize
---
## Related Documentation
- **[Creating Custom Third-Party Agents](../../tutorials/creating_third_party_agents.md)** - Complete tutorial for building third-party agents
- **[Linux Agent as Galaxy Device](../../linux/as_galaxy_device.md)** - Using LinuxAgent in UFO³
- **[Linux Agent Quick Start](../../getting_started/quick_start_linux.md)** - Setting up Linux Agent
- **[Agent Configuration](./agents_config.md)** - Core agent LLM settings
- **[Galaxy Devices Configuration](./galaxy_devices.md)** - Multi-device setup
---
## Summary
!!!success "Key Takeaways"
✅ **third_party.yaml is optional** - Only needed for specialized agents
✅ **LinuxAgent for UFO³** - Required when using Linux devices in Galaxy
✅ **HardwareAgent is experimental** - For physical hardware control
✅ **Enable selectively** - Only enable agents you actually use
✅ **Configuration is simple** - Just add agent names to enabled list
**Extend UFO² with specialized capabilities!** 🔧
---
### Documents Docs Galaxy Agent Registration Agent Profile (`documents/docs/galaxy/agent_registration/agent_profile.md`)
# 📊 AgentProfile - Comprehensive Agent Representation
The **AgentProfile** is a multi-source data structure that consolidates administrator configuration, service-level capabilities, and real-time client telemetry into a unified, dynamically updated representation of each constellation agent.
---
## 📋 Overview
The **AgentProfile** is the primary data structure representing a registered constellation agent. It aggregates information from **three distinct sources** to provide a comprehensive view of each agent's identity, capabilities, operational status, and hardware characteristics.
For a complete understanding of how agents work in the constellation system, see:
- [Constellation Overview](../constellation/overview.md) - Architecture and multi-device coordination
- [Constellation Agent](../constellation_agent/overview.md) - Agent behavior and lifecycle
| Function | Description |
|----------|-------------|
| **Identity Management** | Unique identification and endpoint tracking |
| **Capability Advertisement** | Declare supported features and tools |
| **Status Monitoring** | Real-time operational state tracking |
| **Resource Profiling** | Hardware and system information |
| **Task Assignment** | Enable intelligent task routing decisions |
---
## 🏗️ Structure Definition
### Core Dataclass
```
/* Detailed source-code truncated for AI context efficiency. */
```
---
## 🔍 Field Reference
### Identity Fields
| Field | Type | Source | Description | Example |
|-------|------|--------|-------------|---------|
| `device_id` | `str` | User Config | Unique identifier for the device | `"windowsagent"`, `"linux_gpu_01"` |
| `server_url` | `str` | User Config | WebSocket endpoint of device agent server | `"ws://localhost:5005/ws"` |
The `device_id` must be unique across the entire constellation. Attempting to register a duplicate `device_id` will fail.
### Platform & Capabilities
| Field | Type | Source | Description | Example |
|-------|------|--------|-------------|---------|
| `os` | `Optional[str]` | User Config + Telemetry | Operating system type | `"windows"`, `"linux"`, `"darwin"` |
| `capabilities` | `List[str]` | User Config + Telemetry | Advertised capabilities/features | `["gui", "browser", "office"]` |
| `metadata` | `Dict[str, Any]` | All Sources | Multi-source metadata aggregation | See [Metadata Structure](#metadata-structure) |
**Capabilities Merging:**
```python
# Initial capabilities from user config
capabilities = ["web_browsing", "office_applications"]
# After telemetry collection, auto-detected features are merged
# Result: ["web_browsing", "office_applications", "gui", "cli", "browser", "file_system"]
```
### Operational Status
| Field | Type | Source | Description | Example |
|-------|------|--------|-------------|---------|
| `status` | `DeviceStatus` | Runtime | Current connection/operational state | `DeviceStatus.IDLE` |
| `last_heartbeat` | `Optional[datetime]` | Runtime | Timestamp of last heartbeat | `2025-11-06T10:30:45Z` |
**Status Values:**
```python
DeviceStatus.DISCONNECTED # Not connected
DeviceStatus.CONNECTING # Connection in progress
DeviceStatus.CONNECTED # WebSocket established
DeviceStatus.REGISTERING # Performing AIP registration
DeviceStatus.IDLE # Ready for tasks
DeviceStatus.BUSY # Executing a task
DeviceStatus.FAILED # Connection or execution failed
```
### Connection Management
| Field | Type | Source | Description | Example |
|-------|------|--------|-------------|---------|
| `connection_attempts` | `int` | Runtime | Number of connection attempts made | `0`, `3` |
| `max_retries` | `int` | User Config | Maximum reconnection attempts before giving up | `5`, `10` |
When a device disconnects, the system automatically retries connection up to `max_retries` times with exponential backoff.
### Task Execution
| Field | Type | Source | Description | Example |
|-------|------|--------|-------------|---------|
| `current_task_id` | `Optional[str]` | Runtime | ID of task currently being executed | `"task_12345"`, `None` |
**Usage in Task Queue:**
```python
# When task is assigned
profile.status = DeviceStatus.BUSY
profile.current_task_id = "task_12345"
# When task completes
profile.status = DeviceStatus.IDLE
profile.current_task_id = None
```
---
## 🗂️ Metadata Structure
The `metadata` dictionary is a flexible container that aggregates information from all three profiling sources:
### Metadata Schema
```python
metadata = {
# ===== Source 1: User Configuration =====
"location": str, # Physical location
"performance": str, # Performance tier
"description": str, # Human-readable description
"operation_engineer_email": str, # Contact information
"tags": List[str], # Custom tags
# ... any custom user-defined fields
# ===== Source 2: Service Manifest =====
"platform": str, # Platform type (from registration)
"registration_time": str, # ISO timestamp of registration
# ===== Source 3: Client Telemetry =====
"system_info": {
"platform": str, # OS platform (windows, linux, darwin)
"os_version": str, # OS version string
"cpu_count": int, # Number of CPU cores
"memory_total_gb": float, # Total RAM in GB
"hostname": str, # Device hostname
"ip_address": str, # Device IP address
"platform_type": str, # Device category (computer, mobile, etc.)
"schema_version": str # Telemetry schema version
},
"custom_metadata": { # Optional custom metadata from config
"datacenter": str,
"tier": str,
# ... server-configured metadata
}
}
```
### Example Metadata
```python
# Complete metadata example from a Windows GPU workstation
metadata = {
# User Configuration
"location": "office_desktop",
"performance": "very_high",
"description": "Primary Windows workstation with GPU",
"operation_engineer_email": "admin@example.com",
"tags": ["production", "gpu-enabled", "high-priority"],
# Service Manifest
"platform": "windows",
"registration_time": "2025-11-06T10:30:00.000Z",
# Client Telemetry
"system_info": {
"platform": "windows",
"os_version": "10.0.22631",
"cpu_count": 16,
"memory_total_gb": 32.0,
"hostname": "DESKTOP-GPU01",
"ip_address": "192.168.1.100",
"platform_type": "computer",
"schema_version": "1.0"
},
"custom_metadata": {
"datacenter": "us-west-2",
"tier": "premium",
"gpu_type": "NVIDIA RTX 4090",
"gpu_count": 1
}
}
```
---
## 🔄 Multi-Source Construction
### Three-Source Architecture
```mermaid
graph LR
A[User Config devices.yaml]
B[AIP Registration Service Manifest]
C[Device Telemetry DeviceInfoProvider]
A -->|device_id, server_url capabilities, metadata| D[AgentProfile]
B -->|platform, registration_time| D
C -->|system_info, features| D
style A fill:#e1f5ff
style B fill:#fff4e1
style C fill:#e8f5e9
style D fill:#f3e5f5
```
### Construction Timeline
```mermaid
sequenceDiagram
participant Config as devices.yaml
participant Manager as DeviceManager
participant Server as UFO Server
participant Telemetry as DeviceInfoProvider
Note over Config,Telemetry: Phase 1: Initial Registration
Config->>Manager: Load device config
Manager->>Manager: Create AgentProfile (device_id, server_url, capabilities)
Note over Config,Telemetry: Phase 2: Service Registration
Manager->>Server: WebSocket REGISTER
Server-->>Manager: Add platform, registration_time
Note over Config,Telemetry: Phase 3: Telemetry Collection
Manager->>Server: request_device_info()
Server->>Telemetry: collect_system_info()
Telemetry-->>Server: system_info
Server-->>Manager: system_info
Manager->>Manager: Update AgentProfile (merge system_info & features)
```
### Merging Strategy
**1. User Configuration (Priority: Baseline)**
```python
# Initial AgentProfile creation
profile = AgentProfile(
device_id="windowsagent",
server_url="ws://localhost:5005/ws",
os="windows",
capabilities=["web_browsing", "office_applications"],
metadata={
"location": "office_desktop",
"performance": "high"
}
)
```
**2. Service Manifest (Priority: Override `os`, Add registration data)**
```python
# During AIP registration
profile.metadata.update({
"platform": "windows", # From registration message
"registration_time": "2025-11-06T10:30:00Z"
})
```
**3. Client Telemetry (Priority: Merge capabilities, Add system_info)**
```python
# After DeviceInfoProvider collects data
system_info = {
"platform": "windows",
"os_version": "10.0.22631",
"cpu_count": 16,
"memory_total_gb": 32.0,
"hostname": "DESKTOP-GPU01",
"ip_address": "192.168.1.100",
"supported_features": ["gui", "cli", "browser", "file_system", "office", "windows_apps"],
"platform_type": "computer"
}
# Update OS if not already set
if not profile.os:
profile.os = system_info["platform"]
# Merge capabilities (avoid duplicates)
existing_caps = set(profile.capabilities)
new_caps = set(system_info["supported_features"])
profile.capabilities = list(existing_caps.union(new_caps))
# Result: ["web_browsing", "office_applications", "gui", "cli", "browser", "file_system", "windows_apps"]
# Add system_info to metadata
profile.metadata["system_info"] = system_info
```
---
## 📊 Example Profiles
### Example 1: Windows GPU Workstation
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Profile Summary
```mermaid
graph TB
subgraph "AgentProfile: gpu_workstation_01"
A["Status: IDLE Last Heartbeat: 10:45:30"]
B["System ━━━━━ OS: Windows 10.0.22631 CPU: 16 cores Memory: 64.0 GB Host: DESKTOP-GPU01 IP: 192.168.1.100"]
C["Capabilities ━━━━━ • web_browsing • office_applications • gpu_computation • model_training • gui, cli, browser • file_system"]
D["Metadata ━━━━━ Location: office_desktop Performance: very_high Tags: production, gpu, ml GPU: 2× NVIDIA RTX 4090"]
end
style A fill:#e3f2fd
style B fill:#f3e5f5
style C fill:#e8f5e9
style D fill:#fff3e0
```
### Example 2: Linux Server
```python
AgentProfile(
# Identity
device_id="linux_server_01",
server_url="ws://10.0.0.50:5001/ws",
# Platform & Capabilities
os="linux",
capabilities=[
# User-configured
"server_management",
"log_monitoring",
"database_operations",
# Auto-detected
"cli",
"file_system",
"linux_apps"
],
# Metadata
metadata={
# User Configuration
"location": "datacenter_rack_a42",
"performance": "medium",
"description": "Production Linux server for backend services",
"logs_file_path": "/var/log/application.log",
"dev_path": "/home/deploy/",
# Service Manifest
"platform": "linux",
"registration_time": "2025-11-06T09:15:00Z",
# Client Telemetry
"system_info": {
"platform": "linux",
"os_version": "#1 SMP PREEMPT_DYNAMIC Wed Nov 1 15:36:23 UTC 2023",
"cpu_count": 8,
"memory_total_gb": 16.0,
"hostname": "prod-server-01",
"ip_address": "10.0.0.50",
"platform_type": "computer",
"schema_version": "1.0"
}
},
# Status
status=DeviceStatus.BUSY,
last_heartbeat=datetime(2025, 11, 6, 10, 44, 15),
# Connection
connection_attempts=0,
max_retries=3,
# Task
current_task_id="task_monitoring_567"
)
```
---
## 🔄 Lifecycle Operations
### Creation
```python
from galaxy.client.components import DeviceRegistry, AgentProfile, DeviceStatus
registry = DeviceRegistry()
# Create AgentProfile during registration
profile = registry.register_device(
device_id="windowsagent",
server_url="ws://localhost:5005/ws",
os="windows",
capabilities=["web_browsing", "office"],
metadata={"location": "office"},
max_retries=5
)
print(f"Created: {profile.device_id}")
print(f"Status: {profile.status.value}") # "disconnected"
```
### Status Updates
```python
# Update connection status
registry.update_device_status("windowsagent", DeviceStatus.CONNECTING)
registry.update_device_status("windowsagent", DeviceStatus.CONNECTED)
registry.update_device_status("windowsagent", DeviceStatus.IDLE)
# Set device busy with task
registry.set_device_busy("windowsagent", task_id="task_123")
profile = registry.get_device("windowsagent")
print(f"Status: {profile.status.value}") # "busy"
print(f"Current Task: {profile.current_task_id}") # "task_123"
# Set device idle (task complete)
registry.set_device_idle("windowsagent")
profile = registry.get_device("windowsagent")
print(f"Status: {profile.status.value}") # "idle"
print(f"Current Task: {profile.current_task_id}") # None
```
### System Info Updates
```python
# Update with telemetry data
system_info = {
"platform": "windows",
"os_version": "10.0.22631",
"cpu_count": 16,
"memory_total_gb": 32.0,
"hostname": "DESKTOP-DEV01",
"ip_address": "192.168.1.100",
"supported_features": ["gui", "cli", "browser", "file_system", "office"],
"platform_type": "computer",
"schema_version": "1.0"
}
registry.update_device_system_info("windowsagent", system_info)
# Verify update
profile = registry.get_device("windowsagent")
print(f"OS: {profile.os}") # "windows"
print(f"CPU Cores: {profile.metadata['system_info']['cpu_count']}") # 16
print(f"Memory: {profile.metadata['system_info']['memory_total_gb']} GB") # 32.0
print(f"Capabilities: {profile.capabilities}")
# ["web_browsing", "office", "gui", "cli", "browser", "file_system"]
```
### Heartbeat Tracking
```python
from datetime import datetime, timezone
# Update heartbeat
registry.update_heartbeat("windowsagent")
profile = registry.get_device("windowsagent")
print(f"Last Heartbeat: {profile.last_heartbeat}")
# 2025-11-06 10:45:30.123456+00:00
```
### Connection Retry Management
```python
# Increment connection attempts
attempts = registry.increment_connection_attempts("windowsagent")
print(f"Attempts: {attempts}/{profile.max_retries}")
# Reset after successful connection
registry.reset_connection_attempts("windowsagent")
profile = registry.get_device("windowsagent")
print(f"Attempts: {profile.connection_attempts}") # 0
```
---
## 🎯 Usage Patterns
The following patterns demonstrate how AgentProfile is used for intelligent task routing and device management. For more details on task constellation concepts, see [Constellation Overview](../constellation/overview.md).
### Task Assignment Decision
```python
def can_assign_task(profile: AgentProfile, required_capabilities: List[str]) -> bool:
"""
Check if device can handle a task based on its profile.
"""
# Check if device is available
if profile.status != DeviceStatus.IDLE:
return False
# Check if all required capabilities are supported
device_caps = set(profile.capabilities)
required_caps = set(required_capabilities)
if not required_caps.issubset(device_caps):
return False
# Optional: Check system resources
system_info = profile.metadata.get("system_info", {})
if system_info.get("memory_total_gb", 0) < 8: # Require at least 8GB
return False
return True
# Usage
profile = registry.get_device("windowsagent")
if can_assign_task(profile, ["browser", "gui"]):
await manager.assign_task_to_device(
task_id="task_web_001",
device_id="windowsagent",
task_description="Navigate to website and extract data",
task_data={"url": "https://example.com"}
)
```
### Device Selection
```python
def select_best_device(
all_devices: Dict[str, AgentProfile],
required_capabilities: List[str],
prefer_high_performance: bool = True
) -> Optional[str]:
"""
Select the best available device for a task.
"""
candidates = []
for device_id, profile in all_devices.items():
# Must be idle
if profile.status != DeviceStatus.IDLE:
continue
# Must have required capabilities
device_caps = set(profile.capabilities)
if not set(required_capabilities).issubset(device_caps):
continue
# Calculate score
score = 0
if profile.metadata.get("performance") == "very_high":
score += 10
elif profile.metadata.get("performance") == "high":
score += 5
# Prefer devices with more memory
system_info = profile.metadata.get("system_info", {})
score += system_info.get("memory_total_gb", 0) / 10
candidates.append((device_id, score))
if not candidates:
return None
# Sort by score (descending)
candidates.sort(key=lambda x: x[1], reverse=True)
return candidates[0][0]
# Usage
all_devices = registry.get_all_devices(connected=True)
best_device = select_best_device(
all_devices,
required_capabilities=["gpu_computation", "model_training"],
prefer_high_performance=True
)
print(f"Selected device: {best_device}")
```
### Health Monitoring
```python
from datetime import datetime, timezone, timedelta
def check_device_health(profile: AgentProfile) -> Dict[str, Any]:
"""
Check device health based on profile data.
"""
health = {
"device_id": profile.device_id,
"healthy": True,
"warnings": [],
"errors": []
}
# Check heartbeat freshness
if profile.last_heartbeat:
age = datetime.now(timezone.utc) - profile.last_heartbeat
if age > timedelta(minutes=5):
health["warnings"].append(
f"No heartbeat for {age.total_seconds():.0f} seconds"
)
if age > timedelta(minutes=10):
health["errors"].append("Heartbeat timeout")
health["healthy"] = False
# Check connection attempts
if profile.connection_attempts > profile.max_retries / 2:
health["warnings"].append(
f"High connection attempts: {profile.connection_attempts}/{profile.max_retries}"
)
# Check if device is stuck in BUSY state
if profile.status == DeviceStatus.BUSY and profile.current_task_id:
# Would need to check task age here
health["warnings"].append(f"Device busy with task {profile.current_task_id}")
return health
# Usage
profile = registry.get_device("windowsagent")
health = check_device_health(profile)
print(f"Health: {health['healthy']}")
print(f"Warnings: {health['warnings']}")
print(f"Errors: {health['errors']}")
```
---
## 🔗 Related Documentation
| Topic | Document | Description |
|-------|----------|-------------|
| **Overview** | [Agent Registration Overview](./overview.md) | Registration architecture and process |
| **Registration Flow** | [Registration Flow](./registration_flow.md) | Step-by-step registration process |
| **Device Registry** | [Device Registry](./device_registry.md) | Registry component implementation |
| **Galaxy Devices Config** | [Galaxy Devices Configuration](../../configuration/system/galaxy_devices.md) | YAML configuration reference |
| **Device Info** | [Device Info Provider](../../client/device_info.md) | Telemetry collection details |
| **AIP Protocol** | [AIP Overview](../../aip/overview.md) | Agent Interaction Protocol |
| **Constellation System** | [Constellation Overview](../constellation/overview.md) | Multi-device coordination |
| **WebSocket Client** | [Client AIP Integration](../client/aip_integration.md) | Client-side implementation |
---
## 💡 Best Practices
### 1. Meaningful Capabilities
```python
# ✅ Good: Specific, actionable capabilities
capabilities = ["web_browsing", "office_excel", "file_management", "email_sending"]
# ❌ Bad: Vague capabilities
capabilities = ["desktop", "general"]
```
### 2. Rich Metadata
```python
# ✅ Good: Comprehensive metadata for smart routing
metadata = {
"location": "datacenter_us_west",
"performance": "high",
"description": "GPU workstation for ML training",
"tags": ["production", "gpu", "ml"],
"operation_engineer_email": "ml-team@example.com"
}
```
### 3. Monitor Heartbeats
```python
# Regularly check heartbeat freshness
if profile.last_heartbeat:
age = datetime.now(timezone.utc) - profile.last_heartbeat
if age > timedelta(minutes=5):
logger.warning(f"Device {profile.device_id} heartbeat stale")
```
### 4. Use System Info for Resource-Aware Routing
```python
# Check if device has enough resources
system_info = profile.metadata.get("system_info", {})
if system_info.get("memory_total_gb", 0) >= 16:
# Assign memory-intensive task
pass
```
---
## 🚀 Next Steps
1. **Learn Registration Process**: Read [Registration Flow](./registration_flow.md)
2. **Configure Devices**: See [Galaxy Devices Configuration](../../configuration/system/galaxy_devices.md)
3. **Understand DeviceRegistry**: Check [Device Registry](./device_registry.md)
4. **Study Telemetry**: Read [Device Info Provider](../../client/device_info.md)
---
## 📚 Source Code References
- **AgentProfile Definition**: `galaxy/client/components/types.py`
- **DeviceRegistry**: `galaxy/client/components/device_registry.py`
- **ConstellationDeviceManager**: `galaxy/client/device_manager.py`
- **DeviceInfoProvider**: `ufo/client/device_info_provider.py`
---
### Documents Docs Galaxy Agent Registration Device Registry (`documents/docs/galaxy/agent_registration/device_registry.md`)
# 🗄️ DeviceRegistry - Device Data Management
## 📋 Overview
The **DeviceRegistry** is a focused component that manages device registration and information storage, providing a clean separation of concerns in the constellation architecture. It is responsible for **device data management only** - storing, retrieving, and updating AgentProfile instances without handling networking, task execution, or protocol logic.
> For details on how devices connect and register using the AIP protocol, see [Registration Flow](./registration_flow.md).
**Core Responsibilities:**
| Responsibility | Description |
|----------------|-------------|
| **Registration** | Create and store AgentProfile instances |
| **Status Tracking** | Update device connection and operational states |
| **Metadata Management** | Store and update device metadata from all sources |
| **Information Retrieval** | Provide device information to other components |
| **Task State Tracking** | Track which device is executing which task |
**Delegation to Other Components:**
- Network communication → [`WebSocketConnectionManager`](../client/components.md#websocketconnectionmanager-network-communication-handler)
- Message processing → [`MessageProcessor`](../client/components.md#messageprocessor-message-router-and-handler)
- Task execution → [`TaskQueueManager`](../client/components.md#taskqueuemanager-task-scheduling-and-queuing)
- Heartbeat monitoring → [`HeartbeatManager`](../client/components.md#heartbeatmanager-connection-health-monitor)
## 🏗️ Architecture
### Class Structure
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Internal Storage
```python
class DeviceRegistry:
def __init__(self):
# Primary storage: device_id -> AgentProfile
self._devices: Dict[str, AgentProfile] = {}
# Secondary storage: device_id -> capabilities dict
# (Legacy, mostly superseded by AgentProfile.capabilities)
self._device_capabilities: Dict[str, Dict[str, Any]] = {}
self.logger = logging.getLogger(f"{__name__}.DeviceRegistry")
```
**Storage Structure:**
```python
# Internal state example
_devices = {
"windowsagent": AgentProfile(
device_id="windowsagent",
server_url="ws://localhost:5005/ws",
os="windows",
capabilities=["gui", "browser", "office"],
metadata={...},
status=DeviceStatus.IDLE,
...
),
"linux_server_01": AgentProfile(
device_id="linux_server_01",
server_url="ws://10.0.0.50:5001/ws",
os="linux",
capabilities=["cli", "server"],
metadata={...},
status=DeviceStatus.BUSY,
current_task_id="task_123"
)
}
```
---
## 🔧 Core Operations
### 1. Device Registration
#### Method: `register_device()`
```python
def register_device(
self,
device_id: str,
server_url: str,
os: Optional[str] = None,
capabilities: Optional[List[str]] = None,
metadata: Optional[Dict[str, Any]] = None,
max_retries: int = 5,
) -> AgentProfile:
"""
Register a new device.
:param device_id: Unique device identifier
:param server_url: UFO WebSocket server URL
:param os: Operating system type
:param capabilities: Device capabilities
:param metadata: Additional metadata
:param max_retries: Maximum connection retry attempts
:return: Created AgentProfile object
"""
```
**Process:**
```mermaid
sequenceDiagram
participant Caller
participant Registry as DeviceRegistry
participant Profile as AgentProfile
Caller->>Registry: register_device(device_id, server_url, ...)
Registry->>Profile: Create AgentProfile
Note over Profile: device_id, server_url os, capabilities metadata, max_retries status=DISCONNECTED
Registry->>Registry: Store in _devices[device_id]
Registry->>Registry: Log registration
Registry-->>Caller: Return AgentProfile
```
**Example:**
```python
registry = DeviceRegistry()
# Register device
profile = registry.register_device(
device_id="windowsagent",
server_url="ws://localhost:5005/ws",
os="windows",
capabilities=["web_browsing", "office_applications"],
metadata={
"location": "office_desktop",
"performance": "high"
},
max_retries=5
)
print(f"Registered: {profile.device_id}")
print(f"Status: {profile.status.value}") # "disconnected"
```
> **Note:** The `register_device()` method will overwrite an existing device if the same `device_id` is used. Consider adding validation if duplicate prevention is needed.
### 2. Device Retrieval
#### Method: `get_device()`
```python
def get_device(self, device_id: str) -> Optional[AgentProfile]:
"""Get device information by ID"""
return self._devices.get(device_id)
```
**Example:**
```python
profile = registry.get_device("windowsagent")
if profile:
print(f"Device: {profile.device_id}")
print(f"Status: {profile.status.value}")
print(f"Capabilities: {profile.capabilities}")
else:
print("Device not found")
```
#### Method: `get_all_devices()`
```python
def get_all_devices(self, connected: bool = False) -> Dict[str, AgentProfile]:
"""
Get all registered devices
:param connected: If True, return only connected devices
:return: Dictionary of device_id to AgentProfile
"""
```
**Example:**
```python
# Get all devices
all_devices = registry.get_all_devices(connected=False)
print(f"Total devices: {len(all_devices)}")
# Get only connected devices
connected_devices = registry.get_all_devices(connected=True)
print(f"Connected devices: {len(connected_devices)}")
for device_id, profile in connected_devices.items():
print(f" - {device_id}: {profile.status.value}")
```
**Connected Device Filter:**
```python
# Implementation detail
if connected:
return {
device_id: device_info
for device_id, device_info in self._devices.items()
if device_info.status in [
DeviceStatus.CONNECTED,
DeviceStatus.IDLE,
DeviceStatus.BUSY
]
}
```
#### Method: `get_connected_devices()`
```python
def get_connected_devices(self) -> List[str]:
"""Get list of connected device IDs"""
return [
device_id
for device_id, device_info in self._devices.items()
if device_info.status == DeviceStatus.CONNECTED
]
```
**Example:**
```python
connected = registry.get_connected_devices()
print(f"Connected: {connected}")
# ['windowsagent', 'linux_server_01']
```
---
### 3. Status Management
#### Method: `update_device_status()`
```python
def update_device_status(self, device_id: str, status: DeviceStatus) -> None:
"""Update device connection status"""
if device_id in self._devices:
self._devices[device_id].status = status
```
**Example:**
```python
# Update status progression
registry.update_device_status("windowsagent", DeviceStatus.CONNECTING)
registry.update_device_status("windowsagent", DeviceStatus.CONNECTED)
registry.update_device_status("windowsagent", DeviceStatus.IDLE)
```
**Status Lifecycle:**
```mermaid
stateDiagram-v2
[*] --> DISCONNECTED: register_device()
DISCONNECTED --> CONNECTING: update_device_status()
CONNECTING --> CONNECTED: update_device_status()
CONNECTING --> FAILED: update_device_status()
CONNECTED --> REGISTERING: update_device_status()
REGISTERING --> IDLE: update_device_status()
REGISTERING --> FAILED: update_device_status()
IDLE --> BUSY: set_device_busy()
BUSY --> IDLE: set_device_idle()
IDLE --> DISCONNECTED: update_device_status()
BUSY --> DISCONNECTED: update_device_status()
FAILED --> CONNECTING: update_device_status()
DISCONNECTED --> [*]: remove_device()
```
---
### 4. Task State Management
#### Method: `set_device_busy()`
```python
def set_device_busy(self, device_id: str, task_id: str) -> None:
"""
Set device to BUSY status and track current task.
:param device_id: Device ID
:param task_id: Task ID being executed
"""
if device_id in self._devices:
self._devices[device_id].status = DeviceStatus.BUSY
self._devices[device_id].current_task_id = task_id
self.logger.info(f"🔄 Device {device_id} set to BUSY (task: {task_id})")
```
**Example:**
```python
# Assign task to device
registry.set_device_busy("windowsagent", task_id="task_12345")
profile = registry.get_device("windowsagent")
print(f"Status: {profile.status.value}") # "busy"
print(f"Current Task: {profile.current_task_id}") # "task_12345"
```
#### Method: `set_device_idle()`
```python
def set_device_idle(self, device_id: str) -> None:
"""
Set device to IDLE status and clear current task.
:param device_id: Device ID
"""
if device_id in self._devices:
self._devices[device_id].status = DeviceStatus.IDLE
self._devices[device_id].current_task_id = None
self.logger.info(f"✅ Device {device_id} set to IDLE")
```
**Example:**
```python
# Task completes
registry.set_device_idle("windowsagent")
profile = registry.get_device("windowsagent")
print(f"Status: {profile.status.value}") # "idle"
print(f"Current Task: {profile.current_task_id}") # None
```
#### Method: `is_device_busy()`
```python
def is_device_busy(self, device_id: str) -> bool:
"""
Check if device is currently busy.
:param device_id: Device ID
:return: True if device is busy
"""
if device_id in self._devices:
return self._devices[device_id].status == DeviceStatus.BUSY
return False
```
**Example:**
```python
if registry.is_device_busy("windowsagent"):
print("Device is busy, task will be queued")
else:
print("Device is available")
```
#### Method: `get_current_task()`
```python
def get_current_task(self, device_id: str) -> Optional[str]:
"""
Get the current task ID being executed on device.
:param device_id: Device ID
:return: Current task ID or None
"""
if device_id in self._devices:
return self._devices[device_id].current_task_id
return None
```
**Example:**
```python
task_id = registry.get_current_task("windowsagent")
if task_id:
print(f"Device executing: {task_id}")
else:
print("Device idle")
```
---
### 5. Connection Management
#### Method: `increment_connection_attempts()`
```python
def increment_connection_attempts(self, device_id: str) -> int:
"""Increment connection attempts counter"""
if device_id in self._devices:
self._devices[device_id].connection_attempts += 1
return self._devices[device_id].connection_attempts
return 0
```
**Example:**
```python
attempts = registry.increment_connection_attempts("windowsagent")
print(f"Attempts: {attempts}")
profile = registry.get_device("windowsagent")
if profile.connection_attempts >= profile.max_retries:
print("Max retries reached, giving up")
```
#### Method: `reset_connection_attempts()`
```python
def reset_connection_attempts(self, device_id: str) -> None:
"""Reset connection attempts counter to 0"""
if device_id in self._devices:
self._devices[device_id].connection_attempts = 0
self.logger.info(f"🔄 Reset connection attempts for device {device_id}")
```
**Example:**
```python
# After successful connection
registry.reset_connection_attempts("windowsagent")
profile = registry.get_device("windowsagent")
print(f"Attempts: {profile.connection_attempts}") # 0
```
---
### 6. Heartbeat Tracking
#### Method: `update_heartbeat()`
```python
def update_heartbeat(self, device_id: str) -> None:
"""Update last heartbeat timestamp"""
if device_id in self._devices:
self._devices[device_id].last_heartbeat = datetime.now(timezone.utc)
```
**Example:**
```python
from datetime import datetime, timezone, timedelta
# Update heartbeat
registry.update_heartbeat("windowsagent")
profile = registry.get_device("windowsagent")
print(f"Last heartbeat: {profile.last_heartbeat}")
# Check heartbeat freshness
age = datetime.now(timezone.utc) - profile.last_heartbeat
if age > timedelta(minutes=5):
print("⚠️ Heartbeat stale!")
```
---
### 7. System Information Management
#### Method: `update_device_system_info()`
```python
def update_device_system_info(
self, device_id: str, system_info: Dict[str, Any]
) -> bool:
"""
Update AgentProfile with system information retrieved from server.
This method updates the device's OS, capabilities, and metadata with
the system information that was automatically collected by the device
and stored on the server.
:param device_id: Device ID
:param system_info: System information dictionary from server
:return: True if update successful, False if device not found
"""
```
> **Note:** System information is collected from the device agent and retrieved via the server. See [Client Connection Manager](../../server/client_connection_manager.md) for server-side information management.
**Process:**
```mermaid
sequenceDiagram
participant Caller
participant Registry as DeviceRegistry
participant Profile as AgentProfile
Caller->>Registry: update_device_system_info(device_id, system_info)
Registry->>Profile: Get device
alt Device exists
Registry->>Profile: Update os = system_info["platform"]
Registry->>Profile: Merge supported_features into capabilities
Registry->>Profile: Add system_info to metadata
Registry->>Profile: Add custom_metadata if present
Registry->>Registry: Log update
Registry-->>Caller: True
else Device not found
Registry->>Registry: Log warning
Registry-->>Caller: False
end
```
**Implementation:**
```python
device_info = self.get_device(device_id)
if not device_info:
self.logger.warning(f"Cannot update system info: device {device_id} not found")
return False
# 1. Update OS information
if "platform" in system_info:
device_info.os = system_info["platform"]
# 2. Merge capabilities with supported features (avoid duplicates)
if "supported_features" in system_info:
features = system_info["supported_features"]
existing_caps = set(device_info.capabilities)
new_caps = existing_caps.union(set(features))
device_info.capabilities = list(new_caps)
# 3. Update metadata with system information
device_info.metadata.update({
"system_info": {
"platform": system_info.get("platform"),
"os_version": system_info.get("os_version"),
"cpu_count": system_info.get("cpu_count"),
"memory_total_gb": system_info.get("memory_total_gb"),
"hostname": system_info.get("hostname"),
"ip_address": system_info.get("ip_address"),
"platform_type": system_info.get("platform_type"),
"schema_version": system_info.get("schema_version"),
}
})
# 4. Add custom metadata if present
if "custom_metadata" in system_info:
device_info.metadata["custom_metadata"] = system_info["custom_metadata"]
# 5. Add tags if present
if "tags" in system_info:
device_info.metadata["tags"] = system_info["tags"]
return True
```
**Example:**
```python
system_info = {
"platform": "windows",
"os_version": "10.0.22631",
"cpu_count": 16,
"memory_total_gb": 32.0,
"hostname": "DESKTOP-DEV01",
"ip_address": "192.168.1.100",
"supported_features": ["gui", "cli", "browser", "file_system", "office"],
"platform_type": "computer",
"schema_version": "1.0"
}
success = registry.update_device_system_info("windowsagent", system_info)
if success:
profile = registry.get_device("windowsagent")
print(f"OS: {profile.os}") # "windows"
print(f"CPU: {profile.metadata['system_info']['cpu_count']}") # 16
print(f"Memory: {profile.metadata['system_info']['memory_total_gb']} GB") # 32.0
```
#### Method: `get_device_system_info()`
```python
def get_device_system_info(self, device_id: str) -> Optional[Dict[str, Any]]:
"""
Get device system information (hardware, OS, features).
:param device_id: Device ID
:return: System information dictionary or None if not available
"""
device_info = self.get_device(device_id)
if not device_info:
return None
return device_info.metadata.get("system_info")
```
**Example:**
```python
system_info = registry.get_device_system_info("windowsagent")
if system_info:
print(f"Platform: {system_info['platform']}")
print(f"CPU Cores: {system_info['cpu_count']}")
print(f"Memory: {system_info['memory_total_gb']} GB")
print(f"Hostname: {system_info['hostname']}")
else:
print("System info not available")
```
---
### 8. Capabilities Management
#### Method: `set_device_capabilities()`
```python
def set_device_capabilities(
self, device_id: str, capabilities: Dict[str, Any]
) -> None:
"""Store device capabilities information"""
self._device_capabilities[device_id] = capabilities
# Also update device info with capabilities
if device_id in self._devices:
device_info = self._devices[device_id]
if "capabilities" in capabilities:
device_info.capabilities.extend(capabilities["capabilities"])
if "metadata" in capabilities:
device_info.metadata.update(capabilities["metadata"])
```
> **Note:** This method is primarily for backwards compatibility. Modern code should use `update_device_system_info()` instead.
#### Method: `get_device_capabilities()`
```python
def get_device_capabilities(self, device_id: str) -> Dict[str, Any]:
"""Get device capabilities"""
return self._device_capabilities.get(device_id, {})
```
---
### 9. Utility Methods
#### Method: `is_device_registered()`
```python
def is_device_registered(self, device_id: str) -> bool:
"""Check if device is registered"""
return device_id in self._devices
```
**Example:**
```python
if registry.is_device_registered("windowsagent"):
print("Device exists")
else:
print("Device not registered")
```
#### Method: `remove_device()`
```python
def remove_device(self, device_id: str) -> bool:
"""Remove a device from registry"""
if device_id in self._devices:
del self._devices[device_id]
self._device_capabilities.pop(device_id, None)
return True
return False
```
**Example:**
```python
success = registry.remove_device("windowsagent")
if success:
print("Device removed")
else:
print("Device not found")
```
---
## 💡 Usage Patterns
### Pattern 1: Complete Registration Flow
```python
from galaxy.client.components import DeviceRegistry, DeviceStatus
registry = DeviceRegistry()
# 1. Register device
profile = registry.register_device(
device_id="windowsagent",
server_url="ws://localhost:5005/ws",
os="windows",
capabilities=["web_browsing"],
metadata={"location": "office"},
max_retries=5
)
# 2. Update status through connection process
registry.update_device_status("windowsagent", DeviceStatus.CONNECTING)
registry.increment_connection_attempts("windowsagent")
registry.update_device_status("windowsagent", DeviceStatus.CONNECTED)
registry.reset_connection_attempts("windowsagent")
# 3. Update with system info
system_info = {
"platform": "windows",
"cpu_count": 16,
"memory_total_gb": 32.0,
"supported_features": ["gui", "cli", "browser"]
}
registry.update_device_system_info("windowsagent", system_info)
# 4. Set to IDLE (ready for tasks)
registry.set_device_idle("windowsagent")
# 5. Update heartbeat
registry.update_heartbeat("windowsagent")
```
### Pattern 2: Task Assignment
```python
# Check if device can accept task
if not registry.is_device_busy("windowsagent"):
# Assign task
registry.set_device_busy("windowsagent", task_id="task_123")
# ... execute task ...
# Task complete
registry.set_device_idle("windowsagent")
else:
print("Device busy, task queued")
```
### Pattern 3: Device Selection
```python
def find_available_device_with_capability(
registry: DeviceRegistry,
required_capability: str
) -> Optional[str]:
"""Find an idle device with specific capability."""
all_devices = registry.get_all_devices(connected=True)
for device_id, profile in all_devices.items():
# Check if idle
if profile.status != DeviceStatus.IDLE:
continue
# Check capability
if required_capability in profile.capabilities:
return device_id
return None
# Usage
device_id = find_available_device_with_capability(registry, "browser")
if device_id:
print(f"Selected: {device_id}")
```
### Pattern 4: Health Monitoring
```python
from datetime import datetime, timezone, timedelta
def check_all_devices_health(registry: DeviceRegistry):
"""Check health of all registered devices."""
all_devices = registry.get_all_devices()
for device_id, profile in all_devices.items():
print(f"\n{device_id}:")
print(f" Status: {profile.status.value}")
# Check heartbeat
if profile.last_heartbeat:
age = datetime.now(timezone.utc) - profile.last_heartbeat
print(f" Heartbeat age: {age.total_seconds():.0f}s")
if age > timedelta(minutes=5):
print(f" ⚠️ WARNING: Stale heartbeat!")
else:
print(f" ⚠️ WARNING: No heartbeat recorded")
# Check connection attempts
if profile.connection_attempts > 0:
print(f" Connection attempts: {profile.connection_attempts}/{profile.max_retries}")
# Check task status
if profile.current_task_id:
print(f" Current task: {profile.current_task_id}")
```
---
## 🔗 Integration with Other Components
DeviceRegistry is used internally by other components in the constellation system. See [Components Overview](../client/components.md) for details on the component architecture.
### With ConstellationDeviceManager
```python
# ConstellationDeviceManager uses DeviceRegistry internally
class ConstellationDeviceManager:
def __init__(self, ...):
self.device_registry = DeviceRegistry() # Internal registry
async def register_device(self, ...):
# Delegate to registry
self.device_registry.register_device(...)
def get_device_info(self, device_id: str):
# Delegate to registry
return self.device_registry.get_device(device_id)
```
### With MessageProcessor
```python
# MessageProcessor updates registry when messages arrive
class MessageProcessor:
def __init__(self, device_registry: DeviceRegistry, ...):
self.device_registry = device_registry
async def handle_heartbeat(self, device_id: str):
# Update heartbeat in registry
self.device_registry.update_heartbeat(device_id)
```
### With TaskQueueManager
```python
# TaskQueueManager checks device status via registry
class TaskQueueManager:
def can_assign_task(self, device_id: str) -> bool:
# Check if device is busy
return not self.device_registry.is_device_busy(device_id)
```
---
## 🔗 Related Documentation
| Topic | Document | Description |
|-------|----------|-------------|
| **Overview** | [Agent Registration Overview](./overview.md) | Registration architecture |
| **AgentProfile** | [AgentProfile](./agent_profile.md) | Profile structure details |
| **Registration Flow** | [Registration Flow](./registration_flow.md) | Step-by-step registration |
| **Galaxy Devices Config** | [Galaxy Devices Configuration](../../configuration/system/galaxy_devices.md) | YAML config reference |
| **Components** | [Client Components](../client/components.md) | Component architecture |
---
## 💡 Best Practices
**1. Always Check Device Exists**
```python
profile = registry.get_device(device_id)
if not profile:
logger.error(f"Device {device_id} not found")
return
```
**2. Use Defensive Copies for Lists/Dicts**
```python
# Registry already creates copies, but be aware
capabilities = ["web", "office"]
registry.register_device(..., capabilities=capabilities)
# Modifying original list won't affect registry
capabilities.append("new") # Safe
```
**3. Monitor Heartbeats Regularly**
```python
# Periodic check
for device_id in registry.get_all_devices():
profile = registry.get_device(device_id)
if profile.last_heartbeat:
age = datetime.now(timezone.utc) - profile.last_heartbeat
if age > timedelta(minutes=5):
logger.warning(f"Stale heartbeat: {device_id}")
```
**4. Clear Task State After Completion**
```python
# Always set to IDLE after task completes
registry.set_device_idle(device_id)
# This automatically clears current_task_id
```
---
## 🚀 Next Steps
1. **Understand AgentProfile**: Read [AgentProfile Documentation](./agent_profile.md)
2. **Learn Configuration**: See [Galaxy Devices Configuration](../../configuration/system/galaxy_devices.md)
3. **Study Registration**: Check [Registration Flow](./registration_flow.md)
4. **Explore Components**: See ConstellationDeviceManager implementation
---
## 📚 Source Code Reference
- **DeviceRegistry**: `galaxy/client/components/device_registry.py`
- **AgentProfile**: `galaxy/client/components/types.py`
- **ConstellationDeviceManager**: `galaxy/client/device_manager.py`
---
--- METRICS ---
- Files Extracted: 101
- Estimated Token Budget: ~388579 tokens
- Canonical Reference: https://codewiki.google/github.com/microsoft/UFO
: Weaving the Digital Agent Galaxy
**UFO² Desktop AgentOS**
**STABLE & BATTLE-TESTED**
**Perfect for:**
- 💻 Single Windows automation
- ⚡ Quick task execution
- 🎓 Learning agent basics
- 🛠️ Simple workflows
**Key Features:**
- Deep Windows OS integration
- Hybrid GUI + API actions
- Proven reliability
- Easy setup
- Can serve as Galaxy device agent
**📖 [UFO² Documentation →](./ufo/README.md)**
: Weaving the Digital Agent Galaxy
