# Technical Documentation: sentient-agi/ROMA > ℹ️ **Provenance:** Hybrid Fusion: `sentient-agi/ROMA` (README + 10 In-Tree Chapters) Β· [CodeWiki Reference](https://codewiki.google/github.com/sentient-agi/ROMA) Β· Recency: Legacy (> 180 days) > [!WARNING] > Documentation Recency Notice: This repository has not had active commits or releases within the last 180 days. Some APIs or patterns may be superseded. ## 1. Project Overview & Quickstart (sentient-agi/ROMA) ## πŸ“‘ Table of Contents - [🧠 Conceptual Overview](#-conceptual-overview) - [πŸ“¦ Installation & Setup](#-installation--setup) - [⚑ Quickstart: End-to-End Workflow](#-quickstart-end-to-end-workflow) - [βš™οΈ Configuration & Storage](#-configuration--storage) - [🧰 Toolkits](#-toolkits) - [🌐 REST API & CLI](#-rest-api--cli) - [πŸ—οΈ Core Building Block: `BaseModule`](#-core-building-block-basemodule) - [πŸ“š Module Reference](#-module-reference) - [βš›οΈ Atomizer](#-atomizer) - [πŸ“‹ Planner](#-planner) - [βš™οΈ Executor](#-executor) - [πŸ”€ Aggregator](#-aggregator) - [βœ… Verifier](#-verifier) - [🎯 Advanced Patterns](#-advanced-patterns) - [πŸ§ͺ Testing](#-testing) - [πŸ’‘ Troubleshooting & Tips](#-troubleshooting--tips) - [πŸ“– Glossary](#-glossary) --- ## 🎯 What is ROMA? **ROMA** is a **meta-agent framework** that uses recursive hierarchical structures to solve complex problems. By breaking down tasks into parallelizable components, ROMA enables agents to tackle sophisticated reasoning challenges while maintaining transparency that makes context-engineering and iteration straightforward. The framework offers **parallel problem solving** where agents work simultaneously on different parts of complex tasks, **transparent development** with a clear structure for easy debugging, and **proven performance** demonstrated through our search agent's strong benchmark results. We've shown the framework's effectiveness, but this is just the beginning. As an **open-source and extensible** platform, ROMA is designed for community-driven development, allowing you to build and customize agents for your specific needs while benefiting from the collective improvements of the community. ## πŸ—οΈ How It Works **ROMA** framework processes tasks through a recursive **plan–execute loop**: ```python def solve(task): if is_atomic(task): # Step 1: Atomizer return execute(task) # Step 2: Executor else: subtasks = plan(task) # Step 2: Planner results = [] for subtask in subtasks: results.append(solve(subtask)) # Recursive call return aggregate(results) # Step 3: Aggregator # Entry point: answer = solve(initial_request) ``` 1. **Atomizer** – Decides whether a request is **atomic** (directly executable) or requires **planning**. 2. **Planner** – If planning is needed, the task is broken into smaller **subtasks**. Each subtask is fed back into the **Atomizer**, making the process recursive. 3. **Executors** – Handle atomic tasks. Executors can be **LLMs, APIs, or even other agents** β€” as long as they implement an `agent.execute()` interface. 4. **Aggregator** – Collects and integrates results from subtasks. Importantly, the Aggregator produces the **answer to the original parent task**, not just raw child outputs. #### πŸ“ Information Flow - **Top-down:** Tasks are decomposed into subtasks recursively. - **Bottom-up:** Subtask results are aggregated upwards into solutions for parent tasks. - **Left-to-right:** If a subtask depends on the output of a previous one, it waits until that subtask completes before execution. This structure makes the system flexible, recursive, and dependency-aware β€” capable of decomposing complex problems into smaller steps while ensuring results are integrated coherently. Click to view the system flow diagram ```mermaid flowchart TB A[Your Request] --> B{Atomizer} B -->|Plan Needed| C[Planner] B -->|Atomic Task| D[Executor] %% Planner spawns subtasks C --> E[Subtasks] E --> G[Aggregator] %% Recursion E -.-> B %% Execution + Aggregation D --> F[Final Result] G --> F style A fill:#e1f5fe style F fill:#c8e6c9 style B fill:#fff3e0 style C fill:#ffe0b2 style D fill:#d1c4e9 style G fill:#c5cae9 ``` ## πŸš€ Quick Start ### Fastest Way: Minimal Installation (Recommended for Evaluation) Get started in **under 30 seconds** with no infrastructure required: ```bash # Install with uv (10-100x faster) uv pip install roma-dspy # Or with pip pip install roma-dspy # Set your OpenRouter API key (default uses Claude Sonnet 4.5 + Gemini 2.5 Flash) export OPENROUTER_API_KEY="sk-or-v1-..." # Start solving tasks immediately python -c "from roma_dspy.core.engine.solve import solve; print(solve('What is 2+2?'))" ``` > **Note**: The default configuration uses OpenRouter with Claude Sonnet 4.5 (executor) and Gemini 2.5 Flash (other agents). You can also use OpenAI directly by setting `OPENAI_API_KEY` and customizing the config. **What you get:** - βœ… Core agent framework (Atomizer, Planner, Executor, Aggregator, Verifier) - βœ… All DSPy prediction strategies (CoT, ReAct, CodeAct, etc.) - βœ… File storage (no database required) - βœ… Built-in toolkits (Calculator, File operations) - βœ… Works with any LLM provider (OpenRouter, OpenAI, Anthropic, etc.) **No Docker, no database, no setup - just install and go!** ### Production Setup: Full Features with Docker For production use with persistence, observability, and API server: ```bash # One-command setup (builds Docker, starts services) just setup # Or with specific profile just setup crypto_agent # Verify services running curl http://localhost:8000/health # Solve tasks via API just solve "What is the capital of France?" ``` **Additional features with Docker:** - πŸ“Š PostgreSQL persistence (execution history, checkpoints) - πŸ“ˆ MLflow observability (experiment tracking, visualization) - 🌐 REST API server (FastAPI with interactive docs) - πŸ“¦ S3-compatible storage (MinIO) - πŸ”§ E2B code execution sandboxes - 🎨 Interactive TUI visualization **Services Available:** - πŸš€ **REST API**: http://localhost:8000/docs - πŸ—„οΈ **PostgreSQL**: Automatic persistence - πŸ“¦ **MinIO**: S3-compatible storage (http://localhost:9001) - πŸ“Š **MLflow**: http://localhost:5000 (with `docker-up-full`) See [Quick Start Guide](docs/QUICKSTART.md) and [Deployment Guide](docs/DEPLOYMENT.md) for details. --- ## 🧠 Conceptual Overview ROMA's module layer wraps canonical DSPy patterns into purpose-built components that reflect the lifecycle of complex task execution: 1. **Atomizer** decides whether a request can be handled directly or needs decomposition. 2. **Planner** breaks non-atomic goals into an ordered graph of subtasks. 3. **Executor** resolves individual subtasks, optionally routing through function/tool calls. 4. **Aggregator** synthesizes subtask outputs back into a coherent answer. 5. **Verifier** (optional) inspects the aggregate output against the original goal before delivering. Every module shares the same ergonomics: instantiate it with a language model (LM) or provider string, choose a prediction strategy, then call `.forward()` (or `.aforward()` for async) with the task-specific fields. All modules ultimately delegate to DSPy signatures defined in `roma_dspy.core.signatures`. This keeps interfaces stable even as the internals evolve. ## πŸ“¦ Installation & Setup ### Option 1: Minimal Installation (Fastest - Recommended for Evaluation) **Perfect for:** Evaluating ROMA, development, testing, quick prototyping **Install in under 30 seconds:** ```bash # With uv (recommended - 10-100x faster) uv pip install roma-dspy # Or with pip pip install roma-dspy ``` **Set your API key:** ```bash export OPENROUTER_API_KEY="sk-or-v1-..." # Recommended # OR export OPENAI_API_KEY="sk-..." export ANTHROPIC_API_KEY="sk-ant-..." ``` **Start using immediately:** ```python from roma_dspy.core.engine.solve import solve # Solve any task result = solve("What is the capital of France?") print(result) ``` **What's included:** - βœ… All core modules (Atomizer, Planner, Executor, Aggregator, Verifier) - βœ… All DSPy prediction strategies - βœ… File-based storage (no database needed) - βœ… Core toolkits (Calculator, File operations) - βœ… Works with any LLM provider **What's NOT included (install separately if needed):** - PostgreSQL persistence β†’ `uv pip install roma-dspy[persistence]` - MLflow observability β†’ `uv pip install roma-dspy[observability]` - E2B code execution β†’ `uv pip install roma-dspy[e2b]` - REST API server β†’ `uv pip install roma-dspy[api]` - S3 storage β†’ `uv pip install roma-dspy[s3]` - All features β†’ `uv pip install roma-dspy[all]` --- ### Option 2: Full Installation with Docker (Production) **Perfect for:** Production deployment, teams, full observability **Prerequisites:** - Docker & Docker Compose - Python 3.12+ (for local development) - [Just](https://github.com/casey/just) command runner (optional, recommended) **One-command setup:** ```bash # Interactive setup (prompts for E2B, S3, etc.) just setup # Or with specific profile just setup crypto_agent ``` **Manual Docker start:** ```bash just docker-up # Basic (PostgreSQL + MinIO + API) just docker-up-full # With MLflow observability ``` **Environment variables** (auto-configured by `just setup`): ```bash # LLM Provider (required) OPENROUTER_API_KEY=... # Recommended # OR OPENAI_API_KEY=... ANTHROPIC_API_KEY=... # Optional: Advanced features E2B_API_KEY=... # Code execution COINGECKO_API_KEY=... # Crypto toolkit ``` **Additional Docker features:** - πŸ“Š PostgreSQL (execution history, checkpoints) - πŸ“ˆ MLflow (experiment tracking, metrics) - 🌐 REST API (FastAPI with docs) - πŸ“¦ MinIO (S3-compatible storage) - 🎨 TUI visualization --- ### Option 3: Development Installation **For contributing or extending ROMA:** ```bash # Clone repository git clone https://github.com/sentient-agi/roma.git cd roma # Install with dev tools (includes pytest, ruff, mypy) uv pip install -e ".[dev]" # Run tests just test # Format code just format # Type check just typecheck ``` --- ### Comparison: Which Installation is Right for You? | Feature | Minimal (`pip install roma-dspy`) | Docker (`just setup`) | |---------|----------------------------------|----------------------| | Installation time | **< 30 seconds** | ~5-10 minutes | | Infrastructure required | **None** | Docker | | Core agent framework | βœ… | βœ… | | File storage | βœ… | βœ… | | PostgreSQL persistence | ❌ (install `[persistence]`) | βœ… | | MLflow observability | ❌ (install `[observability]`) | βœ… | | REST API | ❌ (install `[api]`) | βœ… | | Best for | Evaluation, development | Production, teams | ## ⚑ Quickstart: End-to-End Workflow The following example mirrors a typical orchestration loop. It uses three different providers to showcase how easily each module can work with distinct models and strategies. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Highlights: - Different modules can run on different LMs and temperatures. - Tools are provided either at construction or per-call. - `context_defaults` ensures each `.forward()` call enters a proper `dspy.context()` with the module's LM. --- ## βš™οΈ Configuration & Storage ROMA-DSPy uses **OmegaConf** for layered configuration with **Pydantic** validation, and provides **execution-scoped storage** for complete task isolation. ### Quick Configuration Example ```python from roma_dspy.config import load_config # Load with profile and overrides config = load_config( profile="crypto_agent", overrides=["agents.executor.llm.temperature=0.3"] ) ``` **Available Profiles**: `general`, `crypto_agent` (list with `just list-profiles`) **See**: [Configuration Guide](docs/CONFIGURATION.md) for complete documentation on profiles, agent configuration, LLM settings, toolkit configuration, and task-aware agent mapping. ### Storage Storage is automatic and execution-scoped - each task gets an isolated directory. Large toolkit responses (>100KB) are automatically stored as Parquet files. ```python from roma_dspy.core.engine.solve import solve # Storage created automatically at: {base_path}/executions/{execution_id}/ result = solve("Analyze blockchain transactions") ``` **Features**: Execution isolation, S3-compatible, automatic Parquet storage, Docker-managed **See**: [Deployment Guide](docs/DEPLOYMENT.md) for production storage configuration including S3 integration. --- ## 🧰 Toolkits ROMA-DSPy includes 9 built-in toolkits that extend agent capabilities: **Core**: FileToolkit, CalculatorToolkit, E2BToolkit (code execution) **Crypto**: CoinGeckoToolkit, BinanceToolkit, DefiLlamaToolkit, ArkhamToolkit **Search**: SerperToolkit (web search) **Universal**: MCPToolkit (connect to any [MCP server](https://github.com/wong2/awesome-mcp-servers)) ### Quick Configuration ```yaml agents: executor: toolkits: - class_name: "FileToolkit" enabled: true - class_name: "E2BToolkit" enabled: true toolkit_config: timeout: 600 ``` **See**: [Toolkits Reference](docs/TOOLKITS.md) for complete toolkit documentation including all tools, configuration options, MCP integration, and custom toolkit development. --- ## 🌐 REST API & CLI ROMA-DSPy provides both a REST API and CLI for production use. ### REST API FastAPI server with interactive documentation: ```bash # Starts automatically with Docker just docker-up # API Documentation: http://localhost:8000/docs # Health check: http://localhost:8000/health ``` **Endpoints**: Execution management, checkpoints, visualization, metrics ### CLI ```bash # Local task execution roma-dspy solve "Your task" --profile general # Server management roma-dspy server start roma-dspy server health # Execution management roma-dspy exec create "Task" roma-dspy exec status --watch # Interactive TUI visualization (requires MLflow for best results) just viz # Full help roma-dspy --help ``` **See**: API documentation at `/docs` endpoint for complete OpenAPI specification and interactive testing. --- ## πŸ—οΈ Core Building Block: `BaseModule` All modules inherit from `BaseModule`, located at `roma_dspy/core/modules/base_module.py`. It standardizes: - signature binding via DSPy prediction strategies, - LM instantiation and context management, - tool normalization and merging, - sync/async entrypoints with safe keyword filtering. ### Context & LM Management When you instantiate a module, you can either provide an existing `dspy.LM` or let the module build one from a provider string (`model`) and optional keyword arguments (`model_config`). ```python from roma_dspy import Executor executor = Executor( model="openrouter/openai/gpt-4o-mini", model_config={"temperature": 0.5, "cache": True}, ) ``` Internally, `BaseModule` ensures that every `.forward()` call wraps the predictor invocation in: ```python with dspy.context(lm=self._lm, **context_defaults): ... ``` You can inspect the effective LM configuration via `get_model_config()` to confirm provider, cache settings, or sanitized kwargs. ### Working with Tools Tools can be supplied as a list, tuple, or mapping of callables accepted by DSPy’s ReAct/CodeAct strategies. ```python executor = Executor(tools=[get_weather]) executor.forward("What is the weather in Amman?", tools=[another_function]) ``` `BaseModule` automatically deduplicates tools based on object identity and merges constructor defaults with per-call overrides. ### Prediction Strategies ROMA exposes DSPy's strategies through the `PredictionStrategy` enum (`roma_dspy/types/prediction_strategy.py`). Use either the enum or a case-insensitive string alias: ```python from roma_dspy.types import PredictionStrategy planner = Planner(prediction_strategy=PredictionStrategy.CHAIN_OF_THOUGHT) executor = Executor(prediction_strategy="react") ``` Available options include `Predict`, `ChainOfThought`, `ReAct`, `CodeAct`, `BestOfN`, `Refine`, `Parallel`, `majority`, and more. Strategies that require tools (`ReAct`, `CodeAct`) automatically receive any tools you pass to the module. ### Async Execution Every module offers an `aforward()` method. When the underlying DSPy predictor supports async (`acall`/`aforward`), ROMA dispatches asynchronously; otherwise, it gracefully falls back to the sync implementation while preserving awaitability. ```python result = await executor.aforward("Download the latest sales report") ``` ## πŸ“š Module Reference ### βš›οΈ Atomizer **Location**: `roma_dspy/core/modules/atomizer.py` **Purpose**: Decide whether a goal is atomic or needs planning. **Constructor**: ```python Atomizer( prediction_strategy: Union[PredictionStrategy, str] = "ChainOfThought", *, lm: Optional[dspy.LM] = None, model: Optional[str] = None, model_config: Optional[Mapping[str, Any]] = None, tools: Optional[Sequence|Mapping] = None, **strategy_kwargs, ) ``` **Inputs** (`AtomizerSignature`): - `goal: str` **Outputs** (`AtomizerResponse`): - `is_atomic: bool` β€” whether the task can run directly. - `node_type: NodeType` β€” `PLAN` or `EXECUTE` hint for downstream routing. **Usage**: ```python atomized = atomizer.forward("Curate a 5-day Tokyo itinerary with restaurant reservations") if atomized.is_atomic: ... # send directly to Executor else: ... # hand off to Planner ``` The Atomizer is strategy-agnostic but typically uses `ChainOfThought` or `Predict`. You can pass hints (e.g., `max_tokens`) via `call_params`: ```python atomizer.forward( "Summarize this PDF", call_params={"max_tokens": 200}, ) ``` ### πŸ“‹ Planner **Location**: `roma_dspy/core/modules/planner.py` **Purpose**: Break a goal into ordered subtasks with optional dependency graph. **Constructor**: identical pattern as the Atomizer. **Inputs** (`PlannerSignature`): - `goal: str` **Outputs** (`PlannerResult`): - `subtasks: List[SubTask]` β€” each has `goal`, `task_type`, and `dependencies`. - `dependencies_graph: Optional[Dict[str, List[str]]]` β€” explicit adjacency mapping when returned by the LM. **Usage**: ```python plan = planner.forward("Launch a B2B webinar in 6 weeks") for subtask in plan.subtasks: print(subtask.goal, subtask.task_type) ``` `SubTask.task_type` is a `TaskType` enum that follows the ROMA MECE framework (Retrieve, Write, Think, Code Interpret, Image Generation). ### βš™οΈ Executor **Location**: `roma_dspy/core/modules/executor.py` **Purpose**: Resolve atomic goals, optionally calling tools/functions through DSPy's ReAct, CodeAct, or similar strategies. **Constructor**: same pattern; the most common strategies are `ReAct`, `CodeAct`, or `ChainOfThought`. **Inputs** (`ExecutorSignature`): - `goal: str` **Outputs** (`ExecutorResult`): - `output: str | Any` - `sources: Optional[List[str]]` β€” provenance or citations. **Usage**: ```python execution = executor.forward( "Compile a packing list for a 3-day ski trip", config={"temperature": 0.4}, # per-call LM override ) print(execution.output) ``` To expose tools only for certain calls: ```python execution = executor.forward( "What is the weather in Paris?", tools=[get_weather], ) ``` ### πŸ”€ Aggregator **Location**: `roma_dspy/core/modules/aggregator.py` **Purpose**: Combine multiple subtask results into a final narrative or decision. **Constructor**: identical pattern. **Inputs** (`AggregatorResult` signature): - `original_goal: str` - `subtasks_results: List[SubTask]` β€” usually the planner’s proposals augmented with execution outputs. **Outputs** (`AggregatorResult` base model): - `synthesized_result: str` **Usage**: ```python aggregated = aggregator.forward( original_goal="Plan a data migration", subtasks_results=[ SubTask(goal="Inventory current databases", task_type=TaskType.RETRIEVE), SubTask(goal="Draft migration timeline", task_type=TaskType.WRITE), ], ) print(aggregated.synthesized_result) ``` Because it inherits `BaseModule`, you can still attach tools (e.g., a knowledge-base retrieval function) if your aggregation strategy requires external calls. ### βœ… Verifier **Location**: `roma_dspy/core/modules/verifier.py` **Purpose**: Validate that the synthesized output satisfies the original goal. **Inputs** (`VerifierSignature`): - `goal: str` - `candidate_output: str` **Outputs**: - `verdict: bool` - `feedback: Optional[str]` **Usage**: ```python verdict = verifier.forward( goal="Draft a GDPR-compliant privacy policy", candidate_output=aggregated.synthesized_result, ) if not verdict.verdict: print("Needs revision:", verdict.feedback) ``` ## 🎯 Advanced Patterns ### Swapping Models at Runtime Use `replace_lm()` to reuse the same module with a different LM (useful for A/B testing or fallbacks). ```python fast_executor = executor.replace_lm(dspy.LM("openrouter/anthropic/claude-3-haiku")) ``` ### Per-Call Overrides You can alter LM behavior or provide extra parameters without rebuilding the module. ```python executor.forward( "Summarize the meeting notes", config={"temperature": 0.1, "max_tokens": 300}, context={"stop": ["Observation:"]}, ) ``` `call_params` (or keyword arguments) are filtered to match the DSPy predictor’s accepted kwargs, preventing accidental errors. ### Tool-Only Execution If you want deterministic tool routing, you can set a dummy LM (or a very low-temperature model) and pass pure Python callables. ```python from roma_dspy import Executor executor = Executor( prediction_strategy="code_act", lm=dspy.LM("openrouter/openai/gpt-4o-mini", temperature=0.0), tools={"get_weather": get_weather, "lookup_user": lookup_user}, ) ``` ROMA will ensure both constructor and per-call tools are available to the strategy. ## πŸ§ͺ Testing ```bash # Run all tests just test # Run specific tests pytest tests/unit/ -v pytest tests/integration/ -v ``` **See**: `justfile` for all available test commands. ## πŸ’‘ Troubleshooting & Tips - **`ValueError: Either provide an existing lm`** β€” supply `lm=` or `model=` when constructing the module. - **`Invalid prediction strategy`** β€” check spelling; strings are case-insensitive but must match a known alias. - **Caching** β€” pass `cache=True` on your LM or set it in `model_config` to reutilize previous completions. - **Async contexts** β€” when mixing sync and async calls, ensure your event loop is running (e.g., use `asyncio.run`). - **Tool duplicates** β€” tools are deduplicated by identity; create distinct functions if you need variations. ## πŸ“– Glossary ### Core Concepts - **DSPy**: Stanford's declarative framework for prompting, planning, and tool integration. - **Prediction Strategy**: The DSPy class/function that powers reasoning (CoT, ReAct, etc.). - **SubTask**: Pydantic model describing a decomposed unit of work (`goal`, `task_type`, `dependencies`). - **NodeType**: Whether the Atomizer chose to `PLAN` or `EXECUTE`. - **TaskType**: MECE classification for subtasks (`RETRIEVE`, `WRITE`, `THINK`, `CODE_INTERPRET`, `IMAGE_GENERATION`). - **Context Defaults**: Keyword arguments provided to `dspy.context(...)` on every call. ### Configuration & Storage - **FileStorage**: Execution-scoped storage manager providing isolated directories per task execution. - **DataStorage**: Automatic Parquet storage system for large toolkit responses (threshold-based). - **Execution ID**: Unique identifier for each task execution, used for storage isolation. - **Base Path**: Root directory for all storage operations (local path or S3 bucket). - **Profile**: Named configuration preset (e.g., `general`, `crypto_agent`). - **Configuration Override**: Runtime value that supersedes profile/default settings. ### Toolkits - **BaseToolkit**: Abstract base class for all toolkits providing storage integration and tool registration. - **REQUIRES_FILE_STORAGE**: Metadata flag indicating a toolkit requires FileStorage (e.g., FileToolkit). - **Toolkit Config**: Toolkit-specific settings like API keys, timeouts, and thresholds. - **Tool Selection**: Include/exclude lists to filter which tools from a toolkit are available. - **Storage Threshold**: Size limit (KB) above which responses are stored in Parquet format. ### Architecture - **Execution-Scoped Isolation**: Pattern where each execution gets unique storage directory. - **Parquet Integration**: Automatic columnar storage for large structured data. - **S3 Compatibility**: Ability to use S3-compatible storage via Docker volume mounts. - **Tool Registration**: Automatic discovery and registration of toolkit methods as callable tools. --- Happy building! If you extend or customize a module, keep the signatures aligned so your higher-level orchestration remains stable. **Additional Resources:** - [Quick Start Guide](docs/QUICKSTART.md) - Get started in under 10 minutes - [Configuration Guide](docs/CONFIGURATION.md) - Complete configuration reference - [Toolkits Reference](docs/TOOLKITS.md) - All built-in and custom toolkits - [Deployment Guide](docs/DEPLOYMENT.md) - Production deployment with Docker - [E2B Setup](docs/E2B_SETUP.md) - Code execution toolkit setup - [Observability](docs/OBSERVABILITY.md) - MLflow tracking and monitoring - [Configuration System](config/README.md) - Configuration profiles and examples ## πŸ“Š Benchmarks We evaluate our simple implementation of a search system using ROMA, called ROMA-Search across three benchmarks: **SEAL-0**, **FRAMES**, and **SimpleQA**. Below are the performance graphs for each benchmark. ### [SEAL-0](https://huggingface.co/datasets/vtllms/sealqa) SealQA is a new challenging benchmark for evaluating Search-Augmented Language models on fact-seeking questions where web search yields conflicting, noisy, or unhelpful results. --- ### [FRAMES](https://huggingface.co/datasets/google/frames-benchmark) View full results A comprehensive evaluation dataset designed to test the capabilities of Retrieval-Augmented Generation (RAG) systems across factuality, retrieval accuracy, and reasoning. --- ### [SimpleQA](https://openai.com/index/introducing-simpleqa/) View full results Factuality benchmark that measures the ability for language models to answer short, fact-seeking questions. ## 🧩 Foundations & Lineage While ROMA introduces a practical, open-source framework for hierarchical task execution, it is directly built upon two foundational research contributions introduced in [WriteHERE](https://arxiv.org/abs/2503.08275): - **Heterogeneous Recursive Planning** β€” The overall architecture of ROMA follows the framework first introduced in prior work on *heterogeneous recursive planning*, where complex tasks are recursively decomposed into a graph of subtasks, each assigned a distinct cognitive type. - **Type Specification in Decomposition** β€” ROMA’s β€œThree Universal Operations” (THINK πŸ€”, WRITE ✍️, SEARCH πŸ”) generalize the *type specification in decomposition* hypothesis, which identified reasoning, composition, and retrieval as the three fundamental cognitive types. These contributions are described in detail in the WriteHERE repository and paper. By explicitly adopting and extending this foundation, ROMA provides a **generalizable scaffold, agent system, versatility, and extensibility** that builds upon these insights and makes them usable for builders across domains. ## πŸ™ Acknowledgments This framework would not have been possible if it wasn't for these amazing open-source contributions! - Inspired by the hierarchical planning approach described in ["Beyond Outlining: Heterogeneous Recursive Planning"](https://arxiv.org/abs/2503.08275) by Xiong et al. - [Pydantic](https://github.com/pydantic/pydantic) - Data validation using Python type annotations - [DSPy]([https://dspy.ai/)) - Framework for programming AI agents - [E2B](https://github.com/e2b-dev/e2b) - Cloud runtime for AI agents ## πŸ“š Citation If you use the ROMA repo in your research, please cite: ```bibtex @misc{alzubi2026romarecursiveopenmetaagent, title={ROMA: Recursive Open Meta-Agent Framework for Long-Horizon Multi-Agent Systems}, author={Salaheddin Alzu'bi and Baran Nama and Arda Kaz and Anushri Eswaran and Weiyuan Chen and Sarvesh Khetan and Rishab Bala and Tu Vu and Sewoong Oh}, year={2026}, eprint={2602.01848}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2602.01848}, } ``` ## 🌟 Star History ## πŸ“„ License This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details. ## 2. In-Tree Documentation Chapters (sentient-agi/ROMA) ## File: README.md ## πŸ“‘ Table of Contents - [🧠 Conceptual Overview](#-conceptual-overview) - [πŸ“¦ Installation & Setup](#-installation--setup) - [⚑ Quickstart: End-to-End Workflow](#-quickstart-end-to-end-workflow) - [βš™οΈ Configuration & Storage](#-configuration--storage) - [🧰 Toolkits](#-toolkits) - [🌐 REST API & CLI](#-rest-api--cli) - [πŸ—οΈ Core Building Block: `BaseModule`](#-core-building-block-basemodule) - [πŸ“š Module Reference](#-module-reference) - [βš›οΈ Atomizer](#-atomizer) - [πŸ“‹ Planner](#-planner) - [βš™οΈ Executor](#-executor) - [πŸ”€ Aggregator](#-aggregator) - [βœ… Verifier](#-verifier) - [🎯 Advanced Patterns](#-advanced-patterns) - [πŸ§ͺ Testing](#-testing) - [πŸ’‘ Troubleshooting & Tips](#-troubleshooting--tips) - [πŸ“– Glossary](#-glossary) --- ## 🎯 What is ROMA? **ROMA** is a **meta-agent framework** that uses recursive hierarchical structures to solve complex problems. By breaking down tasks into parallelizable components, ROMA enables agents to tackle sophisticated reasoning challenges while maintaining transparency that makes context-engineering and iteration straightforward. The framework offers **parallel problem solving** where agents work simultaneously on different parts of complex tasks, **transparent development** with a clear structure for easy debugging, and **proven performance** demonstrated through our search agent's strong benchmark results. We've shown the framework's effectiveness, but this is just the beginning. As an **open-source and extensible** platform, ROMA is designed for community-driven development, allowing you to build and customize agents for your specific needs while benefiting from the collective improvements of the community. ## πŸ—οΈ How It Works **ROMA** framework processes tasks through a recursive **plan–execute loop**: ```python def solve(task): if is_atomic(task): # Step 1: Atomizer return execute(task) # Step 2: Executor else: subtasks = plan(task) # Step 2: Planner results = [] for subtask in subtasks: results.append(solve(subtask)) # Recursive call return aggregate(results) # Step 3: Aggregator # Entry point: answer = solve(initial_request) ``` 1. **Atomizer** – Decides whether a request is **atomic** (directly executable) or requires **planning**. 2. **Planner** – If planning is needed, the task is broken into smaller **subtasks**. Each subtask is fed back into the **Atomizer**, making the process recursive. 3. **Executors** – Handle atomic tasks. Executors can be **LLMs, APIs, or even other agents** β€” as long as they implement an `agent.execute()` interface. 4. **Aggregator** – Collects and integrates results from subtasks. Importantly, the Aggregator produces the **answer to the original parent task**, not just raw child outputs. #### πŸ“ Information Flow - **Top-down:** Tasks are decomposed into subtasks recursively. - **Bottom-up:** Subtask results are aggregated upwards into solutions for parent tasks. - **Left-to-right:** If a subtask depends on the output of a previous one, it waits until that subtask completes before execution. This structure makes the system flexible, recursive, and dependency-aware β€” capable of decomposing complex problems into smaller steps while ensuring results are integrated coherently. Click to view the system flow diagram ```mermaid flowchart TB A[Your Request] --> B{Atomizer} B -->|Plan Needed| C[Planner] B -->|Atomic Task| D[Executor] %% Planner spawns subtasks C --> E[Subtasks] E --> G[Aggregator] %% Recursion E -.-> B %% Execution + Aggregation D --> F[Final Result] G --> F style A fill:#e1f5fe style F fill:#c8e6c9 style B fill:#fff3e0 style C fill:#ffe0b2 style D fill:#d1c4e9 style G fill:#c5cae9 ``` ## πŸš€ Quick Start ### Fastest Way: Minimal Installation (Recommended for Evaluation) Get started in **under 30 seconds** with no infrastructure required: ```bash # Install with uv (10-100x faster) uv pip install roma-dspy # Or with pip pip install roma-dspy # Set your OpenRouter API key (default uses Claude Sonnet 4.5 + Gemini 2.5 Flash) export OPENROUTER_API_KEY="sk-or-v1-..." # Start solving tasks immediately python -c "from roma_dspy.core.engine.solve import solve; print(solve('What is 2+2?'))" ``` > **Note**: The default configuration uses OpenRouter with Claude Sonnet 4.5 (executor) and Gemini 2.5 Flash (other agents). You can also use OpenAI directly by setting `OPENAI_API_KEY` and customizing the config. **What you get:** - βœ… Core agent framework (Atomizer, Planner, Executor, Aggregator, Verifier) - βœ… All DSPy prediction strategies (CoT, ReAct, CodeAct, etc.) - βœ… File storage (no database required) - βœ… Built-in toolkits (Calculator, File operations) - βœ… Works with any LLM provider (OpenRouter, OpenAI, Anthropic, etc.) **No Docker, no database, no setup - just install and go!** ### Production Setup: Full Features with Docker For production use with persistence, observability, and API server: ```bash # One-command setup (builds Docker, starts services) just setup # Or with specific profile just setup crypto_agent # Verify services running curl http://localhost:8000/health # Solve tasks via API just solve "What is the capital of France?" ``` **Additional features with Docker:** - πŸ“Š PostgreSQL persistence (execution history, checkpoints) - πŸ“ˆ MLflow observability (experiment tracking, visualization) - 🌐 REST API server (FastAPI with interactive docs) - πŸ“¦ S3-compatible storage (MinIO) - πŸ”§ E2B code execution sandboxes - 🎨 Interactive TUI visualization **Services Available:** - πŸš€ **REST API**: http://localhost:8000/docs - πŸ—„οΈ **PostgreSQL**: Automatic persistence - πŸ“¦ **MinIO**: S3-compatible storage (http://localhost:9001) - πŸ“Š **MLflow**: http://localhost:5000 (with `docker-up-full`) See [Quick Start Guide](docs/QUICKSTART.md) and [Deployment Guide](docs/DEPLOYMENT.md) for details. --- ## 🧠 Conceptual Overview ROMA's module layer wraps canonical DSPy patterns into purpose-built components that reflect the lifecycle of complex task execution: 1. **Atomizer** decides whether a request can be handled directly or needs decomposition. 2. **Planner** breaks non-atomic goals into an ordered graph of subtasks. 3. **Executor** resolves individual subtasks, optionally routing through function/tool calls. 4. **Aggregator** synthesizes subtask outputs back into a coherent answer. 5. **Verifier** (optional) inspects the aggregate output against the original goal before delivering. Every module shares the same ergonomics: instantiate it with a language model (LM) or provider string, choose a prediction strategy, then call `.forward()` (or `.aforward()` for async) with the task-specific fields. All modules ultimately delegate to DSPy signatures defined in `roma_dspy.core.signatures`. This keeps interfaces stable even as the internals evolve. ## πŸ“¦ Installation & Setup ### Option 1: Minimal Installation (Fastest - Recommended for Evaluation) **Perfect for:** Evaluating ROMA, development, testing, quick prototyping **Install in under 30 seconds:** ```bash # With uv (recommended - 10-100x faster) uv pip install roma-dspy # Or with pip pip install roma-dspy ``` **Set your API key:** ```bash export OPENROUTER_API_KEY="sk-or-v1-..." # Recommended # OR export OPENAI_API_KEY="sk-..." export ANTHROPIC_API_KEY="sk-ant-..." ``` **Start using immediately:** ```python from roma_dspy.core.engine.solve import solve # Solve any task result = solve("What is the capital of France?") print(result) ``` **What's included:** - βœ… All core modules (Atomizer, Planner, Executor, Aggregator, Verifier) - βœ… All DSPy prediction strategies - βœ… File-based storage (no database needed) - βœ… Core toolkits (Calculator, File operations) - βœ… Works with any LLM provider **What's NOT included (install separately if needed):** - PostgreSQL persistence β†’ `uv pip install roma-dspy[persistence]` - MLflow observability β†’ `uv pip install roma-dspy[observability]` - E2B code execution β†’ `uv pip install roma-dspy[e2b]` - REST API server β†’ `uv pip install roma-dspy[api]` - S3 storage β†’ `uv pip install roma-dspy[s3]` - All features β†’ `uv pip install roma-dspy[all]` --- ### Option 2: Full Installation with Docker (Production) **Perfect for:** Production deployment, teams, full observability **Prerequisites:** - Docker & Docker Compose - Python 3.12+ (for local development) - [Just](https://github.com/casey/just) command runner (optional, recommended) **One-command setup:** ```bash # Interactive setup (prompts for E2B, S3, etc.) just setup # Or with specific profile just setup crypto_agent ``` **Manual Docker start:** ```bash just docker-up # Basic (PostgreSQL + MinIO + API) just docker-up-full # With MLflow observability ``` **Environment variables** (auto-configured by `just setup`): ```bash # LLM Provider (required) OPENROUTER_API_KEY=... # Recommended # OR OPENAI_API_KEY=... ANTHROPIC_API_KEY=... # Optional: Advanced features E2B_API_KEY=... # Code execution COINGECKO_API_KEY=... # Crypto toolkit ``` **Additional Docker features:** - πŸ“Š PostgreSQL (execution history, checkpoints) - πŸ“ˆ MLflow (experiment tracking, metrics) - 🌐 REST API (FastAPI with docs) - πŸ“¦ MinIO (S3-compatible storage) - 🎨 TUI visualization --- ### Option 3: Development Installation **For contributing or extending ROMA:** ```bash # Clone repository git clone https://github.com/sentient-agi/roma.git cd roma # Install with dev tools (includes pytest, ruff, mypy) uv pip install -e ".[dev]" # Run tests just test # Format code just format # Type check just typecheck ``` --- ### Comparison: Which Installation is Right for You? | Feature | Minimal (`pip install roma-dspy`) | Docker (`just setup`) | |---------|----------------------------------|----------------------| | Installation time | **< 30 seconds** | ~5-10 minutes | | Infrastructure required | **None** | Docker | | Core agent framework | βœ… | βœ… | | File storage | βœ… | βœ… | | PostgreSQL persistence | ❌ (install `[persistence]`) | βœ… | | MLflow observability | ❌ (install `[observability]`) | βœ… | | REST API | ❌ (install `[api]`) | βœ… | | Best for | Evaluation, development | Production, teams | ## ⚑ Quickstart: End-to-End Workflow The following example mirrors a typical orchestration loop. It uses three different providers to showcase how easily each module can work with distinct models and strategies. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Highlights: - Different modules can run on different LMs and temperatures. - Tools are provided either at construction or per-call. - `context_defaults` ensures each `.forward()` call enters a proper `dspy.context()` with the module's LM. --- ## βš™οΈ Configuration & Storage ROMA-DSPy uses **OmegaConf** for layered configuration with **Pydantic** validation, and provides **execution-scoped storage** for complete task isolation. ### Quick Configuration Example ```python from roma_dspy.config import load_config # Load with profile and overrides config = load_config( profile="crypto_agent", overrides=["agents.executor.llm.temperature=0.3"] ) ``` **Available Profiles**: `general`, `crypto_agent` (list with `just list-profiles`) **See**: [Configuration Guide](docs/CONFIGURATION.md) for complete documentation on profiles, agent configuration, LLM settings, toolkit configuration, and task-aware agent mapping. ### Storage Storage is automatic and execution-scoped - each task gets an isolated directory. Large toolkit responses (>100KB) are automatically stored as Parquet files. ```python from roma_dspy.core.engine.solve import solve # Storage created automatically at: {base_path}/executions/{execution_id}/ result = solve("Analyze blockchain transactions") ``` **Features**: Execution isolation, S3-compatible, automatic Parquet storage, Docker-managed **See**: [Deployment Guide](docs/DEPLOYMENT.md) for production storage configuration including S3 integration. --- ## 🧰 Toolkits ROMA-DSPy includes 9 built-in toolkits that extend agent capabilities: **Core**: FileToolkit, CalculatorToolkit, E2BToolkit (code execution) **Crypto**: CoinGeckoToolkit, BinanceToolkit, DefiLlamaToolkit, ArkhamToolkit **Search**: SerperToolkit (web search) **Universal**: MCPToolkit (connect to any [MCP server](https://github.com/wong2/awesome-mcp-servers)) ### Quick Configuration ```yaml agents: executor: toolkits: - class_name: "FileToolkit" enabled: true - class_name: "E2BToolkit" enabled: true toolkit_config: timeout: 600 ``` **See**: [Toolkits Reference](docs/TOOLKITS.md) for complete toolkit documentation including all tools, configuration options, MCP integration, and custom toolkit development. --- ## 🌐 REST API & CLI ROMA-DSPy provides both a REST API and CLI for production use. ### REST API FastAPI server with interactive documentation: ```bash # Starts automatically with Docker just docker-up # API Documentation: http://localhost:8000/docs # Health check: http://localhost:8000/health ``` **Endpoints**: Execution management, checkpoints, visualization, metrics ### CLI ```bash # Local task execution roma-dspy solve "Your task" --profile general # Server management roma-dspy server start roma-dspy server health # Execution management roma-dspy exec create "Task" roma-dspy exec status --watch # Interactive TUI visualization (requires MLflow for best results) just viz # Full help roma-dspy --help ``` **See**: API documentation at `/docs` endpoint for complete OpenAPI specification and interactive testing. --- ## πŸ—οΈ Core Building Block: `BaseModule` All modules inherit from `BaseModule`, located at `roma_dspy/core/modules/base_module.py`. It standardizes: - signature binding via DSPy prediction strategies, - LM instantiation and context management, - tool normalization and merging, - sync/async entrypoints with safe keyword filtering. ### Context & LM Management When you instantiate a module, you can either provide an existing `dspy.LM` or let the module build one from a provider string (`model`) and optional keyword arguments (`model_config`). ```python from roma_dspy import Executor executor = Executor( model="openrouter/openai/gpt-4o-mini", model_config={"temperature": 0.5, "cache": True}, ) ``` Internally, `BaseModule` ensures that every `.forward()` call wraps the predictor invocation in: ```python with dspy.context(lm=self._lm, **context_defaults): ... ``` You can inspect the effective LM configuration via `get_model_config()` to confirm provider, cache settings, or sanitized kwargs. ### Working with Tools Tools can be supplied as a list, tuple, or mapping of callables accepted by DSPy’s ReAct/CodeAct strategies. ```python executor = Executor(tools=[get_weather]) executor.forward("What is the weather in Amman?", tools=[another_function]) ``` `BaseModule` automatically deduplicates tools based on object identity and merges constructor defaults with per-call overrides. ### Prediction Strategies ROMA exposes DSPy's strategies through the `PredictionStrategy` enum (`roma_dspy/types/prediction_strategy.py`). Use either the enum or a case-insensitive string alias: ```python from roma_dspy.types import PredictionStrategy planner = Planner(prediction_strategy=PredictionStrategy.CHAIN_OF_THOUGHT) executor = Executor(prediction_strategy="react") ``` Available options include `Predict`, `ChainOfThought`, `ReAct`, `CodeAct`, `BestOfN`, `Refine`, `Parallel`, `majority`, and more. Strategies that require tools (`ReAct`, `CodeAct`) automatically receive any tools you pass to the module. ### Async Execution Every module offers an `aforward()` method. When the underlying DSPy predictor supports async (`acall`/`aforward`), ROMA dispatches asynchronously; otherwise, it gracefully falls back to the sync implementation while preserving awaitability. ```python result = await executor.aforward("Download the latest sales report") ``` ## πŸ“š Module Reference ### βš›οΈ Atomizer **Location**: `roma_dspy/core/modules/atomizer.py` **Purpose**: Decide whether a goal is atomic or needs planning. **Constructor**: ```python Atomizer( prediction_strategy: Union[PredictionStrategy, str] = "ChainOfThought", *, lm: Optional[dspy.LM] = None, model: Optional[str] = None, model_config: Optional[Mapping[str, Any]] = None, tools: Optional[Sequence|Mapping] = None, **strategy_kwargs, ) ``` **Inputs** (`AtomizerSignature`): - `goal: str` **Outputs** (`AtomizerResponse`): - `is_atomic: bool` β€” whether the task can run directly. - `node_type: NodeType` β€” `PLAN` or `EXECUTE` hint for downstream routing. **Usage**: ```python atomized = atomizer.forward("Curate a 5-day Tokyo itinerary with restaurant reservations") if atomized.is_atomic: ... # send directly to Executor else: ... # hand off to Planner ``` The Atomizer is strategy-agnostic but typically uses `ChainOfThought` or `Predict`. You can pass hints (e.g., `max_tokens`) via `call_params`: ```python atomizer.forward( "Summarize this PDF", call_params={"max_tokens": 200}, ) ``` ### πŸ“‹ Planner **Location**: `roma_dspy/core/modules/planner.py` **Purpose**: Break a goal into ordered subtasks with optional dependency graph. **Constructor**: identical pattern as the Atomizer. **Inputs** (`PlannerSignature`): - `goal: str` **Outputs** (`PlannerResult`): - `subtasks: List[SubTask]` β€” each has `goal`, `task_type`, and `dependencies`. - `dependencies_graph: Optional[Dict[str, List[str]]]` β€” explicit adjacency mapping when returned by the LM. **Usage**: ```python plan = planner.forward("Launch a B2B webinar in 6 weeks") for subtask in plan.subtasks: print(subtask.goal, subtask.task_type) ``` `SubTask.task_type` is a `TaskType` enum that follows the ROMA MECE framework (Retrieve, Write, Think, Code Interpret, Image Generation). ### βš™οΈ Executor **Location**: `roma_dspy/core/modules/executor.py` **Purpose**: Resolve atomic goals, optionally calling tools/functions through DSPy's ReAct, CodeAct, or similar strategies. **Constructor**: same pattern; the most common strategies are `ReAct`, `CodeAct`, or `ChainOfThought`. **Inputs** (`ExecutorSignature`): - `goal: str` **Outputs** (`ExecutorResult`): - `output: str | Any` - `sources: Optional[List[str]]` β€” provenance or citations. **Usage**: ```python execution = executor.forward( "Compile a packing list for a 3-day ski trip", config={"temperature": 0.4}, # per-call LM override ) print(execution.output) ``` To expose tools only for certain calls: ```python execution = executor.forward( "What is the weather in Paris?", tools=[get_weather], ) ``` ### πŸ”€ Aggregator **Location**: `roma_dspy/core/modules/aggregator.py` **Purpose**: Combine multiple subtask results into a final narrative or decision. **Constructor**: identical pattern. **Inputs** (`AggregatorResult` signature): - `original_goal: str` - `subtasks_results: List[SubTask]` β€” usually the planner’s proposals augmented with execution outputs. **Outputs** (`AggregatorResult` base model): - `synthesized_result: str` **Usage**: ```python aggregated = aggregator.forward( original_goal="Plan a data migration", subtasks_results=[ SubTask(goal="Inventory current databases", task_type=TaskType.RETRIEVE), SubTask(goal="Draft migration timeline", task_type=TaskType.WRITE), ], ) print(aggregated.synthesized_result) ``` Because it inherits `BaseModule`, you can still attach tools (e.g., a knowledge-base retrieval function) if your aggregation strategy requires external calls. ### βœ… Verifier **Location**: `roma_dspy/core/modules/verifier.py` **Purpose**: Validate that the synthesized output satisfies the original goal. **Inputs** (`VerifierSignature`): - `goal: str` - `candidate_output: str` **Outputs**: - `verdict: bool` - `feedback: Optional[str]` **Usage**: ```python verdict = verifier.forward( goal="Draft a GDPR-compliant privacy policy", candidate_output=aggregated.synthesized_result, ) if not verdict.verdict: print("Needs revision:", verdict.feedback) ``` ## 🎯 Advanced Patterns ### Swapping Models at Runtime Use `replace_lm()` to reuse the same module with a different LM (useful for A/B testing or fallbacks). ```python fast_executor = executor.replace_lm(dspy.LM("openrouter/anthropic/claude-3-haiku")) ``` ### Per-Call Overrides You can alter LM behavior or provide extra parameters without rebuilding the module. ```python executor.forward( "Summarize the meeting notes", config={"temperature": 0.1, "max_tokens": 300}, context={"stop": ["Observation:"]}, ) ``` `call_params` (or keyword arguments) are filtered to match the DSPy predictor’s accepted kwargs, preventing accidental errors. ### Tool-Only Execution If you want deterministic tool routing, you can set a dummy LM (or a very low-temperature model) and pass pure Python callables. ```python from roma_dspy import Executor executor = Executor( prediction_strategy="code_act", lm=dspy.LM("openrouter/openai/gpt-4o-mini", temperature=0.0), tools={"get_weather": get_weather, "lookup_user": lookup_user}, ) ``` ROMA will ensure both constructor and per-call tools are available to the strategy. ## πŸ§ͺ Testing ```bash # Run all tests just test # Run specific tests pytest tests/unit/ -v pytest tests/integration/ -v ``` **See**: `justfile` for all available test commands. ## πŸ’‘ Troubleshooting & Tips - **`ValueError: Either provide an existing lm`** β€” supply `lm=` or `model=` when constructing the module. - **`Invalid prediction strategy`** β€” check spelling; strings are case-insensitive but must match a known alias. - **Caching** β€” pass `cache=True` on your LM or set it in `model_config` to reutilize previous completions. - **Async contexts** β€” when mixing sync and async calls, ensure your event loop is running (e.g., use `asyncio.run`). - **Tool duplicates** β€” tools are deduplicated by identity; create distinct functions if you need variations. ## πŸ“– Glossary ### Core Concepts - **DSPy**: Stanford's declarative framework for prompting, planning, and tool integration. - **Prediction Strategy**: The DSPy class/function that powers reasoning (CoT, ReAct, etc.). - **SubTask**: Pydantic model describing a decomposed unit of work (`goal`, `task_type`, `dependencies`). - **NodeType**: Whether the Atomizer chose to `PLAN` or `EXECUTE`. - **TaskType**: MECE classification for subtasks (`RETRIEVE`, `WRITE`, `THINK`, `CODE_INTERPRET`, `IMAGE_GENERATION`). - **Context Defaults**: Keyword arguments provided to `dspy.context(...)` on every call. ### Configuration & Storage - **FileStorage**: Execution-scoped storage manager providing isolated directories per task execution. - **DataStorage**: Automatic Parquet storage system for large toolkit responses (threshold-based). - **Execution ID**: Unique identifier for each task execution, used for storage isolation. - **Base Path**: Root directory for all storage operations (local path or S3 bucket). - **Profile**: Named configuration preset (e.g., `general`, `crypto_agent`). - **Configuration Override**: Runtime value that supersedes profile/default settings. ### Toolkits - **BaseToolkit**: Abstract base class for all toolkits providing storage integration and tool registration. - **REQUIRES_FILE_STORAGE**: Metadata flag indicating a toolkit requires FileStorage (e.g., FileToolkit). - **Toolkit Config**: Toolkit-specific settings like API keys, timeouts, and thresholds. - **Tool Selection**: Include/exclude lists to filter which tools from a toolkit are available. - **Storage Threshold**: Size limit (KB) above which responses are stored in Parquet format. ### Architecture - **Execution-Scoped Isolation**: Pattern where each execution gets unique storage directory. - **Parquet Integration**: Automatic columnar storage for large structured data. - **S3 Compatibility**: Ability to use S3-compatible storage via Docker volume mounts. - **Tool Registration**: Automatic discovery and registration of toolkit methods as callable tools. --- Happy building! If you extend or customize a module, keep the signatures aligned so your higher-level orchestration remains stable. **Additional Resources:** - [Quick Start Guide](docs/QUICKSTART.md) - Get started in under 10 minutes - [Configuration Guide](docs/CONFIGURATION.md) - Complete configuration reference - [Toolkits Reference](docs/TOOLKITS.md) - All built-in and custom toolkits - [Deployment Guide](docs/DEPLOYMENT.md) - Production deployment with Docker - [E2B Setup](docs/E2B_SETUP.md) - Code execution toolkit setup - [Observability](docs/OBSERVABILITY.md) - MLflow tracking and monitoring - [Configuration System](config/README.md) - Configuration profiles and examples ## πŸ“Š Benchmarks We evaluate our simple implementation of a search system using ROMA, called ROMA-Search across three benchmarks: **SEAL-0**, **FRAMES**, and **SimpleQA**. Below are the performance graphs for each benchmark. ### [SEAL-0](https://huggingface.co/datasets/vtllms/sealqa) SealQA is a new challenging benchmark for evaluating Search-Augmented Language models on fact-seeking questions where web search yields conflicting, noisy, or unhelpful results. --- ### [FRAMES](https://huggingface.co/datasets/google/frames-benchmark) View full results A comprehensive evaluation dataset designed to test the capabilities of Retrieval-Augmented Generation (RAG) systems across factuality, retrieval accuracy, and reasoning. --- ### [SimpleQA](https://openai.com/index/introducing-simpleqa/) View full results Factuality benchmark that measures the ability for language models to answer short, fact-seeking questions. ## 🧩 Foundations & Lineage While ROMA introduces a practical, open-source framework for hierarchical task execution, it is directly built upon two foundational research contributions introduced in [WriteHERE](https://arxiv.org/abs/2503.08275): - **Heterogeneous Recursive Planning** β€” The overall architecture of ROMA follows the framework first introduced in prior work on *heterogeneous recursive planning*, where complex tasks are recursively decomposed into a graph of subtasks, each assigned a distinct cognitive type. - **Type Specification in Decomposition** β€” ROMA’s β€œThree Universal Operations” (THINK πŸ€”, WRITE ✍️, SEARCH πŸ”) generalize the *type specification in decomposition* hypothesis, which identified reasoning, composition, and retrieval as the three fundamental cognitive types. These contributions are described in detail in the WriteHERE repository and paper. By explicitly adopting and extending this foundation, ROMA provides a **generalizable scaffold, agent system, versatility, and extensibility** that builds upon these insights and makes them usable for builders across domains. ## πŸ™ Acknowledgments This framework would not have been possible if it wasn't for these amazing open-source contributions! - Inspired by the hierarchical planning approach described in ["Beyond Outlining: Heterogeneous Recursive Planning"](https://arxiv.org/abs/2503.08275) by Xiong et al. - [Pydantic](https://github.com/pydantic/pydantic) - Data validation using Python type annotations - [DSPy]([https://dspy.ai/)) - Framework for programming AI agents - [E2B](https://github.com/e2b-dev/e2b) - Cloud runtime for AI agents ## πŸ“š Citation If you use the ROMA repo in your research, please cite: ```bibtex @misc{alzubi2026romarecursiveopenmetaagent, title={ROMA: Recursive Open Meta-Agent Framework for Long-Horizon Multi-Agent Systems}, author={Salaheddin Alzu'bi and Baran Nama and Arda Kaz and Anushri Eswaran and Weiyuan Chen and Sarvesh Khetan and Rishab Bala and Tu Vu and Sewoong Oh}, year={2026}, eprint={2602.01848}, archivePrefix={arXiv}, primaryClass={cs.AI}, url={https://arxiv.org/abs/2602.01848}, } ``` ## 🌟 Star History ## πŸ“„ License This project is licensed under the Apache 2.0 License - see the [LICENSE](LICENSE) file for details. --- ## File: config/examples/README.md # ROMA-DSPy Configuration Examples This directory contains curated examples demonstrating key concepts and patterns for configuring ROMA-DSPy agents. ## Directory Structure ``` config/examples/ β”œβ”€β”€ basic/ # Basic concepts and patterns β”œβ”€β”€ mcp/ # Model Context Protocol (MCP) servers β”œβ”€β”€ crypto/ # Crypto/finance domain examples β”œβ”€β”€ advanced/ # Advanced patterns and techniques └── prompts/ # Custom prompt templates (Jinja) ``` ## Quick Start ### 1. Minimal Configuration ```bash uv run python -m roma_dspy.cli solve "Your task here" --config config/examples/basic/minimal.yaml ``` ### 2. Try MCP Public Server (No Setup!) ```bash uv run python -m roma_dspy.cli solve "What is the current price of Bitcoin?" --config config/examples/mcp/http_public_server.yaml ``` ## Examples by Category ### Basic (`basic/`) **Concepts**: Fundamentals, toolkit usage, multi-toolkit patterns | Example | Demonstrates | Setup Required | |---------|-------------|----------------| | `minimal.yaml` | Simplest possible configuration | ❌ No | | `multi_toolkit.yaml` | Combining multiple toolkits | E2B API key | **Key Learnings:** - How to configure agents and toolkits - Combining multiple tools in one agent - Basic runtime settings ### MCP (`mcp/`) **Concepts**: MCP servers, HTTP vs stdio, multi-server orchestration | Example | Demonstrates | Setup Required | |---------|-------------|----------------| | `http_public_server.yaml` | Public HTTP MCP server | ❌ No | | `stdio_local_server.yaml` | Local stdio MCP server | npm install | | `multi_server.yaml` | Multiple MCP servers | npm + API keys | **Key Learnings:** - MCP HTTP servers (remote, no installation) - MCP stdio servers (local subprocess) - Combining multiple MCP servers - Storage configuration for large data ### Crypto (`crypto/`) **Concepts**: Real-world domain-specific agents | Example | Demonstrates | Setup Required | |---------|-------------|----------------| | `crypto_agent.yaml` | Comprehensive crypto analysis | Optional API keys | **Key Learnings:** - Combining MCP + native toolkits - Multi-source data aggregation - Domain-specific agent design ### Advanced (`advanced/`) **Concepts**: Advanced patterns, optimization, customization | Example | Demonstrates | Setup Required | |---------|-------------|----------------| | `task_aware_mapping.yaml` | Task-specific executor configs | API keys | | `custom_prompts.yaml` | Custom prompts and demos | ❌ No | **Key Learnings:** - Task-aware agent mapping (RETRIEVE, CODE_INTERPRET, THINK, WRITE) - Cost/quality optimization per task type - Loading custom signature instructions - Few-shot learning with demos ## Configuration Patterns ### Basic Agent Structure ```yaml agents: executor: llm: model: openai/gpt-4o-mini temperature: 0.3 max_tokens: 2000 prediction_strategy: react # Required for tools toolkits: - class_name: ToolkitName enabled: true toolkit_config: # Toolkit-specific settings ``` ### Task-Aware Mapping ```yaml agents: executor: # Default configuration agent_mapping: executors: RETRIEVE: # Fast model + web search CODE_INTERPRET: # Powerful model + code execution THINK: # Reasoning-focused WRITE: # Creative writing ``` ### Custom Prompts ```yaml agents: executor: signature_instructions: "module.path:PROMPT_VAR" demos: "module.path:DEMOS_VAR" ``` ## Environment Variables Required for specific examples: ```bash # E2B (code execution) export E2B_API_KEY=your_key # Exa (web search via MCP) export EXA_API_KEY=your_key # GitHub MCP server export GITHUB_PERSONAL_ACCESS_TOKEN=your_token # Serper (web search toolkit) export SERPER_API_KEY=your_key # OpenRouter (recommended LLM provider) export OPENROUTER_API_KEY=your_key ``` ## Available Toolkits ### Native Toolkits (Built into ROMA-DSPy) | Toolkit | Purpose | API Key Required | |---------|---------|------------------| | **FileToolkit** | File operations | ❌ No | | **CalculatorToolkit** | Math operations | ❌ No | | **E2BToolkit** | Code execution | βœ… Yes | | **SerperToolkit** | Web search | βœ… Yes | | **BinanceToolkit** | Crypto market data | ❌ No (public endpoints) | | **CoinGeckoToolkit** | Crypto prices | ❌ No (public) | | **DefiLlamaToolkit** | DeFi protocol data | ❌ No | | **ArkhamToolkit** | Blockchain analytics | ❌ No | ### MCP Toolkits (via MCPToolkit) **Public HTTP Servers** (no setup): - CoinGecko: `https://mcp.api.coingecko.com/sse` - Exa: `https://mcp.exa.ai/mcp` (requires API key) **NPM Stdio Servers** (require `npm install -g`): - Filesystem: `@modelcontextprotocol/server-filesystem` - GitHub: `@modelcontextprotocol/server-github` - SQLite: `@modelcontextprotocol/server-sqlite` - Slack: `@modelcontextprotocol/server-slack` ## Tips for Success ### 1. Start Simple Begin with `basic/minimal.yaml`, then add complexity. ### 2. Use Public MCP Servers First Try `mcp/http_public_server.yaml` - no installation needed! ### 3. Task-Aware Mapping for Cost Optimization Use different models for different task types: - RETRIEVE: Fast, cheap models - CODE_INTERPRET: Powerful models - THINK: Reasoning-focused - WRITE: Creative models ### 4. Enable Storage for Large Data ```yaml use_storage: true storage_threshold_kb: 100 # Store results > 100KB ``` ### 5. Custom Prompts for Better Performance Load optimized prompts from `prompt_optimization/seed_prompts/` ## Common Issues ### "Unknown toolkit class" - Check spelling of `class_name` - Ensure toolkit is imported/registered ### "Tools don't support strategy" - Use `prediction_strategy: react` for tool usage - `chain_of_thought` doesn't support tools ### "API key required" - Set environment variable: `export API_KEY=value` - Or use `${oc.env:API_KEY}` in config ### MCP Server Connection Failed - **HTTP**: Check URL and network - **Stdio**: Ensure npm package installed globally ## Next Steps 1. **Copy and modify** examples for your use case 2. **Combine patterns** from different examples 3. **See profiles** in `config/profiles/` for complete configurations 4. **Read seed prompts** in `prompt_optimization/seed_prompts/` for inspiration ## Resources - **Main Documentation**: `/CLAUDE.md` - **Agent Profiles**: `config/profiles/` - **Seed Prompts**: `prompt_optimization/seed_prompts/` - **MCP Documentation**: https://modelcontextprotocol.io/ - **Awesome MCP Servers**: https://github.com/wong2/awesome-mcp-servers --- ## File: config/README.md # ROMA-DSPy Configuration System This directory contains the YAML-based configuration system for ROMA-DSPy, built with OmegaConf for configuration operations and Pydantic for validation. ## Quick Start ```python from roma_dspy.config import load_config # Load with defaults config = load_config() # Load with profile config = load_config(profile="lightweight") # Load with overrides config = load_config(overrides=["agents.executor.llm.temperature=0.5"]) ``` ## Configuration Structure ### Base Configuration (`defaults/config.yaml`) Default settings that override Pydantic defaults: - Project metadata - Agent configuration overrides - Runtime settings - Resilience parameters ### Profiles (`profiles/*.yaml`) Delta configurations that overlay specific use cases: - **lightweight.yaml**: Reduced resource usage, lower token limits - **tool_enabled.yaml**: Prepared for future tool implementation ## Configuration Schema ### LLM Configuration ```yaml agents: executor: llm: model: "gpt-4o-mini" temperature: 0.7 max_tokens: 2000 timeout: 30 api_key: ${oc.env:OPENAI_API_KEY} # Environment variable ``` ### Agent Configuration ```yaml agents: executor: prediction_strategy: "chain_of_thought" tools: [] enabled: true agent_config: # Agent business logic parameters max_subtasks: 10 strategy_config: {} # Prediction strategy algorithm parameters ``` ### Runtime Configuration ```yaml runtime: max_concurrency: 5 timeout: 30 verbose: ${oc.env:ROMA_VERBOSE,false} cache_dir: ".cache/dspy" ``` ### Resilience Configuration ```yaml resilience: max_retries: 3 retry_delay: 1.0 circuit_breaker_threshold: 5 circuit_breaker_timeout: 60 ``` ## Configuration Resolution Order Later sources override earlier ones: 1. **Pydantic defaults** (in schema classes) 2. **Base YAML** (`defaults/config.yaml`) 3. **Profile YAML** (`profiles/{profile}.yaml`) 4. **Override strings** (`["key=value"]`) 5. **Environment variables** (`ROMA_*`) ## Environment Variables ### Naming Convention - Prefix: `ROMA_` - Nested keys: double underscore `__` - Example: `ROMA_AGENTS__EXECUTOR__LLM__TEMPERATURE=0.5` ### Common Variables ```bash # API Keys export OPENAI_API_KEY="your-key" export FIREWORKS_API_KEY="your-key" # Runtime settings export ROMA_VERBOSE=true export ROMA_MAX_RETRIES=5 export ROMA_CACHE_DIR="/custom/cache" # Agent settings export ROMA_AGENTS__EXECUTOR__LLM__TEMPERATURE=0.3 ``` ## Profile Usage ### Creating Custom Profiles Create `profiles/my_profile.yaml`: ```yaml # My custom profile agents: executor: llm: temperature: 0.1 agent_config: max_iterations: 20 runtime: max_concurrency: 10 ``` ### Using Profiles ```python config = load_config(profile="my_profile") ``` ## Advanced Features ### OmegaConf Interpolation ```yaml # Variable interpolation base_timeout: 30 runtime: timeout: ${base_timeout} # Environment variable with default cache_dir: ${oc.env:ROMA_CACHE_DIR,.cache/dspy} ``` ### Configuration Caching The ConfigManager automatically caches loaded configurations for performance: ```python manager = ConfigManager() config1 = manager.load_config() # Loads from file config2 = manager.load_config() # Uses cache manager.clear_cache() # Clears cache ``` ### Validation The system provides two-stage validation: 1. **OmegaConf**: YAML structure and type checking 2. **Pydantic**: Business logic validation Example validations: - Temperature must be between 0.0 and 2.0 - max_tokens must be between 1 and 100,000 - Tool-strategy compatibility checking - Timeout consistency validation ## Module Integration ### BaseModule Integration ```python from roma_dspy.config import load_config from roma_dspy.core.modules import Executor # Load configuration config = load_config(profile="lightweight") # Create module with config executor = Executor( signature=MySignature, config=config.agents.executor ) ``` ### RecursiveSolver Integration ```python from roma_dspy.core.engine.solve import RecursiveSolver # Create solver with config solver = RecursiveSolver(config=config) result = solver.solve("Complex task") ``` ## Configuration Files - `defaults/config.yaml` - Base configuration overrides - `profiles/lightweight.yaml` - Minimal resource usage - `profiles/tool_enabled.yaml` - Tool-ready configuration ## Best Practices 1. **Use profiles** for different deployment environments 2. **Environment variables** for secrets and environment-specific settings 3. **Override strings** for quick testing and experimentation 4. **Base config** for organization-wide defaults 5. **Separate agent_config and strategy_config** for proper parameter isolation ## Troubleshooting ### Common Issues - **OmegaConf type errors**: Check YAML syntax and avoid Pydantic Field objects - **Validation errors**: Review Pydantic validators and constraints - **Missing profiles**: Ensure profile files exist in `profiles/` directory - **Environment variables**: Use correct naming convention with `ROMA_` prefix ### Debugging ```python # Enable verbose logging config = load_config(overrides=["runtime.verbose=true"]) # Check resolved configuration print(OmegaConf.to_yaml(config)) # Validate specific sections from roma_dspy.config.schemas import LLMConfig llm_config = LLMConfig(**config.agents.executor.llm) ``` --- ## File: docs/CONFIGURATION.md # ROMA-DSPy Configuration Guide Complete reference for configuring ROMA-DSPy agents, profiles, toolkits, and runtime settings. ## Table of Contents - [Overview](#overview) - [Configuration System](#configuration-system) - [Profiles](#profiles) - [Agents Configuration](#agents-configuration) - [Task-Aware Agent Mapping](#task-aware-agent-mapping) - [Toolkit Configuration](#toolkit-configuration) - [LLM Configuration](#llm-configuration) - [Runtime Settings](#runtime-settings) - [Storage Configuration](#storage-configuration) - [Observability (MLflow)](#observability-mlflow) - [Resilience Settings](#resilience-settings) - [Logging Configuration](#logging-configuration) - [Environment Variables](#environment-variables) - [Custom Prompts and Demos](#custom-prompts-and-demos) - [Configuration Examples](#configuration-examples) - [Best Practices](#best-practices) --- ## Overview ROMA-DSPy uses a **layered configuration system** combining: - **OmegaConf**: Flexible YAML configuration with interpolation - **Pydantic**: Type-safe validation and defaults - **Profiles**: Pre-configured setups for different use cases - **Environment Variables**: Runtime overrides ### Key Features - **Hierarchical Merging**: Combine defaults, profiles, and overrides - **Type Validation**: Catch configuration errors early - **Environment Interpolation**: `${oc.env:API_KEY}` for secrets - **Profile System**: Pre-configured agents for different domains - **Task-Aware Mapping**: Different executors for different task types --- ## Configuration System ### Resolution Order Configuration is loaded and merged in this order: 1. **Pydantic Defaults** - Base defaults from schema classes 2. **YAML Config** - Explicit configuration file 3. **Profile** - Profile overlay (if specified) 4. **CLI/Runtime Overrides** - Command-line arguments 5. **Environment Variables** - `ROMA__*` variables 6. **Validation** - Final validation via Pydantic Later layers override earlier ones. ### Using Configuration #### Via CLI ```bash # Use a profile uv run python -m roma_dspy.cli solve "task" --profile crypto_agent # Use custom config file uv run python -m roma_dspy.cli solve "task" --config config/examples/basic/minimal.yaml # With overrides uv run python -m roma_dspy.cli solve "task" \ --profile general \ --override agents.executor.llm.temperature=0.5 ``` #### Via Docker (Just) ```bash # Use profile just solve "task" crypto_agent # With all parameters just solve "task" general 2 true json # Parameters: [profile] [max_depth] [verbose] [output] ``` #### Via API ```bash curl -X POST http://localhost:8000/api/v1/executions \ -H "Content-Type: application/json" \ -d '{ "goal": "Your task", "config_profile": "general", "max_depth": 2 }' ``` #### Programmatically ```python from roma_dspy.config.manager import ConfigManager # Load profile config_mgr = ConfigManager() config = config_mgr.load_config(profile="general") # Load custom config config = config_mgr.load_config( config_path="config/custom.yaml", overrides=["runtime.max_depth=2"] ) # With environment prefix config = config_mgr.load_config( profile="crypto_agent", env_prefix="ROMA_" ) ``` --- ## Profiles Profiles are pre-configured agent setups for different use cases. Located in `config/profiles/`. ### Available Profiles | Profile | Purpose | Use Cases | Models | |---------|---------|-----------|--------| | **general** | General-purpose agent | Web research, code execution, file ops, calculations | Gemini Flash + Claude Sonnet 4.5 | | **crypto_agent** | Cryptocurrency analysis | Price tracking, DeFi analysis, on-chain data | Task-aware (Gemini Flash / Claude Sonnet 4.5) | ### Profile Structure ```yaml # config/profiles/general.yaml agents: atomizer: llm: model: openrouter/google/gemini-2.5-flash temperature: 0.0 max_tokens: 8000 signature_instructions: "prompt_optimization.seed_prompts.atomizer_seed:ATOMIZER_PROMPT" demos: "prompt_optimization.seed_prompts.atomizer_seed:ATOMIZER_DEMOS" executor: llm: model: openrouter/anthropic/claude-sonnet-4.5 temperature: 0.2 max_tokens: 32000 prediction_strategy: react toolkits: - class_name: E2BToolkit enabled: true - class_name: FileToolkit enabled: true runtime: max_depth: 6 enable_logging: true ``` ### Creating Custom Profiles Create `config/profiles/my_profile.yaml`: ```yaml agents: executor: llm: model: openrouter/anthropic/claude-sonnet-4.5 temperature: 0.3 max_tokens: 16000 prediction_strategy: react toolkits: - class_name: MCPToolkit enabled: true toolkit_config: server_name: my_server server_type: http url: https://my-mcp-server.com runtime: max_depth: 2 # Recommended: 1-2 for most tasks timeout: 120 enable_logging: true ``` Use it: ```bash just solve "task" my_profile ``` --- ## Agents Configuration ROMA-DSPy has 5 core agent modules. Each can be configured independently. ### Agent Types | Agent | Role | Default Strategy | Toolkits | |-------|------|-----------------|----------| | **Atomizer** | Classifies tasks as atomic or decomposable | chain_of_thought | None | | **Planner** | Breaks complex tasks into subtasks | chain_of_thought | None | | **Executor** | Executes atomic tasks | react | All toolkits | | **Aggregator** | Synthesizes subtask results | chain_of_thought | None | | **Verifier** | Validates outputs | chain_of_thought | None | ### Agent Configuration Schema ```yaml agents: executor: # Agent type: atomizer, planner, executor, aggregator, verifier # LLM configuration llm: model: openrouter/anthropic/claude-sonnet-4.5 temperature: 0.2 max_tokens: 32000 timeout: 30 num_retries: 3 cache: true # Prediction strategy (chain_of_thought or react) prediction_strategy: react # Custom prompts (optional) signature_instructions: "module.path:VARIABLE_NAME" demos: "module.path:DEMOS_LIST" # Agent-specific settings agent_config: max_executions: 10 # Max iterations for executor # max_subtasks: 12 # Max subtasks for planner # Strategy-specific settings strategy_config: # ReAct-specific settings # Toolkits (executor only) toolkits: - class_name: E2BToolkit enabled: true toolkit_config: timeout: 600 ``` ### Per-Agent Defaults Each agent has sensible defaults. Override only what you need: ```yaml # Minimal executor override agents: executor: llm: temperature: 0.3 # Override just temperature # All other settings use defaults ``` ### Agent-Specific Settings #### Atomizer ```yaml atomizer: agent_config: confidence_threshold: 0.8 # Threshold for atomic classification ``` #### Planner ```yaml planner: agent_config: max_subtasks: 12 # Maximum subtasks to generate ``` #### Executor ```yaml executor: agent_config: max_executions: 10 # Maximum ReAct iterations ``` #### Aggregator ```yaml aggregator: agent_config: synthesis_strategy: hierarchical # How to aggregate results ``` #### Verifier ```yaml verifier: agent_config: verification_depth: moderate # Verification thoroughness ``` --- ## Task-Aware Agent Mapping **Advanced Feature**: Use different executor configurations for different task types. ### Task Types ROMA-DSPy classifies tasks into 5 types: | Task Type | Description | Example Tasks | |-----------|-------------|---------------| | **RETRIEVE** | Data fetching, web search | "price of bitcoin", "find documentation" | | **CODE_INTERPRET** | Code execution, analysis | "run this script", "analyze CSV data" | | **THINK** | Deep reasoning, analysis | "compare approaches", "analyze sentiment" | | **WRITE** | Content creation | "write report", "create documentation" | | **IMAGE_GENERATION** | Image creation | "generate diagram", "create visualization" | ### Mapping Configuration ```yaml # Default agents (used for atomizer, planner, aggregator, verifier) agents: executor: llm: model: openrouter/anthropic/claude-sonnet-4.5 prediction_strategy: react toolkits: - class_name: FileToolkit enabled: true # Task-specific executor configurations agent_mapping: executors: # RETRIEVE: Fast model + web search RETRIEVE: llm: model: openrouter/google/gemini-2.5-flash # Fast & cheap temperature: 0.0 max_tokens: 16000 prediction_strategy: react agent_config: max_executions: 6 toolkits: - class_name: MCPToolkit enabled: true toolkit_config: server_name: exa server_type: http url: https://mcp.exa.ai/mcp # CODE_INTERPRET: Powerful model + code execution CODE_INTERPRET: llm: model: openrouter/anthropic/claude-sonnet-4.5 # Powerful temperature: 0.1 max_tokens: 32000 agent_config: max_executions: 15 toolkits: - class_name: E2BToolkit enabled: true - class_name: FileToolkit enabled: true ``` ### Benefits - **Cost Optimization**: Use cheap models for simple tasks - **Quality Optimization**: Use powerful models for complex tasks - **Right Tools**: Each task type gets appropriate toolkits - **Performance**: Faster execution with task-specific configs ### Example: crypto_agent Profile The `crypto_agent` profile uses task-aware mapping: - **RETRIEVE**: Gemini Flash (fast, cheap) + CoinGecko/Binance - **CODE_INTERPRET**: Claude Sonnet 4.5 (powerful) + E2B + crypto data - **THINK**: Claude Sonnet 4.5 + all toolkits - **WRITE**: Claude Sonnet 4.5 (creative) + FileToolkit + research --- ## Toolkit Configuration Toolkits provide tools (functions) that agents can use. Configured per-agent. ### Available Toolkits #### Native Toolkits ROMA-DSPy includes these built-in toolkits: | Toolkit | Purpose | API Key Required | Config Options | |---------|---------|-----------------|----------------| | **FileToolkit** | File I/O operations | ❌ No | `enable_delete`, `max_file_size` | | **CalculatorToolkit** | Math operations | ❌ No | None | | **E2BToolkit** | Code execution sandbox | βœ… Yes | `timeout`, `auto_reinitialize` | | **SerperToolkit** | Web search | βœ… Yes | `num_results`, `search_type` | | **BinanceToolkit** | Crypto market data | ❌ No | `default_market`, `enable_analysis` | | **CoinGeckoToolkit** | Crypto prices | ❌ No | `use_pro_api` | | **DefiLlamaToolkit** | DeFi protocol data | ❌ No | `enable_pro_features` | | **ArkhamToolkit** | Blockchain analytics | ❌ No | `enable_analysis` | #### MCP Toolkit The **MCPToolkit** is special - it can connect to **any** MCP (Model Context Protocol) server, giving you access to thousands of potential tools. **Two Types of MCP Servers:** 1. **HTTP MCP Servers** (Remote) - Public or private HTTP endpoints - No installation required - Examples: CoinGecko MCP, Exa MCP, or any custom HTTP MCP server 2. **Stdio MCP Servers** (Local) - Run as local subprocesses - Typically npm packages or custom scripts - Examples: Filesystem, GitHub, SQLite, or any npm MCP server **Finding MCP Servers:** - **Awesome MCP Servers**: https://github.com/wong2/awesome-mcp-servers (hundreds of servers) - **MCP Documentation**: https://modelcontextprotocol.io/ - **Build your own**: Any server implementing MCP protocol ### Basic Toolkit Configuration ```yaml agents: executor: toolkits: # Simple toolkit with no config - class_name: CalculatorToolkit enabled: true # Toolkit with basic config - class_name: FileToolkit enabled: true toolkit_config: enable_delete: false max_file_size: 10485760 # 10MB ``` ### E2B Toolkit Configuration ```yaml - class_name: E2BToolkit enabled: true toolkit_config: timeout: 600 # Execution timeout (seconds) max_lifetime_hours: 23.5 # Sandbox lifetime auto_reinitialize: true # Auto-restart on failure ``` **Environment Variables**: ```bash E2B_API_KEY=your_e2b_api_key E2B_TEMPLATE_ID=roma-dspy-sandbox # Custom template STORAGE_BASE_PATH=/opt/sentient # Shared storage ``` ### MCP Toolkit Configuration #### HTTP MCP Server (Public) Connect to any public HTTP MCP server: ```yaml - class_name: MCPToolkit enabled: true toolkit_config: server_name: coingecko_mcp server_type: http url: "https://mcp.api.coingecko.com/sse" use_storage: true # Store large results to Parquet storage_threshold_kb: 10 # Store if > 10KB ``` #### HTTP MCP Server (With Authentication) Connect to any authenticated HTTP MCP server: ```yaml - class_name: MCPToolkit enabled: true toolkit_config: server_name: exa server_type: http url: https://mcp.exa.ai/mcp headers: Authorization: Bearer ${oc.env:EXA_API_KEY} # Add any custom headers your MCP server needs transport_type: streamable use_storage: false tool_timeout: 60 ``` #### Stdio MCP Server (Local) Connect to any stdio MCP server (npm package or custom script): ```yaml - class_name: MCPToolkit enabled: true toolkit_config: server_name: filesystem server_type: stdio command: npx # or python, node, etc. args: - "-y" - "@modelcontextprotocol/server-filesystem" - "/Users/yourname/Documents" # Server-specific arguments env: # Optional environment variables for the server CUSTOM_VAR: value use_storage: false ``` **Common Stdio Examples:** ```yaml # GitHub MCP Server - class_name: MCPToolkit toolkit_config: server_name: github server_type: stdio command: npx args: - "-y" - "@modelcontextprotocol/server-github" env: GITHUB_PERSONAL_ACCESS_TOKEN: ${oc.env:GITHUB_PERSONAL_ACCESS_TOKEN} # SQLite MCP Server - class_name: MCPToolkit toolkit_config: server_name: sqlite server_type: stdio command: npx args: - "-y" - "@modelcontextprotocol/server-sqlite" - "/path/to/database.db" # Custom Python MCP Server - class_name: MCPToolkit toolkit_config: server_name: my_custom_server server_type: stdio command: python args: - "/path/to/my_mcp_server.py" ``` **Prerequisites for Stdio Servers**: - npm packages: `npm install ` - Custom scripts: Ensure executable and implements MCP protocol ### Crypto Toolkit Configuration #### Binance ```yaml - class_name: BinanceToolkit enabled: true include_tools: # Optional: limit to specific tools - get_current_price - get_ticker_stats - get_klines toolkit_config: enable_analysis: true default_market: spot # spot or futures ``` #### DefiLlama ```yaml - class_name: DefiLlamaToolkit enabled: true include_tools: - get_protocols - get_protocol_tvl - get_chains toolkit_config: enable_analysis: true enable_pro_features: true ``` ### Tool Filtering Limit which tools from a toolkit are available: ```yaml - class_name: MCPToolkit enabled: true include_tools: # Only these tools - get_simple_price - get_coins_markets - get_search toolkit_config: server_name: coingecko_mcp server_type: http url: "https://mcp.api.coingecko.com/sse" ``` --- ## LLM Configuration Configure language models for each agent. ### Basic LLM Config ```yaml agents: executor: llm: model: openrouter/anthropic/claude-sonnet-4.5 temperature: 0.2 max_tokens: 32000 timeout: 30 num_retries: 3 cache: true ``` ### LLM Parameters | Parameter | Description | Range | Default | |-----------|-------------|-------|---------| | **model** | Model identifier | Provider-specific | `gpt-4o-mini` | | **temperature** | Randomness (0=deterministic, 2=creative) | 0.0 - 2.0 | 0.7 | | **max_tokens** | Maximum output tokens | 1 - 200000 | 2000 | | **timeout** | Request timeout (seconds) | > 0 | 30 | | **num_retries** | Retry attempts on failure | 0 - 10 | 3 | | **cache** | Enable DSPy caching | true/false | true | | **adapter_type** | DSPy adapter type | `json` or `chat` | `json` | | **use_native_function_calling** | Enable native tool calling | true/false | `true` | ### DSPy Adapter Configuration ROMA-DSPy uses DSPy adapters to format inputs/outputs for LLMs. Two adapter types are available: **JSONAdapter** (default, recommended): - Uses structured JSON for inputs/outputs - Better performance for Claude and Gemini models - Cleaner prompt formatting **ChatAdapter**: - Uses chat message format - Better performance for some OpenAI models - More conversational style **Native Function Calling** (enabled by default): - Leverages LLM provider's native tool calling APIs (OpenAI, Anthropic, etc.) - Automatic fallback to text-based parsing for unsupported models - No reliability difference, cleaner provider integration Both parameters have sensible defaults and are **optional** in configuration: ```yaml agents: executor: llm: model: openrouter/anthropic/claude-sonnet-4.5 temperature: 0.2 max_tokens: 16000 # Defaults: adapter_type=json, use_native_function_calling=true # Uncomment to override: # adapter_type: chat # use_native_function_calling: false ``` ### Model Naming #### OpenRouter (Recommended) Single API key for all models: ```yaml model: openrouter/anthropic/claude-sonnet-4.5 model: openrouter/google/gemini-2.5-flash model: openrouter/openai/gpt-4o ``` **Environment**: `OPENROUTER_API_KEY=your_key` #### Direct Providers ```yaml # Anthropic model: claude-sonnet-4.5 # Requires: ANTHROPIC_API_KEY # OpenAI model: gpt-4o # Requires: OPENAI_API_KEY # Google model: gemini-2.5-flash # Requires: GOOGLE_API_KEY ``` ### Temperature Guidelines | Temperature | Use Case | Example | |-------------|----------|---------| | **0.0** | Deterministic, factual | Data retrieval, classification | | **0.1-0.2** | Slight creativity | Code generation, analysis | | **0.3-0.5** | Balanced | General tasks, reasoning | | **0.6-1.0** | Creative | Content writing, brainstorming | | **1.0+** | Very creative | Poetry, experimental | ### Token Limits Recommended `max_tokens` by agent: | Agent | Recommended | Rationale | |-------|-------------|-----------| | **Atomizer** | 1000-8000 | Simple classification | | **Planner** | 4000-32000 | Complex task breakdowns | | **Executor** | 16000-32000 | Detailed execution | | **Aggregator** | 5000-32000 | Result synthesis | | **Verifier** | 3000-16000 | Validation checks | ### Provider-Specific Parameters (`extra_body`) Pass provider-specific features via the `extra_body` parameter. This is particularly useful for OpenRouter's advanced features like web search, model routing, and provider preferences. **Security Note**: Never include sensitive keys (api_key, secret, token) in `extra_body`. Use the `api_key` field instead. #### OpenRouter Web Search Enable real-time web search for up-to-date information: ```yaml agents: executor: llm: model: openrouter/google/gemini-2.5-flash temperature: 0.0 extra_body: plugins: - id: web engine: exa # Options: "exa", "native", or omit for auto max_results: 3 ``` **Alternative**: Use the `:online` suffix for quick setup: ```yaml model: openrouter/anthropic/claude-sonnet-4.5:online ``` #### OpenRouter Native Search with Context Size For OpenRouter's native search engine with customizable context: ```yaml extra_body: plugins: - id: web engine: native web_search_options: search_context_size: high # Options: "low", "medium", "high" ``` #### Model Fallback Array Automatic failover to alternative models: ```yaml extra_body: models: - anthropic/claude-sonnet-4.5 - openai/gpt-4o - google/gemini-2.5-pro route: fallback # Options: "fallback", "lowest-cost", "lowest-latency" ``` #### Provider Preferences Control which providers to use: ```yaml extra_body: provider: order: - Anthropic - OpenAI data_collection: deny # Privacy control: "allow" or "deny" ``` #### Full OpenRouter Web Search Example ```yaml agents: executor: llm: model: openrouter/anthropic/claude-sonnet-4.5 temperature: 0.2 max_tokens: 16000 extra_body: # Enable web search with custom settings plugins: - id: web engine: exa max_results: 5 search_prompt: "Relevant information:" # Fallback models for reliability models: - anthropic/claude-sonnet-4.5 - openai/gpt-4o route: fallback ``` **Documentation**: See [OpenRouter Web Search Docs](https://openrouter.ai/docs/features/web-search) for all available options. **Cost Warning**: Web search plugins may significantly increase API costs per request. --- ## Runtime Settings Control execution behavior, timeouts, and logging. ### Runtime Configuration ```yaml runtime: max_depth: 6 # Recursion depth (recommended: 1-2) max_concurrency: 5 # Parallel task limit timeout: 120 # Global timeout (seconds) verbose: true # Detailed output enable_logging: true # Log to file log_level: INFO # DEBUG, INFO, WARNING, ERROR # Cache configuration cache: enabled: true enable_disk_cache: true enable_memory_cache: true disk_cache_dir: .cache/dspy disk_size_limit_bytes: 30000000000 # 30GB memory_max_entries: 1000000 ``` ### Runtime Parameters | Parameter | Description | Range | Default | Recommended | |-----------|-------------|-------|---------|-------------| | **max_depth** | Maximum task decomposition depth | 1-20 | 5 | **1-2** | | **max_concurrency** | Parallel subtasks | 1-50 | 5 | 5-10 | | **timeout** | Global execution timeout (sec) | 1-300 | 30 | 120-300 | | **verbose** | Print detailed output | bool | false | true (dev) | | **enable_logging** | File logging | bool | false | true | | **log_level** | Logging verbosity | DEBUG-CRITICAL | INFO | INFO | ### Max Depth Guidelines **IMPORTANT**: Lower max_depth = faster, cheaper, more reliable execution. | max_depth | Use Case | Trade-offs | |-----------|----------|-----------| | **1** | Simple, atomic tasks | Fast, cheap, limited decomposition | | **2** | Most production use cases | **Recommended** - good balance | | **3-4** | Complex multi-step tasks | Slower, more expensive | | **5+** | Highly complex hierarchical tasks | Very slow, expensive, may fail | **Best Practice**: Start with `max_depth=1`, increase only if needed. --- ## Storage Configuration Configure persistent storage for execution data and tool results. ### Storage Config ```yaml storage: base_path: ${oc.env:STORAGE_BASE_PATH,/opt/sentient} max_file_size: 104857600 # 100MB # PostgreSQL (execution tracking) postgres: enabled: ${oc.env:POSTGRES_ENABLED,true} connection_url: ${oc.env:DATABASE_URL,postgresql+asyncpg://localhost/roma_dspy} pool_size: 10 max_overflow: 20 ``` ### Storage Backends #### Local Filesystem ```bash # .env STORAGE_BASE_PATH=/opt/sentient ``` #### S3 via goofys ```bash # .env STORAGE_BASE_PATH=/opt/sentient ROMA_S3_BUCKET=my-bucket AWS_REGION=us-east-1 AWS_ACCESS_KEY_ID=your_key AWS_SECRET_ACCESS_KEY=your_secret ``` #### PostgreSQL ```bash # .env POSTGRES_ENABLED=true DATABASE_URL=postgresql+asyncpg://user:pass@host:5432/roma_dspy ``` Or via docker-compose (automatic): ```bash just docker-up # Starts postgres automatically ``` ### Tool Result Storage MCP and native toolkits can store large results to Parquet: ```yaml toolkits: - class_name: MCPToolkit toolkit_config: use_storage: true storage_threshold_kb: 10 # Store results > 10KB ``` **Benefits**: - Reduces context size - Enables large dataset handling - Automatic compression - Queryable via DuckDB --- ## Observability (MLflow) Track execution metrics, traces, and model performance with MLflow. ### MLflow Configuration ```yaml observability: mlflow: enabled: ${oc.env:MLFLOW_ENABLED,false} tracking_uri: ${oc.env:MLFLOW_TRACKING_URI,http://mlflow:5000} experiment_name: ROMA-General-Agent log_traces: true # Log full execution traces log_compiles: true # Log DSPy compilations log_evals: true # Log evaluations ``` ### Environment Variables ```bash # .env MLFLOW_ENABLED=true MLFLOW_TRACKING_URI=http://mlflow:5000 MLFLOW_EXPERIMENT=ROMA-DSPy ``` ### Using MLflow #### Start MLflow Server ```bash # Via Docker Compose (recommended) just docker-up-full # Includes MLflow # Access UI open http://localhost:5000 ``` #### Track Executions ```bash # Run task with MLflow enabled MLFLOW_ENABLED=true just solve "analyze bitcoin price" # View traces in MLflow UI open http://localhost:5000 ``` ### What MLflow Tracks - **Execution metrics**: Duration, depth, token usage - **LLM calls**: Model, parameters, latency - **Tool usage**: Tool calls, results, errors - **Traces**: Full execution tree with spans - **Parameters**: All config values - **Artifacts**: Outputs, logs, checkpoints --- ## Resilience Settings Automatic error handling, retries, and recovery. ### Resilience Configuration ```yaml resilience: # Retry configuration retry: enabled: true max_attempts: 5 strategy: exponential_backoff base_delay: 2.0 # Initial delay (seconds) max_delay: 60.0 # Maximum delay # Circuit breaker circuit_breaker: enabled: true failure_threshold: 5 # Failures before opening recovery_timeout: 120.0 # Seconds before retry half_open_max_calls: 3 # Test calls when recovering # Checkpointing checkpoint: enabled: true storage_path: ${oc.env:ROMA_CHECKPOINT_PATH,.checkpoints} max_checkpoints: 20 max_age_hours: 48.0 compress_checkpoints: true verify_integrity: true ``` ### Retry Strategies | Strategy | Behavior | Use Case | |----------|----------|----------| | **exponential_backoff** | Delay doubles each retry | Most cases (default) | | **fixed_delay** | Same delay each retry | Predictable timing | | **random_backoff** | Random jitter | Avoid thundering herd | ### Circuit Breaker States - **Closed**: Normal operation - **Open**: Failing, reject new requests - **Half-Open**: Testing recovery ### Checkpoint Recovery Automatic recovery from failures: ```python from roma_dspy.core.engine.solve import solve # Execution will checkpoint automatically result = solve("complex task", max_depth=3) # If interrupted, resume from checkpoint result = solve("complex task", resume_from_checkpoint=True) ``` --- ## Logging Configuration Structured logging with loguru. ### Logging Config ```yaml logging: level: ${oc.env:LOG_LEVEL,INFO} log_dir: ${oc.env:LOG_DIR,logs} # null = console only console_format: detailed # minimal, default, detailed file_format: json # default, detailed, json colorize: true serialize: true # JSON serialization rotation: 500 MB # File rotation size retention: 90 days # Keep logs for compression: zip # Compress rotated logs backtrace: true # Full tracebacks diagnose: false # Variable values (disable in prod) enqueue: true # Thread-safe ``` ### Log Levels | Level | Use Case | |-------|----------| | **DEBUG** | Development, detailed tracing | | **INFO** | Production, important events | | **WARNING** | Potential issues | | **ERROR** | Errors, exceptions | | **CRITICAL** | Fatal errors | ### Environment Variables ```bash # .env LOG_LEVEL=INFO LOG_DIR=logs # or null for console only LOG_CONSOLE_FORMAT=detailed LOG_FILE_FORMAT=json ``` ### Log Formats #### Console Formats - **minimal**: Level + message - **default**: Time, level, module, message (colored) - **detailed**: Full context with execution_id, line numbers #### File Formats - **default**: Standard text format - **detailed**: Includes process/thread info - **json**: Machine-parseable structured logs --- ## Environment Variables Environment variables override configuration values. ### LLM Provider Keys ```bash # OpenRouter (recommended - single key for all models) OPENROUTER_API_KEY=your_key # Or individual providers OPENAI_API_KEY=your_key ANTHROPIC_API_KEY=your_key GOOGLE_API_KEY=your_key ``` ### Toolkit Keys ```bash # Code Execution E2B_API_KEY=your_key E2B_TEMPLATE_ID=roma-dspy-sandbox # Web Search EXA_API_KEY=your_key SERPER_API_KEY=your_key # Crypto APIs (all optional, public endpoints work without keys) COINGECKO_API_KEY=your_key # For Pro API DEFILLAMA_API_KEY=your_key # For Pro features ARKHAM_API_KEY=your_key BINANCE_API_KEY=your_key BINANCE_API_SECRET=your_secret # GitHub MCP GITHUB_PERSONAL_ACCESS_TOKEN=your_token # Any MCP server may require its own environment variables ``` ### Storage & Database ```bash # Storage STORAGE_BASE_PATH=/opt/sentient ROMA_S3_BUCKET=my-bucket AWS_ACCESS_KEY_ID=your_key AWS_SECRET_ACCESS_KEY=your_secret # PostgreSQL POSTGRES_ENABLED=true DATABASE_URL=postgresql+asyncpg://user:pass@host:5432/roma_dspy ``` ### MLflow ```bash MLFLOW_ENABLED=true MLFLOW_TRACKING_URI=http://mlflow:5000 MLFLOW_EXPERIMENT=ROMA-DSPy ``` ### Runtime Overrides Use `ROMA__` prefix with double underscores: ```bash # Override agents.executor.llm.temperature ROMA__AGENTS__EXECUTOR__LLM__TEMPERATURE=0.5 # Override runtime.max_depth ROMA__RUNTIME__MAX_DEPTH=2 ``` **Format**: `ROMA______=value` ### Docker Compose In docker-compose, set in `.env`: ```bash # .env OPENROUTER_API_KEY=your_key E2B_API_KEY=your_key POSTGRES_ENABLED=true MLFLOW_ENABLED=true ``` Then: ```bash just docker-up # Automatically loads .env ``` --- ## Custom Prompts and Demos Enhance agent performance with optimized prompts and few-shot examples. ### Signature Instructions Custom instructions guide the agent's behavior. #### Three Formats **1. Inline String** ```yaml agents: executor: signature_instructions: "Execute the task step-by-step with clear reasoning." ``` **2. Jinja Template File** ```yaml agents: executor: signature_instructions: "config/prompts/executor.jinja" ``` **3. Python Module Variable** ```yaml agents: executor: signature_instructions: "prompt_optimization.seed_prompts.executor_seed:EXECUTOR_PROMPT" ``` #### Seed Prompts ROMA-DSPy includes optimized seed prompts in `prompt_optimization/seed_prompts/`: | Module | Variable | Purpose | |--------|----------|---------| | `atomizer_seed` | `ATOMIZER_PROMPT` | Task classification | | `planner_seed` | `PLANNER_PROMPT` | Task decomposition | | `executor_seed` | `EXECUTOR_PROMPT` | General execution | | `executor_retrieve_seed` | `EXECUTOR_RETRIEVE_PROMPT` | Data retrieval | | `executor_code_seed` | `EXECUTOR_CODE_PROMPT` | Code execution | | `executor_think_seed` | `EXECUTOR_THINK_PROMPT` | Deep reasoning | | `executor_write_seed` | `EXECUTOR_WRITE_PROMPT` | Content creation | | `aggregator_seed` | `AGGREGATOR_PROMPT` | Result synthesis | | `verifier_seed` | `VERIFIER_PROMPT` | Output validation | ### Demos (Few-Shot Examples) Provide examples to guide the agent. #### Format ```yaml agents: executor: demos: "prompt_optimization.seed_prompts.executor_seed:EXECUTOR_DEMOS" ``` #### Creating Custom Demos ```python # my_prompts/executor_demos.py import dspy EXECUTOR_DEMOS = [ dspy.Example( goal="Calculate 15% of 2500", answer="375" ).with_inputs("goal"), dspy.Example( goal="What is the capital of France?", answer="Paris" ).with_inputs("goal") ] ``` Use in config: ```yaml agents: executor: demos: "my_prompts.executor_demos:EXECUTOR_DEMOS" ``` ### Custom Signature Override the default DSPy signature: ```yaml agents: executor: signature: "goal -> answer: str, confidence: float" ``` **Note**: Most users don't need this. Use `signature_instructions` instead. --- ## Configuration Examples ROMA-DSPy includes comprehensive configuration examples in `config/examples/`. These are real, working configurations that demonstrate different concepts and patterns. ### Available Examples #### Basic Examples (`config/examples/basic/`) | Example | Description | Use It | |---------|-------------|--------| | **minimal.yaml** | Simplest possible configuration | `just solve "task" -c config/examples/basic/minimal.yaml` | | **multi_toolkit.yaml** | Multiple toolkits (E2B + File + Calculator) | `just solve "task" -c config/examples/basic/multi_toolkit.yaml` | **Demonstrates**: Fundamentals, toolkit usage, basic configuration patterns #### MCP Examples (`config/examples/mcp/`) | Example | Description | Use It | |---------|-------------|--------| | **http_public_server.yaml** | Public HTTP MCP server (CoinGecko) - no setup | `just solve "task" -c config/examples/mcp/http_public_server.yaml` | | **stdio_local_server.yaml** | Local stdio MCP server via npx | `just solve "task" -c config/examples/mcp/stdio_local_server.yaml` | | **multi_server.yaml** | Multiple MCP servers (HTTP + stdio) | `just solve "task" -c config/examples/mcp/multi_server.yaml` | | **common_servers.yaml** | Common MCP servers (GitHub, Filesystem, SQLite) | `just solve "task" -c config/examples/mcp/common_servers.yaml` | **Demonstrates**: HTTP vs stdio MCP servers, multi-server orchestration, storage configuration #### Crypto Example (`config/examples/crypto/`) | Example | Description | Use It | |---------|-------------|--------| | **crypto_agent.yaml** | Real-world crypto analysis agent | `just solve "task" -c config/examples/crypto/crypto_agent.yaml` | **Demonstrates**: Domain-specific agent, combining MCP + native toolkits, multi-source data aggregation #### Advanced Examples (`config/examples/advanced/`) | Example | Description | Use It | |---------|-------------|--------| | **task_aware_mapping.yaml** | Task-specific executor configurations | `just solve "task" -c config/examples/advanced/task_aware_mapping.yaml` | | **custom_prompts.yaml** | Custom prompts and demos | `just solve "task" -c config/examples/advanced/custom_prompts.yaml` | **Demonstrates**: Task-aware agent mapping, cost/quality optimization per task type, loading custom signature instructions and demos ### Quick Reference ```bash # Use a profile (recommended) just solve "task" general # Use an example configuration just solve "task" -c config/examples/basic/minimal.yaml # With CLI parameters uv run python -m roma_dspy.cli solve "task" \ --config config/examples/basic/minimal.yaml \ --override runtime.max_depth=1 ``` ### Example Structure Each example includes: - **Inline comments** explaining each section - **Setup requirements** (API keys, npm packages) - **Usage examples** showing how to run - **Key learnings** about what the example demonstrates ### Detailed Guide See **[config/examples/README.md](../config/examples/README.md)** for: - Complete examples directory structure - Detailed descriptions of each example - Setup instructions - Common issues and solutions - Tips for success --- ## Best Practices ### 1. Start Simple ```yaml # Start with minimal config agents: executor: llm: model: openrouter/anthropic/claude-sonnet-4.5 prediction_strategy: react toolkits: - class_name: FileToolkit enabled: true runtime: max_depth: 1 # Start with 1, increase if needed ``` Add complexity only when needed. ### 2. Use Profiles Don't create configs from scratch. Start with a profile: ```bash # Use existing profile just solve "task" general # Or copy and customize cp config/profiles/general.yaml config/profiles/my_profile.yaml # Edit my_profile.yaml just solve "task" my_profile ``` ### 3. Environment Variables for Secrets Never hardcode API keys in config files: ```yaml # ❌ Bad headers: Authorization: Bearer sk-1234567890 # βœ… Good headers: Authorization: Bearer ${oc.env:EXA_API_KEY} ``` ### 4. Optimize max_depth **Most tasks need max_depth=1 or 2**: - Start with 1 - Increase to 2 if task needs decomposition - Only use 3+ for complex hierarchical tasks - Higher depth = slower + more expensive ### 5. Task-Aware Mapping for Cost Optimization Use cheap models for simple tasks: ```yaml agent_mapping: executors: RETRIEVE: llm: model: openrouter/google/gemini-2.5-flash # $0.075/1M tokens CODE_INTERPRET: llm: model: openrouter/anthropic/claude-sonnet-4.5 # $3/1M tokens ``` ### 6. Enable Caching ```yaml agents: executor: llm: cache: true # Enable DSPy caching runtime: cache: enabled: true enable_disk_cache: true ``` Saves money and improves speed. ### 7. Use Storage for Large Data ```yaml toolkits: - class_name: MCPToolkit toolkit_config: use_storage: true storage_threshold_kb: 10 # Store results > 10KB ``` Prevents context overflow. ### 8. Monitor with MLflow ```yaml observability: mlflow: enabled: true log_traces: true ``` Track performance, costs, and errors. ### 9. Configure Resilience ```yaml resilience: retry: enabled: true max_attempts: 5 circuit_breaker: enabled: true checkpoint: enabled: true ``` Automatic recovery from failures. ### 10. Validate Configuration ```python from roma_dspy.config.manager import ConfigManager # Validate before using try: config = ConfigManager().load_config(profile="my_profile") print("βœ… Configuration valid") except ValueError as e: print(f"❌ Invalid configuration: {e}") ``` --- ## Next Steps - **[QUICKSTART.md](QUICKSTART.md)** - Get started quickly - **[TOOLKITS.md](TOOLKITS.md)** - Complete toolkit reference - **[MCP.md](MCP.md)** - MCP integration guide - **[API.md](API.md)** - REST API reference - **[DEPLOYMENT.md](DEPLOYMENT.md)** - Production deployment - **Examples**: `config/examples/` - Real-world examples --- **Questions?** Check the examples in `config/examples/` or create an issue on GitHub. --- ## File: docs/DEPLOYMENT.md # ROMA-DSPy Deployment Guide Production deployment guide for ROMA-DSPy. ## Table of Contents - [Overview](#overview) - [Quick Deploy](#quick-deploy) - [Architecture](#architecture) - [Environment Configuration](#environment-configuration) - [Docker Deployment](#docker-deployment) - [Production Checklist](#production-checklist) - [Monitoring & Observability](#monitoring--observability) - [Scaling](#scaling) - [Security](#security) - [Troubleshooting](#troubleshooting) --- ## Overview ROMA-DSPy is designed for production deployment with Docker Compose, providing: **Infrastructure:** - PostgreSQL (execution/checkpoint persistence) - MinIO (S3-compatible object storage for MLflow artifacts) - MLflow (optional, experiment tracking) - ROMA API (FastAPI server) **Features:** - Health checks and auto-restart - Volume persistence - Network isolation - Multi-stage Docker builds - Non-root containers --- ## Quick Deploy ### Prerequisites - Docker 24.0+ and Docker Compose 2.0+ - 4GB RAM minimum (8GB recommended) - 20GB disk space - Ports available: 8000 (API), 5432 (Postgres), 9000/9001 (MinIO), 5000 (MLflow) ### 1. Clone Repository ```bash git clone https://github.com/your-org/ROMA-DSPy.git cd ROMA-DSPy ``` ### 2. Configure Environment ```bash # Copy environment template cp .env.example .env # Edit .env and set required values nano .env ``` **Minimum Required:** ```bash # LLM Provider OPENROUTER_API_KEY=your_key_here # Database POSTGRES_PASSWORD=secure_password_here # MinIO/S3 MINIO_ROOT_PASSWORD=secure_password_here ``` ### 3. Start Services ```bash # Basic deployment (API + PostgreSQL + MinIO) just docker-up # Full deployment (includes MLflow observability) just docker-up-full # Verify health curl http://localhost:8000/health ``` ### 4. Test ```bash # Via API curl -X POST http://localhost:8000/api/v1/executions \ -H "Content-Type: application/json" \ -d '{"goal": "What is 2+2?", "max_depth": 1}' | jq # Via CLI (inside container) docker exec -it roma-dspy-api roma-dspy solve "What is 2+2?" ``` --- ## Architecture ### Docker Compose Stack ``` β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ Docker Network β”‚ β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ ROMA API │───▢│ PostgreSQL β”‚ β”‚ β”‚ β”‚ Port: 8000 β”‚ β”‚ Port: 5432 β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ └───────────▢│ MinIO β”‚ β”‚ β”‚ β”‚ Port: 9000 β”‚ β”‚ β”‚ β”‚ Console:9001 β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β”‚ β”‚ β”‚ β”‚ β”Œβ”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β” β”‚ β”‚ β”‚ MLflow β”‚ (optional) β”‚ β”‚ β”‚ Port: 5000 β”‚ β”‚ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β”‚ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ### Service Descriptions **roma-api:** - FastAPI application server - Handles execution management - Exposes REST API - Health check: `http://localhost:8000/health` **postgres:** - PostgreSQL 16 Alpine - Stores execution metadata, checkpoints, traces - Persistent volume: `postgres_data` - Health check: `pg_isready` **minio:** - S3-compatible object storage - Stores MLflow artifacts - Persistent volume: `minio_data` - UI: `http://localhost:9001` **mlflow** (optional): - Experiment tracking server - Requires `--profile observability` - UI: `http://localhost:5000` --- ## Environment Configuration ### Required Variables ```bash # LLM Provider (at least one required) OPENROUTER_API_KEY=your_key_here # Recommended # OR OPENAI_API_KEY=your_key_here ANTHROPIC_API_KEY=your_key_here GOOGLE_API_KEY=your_key_here # Database POSTGRES_DB=roma_dspy # Database name POSTGRES_USER=postgres # Database user POSTGRES_PASSWORD=CHANGE_ME_IN_PROD # Database password POSTGRES_PORT=5432 # Host port # MinIO/S3 MINIO_ROOT_USER=minioadmin # MinIO access key MINIO_ROOT_PASSWORD=CHANGE_ME_IN_PROD # MinIO secret key MINIO_PORT=9000 # S3 API port MINIO_CONSOLE_PORT=9001 # Console port # API API_PORT=8000 # API port POSTGRES_ENABLED=true # Enable PostgreSQL storage ``` ### Optional Variables ```bash # Toolkit API Keys E2B_API_KEY=your_key_here # Code execution EXA_API_KEY=your_key_here # Web search via MCP SERPER_API_KEY=your_key_here # Web search toolkit GITHUB_PERSONAL_ACCESS_TOKEN=your_token # GitHub MCP server COINGECKO_API_KEY=your_key_here # CoinGecko Pro API # MLflow (for observability profile) MLFLOW_PORT=5000 MLFLOW_TRACKING_URI=http://mlflow:5000 # Storage STORAGE_BASE_PATH=/opt/sentient # Base path for file storage # Security ALLOWED_ORIGINS=https://yourdomain.com # CORS origins (comma-separated) # Logging LOG_LEVEL=INFO # DEBUG, INFO, WARNING, ERROR LOG_DIR=/app/logs # Log directory ``` --- ## Docker Deployment ### Build and Start **Build from scratch:** ```bash # Clean build just docker-build-clean # Start services just docker-up ``` **Start with existing images:** ```bash # Basic (API + Postgres + MinIO) just docker-up # Full (includes MLflow) just docker-up-full ``` ### Verify Deployment ```bash # Check all services running just docker-ps # Check health curl http://localhost:8000/health # View logs just docker-logs # View specific service logs just docker-logs-service roma-api just docker-logs-service postgres just docker-logs-service mlflow ``` ### Stop Services ```bash # Stop (preserves data) just docker-down # Stop and remove volumes (data loss!) just docker-down-clean ``` --- ## Production Checklist ### Security - [ ] Change default passwords in `.env`: - `POSTGRES_PASSWORD` - `MINIO_ROOT_PASSWORD` - [ ] Set `ALLOWED_ORIGINS` for CORS (don't use `*` in production) - [ ] Use HTTPS reverse proxy (nginx, Caddy, Traefik) - [ ] Enable authentication on API (add middleware) - [ ] Restrict network access (firewall rules) - [ ] Use secrets management (Docker secrets, Vault, AWS Secrets Manager) - [ ] Regularly update base images: ```bash docker-compose pull docker-compose up -d ``` ### Reliability - [ ] Configure automatic backups: ```bash # PostgreSQL backup docker exec roma-dspy-postgres pg_dump -U postgres roma_dspy > backup.sql ``` - [ ] Set resource limits in `docker-compose.yaml`: ```yaml roma-api: deploy: resources: limits: cpus: '2.0' memory: 4G reservations: cpus: '1.0' memory: 2G ``` - [ ] Monitor disk usage: ```bash docker system df docker volume ls ``` - [ ] Configure log rotation: ```yaml roma-api: logging: driver: "json-file" options: max-size: "10m" max-file: "3" ``` ### Observability - [ ] Enable MLflow tracking: ```bash just docker-up-full ``` - [ ] Set up health check monitoring (Prometheus, Datadog, etc.) - [ ] Configure log aggregation (ELK, Grafana Loki, Datadog) - [ ] Monitor resource usage (CPU, memory, disk) - [ ] Set up alerts for service failures --- ## Monitoring & Observability ### Health Checks **API Health:** ```bash curl http://localhost:8000/health ``` **Response:** ```json { "status": "healthy", "version": "0.1.0", "uptime_seconds": 3600.5, "active_executions": 2, "storage_connected": true, "cache_size": 5, "timestamp": "2024-10-21T12:00:00.000Z" } ``` **PostgreSQL Health:** ```bash docker exec roma-dspy-postgres pg_isready -U postgres ``` **MinIO Health:** ```bash curl http://localhost:9000/minio/health/live ``` ### MLflow UI Access at http://localhost:5000 **Features:** - Experiment tracking - Run comparison - Model registry - Artifact storage **View Executions:** 1. Navigate to http://localhost:5000 2. Filter by experiment name 3. Click execution ID to view details ### Metrics Endpoints ```bash # Execution metrics curl http://localhost:8000/api/v1/executions//metrics | jq # Cost summary curl http://localhost:8000/api/v1/executions//costs | jq # Toolkit metrics curl http://localhost:8000/api/v1/executions//toolkit-metrics | jq # LM traces curl http://localhost:8000/api/v1/executions//lm-traces | jq ``` ### Log Aggregation **View logs:** ```bash # All services just docker-logs # Specific service just docker-logs-service roma-api # Follow logs docker-compose logs -f roma-api ``` **Export logs:** ```bash docker-compose logs roma-api > roma-api.log ``` --- ## Scaling ### Horizontal Scaling (Multiple API Instances) **docker-compose.yaml:** ```yaml roma-api: # ... existing config ... deploy: replicas: 3 # Run 3 instances # Load balancer labels: - "traefik.enable=true" - "traefik.http.routers.roma.rule=Host(`api.yourdomain.com`)" ``` **With nginx load balancer:** ```nginx upstream roma_api { server localhost:8001; server localhost:8002; server localhost:8003; } server { listen 80; server_name api.yourdomain.com; location / { proxy_pass http://roma_api; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; } } ``` ### Vertical Scaling (Resource Limits) **docker-compose.yaml:** ```yaml roma-api: deploy: resources: limits: cpus: '4.0' memory: 8G reservations: cpus: '2.0' memory: 4G postgres: deploy: resources: limits: cpus: '2.0' memory: 4G reservations: cpus: '1.0' memory: 2G ``` ### Database Scaling **PostgreSQL optimization:** ```bash # Connect to database docker exec -it roma-dspy-postgres psql -U postgres -d roma_dspy # Analyze tables ANALYZE executions; ANALYZE checkpoints; ANALYZE lm_traces; # Vacuum VACUUM ANALYZE; # Check indexes \di ``` **Connection pooling** (add PgBouncer if needed): ```yaml pgbouncer: image: pgbouncer/pgbouncer:latest environment: DATABASE_URL: postgres://postgres:password@postgres:5432/roma_dspy POOL_MODE: transaction MAX_CLIENT_CONN: 1000 DEFAULT_POOL_SIZE: 20 ``` --- ## Security ### HTTPS/TLS **Option 1: nginx reverse proxy** ```nginx server { listen 443 ssl; server_name api.yourdomain.com; ssl_certificate /etc/nginx/ssl/cert.pem; ssl_certificate_key /etc/nginx/ssl/key.pem; location / { proxy_pass http://localhost:8000; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; } } ``` **Option 2: Caddy (auto HTTPS)** ```caddy api.yourdomain.com { reverse_proxy localhost:8000 } ``` ### Authentication **Add API key middleware** (example): ```python # src/roma_dspy/api/middleware.py from fastapi import HTTPException, Request from starlette.middleware.base import BaseHTTPMiddleware class APIKeyMiddleware(BaseHTTPMiddleware): async def dispatch(self, request: Request, call_next): api_key = request.headers.get("X-API-Key") if not api_key or api_key != os.getenv("API_KEY"): raise HTTPException(status_code=401, detail="Invalid API key") return await call_next(request) ``` **Use:** ```python # src/roma_dspy/api/main.py app.add_middleware(APIKeyMiddleware) ``` ### Network Security **Firewall rules:** ```bash # Allow only specific IPs sudo ufw allow from 203.0.113.0/24 to any port 8000 # Or use Docker network policies ``` **Internal network only:** ```yaml # docker-compose.yaml services: postgres: ports: [] # Don't expose to host networks: - roma-network networks: roma-network: internal: true # No external access ``` ### Secrets Management **Using Docker secrets:** ```yaml secrets: postgres_password: file: ./secrets/postgres_password.txt openrouter_api_key: file: ./secrets/openrouter_api_key.txt services: roma-api: secrets: - postgres_password - openrouter_api_key environment: POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password OPENROUTER_API_KEY_FILE: /run/secrets/openrouter_api_key ``` --- ## Troubleshooting ### Service Won't Start **Check logs:** ```bash just docker-logs-service roma-api just docker-logs-service postgres ``` **Common issues:** 1. **Port already in use:** ```bash # Find process using port lsof -i :8000 # Kill process or change API_PORT in .env ``` 2. **Database connection failed:** ```bash # Check postgres health docker exec roma-dspy-postgres pg_isready -U postgres # Verify DATABASE_URL in .env ``` 3. **Out of memory:** ```bash # Check Docker resources docker stats # Increase Docker memory limit # Docker Desktop β†’ Settings β†’ Resources β†’ Memory ``` ### Data Persistence Issues **Check volumes:** ```bash # List volumes docker volume ls | grep roma # Inspect volume docker volume inspect roma-dspy_postgres_data # Backup volume docker run --rm -v roma-dspy_postgres_data:/data -v $(pwd):/backup alpine tar czf /backup/postgres_backup.tar.gz /data ``` ### Performance Issues **Monitor resources:** ```bash # Real-time stats docker stats # Check disk usage docker system df # Prune unused data docker system prune -a ``` **Database slow queries:** ```bash # Enable query logging docker exec -it roma-dspy-postgres psql -U postgres -d roma_dspy # Show slow queries SELECT query, calls, total_time, mean_time FROM pg_stat_statements ORDER BY total_time DESC LIMIT 10; ``` ### MLflow Not Accessible **Check service:** ```bash # Ensure started with observability profile just docker-up-full # Check logs docker-compose logs mlflow # Verify port curl http://localhost:5000 ``` --- ## Additional Resources - **Quick Start**: [QUICKSTART.md](QUICKSTART.md) - **Configuration**: [CONFIGURATION.md](CONFIGURATION.md) - **API Reference**: http://localhost:8000/docs - **Docker Compose Docs**: https://docs.docker.com/compose/ - **FastAPI Deployment**: https://fastapi.tiangolo.com/deployment/ --- **Production Ready!** πŸš€ For questions or issues, check logs first (`just docker-logs`), then consult the documentation or open an issue. --- ## File: docs/E2B_SETUP.md # E2B Integration Setup Guide ## Overview This guide explains how to set up E2B code execution sandboxes with S3 storage integration for ROMA-DSPy. The setup enables agents to execute code in isolated sandboxes while maintaining access to shared S3 storage via goofys. ## Architecture ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Prerequisites 1. **E2B Account** - Sign up at [e2b.dev](https://e2b.dev) - Get API key from dashboard 2. **AWS S3 Bucket** - Create S3 bucket for storage - Configure AWS credentials with S3 access 3. **E2B CLI** (for template creation) ```bash npm install @e2b/cli # or yarn global add @e2b/cli ``` ## Step 1: Environment Configuration ### 1.1 Configure Environment Variables Copy `.env.example` to `.env` and fill in: ```bash # Storage Configuration STORAGE_BASE_PATH=/opt/sentient ROMA_S3_BUCKET=your-s3-bucket-name AWS_REGION=us-east-1 # AWS Credentials AWS_ACCESS_KEY_ID=your_aws_access_key AWS_SECRET_ACCESS_KEY=your_aws_secret_key # E2B Configuration E2B_API_KEY=your_e2b_api_key E2B_TEMPLATE_ID=roma-dspy-sandbox ``` ### 1.2 Verify Configuration ```python from roma_dspy.config.manager import ConfigManager config = ConfigManager().load_config() print(f"Storage path: {config.storage.base_path}") print(f"S3 bucket: {os.getenv('ROMA_S3_BUCKET')}") ``` ## Step 2: Create Custom E2B Template E2B templates define the sandbox environment. We need a custom template that includes our S3 mounting script. ### 2.1 Initialize Template ```bash cd /Users/barannama/ROMA-DSPy # Initialize E2B template e2b template init roma-dspy-sandbox ``` This creates `.e2b/` directory with template configuration. ### 2.2 Add Startup Script Copy our startup script to the template: ```bash # Copy start-up.sh to template cp docker/e2b/start-up.sh .e2b/start-up.sh # Verify script is executable chmod +x .e2b/start-up.sh ``` The `start-up.sh` script: - Installs goofys in the sandbox - Mounts S3 bucket to `$STORAGE_BASE_PATH` - Validates write access - Sets up Python dependencies ### 2.3 Build and Publish Template ```bash # Build template (creates Docker image) e2b template build # This will output a template ID like: # βœ“ Template built successfully # Template ID: roma-dspy-sandbox-abc123 # Copy the template ID to your .env file echo "E2B_TEMPLATE_ID=" >> .env ``` ### 2.4 Verify Template ```bash # List your templates e2b template list # You should see your template with ID matching .env ``` ## Step 3: Local Storage Setup ### 3.1 Run Local Setup Script ```bash # Make script executable chmod +x scripts/setup_local.sh # Run setup (mounts S3 locally via goofys) ./scripts/setup_local.sh ``` This script: 1. Installs goofys (if not present) 2. Mounts S3 bucket to local path 3. Creates symlink if needed 4. Validates write access ### 3.2 Verify Local Storage ```bash # Check mount mount | grep goofys # Verify directory structure ls -la /opt/sentient/executions/ # Test write access echo "test" > /opt/sentient/executions/test.txt cat /opt/sentient/executions/test.txt rm /opt/sentient/executions/test.txt ``` ## Step 4: Test E2B Integration ### 4.1 Basic Test ```python from roma_dspy.tools.core import E2BToolkit from roma_dspy.config.manager import ConfigManager from roma_dspy.core.storage import FileStorage # Load config config = ConfigManager().load_config() # Create storage storage = FileStorage( config=config.storage, execution_id="test_e2b_001" ) # Write file on host test_data = b"Hello from host!" await storage.put("test.txt", test_data) print(f"Wrote to: {storage.get_artifacts_path('test.txt')}") # Create E2B toolkit e2b = E2BToolkit() # Read file in E2B sandbox code = f""" import os file_path = '{storage.get_artifacts_path('test.txt')}' print(f'Reading from: {{file_path}}') with open(file_path, 'r') as f: content = f.read() print(f'Content: {{content}}') """ result = e2b.run_python_code(code) print(result) ``` Expected output: ```json { "success": true, "results": [], "stdout": [ "Reading from: /opt/sentient/executions/test_e2b_001/artifacts/test.txt", "Content: Hello from host!" ], "stderr": [], "error": null, "sandbox_id": "..." } ``` ### 4.2 Run Integration Tests ```bash # Run E2B integration tests pytest tests/integration/test_e2b_integration.py -v # Run E2E storage test pytest tests/integration/test_e2e_storage.py -v ``` ## Step 5: Production Deployment ### 5.1 Environment-Specific Configuration **Development (.env.development)**: ```bash STORAGE_BASE_PATH=/opt/sentient/dev ROMA_S3_BUCKET=roma-storage-dev E2B_TEMPLATE_ID=roma-dspy-sandbox-dev ``` **Production (.env.production)**: ```bash STORAGE_BASE_PATH=/opt/sentient/prod ROMA_S3_BUCKET=roma-storage-prod E2B_TEMPLATE_ID=roma-dspy-sandbox-prod ``` ### 5.2 Update Template When updating startup script: ```bash # Modify docker/e2b/start-up.sh # Then update template: cp docker/e2b/start-up.sh .e2b/start-up.sh e2b template build ``` ## Troubleshooting ### Issue: Sandbox Can't Access S3 **Symptoms**: E2B code execution fails with file not found errors **Solution**: 1. Verify env vars are passed to sandbox: ```python e2b = E2BToolkit() status = e2b.get_sandbox_status() print(status) # Check env vars ``` 2. Check start-up.sh logs in E2B dashboard 3. Verify AWS credentials are valid: ```bash aws s3 ls s3://$ROMA_S3_BUCKET ``` ### Issue: goofys Mount Fails **Symptoms**: Local setup script fails or mount point empty **Solution**: 1. Check AWS credentials: ```bash aws sts get-caller-identity ``` 2. Verify S3 bucket exists: ```bash aws s3 ls | grep $ROMA_S3_BUCKET ``` 3. Check goofys installation: ```bash which goofys goofys --version ``` ### Issue: Path Mismatch Between Host and E2B **Symptoms**: Files written on host not visible in E2B **Solution**: 1. Verify `STORAGE_BASE_PATH` is same in: - `.env` file - Local setup script output - E2B start-up.sh 2. Check both systems point to same S3 bucket: ```bash # Local mount | grep goofys # E2B (run in sandbox) mount | grep goofys ``` ### Issue: Template Not Found **Symptoms**: E2B toolkit fails with template not found **Solution**: 1. Verify template exists: ```bash e2b template list ``` 2. Check `E2B_TEMPLATE_ID` in .env matches template ID 3. Rebuild template if needed: ```bash e2b template build ``` ## Advanced Configuration ### Custom Goofys Options Edit `start-up.sh` to customize goofys mount: ```bash # In start-up.sh, modify goofys command: goofys \ --region "${AWS_REGION}" \ --stat-cache-ttl 5m \ # Longer cache --type-cache-ttl 5m \ --max-idle-handles 1000 \ # More file handles --dir-mode 0755 \ --file-mode 0644 \ "${S3_BUCKET}" \ "${STORAGE_BASE_PATH}" ``` ### Multiple E2B Templates Create environment-specific templates: ```bash # Development template e2b template init roma-dspy-dev cp docker/e2b/start-up.sh .e2b/start-up.sh e2b template build # Production template (with optimizations) e2b template init roma-dspy-prod # Edit .e2b/start-up.sh with production settings e2b template build ``` ### Monitoring Storage Usage ```python from roma_dspy.core.storage import FileStorage storage = FileStorage(config=config.storage, execution_id="exec_123") info = await storage.get_storage_info() print(f"Total size: {info['total_size_mb']} MB") print(f"File count: {info['file_count']}") ``` ## Best Practices 1. **Use Execution IDs**: Always scope storage to execution IDs for isolation 2. **Clean Up Temp Files**: Use `cleanup_temp_files()` after execution: ```python await storage.cleanup_execution_temp_files() ``` 3. **Monitor Costs**: Track S3 storage and E2B sandbox usage 4. **Template Versioning**: Version E2B templates in production: ```bash e2b template build --name roma-dspy-prod-v1.0.0 ``` 5. **Error Handling**: Always check E2B execution results: ```python result = e2b.run_python_code(code) result_data = json.loads(result) if not result_data["success"]: logger.error(f"E2B execution failed: {result_data['error']}") ``` ## References - [E2B Documentation](https://e2b.dev/docs) - [Goofys GitHub](https://github.com/kahing/goofys) - [AWS S3 Documentation](https://docs.aws.amazon.com/s3/) - [ROMA-DSPy Storage Architecture](/docs/STORAGE_ARCHITECTURE.md) --- ## File: docs/OBSERVABILITY.md # Observability and Monitoring ROMA-DSPy provides comprehensive observability through MLflow integration, enabling experiment tracking, metrics logging, and execution tracing. ## Overview The observability system captures: - **Execution traces** - Task decomposition and execution flow - **LLM metrics** - Token usage, costs, and latency for each LLM call - **Performance metrics** - Task duration, depth, and success rates - **Compilation artifacts** - Optimized prompts and few-shot examples ## MLflow Integration ### Configuration Enable MLflow tracking in your configuration: ```yaml # config/defaults/config.yaml observability: mlflow: enabled: true tracking_uri: "http://127.0.0.1:5000" # Local MLflow server experiment_name: "ROMA-DSPy" log_traces: true log_compiles: true log_evals: true log_traces_from_compile: false # Expensive, disabled by default ``` Or via environment variables: ```bash export MLFLOW_ENABLED=true export MLFLOW_TRACKING_URI=http://127.0.0.1:5000 export MLFLOW_EXPERIMENT=ROMA-DSPy ``` ### Starting MLflow Server ```bash # Start MLflow UI mlflow ui --port 5000 # Or with specific backend store mlflow ui --backend-store-uri sqlite:///mlflow.db --port 5000 ``` Access the UI at http://localhost:5000 ## What Gets Logged ### 1. Run-Level Metrics Each solver execution creates an MLflow run with: **Parameters:** - `task` - The original goal/task - `max_depth` - Maximum decomposition depth - `config_version` - Configuration version - `solver_type` - RecursiveSolver identifier **Metrics:** - `total_tasks` - Number of tasks created - `completed_tasks` - Successfully completed tasks - `failed_tasks` - Failed tasks - `total_cost` - Total LLM API cost (USD) - `total_tokens` - Total tokens consumed - `execution_duration` - Total execution time (seconds) - `max_depth_reached` - Actual maximum depth reached ### 2. LLM Traces For each Language Model call: **Logged Information:** - Module name (atomizer, planner, executor, etc.) - Model identifier (gpt-4, claude-3, etc.) - Token usage (prompt, completion, total) - Cost breakdown - Latency (milliseconds) - Input/output (if enabled) **Metrics per module:** - `{module}_calls` - Number of calls - `{module}_tokens` - Total tokens - `{module}_cost` - Total cost - `{module}_avg_latency` - Average latency ### 3. Compilation Artifacts When using DSPy optimization: - Compiled predictor signatures - Few-shot examples - Optimization metrics - Prompt templates ## Usage Examples ### Basic Usage ```python from roma_dspy.config.manager import ConfigManager from roma_dspy.core.engine.solve import RecursiveSolver # Load config with MLflow enabled config = ConfigManager(profile="high_quality").get_config() config.observability.mlflow.enabled = True # Create solver solver = RecursiveSolver(config=config) # Solve task - automatically logged to MLflow result = await solver.async_solve("Plan a weekend in Barcelona") ``` ### Custom Experiment Names ```python config.observability.mlflow.experiment_name = "Barcelona-Planning-v2" solver = RecursiveSolver(config=config) ``` ### Programmatic Access ```python from roma_dspy.observability.mlflow_manager import MLflowManager # Initialize mlflow_mgr = MLflowManager(config.observability.mlflow) await mlflow_mgr.initialize() # Start run run_id = await mlflow_mgr.start_run( run_name="custom-run", tags={"version": "1.0", "experiment_type": "production"} ) # Log metrics await mlflow_mgr.log_metric("custom_metric", 42.0) await mlflow_mgr.log_param("custom_param", "value") # End run await mlflow_mgr.end_run(status="FINISHED") ``` ## Querying MLflow Data ### Using MLflow UI 1. Navigate to http://localhost:5000 2. Select your experiment 3. Compare runs, view metrics, download artifacts ### Using MLflow API ```python import mlflow # Set tracking URI mlflow.set_tracking_uri("http://localhost:5000") # Search runs runs = mlflow.search_runs( experiment_names=["ROMA-DSPy"], filter_string="metrics.total_cost < 1.0", order_by=["metrics.execution_duration ASC"] ) # Get best run best_run = runs.sort_values("metrics.total_cost").iloc[0] print(f"Best run: {best_run.run_id}") print(f"Cost: ${best_run['metrics.total_cost']:.4f}") ``` ### Analyzing Costs ```python # Get all runs runs = mlflow.search_runs(experiment_names=["ROMA-DSPy"]) # Cost analysis total_cost = runs["metrics.total_cost"].sum() avg_cost = runs["metrics.total_cost"].mean() cost_by_depth = runs.groupby("params.max_depth")["metrics.total_cost"].mean() print(f"Total spent: ${total_cost:.2f}") print(f"Average per run: ${avg_cost:.4f}") print("\nCost by depth:") print(cost_by_depth) ``` ## Cost Tracking ### Token Costs ROMA-DSPy tracks costs for common LLM providers: - **OpenAI**: gpt-4, gpt-3.5-turbo, etc. - **Anthropic**: claude-3-opus, claude-3-sonnet, etc. - **Fireworks AI**: Various models - **OpenRouter**: Pass-through pricing Costs are calculated using: ```python cost = (prompt_tokens * prompt_price_per_1k / 1000) + (completion_tokens * completion_price_per_1k / 1000) ``` ### Cost Optimization Monitor these metrics to optimize costs: 1. **Tokens per task** - Identify verbose modules 2. **Failed task cost** - Wasted spend on failures 3. **Model selection** - Compare costs across models 4. **Depth vs cost** - Find optimal decomposition depth ## Performance Monitoring ### Key Metrics **Latency:** - Total execution time - Per-module latency - LLM call latency **Throughput:** - Tasks per minute - Subtasks per decomposition - Success rate **Resource Usage:** - Token consumption rate - API calls per task - Checkpoint frequency ### Alerts and Thresholds Set up alerts for: ```python # High cost runs if run.metrics.total_cost > 5.0: alert("High cost run detected") # Slow execution if run.metrics.execution_duration > 300: alert("Slow execution") # High failure rate failure_rate = run.metrics.failed_tasks / run.metrics.total_tasks if failure_rate > 0.2: alert("High failure rate") ``` ## Integration with Postgres When both MLflow and Postgres are enabled, you get dual observability: **MLflow**: Experiment tracking, visualization, comparison **Postgres**: Detailed execution traces, queryable history, audit trail ```python # Query both sources import mlflow from roma_dspy.core.storage.postgres_storage import PostgresStorage # MLflow - high-level metrics runs = mlflow.search_runs(experiment_names=["ROMA-DSPy"]) # Postgres - detailed traces storage = PostgresStorage(config.storage.postgres) await storage.initialize() for _, run in runs.iterrows(): execution_id = run["tags.execution_id"] costs = await storage.get_execution_costs(execution_id) print(f"Run {execution_id}: ${costs['total_cost']:.4f}") ``` ## Best Practices 1. **Use descriptive experiment names** - Organize by project/feature 2. **Tag runs appropriately** - Add version, environment, user tags 3. **Monitor costs regularly** - Set up cost alerts 4. **Archive old experiments** - Keep MLflow database manageable 5. **Disable expensive logging in production** - `log_traces_from_compile: false` 6. **Use remote tracking server** - For team collaboration 7. **Back up MLflow data** - Especially artifact stores ## Troubleshooting ### MLflow Connection Issues ```bash # Check server is running curl http://localhost:5000/health # Check environment variable echo $MLFLOW_TRACKING_URI # Test connection python -c "import mlflow; print(mlflow.get_tracking_uri())" ``` ### Missing Metrics ```python # Verify logging is enabled print(config.observability.mlflow.enabled) print(config.observability.mlflow.log_traces) # Check MLflow manager initialization print(solver.mlflow_manager._initialized) ``` ### High Storage Usage ```bash # Check artifact store size du -sh ~/.mlflow # Clean up old runs (use with caution) mlflow gc --backend-store-uri sqlite:///mlflow.db ``` ## Advanced Topics ### Custom Metrics ```python # Add custom metrics to MLflow async with solver.mlflow_manager.run_context(): await solver.mlflow_manager.log_metric("custom_score", score) await solver.mlflow_manager.log_param("algorithm", "custom") ``` ### Distributed Tracking For multi-machine setups: ```yaml observability: mlflow: tracking_uri: "http://mlflow-server.company.com:5000" # Use S3/GCS for artifacts artifact_location: "s3://my-bucket/mlflow-artifacts" ``` ### Integration with Other Tools MLflow integrates with: - **Prometheus** - For operational metrics - **Grafana** - For dashboards - **Databricks** - For managed MLflow - **Weights & Biases** - Via exporters ## Toolkit Metrics and Traceability ROMA-DSPy provides comprehensive tracking of toolkit lifecycle and tool invocation metrics, enabling deep visibility into tool usage patterns, performance, and reliability. ### Overview The toolkit metrics system automatically tracks: - **Toolkit Lifecycle** - Creation, caching, cleanup operations - **Tool Invocations** - Individual tool calls with timing and I/O metrics - **Performance** - Duration, success rates, error patterns - **Attribution** - Cost and usage per toolkit/tool ### Configuration Enable toolkit metrics tracking: ```yaml # config/defaults/config.yaml observability: toolkit_metrics: enabled: true # Enable/disable tracking track_lifecycle: true # Track toolkit operations track_invocations: true # Track tool calls sample_rate: 1.0 # Sample rate (0.0-1.0) persist_to_db: true # Save to PostgreSQL persist_to_mlflow: false # Save to MLflow batch_size: 100 # Batch size for persistence async_persist: true # Async persistence ``` Or via environment variables: ```bash export TOOLKIT_METRICS_ENABLED=true export TOOLKIT_TRACK_LIFECYCLE=true export TOOLKIT_TRACK_INVOCATIONS=true export TOOLKIT_SAMPLE_RATE=1.0 export TOOLKIT_PERSIST_DB=true ``` ### What Gets Tracked #### 1. Toolkit Lifecycle Events **Tracked Operations:** - `create` - Toolkit instantiation - `cache_hit` - Retrieved from cache - `cache_miss` - Cache lookup failed - `cleanup` - Toolkit disposal **Captured Data:** - Operation timestamp - Toolkit class name - Duration (milliseconds) - Success/failure status - Error details (if failed) - Custom metadata #### 2. Tool Invocation Events **Tracked for Each Call:** - Tool name and toolkit class - Invocation timestamp - Duration (milliseconds) - Input size (bytes) - Output size (bytes) - Success/failure status - Error details (if failed) - Custom metadata ### API Endpoints Query toolkit metrics via REST API: #### Get Aggregated Summary ```bash curl http://localhost:8000/executions/{execution_id}/toolkit-metrics ``` **Response:** ```json { "execution_id": "exec_123", "toolkit_lifecycle": { "total_created": 5, "cache_hit_rate": 0.75 }, "tool_invocations": { "total_calls": 50, "successful_calls": 48, "failed_calls": 2, "success_rate": 0.96, "avg_duration_ms": 125.5, "total_duration_ms": 6275.0 }, "by_toolkit": { "SerperToolkit": { "calls": 20, "successful": 20, "failed": 0, "avg_duration_ms": 150.0 } }, "by_tool": { "SerperToolkit.search_web": { "calls": 15, "successful": 15, "avg_duration_ms": 145.0 } } } ``` #### Get Raw Lifecycle Traces ```bash # All lifecycle traces curl http://localhost:8000/executions/{execution_id}/toolkit-traces # Filter by operation curl http://localhost:8000/executions/{execution_id}/toolkit-traces?operation=create # Filter by toolkit class curl http://localhost:8000/executions/{execution_id}/toolkit-traces?toolkit_class=SerperToolkit # Limit results curl http://localhost:8000/executions/{execution_id}/toolkit-traces?limit=100 ``` #### Get Raw Tool Invocations ```bash # All tool invocations curl http://localhost:8000/executions/{execution_id}/tool-invocations # Filter by toolkit curl http://localhost:8000/executions/{execution_id}/tool-invocations?toolkit_class=SerperToolkit # Filter by tool name curl http://localhost:8000/executions/{execution_id}/tool-invocations?tool_name=search_web # Combined filters curl http://localhost:8000/executions/{execution_id}/tool-invocations?toolkit_class=SerperToolkit&tool_name=search_web ``` ### Database Schema #### toolkit_traces Table ```sql CREATE TABLE toolkit_traces ( trace_id BIGSERIAL PRIMARY KEY, execution_id VARCHAR(64) NOT NULL, timestamp TIMESTAMP WITH TIME ZONE NOT NULL, operation VARCHAR(32) NOT NULL, toolkit_class VARCHAR(128), duration_ms FLOAT NOT NULL, success BOOLEAN NOT NULL, error TEXT, metadata JSONB NOT NULL DEFAULT '{}', FOREIGN KEY (execution_id) REFERENCES executions(execution_id) ON DELETE CASCADE ); -- Indexes for query performance CREATE INDEX idx_toolkit_traces_execution ON toolkit_traces (execution_id, timestamp); CREATE INDEX idx_toolkit_traces_operation ON toolkit_traces (operation); CREATE INDEX idx_toolkit_traces_toolkit_class ON toolkit_traces (toolkit_class); CREATE INDEX idx_toolkit_traces_success ON toolkit_traces (success); ``` #### tool_invocation_traces Table ```sql CREATE TABLE tool_invocation_traces ( trace_id BIGSERIAL PRIMARY KEY, execution_id VARCHAR(64) NOT NULL, toolkit_class VARCHAR(128) NOT NULL, tool_name VARCHAR(128) NOT NULL, invoked_at TIMESTAMP WITH TIME ZONE NOT NULL, duration_ms FLOAT NOT NULL, input_size_bytes INTEGER NOT NULL, output_size_bytes INTEGER NOT NULL, success BOOLEAN NOT NULL, error TEXT, metadata JSONB NOT NULL DEFAULT '{}', FOREIGN KEY (execution_id) REFERENCES executions(execution_id) ON DELETE CASCADE ); -- Indexes for query performance CREATE INDEX idx_tool_invocations_execution ON tool_invocation_traces (execution_id, invoked_at); CREATE INDEX idx_tool_invocations_toolkit ON tool_invocation_traces (toolkit_class); CREATE INDEX idx_tool_invocations_tool ON tool_invocation_traces (tool_name); CREATE INDEX idx_tool_invocations_toolkit_tool ON tool_invocation_traces (toolkit_class, tool_name); CREATE INDEX idx_tool_invocations_success ON tool_invocation_traces (success); ``` ### Database Migration Apply the migration to create toolkit metrics tables: ```bash # Navigate to project root cd /path/to/ROMA-DSPy # Run migration alembic upgrade head ``` Or manually: ```bash # Check current version alembic current # Upgrade to toolkit metrics migration alembic upgrade 004_toolkit_metrics # Rollback if needed alembic downgrade 003_add_dag_snapshot ``` ### Usage Examples #### Analyzing Toolkit Performance ```python from roma_dspy.core.storage.postgres_storage import PostgresStorage # Get toolkit metrics summary summary = await storage.get_toolkit_metrics_summary("exec_123") print(f"Total tool calls: {summary['tool_invocations']['total_calls']}") print(f"Success rate: {summary['tool_invocations']['success_rate']:.2%}") print(f"Average duration: {summary['tool_invocations']['avg_duration_ms']:.2f}ms") # Analyze by toolkit for toolkit, metrics in summary['by_toolkit'].items(): print(f"\n{toolkit}:") print(f" Calls: {metrics['calls']}") print(f" Success rate: {metrics['successful'] / metrics['calls']:.2%}") print(f" Avg duration: {metrics['avg_duration_ms']:.2f}ms") ``` #### Identifying Slow Tools ```python # Get all tool invocations invocations = await storage.get_tool_invocation_traces("exec_123") # Sort by duration slow_tools = sorted(invocations, key=lambda x: x.duration_ms, reverse=True)[:10] print("Top 10 slowest tool calls:") for inv in slow_tools: print(f"{inv.toolkit_class}.{inv.tool_name}: {inv.duration_ms:.2f}ms") ``` #### Tracking Failures ```python # Get failed tool invocations failed = await storage.get_tool_invocation_traces( execution_id="exec_123", limit=1000 ) failed = [inv for inv in failed if not inv.success] # Group by error type from collections import Counter error_types = Counter(inv.metadata.get('error_type', 'Unknown') for inv in failed) print("Failure breakdown:") for error_type, count in error_types.most_common(): print(f" {error_type}: {count}") ``` #### Cache Performance Analysis ```python # Get lifecycle traces traces = await storage.get_toolkit_traces("exec_123") # Calculate cache metrics cache_hits = sum(1 for t in traces if t.operation == "cache_hit") cache_misses = sum(1 for t in traces if t.operation == "cache_miss") total = cache_hits + cache_misses if total > 0: hit_rate = cache_hits / total print(f"Cache hit rate: {hit_rate:.2%}") print(f"Cache hits: {cache_hits}") print(f"Cache misses: {cache_misses}") ``` ### Monitoring and Alerting #### Key Metrics to Monitor 1. **Success Rate** - Alert if below 95% 2. **Average Duration** - Alert on significant increases 3. **Error Rate** - Alert on spikes 4. **Cache Hit Rate** - Alert if drops significantly #### Example Prometheus Queries ```promql # Success rate by toolkit sum(rate(tool_invocations_success_total[5m])) by (toolkit_class) / sum(rate(tool_invocations_total[5m])) by (toolkit_class) # P95 latency histogram_quantile(0.95, sum(rate(tool_duration_ms_bucket[5m])) by (le, tool_name)) # Error rate sum(rate(tool_invocations_failed_total[5m])) by (toolkit_class, error_type) ``` ### Performance Tuning #### Reduce Storage Overhead ```yaml # Sample only 10% of calls in high-volume environments observability: toolkit_metrics: sample_rate: 0.1 ``` #### Batch Persistence ```yaml # Increase batch size for better write performance observability: toolkit_metrics: batch_size: 500 async_persist: true ``` #### Disable Specific Tracking ```yaml # Track only lifecycle, skip invocations observability: toolkit_metrics: track_lifecycle: true track_invocations: false ``` ### Troubleshooting #### Metrics Not Appearing 1. **Check PostgreSQL is enabled:** ```yaml storage: postgres: enabled: true ``` 2. **Verify migration applied:** ```bash alembic current # Should show: 004_toolkit_metrics (head) ``` 3. **Check configuration:** ```python print(config.observability.toolkit_metrics.enabled) print(config.observability.toolkit_metrics.persist_to_db) ``` #### High Storage Usage ```sql -- Check table sizes SELECT schemaname, tablename, pg_size_pretty(pg_total_relation_size(schemaname||'.'||tablename)) AS size FROM pg_tables WHERE tablename LIKE '%trace%' ORDER BY pg_total_relation_size(schemaname||'.'||tablename) DESC; -- Clean old executions (careful!) DELETE FROM executions WHERE created_at < NOW() - INTERVAL '30 days'; ``` #### Missing Traces ```python # Check if context is properly set from roma_dspy.core.context import ExecutionContext ctx = ExecutionContext.get() if ctx: print(f"Execution ID: {ctx.execution_id}") print(f"Toolkit events: {len(ctx.toolkit_events)}") print(f"Tool invocations: {len(ctx.tool_invocations)}") else: print("No ExecutionContext found!") ``` ### Best Practices 1. **Enable in Development** - Use full tracking (sample_rate=1.0) 2. **Sample in Production** - Use lower sample rates for high-volume systems 3. **Monitor Key Metrics** - Set up alerts on success rates and latencies 4. **Regular Cleanup** - Archive or delete old execution data 5. **Index Management** - Monitor index size and query performance 6. **Correlate with LM Traces** - Combine with LM metrics for cost attribution ### Integration with MLflow ```python # Log toolkit metrics to MLflow from roma_dspy.core.observability import MLflowManager async with mlflow_manager.run_context(): summary = await storage.get_toolkit_metrics_summary(execution_id) # Log aggregate metrics await mlflow_manager.log_metric( "toolkit_success_rate", summary["tool_invocations"]["success_rate"] ) await mlflow_manager.log_metric( "avg_tool_duration_ms", summary["tool_invocations"]["avg_duration_ms"] ) # Log per-toolkit metrics for toolkit, metrics in summary["by_toolkit"].items(): await mlflow_manager.log_metric( f"{toolkit}_calls", metrics["calls"] ) ``` ## Further Reading - [MLflow Documentation](https://mlflow.org/docs/latest/index.html) - [MLflow Tracking Guide](https://mlflow.org/docs/latest/tracking.html) - [DSPy Observability](https://dspy-docs.vercel.app/) - [PostgreSQL Performance Tuning](https://www.postgresql.org/docs/current/performance-tips.html) --- ## File: docs/QUICKSTART.md # ROMA-DSPy Quick Start Get started in **under 30 seconds** with no infrastructure required! ## What is ROMA-DSPy? ROMA-DSPy is a framework for building production-ready AI agents using [DSPy](https://github.com/stanfordnlp/dspy). It provides: - **Hierarchical Task Decomposition** - Break complex tasks into manageable subtasks - **Modular Agent Architecture** - Atomizer, Planner, Executor, Aggregator, Verifier - **Extensive Toolkit System** - File ops, code execution, web search, crypto data, and more - **MCP Integration** - Connect to any Model Context Protocol server - **Optional Production Features** - REST API, PostgreSQL persistence, MLflow observability, Docker deployment ## Prerequisites ### Minimal Installation (Recommended) - **Python 3.12+** - **API key** from OpenRouter, OpenAI, Anthropic, or Fireworks ### Full Installation (Optional) - **Docker & Docker Compose** (for production features) - **Just** command runner (optional but recommended) --- ## Quick Start (3 paths) Choose your preferred setup method: ### Path A: Minimal Installation (Recommended - Start in 30 Seconds) **Best for**: Quick evaluation, development, testing - no infrastructure required **What you get:** - βœ… Core agent framework (all modules) - βœ… All DSPy prediction strategies - βœ… File storage (no database needed) - βœ… Built-in toolkits (Calculator, File ops) - βœ… Works with any LLM provider **What you DON'T need:** - ❌ No Docker - ❌ No PostgreSQL - ❌ No MLflow - ❌ No infrastructure setup **Install in 30 seconds:** ```bash # Install with uv (10-100x faster, recommended) uv pip install roma-dspy # Or with pip pip install roma-dspy # Set your API key export OPENROUTER_API_KEY="sk-or-v1-..." # Start solving tasks immediately python -c "from roma_dspy.core.engine.solve import solve; print(solve('What is 2+2?'))" ``` **Python usage:** ```python from roma_dspy.core.engine.solve import solve # Simple task result = solve("What is 25 * 47?") print(result) # More complex task result = solve("Analyze the pros and cons of electric vehicles") print(result) ``` **Installation time:** < 30 seconds **Package size:** ~15 core dependencies **Ready to use:** Immediately --- ### Path B: Full Installation with Docker (Production Features) **Best for**: Production deployment with persistence, observability, and REST API **Additional features:** - βœ… REST API server - βœ… PostgreSQL persistence - βœ… MLflow observability - βœ… S3 storage integration - βœ… E2B code execution sandbox - βœ… Interactive TUI visualization 1. **Clone and Configure** ```bash git clone https://github.com/your-org/ROMA-DSPy.git cd ROMA-DSPy # Copy environment template cp .env.example .env ``` 2. **Configure Environment** Edit `.env` and add your API keys: ```bash # Required OPENROUTER_API_KEY=your_key_here # Optional (for specific features) E2B_API_KEY=your_key_here EXA_API_KEY=your_key_here ``` 3. **Start Services** ```bash # Build and start all services just docker-up # Or with MLflow observability just docker-up-full # Check health curl http://localhost:8000/health ``` 4. **Run Your First Task** ```bash # Via Docker CLI just solve "What is the capital of France?" # Or via REST API curl -X POST http://localhost:8000/api/v1/executions \ -H "Content-Type: application/json" \ -d '{"goal": "What is the capital of France?"}' ``` **Services Running:** - API: http://localhost:8000 - PostgreSQL: localhost:5432 - MinIO: http://localhost:9001 - MLflow: http://localhost:5000 (with `--profile observability`) --- ### Path C: Crypto Agent (Domain-Specific Example) **Best for**: Cryptocurrency analysis use case 1. **Quick Setup** ```bash just docker-up ``` 2. **Run Crypto Analysis** ```bash # Get Bitcoin price just solve "What is the current price of Bitcoin?" crypto_agent # Complex analysis just solve "Compare Bitcoin and Ethereum prices, analyze 7-day trends" crypto_agent # DeFi analysis just solve "Show top 10 DeFi protocols by TVL" crypto_agent ``` **Crypto Agent Includes:** - CoinGecko (15,000+ cryptocurrencies) - Binance (spot/futures markets) - DefiLlama (DeFi protocol data) - Arkham (blockchain analytics) - Exa (web search) --- ## Installation Comparison | Feature | Minimal | Docker Full | |---------|---------|-------------| | **Install time** | < 30 seconds | 2-5 minutes | | **Prerequisites** | Python 3.12+ | Docker + Docker Compose | | **Infrastructure** | None required | PostgreSQL, MinIO, MLflow (auto-deployed) | | **Package size** | ~15 dependencies | All features | | **Use case** | Quick eval, dev, testing | Production deployment | | **Core framework** | βœ… | βœ… | | **DSPy strategies** | βœ… | βœ… | | **File storage** | βœ… | βœ… | | **Built-in toolkits** | βœ… | βœ… | | **REST API** | ❌ | βœ… | | **PostgreSQL persistence** | ❌ | βœ… | | **MLflow tracking** | ❌ | βœ… | | **S3 storage** | ❌ | βœ… | | **E2B sandbox** | ❌ | βœ… | | **TUI visualization** | ❌ | βœ… | **Key difference**: - **Minimal** = Just Python package (no Docker, no services) - **Docker** = Complete production stack (PostgreSQL, MLflow, API, all features via docker-compose) --- ## Adding Features to Minimal Install You can install Python dependencies for optional features: ```bash # Install dependencies for specific features uv pip install roma-dspy[api] # REST API dependencies uv pip install roma-dspy[persistence] # PostgreSQL client dependencies uv pip install roma-dspy[observability] # MLflow client dependencies uv pip install roma-dspy[e2b] # E2B code execution uv pip install roma-dspy[tui] # TUI visualization uv pip install roma-dspy[dev] # Development tools # Install all Python dependencies uv pip install roma-dspy[all] ``` **Important**: Installing extras only adds Python dependencies. Services like PostgreSQL, MLflow, and the API server require Docker or separate deployment. **For production use with all features, use Docker (Path B)**. --- ## Just Commands Cheat Sheet ### Basic Usage ```bash just # List all commands just solve "task" # Solve task with Docker just viz # Visualize execution DAG ``` ### Docker Management ```bash just docker-up # Start services just docker-up-full # Start with MLflow just docker-down # Stop services just docker-logs # View logs just docker-ps # Check status just docker-shell # Open shell in container ``` ### Development ```bash just install # Install dependencies just test # Run tests just lint # Check code quality just format # Format code just clean # Clean cache ``` ### List Available Profiles ```bash just list-profiles # Output: # - crypto_agent # - general ``` --- ## Verify Installation ### 1. Check Health ```bash curl http://localhost:8000/health ``` Expected response: ```json { "status": "healthy", "version": "1.0.0", "storage_connected": true, "active_executions": 0, "uptime_seconds": 123.45 } ``` ### 2. Test via CLI ```bash # Simple calculation just solve "Calculate 15% of 2500" # Get execution ID from output, then visualize just viz ``` ### 3. Test via API ```bash # Create execution (max_depth=1 or 2 recommended) curl -X POST http://localhost:8000/api/v1/executions \ -H "Content-Type: application/json" \ -d '{ "goal": "What are the prime numbers between 1 and 20?", "max_depth": 2 }' | jq # Poll status (use execution_id from response) curl http://localhost:8000/api/v1/executions//status | jq ``` --- ## Configuration Profiles ROMA-DSPy uses profiles to pre-configure agents for different use cases. ### Available Profiles | Profile | Purpose | Models | Toolkits | |---------|---------|--------|----------| | **general** | General-purpose tasks | Gemini Flash + Claude Sonnet | E2B, FileToolkit, CalculatorToolkit, Exa MCP | | **crypto_agent** | Cryptocurrency analysis | Multiple (task-aware) | CoinGecko, Binance, DefiLlama, Arkham, E2B | ### Using a Profile ```bash # Via CLI (defaults to 'general' if not specified) just solve "your task" just solve "crypto task" crypto_agent # Via API curl -X POST http://localhost:8000/api/v1/executions \ -H "Content-Type: application/json" \ -d '{ "goal": "Your task", "config_profile": "general" }' ``` ### Custom Profile Create `config/profiles/my_profile.yaml`: ```yaml agents: executor: llm: model: openai/gpt-4o temperature: 0.3 prediction_strategy: react toolkits: - class_name: FileToolkit enabled: true - class_name: CalculatorToolkit enabled: true runtime: max_depth: 2 # 1-2 recommended for most tasks ``` Use it: ```bash just solve "task" my_profile ``` See [CONFIGURATION.md](CONFIGURATION.md) for complete guide. --- ## Environment Variables ### Required ```bash # LLM Provider (choose one or use OpenRouter for all) OPENROUTER_API_KEY=xxx # Recommended (single key for all models) # OR individual providers: OPENAI_API_KEY=xxx ANTHROPIC_API_KEY=xxx GOOGLE_API_KEY=xxx ``` ### Optional Features ```bash # Code Execution (E2B) E2B_API_KEY=xxx # Web Search (Exa MCP) EXA_API_KEY=xxx # Web Search (Serper Toolkit) SERPER_API_KEY=xxx # Crypto APIs (all public, no keys needed) # CoinGecko, Binance, DefiLlama, Arkham work without keys ``` ### Storage & Database ```bash # PostgreSQL (auto-configured in Docker) DATABASE_URL=postgresql+asyncpg://postgres:postgres@localhost:5432/roma_dspy POSTGRES_ENABLED=true # S3 Storage (optional) STORAGE_BASE_PATH=/opt/sentient ROMA_S3_BUCKET=your-bucket AWS_ACCESS_KEY_ID=xxx AWS_SECRET_ACCESS_KEY=xxx ``` --- ## Common Tasks ### 1. Solve a Task ```bash # Simple (uses 'general' profile by default) just solve "What is 2+2?" # With specific profile just solve "Analyze Bitcoin" crypto_agent # With all options just solve "Complex task" crypto_agent 5 true json # Parameters: [profile] [max_depth] [verbose] [output_format] ``` ### 2. Check Execution ```bash # List all executions curl http://localhost:8000/api/v1/executions | jq # Get specific execution curl http://localhost:8000/api/v1/executions/ | jq # Get execution status curl http://localhost:8000/api/v1/executions//status | jq ``` ### 3. View Logs ```bash # All services just docker-logs # Specific service just docker-logs-service roma-api just docker-logs-service postgres just docker-logs-service mlflow ``` ### 4. Interactive Visualization ```bash # After solving a task, get execution_id just solve "Complex task" # Visualize execution tree just viz ``` --- ## Examples ### Example 1: Simple Calculation ```bash just solve "Calculate compound interest on $10,000 at 5% annual rate for 10 years" ``` ### Example 2: Web Research ```bash just solve "Research the latest developments in quantum computing and summarize in 3 bullet points" ``` ### Example 3: Code Execution ```bash just solve "Generate a Python script that creates a fibonacci sequence up to 100, execute it, and show results" ``` ### Example 4: Crypto Analysis ```bash just solve "Compare Bitcoin and Ethereum market caps, 24h volumes, and price changes" crypto_agent ``` ### Example 5: File Operations ```bash just solve "Create a JSON file with data about the top 5 programming languages and their use cases" ``` --- ## Troubleshooting ### Docker not starting ```bash # Check Docker is running docker ps # Rebuild images just docker-down just docker-build-clean just docker-up # Check logs just docker-logs ``` ### API not responding ```bash # Check health curl http://localhost:8000/health # Check container status just docker-ps # View logs just docker-logs-service roma-api ``` ### Database connection errors ```bash # Check postgres is running docker ps | grep postgres # Check connection docker exec -it roma-dspy-postgres psql -U postgres -d roma_dspy -c "SELECT 1" # Verify DATABASE_URL in .env matches docker-compose.yaml ``` ### Missing API keys ```bash # Verify keys are set docker exec -it roma-dspy-api env | grep API_KEY # Restart after changing .env just docker-restart ``` ### E2B not working ```bash # Check E2B key is set echo $E2B_API_KEY # Test E2B connection just e2b-test # Build custom template (if using S3 mount) just e2b-build ``` --- ## Next Steps ### Learn More - **[Configuration Guide](CONFIGURATION.md)** - Profiles, agents, settings - **[Toolkits Reference](TOOLKITS.md)** - All available toolkits - **[MCP Integration](MCP.md)** - Using MCP servers - **[API Reference](API.md)** - REST API endpoints - **[Deployment Guide](DEPLOYMENT.md)** - Production deployment - **[Observability](OBSERVABILITY.md)** - MLflow tracking ### Explore Examples ```bash # See all example configurations ls config/examples/*/ # Try different examples just solve "task" -c config/examples/basic/minimal.yaml ``` ### Customize 1. Create custom profiles in `config/profiles/` 2. Add custom toolkits (see [TOOLKITS.md](TOOLKITS.md)) 3. Configure agents per task type (see [CONFIGURATION.md](CONFIGURATION.md)) ### Deploy ```bash # Production deployment just deploy-full # Check deployment just health-check ``` --- ## REST API ROMA-DSPy includes a production-ready REST API for programmatic access. ### Quick Start ```bash # Start API server (via Docker) just docker-up # Verify server is running curl http://localhost:8000/health ``` ### API Documentation FastAPI provides interactive API documentation: - **Swagger UI** (interactive testing): http://localhost:8000/docs - **ReDoc** (clean reference): http://localhost:8000/redoc - **OpenAPI JSON**: http://localhost:8000/openapi.json ### Example Usage ```bash # Start execution curl -X POST http://localhost:8000/api/v1/executions \ -H "Content-Type: application/json" \ -d '{"goal": "What is 2+2?", "max_depth": 1}' | jq # Get status (use execution_id from response) curl http://localhost:8000/api/v1/executions//status | jq # Get metrics curl http://localhost:8000/api/v1/executions//metrics | jq ``` **See** http://localhost:8000/docs **for complete API reference with all endpoints, schemas, and interactive testing.** --- ## Getting Help - **Documentation**: `docs/` directory - **Examples**: `config/examples/` - **Issues**: GitHub Issues - **Just Commands**: Run `just` to see all available commands --- **You're all set!** Start building with ROMA-DSPy πŸš€ --- ## File: docs/TOOLKITS.md # ROMA-DSPy Toolkits Reference Complete guide to using toolkits in ROMA-DSPy agents. ## Table of Contents - [Overview](#overview) - [Quick Start](#quick-start) - [Native Toolkits](#native-toolkits) - [MCP Integration](#mcp-integration) - [Configuration Guide](#configuration-guide) - [Examples](#examples) - [Creating Custom Toolkits](#creating-custom-toolkits) - [Best Practices](#best-practices) --- ## Overview ROMA-DSPy provides a powerful toolkit system that enables agents to interact with external systems, execute code, access data, and perform specialized operations. The toolkit architecture supports: - **10 Built-in Toolkits** for common operations (files, math, web, crypto, code execution) - **MCP Integration** to connect to any Model Context Protocol server (1000+ available) - **Smart Data Handling** with optional Parquet storage for large results - **Execution Isolation** with per-execution file scoping - **Tool Metrics** tracking invocations, latency, and errors - **Flexible Configuration** via YAML profiles ### Architecture ``` Agent (Executor) β”œβ”€β”€ Toolkit Manager β”‚ β”œβ”€β”€ Native Toolkits (FileToolkit, CalculatorToolkit, etc.) β”‚ β”œβ”€β”€ MCP Toolkits (connects to external MCP servers) β”‚ └── Custom Toolkits (user-defined) β”œβ”€β”€ Tool Storage (optional Parquet for large data) └── Tool Metrics (tracking and observability) ``` Each toolkit: - Auto-registers tools with DSPy's tool system - Provides full parameter schemas for LLM tool selection - Handles errors gracefully with structured responses - Optionally stores large results to reduce context usage --- ## Quick Start ### 1. Using Built-in Toolkits ```yaml # config/profiles/my_profile.yaml agents: executor: llm: model: openai/gpt-4o-mini temperature: 0.3 prediction_strategy: react # Required for tool usage toolkits: - class_name: FileToolkit enabled: true - class_name: CalculatorToolkit enabled: true - class_name: E2BToolkit enabled: true toolkit_config: timeout: 300 ``` **Usage:** ```bash just solve "Calculate 15% of 2500 and save to results.txt" -c config/profiles/my_profile.yaml ``` ### 2. Using MCP Servers ```yaml agents: executor: llm: model: openai/gpt-4o-mini prediction_strategy: react toolkits: # Public HTTP MCP server (no installation needed) - class_name: MCPToolkit enabled: true toolkit_config: server_name: coingecko server_type: http url: https://mcp.api.coingecko.com/sse use_storage: false ``` **Usage:** ```bash just solve "What is the current price of Bitcoin?" -c config/profiles/my_profile.yaml ``` --- ## Native Toolkits ROMA-DSPy includes 10 built-in toolkits registered in `ToolkitManager.BUILTIN_TOOLKITS`. ### 1. FileToolkit File operations with execution-scoped isolation. **Tools:** - `save_file(file_path: str, content: str, encoding: str = 'utf-8')` - Save content to file - `read_file(file_path: str, encoding: str = 'utf-8')` - Read file content - `list_files(directory: str = ".", pattern: str = "*")` - List files matching pattern - `search_files(query: str, directory: str = ".", extensions: list = None)` - Search file contents - `create_directory(directory_path: str)` - Create directory - `delete_file(file_path: str)` - Delete file (requires enable_delete=True) **Configuration:** ```yaml - class_name: FileToolkit enabled: true toolkit_config: enable_delete: false # Safety: disable destructive operations max_file_size: 10485760 # 10MB limit ``` **Security:** - All file paths are scoped to execution-specific directories - Path traversal attacks prevented - File size limits enforced - Delete operations disabled by default **Example:** See `config/examples/basic/minimal.yaml` --- ### 2. CalculatorToolkit Mathematical operations with precision control. **Tools:** - `add(a: float, b: float)` - Add two numbers - `subtract(a: float, b: float)` - Subtract b from a - `multiply(a: float, b: float)` - Multiply two numbers - `divide(a: float, b: float)` - Divide a by b - `exponentiate(base: float, exponent: float)` - Calculate base^exponent - `factorial(n: int)` - Calculate factorial of n - `is_prime(n: int)` - Check if n is prime - `square_root(n: float)` - Calculate square root **Configuration:** ```yaml - class_name: CalculatorToolkit enabled: true toolkit_config: precision: 10 # Decimal places (default: 10) ``` **Response Format:** ```json { "success": true, "operation": "addition", "operands": [25, 47], "result": 72.0 } ``` **Example:** See `config/examples/basic/minimal.yaml` --- ### 3. E2BToolkit Secure sandboxed code execution via [E2B](https://e2b.dev). **Features:** - Isolated Python/Node.js execution environments - Automatic sandbox health checks - Sandbox lifecycle management - File system access within sandbox - Network access for data fetching **Configuration:** ```yaml - class_name: E2BToolkit enabled: true toolkit_config: timeout: 300 # Execution timeout (seconds) max_lifetime_hours: 23.5 # Auto-restart before 24h limit template: base # E2B template ID auto_reinitialize: true # Auto-restart on failure ``` **Environment Variables:** ```bash export E2B_API_KEY=your_key_here export E2B_TEMPLATE_ID=base # Optional: custom template ``` **Example:** See `config/examples/basic/multi_toolkit.yaml` --- ### 4. SerperToolkit Web search via [Serper.dev](https://serper.dev) API. **Tools:** - `search(query: str, num_results: int = 10)` - Search the web **Configuration:** ```yaml - class_name: SerperToolkit enabled: true toolkit_config: location: "United States" # Search location language: "en" # Results language num_results: 10 # Number of results date_range: null # Optional: "d" (day), "w" (week), "m" (month), "y" (year) ``` **Environment Variables:** ```bash export SERPER_API_KEY=your_key_here ``` **Example:** See `config/examples/basic/multi_toolkit.yaml` --- ### 5. WebSearchToolkit Native web search using DSPy with LLM-powered web search capabilities. **Features:** - DSPy-native integration with web search enabled models - Supports OpenRouter (with plugins) and OpenAI (Responses API) - Automatic citation extraction - Expert searcher prompts for comprehensive data retrieval - Prioritizes reliable sources (Wikipedia, government, academic) - Configurable search context depth **Tool:** - `web_search(query: str, max_results: int = None, search_context_size: str = None)` - Search the web with comprehensive data retrieval **Configuration:** ```yaml - class_name: WebSearchToolkit enabled: true toolkit_config: model: openrouter/openai/gpt-5-mini # Auto-detects provider from prefix search_engine: exa # For OpenRouter (omit for native search) max_results: 5 # Number of search results search_context_size: medium # low, medium, or high temperature: 1.0 # Model temperature (1.0 required for GPT-5) max_tokens: 16000 # Max response tokens (16000+ for GPT-5) ``` **Provider Detection:** - Models starting with `openrouter/` use OpenRouter plugins API - Models starting with `openai/` use OpenAI Responses API - No separate provider parameter needed **Search Behavior:** The toolkit uses expert searcher instructions that guide the LLM to: 1. Retrieve COMPLETE datasets (entire tables, all list items, all data points) 2. Prioritize reliable sources (Wikipedia first, then gov/academic/news) 3. Present data EXACTLY as found (no summarization) 4. Include temporal awareness for time-sensitive queries **Environment Variables:** ```bash export OPENROUTER_API_KEY=your_key_here # For OpenRouter models # OR export OPENAI_API_KEY=your_key_here # For OpenAI models ``` **Response Format:** ```json { "success": true, "data": "Comprehensive answer with complete data...", "citations": [ {"url": "https://en.wikipedia.org/..."}, {"url": "https://example.com/..."} ], "tool": "web_search", "model": "openrouter/openai/gpt-5-mini", "provider": "openrouter" } ``` **Example Usage:** ```yaml # OpenRouter native search (GPT-5-mini) - class_name: WebSearchToolkit toolkit_config: model: openrouter/openai/gpt-5-mini # No search_engine = native search max_results: 5 search_context_size: medium temperature: 1.0 max_tokens: 16000 # OpenRouter with Exa search engine - class_name: WebSearchToolkit toolkit_config: model: openrouter/anthropic/claude-sonnet-4 search_engine: exa max_results: 10 search_context_size: high # OpenAI Responses API - class_name: WebSearchToolkit toolkit_config: model: openai/gpt-4o search_context_size: medium max_results: 5 ``` **Example:** See `config/profiles/crypto_agent.yaml` --- ### 6. BinanceToolkit Cryptocurrency market data from Binance. **Features:** - Spot, USDT-margined futures, and coin-margined futures - Real-time prices and ticker stats - Orderbook depth and recent trades - OHLCV candlestick data - Optional statistical analysis **Tools:** - `get_current_price(symbol: str, market: str = "spot")` - Current price - `get_ticker_stats(symbol: str, market: str = "spot")` - 24h ticker statistics - `get_book_ticker(symbol: str, market: str = "spot")` - Best bid/ask prices - `get_klines(symbol: str, interval: str, limit: int = 100, market: str = "spot")` - OHLCV data - `get_order_book(symbol: str, limit: int = 100, market: str = "spot")` - Order book depth - `get_recent_trades(symbol: str, limit: int = 100, market: str = "spot")` - Recent trades **Configuration:** ```yaml - class_name: BinanceToolkit enabled: true toolkit_config: default_market: spot # spot, usdm, coinm enable_analysis: false # Statistical analysis ``` **No API Key Required** - Uses public Binance endpoints **Example:** See `config/profiles/crypto_agent.yaml` --- ### 7. CoinGeckoToolkit Comprehensive cryptocurrency data from [CoinGecko](https://coingecko.com). **Features:** - 17,000+ cryptocurrencies - Real-time prices in 100+ currencies - Historical price and market data - OHLCV candlestick data - Market rankings and statistics - Contract address lookups - Global market metrics **Tools:** - `get_coin_price(coin_name_or_id: str, vs_currency: str = "usd")` - Current price - `get_coin_market_chart(coin_name_or_id: str, vs_currency: str = "usd", days: int = 30)` - Historical data - More tools available - see toolkit implementation **Configuration:** ```yaml - class_name: CoinGeckoToolkit enabled: true toolkit_config: coins: null # Restrict to specific coins (null = all) default_vs_currency: usd # Default quote currency use_pro: false # Use CoinGecko Pro API enable_analysis: false # Statistical analysis ``` **Environment Variables:** ```bash export COINGECKO_API_KEY=your_key_here # Optional: for Pro API ``` **No API Key Required** for public endpoints **Example:** See `config/profiles/crypto_agent.yaml` --- ### 8. DefiLlamaToolkit DeFi protocol analytics from [DefiLlama](https://defillama.com). **Features:** - Protocol TVL (Total Value Locked) tracking - Daily fees and revenue analysis - Yield farming pools and APY data (Pro) - User activity metrics (Pro) - Cross-chain analytics - Statistical analysis **Tools (Public):** - `get_protocol_fees(protocol_name: str)` - Protocol fees and revenue - `get_protocol_tvl(protocol_name: str)` - Total Value Locked - More public tools available **Tools (Pro - requires API key):** - `get_yield_pools()` - Yield farming opportunities - `get_yield_chart(pool_id: str)` - Historical APY data - `get_active_users(protocol_name: str)` - User activity - More Pro tools available **Configuration:** ```yaml - class_name: DefiLlamaToolkit enabled: true toolkit_config: enable_pro_features: false # Requires API key default_chain: ethereum enable_analysis: true ``` **Environment Variables:** ```bash export DEFILLAMA_API_KEY=your_key_here # For Pro features ``` **No API Key Required** for public endpoints **Example:** See `config/profiles/crypto_agent.yaml` --- ### 9. ArkhamToolkit Blockchain analytics from [Arkham Intelligence](https://arkhamintelligence.com). **Features:** - Token analytics (top tokens, holders, flows) - Transfer tracking with entity attribution - Wallet balance monitoring across chains - Statistical analysis of distributions - Rate limiting (20 req/sec standard, 1 req/sec heavy) **Tools:** - Token analytics tools - Transfer tracking tools - Wallet balance tools - More tools available - see toolkit implementation **Configuration:** ```yaml - class_name: ArkhamToolkit enabled: true toolkit_config: default_chain: ethereum enable_analysis: true ``` **Environment Variables:** ```bash export ARKHAM_API_KEY=your_key_here # Required ``` **API Key Required** --- ### 10. CoinglassToolkit Derivatives market data from [Coinglass](https://coinglass.com). **Features:** - Historical funding rates weighted by open interest (OHLC data) - Real-time funding rates across 20+ exchanges - Funding rate arbitrage opportunity detection - Open interest tracking and historical analysis - Taker buy/sell volume ratios (market sentiment) - Liquidation data by exchange and position type **Tools:** - `get_funding_rates_weighted_by_oi` - Historical funding rate OHLC data - `get_funding_rates_per_exchange` - Current funding rates across exchanges - `get_arbitrage_opportunities` - Funding rate arbitrage opportunities - `get_open_interest_by_exchange` - Current open interest by exchange - `get_open_interest_history` - Historical open interest data - `get_taker_buy_sell_volume` - Buy/sell volume ratios - `get_liquidations_by_exchange` - Liquidation data **Configuration:** ```yaml - class_name: CoinglassToolkit enabled: true toolkit_config: symbols: ["BTC", "ETH", "SOL"] # Restrict to specific symbols (null = all) default_symbol: BTC storage_threshold_kb: 500 # Auto-store responses > 500KB ``` **Environment Variables:** ```bash export COINGLASS_API_KEY=your_key_here # Required ``` **API Key Required** - Get yours at [Coinglass API](https://coinglass.com/api) **Example:** See `config/profiles/crypto_agent.yaml` --- ### 11. MCPToolkit Universal connector for Model Context Protocol servers. **Special Property:** The MCPToolkit can connect to **any** MCP server - there are 1000+ available! See [MCP Integration](#mcp-integration) section below for complete details. --- ## MCP Integration The **MCPToolkit** enables ROMA-DSPy agents to use tools from **any** MCP (Model Context Protocol) server. This provides unlimited extensibility beyond the 10 built-in toolkits. ### What is MCP? MCP is an open protocol for connecting AI applications to data sources and tools. It's like USB-C for AI - a universal connector. **Resources:** - **Awesome MCP Servers**: [700+ servers](https://github.com/wong2/awesome-mcp-servers) - **MCP Documentation**: [modelcontextprotocol.io](https://modelcontextprotocol.io/) - **Build Your Own**: Any server implementing the MCP protocol ### Connection Types #### 1. HTTP/SSE Servers (Remote) **Best for:** Public APIs, cloud services, no installation needed **Example - CoinGecko Public Server:** ```yaml - class_name: MCPToolkit enabled: true toolkit_config: server_name: coingecko server_type: http url: https://mcp.api.coingecko.com/sse use_storage: false ``` **Example - Exa Search (with API key):** ```yaml - class_name: MCPToolkit enabled: true toolkit_config: server_name: exa server_type: http url: https://mcp.exa.ai/mcp headers: Authorization: "Bearer ${oc.env:EXA_API_KEY}" use_storage: true # Exa returns large search results storage_threshold_kb: 50 ``` **No installation required** - connects over HTTP #### 2. Stdio Servers (Local Subprocess) **Best for:** Local tools, filesystem access, databases, git operations **Example - GitHub Operations:** ```yaml - class_name: MCPToolkit enabled: true toolkit_config: server_name: github server_type: stdio command: npx args: - "-y" - "@modelcontextprotocol/server-github" env: GITHUB_PERSONAL_ACCESS_TOKEN: "${oc.env:GITHUB_PERSONAL_ACCESS_TOKEN}" use_storage: false ``` **Example - Filesystem Access:** ```yaml - class_name: MCPToolkit enabled: true toolkit_config: server_name: filesystem server_type: stdio command: npx args: - "-y" - "@modelcontextprotocol/server-filesystem" - "/Users/yourname/Documents" # Allowed directory use_storage: false ``` **Requires installation:** ```bash npm install @modelcontextprotocol/server-github npm install @modelcontextprotocol/server-filesystem ``` ### Storage Configuration MCP tools can return large datasets (search results, database queries, etc.). The toolkit provides smart data handling: **Small Data (default):** ```yaml use_storage: false # Returns raw text/JSON directly ``` **Large Data (with storage):** ```yaml use_storage: true # Stores data in Parquet, returns reference storage_threshold_kb: 100 # Store results > 100KB (default) ``` **How it works:** 1. Tool executes and returns data 2. If data size > threshold, saves to Parquet file 3. Returns file reference instead of full data 4. Reduces context usage for large datasets ### Finding MCP Servers **Popular Categories:** | Category | Examples | |----------|----------| | **Web Search** | Exa, Brave Search, Google Search | | **Development** | GitHub, GitLab, Linear, Sentry | | **Data** | PostgreSQL, SQLite, MongoDB, Redis | | **Cloud** | AWS, Google Cloud, Kubernetes | | **Productivity** | Google Drive, Slack, Notion, Confluence | | **Finance** | Stripe, QuickBooks | | **AI/ML** | OpenAI, Anthropic, Hugging Face | **Browse all:** - [awesome-mcp-servers](https://github.com/wong2/awesome-mcp-servers) - 700+ servers - [MCP Server Registry](https://modelcontextprotocol.io/servers) - Official registry ### Multiple MCP Servers You can use **multiple** MCP servers in one agent: ```yaml agents: executor: llm: model: openai/gpt-4o-mini prediction_strategy: react toolkits: # GitHub for code - class_name: MCPToolkit toolkit_config: server_name: github server_type: stdio command: npx args: ["-y", "@modelcontextprotocol/server-github"] env: GITHUB_PERSONAL_ACCESS_TOKEN: "${oc.env:GITHUB_TOKEN}" # Exa for web search - class_name: MCPToolkit toolkit_config: server_name: exa server_type: http url: https://mcp.exa.ai/mcp headers: Authorization: "Bearer ${oc.env:EXA_API_KEY}" use_storage: true # Filesystem for local files - class_name: MCPToolkit toolkit_config: server_name: filesystem server_type: stdio command: npx args: ["-y", "@modelcontextprotocol/server-filesystem", "/path/to/allowed/dir"] ``` **Example:** See `config/examples/mcp/multi_server.yaml` --- ## Configuration Guide ### Basic Structure ```yaml agents: executor: llm: model: openai/gpt-4o-mini temperature: 0.3 prediction_strategy: react # REQUIRED for tool usage toolkits: - class_name: ToolkitName enabled: true include_tools: null # Optional: whitelist specific tools exclude_tools: null # Optional: blacklist specific tools toolkit_config: # Toolkit-specific settings ``` ### Tool Filtering **Include specific tools only:** ```yaml - class_name: CalculatorToolkit enabled: true include_tools: - add - subtract - multiply # Only these 3 tools will be available ``` **Exclude specific tools:** ```yaml - class_name: FileToolkit enabled: true exclude_tools: - delete_file # Safety: disable deletions # All tools except delete_file will be available ``` ### Environment Variables **Via OmegaConf:** ```yaml toolkit_config: api_key: "${oc.env:MY_API_KEY}" # Reads from environment timeout: "${oc.env:TIMEOUT,300}" # Default value: 300 ``` **Via .env file:** ```bash # .env E2B_API_KEY=your_key SERPER_API_KEY=your_key GITHUB_PERSONAL_ACCESS_TOKEN=your_token ``` ### Storage Integration Some toolkits support optional Parquet storage for large data: ```yaml - class_name: MCPToolkit enabled: true toolkit_config: server_name: database server_type: stdio command: npx args: ["-y", "@modelcontextprotocol/server-sqlite", "/path/to/db.db"] use_storage: true # Enable storage wrapper storage_threshold_kb: 100 # Store results > 100KB ``` **Toolkits with storage support:** - MCPToolkit - DefiLlamaToolkit - ArkhamToolkit - BinanceToolkit (for large responses) - CoinGeckoToolkit (for large responses) - CoinglassToolkit (for large responses) --- ## Examples All examples available in `config/examples/`. See `config/examples/README.md` for complete guide. ### Example 1: Minimal Configuration **File:** `config/examples/basic/minimal.yaml` Simple agent with FileToolkit and CalculatorToolkit. **Usage:** ```bash just solve "Calculate 15% of 2500 and save to results.txt" -c config/examples/basic/minimal.yaml ``` --- ### Example 2: Multi-Toolkit **File:** `config/examples/basic/multi_toolkit.yaml` Combines E2B (code execution), FileToolkit, CalculatorToolkit, and SerperToolkit. **Usage:** ```bash just solve "Search for Python fibonacci implementation, execute it, and save results" \ -c config/examples/basic/multi_toolkit.yaml ``` --- ### Example 3: Public HTTP MCP Server **File:** `config/examples/mcp/http_public_server.yaml` Uses CoinGecko public MCP server - **no installation or API key required!** **Usage:** ```bash just solve "What is the current price of Bitcoin?" \ -c config/examples/mcp/http_public_server.yaml ``` --- ### Example 4: Local Stdio MCP Server **File:** `config/examples/mcp/stdio_local_server.yaml` Uses local Exa MCP server for web search. **Setup:** ```bash export EXA_API_KEY=your_key npm install @exa-labs/exa-mcp-server ``` **Usage:** ```bash just solve "Search for latest LLM research papers" \ -c config/examples/mcp/stdio_local_server.yaml ``` --- ### Example 5: Multiple MCP Servers **File:** `config/examples/mcp/multi_server.yaml` Combines GitHub, Exa (web search), and CoinGecko MCP servers. **Usage:** ```bash just solve "Search recent AI news, check Bitcoin price, and create GitHub issue summary" \ -c config/examples/mcp/multi_server.yaml ``` --- ### Example 6: Crypto Agent (Domain-Specific) **File:** `config/profiles/crypto_agent.yaml` Comprehensive crypto analysis with: - CoinGeckoToolkit (17,000+ coins) - CoinglassToolkit (derivatives market data) - BinanceToolkit (spot + futures) - DefiLlamaToolkit (DeFi protocols) - ArkhamToolkit (blockchain analytics) - Exa MCP (web search) **Usage:** ```bash just solve "Compare Bitcoin and Ethereum: prices, market caps, 24h volumes, and analyze trends" \ crypto_agent ``` --- ## Creating Custom Toolkits ### Step 1: Create Toolkit Class ```python # my_custom_toolkit.py from roma_dspy.tools.base.base import BaseToolkit from typing import Optional, List class MyCustomToolkit(BaseToolkit): """My custom toolkit for XYZ operations.""" def __init__( self, enabled: bool = True, include_tools: Optional[List[str]] = None, exclude_tools: Optional[List[str]] = None, **config, ): super().__init__( enabled=enabled, include_tools=include_tools, exclude_tools=exclude_tools, **config, ) # Your initialization self.api_key = config.get("api_key") def _setup_dependencies(self) -> None: """Setup external dependencies.""" # Optional: validate API keys, initialize clients pass def _initialize_tools(self) -> None: """Initialize toolkit-specific configuration.""" # Optional: additional setup pass # Tool methods (auto-registered by BaseToolkit) async def my_tool(self, param1: str, param2: int) -> str: """ Tool description that the LLM will see. Args: param1: Description of param1 param2: Description of param2 Returns: Result description """ # Your tool implementation result = f"Processed {param1} with {param2}" return result async def another_tool(self, query: str) -> dict: """Another tool that returns structured data.""" return { "success": True, "query": query, "results": ["result1", "result2"] } ``` ### Step 2: Register Toolkit Add to `src/roma_dspy/tools/base/manager.py`: ```python BUILTIN_TOOLKITS = { # ... existing toolkits ... "MyCustomToolkit": "path.to.my_custom_toolkit", } ``` ### Step 3: Use in Configuration ```yaml agents: executor: llm: model: openai/gpt-4o-mini prediction_strategy: react toolkits: - class_name: MyCustomToolkit enabled: true toolkit_config: api_key: "${oc.env:MY_API_KEY}" ``` ### Best Practices 1. **Tool Design:** - Clear, descriptive tool names - Comprehensive docstrings (LLM sees these) - Type hints for all parameters - Return structured data (JSON dicts or strings) 2. **Error Handling:** ```python async def my_tool(self, param: str) -> dict: try: result = await self._do_something(param) return {"success": True, "data": result} except Exception as e: logger.error(f"Tool failed: {e}") return {"success": False, "error": str(e)} ``` 3. **Storage for Large Data:** ```python class MyToolkit(BaseToolkit): REQUIRES_FILE_STORAGE = False # Optional storage def __init__(self, use_storage: bool = False, **config): super().__init__(**config) self.use_storage = use_storage async def big_data_tool(self, query: str) -> str: result = await self._fetch_large_dataset(query) if self.use_storage and len(result) > threshold: # Store to Parquet and return reference path = await self.file_storage.save_tool_result(...) return f"Data stored at: {path}" return result ``` 4. **Testing:** ```python # tests/test_my_toolkit.py import pytest from my_custom_toolkit import MyCustomToolkit @pytest.mark.asyncio async def test_my_tool(): toolkit = MyCustomToolkit() result = await toolkit.my_tool("test", 42) assert "Processed" in result ``` --- ## Best Practices ### 1. Toolkit Selection **Choose the right tools for the task:** ```yaml # For file operations + math toolkits: - class_name: FileToolkit - class_name: CalculatorToolkit # For web research toolkits: - class_name: SerperToolkit # Native # OR - class_name: MCPToolkit # MCP (Exa, Brave, etc.) toolkit_config: server_name: exa server_type: http url: https://mcp.exa.ai/mcp # For code execution toolkits: - class_name: E2BToolkit ``` ### 2. Security **File operations:** ```yaml - class_name: FileToolkit toolkit_config: enable_delete: false # Disable destructive operations max_file_size: 10485760 # 10MB limit ``` **MCP servers:** - Only use trusted MCP servers - Validate server URLs and signatures - Use environment variables for sensitive data ### 3. Performance **Use storage for large data:** ```yaml - class_name: MCPToolkit toolkit_config: use_storage: true storage_threshold_kb: 50 # Aggressive threshold for faster responses ``` **Limit tool scope:** ```yaml - class_name: CalculatorToolkit include_tools: - add - multiply # Faster tool selection with fewer options ``` ### 4. Cost Optimization **Use task-aware mapping** to assign different toolkits to different task types: ```yaml agent_mapping: executors: RETRIEVE: # Cheap model + web search llm: model: openrouter/google/gemini-2.0-flash-exp:free toolkits: - class_name: SerperToolkit CODE_INTERPRET: # Powerful model + code execution llm: model: openrouter/anthropic/claude-sonnet-4 toolkits: - class_name: E2BToolkit - class_name: FileToolkit ``` **Example:** See `config/examples/advanced/task_aware_mapping.yaml` ### 5. Observability **Enable logging:** ```yaml runtime: enable_logging: true ``` **Track tool metrics:** - Tool invocations logged automatically - Latency tracking - Error rates - View in MLflow (if observability enabled) ### 6. API Key Management **Never hardcode keys:** ```yaml # ❌ BAD toolkit_config: api_key: "sk-1234567890abcdef" # βœ… GOOD toolkit_config: api_key: "${oc.env:MY_API_KEY}" ``` **Use .env file:** ```bash # .env E2B_API_KEY=your_key SERPER_API_KEY=your_key GITHUB_PERSONAL_ACCESS_TOKEN=your_token ``` --- ## Troubleshooting ### "Unknown toolkit class: XYZ" **Cause:** Toolkit not registered or typo in class_name **Fix:** ```bash # Check available toolkits python -c "from roma_dspy.tools.base.manager import ToolkitManager; print(ToolkitManager.BUILTIN_TOOLKITS.keys())" # Verify spelling matches exactly (case-sensitive) ``` ### "Tools don't support strategy: chain_of_thought" **Cause:** Chain-of-thought strategy doesn't support tool usage **Fix:** ```yaml agents: executor: prediction_strategy: react # Use react or codeact for tools ``` ### MCP Server Connection Failed **HTTP servers:** ```bash # Test connectivity curl -I https://mcp.api.coingecko.com/sse # Check headers/auth curl -H "Authorization: Bearer YOUR_KEY" https://mcp.exa.ai/mcp ``` **Stdio servers:** ```bash # Verify installation npx @modelcontextprotocol/server-github --version # Test manually npx -y @modelcontextprotocol/server-github ``` ### E2B Not Working ```bash # Verify API key echo $E2B_API_KEY # Test connection python -c "from e2b import Sandbox; s = Sandbox(); print(s.is_running())" # Check template export E2B_TEMPLATE_ID=base ``` ### Large Data Timeouts **Enable storage:** ```yaml toolkit_config: use_storage: true storage_threshold_kb: 50 # Lower threshold ``` --- ## Additional Resources - **Configuration Guide**: [CONFIGURATION.md](CONFIGURATION.md) - **MCP Deep Dive**: [MCP.md](MCP.md) - **Example Configurations**: `config/examples/` - **Awesome MCP Servers**: https://github.com/wong2/awesome-mcp-servers - **MCP Documentation**: https://modelcontextprotocol.io/ - **E2B Documentation**: https://e2b.dev/docs --- **Ready to build?** Start with the examples in `config/examples/` and customize for your use case! πŸš€ --- ## File: prompt_optimization/experiment_cli/README.md # GEPA Optimization Experiment CLI CLI tool for running configurable GEPA optimization experiments with MLflow tracking. ## Features - **YAML-based configuration**: All experiment settings in YAML files - **MLflow autolog integration**: Automatic tracking of params, metrics, datasets, traces - **OmegaConf support**: Type-safe config loading with defaults and overrides - **Minimal CLI overrides**: Quick tweaks without editing config files - **GEPA observability**: Full support for track_stats, log_dir, checkpointing - **Leverages ROMA profiles**: Agent configurations come from existing ROMA profile YAMLs ## Quick Start ### Recommended: Using Just Commands The easiest way to run experiments is using the `just` task runner from the project root: ```bash # Start services (MLflow, MinIO, PostgreSQL) just docker-up-full # Run optimization with default config (quick_test) just optimize # Run with specific config and name just optimize balanced my-experiment # Run with specific profile and verbose logging just optimize balanced my-experiment default true # Run without MLflow tracking just optimize-no-mlflow quick_test test-run # List available configs just list-optimize-configs # Open MLflow UI in browser just mlflow-ui ``` **Just Command Arguments:** ```bash just optimize [config] [name] [profile] [verbose] config - Config name from configs/ (without .yaml, default: quick_test) name - Experiment name (default: auto-generated timestamp) profile - ROMA profile to use (default: test) verbose - Enable verbose logging (default: false) ``` **Why use Just commands?** - βœ… Runs inside Docker with all dependencies configured - βœ… MLflow tracking pre-configured (`http://mlflow:5000`) - βœ… MinIO S3 artifact storage pre-configured - βœ… Simple, memorable commands from project root ### Alternative: Direct Docker Commands If you prefer to run Docker commands directly: ```bash # Start services docker compose --profile observability up -d # Run experiment inside container (using uv) docker exec -it roma-dspy-api bash -c "cd /app/prompt_optimization/experiment_cli && uv run python run_experiment.py --config configs/quick_test.yaml" # Or enter container interactively docker exec -it roma-dspy-api bash cd /app/prompt_optimization/experiment_cli uv run python run_experiment.py --config configs/balanced.yaml ``` **Note:** The container has `uv` installed for fast Python package management. Always use `uv run python` for consistency. ### For Local Development (Outside Docker) For local development, use `uv` as well: ```bash # Install uv if not already installed curl -LsSf https://astral.sh/uv/install.sh | sh # Run experiment locally cd prompt_optimization/experiment_cli uv run python run_experiment.py --config configs/quick_test.yaml ``` Ensure your `.env` has the correct endpoints: ```bash MLFLOW_TRACKING_URI=http://mlflow:5000 # For Docker, or http://localhost:5000 for local MINIO_ROOT_USER=minioadmin MINIO_ROOT_PASSWORD=minioadmin123 ``` ### Example Configs - **`configs/quick_test.yaml`**: Fast iteration (8 train, 10 metric calls) - **`configs/balanced.yaml`**: Good balance (32 train, 48 metric calls) - **`configs/thorough.yaml`**: Maximum quality (64 train, 100 metric calls) - **`configs/custom_lms.yaml`**: Custom LM configuration example ## Configuration ### YAML Structure The config YAML maps directly to `OptimizationConfig` dataclass: ```yaml # Dataset configs train_size: 32 val_size: 8 test_size: 12 dataset_seed: 0 # GEPA configs max_metric_calls: 48 num_threads: 8 reflection_minibatch_size: 8 component_selector: "round_robin" # or planner_only, atomizer_only, etc. # GEPA observability track_stats: true track_best_outputs: true log_dir: "logs/my_experiment" use_mlflow: true # Solver configs max_depth: 1 enable_logging: false # Execution max_parallel: 12 # Output output_path: "outputs/my_experiment" ``` ### Custom LM Configurations Override LM configs for specific components: ```yaml # Judge LM (for component feedback) judge_lm: model: "openrouter/anthropic/claude-sonnet-4.5" temperature: 0.75 max_tokens: 64000 cache: true # Reflection LM (for GEPA optimization) reflection_lm: model: "openrouter/anthropic/claude-sonnet-4.5" temperature: 1.0 max_tokens: 64000 cache: true ``` **Note**: Agent LMs (atomizer, planner, executor, aggregator) come from ROMA profile YAMLs, not optimization config. ## CLI Options ### Required None (uses defaults if no config specified) ### Optional ``` --config, -c Path to YAML config file --name Experiment name (default: auto-generated timestamp) --dataset Dataset type: aimo, frames, simpleqa, simpleqa_verified, seal0 --profile ROMA config profile (default: test) --num-threads GEPA num_threads override --selector Component selector override --mlflow-uri MLflow tracking URI (default: http://localhost:5000) --mlflow-experiment MLflow experiment name (default: roma-optimization) --no-mlflow Disable MLflow tracking --output-dir Output directory override --save-config Save effective config to path (useful for debugging) --verbose Enable verbose logging ``` ## MLflow Integration ### Setup 1. **Start MLflow server** (one-time): ```bash mlflow server --backend-store-uri sqlite:///mlflow.db --host 0.0.0.0 --port 5000 ``` 2. **Run experiment** (MLflow autolog handles tracking automatically): ```bash python run_experiment.py --config configs/balanced.yaml ``` 3. **View results**: Open http://localhost:5000 in browser ### What Gets Tracked Automatically MLflow's `dspy.autolog()` captures: - βœ… All GEPA optimizer parameters - βœ… Training/validation metrics over time - βœ… Optimized program states (JSON artifacts) - βœ… Datasets used - βœ… Full execution traces - βœ… Intermediate program versions ### Manual Logging Only a few things need manual logging: - Experiment metadata (name, dataset type) - Test set evaluation results - Custom metrics/tags The CLI handles these automatically. ## Example Workflows ### Quick Iteration ```bash # Fast test with minimal budget python run_experiment.py \ --config configs/quick_test.yaml \ --name quick_test_001 \ --dataset aimo ``` ### Production Run ```bash # Thorough optimization with full tracking python run_experiment.py \ --config configs/thorough.yaml \ --name prod_frames_experiment \ --dataset frames \ --profile default ``` ### Experiment Sweep ```bash # Test different selectors for selector in planner_only round_robin; do python run_experiment.py \ --config configs/balanced.yaml \ --selector $selector \ --name "balanced_${selector}" done ``` ### Custom Configuration ```bash # Create custom config cat > configs/my_experiment.yaml < 180 days) - Canonical Reference: https://codewiki.google/github.com/sentient-agi/ROMA