## File: ckpt/README.md

# Paper-aligned SkillOpt reference skills (GPT-5.5)

This folder provides a subset of the paper's main Table 1 GPT-5.5 optimized
skills as reference artifacts — one `gpt5.5_skill.md` per currently included
benchmark. You can plug them into `scripts/eval_only.py` to evaluate the
provided skills on a given split without re-running the training loop.

> These are checkpoints associated with the paper, not a general-purpose
> tool. They're here so you can verify the reported numbers and use the
> skills as portable artifacts. If you want to *train* your own skill,
> use `scripts/train.py` per the top-level README.
>
> This is the first optimized-skill artifact batch. We plan to continue
> uploading remaining paper artifacts as they are cleaned and verified. All
> six lightweight ID/path split manifests are already checked in under
> `data/`; most still require materializing their upstream benchmark payload.

## What's here

| Benchmark | Skill artifact | Matching config |
|---|---|---|
| SearchQA | `ckpt/searchqa/gpt5.5_skill.md` | `configs/searchqa/default.yaml` |
| ALFWorld | `ckpt/alfworld/gpt5.5_skill.md` | `configs/alfworld/default.yaml` |
| DocVQA | `ckpt/docvqa/gpt5.5_skill.md` | `configs/docvqa/default.yaml` |
| LiveMathematicianBench | `ckpt/livemath/gpt5.5_skill.md` | `configs/livemathematicianbench/default.yaml` |
| OfficeQA | `ckpt/officeqa/gpt5.5_skill.md` | `configs/officeqa/default.yaml` |
| SpreadsheetBench | `ckpt/spreadsheetbench/gpt5.5_skill.md` | `configs/spreadsheetbench/default.yaml` |

Each file is a plain Markdown skill document (~2k–13k chars). It contains a
protected `SLOW_UPDATE` section at the end that holds epoch-wise
longitudinal guidance — that's expected, not a formatting issue.

## How to evaluate a provided skill

`scripts/eval_only.py` runs a single skill against a data split without
invoking the optimizer. Example for SearchQA against the test split:

```bash
# The checked-in SearchQA split is ID-only; materialize full examples first.
python -m pip install -e ".[searchqa]"
python scripts/materialize_searchqa.py

python scripts/eval_only.py \
  --config configs/searchqa/default.yaml \
  --skill ckpt/searchqa/gpt5.5_skill.md \
  --split valid_unseen \
  --split_dir data/searchqa_split \
  --azure_openai_endpoint https://your-resource.openai.azure.com/ \
  --azure_openai_auth_mode api_key \
  --target_model gpt-5.5
```

Substitute the benchmark, config, skill path, and `--split_dir` to evaluate
any of the other five. `--split valid_unseen` is the test split, `valid_seen`
is the selection / validation split, `train` is the training split, and
`all` runs all three.

## On comparing to the paper numbers

To compare against the paper-reported cells, use the same dataset split and
scorer. SearchQA's ID manifest is checked in at `data/searchqa_id_split/` (400
train / 200 selection / 1400 test); the materializer writes the runnable
payload to `data/searchqa_split/`. All six lightweight split manifests are
checked in under `data/`. ALFWorld's manifest records game-file paths; the
other ID manifests still require you to materialize the corresponding
upstream benchmark payload into the documented `split_dir`. See
[`data/README.md`](../data/README.md) for the exact status of each benchmark.
When using `split_mode: ratio` instead, the loader is deterministic from
`split_seed` (default `42`) + `split_ratio` (default `2:1:7`), so a given
`data_path` + seed reproduces across machines.

## Why force-accept vs. gated slow-update matters

These `ckpt/` skills were produced with the gated slow-update semantics
described in paper Section 3.6:

```yaml
optimizer:
  slow_update_gate_with_selection: true
```

Current `main` defaults to `false` (force-accept mode), a newer
post-submission behavior where the slow-update guidance is written into
`current_skill` and `best_skill` unconditionally at the epoch boundary. If
you re-train with the current default, you may produce a *different*
`best_skill.md` than the one checked in here. Both modes are supported; see
the [configuration reference](../docs/reference/config.md).

---

## File: docs/guide/configuration.md

# Configuration Guide

SkillOpt uses YAML configuration files with a hierarchical override system.

## Config Structure

```
configs/
├── _base_/
│   └── default.yaml          # Global defaults
├── searchqa/
│   └── default.yaml          # SearchQA overrides
├── docvqa/
│   └── default.yaml          # DocVQA overrides
└── alfworld/
    └── default.yaml          # ALFWorld overrides
```

Benchmark configs inherit from `_base_/default.yaml` and override specific values.

## Key Parameters

### Model Backends

`optimizer_backend` controls reflection and skill editing;
`target_backend` controls task rollout. The legacy `backend` field remains for
backward compatibility, but explicit role fields are the clearest configuration.

```yaml
model:
  backend: azure_openai          # High-level compatibility label
  optimizer_backend: openai_chat
  target_backend: openai_chat
  optimizer: gpt-5.5             # Optimizer deployment/model
  target: gpt-5.5                # Target deployment/model
  azure_openai_auth_mode: api_key
```

| Backend | Optimizer | Target | Configuration |
|---|:---:|:---:|---|
| `openai_chat` | ✓ | ✓ | Azure OpenAI, or its explicit compatibility auth mode |
| `openai_compatible` | ✓ | ✓ | Generic OpenAI Chat Completions endpoint |
| `claude_chat` | ✓ | ✓ | Claude Code CLI (`claude -p`) |
| `qwen_chat` | ✓ | ✓ | Qwen served through an OpenAI-compatible endpoint (self-hosted vLLM/SGLang or a hosted gateway) |
| `minimax_chat` | ✓ | ✓ | MiniMax API |
| `copilot_chat` | ✓ | ✓ | GitHub Copilot CLI (`copilot -p`); alias `copilot` |
| `codex_exec` | ✓ | ✓ | Codex CLI execution harness |
| `claude_code_exec` | ✓ | ✓ | Claude Code CLI execution harness |
| `cursor_exec` | — | ✓ | Cursor Agent CLI execution harness |
| `copilot_exec` | — | ✓ | GitHub Copilot CLI execution harness |

`copilot_chat` fills **both** roles through a locally installed,
CLI-authenticated client, so `--backend copilot` runs a complete training loop
without a separate provider API key. Inference still uses the GitHub Copilot
cloud service. Sign in once with `copilot` (GitHub Copilot CLI) beforehand.
Expect roughly 20-40 s per call: the CLI is an agent, not a completions
endpoint, so a full-size run is far slower than a hosted backend. It also
reports no token counts, so usage totals are zero.
It does not support caller-supplied tools or structured tool calls; environments
that require those features must use another chat backend.

The current MiniMax adapter has one shared deployment. Set
`model.minimax_model` when MiniMax is the target; a mixed-backend run cannot
independently select a MiniMax optimizer model and a different target model.

MiniMax is served from region-specific hosts. Select one with
`model.minimax_region` (or `MINIMAX_REGION`) instead of hardcoding a host:
`global_en` (default) resolves to `https://api.minimax.io/v1` and `cn_zh`
resolves to `https://api.minimaxi.com/v1`. An explicit
`model.minimax_base_url` or `MINIMAX_BASE_URL` overrides the region default.

For a generic compatible provider, select the role backends explicitly rather
than relying on a high-level shorthand:

```yaml
model:
  optimizer_backend: openai_compatible
  target_backend: openai_compatible
  optimizer: deepseek-chat
  target: deepseek-chat
```

The train/eval entry points apply `model.optimizer` and `model.target` after
backend initialization. For the selected roles, these YAML values override
`OPENAI_COMPATIBLE_MODEL`, `QWEN_CHAT_MODEL`, and their per-role environment
forms. The environment model variables mainly seed direct library use; always
set the role models in a training or evaluation config.

### Training

```yaml
train:
  num_epochs: 4                  # Number of training epochs
  batch_size: 40                 # Tasks per step (batch size)
  accumulation: 1                # Gradient accumulation
  seed: 42
```

### Gradient (Reflection)

```yaml
gradient:
  minibatch_size: 8              # Reflect minibatch size
  analyst_workers: 16            # Parallel reflection workers
  failure_only: false            # Only reflect on failures
```

### Optimizer

```yaml
optimizer:
  learning_rate: 4               # Max edits per step (edit budget)
  min_learning_rate: 2           # Min edits for decay schedulers
  lr_scheduler: cosine           # constant | linear | cosine | autonomous
  use_slow_update: true          # Momentum-like blending at epoch boundary
  slow_update_samples: 20        # Samples for slow update evaluation
  use_meta_skill: true           # Cross-epoch strategy memory
```

### Skill-Aware Reflection (optional, off by default)

EmbodiSkill-style failure routing: the failure analyst classifies each
failure pattern as **SKILL_DEFECT** (the rule is wrong or missing → normal
gated body edit) or **EXECUTION_LAPSE** (a valid rule exists but was not
followed → a short reminder appended to a protected appendix region inside
the skill that step-level edits can never modify).

```yaml
optimizer:
  use_skill_aware_reflection: false    # Master switch (default off = baseline-identical)
  skill_aware_appendix_source: both    # both | failure_only (paper-faithful S_app)
  skill_aware_consolidate_threshold: 0 # >0: LLM-compact the appendix past N notes (experimental)
```

Notes:

- The switch is resolved process-wide from the config
  (`configure_skill_aware_reflection`), so it applies to every benchmark
  with no per-adapter wiring.
- `failure_only` restricts appendix notes to the failure analyst, matching
  the original S_app formulation; `both` additionally lets the success
  analyst re-emphasize existing rules.
- Appendix notes bypass the validation gate by design and accumulate with
  order-preserving dedup; lapse-only steps (no body edits) still flush
  their notes.
- Not supported together with `skill_update_mode=rewrite_from_suggestions`
  or the full-rewrite modes: whole-document rewrites can drop the appendix
  region.

### Evaluation

```yaml
evaluation:
  use_gate: true                 # Validation gating (accept/reject updates)
  gate_metric: hard              # hard | soft | mixed
  gate_mixed_weight: 0.5         # Soft-score weight when metric=mixed
  use_semantic_density: false    # Optional instruction-density bonus
  eval_test: true                # Run test evaluation after training
```

The default and paper-style setting is `use_gate: true`. Setting it to `false`
still records selection scores but force-accepts every candidate, so it changes
the optimization semantics and should be reported explicitly.

### Environment (Data)

```yaml
env:
  name: searchqa                 # Benchmark name
  split_mode: ratio              # ratio | split_dir
  split_ratio: "2:1:7"           # train:val:test ratio
  data_path: ""                  # Path to dataset
  exec_timeout: 120              # Per-task timeout (seconds)
```

## CLI Overrides

Override any config value from the command line:

```bash
python scripts/train.py \
  --config configs/searchqa/default.yaml \
  --cfg-options \
    optimizer.learning_rate=16 \
    optimizer.lr_scheduler=linear \
    gradient.analyst_workers=8
```

## Environment Variables

Model credentials are loaded from environment variables:

| Variable | Backend | Description |
|---|---|---|
| `AZURE_OPENAI_ENDPOINT` | `openai_chat` | Azure resource URL, or compatibility-mode base URL |
| `AZURE_OPENAI_API_VERSION` | `openai_chat` | Azure API version |
| `AZURE_OPENAI_AUTH_MODE` | `openai_chat` | `api_key`, `azure_cli`, `managed_identity`, or `openai_compatible` |
| `AZURE_OPENAI_API_KEY` | `openai_chat` | Required when auth mode is `api_key` or `openai_compatible` |
| `OPENAI_COMPATIBLE_BASE_URL` | `openai_compatible` | Generic Chat Completions base URL |
| `OPENAI_COMPATIBLE_API_KEY` | `openai_compatible` | Provider API key; optional for local servers |
| `OPENAI_COMPATIBLE_MODEL` | `openai_compatible` | Shared provider model ID for direct library use; train/eval YAML role models take precedence |
| `CLAUDE_CLI_BIN` | `claude_chat` | Optional path to the `claude` executable; defaults to `claude` |
| `ANTHROPIC_API_KEY` | `claude_chat` | Optional authentication method understood by the Claude CLI, not a direct SkillOpt API client |
| `CURSOR_EXEC_PATH` | `cursor_exec` | Optional path to `cursor-agent`; defaults to `cursor-agent` |
| `CURSOR_EXEC_SANDBOX` | `cursor_exec` | Cursor sandbox mode: `enabled` (default) or `disabled` |
| `CURSOR_API_KEY` | `cursor_exec` | Optional authentication method understood directly by Cursor Agent |
| `QWEN_CHAT_BASE_URL` | `qwen_chat` | OpenAI-compatible Qwen endpoint: self-hosted vLLM/SGLang or a hosted gateway |
| `QWEN_CHAT_THINKING_MODE` | `qwen_chat` | `server_default` (default; omit `chat_template_kwargs`), `enabled`, or `disabled`; per-role `OPTIMIZER_`/`TARGET_` variants take precedence |
| `QWEN_CHAT_MODEL` | `qwen_chat` | Served model name for direct library use; train/eval YAML role models take precedence |
| `MINIMAX_REGION` | `minimax_chat` | Service region: `global_en` (default) or `cn_zh`; selects the base URL |
| `MINIMAX_BASE_URL` | `minimax_chat` | MiniMax-compatible base URL; overrides the region default |
| `MINIMAX_API_KEY` | `minimax_chat` | MiniMax API key |
| `COPILOT_EXEC_PATH` | `copilot_chat`, `copilot_exec` | Optional path to `copilot`; defaults to `copilot` |
| `COPILOT_EXEC_HOME` | `copilot_chat`, `copilot_exec` | Optional `COPILOT_HOME` override isolating CLI config; sign-in state lives outside it |
| `COPILOT_EXEC_ALLOW_ALL_TOOLS` | `copilot_exec` | Opt in to `--allow-all-tools` for file-edit rollouts; `false` by default |
| `COPILOT_CHAT_OPTIMIZER_MODEL` | `copilot_chat` | Optional model ID passed as `--model` for optimizer calls |
| `COPILOT_CHAT_TARGET_MODEL` | `copilot_chat` | Optional model ID passed as `--model` for target calls |
| `COPILOT_CHAT_TIMEOUT` | `copilot_chat` | Per-call timeout in seconds |

`OPTIMIZER_` and `TARGET_` prefixes provide per-role overrides for the
Azure, OpenAI-compatible, and Qwen variable families. See the
[Configuration Reference](../reference/config.md) for exact names.

`claude_chat` launches the installed Claude Code CLI with `claude -p`; install
and authenticate that CLI before use. Setting `ANTHROPIC_API_KEY` is one way
the CLI may authenticate, but SkillOpt does not call the Anthropic API
directly through this backend.

`cursor_exec` is a target-only benchmark harness. Install and authenticate
Cursor Agent first, then select it with `model.target_backend=cursor_exec`.
Read-only rollouts use Ask mode; artifact-producing rollouts add Cursor's
headless `--force` flag inside the benchmark workspace. SkillOpt enables the
Cursor sandbox by default and rejects file-edit rollouts if it is disabled;
read-only Ask-mode rollouts may explicitly disable it. SkillOpt does not approve
MCP servers automatically.

`copilot_chat` drives the GitHub Copilot CLI as a chat model for either role.
Because the CLI carries its own sign-in, selecting it for both roles
(`--backend copilot`) avoids separate provider API-key configuration:

```bash
copilot login
skillopt-train --config configs/searchqa/default.yaml --backend copilot
```

Chat calls use an empty tool allowlist and disable built-in MCP servers and
custom instructions, so the model sees only the prompt SkillOpt sends.
`COPILOT_ALLOW_ALL` is removed from child environments, and
`--allow-all-tools` is never passed for chat calls. `copilot_exec` is the
separate target-only harness that runs the CLI as an agent inside a benchmark
workspace; unlike the other exec harnesses it does not grant unattended tool
use unless `COPILOT_EXEC_ALLOW_ALL_TOOLS` is set.

### Three OpenAI-compatible paths

- Research, generic provider: select `openai_compatible` and use
  `OPENAI_COMPATIBLE_*`.
- Research, Azure-family compatibility mode: keep `openai_chat`, set
  `AZURE_OPENAI_AUTH_MODE=openai_compatible`, and use `AZURE_OPENAI_*`.
- SkillOpt-Sleep: run with `--backend azure_openai` and use the same
  compatibility-mode `AZURE_OPENAI_*` variables. Sleep does not read the
  research backend's role-specific variables.

## Full Reference

See [Configuration Reference](../reference/config.md) for the complete parameter list.

---

## File: docs/guide/dl-analogy.md

# Deep Learning ↔ SkillOpt Analogy

SkillOpt is designed around a core insight: **optimizing natural-language prompts follows the same structure as training neural networks**. This page maps every DL concept to its SkillOpt counterpart.

## Complete Mapping

| Deep Learning | SkillOpt | Description |
|---|---|---|
| **Model weights** | Skill document (Markdown) | The thing being optimized |
| **Forward pass** | Rollout | Target executes tasks using current skill |
| **Loss function** | Task evaluator | Scores task execution quality |
| **Backpropagation** | Reflect | Optimizer analyzes failures → edit patches |
| **Gradients** | Edit patches | Proposed changes to the skill |
| **Gradient aggregation** | Patch aggregation | Merge similar edits |
| **Gradient clipping** | Edit selection | Cap max edits per step |
| **Learning rate** | `learning_rate` | Max number of edits applied per step |
| **LR scheduler** | `lr_scheduler` | Edit-budget schedule: cosine, linear, constant, or autonomous |
| **SGD step** | Skill update | Apply selected patches to document |
| **Validation set** | Selection split | Gate checks improvement before accepting |
| **Training step** | Step | One rollout → reflect → update cycle |
| **Epoch** | Epoch | Full pass with slow update + meta memory |
| **Momentum** | Slow update | Longitudinal comparison at epoch boundary |
| **Meta-learning** | Meta skill | Cross-epoch optimizer strategy memory |
| **Batch size** | `batch_size` | Tasks sampled per rollout |
| **Data parallelism** | `analyst_workers` | Parallel reflection workers |
| **Training set** | Train split | Items used for rollout |
| **Test set** | Test split | Held-out final evaluation |
| **Warm-up** | (implicit) | High LR early steps explore broadly |
| **Checkpointing** | Skill snapshots | Saved after each accepted step |
| **Transfer learning** | Seed skill / cross-benchmark init | Start from pre-trained skill |

## Why This Analogy Matters

1. **Familiar mental model**: ML practitioners immediately understand how to tune SkillOpt
2. **Principled hyperparameter search**: Grid search over `learning_rate` × `lr_scheduler` works just like in DL
3. **Reusable mechanisms**: Gating provides validation-based model selection, while slow update plays a momentum-like role across epochs

The gate is a per-candidate accept/reject decision. SkillOpt does not implement
a gate-patience counter or stop training after a run of rejected candidates.

## Hyperparameter Transfer Rules

From our experiments, these DL intuitions transfer well:

!!! success "What transfers"
    - **Cosine schedule > constant** — same as in DL, cosine annealing helps convergence
    - **Moderate LR (4-16) > very high/low** — too few edits = slow learning, too many = noisy
    - **Slow update helps** — longitudinal comparison prevents catastrophic forgetting across epochs
    - **Meta skill memory improves reflection** — optimizer benefits from cross-epoch strategy notes

!!! warning "What doesn't transfer"
    - **Batch size ≠ better** — larger rollout batches have diminishing returns due to API costs
    - **More epochs ≠ better** — skills converge faster than neural networks (2-4 epochs usually enough)

---

## File: docs/guide/first-experiment.md

# Your First Experiment

This guide walks through running a complete SkillOpt training on SearchQA.

## 1. Choose a Benchmark

SkillOpt includes ready-to-use configs for several benchmarks. End-to-end
runtime depends on the chosen models, provider latency, worker limits, and
dataset size, so the project does not promise fixed wall-clock estimates.

| Benchmark | Modality | Additional setup |
|---|---|---|
| SearchQA | Text QA | Materialize the released ID manifest |
| DocVQA | Document/image QA | Obtain and materialize images and examples |
| ALFWorld | Embodied agent | Install ALFWorld and download its assets |

We'll use **SearchQA** because it is the simplest text-only walkthrough.

## 2. Install and Materialize SearchQA

The repository contains a stable SearchQA ID manifest, not the full runnable
examples. From a source checkout, install the data extra and materialize the
split once:

```bash
python -m pip install -e ".[searchqa]"
python scripts/materialize_searchqa.py
```

By default, the materializer reads `data/searchqa_id_split/` and writes the
train/validation/test payloads expected by the config to
`data/searchqa_split/`; both paths have command-line overrides.

## 3. Configure

Configure and export one model backend as described in
[Installation](installation.md#environment-variables). For example:

```bash
cp .env.example .env
# Edit .env, choose one authentication mode, then export it:
set -a; source .env; set +a
```

Review the config file:

```bash
cat configs/searchqa/default.yaml
```

Key parameters (deep learning analogy in parentheses):

```yaml
train:
  num_epochs: 4           # (epochs)
  batch_size: 40          # (batch size)

optimizer:
  learning_rate: 4        # (max edits per step)
  lr_scheduler: cosine    # (learning rate schedule)
  use_slow_update: true   # (momentum at epoch boundary)
  use_meta_skill: true    # (cross-epoch optimizer memory)

gradient:
  analyst_workers: 16     # (parallel reflection workers)

evaluation:
  use_gate: true          # (validation gating)
```

## 4. Train

```bash
python scripts/train.py \
  --config configs/searchqa/default.yaml \
  --out_root outputs/searchqa_first_run
```

The command prints the resolved backend/data configuration, per-step rollout
and gate progress, and the generated output directory.

## 5. Monitor

The explicit `--out_root` above creates this run directory:

```
outputs/searchqa_first_run/
├── config.json
├── runtime_state.json
├── history.json
├── best_skill.md
├── skills/
│   └── skill_vXXXX.md
├── steps/
│   └── step_XXXX/
│       ├── candidate_skill.md
│       ├── step_record.json
│       └── trajectory_digest.json
├── slow_update/
│   └── epoch_XX/
└── meta_skill/
    └── epoch_XX/
```

## 6. Evaluate

Evaluate the best skill on the test split:

```bash
python scripts/eval_only.py \
  --config configs/searchqa/default.yaml \
  --skill outputs/searchqa_first_run/best_skill.md \
  --split valid_unseen
```

The `--skill` path above is the training artifact. Evaluation writes
`eval_summary.json` to its own timestamped `outputs/eval_.../` directory unless
you pass an explicit `--out_root`; it does not overwrite the training run.

## WebUI

Prefer a graphical interface? Launch the WebUI:

```bash
pip install -e ".[webui]"
python -m skillopt_webui.app
```

Then open `http://localhost:7860` in your browser to configure parameters and
launch training. The default host is `0.0.0.0`; pass `--host 127.0.0.1` for a
local-only dashboard.

## Next Steps

- [Understand the training loop](training-loop.md)
- [Configuration reference](../reference/config.md)
- [Add a new benchmark](new-benchmark.md)

---

## File: docs/guide/installation.md

# Installation

## Requirements

- Python ≥ 3.10
- For research training/evaluation, access to at least one configured model
  backend (hosted API, local server, or an installed execution CLI)
- The SkillOpt-Sleep `mock` backend needs no credentials

## Choose an Install

### PyPI

Use PyPI for the Python packages and installed commands:

```bash
python -m pip install skillopt
skillopt-sleep --help
```

This installs `skillopt-train`, `skillopt-eval`, and `skillopt-sleep`. The wheel
does not include the repository's benchmark configs, data materializers,
agent-integration shells/MCP servers, or development tests; use a source
checkout for those files.

!!! important "PyPI versus `main`"
    These docs track the latest `main`. The current PyPI release is `0.2.0`.
    The generic research `openai_compatible` backend, SkillOpt-Sleep handoff,
    Sleep support for non-Azure OpenAI-compatible endpoints, the Sleep
    `--preferences` flag, Cursor source/backend/plugin support, and Pi
    source/backend support, multi-skill fan-out, and reviewed subset adoption
    landed after that release and require a source install from `main` until
    the next release.

### Source checkout

```bash
git clone https://github.com/microsoft/SkillOpt.git
cd SkillOpt
python -m pip install -e .
```

Use the source checkout for paper reproduction, built-in benchmark configs,
and contributions.

## Optional Dependencies

Install extras for specific benchmarks or backends:

=== "ALFWorld"

    ```bash
    python -m pip install -e ".[alfworld]"
    ```

=== "Claude agent SDK (optional)"

    ```bash
    python -m pip install -e ".[claude]"
    ```

    This extra does not install the `claude` executable. The research
    `claude_chat` backend launches `claude -p`, so install and authenticate the
    Claude Code CLI separately. The SDK extra is only needed when selecting an
    SDK-backed Claude Code exec path.

=== "Pi coding-agent CLI (optional)"

    ```bash
    npm install -g --ignore-scripts @earendil-works/pi-coding-agent
    ```

    Install and authenticate the [Pi coding-agent CLI](https://github.com/earendil-works/pi)
    only when using SkillOpt-Sleep with `--backend pi`. Harvesting local Pi
    transcripts with `--source pi` does not require the CLI or provider
    authentication. By default, the source reads below
    `~/.pi/agent/sessions`; `--pi-home` selects the parent directory that
    contains `agent/sessions`.

=== "Qwen (Local)"

    ```bash
    python -m pip install -e ".[qwen]"
    ```

=== "SearchQA data"

    ```bash
    python -m pip install -e ".[searchqa]"
    ```

=== "WebUI"

    ```bash
    python -m pip install -e ".[webui]"
    ```

=== "Development"

    ```bash
    python -m pip install -e ".[dev]"
    ```

=== "All"

    ```bash
    python -m pip install -e ".[alfworld,claude,qwen,searchqa,webui,docs,dev]"
    ```

## Environment Variables

From a source checkout, copy the template and fill in only the backend you
will use:

```bash
cp .env.example .env
```

SkillOpt does not automatically load `.env`; export it into the current shell
before running commands:

```bash
set -a
source .env
set +a
```

For Azure OpenAI with API-key authentication, the minimum settings are:

```ini
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_API_VERSION=2024-12-01-preview
AZURE_OPENAI_API_KEY=your-key
AZURE_OPENAI_AUTH_MODE=api_key
```

Use `AZURE_OPENAI_AUTH_MODE=azure_cli` for Azure CLI credentials, or
`managed_identity` with an optional
`AZURE_OPENAI_MANAGED_IDENTITY_CLIENT_ID`.

The research `claude_chat` backend is a Claude Code CLI adapter, not a direct
Anthropic API client. Install and authenticate `claude`, and set
`CLAUDE_CLI_BIN` only if the executable is not available as `claude` on
`PATH`. `ANTHROPIC_API_KEY` is one authentication option the CLI may consume.

The SkillOpt-Sleep `cursor` backend similarly requires a separately installed
and authenticated `cursor-agent`; harvesting with `--source cursor` alone does
not. Set `SKILLOPT_SLEEP_CURSOR_PATH` when the executable is not on `PATH`, and
`SKILLOPT_SLEEP_CURSOR_MODEL` to override its model. Cursor plugin installation
and the explicit project skill target are documented in the
[Cursor integration guide](https://github.com/microsoft/SkillOpt/blob/main/plugins/cursor/README.md).

OpenAI-compatible servers have three distinct entry points:

1. The research engine's generic `openai_compatible` backend uses
   `OPENAI_COMPATIBLE_BASE_URL`, `OPENAI_COMPATIBLE_API_KEY`, and
   `OPENAI_COMPATIBLE_MODEL`.
2. The research `openai_chat` backend can use
   `AZURE_OPENAI_AUTH_MODE=openai_compatible` with
   `AZURE_OPENAI_ENDPOINT` and `AZURE_OPENAI_API_KEY`.
3. SkillOpt-Sleep uses the same Azure-family variables as item 2 with
   `skillopt-sleep run --backend azure_openai`.

For research train/eval commands, `model.optimizer` and `model.target` in the
YAML config are applied after backend initialization. They override model-name
environment variables such as `OPENAI_COMPATIBLE_MODEL` and
`QWEN_CHAT_MODEL`; set both role models explicitly when selecting those
backends.

!!! tip
    You only need to configure the backend you plan to use. See
    [Configuration](configuration.md#model-backends) for exact backend names
    and role-specific overrides.

## Verify Installation

```bash
python -c "import skillopt; print('SkillOpt ready!')"
skillopt-train --help
skillopt-eval --help
skillopt-sleep --help
```

## Next Steps

→ [Run your first experiment](first-experiment.md)

---

## File: docs/guide/local-env-smoke.md

# Local Environment Smoke Tests

This guide describes a lightweight pattern for testing a custom SkillOpt environment before connecting it to expensive model calls or a full benchmark dataset.

The goal is to validate the training loop plumbing first:

- config loading
- adapter construction
- dataloader splits
- rollout output shape
- reflection patch shape
- merge/rank/update control flow
- artifact creation under `out_root`

Once those are stable, you can switch the same environment to real model calls and larger evaluation splits.

## 1. Add a tiny fixture split

Start with a handful of deterministic examples that cover the expected pass/fail cases for your environment. Keep them small enough that a single training step can run locally.

A minimal fixture item usually needs:

```json
{
  "id": "example-1",
  "split": "train",
  "question": "...",
  "expected": "..."
}
```

Use the split names your adapter maps to SkillOpt phases:

- `train` for optimization rollouts
- `val` or `valid_seen` for selection/gating
- `test` or `valid_unseen` for final evaluation

## 2. Support a genuinely offline mock mode

Add a configuration flag such as `mock: true` to your adapter. In mock mode,
`rollout()` should return deterministic responses without calling external
model APIs. The inherited `EnvAdapter.reflect()` does call the configured
optimizer backend, so a no-credential smoke test must also override
`reflect()` in mock mode to return a small, schema-valid deterministic patch
(and delegate to `super().reflect(...)` otherwise).

This lets you verify the SkillOpt loop with a fast command such as:

```bash
python scripts/train.py \
  --config configs/myenv/tiny_mock.yaml
```

Mock mode should still exercise the trainer's normal artifact paths, including:

- `config.json`, `runtime_state.json`, and `history.json`
- `skills/skill_vXXXX.md`
- `steps/step_XXXX/ranked_edits.json`
- `steps/step_XXXX/candidate_skill.md`
- `steps/step_XXXX/step_record.json`
- the final `summary.json`

## 3. Keep the smoke config tiny

A CI-friendly smoke config should run a single small step:

```yaml
train:
  num_epochs: 1
  train_size: 3
  batch_size: 3

gradient:
  minibatch_size: 1
  merge_batch_size: 2
  analyst_workers: 1

optimizer:
  learning_rate: 1
  min_learning_rate: 1
  lr_scheduler: constant
  skill_update_mode: patch
  use_slow_update: false

evaluation:
  use_gate: true
  sel_env_num: 2
  test_env_num: 2
  eval_test: false

env:
  name: myenv
  out_root: outputs/myenv_tiny_mock
  mock: true
```

Prefer a mock config that runs without credentials. That makes it useful for contributors and CI.

## 4. Validate optimizer JSON before returning it

If your environment or extension asks an LLM to merge or rank skill edits, validate the returned JSON before passing it back into SkillOpt. This avoids silent fallbacks from empty, malformed, or out-of-range responses.

Useful checks for edit payloads:

- response is a JSON object
- `edits` is a non-empty list
- every edit is an object
- every edit has an allowed operation
- required fields such as `content` or `target` are present for that operation

Useful checks for ranking payloads:

- `selected_indices` exists
- indices are integers
- indices are unique
- indices are within the candidate edit range
- selected count does not exceed the edit budget

On failure, retry with a compact prompt that includes the schema error. If retries fail, raise an explicit error instead of silently accepting malformed output.

## 5. Run progressively stronger checks

A good development sequence is:

```bash
python -m py_compile scripts/train.py skillopt/envs/myenv/adapter.py
python scripts/train.py --config configs/myenv/tiny_mock.yaml
python scripts/train.py --config configs/myenv/tiny.yaml
```

For the real tiny run, verify that:

- the run completes
- `summary.json` is written
- the step directory's `ranked_edits.json` contains the expected ranking metadata
- any optimizer bridge log marks the response schema as valid
- no generated files are written outside `out_root`

## 6. Keep custom environments isolated

When adding a custom environment to the registry, avoid side effects for existing benchmarks:

- lazy-import optional dependencies
- install environment-specific hooks only when `cfg["env"]` matches your environment
- keep mock behavior behind an explicit config flag
- write generated artifacts only under `out_root`

This makes it easier to review and test a custom integration without affecting the built-in benchmarks.

---

## File: docs/guide/new-backend.md

# Add a New Model Backend

SkillOpt's model layer is function-based: each chat backend is a Python module
that exposes the call, token-tracking, and deployment-setting functions used by
`skillopt.model`. There is no backend base class or registry object to subclass.

## Built-in: the generic OpenAI-compatible backend

!!! note "Version requirement"
    This backend landed after v0.2.0. Install from the latest `main` until it is
    included in the next release.

Before writing a new backend, check whether your provider already speaks the
OpenAI Chat Completions protocol. Most do, in which case you can use the
built-in **`openai_compatible`** backend
(`skillopt/model/openai_compatible_backend.py`) with no code changes.

A single `base_url` + `api_key` pair lets you point SkillOpt at, for example:

| Provider | `base_url` | Example model |
|---|---|---|
| DeepSeek | `https://api.deepseek.com/v1` | `deepseek-chat` |
| Groq | `https://api.groq.com/openai/v1` | `llama-3.3-70b-versatile` |
| Together AI | `https://api.together.xyz/v1` | `meta-llama/Llama-3.3-70B-Instruct-Turbo` |
| Novita AI | `https://api.novita.ai/openai/v1` | `zai-org/glm-5.2` |
| Ollama (local) | `http://localhost:11434/v1` | `qwen2.5:7b` |
| vLLM / SGLang / TGI | `http://localhost:8000/v1` | your served model |
| LiteLLM proxy | `http://localhost:4000` | any proxied model |
| OpenRouter / Fireworks / xAI / … | provider base URL | provider model id |
| OrcaRouter | `https://api.orcarouter.ai/v1` | `openai/gpt-5.5` |
| Cheaper Inference | `https://api.cheaperinference.com/v1` | `gpt-5.4-mini` |

[OrcaRouter](https://www.orcarouter.ai) exposes OpenAI, Anthropic, Google,
DeepSeek and other models behind one OpenAI-compatible endpoint. Its model IDs
are namespaced by upstream provider (e.g. `openai/gpt-5.5`,
`anthropic/claude-sonnet-4.6`), so pass a namespaced ID as the model.

Cheaper Inference model IDs are not namespaced (e.g. `gpt-5.4-mini`), so pass
the bare ID as the model. See its [API docs](https://www.cheaperinference.com/docs)
for the base URL and its [model list](https://www.cheaperinference.com/markets)
for the model IDs.

### Python API

Select and configure the backend directly when embedding SkillOpt as a Python
library:

```python
import skillopt.model as model

# Use the generic backend for both optimizer and target calls.
model.set_backend("openai_compatible")
model.configure_openai_compatible(
    base_url="https://api.deepseek.com/v1",
    api_key="sk-...",
    model="deepseek-chat",
)
```

`configure_openai_compatible()` also accepts `optimizer_*` and `target_*`
arguments when the two roles use different endpoints or models.

### Environment variables

The shared variables below configure both roles. Role-specific
`OPTIMIZER_OPENAI_COMPATIBLE_*` and `TARGET_OPENAI_COMPATIBLE_*` variables take
precedence:

```bash
export OPENAI_COMPATIBLE_BASE_URL="https://api.groq.com/openai/v1"
export OPENAI_COMPATIBLE_API_KEY="gsk_..."
export OPENAI_COMPATIBLE_MODEL="llama-3.3-70b-versatile"
# Optional: OPENAI_COMPATIBLE_TEMPERATURE, _MAX_TOKENS, _TIMEOUT_SECONDS
```

For direct library use, `OPTIMIZER_BACKEND=openai_compatible` and/or
`TARGET_BACKEND=openai_compatible` select the role. The training and evaluation
scripts resolve backend selection from their config, so set the split fields
explicitly there:

```yaml
model:
  optimizer_backend: openai_compatible
  target_backend: openai_compatible
  optimizer: llama-3.3-70b-versatile
  target: llama-3.3-70b-versatile
```

Equivalently, override those fields on the command line:

```bash
python scripts/train.py --config configs/searchqa/default.yaml \
  --cfg-options \
  model.optimizer_backend=openai_compatible \
  model.target_backend=openai_compatible \
  model.optimizer=llama-3.3-70b-versatile \
  model.target=llama-3.3-70b-versatile
```

`model.backend` is a high-level shorthand for SkillOpt's built-in role
pairings. It replaces role fields that are still at the shipped
`openai_chat` default, while non-default role fields are preserved. For a
custom optimizer/target split, set both role-specific fields and do not combine
them with a conflicting high-level label.

The generic backend uses the official `openai` SDK and the Chat Completions
API. It records token usage through the shared tracker, supports provider tool
calling through `chat_*_messages(..., tools=...)`, and exposes `count_tokens()`
(tiktoken when available, with a character-based fallback). Provider-specific
Responses API features are outside this backend's contract.

Only write a new backend when the provider is not compatible with this surface
or requires behavior that cannot be expressed by its configuration.

## Backend architecture

The active split optimizer/target dispatcher is the public
`skillopt/model/__init__.py` module:

```text
skillopt/model/
├── common.py                       # aliases, default models, token/response helpers
├── backend_config.py               # optimizer/target whitelists and runtime selection
├── __init__.py                     # public API and split-role dispatch
├── openai_compatible_backend.py    # generic Chat Completions example
├── qwen_backend.py                 # raw-HTTP chat example with per-role config
├── minimax_backend.py              # compact raw-HTTP chat example
├── codex_harness.py                # target-only exec harnesses
└── router.py                       # legacy single-backend compatibility surface
```

`router.py` is not the dispatcher used by the current training loop. Update it
only if the new backend must also be exposed through that legacy single-backend
API.

## Step 1: implement the module contract

Create a module such as `skillopt/model/your_backend.py`. Copy the signatures
from `openai_compatible_backend.py` or `qwen_backend.py`; model calls in the
current framework are synchronous.

For a chat backend that supports both roles, the public module surface is:

| Function | Purpose |
|---|---|
| `chat_optimizer(...)` | Optimizer system/user call; returns `(text, usage)` |
| `chat_target(...)` | Target system/user call; returns `(text, usage)` |
| `chat_optimizer_messages(...)` | Optimizer message-list call, including optional tools |
| `chat_target_messages(...)` | Target message-list call, including optional tools |
| `get_token_summary()` | Return per-stage counters plus `_total` |
| `reset_token_tracker()` | Clear this backend's counters |
| `set_optimizer_deployment(name)` | Change the optimizer model at runtime |
| `set_target_deployment(name)` | Change the target model at runtime |
| `set_reasoning_effort(effort)` | Apply or safely ignore the shared reasoning setting |

Every call returns a usage dict with `prompt_tokens`, `completion_tokens`, and
`total_tokens`. Use `TokenTracker` from `skillopt.model.common` and record each
call exactly once. Message-list calls that accept tools should return the
compatibility message objects from `common.py` when `return_message=True`.

Provider-specific configuration helpers and `count_tokens()` are optional, but
their state must be safe to update while calls may run concurrently. Keep
credentials out of logs and persisted artifacts.

Exec-style targets such as `claude_code_exec` and `cursor_exec` do not
implement this chat contract. They are target-only and are integrated through
`codex_harness.py` plus environment-specific rollout code.

## Step 2: register and route the backend

A new backend normally requires all of the following:

1. Add its canonical name, aliases, and default model to
   `skillopt/model/common.py`.
2. Add the canonical name to the appropriate optimizer and/or target whitelist
   in `skillopt/model/backend_config.py`. Do not advertise a role the module
   cannot execute.
3. Import the module in `skillopt/model/__init__.py` and add dispatch branches
   for every supported call surface.
4. Include its counters in `get_token_summary()` / `reset_token_tracker()` and
   forward the shared deployment/reasoning setters where applicable.
5. If it has YAML settings, add structured-to-flat mappings in
   `skillopt/config.py`, wire them through `scripts/train.py` and
   `scripts/eval_only.py`, and document their precedence over environment
   variables.
6. Update `router.py` only when legacy single-backend compatibility is part of
   the intended feature.

Backend selection in `scripts/train.py` must support
`model.optimizer_backend` and `model.target_backend`. If the backend also has a
high-level `model.backend` alias, resolve its built-in role pairing before
applying backend-specific model defaults, without replacing non-default role
fields.

## Step 3: test the integration

Add focused tests under `tests/` that do not call a live provider. At minimum,
cover:

- optimizer and target whitelist validation;
- routing for text and message-list calls;
- role-specific configuration precedence;
- tool-call compatibility, if supported;
- deployment/reasoning setters;
- token accounting, including a single correct `_total`;
- actionable errors for missing credentials or invalid responses.

Then run the focused test, the full suite, and the documentation build:

```bash
python -m pytest tests/test_your_backend.py -q
python -m pytest tests/ -q
mkdocs build --strict
```

Also update `.env.example`, the configuration reference, and the backend table
in the API reference. Add an optional dependency extra only when the backend
requires a package that is not already a core dependency.

---

## File: docs/guide/new-benchmark.md

# Add a New Benchmark

Extend SkillOpt with your own benchmark in ~200 lines of code. We will use
a tiny worked example, `docfaithful`, that scores a target model on
how faithfully it answers questions grounded in a small reference doc.

> **Working reference.** The easiest way to copy-cargo-cult a new env is
> to read [`skillopt/envs/officeqa/`](https://github.com/microsoft/SkillOpt/tree/main/skillopt/envs/officeqa).
> Everything below is the same shape, simplified.

## What you need to build

To add a benchmark you implement four things:

1. **A `SplitDataLoader` subclass** — knows how to load train / val / test
   item dicts from disk.
2. **A rollout helper** — runs the target model on a batch of items, scores
   each prediction, and persists the per-item conversation consumed by the
   shared reflection stage.
3. **An `EnvAdapter` subclass** — wires the loader + rollout helper into
   SkillOpt's lifecycle (`build_*_env`, `rollout`, and `get_task_types`).
   The shared `reflect()` implementation is inherited unless the benchmark
   needs custom reflection logic.
4. **A YAML config** — references your env name plus the standard
   train / optimizer / gradient knobs.

Then lazy registration in the training and evaluation scripts makes it
discoverable without importing optional dependencies at startup.

---

## Step 1 — Create the package

```bash
mkdir -p skillopt/envs/docfaithful
touch skillopt/envs/docfaithful/__init__.py
```

## Step 2 — Implement the data loader

`skillopt/envs/docfaithful/dataloader.py`:

```python
from __future__ import annotations

import json
from pathlib import Path

from skillopt.datasets.base import SplitDataLoader


def _normalize(raw: dict) -> dict:
    """Make sure every item has an ``id``. Other keys are env-specific."""
    return {
        "id": str(raw["uid"]),
        "question": raw["question"],
        "ground_truth": raw["answer"],
        "reference_text": raw.get("reference", ""),
        "task_type": raw.get("category", "docfaithful"),
    }


class DocFaithfulDataLoader(SplitDataLoader):
    """Load DocFaithful items from JSON files inside each split dir."""

    def load_split_items(self, split_path: str) -> list[dict]:
        # split_path is e.g. data/docfaithful_split/train/
        json_files = sorted(Path(split_path).glob("*.json"))
        if not json_files:
            raise FileNotFoundError(f"No .json file found in {split_path}")
        with json_files[0].open(encoding="utf-8") as f:
            raw = json.load(f)
        return [_normalize(item) for item in raw]
```

Only `load_split_items()` is mandatory. If you also want to support
`split_mode="ratio"` (auto-split a single raw file into train/val/test),
override `load_raw_items(data_path)` as well — see
`skillopt/datasets/base.py` docstrings.

## Step 3 — Write the rollout helper

`skillopt/envs/docfaithful/rollout.py`:

```python
from __future__ import annotations

import json
import os
from pathlib import Path

from skillopt.model import chat_target


def _score(prediction: str, ground_truth: str) -> tuple[int, float]:
    """Trivial exact-match scorer. Replace with F1 / ROUGE / LLM-judge."""
    p = (prediction or "").strip().lower()
    g = (ground_truth or "").strip().lower()
    hard = int(p == g and bool(g))
    soft = 1.0 if hard else 0.0
    return hard, soft


def _rollout_one(item: dict, skill_content: str, *, prediction_dir: Path,
                 max_completion_tokens: int) -> dict:
    system = skill_content
    user = (
        f"Question: {item['question']}\n\n"
        f"Reference:\n{item.get('reference_text', '')}\n\n"
        "Answer:"
    )
    prediction, _usage = chat_target(
        system=system,
        user=user,
        max_completion_tokens=max_completion_tokens,
    )
    hard, soft = _score(prediction, item.get("ground_truth", ""))

    # EnvAdapter.reflect() reads this exact trajectory path. Keep item IDs
    # unique and filesystem-safe.
    task_dir = prediction_dir / str(item["id"])
    task_dir.mkdir(parents=True, exist_ok=True)
    conversation = [
        {"role": "system", "content": system},
        {"role": "user", "content": user},
        {"role": "assistant", "content": prediction},
    ]
    (task_dir / "conversation.json").write_text(
        json.dumps(conversation, ensure_ascii=False, indent=2),
        encoding="utf-8",
    )

    return {
        "id": str(item["id"]),
        "hard": hard,
        "soft": soft,
        "predicted_answer": prediction,
        "task_description": item.get("question", ""),
        "question": item.get("question", ""),
        "reference_text": item.get("reference_text", ""),
        "task_type": item.get("task_type", "docfaithful"),
        "target_system_prompt": system,
        "target_user_prompt": user,
        "n_turns": 1,
    }


def run_batch(*, items: list[dict], skill_content: str, out_root: str,
              workers: int = 4, max_completion_tokens: int = 4096) -> list[dict]:
    """Run a batch of episodes sequentially or with a thread pool."""
    os.makedirs(out_root, exist_ok=True)
    prediction_dir = Path(out_root, "predictions")
    # For brevity we go sequentially — swap in concurrent.futures.ThreadPoolExecutor
    # when network / model latency dominates.
    results = [
        _rollout_one(item, skill_content,
                     prediction_dir=prediction_dir,
                     max_completion_tokens=max_completion_tokens)
        for item in items
    ]
    Path(out_root, "rollouts.json").write_text(
        json.dumps(results, ensure_ascii=False, indent=2),
        encoding="utf-8",
    )
    return results
```

Two design points worth flagging:

- **Scoring lives here, not in `EnvAdapter`.** There is no `evaluate()`
  method on the ABC. Whatever signal you put in `hard` (0/1, or a float
  in [0, 1] for smoothed reward) and `soft` (float in [0, 1]) is what
  the optimizer reads.
- **Use `skillopt.model.chat_target`**, not raw OpenAI/Claude calls.
  That routes through whichever **chat** target backend the user
  configured (`openai_chat` / `claude_chat` / `qwen_chat` /
  `minimax_chat` / `openai_compatible`) without your adapter caring.
  Exec-style backends (`codex_exec`, `claude_code_exec`) need
  environment-specific rollout code —
  see `skillopt/model/codex_harness.py` together with the rollout modules in
  `skillopt/envs/searchqa/`, `skillopt/envs/docvqa/`, or
  `skillopt/envs/officeqa/` for working examples.
- **Persist a conversation for reflection.** The shared `EnvAdapter.reflect()`
  looks under `<rollout_dir>/predictions/<result-id>/conversation.json` and
  skips results whose trajectory is absent or empty. Returning `hard`/`soft`
  scores alone is sufficient for evaluation, but it cannot produce learning
  patches.

## Step 4 — Implement the environment adapter

`skillopt/envs/docfaithful/adapter.py`:

```python
from __future__ import annotations

from skillopt.datasets.base import BatchSpec
from skillopt.envs.base import EnvAdapter
from skillopt.envs.docfaithful.dataloader import DocFaithfulDataLoader
from skillopt.envs.docfaithful.rollout import run_batch


class DocFaithfulAdapter(EnvAdapter):
    """SkillOpt adapter for the DocFaithful benchmark."""

    def __init__(
        self,
        split_dir: str = "",
        data_path: str = "",
        split_mode: str = "split_dir",
        split_ratio: str = "2:1:7",
        split_seed: int = 42,
        split_output_dir: str = "",
        workers: int = 4,
        analyst_workers: int = 4,
        failure_only: bool = False,
        minibatch_size: int = 8,
        edit_budget: int = 4,
        seed: int = 42,
        limit: int = 0,
        max_completion_tokens: int = 4096,
    ) -> None:
        self.workers = workers
        self.analyst_workers = analyst_workers
        self.failure_only = failure_only
        self.minibatch_size = minibatch_size
        self.edit_budget = edit_budget
        self.max_completion_tokens = int(max_completion_tokens)
        self.dataloader = DocFaithfulDataLoader(
            split_dir=split_dir,
            data_path=data_path,
            split_mode=split_mode,
            split_ratio=split_ratio,
            split_seed=split_seed,
            split_output_dir=split_output_dir,
            seed=seed,
            limit=limit,
        )

    # ── Lifecycle ───────────────────────────────────────────────────────

    def setup(self, cfg: dict) -> None:
        super().setup(cfg)
        self.dataloader.setup(cfg)

    def get_dataloader(self):
        return self.dataloader

    # ── Env construction ────────────────────────────────────────────────

    def build_env_from_batch(self, batch: BatchSpec, **kwargs):
        # For dataset-backed envs the "manager" is just the items list.
        return list(batch.payload or [])

    def build_train_env(self, batch_size: int, seed: int, **kwargs):
        batch = self.dataloader.build_train_batch(
            batch_size=batch_size, seed=seed, **kwargs
        )
        return self.build_env_from_batch(batch, **kwargs)

    def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
        batch = self.dataloader.build_eval_batch(
            env_num=env_num, split=split, seed=seed, **kwargs
        )
        return self.build_env_from_batch(batch, **kwargs)

    # ── The rollout method (reflect is inherited) ───────────────────────

    def rollout(self, env_manager, skill_content: str,
                out_dir: str, **kwargs) -> list[dict]:
        items: list[dict] = env_manager
        return run_batch(
            items=items,
            skill_content=skill_content,
            out_root=out_dir,
            workers=self.workers,
            max_completion_tokens=self.max_completion_tokens,
        )

    # reflect() is inherited from EnvAdapter — it delegates to
    # run_minibatch_reflect with your analyst_error_* / analyst_success_*
    # prompts. Override it only if you need custom reflection logic.

    def get_task_types(self) -> list[str]:
        seen: list[str] = []
        for item in (
            self.dataloader.train_items
            + self.dataloader.val_items
            + self.dataloader.test_items
        ):
            tt = str(item.get("task_type") or "docfaithful")
            if tt not in seen:
                seen.append(tt)
        return seen or ["docfaithful"]
```

### What the rollout actually does

Look back at `run_batch` from Step 3 — it sends each `item["question"]`
to the target model with `skill_content` as the system prompt, scores
the answer against `item["ground_truth"]`, and returns a list of dicts:

```python
[
    {"id": "ex_001", "hard": 1, "soft": 0.92,
     "predicted_answer": "...", "question": "...",
     "reference_text": item["reference_text"]},
    {"id": "ex_002", "hard": 0, "soft": 0.13, "fail_reason": "...", ...},
    ...
]
```

The trainer requires `id`, `hard`, and `soft` for scoring. The remaining fields
are preserved on `RolloutResult.extras` (see `skillopt/types.py`). The shared
reflection implementation combines those fields with each persisted
`predictions//conversation.json`; without that file the result is omitted
from reflection.

## Step 5 — Register the adapter

Edit [`scripts/train.py`](https://github.com/microsoft/SkillOpt/blob/main/scripts/train.py)
and add to `_register_builtins()`:

```python
    try:
        from skillopt.envs.docfaithful.adapter import DocFaithfulAdapter
        _ENV_REGISTRY["docfaithful"] = DocFaithfulAdapter
    except ImportError:
        pass  # docfaithful deps not installed — skip
```

Mirror the same lazy registration in
[`scripts/eval_only.py`](https://github.com/microsoft/SkillOpt/blob/main/scripts/eval_only.py)
so standalone evaluation can resolve the environment too. There is **no
`BENCHMARK_REGISTRY` dict in `skillopt/envs/__init__.py`**; both entry points
keep a small lazy registry so optional dependencies do not break `--help`.

## Step 6 — Create the YAML config

`configs/docfaithful/default.yaml`:

```yaml
_base_: ../_base_/default.yaml      # NOTE: string, not list

model:
  reasoning_effort: medium

train:
  batch_size: 16
  accumulation: 1
  num_epochs: 4

gradient:
  minibatch_size: 8
  merge_batch_size: 8

optimizer:
  learning_rate: 4

env:
  name: docfaithful
  # Point to an existing Markdown file. Use an empty file to start blank.
  skill_init: skillopt/envs/docfaithful/skills/initial.md
  split_mode: split_dir
  split_dir: data/docfaithful_split
  workers: 4
  max_completion_tokens: 4096
  limit: 0
```

> ⚠️ `_base_` is currently parsed as a **string path**, not a list. Write
> `_base_: ../_base_/default.yaml`, not `_base_: ['../_base_/default.yaml']`.
> See [`skillopt/config.py`](https://github.com/microsoft/SkillOpt/blob/main/skillopt/config.py)
> if you want to add list-form inheritance.

## Step 7 — Run

```bash
# Create the file referenced by env.skill_init before the first run:
#   mkdir -p skillopt/envs/docfaithful/skills
#   echo "# DocFaithful initial skill" > skillopt/envs/docfaithful/skills/initial.md

python scripts/train.py --config configs/docfaithful/default.yaml
```

If you get `ValueError: Unknown environment 'docfaithful'. Available: [...]`,
you forgot Step 5.

If you get `TypeError: Can't instantiate abstract class DocFaithfulAdapter`,
you forgot to implement one of the four abstract methods on `EnvAdapter`:
`build_train_env`, `build_eval_env`, `rollout`, `get_task_types`.

## Tips

- Start with `train.batch_size: 4` and `limit: 10` while debugging.
- The `evaluate` half lives **inside your `rollout`**, not as a separate
  method — there is no `evaluate()` in the `EnvAdapter` ABC. Score the
  prediction in `run_batch` and put the score on each result dict's
  `hard` / `soft`.
- Noisy scoring kills the optimizer. Spend time on `run_batch`'s scoring
  before you spend time on prompts.
- If training repeatedly reports `skip_no_patches`, first verify that every
  rollout result has a non-empty
  `rollout/predictions//conversation.json` using the same `id` string.
- If your benchmark needs heavy optional deps (selenium, vllm, ...),
  wrap both registration blocks with `try / except ImportError` (Step 5)
  so people without those deps can still `--help`.
- Copy `skillopt/envs/_template/` as a starting skeleton — it now
  implements the real abstract methods.

---

## File: docs/guide/skill-document.md

# Skill Document

A **skill document** is a Markdown file that serves as the "prompt weights" of your agent. SkillOpt trains this document through iterative optimization.

## What is a Skill Document?

A skill document is a structured set of instructions that tells a language model **how** to approach a specific type of task. It's analogous to learned weights in a neural network — encoding task-specific knowledge in natural language rather than floating-point parameters.

## Structure

A typical skill document contains:

```markdown
# Task Strategy

## General Approach
- Break complex problems into sub-steps
- Always verify intermediate results

## Common Patterns
- When you see X, try approach Y
- Avoid Z because it leads to errors

## Edge Cases
- If the input contains A, handle it specially by...
- Watch out for B — it requires C

## Output Format
- Always include reasoning before the answer
- Format numbers with proper units
```

## How It Evolves

During training, the skill document is modified by **edit patches**:

1. **Additions**: New rules or strategies discovered from failed trajectories
2. **Modifications**: Refining existing rules that are partially correct
3. **Deletions**: Removing rules that consistently lead to errors

Selected edits are applied together to produce a candidate skill. With the
validation gate enabled, that candidate replaces the current skill only when
its score on the selection split strictly improves.

SkillOpt may maintain two protected, machine-managed regions:

```markdown
<!-- SLOW_UPDATE_START -->
... epoch-level longitudinal guidance ...
<!-- SLOW_UPDATE_END -->

<!-- APPENDIX_START -->
... skill-aware execution reminders ...
<!-- APPENDIX_END -->
```

Normal edit patches cannot modify either region. Slow update owns the first;
optional skill-aware reflection owns the second. Preserve these markers when
copying or manually inspecting a trained skill.

## Initial Skill

You can start training with:

- **Empty skill**: Point `env.skill_init` to an empty Markdown file
- **Seed skill**: Provide initial instructions to bootstrap training
- **Pre-trained skill**: Transfer a skill from a related benchmark

Configure the initial skill in your YAML:

```yaml
env:
  skill_init: path/to/initial_skill.md
```

To start from scratch, create an empty Markdown file and use its path. A missing
path currently also starts blank, so using an explicit file avoids silently
treating a typo as an empty skill.

## Skill Quality Metrics

Track your skill's evolution through:

- **Validation score**: Primary metric on the selection split
- **Test score**: Final metric on held-out test data
- **Skill length**: Total tokens in the document
- **Candidate acceptance rate**: Fraction of candidate skill updates that pass
  gating; multiple proposed edits can be combined into one candidate

## Best Practices

!!! tip "Tips for better skills"
    1. **Start with a seed skill** (`env.skill_init`) if you have domain knowledge — it converges faster
    2. **Use cosine LR schedule** — aggressive early exploration + careful late refinement
    3. **Enable slow update** (`optimizer.use_slow_update: true`) to counter forgetting across epochs
    4. **Enable meta skill** (`optimizer.use_meta_skill: true`) so the optimizer accumulates strategy memory

## Next Steps

- [Deep Learning Analogy](dl-analogy.md)
- [Configuration Reference](../reference/config.md)

---

## File: docs/guide/training-loop.md

# The Training Loop

SkillOpt's core insight: **optimizing natural-language skill documents follows the same structure as training neural networks**.

## Overview

```
┌─────────────────────────────────────────────────────────┐
│                    Training Loop                         │
│                                                         │
│  for epoch in epochs:                                   │
│    for step in steps:                                   │
│      1. Rollout   — Target executes tasks              │
│      2. Reflect   — Optimizer analyzes trajectories       │
│      3. Aggregate — Hierarchical merge of patches       │
│      4. Select    — Rank & clip edits (learning rate)   │
│      5. Update    — Apply patches to skill doc          │
│      6. Gate      — Validate & accept/reject            │
│                                                         │
│    Epoch Boundary:                                       │
│      • Slow Update (longitudinal comparison & guidance) │
│      • Meta Skill  (cross-epoch strategy memory)        │
└─────────────────────────────────────────────────────────┘
```

## Stage Details

### 1. Rollout (Forward Pass)

The **target** model executes tasks using the current skill document as its prompt. Each task produces a trajectory and a score.

```python
# Analogy: forward pass through the network
predictions = model(input, skill_document)
scores = evaluate(predictions, ground_truth)
```

### 2. Reflect (Backward Pass)

The **optimizer** model analyzes trajectory minibatches and produces **edit
patches** — structured suggestions for improving the skill document. Failure
minibatches are always eligible for analysis; successful trajectories are also
analyzed unless `gradient.failure_only` is enabled. Independent minibatches can
run concurrently according to `gradient.analyst_workers`.

```python
# Analogy: computing gradients
gradients = loss.backward()  # → edit patches
```

### 3. Aggregate

Semantically similar edit patches are merged to avoid redundant edits.

### 4. Select (Gradient Clipping)

Edits are ranked by relevance score. The `learning_rate` parameter caps how many edits are applied per step — just like gradient clipping prevents overshooting.

```python
# Analogy: gradient clipping + optimizer step size
selected = top_k(edits, k=learning_rate)
```

The `lr_scheduler` adjusts this over training:

- **cosine**: Start aggressive, taper smoothly
- **linear**: Linear decay
- **constant**: Fixed rate

### 5. Update (Parameter Update)

Selected edits are applied to the skill document, producing a new version.

### 6. Gate (Validation)

The updated skill is evaluated on a **selection split** (analogous to a
validation set). With the gate enabled, the candidate is accepted only when its
configured gate score (`hard`, `soft`, or `mixed`) is strictly higher than the
current skill's score. With `evaluation.use_gate: false`, validation is still
recorded but candidates are force-accepted.

#### Why current and best are separate

On the strictly gated path, the current and best skills are initialized to the
same document and score. Every accepted candidate must beat the current score,
which is also the best score, so both states advance together. Thus
`current_skill == best_skill` remains an invariant for the procedure described
by the paper; the nested "new best" check is redundant on that path.

The implementation keeps separate current and best snapshots for optional
extensions where they can diverge. Disabling the gate can force-accept a new
current skill without replacing the validation-best snapshot, and an ungated
slow update can likewise modify only the current skill. The final evaluation
reports the validation-best snapshot separately from the last current skill.

## Epoch Boundary Mechanisms

### Slow Update

At the end of each epoch (starting from epoch 2), the system performs a
**longitudinal comparison**: it rolls out both the previous epoch's skill and
the current skill on the same samples, categorizes items as
improved/regressed/persistent-fail/stable-success, then generates high-level
**guidance** for the skill document. Depending on
`optimizer.slow_update_gate_with_selection`, that guidance is either checked on
the selection split or applied unconditionally. Its purpose is to counter
cross-epoch forgetting.

### Meta Skill

A **meta-skill memory** accumulates high-level strategy notes across the training
run. Starting at the end of epoch 2, the optimizer compares the previous and
current epoch, writes a compact memory, and provides the prior epoch's memory as
additional context during later reflection and update stages.

## Next Steps

- [Understand Skill Documents](skill-document.md)
- [DL ↔ SkillOpt analogy table](dl-analogy.md)

---

## File: docs/reference/api.md

# API Reference

This page documents the public Python API SkillOpt exposes for **extending the
framework** with new environments / benchmarks. For ready-made adapters,
browse [`skillopt/envs/`](https://github.com/microsoft/SkillOpt/tree/main/skillopt/envs).

> **Source of truth.** The classes below are real Python ABCs defined in
> `skillopt/envs/base.py`, `skillopt/datasets/base.py`, `skillopt/types.py`,
> and `skillopt/evaluation/gate.py`. If this page ever drifts, the code
> wins — please open an issue.

---

## Core Classes

### `EnvAdapter`

`skillopt/envs/base.py` — abstract adapter that connects the SkillOpt
trainer to an environment (benchmark, simulator, REST API, ...).
Subclasses **must** implement the four abstract methods below. Reflection has a
shared default implementation and only needs to be overridden for
environment-specific behavior.

```python
from abc import ABC, abstractmethod
from skillopt.datasets.base import BaseDataLoader, BatchSpec

class EnvAdapter(ABC):

    # ── Lifecycle hooks (have defaults; override only if needed) ────────

    def setup(self, cfg: dict) -> None: ...
    def get_dataloader(self) -> BaseDataLoader | None: ...
    def requires_ray(self) -> bool: ...                 # default False
    def reflect(self, results: list[dict], skill_content: str,
                out_dir: str, **kwargs) -> list[dict | None]:
        """Delegate to the shared minibatch reflection pipeline."""
        ...

    # ── Abstract methods (subclasses MUST implement) ────────────────────

    @abstractmethod
    def build_train_env(self, batch_size: int, seed: int, **kwargs):
        """Return an environment-manager object to be passed to rollout()."""

    @abstractmethod
    def build_eval_env(self, env_num: int, split: str, seed: int, **kwargs):
        """Like build_train_env() but for a fixed eval split."""

    @abstractmethod
    def rollout(self, env_manager, skill_content: str,
                out_dir: str, **kwargs) -> list[dict]:
        """Run a batch of episodes with the current skill.

        Each returned dict MUST contain:
          - "id":   str        episode/task identifier
          - "hard": int (0|1)  pass/fail (may be float 0.0-1.0 if smoothed)
          - "soft": float      partial-credit score in [0.0, 1.0]
        It MAY contain env-specific extra keys (parsed into RolloutResult.extras).
        """

    @abstractmethod
    def get_task_types(self) -> list[str]:
        """Distinct task-type strings used for stratified sampling."""
```

The default `reflect()` delegates to `run_minibatch_reflect` and returns raw
patch dicts with a `patch` payload plus a `failure` or `success` source type.
It expects each rollout to persist a non-empty trajectory at
`<rollout_dir>/predictions/<result-id>/conversation.json`; results without that
file can be scored but are skipped during reflection.
The trainer also calls several default-implemented helpers on every adapter:
`build_reference_text`, `get_reference_metadata`, `attach_reference_context`,
`select_representative_items`, and `build_env_from_batch`. Read the docstrings
in `skillopt/envs/base.py` if you need to override any of these — most
benchmarks do not.

### `BaseDataLoader` / `SplitDataLoader`

`skillopt/datasets/base.py` — episode-planning loaders.

```python
class BaseDataLoader(ABC):
    def setup(self, cfg: dict) -> None: ...
    @abstractmethod
    def build_train_batch(self, batch_size: int, seed: int, **kwargs) -> BatchSpec: ...
    @abstractmethod
    def build_eval_batch(self, env_num: int, split: str, seed: int, **kwargs) -> BatchSpec: ...

class SplitDataLoader(BaseDataLoader):
    """Concrete base for dataset-backed envs with on-disk train/val/test splits.

    Subclasses only need to implement load_split_items() (and optionally
    load_raw_items() if you also want ``split_mode='ratio'``).
    """
    def load_split_items(self, split_path: str) -> list[dict]: ...
    def load_raw_items(self, data_path: str) -> list[dict]: ...   # optional
```

`SplitDataLoader` handles two layout modes:

| `split_mode` | What it expects |
|---|---|
| `"split_dir"` | A directory with `train/`, `val/`, `test/` subdirs already split. |
| `"ratio"` | A raw dataset path + `split_ratio: "2:1:7"` style string. |

In either case the items returned by `load_split_items()` are plain
`dict` objects with at minimum an `"id"` key.

### `BatchSpec`

`skillopt/datasets/base.py` — a slotted dataclass describing one batch
request the trainer hands to the adapter.

```python
@dataclass(slots=True)
class BatchSpec:
    phase: str                 # "train" | "eval"
    split: str                 # "train" | "val" | "test" | "valid_seen" | ...
    seed: int
    batch_size: int
    payload: object | None = None     # what the loader produced (e.g. list[dict])
    metadata: dict = field(default_factory=dict)
```

### `Edit` / `Patch`

`skillopt/types.py` — the I/O types Reflect / Aggregate / Update produce
and consume.

```python
EditOp = Literal["append", "insert_after", "replace", "delete"]

@dataclass
class Edit:
    op: EditOp
    content: str = ""
    target: str = ""
    support_count: int | None = None
    source_type: Literal["failure", "success"] | None = None
    merge_level: int | None = None
    update_origin: str = ""
    update_target: str = ""

@dataclass
class Patch:
    edits: list[Edit] = field(default_factory=list)
    reasoning: str = ""
    ranking_details: dict[str, Any] | None = None
```

Both types support `to_dict()` / `from_dict()` for serialization.

### `RolloutResult`

`skillopt/types.py` — the normalised rollout return type. The trainer
calls `RolloutResult.from_dict(...)` on each dict returned from
`EnvAdapter.rollout()`, so the only **hard** requirement on those dicts is
the three keys above (`id`, `hard`, `soft`). Extra fields are preserved
into `RolloutResult.extras`.

### `GateResult` / `GateAction`

`skillopt/evaluation/gate.py` — the validation-gate decision types
returned for each candidate optimization step, and optionally for a separate
epoch-end slow-update candidate.

---

## Registering an environment

Environments are not registered via decorators or a `BENCHMARK_REGISTRY`
dict. The training and standalone-evaluation entry points each keep a lazy
`_ENV_REGISTRY`, populated by `_register_builtins()` in `scripts/train.py` and
`scripts/eval_only.py`. Add the environment to both entry points. See
[Add a New Benchmark](../guide/new-benchmark.md) for the full step-by-step.

---

## Backends (model layer)

The model layer lives under `skillopt.model.*`. Backends are selected
via `model.optimizer_backend` and `model.target_backend` in the config —
not via a base class subclass. Supported values (as of this writing):

| Backend | Optimizer? | Target? |
|---|---|---|
| `openai_chat` | ✓ | ✓ |
| `claude_chat` | ✓ | ✓ |
| `qwen_chat` | ✓ | ✓ |
| `minimax_chat` | ✓ | ✓ |
| `copilot_chat` | ✓ | ✓ |
| `openai_compatible` | ✓ | ✓ |
| `codex_exec` | ✓ | ✓ |
| `claude_code_exec` | — | ✓ |
| `cursor_exec` | — | ✓ |
| `copilot_exec` | — | ✓ |

See `skillopt/model/backend_config.py` for the live whitelist and
[`docs/reference/config.md`](./config.md) for the per-backend
configuration keys.

---

## File: docs/reference/cli.md

# CLI Reference

> **Version note.** This reference tracks `main`. PyPI 0.2.0 does not yet
> include the generic research `openai_compatible` backend, Sleep handoff,
> Sleep support for non-Azure OpenAI-compatible endpoints, the Sleep
> `--preferences` flag, the research `cursor_exec` target harness, or Cursor
> source/backend/plugin support, Pi source/backend support, OpenCode Sleep
> source/backend support, VS Code Copilot transcript harvesting, or multi-skill
> fan-out and subset adoption; use a source install from `main` for those
> features until the next release.

## Training

```bash
python scripts/train.py --config <config.yaml> [overrides...]
# Installed equivalent:
skillopt-train --config <config.yaml> [overrides...]
```

### Arguments

| Argument | Description |
|---|---|
| `--config` | Path to YAML config file (required) |
| `--cfg-options key=value [...]` | Override structured config parameters |

### Examples

```bash
# Basic training
python scripts/train.py \
  --config configs/searchqa/default.yaml \
  --out_root outputs/searchqa_run

# With overrides
python scripts/train.py \
  --config configs/searchqa/default.yaml \
  --cfg-options optimizer.learning_rate=16 optimizer.lr_scheduler=linear

# With custom initial skill
python scripts/train.py \
  --config configs/searchqa/default.yaml \
  --cfg-options env.skill_init=skills/my_seed.md
```

## Evaluation

```bash
python scripts/eval_only.py --config <config.yaml> --skill <skill.md>
# Installed equivalent:
skillopt-eval --config <config.yaml> --skill <skill.md>
```

### Arguments

| Argument | Description |
|---|---|
| `--config` | Path to YAML config file (required) |
| `--skill` | Path to skill document to evaluate (required) |
| `--split` | `train`, `valid_seen`, `valid_unseen`, or `all` (default) |
| `--cfg-options` | One or more `section.key=value` overrides |

### Examples

```bash
# Evaluate best skill on test set
python scripts/eval_only.py \
  --config configs/searchqa/default.yaml \
  --skill outputs/searchqa_run/best_skill.md \
  --split valid_unseen

# Evaluate on validation set
python scripts/eval_only.py \
  --config configs/searchqa/default.yaml \
  --skill outputs/searchqa_run/best_skill.md \
  --split valid_seen
```

`--skill` consumes the artifact produced by training. Unless `--out_root` is
set for evaluation, `eval_only.py` creates a separate timestamped
`outputs/eval_<env>_<model>_<timestamp>/` directory and writes
`eval_summary.json` there; it does not modify the training run directory.

For the generic OpenAI-compatible research backend, select the role backends
explicitly:

```bash
python scripts/train.py \
  --config configs/searchqa/default.yaml \
  --cfg-options \
    model.optimizer_backend=openai_compatible \
    model.target_backend=openai_compatible \
    model.optimizer=deepseek-chat \
    model.target=deepseek-chat
```

To benchmark an installed, authenticated Cursor Agent through an environment
that supports exec targets:

```bash
python scripts/eval_only.py \
  --config configs/searchqa/default.yaml \
  --skill skills/my_skill.md \
  --cfg-options \
    model.optimizer_backend=openai_chat \
    model.target_backend=cursor_exec \
    model.target=composer-2.5
```

`cursor_exec` runs the target only; the optimizer remains separately
configured. Read-only rollouts use Cursor Ask mode. Rollouts that request file
edits use `--force` inside the benchmark workspace, with Cursor sandboxing
enabled. The harness refuses file-edit rollouts when the Cursor sandbox is
disabled. Read-only Ask-mode rollouts may explicitly disable it. Override the
executable or sandbox through `model.cursor_exec_path` and
`model.cursor_exec_sandbox`.

To train or evaluate with the Claude Code CLI execution harness (`claude_code_exec`):

```bash
# Target-only rollout through Claude Code (optimizer defaults to configured chat backend):
python scripts/train.py \
  --config configs/searchqa/default.yaml \
  --backend claude_code_exec

# Explicitly drive both target and optimizer roles with Claude Code:
python scripts/train.py \
  --config configs/searchqa/default.yaml \
  --cfg-options \
    model.optimizer_backend=claude_code_exec \
    model.target_backend=claude_code_exec
```

When `claude_code_exec` is selected as the target backend, SkillOpt streams SDK
messages from the Claude Code process and parses text, tool calls, and tool
results into structured trace steps (`claude_trace_steps.txt`). When
`model.claude_trace_to_optimizer` is enabled (`true` by default), these trace
steps are injected into the reflection prompt so the optimizer can inspect the
agent's intermediate actions.


## SkillOpt-Sleep

```bash
skillopt-sleep <action> [options]
# Equivalent from a source checkout:
python -m skillopt_sleep <action> [options]
```

Actions are `run`, `dry-run`, `status`, `adopt`, `harvest`, `schedule`,
`unschedule`, and `evalkit`. `evalkit` is also available as
`python -m skillopt_sleep.evalkit` and compares two conditions on one fixed
task manifest (McNemar + bootstrap CI). Exactly one of its `--b` comparison
input or `--aa` identity-check flag is required. See `docs/sleep/evalkit.md`.
Common options for the nightly actions include:

| Argument | Description |
|---|---|
| `--project PATH` | Project used for transcript scope, targets, state, and staging (default: current directory) |
| `--scope invoked\|all` | Harvest this project or all projects |
| `--source claude\|codex\|copilot\|cursor\|pi\|opencode\|auto` | Transcript source; `auto` keeps Codex-then-Claude precedence and does not select Copilot, Cursor, Pi, or OpenCode |
| `--backend mock\|claude\|codex\|copilot\|cursor\|pi\|opencode\|handoff\|azure_openai` | Replay/optimizer backend |
| `--model NAME` | Backend-specific model override |
| `--cursor-home PATH` | Override `~/.cursor` for Cursor transcript harvesting |
| `--pi-home PATH` | Parent directory containing Pi's `agent/sessions` tree (default: `~/.pi`) |
| `--vscode-workspace-storage PATH` | Override VS Code's `User/workspaceStorage` root for Copilot transcript harvesting |
| `--cursor-path PATH` | Path to the installed Cursor Agent CLI |
| `--pi-path PATH` | Path to the installed Pi coding-agent CLI |
| `--opencode-path PATH` | Path to the installed OpenCode CLI |
| `--opencode-db PATH` | Path to the OpenCode SQLite history database |
| `--opencode-tool-replay` | Enable OpenCode tool-aware replay for `tool_called` checks in rule judges |
| `--preferences TEXT` | House rules supplied to reflection |
| `--lookback-hours N` | Initial transcript lookback; `0` scans all history |
| `--max-sessions N` / `--max-tasks N` | Bound the harvested workload |
| `--target-skill-path PATH` | Explicit skill document to stage/adopt |
| `--skill-root PATH` | Add a skill-resolution root; repeatable, with relative paths resolved below `--project` |
| `--tasks-file PATH` | Replay a reviewed task JSON file instead of harvesting |
| `--edit-budget N` | Maximum bounded edits for the night |
| `--progress` / `--json` | Progress or machine-readable output |
| `--auto-adopt` | Apply an accepted staged proposal automatically |

`adopt` also accepts `--skill NAME` (repeatable) and `--all-skills` for a night
that staged per-skill proposals. Use `--legacy` to adopt only a co-staged
managed `SKILL.md` / `CLAUDE.md` proposal. Bare `adopt` on a fan-out night lists
the names and exits instead of promoting anything. A leading-dash name must use
the unambiguous `--skill=--name` form. See
[multi-skill staging](../sleep/multi-skill-staging.md).

Fan-out resolves existing project-native `.agents/skills`, `.claude/skills`,
`.cursor/skills`, and `.devin/skills` directories plus the established Claude
roots. Use `--skill-root` for another integration-specific location. Configure
the canonical `multi_skill_fanout` key to enable proposal fan-out;
`multi_skill_report` remains a compatibility alias.

The `mock` and `handoff` backends make no network calls. A real backend sends
mining, replay, judging, and reflection prompts derived from harvested
transcripts and tasks to its selected provider. Review that provider's
data-retention and privacy policy before processing sensitive sessions.

### VS Code GitHub Copilot Chat source

`--source copilot` reads local VS Code GitHub Copilot Chat session logs from
the platform's stable and Insiders `User/workspaceStorage` locations, plus the
portable location when `VSCODE_PORTABLE` is available. Use
`--vscode-workspace-storage PATH` for a nonstandard root. Each workspace must
have a readable adjacent `workspace.json` mapping to a local project; unmapped
windows are skipped so project scoping cannot silently mix unrelated sessions.
`--source auto` does not select Copilot.

The harvester retains user-entered prompts, visible assistant Markdown, and
tool names from requests confirmed as GitHub Copilot Chat. It excludes
system-initiated notifications, reasoning, tool inputs and outputs, rendered
context, and account/model metadata. Known secret-shaped strings are redacted
as defense in depth, but review harvested tasks before sending them to a model
provider. Transcript harvesting is independent of `--backend copilot`, which
invokes the GitHub Copilot CLI for model calls.

The managed `schedule` command does not persist `--source` or
`--vscode-workspace-storage`. Before scheduling this source, set
`"transcript_source": "copilot"` and, when needed,
`"vscode_workspace_storage": "/absolute/path/to/workspaceStorage"` in
`~/.skillopt-sleep/config.json`.

### Pi source and backend

`--source pi` reads local session JSONL files below
`~/.pi/agent/sessions`; use `--pi-home PATH` to select the parent directory that
contains `agent/sessions`. This local source does not require the Pi CLI or
provider authentication. It retains user/assistant text, tool names, and lexical
feedback found in user text, while excluding thinking, tool arguments, tool
outputs, images, and unrelated metadata. The absolute project `cwd` from the
session header is retained for scope filtering and may appear in miner prompts
sent to a real backend and its provider. Known secret-shaped strings in retained
message text are redacted as defense in depth, not as a guarantee. Pi is an explicit source: `--source auto`
retains Codex-then-Claude precedence and does not select it.

Transcript source and model backend are independent. `--backend pi` launches a
locally installed, authenticated Pi CLI and makes real provider calls for
mining, replay, judging, and reflection. Use `--pi-path PATH` to select its
executable and `--model NAME` to override its configured model:

```bash
skillopt-sleep run --project "$(pwd)" \
  --source pi --backend pi --pi-path /absolute/path/to/pi \
  --model provider/model --max-sessions 5 --max-tasks 3 --progress
```

For these calls, SkillOpt disables Pi tools, skills, context files, extensions,
prompt templates, themes, and session writes. Pi authentication and model
configuration remain available. It also enables Pi's offline startup mode, so
configured npm/git packages are not installed or updated and model catalogs are
not refreshed; the selected model provider is still contacted for generation.
These controls should not be treated as permanent or complete isolation. The
provider selected in Pi receives the transcript-derived prompts.

The managed `schedule` command preserves the backend but not `--source`,
`--pi-home`, `--pi-path`, or `--model`. Before scheduling Pi, put
`transcript_source`, `pi_home`, `pi_path`, and `model` in
`~/.skillopt-sleep/config.json`; use an absolute `pi_path` and verify
authentication for the scheduled account.

### OpenCode source and backend

`--source opencode` reads OpenCode's local SQLite history directly in read-only
mode. The source does not launch OpenCode, require the OpenCode CLI, use its
login, or contact a model provider. Source selection remains explicit:
`--source auto` keeps Codex-then-Claude precedence and does not select OpenCode.

The database path is selected from `--opencode-db` or the `opencode_db` config
key, then `OPENCODE_DB`, then `%LOCALAPPDATA%\opencode\opencode.db` or `%APPDATA%\opencode\opencode.db` on Windows or
`${XDG_DATA_HOME:-~/.local/share}/opencode/opencode.db` on POSIX. A relative
`OPENCODE_DB` value is resolved below OpenCode's data directory;
`OPENCODE_DB=:memory:` has no persistent history to harvest.

The harvester keeps visible user and assistant text, short tool names, the
recorded project directory, Git branch, and session timestamps. It excludes
reasoning, tool arguments and results, file contents, patches, and
provider/model/account metadata. Only root sessions are considered, and
sessions produced by SkillOpt's own OpenCode backend are excluded. Known
secret-shaped strings in retained text are redacted as defense in depth;
inspect harvested tasks before sending them to a real backend. The database is
opened read-only, although SQLite may still update its transient `-shm` file
while coordinating an active WAL database.

The transcript source and model backend are independent. For example, export
OpenCode-derived tasks for review before using any configured backend:

```bash
skillopt-sleep harvest --project "$(pwd)" \
  --source opencode --output reviewed-tasks.json --progress
```

`--backend opencode` runs SkillOpt's model calls for mining, plain task replay,
judging, and reflection through an installed OpenCode CLI. It uses the user's
existing OpenCode login, provider environment variables, and file-based global
configuration; SkillOpt does not manage OpenCode accounts or provider
credentials.

Install and configure OpenCode using its
[official documentation](https://opencode.ai/docs/), then confirm the CLI is
available with `opencode --version`.

SkillOpt's OpenCode backend, including tool-aware replay, has been tested with
OpenCode CLI `1.18.15`. Other versions may work but have not been validated.

If OpenCode is on `PATH`, no path option is needed. Otherwise use
`--opencode-path`, the `opencode_path` config key, or
`SKILLOPT_SLEEP_OPENCODE_PATH`. Use `--model`, the `model` config key, or
`SKILLOPT_SLEEP_OPENCODE_MODEL` to override OpenCode's configured model:

```bash
skillopt-sleep run --project "$(pwd)" \
  --source opencode --backend opencode \
  --opencode-path /absolute/path/to/opencode \
  --model provider/model --max-sessions 5 --max-tasks 3 --progress
```

Plain calls run from a temporary directory with project configuration and
model-initiated tool invocation disabled. Before contacting the model, SkillOpt
discovers the resolved MCP configuration, disables every configured MCP server
for the call, and verifies that none remains enabled. If that check fails, the
model call is not made.

OpenCode tool-aware replay is disabled by default. Enable it explicitly with
`--opencode-tool-replay` or `"opencode_tool_replay": true` in
`~/.skillopt-sleep/config.json`. It applies only to tasks whose rule judge
contains a non-empty `tool_called` check. In a fresh temporary workspace,
SkillOpt creates synthetic tools with randomized names and fixed results, then
verifies which tools OpenCode actually invoked. Historical tool arguments and
results are neither retained nor replayed. Configured MCP servers remain
disabled, and the invocation allowlist includes only these temporary tools.

Both modes continue to use OpenCode's normal data directory and file-based
global configuration. OpenCode may discover or initialize custom JS/TS tools
from its global configuration directories, although SkillOpt does not allow the
model to invoke them. SkillOpt replaces `OPENCODE_CONFIG_CONTENT` in the child
process to define the temporary agent and disable configured MCP servers, so
settings present only in the user's original value are unavailable. Because
`--pure` skips external plugins,
authentication or provider setup that depends on one of those plugins is also
unavailable. Calls may appear in the user's OpenCode session history. During
tool-aware replay, the fixed input (`synthetic`), fixed result, and temporary
project metadata may remain there as well. These controls are invocation
settings, not complete account or process isolation.

The managed scheduler stores the backend but not `--source`, `--opencode-db`,
`--opencode-path`, `--opencode-tool-replay`, or `--model`. Put
`transcript_source`, `opencode_db`, `opencode_path`, and `model` in
`~/.skillopt-sleep/config.json` as needed. Add
`"opencode_tool_replay": true` only when the scheduled run should enable
tool-aware replay. Use absolute database and executable paths, and verify
OpenCode access when the scheduled run uses the backend.

### Cursor source and backend

`--source cursor` reads local Cursor JSONL transcripts from
`~/.cursor/projects/<workspace>/agent-transcripts/*/*.jsonl`. Invoked scope uses
Cursor's recorded workspace path, including when `--project` is a nested
directory, and falls back to the sanitized storage name when metadata is not
available. `--scope all` scans every workspace below `cursor_home`. The
harvester retains user/assistant text, explicit turn errors, and tool names,
while excluding tool arguments, tool outputs, and non-message records. It
redacts known secret patterns and filters SkillOpt-generated replay sessions,
but redaction is not a guarantee that outbound prompts contain no sensitive
data.

`--backend cursor` launches an installed, authenticated `cursor-agent`, sends
prompts over stdin, and parses its JSON result. SkillOpt reads the target skill
and includes its text in replay prompts; it does not invoke that file as a native
Cursor skill. Ordinary mining, replay, judging, and reflection calls use
read-only Ask mode in a new empty temporary workspace. Project file reads, file
writes, and MCP tools are denied. `--project` does not change that execution
workspace.

Cursor tool-aware replay is temporarily disabled pending live Cursor
permission-boundary validation. A task with a `tool_called` check fails nonzero
before Agent mode starts and does not stage, adopt, cache, persist state, or
advance the harvest checkpoint. Use another backend for such tasks. The current
Cursor backend therefore does not provide end-to-end validation for skills that
need repository inspection, real CLIs, browsers, running services, or file
changes.

There is no implemented fresh-worktree Cursor replay. If a report says
`replay: mock`, that is the prompt-replay label and does not mean the mock model
backend was selected. Both `run` and `dry-run` perform real-backend provider
calls; `dry-run` suppresses staging, adoption, and persisted state changes, not
spend. Session and task limits do not impose hard provider-call, token, time, or
monetary budgets.
Cursor and its selected model provider can receive the prompt content.

Cursor-specific settings are available through the CLI, config, and environment:

| Purpose | CLI | `~/.skillopt-sleep/config.json` | Environment |
|---|---|---|---|
| Transcript home | `--cursor-home PATH` | `"cursor_home": "/path/to/.cursor"` | none |
| Agent executable | `--cursor-path PATH` | `"cursor_path": "/path/to/cursor-agent"` | `SKILLOPT_SLEEP_CURSOR_PATH` |
| Model | `--model NAME` | `"model": "NAME"` | `SKILLOPT_SLEEP_CURSOR_MODEL` |

Use `cursor-agent --list-models` to inspect model identifiers available to the
authenticated account. When cost depends on a model variant, confirm the billed
variant in Cursor's usage reporting rather than relying only on its display
name.

Target the learned project skill explicitly so accepted updates are visible to
Cursor without modifying the plugin's own `skillopt-sleep` workflow skill:

```bash
skillopt-sleep run --project "$(pwd)" \
  --source cursor --backend cursor \
  --target-skill-path .cursor/skills/skillopt-sleep-learned/SKILL.md \
  --max-sessions 5 --max-tasks 3 --progress
```

The first harvest uses a 72-hour lookback unless `--lookback-hours` is set. A
value of `0` considers all available history while still respecting
`--max-sessions`. A stateful `run`, including a run that mines no tasks, records
a new harvest checkpoint; subsequent runs use that checkpoint rather than the
initial lookback. Use `harvest` or `dry-run` to verify counts before the first
stateful run.

There is one checkpoint per project **and transcript source**, because each
source reads a different store with its own timestamps. Feeding one project from
several sources therefore keeps a separate window per source, and a run of one
source does not advance another source's window.

The managed `schedule` command persists the project, backend, time, and optional
auto-adopt setting only. It does not copy source, Cursor paths, model, or target
skill flags into the scheduled command. Put `transcript_source`, `cursor_home`,
`cursor_path`, `model`, and `target_skill_path` in the user config before
scheduling Cursor. Keep `target_skill_path` project-relative as
`.cursor/skills/skillopt-sleep-learned/SKILL.md`, prefer an absolute
`cursor_path`, and verify authentication for the scheduled account because cron
and Task Scheduler may have a minimal environment.

Backend-specific setup for compatible endpoints is documented in
[OpenAI-compatible endpoints for SkillOpt-Sleep](../sleep/openai-compatible-endpoints.md).

## WebUI

```bash
python -m skillopt_webui.app [--port PORT] [--share]
```

| Argument | Default | Description |
|---|---|---|
| `--port` | 7860 | Port number |
| `--host` | `127.0.0.1` | Server bind address |
| `--share` | false | Create public Gradio link |

The default host binds loopback only; pass `--host 0.0.0.0` to expose the dashboard.