{"owner":"gepa-ai","repo":"gepa","hasSkills":true,"totalSkillsCount":9,"totalTokensCount":15864,"categories":["marketplace","root-instruction","anthropic-skill","plugin-manifest","claude-rule"],"hasMcp":false,"mcpConfig":null,"found":[".claude-plugin/marketplace.json","AGENTS.md",".claude/skills/gepa-optimize-anything/references/api.md",".claude/skills/gepa-optimize-anything/SKILL.md",".claude/skills/gepa-optimize-anything/references/tracking.md",".claude/skills/gepa-optimize-anything/.claude-plugin/plugin.json",".claude/skills/gepa-optimize-anything/references/writing_evaluators.md",".claude/skills/gepa-optimize-anything/scripts/preflight.py",".claude/skills/gepa-optimize-anything/references/gotchas.md"],"skills":{".claude-plugin/marketplace.json":"{\n  \"name\": \"gepa\",\n  \"owner\": { \"name\": \"GEPA\", \"url\": \"https://github.com/gepa-ai/gepa\" },\n  \"description\": \"Official GEPA Claude Code plugins.\",\n  \"plugins\": [\n    {\n      \"name\": \"gepa-optimize-anything\",\n      \"source\": \"./.claude/skills/gepa-optimize-anything\",\n      \"description\": \"Optimize any scorable text artifact (prompts, code, configs, agent scaffolds) with GEPA's optimize_anything.\",\n      \"version\": \"0.1.0\",\n      \"homepage\": \"https://gepa-ai.github.io/gepa/\",\n      \"license\": \"MIT\",\n      \"category\": \"optimization\",\n      \"keywords\": [\"gepa\", \"prompt-optimization\", \"optimize-anything\"]\n    }\n  ]\n}\n","AGENTS.md":"# GEPA\n\nGEPA (Genetic-Pareto) is a Python framework for optimizing text components (AI prompts, code, instructions) using LLM-based reflection and Pareto-efficient evolutionary search.\n\n## Agent Skill\n\nThis repository ships an **Agent Skill** at `.claude/skills/gepa-optimize-anything/`. If you are a coding agent and the user wants to **auto-optimize, tune, or search over any scorable text artifact** (a prompt, program/code, config, regex/SQL, or agent scaffold) with `gepa.optimize_anything`, read `.claude/skills/gepa-optimize-anything/SKILL.md` first and follow it. It is auto-discovered by Claude Code and by other agents that read `.claude/skills/` (Cursor, VS Code/Copilot, Codex, Gemini CLI). To install it standalone in another project: `/plugin marketplace add gepa-ai/gepa` then `/plugin install gepa-optimize-anything@gepa`.\n\n## Setup\n\nWe use **uv** for dependency management. The project uses setuptools as the build backend. All python executions must be done through uv.\n\n```bash\nuv sync --extra dev\n```\n\n## Project Structure\n\n- `src/gepa/` — main package source\n  - `core/` — optimization loop, state, evaluation\n  - `proposer/` — candidate proposal and mutation logic\n  - `adapters/` — integration adapters (DSPy, RAG, MCP, etc.)\n  - `strategies/` — batch sampling and candidate selection\n  - `logging/` — experiment tracking and logging\n- `tests/` — pytest test suite\n- `docs/` — mkdocs documentation site\n\n## Build & Test\n\n```bash\nuv run pytest\nuv run ruff check src/\nuv run ruff format src/\nuv run pyright src/\n```\n\n## Code Style\n\n- Linter/formatter: ruff (line length 120, double quotes, space indent)\n- Type checking: pyright\n- Python target: 3.10+\n- No relative imports (enforced by ruff)\n",".claude/skills/gepa-optimize-anything/references/api.md":"# API reference — `optimize_anything`\n\nThe public entry point is `gepa.optimize_anything.optimize_anything`.\n\n## Signature\n```python\nfrom gepa.optimize_anything import GEPAResult, optimize_anything, OptimizeAnythingConfig\n\nresult = optimize_anything(\n    seed_candidate: str | None = None,   # starting text; None = seedless (engine bootstraps from objective/background)\n    *,\n    evaluator: Callable | None = None,       # (candidate) or (candidate, example) -> (score, info)\n    batch_evaluator: Callable | None = None, # list[(candidate, example)] -> one result per pair (see below)\n    dataset:  list | None = None,        # training examples used DURING optimization\n    valset:   list | None = None,        # validation examples used to SELECT the best candidate\n    objective: str | None = None,        # short goal, surfaced verbatim to the engine\n    background: str | None = None,       # long-form rules/constraints/domain notes\n    test_set: list | None = None,        # reporting-only: seed + final candidate scored here at the end\n    config: OptimizeAnythingConfig | None = None,\n) -> GEPAResult\n```\nExamples in `dataset`/`valset`/`test_set` are opaque — any object your `evaluator` understands.\n`seed_candidate` is a **single string** at this API (multi-component `dict[str, str]` candidates\nexist only in the lower-level `gepa.gepa_launcher.optimize_anything`).\n\n**`batch_evaluator`** — grouped scoring for when evals batch better than they stream (a provider\nbatch API, fan-out over your own cluster): it receives ALL `(candidate, example)` pairs of an\nevaluation stage (minibatch, valset, held-out test pass) in **one call** and returns one result per\npair (`score` or `(score, info)`). At least one of `evaluator` / `batch_evaluator` is required.\nWhen both are given, multi-pair stages use the batch function and single-pair evaluations use\n`evaluator`; with only `batch_evaluator`, singles route through it as singleton batches.\n\n### The three optimization modes (set by `dataset` / `valset`)\n| mode | pass | evaluator called | use when |\n|---|---|---|---|\n| **Single-task** | `dataset=None, valset=None` | `evaluator(candidate)` | solving one hard problem; the candidate *is* the solution (one kernel, a packing layout, one document). |\n| **Multi-task** | `dataset=<list>, valset=None` | `evaluator(candidate, example)` | one shared candidate must do well across a batch of related problems; insight transfers across them. |\n| **Generalization** | `dataset=<list>, valset=<list>` | `evaluator(candidate, example)` | the candidate must transfer to **unseen** problems: optimize on `dataset`, select on `valset`. |\n\n*Note on `valset`:* held-out selection is currently implemented only by the gepa backend (GEPA is\ndesigned around this mode — `valset` selection plus a Pareto frontier over val instances). The\nother backends fold `valset` into the pool the eval server exposes (`train_set + val_set` merged),\nso they optimize over the combined set with no separate selection split — they may still produce\ncandidates that generalize.\n\n**`test_set` is reporting-only and orthogonal to the mode.** Only `dataset` and `valset` affect\noptimization (search and selection). If you pass `test_set`, the seed and the final chosen candidate\nare scored on it *after* optimization, outside the budget, and reported in `result.metadata`. The\ntest set never enters the eval server's HTTP surface, so agentic backends cannot see it. Use it when\nyou want an unbiased number to report; skip it otherwise.\n\n## `OptimizeAnythingConfig` (top-level fields)\n| field | default | meaning |\n|---|---|---|\n| `engine` | `\"gepa\"` | `\"gepa\"` / `\"autoresearch\"` / `\"meta_harness\"` (or `\"best_of_n\"`, a baseline), or a constructed `Engine` instance (custom engines can be added via `gepa.oa.registry.register_engine`). |\n| `name` | `None` | run id (logging + default output dir). Auto-generated (`<engine>-<uuid>-<timestamp>`) if `None`. |\n| `max_evals` | **`100`** | server-side cap on eval calls. `None` = unlimited. Size it — the default is rarely right (see SKILL.md). |\n| `max_token_cost` | `None` | USD cap on the engine's **own** optimizer-LLM spend (reflection/agent). Enforced by the engine (gepa: `max_reflection_cost` stopper; agent engines: `--max-budget-usd`), not the eval server. |\n| `max_concurrency` | `8` | eval-server thread-pool size. |\n| `output_dir` | `None` | where the eval server writes per-eval JSON, `progress_log.jsonl`, `summary.json`. `None` → `outputs/optimize_anything/<task>/<engine>/<timestamp>/`. |\n| `run_dir` | `None` | engine workspace (gepa run dir / agent work dir; with `engine.write_agent_state=True` the gepa backend writes an agent-readable `iterations/` + `pareto/` tree here). Distinct from `output_dir`. `None` → subprocess engines use a tempdir; set it to persist artifacts. |\n| `stop_at_score` | `None` | early-stop score threshold; each engine interprets it. |\n| `sandbox` | **`True`** | OS-jail the subprocess engines' `claude` sessions: bwrap on Linux (needs the `bubblewrap` package — the run aborts at launch if `bwrap` is missing), Claude Code's Seatbelt sandbox on macOS. Also forces the work dir to a tempdir even when `run_dir` is set (artifacts are mirrored back). `False` prints a loud warning and runs the agent unsandboxed (`bypassPermissions`, full host access). |\n| `engine_config` | `{}` | dict of **engine-specific** options — see the per-backend sections below. Parsed into a typed per-engine config dataclass; **an unknown key raises `TypeError` immediately** (fail fast, not warn-and-drop). |\n\n**If both `max_evals` and `max_token_cost` are `None`, the run is unbounded** (only a\n`warnings.warn`). Note the former fields `tracker`, `effort`, and `max_thinking_tokens` no longer\nexist here: tracking is configured via the gepa backend's `engine_config[\"tracking\"]`\n(see `tracking.md`), and `effort` / `max_thinking_tokens` are per-agent-engine `engine_config` keys.\n\n## The backends\n\nAll are selected with `engine=` and run the **same** task/evaluator — switch optimizer by\nchanging one argument. Each backend parses `engine_config` into its own typed dataclass, so swapping\n`engine=` also means swapping the `engine_config` block (leftover keys → `TypeError`).\n\n| engine | how it proposes candidates | runs | needs |\n|---|---|---|---|\n| `gepa` | an LLM reflects on evaluator feedback and mutates the candidate; keeps a Pareto frontier | in-process | reflection-LM creds (default `openai/gpt-5.1`) or a custom LM |\n| `autoresearch` | one Claude Code subprocess iterates in a work dir (`program.md`, `candidate.txt`, `eval.sh` → HTTP eval server) | subprocess | `claude` on PATH + headless auth, `jq` |\n| `meta_harness` | a Claude subprocess reads frontier/history and writes `pending_eval.json` candidates; the engine benchmarks each | subprocess | `claude` on PATH + headless auth |\n| `best_of_n` *(baseline)* | independent single-shot samples from one LLM; keep the best — no feedback, no history | in-process | LiteLLM creds for `model` (default `claude-sonnet-4-6`) |\n\n`scripts/preflight.py` checks a backend's prerequisites before a long run.\n\n### `gepa` — `engine_config` is a `GEPAConfig`-shaped dict (1-to-1 passthrough)\nThe dict is passed to `gepa.gepa_launcher.GEPAConfig(**engine_config)`; valid top-level keys are\nexactly its fields — `engine`, `reflection`, `tracking`, `merge`, `refiner`, `callbacks`,\n`stop_callbacks` — and nested dicts are coerced to the matching dataclasses. The OA layer only\noverlays the eval budget (`max_evals` → `engine.max_metric_calls`), `run_dir`, `stop_at_score`, and\nthe `max_token_cost` → `engine.max_reflection_cost` cap.\n\n```python\nengine_config={\n    \"reflection\": {                     # -> ReflectionConfig\n        \"reflection_lm\": \"anthropic/claude-sonnet-4-6\",  # default \"openai/gpt-5.1\"; see \"Proposer LM\"\n        \"reflection_lm_kwargs\": {\"reasoning_effort\": \"high\"},  # litellm kwargs (temperature, thinking, …)\n        \"reflection_minibatch_size\": 5,  # default: 1 single-task, 3 otherwise\n        # \"custom_candidate_proposer\": ClaudeCodeAgentProposer(...),  # replace the reflection LM with\n        #                                # a Claude Code proposer (from gepa.oa.proposers)\n        # \"reflection_strategy\": ...,    # advanced: a ReflectionLM impl owning how reflection is called\n    },\n    \"engine\": {                          # -> EngineConfig (all optional, sensible defaults)\n        \"max_workers\": 32,               # parallel eval workers (default: cpu_count or 32)\n        \"seed\": 0,                       # reproducibility\n        \"frontier_type\": \"hybrid\",       # \"instance\" | \"objective\" | \"hybrid\" (default) | \"cartesian\"\n        \"candidate_selection_strategy\": \"pareto\",      # | \"current_best\" | \"epsilon_greedy\" | \"top_k_pareto\"\n        \"acceptance_criterion\": \"strict_improvement\",  # | \"improvement_or_equal\"\n        \"cache_evaluation\": False,       # opt-in: cache (candidate, split, example) evals.\n                                         # Distinct valset is isolated from trainset ids.\n                                         # valset=None still shares minibatch rollouts.\n        \"capture_stdio\": False,          # opt-in: route evaluator print() output into feedback\n        \"raise_on_exception\": True,      # False → evaluator exceptions become score 0 + info[\"error\"]\n        # \"write_agent_state\": True,     # agent-readable iterations/ + pareto/ tree under run_dir\n    },\n    \"tracking\": {\"use_wandb\": True},     # -> TrackingConfig (see references/tracking.md)\n    \"merge\":  {...},                     # -> MergeConfig (cross-candidate merging) or omit\n    \"refiner\": {...},                    # -> RefinerConfig (auto per-eval refinement) or omit\n    # \"callbacks\": [GEPACallback, ...],  # observation hooks (on_iteration_end, on_candidate_accepted, …)\n    # \"stop_callbacks\": [...],           # custom StopperProtocol stop conditions\n}\n```\nThe nested dataclasses live in `gepa.gepa_launcher` (`EngineConfig`, `ReflectionConfig`,\n`TrackingConfig`, `MergeConfig`, `RefinerConfig`); their knobs are documented in the GEPA guides —\ne.g. **candidate-selection**, **acceptance-criterion**, **batch-sampling**, **callbacks**,\n**cost-tracking**, **experiment-tracking** — see <https://gepa-ai.github.io/gepa/guides/>.\n\n### `autoresearch` — `engine_config` → `AutoResearchConfig`\n| key | default | meaning |\n|---|---|---|\n| `model` | `\"claude-sonnet-4-6\"` | Claude model id (pass `\"sonnet\"`/`\"opus\"` to track the current default). |\n| `ralph` | `True` | keep resuming one Claude session (`claude --resume`) while budget remains. |\n| `max_no_eval_seconds` | `None` | kill the subprocess after this long with no eval call. |\n| `handoffs` | `None` | prior-stage artifacts for sequential compositions (materialized under `handoff/`). |\n| `effort` | `None` | `claude --effort` value. |\n| `max_thinking_tokens` | `None` | fixed thinking-token budget (`MAX_THINKING_TOKENS`). |\n\nThe engine lays out a work dir (`program.md`, `candidate.txt`, `best_candidate.txt`, `eval.sh`) and\nlaunches `claude --print`; `eval.sh` POSTs candidates to the eval server, which enforces the budget\nserver-side (HTTP 429 on exhaustion) and caps LLM spend via `--max-budget-usd` (from\n`max_token_cost`). Train and val are presented to the agent as one combined pool; the test set is\nunreachable over HTTP.\n\n### `meta_harness` — `engine_config` → `MetaHarnessConfig`\n| key | default | meaning |\n|---|---|---|\n| `model` | `\"claude-sonnet-4-6\"` | proposer model id. |\n| `max_iterations` | `None` | hard cap on proposer sessions; `None` = until budget. |\n| `max_candidates_per_iter` | `3` | upper bound on candidates proposed per iteration. |\n| `effort` | `None` | `claude --effort` value. |\n| `max_thinking_tokens` | `None` | fixed thinking-token budget. |\n\nEach iteration the proposer subprocess reads the frontier + history state files, writes\n`pending_eval.json` with 1+ candidates, and the engine benchmarks each through the eval server.\n\n### `best_of_n` (baseline) — `engine_config` → `BestOfNConfig`\nDeliberately naive: each sample is one independent LLM call — no feedback, no history, no\nselection pressure beyond keep-the-best. Use it as a **comparison floor** when evaluating an\noptimizer, not as the optimizer. Stops on budget exhaustion, `stop_at_score`, or `max_n`.\n\n| key | default | meaning |\n|---|---|---|\n| `model` | `\"claude-sonnet-4-6\"` | LiteLLM model id used to sample candidates. |\n| `temperature` | `1.0` | sampling on by default so N calls don't collapse to one response. |\n| `max_n` | `None` | optional hard cap on samples; `None` = run until budget out. |\n| `lm_kwargs` | `{}` | extra kwargs forwarded to `gepa.lm.LM`. |\n| `effort` | `None` | threaded into the LM as `reasoning_effort`. |\n| `max_thinking_tokens` | `None` | fixed thinking-token budget for the LM. |\n\n## Proposer LM (not limited to LiteLLM)\n`reflection.reflection_lm` (gepa) accepts either:\n- a **model-id string** resolved through LiteLLM (`\"openai/gpt-5.1\"` — the default —\n  `\"anthropic/claude-sonnet-4-6\"`, a Bedrock inference-profile ARN, …); set the provider's\n  credentials, and pass litellm kwargs via `reflection.reflection_lm_kwargs`; or\n- **any object/callable implementing GEPA's LM protocol** — `__call__(prompt) -> str` (a\n  `str`-or-messages prompt in, completion text out). This is how you plug a **self-hosted or custom\n  inference engine** (vLLM, a local server, your own client) instead of LiteLLM.\n  `reflection_lm_kwargs` is ignored for callables.\n\n## Composition / pipeline helpers (ensembles)\n`gepa.optimize_anything` also exports helpers that compose several backends over the same\ntask/evaluator. Each takes the same flat task fields as `optimize_anything` (`seed_candidate`\npositional; `evaluator=`, `dataset=`, `valset=`, `objective=`, `background=`, `test_set=`, `name=`\nkeywords) plus `configs=` — a list of `OptimizeAnythingConfig`, one per stage/branch, each carrying\nits own budget. Budgets are pre-partitioned per config; an early-finishing stage's leftover is not\nredistributed.\n\n- `optimize_sequential` — **pipeline**: run configs in order; each stage's best becomes the next\n  stage's seed. Monotonic — the running best is carried forward, so a regressing stage doesn't\n  poison the chain.\n- `optimize_parallel` — run all configs concurrently; returns the list of `Result`s (config order).\n- `optimize_best_of` — parallel, then keep the result with the highest `best_score` (each engine's\n  own number — no re-scoring).\n- `optimize_vote` — parallel, then re-score each branch's best candidate once via `evaluator`\n  (outside any budget) and return the highest re-score. Use when engines score with different\n  quirks and you want a fair cross-engine comparison.\n- `optimize_adaptive_sequential` — **plateau scheduler**: give the active engine bounded eval\n  slices (`plateau_evals`) and rotate to the next config after `patience` non-improving slices,\n  always feeding the running best forward. The exception to per-config budgets: one shared eval\n  pool set by its `max_evals` keyword (per-config `max_evals` is ignored; per-config\n  `max_token_cost` still applies). Extra knobs: `min_evals_per_stage`, `improvement_epsilon`,\n  `cycle`, `max_switches`.\n\n```python\nfrom gepa.optimize_anything import optimize_sequential, OptimizeAnythingConfig\n\nresult = optimize_sequential(\n    SEED,\n    evaluator=evaluate,\n    dataset=trainset, valset=valset,\n    objective=\"...\",\n    configs=[\n        OptimizeAnythingConfig(engine=\"autoresearch\", max_evals=100, max_token_cost=5.0),\n        OptimizeAnythingConfig(engine=\"gepa\", max_evals=200),  # refines autoresearch's best\n    ],\n)\n```\n\n`*_with_server` variants (`optimize_sequential_with_server`, `optimize_parallel_with_server`, …)\ntake caller-owned `EvalServer`s — one per config (one shared server for\n`optimize_adaptive_sequential_with_server`) — for embedding in outer frameworks that route every\neval through their own server. Related: the autoresearch backend's `handoffs` key materializes\nprior-stage artifacts into the agent's work dir for hand-rolled sequential compositions.\n\n## `GEPAResult`\n\n`optimize_anything` always returns a `GEPAResult` (the same type as the legacy\n`gepa.gepa_launcher` path). The mutable engine-internal `Result` is still what\nengines and ensemble helpers thread; it is not the public return.\n\n```python\nresult.best_candidate   # str (or dict[str, str] for multi-component seeds)\nresult.best_score       # float (on valset/selection set)\nresult.total_evals      # int — eval-server call count (budget.used)\nresult.eval_log         # list[dict]\nresult.metadata         # dict (verified keys):\n#   \"test_score\"             avg over test_set         } only present\n#   \"test_scores\"            per-example dict          } if you passed\n#   \"baseline_test_score\"    seed avg over test_set    } a test_set\n#   \"baseline_test_scores\"   seed per-example dict     }\n#   \"budget\", \"total_cost\", \"adapter_cost\", \"wall_time\", \"engine\", \"output_dir\", \"progress_log\"\nresult.candidates            # full candidate pool (length 1 on non-gepa engines)\nresult.best_idx              # index of the highest val score\nresult.val_aggregate_scores  # per-candidate average validation score\nresult.to_dict()             # JSON-safe pool snapshot (does not include eval_log / metadata)\n```\n\nThe candidate pool lives on the result itself. `metadata` does **not** contain a nested\n`gepa_result`. `total_evals` is the eval-server count; `total_metric_calls` is GEPA core's\nown counter and may differ (e.g. after `BudgetExhausted`). `to_dict()` / `from_dict()` omit\n`eval_log`, `metadata`, and `eval_server_calls` so the frozen JSON pool contract stays\nbyte-stable.\n\nThe gepa engine also writes `run_dir/` artifacts, and the eval server writes\n`output_dir/summary.json`. (If the best candidate equals the seed, the test scores are reported from the\nseed's single scoring pass rather than re-scored.)\n\n**Migration:** callers that used `isinstance(result, Result)` or\n`result.metadata[\"gepa_result\"]` after `optimize_anything(...)` should switch to\n`GEPAResult` and `result.candidates`. Ensemble helpers (`optimize_sequential`, …)\nstill return the mutable `Result`.\n",".claude/skills/gepa-optimize-anything/SKILL.md":"---\nname: gepa-optimize-anything\ndescription: >-\n  Automatically improve any text artifact that can be scored — prompts, programs/code, configs,\n  specs, regex/SQL/schemas, agent scaffolds, or encoded search solutions — with optimize_anything,\n  an LLM-driven optimization API: a search backend repeatedly proposes better versions from\n  execution feedback and an evaluator you provide assigns the score (an objective metric or an\n  LLM-as-judge for subjective tasks). optimize_anything is one interface across several optimizer\n  backends — GEPA (reflective evolutionary search, the default), AutoResearch, MetaHarness — plus a\n  best-of-N baseline: write the evaluator once and switch backend with a single argument.\n  Use whenever a task needs auto-optimizing, tuning, or searching over text with any quality signal\n  (accuracy, pass rate, latency, cost, judge rating); when building an evaluator + proposer loop;\n  when running GEPA; or when comparing optimizers. The candidate can be any string an evaluator can\n  grade, scored on one problem or generalized across many.\n---\n\n# `optimize_anything`\n\n**Naming, precisely.** `optimize_anything` is the tool: a general API for optimizing text\nartifacts. **GEPA** is one specific optimizer behind it — reflective evolutionary search, the\ndefault backend (`engine=\"gepa\"`) — and, for legacy reasons, also the name of the Python package\nthat ships all of this. In this skill, \"the gepa backend\" always means the optimizer; statements\nabout \"the optimizer\" or \"the backend\" apply to whichever engine you chose.\n\n`optimize_anything` does **black-box optimization**: you provide (1) a seed artifact, (2) an\n**evaluator** that scores any artifact and returns feedback, and (3) a backend, which repeatedly\nproposes improved artifacts and scores them through your evaluator. \"Black-box\" refers to the\n**evaluator**, not the artifact: the backend never sees how the score is computed — no gradients,\nno metric internals — only the scalar score and the feedback text you emit. The candidate itself\n*is* visible: the proposer reads and rewrites it, applying the LLM's understanding of your artifact.\nThe framework just imposes no structure on it — any string an evaluator can score works. The\nleverage is in your score and your feedback.\n\n**You write the task and evaluator once, then choose the search algorithm with one `engine`\nargument** — and the same code runs under any of them:\n- **`gepa`** — the GEPA optimizer: reflective evolutionary search, in-process (an LLM reflects on\n  feedback and mutates candidates; keeps a Pareto frontier). The default; strongest when feedback\n  is rich.\n- **`autoresearch`** — an agentic optimizer: one Claude Code subprocess iterates like a researcher in\n  a work dir, scoring candidates through an HTTP eval server.\n- **`meta_harness`** — an agentic proposer (Claude subprocess) that reads the frontier/history each\n  iteration and writes new candidates for the engine to benchmark.\n\n(There is also a `best_of_n` engine — sample N independent candidates, keep the best. It is\ndeliberately naive: use it as a **baseline** to compare an optimizer against, not as the optimizer.)\n\nThis makes it easy to start with one backend and benchmark others on the identical task/evaluator.\nThere are also **composition/pipeline helpers** that combine backends over the same task:\n`optimize_sequential` (a pipeline — each stage's best seeds the next), `optimize_parallel`,\n`optimize_best_of`, `optimize_vote` (re-score each branch's best for a fair pick), and an adaptive\nscheduler that rotates backends on score plateaus — see `references/api.md`.\n\n## What can be a candidate\nA candidate is **any string your evaluator can score**. \"Text in → a number out (higher is better),\nplus optional feedback\" is the entire contract, which covers a wide range of artifacts:\n- **prompts** — system/user prompts, instruction templates, rubrics, few-shot exemplars\n- **programs / code** — functions, whole files, CUDA kernels, scored by compiling + running + benchmarking\n- **configs / specs / schemas / regex / SQL** — any text whose effect you can measure\n- **agent scaffolds** — tool instructions, planner/critic prompts, orchestration text\n- **pure search artifacts** — a mathematical construction, a packing layout, a plan, encoded as text\n\nThe score can be an **objective metric** (accuracy, pass rate, runtime, cost) **or an LLM-as-judge**\nrating for subjective tasks (writing quality, helpfulness, style). At this API the candidate is a\nsingle string (`seed_candidate: str | None`; `None` = seedless — the engine bootstraps from\n`objective`/`background`). Multi-component dict candidates exist only in the lower-level\n`gepa.gepa_launcher.optimize_anything` API, not here.\n\n## Three optimization modes (choose by how you pass data)\nThe mode is determined by whether you provide `dataset` and `valset`:\n1. **Single-task** (`dataset=None, valset=None`) — solve one hard problem; the candidate *is* the\n   solution; the evaluator is called with no example. *E.g. one CUDA kernel, a circle-packing layout.*\n2. **Multi-task** (`dataset=<list>, valset=None`) — solve a batch of related problems with one shared\n   candidate, transferring insight across them; evaluator called per example. *E.g. a single prompt\n   that works across many tasks.*\n3. **Generalization** (`dataset=<list>, valset=<list>`) — build a candidate that transfers to\n   **unseen** problems; optimize on `dataset`, select on `valset`. *E.g. a prompt tuned to generalize.*\n   *Note:* GEPA is the algorithm designed around this mode, and `valset`-based held-out selection is\n   currently implemented only by the gepa backend — the other backends fold `valset` into the\n   training pool (they can still generalize; there's just no separate selection split).\n\n`test_set` is **separate from the modes and reporting-only**: the seed and the final candidate are\nscored on it after optimization for an unbiased number — it never enters the search, selection, or\nbudget (for the agentic backends it is sealed at the eval server's HTTP layer, so the agent cannot\neven see it). See `references/api.md` for details and when to use each mode.\n\n## Install\n```bash\npip install \"gepa[full]\"   # [full] pulls cloudpickle — needed to pickle closure evaluators for\n                           # parallel workers / opt-in evaluation caching; plain `pip install gepa`\n                           # can fail there when your evaluator closes over data.\n# Proposer LLM: the gepa backend's reflection LM defaults to \"openai/gpt-5.1\" (a LiteLLM id) — set\n# that provider's key (OPENAI_API_KEY), or pass your own id (e.g. \"anthropic/claude-sonnet-4-6\" with\n# ANTHROPIC_API_KEY, or a Bedrock ARN with AWS creds). You can also pass any callable implementing\n# GEPA's LM protocol — a self-hosted / custom inference engine — instead of a model-id string.\n# Agentic backends (autoresearch, meta_harness) additionally need the `claude` CLI on PATH (plus\n# `jq` for the generated eval.sh):\nnpm install -g @anthropic-ai/claude-code   # or: curl -fsSL https://claude.ai/install.sh | bash\n#   ...then run `claude` once to authenticate.\n# On Linux they also need bubblewrap: the default sandbox=True jails the claude subprocess with\n# bwrap (`sudo apt install bubblewrap` / `sudo dnf install bubblewrap`) and the run aborts at\n# launch if it's missing. sandbox=False opts out — the agent then runs unsandboxed (loud warning).\n```\n\n## Mental model (4 pieces)\n1. **Candidate** — the text you optimize (a `str`). The seed is the start; `None` = seedless.\n2. **`evaluate(candidate, example) -> (score, info)`** — `score` is a float (higher is better);\n   `info` is a free-form dict of **feedback the backend's proposer reads** to make better candidates\n   (the gepa backend's reflection LM, or the agentic backends' agent; the `best_of_n` baseline\n   ignores feedback). When evals batch better than they stream (e.g. a provider batch API), pass\n   `batch_evaluator=` instead — all pending `(candidate, example)` pairs in one call; see\n   `references/api.md`.\n3. **Engine (backend)** — `\"gepa\"` (default), `\"autoresearch\"`, `\"meta_harness\"`, the `\"best_of_n\"`\n   baseline, or a constructed `Engine` instance.\n4. **Budget** — `max_evals` (server-side eval-call cap, **default 100**) and/or `max_token_cost`\n   (USD cap on the backend's own proposer-LLM spend). Setting both to `None` makes the run unbounded\n   (only a warning). **Size `max_evals` for many proposal rounds, not one** (see below) — this is the\n   most common way agents misuse this API.\n\n## Sizing the valset and the budget (read this — the #1 mistake)\n`max_evals` is the *main* control over how long the optimizer runs. Leave it at the default (100)\nwith a large valset, or set it too low, and the run stops after a **single proposal**, then reports\na \"best candidate\" that looks fine but is barely optimized.\n\n- **valset** — on the gepa backend the best candidate is selected by its scores on the valset, so\n  make it a **representative subset of your data**. There is no fixed size: pick what represents the task (a\n  handful for small/expensive tasks, more when data is cheap and plentiful — anywhere from a few to\n  hundreds). Bigger valset = less noisy selection but more eval calls per candidate.\n- **budget** — every proposed candidate is scored on the **whole selection set**, so size\n  `max_evals` off that set, not off a single proposal. Which set that is depends on the mode:\n  ```\n  generalization (dataset + valset):  max_evals ≳ 15–20 × len(valset)    # scored & selected on valset\n  multi-task     (dataset only):      max_evals ≳ 15–20 × len(dataset)   # scored & selected on dataset\n  single-task    (no dataset/valset): max_evals ≳ 15–20                  # 1 eval per candidate, so this\n                                                                         #   IS the number of proposals\n  ```\n  The constant is the same everywhere — **let the backend propose AND evaluate ~15–20 candidates**\n  (more if you can afford it). Anything much less and the run tries only a couple of candidates —\n  i.e. it's barely optimizing. (This arithmetic is exact for the gepa backend — and the best_of_n\n  baseline — which score every candidate on the full selection set; the agentic backends decide\n  themselves how to spend eval calls, so treat it as a floor.)\n\nAfter the run, check how many proposals actually happened (on the gepa backend,\n`result.candidates` is the full pool; with `engine.write_agent_state=True` the\n`run_dir/iterations/` tree shows each one). **If it stopped after one proposal, the budget was too\nlow** — raise it and rerun.\n\n### Give every run a real stop condition\n`max_evals` bounds eval calls, but add explicit stops so runs end at the right moment:\n- **`stop_at_score`** — set it whenever your metric has a known ceiling (e.g. `1.0` for a pass rate /\n  accuracy). The backend stops the moment a candidate reaches it instead of burning the rest of the\n  budget at the optimum.\n- **`max_token_cost`** — a hard USD cap on the backend's own proposer/agent LLM spend. Especially\n  important for the agentic backends (`autoresearch`, `meta_harness`), whose Claude subprocesses\n  spend tokens between eval calls.\n- **a wall-clock `timeout`** on the process you launch (e.g. `timeout 1200 python run.py`) as a\n  backstop.\n- If you **opt in** to evaluation caching (`engine_config={\"engine\": {\"cache_evaluation\": True}}` on\n  the gepa backend — it is **off by default**), be aware `max_evals` then counts only cache *misses*:\n  a converged search can keep proposing cache-hitting candidates without consuming eval budget, so\n  `stop_at_score`/`max_token_cost` become mandatory, not optional. A distinct `valset` is cached\n  separately from the trainset; `valset=None` still reuses minibatch rollouts.\n\n## Minimal working example\nThe example optimizes a system prompt for concreteness, but the **shape is identical** for any\ncandidate — swap `SEED` for a code file / config / etc. and have `evaluate` compile/run/measure it.\n```python\nfrom gepa.optimize_anything import optimize_anything, OptimizeAnythingConfig\n\nSEED = \"You are an expert. Solve the task. Output only the final answer.\"\n\ndef evaluate(candidate: str, example) -> tuple[float, dict]:\n    output = run_my_model(system_prompt=candidate, user_prompt=example[\"prompt\"])  # your call\n    score = grade(output, example)                                                 # float, higher=better\n    return score, {                       # everything here is shown to the proposer LLM\n        \"score\": score,\n        \"output\": output,\n        \"error\": example.get(\"error\"),    # concrete, actionable feedback drives good proposals\n    }\n\nresult = optimize_anything(\n    seed_candidate=SEED,\n    evaluator=evaluate,\n    dataset=trainset,            # optimize on these (multi-task/generalization mode)\n    valset=valset,               # select the best candidate on these (generalization mode)\n    test_set=testset,            # OPTIONAL, reporting-only: seed + final candidate scored here at the end\n    objective=\"Produce a prompt that maximizes task accuracy.\",\n    background=\"Domain rules, constraints, output format the model must follow.\",\n    config=OptimizeAnythingConfig(\n        engine=\"gepa\",               # swap to \"autoresearch\" / \"meta_harness\" — same code\n                                     #   (\"best_of_n\" runs the same way, as a comparison baseline)\n        name=\"my_run\",\n        max_evals=300,               # ≳ 15-20 × len(valset): enough for ~15-20 proposals (see above)\n        stop_at_score=1.0,           # stop at the optimum (set when your metric has a known ceiling)\n        max_concurrency=16,\n        run_dir=\"runs/my_run\",       # engine workspace (gepa run dir / agent work dir)\n        output_dir=\"outputs/my_run\", # eval server: per-eval JSON, progress_log.jsonl, summary.json\n        engine_config={              # gepa backend: a GEPAConfig-shaped dict, validated strictly —\n            \"reflection\": {          #   an unknown key raises TypeError immediately (fail fast)\n                # a LiteLLM id (set the provider key) OR any callable implementing the LM protocol\n                \"reflection_lm\": \"anthropic/claude-sonnet-4-6\",\n                \"reflection_minibatch_size\": 5,\n            },\n            \"engine\": {\"max_workers\": 32, \"seed\": 0},  # seed = reproducibility\n        },\n    ),\n)\nprint(result.best_candidate, result.best_score)\n# held-out (only present if you passed test_set): \"test_score\" = average, \"test_scores\" = per-example\nprint(\"held-out:\", result.metadata.get(\"test_score\"),\n      \"seed held-out:\", result.metadata.get(\"baseline_test_score\"))\n```\n\n## Standard workflow\n1. **Pick the mode** (single-task / multi-task / generalization) by which of `dataset`/`valset` you pass.\n2. **Define the score deliberately.** The optimizer optimizes exactly what you measure — gate the\n   score on what you actually care about (see `references/gotchas.md`, reward hacking).\n3. **Write a feedback-rich `evaluate`.** The `info` dict is the proposer's signal — return errors,\n   diffs, partial credit, not just a number (`oa.log()` and `capture_stdio` can route diagnostics in\n   automatically). See `references/writing_evaluators.md`.\n4. **Pick a proposer LLM** — a LiteLLM id (set the provider key) or a custom LM-protocol callable.\n   Validate it with a 1-call test before a long run.\n5. **Set a budget** (`max_evals` sized per above, and/or `max_token_cost`) plus `stop_at_score` when\n   the metric has a ceiling.\n6. **Run `python scripts/preflight.py`** to fail fast on missing creds / CLI before a long run.\n7. **Launch**, watch the first 1-2 evals (the eval→model→score chain), then let it run.\n8. **Read `result.best_candidate` and `run_dir/`** (and `result.metadata[\"test_score\"]` if you\n   passed a `test_set`).\n\n## Critical gotchas (read before a real run)\nThese silently degrade *results* — skim before launching:\n- **Reward hacking.** Every backend optimizes exactly what you score; a weak proxy gets gamed (e.g. a\n  \"correct\"-only score → a do-nothing wrapper). Gate the score on the real goal. → `references/gotchas.md`.\n- **Selection bias.** In generalization mode the best candidate is the max over many scored on\n  `valset` — an optimistic estimate. Use enough `valset` examples (and N>1 for stochastic models), and\n  report on a `test_set` for an unbiased number. → `references/gotchas.md`.\n- **Stochastic models default to N=1 per eval** → noisy selection. Average N samples *inside*\n  `evaluate`. → `references/writing_evaluators.md`.\n- **Saturated signal → the gepa backend returns the seed unchanged.** If the seed already aces the\n  training examples, every proposal looks \"not better\" and is rejected (many proposals, ~0 accepted).\n  Reflection needs examples the seed gets *wrong* to learn from — ensure `dataset` has real\n  failures. → `references/gotchas.md`.\n- **`engine_config` is validated strictly per backend** — an unknown key (including a leftover key\n  from a different backend after swapping `engine=`) raises `TypeError` at construction. Swapping\n  `engine=` means swapping the `engine_config` block. → `references/api.md`.\n- **Agentic backends (`autoresearch`, `meta_harness`) shell out to the `claude` CLI** and abort at\n  launch with install instructions if it's missing — likewise for `bwrap` (bubblewrap) on Linux,\n  which the default `sandbox=True` needs. `sandbox=False` runs the agent unconfined (loud warning).\n  Run `scripts/preflight.py` first; details in `references/api.md`.\n\n## Reference files (load as needed)\n- `references/api.md` — `OptimizeAnythingConfig`, the backends and their typed `engine_config`\n  options, the three modes, the LM protocol, budget/cost semantics, `GEPAResult` shape, and the\n  composition/pipeline helpers.\n- `references/writing_evaluators.md` — the `(score, info)` contract, `oa.log()`/`capture_stdio`,\n  LLM-as-judge scoring, multi-objective via `info[\"scores\"]`, N>1 averaging, feedback design.\n- `references/tracking.md` — enabling wandb / mlflow experiment tracking and what gets logged.\n- `references/gotchas.md` — reward hacking, selection bias, the three modes, backend prerequisites.\n- `scripts/preflight.py` — validate creds / proposer LM / `claude` CLI before launching.\n",".claude/skills/gepa-optimize-anything/references/tracking.md":"# Experiment tracking\n\nThe gepa backend logs to **Weights & Biases** and **MLflow** via `TrackingConfig` — configure it\nunder `engine_config[\"tracking\"]` (a valid `GEPAConfig` field, coerced to `TrackingConfig`). The\nother backends have no built-in tracker; for them, rely on the eval server's `output_dir` records\n(always written, whatever the backend):\n\n```python\nresult = optimize_anything(\n    seed_candidate=SEED, evaluator=evaluate, dataset=trainset, valset=valset,\n    config=OptimizeAnythingConfig(\n        engine=\"gepa\", max_evals=300,\n        engine_config={\n            \"reflection\": {\"reflection_lm\": \"anthropic/claude-sonnet-4-6\"},\n            \"tracking\": {                       # -> TrackingConfig\n                \"use_wandb\": True,\n                \"wandb_init_kwargs\": {\"project\": \"my-gepa-run\", \"name\": \"experiment-1\"},\n            },\n        },\n    ),\n)\n```\n\n## `TrackingConfig` fields\n- `use_wandb`, `wandb_api_key`, `wandb_init_kwargs`, `wandb_attach_existing`, `wandb_step_metric`\n- `use_mlflow`, `mlflow_tracking_uri`, `mlflow_experiment_name`, `mlflow_attach_existing`\n- `key_prefix` — prepended to every logged key/name (e.g. `\"gepa/\"` → `gepa/val_score`)\n- `logger` — a custom `LoggerProtocol` for plain-text log lines\n\n## What gets logged\n- **scalars** — `val_program_average`, `best_score_on_valset`, `total_metric_calls`, …\n- **tables** — `candidates`, `proposals`, `valset_scores`, `valset_pareto_front`\n- **run summary** — seed and best-found component text\n- **interactive candidate-tree HTML**\n\n## Attaching to an existing run\nIf you've already called `wandb.init()` (or started an MLflow run) yourself — e.g. a parent script\nthat also logs other things — set `wandb_attach_existing=True` (or `mlflow_attach_existing=True`).\nThe tracker then logs **into your active run** without calling `init()`/`finish()`, so it won't\ndisrupt the run lifecycle. Combine with `key_prefix=\"gepa/\"` to keep its metrics in their own\nnamespace.\n\nWhen the host loop manages its own wandb step counter, also set `wandb_step_metric` (e.g.\n`\"gepa/iteration\"`): the tracker then declares its own x-axis via `wandb.define_metric` instead of\npassing `step=`, so its `1, 2, 3, …` steps don't collide with the host's counter (a collision makes\nwandb silently drop the data).\n\n## Custom hooks\nFor programmatic observation beyond metric logging, pass `engine_config={\"callbacks\": [...]}` —\n`GEPACallback` objects receive events like `on_optimization_start`, `on_iteration_end`,\n`on_candidate_accepted`, `on_proposal_end`. The eval server also always writes a flat record\n(per-eval JSON + `progress_log.jsonl` + `summary.json`) under `output_dir`, whatever tracking is\nconfigured.\n\nSee the official guide for the authoritative list of fields and behaviors:\n<https://gepa-ai.github.io/gepa/guides/experiment-tracking/>.\n",".claude/skills/gepa-optimize-anything/.claude-plugin/plugin.json":"{\n  \"name\": \"gepa-optimize-anything\",\n  \"version\": \"0.1.0\",\n  \"description\": \"Optimize any scorable text artifact (prompts, code, configs, agent scaffolds, search solutions) with optimize_anything — one evaluator-driven API across the GEPA optimizer (reflective evolutionary search, the default), AutoResearch, and MetaHarness backends, with a best-of-N baseline for comparisons.\",\n  \"author\": { \"name\": \"GEPA\", \"url\": \"https://github.com/gepa-ai/gepa\" },\n  \"homepage\": \"https://gepa-ai.github.io/gepa/\",\n  \"repository\": \"https://github.com/gepa-ai/gepa\",\n  \"license\": \"MIT\",\n  \"keywords\": [\"gepa\", \"optimization\", \"prompt-optimization\", \"optimize-anything\", \"agent-skill\", \"dspy\"]\n}\n",".claude/skills/gepa-optimize-anything/references/writing_evaluators.md":"# Writing evaluators\n\nThe evaluator is where almost all of your effort and almost all of the quality come from. Whatever\nbackend you choose can only be as good as the score you compute and the feedback you return.\n\n## The contract\n```python\ndef evaluate(candidate: str, example) -> tuple[float, dict]:\n    return score, info\n```\n- `score: float` — **higher is better**. This is what the optimizer maximizes and selects on.\n- `info: dict` — **free-form feedback shown to the proposer**: the gepa backend's reflection LM\n  (as \"Actionable Side Information\", ASI) or the agentic backends' agent (the `best_of_n` baseline\n  ignores it). This is the single biggest lever on mutation quality.\n- For single-task runs the signature is `evaluate(candidate)`; with `dataset`/`valset`/`test_set`\n  it's `evaluate(candidate, example)`. (Returning a bare `float` instead of a tuple also works —\n  the wrapper normalizes it — but then the proposer gets no feedback; always return the tuple.)\n\n### Batched form (`batch_evaluator=`)\nWhen evals batch better than they stream — a provider batch API, one job submission for a whole\nstage, fan-out over your own infrastructure — write the batched form instead of (or alongside) the\nper-pair one:\n```python\ndef batch_evaluate(pairs: list[tuple[str, Any]]) -> list:  # pairs = [(candidate, example), ...]\n    return [score_or_score_info_tuple for candidate, example in pairs]  # one result per pair, same order\n```\nEach evaluation stage (minibatch, valset, held-out test pass) arrives as ONE call with all its\npairs. Everything above about feedback-rich `info` applies per pair. Put diagnostics in each\nreturned `info` — the per-call channels below (`oa.log()`, `capture_stdio`) don't apply to the\ngrouped call.\n\n## Make `info` feedback-rich (not just a number)\nThe proposer (reflection LM or agent) writes the next candidate by reading `info`. Give it specifics:\n```python\nreturn score, {\n    \"score\": score,\n    \"output\": output,                 # what the candidate produced\n    \"expected\": example.get(\"gold\"),  # what was wanted (if available)\n    \"error_type\": err_type,           # compile error / wrong answer / format violation / timeout\n    \"error_detail\": traceback_or_diff,# the actual message — concrete > vague\n    \"passed_checks\": [...],           # partial credit signals\n    \"failed_checks\": [...],\n}\n```\nRule of thumb: if a smart human reading only `info` could tell you *how to fix the candidate*, the\nproposer LLM can too. If `info` is just `{\"score\": 0.0}`, the search is blind.\n\n### Two built-in channels for diagnostics\n- **`oa.log()`** — `import gepa.gepa_launcher as oa; oa.log(\"landing distance:\", d)` inside your\n  evaluator. Same calling convention as `print()`; output is captured per-eval (thread-safe) and\n  auto-included in the feedback under `info[\"log\"]`. For child threads, propagate the context via\n  `oa.get_log_context()` / `oa.set_log_context()`.\n- **`capture_stdio`** — set `engine_config={\"engine\": {\"capture_stdio\": True}}` (gepa backend) and\n  any `print()`/stdout/stderr during evaluation lands in the feedback under `\"stdout\"`/`\"stderr\"` —\n  useful for wrapping existing scripts with no code changes. (Doesn't catch C-extension or\n  subprocess output that bypasses Python's `sys.stdout`; route that through `oa.log()`.)\n\n## LLM-as-judge scoring (subjective tasks)\nThe score doesn't have to come from an objective metric. For open-ended tasks (writing quality,\nhelpfulness, tone, adherence to a rubric) the evaluator can call an **LLM judge** and use its rating\nas the score — and, importantly, return the judge's **written critique** as feedback:\n```python\ndef evaluate(candidate, example):\n    output = run_my_model(candidate, example)\n    verdict = judge_lm(JUDGE_RUBRIC.format(task=example, answer=output))  # your judge call\n    score = verdict[\"rating\"] / 10.0                  # normalize to a float, higher = better\n    return score, {\n        \"score\": score,\n        \"output\": output,\n        \"critique\": verdict[\"critique\"],              # the judge's reasoning → drives better proposals\n        \"rubric_breakdown\": verdict.get(\"by_criterion\"),\n    }\n```\nTips: pin the judge (fixed model + temperature, ideally a stronger model than the one being\noptimized), give it a concrete rubric, and average a few judge calls if its ratings are noisy\n(this is the N>1 idea below). The critique is often more valuable to the proposer than the number.\n\n## Stochastic models: beat the silent N=1 default\nThe eval server calls your function **once** per (candidate, example) — there is no\n`samples_per_eval` knob. For a temperature>0 model that makes every score a single-sample estimate,\nand candidate **selection** then runs on noisy numbers (this is how you get winner's-curse inflation).\nFix it inside `evaluate` by averaging N samples:\n```python\ndef evaluate(candidate, example, N=4):\n    outs = [run_my_model(candidate, example) for _ in range(N)]\n    scores = [grade(o, example) for o in outs]\n    score = sum(scores) / len(scores)           # mean correctness ~ pass@1 estimate\n    return score, {\"score\": score, \"n\": N,\n                   \"samples\": [{\"out\": o, \"s\": s} for o, s in zip(outs, scores)]}\n```\nTrade-off: N× more eval calls (budget). Pick N to balance variance vs `max_evals`.\n\n## Multi-objective optimization (e.g. correctness AND speed)\nThe gepa backend keeps an objective-level Pareto front (the other backends ignore\n`info[\"scores\"]`). To use it, **return per-objective metrics under `info[\"scores\"]`** — the adapter\nforwards them as `objective_scores`:\n```python\nreturn score, {\n    \"score\": score,                        # the scalar the optimizer selects on\n    \"scores\": {\"correct\": correct_rate,    # per-objective metrics for the Pareto frontier\n               \"speedup\": speedup},        # all \"higher is better\"\n    ...\n}\n```\n`engine_config={\"engine\": {\"frontier_type\": \"hybrid\"}}` is the default (instance- and\nobjective-level fronts combined); `\"objective\"`, `\"instance\"`, and `\"cartesian\"` are the\nalternatives. The scalar `score` still drives final selection; the per-objective scores shape the\nfrontier that candidates are drawn from.\n\n## Choose the score to match the real goal (avoid reward hacking)\nThe optimizer maximizes exactly what you write. A correctness-only score is gameable — e.g. in code/kernel\ntasks the optimizer learns to emit a trivial wrapper that is \"correct\" but does nothing useful.\nPrefer a **gated** objective:\n```python\ndef score_fn(result):\n    if not (result[\"compiled\"] and result[\"correct\"]):\n        return 0.0\n    return f(result[\"speedup\"])   # e.g. min(speedup / target, 1.0), monotonically increasing\n```\nThen the only way to raise the score is to be correct **and** better on what you care about. See\n`gotchas.md` for the full reward-hacking story.\n\n## Determinism & robustness\n- Make `evaluate` side-effect-free and resumable; it may run concurrently (`max_concurrency`,\n  `engine.max_workers`) and be retried.\n- Set a seed in `engine_config={\"engine\": {\"seed\": 0}}` for reproducible search order.\n- Log your own per-eval record (id, score, sub-metrics, candidate hash) — you'll want it for\n  analysis; the eval server's `output_dir` per-eval JSON covers the basics, and `oa.log()` covers\n  in-feedback diagnostics.\n- Catch and *return* failures as low scores with `info[\"error_*\"]`, rather than raising —\n  `EngineConfig.raise_on_exception` defaults to `True`, so an uncaught exception aborts the run.\n  (Setting it to `False` converts exceptions to score `0.0` with `info[\"error\"]` instead.)\n",".claude/skills/gepa-optimize-anything/scripts/preflight.py":"#!/usr/bin/env python3\n\"\"\"Pre-flight checks for an optimize_anything run (gepa package) — fail fast before a long job.\n\n    python preflight.py                      # checks gepa + reflection-LM creds\n    python preflight.py --engine autoresearch  # also checks the `claude` CLI + jq\n    GEPA_REFLECTION_LM=anthropic/claude-sonnet-4-6 python preflight.py --test-lm\n\nExit code 0 = all good; non-zero = at least one blocker.\n\"\"\"\nfrom __future__ import annotations\n\nimport argparse\nimport os\nimport shutil\nimport sys\n\nOK, BAD = \"\\033[32mOK\\033[0m\", \"\\033[31mFAIL\\033[0m\"\nproblems: list[str] = []\n\n# The gepa backend's reflection LM defaults to openai/gpt-5.1; best_of_n's\n# sampling model defaults to claude-sonnet-4-6 (see references/api.md).\nDEFAULT_LM_BY_ENGINE = {\"gepa\": \"openai/gpt-5.1\", \"best_of_n\": \"claude-sonnet-4-6\"}\n\n\ndef check(label: str, ok: bool, fix: str = \"\") -> None:\n    print(f\"  [{OK if ok else BAD}] {label}\")\n    if not ok:\n        problems.append(f\"{label} — {fix}\" if fix else label)\n\n\ndef _creds_for(lm: str) -> tuple[bool, str]:\n    \"\"\"Best-effort provider-credential check for a LiteLLM model id.\"\"\"\n    has_aws = bool(\n        os.environ.get(\"AWS_BEARER_TOKEN_BEDROCK\")\n        or os.environ.get(\"AWS_ACCESS_KEY_ID\")\n        or os.environ.get(\"AWS_PROFILE\")\n    )\n    if \"bedrock\" in lm:\n        return has_aws, \"export AWS creds (AWS_BEARER_TOKEN_BEDROCK / AWS_ACCESS_KEY_ID / AWS_PROFILE)\"\n    if lm.startswith(\"openai/\") or lm.startswith(\"gpt-\") or \"gpt-5\" in lm:\n        return bool(os.environ.get(\"OPENAI_API_KEY\")), \"export OPENAI_API_KEY\"\n    if \"claude\" in lm or lm.startswith(\"anthropic/\"):\n        return bool(os.environ.get(\"ANTHROPIC_API_KEY\")) or has_aws, \"export ANTHROPIC_API_KEY (or AWS creds)\"\n    # Unknown provider: accept any common key being present.\n    any_key = bool(\n        os.environ.get(\"OPENAI_API_KEY\") or os.environ.get(\"ANTHROPIC_API_KEY\") or has_aws\n    )\n    return any_key, \"export your LiteLLM provider's API key\"\n\n\ndef main() -> int:\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--engine\", default=\"gepa\",\n                    choices=[\"gepa\", \"best_of_n\", \"autoresearch\", \"meta_harness\"])\n    ap.add_argument(\"--test-lm\", action=\"store_true\",\n                    help=\"make a 1-call round-trip to the reflection LM (costs a few tokens)\")\n    a = ap.parse_args()\n\n    print(\"== optimize_anything preflight ==\")\n\n    # 1) import + the correct API surface\n    try:\n        import gepa  # noqa\n        from gepa.optimize_anything import OptimizeAnythingConfig, optimize_anything  # noqa\n        check(f\"import gepa ({getattr(gepa, '__version__', '?')}) + optimize_anything\", True)\n    except Exception as e:  # noqa\n        check(\"import gepa + optimize_anything\", False, \"pip install 'gepa[full]'\")\n        print(f\"      {e}\")\n        return _report()\n\n    # 2) LM credentials (in-process engines that call an LLM directly)\n    lm = os.environ.get(\"GEPA_REFLECTION_LM\", \"\")\n    if a.engine in (\"gepa\", \"best_of_n\"):\n        effective_lm = lm or DEFAULT_LM_BY_ENGINE[a.engine]\n        if not lm:\n            print(f\"      GEPA_REFLECTION_LM unset -> engine default '{effective_lm}'\")\n        ok, fix = _creds_for(effective_lm)\n        check(f\"LLM creds present for '{effective_lm}'\", ok, fix)\n\n    # 3) agentic engines need the claude CLI (and autoresearch's eval.sh uses jq)\n    if a.engine in (\"autoresearch\", \"meta_harness\"):\n        cli = shutil.which(\"claude\")\n        check(f\"`claude` CLI on PATH (required by {a.engine})\", bool(cli),\n              \"install + authenticate the Claude Code CLI headless\")\n        if cli:\n            print(f\"      claude -> {cli}\")\n        if a.engine == \"autoresearch\":\n            check(\"`jq` on PATH (used by the generated eval.sh)\", bool(shutil.which(\"jq\")),\n                  \"install jq\")\n        if sys.platform.startswith(\"linux\"):\n            check(\"`bwrap` on PATH (default sandbox=True jails claude with bubblewrap)\",\n                  bool(shutil.which(\"bwrap\")),\n                  \"sudo apt/dnf install bubblewrap, or pass sandbox=False (runs unconfined)\")\n\n    # 4) optional live LM round-trip\n    if a.test_lm and a.engine in (\"gepa\", \"best_of_n\"):\n        target = lm or DEFAULT_LM_BY_ENGINE[a.engine]\n        try:\n            from gepa.lm import LM\n            out = LM(target)(\"Reply with the single word: ok\")\n            check(f\"LM 1-call round-trip ({target})\", bool(out),\n                  \"LM returned empty; check model id / creds / region\")\n        except Exception as e:  # noqa\n            check(f\"LM 1-call round-trip ({target})\", False, str(e)[:160])\n\n    return _report()\n\n\ndef _report() -> int:\n    print()\n    if problems:\n        print(f\"\\033[31m{len(problems)} blocker(s):\\033[0m\")\n        for p in problems:\n            print(f\"  - {p}\")\n        return 1\n    print(\"\\033[32mAll preflight checks passed.\\033[0m\")\n    return 0\n\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n",".claude/skills/gepa-optimize-anything/references/gotchas.md":"# Gotchas and pitfalls\n\nRead before any real run.\n\n## 1. Reward hacking — the optimizer exploits a weak proxy ruthlessly\nEvery backend maximizes exactly the score you compute; if it is gameable, it *will* be gamed. Real\nexample (KernelBench, gepa backend): a correctness-only score drove the optimizer to a prompt that\ntells the model to **wrap the reference implementation** — always \"correct\", zero real work. Result: pass@1 ~0.87 but the actual goal (fast kernels) collapsed to ~base.\nMitigation: a **gated objective** — score 0 unless valid+correct, then increasing in the thing you\ncare about (see `writing_evaluators.md`). Sanity-check the *winning* candidate against your true goal,\nnot just the score.\n\n## 2. Selection bias / winner's curse (generalization mode)\nIn generalization mode the chosen candidate is the **max** over many candidates scored on `valset`.\nWith noisy (especially N=1) scores over few examples, that max is optimistically inflated — it can sit\nwell above true generalization (e.g. a selected best of 0.87 with runner-up 0.60 and median 0.57).\nThis bias lives in **selection on `valset`**, so reduce it there:\n- use **enough `valset` examples**, and **N>1 samples per eval** for stochastic models, to shrink the\n  variance that inflates the max;\n- treat `best_score` (the selection-set number) as optimistic.\n\n`test_set` does **not** reduce this bias — it is reporting-only. But scoring the final candidate on a\nheld-out `test_set` gives you an *honest* number to report — `metadata[\"test_score\"]` (average; also\n`[\"test_scores\"]` per-example) vs `[\"baseline_test_score\"]` for the seed — separate from the inflated\nselection score. (Note the exact keys: `test_score`/`test_scores`, **not** `test_score(s)`.)\n\n## 3. Stochastic N=1 default\nOne eval call per (candidate, example). For a temperature>0 model that means single-sample scores,\nwhich feeds the selection bias in #2. Average N samples inside `evaluate` (`writing_evaluators.md`).\n\n## 4. The default budget is probably wrong for your run\n`max_evals` defaults to **100** — bounded, but rarely sized for your selection set (see SKILL.md's\nsizing rule: ≳ 15–20 × the selection set). Set it explicitly. Only if you set **both**\n`max_evals=None` and `max_token_cost=None` is the run unbounded, and even that is just a\n`warnings.warn`, not an error.\n\n## 5. `engine_config` is validated strictly — unknown keys raise `TypeError`\nEvery backend parses `engine_config` into a typed dataclass (`GEPAConfig` for gepa,\n`BestOfNConfig` / `AutoResearchConfig` / `MetaHarnessConfig` for the others), so a typo'd or\nwrong-backend key fails **immediately at construction** with a `TypeError` — it is *not* silently\ndropped. The practical consequence: swapping `engine=` requires swapping the whole `engine_config`\nblock, and old-API keys (`claude_code_agent`, top-level `reflection_lm_kwargs`, `objective`,\n`background` inside `engine_config`) now crash. See `api.md` for each backend's valid keys.\n\n## 6. Agentic backends have launch-time prerequisites\n`autoresearch` / `meta_harness` `subprocess.Popen([\"claude\", ...])`. A missing `claude` CLI — or, on\nLinux, a missing `bwrap` (bubblewrap) while the default `sandbox=True` is in effect — aborts the run\nat launch with a boxed message and install instructions (`npm install -g @anthropic-ai/claude-code`;\n`sudo apt/dnf install bubblewrap`). An *unauthenticated* CLI or a missing `jq` (used by\nautoresearch's generated `eval.sh`) still surfaces only mid-run, and `sandbox=False` runs the agent\nunconfined (loud warning) — so run `scripts/preflight.py` first either way.\n\n## 7. Give runs a real stop condition (`stop_at_score` / `max_token_cost`)\n`max_evals` caps eval calls, but two situations still burn money or time past the point of useful\nwork:\n- **The metric has a ceiling** and a candidate reaches it — without `stop_at_score` the run keeps\n  proposing at the optimum until the budget is gone. For a bounded metric (pass rate, accuracy),\n  *always* pass `stop_at_score`.\n- **You opted into evaluation caching** (`engine_config={\"engine\": {\"cache_evaluation\": True}}` —\n  it is **off by default**). Then `max_evals` counts only cache *misses*: a converged search keeps\n  emitting cache-hitting candidates without consuming eval budget and can spin until your process\n  times out, still spending proposer-LLM tokens. With caching on, `stop_at_score` and/or\n  `max_token_cost` (plus a wall-clock `timeout` on the launched process) are mandatory.\n  A distinct `valset` is a separate cache namespace from the trainset (list-position ids would\n  otherwise collide). `valset=None` still shares cached rollouts with minibatches. Resuming a\n  `run_dir` whose cache predates that namespacing drops the old entries.\nAgentic backends also spend LLM tokens between evals — cap them with `max_token_cost`\n(enforced as `--max-budget-usd`).\n\n## 8. Pick the right mode\nThe mode is implicit in which sets you pass (`api.md`): no `dataset`/`valset` → single-task;\n`dataset` only → multi-task; `dataset`+`valset` → generalization. A common mistake is wanting a\ncandidate that generalizes but passing only `dataset` (multi-task) — then there's no held-out\nselection and the candidate can overfit the training examples. Pass a `valset` for generalization.\n(Held-out `valset` selection is implemented by the gepa backend; the other backends fold `valset`\ninto the scoring pool — see `api.md`.)\n\n## 9. Saturated signal — the gepa backend returns the seed unchanged (no failure to learn from)\nThis one is specific to the gepa backend, whose optimizer improves by reflecting on examples the\ncurrent candidate gets **wrong**. If the seed already scores near-perfect on the train minibatch the\nproposer reflects over, every mutation looks \"not better\" and is rejected by the acceptance gate —\nso the run spends its full budget, accepts **zero** proposals, and hands back the seed. Symptom:\nmany proposals made, ~0 accepted, `best_candidate` == seed. This is \"baseline saturation,\" not a\nbug: there's no gradient for reflection to climb.\nMitigations: make sure `dataset` contains examples the seed actually **fails** (harder/larger train\nset, or a task-LM weak enough to generate failures); if the seed is already at ceiling, reflective\nmutation has little to do — the `best_of_n` baseline (independent full rewrites, no acceptance\ngate) or just keeping the seed may be the better call.\nAlways check **proposals accepted**, not just proposals made.\n\n## 10. Uncaught evaluator exceptions abort the run\nOn the gepa backend, `EngineConfig.raise_on_exception` defaults to `True`: an exception escaping\nyour evaluator stops the whole optimization. Either catch failures yourself and return them as low scores with rich\n`info[\"error_*\"]` feedback (preferred — the proposer learns from them), or set\n`engine_config={\"engine\": {\"raise_on_exception\": False}}` to have them auto-converted to score\n`0.0` with `info[\"error\"]`.\n\n## Quick pre-flight checklist\n- [ ] mode matches intent (single-task / multi-task / generalization) via `dataset`/`valset`\n- [ ] `dataset` contains examples the seed gets WRONG — reflection has no failure signal to learn\n      from otherwise, and the gepa backend hands back the seed unchanged (#9)\n- [ ] `score` matches the real goal (gated, not gameable)\n- [ ] `info` carries actionable feedback (errors/diffs/partial credit)\n- [ ] N>1 if the model is stochastic\n- [ ] proposer LM valid + 1-call tested; creds present (default `openai/gpt-5.1` needs\n      `OPENAI_API_KEY` — or set your own via `reflection.reflection_lm`)\n- [ ] `max_evals` set explicitly and sized for many proposals (≳ 15–20 × the size of the selection\n      set: `len(valset)` in generalization, `len(dataset)` in multi-task, or just ≳ 15–20 in\n      single-task) — see SKILL.md\n- [ ] `stop_at_score` set when the metric has a ceiling; `max_token_cost` for agentic backends\n- [ ] `engine_config` keys match the chosen backend (typos raise `TypeError`, #5)\n- [ ] `run_dir` + `output_dir` set (so artifacts persist)\n- [ ] `test_set` passed if you need an unbiased number to report\n- [ ] for agentic backends: `claude` CLI on PATH + authed, `jq` installed, and on Linux `bwrap`\n      (bubblewrap) for the default `sandbox=True` (`scripts/preflight.py`)\n"},"files":{".claude-plugin/marketplace.json":"{\n  \"name\": \"gepa\",\n  \"owner\": { \"name\": \"GEPA\", \"url\": \"https://github.com/gepa-ai/gepa\" },\n  \"description\": \"Official GEPA Claude Code plugins.\",\n  \"plugins\": [\n    {\n      \"name\": \"gepa-optimize-anything\",\n      \"source\": \"./.claude/skills/gepa-optimize-anything\",\n      \"description\": \"Optimize any scorable text artifact (prompts, code, configs, agent scaffolds) with GEPA's optimize_anything.\",\n      \"version\": \"0.1.0\",\n      \"homepage\": \"https://gepa-ai.github.io/gepa/\",\n      \"license\": \"MIT\",\n      \"category\": \"optimization\",\n      \"keywords\": [\"gepa\", \"prompt-optimization\", \"optimize-anything\"]\n    }\n  ]\n}\n","AGENTS.md":"# GEPA\n\nGEPA (Genetic-Pareto) is a Python framework for optimizing text components (AI prompts, code, instructions) using LLM-based reflection and Pareto-efficient evolutionary search.\n\n## Agent Skill\n\nThis repository ships an **Agent Skill** at `.claude/skills/gepa-optimize-anything/`. If you are a coding agent and the user wants to **auto-optimize, tune, or search over any scorable text artifact** (a prompt, program/code, config, regex/SQL, or agent scaffold) with `gepa.optimize_anything`, read `.claude/skills/gepa-optimize-anything/SKILL.md` first and follow it. It is auto-discovered by Claude Code and by other agents that read `.claude/skills/` (Cursor, VS Code/Copilot, Codex, Gemini CLI). To install it standalone in another project: `/plugin marketplace add gepa-ai/gepa` then `/plugin install gepa-optimize-anything@gepa`.\n\n## Setup\n\nWe use **uv** for dependency management. The project uses setuptools as the build backend. All python executions must be done through uv.\n\n```bash\nuv sync --extra dev\n```\n\n## Project Structure\n\n- `src/gepa/` — main package source\n  - `core/` — optimization loop, state, evaluation\n  - `proposer/` — candidate proposal and mutation logic\n  - `adapters/` — integration adapters (DSPy, RAG, MCP, etc.)\n  - `strategies/` — batch sampling and candidate selection\n  - `logging/` — experiment tracking and logging\n- `tests/` — pytest test suite\n- `docs/` — mkdocs documentation site\n\n## Build & Test\n\n```bash\nuv run pytest\nuv run ruff check src/\nuv run ruff format src/\nuv run pyright src/\n```\n\n## Code Style\n\n- Linter/formatter: ruff (line length 120, double quotes, space indent)\n- Type checking: pyright\n- Python target: 3.10+\n- No relative imports (enforced by ruff)\n",".claude/skills/gepa-optimize-anything/references/api.md":"# API reference — `optimize_anything`\n\nThe public entry point is `gepa.optimize_anything.optimize_anything`.\n\n## Signature\n```python\nfrom gepa.optimize_anything import GEPAResult, optimize_anything, OptimizeAnythingConfig\n\nresult = optimize_anything(\n    seed_candidate: str | None = None,   # starting text; None = seedless (engine bootstraps from objective/background)\n    *,\n    evaluator: Callable | None = None,       # (candidate) or (candidate, example) -> (score, info)\n    batch_evaluator: Callable | None = None, # list[(candidate, example)] -> one result per pair (see below)\n    dataset:  list | None = None,        # training examples used DURING optimization\n    valset:   list | None = None,        # validation examples used to SELECT the best candidate\n    objective: str | None = None,        # short goal, surfaced verbatim to the engine\n    background: str | None = None,       # long-form rules/constraints/domain notes\n    test_set: list | None = None,        # reporting-only: seed + final candidate scored here at the end\n    config: OptimizeAnythingConfig | None = None,\n) -> GEPAResult\n```\nExamples in `dataset`/`valset`/`test_set` are opaque — any object your `evaluator` understands.\n`seed_candidate` is a **single string** at this API (multi-component `dict[str, str]` candidates\nexist only in the lower-level `gepa.gepa_launcher.optimize_anything`).\n\n**`batch_evaluator`** — grouped scoring for when evals batch better than they stream (a provider\nbatch API, fan-out over your own cluster): it receives ALL `(candidate, example)` pairs of an\nevaluation stage (minibatch, valset, held-out test pass) in **one call** and returns one result per\npair (`score` or `(score, info)`). At least one of `evaluator` / `batch_evaluator` is required.\nWhen both are given, multi-pair stages use the batch function and single-pair evaluations use\n`evaluator`; with only `batch_evaluator`, singles route through it as singleton batches.\n\n### The three optimization modes (set by `dataset` / `valset`)\n| mode | pass | evaluator called | use when |\n|---|---|---|---|\n| **Single-task** | `dataset=None, valset=None` | `evaluator(candidate)` | solving one hard problem; the candidate *is* the solution (one kernel, a packing layout, one document). |\n| **Multi-task** | `dataset=<list>, valset=None` | `evaluator(candidate, example)` | one shared candidate must do well across a batch of related problems; insight transfers across them. |\n| **Generalization** | `dataset=<list>, valset=<list>` | `evaluator(candidate, example)` | the candidate must transfer to **unseen** problems: optimize on `dataset`, select on `valset`. |\n\n*Note on `valset`:* held-out selection is currently implemented only by the gepa backend (GEPA is\ndesigned around this mode — `valset` selection plus a Pareto frontier over val instances). The\nother backends fold `valset` into the pool the eval server exposes (`train_set + val_set` merged),\nso they optimize over the combined set with no separate selection split — they may still produce\ncandidates that generalize.\n\n**`test_set` is reporting-only and orthogonal to the mode.** Only `dataset` and `valset` affect\noptimization (search and selection). If you pass `test_set`, the seed and the final chosen candidate\nare scored on it *after* optimization, outside the budget, and reported in `result.metadata`. The\ntest set never enters the eval server's HTTP surface, so agentic backends cannot see it. Use it when\nyou want an unbiased number to report; skip it otherwise.\n\n## `OptimizeAnythingConfig` (top-level fields)\n| field | default | meaning |\n|---|---|---|\n| `engine` | `\"gepa\"` | `\"gepa\"` / `\"autoresearch\"` / `\"meta_harness\"` (or `\"best_of_n\"`, a baseline), or a constructed `Engine` instance (custom engines can be added via `gepa.oa.registry.register_engine`). |\n| `name` | `None` | run id (logging + default output dir). Auto-generated (`<engine>-<uuid>-<timestamp>`) if `None`. |\n| `max_evals` | **`100`** | server-side cap on eval calls. `None` = unlimited. Size it — the default is rarely right (see SKILL.md). |\n| `max_token_cost` | `None` | USD cap on the engine's **own** optimizer-LLM spend (reflection/agent). Enforced by the engine (gepa: `max_reflection_cost` stopper; agent engines: `--max-budget-usd`), not the eval server. |\n| `max_concurrency` | `8` | eval-server thread-pool size. |\n| `output_dir` | `None` | where the eval server writes per-eval JSON, `progress_log.jsonl`, `summary.json`. `None` → `outputs/optimize_anything/<task>/<engine>/<timestamp>/`. |\n| `run_dir` | `None` | engine workspace (gepa run dir / agent work dir; with `engine.write_agent_state=True` the gepa backend writes an agent-readable `iterations/` + `pareto/` tree here). Distinct from `output_dir`. `None` → subprocess engines use a tempdir; set it to persist artifacts. |\n| `stop_at_score` | `None` | early-stop score threshold; each engine interprets it. |\n| `sandbox` | **`True`** | OS-jail the subprocess engines' `claude` sessions: bwrap on Linux (needs the `bubblewrap` package — the run aborts at launch if `bwrap` is missing), Claude Code's Seatbelt sandbox on macOS. Also forces the work dir to a tempdir even when `run_dir` is set (artifacts are mirrored back). `False` prints a loud warning and runs the agent unsandboxed (`bypassPermissions`, full host access). |\n| `engine_config` | `{}` | dict of **engine-specific** options — see the per-backend sections below. Parsed into a typed per-engine config dataclass; **an unknown key raises `TypeError` immediately** (fail fast, not warn-and-drop). |\n\n**If both `max_evals` and `max_token_cost` are `None`, the run is unbounded** (only a\n`warnings.warn`). Note the former fields `tracker`, `effort`, and `max_thinking_tokens` no longer\nexist here: tracking is configured via the gepa backend's `engine_config[\"tracking\"]`\n(see `tracking.md`), and `effort` / `max_thinking_tokens` are per-agent-engine `engine_config` keys.\n\n## The backends\n\nAll are selected with `engine=` and run the **same** task/evaluator — switch optimizer by\nchanging one argument. Each backend parses `engine_config` into its own typed dataclass, so swapping\n`engine=` also means swapping the `engine_config` block (leftover keys → `TypeError`).\n\n| engine | how it proposes candidates | runs | needs |\n|---|---|---|---|\n| `gepa` | an LLM reflects on evaluator feedback and mutates the candidate; keeps a Pareto frontier | in-process | reflection-LM creds (default `openai/gpt-5.1`) or a custom LM |\n| `autoresearch` | one Claude Code subprocess iterates in a work dir (`program.md`, `candidate.txt`, `eval.sh` → HTTP eval server) | subprocess | `claude` on PATH + headless auth, `jq` |\n| `meta_harness` | a Claude subprocess reads frontier/history and writes `pending_eval.json` candidates; the engine benchmarks each | subprocess | `claude` on PATH + headless auth |\n| `best_of_n` *(baseline)* | independent single-shot samples from one LLM; keep the best — no feedback, no history | in-process | LiteLLM creds for `model` (default `claude-sonnet-4-6`) |\n\n`scripts/preflight.py` checks a backend's prerequisites before a long run.\n\n### `gepa` — `engine_config` is a `GEPAConfig`-shaped dict (1-to-1 passthrough)\nThe dict is passed to `gepa.gepa_launcher.GEPAConfig(**engine_config)`; valid top-level keys are\nexactly its fields — `engine`, `reflection`, `tracking`, `merge`, `refiner`, `callbacks`,\n`stop_callbacks` — and nested dicts are coerced to the matching dataclasses. The OA layer only\noverlays the eval budget (`max_evals` → `engine.max_metric_calls`), `run_dir`, `stop_at_score`, and\nthe `max_token_cost` → `engine.max_reflection_cost` cap.\n\n```python\nengine_config={\n    \"reflection\": {                     # -> ReflectionConfig\n        \"reflection_lm\": \"anthropic/claude-sonnet-4-6\",  # default \"openai/gpt-5.1\"; see \"Proposer LM\"\n        \"reflection_lm_kwargs\": {\"reasoning_effort\": \"high\"},  # litellm kwargs (temperature, thinking, …)\n        \"reflection_minibatch_size\": 5,  # default: 1 single-task, 3 otherwise\n        # \"custom_candidate_proposer\": ClaudeCodeAgentProposer(...),  # replace the reflection LM with\n        #                                # a Claude Code proposer (from gepa.oa.proposers)\n        # \"reflection_strategy\": ...,    # advanced: a ReflectionLM impl owning how reflection is called\n    },\n    \"engine\": {                          # -> EngineConfig (all optional, sensible defaults)\n        \"max_workers\": 32,               # parallel eval workers (default: cpu_count or 32)\n        \"seed\": 0,                       # reproducibility\n        \"frontier_type\": \"hybrid\",       # \"instance\" | \"objective\" | \"hybrid\" (default) | \"cartesian\"\n        \"candidate_selection_strategy\": \"pareto\",      # | \"current_best\" | \"epsilon_greedy\" | \"top_k_pareto\"\n        \"acceptance_criterion\": \"strict_improvement\",  # | \"improvement_or_equal\"\n        \"cache_evaluation\": False,       # opt-in: cache (candidate, split, example) evals.\n                                         # Distinct valset is isolated from trainset ids.\n                                         # valset=None still shares minibatch rollouts.\n        \"capture_stdio\": False,          # opt-in: route evaluator print() output into feedback\n        \"raise_on_exception\": True,      # False → evaluator exceptions become score 0 + info[\"error\"]\n        # \"write_agent_state\": True,     # agent-readable iterations/ + pareto/ tree under run_dir\n    },\n    \"tracking\": {\"use_wandb\": True},     # -> TrackingConfig (see references/tracking.md)\n    \"merge\":  {...},                     # -> MergeConfig (cross-candidate merging) or omit\n    \"refiner\": {...},                    # -> RefinerConfig (auto per-eval refinement) or omit\n    # \"callbacks\": [GEPACallback, ...],  # observation hooks (on_iteration_end, on_candidate_accepted, …)\n    # \"stop_callbacks\": [...],           # custom StopperProtocol stop conditions\n}\n```\nThe nested dataclasses live in `gepa.gepa_launcher` (`EngineConfig`, `ReflectionConfig`,\n`TrackingConfig`, `MergeConfig`, `RefinerConfig`); their knobs are documented in the GEPA guides —\ne.g. **candidate-selection**, **acceptance-criterion**, **batch-sampling**, **callbacks**,\n**cost-tracking**, **experiment-tracking** — see <https://gepa-ai.github.io/gepa/guides/>.\n\n### `autoresearch` — `engine_config` → `AutoResearchConfig`\n| key | default | meaning |\n|---|---|---|\n| `model` | `\"claude-sonnet-4-6\"` | Claude model id (pass `\"sonnet\"`/`\"opus\"` to track the current default). |\n| `ralph` | `True` | keep resuming one Claude session (`claude --resume`) while budget remains. |\n| `max_no_eval_seconds` | `None` | kill the subprocess after this long with no eval call. |\n| `handoffs` | `None` | prior-stage artifacts for sequential compositions (materialized under `handoff/`). |\n| `effort` | `None` | `claude --effort` value. |\n| `max_thinking_tokens` | `None` | fixed thinking-token budget (`MAX_THINKING_TOKENS`). |\n\nThe engine lays out a work dir (`program.md`, `candidate.txt`, `best_candidate.txt`, `eval.sh`) and\nlaunches `claude --print`; `eval.sh` POSTs candidates to the eval server, which enforces the budget\nserver-side (HTTP 429 on exhaustion) and caps LLM spend via `--max-budget-usd` (from\n`max_token_cost`). Train and val are presented to the agent as one combined pool; the test set is\nunreachable over HTTP.\n\n### `meta_harness` — `engine_config` → `MetaHarnessConfig`\n| key | default | meaning |\n|---|---|---|\n| `model` | `\"claude-sonnet-4-6\"` | proposer model id. |\n| `max_iterations` | `None` | hard cap on proposer sessions; `None` = until budget. |\n| `max_candidates_per_iter` | `3` | upper bound on candidates proposed per iteration. |\n| `effort` | `None` | `claude --effort` value. |\n| `max_thinking_tokens` | `None` | fixed thinking-token budget. |\n\nEach iteration the proposer subprocess reads the frontier + history state files, writes\n`pending_eval.json` with 1+ candidates, and the engine benchmarks each through the eval server.\n\n### `best_of_n` (baseline) — `engine_config` → `BestOfNConfig`\nDeliberately naive: each sample is one independent LLM call — no feedback, no history, no\nselection pressure beyond keep-the-best. Use it as a **comparison floor** when evaluating an\noptimizer, not as the optimizer. Stops on budget exhaustion, `stop_at_score`, or `max_n`.\n\n| key | default | meaning |\n|---|---|---|\n| `model` | `\"claude-sonnet-4-6\"` | LiteLLM model id used to sample candidates. |\n| `temperature` | `1.0` | sampling on by default so N calls don't collapse to one response. |\n| `max_n` | `None` | optional hard cap on samples; `None` = run until budget out. |\n| `lm_kwargs` | `{}` | extra kwargs forwarded to `gepa.lm.LM`. |\n| `effort` | `None` | threaded into the LM as `reasoning_effort`. |\n| `max_thinking_tokens` | `None` | fixed thinking-token budget for the LM. |\n\n## Proposer LM (not limited to LiteLLM)\n`reflection.reflection_lm` (gepa) accepts either:\n- a **model-id string** resolved through LiteLLM (`\"openai/gpt-5.1\"` — the default —\n  `\"anthropic/claude-sonnet-4-6\"`, a Bedrock inference-profile ARN, …); set the provider's\n  credentials, and pass litellm kwargs via `reflection.reflection_lm_kwargs`; or\n- **any object/callable implementing GEPA's LM protocol** — `__call__(prompt) -> str` (a\n  `str`-or-messages prompt in, completion text out). This is how you plug a **self-hosted or custom\n  inference engine** (vLLM, a local server, your own client) instead of LiteLLM.\n  `reflection_lm_kwargs` is ignored for callables.\n\n## Composition / pipeline helpers (ensembles)\n`gepa.optimize_anything` also exports helpers that compose several backends over the same\ntask/evaluator. Each takes the same flat task fields as `optimize_anything` (`seed_candidate`\npositional; `evaluator=`, `dataset=`, `valset=`, `objective=`, `background=`, `test_set=`, `name=`\nkeywords) plus `configs=` — a list of `OptimizeAnythingConfig`, one per stage/branch, each carrying\nits own budget. Budgets are pre-partitioned per config; an early-finishing stage's leftover is not\nredistributed.\n\n- `optimize_sequential` — **pipeline**: run configs in order; each stage's best becomes the next\n  stage's seed. Monotonic — the running best is carried forward, so a regressing stage doesn't\n  poison the chain.\n- `optimize_parallel` — run all configs concurrently; returns the list of `Result`s (config order).\n- `optimize_best_of` — parallel, then keep the result with the highest `best_score` (each engine's\n  own number — no re-scoring).\n- `optimize_vote` — parallel, then re-score each branch's best candidate once via `evaluator`\n  (outside any budget) and return the highest re-score. Use when engines score with different\n  quirks and you want a fair cross-engine comparison.\n- `optimize_adaptive_sequential` — **plateau scheduler**: give the active engine bounded eval\n  slices (`plateau_evals`) and rotate to the next config after `patience` non-improving slices,\n  always feeding the running best forward. The exception to per-config budgets: one shared eval\n  pool set by its `max_evals` keyword (per-config `max_evals` is ignored; per-config\n  `max_token_cost` still applies). Extra knobs: `min_evals_per_stage`, `improvement_epsilon`,\n  `cycle`, `max_switches`.\n\n```python\nfrom gepa.optimize_anything import optimize_sequential, OptimizeAnythingConfig\n\nresult = optimize_sequential(\n    SEED,\n    evaluator=evaluate,\n    dataset=trainset, valset=valset,\n    objective=\"...\",\n    configs=[\n        OptimizeAnythingConfig(engine=\"autoresearch\", max_evals=100, max_token_cost=5.0),\n        OptimizeAnythingConfig(engine=\"gepa\", max_evals=200),  # refines autoresearch's best\n    ],\n)\n```\n\n`*_with_server` variants (`optimize_sequential_with_server`, `optimize_parallel_with_server`, …)\ntake caller-owned `EvalServer`s — one per config (one shared server for\n`optimize_adaptive_sequential_with_server`) — for embedding in outer frameworks that route every\neval through their own server. Related: the autoresearch backend's `handoffs` key materializes\nprior-stage artifacts into the agent's work dir for hand-rolled sequential compositions.\n\n## `GEPAResult`\n\n`optimize_anything` always returns a `GEPAResult` (the same type as the legacy\n`gepa.gepa_launcher` path). The mutable engine-internal `Result` is still what\nengines and ensemble helpers thread; it is not the public return.\n\n```python\nresult.best_candidate   # str (or dict[str, str] for multi-component seeds)\nresult.best_score       # float (on valset/selection set)\nresult.total_evals      # int — eval-server call count (budget.used)\nresult.eval_log         # list[dict]\nresult.metadata         # dict (verified keys):\n#   \"test_score\"             avg over test_set         } only present\n#   \"test_scores\"            per-example dict          } if you passed\n#   \"baseline_test_score\"    seed avg over test_set    } a test_set\n#   \"baseline_test_scores\"   seed per-example dict     }\n#   \"budget\", \"total_cost\", \"adapter_cost\", \"wall_time\", \"engine\", \"output_dir\", \"progress_log\"\nresult.candidates            # full candidate pool (length 1 on non-gepa engines)\nresult.best_idx              # index of the highest val score\nresult.val_aggregate_scores  # per-candidate average validation score\nresult.to_dict()             # JSON-safe pool snapshot (does not include eval_log / metadata)\n```\n\nThe candidate pool lives on the result itself. `metadata` does **not** contain a nested\n`gepa_result`. `total_evals` is the eval-server count; `total_metric_calls` is GEPA core's\nown counter and may differ (e.g. after `BudgetExhausted`). `to_dict()` / `from_dict()` omit\n`eval_log`, `metadata`, and `eval_server_calls` so the frozen JSON pool contract stays\nbyte-stable.\n\nThe gepa engine also writes `run_dir/` artifacts, and the eval server writes\n`output_dir/summary.json`. (If the best candidate equals the seed, the test scores are reported from the\nseed's single scoring pass rather than re-scored.)\n\n**Migration:** callers that used `isinstance(result, Result)` or\n`result.metadata[\"gepa_result\"]` after `optimize_anything(...)` should switch to\n`GEPAResult` and `result.candidates`. Ensemble helpers (`optimize_sequential`, …)\nstill return the mutable `Result`.\n",".claude/skills/gepa-optimize-anything/SKILL.md":"---\nname: gepa-optimize-anything\ndescription: >-\n  Automatically improve any text artifact that can be scored — prompts, programs/code, configs,\n  specs, regex/SQL/schemas, agent scaffolds, or encoded search solutions — with optimize_anything,\n  an LLM-driven optimization API: a search backend repeatedly proposes better versions from\n  execution feedback and an evaluator you provide assigns the score (an objective metric or an\n  LLM-as-judge for subjective tasks). optimize_anything is one interface across several optimizer\n  backends — GEPA (reflective evolutionary search, the default), AutoResearch, MetaHarness — plus a\n  best-of-N baseline: write the evaluator once and switch backend with a single argument.\n  Use whenever a task needs auto-optimizing, tuning, or searching over text with any quality signal\n  (accuracy, pass rate, latency, cost, judge rating); when building an evaluator + proposer loop;\n  when running GEPA; or when comparing optimizers. The candidate can be any string an evaluator can\n  grade, scored on one problem or generalized across many.\n---\n\n# `optimize_anything`\n\n**Naming, precisely.** `optimize_anything` is the tool: a general API for optimizing text\nartifacts. **GEPA** is one specific optimizer behind it — reflective evolutionary search, the\ndefault backend (`engine=\"gepa\"`) — and, for legacy reasons, also the name of the Python package\nthat ships all of this. In this skill, \"the gepa backend\" always means the optimizer; statements\nabout \"the optimizer\" or \"the backend\" apply to whichever engine you chose.\n\n`optimize_anything` does **black-box optimization**: you provide (1) a seed artifact, (2) an\n**evaluator** that scores any artifact and returns feedback, and (3) a backend, which repeatedly\nproposes improved artifacts and scores them through your evaluator. \"Black-box\" refers to the\n**evaluator**, not the artifact: the backend never sees how the score is computed — no gradients,\nno metric internals — only the scalar score and the feedback text you emit. The candidate itself\n*is* visible: the proposer reads and rewrites it, applying the LLM's understanding of your artifact.\nThe framework just imposes no structure on it — any string an evaluator can score works. The\nleverage is in your score and your feedback.\n\n**You write the task and evaluator once, then choose the search algorithm with one `engine`\nargument** — and the same code runs under any of them:\n- **`gepa`** — the GEPA optimizer: reflective evolutionary search, in-process (an LLM reflects on\n  feedback and mutates candidates; keeps a Pareto frontier). The default; strongest when feedback\n  is rich.\n- **`autoresearch`** — an agentic optimizer: one Claude Code subprocess iterates like a researcher in\n  a work dir, scoring candidates through an HTTP eval server.\n- **`meta_harness`** — an agentic proposer (Claude subprocess) that reads the frontier/history each\n  iteration and writes new candidates for the engine to benchmark.\n\n(There is also a `best_of_n` engine — sample N independent candidates, keep the best. It is\ndeliberately naive: use it as a **baseline** to compare an optimizer against, not as the optimizer.)\n\nThis makes it easy to start with one backend and benchmark others on the identical task/evaluator.\nThere are also **composition/pipeline helpers** that combine backends over the same task:\n`optimize_sequential` (a pipeline — each stage's best seeds the next), `optimize_parallel`,\n`optimize_best_of`, `optimize_vote` (re-score each branch's best for a fair pick), and an adaptive\nscheduler that rotates backends on score plateaus — see `references/api.md`.\n\n## What can be a candidate\nA candidate is **any string your evaluator can score**. \"Text in → a number out (higher is better),\nplus optional feedback\" is the entire contract, which covers a wide range of artifacts:\n- **prompts** — system/user prompts, instruction templates, rubrics, few-shot exemplars\n- **programs / code** — functions, whole files, CUDA kernels, scored by compiling + running + benchmarking\n- **configs / specs / schemas / regex / SQL** — any text whose effect you can measure\n- **agent scaffolds** — tool instructions, planner/critic prompts, orchestration text\n- **pure search artifacts** — a mathematical construction, a packing layout, a plan, encoded as text\n\nThe score can be an **objective metric** (accuracy, pass rate, runtime, cost) **or an LLM-as-judge**\nrating for subjective tasks (writing quality, helpfulness, style). At this API the candidate is a\nsingle string (`seed_candidate: str | None`; `None` = seedless — the engine bootstraps from\n`objective`/`background`). Multi-component dict candidates exist only in the lower-level\n`gepa.gepa_launcher.optimize_anything` API, not here.\n\n## Three optimization modes (choose by how you pass data)\nThe mode is determined by whether you provide `dataset` and `valset`:\n1. **Single-task** (`dataset=None, valset=None`) — solve one hard problem; the candidate *is* the\n   solution; the evaluator is called with no example. *E.g. one CUDA kernel, a circle-packing layout.*\n2. **Multi-task** (`dataset=<list>, valset=None`) — solve a batch of related problems with one shared\n   candidate, transferring insight across them; evaluator called per example. *E.g. a single prompt\n   that works across many tasks.*\n3. **Generalization** (`dataset=<list>, valset=<list>`) — build a candidate that transfers to\n   **unseen** problems; optimize on `dataset`, select on `valset`. *E.g. a prompt tuned to generalize.*\n   *Note:* GEPA is the algorithm designed around this mode, and `valset`-based held-out selection is\n   currently implemented only by the gepa backend — the other backends fold `valset` into the\n   training pool (they can still generalize; there's just no separate selection split).\n\n`test_set` is **separate from the modes and reporting-only**: the seed and the final candidate are\nscored on it after optimization for an unbiased number — it never enters the search, selection, or\nbudget (for the agentic backends it is sealed at the eval server's HTTP layer, so the agent cannot\neven see it). See `references/api.md` for details and when to use each mode.\n\n## Install\n```bash\npip install \"gepa[full]\"   # [full] pulls cloudpickle — needed to pickle closure evaluators for\n                           # parallel workers / opt-in evaluation caching; plain `pip install gepa`\n                           # can fail there when your evaluator closes over data.\n# Proposer LLM: the gepa backend's reflection LM defaults to \"openai/gpt-5.1\" (a LiteLLM id) — set\n# that provider's key (OPENAI_API_KEY), or pass your own id (e.g. \"anthropic/claude-sonnet-4-6\" with\n# ANTHROPIC_API_KEY, or a Bedrock ARN with AWS creds). You can also pass any callable implementing\n# GEPA's LM protocol — a self-hosted / custom inference engine — instead of a model-id string.\n# Agentic backends (autoresearch, meta_harness) additionally need the `claude` CLI on PATH (plus\n# `jq` for the generated eval.sh):\nnpm install -g @anthropic-ai/claude-code   # or: curl -fsSL https://claude.ai/install.sh | bash\n#   ...then run `claude` once to authenticate.\n# On Linux they also need bubblewrap: the default sandbox=True jails the claude subprocess with\n# bwrap (`sudo apt install bubblewrap` / `sudo dnf install bubblewrap`) and the run aborts at\n# launch if it's missing. sandbox=False opts out — the agent then runs unsandboxed (loud warning).\n```\n\n## Mental model (4 pieces)\n1. **Candidate** — the text you optimize (a `str`). The seed is the start; `None` = seedless.\n2. **`evaluate(candidate, example) -> (score, info)`** — `score` is a float (higher is better);\n   `info` is a free-form dict of **feedback the backend's proposer reads** to make better candidates\n   (the gepa backend's reflection LM, or the agentic backends' agent; the `best_of_n` baseline\n   ignores feedback). When evals batch better than they stream (e.g. a provider batch API), pass\n   `batch_evaluator=` instead — all pending `(candidate, example)` pairs in one call; see\n   `references/api.md`.\n3. **Engine (backend)** — `\"gepa\"` (default), `\"autoresearch\"`, `\"meta_harness\"`, the `\"best_of_n\"`\n   baseline, or a constructed `Engine` instance.\n4. **Budget** — `max_evals` (server-side eval-call cap, **default 100**) and/or `max_token_cost`\n   (USD cap on the backend's own proposer-LLM spend). Setting both to `None` makes the run unbounded\n   (only a warning). **Size `max_evals` for many proposal rounds, not one** (see below) — this is the\n   most common way agents misuse this API.\n\n## Sizing the valset and the budget (read this — the #1 mistake)\n`max_evals` is the *main* control over how long the optimizer runs. Leave it at the default (100)\nwith a large valset, or set it too low, and the run stops after a **single proposal**, then reports\na \"best candidate\" that looks fine but is barely optimized.\n\n- **valset** — on the gepa backend the best candidate is selected by its scores on the valset, so\n  make it a **representative subset of your data**. There is no fixed size: pick what represents the task (a\n  handful for small/expensive tasks, more when data is cheap and plentiful — anywhere from a few to\n  hundreds). Bigger valset = less noisy selection but more eval calls per candidate.\n- **budget** — every proposed candidate is scored on the **whole selection set**, so size\n  `max_evals` off that set, not off a single proposal. Which set that is depends on the mode:\n  ```\n  generalization (dataset + valset):  max_evals ≳ 15–20 × len(valset)    # scored & selected on valset\n  multi-task     (dataset only):      max_evals ≳ 15–20 × len(dataset)   # scored & selected on dataset\n  single-task    (no dataset/valset): max_evals ≳ 15–20                  # 1 eval per candidate, so this\n                                                                         #   IS the number of proposals\n  ```\n  The constant is the same everywhere — **let the backend propose AND evaluate ~15–20 candidates**\n  (more if you can afford it). Anything much less and the run tries only a couple of candidates —\n  i.e. it's barely optimizing. (This arithmetic is exact for the gepa backend — and the best_of_n\n  baseline — which score every candidate on the full selection set; the agentic backends decide\n  themselves how to spend eval calls, so treat it as a floor.)\n\nAfter the run, check how many proposals actually happened (on the gepa backend,\n`result.candidates` is the full pool; with `engine.write_agent_state=True` the\n`run_dir/iterations/` tree shows each one). **If it stopped after one proposal, the budget was too\nlow** — raise it and rerun.\n\n### Give every run a real stop condition\n`max_evals` bounds eval calls, but add explicit stops so runs end at the right moment:\n- **`stop_at_score`** — set it whenever your metric has a known ceiling (e.g. `1.0` for a pass rate /\n  accuracy). The backend stops the moment a candidate reaches it instead of burning the rest of the\n  budget at the optimum.\n- **`max_token_cost`** — a hard USD cap on the backend's own proposer/agent LLM spend. Especially\n  important for the agentic backends (`autoresearch`, `meta_harness`), whose Claude subprocesses\n  spend tokens between eval calls.\n- **a wall-clock `timeout`** on the process you launch (e.g. `timeout 1200 python run.py`) as a\n  backstop.\n- If you **opt in** to evaluation caching (`engine_config={\"engine\": {\"cache_evaluation\": True}}` on\n  the gepa backend — it is **off by default**), be aware `max_evals` then counts only cache *misses*:\n  a converged search can keep proposing cache-hitting candidates without consuming eval budget, so\n  `stop_at_score`/`max_token_cost` become mandatory, not optional. A distinct `valset` is cached\n  separately from the trainset; `valset=None` still reuses minibatch rollouts.\n\n## Minimal working example\nThe example optimizes a system prompt for concreteness, but the **shape is identical** for any\ncandidate — swap `SEED` for a code file / config / etc. and have `evaluate` compile/run/measure it.\n```python\nfrom gepa.optimize_anything import optimize_anything, OptimizeAnythingConfig\n\nSEED = \"You are an expert. Solve the task. Output only the final answer.\"\n\ndef evaluate(candidate: str, example) -> tuple[float, dict]:\n    output = run_my_model(system_prompt=candidate, user_prompt=example[\"prompt\"])  # your call\n    score = grade(output, example)                                                 # float, higher=better\n    return score, {                       # everything here is shown to the proposer LLM\n        \"score\": score,\n        \"output\": output,\n        \"error\": example.get(\"error\"),    # concrete, actionable feedback drives good proposals\n    }\n\nresult = optimize_anything(\n    seed_candidate=SEED,\n    evaluator=evaluate,\n    dataset=trainset,            # optimize on these (multi-task/generalization mode)\n    valset=valset,               # select the best candidate on these (generalization mode)\n    test_set=testset,            # OPTIONAL, reporting-only: seed + final candidate scored here at the end\n    objective=\"Produce a prompt that maximizes task accuracy.\",\n    background=\"Domain rules, constraints, output format the model must follow.\",\n    config=OptimizeAnythingConfig(\n        engine=\"gepa\",               # swap to \"autoresearch\" / \"meta_harness\" — same code\n                                     #   (\"best_of_n\" runs the same way, as a comparison baseline)\n        name=\"my_run\",\n        max_evals=300,               # ≳ 15-20 × len(valset): enough for ~15-20 proposals (see above)\n        stop_at_score=1.0,           # stop at the optimum (set when your metric has a known ceiling)\n        max_concurrency=16,\n        run_dir=\"runs/my_run\",       # engine workspace (gepa run dir / agent work dir)\n        output_dir=\"outputs/my_run\", # eval server: per-eval JSON, progress_log.jsonl, summary.json\n        engine_config={              # gepa backend: a GEPAConfig-shaped dict, validated strictly —\n            \"reflection\": {          #   an unknown key raises TypeError immediately (fail fast)\n                # a LiteLLM id (set the provider key) OR any callable implementing the LM protocol\n                \"reflection_lm\": \"anthropic/claude-sonnet-4-6\",\n                \"reflection_minibatch_size\": 5,\n            },\n            \"engine\": {\"max_workers\": 32, \"seed\": 0},  # seed = reproducibility\n        },\n    ),\n)\nprint(result.best_candidate, result.best_score)\n# held-out (only present if you passed test_set): \"test_score\" = average, \"test_scores\" = per-example\nprint(\"held-out:\", result.metadata.get(\"test_score\"),\n      \"seed held-out:\", result.metadata.get(\"baseline_test_score\"))\n```\n\n## Standard workflow\n1. **Pick the mode** (single-task / multi-task / generalization) by which of `dataset`/`valset` you pass.\n2. **Define the score deliberately.** The optimizer optimizes exactly what you measure — gate the\n   score on what you actually care about (see `references/gotchas.md`, reward hacking).\n3. **Write a feedback-rich `evaluate`.** The `info` dict is the proposer's signal — return errors,\n   diffs, partial credit, not just a number (`oa.log()` and `capture_stdio` can route diagnostics in\n   automatically). See `references/writing_evaluators.md`.\n4. **Pick a proposer LLM** — a LiteLLM id (set the provider key) or a custom LM-protocol callable.\n   Validate it with a 1-call test before a long run.\n5. **Set a budget** (`max_evals` sized per above, and/or `max_token_cost`) plus `stop_at_score` when\n   the metric has a ceiling.\n6. **Run `python scripts/preflight.py`** to fail fast on missing creds / CLI before a long run.\n7. **Launch**, watch the first 1-2 evals (the eval→model→score chain), then let it run.\n8. **Read `result.best_candidate` and `run_dir/`** (and `result.metadata[\"test_score\"]` if you\n   passed a `test_set`).\n\n## Critical gotchas (read before a real run)\nThese silently degrade *results* — skim before launching:\n- **Reward hacking.** Every backend optimizes exactly what you score; a weak proxy gets gamed (e.g. a\n  \"correct\"-only score → a do-nothing wrapper). Gate the score on the real goal. → `references/gotchas.md`.\n- **Selection bias.** In generalization mode the best candidate is the max over many scored on\n  `valset` — an optimistic estimate. Use enough `valset` examples (and N>1 for stochastic models), and\n  report on a `test_set` for an unbiased number. → `references/gotchas.md`.\n- **Stochastic models default to N=1 per eval** → noisy selection. Average N samples *inside*\n  `evaluate`. → `references/writing_evaluators.md`.\n- **Saturated signal → the gepa backend returns the seed unchanged.** If the seed already aces the\n  training examples, every proposal looks \"not better\" and is rejected (many proposals, ~0 accepted).\n  Reflection needs examples the seed gets *wrong* to learn from — ensure `dataset` has real\n  failures. → `references/gotchas.md`.\n- **`engine_config` is validated strictly per backend** — an unknown key (including a leftover key\n  from a different backend after swapping `engine=`) raises `TypeError` at construction. Swapping\n  `engine=` means swapping the `engine_config` block. → `references/api.md`.\n- **Agentic backends (`autoresearch`, `meta_harness`) shell out to the `claude` CLI** and abort at\n  launch with install instructions if it's missing — likewise for `bwrap` (bubblewrap) on Linux,\n  which the default `sandbox=True` needs. `sandbox=False` runs the agent unconfined (loud warning).\n  Run `scripts/preflight.py` first; details in `references/api.md`.\n\n## Reference files (load as needed)\n- `references/api.md` — `OptimizeAnythingConfig`, the backends and their typed `engine_config`\n  options, the three modes, the LM protocol, budget/cost semantics, `GEPAResult` shape, and the\n  composition/pipeline helpers.\n- `references/writing_evaluators.md` — the `(score, info)` contract, `oa.log()`/`capture_stdio`,\n  LLM-as-judge scoring, multi-objective via `info[\"scores\"]`, N>1 averaging, feedback design.\n- `references/tracking.md` — enabling wandb / mlflow experiment tracking and what gets logged.\n- `references/gotchas.md` — reward hacking, selection bias, the three modes, backend prerequisites.\n- `scripts/preflight.py` — validate creds / proposer LM / `claude` CLI before launching.\n",".claude/skills/gepa-optimize-anything/references/tracking.md":"# Experiment tracking\n\nThe gepa backend logs to **Weights & Biases** and **MLflow** via `TrackingConfig` — configure it\nunder `engine_config[\"tracking\"]` (a valid `GEPAConfig` field, coerced to `TrackingConfig`). The\nother backends have no built-in tracker; for them, rely on the eval server's `output_dir` records\n(always written, whatever the backend):\n\n```python\nresult = optimize_anything(\n    seed_candidate=SEED, evaluator=evaluate, dataset=trainset, valset=valset,\n    config=OptimizeAnythingConfig(\n        engine=\"gepa\", max_evals=300,\n        engine_config={\n            \"reflection\": {\"reflection_lm\": \"anthropic/claude-sonnet-4-6\"},\n            \"tracking\": {                       # -> TrackingConfig\n                \"use_wandb\": True,\n                \"wandb_init_kwargs\": {\"project\": \"my-gepa-run\", \"name\": \"experiment-1\"},\n            },\n        },\n    ),\n)\n```\n\n## `TrackingConfig` fields\n- `use_wandb`, `wandb_api_key`, `wandb_init_kwargs`, `wandb_attach_existing`, `wandb_step_metric`\n- `use_mlflow`, `mlflow_tracking_uri`, `mlflow_experiment_name`, `mlflow_attach_existing`\n- `key_prefix` — prepended to every logged key/name (e.g. `\"gepa/\"` → `gepa/val_score`)\n- `logger` — a custom `LoggerProtocol` for plain-text log lines\n\n## What gets logged\n- **scalars** — `val_program_average`, `best_score_on_valset`, `total_metric_calls`, …\n- **tables** — `candidates`, `proposals`, `valset_scores`, `valset_pareto_front`\n- **run summary** — seed and best-found component text\n- **interactive candidate-tree HTML**\n\n## Attaching to an existing run\nIf you've already called `wandb.init()` (or started an MLflow run) yourself — e.g. a parent script\nthat also logs other things — set `wandb_attach_existing=True` (or `mlflow_attach_existing=True`).\nThe tracker then logs **into your active run** without calling `init()`/`finish()`, so it won't\ndisrupt the run lifecycle. Combine with `key_prefix=\"gepa/\"` to keep its metrics in their own\nnamespace.\n\nWhen the host loop manages its own wandb step counter, also set `wandb_step_metric` (e.g.\n`\"gepa/iteration\"`): the tracker then declares its own x-axis via `wandb.define_metric` instead of\npassing `step=`, so its `1, 2, 3, …` steps don't collide with the host's counter (a collision makes\nwandb silently drop the data).\n\n## Custom hooks\nFor programmatic observation beyond metric logging, pass `engine_config={\"callbacks\": [...]}` —\n`GEPACallback` objects receive events like `on_optimization_start`, `on_iteration_end`,\n`on_candidate_accepted`, `on_proposal_end`. The eval server also always writes a flat record\n(per-eval JSON + `progress_log.jsonl` + `summary.json`) under `output_dir`, whatever tracking is\nconfigured.\n\nSee the official guide for the authoritative list of fields and behaviors:\n<https://gepa-ai.github.io/gepa/guides/experiment-tracking/>.\n",".claude/skills/gepa-optimize-anything/.claude-plugin/plugin.json":"{\n  \"name\": \"gepa-optimize-anything\",\n  \"version\": \"0.1.0\",\n  \"description\": \"Optimize any scorable text artifact (prompts, code, configs, agent scaffolds, search solutions) with optimize_anything — one evaluator-driven API across the GEPA optimizer (reflective evolutionary search, the default), AutoResearch, and MetaHarness backends, with a best-of-N baseline for comparisons.\",\n  \"author\": { \"name\": \"GEPA\", \"url\": \"https://github.com/gepa-ai/gepa\" },\n  \"homepage\": \"https://gepa-ai.github.io/gepa/\",\n  \"repository\": \"https://github.com/gepa-ai/gepa\",\n  \"license\": \"MIT\",\n  \"keywords\": [\"gepa\", \"optimization\", \"prompt-optimization\", \"optimize-anything\", \"agent-skill\", \"dspy\"]\n}\n",".claude/skills/gepa-optimize-anything/references/writing_evaluators.md":"# Writing evaluators\n\nThe evaluator is where almost all of your effort and almost all of the quality come from. Whatever\nbackend you choose can only be as good as the score you compute and the feedback you return.\n\n## The contract\n```python\ndef evaluate(candidate: str, example) -> tuple[float, dict]:\n    return score, info\n```\n- `score: float` — **higher is better**. This is what the optimizer maximizes and selects on.\n- `info: dict` — **free-form feedback shown to the proposer**: the gepa backend's reflection LM\n  (as \"Actionable Side Information\", ASI) or the agentic backends' agent (the `best_of_n` baseline\n  ignores it). This is the single biggest lever on mutation quality.\n- For single-task runs the signature is `evaluate(candidate)`; with `dataset`/`valset`/`test_set`\n  it's `evaluate(candidate, example)`. (Returning a bare `float` instead of a tuple also works —\n  the wrapper normalizes it — but then the proposer gets no feedback; always return the tuple.)\n\n### Batched form (`batch_evaluator=`)\nWhen evals batch better than they stream — a provider batch API, one job submission for a whole\nstage, fan-out over your own infrastructure — write the batched form instead of (or alongside) the\nper-pair one:\n```python\ndef batch_evaluate(pairs: list[tuple[str, Any]]) -> list:  # pairs = [(candidate, example), ...]\n    return [score_or_score_info_tuple for candidate, example in pairs]  # one result per pair, same order\n```\nEach evaluation stage (minibatch, valset, held-out test pass) arrives as ONE call with all its\npairs. Everything above about feedback-rich `info` applies per pair. Put diagnostics in each\nreturned `info` — the per-call channels below (`oa.log()`, `capture_stdio`) don't apply to the\ngrouped call.\n\n## Make `info` feedback-rich (not just a number)\nThe proposer (reflection LM or agent) writes the next candidate by reading `info`. Give it specifics:\n```python\nreturn score, {\n    \"score\": score,\n    \"output\": output,                 # what the candidate produced\n    \"expected\": example.get(\"gold\"),  # what was wanted (if available)\n    \"error_type\": err_type,           # compile error / wrong answer / format violation / timeout\n    \"error_detail\": traceback_or_diff,# the actual message — concrete > vague\n    \"passed_checks\": [...],           # partial credit signals\n    \"failed_checks\": [...],\n}\n```\nRule of thumb: if a smart human reading only `info` could tell you *how to fix the candidate*, the\nproposer LLM can too. If `info` is just `{\"score\": 0.0}`, the search is blind.\n\n### Two built-in channels for diagnostics\n- **`oa.log()`** — `import gepa.gepa_launcher as oa; oa.log(\"landing distance:\", d)` inside your\n  evaluator. Same calling convention as `print()`; output is captured per-eval (thread-safe) and\n  auto-included in the feedback under `info[\"log\"]`. For child threads, propagate the context via\n  `oa.get_log_context()` / `oa.set_log_context()`.\n- **`capture_stdio`** — set `engine_config={\"engine\": {\"capture_stdio\": True}}` (gepa backend) and\n  any `print()`/stdout/stderr during evaluation lands in the feedback under `\"stdout\"`/`\"stderr\"` —\n  useful for wrapping existing scripts with no code changes. (Doesn't catch C-extension or\n  subprocess output that bypasses Python's `sys.stdout`; route that through `oa.log()`.)\n\n## LLM-as-judge scoring (subjective tasks)\nThe score doesn't have to come from an objective metric. For open-ended tasks (writing quality,\nhelpfulness, tone, adherence to a rubric) the evaluator can call an **LLM judge** and use its rating\nas the score — and, importantly, return the judge's **written critique** as feedback:\n```python\ndef evaluate(candidate, example):\n    output = run_my_model(candidate, example)\n    verdict = judge_lm(JUDGE_RUBRIC.format(task=example, answer=output))  # your judge call\n    score = verdict[\"rating\"] / 10.0                  # normalize to a float, higher = better\n    return score, {\n        \"score\": score,\n        \"output\": output,\n        \"critique\": verdict[\"critique\"],              # the judge's reasoning → drives better proposals\n        \"rubric_breakdown\": verdict.get(\"by_criterion\"),\n    }\n```\nTips: pin the judge (fixed model + temperature, ideally a stronger model than the one being\noptimized), give it a concrete rubric, and average a few judge calls if its ratings are noisy\n(this is the N>1 idea below). The critique is often more valuable to the proposer than the number.\n\n## Stochastic models: beat the silent N=1 default\nThe eval server calls your function **once** per (candidate, example) — there is no\n`samples_per_eval` knob. For a temperature>0 model that makes every score a single-sample estimate,\nand candidate **selection** then runs on noisy numbers (this is how you get winner's-curse inflation).\nFix it inside `evaluate` by averaging N samples:\n```python\ndef evaluate(candidate, example, N=4):\n    outs = [run_my_model(candidate, example) for _ in range(N)]\n    scores = [grade(o, example) for o in outs]\n    score = sum(scores) / len(scores)           # mean correctness ~ pass@1 estimate\n    return score, {\"score\": score, \"n\": N,\n                   \"samples\": [{\"out\": o, \"s\": s} for o, s in zip(outs, scores)]}\n```\nTrade-off: N× more eval calls (budget). Pick N to balance variance vs `max_evals`.\n\n## Multi-objective optimization (e.g. correctness AND speed)\nThe gepa backend keeps an objective-level Pareto front (the other backends ignore\n`info[\"scores\"]`). To use it, **return per-objective metrics under `info[\"scores\"]`** — the adapter\nforwards them as `objective_scores`:\n```python\nreturn score, {\n    \"score\": score,                        # the scalar the optimizer selects on\n    \"scores\": {\"correct\": correct_rate,    # per-objective metrics for the Pareto frontier\n               \"speedup\": speedup},        # all \"higher is better\"\n    ...\n}\n```\n`engine_config={\"engine\": {\"frontier_type\": \"hybrid\"}}` is the default (instance- and\nobjective-level fronts combined); `\"objective\"`, `\"instance\"`, and `\"cartesian\"` are the\nalternatives. The scalar `score` still drives final selection; the per-objective scores shape the\nfrontier that candidates are drawn from.\n\n## Choose the score to match the real goal (avoid reward hacking)\nThe optimizer maximizes exactly what you write. A correctness-only score is gameable — e.g. in code/kernel\ntasks the optimizer learns to emit a trivial wrapper that is \"correct\" but does nothing useful.\nPrefer a **gated** objective:\n```python\ndef score_fn(result):\n    if not (result[\"compiled\"] and result[\"correct\"]):\n        return 0.0\n    return f(result[\"speedup\"])   # e.g. min(speedup / target, 1.0), monotonically increasing\n```\nThen the only way to raise the score is to be correct **and** better on what you care about. See\n`gotchas.md` for the full reward-hacking story.\n\n## Determinism & robustness\n- Make `evaluate` side-effect-free and resumable; it may run concurrently (`max_concurrency`,\n  `engine.max_workers`) and be retried.\n- Set a seed in `engine_config={\"engine\": {\"seed\": 0}}` for reproducible search order.\n- Log your own per-eval record (id, score, sub-metrics, candidate hash) — you'll want it for\n  analysis; the eval server's `output_dir` per-eval JSON covers the basics, and `oa.log()` covers\n  in-feedback diagnostics.\n- Catch and *return* failures as low scores with `info[\"error_*\"]`, rather than raising —\n  `EngineConfig.raise_on_exception` defaults to `True`, so an uncaught exception aborts the run.\n  (Setting it to `False` converts exceptions to score `0.0` with `info[\"error\"]` instead.)\n",".claude/skills/gepa-optimize-anything/scripts/preflight.py":"#!/usr/bin/env python3\n\"\"\"Pre-flight checks for an optimize_anything run (gepa package) — fail fast before a long job.\n\n    python preflight.py                      # checks gepa + reflection-LM creds\n    python preflight.py --engine autoresearch  # also checks the `claude` CLI + jq\n    GEPA_REFLECTION_LM=anthropic/claude-sonnet-4-6 python preflight.py --test-lm\n\nExit code 0 = all good; non-zero = at least one blocker.\n\"\"\"\nfrom __future__ import annotations\n\nimport argparse\nimport os\nimport shutil\nimport sys\n\nOK, BAD = \"\\033[32mOK\\033[0m\", \"\\033[31mFAIL\\033[0m\"\nproblems: list[str] = []\n\n# The gepa backend's reflection LM defaults to openai/gpt-5.1; best_of_n's\n# sampling model defaults to claude-sonnet-4-6 (see references/api.md).\nDEFAULT_LM_BY_ENGINE = {\"gepa\": \"openai/gpt-5.1\", \"best_of_n\": \"claude-sonnet-4-6\"}\n\n\ndef check(label: str, ok: bool, fix: str = \"\") -> None:\n    print(f\"  [{OK if ok else BAD}] {label}\")\n    if not ok:\n        problems.append(f\"{label} — {fix}\" if fix else label)\n\n\ndef _creds_for(lm: str) -> tuple[bool, str]:\n    \"\"\"Best-effort provider-credential check for a LiteLLM model id.\"\"\"\n    has_aws = bool(\n        os.environ.get(\"AWS_BEARER_TOKEN_BEDROCK\")\n        or os.environ.get(\"AWS_ACCESS_KEY_ID\")\n        or os.environ.get(\"AWS_PROFILE\")\n    )\n    if \"bedrock\" in lm:\n        return has_aws, \"export AWS creds (AWS_BEARER_TOKEN_BEDROCK / AWS_ACCESS_KEY_ID / AWS_PROFILE)\"\n    if lm.startswith(\"openai/\") or lm.startswith(\"gpt-\") or \"gpt-5\" in lm:\n        return bool(os.environ.get(\"OPENAI_API_KEY\")), \"export OPENAI_API_KEY\"\n    if \"claude\" in lm or lm.startswith(\"anthropic/\"):\n        return bool(os.environ.get(\"ANTHROPIC_API_KEY\")) or has_aws, \"export ANTHROPIC_API_KEY (or AWS creds)\"\n    # Unknown provider: accept any common key being present.\n    any_key = bool(\n        os.environ.get(\"OPENAI_API_KEY\") or os.environ.get(\"ANTHROPIC_API_KEY\") or has_aws\n    )\n    return any_key, \"export your LiteLLM provider's API key\"\n\n\ndef main() -> int:\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--engine\", default=\"gepa\",\n                    choices=[\"gepa\", \"best_of_n\", \"autoresearch\", \"meta_harness\"])\n    ap.add_argument(\"--test-lm\", action=\"store_true\",\n                    help=\"make a 1-call round-trip to the reflection LM (costs a few tokens)\")\n    a = ap.parse_args()\n\n    print(\"== optimize_anything preflight ==\")\n\n    # 1) import + the correct API surface\n    try:\n        import gepa  # noqa\n        from gepa.optimize_anything import OptimizeAnythingConfig, optimize_anything  # noqa\n        check(f\"import gepa ({getattr(gepa, '__version__', '?')}) + optimize_anything\", True)\n    except Exception as e:  # noqa\n        check(\"import gepa + optimize_anything\", False, \"pip install 'gepa[full]'\")\n        print(f\"      {e}\")\n        return _report()\n\n    # 2) LM credentials (in-process engines that call an LLM directly)\n    lm = os.environ.get(\"GEPA_REFLECTION_LM\", \"\")\n    if a.engine in (\"gepa\", \"best_of_n\"):\n        effective_lm = lm or DEFAULT_LM_BY_ENGINE[a.engine]\n        if not lm:\n            print(f\"      GEPA_REFLECTION_LM unset -> engine default '{effective_lm}'\")\n        ok, fix = _creds_for(effective_lm)\n        check(f\"LLM creds present for '{effective_lm}'\", ok, fix)\n\n    # 3) agentic engines need the claude CLI (and autoresearch's eval.sh uses jq)\n    if a.engine in (\"autoresearch\", \"meta_harness\"):\n        cli = shutil.which(\"claude\")\n        check(f\"`claude` CLI on PATH (required by {a.engine})\", bool(cli),\n              \"install + authenticate the Claude Code CLI headless\")\n        if cli:\n            print(f\"      claude -> {cli}\")\n        if a.engine == \"autoresearch\":\n            check(\"`jq` on PATH (used by the generated eval.sh)\", bool(shutil.which(\"jq\")),\n                  \"install jq\")\n        if sys.platform.startswith(\"linux\"):\n            check(\"`bwrap` on PATH (default sandbox=True jails claude with bubblewrap)\",\n                  bool(shutil.which(\"bwrap\")),\n                  \"sudo apt/dnf install bubblewrap, or pass sandbox=False (runs unconfined)\")\n\n    # 4) optional live LM round-trip\n    if a.test_lm and a.engine in (\"gepa\", \"best_of_n\"):\n        target = lm or DEFAULT_LM_BY_ENGINE[a.engine]\n        try:\n            from gepa.lm import LM\n            out = LM(target)(\"Reply with the single word: ok\")\n            check(f\"LM 1-call round-trip ({target})\", bool(out),\n                  \"LM returned empty; check model id / creds / region\")\n        except Exception as e:  # noqa\n            check(f\"LM 1-call round-trip ({target})\", False, str(e)[:160])\n\n    return _report()\n\n\ndef _report() -> int:\n    print()\n    if problems:\n        print(f\"\\033[31m{len(problems)} blocker(s):\\033[0m\")\n        for p in problems:\n            print(f\"  - {p}\")\n        return 1\n    print(\"\\033[32mAll preflight checks passed.\\033[0m\")\n    return 0\n\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n",".claude/skills/gepa-optimize-anything/references/gotchas.md":"# Gotchas and pitfalls\n\nRead before any real run.\n\n## 1. Reward hacking — the optimizer exploits a weak proxy ruthlessly\nEvery backend maximizes exactly the score you compute; if it is gameable, it *will* be gamed. Real\nexample (KernelBench, gepa backend): a correctness-only score drove the optimizer to a prompt that\ntells the model to **wrap the reference implementation** — always \"correct\", zero real work. Result: pass@1 ~0.87 but the actual goal (fast kernels) collapsed to ~base.\nMitigation: a **gated objective** — score 0 unless valid+correct, then increasing in the thing you\ncare about (see `writing_evaluators.md`). Sanity-check the *winning* candidate against your true goal,\nnot just the score.\n\n## 2. Selection bias / winner's curse (generalization mode)\nIn generalization mode the chosen candidate is the **max** over many candidates scored on `valset`.\nWith noisy (especially N=1) scores over few examples, that max is optimistically inflated — it can sit\nwell above true generalization (e.g. a selected best of 0.87 with runner-up 0.60 and median 0.57).\nThis bias lives in **selection on `valset`**, so reduce it there:\n- use **enough `valset` examples**, and **N>1 samples per eval** for stochastic models, to shrink the\n  variance that inflates the max;\n- treat `best_score` (the selection-set number) as optimistic.\n\n`test_set` does **not** reduce this bias — it is reporting-only. But scoring the final candidate on a\nheld-out `test_set` gives you an *honest* number to report — `metadata[\"test_score\"]` (average; also\n`[\"test_scores\"]` per-example) vs `[\"baseline_test_score\"]` for the seed — separate from the inflated\nselection score. (Note the exact keys: `test_score`/`test_scores`, **not** `test_score(s)`.)\n\n## 3. Stochastic N=1 default\nOne eval call per (candidate, example). For a temperature>0 model that means single-sample scores,\nwhich feeds the selection bias in #2. Average N samples inside `evaluate` (`writing_evaluators.md`).\n\n## 4. The default budget is probably wrong for your run\n`max_evals` defaults to **100** — bounded, but rarely sized for your selection set (see SKILL.md's\nsizing rule: ≳ 15–20 × the selection set). Set it explicitly. Only if you set **both**\n`max_evals=None` and `max_token_cost=None` is the run unbounded, and even that is just a\n`warnings.warn`, not an error.\n\n## 5. `engine_config` is validated strictly — unknown keys raise `TypeError`\nEvery backend parses `engine_config` into a typed dataclass (`GEPAConfig` for gepa,\n`BestOfNConfig` / `AutoResearchConfig` / `MetaHarnessConfig` for the others), so a typo'd or\nwrong-backend key fails **immediately at construction** with a `TypeError` — it is *not* silently\ndropped. The practical consequence: swapping `engine=` requires swapping the whole `engine_config`\nblock, and old-API keys (`claude_code_agent`, top-level `reflection_lm_kwargs`, `objective`,\n`background` inside `engine_config`) now crash. See `api.md` for each backend's valid keys.\n\n## 6. Agentic backends have launch-time prerequisites\n`autoresearch` / `meta_harness` `subprocess.Popen([\"claude\", ...])`. A missing `claude` CLI — or, on\nLinux, a missing `bwrap` (bubblewrap) while the default `sandbox=True` is in effect — aborts the run\nat launch with a boxed message and install instructions (`npm install -g @anthropic-ai/claude-code`;\n`sudo apt/dnf install bubblewrap`). An *unauthenticated* CLI or a missing `jq` (used by\nautoresearch's generated `eval.sh`) still surfaces only mid-run, and `sandbox=False` runs the agent\nunconfined (loud warning) — so run `scripts/preflight.py` first either way.\n\n## 7. Give runs a real stop condition (`stop_at_score` / `max_token_cost`)\n`max_evals` caps eval calls, but two situations still burn money or time past the point of useful\nwork:\n- **The metric has a ceiling** and a candidate reaches it — without `stop_at_score` the run keeps\n  proposing at the optimum until the budget is gone. For a bounded metric (pass rate, accuracy),\n  *always* pass `stop_at_score`.\n- **You opted into evaluation caching** (`engine_config={\"engine\": {\"cache_evaluation\": True}}` —\n  it is **off by default**). Then `max_evals` counts only cache *misses*: a converged search keeps\n  emitting cache-hitting candidates without consuming eval budget and can spin until your process\n  times out, still spending proposer-LLM tokens. With caching on, `stop_at_score` and/or\n  `max_token_cost` (plus a wall-clock `timeout` on the launched process) are mandatory.\n  A distinct `valset` is a separate cache namespace from the trainset (list-position ids would\n  otherwise collide). `valset=None` still shares cached rollouts with minibatches. Resuming a\n  `run_dir` whose cache predates that namespacing drops the old entries.\nAgentic backends also spend LLM tokens between evals — cap them with `max_token_cost`\n(enforced as `--max-budget-usd`).\n\n## 8. Pick the right mode\nThe mode is implicit in which sets you pass (`api.md`): no `dataset`/`valset` → single-task;\n`dataset` only → multi-task; `dataset`+`valset` → generalization. A common mistake is wanting a\ncandidate that generalizes but passing only `dataset` (multi-task) — then there's no held-out\nselection and the candidate can overfit the training examples. Pass a `valset` for generalization.\n(Held-out `valset` selection is implemented by the gepa backend; the other backends fold `valset`\ninto the scoring pool — see `api.md`.)\n\n## 9. Saturated signal — the gepa backend returns the seed unchanged (no failure to learn from)\nThis one is specific to the gepa backend, whose optimizer improves by reflecting on examples the\ncurrent candidate gets **wrong**. If the seed already scores near-perfect on the train minibatch the\nproposer reflects over, every mutation looks \"not better\" and is rejected by the acceptance gate —\nso the run spends its full budget, accepts **zero** proposals, and hands back the seed. Symptom:\nmany proposals made, ~0 accepted, `best_candidate` == seed. This is \"baseline saturation,\" not a\nbug: there's no gradient for reflection to climb.\nMitigations: make sure `dataset` contains examples the seed actually **fails** (harder/larger train\nset, or a task-LM weak enough to generate failures); if the seed is already at ceiling, reflective\nmutation has little to do — the `best_of_n` baseline (independent full rewrites, no acceptance\ngate) or just keeping the seed may be the better call.\nAlways check **proposals accepted**, not just proposals made.\n\n## 10. Uncaught evaluator exceptions abort the run\nOn the gepa backend, `EngineConfig.raise_on_exception` defaults to `True`: an exception escaping\nyour evaluator stops the whole optimization. Either catch failures yourself and return them as low scores with rich\n`info[\"error_*\"]` feedback (preferred — the proposer learns from them), or set\n`engine_config={\"engine\": {\"raise_on_exception\": False}}` to have them auto-converted to score\n`0.0` with `info[\"error\"]`.\n\n## Quick pre-flight checklist\n- [ ] mode matches intent (single-task / multi-task / generalization) via `dataset`/`valset`\n- [ ] `dataset` contains examples the seed gets WRONG — reflection has no failure signal to learn\n      from otherwise, and the gepa backend hands back the seed unchanged (#9)\n- [ ] `score` matches the real goal (gated, not gameable)\n- [ ] `info` carries actionable feedback (errors/diffs/partial credit)\n- [ ] N>1 if the model is stochastic\n- [ ] proposer LM valid + 1-call tested; creds present (default `openai/gpt-5.1` needs\n      `OPENAI_API_KEY` — or set your own via `reflection.reflection_lm`)\n- [ ] `max_evals` set explicitly and sized for many proposals (≳ 15–20 × the size of the selection\n      set: `len(valset)` in generalization, `len(dataset)` in multi-task, or just ≳ 15–20 in\n      single-task) — see SKILL.md\n- [ ] `stop_at_score` set when the metric has a ceiling; `max_token_cost` for agentic backends\n- [ ] `engine_config` keys match the chosen backend (typos raise `TypeError`, #5)\n- [ ] `run_dir` + `output_dir` set (so artifacts persist)\n- [ ] `test_set` passed if you need an unbiased number to report\n- [ ] for agentic backends: `claude` CLI on PATH + authed, `jq` installed, and on Linux `bwrap`\n      (bubblewrap) for the default `sandbox=True` (`scripts/preflight.py`)\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/gepa-ai/gepa/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# GEPA\n\nGEPA (Genetic-Pareto) is a Python framework for optimizing text components (AI prompts, code, instructions) using LLM-based reflection and Pareto-efficient evolutionary search.\n\n## Agent Skill\n\nThis repository ships an **Agent Skill** at `.claude/skills/gepa-optimize-anything/`. If you are a coding agent and the user wants to **auto-optimize, tune, or search over any scorable text artifact** (a prompt, program/code, config, regex/SQL, or agent scaffold) with `gepa.optimize_anything`, read `.claude/skills/gepa-optimize-anything/SKILL.md` first and follow it. It is auto-discovered by Claude Code and by other agents that read `.claude/skills/` (Cursor, VS Code/Copilot, Codex, Gemini CLI). To install it standalone in another project: `/plugin marketplace add gepa-ai/gepa` then `/plugin install gepa-optimize-anything@gepa`.\n\n## Setup\n\nWe use **uv** for dependency management. The project uses setuptools as the build backend. All python executions must be done through uv.\n\n```bash\nuv sync --extra dev\n```\n\n## Project Structure\n\n- `src/gepa/` — main package source\n  - `core/` — optimization loop, state, evaluation\n  - `proposer/` — candidate proposal and mutation logic\n  - `adapters/` — integration adapters (DSPy, RAG, MCP, etc.)\n  - `strategies/` — batch sampling and candidate selection\n  - `logging/` — experiment tracking and logging\n- `tests/` — pytest test suite\n- `docs/` — mkdocs documentation site\n\n## Build & Test\n\n```bash\nuv run pytest\nuv run ruff check src/\nuv run ruff format src/\nuv run pyright src/\n```\n\n## Code Style\n\n- Linter/formatter: ruff (line length 120, double quotes, space indent)\n- Type checking: pyright\n- Python target: 3.10+\n- No relative imports (enforced by ruff)\n","isInternal":false,"tokens":448,"sizeBytes":1730},{"name":"api.md","path":".claude/skills/gepa-optimize-anything/references/api.md","rawUrl":"https://raw.githubusercontent.com/gepa-ai/gepa/HEAD/.claude/skills/gepa-optimize-anything/references/api.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# API reference — `optimize_anything`\n\nThe public entry point is `gepa.optimize_anything.optimize_anything`.\n\n## Signature\n```python\nfrom gepa.optimize_anything import GEPAResult, optimize_anything, OptimizeAnythingConfig\n\nresult = optimize_anything(\n    seed_candidate: str | None = None,   # starting text; None = seedless (engine bootstraps from objective/background)\n    *,\n    evaluator: Callable | None = None,       # (candidate) or (candidate, example) -> (score, info)\n    batch_evaluator: Callable | None = None, # list[(candidate, example)] -> one result per pair (see below)\n    dataset:  list | None = None,        # training examples used DURING optimization\n    valset:   list | None = None,        # validation examples used to SELECT the best candidate\n    objective: str | None = None,        # short goal, surfaced verbatim to the engine\n    background: str | None = None,       # long-form rules/constraints/domain notes\n    test_set: list | None = None,        # reporting-only: seed + final candidate scored here at the end\n    config: OptimizeAnythingConfig | None = None,\n) -> GEPAResult\n```\nExamples in `dataset`/`valset`/`test_set` are opaque — any object your `evaluator` understands.\n`seed_candidate` is a **single string** at this API (multi-component `dict[str, str]` candidates\nexist only in the lower-level `gepa.gepa_launcher.optimize_anything`).\n\n**`batch_evaluator`** — grouped scoring for when evals batch better than they stream (a provider\nbatch API, fan-out over your own cluster): it receives ALL `(candidate, example)` pairs of an\nevaluation stage (minibatch, valset, held-out test pass) in **one call** and returns one result per\npair (`score` or `(score, info)`). At least one of `evaluator` / `batch_evaluator` is required.\nWhen both are given, multi-pair stages use the batch function and single-pair evaluations use\n`evaluator`; with only `batch_evaluator`, singles route through it as singleton batches.\n\n### The three optimization modes (set by `dataset` / `valset`)\n| mode | pass | evaluator called | use when |\n|---|---|---|---|\n| **Single-task** | `dataset=None, valset=None` | `evaluator(candidate)` | solving one hard problem; the candidate *is* the solution (one kernel, a packing layout, one document). |\n| **Multi-task** | `dataset=<list>, valset=None` | `evaluator(candidate, example)` | one shared candidate must do well across a batch of related problems; insight transfers across them. |\n| **Generalization** | `dataset=<list>, valset=<list>` | `evaluator(candidate, example)` | the candidate must transfer to **unseen** problems: optimize on `dataset`, select on `valset`. |\n\n*Note on `valset`:* held-out selection is currently implemented only by the gepa backend (GEPA is\ndesigned around this mode — `valset` selection plus a Pareto frontier over val instances). The\nother backends fold `valset` into the pool the eval server exposes (`train_set + val_set` merged),\nso they optimize over the combined set with no separate selection split — they may still produce\ncandidates that generalize.\n\n**`test_set` is reporting-only and orthogonal to the mode.** Only `dataset` and `valset` affect\noptimization (search and selection). If you pass `test_set`, the seed and the final chosen candidate\nare scored on it *after* optimization, outside the budget, and reported in `result.metadata`. The\ntest set never enters the eval server's HTTP surface, so agentic backends cannot see it. Use it when\nyou want an unbiased number to report; skip it otherwise.\n\n## `OptimizeAnythingConfig` (top-level fields)\n| field | default | meaning |\n|---|---|---|\n| `engine` | `\"gepa\"` | `\"gepa\"` / `\"autoresearch\"` / `\"meta_harness\"` (or `\"best_of_n\"`, a baseline), or a constructed `Engine` instance (custom engines can be added via `gepa.oa.registry.register_engine`). |\n| `name` | `None` | run id (logging + default output dir). Auto-generated (`<engine>-<uuid>-<timestamp>`) if `None`. |\n| `max_evals` | **`100`** | server-side cap on eval calls. `None` = unlimited. Size it — the default is rarely right (see SKILL.md). |\n| `max_token_cost` | `None` | USD cap on the engine's **own** optimizer-LLM spend (reflection/agent). Enforced by the engine (gepa: `max_reflection_cost` stopper; agent engines: `--max-budget-usd`), not the eval server. |\n| `max_concurrency` | `8` | eval-server thread-pool size. |\n| `output_dir` | `None` | where the eval server writes per-eval JSON, `progress_log.jsonl`, `summary.json`. `None` → `outputs/optimize_anything/<task>/<engine>/<timestamp>/`. |\n| `run_dir` | `None` | engine workspace (gepa run dir / agent work dir; with `engine.write_agent_state=True` the gepa backend writes an agent-readable `iterations/` + `pareto/` tree here). Distinct from `output_dir`. `None` → subprocess engines use a tempdir; set it to persist artifacts. |\n| `stop_at_score` | `None` | early-stop score threshold; each engine interprets it. |\n| `sandbox` | **`True`** | OS-jail the subprocess engines' `claude` sessions: bwrap on Linux (needs the `bubblewrap` package — the run aborts at launch if `bwrap` is missing), Claude Code's Seatbelt sandbox on macOS. Also forces the work dir to a tempdir even when `run_dir` is set (artifacts are mirrored back). `False` prints a loud warning and runs the agent unsandboxed (`bypassPermissions`, full host access). |\n| `engine_config` | `{}` | dict of **engine-specific** options — see the per-backend sections below. Parsed into a typed per-engine config dataclass; **an unknown key raises `TypeError` immediately** (fail fast, not warn-and-drop). |\n\n**If both `max_evals` and `max_token_cost` are `None`, the run is unbounded** (only a\n`warnings.warn`). Note the former fields `tracker`, `effort`, and `max_thinking_tokens` no longer\nexist here: tracking is configured via the gepa backend's `engine_config[\"tracking\"]`\n(see `tracking.md`), and `effort` / `max_thinking_tokens` are per-agent-engine `engine_config` keys.\n\n## The backends\n\nAll are selected with `engine=` and run the **same** task/evaluator — switch optimizer by\nchanging one argument. Each backend parses `engine_config` into its own typed dataclass, so swapping\n`engine=` also means swapping the `engine_config` block (leftover keys → `TypeError`).\n\n| engine | how it proposes candidates | runs | needs |\n|---|---|---|---|\n| `gepa` | an LLM reflects on evaluator feedback and mutates the candidate; keeps a Pareto frontier | in-process | reflection-LM creds (default `openai/gpt-5.1`) or a custom LM |\n| `autoresearch` | one Claude Code subprocess iterates in a work dir (`program.md`, `candidate.txt`, `eval.sh` → HTTP eval server) | subprocess | `claude` on PATH + headless auth, `jq` |\n| `meta_harness` | a Claude subprocess reads frontier/history and writes `pending_eval.json` candidates; the engine benchmarks each | subprocess | `claude` on PATH + headless auth |\n| `best_of_n` *(baseline)* | independent single-shot samples from one LLM; keep the best — no feedback, no history | in-process | LiteLLM creds for `model` (default `claude-sonnet-4-6`) |\n\n`scripts/preflight.py` checks a backend's prerequisites before a long run.\n\n### `gepa` — `engine_config` is a `GEPAConfig`-shaped dict (1-to-1 passthrough)\nThe dict is passed to `gepa.gepa_launcher.GEPAConfig(**engine_config)`; valid top-level keys are\nexactly its fields — `engine`, `reflection`, `tracking`, `merge`, `refiner`, `callbacks`,\n`stop_callbacks` — and nested dicts are coerced to the matching dataclasses. The OA layer only\noverlays the eval budget (`max_evals` → `engine.max_metric_calls`), `run_dir`, `stop_at_score`, and\nthe `max_token_cost` → `engine.max_reflection_cost` cap.\n\n```python\nengine_config={\n    \"reflection\": {                     # -> ReflectionConfig\n        \"reflection_lm\": \"anthropic/claude-sonnet-4-6\",  # default \"openai/gpt-5.1\"; see \"Proposer LM\"\n        \"reflection_lm_kwargs\": {\"reasoning_effort\": \"high\"},  # litellm kwargs (temperature, thinking, …)\n        \"reflection_minibatch_size\": 5,  # default: 1 single-task, 3 otherwise\n        # \"custom_candidate_proposer\": ClaudeCodeAgentProposer(...),  # replace the reflection LM with\n        #                                # a Claude Code proposer (from gepa.oa.proposers)\n        # \"reflection_strategy\": ...,    # advanced: a ReflectionLM impl owning how reflection is called\n    },\n    \"engine\": {                          # -> EngineConfig (all optional, sensible defaults)\n        \"max_workers\": 32,               # parallel eval workers (default: cpu_count or 32)\n        \"seed\": 0,                       # reproducibility\n        \"frontier_type\": \"hybrid\",       # \"instance\" | \"objective\" | \"hybrid\" (default) | \"cartesian\"\n        \"candidate_selection_strategy\": \"pareto\",      # | \"current_best\" | \"epsilon_greedy\" | \"top_k_pareto\"\n        \"acceptance_criterion\": \"strict_improvement\",  # | \"improvement_or_equal\"\n        \"cache_evaluation\": False,       # opt-in: cache (candidate, split, example) evals.\n                                         # Distinct valset is isolated from trainset ids.\n                                         # valset=None still shares minibatch rollouts.\n        \"capture_stdio\": False,          # opt-in: route evaluator print() output into feedback\n        \"raise_on_exception\": True,      # False → evaluator exceptions become score 0 + info[\"error\"]\n        # \"write_agent_state\": True,     # agent-readable iterations/ + pareto/ tree under run_dir\n    },\n    \"tracking\": {\"use_wandb\": True},     # -> TrackingConfig (see references/tracking.md)\n    \"merge\":  {...},                     # -> MergeConfig (cross-candidate merging) or omit\n    \"refiner\": {...},                    # -> RefinerConfig (auto per-eval refinement) or omit\n    # \"callbacks\": [GEPACallback, ...],  # observation hooks (on_iteration_end, on_candidate_accepted, …)\n    # \"stop_callbacks\": [...],           # custom StopperProtocol stop conditions\n}\n```\nThe nested dataclasses live in `gepa.gepa_launcher` (`EngineConfig`, `ReflectionConfig`,\n`TrackingConfig`, `MergeConfig`, `RefinerConfig`); their knobs are documented in the GEPA guides —\ne.g. **candidate-selection**, **acceptance-criterion**, **batch-sampling**, **callbacks**,\n**cost-tracking**, **experiment-tracking** — see <https://gepa-ai.github.io/gepa/guides/>.\n\n### `autoresearch` — `engine_config` → `AutoResearchConfig`\n| key | default | meaning |\n|---|---|---|\n| `model` | `\"claude-sonnet-4-6\"` | Claude model id (pass `\"sonnet\"`/`\"opus\"` to track the current default). |\n| `ralph` | `True` | keep resuming one Claude session (`claude --resume`) while budget remains. |\n| `max_no_eval_seconds` | `None` | kill the subprocess after this long with no eval call. |\n| `handoffs` | `None` | prior-stage artifacts for sequential compositions (materialized under `handoff/`). |\n| `effort` | `None` | `claude --effort` value. |\n| `max_thinking_tokens` | `None` | fixed thinking-token budget (`MAX_THINKING_TOKENS`). |\n\nThe engine lays out a work dir (`program.md`, `candidate.txt`, `best_candidate.txt`, `eval.sh`) and\nlaunches `claude --print`; `eval.sh` POSTs candidates to the eval server, which enforces the budget\nserver-side (HTTP 429 on exhaustion) and caps LLM spend via `--max-budget-usd` (from\n`max_token_cost`). Train and val are presented to the agent as one combined pool; the test set is\nunreachable over HTTP.\n\n### `meta_harness` — `engine_config` → `MetaHarnessConfig`\n| key | default | meaning |\n|---|---|---|\n| `model` | `\"claude-sonnet-4-6\"` | proposer model id. |\n| `max_iterations` | `None` | hard cap on proposer sessions; `None` = until budget. |\n| `max_candidates_per_iter` | `3` | upper bound on candidates proposed per iteration. |\n| `effort` | `None` | `claude --effort` value. |\n| `max_thinking_tokens` | `None` | fixed thinking-token budget. |\n\nEach iteration the proposer subprocess reads the frontier + history state files, writes\n`pending_eval.json` with 1+ candidates, and the engine benchmarks each through the eval server.\n\n### `best_of_n` (baseline) — `engine_config` → `BestOfNConfig`\nDeliberately naive: each sample is one independent LLM call — no feedback, no history, no\nselection pressure beyond keep-the-best. Use it as a **comparison floor** when evaluating an\noptimizer, not as the optimizer. Stops on budget exhaustion, `stop_at_score`, or `max_n`.\n\n| key | default | meaning |\n|---|---|---|\n| `model` | `\"claude-sonnet-4-6\"` | LiteLLM model id used to sample candidates. |\n| `temperature` | `1.0` | sampling on by default so N calls don't collapse to one response. |\n| `max_n` | `None` | optional hard cap on samples; `None` = run until budget out. |\n| `lm_kwargs` | `{}` | extra kwargs forwarded to `gepa.lm.LM`. |\n| `effort` | `None` | threaded into the LM as `reasoning_effort`. |\n| `max_thinking_tokens` | `None` | fixed thinking-token budget for the LM. |\n\n## Proposer LM (not limited to LiteLLM)\n`reflection.reflection_lm` (gepa) accepts either:\n- a **model-id string** resolved through LiteLLM (`\"openai/gpt-5.1\"` — the default —\n  `\"anthropic/claude-sonnet-4-6\"`, a Bedrock inference-profile ARN, …); set the provider's\n  credentials, and pass litellm kwargs via `reflection.reflection_lm_kwargs`; or\n- **any object/callable implementing GEPA's LM protocol** — `__call__(prompt) -> str` (a\n  `str`-or-messages prompt in, completion text out). This is how you plug a **self-hosted or custom\n  inference engine** (vLLM, a local server, your own client) instead of LiteLLM.\n  `reflection_lm_kwargs` is ignored for callables.\n\n## Composition / pipeline helpers (ensembles)\n`gepa.optimize_anything` also exports helpers that compose several backends over the same\ntask/evaluator. Each takes the same flat task fields as `optimize_anything` (`seed_candidate`\npositional; `evaluator=`, `dataset=`, `valset=`, `objective=`, `background=`, `test_set=`, `name=`\nkeywords) plus `configs=` — a list of `OptimizeAnythingConfig`, one per stage/branch, each carrying\nits own budget. Budgets are pre-partitioned per config; an early-finishing stage's leftover is not\nredistributed.\n\n- `optimize_sequential` — **pipeline**: run configs in order; each stage's best becomes the next\n  stage's seed. Monotonic — the running best is carried forward, so a regressing stage doesn't\n  poison the chain.\n- `optimize_parallel` — run all configs concurrently; returns the list of `Result`s (config order).\n- `optimize_best_of` — parallel, then keep the result with the highest `best_score` (each engine's\n  own number — no re-scoring).\n- `optimize_vote` — parallel, then re-score each branch's best candidate once via `evaluator`\n  (outside any budget) and return the highest re-score. Use when engines score with different\n  quirks and you want a fair cross-engine comparison.\n- `optimize_adaptive_sequential` — **plateau scheduler**: give the active engine bounded eval\n  slices (`plateau_evals`) and rotate to the next config after `patience` non-improving slices,\n  always feeding the running best forward. The exception to per-config budgets: one shared eval\n  pool set by its `max_evals` keyword (per-config `max_evals` is ignored; per-config\n  `max_token_cost` still applies). Extra knobs: `min_evals_per_stage`, `improvement_epsilon`,\n  `cycle`, `max_switches`.\n\n```python\nfrom gepa.optimize_anything import optimize_sequential, OptimizeAnythingConfig\n\nresult = optimize_sequential(\n    SEED,\n    evaluator=evaluate,\n    dataset=trainset, valset=valset,\n    objective=\"...\",\n    configs=[\n        OptimizeAnythingConfig(engine=\"autoresearch\", max_evals=100, max_token_cost=5.0),\n        OptimizeAnythingConfig(engine=\"gepa\", max_evals=200),  # refines autoresearch's best\n    ],\n)\n```\n\n`*_with_server` variants (`optimize_sequential_with_server`, `optimize_parallel_with_server`, …)\ntake caller-owned `EvalServer`s — one per config (one shared server for\n`optimize_adaptive_sequential_with_server`) — for embedding in outer frameworks that route every\neval through their own server. Related: the autoresearch backend's `handoffs` key materializes\nprior-stage artifacts into the agent's work dir for hand-rolled sequential compositions.\n\n## `GEPAResult`\n\n`optimize_anything` always returns a `GEPAResult` (the same type as the legacy\n`gepa.gepa_launcher` path). The mutable engine-internal `Result` is still what\nengines and ensemble helpers thread; it is not the public return.\n\n```python\nresult.best_candidate   # str (or dict[str, str] for multi-component seeds)\nresult.best_score       # float (on valset/selection set)\nresult.total_evals      # int — eval-server call count (budget.used)\nresult.eval_log         # list[dict]\nresult.metadata         # dict (verified keys):\n#   \"test_score\"             avg over test_set         } only present\n#   \"test_scores\"            per-example dict          } if you passed\n#   \"baseline_test_score\"    seed avg over test_set    } a test_set\n#   \"baseline_test_scores\"   seed per-example dict     }\n#   \"budget\", \"total_cost\", \"adapter_cost\", \"wall_time\", \"engine\", \"output_dir\", \"progress_log\"\nresult.candidates            # full candidate pool (length 1 on non-gepa engines)\nresult.best_idx              # index of the highest val score\nresult.val_aggregate_scores  # per-candidate average validation score\nresult.to_dict()             # JSON-safe pool snapshot (does not include eval_log / metadata)\n```\n\nThe candidate pool lives on the result itself. `metadata` does **not** contain a nested\n`gepa_result`. `total_evals` is the eval-server count; `total_metric_calls` is GEPA core's\nown counter and may differ (e.g. after `BudgetExhausted`). `to_dict()` / `from_dict()` omit\n`eval_log`, `metadata`, and `eval_server_calls` so the frozen JSON pool contract stays\nbyte-stable.\n\nThe gepa engine also writes `run_dir/` artifacts, and the eval server writes\n`output_dir/summary.json`. (If the best candidate equals the seed, the test scores are reported from the\nseed's single scoring pass rather than re-scored.)\n\n**Migration:** callers that used `isinstance(result, Result)` or\n`result.metadata[\"gepa_result\"]` after `optimize_anything(...)` should switch to\n`GEPAResult` and `result.candidates`. Ensemble helpers (`optimize_sequential`, …)\nstill return the mutable `Result`.\n","isInternal":false,"tokens":4634,"sizeBytes":18252},{"name":"gotchas.md","path":".claude/skills/gepa-optimize-anything/references/gotchas.md","rawUrl":"https://raw.githubusercontent.com/gepa-ai/gepa/HEAD/.claude/skills/gepa-optimize-anything/references/gotchas.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Gotchas and pitfalls\n\nRead before any real run.\n\n## 1. Reward hacking — the optimizer exploits a weak proxy ruthlessly\nEvery backend maximizes exactly the score you compute; if it is gameable, it *will* be gamed. Real\nexample (KernelBench, gepa backend): a correctness-only score drove the optimizer to a prompt that\ntells the model to **wrap the reference implementation** — always \"correct\", zero real work. Result: pass@1 ~0.87 but the actual goal (fast kernels) collapsed to ~base.\nMitigation: a **gated objective** — score 0 unless valid+correct, then increasing in the thing you\ncare about (see `writing_evaluators.md`). Sanity-check the *winning* candidate against your true goal,\nnot just the score.\n\n## 2. Selection bias / winner's curse (generalization mode)\nIn generalization mode the chosen candidate is the **max** over many candidates scored on `valset`.\nWith noisy (especially N=1) scores over few examples, that max is optimistically inflated — it can sit\nwell above true generalization (e.g. a selected best of 0.87 with runner-up 0.60 and median 0.57).\nThis bias lives in **selection on `valset`**, so reduce it there:\n- use **enough `valset` examples**, and **N>1 samples per eval** for stochastic models, to shrink the\n  variance that inflates the max;\n- treat `best_score` (the selection-set number) as optimistic.\n\n`test_set` does **not** reduce this bias — it is reporting-only. But scoring the final candidate on a\nheld-out `test_set` gives you an *honest* number to report — `metadata[\"test_score\"]` (average; also\n`[\"test_scores\"]` per-example) vs `[\"baseline_test_score\"]` for the seed — separate from the inflated\nselection score. (Note the exact keys: `test_score`/`test_scores`, **not** `test_score(s)`.)\n\n## 3. Stochastic N=1 default\nOne eval call per (candidate, example). For a temperature>0 model that means single-sample scores,\nwhich feeds the selection bias in #2. Average N samples inside `evaluate` (`writing_evaluators.md`).\n\n## 4. The default budget is probably wrong for your run\n`max_evals` defaults to **100** — bounded, but rarely sized for your selection set (see SKILL.md's\nsizing rule: ≳ 15–20 × the selection set). Set it explicitly. Only if you set **both**\n`max_evals=None` and `max_token_cost=None` is the run unbounded, and even that is just a\n`warnings.warn`, not an error.\n\n## 5. `engine_config` is validated strictly — unknown keys raise `TypeError`\nEvery backend parses `engine_config` into a typed dataclass (`GEPAConfig` for gepa,\n`BestOfNConfig` / `AutoResearchConfig` / `MetaHarnessConfig` for the others), so a typo'd or\nwrong-backend key fails **immediately at construction** with a `TypeError` — it is *not* silently\ndropped. The practical consequence: swapping `engine=` requires swapping the whole `engine_config`\nblock, and old-API keys (`claude_code_agent`, top-level `reflection_lm_kwargs`, `objective`,\n`background` inside `engine_config`) now crash. See `api.md` for each backend's valid keys.\n\n## 6. Agentic backends have launch-time prerequisites\n`autoresearch` / `meta_harness` `subprocess.Popen([\"claude\", ...])`. A missing `claude` CLI — or, on\nLinux, a missing `bwrap` (bubblewrap) while the default `sandbox=True` is in effect — aborts the run\nat launch with a boxed message and install instructions (`npm install -g @anthropic-ai/claude-code`;\n`sudo apt/dnf install bubblewrap`). An *unauthenticated* CLI or a missing `jq` (used by\nautoresearch's generated `eval.sh`) still surfaces only mid-run, and `sandbox=False` runs the agent\nunconfined (loud warning) — so run `scripts/preflight.py` first either way.\n\n## 7. Give runs a real stop condition (`stop_at_score` / `max_token_cost`)\n`max_evals` caps eval calls, but two situations still burn money or time past the point of useful\nwork:\n- **The metric has a ceiling** and a candidate reaches it — without `stop_at_score` the run keeps\n  proposing at the optimum until the budget is gone. For a bounded metric (pass rate, accuracy),\n  *always* pass `stop_at_score`.\n- **You opted into evaluation caching** (`engine_config={\"engine\": {\"cache_evaluation\": True}}` —\n  it is **off by default**). Then `max_evals` counts only cache *misses*: a converged search keeps\n  emitting cache-hitting candidates without consuming eval budget and can spin until your process\n  times out, still spending proposer-LLM tokens. With caching on, `stop_at_score` and/or\n  `max_token_cost` (plus a wall-clock `timeout` on the launched process) are mandatory.\n  A distinct `valset` is a separate cache namespace from the trainset (list-position ids would\n  otherwise collide). `valset=None` still shares cached rollouts with minibatches. Resuming a\n  `run_dir` whose cache predates that namespacing drops the old entries.\nAgentic backends also spend LLM tokens between evals — cap them with `max_token_cost`\n(enforced as `--max-budget-usd`).\n\n## 8. Pick the right mode\nThe mode is implicit in which sets you pass (`api.md`): no `dataset`/`valset` → single-task;\n`dataset` only → multi-task; `dataset`+`valset` → generalization. A common mistake is wanting a\ncandidate that generalizes but passing only `dataset` (multi-task) — then there's no held-out\nselection and the candidate can overfit the training examples. Pass a `valset` for generalization.\n(Held-out `valset` selection is implemented by the gepa backend; the other backends fold `valset`\ninto the scoring pool — see `api.md`.)\n\n## 9. Saturated signal — the gepa backend returns the seed unchanged (no failure to learn from)\nThis one is specific to the gepa backend, whose optimizer improves by reflecting on examples the\ncurrent candidate gets **wrong**. If the seed already scores near-perfect on the train minibatch the\nproposer reflects over, every mutation looks \"not better\" and is rejected by the acceptance gate —\nso the run spends its full budget, accepts **zero** proposals, and hands back the seed. Symptom:\nmany proposals made, ~0 accepted, `best_candidate` == seed. This is \"baseline saturation,\" not a\nbug: there's no gradient for reflection to climb.\nMitigations: make sure `dataset` contains examples the seed actually **fails** (harder/larger train\nset, or a task-LM weak enough to generate failures); if the seed is already at ceiling, reflective\nmutation has little to do — the `best_of_n` baseline (independent full rewrites, no acceptance\ngate) or just keeping the seed may be the better call.\nAlways check **proposals accepted**, not just proposals made.\n\n## 10. Uncaught evaluator exceptions abort the run\nOn the gepa backend, `EngineConfig.raise_on_exception` defaults to `True`: an exception escaping\nyour evaluator stops the whole optimization. Either catch failures yourself and return them as low scores with rich\n`info[\"error_*\"]` feedback (preferred — the proposer learns from them), or set\n`engine_config={\"engine\": {\"raise_on_exception\": False}}` to have them auto-converted to score\n`0.0` with `info[\"error\"]`.\n\n## Quick pre-flight checklist\n- [ ] mode matches intent (single-task / multi-task / generalization) via `dataset`/`valset`\n- [ ] `dataset` contains examples the seed gets WRONG — reflection has no failure signal to learn\n      from otherwise, and the gepa backend hands back the seed unchanged (#9)\n- [ ] `score` matches the real goal (gated, not gameable)\n- [ ] `info` carries actionable feedback (errors/diffs/partial credit)\n- [ ] N>1 if the model is stochastic\n- [ ] proposer LM valid + 1-call tested; creds present (default `openai/gpt-5.1` needs\n      `OPENAI_API_KEY` — or set your own via `reflection.reflection_lm`)\n- [ ] `max_evals` set explicitly and sized for many proposals (≳ 15–20 × the size of the selection\n      set: `len(valset)` in generalization, `len(dataset)` in multi-task, or just ≳ 15–20 in\n      single-task) — see SKILL.md\n- [ ] `stop_at_score` set when the metric has a ceiling; `max_token_cost` for agentic backends\n- [ ] `engine_config` keys match the chosen backend (typos raise `TypeError`, #5)\n- [ ] `run_dir` + `output_dir` set (so artifacts persist)\n- [ ] `test_set` passed if you need an unbiased number to report\n- [ ] for agentic backends: `claude` CLI on PATH + authed, `jq` installed, and on Linux `bwrap`\n      (bubblewrap) for the default `sandbox=True` (`scripts/preflight.py`)\n","isInternal":false,"tokens":2086,"sizeBytes":8306},{"name":"tracking.md","path":".claude/skills/gepa-optimize-anything/references/tracking.md","rawUrl":"https://raw.githubusercontent.com/gepa-ai/gepa/HEAD/.claude/skills/gepa-optimize-anything/references/tracking.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Experiment tracking\n\nThe gepa backend logs to **Weights & Biases** and **MLflow** via `TrackingConfig` — configure it\nunder `engine_config[\"tracking\"]` (a valid `GEPAConfig` field, coerced to `TrackingConfig`). The\nother backends have no built-in tracker; for them, rely on the eval server's `output_dir` records\n(always written, whatever the backend):\n\n```python\nresult = optimize_anything(\n    seed_candidate=SEED, evaluator=evaluate, dataset=trainset, valset=valset,\n    config=OptimizeAnythingConfig(\n        engine=\"gepa\", max_evals=300,\n        engine_config={\n            \"reflection\": {\"reflection_lm\": \"anthropic/claude-sonnet-4-6\"},\n            \"tracking\": {                       # -> TrackingConfig\n                \"use_wandb\": True,\n                \"wandb_init_kwargs\": {\"project\": \"my-gepa-run\", \"name\": \"experiment-1\"},\n            },\n        },\n    ),\n)\n```\n\n## `TrackingConfig` fields\n- `use_wandb`, `wandb_api_key`, `wandb_init_kwargs`, `wandb_attach_existing`, `wandb_step_metric`\n- `use_mlflow`, `mlflow_tracking_uri`, `mlflow_experiment_name`, `mlflow_attach_existing`\n- `key_prefix` — prepended to every logged key/name (e.g. `\"gepa/\"` → `gepa/val_score`)\n- `logger` — a custom `LoggerProtocol` for plain-text log lines\n\n## What gets logged\n- **scalars** — `val_program_average`, `best_score_on_valset`, `total_metric_calls`, …\n- **tables** — `candidates`, `proposals`, `valset_scores`, `valset_pareto_front`\n- **run summary** — seed and best-found component text\n- **interactive candidate-tree HTML**\n\n## Attaching to an existing run\nIf you've already called `wandb.init()` (or started an MLflow run) yourself — e.g. a parent script\nthat also logs other things — set `wandb_attach_existing=True` (or `mlflow_attach_existing=True`).\nThe tracker then logs **into your active run** without calling `init()`/`finish()`, so it won't\ndisrupt the run lifecycle. Combine with `key_prefix=\"gepa/\"` to keep its metrics in their own\nnamespace.\n\nWhen the host loop manages its own wandb step counter, also set `wandb_step_metric` (e.g.\n`\"gepa/iteration\"`): the tracker then declares its own x-axis via `wandb.define_metric` instead of\npassing `step=`, so its `1, 2, 3, …` steps don't collide with the host's counter (a collision makes\nwandb silently drop the data).\n\n## Custom hooks\nFor programmatic observation beyond metric logging, pass `engine_config={\"callbacks\": [...]}` —\n`GEPACallback` objects receive events like `on_optimization_start`, `on_iteration_end`,\n`on_candidate_accepted`, `on_proposal_end`. The eval server also always writes a flat record\n(per-eval JSON + `progress_log.jsonl` + `summary.json`) under `output_dir`, whatever tracking is\nconfigured.\n\nSee the official guide for the authoritative list of fields and behaviors:\n<https://gepa-ai.github.io/gepa/guides/experiment-tracking/>.\n","isInternal":false,"tokens":725,"sizeBytes":2844},{"name":"writing_evaluators.md","path":".claude/skills/gepa-optimize-anything/references/writing_evaluators.md","rawUrl":"https://raw.githubusercontent.com/gepa-ai/gepa/HEAD/.claude/skills/gepa-optimize-anything/references/writing_evaluators.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Writing evaluators\n\nThe evaluator is where almost all of your effort and almost all of the quality come from. Whatever\nbackend you choose can only be as good as the score you compute and the feedback you return.\n\n## The contract\n```python\ndef evaluate(candidate: str, example) -> tuple[float, dict]:\n    return score, info\n```\n- `score: float` — **higher is better**. This is what the optimizer maximizes and selects on.\n- `info: dict` — **free-form feedback shown to the proposer**: the gepa backend's reflection LM\n  (as \"Actionable Side Information\", ASI) or the agentic backends' agent (the `best_of_n` baseline\n  ignores it). This is the single biggest lever on mutation quality.\n- For single-task runs the signature is `evaluate(candidate)`; with `dataset`/`valset`/`test_set`\n  it's `evaluate(candidate, example)`. (Returning a bare `float` instead of a tuple also works —\n  the wrapper normalizes it — but then the proposer gets no feedback; always return the tuple.)\n\n### Batched form (`batch_evaluator=`)\nWhen evals batch better than they stream — a provider batch API, one job submission for a whole\nstage, fan-out over your own infrastructure — write the batched form instead of (or alongside) the\nper-pair one:\n```python\ndef batch_evaluate(pairs: list[tuple[str, Any]]) -> list:  # pairs = [(candidate, example), ...]\n    return [score_or_score_info_tuple for candidate, example in pairs]  # one result per pair, same order\n```\nEach evaluation stage (minibatch, valset, held-out test pass) arrives as ONE call with all its\npairs. Everything above about feedback-rich `info` applies per pair. Put diagnostics in each\nreturned `info` — the per-call channels below (`oa.log()`, `capture_stdio`) don't apply to the\ngrouped call.\n\n## Make `info` feedback-rich (not just a number)\nThe proposer (reflection LM or agent) writes the next candidate by reading `info`. Give it specifics:\n```python\nreturn score, {\n    \"score\": score,\n    \"output\": output,                 # what the candidate produced\n    \"expected\": example.get(\"gold\"),  # what was wanted (if available)\n    \"error_type\": err_type,           # compile error / wrong answer / format violation / timeout\n    \"error_detail\": traceback_or_diff,# the actual message — concrete > vague\n    \"passed_checks\": [...],           # partial credit signals\n    \"failed_checks\": [...],\n}\n```\nRule of thumb: if a smart human reading only `info` could tell you *how to fix the candidate*, the\nproposer LLM can too. If `info` is just `{\"score\": 0.0}`, the search is blind.\n\n### Two built-in channels for diagnostics\n- **`oa.log()`** — `import gepa.gepa_launcher as oa; oa.log(\"landing distance:\", d)` inside your\n  evaluator. Same calling convention as `print()`; output is captured per-eval (thread-safe) and\n  auto-included in the feedback under `info[\"log\"]`. For child threads, propagate the context via\n  `oa.get_log_context()` / `oa.set_log_context()`.\n- **`capture_stdio`** — set `engine_config={\"engine\": {\"capture_stdio\": True}}` (gepa backend) and\n  any `print()`/stdout/stderr during evaluation lands in the feedback under `\"stdout\"`/`\"stderr\"` —\n  useful for wrapping existing scripts with no code changes. (Doesn't catch C-extension or\n  subprocess output that bypasses Python's `sys.stdout`; route that through `oa.log()`.)\n\n## LLM-as-judge scoring (subjective tasks)\nThe score doesn't have to come from an objective metric. For open-ended tasks (writing quality,\nhelpfulness, tone, adherence to a rubric) the evaluator can call an **LLM judge** and use its rating\nas the score — and, importantly, return the judge's **written critique** as feedback:\n```python\ndef evaluate(candidate, example):\n    output = run_my_model(candidate, example)\n    verdict = judge_lm(JUDGE_RUBRIC.format(task=example, answer=output))  # your judge call\n    score = verdict[\"rating\"] / 10.0                  # normalize to a float, higher = better\n    return score, {\n        \"score\": score,\n        \"output\": output,\n        \"critique\": verdict[\"critique\"],              # the judge's reasoning → drives better proposals\n        \"rubric_breakdown\": verdict.get(\"by_criterion\"),\n    }\n```\nTips: pin the judge (fixed model + temperature, ideally a stronger model than the one being\noptimized), give it a concrete rubric, and average a few judge calls if its ratings are noisy\n(this is the N>1 idea below). The critique is often more valuable to the proposer than the number.\n\n## Stochastic models: beat the silent N=1 default\nThe eval server calls your function **once** per (candidate, example) — there is no\n`samples_per_eval` knob. For a temperature>0 model that makes every score a single-sample estimate,\nand candidate **selection** then runs on noisy numbers (this is how you get winner's-curse inflation).\nFix it inside `evaluate` by averaging N samples:\n```python\ndef evaluate(candidate, example, N=4):\n    outs = [run_my_model(candidate, example) for _ in range(N)]\n    scores = [grade(o, example) for o in outs]\n    score = sum(scores) / len(scores)           # mean correctness ~ pass@1 estimate\n    return score, {\"score\": score, \"n\": N,\n                   \"samples\": [{\"out\": o, \"s\": s} for o, s in zip(outs, scores)]}\n```\nTrade-off: N× more eval calls (budget). Pick N to balance variance vs `max_evals`.\n\n## Multi-objective optimization (e.g. correctness AND speed)\nThe gepa backend keeps an objective-level Pareto front (the other backends ignore\n`info[\"scores\"]`). To use it, **return per-objective metrics under `info[\"scores\"]`** — the adapter\nforwards them as `objective_scores`:\n```python\nreturn score, {\n    \"score\": score,                        # the scalar the optimizer selects on\n    \"scores\": {\"correct\": correct_rate,    # per-objective metrics for the Pareto frontier\n               \"speedup\": speedup},        # all \"higher is better\"\n    ...\n}\n```\n`engine_config={\"engine\": {\"frontier_type\": \"hybrid\"}}` is the default (instance- and\nobjective-level fronts combined); `\"objective\"`, `\"instance\"`, and `\"cartesian\"` are the\nalternatives. The scalar `score` still drives final selection; the per-objective scores shape the\nfrontier that candidates are drawn from.\n\n## Choose the score to match the real goal (avoid reward hacking)\nThe optimizer maximizes exactly what you write. A correctness-only score is gameable — e.g. in code/kernel\ntasks the optimizer learns to emit a trivial wrapper that is \"correct\" but does nothing useful.\nPrefer a **gated** objective:\n```python\ndef score_fn(result):\n    if not (result[\"compiled\"] and result[\"correct\"]):\n        return 0.0\n    return f(result[\"speedup\"])   # e.g. min(speedup / target, 1.0), monotonically increasing\n```\nThen the only way to raise the score is to be correct **and** better on what you care about. See\n`gotchas.md` for the full reward-hacking story.\n\n## Determinism & robustness\n- Make `evaluate` side-effect-free and resumable; it may run concurrently (`max_concurrency`,\n  `engine.max_workers`) and be retried.\n- Set a seed in `engine_config={\"engine\": {\"seed\": 0}}` for reproducible search order.\n- Log your own per-eval record (id, score, sub-metrics, candidate hash) — you'll want it for\n  analysis; the eval server's `output_dir` per-eval JSON covers the basics, and `oa.log()` covers\n  in-feedback diagnostics.\n- Catch and *return* failures as low scores with `info[\"error_*\"]`, rather than raising —\n  `EngineConfig.raise_on_exception` defaults to `True`, so an uncaught exception aborts the run.\n  (Setting it to `False` converts exceptions to score `0.0` with `info[\"error\"]` instead.)\n","isInternal":false,"tokens":1854,"sizeBytes":7573},{"name":"SKILL.md","path":".claude/skills/gepa-optimize-anything/SKILL.md","rawUrl":"https://raw.githubusercontent.com/gepa-ai/gepa/HEAD/.claude/skills/gepa-optimize-anything/SKILL.md","title":"gepa-optimize-anything","category":"anthropic-skill","format":"markdown","content":"---\nname: gepa-optimize-anything\ndescription: >-\n  Automatically improve any text artifact that can be scored — prompts, programs/code, configs,\n  specs, regex/SQL/schemas, agent scaffolds, or encoded search solutions — with optimize_anything,\n  an LLM-driven optimization API: a search backend repeatedly proposes better versions from\n  execution feedback and an evaluator you provide assigns the score (an objective metric or an\n  LLM-as-judge for subjective tasks). optimize_anything is one interface across several optimizer\n  backends — GEPA (reflective evolutionary search, the default), AutoResearch, MetaHarness — plus a\n  best-of-N baseline: write the evaluator once and switch backend with a single argument.\n  Use whenever a task needs auto-optimizing, tuning, or searching over text with any quality signal\n  (accuracy, pass rate, latency, cost, judge rating); when building an evaluator + proposer loop;\n  when running GEPA; or when comparing optimizers. The candidate can be any string an evaluator can\n  grade, scored on one problem or generalized across many.\n---\n\n# `optimize_anything`\n\n**Naming, precisely.** `optimize_anything` is the tool: a general API for optimizing text\nartifacts. **GEPA** is one specific optimizer behind it — reflective evolutionary search, the\ndefault backend (`engine=\"gepa\"`) — and, for legacy reasons, also the name of the Python package\nthat ships all of this. In this skill, \"the gepa backend\" always means the optimizer; statements\nabout \"the optimizer\" or \"the backend\" apply to whichever engine you chose.\n\n`optimize_anything` does **black-box optimization**: you provide (1) a seed artifact, (2) an\n**evaluator** that scores any artifact and returns feedback, and (3) a backend, which repeatedly\nproposes improved artifacts and scores them through your evaluator. \"Black-box\" refers to the\n**evaluator**, not the artifact: the backend never sees how the score is computed — no gradients,\nno metric internals — only the scalar score and the feedback text you emit. The candidate itself\n*is* visible: the proposer reads and rewrites it, applying the LLM's understanding of your artifact.\nThe framework just imposes no structure on it — any string an evaluator can score works. The\nleverage is in your score and your feedback.\n\n**You write the task and evaluator once, then choose the search algorithm with one `engine`\nargument** — and the same code runs under any of them:\n- **`gepa`** — the GEPA optimizer: reflective evolutionary search, in-process (an LLM reflects on\n  feedback and mutates candidates; keeps a Pareto frontier). The default; strongest when feedback\n  is rich.\n- **`autoresearch`** — an agentic optimizer: one Claude Code subprocess iterates like a researcher in\n  a work dir, scoring candidates through an HTTP eval server.\n- **`meta_harness`** — an agentic proposer (Claude subprocess) that reads the frontier/history each\n  iteration and writes new candidates for the engine to benchmark.\n\n(There is also a `best_of_n` engine — sample N independent candidates, keep the best. It is\ndeliberately naive: use it as a **baseline** to compare an optimizer against, not as the optimizer.)\n\nThis makes it easy to start with one backend and benchmark others on the identical task/evaluator.\nThere are also **composition/pipeline helpers** that combine backends over the same task:\n`optimize_sequential` (a pipeline — each stage's best seeds the next), `optimize_parallel`,\n`optimize_best_of`, `optimize_vote` (re-score each branch's best for a fair pick), and an adaptive\nscheduler that rotates backends on score plateaus — see `references/api.md`.\n\n## What can be a candidate\nA candidate is **any string your evaluator can score**. \"Text in → a number out (higher is better),\nplus optional feedback\" is the entire contract, which covers a wide range of artifacts:\n- **prompts** — system/user prompts, instruction templates, rubrics, few-shot exemplars\n- **programs / code** — functions, whole files, CUDA kernels, scored by compiling + running + benchmarking\n- **configs / specs / schemas / regex / SQL** — any text whose effect you can measure\n- **agent scaffolds** — tool instructions, planner/critic prompts, orchestration text\n- **pure search artifacts** — a mathematical construction, a packing layout, a plan, encoded as text\n\nThe score can be an **objective metric** (accuracy, pass rate, runtime, cost) **or an LLM-as-judge**\nrating for subjective tasks (writing quality, helpfulness, style). At this API the candidate is a\nsingle string (`seed_candidate: str | None`; `None` = seedless — the engine bootstraps from\n`objective`/`background`). Multi-component dict candidates exist only in the lower-level\n`gepa.gepa_launcher.optimize_anything` API, not here.\n\n## Three optimization modes (choose by how you pass data)\nThe mode is determined by whether you provide `dataset` and `valset`:\n1. **Single-task** (`dataset=None, valset=None`) — solve one hard problem; the candidate *is* the\n   solution; the evaluator is called with no example. *E.g. one CUDA kernel, a circle-packing layout.*\n2. **Multi-task** (`dataset=<list>, valset=None`) — solve a batch of related problems with one shared\n   candidate, transferring insight across them; evaluator called per example. *E.g. a single prompt\n   that works across many tasks.*\n3. **Generalization** (`dataset=<list>, valset=<list>`) — build a candidate that transfers to\n   **unseen** problems; optimize on `dataset`, select on `valset`. *E.g. a prompt tuned to generalize.*\n   *Note:* GEPA is the algorithm designed around this mode, and `valset`-based held-out selection is\n   currently implemented only by the gepa backend — the other backends fold `valset` into the\n   training pool (they can still generalize; there's just no separate selection split).\n\n`test_set` is **separate from the modes and reporting-only**: the seed and the final candidate are\nscored on it after optimization for an unbiased number — it never enters the search, selection, or\nbudget (for the agentic backends it is sealed at the eval server's HTTP layer, so the agent cannot\neven see it). See `references/api.md` for details and when to use each mode.\n\n## Install\n```bash\npip install \"gepa[full]\"   # [full] pulls cloudpickle — needed to pickle closure evaluators for\n                           # parallel workers / opt-in evaluation caching; plain `pip install gepa`\n                           # can fail there when your evaluator closes over data.\n# Proposer LLM: the gepa backend's reflection LM defaults to \"openai/gpt-5.1\" (a LiteLLM id) — set\n# that provider's key (OPENAI_API_KEY), or pass your own id (e.g. \"anthropic/claude-sonnet-4-6\" with\n# ANTHROPIC_API_KEY, or a Bedrock ARN with AWS creds). You can also pass any callable implementing\n# GEPA's LM protocol — a self-hosted / custom inference engine — instead of a model-id string.\n# Agentic backends (autoresearch, meta_harness) additionally need the `claude` CLI on PATH (plus\n# `jq` for the generated eval.sh):\nnpm install -g @anthropic-ai/claude-code   # or: curl -fsSL https://claude.ai/install.sh | bash\n#   ...then run `claude` once to authenticate.\n# On Linux they also need bubblewrap: the default sandbox=True jails the claude subprocess with\n# bwrap (`sudo apt install bubblewrap` / `sudo dnf install bubblewrap`) and the run aborts at\n# launch if it's missing. sandbox=False opts out — the agent then runs unsandboxed (loud warning).\n```\n\n## Mental model (4 pieces)\n1. **Candidate** — the text you optimize (a `str`). The seed is the start; `None` = seedless.\n2. **`evaluate(candidate, example) -> (score, info)`** — `score` is a float (higher is better);\n   `info` is a free-form dict of **feedback the backend's proposer reads** to make better candidates\n   (the gepa backend's reflection LM, or the agentic backends' agent; the `best_of_n` baseline\n   ignores feedback). When evals batch better than they stream (e.g. a provider batch API), pass\n   `batch_evaluator=` instead — all pending `(candidate, example)` pairs in one call; see\n   `references/api.md`.\n3. **Engine (backend)** — `\"gepa\"` (default), `\"autoresearch\"`, `\"meta_harness\"`, the `\"best_of_n\"`\n   baseline, or a constructed `Engine` instance.\n4. **Budget** — `max_evals` (server-side eval-call cap, **default 100**) and/or `max_token_cost`\n   (USD cap on the backend's own proposer-LLM spend). Setting both to `None` makes the run unbounded\n   (only a warning). **Size `max_evals` for many proposal rounds, not one** (see below) — this is the\n   most common way agents misuse this API.\n\n## Sizing the valset and the budget (read this — the #1 mistake)\n`max_evals` is the *main* control over how long the optimizer runs. Leave it at the default (100)\nwith a large valset, or set it too low, and the run stops after a **single proposal**, then reports\na \"best candidate\" that looks fine but is barely optimized.\n\n- **valset** — on the gepa backend the best candidate is selected by its scores on the valset, so\n  make it a **representative subset of your data**. There is no fixed size: pick what represents the task (a\n  handful for small/expensive tasks, more when data is cheap and plentiful — anywhere from a few to\n  hundreds). Bigger valset = less noisy selection but more eval calls per candidate.\n- **budget** — every proposed candidate is scored on the **whole selection set**, so size\n  `max_evals` off that set, not off a single proposal. Which set that is depends on the mode:\n  ```\n  generalization (dataset + valset):  max_evals ≳ 15–20 × len(valset)    # scored & selected on valset\n  multi-task     (dataset only):      max_evals ≳ 15–20 × len(dataset)   # scored & selected on dataset\n  single-task    (no dataset/valset): max_evals ≳ 15–20                  # 1 eval per candidate, so this\n                                                                         #   IS the number of proposals\n  ```\n  The constant is the same everywhere — **let the backend propose AND evaluate ~15–20 candidates**\n  (more if you can afford it). Anything much less and the run tries only a couple of candidates —\n  i.e. it's barely optimizing. (This arithmetic is exact for the gepa backend — and the best_of_n\n  baseline — which score every candidate on the full selection set; the agentic backends decide\n  themselves how to spend eval calls, so treat it as a floor.)\n\nAfter the run, check how many proposals actually happened (on the gepa backend,\n`result.candidates` is the full pool; with `engine.write_agent_state=True` the\n`run_dir/iterations/` tree shows each one). **If it stopped after one proposal, the budget was too\nlow** — raise it and rerun.\n\n### Give every run a real stop condition\n`max_evals` bounds eval calls, but add explicit stops so runs end at the right moment:\n- **`stop_at_score`** — set it whenever your metric has a known ceiling (e.g. `1.0` for a pass rate /\n  accuracy). The backend stops the moment a candidate reaches it instead of burning the rest of the\n  budget at the optimum.\n- **`max_token_cost`** — a hard USD cap on the backend's own proposer/agent LLM spend. Especially\n  important for the agentic backends (`autoresearch`, `meta_harness`), whose Claude subprocesses\n  spend tokens between eval calls.\n- **a wall-clock `timeout`** on the process you launch (e.g. `timeout 1200 python run.py`) as a\n  backstop.\n- If you **opt in** to evaluation caching (`engine_config={\"engine\": {\"cache_evaluation\": True}}` on\n  the gepa backend — it is **off by default**), be aware `max_evals` then counts only cache *misses*:\n  a converged search can keep proposing cache-hitting candidates without consuming eval budget, so\n  `stop_at_score`/`max_token_cost` become mandatory, not optional. A distinct `valset` is cached\n  separately from the trainset; `valset=None` still reuses minibatch rollouts.\n\n## Minimal working example\nThe example optimizes a system prompt for concreteness, but the **shape is identical** for any\ncandidate — swap `SEED` for a code file / config / etc. and have `evaluate` compile/run/measure it.\n```python\nfrom gepa.optimize_anything import optimize_anything, OptimizeAnythingConfig\n\nSEED = \"You are an expert. Solve the task. Output only the final answer.\"\n\ndef evaluate(candidate: str, example) -> tuple[float, dict]:\n    output = run_my_model(system_prompt=candidate, user_prompt=example[\"prompt\"])  # your call\n    score = grade(output, example)                                                 # float, higher=better\n    return score, {                       # everything here is shown to the proposer LLM\n        \"score\": score,\n        \"output\": output,\n        \"error\": example.get(\"error\"),    # concrete, actionable feedback drives good proposals\n    }\n\nresult = optimize_anything(\n    seed_candidate=SEED,\n    evaluator=evaluate,\n    dataset=trainset,            # optimize on these (multi-task/generalization mode)\n    valset=valset,               # select the best candidate on these (generalization mode)\n    test_set=testset,            # OPTIONAL, reporting-only: seed + final candidate scored here at the end\n    objective=\"Produce a prompt that maximizes task accuracy.\",\n    background=\"Domain rules, constraints, output format the model must follow.\",\n    config=OptimizeAnythingConfig(\n        engine=\"gepa\",               # swap to \"autoresearch\" / \"meta_harness\" — same code\n                                     #   (\"best_of_n\" runs the same way, as a comparison baseline)\n        name=\"my_run\",\n        max_evals=300,               # ≳ 15-20 × len(valset): enough for ~15-20 proposals (see above)\n        stop_at_score=1.0,           # stop at the optimum (set when your metric has a known ceiling)\n        max_concurrency=16,\n        run_dir=\"runs/my_run\",       # engine workspace (gepa run dir / agent work dir)\n        output_dir=\"outputs/my_run\", # eval server: per-eval JSON, progress_log.jsonl, summary.json\n        engine_config={              # gepa backend: a GEPAConfig-shaped dict, validated strictly —\n            \"reflection\": {          #   an unknown key raises TypeError immediately (fail fast)\n                # a LiteLLM id (set the provider key) OR any callable implementing the LM protocol\n                \"reflection_lm\": \"anthropic/claude-sonnet-4-6\",\n                \"reflection_minibatch_size\": 5,\n            },\n            \"engine\": {\"max_workers\": 32, \"seed\": 0},  # seed = reproducibility\n        },\n    ),\n)\nprint(result.best_candidate, result.best_score)\n# held-out (only present if you passed test_set): \"test_score\" = average, \"test_scores\" = per-example\nprint(\"held-out:\", result.metadata.get(\"test_score\"),\n      \"seed held-out:\", result.metadata.get(\"baseline_test_score\"))\n```\n\n## Standard workflow\n1. **Pick the mode** (single-task / multi-task / generalization) by which of `dataset`/`valset` you pass.\n2. **Define the score deliberately.** The optimizer optimizes exactly what you measure — gate the\n   score on what you actually care about (see `references/gotchas.md`, reward hacking).\n3. **Write a feedback-rich `evaluate`.** The `info` dict is the proposer's signal — return errors,\n   diffs, partial credit, not just a number (`oa.log()` and `capture_stdio` can route diagnostics in\n   automatically). See `references/writing_evaluators.md`.\n4. **Pick a proposer LLM** — a LiteLLM id (set the provider key) or a custom LM-protocol callable.\n   Validate it with a 1-call test before a long run.\n5. **Set a budget** (`max_evals` sized per above, and/or `max_token_cost`) plus `stop_at_score` when\n   the metric has a ceiling.\n6. **Run `python scripts/preflight.py`** to fail fast on missing creds / CLI before a long run.\n7. **Launch**, watch the first 1-2 evals (the eval→model→score chain), then let it run.\n8. **Read `result.best_candidate` and `run_dir/`** (and `result.metadata[\"test_score\"]` if you\n   passed a `test_set`).\n\n## Critical gotchas (read before a real run)\nThese silently degrade *results* — skim before launching:\n- **Reward hacking.** Every backend optimizes exactly what you score; a weak proxy gets gamed (e.g. a\n  \"correct\"-only score → a do-nothing wrapper). Gate the score on the real goal. → `references/gotchas.md`.\n- **Selection bias.** In generalization mode the best candidate is the max over many scored on\n  `valset` — an optimistic estimate. Use enough `valset` examples (and N>1 for stochastic models), and\n  report on a `test_set` for an unbiased number. → `references/gotchas.md`.\n- **Stochastic models default to N=1 per eval** → noisy selection. Average N samples *inside*\n  `evaluate`. → `references/writing_evaluators.md`.\n- **Saturated signal → the gepa backend returns the seed unchanged.** If the seed already aces the\n  training examples, every proposal looks \"not better\" and is rejected (many proposals, ~0 accepted).\n  Reflection needs examples the seed gets *wrong* to learn from — ensure `dataset` has real\n  failures. → `references/gotchas.md`.\n- **`engine_config` is validated strictly per backend** — an unknown key (including a leftover key\n  from a different backend after swapping `engine=`) raises `TypeError` at construction. Swapping\n  `engine=` means swapping the `engine_config` block. → `references/api.md`.\n- **Agentic backends (`autoresearch`, `meta_harness`) shell out to the `claude` CLI** and abort at\n  launch with install instructions if it's missing — likewise for `bwrap` (bubblewrap) on Linux,\n  which the default `sandbox=True` needs. `sandbox=False` runs the agent unconfined (loud warning).\n  Run `scripts/preflight.py` first; details in `references/api.md`.\n\n## Reference files (load as needed)\n- `references/api.md` — `OptimizeAnythingConfig`, the backends and their typed `engine_config`\n  options, the three modes, the LM protocol, budget/cost semantics, `GEPAResult` shape, and the\n  composition/pipeline helpers.\n- `references/writing_evaluators.md` — the `(score, info)` contract, `oa.log()`/`capture_stdio`,\n  LLM-as-judge scoring, multi-objective via `info[\"scores\"]`, N>1 averaging, feedback design.\n- `references/tracking.md` — enabling wandb / mlflow experiment tracking and what gets logged.\n- `references/gotchas.md` — reward hacking, selection bias, the three modes, backend prerequisites.\n- `scripts/preflight.py` — validate creds / proposer LM / `claude` CLI before launching.\n","frontmatter":{"name":"gepa-optimize-anything","description":"Automatically improve any text artifact that can be scored — prompts, programs/code, configs, specs, regex/SQL/schemas, agent scaffolds, or encoded search solutions — with optimize_anything, an LLM-driven optimization API: a search backend repeatedly proposes better versions from execution feedback and an evaluator you provide assigns the score (an objective metric or an LLM-as-judge for subjective tasks). optimize_anything is one interface across several optimizer backends — GEPA (reflective evolutionary search, the default), AutoResearch, MetaHarness — plus a best-of-N baseline: write the evaluator once and switch backend with a single argument. Use whenever a task needs auto-optimizing, tuning, or searching over text with any quality signal (accuracy, pass rate, latency, cost, judge rating); when building an evaluator + proposer loop; when running GEPA; or when comparing optimizers. The candidate can be any string an evaluator can grade, scored on one problem or generalized across many."},"isInternal":false,"tokens":4423,"sizeBytes":18411},{"name":"preflight.py","path":".claude/skills/gepa-optimize-anything/scripts/preflight.py","rawUrl":"https://raw.githubusercontent.com/gepa-ai/gepa/HEAD/.claude/skills/gepa-optimize-anything/scripts/preflight.py","title":"Claude Directive: preflight.py","category":"claude-rule","format":"markdown","content":"#!/usr/bin/env python3\n\"\"\"Pre-flight checks for an optimize_anything run (gepa package) — fail fast before a long job.\n\n    python preflight.py                      # checks gepa + reflection-LM creds\n    python preflight.py --engine autoresearch  # also checks the `claude` CLI + jq\n    GEPA_REFLECTION_LM=anthropic/claude-sonnet-4-6 python preflight.py --test-lm\n\nExit code 0 = all good; non-zero = at least one blocker.\n\"\"\"\nfrom __future__ import annotations\n\nimport argparse\nimport os\nimport shutil\nimport sys\n\nOK, BAD = \"\\033[32mOK\\033[0m\", \"\\033[31mFAIL\\033[0m\"\nproblems: list[str] = []\n\n# The gepa backend's reflection LM defaults to openai/gpt-5.1; best_of_n's\n# sampling model defaults to claude-sonnet-4-6 (see references/api.md).\nDEFAULT_LM_BY_ENGINE = {\"gepa\": \"openai/gpt-5.1\", \"best_of_n\": \"claude-sonnet-4-6\"}\n\n\ndef check(label: str, ok: bool, fix: str = \"\") -> None:\n    print(f\"  [{OK if ok else BAD}] {label}\")\n    if not ok:\n        problems.append(f\"{label} — {fix}\" if fix else label)\n\n\ndef _creds_for(lm: str) -> tuple[bool, str]:\n    \"\"\"Best-effort provider-credential check for a LiteLLM model id.\"\"\"\n    has_aws = bool(\n        os.environ.get(\"AWS_BEARER_TOKEN_BEDROCK\")\n        or os.environ.get(\"AWS_ACCESS_KEY_ID\")\n        or os.environ.get(\"AWS_PROFILE\")\n    )\n    if \"bedrock\" in lm:\n        return has_aws, \"export AWS creds (AWS_BEARER_TOKEN_BEDROCK / AWS_ACCESS_KEY_ID / AWS_PROFILE)\"\n    if lm.startswith(\"openai/\") or lm.startswith(\"gpt-\") or \"gpt-5\" in lm:\n        return bool(os.environ.get(\"OPENAI_API_KEY\")), \"export OPENAI_API_KEY\"\n    if \"claude\" in lm or lm.startswith(\"anthropic/\"):\n        return bool(os.environ.get(\"ANTHROPIC_API_KEY\")) or has_aws, \"export ANTHROPIC_API_KEY (or AWS creds)\"\n    # Unknown provider: accept any common key being present.\n    any_key = bool(\n        os.environ.get(\"OPENAI_API_KEY\") or os.environ.get(\"ANTHROPIC_API_KEY\") or has_aws\n    )\n    return any_key, \"export your LiteLLM provider's API key\"\n\n\ndef main() -> int:\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--engine\", default=\"gepa\",\n                    choices=[\"gepa\", \"best_of_n\", \"autoresearch\", \"meta_harness\"])\n    ap.add_argument(\"--test-lm\", action=\"store_true\",\n                    help=\"make a 1-call round-trip to the reflection LM (costs a few tokens)\")\n    a = ap.parse_args()\n\n    print(\"== optimize_anything preflight ==\")\n\n    # 1) import + the correct API surface\n    try:\n        import gepa  # noqa\n        from gepa.optimize_anything import OptimizeAnythingConfig, optimize_anything  # noqa\n        check(f\"import gepa ({getattr(gepa, '__version__', '?')}) + optimize_anything\", True)\n    except Exception as e:  # noqa\n        check(\"import gepa + optimize_anything\", False, \"pip install 'gepa[full]'\")\n        print(f\"      {e}\")\n        return _report()\n\n    # 2) LM credentials (in-process engines that call an LLM directly)\n    lm = os.environ.get(\"GEPA_REFLECTION_LM\", \"\")\n    if a.engine in (\"gepa\", \"best_of_n\"):\n        effective_lm = lm or DEFAULT_LM_BY_ENGINE[a.engine]\n        if not lm:\n            print(f\"      GEPA_REFLECTION_LM unset -> engine default '{effective_lm}'\")\n        ok, fix = _creds_for(effective_lm)\n        check(f\"LLM creds present for '{effective_lm}'\", ok, fix)\n\n    # 3) agentic engines need the claude CLI (and autoresearch's eval.sh uses jq)\n    if a.engine in (\"autoresearch\", \"meta_harness\"):\n        cli = shutil.which(\"claude\")\n        check(f\"`claude` CLI on PATH (required by {a.engine})\", bool(cli),\n              \"install + authenticate the Claude Code CLI headless\")\n        if cli:\n            print(f\"      claude -> {cli}\")\n        if a.engine == \"autoresearch\":\n            check(\"`jq` on PATH (used by the generated eval.sh)\", bool(shutil.which(\"jq\")),\n                  \"install jq\")\n        if sys.platform.startswith(\"linux\"):\n            check(\"`bwrap` on PATH (default sandbox=True jails claude with bubblewrap)\",\n                  bool(shutil.which(\"bwrap\")),\n                  \"sudo apt/dnf install bubblewrap, or pass sandbox=False (runs unconfined)\")\n\n    # 4) optional live LM round-trip\n    if a.test_lm and a.engine in (\"gepa\", \"best_of_n\"):\n        target = lm or DEFAULT_LM_BY_ENGINE[a.engine]\n        try:\n            from gepa.lm import LM\n            out = LM(target)(\"Reply with the single word: ok\")\n            check(f\"LM 1-call round-trip ({target})\", bool(out),\n                  \"LM returned empty; check model id / creds / region\")\n        except Exception as e:  # noqa\n            check(f\"LM 1-call round-trip ({target})\", False, str(e)[:160])\n\n    return _report()\n\n\ndef _report() -> int:\n    print()\n    if problems:\n        print(f\"\\033[31m{len(problems)} blocker(s):\\033[0m\")\n        for p in problems:\n            print(f\"  - {p}\")\n        return 1\n    print(\"\\033[32mAll preflight checks passed.\\033[0m\")\n    return 0\n\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n","isInternal":false,"tokens":1319,"sizeBytes":4922},{"name":"marketplace.json","path":".claude-plugin/marketplace.json","rawUrl":"https://raw.githubusercontent.com/gepa-ai/gepa/HEAD/.claude-plugin/marketplace.json","title":"gepa","category":"marketplace","format":"json","content":"{\n  \"name\": \"gepa\",\n  \"owner\": { \"name\": \"GEPA\", \"url\": \"https://github.com/gepa-ai/gepa\" },\n  \"description\": \"Official GEPA Claude Code plugins.\",\n  \"plugins\": [\n    {\n      \"name\": \"gepa-optimize-anything\",\n      \"source\": \"./.claude/skills/gepa-optimize-anything\",\n      \"description\": \"Optimize any scorable text artifact (prompts, code, configs, agent scaffolds) with GEPA's optimize_anything.\",\n      \"version\": \"0.1.0\",\n      \"homepage\": \"https://gepa-ai.github.io/gepa/\",\n      \"license\": \"MIT\",\n      \"category\": \"optimization\",\n      \"keywords\": [\"gepa\", \"prompt-optimization\", \"optimize-anything\"]\n    }\n  ]\n}\n","frontmatter":{"name":"gepa","owner":{"name":"GEPA","url":"https://github.com/gepa-ai/gepa"},"description":"Official GEPA Claude Code plugins.","plugins":[{"name":"gepa-optimize-anything","source":"./.claude/skills/gepa-optimize-anything","description":"Optimize any scorable text artifact (prompts, code, configs, agent scaffolds) with GEPA's optimize_anything.","version":"0.1.0","homepage":"https://gepa-ai.github.io/gepa/","license":"MIT","category":"optimization","keywords":["gepa","prompt-optimization","optimize-anything"]}]},"isInternal":false,"tokens":183,"sizeBytes":621},{"name":"plugin.json","path":".claude/skills/gepa-optimize-anything/.claude-plugin/plugin.json","rawUrl":"https://raw.githubusercontent.com/gepa-ai/gepa/HEAD/.claude/skills/gepa-optimize-anything/.claude-plugin/plugin.json","title":"gepa-optimize-anything","category":"plugin-manifest","format":"json","content":"{\n  \"name\": \"gepa-optimize-anything\",\n  \"version\": \"0.1.0\",\n  \"description\": \"Optimize any scorable text artifact (prompts, code, configs, agent scaffolds, search solutions) with optimize_anything — one evaluator-driven API across the GEPA optimizer (reflective evolutionary search, the default), AutoResearch, and MetaHarness backends, with a best-of-N baseline for comparisons.\",\n  \"author\": { \"name\": \"GEPA\", \"url\": \"https://github.com/gepa-ai/gepa\" },\n  \"homepage\": \"https://gepa-ai.github.io/gepa/\",\n  \"repository\": \"https://github.com/gepa-ai/gepa\",\n  \"license\": \"MIT\",\n  \"keywords\": [\"gepa\", \"optimization\", \"prompt-optimization\", \"optimize-anything\", \"agent-skill\", \"dspy\"]\n}\n","frontmatter":{"name":"gepa-optimize-anything","version":"0.1.0","description":"Optimize any scorable text artifact (prompts, code, configs, agent scaffolds, search solutions) with optimize_anything — one evaluator-driven API across the GEPA optimizer (reflective evolutionary search, the default), AutoResearch, and MetaHarness backends, with a best-of-N baseline for comparisons.","author":{"name":"GEPA","url":"https://github.com/gepa-ai/gepa"},"homepage":"https://gepa-ai.github.io/gepa/","repository":"https://github.com/gepa-ai/gepa","license":"MIT","keywords":["gepa","optimization","prompt-optimization","optimize-anything","agent-skill","dspy"]},"isInternal":false,"tokens":192,"sizeBytes":686}],"systemPromptSnippet":"<yakaai_skills repo=\"gepa-ai/gepa\">\n<!-- File: AGENTS.md (Tokens: ~448 | Category: root-instruction) -->\n# GEPA\n\nGEPA (Genetic-Pareto) is a Python framework for optimizing text components (AI prompts, code, instructions) using LLM-based reflection and Pareto-efficient evolutionary search.\n\n## Agent Skill\n\nThis repository ships an **Agent Skill** at `.claude/skills/gepa-optimize-anything/`. If you are a coding agent and the user wants to **auto-optimize, tune, or search over any scorable text artifact** (a prompt, program/code, config, regex/SQL, or agent scaffold) with `gepa.optimize_anything`, read `.claude/skills/gepa-optimize-anything/SKILL.md` first and follow it. It is auto-discovered by Claude Code and by other agents that read `.claude/skills/` (Cursor, VS Code/Copilot, Codex, Gemini CLI). To install it standalone in another project: `/plugin marketplace add gepa-ai/gepa` then `/plugin install gepa-optimize-anything@gepa`.\n\n## Setup\n\nWe use **uv** for dependency management. The project uses setuptools as the build backend. All python executions must be done through uv.\n\n```bash\nuv sync --extra dev\n```\n\n## Project Structure\n\n- `src/gepa/` — main package source\n  - `core/` — optimization loop, state, evaluation\n  - `proposer/` — candidate proposal and mutation logic\n  - `adapters/` — integration adapters (DSPy, RAG, MCP, etc.)\n  - `strategies/` — batch sampling and candidate selection\n  - `logging/` — experiment tracking and logging\n- `tests/` — pytest test suite\n- `docs/` — mkdocs documentation site\n\n## Build & Test\n\n```bash\nuv run pytest\nuv run ruff check src/\nuv run ruff format src/\nuv run pyright src/\n```\n\n## Code Style\n\n- Linter/formatter: ruff (line length 120, double quotes, space indent)\n- Type checking: pyright\n- Python target: 3.10+\n- No relative imports (enforced by ruff)\n\n<!-- File: .claude/skills/gepa-optimize-anything/references/api.md (Tokens: ~4634 | Category: anthropic-skill) -->\n# API reference — `optimize_anything`\n\nThe public entry point is `gepa.optimize_anything.optimize_anything`.\n\n## Signature\n```python\nfrom gepa.optimize_anything import GEPAResult, optimize_anything, OptimizeAnythingConfig\n\nresult = optimize_anything(\n    seed_candidate: str | None = None,   # starting text; None = seedless (engine bootstraps from objective/background)\n    *,\n    evaluator: Callable | None = None,       # (candidate) or (candidate, example) -> (score, info)\n    batch_evaluator: Callable | None = None, # list[(candidate, example)] -> one result per pair (see below)\n    dataset:  list | None = None,        # training examples used DURING optimization\n    valset:   list | None = None,        # validation examples used to SELECT the best candidate\n    objective: str | None = None,        # short goal, surfaced verbatim to the engine\n    background: str | None = None,       # long-form rules/constraints/domain notes\n    test_set: list | None = None,        # reporting-only: seed + final candidate scored here at the end\n    config: OptimizeAnythingConfig | None = None,\n) -> GEPAResult\n```\nExamples in `dataset`/`valset`/`test_set` are opaque — any object your `evaluator` understands.\n`seed_candidate` is a **single string** at this API (multi-component `dict[str, str]` candidates\nexist only in the lower-level `gepa.gepa_launcher.optimize_anything`).\n\n**`batch_evaluator`** — grouped scoring for when evals batch better than they stream (a provider\nbatch API, fan-out over your own cluster): it receives ALL `(candidate, example)` pairs of an\nevaluation stage (minibatch, valset, held-out test pass) in **one call** and returns one result per\npair (`score` or `(score, info)`). At least one of `evaluator` / `batch_evaluator` is required.\nWhen both are given, multi-pair stages use the batch function and single-pair evaluations use\n`evaluator`; with only `batch_evaluator`, singles route through it as singleton batches.\n\n### The three optimization modes (set by `dataset` / `valset`)\n| mode | pass | evaluator called | use when |\n|---|---|---|---|\n| **Single-task** | `dataset=None, valset=None` | `evaluator(candidate)` | solving one hard problem; the candidate *is* the solution (one kernel, a packing layout, one document). |\n| **Multi-task** | `dataset=<list>, valset=None` | `evaluator(candidate, example)` | one shared candidate must do well across a batch of related problems; insight transfers across them. |\n| **Generalization** | `dataset=<list>, valset=<list>` | `evaluator(candidate, example)` | the candidate must transfer to **unseen** problems: optimize on `dataset`, select on `valset`. |\n\n*Note on `valset`:* held-out selection is currently implemented only by the gepa backend (GEPA is\ndesigned around this mode — `valset` selection plus a Pareto frontier over val instances). The\nother backends fold `valset` into the pool the eval server exposes (`train_set + val_set` merged),\nso they optimize over the combined set with no separate selection split — they may still produce\ncandidates that generalize.\n\n**`test_set` is reporting-only and orthogonal to the mode.** Only `dataset` and `valset` affect\noptimization (search and selection). If you pass `test_set`, the seed and the final chosen candidate\nare scored on it *after* optimization, outside the budget, and reported in `result.metadata`. The\ntest set never enters the eval server's HTTP surface, so agentic backends cannot see it. Use it when\nyou want an unbiased number to report; skip it otherwise.\n\n## `OptimizeAnythingConfig` (top-level fields)\n| field | default | meaning |\n|---|---|---|\n| `engine` | `\"gepa\"` | `\"gepa\"` / `\"autoresearch\"` / `\"meta_harness\"` (or `\"best_of_n\"`, a baseline), or a constructed `Engine` instance (custom engines can be added via `gepa.oa.registry.register_engine`). |\n| `name` | `None` | run id (logging + default output dir). Auto-generated (`<engine>-<uuid>-<timestamp>`) if `None`. |\n| `max_evals` | **`100`** | server-side cap on eval calls. `None` = unlimited. Size it — the default is rarely right (see SKILL.md). |\n| `max_token_cost` | `None` | USD cap on the engine's **own** optimizer-LLM spend (reflection/agent). Enforced by the engine (gepa: `max_reflection_cost` stopper; agent engines: `--max-budget-usd`), not the eval server. |\n| `max_concurrency` | `8` | eval-server thread-pool size. |\n| `output_dir` | `None` | where the eval server writes per-eval JSON, `progress_log.jsonl`, `summary.json`. `None` → `outputs/optimize_anything/<task>/<engine>/<timestamp>/`. |\n| `run_dir` | `None` | engine workspace (gepa run dir / agent work dir; with `engine.write_agent_state=True` the gepa backend writes an agent-readable `iterations/` + `pareto/` tree here). Distinct from `output_dir`. `None` → subprocess engines use a tempdir; set it to persist artifacts. |\n| `stop_at_score` | `None` | early-stop score threshold; each engine interprets it. |\n| `sandbox` | **`True`** | OS-jail the subprocess engines' `claude` sessions: bwrap on Linux (needs the `bubblewrap` package — the run aborts at launch if `bwrap` is missing), Claude Code's Seatbelt sandbox on macOS. Also forces the work dir to a tempdir even when `run_dir` is set (artifacts are mirrored back). `False` prints a loud warning and runs the agent unsandboxed (`bypassPermissions`, full host access). |\n| `engine_config` | `{}` | dict of **engine-specific** options — see the per-backend sections below. Parsed into a typed per-engine config dataclass; **an unknown key raises `TypeError` immediately** (fail fast, not warn-and-drop). |\n\n**If both `max_evals` and `max_token_cost` are `None`, the run is unbounded** (only a\n`warnings.warn`). Note the former fields `tracker`, `effort`, and `max_thinking_tokens` no longer\nexist here: tracking is configured via the gepa backend's `engine_config[\"tracking\"]`\n(see `tracking.md`), and `effort` / `max_thinking_tokens` are per-agent-engine `engine_config` keys.\n\n## The backends\n\nAll are selected with `engine=` and run the **same** task/evaluator — switch optimizer by\nchanging one argument. Each backend parses `engine_config` into its own typed dataclass, so swapping\n`engine=` also means swapping the `engine_config` block (leftover keys → `TypeError`).\n\n| engine | how it proposes candidates | runs | needs |\n|---|---|---|---|\n| `gepa` | an LLM reflects on evaluator feedback and mutates the candidate; keeps a Pareto frontier | in-process | reflection-LM creds (default `openai/gpt-5.1`) or a custom LM |\n| `autoresearch` | one Claude Code subprocess iterates in a work dir (`program.md`, `candidate.txt`, `eval.sh` → HTTP eval server) | subprocess | `claude` on PATH + headless auth, `jq` |\n| `meta_harness` | a Claude subprocess reads frontier/history and writes `pending_eval.json` candidates; the engine benchmarks each | subprocess | `claude` on PATH + headless auth |\n| `best_of_n` *(baseline)* | independent single-shot samples from one LLM; keep the best — no feedback, no history | in-process | LiteLLM creds for `model` (default `claude-sonnet-4-6`) |\n\n`scripts/preflight.py` checks a backend's prerequisites before a long run.\n\n### `gepa` — `engine_config` is a `GEPAConfig`-shaped dict (1-to-1 passthrough)\nThe dict is passed to `gepa.gepa_launcher.GEPAConfig(**engine_config)`; valid top-level keys are\nexactly its fields — `engine`, `reflection`, `tracking`, `merge`, `refiner`, `callbacks`,\n`stop_callbacks` — and nested dicts are coerced to the matching dataclasses. The OA layer only\noverlays the eval budget (`max_evals` → `engine.max_metric_calls`), `run_dir`, `stop_at_score`, and\nthe `max_token_cost` → `engine.max_reflection_cost` cap.\n\n```python\nengine_config={\n    \"reflection\": {                     # -> ReflectionConfig\n        \"reflection_lm\": \"anthropic/claude-sonnet-4-6\",  # default \"openai/gpt-5.1\"; see \"Proposer LM\"\n        \"reflection_lm_kwargs\": {\"reasoning_effort\": \"high\"},  # litellm kwargs (temperature, thinking, …)\n        \"reflection_minibatch_size\": 5,  # default: 1 single-task, 3 otherwise\n        # \"custom_candidate_proposer\": ClaudeCodeAgentProposer(...),  # replace the reflection LM with\n        #                                # a Claude Code proposer (from gepa.oa.proposers)\n        # \"reflection_strategy\": ...,    # advanced: a ReflectionLM impl owning how reflection is called\n    },\n    \"engine\": {                          # -> EngineConfig (all optional, sensible defaults)\n        \"max_workers\": 32,               # parallel eval workers (default: cpu_count or 32)\n        \"seed\": 0,                       # reproducibility\n        \"frontier_type\": \"hybrid\",       # \"instance\" | \"objective\" | \"hybrid\" (default) | \"cartesian\"\n        \"candidate_selection_strategy\": \"pareto\",      # | \"current_best\" | \"epsilon_greedy\" | \"top_k_pareto\"\n        \"acceptance_criterion\": \"strict_improvement\",  # | \"improvement_or_equal\"\n        \"cache_evaluation\": False,       # opt-in: cache (candidate, split, example) evals.\n                                         # Distinct valset is isolated from trainset ids.\n                                         # valset=None still shares minibatch rollouts.\n        \"capture_stdio\": False,          # opt-in: route evaluator print() output into feedback\n        \"raise_on_exception\": True,      # False → evaluator exceptions become score 0 + info[\"error\"]\n        # \"write_agent_state\": True,     # agent-readable iterations/ + pareto/ tree under run_dir\n    },\n    \"tracking\": {\"use_wandb\": True},     # -> TrackingConfig (see references/tracking.md)\n    \"merge\":  {...},                     # -> MergeConfig (cross-candidate merging) or omit\n    \"refiner\": {...},                    # -> RefinerConfig (auto per-eval refinement) or omit\n    # \"callbacks\": [GEPACallback, ...],  # observation hooks (on_iteration_end, on_candidate_accepted, …)\n    # \"stop_callbacks\": [...],           # custom StopperProtocol stop conditions\n}\n```\nThe nested dataclasses live in `gepa.gepa_launcher` (`EngineConfig`, `ReflectionConfig`,\n`TrackingConfig`, `MergeConfig`, `RefinerConfig`); their knobs are documented in the GEPA guides —\ne.g. **candidate-selection**, **acceptance-criterion**, **batch-sampling**, **callbacks**,\n**cost-tracking**, **experiment-tracking** — see <https://gepa-ai.github.io/gepa/guides/>.\n\n### `autoresearch` — `engine_config` → `AutoResearchConfig`\n| key | default | meaning |\n|---|---|---|\n| `model` | `\"claude-sonnet-4-6\"` | Claude model id (pass `\"sonnet\"`/`\"opus\"` to track the current default). |\n| `ralph` | `True` | keep resuming one Claude session (`claude --resume`) while budget remains. |\n| `max_no_eval_seconds` | `None` | kill the subprocess after this long with no eval call. |\n| `handoffs` | `None` | prior-stage artifacts for sequential compositions (materialized under `handoff/`). |\n| `effort` | `None` | `claude --effort` value. |\n| `max_thinking_tokens` | `None` | fixed thinking-token budget (`MAX_THINKING_TOKENS`). |\n\nThe engine lays out a work dir (`program.md`, `candidate.txt`, `best_candidate.txt`, `eval.sh`) and\nlaunches `claude --print`; `eval.sh` POSTs candidates to the eval server, which enforces the budget\nserver-side (HTTP 429 on exhaustion) and caps LLM spend via `--max-budget-usd` (from\n`max_token_cost`). Train and val are presented to the agent as one combined pool; the test set is\nunreachable over HTTP.\n\n### `meta_harness` — `engine_config` → `MetaHarnessConfig`\n| key | default | meaning |\n|---|---|---|\n| `model` | `\"claude-sonnet-4-6\"` | proposer model id. |\n| `max_iterations` | `None` | hard cap on proposer sessions; `None` = until budget. |\n| `max_candidates_per_iter` | `3` | upper bound on candidates proposed per iteration. |\n| `effort` | `None` | `claude --effort` value. |\n| `max_thinking_tokens` | `None` | fixed thinking-token budget. |\n\nEach iteration the proposer subprocess reads the frontier + history state files, writes\n`pending_eval.json` with 1+ candidates, and the engine benchmarks each through the eval server.\n\n### `best_of_n` (baseline) — `engine_config` → `BestOfNConfig`\nDeliberately naive: each sample is one independent LLM call — no feedback, no history, no\nselection pressure beyond keep-the-best. Use it as a **comparison floor** when evaluating an\noptimizer, not as the optimizer. Stops on budget exhaustion, `stop_at_score`, or `max_n`.\n\n| key | default | meaning |\n|---|---|---|\n| `model` | `\"claude-sonnet-4-6\"` | LiteLLM model id used to sample candidates. |\n| `temperature` | `1.0` | sampling on by default so N calls don't collapse to one response. |\n| `max_n` | `None` | optional hard cap on samples; `None` = run until budget out. |\n| `lm_kwargs` | `{}` | extra kwargs forwarded to `gepa.lm.LM`. |\n| `effort` | `None` | threaded into the LM as `reasoning_effort`. |\n| `max_thinking_tokens` | `None` | fixed thinking-token budget for the LM. |\n\n## Proposer LM (not limited to LiteLLM)\n`reflection.reflection_lm` (gepa) accepts either:\n- a **model-id string** resolved through LiteLLM (`\"openai/gpt-5.1\"` — the default —\n  `\"anthropic/claude-sonnet-4-6\"`, a Bedrock inference-profile ARN, …); set the provider's\n  credentials, and pass litellm kwargs via `reflection.reflection_lm_kwargs`; or\n- **any object/callable implementing GEPA's LM protocol** — `__call__(prompt) -> str` (a\n  `str`-or-messages prompt in, completion text out). This is how you plug a **self-hosted or custom\n  inference engine** (vLLM, a local server, your own client) instead of LiteLLM.\n  `reflection_lm_kwargs` is ignored for callables.\n\n## Composition / pipeline helpers (ensembles)\n`gepa.optimize_anything` also exports helpers that compose several backends over the same\ntask/evaluator. Each takes the same flat task fields as `optimize_anything` (`seed_candidate`\npositional; `evaluator=`, `dataset=`, `valset=`, `objective=`, `background=`, `test_set=`, `name=`\nkeywords) plus `configs=` — a list of `OptimizeAnythingConfig`, one per stage/branch, each carrying\nits own budget. Budgets are pre-partitioned per config; an early-finishing stage's leftover is not\nredistributed.\n\n- `optimize_sequential` — **pipeline**: run configs in order; each stage's best becomes the next\n  stage's seed. Monotonic — the running best is carried forward, so a regressing stage doesn't\n  poison the chain.\n- `optimize_parallel` — run all configs concurrently; returns the list of `Result`s (config order).\n- `optimize_best_of` — parallel, then keep the result with the highest `best_score` (each engine's\n  own number — no re-scoring).\n- `optimize_vote` — parallel, then re-score each branch's best candidate once via `evaluator`\n  (outside any budget) and return the highest re-score. Use when engines score with different\n  quirks and you want a fair cross-engine comparison.\n- `optimize_adaptive_sequential` — **plateau scheduler**: give the active engine bounded eval\n  slices (`plateau_evals`) and rotate to the next config after `patience` non-improving slices,\n  always feeding the running best forward. The exception to per-config budgets: one shared eval\n  pool set by its `max_evals` keyword (per-config `max_evals` is ignored; per-config\n  `max_token_cost` still applies). Extra knobs: `min_evals_per_stage`, `improvement_epsilon`,\n  `cycle`, `max_switches`.\n\n```python\nfrom gepa.optimize_anything import optimize_sequential, OptimizeAnythingConfig\n\nresult = optimize_sequential(\n    SEED,\n    evaluator=evaluate,\n    dataset=trainset, valset=valset,\n    objective=\"...\",\n    configs=[\n        OptimizeAnythingConfig(engine=\"autoresearch\", max_evals=100, max_token_cost=5.0),\n        OptimizeAnythingConfig(engine=\"gepa\", max_evals=200),  # refines autoresearch's best\n    ],\n)\n```\n\n`*_with_server` variants (`optimize_sequential_with_server`, `optimize_parallel_with_server`, …)\ntake caller-owned `EvalServer`s — one per config (one shared server for\n`optimize_adaptive_sequential_with_server`) — for embedding in outer frameworks that route every\neval through their own server. Related: the autoresearch backend's `handoffs` key materializes\nprior-stage artifacts into the agent's work dir for hand-rolled sequential compositions.\n\n## `GEPAResult`\n\n`optimize_anything` always returns a `GEPAResult` (the same type as the legacy\n`gepa.gepa_launcher` path). The mutable engine-internal `Result` is still what\nengines and ensemble helpers thread; it is not the public return.\n\n```python\nresult.best_candidate   # str (or dict[str, str] for multi-component seeds)\nresult.best_score       # float (on valset/selection set)\nresult.total_evals      # int — eval-server call count (budget.used)\nresult.eval_log         # list[dict]\nresult.metadata         # dict (verified keys):\n#   \"test_score\"             avg over test_set         } only present\n#   \"test_scores\"            per-example dict          } if you passed\n#   \"baseline_test_score\"    seed avg over test_set    } a test_set\n#   \"baseline_test_scores\"   seed per-example dict     }\n#   \"budget\", \"total_cost\", \"adapter_cost\", \"wall_time\", \"engine\", \"output_dir\", \"progress_log\"\nresult.candidates            # full candidate pool (length 1 on non-gepa engines)\nresult.best_idx              # index of the highest val score\nresult.val_aggregate_scores  # per-candidate average validation score\nresult.to_dict()             # JSON-safe pool snapshot (does not include eval_log / metadata)\n```\n\nThe candidate pool lives on the result itself. `metadata` does **not** contain a nested\n`gepa_result`. `total_evals` is the eval-server count; `total_metric_calls` is GEPA core's\nown counter and may differ (e.g. after `BudgetExhausted`). `to_dict()` / `from_dict()` omit\n`eval_log`, `metadata`, and `eval_server_calls` so the frozen JSON pool contract stays\nbyte-stable.\n\nThe gepa engine also writes `run_dir/` artifacts, and the eval server writes\n`output_dir/summary.json`. (If the best candidate equals the seed, the test scores are reported from the\nseed's single scoring pass rather than re-scored.)\n\n**Migration:** callers that used `isinstance(result, Result)` or\n`result.metadata[\"gepa_result\"]` after `optimize_anything(...)` should switch to\n`GEPAResult` and `result.candidates`. Ensemble helpers (`optimize_sequential`, …)\nstill return the mutable `Result`.\n\n<!-- File: .claude/skills/gepa-optimize-anything/references/gotchas.md (Tokens: ~2086 | Category: anthropic-skill) -->\n# Gotchas and pitfalls\n\nRead before any real run.\n\n## 1. Reward hacking — the optimizer exploits a weak proxy ruthlessly\nEvery backend maximizes exactly the score you compute; if it is gameable, it *will* be gamed. Real\nexample (KernelBench, gepa backend): a correctness-only score drove the optimizer to a prompt that\ntells the model to **wrap the reference implementation** — always \"correct\", zero real work. Result: pass@1 ~0.87 but the actual goal (fast kernels) collapsed to ~base.\nMitigation: a **gated objective** — score 0 unless valid+correct, then increasing in the thing you\ncare about (see `writing_evaluators.md`). Sanity-check the *winning* candidate against your true goal,\nnot just the score.\n\n## 2. Selection bias / winner's curse (generalization mode)\nIn generalization mode the chosen candidate is the **max** over many candidates scored on `valset`.\nWith noisy (especially N=1) scores over few examples, that max is optimistically inflated — it can sit\nwell above true generalization (e.g. a selected best of 0.87 with runner-up 0.60 and median 0.57).\nThis bias lives in **selection on `valset`**, so reduce it there:\n- use **enough `valset` examples**, and **N>1 samples per eval** for stochastic models, to shrink the\n  variance that inflates the max;\n- treat `best_score` (the selection-set number) as optimistic.\n\n`test_set` does **not** reduce this bias — it is reporting-only. But scoring the final candidate on a\nheld-out `test_set` gives you an *honest* number to report — `metadata[\"test_score\"]` (average; also\n`[\"test_scores\"]` per-example) vs `[\"baseline_test_score\"]` for the seed — separate from the inflated\nselection score. (Note the exact keys: `test_score`/`test_scores`, **not** `test_score(s)`.)\n\n## 3. Stochastic N=1 default\nOne eval call per (candidate, example). For a temperature>0 model that means single-sample scores,\nwhich feeds the selection bias in #2. Average N samples inside `evaluate` (`writing_evaluators.md`).\n\n## 4. The default budget is probably wrong for your run\n`max_evals` defaults to **100** — bounded, but rarely sized for your selection set (see SKILL.md's\nsizing rule: ≳ 15–20 × the selection set). Set it explicitly. Only if you set **both**\n`max_evals=None` and `max_token_cost=None` is the run unbounded, and even that is just a\n`warnings.warn`, not an error.\n\n## 5. `engine_config` is validated strictly — unknown keys raise `TypeError`\nEvery backend parses `engine_config` into a typed dataclass (`GEPAConfig` for gepa,\n`BestOfNConfig` / `AutoResearchConfig` / `MetaHarnessConfig` for the others), so a typo'd or\nwrong-backend key fails **immediately at construction** with a `TypeError` — it is *not* silently\ndropped. The practical consequence: swapping `engine=` requires swapping the whole `engine_config`\nblock, and old-API keys (`claude_code_agent`, top-level `reflection_lm_kwargs`, `objective`,\n`background` inside `engine_config`) now crash. See `api.md` for each backend's valid keys.\n\n## 6. Agentic backends have launch-time prerequisites\n`autoresearch` / `meta_harness` `subprocess.Popen([\"claude\", ...])`. A missing `claude` CLI — or, on\nLinux, a missing `bwrap` (bubblewrap) while the default `sandbox=True` is in effect — aborts the run\nat launch with a boxed message and install instructions (`npm install -g @anthropic-ai/claude-code`;\n`sudo apt/dnf install bubblewrap`). An *unauthenticated* CLI or a missing `jq` (used by\nautoresearch's generated `eval.sh`) still surfaces only mid-run, and `sandbox=False` runs the agent\nunconfined (loud warning) — so run `scripts/preflight.py` first either way.\n\n## 7. Give runs a real stop condition (`stop_at_score` / `max_token_cost`)\n`max_evals` caps eval calls, but two situations still burn money or time past the point of useful\nwork:\n- **The metric has a ceiling** and a candidate reaches it — without `stop_at_score` the run keeps\n  proposing at the optimum until the budget is gone. For a bounded metric (pass rate, accuracy),\n  *always* pass `stop_at_score`.\n- **You opted into evaluation caching** (`engine_config={\"engine\": {\"cache_evaluation\": True}}` —\n  it is **off by default**). Then `max_evals` counts only cache *misses*: a converged search keeps\n  emitting cache-hitting candidates without consuming eval budget and can spin until your process\n  times out, still spending proposer-LLM tokens. With caching on, `stop_at_score` and/or\n  `max_token_cost` (plus a wall-clock `timeout` on the launched process) are mandatory.\n  A distinct `valset` is a separate cache namespace from the trainset (list-position ids would\n  otherwise collide). `valset=None` still shares cached rollouts with minibatches. Resuming a\n  `run_dir` whose cache predates that namespacing drops the old entries.\nAgentic backends also spend LLM tokens between evals — cap them with `max_token_cost`\n(enforced as `--max-budget-usd`).\n\n## 8. Pick the right mode\nThe mode is implicit in which sets you pass (`api.md`): no `dataset`/`valset` → single-task;\n`dataset` only → multi-task; `dataset`+`valset` → generalization. A common mistake is wanting a\ncandidate that generalizes but passing only `dataset` (multi-task) — then there's no held-out\nselection and the candidate can overfit the training examples. Pass a `valset` for generalization.\n(Held-out `valset` selection is implemented by the gepa backend; the other backends fold `valset`\ninto the scoring pool — see `api.md`.)\n\n## 9. Saturated signal — the gepa backend returns the seed unchanged (no failure to learn from)\nThis one is specific to the gepa backend, whose optimizer improves by reflecting on examples the\ncurrent candidate gets **wrong**. If the seed already scores near-perfect on the train minibatch the\nproposer reflects over, every mutation looks \"not better\" and is rejected by the acceptance gate —\nso the run spends its full budget, accepts **zero** proposals, and hands back the seed. Symptom:\nmany proposals made, ~0 accepted, `best_candidate` == seed. This is \"baseline saturation,\" not a\nbug: there's no gradient for reflection to climb.\nMitigations: make sure `dataset` contains examples the seed actually **fails** (harder/larger train\nset, or a task-LM weak enough to generate failures); if the seed is already at ceiling, reflective\nmutation has little to do — the `best_of_n` baseline (independent full rewrites, no acceptance\ngate) or just keeping the seed may be the better call.\nAlways check **proposals accepted**, not just proposals made.\n\n## 10. Uncaught evaluator exceptions abort the run\nOn the gepa backend, `EngineConfig.raise_on_exception` defaults to `True`: an exception escaping\nyour evaluator stops the whole optimization. Either catch failures yourself and return them as low scores with rich\n`info[\"error_*\"]` feedback (preferred — the proposer learns from them), or set\n`engine_config={\"engine\": {\"raise_on_exception\": False}}` to have them auto-converted to score\n`0.0` with `info[\"error\"]`.\n\n## Quick pre-flight checklist\n- [ ] mode matches intent (single-task / multi-task / generalization) via `dataset`/`valset`\n- [ ] `dataset` contains examples the seed gets WRONG — reflection has no failure signal to learn\n      from otherwise, and the gepa backend hands back the seed unchanged (#9)\n- [ ] `score` matches the real goal (gated, not gameable)\n- [ ] `info` carries actionable feedback (errors/diffs/partial credit)\n- [ ] N>1 if the model is stochastic\n- [ ] proposer LM valid + 1-call tested; creds present (default `openai/gpt-5.1` needs\n      `OPENAI_API_KEY` — or set your own via `reflection.reflection_lm`)\n- [ ] `max_evals` set explicitly and sized for many proposals (≳ 15–20 × the size of the selection\n      set: `len(valset)` in generalization, `len(dataset)` in multi-task, or just ≳ 15–20 in\n      single-task) — see SKILL.md\n- [ ] `stop_at_score` set when the metric has a ceiling; `max_token_cost` for agentic backends\n- [ ] `engine_config` keys match the chosen backend (typos raise `TypeError`, #5)\n- [ ] `run_dir` + `output_dir` set (so artifacts persist)\n- [ ] `test_set` passed if you need an unbiased number to report\n- [ ] for agentic backends: `claude` CLI on PATH + authed, `jq` installed, and on Linux `bwrap`\n      (bubblewrap) for the default `sandbox=True` (`scripts/preflight.py`)\n\n<!-- File: .claude/skills/gepa-optimize-anything/references/tracking.md (Tokens: ~725 | Category: anthropic-skill) -->\n# Experiment tracking\n\nThe gepa backend logs to **Weights & Biases** and **MLflow** via `TrackingConfig` — configure it\nunder `engine_config[\"tracking\"]` (a valid `GEPAConfig` field, coerced to `TrackingConfig`). The\nother backends have no built-in tracker; for them, rely on the eval server's `output_dir` records\n(always written, whatever the backend):\n\n```python\nresult = optimize_anything(\n    seed_candidate=SEED, evaluator=evaluate, dataset=trainset, valset=valset,\n    config=OptimizeAnythingConfig(\n        engine=\"gepa\", max_evals=300,\n        engine_config={\n            \"reflection\": {\"reflection_lm\": \"anthropic/claude-sonnet-4-6\"},\n            \"tracking\": {                       # -> TrackingConfig\n                \"use_wandb\": True,\n                \"wandb_init_kwargs\": {\"project\": \"my-gepa-run\", \"name\": \"experiment-1\"},\n            },\n        },\n    ),\n)\n```\n\n## `TrackingConfig` fields\n- `use_wandb`, `wandb_api_key`, `wandb_init_kwargs`, `wandb_attach_existing`, `wandb_step_metric`\n- `use_mlflow`, `mlflow_tracking_uri`, `mlflow_experiment_name`, `mlflow_attach_existing`\n- `key_prefix` — prepended to every logged key/name (e.g. `\"gepa/\"` → `gepa/val_score`)\n- `logger` — a custom `LoggerProtocol` for plain-text log lines\n\n## What gets logged\n- **scalars** — `val_program_average`, `best_score_on_valset`, `total_metric_calls`, …\n- **tables** — `candidates`, `proposals`, `valset_scores`, `valset_pareto_front`\n- **run summary** — seed and best-found component text\n- **interactive candidate-tree HTML**\n\n## Attaching to an existing run\nIf you've already called `wandb.init()` (or started an MLflow run) yourself — e.g. a parent script\nthat also logs other things — set `wandb_attach_existing=True` (or `mlflow_attach_existing=True`).\nThe tracker then logs **into your active run** without calling `init()`/`finish()`, so it won't\ndisrupt the run lifecycle. Combine with `key_prefix=\"gepa/\"` to keep its metrics in their own\nnamespace.\n\nWhen the host loop manages its own wandb step counter, also set `wandb_step_metric` (e.g.\n`\"gepa/iteration\"`): the tracker then declares its own x-axis via `wandb.define_metric` instead of\npassing `step=`, so its `1, 2, 3, …` steps don't collide with the host's counter (a collision makes\nwandb silently drop the data).\n\n## Custom hooks\nFor programmatic observation beyond metric logging, pass `engine_config={\"callbacks\": [...]}` —\n`GEPACallback` objects receive events like `on_optimization_start`, `on_iteration_end`,\n`on_candidate_accepted`, `on_proposal_end`. The eval server also always writes a flat record\n(per-eval JSON + `progress_log.jsonl` + `summary.json`) under `output_dir`, whatever tracking is\nconfigured.\n\nSee the official guide for the authoritative list of fields and behaviors:\n<https://gepa-ai.github.io/gepa/guides/experiment-tracking/>.\n\n<!-- File: .claude/skills/gepa-optimize-anything/references/writing_evaluators.md (Tokens: ~1854 | Category: anthropic-skill) -->\n# Writing evaluators\n\nThe evaluator is where almost all of your effort and almost all of the quality come from. Whatever\nbackend you choose can only be as good as the score you compute and the feedback you return.\n\n## The contract\n```python\ndef evaluate(candidate: str, example) -> tuple[float, dict]:\n    return score, info\n```\n- `score: float` — **higher is better**. This is what the optimizer maximizes and selects on.\n- `info: dict` — **free-form feedback shown to the proposer**: the gepa backend's reflection LM\n  (as \"Actionable Side Information\", ASI) or the agentic backends' agent (the `best_of_n` baseline\n  ignores it). This is the single biggest lever on mutation quality.\n- For single-task runs the signature is `evaluate(candidate)`; with `dataset`/`valset`/`test_set`\n  it's `evaluate(candidate, example)`. (Returning a bare `float` instead of a tuple also works —\n  the wrapper normalizes it — but then the proposer gets no feedback; always return the tuple.)\n\n### Batched form (`batch_evaluator=`)\nWhen evals batch better than they stream — a provider batch API, one job submission for a whole\nstage, fan-out over your own infrastructure — write the batched form instead of (or alongside) the\nper-pair one:\n```python\ndef batch_evaluate(pairs: list[tuple[str, Any]]) -> list:  # pairs = [(candidate, example), ...]\n    return [score_or_score_info_tuple for candidate, example in pairs]  # one result per pair, same order\n```\nEach evaluation stage (minibatch, valset, held-out test pass) arrives as ONE call with all its\npairs. Everything above about feedback-rich `info` applies per pair. Put diagnostics in each\nreturned `info` — the per-call channels below (`oa.log()`, `capture_stdio`) don't apply to the\ngrouped call.\n\n## Make `info` feedback-rich (not just a number)\nThe proposer (reflection LM or agent) writes the next candidate by reading `info`. Give it specifics:\n```python\nreturn score, {\n    \"score\": score,\n    \"output\": output,                 # what the candidate produced\n    \"expected\": example.get(\"gold\"),  # what was wanted (if available)\n    \"error_type\": err_type,           # compile error / wrong answer / format violation / timeout\n    \"error_detail\": traceback_or_diff,# the actual message — concrete > vague\n    \"passed_checks\": [...],           # partial credit signals\n    \"failed_checks\": [...],\n}\n```\nRule of thumb: if a smart human reading only `info` could tell you *how to fix the candidate*, the\nproposer LLM can too. If `info` is just `{\"score\": 0.0}`, the search is blind.\n\n### Two built-in channels for diagnostics\n- **`oa.log()`** — `import gepa.gepa_launcher as oa; oa.log(\"landing distance:\", d)` inside your\n  evaluator. Same calling convention as `print()`; output is captured per-eval (thread-safe) and\n  auto-included in the feedback under `info[\"log\"]`. For child threads, propagate the context via\n  `oa.get_log_context()` / `oa.set_log_context()`.\n- **`capture_stdio`** — set `engine_config={\"engine\": {\"capture_stdio\": True}}` (gepa backend) and\n  any `print()`/stdout/stderr during evaluation lands in the feedback under `\"stdout\"`/`\"stderr\"` —\n  useful for wrapping existing scripts with no code changes. (Doesn't catch C-extension or\n  subprocess output that bypasses Python's `sys.stdout`; route that through `oa.log()`.)\n\n## LLM-as-judge scoring (subjective tasks)\nThe score doesn't have to come from an objective metric. For open-ended tasks (writing quality,\nhelpfulness, tone, adherence to a rubric) the evaluator can call an **LLM judge** and use its rating\nas the score — and, importantly, return the judge's **written critique** as feedback:\n```python\ndef evaluate(candidate, example):\n    output = run_my_model(candidate, example)\n    verdict = judge_lm(JUDGE_RUBRIC.format(task=example, answer=output))  # your judge call\n    score = verdict[\"rating\"] / 10.0                  # normalize to a float, higher = better\n    return score, {\n        \"score\": score,\n        \"output\": output,\n        \"critique\": verdict[\"critique\"],              # the judge's reasoning → drives better proposals\n        \"rubric_breakdown\": verdict.get(\"by_criterion\"),\n    }\n```\nTips: pin the judge (fixed model + temperature, ideally a stronger model than the one being\noptimized), give it a concrete rubric, and average a few judge calls if its ratings are noisy\n(this is the N>1 idea below). The critique is often more valuable to the proposer than the number.\n\n## Stochastic models: beat the silent N=1 default\nThe eval server calls your function **once** per (candidate, example) — there is no\n`samples_per_eval` knob. For a temperature>0 model that makes every score a single-sample estimate,\nand candidate **selection** then runs on noisy numbers (this is how you get winner's-curse inflation).\nFix it inside `evaluate` by averaging N samples:\n```python\ndef evaluate(candidate, example, N=4):\n    outs = [run_my_model(candidate, example) for _ in range(N)]\n    scores = [grade(o, example) for o in outs]\n    score = sum(scores) / len(scores)           # mean correctness ~ pass@1 estimate\n    return score, {\"score\": score, \"n\": N,\n                   \"samples\": [{\"out\": o, \"s\": s} for o, s in zip(outs, scores)]}\n```\nTrade-off: N× more eval calls (budget). Pick N to balance variance vs `max_evals`.\n\n## Multi-objective optimization (e.g. correctness AND speed)\nThe gepa backend keeps an objective-level Pareto front (the other backends ignore\n`info[\"scores\"]`). To use it, **return per-objective metrics under `info[\"scores\"]`** — the adapter\nforwards them as `objective_scores`:\n```python\nreturn score, {\n    \"score\": score,                        # the scalar the optimizer selects on\n    \"scores\": {\"correct\": correct_rate,    # per-objective metrics for the Pareto frontier\n               \"speedup\": speedup},        # all \"higher is better\"\n    ...\n}\n```\n`engine_config={\"engine\": {\"frontier_type\": \"hybrid\"}}` is the default (instance- and\nobjective-level fronts combined); `\"objective\"`, `\"instance\"`, and `\"cartesian\"` are the\nalternatives. The scalar `score` still drives final selection; the per-objective scores shape the\nfrontier that candidates are drawn from.\n\n## Choose the score to match the real goal (avoid reward hacking)\nThe optimizer maximizes exactly what you write. A correctness-only score is gameable — e.g. in code/kernel\ntasks the optimizer learns to emit a trivial wrapper that is \"correct\" but does nothing useful.\nPrefer a **gated** objective:\n```python\ndef score_fn(result):\n    if not (result[\"compiled\"] and result[\"correct\"]):\n        return 0.0\n    return f(result[\"speedup\"])   # e.g. min(speedup / target, 1.0), monotonically increasing\n```\nThen the only way to raise the score is to be correct **and** better on what you care about. See\n`gotchas.md` for the full reward-hacking story.\n\n## Determinism & robustness\n- Make `evaluate` side-effect-free and resumable; it may run concurrently (`max_concurrency`,\n  `engine.max_workers`) and be retried.\n- Set a seed in `engine_config={\"engine\": {\"seed\": 0}}` for reproducible search order.\n- Log your own per-eval record (id, score, sub-metrics, candidate hash) — you'll want it for\n  analysis; the eval server's `output_dir` per-eval JSON covers the basics, and `oa.log()` covers\n  in-feedback diagnostics.\n- Catch and *return* failures as low scores with `info[\"error_*\"]`, rather than raising —\n  `EngineConfig.raise_on_exception` defaults to `True`, so an uncaught exception aborts the run.\n  (Setting it to `False` converts exceptions to score `0.0` with `info[\"error\"]` instead.)\n\n<!-- File: .claude/skills/gepa-optimize-anything/SKILL.md (Tokens: ~4423 | Category: anthropic-skill) -->\n---\nname: gepa-optimize-anything\ndescription: >-\n  Automatically improve any text artifact that can be scored — prompts, programs/code, configs,\n  specs, regex/SQL/schemas, agent scaffolds, or encoded search solutions — with optimize_anything,\n  an LLM-driven optimization API: a search backend repeatedly proposes better versions from\n  execution feedback and an evaluator you provide assigns the score (an objective metric or an\n  LLM-as-judge for subjective tasks). optimize_anything is one interface across several optimizer\n  backends — GEPA (reflective evolutionary search, the default), AutoResearch, MetaHarness — plus a\n  best-of-N baseline: write the evaluator once and switch backend with a single argument.\n  Use whenever a task needs auto-optimizing, tuning, or searching over text with any quality signal\n  (accuracy, pass rate, latency, cost, judge rating); when building an evaluator + proposer loop;\n  when running GEPA; or when comparing optimizers. The candidate can be any string an evaluator can\n  grade, scored on one problem or generalized across many.\n---\n\n# `optimize_anything`\n\n**Naming, precisely.** `optimize_anything` is the tool: a general API for optimizing text\nartifacts. **GEPA** is one specific optimizer behind it — reflective evolutionary search, the\ndefault backend (`engine=\"gepa\"`) — and, for legacy reasons, also the name of the Python package\nthat ships all of this. In this skill, \"the gepa backend\" always means the optimizer; statements\nabout \"the optimizer\" or \"the backend\" apply to whichever engine you chose.\n\n`optimize_anything` does **black-box optimization**: you provide (1) a seed artifact, (2) an\n**evaluator** that scores any artifact and returns feedback, and (3) a backend, which repeatedly\nproposes improved artifacts and scores them through your evaluator. \"Black-box\" refers to the\n**evaluator**, not the artifact: the backend never sees how the score is computed — no gradients,\nno metric internals — only the scalar score and the feedback text you emit. The candidate itself\n*is* visible: the proposer reads and rewrites it, applying the LLM's understanding of your artifact.\nThe framework just imposes no structure on it — any string an evaluator can score works. The\nleverage is in your score and your feedback.\n\n**You write the task and evaluator once, then choose the search algorithm with one `engine`\nargument** — and the same code runs under any of them:\n- **`gepa`** — the GEPA optimizer: reflective evolutionary search, in-process (an LLM reflects on\n  feedback and mutates candidates; keeps a Pareto frontier). The default; strongest when feedback\n  is rich.\n- **`autoresearch`** — an agentic optimizer: one Claude Code subprocess iterates like a researcher in\n  a work dir, scoring candidates through an HTTP eval server.\n- **`meta_harness`** — an agentic proposer (Claude subprocess) that reads the frontier/history each\n  iteration and writes new candidates for the engine to benchmark.\n\n(There is also a `best_of_n` engine — sample N independent candidates, keep the best. It is\ndeliberately naive: use it as a **baseline** to compare an optimizer against, not as the optimizer.)\n\nThis makes it easy to start with one backend and benchmark others on the identical task/evaluator.\nThere are also **composition/pipeline helpers** that combine backends over the same task:\n`optimize_sequential` (a pipeline — each stage's best seeds the next), `optimize_parallel`,\n`optimize_best_of`, `optimize_vote` (re-score each branch's best for a fair pick), and an adaptive\nscheduler that rotates backends on score plateaus — see `references/api.md`.\n\n## What can be a candidate\nA candidate is **any string your evaluator can score**. \"Text in → a number out (higher is better),\nplus optional feedback\" is the entire contract, which covers a wide range of artifacts:\n- **prompts** — system/user prompts, instruction templates, rubrics, few-shot exemplars\n- **programs / code** — functions, whole files, CUDA kernels, scored by compiling + running + benchmarking\n- **configs / specs / schemas / regex / SQL** — any text whose effect you can measure\n- **agent scaffolds** — tool instructions, planner/critic prompts, orchestration text\n- **pure search artifacts** — a mathematical construction, a packing layout, a plan, encoded as text\n\nThe score can be an **objective metric** (accuracy, pass rate, runtime, cost) **or an LLM-as-judge**\nrating for subjective tasks (writing quality, helpfulness, style). At this API the candidate is a\nsingle string (`seed_candidate: str | None`; `None` = seedless — the engine bootstraps from\n`objective`/`background`). Multi-component dict candidates exist only in the lower-level\n`gepa.gepa_launcher.optimize_anything` API, not here.\n\n## Three optimization modes (choose by how you pass data)\nThe mode is determined by whether you provide `dataset` and `valset`:\n1. **Single-task** (`dataset=None, valset=None`) — solve one hard problem; the candidate *is* the\n   solution; the evaluator is called with no example. *E.g. one CUDA kernel, a circle-packing layout.*\n2. **Multi-task** (`dataset=<list>, valset=None`) — solve a batch of related problems with one shared\n   candidate, transferring insight across them; evaluator called per example. *E.g. a single prompt\n   that works across many tasks.*\n3. **Generalization** (`dataset=<list>, valset=<list>`) — build a candidate that transfers to\n   **unseen** problems; optimize on `dataset`, select on `valset`. *E.g. a prompt tuned to generalize.*\n   *Note:* GEPA is the algorithm designed around this mode, and `valset`-based held-out selection is\n   currently implemented only by the gepa backend — the other backends fold `valset` into the\n   training pool (they can still generalize; there's just no separate selection split).\n\n`test_set` is **separate from the modes and reporting-only**: the seed and the final candidate are\nscored on it after optimization for an unbiased number — it never enters the search, selection, or\nbudget (for the agentic backends it is sealed at the eval server's HTTP layer, so the agent cannot\neven see it). See `references/api.md` for details and when to use each mode.\n\n## Install\n```bash\npip install \"gepa[full]\"   # [full] pulls cloudpickle — needed to pickle closure evaluators for\n                           # parallel workers / opt-in evaluation caching; plain `pip install gepa`\n                           # can fail there when your evaluator closes over data.\n# Proposer LLM: the gepa backend's reflection LM defaults to \"openai/gpt-5.1\" (a LiteLLM id) — set\n# that provider's key (OPENAI_API_KEY), or pass your own id (e.g. \"anthropic/claude-sonnet-4-6\" with\n# ANTHROPIC_API_KEY, or a Bedrock ARN with AWS creds). You can also pass any callable implementing\n# GEPA's LM protocol — a self-hosted / custom inference engine — instead of a model-id string.\n# Agentic backends (autoresearch, meta_harness) additionally need the `claude` CLI on PATH (plus\n# `jq` for the generated eval.sh):\nnpm install -g @anthropic-ai/claude-code   # or: curl -fsSL https://claude.ai/install.sh | bash\n#   ...then run `claude` once to authenticate.\n# On Linux they also need bubblewrap: the default sandbox=True jails the claude subprocess with\n# bwrap (`sudo apt install bubblewrap` / `sudo dnf install bubblewrap`) and the run aborts at\n# launch if it's missing. sandbox=False opts out — the agent then runs unsandboxed (loud warning).\n```\n\n## Mental model (4 pieces)\n1. **Candidate** — the text you optimize (a `str`). The seed is the start; `None` = seedless.\n2. **`evaluate(candidate, example) -> (score, info)`** — `score` is a float (higher is better);\n   `info` is a free-form dict of **feedback the backend's proposer reads** to make better candidates\n   (the gepa backend's reflection LM, or the agentic backends' agent; the `best_of_n` baseline\n   ignores feedback). When evals batch better than they stream (e.g. a provider batch API), pass\n   `batch_evaluator=` instead — all pending `(candidate, example)` pairs in one call; see\n   `references/api.md`.\n3. **Engine (backend)** — `\"gepa\"` (default), `\"autoresearch\"`, `\"meta_harness\"`, the `\"best_of_n\"`\n   baseline, or a constructed `Engine` instance.\n4. **Budget** — `max_evals` (server-side eval-call cap, **default 100**) and/or `max_token_cost`\n   (USD cap on the backend's own proposer-LLM spend). Setting both to `None` makes the run unbounded\n   (only a warning). **Size `max_evals` for many proposal rounds, not one** (see below) — this is the\n   most common way agents misuse this API.\n\n## Sizing the valset and the budget (read this — the #1 mistake)\n`max_evals` is the *main* control over how long the optimizer runs. Leave it at the default (100)\nwith a large valset, or set it too low, and the run stops after a **single proposal**, then reports\na \"best candidate\" that looks fine but is barely optimized.\n\n- **valset** — on the gepa backend the best candidate is selected by its scores on the valset, so\n  make it a **representative subset of your data**. There is no fixed size: pick what represents the task (a\n  handful for small/expensive tasks, more when data is cheap and plentiful — anywhere from a few to\n  hundreds). Bigger valset = less noisy selection but more eval calls per candidate.\n- **budget** — every proposed candidate is scored on the **whole selection set**, so size\n  `max_evals` off that set, not off a single proposal. Which set that is depends on the mode:\n  ```\n  generalization (dataset + valset):  max_evals ≳ 15–20 × len(valset)    # scored & selected on valset\n  multi-task     (dataset only):      max_evals ≳ 15–20 × len(dataset)   # scored & selected on dataset\n  single-task    (no dataset/valset): max_evals ≳ 15–20                  # 1 eval per candidate, so this\n                                                                         #   IS the number of proposals\n  ```\n  The constant is the same everywhere — **let the backend propose AND evaluate ~15–20 candidates**\n  (more if you can afford it). Anything much less and the run tries only a couple of candidates —\n  i.e. it's barely optimizing. (This arithmetic is exact for the gepa backend — and the best_of_n\n  baseline — which score every candidate on the full selection set; the agentic backends decide\n  themselves how to spend eval calls, so treat it as a floor.)\n\nAfter the run, check how many proposals actually happened (on the gepa backend,\n`result.candidates` is the full pool; with `engine.write_agent_state=True` the\n`run_dir/iterations/` tree shows each one). **If it stopped after one proposal, the budget was too\nlow** — raise it and rerun.\n\n### Give every run a real stop condition\n`max_evals` bounds eval calls, but add explicit stops so runs end at the right moment:\n- **`stop_at_score`** — set it whenever your metric has a known ceiling (e.g. `1.0` for a pass rate /\n  accuracy). The backend stops the moment a candidate reaches it instead of burning the rest of the\n  budget at the optimum.\n- **`max_token_cost`** — a hard USD cap on the backend's own proposer/agent LLM spend. Especially\n  important for the agentic backends (`autoresearch`, `meta_harness`), whose Claude subprocesses\n  spend tokens between eval calls.\n- **a wall-clock `timeout`** on the process you launch (e.g. `timeout 1200 python run.py`) as a\n  backstop.\n- If you **opt in** to evaluation caching (`engine_config={\"engine\": {\"cache_evaluation\": True}}` on\n  the gepa backend — it is **off by default**), be aware `max_evals` then counts only cache *misses*:\n  a converged search can keep proposing cache-hitting candidates without consuming eval budget, so\n  `stop_at_score`/`max_token_cost` become mandatory, not optional. A distinct `valset` is cached\n  separately from the trainset; `valset=None` still reuses minibatch rollouts.\n\n## Minimal working example\nThe example optimizes a system prompt for concreteness, but the **shape is identical** for any\ncandidate — swap `SEED` for a code file / config / etc. and have `evaluate` compile/run/measure it.\n```python\nfrom gepa.optimize_anything import optimize_anything, OptimizeAnythingConfig\n\nSEED = \"You are an expert. Solve the task. Output only the final answer.\"\n\ndef evaluate(candidate: str, example) -> tuple[float, dict]:\n    output = run_my_model(system_prompt=candidate, user_prompt=example[\"prompt\"])  # your call\n    score = grade(output, example)                                                 # float, higher=better\n    return score, {                       # everything here is shown to the proposer LLM\n        \"score\": score,\n        \"output\": output,\n        \"error\": example.get(\"error\"),    # concrete, actionable feedback drives good proposals\n    }\n\nresult = optimize_anything(\n    seed_candidate=SEED,\n    evaluator=evaluate,\n    dataset=trainset,            # optimize on these (multi-task/generalization mode)\n    valset=valset,               # select the best candidate on these (generalization mode)\n    test_set=testset,            # OPTIONAL, reporting-only: seed + final candidate scored here at the end\n    objective=\"Produce a prompt that maximizes task accuracy.\",\n    background=\"Domain rules, constraints, output format the model must follow.\",\n    config=OptimizeAnythingConfig(\n        engine=\"gepa\",               # swap to \"autoresearch\" / \"meta_harness\" — same code\n                                     #   (\"best_of_n\" runs the same way, as a comparison baseline)\n        name=\"my_run\",\n        max_evals=300,               # ≳ 15-20 × len(valset): enough for ~15-20 proposals (see above)\n        stop_at_score=1.0,           # stop at the optimum (set when your metric has a known ceiling)\n        max_concurrency=16,\n        run_dir=\"runs/my_run\",       # engine workspace (gepa run dir / agent work dir)\n        output_dir=\"outputs/my_run\", # eval server: per-eval JSON, progress_log.jsonl, summary.json\n        engine_config={              # gepa backend: a GEPAConfig-shaped dict, validated strictly —\n            \"reflection\": {          #   an unknown key raises TypeError immediately (fail fast)\n                # a LiteLLM id (set the provider key) OR any callable implementing the LM protocol\n                \"reflection_lm\": \"anthropic/claude-sonnet-4-6\",\n                \"reflection_minibatch_size\": 5,\n            },\n            \"engine\": {\"max_workers\": 32, \"seed\": 0},  # seed = reproducibility\n        },\n    ),\n)\nprint(result.best_candidate, result.best_score)\n# held-out (only present if you passed test_set): \"test_score\" = average, \"test_scores\" = per-example\nprint(\"held-out:\", result.metadata.get(\"test_score\"),\n      \"seed held-out:\", result.metadata.get(\"baseline_test_score\"))\n```\n\n## Standard workflow\n1. **Pick the mode** (single-task / multi-task / generalization) by which of `dataset`/`valset` you pass.\n2. **Define the score deliberately.** The optimizer optimizes exactly what you measure — gate the\n   score on what you actually care about (see `references/gotchas.md`, reward hacking).\n3. **Write a feedback-rich `evaluate`.** The `info` dict is the proposer's signal — return errors,\n   diffs, partial credit, not just a number (`oa.log()` and `capture_stdio` can route diagnostics in\n   automatically). See `references/writing_evaluators.md`.\n4. **Pick a proposer LLM** — a LiteLLM id (set the provider key) or a custom LM-protocol callable.\n   Validate it with a 1-call test before a long run.\n5. **Set a budget** (`max_evals` sized per above, and/or `max_token_cost`) plus `stop_at_score` when\n   the metric has a ceiling.\n6. **Run `python scripts/preflight.py`** to fail fast on missing creds / CLI before a long run.\n7. **Launch**, watch the first 1-2 evals (the eval→model→score chain), then let it run.\n8. **Read `result.best_candidate` and `run_dir/`** (and `result.metadata[\"test_score\"]` if you\n   passed a `test_set`).\n\n## Critical gotchas (read before a real run)\nThese silently degrade *results* — skim before launching:\n- **Reward hacking.** Every backend optimizes exactly what you score; a weak proxy gets gamed (e.g. a\n  \"correct\"-only score → a do-nothing wrapper). Gate the score on the real goal. → `references/gotchas.md`.\n- **Selection bias.** In generalization mode the best candidate is the max over many scored on\n  `valset` — an optimistic estimate. Use enough `valset` examples (and N>1 for stochastic models), and\n  report on a `test_set` for an unbiased number. → `references/gotchas.md`.\n- **Stochastic models default to N=1 per eval** → noisy selection. Average N samples *inside*\n  `evaluate`. → `references/writing_evaluators.md`.\n- **Saturated signal → the gepa backend returns the seed unchanged.** If the seed already aces the\n  training examples, every proposal looks \"not better\" and is rejected (many proposals, ~0 accepted).\n  Reflection needs examples the seed gets *wrong* to learn from — ensure `dataset` has real\n  failures. → `references/gotchas.md`.\n- **`engine_config` is validated strictly per backend** — an unknown key (including a leftover key\n  from a different backend after swapping `engine=`) raises `TypeError` at construction. Swapping\n  `engine=` means swapping the `engine_config` block. → `references/api.md`.\n- **Agentic backends (`autoresearch`, `meta_harness`) shell out to the `claude` CLI** and abort at\n  launch with install instructions if it's missing — likewise for `bwrap` (bubblewrap) on Linux,\n  which the default `sandbox=True` needs. `sandbox=False` runs the agent unconfined (loud warning).\n  Run `scripts/preflight.py` first; details in `references/api.md`.\n\n## Reference files (load as needed)\n- `references/api.md` — `OptimizeAnythingConfig`, the backends and their typed `engine_config`\n  options, the three modes, the LM protocol, budget/cost semantics, `GEPAResult` shape, and the\n  composition/pipeline helpers.\n- `references/writing_evaluators.md` — the `(score, info)` contract, `oa.log()`/`capture_stdio`,\n  LLM-as-judge scoring, multi-objective via `info[\"scores\"]`, N>1 averaging, feedback design.\n- `references/tracking.md` — enabling wandb / mlflow experiment tracking and what gets logged.\n- `references/gotchas.md` — reward hacking, selection bias, the three modes, backend prerequisites.\n- `scripts/preflight.py` — validate creds / proposer LM / `claude` CLI before launching.\n\n<!-- File: .claude/skills/gepa-optimize-anything/scripts/preflight.py (Tokens: ~1319 | Category: claude-rule) -->\n#!/usr/bin/env python3\n\"\"\"Pre-flight checks for an optimize_anything run (gepa package) — fail fast before a long job.\n\n    python preflight.py                      # checks gepa + reflection-LM creds\n    python preflight.py --engine autoresearch  # also checks the `claude` CLI + jq\n    GEPA_REFLECTION_LM=anthropic/claude-sonnet-4-6 python preflight.py --test-lm\n\nExit code 0 = all good; non-zero = at least one blocker.\n\"\"\"\nfrom __future__ import annotations\n\nimport argparse\nimport os\nimport shutil\nimport sys\n\nOK, BAD = \"\\033[32mOK\\033[0m\", \"\\033[31mFAIL\\033[0m\"\nproblems: list[str] = []\n\n# The gepa backend's reflection LM defaults to openai/gpt-5.1; best_of_n's\n# sampling model defaults to claude-sonnet-4-6 (see references/api.md).\nDEFAULT_LM_BY_ENGINE = {\"gepa\": \"openai/gpt-5.1\", \"best_of_n\": \"claude-sonnet-4-6\"}\n\n\ndef check(label: str, ok: bool, fix: str = \"\") -> None:\n    print(f\"  [{OK if ok else BAD}] {label}\")\n    if not ok:\n        problems.append(f\"{label} — {fix}\" if fix else label)\n\n\ndef _creds_for(lm: str) -> tuple[bool, str]:\n    \"\"\"Best-effort provider-credential check for a LiteLLM model id.\"\"\"\n    has_aws = bool(\n        os.environ.get(\"AWS_BEARER_TOKEN_BEDROCK\")\n        or os.environ.get(\"AWS_ACCESS_KEY_ID\")\n        or os.environ.get(\"AWS_PROFILE\")\n    )\n    if \"bedrock\" in lm:\n        return has_aws, \"export AWS creds (AWS_BEARER_TOKEN_BEDROCK / AWS_ACCESS_KEY_ID / AWS_PROFILE)\"\n    if lm.startswith(\"openai/\") or lm.startswith(\"gpt-\") or \"gpt-5\" in lm:\n        return bool(os.environ.get(\"OPENAI_API_KEY\")), \"export OPENAI_API_KEY\"\n    if \"claude\" in lm or lm.startswith(\"anthropic/\"):\n        return bool(os.environ.get(\"ANTHROPIC_API_KEY\")) or has_aws, \"export ANTHROPIC_API_KEY (or AWS creds)\"\n    # Unknown provider: accept any common key being present.\n    any_key = bool(\n        os.environ.get(\"OPENAI_API_KEY\") or os.environ.get(\"ANTHROPIC_API_KEY\") or has_aws\n    )\n    return any_key, \"export your LiteLLM provider's API key\"\n\n\ndef main() -> int:\n    ap = argparse.ArgumentParser()\n    ap.add_argument(\"--engine\", default=\"gepa\",\n                    choices=[\"gepa\", \"best_of_n\", \"autoresearch\", \"meta_harness\"])\n    ap.add_argument(\"--test-lm\", action=\"store_true\",\n                    help=\"make a 1-call round-trip to the reflection LM (costs a few tokens)\")\n    a = ap.parse_args()\n\n    print(\"== optimize_anything preflight ==\")\n\n    # 1) import + the correct API surface\n    try:\n        import gepa  # noqa\n        from gepa.optimize_anything import OptimizeAnythingConfig, optimize_anything  # noqa\n        check(f\"import gepa ({getattr(gepa, '__version__', '?')}) + optimize_anything\", True)\n    except Exception as e:  # noqa\n        check(\"import gepa + optimize_anything\", False, \"pip install 'gepa[full]'\")\n        print(f\"      {e}\")\n        return _report()\n\n    # 2) LM credentials (in-process engines that call an LLM directly)\n    lm = os.environ.get(\"GEPA_REFLECTION_LM\", \"\")\n    if a.engine in (\"gepa\", \"best_of_n\"):\n        effective_lm = lm or DEFAULT_LM_BY_ENGINE[a.engine]\n        if not lm:\n            print(f\"      GEPA_REFLECTION_LM unset -> engine default '{effective_lm}'\")\n        ok, fix = _creds_for(effective_lm)\n        check(f\"LLM creds present for '{effective_lm}'\", ok, fix)\n\n    # 3) agentic engines need the claude CLI (and autoresearch's eval.sh uses jq)\n    if a.engine in (\"autoresearch\", \"meta_harness\"):\n        cli = shutil.which(\"claude\")\n        check(f\"`claude` CLI on PATH (required by {a.engine})\", bool(cli),\n              \"install + authenticate the Claude Code CLI headless\")\n        if cli:\n            print(f\"      claude -> {cli}\")\n        if a.engine == \"autoresearch\":\n            check(\"`jq` on PATH (used by the generated eval.sh)\", bool(shutil.which(\"jq\")),\n                  \"install jq\")\n        if sys.platform.startswith(\"linux\"):\n            check(\"`bwrap` on PATH (default sandbox=True jails claude with bubblewrap)\",\n                  bool(shutil.which(\"bwrap\")),\n                  \"sudo apt/dnf install bubblewrap, or pass sandbox=False (runs unconfined)\")\n\n    # 4) optional live LM round-trip\n    if a.test_lm and a.engine in (\"gepa\", \"best_of_n\"):\n        target = lm or DEFAULT_LM_BY_ENGINE[a.engine]\n        try:\n            from gepa.lm import LM\n            out = LM(target)(\"Reply with the single word: ok\")\n            check(f\"LM 1-call round-trip ({target})\", bool(out),\n                  \"LM returned empty; check model id / creds / region\")\n        except Exception as e:  # noqa\n            check(f\"LM 1-call round-trip ({target})\", False, str(e)[:160])\n\n    return _report()\n\n\ndef _report() -> int:\n    print()\n    if problems:\n        print(f\"\\033[31m{len(problems)} blocker(s):\\033[0m\")\n        for p in problems:\n            print(f\"  - {p}\")\n        return 1\n    print(\"\\033[32mAll preflight checks passed.\\033[0m\")\n    return 0\n\n\nif __name__ == \"__main__\":\n    sys.exit(main())\n\n<!-- File: .claude-plugin/marketplace.json (Tokens: ~183 | Category: marketplace) -->\n{\n  \"name\": \"gepa\",\n  \"owner\": { \"name\": \"GEPA\", \"url\": \"https://github.com/gepa-ai/gepa\" },\n  \"description\": \"Official GEPA Claude Code plugins.\",\n  \"plugins\": [\n    {\n      \"name\": \"gepa-optimize-anything\",\n      \"source\": \"./.claude/skills/gepa-optimize-anything\",\n      \"description\": \"Optimize any scorable text artifact (prompts, code, configs, agent scaffolds) with GEPA's optimize_anything.\",\n      \"version\": \"0.1.0\",\n      \"homepage\": \"https://gepa-ai.github.io/gepa/\",\n      \"license\": \"MIT\",\n      \"category\": \"optimization\",\n      \"keywords\": [\"gepa\", \"prompt-optimization\", \"optimize-anything\"]\n    }\n  ]\n}\n\n<!-- File: .claude/skills/gepa-optimize-anything/.claude-plugin/plugin.json (Tokens: ~192 | Category: plugin-manifest) -->\n{\n  \"name\": \"gepa-optimize-anything\",\n  \"version\": \"0.1.0\",\n  \"description\": \"Optimize any scorable text artifact (prompts, code, configs, agent scaffolds, search solutions) with optimize_anything — one evaluator-driven API across the GEPA optimizer (reflective evolutionary search, the default), AutoResearch, and MetaHarness backends, with a best-of-N baseline for comparisons.\",\n  \"author\": { \"name\": \"GEPA\", \"url\": \"https://github.com/gepa-ai/gepa\" },\n  \"homepage\": \"https://gepa-ai.github.io/gepa/\",\n  \"repository\": \"https://github.com/gepa-ai/gepa\",\n  \"license\": \"MIT\",\n  \"keywords\": [\"gepa\", \"optimization\", \"prompt-optimization\", \"optimize-anything\", \"agent-skill\", \"dspy\"]\n}\n</yakaai_skills>"}