{"owner":"HKUDS","repo":"OpenHarness","hasSkills":true,"totalSkillsCount":13,"totalTokensCount":8794,"categories":["anthropic-skill"],"hasMcp":false,"mcpConfig":null,"found":[".claude/skills/harness-eval/SKILL.md",".claude/skills/harness-eval/references/feature-matrix.md",".claude/skills/harness-eval/references/test-patterns.md",".claude/skills/pr-merge/SKILL.md",".claude/skills/pr-merge/references/merge-scenarios.md","src/openharness/skills/bundled/content/commit.md","src/openharness/skills/bundled/content/debug.md","src/openharness/skills/bundled/content/diagnose.md","src/openharness/skills/bundled/content/plan.md","src/openharness/skills/bundled/content/review.md","src/openharness/skills/bundled/content/simplify.md","src/openharness/skills/bundled/content/skill-creator.md","src/openharness/skills/bundled/content/test.md"],"skills":{".claude/skills/harness-eval/SKILL.md":"---\nname: harness-eval\ndescription: This skill should be used when the user asks to \"test the harness\", \"run integration tests\", \"validate features with real API\", \"test with real model calls\", \"run agent loop tests\", \"verify end-to-end\", or needs to verify OpenHarness features on a real codebase with actual LLM calls.\nversion: 0.2.0\n---\n\n# Harness Eval — End-to-End Feature Validation\n\nValidate OpenHarness features by running real agent loops against an unfamiliar codebase with actual LLM API calls. Every test exercises the full stack: API client → model → tool calls → execution → result.\n\n## Core Principles\n\n1. **Test on an unfamiliar project** — never test on OpenHarness itself (the agent modifies its own code). Clone a real project as the workspace.\n2. **Use real API calls** — no mocks. Configure a real LLM endpoint.\n3. **Multi-turn conversations** — always test 2+ turns where the model needs prior context.\n4. **Combine features** — test hooks+skills+agent loop together, not in isolation.\n5. **Verify tool execution** — inspect tool call lists and output files, not just model text.\n\n## Workflow\n\n### 1. Prepare Workspace\n\nClone an unfamiliar project (do not use OpenHarness):\n\n```bash\ngit clone https://github.com/HKUDS/AutoAgent /tmp/eval-workspace\n```\n\n### 2. Configure Environment\n\n```bash\nexport ANTHROPIC_API_KEY=sk-xxx\nexport ANTHROPIC_BASE_URL=https://api.moonshot.cn/anthropic  # or any provider\nexport ANTHROPIC_MODEL=kimi-k2.5\n```\n\nFor long-running real evals, do not artificially lower `max_turns`. Use the product default (`200`) unless the user explicitly wants a tighter bound.\n\n### 3. Prepare Real Sandbox Runtime When Relevant\n\nIf the task is validating sandbox behavior, install and verify the actual runtime before running agent loops:\n\n```bash\nnpm install -g @anthropic-ai/sandbox-runtime\nsudo apt-get update\nsudo apt-get install -y bubblewrap ripgrep\nwhich srt\nwhich bwrap\nwhich rg\nsrt --version\n```\n\nThen run a minimal smoke check through OpenHarness, not just raw `srt`, so you verify the real adapter path:\n\n```python\nfrom pathlib import Path\nfrom openharness.config.settings import Settings, SandboxSettings, save_settings\nfrom openharness.tools.bash_tool import BashTool\n\ncfg = Path(\"/tmp/openharness-sandbox-settings.json\")\nsave_settings(Settings(sandbox=SandboxSettings(enabled=True, fail_if_unavailable=True)), cfg)\n# Point config loader at this file, then run BashTool on a tiny command such as `pwd`.\n```\n\nIf sandbox dependencies are missing, treat that as an environment/setup failure, not a feature regression.\n\n### 4. Design Tests\n\nEach test follows this pattern:\n\n```python\nengine = make_engine(system_prompt=\"...\", cwd=UNFAMILIAR_PROJECT)\nevs1 = [ev async for ev in engine.submit_message(\"Read X, analyze Y\")]\nr1 = collect(evs1)  # text, tools, turns, tokens\nevs2 = [ev async for ev in engine.submit_message(\"Based on what you found...\")]\nr2 = collect(evs2)\nassert \"grep\" in r1[\"tools\"]  # verify tools ran\n```\n\nFor detailed code templates and the `make_engine`/`collect` helpers, consult `references/test-patterns.md`.\n\n### 5. Prefer Long-Horizon, Real Agent Loops\n\nFor meaningful end-to-end validation, prefer unfamiliar-repo tasks that force multiple turns, context reuse, and mixed tool usage.\n\nRecommended pattern:\n\n- Use a real external workspace such as `AutoAgent`\n- Use real provider credentials and the actual target model\n- Keep `max_turns=200`\n- Use per-prompt timeouts large enough for real exploration, such as `240-600s`\n- Require at least 2 turns per scenario\n- Verify both text quality and tool traces\n- Keep polling long-running sessions until they finish; do not abandon a run after the first long pause\n\nRecommended long-horizon scenarios:\n\n- `architecture_multiturn`\n  - Turn 1: map architecture, shell/subprocess surfaces, and test entrypoints\n  - Turn 2: identify top risks and propose refactors\n  - Turn 3: condense into onboarding or remediation actions\n  - Success: `bash`, `glob`, `grep`, `read_file` all appear; no timeout; no `MaxTurnsExceeded`\n\n- `hook_block_and_recover`\n  - Force the model to try `bash`\n  - Block it with a real pre-tool hook\n  - Verify the model adapts with `glob`/`grep`/`read_file`\n\n- `sandbox_multiturn`\n  - Enable real sandbox settings with `fail_if_unavailable=true`\n  - First prompt must start with exactly one shell command such as `pwd && ls -la`\n  - Second prompt must explicitly reuse the prior shell findings\n  - Success: `bash` executes via sandbox, non-shell tools continue the task, and the agent recovers from incidental repo errors\n\nWhen a scenario fails, classify it before changing code:\n\n- `MaxTurnsExceeded`: likely eval harness misconfiguration if `max_turns` was manually lowered\n- `timeout`: task is too broad or per-prompt timeout is too small\n- sandbox unavailable: environment missing `srt`, `bwrap`, or `rg`\n- tool error with task still completed: feature may still be healthy; inspect recovery behavior\n\n### 6. Run Tests\n\n```bash\npython tests/test_merged_prs_on_autoagent.py   # PR feature tests\npython tests/test_real_large_tasks.py           # large multi-step tasks\npython tests/test_hooks_skills_plugins_real.py  # hooks/skills/plugins\npython -m pytest tests/ -q -k \"not autoagent\"  # unit tests (no API)\n```\n\nFor ad hoc long-horizon validation, it is acceptable to run a temporary Python driver script as long as it:\n\n- uses real OpenHarness engine/tool objects\n- targets an unfamiliar repository\n- prints per-scenario JSON summaries\n- records tools, errors, turns, and token usage\n- stays attached until completion\n\n### 7. Interpret Results\n\n| Result | Meaning | Action |\n|--------|---------|--------|\n| PASS with tool calls | Feature works end-to-end | Done |\n| PASS without tool calls | Model answered from knowledge | Rewrite prompt to force tool use |\n| FAIL with exception | Code bug | Read traceback |\n| FAIL with wrong output | Model behavior issue | Check system prompt and tool schemas |\n| Timeout | Task too complex | Increase `max_turns` or simplify prompt |\n\nFor long-running real evals, refine the timeout guidance:\n\n- First check whether `max_turns` was manually set too low\n- If `max_turns=200` and the run still fails, the next suspect is wall-clock timeout, not turn count\n- Distinguish environment failures from product failures\n  - Example: missing dependency in the unfamiliar target repo is not automatically an OpenHarness regression\n  - Example: missing `srt`/`bwrap`/`rg` is an eval environment issue\n\n## Feature Coverage Checklist\n\n- [ ] Engine: multi-turn memory, tool chaining, parallel tools, error recovery, auto-compaction\n- [ ] Swarm: InProcessBackend lifecycle, concurrent teammates, coordinator+notifications\n- [ ] Hooks: pre_tool_use blocking → model adapts, post_tool_use firing\n- [ ] Skills: skill tool invocation → model follows instructions\n- [ ] Plugins: plugin-provided skill loaded and used in agent loop\n- [ ] Memory: YAML frontmatter parsing, body content search, context injection\n- [ ] Session: save → load → resume with context preserved\n- [ ] Providers: Anthropic client, OpenAI client (with reasoning_content), multi-turn\n- [ ] Cost: token accumulation across turns\n\n## Common Pitfalls\n\n- Testing on OpenHarness itself — agent modifies its own running code\n- Using mocks — misses serialization and API compatibility bugs\n- Single-turn only — misses context accumulation and compaction bugs\n- Artificially lowering `max_turns` during real evals — can create false failures that do not reflect product defaults\n- Not checking tool call list — model may claim tool use without calling it\n- Hardcoding paths — use `WORKSPACE` variable, skip in CI with `pytest.mark.skipif`\n- Declaring sandbox “tested” after only checking raw `srt` — verify the OpenHarness adapter path too\n- Abandoning long tasks too early — some real tasks pause for minutes before the next event arrives\n\n## Additional Resources\n\n### Reference Files\n\n- **`references/test-patterns.md`** — Complete code templates for `make_engine`, `collect`, and each feature category\n- **`references/feature-matrix.md`** — Detailed test cases for every OpenHarness module\n\n### Existing Test Files\n\nWorking test suites in the repo:\n- `tests/test_merged_prs_on_autoagent.py` — PR feature validation\n- `tests/test_real_large_tasks.py` — Large multi-step tasks\n- `tests/test_hooks_skills_plugins_real.py` — Hooks/skills/plugins in agent loops\n- `tests/test_untested_features.py` — Module-level integration tests\n",".claude/skills/harness-eval/references/feature-matrix.md":"# Feature Test Matrix — Detailed Test Cases\n\n## Engine & Tools\n\n| Test | What to Verify | Key Assertion |\n|------|---------------|---------------|\n| Multi-turn memory | Set fact turn 1, recall turn 3 | Fact appears in turn 3 response |\n| Tool chaining | glob → grep → read in one task | All 3 tools in tool list |\n| Write→Edit→Read | Create, modify, verify file | File content matches expected |\n| Parallel tools | 3+ tool calls in one response | 3+ tools in single turn |\n| Error recovery | Tool fails, model adapts | Alternative tool used after error |\n| Auto-compaction | 5+ tasks on shared engine | No context overflow crash |\n\n## Swarm & Coordinator\n\n| Test | What to Verify | Key Assertion |\n|------|---------------|---------------|\n| InProcessBackend | spawn → active → status → shutdown | All states transition correctly |\n| Concurrent teammates | 2+ agents running simultaneously | Both complete, total time < 2x single |\n| Coordinator + notifications | Multi-turn delegation with XML | Coordinator synthesizes worker results |\n| Permission sync | request → pending → resolve | Pending count goes to 0 after resolve |\n\n## Hooks, Skills, Plugins\n\n| Test | What to Verify | Key Assertion |\n|------|---------------|---------------|\n| Hook blocks → adapt | pre_tool_use blocks bash | \"bash\" in errors, \"glob\" in tools |\n| Skill invocation | Model calls skill tool | \"skill\" in tools, content drives next action |\n| Plugin skill | Plugin provides skill | Loaded via skill tool, model follows it |\n| Hook + skill combined | Hook gates writes, skill guides | Protected file untouched, new file created |\n\n## Memory, Session, Config\n\n| Test | What to Verify | Key Assertion |\n|------|---------------|---------------|\n| Memory frontmatter | YAML parsed, not \"---\" | description != \"---\", body searchable |\n| Session resume | Save → load → continue | Model remembers prior context |\n| Cost tracking | Tokens accumulate | in_tokens strictly increasing |\n| Cron CRUD | Create, toggle, mark_run, delete | Job count correct at each step |\n\n## Provider Compatibility\n\n| Test | What to Verify | Key Assertion |\n|------|---------------|---------------|\n| Anthropic client | Standard tool calling | Tools execute, response coherent |\n| OpenAI client | Tool calling + reasoning_content | No 400 error on tool call round-trip |\n| OpenAI multi-turn | reasoning_content persists | 3+ turns without API error |\n",".claude/skills/harness-eval/references/test-patterns.md":"# Test Patterns — Code Templates for Harness Eval\n\n## Engine Setup Helpers\n\n```python\nimport asyncio, sys, os\nfrom pathlib import Path\nsys.path.insert(0, \"src\")\n\nAPI_KEY = os.environ.get(\"ANTHROPIC_API_KEY\", \"your-key\")\nBASE_URL = os.environ.get(\"ANTHROPIC_BASE_URL\", \"https://api.moonshot.cn/anthropic\")\nOPENAI_BASE = os.environ.get(\"OPENAI_BASE_URL\", \"https://api.moonshot.cn/v1\")\nMODEL = os.environ.get(\"ANTHROPIC_MODEL\", \"kimi-k2.5\")\nWORKSPACE = Path(\"/tmp/eval-workspace\")  # unfamiliar project\n\n\ndef make_anthropic_engine(system_prompt, cwd=None, extra_tools=None):\n    from openharness.api.client import AnthropicApiClient\n    from openharness.config.settings import PermissionSettings\n    from openharness.engine.query_engine import QueryEngine\n    from openharness.permissions.checker import PermissionChecker\n    from openharness.permissions.modes import PermissionMode\n    from openharness.tools.base import ToolRegistry\n    from openharness.tools.bash_tool import BashTool\n    from openharness.tools.file_read_tool import FileReadTool\n    from openharness.tools.file_write_tool import FileWriteTool\n    from openharness.tools.file_edit_tool import FileEditTool\n    from openharness.tools.glob_tool import GlobTool\n    from openharness.tools.grep_tool import GrepTool\n\n    api = AnthropicApiClient(api_key=API_KEY, base_url=BASE_URL)\n    reg = ToolRegistry()\n    for t in [BashTool(), FileReadTool(), FileWriteTool(), FileEditTool(), GlobTool(), GrepTool()]:\n        reg.register(t)\n    for t in (extra_tools or []):\n        reg.register(t)\n    checker = PermissionChecker(PermissionSettings(mode=PermissionMode.FULL_AUTO))\n    return QueryEngine(\n        api_client=api, tool_registry=reg, permission_checker=checker,\n        cwd=Path(cwd or WORKSPACE), model=MODEL, system_prompt=system_prompt, max_tokens=4096,\n    )\n\n\ndef make_openai_engine(system_prompt, cwd=None, extra_tools=None):\n    from openharness.api.openai_client import OpenAICompatibleClient\n    # Same structure as above, but with:\n    api = OpenAICompatibleClient(api_key=API_KEY, base_url=OPENAI_BASE)\n    # ... rest identical\n\n\ndef collect(events):\n    from openharness.engine.stream_events import (\n        AssistantTextDelta, AssistantTurnComplete,\n        ToolExecutionStarted, ToolExecutionCompleted,\n    )\n    r = {\"text\": \"\", \"tools\": [], \"turns\": 0, \"in_tok\": 0, \"out_tok\": 0}\n    for ev in events:\n        if isinstance(ev, AssistantTextDelta):\n            r[\"text\"] += ev.text\n        elif isinstance(ev, ToolExecutionStarted):\n            r[\"tools\"].append(ev.tool_name)\n        elif isinstance(ev, AssistantTurnComplete):\n            r[\"turns\"] += 1\n            r[\"in_tok\"] += ev.usage.input_tokens\n            r[\"out_tok\"] += ev.usage.output_tokens\n    return r\n```\n\n## Multi-Turn Memory Test\n\n```python\nasync def test_multi_turn_memory():\n    engine = make_anthropic_engine(\"Remember what the user tells you.\")\n    [ev async for ev in engine.submit_message(\"My favorite number is 42.\")]\n    [ev async for ev in engine.submit_message(\"What is 2+2?\")]\n    evs = [ev async for ev in engine.submit_message(\"What is my favorite number?\")]\n    r = collect(evs)\n    assert \"42\" in r[\"text\"]\n```\n\n## Hook Blocks Tool → Model Adapts\n\n```python\nasync def test_hook_blocks():\n    from openharness.hooks.events import HookEvent\n    from openharness.hooks.loader import HookRegistry\n    from openharness.hooks.schemas import CommandHookDefinition\n    from openharness.hooks.executor import HookExecutor, HookExecutionContext\n\n    hook_reg = HookRegistry()\n    hook_reg.register(HookEvent.PRE_TOOL_USE, CommandHookDefinition(\n        type=\"command\", command=\"exit 1\",\n        matcher=\"bash\", block_on_failure=True, timeout_seconds=5,\n    ))\n    # ... create engine with hook_executor, model tries bash, gets blocked, adapts to glob\n```\n\n## Skill Tool Invocation\n\n```python\nasync def test_skill_invocation():\n    from openharness.tools.skill_tool import SkillTool\n    engine = make_anthropic_engine(\n        \"Use the 'skill' tool to load instructions before working.\",\n        extra_tools=[SkillTool()],\n    )\n    evs = [ev async for ev in engine.submit_message(\n        \"Load the 'diagnose' skill, then investigate the codebase.\"\n    )]\n    r = collect(evs)\n    assert \"skill\" in r[\"tools\"]\n```\n\n## InProcess Concurrent Teammates\n\n```python\nasync def test_concurrent_teammates():\n    from openharness.swarm.in_process import start_in_process_teammate, TeammateAbortController\n    from openharness.swarm.types import TeammateSpawnConfig\n    from openharness.engine.query import QueryContext\n\n    async def run_one(name, prompt):\n        ctx = QueryContext(api_client=api, tool_registry=reg, ...)\n        config = TeammateSpawnConfig(name=name, team=\"test\", prompt=prompt, ...)\n        abort = TeammateAbortController()\n        await start_in_process_teammate(config=config, agent_id=f\"{name}@test\", abort_controller=abort, query_context=ctx)\n\n    await asyncio.gather(\n        asyncio.wait_for(run_one(\"worker-a\", \"Count .py files\"), timeout=30),\n        asyncio.wait_for(run_one(\"worker-b\", \"Find main class\"), timeout=30),\n    )\n```\n\n## Session Save → Resume\n\n```python\nasync def test_session_resume():\n    from openharness.services.session_storage import save_session_snapshot, load_session_snapshot\n    from openharness.engine.messages import ConversationMessage\n\n    engine1 = make_anthropic_engine(\"Remember context.\")\n    [ev async for ev in engine1.submit_message(\"Project uses FastAPI + React.\")]\n    save_session_snapshot(cwd=tmpdir, model=MODEL, system_prompt=\"...\", messages=engine1.messages, usage=engine1.total_usage)\n\n    loaded = load_session_snapshot(tmpdir)\n    engine2 = make_anthropic_engine(\"Continue analysis.\")\n    engine2.load_messages([ConversationMessage.model_validate(m) for m in loaded[\"messages\"]])\n    evs = [ev async for ev in engine2.submit_message(\"What tech stack did I mention?\")]\n    assert \"fastapi\" in collect(evs)[\"text\"].lower()\n```\n",".claude/skills/pr-merge/SKILL.md":"---\nname: pr-merge\ndescription: This skill should be used when the user asks to \"merge a PR\", \"review and merge pull requests\", \"integrate external contributions\", \"handle PR conflicts\", \"cherry-pick from a PR\", or needs to merge GitHub PRs while maximizing contributor attribution.\nversion: 0.1.0\n---\n\n# PR Merge — Contributor-First Pull Request Integration\n\nMerge external pull requests while maximizing original author attribution. Core principle: **merge first, resolve conflicts after** — never rewrite a contributor's work from scratch.\n\n## Core Principles\n\n1. **Preserve authorship** — use `gh pr merge --squash` for clean PRs. For manual merges, use `--author=\"Name <email>\"`.\n2. **Merge first, fix after** — accept the PR's approach even if it differs from local style. Fix conflicts in a separate commit.\n3. **Selective merge is OK** — exclude files with `--exclude` when a PR contains features already implemented locally. Document what was excluded.\n4. **Never `git apply` + self-commit** — this loses author attribution entirely.\n\n## Workflow\n\n### 1. Triage Open PRs\n\n```bash\ngh pr list --repo OWNER/REPO --state open \\\n  --json number,title,author,additions,deletions,mergeable\n```\n\nClassify each PR: merge directly, merge with conflict resolution, selective merge, or close.\n\n### 2. Merge Clean PRs via GitHub\n\nPrefer `gh pr merge` — preserves author automatically:\n\n```bash\ngh pr merge NUMBER --repo OWNER/REPO --squash \\\n  --subject \"feat: description (#NUMBER)\"\n```\n\n### 3. Handle Conflicting PRs Locally\n\n```bash\ngit fetch origin pull/NUMBER/head:pr-NUMBER\ngit merge pr-NUMBER --no-edit\n# Resolve conflicts keeping both sides where possible\ngit add -A && git commit --no-edit\n```\n\n### 4. Selective Merge (Skip Some Files)\n\nWhen a PR contains features already implemented locally:\n\n```bash\ngh pr diff NUMBER --repo OWNER/REPO | \\\n  git apply --exclude='path/to/skip.py' --exclude='CHANGELOG.md'\ngit commit --author=\"Author Name <author@email>\" \\\n  -m \"feat: description (#NUMBER)\n\nCherry-picked from PR #NUMBER. Excluded: file.py (already implemented).\"\n```\n\n### 5. Close Duplicate/Superseded PRs\n\n```bash\ngh pr close NUMBER --repo OWNER/REPO \\\n  --comment \"Fixed via PR #OTHER. Thank you for the contribution!\"\n```\n\n### 6. Post-Merge Verification\n\n```bash\npython -m ruff check src/ tests/    # lint\npython -m pytest tests/ -q          # unit tests\ngit push origin main                # push\n```\n\nThen run `harness-eval` for end-to-end verification on an unfamiliar codebase.\n\n## Attribution Checklist\n\nBefore pushing a merged PR:\n\n- [ ] Original author appears in `git log` (via `--author` or GitHub squash merge)\n- [ ] Commit message references PR number (`#NUMBER`)\n- [ ] If selectively merged, commit body explains exclusions\n- [ ] Closed PRs have a comment thanking the contributor\n- [ ] Duplicate PRs acknowledge the contributor's investigation\n\n## Common Pitfalls\n\n- `git apply` + self-commit loses author\n- Rewriting from scratch instead of merging — merge their code, fix style after\n- Force-pushing main after merge — may remove contributor commits\n- Forgetting CHANGELOG conflicts — always exclude and handle manually\n- Not testing after merge — clean merge doesn't mean working code\n\n## Additional Resources\n\n### Reference Files\n\n- **`references/merge-scenarios.md`** — Detailed examples for each merge scenario (clean, conflicting, selective, duplicate)\n",".claude/skills/pr-merge/references/merge-scenarios.md":"# Merge Scenarios — Detailed Examples\n\n## Scenario 1: Clean Merge (No Conflicts)\n\nPR adds a new feature, no files overlap with local changes.\n\n```bash\n# Preferred: squash merge via GitHub (preserves author)\ngh pr merge 17 --repo HKUDS/OpenHarness --squash \\\n  --subject \"feat(skills): add diagnose skill (#17)\"\n\n# Result: author \"Qu Zhi\" appears in git log\n```\n\n## Scenario 2: CHANGELOG Conflict Only\n\nAlmost every PR touches CHANGELOG.md. Handle by excluding it:\n\n```bash\ngh pr diff 14 --repo OWNER/REPO | \\\n  git apply --exclude='CHANGELOG.md'\ngit add -A\n\n# Manually merge CHANGELOG sections (keep both)\n# Then commit with original author\ngit commit --author=\"washi4 <washi4@users.noreply.github.com>\" \\\n  -m \"feat: add OpenAI-compatible client (#14)\"\n```\n\n## Scenario 3: Conflicting UI Files\n\nPR #14 modified ui/app.py and ui/backend_host.py which were also changed by PR #13.\n\n```bash\n# Apply excluding conflicting files\ngh pr diff 14 --repo OWNER/REPO | \\\n  git apply --exclude='src/ui/app.py' --exclude='src/ui/backend_host.py' --exclude='CHANGELOG.md'\n\n# Manually add the PR's changes to conflicting files\n# Read the PR diff to understand what they added, apply by hand\n# Commit with author\ngit commit --author=\"Author <email>\" -m \"feat: description (#14)\"\n```\n\n## Scenario 4: Selective Merge (Skip Features)\n\nPR #16 has auto-compact (already implemented locally) + --resume (wanted) + cron (wanted).\n\n```bash\n# Exclude the file containing the unwanted feature\ngh pr diff 16 --repo OWNER/REPO | git apply \\\n  --exclude='src/engine/query_engine.py' \\  # skip auto-compact\n  --exclude='CHANGELOG.md' \\\n  --exclude='README.md'\n\n# Commit with explanation\ngit commit --author=\"Chao Qin <win4r@users.noreply.github.com>\" \\\n  -m \"feat: wire --resume/--continue and cron scheduler (#16)\n\nCherry-picked from PR #16. Excluded auto-compact (already implemented\nwith LLM-based approach from reference source).\"\n```\n\n## Scenario 5: Duplicate PRs (Same Bug, Different Fix)\n\nPR #11 and #13 both fix the double-Enter bug. #13 is smaller and cleaner.\n\n```bash\n# Merge #13 (the better fix)\ngh pr merge 13 --repo OWNER/REPO --squash\n\n# Close #11 with acknowledgment\ngh pr close 11 --repo OWNER/REPO --comment \\\n  \"Fixed via PR #13 (smaller patch for the same bug). \\\nThank you for the detailed investigation!\"\n```\n\n## Scenario 6: PR From a Fork\n\nThe contributor pushed to their fork, not a branch on the repo.\n\n```bash\n# Fetch via PR ref (works for any PR regardless of source)\ngit fetch origin pull/14/head:pr-14\ngit merge pr-14 --no-edit\n\n# If merge conflict:\ngit mergetool  # or manually resolve\ngit add -A && git commit --no-edit\n\ngit push origin main\n```\n\n## Scenario 7: Batch Merge (Multiple PRs)\n\nMerge in dependency order, test after each:\n\n```bash\n# 1. Small fixes first (least risk)\ngh pr merge 17 --squash  # skill file\ngh pr merge 13 --squash  # 1-file bug fix\n\n# 2. Medium changes\ngh pr merge 12 --squash  # memory improvement\n\n# 3. Large features last (most conflict risk)\n# PR #14 needs manual conflict resolution\ngit fetch origin pull/14/head:pr-14\ngit merge pr-14\n# resolve CHANGELOG conflict\ngit push origin main\n\n# 4. Test after all merges\npython -m ruff check src/ tests/\npython -m pytest tests/ -q\n```\n\n## Post-Merge Fix Commit Pattern\n\nWhen the merged PR introduces a compatibility issue (e.g., API mismatch with a provider):\n\n```bash\n# Fix in a SEPARATE commit (don't amend the author's commit)\ngit commit -m \"fix(api): handle Kimi reasoning_content in OpenAI client\n\nKimi k2.5 requires reasoning_content on assistant tool_call messages.\nFix: capture during streaming, replay when converting back.\nFound during post-merge testing with harness-eval.\"\n```\n\nThis preserves the original author's commit intact while documenting the fix separately.\n","src/openharness/skills/bundled/content/commit.md":"# commit\n\nCreate clean, well-structured git commits.\n\n## When to use\n\nUse when the user asks to commit changes, create a PR, or prepare code for review.\n\n## Workflow\n\n1. Run `git status` and `git diff` to understand all changes\n2. Analyze changes: categorize as feature, fix, refactor, docs, test, etc.\n3. Draft a concise commit message:\n   - First line: imperative mood, under 72 chars, describes the \"why\"\n   - Body (if needed): explain context, trade-offs, or breaking changes\n4. Stage only relevant files — never stage .env, credentials, or large binaries\n5. Create the commit\n\n## Rules\n\n- Prefer specific `git add <file>` over `git add -A`\n- Never use `--no-verify` unless explicitly asked\n- Never amend published commits unless explicitly asked\n- If a pre-commit hook fails, fix the issue and create a NEW commit (don't --amend)\n- Include `Co-Authored-By` if pair programming\n","src/openharness/skills/bundled/content/debug.md":"# debug\n\nDiagnose and fix bugs systematically.\n\n## When to use\n\nUse when the user reports a bug, error, or unexpected behavior.\n\n## Workflow\n\n1. Reproduce: understand the exact steps that trigger the issue\n2. Read the error: stack traces, log messages, error codes\n3. Locate: use Grep/Read to find the relevant code path\n4. Hypothesize: form a theory about the root cause\n5. Verify: add logging or read surrounding code to confirm\n6. Fix: make the minimal change that addresses the root cause\n7. Test: verify the fix works and doesn't break other things\n\n## Rules\n\n- Read the error message carefully before searching code\n- Don't guess — verify your hypothesis before changing code\n- Fix the root cause, not the symptom\n- Don't retry the same approach if it failed — investigate why\n- If stuck after 3 attempts, explain what you've tried and ask for help\n","src/openharness/skills/bundled/content/diagnose.md":"# diagnose\n\nDiagnose why an agent run failed, regressed, or produced unexpected output — using structured evidence instead of intuition.\n\n## When to use\n\nUse when the user asks:\n- \"Why did this run fail?\"\n- \"What changed between this run and the last one?\"\n- \"The output looks wrong — what happened?\"\n- \"Why is this run worse than before?\"\n\n## Workflow\n\n1. **Locate the run artifacts** — check `artifacts/runs/<run_id>/` or the latest run directory for:\n   - `manifest.json` — run metadata, task type, input hash, timestamps\n   - `execution_trace.jsonl` — the full tool-call chain, one event per line\n   - `verification_report.json` — what was verified and whether it passed\n   - `failure_signature.json` — which stage failed and why (if present)\n2. **Read the failure signature first** — if it exists, it already localizes the failure to a specific stage (routing, execution, verification, or governance)\n3. **Trace the execution chain** — walk `execution_trace.jsonl` forward to find where output diverged from expectation\n4. **Compare with a passing run** — if a previous run succeeded, diff the two manifests and traces to identify what changed\n5. **Identify the failure layer**:\n   - **Routing**: wrong task type or profile selected\n   - **Execution**: tool call error, permission block, or unexpected result\n   - **Verification**: output produced but did not match expected artifact\n   - **Governance**: run was halted by a safety or boundary check\n6. **Report with evidence** — cite specific file paths, line numbers in the trace, or field values from the manifest; never summarize without pointing to the source\n\n## Rules\n\n- Evidence before intuition: always read the trace before forming a hypothesis\n- Localize before explaining: identify which stage failed before describing why\n- Compare, don't guess: if a previous run succeeded, diff it — don't speculate about what changed\n- If no artifacts exist, say so clearly and ask the user to re-run with archive mode enabled\n","src/openharness/skills/bundled/content/plan.md":"# plan\n\nDesign an implementation plan before coding.\n\n## When to use\n\nUse when the user asks to plan, design, or architect a feature before implementing it.\n\n## Workflow\n\n1. Understand the requirement:\n   - What problem does this solve?\n   - What are the constraints?\n   - What's the expected outcome?\n2. Explore the codebase to find:\n   - Existing functions/utilities that can be reused\n   - Patterns used elsewhere in the project\n   - Files that will need modification\n3. Design the approach:\n   - Break into discrete steps\n   - Identify dependencies between steps\n   - Consider edge cases and error handling\n4. Present the plan:\n   - Start with a Context section (why this change)\n   - List concrete steps with file paths\n   - Include a verification section (how to test)\n\n## Rules\n\n- Read before suggesting — never propose changes to code you haven't read\n- Prefer editing existing files over creating new ones\n- Reuse existing patterns and utilities\n- Include file paths and line numbers when referencing code\n- Don't over-plan: match complexity to the task\n","src/openharness/skills/bundled/content/review.md":"# review\n\nReview code for bugs, security issues, and quality.\n\n## When to use\n\nUse when the user asks to review code, a PR, or a diff.\n\n## Workflow\n\n1. Read the changed files or diff thoroughly\n2. Check for:\n   - **Bugs**: logic errors, off-by-one, null/undefined access, race conditions\n   - **Security**: injection, XSS, hardcoded secrets, path traversal\n   - **Performance**: N+1 queries, unnecessary allocations, missing indexes\n   - **Tests**: are new code paths covered? Are edge cases tested?\n   - **Style**: naming consistency, dead code, unnecessary complexity\n3. Provide concrete, actionable feedback with file:line references\n4. Prioritize findings by severity (critical > major > minor > nit)\n\n## Rules\n\n- Be specific: \"line 42 may throw if `user` is null\" not \"check for null\"\n- Suggest fixes, don't just point out problems\n- Acknowledge good patterns too\n- Don't nitpick formatting if there's a linter\n","src/openharness/skills/bundled/content/simplify.md":"# simplify\n\nRefactor code to be simpler and more maintainable.\n\n## When to use\n\nUse when the user asks to simplify, clean up, or refactor code.\n\n## Workflow\n\n1. Read the target code completely\n2. Identify complexity sources:\n   - Unnecessary abstractions or indirection\n   - Duplicated logic that should be consolidated\n   - Over-engineered patterns (factories, builders, strategies for simple cases)\n   - Dead code paths or unused variables\n3. Propose changes that reduce line count while preserving behavior\n4. Verify the refactored code passes existing tests\n\n## Rules\n\n- Don't add features or change behavior — simplification only\n- Prefer deleting code over refactoring it\n- Three similar lines are better than a premature abstraction\n- Remove backwards-compatibility shims for removed features\n- Don't add comments to explain simple code\n","src/openharness/skills/bundled/content/skill-creator.md":"---\nname: skill-creator\ndescription: >\n  Create, improve, and verify OpenHarness skills. Use this whenever the user\n  asks to create a new skill, convert a workflow into a skill, update an\n  existing SKILL.md, add skills for oh/ohmo, design skill trigger behavior,\n  or test whether a skill loads and works correctly.\n---\n\n# skill-creator\n\nCreate or improve OpenHarness skills in the directory-based `SKILL.md` format.\n\n## When to use\n\nUse this skill when the user wants to:\n\n- create a new skill from a workflow, repeated task, domain guide, or tool process\n- convert existing notes, prompts, or instructions into a reusable skill\n- modify or harden an existing skill\n- make a skill available to `oh`, `ohmo`, or a plugin\n- debug why the `skill` tool cannot find or load a skill\n- test trigger wording, skill metadata, or skill behavior\n\n## OpenHarness skill locations\n\nChoose the target deliberately:\n\n- Built-in OpenHarness skills live in `src/openharness/skills/bundled/content/*.md`.\n- User skills for `oh` live in `~/.openharness/skills/<skill-dir>/SKILL.md`.\n- Private `ohmo` skills live in `~/.ohmo/skills/<skill-dir>/SKILL.md`.\n- Plugin skills live in `<plugin-root>/skills/<skill-dir>/SKILL.md`.\n\nOpenHarness user and plugin skills use a directory layout. Do not create flat\n`*.md` user skills under `skills/`; they will not be loaded by the normal loader.\n\n## Skill anatomy\n\nUse this structure for user, ohmo, and plugin skills:\n\n```text\nmy-skill/\n├── SKILL.md\n├── scripts/      # optional deterministic helpers\n├── references/   # optional long docs loaded only when relevant\n└── assets/       # optional templates or static files\n```\n\nUse this structure inside `SKILL.md`:\n\n```markdown\n---\nname: my-skill\ndescription: >\n  What the skill does and exactly when to use it.\n---\n\n# my-skill\n\nShort purpose statement.\n\n## When to use\n\nTrigger contexts in plain language.\n\n## Workflow\n\nConcrete steps the agent should follow.\n\n## Rules\n\nImportant boundaries, verification, and failure handling.\n```\n\n## Workflow\n\n1. Capture intent before writing.\n   Ask or infer what the skill should help with, when it should trigger, what\n   output it should produce, and what success looks like. If the conversation\n   already contains the workflow, extract the sequence from history before\n   asking more questions.\n\n2. Inspect existing examples.\n   Read nearby skills and loader behavior before choosing a layout. For\n   OpenHarness, confirm whether the target is bundled, user, ohmo-private, or\n   plugin-provided.\n\n3. Write the metadata first.\n   The `description` is the primary trigger hint shown in the available skills\n   list. Make it specific and slightly assertive: include both what the skill\n   does and the phrases or situations that should trigger it. Avoid vague\n   descriptions like \"Helpful notes for X\".\n\n4. Keep the body procedural.\n   Use concise imperative steps. Explain why non-obvious constraints matter.\n   Prefer general principles over overfitted examples. If a skill is getting\n   long, move details into `references/` and point to the relevant file.\n\n5. Add deterministic resources only when they pay for themselves.\n   If the skill repeatedly needs the same script, template, schema, or checklist,\n   place it under `scripts/`, `assets/`, or `references/` rather than making the\n   model regenerate it every time.\n\n6. Verify loading.\n   Run a local loader check for the chosen target. For bundled skills, use:\n\n   ```bash\n   PYTHONPATH=src python - <<'PY'\n   from openharness.skills import load_skill_registry\n   for skill in load_skill_registry().list_skills():\n       print(skill.name, skill.source, skill.path)\n   PY\n   ```\n\n   For ohmo-private skills, include the extra skill directory:\n\n   ```bash\n   PYTHONPATH=src python - <<'PY'\n   from openharness.skills import load_skill_registry\n   from ohmo.workspace import get_skills_dir\n   for skill in load_skill_registry(extra_skill_dirs=[get_skills_dir()]).list_skills():\n       print(skill.name, skill.source, skill.path)\n   PY\n   ```\n\n7. Test behavior.\n   Create two or three realistic prompts that should use the skill. If the skill\n   changes code or files, add regression tests around loading or command\n   behavior. Run targeted tests first, then broader tests if the loader or\n   runtime path changed.\n\n8. Iterate from evidence.\n   If a skill under-triggers, improve the frontmatter description. If it\n   triggers but produces poor work, improve the body workflow. If agents repeat\n   deterministic work, add a helper script or reference file.\n\n## Writing rules\n\n- Preserve the skill name when updating an existing skill unless the user\n  explicitly asks to rename it.\n- Do not overwrite user-created skills without reading the existing `SKILL.md`.\n- Keep `SKILL.md` under roughly 500 lines; use `references/` for long material.\n- Keep trigger guidance in frontmatter `description`, not buried only in the\n  body.\n- Make skills safe and unsurprising. Do not create skills that hide behavior,\n  bypass permissions, exfiltrate data, or encourage unauthorized access.\n- Prefer directory names that are lowercase kebab-case, even if the displayed\n  skill `name` is more human-readable.\n- Mention exact install or verification paths in the final response.\n\n## Test prompts\n\nAfter drafting a skill, propose a small eval set:\n\n```json\n{\n  \"skill_name\": \"my-skill\",\n  \"evals\": [\n    {\n      \"id\": \"happy-path\",\n      \"prompt\": \"A realistic user request that should trigger the skill.\",\n      \"expected_output\": \"What good behavior looks like.\"\n    },\n    {\n      \"id\": \"near-miss\",\n      \"prompt\": \"A similar request that should not trigger this skill.\",\n      \"expected_output\": \"The agent should choose a different path.\"\n    }\n  ]\n}\n```\n\nFor objective skills, add assertions or tests. For subjective skills, collect\nhuman feedback and revise the workflow from concrete complaints rather than\nadding rigid rules.\n","src/openharness/skills/bundled/content/test.md":"# test\n\nWrite and run tests for code.\n\n## When to use\n\nUse when the user asks to write tests, verify behavior, or improve test coverage.\n\n## Workflow\n\n1. Understand what needs testing:\n   - New feature? Write unit tests for the happy path and edge cases\n   - Bug fix? Write a regression test that would have caught the bug\n   - Refactor? Ensure existing tests still pass\n2. Follow the project's testing patterns:\n   - Check existing tests for framework (pytest, jest, go test, etc.)\n   - Match naming conventions and file organization\n   - Use the same fixtures and helpers\n3. Write tests that are:\n   - Independent: each test can run alone\n   - Deterministic: same result every time\n   - Fast: mock external services, use in-memory databases\n4. Run the tests and verify they pass\n\n## Rules\n\n- Test behavior, not implementation details\n- One assertion per test when possible\n- Use descriptive test names that explain the scenario\n- Don't test framework or library code\n- Mock at system boundaries (external APIs, filesystem, network)\n"},"files":{".claude/skills/harness-eval/SKILL.md":"---\nname: harness-eval\ndescription: This skill should be used when the user asks to \"test the harness\", \"run integration tests\", \"validate features with real API\", \"test with real model calls\", \"run agent loop tests\", \"verify end-to-end\", or needs to verify OpenHarness features on a real codebase with actual LLM calls.\nversion: 0.2.0\n---\n\n# Harness Eval — End-to-End Feature Validation\n\nValidate OpenHarness features by running real agent loops against an unfamiliar codebase with actual LLM API calls. Every test exercises the full stack: API client → model → tool calls → execution → result.\n\n## Core Principles\n\n1. **Test on an unfamiliar project** — never test on OpenHarness itself (the agent modifies its own code). Clone a real project as the workspace.\n2. **Use real API calls** — no mocks. Configure a real LLM endpoint.\n3. **Multi-turn conversations** — always test 2+ turns where the model needs prior context.\n4. **Combine features** — test hooks+skills+agent loop together, not in isolation.\n5. **Verify tool execution** — inspect tool call lists and output files, not just model text.\n\n## Workflow\n\n### 1. Prepare Workspace\n\nClone an unfamiliar project (do not use OpenHarness):\n\n```bash\ngit clone https://github.com/HKUDS/AutoAgent /tmp/eval-workspace\n```\n\n### 2. Configure Environment\n\n```bash\nexport ANTHROPIC_API_KEY=sk-xxx\nexport ANTHROPIC_BASE_URL=https://api.moonshot.cn/anthropic  # or any provider\nexport ANTHROPIC_MODEL=kimi-k2.5\n```\n\nFor long-running real evals, do not artificially lower `max_turns`. Use the product default (`200`) unless the user explicitly wants a tighter bound.\n\n### 3. Prepare Real Sandbox Runtime When Relevant\n\nIf the task is validating sandbox behavior, install and verify the actual runtime before running agent loops:\n\n```bash\nnpm install -g @anthropic-ai/sandbox-runtime\nsudo apt-get update\nsudo apt-get install -y bubblewrap ripgrep\nwhich srt\nwhich bwrap\nwhich rg\nsrt --version\n```\n\nThen run a minimal smoke check through OpenHarness, not just raw `srt`, so you verify the real adapter path:\n\n```python\nfrom pathlib import Path\nfrom openharness.config.settings import Settings, SandboxSettings, save_settings\nfrom openharness.tools.bash_tool import BashTool\n\ncfg = Path(\"/tmp/openharness-sandbox-settings.json\")\nsave_settings(Settings(sandbox=SandboxSettings(enabled=True, fail_if_unavailable=True)), cfg)\n# Point config loader at this file, then run BashTool on a tiny command such as `pwd`.\n```\n\nIf sandbox dependencies are missing, treat that as an environment/setup failure, not a feature regression.\n\n### 4. Design Tests\n\nEach test follows this pattern:\n\n```python\nengine = make_engine(system_prompt=\"...\", cwd=UNFAMILIAR_PROJECT)\nevs1 = [ev async for ev in engine.submit_message(\"Read X, analyze Y\")]\nr1 = collect(evs1)  # text, tools, turns, tokens\nevs2 = [ev async for ev in engine.submit_message(\"Based on what you found...\")]\nr2 = collect(evs2)\nassert \"grep\" in r1[\"tools\"]  # verify tools ran\n```\n\nFor detailed code templates and the `make_engine`/`collect` helpers, consult `references/test-patterns.md`.\n\n### 5. Prefer Long-Horizon, Real Agent Loops\n\nFor meaningful end-to-end validation, prefer unfamiliar-repo tasks that force multiple turns, context reuse, and mixed tool usage.\n\nRecommended pattern:\n\n- Use a real external workspace such as `AutoAgent`\n- Use real provider credentials and the actual target model\n- Keep `max_turns=200`\n- Use per-prompt timeouts large enough for real exploration, such as `240-600s`\n- Require at least 2 turns per scenario\n- Verify both text quality and tool traces\n- Keep polling long-running sessions until they finish; do not abandon a run after the first long pause\n\nRecommended long-horizon scenarios:\n\n- `architecture_multiturn`\n  - Turn 1: map architecture, shell/subprocess surfaces, and test entrypoints\n  - Turn 2: identify top risks and propose refactors\n  - Turn 3: condense into onboarding or remediation actions\n  - Success: `bash`, `glob`, `grep`, `read_file` all appear; no timeout; no `MaxTurnsExceeded`\n\n- `hook_block_and_recover`\n  - Force the model to try `bash`\n  - Block it with a real pre-tool hook\n  - Verify the model adapts with `glob`/`grep`/`read_file`\n\n- `sandbox_multiturn`\n  - Enable real sandbox settings with `fail_if_unavailable=true`\n  - First prompt must start with exactly one shell command such as `pwd && ls -la`\n  - Second prompt must explicitly reuse the prior shell findings\n  - Success: `bash` executes via sandbox, non-shell tools continue the task, and the agent recovers from incidental repo errors\n\nWhen a scenario fails, classify it before changing code:\n\n- `MaxTurnsExceeded`: likely eval harness misconfiguration if `max_turns` was manually lowered\n- `timeout`: task is too broad or per-prompt timeout is too small\n- sandbox unavailable: environment missing `srt`, `bwrap`, or `rg`\n- tool error with task still completed: feature may still be healthy; inspect recovery behavior\n\n### 6. Run Tests\n\n```bash\npython tests/test_merged_prs_on_autoagent.py   # PR feature tests\npython tests/test_real_large_tasks.py           # large multi-step tasks\npython tests/test_hooks_skills_plugins_real.py  # hooks/skills/plugins\npython -m pytest tests/ -q -k \"not autoagent\"  # unit tests (no API)\n```\n\nFor ad hoc long-horizon validation, it is acceptable to run a temporary Python driver script as long as it:\n\n- uses real OpenHarness engine/tool objects\n- targets an unfamiliar repository\n- prints per-scenario JSON summaries\n- records tools, errors, turns, and token usage\n- stays attached until completion\n\n### 7. Interpret Results\n\n| Result | Meaning | Action |\n|--------|---------|--------|\n| PASS with tool calls | Feature works end-to-end | Done |\n| PASS without tool calls | Model answered from knowledge | Rewrite prompt to force tool use |\n| FAIL with exception | Code bug | Read traceback |\n| FAIL with wrong output | Model behavior issue | Check system prompt and tool schemas |\n| Timeout | Task too complex | Increase `max_turns` or simplify prompt |\n\nFor long-running real evals, refine the timeout guidance:\n\n- First check whether `max_turns` was manually set too low\n- If `max_turns=200` and the run still fails, the next suspect is wall-clock timeout, not turn count\n- Distinguish environment failures from product failures\n  - Example: missing dependency in the unfamiliar target repo is not automatically an OpenHarness regression\n  - Example: missing `srt`/`bwrap`/`rg` is an eval environment issue\n\n## Feature Coverage Checklist\n\n- [ ] Engine: multi-turn memory, tool chaining, parallel tools, error recovery, auto-compaction\n- [ ] Swarm: InProcessBackend lifecycle, concurrent teammates, coordinator+notifications\n- [ ] Hooks: pre_tool_use blocking → model adapts, post_tool_use firing\n- [ ] Skills: skill tool invocation → model follows instructions\n- [ ] Plugins: plugin-provided skill loaded and used in agent loop\n- [ ] Memory: YAML frontmatter parsing, body content search, context injection\n- [ ] Session: save → load → resume with context preserved\n- [ ] Providers: Anthropic client, OpenAI client (with reasoning_content), multi-turn\n- [ ] Cost: token accumulation across turns\n\n## Common Pitfalls\n\n- Testing on OpenHarness itself — agent modifies its own running code\n- Using mocks — misses serialization and API compatibility bugs\n- Single-turn only — misses context accumulation and compaction bugs\n- Artificially lowering `max_turns` during real evals — can create false failures that do not reflect product defaults\n- Not checking tool call list — model may claim tool use without calling it\n- Hardcoding paths — use `WORKSPACE` variable, skip in CI with `pytest.mark.skipif`\n- Declaring sandbox “tested” after only checking raw `srt` — verify the OpenHarness adapter path too\n- Abandoning long tasks too early — some real tasks pause for minutes before the next event arrives\n\n## Additional Resources\n\n### Reference Files\n\n- **`references/test-patterns.md`** — Complete code templates for `make_engine`, `collect`, and each feature category\n- **`references/feature-matrix.md`** — Detailed test cases for every OpenHarness module\n\n### Existing Test Files\n\nWorking test suites in the repo:\n- `tests/test_merged_prs_on_autoagent.py` — PR feature validation\n- `tests/test_real_large_tasks.py` — Large multi-step tasks\n- `tests/test_hooks_skills_plugins_real.py` — Hooks/skills/plugins in agent loops\n- `tests/test_untested_features.py` — Module-level integration tests\n",".claude/skills/harness-eval/references/feature-matrix.md":"# Feature Test Matrix — Detailed Test Cases\n\n## Engine & Tools\n\n| Test | What to Verify | Key Assertion |\n|------|---------------|---------------|\n| Multi-turn memory | Set fact turn 1, recall turn 3 | Fact appears in turn 3 response |\n| Tool chaining | glob → grep → read in one task | All 3 tools in tool list |\n| Write→Edit→Read | Create, modify, verify file | File content matches expected |\n| Parallel tools | 3+ tool calls in one response | 3+ tools in single turn |\n| Error recovery | Tool fails, model adapts | Alternative tool used after error |\n| Auto-compaction | 5+ tasks on shared engine | No context overflow crash |\n\n## Swarm & Coordinator\n\n| Test | What to Verify | Key Assertion |\n|------|---------------|---------------|\n| InProcessBackend | spawn → active → status → shutdown | All states transition correctly |\n| Concurrent teammates | 2+ agents running simultaneously | Both complete, total time < 2x single |\n| Coordinator + notifications | Multi-turn delegation with XML | Coordinator synthesizes worker results |\n| Permission sync | request → pending → resolve | Pending count goes to 0 after resolve |\n\n## Hooks, Skills, Plugins\n\n| Test | What to Verify | Key Assertion |\n|------|---------------|---------------|\n| Hook blocks → adapt | pre_tool_use blocks bash | \"bash\" in errors, \"glob\" in tools |\n| Skill invocation | Model calls skill tool | \"skill\" in tools, content drives next action |\n| Plugin skill | Plugin provides skill | Loaded via skill tool, model follows it |\n| Hook + skill combined | Hook gates writes, skill guides | Protected file untouched, new file created |\n\n## Memory, Session, Config\n\n| Test | What to Verify | Key Assertion |\n|------|---------------|---------------|\n| Memory frontmatter | YAML parsed, not \"---\" | description != \"---\", body searchable |\n| Session resume | Save → load → continue | Model remembers prior context |\n| Cost tracking | Tokens accumulate | in_tokens strictly increasing |\n| Cron CRUD | Create, toggle, mark_run, delete | Job count correct at each step |\n\n## Provider Compatibility\n\n| Test | What to Verify | Key Assertion |\n|------|---------------|---------------|\n| Anthropic client | Standard tool calling | Tools execute, response coherent |\n| OpenAI client | Tool calling + reasoning_content | No 400 error on tool call round-trip |\n| OpenAI multi-turn | reasoning_content persists | 3+ turns without API error |\n",".claude/skills/harness-eval/references/test-patterns.md":"# Test Patterns — Code Templates for Harness Eval\n\n## Engine Setup Helpers\n\n```python\nimport asyncio, sys, os\nfrom pathlib import Path\nsys.path.insert(0, \"src\")\n\nAPI_KEY = os.environ.get(\"ANTHROPIC_API_KEY\", \"your-key\")\nBASE_URL = os.environ.get(\"ANTHROPIC_BASE_URL\", \"https://api.moonshot.cn/anthropic\")\nOPENAI_BASE = os.environ.get(\"OPENAI_BASE_URL\", \"https://api.moonshot.cn/v1\")\nMODEL = os.environ.get(\"ANTHROPIC_MODEL\", \"kimi-k2.5\")\nWORKSPACE = Path(\"/tmp/eval-workspace\")  # unfamiliar project\n\n\ndef make_anthropic_engine(system_prompt, cwd=None, extra_tools=None):\n    from openharness.api.client import AnthropicApiClient\n    from openharness.config.settings import PermissionSettings\n    from openharness.engine.query_engine import QueryEngine\n    from openharness.permissions.checker import PermissionChecker\n    from openharness.permissions.modes import PermissionMode\n    from openharness.tools.base import ToolRegistry\n    from openharness.tools.bash_tool import BashTool\n    from openharness.tools.file_read_tool import FileReadTool\n    from openharness.tools.file_write_tool import FileWriteTool\n    from openharness.tools.file_edit_tool import FileEditTool\n    from openharness.tools.glob_tool import GlobTool\n    from openharness.tools.grep_tool import GrepTool\n\n    api = AnthropicApiClient(api_key=API_KEY, base_url=BASE_URL)\n    reg = ToolRegistry()\n    for t in [BashTool(), FileReadTool(), FileWriteTool(), FileEditTool(), GlobTool(), GrepTool()]:\n        reg.register(t)\n    for t in (extra_tools or []):\n        reg.register(t)\n    checker = PermissionChecker(PermissionSettings(mode=PermissionMode.FULL_AUTO))\n    return QueryEngine(\n        api_client=api, tool_registry=reg, permission_checker=checker,\n        cwd=Path(cwd or WORKSPACE), model=MODEL, system_prompt=system_prompt, max_tokens=4096,\n    )\n\n\ndef make_openai_engine(system_prompt, cwd=None, extra_tools=None):\n    from openharness.api.openai_client import OpenAICompatibleClient\n    # Same structure as above, but with:\n    api = OpenAICompatibleClient(api_key=API_KEY, base_url=OPENAI_BASE)\n    # ... rest identical\n\n\ndef collect(events):\n    from openharness.engine.stream_events import (\n        AssistantTextDelta, AssistantTurnComplete,\n        ToolExecutionStarted, ToolExecutionCompleted,\n    )\n    r = {\"text\": \"\", \"tools\": [], \"turns\": 0, \"in_tok\": 0, \"out_tok\": 0}\n    for ev in events:\n        if isinstance(ev, AssistantTextDelta):\n            r[\"text\"] += ev.text\n        elif isinstance(ev, ToolExecutionStarted):\n            r[\"tools\"].append(ev.tool_name)\n        elif isinstance(ev, AssistantTurnComplete):\n            r[\"turns\"] += 1\n            r[\"in_tok\"] += ev.usage.input_tokens\n            r[\"out_tok\"] += ev.usage.output_tokens\n    return r\n```\n\n## Multi-Turn Memory Test\n\n```python\nasync def test_multi_turn_memory():\n    engine = make_anthropic_engine(\"Remember what the user tells you.\")\n    [ev async for ev in engine.submit_message(\"My favorite number is 42.\")]\n    [ev async for ev in engine.submit_message(\"What is 2+2?\")]\n    evs = [ev async for ev in engine.submit_message(\"What is my favorite number?\")]\n    r = collect(evs)\n    assert \"42\" in r[\"text\"]\n```\n\n## Hook Blocks Tool → Model Adapts\n\n```python\nasync def test_hook_blocks():\n    from openharness.hooks.events import HookEvent\n    from openharness.hooks.loader import HookRegistry\n    from openharness.hooks.schemas import CommandHookDefinition\n    from openharness.hooks.executor import HookExecutor, HookExecutionContext\n\n    hook_reg = HookRegistry()\n    hook_reg.register(HookEvent.PRE_TOOL_USE, CommandHookDefinition(\n        type=\"command\", command=\"exit 1\",\n        matcher=\"bash\", block_on_failure=True, timeout_seconds=5,\n    ))\n    # ... create engine with hook_executor, model tries bash, gets blocked, adapts to glob\n```\n\n## Skill Tool Invocation\n\n```python\nasync def test_skill_invocation():\n    from openharness.tools.skill_tool import SkillTool\n    engine = make_anthropic_engine(\n        \"Use the 'skill' tool to load instructions before working.\",\n        extra_tools=[SkillTool()],\n    )\n    evs = [ev async for ev in engine.submit_message(\n        \"Load the 'diagnose' skill, then investigate the codebase.\"\n    )]\n    r = collect(evs)\n    assert \"skill\" in r[\"tools\"]\n```\n\n## InProcess Concurrent Teammates\n\n```python\nasync def test_concurrent_teammates():\n    from openharness.swarm.in_process import start_in_process_teammate, TeammateAbortController\n    from openharness.swarm.types import TeammateSpawnConfig\n    from openharness.engine.query import QueryContext\n\n    async def run_one(name, prompt):\n        ctx = QueryContext(api_client=api, tool_registry=reg, ...)\n        config = TeammateSpawnConfig(name=name, team=\"test\", prompt=prompt, ...)\n        abort = TeammateAbortController()\n        await start_in_process_teammate(config=config, agent_id=f\"{name}@test\", abort_controller=abort, query_context=ctx)\n\n    await asyncio.gather(\n        asyncio.wait_for(run_one(\"worker-a\", \"Count .py files\"), timeout=30),\n        asyncio.wait_for(run_one(\"worker-b\", \"Find main class\"), timeout=30),\n    )\n```\n\n## Session Save → Resume\n\n```python\nasync def test_session_resume():\n    from openharness.services.session_storage import save_session_snapshot, load_session_snapshot\n    from openharness.engine.messages import ConversationMessage\n\n    engine1 = make_anthropic_engine(\"Remember context.\")\n    [ev async for ev in engine1.submit_message(\"Project uses FastAPI + React.\")]\n    save_session_snapshot(cwd=tmpdir, model=MODEL, system_prompt=\"...\", messages=engine1.messages, usage=engine1.total_usage)\n\n    loaded = load_session_snapshot(tmpdir)\n    engine2 = make_anthropic_engine(\"Continue analysis.\")\n    engine2.load_messages([ConversationMessage.model_validate(m) for m in loaded[\"messages\"]])\n    evs = [ev async for ev in engine2.submit_message(\"What tech stack did I mention?\")]\n    assert \"fastapi\" in collect(evs)[\"text\"].lower()\n```\n",".claude/skills/pr-merge/SKILL.md":"---\nname: pr-merge\ndescription: This skill should be used when the user asks to \"merge a PR\", \"review and merge pull requests\", \"integrate external contributions\", \"handle PR conflicts\", \"cherry-pick from a PR\", or needs to merge GitHub PRs while maximizing contributor attribution.\nversion: 0.1.0\n---\n\n# PR Merge — Contributor-First Pull Request Integration\n\nMerge external pull requests while maximizing original author attribution. Core principle: **merge first, resolve conflicts after** — never rewrite a contributor's work from scratch.\n\n## Core Principles\n\n1. **Preserve authorship** — use `gh pr merge --squash` for clean PRs. For manual merges, use `--author=\"Name <email>\"`.\n2. **Merge first, fix after** — accept the PR's approach even if it differs from local style. Fix conflicts in a separate commit.\n3. **Selective merge is OK** — exclude files with `--exclude` when a PR contains features already implemented locally. Document what was excluded.\n4. **Never `git apply` + self-commit** — this loses author attribution entirely.\n\n## Workflow\n\n### 1. Triage Open PRs\n\n```bash\ngh pr list --repo OWNER/REPO --state open \\\n  --json number,title,author,additions,deletions,mergeable\n```\n\nClassify each PR: merge directly, merge with conflict resolution, selective merge, or close.\n\n### 2. Merge Clean PRs via GitHub\n\nPrefer `gh pr merge` — preserves author automatically:\n\n```bash\ngh pr merge NUMBER --repo OWNER/REPO --squash \\\n  --subject \"feat: description (#NUMBER)\"\n```\n\n### 3. Handle Conflicting PRs Locally\n\n```bash\ngit fetch origin pull/NUMBER/head:pr-NUMBER\ngit merge pr-NUMBER --no-edit\n# Resolve conflicts keeping both sides where possible\ngit add -A && git commit --no-edit\n```\n\n### 4. Selective Merge (Skip Some Files)\n\nWhen a PR contains features already implemented locally:\n\n```bash\ngh pr diff NUMBER --repo OWNER/REPO | \\\n  git apply --exclude='path/to/skip.py' --exclude='CHANGELOG.md'\ngit commit --author=\"Author Name <author@email>\" \\\n  -m \"feat: description (#NUMBER)\n\nCherry-picked from PR #NUMBER. Excluded: file.py (already implemented).\"\n```\n\n### 5. Close Duplicate/Superseded PRs\n\n```bash\ngh pr close NUMBER --repo OWNER/REPO \\\n  --comment \"Fixed via PR #OTHER. Thank you for the contribution!\"\n```\n\n### 6. Post-Merge Verification\n\n```bash\npython -m ruff check src/ tests/    # lint\npython -m pytest tests/ -q          # unit tests\ngit push origin main                # push\n```\n\nThen run `harness-eval` for end-to-end verification on an unfamiliar codebase.\n\n## Attribution Checklist\n\nBefore pushing a merged PR:\n\n- [ ] Original author appears in `git log` (via `--author` or GitHub squash merge)\n- [ ] Commit message references PR number (`#NUMBER`)\n- [ ] If selectively merged, commit body explains exclusions\n- [ ] Closed PRs have a comment thanking the contributor\n- [ ] Duplicate PRs acknowledge the contributor's investigation\n\n## Common Pitfalls\n\n- `git apply` + self-commit loses author\n- Rewriting from scratch instead of merging — merge their code, fix style after\n- Force-pushing main after merge — may remove contributor commits\n- Forgetting CHANGELOG conflicts — always exclude and handle manually\n- Not testing after merge — clean merge doesn't mean working code\n\n## Additional Resources\n\n### Reference Files\n\n- **`references/merge-scenarios.md`** — Detailed examples for each merge scenario (clean, conflicting, selective, duplicate)\n",".claude/skills/pr-merge/references/merge-scenarios.md":"# Merge Scenarios — Detailed Examples\n\n## Scenario 1: Clean Merge (No Conflicts)\n\nPR adds a new feature, no files overlap with local changes.\n\n```bash\n# Preferred: squash merge via GitHub (preserves author)\ngh pr merge 17 --repo HKUDS/OpenHarness --squash \\\n  --subject \"feat(skills): add diagnose skill (#17)\"\n\n# Result: author \"Qu Zhi\" appears in git log\n```\n\n## Scenario 2: CHANGELOG Conflict Only\n\nAlmost every PR touches CHANGELOG.md. Handle by excluding it:\n\n```bash\ngh pr diff 14 --repo OWNER/REPO | \\\n  git apply --exclude='CHANGELOG.md'\ngit add -A\n\n# Manually merge CHANGELOG sections (keep both)\n# Then commit with original author\ngit commit --author=\"washi4 <washi4@users.noreply.github.com>\" \\\n  -m \"feat: add OpenAI-compatible client (#14)\"\n```\n\n## Scenario 3: Conflicting UI Files\n\nPR #14 modified ui/app.py and ui/backend_host.py which were also changed by PR #13.\n\n```bash\n# Apply excluding conflicting files\ngh pr diff 14 --repo OWNER/REPO | \\\n  git apply --exclude='src/ui/app.py' --exclude='src/ui/backend_host.py' --exclude='CHANGELOG.md'\n\n# Manually add the PR's changes to conflicting files\n# Read the PR diff to understand what they added, apply by hand\n# Commit with author\ngit commit --author=\"Author <email>\" -m \"feat: description (#14)\"\n```\n\n## Scenario 4: Selective Merge (Skip Features)\n\nPR #16 has auto-compact (already implemented locally) + --resume (wanted) + cron (wanted).\n\n```bash\n# Exclude the file containing the unwanted feature\ngh pr diff 16 --repo OWNER/REPO | git apply \\\n  --exclude='src/engine/query_engine.py' \\  # skip auto-compact\n  --exclude='CHANGELOG.md' \\\n  --exclude='README.md'\n\n# Commit with explanation\ngit commit --author=\"Chao Qin <win4r@users.noreply.github.com>\" \\\n  -m \"feat: wire --resume/--continue and cron scheduler (#16)\n\nCherry-picked from PR #16. Excluded auto-compact (already implemented\nwith LLM-based approach from reference source).\"\n```\n\n## Scenario 5: Duplicate PRs (Same Bug, Different Fix)\n\nPR #11 and #13 both fix the double-Enter bug. #13 is smaller and cleaner.\n\n```bash\n# Merge #13 (the better fix)\ngh pr merge 13 --repo OWNER/REPO --squash\n\n# Close #11 with acknowledgment\ngh pr close 11 --repo OWNER/REPO --comment \\\n  \"Fixed via PR #13 (smaller patch for the same bug). \\\nThank you for the detailed investigation!\"\n```\n\n## Scenario 6: PR From a Fork\n\nThe contributor pushed to their fork, not a branch on the repo.\n\n```bash\n# Fetch via PR ref (works for any PR regardless of source)\ngit fetch origin pull/14/head:pr-14\ngit merge pr-14 --no-edit\n\n# If merge conflict:\ngit mergetool  # or manually resolve\ngit add -A && git commit --no-edit\n\ngit push origin main\n```\n\n## Scenario 7: Batch Merge (Multiple PRs)\n\nMerge in dependency order, test after each:\n\n```bash\n# 1. Small fixes first (least risk)\ngh pr merge 17 --squash  # skill file\ngh pr merge 13 --squash  # 1-file bug fix\n\n# 2. Medium changes\ngh pr merge 12 --squash  # memory improvement\n\n# 3. Large features last (most conflict risk)\n# PR #14 needs manual conflict resolution\ngit fetch origin pull/14/head:pr-14\ngit merge pr-14\n# resolve CHANGELOG conflict\ngit push origin main\n\n# 4. Test after all merges\npython -m ruff check src/ tests/\npython -m pytest tests/ -q\n```\n\n## Post-Merge Fix Commit Pattern\n\nWhen the merged PR introduces a compatibility issue (e.g., API mismatch with a provider):\n\n```bash\n# Fix in a SEPARATE commit (don't amend the author's commit)\ngit commit -m \"fix(api): handle Kimi reasoning_content in OpenAI client\n\nKimi k2.5 requires reasoning_content on assistant tool_call messages.\nFix: capture during streaming, replay when converting back.\nFound during post-merge testing with harness-eval.\"\n```\n\nThis preserves the original author's commit intact while documenting the fix separately.\n","src/openharness/skills/bundled/content/commit.md":"# commit\n\nCreate clean, well-structured git commits.\n\n## When to use\n\nUse when the user asks to commit changes, create a PR, or prepare code for review.\n\n## Workflow\n\n1. Run `git status` and `git diff` to understand all changes\n2. Analyze changes: categorize as feature, fix, refactor, docs, test, etc.\n3. Draft a concise commit message:\n   - First line: imperative mood, under 72 chars, describes the \"why\"\n   - Body (if needed): explain context, trade-offs, or breaking changes\n4. Stage only relevant files — never stage .env, credentials, or large binaries\n5. Create the commit\n\n## Rules\n\n- Prefer specific `git add <file>` over `git add -A`\n- Never use `--no-verify` unless explicitly asked\n- Never amend published commits unless explicitly asked\n- If a pre-commit hook fails, fix the issue and create a NEW commit (don't --amend)\n- Include `Co-Authored-By` if pair programming\n","src/openharness/skills/bundled/content/debug.md":"# debug\n\nDiagnose and fix bugs systematically.\n\n## When to use\n\nUse when the user reports a bug, error, or unexpected behavior.\n\n## Workflow\n\n1. Reproduce: understand the exact steps that trigger the issue\n2. Read the error: stack traces, log messages, error codes\n3. Locate: use Grep/Read to find the relevant code path\n4. Hypothesize: form a theory about the root cause\n5. Verify: add logging or read surrounding code to confirm\n6. Fix: make the minimal change that addresses the root cause\n7. Test: verify the fix works and doesn't break other things\n\n## Rules\n\n- Read the error message carefully before searching code\n- Don't guess — verify your hypothesis before changing code\n- Fix the root cause, not the symptom\n- Don't retry the same approach if it failed — investigate why\n- If stuck after 3 attempts, explain what you've tried and ask for help\n","src/openharness/skills/bundled/content/diagnose.md":"# diagnose\n\nDiagnose why an agent run failed, regressed, or produced unexpected output — using structured evidence instead of intuition.\n\n## When to use\n\nUse when the user asks:\n- \"Why did this run fail?\"\n- \"What changed between this run and the last one?\"\n- \"The output looks wrong — what happened?\"\n- \"Why is this run worse than before?\"\n\n## Workflow\n\n1. **Locate the run artifacts** — check `artifacts/runs/<run_id>/` or the latest run directory for:\n   - `manifest.json` — run metadata, task type, input hash, timestamps\n   - `execution_trace.jsonl` — the full tool-call chain, one event per line\n   - `verification_report.json` — what was verified and whether it passed\n   - `failure_signature.json` — which stage failed and why (if present)\n2. **Read the failure signature first** — if it exists, it already localizes the failure to a specific stage (routing, execution, verification, or governance)\n3. **Trace the execution chain** — walk `execution_trace.jsonl` forward to find where output diverged from expectation\n4. **Compare with a passing run** — if a previous run succeeded, diff the two manifests and traces to identify what changed\n5. **Identify the failure layer**:\n   - **Routing**: wrong task type or profile selected\n   - **Execution**: tool call error, permission block, or unexpected result\n   - **Verification**: output produced but did not match expected artifact\n   - **Governance**: run was halted by a safety or boundary check\n6. **Report with evidence** — cite specific file paths, line numbers in the trace, or field values from the manifest; never summarize without pointing to the source\n\n## Rules\n\n- Evidence before intuition: always read the trace before forming a hypothesis\n- Localize before explaining: identify which stage failed before describing why\n- Compare, don't guess: if a previous run succeeded, diff it — don't speculate about what changed\n- If no artifacts exist, say so clearly and ask the user to re-run with archive mode enabled\n","src/openharness/skills/bundled/content/plan.md":"# plan\n\nDesign an implementation plan before coding.\n\n## When to use\n\nUse when the user asks to plan, design, or architect a feature before implementing it.\n\n## Workflow\n\n1. Understand the requirement:\n   - What problem does this solve?\n   - What are the constraints?\n   - What's the expected outcome?\n2. Explore the codebase to find:\n   - Existing functions/utilities that can be reused\n   - Patterns used elsewhere in the project\n   - Files that will need modification\n3. Design the approach:\n   - Break into discrete steps\n   - Identify dependencies between steps\n   - Consider edge cases and error handling\n4. Present the plan:\n   - Start with a Context section (why this change)\n   - List concrete steps with file paths\n   - Include a verification section (how to test)\n\n## Rules\n\n- Read before suggesting — never propose changes to code you haven't read\n- Prefer editing existing files over creating new ones\n- Reuse existing patterns and utilities\n- Include file paths and line numbers when referencing code\n- Don't over-plan: match complexity to the task\n","src/openharness/skills/bundled/content/review.md":"# review\n\nReview code for bugs, security issues, and quality.\n\n## When to use\n\nUse when the user asks to review code, a PR, or a diff.\n\n## Workflow\n\n1. Read the changed files or diff thoroughly\n2. Check for:\n   - **Bugs**: logic errors, off-by-one, null/undefined access, race conditions\n   - **Security**: injection, XSS, hardcoded secrets, path traversal\n   - **Performance**: N+1 queries, unnecessary allocations, missing indexes\n   - **Tests**: are new code paths covered? Are edge cases tested?\n   - **Style**: naming consistency, dead code, unnecessary complexity\n3. Provide concrete, actionable feedback with file:line references\n4. Prioritize findings by severity (critical > major > minor > nit)\n\n## Rules\n\n- Be specific: \"line 42 may throw if `user` is null\" not \"check for null\"\n- Suggest fixes, don't just point out problems\n- Acknowledge good patterns too\n- Don't nitpick formatting if there's a linter\n","src/openharness/skills/bundled/content/simplify.md":"# simplify\n\nRefactor code to be simpler and more maintainable.\n\n## When to use\n\nUse when the user asks to simplify, clean up, or refactor code.\n\n## Workflow\n\n1. Read the target code completely\n2. Identify complexity sources:\n   - Unnecessary abstractions or indirection\n   - Duplicated logic that should be consolidated\n   - Over-engineered patterns (factories, builders, strategies for simple cases)\n   - Dead code paths or unused variables\n3. Propose changes that reduce line count while preserving behavior\n4. Verify the refactored code passes existing tests\n\n## Rules\n\n- Don't add features or change behavior — simplification only\n- Prefer deleting code over refactoring it\n- Three similar lines are better than a premature abstraction\n- Remove backwards-compatibility shims for removed features\n- Don't add comments to explain simple code\n","src/openharness/skills/bundled/content/skill-creator.md":"---\nname: skill-creator\ndescription: >\n  Create, improve, and verify OpenHarness skills. Use this whenever the user\n  asks to create a new skill, convert a workflow into a skill, update an\n  existing SKILL.md, add skills for oh/ohmo, design skill trigger behavior,\n  or test whether a skill loads and works correctly.\n---\n\n# skill-creator\n\nCreate or improve OpenHarness skills in the directory-based `SKILL.md` format.\n\n## When to use\n\nUse this skill when the user wants to:\n\n- create a new skill from a workflow, repeated task, domain guide, or tool process\n- convert existing notes, prompts, or instructions into a reusable skill\n- modify or harden an existing skill\n- make a skill available to `oh`, `ohmo`, or a plugin\n- debug why the `skill` tool cannot find or load a skill\n- test trigger wording, skill metadata, or skill behavior\n\n## OpenHarness skill locations\n\nChoose the target deliberately:\n\n- Built-in OpenHarness skills live in `src/openharness/skills/bundled/content/*.md`.\n- User skills for `oh` live in `~/.openharness/skills/<skill-dir>/SKILL.md`.\n- Private `ohmo` skills live in `~/.ohmo/skills/<skill-dir>/SKILL.md`.\n- Plugin skills live in `<plugin-root>/skills/<skill-dir>/SKILL.md`.\n\nOpenHarness user and plugin skills use a directory layout. Do not create flat\n`*.md` user skills under `skills/`; they will not be loaded by the normal loader.\n\n## Skill anatomy\n\nUse this structure for user, ohmo, and plugin skills:\n\n```text\nmy-skill/\n├── SKILL.md\n├── scripts/      # optional deterministic helpers\n├── references/   # optional long docs loaded only when relevant\n└── assets/       # optional templates or static files\n```\n\nUse this structure inside `SKILL.md`:\n\n```markdown\n---\nname: my-skill\ndescription: >\n  What the skill does and exactly when to use it.\n---\n\n# my-skill\n\nShort purpose statement.\n\n## When to use\n\nTrigger contexts in plain language.\n\n## Workflow\n\nConcrete steps the agent should follow.\n\n## Rules\n\nImportant boundaries, verification, and failure handling.\n```\n\n## Workflow\n\n1. Capture intent before writing.\n   Ask or infer what the skill should help with, when it should trigger, what\n   output it should produce, and what success looks like. If the conversation\n   already contains the workflow, extract the sequence from history before\n   asking more questions.\n\n2. Inspect existing examples.\n   Read nearby skills and loader behavior before choosing a layout. For\n   OpenHarness, confirm whether the target is bundled, user, ohmo-private, or\n   plugin-provided.\n\n3. Write the metadata first.\n   The `description` is the primary trigger hint shown in the available skills\n   list. Make it specific and slightly assertive: include both what the skill\n   does and the phrases or situations that should trigger it. Avoid vague\n   descriptions like \"Helpful notes for X\".\n\n4. Keep the body procedural.\n   Use concise imperative steps. Explain why non-obvious constraints matter.\n   Prefer general principles over overfitted examples. If a skill is getting\n   long, move details into `references/` and point to the relevant file.\n\n5. Add deterministic resources only when they pay for themselves.\n   If the skill repeatedly needs the same script, template, schema, or checklist,\n   place it under `scripts/`, `assets/`, or `references/` rather than making the\n   model regenerate it every time.\n\n6. Verify loading.\n   Run a local loader check for the chosen target. For bundled skills, use:\n\n   ```bash\n   PYTHONPATH=src python - <<'PY'\n   from openharness.skills import load_skill_registry\n   for skill in load_skill_registry().list_skills():\n       print(skill.name, skill.source, skill.path)\n   PY\n   ```\n\n   For ohmo-private skills, include the extra skill directory:\n\n   ```bash\n   PYTHONPATH=src python - <<'PY'\n   from openharness.skills import load_skill_registry\n   from ohmo.workspace import get_skills_dir\n   for skill in load_skill_registry(extra_skill_dirs=[get_skills_dir()]).list_skills():\n       print(skill.name, skill.source, skill.path)\n   PY\n   ```\n\n7. Test behavior.\n   Create two or three realistic prompts that should use the skill. If the skill\n   changes code or files, add regression tests around loading or command\n   behavior. Run targeted tests first, then broader tests if the loader or\n   runtime path changed.\n\n8. Iterate from evidence.\n   If a skill under-triggers, improve the frontmatter description. If it\n   triggers but produces poor work, improve the body workflow. If agents repeat\n   deterministic work, add a helper script or reference file.\n\n## Writing rules\n\n- Preserve the skill name when updating an existing skill unless the user\n  explicitly asks to rename it.\n- Do not overwrite user-created skills without reading the existing `SKILL.md`.\n- Keep `SKILL.md` under roughly 500 lines; use `references/` for long material.\n- Keep trigger guidance in frontmatter `description`, not buried only in the\n  body.\n- Make skills safe and unsurprising. Do not create skills that hide behavior,\n  bypass permissions, exfiltrate data, or encourage unauthorized access.\n- Prefer directory names that are lowercase kebab-case, even if the displayed\n  skill `name` is more human-readable.\n- Mention exact install or verification paths in the final response.\n\n## Test prompts\n\nAfter drafting a skill, propose a small eval set:\n\n```json\n{\n  \"skill_name\": \"my-skill\",\n  \"evals\": [\n    {\n      \"id\": \"happy-path\",\n      \"prompt\": \"A realistic user request that should trigger the skill.\",\n      \"expected_output\": \"What good behavior looks like.\"\n    },\n    {\n      \"id\": \"near-miss\",\n      \"prompt\": \"A similar request that should not trigger this skill.\",\n      \"expected_output\": \"The agent should choose a different path.\"\n    }\n  ]\n}\n```\n\nFor objective skills, add assertions or tests. For subjective skills, collect\nhuman feedback and revise the workflow from concrete complaints rather than\nadding rigid rules.\n","src/openharness/skills/bundled/content/test.md":"# test\n\nWrite and run tests for code.\n\n## When to use\n\nUse when the user asks to write tests, verify behavior, or improve test coverage.\n\n## Workflow\n\n1. Understand what needs testing:\n   - New feature? Write unit tests for the happy path and edge cases\n   - Bug fix? Write a regression test that would have caught the bug\n   - Refactor? Ensure existing tests still pass\n2. Follow the project's testing patterns:\n   - Check existing tests for framework (pytest, jest, go test, etc.)\n   - Match naming conventions and file organization\n   - Use the same fixtures and helpers\n3. Write tests that are:\n   - Independent: each test can run alone\n   - Deterministic: same result every time\n   - Fast: mock external services, use in-memory databases\n4. Run the tests and verify they pass\n\n## Rules\n\n- Test behavior, not implementation details\n- One assertion per test when possible\n- Use descriptive test names that explain the scenario\n- Don't test framework or library code\n- Mock at system boundaries (external APIs, filesystem, network)\n"},"items":[{"name":"feature-matrix.md","path":".claude/skills/harness-eval/references/feature-matrix.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/OpenHarness/HEAD/.claude/skills/harness-eval/references/feature-matrix.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Feature Test Matrix — Detailed Test Cases\n\n## Engine & Tools\n\n| Test | What to Verify | Key Assertion |\n|------|---------------|---------------|\n| Multi-turn memory | Set fact turn 1, recall turn 3 | Fact appears in turn 3 response |\n| Tool chaining | glob → grep → read in one task | All 3 tools in tool list |\n| Write→Edit→Read | Create, modify, verify file | File content matches expected |\n| Parallel tools | 3+ tool calls in one response | 3+ tools in single turn |\n| Error recovery | Tool fails, model adapts | Alternative tool used after error |\n| Auto-compaction | 5+ tasks on shared engine | No context overflow crash |\n\n## Swarm & Coordinator\n\n| Test | What to Verify | Key Assertion |\n|------|---------------|---------------|\n| InProcessBackend | spawn → active → status → shutdown | All states transition correctly |\n| Concurrent teammates | 2+ agents running simultaneously | Both complete, total time < 2x single |\n| Coordinator + notifications | Multi-turn delegation with XML | Coordinator synthesizes worker results |\n| Permission sync | request → pending → resolve | Pending count goes to 0 after resolve |\n\n## Hooks, Skills, Plugins\n\n| Test | What to Verify | Key Assertion |\n|------|---------------|---------------|\n| Hook blocks → adapt | pre_tool_use blocks bash | \"bash\" in errors, \"glob\" in tools |\n| Skill invocation | Model calls skill tool | \"skill\" in tools, content drives next action |\n| Plugin skill | Plugin provides skill | Loaded via skill tool, model follows it |\n| Hook + skill combined | Hook gates writes, skill guides | Protected file untouched, new file created |\n\n## Memory, Session, Config\n\n| Test | What to Verify | Key Assertion |\n|------|---------------|---------------|\n| Memory frontmatter | YAML parsed, not \"---\" | description != \"---\", body searchable |\n| Session resume | Save → load → continue | Model remembers prior context |\n| Cost tracking | Tokens accumulate | in_tokens strictly increasing |\n| Cron CRUD | Create, toggle, mark_run, delete | Job count correct at each step |\n\n## Provider Compatibility\n\n| Test | What to Verify | Key Assertion |\n|------|---------------|---------------|\n| Anthropic client | Standard tool calling | Tools execute, response coherent |\n| OpenAI client | Tool calling + reasoning_content | No 400 error on tool call round-trip |\n| OpenAI multi-turn | reasoning_content persists | 3+ turns without API error |\n","isInternal":false,"tokens":521,"sizeBytes":2423},{"name":"test-patterns.md","path":".claude/skills/harness-eval/references/test-patterns.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/OpenHarness/HEAD/.claude/skills/harness-eval/references/test-patterns.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Test Patterns — Code Templates for Harness Eval\n\n## Engine Setup Helpers\n\n```python\nimport asyncio, sys, os\nfrom pathlib import Path\nsys.path.insert(0, \"src\")\n\nAPI_KEY = os.environ.get(\"ANTHROPIC_API_KEY\", \"your-key\")\nBASE_URL = os.environ.get(\"ANTHROPIC_BASE_URL\", \"https://api.moonshot.cn/anthropic\")\nOPENAI_BASE = os.environ.get(\"OPENAI_BASE_URL\", \"https://api.moonshot.cn/v1\")\nMODEL = os.environ.get(\"ANTHROPIC_MODEL\", \"kimi-k2.5\")\nWORKSPACE = Path(\"/tmp/eval-workspace\")  # unfamiliar project\n\n\ndef make_anthropic_engine(system_prompt, cwd=None, extra_tools=None):\n    from openharness.api.client import AnthropicApiClient\n    from openharness.config.settings import PermissionSettings\n    from openharness.engine.query_engine import QueryEngine\n    from openharness.permissions.checker import PermissionChecker\n    from openharness.permissions.modes import PermissionMode\n    from openharness.tools.base import ToolRegistry\n    from openharness.tools.bash_tool import BashTool\n    from openharness.tools.file_read_tool import FileReadTool\n    from openharness.tools.file_write_tool import FileWriteTool\n    from openharness.tools.file_edit_tool import FileEditTool\n    from openharness.tools.glob_tool import GlobTool\n    from openharness.tools.grep_tool import GrepTool\n\n    api = AnthropicApiClient(api_key=API_KEY, base_url=BASE_URL)\n    reg = ToolRegistry()\n    for t in [BashTool(), FileReadTool(), FileWriteTool(), FileEditTool(), GlobTool(), GrepTool()]:\n        reg.register(t)\n    for t in (extra_tools or []):\n        reg.register(t)\n    checker = PermissionChecker(PermissionSettings(mode=PermissionMode.FULL_AUTO))\n    return QueryEngine(\n        api_client=api, tool_registry=reg, permission_checker=checker,\n        cwd=Path(cwd or WORKSPACE), model=MODEL, system_prompt=system_prompt, max_tokens=4096,\n    )\n\n\ndef make_openai_engine(system_prompt, cwd=None, extra_tools=None):\n    from openharness.api.openai_client import OpenAICompatibleClient\n    # Same structure as above, but with:\n    api = OpenAICompatibleClient(api_key=API_KEY, base_url=OPENAI_BASE)\n    # ... rest identical\n\n\ndef collect(events):\n    from openharness.engine.stream_events import (\n        AssistantTextDelta, AssistantTurnComplete,\n        ToolExecutionStarted, ToolExecutionCompleted,\n    )\n    r = {\"text\": \"\", \"tools\": [], \"turns\": 0, \"in_tok\": 0, \"out_tok\": 0}\n    for ev in events:\n        if isinstance(ev, AssistantTextDelta):\n            r[\"text\"] += ev.text\n        elif isinstance(ev, ToolExecutionStarted):\n            r[\"tools\"].append(ev.tool_name)\n        elif isinstance(ev, AssistantTurnComplete):\n            r[\"turns\"] += 1\n            r[\"in_tok\"] += ev.usage.input_tokens\n            r[\"out_tok\"] += ev.usage.output_tokens\n    return r\n```\n\n## Multi-Turn Memory Test\n\n```python\nasync def test_multi_turn_memory():\n    engine = make_anthropic_engine(\"Remember what the user tells you.\")\n    [ev async for ev in engine.submit_message(\"My favorite number is 42.\")]\n    [ev async for ev in engine.submit_message(\"What is 2+2?\")]\n    evs = [ev async for ev in engine.submit_message(\"What is my favorite number?\")]\n    r = collect(evs)\n    assert \"42\" in r[\"text\"]\n```\n\n## Hook Blocks Tool → Model Adapts\n\n```python\nasync def test_hook_blocks():\n    from openharness.hooks.events import HookEvent\n    from openharness.hooks.loader import HookRegistry\n    from openharness.hooks.schemas import CommandHookDefinition\n    from openharness.hooks.executor import HookExecutor, HookExecutionContext\n\n    hook_reg = HookRegistry()\n    hook_reg.register(HookEvent.PRE_TOOL_USE, CommandHookDefinition(\n        type=\"command\", command=\"exit 1\",\n        matcher=\"bash\", block_on_failure=True, timeout_seconds=5,\n    ))\n    # ... create engine with hook_executor, model tries bash, gets blocked, adapts to glob\n```\n\n## Skill Tool Invocation\n\n```python\nasync def test_skill_invocation():\n    from openharness.tools.skill_tool import SkillTool\n    engine = make_anthropic_engine(\n        \"Use the 'skill' tool to load instructions before working.\",\n        extra_tools=[SkillTool()],\n    )\n    evs = [ev async for ev in engine.submit_message(\n        \"Load the 'diagnose' skill, then investigate the codebase.\"\n    )]\n    r = collect(evs)\n    assert \"skill\" in r[\"tools\"]\n```\n\n## InProcess Concurrent Teammates\n\n```python\nasync def test_concurrent_teammates():\n    from openharness.swarm.in_process import start_in_process_teammate, TeammateAbortController\n    from openharness.swarm.types import TeammateSpawnConfig\n    from openharness.engine.query import QueryContext\n\n    async def run_one(name, prompt):\n        ctx = QueryContext(api_client=api, tool_registry=reg, ...)\n        config = TeammateSpawnConfig(name=name, team=\"test\", prompt=prompt, ...)\n        abort = TeammateAbortController()\n        await start_in_process_teammate(config=config, agent_id=f\"{name}@test\", abort_controller=abort, query_context=ctx)\n\n    await asyncio.gather(\n        asyncio.wait_for(run_one(\"worker-a\", \"Count .py files\"), timeout=30),\n        asyncio.wait_for(run_one(\"worker-b\", \"Find main class\"), timeout=30),\n    )\n```\n\n## Session Save → Resume\n\n```python\nasync def test_session_resume():\n    from openharness.services.session_storage import save_session_snapshot, load_session_snapshot\n    from openharness.engine.messages import ConversationMessage\n\n    engine1 = make_anthropic_engine(\"Remember context.\")\n    [ev async for ev in engine1.submit_message(\"Project uses FastAPI + React.\")]\n    save_session_snapshot(cwd=tmpdir, model=MODEL, system_prompt=\"...\", messages=engine1.messages, usage=engine1.total_usage)\n\n    loaded = load_session_snapshot(tmpdir)\n    engine2 = make_anthropic_engine(\"Continue analysis.\")\n    engine2.load_messages([ConversationMessage.model_validate(m) for m in loaded[\"messages\"]])\n    evs = [ev async for ev in engine2.submit_message(\"What tech stack did I mention?\")]\n    assert \"fastapi\" in collect(evs)[\"text\"].lower()\n```\n","isInternal":false,"tokens":1422,"sizeBytes":5959},{"name":"SKILL.md","path":".claude/skills/harness-eval/SKILL.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/OpenHarness/HEAD/.claude/skills/harness-eval/SKILL.md","title":"harness-eval","category":"anthropic-skill","format":"markdown","content":"---\nname: harness-eval\ndescription: This skill should be used when the user asks to \"test the harness\", \"run integration tests\", \"validate features with real API\", \"test with real model calls\", \"run agent loop tests\", \"verify end-to-end\", or needs to verify OpenHarness features on a real codebase with actual LLM calls.\nversion: 0.2.0\n---\n\n# Harness Eval — End-to-End Feature Validation\n\nValidate OpenHarness features by running real agent loops against an unfamiliar codebase with actual LLM API calls. Every test exercises the full stack: API client → model → tool calls → execution → result.\n\n## Core Principles\n\n1. **Test on an unfamiliar project** — never test on OpenHarness itself (the agent modifies its own code). Clone a real project as the workspace.\n2. **Use real API calls** — no mocks. Configure a real LLM endpoint.\n3. **Multi-turn conversations** — always test 2+ turns where the model needs prior context.\n4. **Combine features** — test hooks+skills+agent loop together, not in isolation.\n5. **Verify tool execution** — inspect tool call lists and output files, not just model text.\n\n## Workflow\n\n### 1. Prepare Workspace\n\nClone an unfamiliar project (do not use OpenHarness):\n\n```bash\ngit clone https://github.com/HKUDS/AutoAgent /tmp/eval-workspace\n```\n\n### 2. Configure Environment\n\n```bash\nexport ANTHROPIC_API_KEY=sk-xxx\nexport ANTHROPIC_BASE_URL=https://api.moonshot.cn/anthropic  # or any provider\nexport ANTHROPIC_MODEL=kimi-k2.5\n```\n\nFor long-running real evals, do not artificially lower `max_turns`. Use the product default (`200`) unless the user explicitly wants a tighter bound.\n\n### 3. Prepare Real Sandbox Runtime When Relevant\n\nIf the task is validating sandbox behavior, install and verify the actual runtime before running agent loops:\n\n```bash\nnpm install -g @anthropic-ai/sandbox-runtime\nsudo apt-get update\nsudo apt-get install -y bubblewrap ripgrep\nwhich srt\nwhich bwrap\nwhich rg\nsrt --version\n```\n\nThen run a minimal smoke check through OpenHarness, not just raw `srt`, so you verify the real adapter path:\n\n```python\nfrom pathlib import Path\nfrom openharness.config.settings import Settings, SandboxSettings, save_settings\nfrom openharness.tools.bash_tool import BashTool\n\ncfg = Path(\"/tmp/openharness-sandbox-settings.json\")\nsave_settings(Settings(sandbox=SandboxSettings(enabled=True, fail_if_unavailable=True)), cfg)\n# Point config loader at this file, then run BashTool on a tiny command such as `pwd`.\n```\n\nIf sandbox dependencies are missing, treat that as an environment/setup failure, not a feature regression.\n\n### 4. Design Tests\n\nEach test follows this pattern:\n\n```python\nengine = make_engine(system_prompt=\"...\", cwd=UNFAMILIAR_PROJECT)\nevs1 = [ev async for ev in engine.submit_message(\"Read X, analyze Y\")]\nr1 = collect(evs1)  # text, tools, turns, tokens\nevs2 = [ev async for ev in engine.submit_message(\"Based on what you found...\")]\nr2 = collect(evs2)\nassert \"grep\" in r1[\"tools\"]  # verify tools ran\n```\n\nFor detailed code templates and the `make_engine`/`collect` helpers, consult `references/test-patterns.md`.\n\n### 5. Prefer Long-Horizon, Real Agent Loops\n\nFor meaningful end-to-end validation, prefer unfamiliar-repo tasks that force multiple turns, context reuse, and mixed tool usage.\n\nRecommended pattern:\n\n- Use a real external workspace such as `AutoAgent`\n- Use real provider credentials and the actual target model\n- Keep `max_turns=200`\n- Use per-prompt timeouts large enough for real exploration, such as `240-600s`\n- Require at least 2 turns per scenario\n- Verify both text quality and tool traces\n- Keep polling long-running sessions until they finish; do not abandon a run after the first long pause\n\nRecommended long-horizon scenarios:\n\n- `architecture_multiturn`\n  - Turn 1: map architecture, shell/subprocess surfaces, and test entrypoints\n  - Turn 2: identify top risks and propose refactors\n  - Turn 3: condense into onboarding or remediation actions\n  - Success: `bash`, `glob`, `grep`, `read_file` all appear; no timeout; no `MaxTurnsExceeded`\n\n- `hook_block_and_recover`\n  - Force the model to try `bash`\n  - Block it with a real pre-tool hook\n  - Verify the model adapts with `glob`/`grep`/`read_file`\n\n- `sandbox_multiturn`\n  - Enable real sandbox settings with `fail_if_unavailable=true`\n  - First prompt must start with exactly one shell command such as `pwd && ls -la`\n  - Second prompt must explicitly reuse the prior shell findings\n  - Success: `bash` executes via sandbox, non-shell tools continue the task, and the agent recovers from incidental repo errors\n\nWhen a scenario fails, classify it before changing code:\n\n- `MaxTurnsExceeded`: likely eval harness misconfiguration if `max_turns` was manually lowered\n- `timeout`: task is too broad or per-prompt timeout is too small\n- sandbox unavailable: environment missing `srt`, `bwrap`, or `rg`\n- tool error with task still completed: feature may still be healthy; inspect recovery behavior\n\n### 6. Run Tests\n\n```bash\npython tests/test_merged_prs_on_autoagent.py   # PR feature tests\npython tests/test_real_large_tasks.py           # large multi-step tasks\npython tests/test_hooks_skills_plugins_real.py  # hooks/skills/plugins\npython -m pytest tests/ -q -k \"not autoagent\"  # unit tests (no API)\n```\n\nFor ad hoc long-horizon validation, it is acceptable to run a temporary Python driver script as long as it:\n\n- uses real OpenHarness engine/tool objects\n- targets an unfamiliar repository\n- prints per-scenario JSON summaries\n- records tools, errors, turns, and token usage\n- stays attached until completion\n\n### 7. Interpret Results\n\n| Result | Meaning | Action |\n|--------|---------|--------|\n| PASS with tool calls | Feature works end-to-end | Done |\n| PASS without tool calls | Model answered from knowledge | Rewrite prompt to force tool use |\n| FAIL with exception | Code bug | Read traceback |\n| FAIL with wrong output | Model behavior issue | Check system prompt and tool schemas |\n| Timeout | Task too complex | Increase `max_turns` or simplify prompt |\n\nFor long-running real evals, refine the timeout guidance:\n\n- First check whether `max_turns` was manually set too low\n- If `max_turns=200` and the run still fails, the next suspect is wall-clock timeout, not turn count\n- Distinguish environment failures from product failures\n  - Example: missing dependency in the unfamiliar target repo is not automatically an OpenHarness regression\n  - Example: missing `srt`/`bwrap`/`rg` is an eval environment issue\n\n## Feature Coverage Checklist\n\n- [ ] Engine: multi-turn memory, tool chaining, parallel tools, error recovery, auto-compaction\n- [ ] Swarm: InProcessBackend lifecycle, concurrent teammates, coordinator+notifications\n- [ ] Hooks: pre_tool_use blocking → model adapts, post_tool_use firing\n- [ ] Skills: skill tool invocation → model follows instructions\n- [ ] Plugins: plugin-provided skill loaded and used in agent loop\n- [ ] Memory: YAML frontmatter parsing, body content search, context injection\n- [ ] Session: save → load → resume with context preserved\n- [ ] Providers: Anthropic client, OpenAI client (with reasoning_content), multi-turn\n- [ ] Cost: token accumulation across turns\n\n## Common Pitfalls\n\n- Testing on OpenHarness itself — agent modifies its own running code\n- Using mocks — misses serialization and API compatibility bugs\n- Single-turn only — misses context accumulation and compaction bugs\n- Artificially lowering `max_turns` during real evals — can create false failures that do not reflect product defaults\n- Not checking tool call list — model may claim tool use without calling it\n- Hardcoding paths — use `WORKSPACE` variable, skip in CI with `pytest.mark.skipif`\n- Declaring sandbox “tested” after only checking raw `srt` — verify the OpenHarness adapter path too\n- Abandoning long tasks too early — some real tasks pause for minutes before the next event arrives\n\n## Additional Resources\n\n### Reference Files\n\n- **`references/test-patterns.md`** — Complete code templates for `make_engine`, `collect`, and each feature category\n- **`references/feature-matrix.md`** — Detailed test cases for every OpenHarness module\n\n### Existing Test Files\n\nWorking test suites in the repo:\n- `tests/test_merged_prs_on_autoagent.py` — PR feature validation\n- `tests/test_real_large_tasks.py` — Large multi-step tasks\n- `tests/test_hooks_skills_plugins_real.py` — Hooks/skills/plugins in agent loops\n- `tests/test_untested_features.py` — Module-level integration tests\n","frontmatter":{"name":"harness-eval","description":"This skill should be used when the user asks to \"test the harness\", \"run integration tests\", \"validate features with real API\", \"test with real model calls\", \"run agent loop tests\", \"verify end-to-end\", or needs to verify OpenHarness features on a real codebase with actual LLM calls.","version":"0.2.0"},"isInternal":false,"tokens":1978,"sizeBytes":8505},{"name":"merge-scenarios.md","path":".claude/skills/pr-merge/references/merge-scenarios.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/OpenHarness/HEAD/.claude/skills/pr-merge/references/merge-scenarios.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Merge Scenarios — Detailed Examples\n\n## Scenario 1: Clean Merge (No Conflicts)\n\nPR adds a new feature, no files overlap with local changes.\n\n```bash\n# Preferred: squash merge via GitHub (preserves author)\ngh pr merge 17 --repo HKUDS/OpenHarness --squash \\\n  --subject \"feat(skills): add diagnose skill (#17)\"\n\n# Result: author \"Qu Zhi\" appears in git log\n```\n\n## Scenario 2: CHANGELOG Conflict Only\n\nAlmost every PR touches CHANGELOG.md. Handle by excluding it:\n\n```bash\ngh pr diff 14 --repo OWNER/REPO | \\\n  git apply --exclude='CHANGELOG.md'\ngit add -A\n\n# Manually merge CHANGELOG sections (keep both)\n# Then commit with original author\ngit commit --author=\"washi4 <washi4@users.noreply.github.com>\" \\\n  -m \"feat: add OpenAI-compatible client (#14)\"\n```\n\n## Scenario 3: Conflicting UI Files\n\nPR #14 modified ui/app.py and ui/backend_host.py which were also changed by PR #13.\n\n```bash\n# Apply excluding conflicting files\ngh pr diff 14 --repo OWNER/REPO | \\\n  git apply --exclude='src/ui/app.py' --exclude='src/ui/backend_host.py' --exclude='CHANGELOG.md'\n\n# Manually add the PR's changes to conflicting files\n# Read the PR diff to understand what they added, apply by hand\n# Commit with author\ngit commit --author=\"Author <email>\" -m \"feat: description (#14)\"\n```\n\n## Scenario 4: Selective Merge (Skip Features)\n\nPR #16 has auto-compact (already implemented locally) + --resume (wanted) + cron (wanted).\n\n```bash\n# Exclude the file containing the unwanted feature\ngh pr diff 16 --repo OWNER/REPO | git apply \\\n  --exclude='src/engine/query_engine.py' \\  # skip auto-compact\n  --exclude='CHANGELOG.md' \\\n  --exclude='README.md'\n\n# Commit with explanation\ngit commit --author=\"Chao Qin <win4r@users.noreply.github.com>\" \\\n  -m \"feat: wire --resume/--continue and cron scheduler (#16)\n\nCherry-picked from PR #16. Excluded auto-compact (already implemented\nwith LLM-based approach from reference source).\"\n```\n\n## Scenario 5: Duplicate PRs (Same Bug, Different Fix)\n\nPR #11 and #13 both fix the double-Enter bug. #13 is smaller and cleaner.\n\n```bash\n# Merge #13 (the better fix)\ngh pr merge 13 --repo OWNER/REPO --squash\n\n# Close #11 with acknowledgment\ngh pr close 11 --repo OWNER/REPO --comment \\\n  \"Fixed via PR #13 (smaller patch for the same bug). \\\nThank you for the detailed investigation!\"\n```\n\n## Scenario 6: PR From a Fork\n\nThe contributor pushed to their fork, not a branch on the repo.\n\n```bash\n# Fetch via PR ref (works for any PR regardless of source)\ngit fetch origin pull/14/head:pr-14\ngit merge pr-14 --no-edit\n\n# If merge conflict:\ngit mergetool  # or manually resolve\ngit add -A && git commit --no-edit\n\ngit push origin main\n```\n\n## Scenario 7: Batch Merge (Multiple PRs)\n\nMerge in dependency order, test after each:\n\n```bash\n# 1. Small fixes first (least risk)\ngh pr merge 17 --squash  # skill file\ngh pr merge 13 --squash  # 1-file bug fix\n\n# 2. Medium changes\ngh pr merge 12 --squash  # memory improvement\n\n# 3. Large features last (most conflict risk)\n# PR #14 needs manual conflict resolution\ngit fetch origin pull/14/head:pr-14\ngit merge pr-14\n# resolve CHANGELOG conflict\ngit push origin main\n\n# 4. Test after all merges\npython -m ruff check src/ tests/\npython -m pytest tests/ -q\n```\n\n## Post-Merge Fix Commit Pattern\n\nWhen the merged PR introduces a compatibility issue (e.g., API mismatch with a provider):\n\n```bash\n# Fix in a SEPARATE commit (don't amend the author's commit)\ngit commit -m \"fix(api): handle Kimi reasoning_content in OpenAI client\n\nKimi k2.5 requires reasoning_content on assistant tool_call messages.\nFix: capture during streaming, replay when converting back.\nFound during post-merge testing with harness-eval.\"\n```\n\nThis preserves the original author's commit intact while documenting the fix separately.\n","isInternal":false,"tokens":990,"sizeBytes":3757},{"name":"SKILL.md","path":".claude/skills/pr-merge/SKILL.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/OpenHarness/HEAD/.claude/skills/pr-merge/SKILL.md","title":"pr-merge","category":"anthropic-skill","format":"markdown","content":"---\nname: pr-merge\ndescription: This skill should be used when the user asks to \"merge a PR\", \"review and merge pull requests\", \"integrate external contributions\", \"handle PR conflicts\", \"cherry-pick from a PR\", or needs to merge GitHub PRs while maximizing contributor attribution.\nversion: 0.1.0\n---\n\n# PR Merge — Contributor-First Pull Request Integration\n\nMerge external pull requests while maximizing original author attribution. Core principle: **merge first, resolve conflicts after** — never rewrite a contributor's work from scratch.\n\n## Core Principles\n\n1. **Preserve authorship** — use `gh pr merge --squash` for clean PRs. For manual merges, use `--author=\"Name <email>\"`.\n2. **Merge first, fix after** — accept the PR's approach even if it differs from local style. Fix conflicts in a separate commit.\n3. **Selective merge is OK** — exclude files with `--exclude` when a PR contains features already implemented locally. Document what was excluded.\n4. **Never `git apply` + self-commit** — this loses author attribution entirely.\n\n## Workflow\n\n### 1. Triage Open PRs\n\n```bash\ngh pr list --repo OWNER/REPO --state open \\\n  --json number,title,author,additions,deletions,mergeable\n```\n\nClassify each PR: merge directly, merge with conflict resolution, selective merge, or close.\n\n### 2. Merge Clean PRs via GitHub\n\nPrefer `gh pr merge` — preserves author automatically:\n\n```bash\ngh pr merge NUMBER --repo OWNER/REPO --squash \\\n  --subject \"feat: description (#NUMBER)\"\n```\n\n### 3. Handle Conflicting PRs Locally\n\n```bash\ngit fetch origin pull/NUMBER/head:pr-NUMBER\ngit merge pr-NUMBER --no-edit\n# Resolve conflicts keeping both sides where possible\ngit add -A && git commit --no-edit\n```\n\n### 4. Selective Merge (Skip Some Files)\n\nWhen a PR contains features already implemented locally:\n\n```bash\ngh pr diff NUMBER --repo OWNER/REPO | \\\n  git apply --exclude='path/to/skip.py' --exclude='CHANGELOG.md'\ngit commit --author=\"Author Name <author@email>\" \\\n  -m \"feat: description (#NUMBER)\n\nCherry-picked from PR #NUMBER. Excluded: file.py (already implemented).\"\n```\n\n### 5. Close Duplicate/Superseded PRs\n\n```bash\ngh pr close NUMBER --repo OWNER/REPO \\\n  --comment \"Fixed via PR #OTHER. Thank you for the contribution!\"\n```\n\n### 6. Post-Merge Verification\n\n```bash\npython -m ruff check src/ tests/    # lint\npython -m pytest tests/ -q          # unit tests\ngit push origin main                # push\n```\n\nThen run `harness-eval` for end-to-end verification on an unfamiliar codebase.\n\n## Attribution Checklist\n\nBefore pushing a merged PR:\n\n- [ ] Original author appears in `git log` (via `--author` or GitHub squash merge)\n- [ ] Commit message references PR number (`#NUMBER`)\n- [ ] If selectively merged, commit body explains exclusions\n- [ ] Closed PRs have a comment thanking the contributor\n- [ ] Duplicate PRs acknowledge the contributor's investigation\n\n## Common Pitfalls\n\n- `git apply` + self-commit loses author\n- Rewriting from scratch instead of merging — merge their code, fix style after\n- Force-pushing main after merge — may remove contributor commits\n- Forgetting CHANGELOG conflicts — always exclude and handle manually\n- Not testing after merge — clean merge doesn't mean working code\n\n## Additional Resources\n\n### Reference Files\n\n- **`references/merge-scenarios.md`** — Detailed examples for each merge scenario (clean, conflicting, selective, duplicate)\n","frontmatter":{"name":"pr-merge","description":"This skill should be used when the user asks to \"merge a PR\", \"review and merge pull requests\", \"integrate external contributions\", \"handle PR conflicts\", \"cherry-pick from a PR\", or needs to merge GitHub PRs while maximizing contributor attribution.","version":"0.1.0"},"isInternal":false,"tokens":816,"sizeBytes":3409},{"name":"commit.md","path":"src/openharness/skills/bundled/content/commit.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/OpenHarness/HEAD/src/openharness/skills/bundled/content/commit.md","title":"Content Skill","category":"anthropic-skill","format":"markdown","content":"# commit\n\nCreate clean, well-structured git commits.\n\n## When to use\n\nUse when the user asks to commit changes, create a PR, or prepare code for review.\n\n## Workflow\n\n1. Run `git status` and `git diff` to understand all changes\n2. Analyze changes: categorize as feature, fix, refactor, docs, test, etc.\n3. Draft a concise commit message:\n   - First line: imperative mood, under 72 chars, describes the \"why\"\n   - Body (if needed): explain context, trade-offs, or breaking changes\n4. Stage only relevant files — never stage .env, credentials, or large binaries\n5. Create the commit\n\n## Rules\n\n- Prefer specific `git add <file>` over `git add -A`\n- Never use `--no-verify` unless explicitly asked\n- Never amend published commits unless explicitly asked\n- If a pre-commit hook fails, fix the issue and create a NEW commit (don't --amend)\n- Include `Co-Authored-By` if pair programming\n","isInternal":false,"tokens":222,"sizeBytes":884},{"name":"debug.md","path":"src/openharness/skills/bundled/content/debug.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/OpenHarness/HEAD/src/openharness/skills/bundled/content/debug.md","title":"Content Skill","category":"anthropic-skill","format":"markdown","content":"# debug\n\nDiagnose and fix bugs systematically.\n\n## When to use\n\nUse when the user reports a bug, error, or unexpected behavior.\n\n## Workflow\n\n1. Reproduce: understand the exact steps that trigger the issue\n2. Read the error: stack traces, log messages, error codes\n3. Locate: use Grep/Read to find the relevant code path\n4. Hypothesize: form a theory about the root cause\n5. Verify: add logging or read surrounding code to confirm\n6. Fix: make the minimal change that addresses the root cause\n7. Test: verify the fix works and doesn't break other things\n\n## Rules\n\n- Read the error message carefully before searching code\n- Don't guess — verify your hypothesis before changing code\n- Fix the root cause, not the symptom\n- Don't retry the same approach if it failed — investigate why\n- If stuck after 3 attempts, explain what you've tried and ask for help\n","isInternal":false,"tokens":201,"sizeBytes":859},{"name":"diagnose.md","path":"src/openharness/skills/bundled/content/diagnose.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/OpenHarness/HEAD/src/openharness/skills/bundled/content/diagnose.md","title":"Content Skill","category":"anthropic-skill","format":"markdown","content":"# diagnose\n\nDiagnose why an agent run failed, regressed, or produced unexpected output — using structured evidence instead of intuition.\n\n## When to use\n\nUse when the user asks:\n- \"Why did this run fail?\"\n- \"What changed between this run and the last one?\"\n- \"The output looks wrong — what happened?\"\n- \"Why is this run worse than before?\"\n\n## Workflow\n\n1. **Locate the run artifacts** — check `artifacts/runs/<run_id>/` or the latest run directory for:\n   - `manifest.json` — run metadata, task type, input hash, timestamps\n   - `execution_trace.jsonl` — the full tool-call chain, one event per line\n   - `verification_report.json` — what was verified and whether it passed\n   - `failure_signature.json` — which stage failed and why (if present)\n2. **Read the failure signature first** — if it exists, it already localizes the failure to a specific stage (routing, execution, verification, or governance)\n3. **Trace the execution chain** — walk `execution_trace.jsonl` forward to find where output diverged from expectation\n4. **Compare with a passing run** — if a previous run succeeded, diff the two manifests and traces to identify what changed\n5. **Identify the failure layer**:\n   - **Routing**: wrong task type or profile selected\n   - **Execution**: tool call error, permission block, or unexpected result\n   - **Verification**: output produced but did not match expected artifact\n   - **Governance**: run was halted by a safety or boundary check\n6. **Report with evidence** — cite specific file paths, line numbers in the trace, or field values from the manifest; never summarize without pointing to the source\n\n## Rules\n\n- Evidence before intuition: always read the trace before forming a hypothesis\n- Localize before explaining: identify which stage failed before describing why\n- Compare, don't guess: if a previous run succeeded, diff it — don't speculate about what changed\n- If no artifacts exist, say so clearly and ask the user to re-run with archive mode enabled\n","isInternal":false,"tokens":444,"sizeBytes":2005},{"name":"plan.md","path":"src/openharness/skills/bundled/content/plan.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/OpenHarness/HEAD/src/openharness/skills/bundled/content/plan.md","title":"Content Skill","category":"anthropic-skill","format":"markdown","content":"# plan\n\nDesign an implementation plan before coding.\n\n## When to use\n\nUse when the user asks to plan, design, or architect a feature before implementing it.\n\n## Workflow\n\n1. Understand the requirement:\n   - What problem does this solve?\n   - What are the constraints?\n   - What's the expected outcome?\n2. Explore the codebase to find:\n   - Existing functions/utilities that can be reused\n   - Patterns used elsewhere in the project\n   - Files that will need modification\n3. Design the approach:\n   - Break into discrete steps\n   - Identify dependencies between steps\n   - Consider edge cases and error handling\n4. Present the plan:\n   - Start with a Context section (why this change)\n   - List concrete steps with file paths\n   - Include a verification section (how to test)\n\n## Rules\n\n- Read before suggesting — never propose changes to code you haven't read\n- Prefer editing existing files over creating new ones\n- Reuse existing patterns and utilities\n- Include file paths and line numbers when referencing code\n- Don't over-plan: match complexity to the task\n","isInternal":false,"tokens":228,"sizeBytes":1065},{"name":"review.md","path":"src/openharness/skills/bundled/content/review.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/OpenHarness/HEAD/src/openharness/skills/bundled/content/review.md","title":"Content Skill","category":"anthropic-skill","format":"markdown","content":"# review\n\nReview code for bugs, security issues, and quality.\n\n## When to use\n\nUse when the user asks to review code, a PR, or a diff.\n\n## Workflow\n\n1. Read the changed files or diff thoroughly\n2. Check for:\n   - **Bugs**: logic errors, off-by-one, null/undefined access, race conditions\n   - **Security**: injection, XSS, hardcoded secrets, path traversal\n   - **Performance**: N+1 queries, unnecessary allocations, missing indexes\n   - **Tests**: are new code paths covered? Are edge cases tested?\n   - **Style**: naming consistency, dead code, unnecessary complexity\n3. Provide concrete, actionable feedback with file:line references\n4. Prioritize findings by severity (critical > major > minor > nit)\n\n## Rules\n\n- Be specific: \"line 42 may throw if `user` is null\" not \"check for null\"\n- Suggest fixes, don't just point out problems\n- Acknowledge good patterns too\n- Don't nitpick formatting if there's a linter\n","isInternal":false,"tokens":222,"sizeBytes":916},{"name":"simplify.md","path":"src/openharness/skills/bundled/content/simplify.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/OpenHarness/HEAD/src/openharness/skills/bundled/content/simplify.md","title":"Content Skill","category":"anthropic-skill","format":"markdown","content":"# simplify\n\nRefactor code to be simpler and more maintainable.\n\n## When to use\n\nUse when the user asks to simplify, clean up, or refactor code.\n\n## Workflow\n\n1. Read the target code completely\n2. Identify complexity sources:\n   - Unnecessary abstractions or indirection\n   - Duplicated logic that should be consolidated\n   - Over-engineered patterns (factories, builders, strategies for simple cases)\n   - Dead code paths or unused variables\n3. Propose changes that reduce line count while preserving behavior\n4. Verify the refactored code passes existing tests\n\n## Rules\n\n- Don't add features or change behavior — simplification only\n- Prefer deleting code over refactoring it\n- Three similar lines are better than a premature abstraction\n- Remove backwards-compatibility shims for removed features\n- Don't add comments to explain simple code\n","isInternal":false,"tokens":179,"sizeBytes":846},{"name":"skill-creator.md","path":"src/openharness/skills/bundled/content/skill-creator.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/OpenHarness/HEAD/src/openharness/skills/bundled/content/skill-creator.md","title":"skill-creator","category":"anthropic-skill","format":"markdown","content":"---\nname: skill-creator\ndescription: >\n  Create, improve, and verify OpenHarness skills. Use this whenever the user\n  asks to create a new skill, convert a workflow into a skill, update an\n  existing SKILL.md, add skills for oh/ohmo, design skill trigger behavior,\n  or test whether a skill loads and works correctly.\n---\n\n# skill-creator\n\nCreate or improve OpenHarness skills in the directory-based `SKILL.md` format.\n\n## When to use\n\nUse this skill when the user wants to:\n\n- create a new skill from a workflow, repeated task, domain guide, or tool process\n- convert existing notes, prompts, or instructions into a reusable skill\n- modify or harden an existing skill\n- make a skill available to `oh`, `ohmo`, or a plugin\n- debug why the `skill` tool cannot find or load a skill\n- test trigger wording, skill metadata, or skill behavior\n\n## OpenHarness skill locations\n\nChoose the target deliberately:\n\n- Built-in OpenHarness skills live in `src/openharness/skills/bundled/content/*.md`.\n- User skills for `oh` live in `~/.openharness/skills/<skill-dir>/SKILL.md`.\n- Private `ohmo` skills live in `~/.ohmo/skills/<skill-dir>/SKILL.md`.\n- Plugin skills live in `<plugin-root>/skills/<skill-dir>/SKILL.md`.\n\nOpenHarness user and plugin skills use a directory layout. Do not create flat\n`*.md` user skills under `skills/`; they will not be loaded by the normal loader.\n\n## Skill anatomy\n\nUse this structure for user, ohmo, and plugin skills:\n\n```text\nmy-skill/\n├── SKILL.md\n├── scripts/      # optional deterministic helpers\n├── references/   # optional long docs loaded only when relevant\n└── assets/       # optional templates or static files\n```\n\nUse this structure inside `SKILL.md`:\n\n```markdown\n---\nname: my-skill\ndescription: >\n  What the skill does and exactly when to use it.\n---\n\n# my-skill\n\nShort purpose statement.\n\n## When to use\n\nTrigger contexts in plain language.\n\n## Workflow\n\nConcrete steps the agent should follow.\n\n## Rules\n\nImportant boundaries, verification, and failure handling.\n```\n\n## Workflow\n\n1. Capture intent before writing.\n   Ask or infer what the skill should help with, when it should trigger, what\n   output it should produce, and what success looks like. If the conversation\n   already contains the workflow, extract the sequence from history before\n   asking more questions.\n\n2. Inspect existing examples.\n   Read nearby skills and loader behavior before choosing a layout. For\n   OpenHarness, confirm whether the target is bundled, user, ohmo-private, or\n   plugin-provided.\n\n3. Write the metadata first.\n   The `description` is the primary trigger hint shown in the available skills\n   list. Make it specific and slightly assertive: include both what the skill\n   does and the phrases or situations that should trigger it. Avoid vague\n   descriptions like \"Helpful notes for X\".\n\n4. Keep the body procedural.\n   Use concise imperative steps. Explain why non-obvious constraints matter.\n   Prefer general principles over overfitted examples. If a skill is getting\n   long, move details into `references/` and point to the relevant file.\n\n5. Add deterministic resources only when they pay for themselves.\n   If the skill repeatedly needs the same script, template, schema, or checklist,\n   place it under `scripts/`, `assets/`, or `references/` rather than making the\n   model regenerate it every time.\n\n6. Verify loading.\n   Run a local loader check for the chosen target. For bundled skills, use:\n\n   ```bash\n   PYTHONPATH=src python - <<'PY'\n   from openharness.skills import load_skill_registry\n   for skill in load_skill_registry().list_skills():\n       print(skill.name, skill.source, skill.path)\n   PY\n   ```\n\n   For ohmo-private skills, include the extra skill directory:\n\n   ```bash\n   PYTHONPATH=src python - <<'PY'\n   from openharness.skills import load_skill_registry\n   from ohmo.workspace import get_skills_dir\n   for skill in load_skill_registry(extra_skill_dirs=[get_skills_dir()]).list_skills():\n       print(skill.name, skill.source, skill.path)\n   PY\n   ```\n\n7. Test behavior.\n   Create two or three realistic prompts that should use the skill. If the skill\n   changes code or files, add regression tests around loading or command\n   behavior. Run targeted tests first, then broader tests if the loader or\n   runtime path changed.\n\n8. Iterate from evidence.\n   If a skill under-triggers, improve the frontmatter description. If it\n   triggers but produces poor work, improve the body workflow. If agents repeat\n   deterministic work, add a helper script or reference file.\n\n## Writing rules\n\n- Preserve the skill name when updating an existing skill unless the user\n  explicitly asks to rename it.\n- Do not overwrite user-created skills without reading the existing `SKILL.md`.\n- Keep `SKILL.md` under roughly 500 lines; use `references/` for long material.\n- Keep trigger guidance in frontmatter `description`, not buried only in the\n  body.\n- Make skills safe and unsurprising. Do not create skills that hide behavior,\n  bypass permissions, exfiltrate data, or encourage unauthorized access.\n- Prefer directory names that are lowercase kebab-case, even if the displayed\n  skill `name` is more human-readable.\n- Mention exact install or verification paths in the final response.\n\n## Test prompts\n\nAfter drafting a skill, propose a small eval set:\n\n```json\n{\n  \"skill_name\": \"my-skill\",\n  \"evals\": [\n    {\n      \"id\": \"happy-path\",\n      \"prompt\": \"A realistic user request that should trigger the skill.\",\n      \"expected_output\": \"What good behavior looks like.\"\n    },\n    {\n      \"id\": \"near-miss\",\n      \"prompt\": \"A similar request that should not trigger this skill.\",\n      \"expected_output\": \"The agent should choose a different path.\"\n    }\n  ]\n}\n```\n\nFor objective skills, add assertions or tests. For subjective skills, collect\nhuman feedback and revise the workflow from concrete complaints rather than\nadding rigid rules.\n","frontmatter":{"name":"skill-creator","description":"Create, improve, and verify OpenHarness skills. Use this whenever the user asks to create a new skill, convert a workflow into a skill, update an existing SKILL.md, add skills for oh/ohmo, design skill trigger behavior, or test whether a skill loads and works correctly.\n"},"isInternal":false,"tokens":1341,"sizeBytes":5914},{"name":"test.md","path":"src/openharness/skills/bundled/content/test.md","rawUrl":"https://raw.githubusercontent.com/HKUDS/OpenHarness/HEAD/src/openharness/skills/bundled/content/test.md","title":"Content Skill","category":"anthropic-skill","format":"markdown","content":"# test\n\nWrite and run tests for code.\n\n## When to use\n\nUse when the user asks to write tests, verify behavior, or improve test coverage.\n\n## Workflow\n\n1. Understand what needs testing:\n   - New feature? Write unit tests for the happy path and edge cases\n   - Bug fix? Write a regression test that would have caught the bug\n   - Refactor? Ensure existing tests still pass\n2. Follow the project's testing patterns:\n   - Check existing tests for framework (pytest, jest, go test, etc.)\n   - Match naming conventions and file organization\n   - Use the same fixtures and helpers\n3. Write tests that are:\n   - Independent: each test can run alone\n   - Deterministic: same result every time\n   - Fast: mock external services, use in-memory databases\n4. Run the tests and verify they pass\n\n## Rules\n\n- Test behavior, not implementation details\n- One assertion per test when possible\n- Use descriptive test names that explain the scenario\n- Don't test framework or library code\n- Mock at system boundaries (external APIs, filesystem, network)\n","isInternal":false,"tokens":230,"sizeBytes":1034}],"systemPromptSnippet":"<yakaai_skills repo=\"HKUDS/OpenHarness\">\n<!-- File: .claude/skills/harness-eval/references/feature-matrix.md (Tokens: ~521 | Category: anthropic-skill) -->\n# Feature Test Matrix — Detailed Test Cases\n\n## Engine & Tools\n\n| Test | What to Verify | Key Assertion |\n|------|---------------|---------------|\n| Multi-turn memory | Set fact turn 1, recall turn 3 | Fact appears in turn 3 response |\n| Tool chaining | glob → grep → read in one task | All 3 tools in tool list |\n| Write→Edit→Read | Create, modify, verify file | File content matches expected |\n| Parallel tools | 3+ tool calls in one response | 3+ tools in single turn |\n| Error recovery | Tool fails, model adapts | Alternative tool used after error |\n| Auto-compaction | 5+ tasks on shared engine | No context overflow crash |\n\n## Swarm & Coordinator\n\n| Test | What to Verify | Key Assertion |\n|------|---------------|---------------|\n| InProcessBackend | spawn → active → status → shutdown | All states transition correctly |\n| Concurrent teammates | 2+ agents running simultaneously | Both complete, total time < 2x single |\n| Coordinator + notifications | Multi-turn delegation with XML | Coordinator synthesizes worker results |\n| Permission sync | request → pending → resolve | Pending count goes to 0 after resolve |\n\n## Hooks, Skills, Plugins\n\n| Test | What to Verify | Key Assertion |\n|------|---------------|---------------|\n| Hook blocks → adapt | pre_tool_use blocks bash | \"bash\" in errors, \"glob\" in tools |\n| Skill invocation | Model calls skill tool | \"skill\" in tools, content drives next action |\n| Plugin skill | Plugin provides skill | Loaded via skill tool, model follows it |\n| Hook + skill combined | Hook gates writes, skill guides | Protected file untouched, new file created |\n\n## Memory, Session, Config\n\n| Test | What to Verify | Key Assertion |\n|------|---------------|---------------|\n| Memory frontmatter | YAML parsed, not \"---\" | description != \"---\", body searchable |\n| Session resume | Save → load → continue | Model remembers prior context |\n| Cost tracking | Tokens accumulate | in_tokens strictly increasing |\n| Cron CRUD | Create, toggle, mark_run, delete | Job count correct at each step |\n\n## Provider Compatibility\n\n| Test | What to Verify | Key Assertion |\n|------|---------------|---------------|\n| Anthropic client | Standard tool calling | Tools execute, response coherent |\n| OpenAI client | Tool calling + reasoning_content | No 400 error on tool call round-trip |\n| OpenAI multi-turn | reasoning_content persists | 3+ turns without API error |\n\n<!-- File: .claude/skills/harness-eval/references/test-patterns.md (Tokens: ~1422 | Category: anthropic-skill) -->\n# Test Patterns — Code Templates for Harness Eval\n\n## Engine Setup Helpers\n\n```python\nimport asyncio, sys, os\nfrom pathlib import Path\nsys.path.insert(0, \"src\")\n\nAPI_KEY = os.environ.get(\"ANTHROPIC_API_KEY\", \"your-key\")\nBASE_URL = os.environ.get(\"ANTHROPIC_BASE_URL\", \"https://api.moonshot.cn/anthropic\")\nOPENAI_BASE = os.environ.get(\"OPENAI_BASE_URL\", \"https://api.moonshot.cn/v1\")\nMODEL = os.environ.get(\"ANTHROPIC_MODEL\", \"kimi-k2.5\")\nWORKSPACE = Path(\"/tmp/eval-workspace\")  # unfamiliar project\n\n\ndef make_anthropic_engine(system_prompt, cwd=None, extra_tools=None):\n    from openharness.api.client import AnthropicApiClient\n    from openharness.config.settings import PermissionSettings\n    from openharness.engine.query_engine import QueryEngine\n    from openharness.permissions.checker import PermissionChecker\n    from openharness.permissions.modes import PermissionMode\n    from openharness.tools.base import ToolRegistry\n    from openharness.tools.bash_tool import BashTool\n    from openharness.tools.file_read_tool import FileReadTool\n    from openharness.tools.file_write_tool import FileWriteTool\n    from openharness.tools.file_edit_tool import FileEditTool\n    from openharness.tools.glob_tool import GlobTool\n    from openharness.tools.grep_tool import GrepTool\n\n    api = AnthropicApiClient(api_key=API_KEY, base_url=BASE_URL)\n    reg = ToolRegistry()\n    for t in [BashTool(), FileReadTool(), FileWriteTool(), FileEditTool(), GlobTool(), GrepTool()]:\n        reg.register(t)\n    for t in (extra_tools or []):\n        reg.register(t)\n    checker = PermissionChecker(PermissionSettings(mode=PermissionMode.FULL_AUTO))\n    return QueryEngine(\n        api_client=api, tool_registry=reg, permission_checker=checker,\n        cwd=Path(cwd or WORKSPACE), model=MODEL, system_prompt=system_prompt, max_tokens=4096,\n    )\n\n\ndef make_openai_engine(system_prompt, cwd=None, extra_tools=None):\n    from openharness.api.openai_client import OpenAICompatibleClient\n    # Same structure as above, but with:\n    api = OpenAICompatibleClient(api_key=API_KEY, base_url=OPENAI_BASE)\n    # ... rest identical\n\n\ndef collect(events):\n    from openharness.engine.stream_events import (\n        AssistantTextDelta, AssistantTurnComplete,\n        ToolExecutionStarted, ToolExecutionCompleted,\n    )\n    r = {\"text\": \"\", \"tools\": [], \"turns\": 0, \"in_tok\": 0, \"out_tok\": 0}\n    for ev in events:\n        if isinstance(ev, AssistantTextDelta):\n            r[\"text\"] += ev.text\n        elif isinstance(ev, ToolExecutionStarted):\n            r[\"tools\"].append(ev.tool_name)\n        elif isinstance(ev, AssistantTurnComplete):\n            r[\"turns\"] += 1\n            r[\"in_tok\"] += ev.usage.input_tokens\n            r[\"out_tok\"] += ev.usage.output_tokens\n    return r\n```\n\n## Multi-Turn Memory Test\n\n```python\nasync def test_multi_turn_memory():\n    engine = make_anthropic_engine(\"Remember what the user tells you.\")\n    [ev async for ev in engine.submit_message(\"My favorite number is 42.\")]\n    [ev async for ev in engine.submit_message(\"What is 2+2?\")]\n    evs = [ev async for ev in engine.submit_message(\"What is my favorite number?\")]\n    r = collect(evs)\n    assert \"42\" in r[\"text\"]\n```\n\n## Hook Blocks Tool → Model Adapts\n\n```python\nasync def test_hook_blocks():\n    from openharness.hooks.events import HookEvent\n    from openharness.hooks.loader import HookRegistry\n    from openharness.hooks.schemas import CommandHookDefinition\n    from openharness.hooks.executor import HookExecutor, HookExecutionContext\n\n    hook_reg = HookRegistry()\n    hook_reg.register(HookEvent.PRE_TOOL_USE, CommandHookDefinition(\n        type=\"command\", command=\"exit 1\",\n        matcher=\"bash\", block_on_failure=True, timeout_seconds=5,\n    ))\n    # ... create engine with hook_executor, model tries bash, gets blocked, adapts to glob\n```\n\n## Skill Tool Invocation\n\n```python\nasync def test_skill_invocation():\n    from openharness.tools.skill_tool import SkillTool\n    engine = make_anthropic_engine(\n        \"Use the 'skill' tool to load instructions before working.\",\n        extra_tools=[SkillTool()],\n    )\n    evs = [ev async for ev in engine.submit_message(\n        \"Load the 'diagnose' skill, then investigate the codebase.\"\n    )]\n    r = collect(evs)\n    assert \"skill\" in r[\"tools\"]\n```\n\n## InProcess Concurrent Teammates\n\n```python\nasync def test_concurrent_teammates():\n    from openharness.swarm.in_process import start_in_process_teammate, TeammateAbortController\n    from openharness.swarm.types import TeammateSpawnConfig\n    from openharness.engine.query import QueryContext\n\n    async def run_one(name, prompt):\n        ctx = QueryContext(api_client=api, tool_registry=reg, ...)\n        config = TeammateSpawnConfig(name=name, team=\"test\", prompt=prompt, ...)\n        abort = TeammateAbortController()\n        await start_in_process_teammate(config=config, agent_id=f\"{name}@test\", abort_controller=abort, query_context=ctx)\n\n    await asyncio.gather(\n        asyncio.wait_for(run_one(\"worker-a\", \"Count .py files\"), timeout=30),\n        asyncio.wait_for(run_one(\"worker-b\", \"Find main class\"), timeout=30),\n    )\n```\n\n## Session Save → Resume\n\n```python\nasync def test_session_resume():\n    from openharness.services.session_storage import save_session_snapshot, load_session_snapshot\n    from openharness.engine.messages import ConversationMessage\n\n    engine1 = make_anthropic_engine(\"Remember context.\")\n    [ev async for ev in engine1.submit_message(\"Project uses FastAPI + React.\")]\n    save_session_snapshot(cwd=tmpdir, model=MODEL, system_prompt=\"...\", messages=engine1.messages, usage=engine1.total_usage)\n\n    loaded = load_session_snapshot(tmpdir)\n    engine2 = make_anthropic_engine(\"Continue analysis.\")\n    engine2.load_messages([ConversationMessage.model_validate(m) for m in loaded[\"messages\"]])\n    evs = [ev async for ev in engine2.submit_message(\"What tech stack did I mention?\")]\n    assert \"fastapi\" in collect(evs)[\"text\"].lower()\n```\n\n<!-- File: .claude/skills/harness-eval/SKILL.md (Tokens: ~1978 | Category: anthropic-skill) -->\n---\nname: harness-eval\ndescription: This skill should be used when the user asks to \"test the harness\", \"run integration tests\", \"validate features with real API\", \"test with real model calls\", \"run agent loop tests\", \"verify end-to-end\", or needs to verify OpenHarness features on a real codebase with actual LLM calls.\nversion: 0.2.0\n---\n\n# Harness Eval — End-to-End Feature Validation\n\nValidate OpenHarness features by running real agent loops against an unfamiliar codebase with actual LLM API calls. Every test exercises the full stack: API client → model → tool calls → execution → result.\n\n## Core Principles\n\n1. **Test on an unfamiliar project** — never test on OpenHarness itself (the agent modifies its own code). Clone a real project as the workspace.\n2. **Use real API calls** — no mocks. Configure a real LLM endpoint.\n3. **Multi-turn conversations** — always test 2+ turns where the model needs prior context.\n4. **Combine features** — test hooks+skills+agent loop together, not in isolation.\n5. **Verify tool execution** — inspect tool call lists and output files, not just model text.\n\n## Workflow\n\n### 1. Prepare Workspace\n\nClone an unfamiliar project (do not use OpenHarness):\n\n```bash\ngit clone https://github.com/HKUDS/AutoAgent /tmp/eval-workspace\n```\n\n### 2. Configure Environment\n\n```bash\nexport ANTHROPIC_API_KEY=sk-xxx\nexport ANTHROPIC_BASE_URL=https://api.moonshot.cn/anthropic  # or any provider\nexport ANTHROPIC_MODEL=kimi-k2.5\n```\n\nFor long-running real evals, do not artificially lower `max_turns`. Use the product default (`200`) unless the user explicitly wants a tighter bound.\n\n### 3. Prepare Real Sandbox Runtime When Relevant\n\nIf the task is validating sandbox behavior, install and verify the actual runtime before running agent loops:\n\n```bash\nnpm install -g @anthropic-ai/sandbox-runtime\nsudo apt-get update\nsudo apt-get install -y bubblewrap ripgrep\nwhich srt\nwhich bwrap\nwhich rg\nsrt --version\n```\n\nThen run a minimal smoke check through OpenHarness, not just raw `srt`, so you verify the real adapter path:\n\n```python\nfrom pathlib import Path\nfrom openharness.config.settings import Settings, SandboxSettings, save_settings\nfrom openharness.tools.bash_tool import BashTool\n\ncfg = Path(\"/tmp/openharness-sandbox-settings.json\")\nsave_settings(Settings(sandbox=SandboxSettings(enabled=True, fail_if_unavailable=True)), cfg)\n# Point config loader at this file, then run BashTool on a tiny command such as `pwd`.\n```\n\nIf sandbox dependencies are missing, treat that as an environment/setup failure, not a feature regression.\n\n### 4. Design Tests\n\nEach test follows this pattern:\n\n```python\nengine = make_engine(system_prompt=\"...\", cwd=UNFAMILIAR_PROJECT)\nevs1 = [ev async for ev in engine.submit_message(\"Read X, analyze Y\")]\nr1 = collect(evs1)  # text, tools, turns, tokens\nevs2 = [ev async for ev in engine.submit_message(\"Based on what you found...\")]\nr2 = collect(evs2)\nassert \"grep\" in r1[\"tools\"]  # verify tools ran\n```\n\nFor detailed code templates and the `make_engine`/`collect` helpers, consult `references/test-patterns.md`.\n\n### 5. Prefer Long-Horizon, Real Agent Loops\n\nFor meaningful end-to-end validation, prefer unfamiliar-repo tasks that force multiple turns, context reuse, and mixed tool usage.\n\nRecommended pattern:\n\n- Use a real external workspace such as `AutoAgent`\n- Use real provider credentials and the actual target model\n- Keep `max_turns=200`\n- Use per-prompt timeouts large enough for real exploration, such as `240-600s`\n- Require at least 2 turns per scenario\n- Verify both text quality and tool traces\n- Keep polling long-running sessions until they finish; do not abandon a run after the first long pause\n\nRecommended long-horizon scenarios:\n\n- `architecture_multiturn`\n  - Turn 1: map architecture, shell/subprocess surfaces, and test entrypoints\n  - Turn 2: identify top risks and propose refactors\n  - Turn 3: condense into onboarding or remediation actions\n  - Success: `bash`, `glob`, `grep`, `read_file` all appear; no timeout; no `MaxTurnsExceeded`\n\n- `hook_block_and_recover`\n  - Force the model to try `bash`\n  - Block it with a real pre-tool hook\n  - Verify the model adapts with `glob`/`grep`/`read_file`\n\n- `sandbox_multiturn`\n  - Enable real sandbox settings with `fail_if_unavailable=true`\n  - First prompt must start with exactly one shell command such as `pwd && ls -la`\n  - Second prompt must explicitly reuse the prior shell findings\n  - Success: `bash` executes via sandbox, non-shell tools continue the task, and the agent recovers from incidental repo errors\n\nWhen a scenario fails, classify it before changing code:\n\n- `MaxTurnsExceeded`: likely eval harness misconfiguration if `max_turns` was manually lowered\n- `timeout`: task is too broad or per-prompt timeout is too small\n- sandbox unavailable: environment missing `srt`, `bwrap`, or `rg`\n- tool error with task still completed: feature may still be healthy; inspect recovery behavior\n\n### 6. Run Tests\n\n```bash\npython tests/test_merged_prs_on_autoagent.py   # PR feature tests\npython tests/test_real_large_tasks.py           # large multi-step tasks\npython tests/test_hooks_skills_plugins_real.py  # hooks/skills/plugins\npython -m pytest tests/ -q -k \"not autoagent\"  # unit tests (no API)\n```\n\nFor ad hoc long-horizon validation, it is acceptable to run a temporary Python driver script as long as it:\n\n- uses real OpenHarness engine/tool objects\n- targets an unfamiliar repository\n- prints per-scenario JSON summaries\n- records tools, errors, turns, and token usage\n- stays attached until completion\n\n### 7. Interpret Results\n\n| Result | Meaning | Action |\n|--------|---------|--------|\n| PASS with tool calls | Feature works end-to-end | Done |\n| PASS without tool calls | Model answered from knowledge | Rewrite prompt to force tool use |\n| FAIL with exception | Code bug | Read traceback |\n| FAIL with wrong output | Model behavior issue | Check system prompt and tool schemas |\n| Timeout | Task too complex | Increase `max_turns` or simplify prompt |\n\nFor long-running real evals, refine the timeout guidance:\n\n- First check whether `max_turns` was manually set too low\n- If `max_turns=200` and the run still fails, the next suspect is wall-clock timeout, not turn count\n- Distinguish environment failures from product failures\n  - Example: missing dependency in the unfamiliar target repo is not automatically an OpenHarness regression\n  - Example: missing `srt`/`bwrap`/`rg` is an eval environment issue\n\n## Feature Coverage Checklist\n\n- [ ] Engine: multi-turn memory, tool chaining, parallel tools, error recovery, auto-compaction\n- [ ] Swarm: InProcessBackend lifecycle, concurrent teammates, coordinator+notifications\n- [ ] Hooks: pre_tool_use blocking → model adapts, post_tool_use firing\n- [ ] Skills: skill tool invocation → model follows instructions\n- [ ] Plugins: plugin-provided skill loaded and used in agent loop\n- [ ] Memory: YAML frontmatter parsing, body content search, context injection\n- [ ] Session: save → load → resume with context preserved\n- [ ] Providers: Anthropic client, OpenAI client (with reasoning_content), multi-turn\n- [ ] Cost: token accumulation across turns\n\n## Common Pitfalls\n\n- Testing on OpenHarness itself — agent modifies its own running code\n- Using mocks — misses serialization and API compatibility bugs\n- Single-turn only — misses context accumulation and compaction bugs\n- Artificially lowering `max_turns` during real evals — can create false failures that do not reflect product defaults\n- Not checking tool call list — model may claim tool use without calling it\n- Hardcoding paths — use `WORKSPACE` variable, skip in CI with `pytest.mark.skipif`\n- Declaring sandbox “tested” after only checking raw `srt` — verify the OpenHarness adapter path too\n- Abandoning long tasks too early — some real tasks pause for minutes before the next event arrives\n\n## Additional Resources\n\n### Reference Files\n\n- **`references/test-patterns.md`** — Complete code templates for `make_engine`, `collect`, and each feature category\n- **`references/feature-matrix.md`** — Detailed test cases for every OpenHarness module\n\n### Existing Test Files\n\nWorking test suites in the repo:\n- `tests/test_merged_prs_on_autoagent.py` — PR feature validation\n- `tests/test_real_large_tasks.py` — Large multi-step tasks\n- `tests/test_hooks_skills_plugins_real.py` — Hooks/skills/plugins in agent loops\n- `tests/test_untested_features.py` — Module-level integration tests\n\n<!-- File: .claude/skills/pr-merge/references/merge-scenarios.md (Tokens: ~990 | Category: anthropic-skill) -->\n# Merge Scenarios — Detailed Examples\n\n## Scenario 1: Clean Merge (No Conflicts)\n\nPR adds a new feature, no files overlap with local changes.\n\n```bash\n# Preferred: squash merge via GitHub (preserves author)\ngh pr merge 17 --repo HKUDS/OpenHarness --squash \\\n  --subject \"feat(skills): add diagnose skill (#17)\"\n\n# Result: author \"Qu Zhi\" appears in git log\n```\n\n## Scenario 2: CHANGELOG Conflict Only\n\nAlmost every PR touches CHANGELOG.md. Handle by excluding it:\n\n```bash\ngh pr diff 14 --repo OWNER/REPO | \\\n  git apply --exclude='CHANGELOG.md'\ngit add -A\n\n# Manually merge CHANGELOG sections (keep both)\n# Then commit with original author\ngit commit --author=\"washi4 <washi4@users.noreply.github.com>\" \\\n  -m \"feat: add OpenAI-compatible client (#14)\"\n```\n\n## Scenario 3: Conflicting UI Files\n\nPR #14 modified ui/app.py and ui/backend_host.py which were also changed by PR #13.\n\n```bash\n# Apply excluding conflicting files\ngh pr diff 14 --repo OWNER/REPO | \\\n  git apply --exclude='src/ui/app.py' --exclude='src/ui/backend_host.py' --exclude='CHANGELOG.md'\n\n# Manually add the PR's changes to conflicting files\n# Read the PR diff to understand what they added, apply by hand\n# Commit with author\ngit commit --author=\"Author <email>\" -m \"feat: description (#14)\"\n```\n\n## Scenario 4: Selective Merge (Skip Features)\n\nPR #16 has auto-compact (already implemented locally) + --resume (wanted) + cron (wanted).\n\n```bash\n# Exclude the file containing the unwanted feature\ngh pr diff 16 --repo OWNER/REPO | git apply \\\n  --exclude='src/engine/query_engine.py' \\  # skip auto-compact\n  --exclude='CHANGELOG.md' \\\n  --exclude='README.md'\n\n# Commit with explanation\ngit commit --author=\"Chao Qin <win4r@users.noreply.github.com>\" \\\n  -m \"feat: wire --resume/--continue and cron scheduler (#16)\n\nCherry-picked from PR #16. Excluded auto-compact (already implemented\nwith LLM-based approach from reference source).\"\n```\n\n## Scenario 5: Duplicate PRs (Same Bug, Different Fix)\n\nPR #11 and #13 both fix the double-Enter bug. #13 is smaller and cleaner.\n\n```bash\n# Merge #13 (the better fix)\ngh pr merge 13 --repo OWNER/REPO --squash\n\n# Close #11 with acknowledgment\ngh pr close 11 --repo OWNER/REPO --comment \\\n  \"Fixed via PR #13 (smaller patch for the same bug). \\\nThank you for the detailed investigation!\"\n```\n\n## Scenario 6: PR From a Fork\n\nThe contributor pushed to their fork, not a branch on the repo.\n\n```bash\n# Fetch via PR ref (works for any PR regardless of source)\ngit fetch origin pull/14/head:pr-14\ngit merge pr-14 --no-edit\n\n# If merge conflict:\ngit mergetool  # or manually resolve\ngit add -A && git commit --no-edit\n\ngit push origin main\n```\n\n## Scenario 7: Batch Merge (Multiple PRs)\n\nMerge in dependency order, test after each:\n\n```bash\n# 1. Small fixes first (least risk)\ngh pr merge 17 --squash  # skill file\ngh pr merge 13 --squash  # 1-file bug fix\n\n# 2. Medium changes\ngh pr merge 12 --squash  # memory improvement\n\n# 3. Large features last (most conflict risk)\n# PR #14 needs manual conflict resolution\ngit fetch origin pull/14/head:pr-14\ngit merge pr-14\n# resolve CHANGELOG conflict\ngit push origin main\n\n# 4. Test after all merges\npython -m ruff check src/ tests/\npython -m pytest tests/ -q\n```\n\n## Post-Merge Fix Commit Pattern\n\nWhen the merged PR introduces a compatibility issue (e.g., API mismatch with a provider):\n\n```bash\n# Fix in a SEPARATE commit (don't amend the author's commit)\ngit commit -m \"fix(api): handle Kimi reasoning_content in OpenAI client\n\nKimi k2.5 requires reasoning_content on assistant tool_call messages.\nFix: capture during streaming, replay when converting back.\nFound during post-merge testing with harness-eval.\"\n```\n\nThis preserves the original author's commit intact while documenting the fix separately.\n\n<!-- File: .claude/skills/pr-merge/SKILL.md (Tokens: ~816 | Category: anthropic-skill) -->\n---\nname: pr-merge\ndescription: This skill should be used when the user asks to \"merge a PR\", \"review and merge pull requests\", \"integrate external contributions\", \"handle PR conflicts\", \"cherry-pick from a PR\", or needs to merge GitHub PRs while maximizing contributor attribution.\nversion: 0.1.0\n---\n\n# PR Merge — Contributor-First Pull Request Integration\n\nMerge external pull requests while maximizing original author attribution. Core principle: **merge first, resolve conflicts after** — never rewrite a contributor's work from scratch.\n\n## Core Principles\n\n1. **Preserve authorship** — use `gh pr merge --squash` for clean PRs. For manual merges, use `--author=\"Name <email>\"`.\n2. **Merge first, fix after** — accept the PR's approach even if it differs from local style. Fix conflicts in a separate commit.\n3. **Selective merge is OK** — exclude files with `--exclude` when a PR contains features already implemented locally. Document what was excluded.\n4. **Never `git apply` + self-commit** — this loses author attribution entirely.\n\n## Workflow\n\n### 1. Triage Open PRs\n\n```bash\ngh pr list --repo OWNER/REPO --state open \\\n  --json number,title,author,additions,deletions,mergeable\n```\n\nClassify each PR: merge directly, merge with conflict resolution, selective merge, or close.\n\n### 2. Merge Clean PRs via GitHub\n\nPrefer `gh pr merge` — preserves author automatically:\n\n```bash\ngh pr merge NUMBER --repo OWNER/REPO --squash \\\n  --subject \"feat: description (#NUMBER)\"\n```\n\n### 3. Handle Conflicting PRs Locally\n\n```bash\ngit fetch origin pull/NUMBER/head:pr-NUMBER\ngit merge pr-NUMBER --no-edit\n# Resolve conflicts keeping both sides where possible\ngit add -A && git commit --no-edit\n```\n\n### 4. Selective Merge (Skip Some Files)\n\nWhen a PR contains features already implemented locally:\n\n```bash\ngh pr diff NUMBER --repo OWNER/REPO | \\\n  git apply --exclude='path/to/skip.py' --exclude='CHANGELOG.md'\ngit commit --author=\"Author Name <author@email>\" \\\n  -m \"feat: description (#NUMBER)\n\nCherry-picked from PR #NUMBER. Excluded: file.py (already implemented).\"\n```\n\n### 5. Close Duplicate/Superseded PRs\n\n```bash\ngh pr close NUMBER --repo OWNER/REPO \\\n  --comment \"Fixed via PR #OTHER. Thank you for the contribution!\"\n```\n\n### 6. Post-Merge Verification\n\n```bash\npython -m ruff check src/ tests/    # lint\npython -m pytest tests/ -q          # unit tests\ngit push origin main                # push\n```\n\nThen run `harness-eval` for end-to-end verification on an unfamiliar codebase.\n\n## Attribution Checklist\n\nBefore pushing a merged PR:\n\n- [ ] Original author appears in `git log` (via `--author` or GitHub squash merge)\n- [ ] Commit message references PR number (`#NUMBER`)\n- [ ] If selectively merged, commit body explains exclusions\n- [ ] Closed PRs have a comment thanking the contributor\n- [ ] Duplicate PRs acknowledge the contributor's investigation\n\n## Common Pitfalls\n\n- `git apply` + self-commit loses author\n- Rewriting from scratch instead of merging — merge their code, fix style after\n- Force-pushing main after merge — may remove contributor commits\n- Forgetting CHANGELOG conflicts — always exclude and handle manually\n- Not testing after merge — clean merge doesn't mean working code\n\n## Additional Resources\n\n### Reference Files\n\n- **`references/merge-scenarios.md`** — Detailed examples for each merge scenario (clean, conflicting, selective, duplicate)\n\n<!-- File: src/openharness/skills/bundled/content/commit.md (Tokens: ~222 | Category: anthropic-skill) -->\n# commit\n\nCreate clean, well-structured git commits.\n\n## When to use\n\nUse when the user asks to commit changes, create a PR, or prepare code for review.\n\n## Workflow\n\n1. Run `git status` and `git diff` to understand all changes\n2. Analyze changes: categorize as feature, fix, refactor, docs, test, etc.\n3. Draft a concise commit message:\n   - First line: imperative mood, under 72 chars, describes the \"why\"\n   - Body (if needed): explain context, trade-offs, or breaking changes\n4. Stage only relevant files — never stage .env, credentials, or large binaries\n5. Create the commit\n\n## Rules\n\n- Prefer specific `git add <file>` over `git add -A`\n- Never use `--no-verify` unless explicitly asked\n- Never amend published commits unless explicitly asked\n- If a pre-commit hook fails, fix the issue and create a NEW commit (don't --amend)\n- Include `Co-Authored-By` if pair programming\n\n<!-- File: src/openharness/skills/bundled/content/debug.md (Tokens: ~201 | Category: anthropic-skill) -->\n# debug\n\nDiagnose and fix bugs systematically.\n\n## When to use\n\nUse when the user reports a bug, error, or unexpected behavior.\n\n## Workflow\n\n1. Reproduce: understand the exact steps that trigger the issue\n2. Read the error: stack traces, log messages, error codes\n3. Locate: use Grep/Read to find the relevant code path\n4. Hypothesize: form a theory about the root cause\n5. Verify: add logging or read surrounding code to confirm\n6. Fix: make the minimal change that addresses the root cause\n7. Test: verify the fix works and doesn't break other things\n\n## Rules\n\n- Read the error message carefully before searching code\n- Don't guess — verify your hypothesis before changing code\n- Fix the root cause, not the symptom\n- Don't retry the same approach if it failed — investigate why\n- If stuck after 3 attempts, explain what you've tried and ask for help\n\n<!-- File: src/openharness/skills/bundled/content/diagnose.md (Tokens: ~444 | Category: anthropic-skill) -->\n# diagnose\n\nDiagnose why an agent run failed, regressed, or produced unexpected output — using structured evidence instead of intuition.\n\n## When to use\n\nUse when the user asks:\n- \"Why did this run fail?\"\n- \"What changed between this run and the last one?\"\n- \"The output looks wrong — what happened?\"\n- \"Why is this run worse than before?\"\n\n## Workflow\n\n1. **Locate the run artifacts** — check `artifacts/runs/<run_id>/` or the latest run directory for:\n   - `manifest.json` — run metadata, task type, input hash, timestamps\n   - `execution_trace.jsonl` — the full tool-call chain, one event per line\n   - `verification_report.json` — what was verified and whether it passed\n   - `failure_signature.json` — which stage failed and why (if present)\n2. **Read the failure signature first** — if it exists, it already localizes the failure to a specific stage (routing, execution, verification, or governance)\n3. **Trace the execution chain** — walk `execution_trace.jsonl` forward to find where output diverged from expectation\n4. **Compare with a passing run** — if a previous run succeeded, diff the two manifests and traces to identify what changed\n5. **Identify the failure layer**:\n   - **Routing**: wrong task type or profile selected\n   - **Execution**: tool call error, permission block, or unexpected result\n   - **Verification**: output produced but did not match expected artifact\n   - **Governance**: run was halted by a safety or boundary check\n6. **Report with evidence** — cite specific file paths, line numbers in the trace, or field values from the manifest; never summarize without pointing to the source\n\n## Rules\n\n- Evidence before intuition: always read the trace before forming a hypothesis\n- Localize before explaining: identify which stage failed before describing why\n- Compare, don't guess: if a previous run succeeded, diff it — don't speculate about what changed\n- If no artifacts exist, say so clearly and ask the user to re-run with archive mode enabled\n\n<!-- File: src/openharness/skills/bundled/content/plan.md (Tokens: ~228 | Category: anthropic-skill) -->\n# plan\n\nDesign an implementation plan before coding.\n\n## When to use\n\nUse when the user asks to plan, design, or architect a feature before implementing it.\n\n## Workflow\n\n1. Understand the requirement:\n   - What problem does this solve?\n   - What are the constraints?\n   - What's the expected outcome?\n2. Explore the codebase to find:\n   - Existing functions/utilities that can be reused\n   - Patterns used elsewhere in the project\n   - Files that will need modification\n3. Design the approach:\n   - Break into discrete steps\n   - Identify dependencies between steps\n   - Consider edge cases and error handling\n4. Present the plan:\n   - Start with a Context section (why this change)\n   - List concrete steps with file paths\n   - Include a verification section (how to test)\n\n## Rules\n\n- Read before suggesting — never propose changes to code you haven't read\n- Prefer editing existing files over creating new ones\n- Reuse existing patterns and utilities\n- Include file paths and line numbers when referencing code\n- Don't over-plan: match complexity to the task\n\n<!-- File: src/openharness/skills/bundled/content/review.md (Tokens: ~222 | Category: anthropic-skill) -->\n# review\n\nReview code for bugs, security issues, and quality.\n\n## When to use\n\nUse when the user asks to review code, a PR, or a diff.\n\n## Workflow\n\n1. Read the changed files or diff thoroughly\n2. Check for:\n   - **Bugs**: logic errors, off-by-one, null/undefined access, race conditions\n   - **Security**: injection, XSS, hardcoded secrets, path traversal\n   - **Performance**: N+1 queries, unnecessary allocations, missing indexes\n   - **Tests**: are new code paths covered? Are edge cases tested?\n   - **Style**: naming consistency, dead code, unnecessary complexity\n3. Provide concrete, actionable feedback with file:line references\n4. Prioritize findings by severity (critical > major > minor > nit)\n\n## Rules\n\n- Be specific: \"line 42 may throw if `user` is null\" not \"check for null\"\n- Suggest fixes, don't just point out problems\n- Acknowledge good patterns too\n- Don't nitpick formatting if there's a linter\n\n<!-- File: src/openharness/skills/bundled/content/simplify.md (Tokens: ~179 | Category: anthropic-skill) -->\n# simplify\n\nRefactor code to be simpler and more maintainable.\n\n## When to use\n\nUse when the user asks to simplify, clean up, or refactor code.\n\n## Workflow\n\n1. Read the target code completely\n2. Identify complexity sources:\n   - Unnecessary abstractions or indirection\n   - Duplicated logic that should be consolidated\n   - Over-engineered patterns (factories, builders, strategies for simple cases)\n   - Dead code paths or unused variables\n3. Propose changes that reduce line count while preserving behavior\n4. Verify the refactored code passes existing tests\n\n## Rules\n\n- Don't add features or change behavior — simplification only\n- Prefer deleting code over refactoring it\n- Three similar lines are better than a premature abstraction\n- Remove backwards-compatibility shims for removed features\n- Don't add comments to explain simple code\n\n<!-- File: src/openharness/skills/bundled/content/skill-creator.md (Tokens: ~1341 | Category: anthropic-skill) -->\n---\nname: skill-creator\ndescription: >\n  Create, improve, and verify OpenHarness skills. Use this whenever the user\n  asks to create a new skill, convert a workflow into a skill, update an\n  existing SKILL.md, add skills for oh/ohmo, design skill trigger behavior,\n  or test whether a skill loads and works correctly.\n---\n\n# skill-creator\n\nCreate or improve OpenHarness skills in the directory-based `SKILL.md` format.\n\n## When to use\n\nUse this skill when the user wants to:\n\n- create a new skill from a workflow, repeated task, domain guide, or tool process\n- convert existing notes, prompts, or instructions into a reusable skill\n- modify or harden an existing skill\n- make a skill available to `oh`, `ohmo`, or a plugin\n- debug why the `skill` tool cannot find or load a skill\n- test trigger wording, skill metadata, or skill behavior\n\n## OpenHarness skill locations\n\nChoose the target deliberately:\n\n- Built-in OpenHarness skills live in `src/openharness/skills/bundled/content/*.md`.\n- User skills for `oh` live in `~/.openharness/skills/<skill-dir>/SKILL.md`.\n- Private `ohmo` skills live in `~/.ohmo/skills/<skill-dir>/SKILL.md`.\n- Plugin skills live in `<plugin-root>/skills/<skill-dir>/SKILL.md`.\n\nOpenHarness user and plugin skills use a directory layout. Do not create flat\n`*.md` user skills under `skills/`; they will not be loaded by the normal loader.\n\n## Skill anatomy\n\nUse this structure for user, ohmo, and plugin skills:\n\n```text\nmy-skill/\n├── SKILL.md\n├── scripts/      # optional deterministic helpers\n├── references/   # optional long docs loaded only when relevant\n└── assets/       # optional templates or static files\n```\n\nUse this structure inside `SKILL.md`:\n\n```markdown\n---\nname: my-skill\ndescription: >\n  What the skill does and exactly when to use it.\n---\n\n# my-skill\n\nShort purpose statement.\n\n## When to use\n\nTrigger contexts in plain language.\n\n## Workflow\n\nConcrete steps the agent should follow.\n\n## Rules\n\nImportant boundaries, verification, and failure handling.\n```\n\n## Workflow\n\n1. Capture intent before writing.\n   Ask or infer what the skill should help with, when it should trigger, what\n   output it should produce, and what success looks like. If the conversation\n   already contains the workflow, extract the sequence from history before\n   asking more questions.\n\n2. Inspect existing examples.\n   Read nearby skills and loader behavior before choosing a layout. For\n   OpenHarness, confirm whether the target is bundled, user, ohmo-private, or\n   plugin-provided.\n\n3. Write the metadata first.\n   The `description` is the primary trigger hint shown in the available skills\n   list. Make it specific and slightly assertive: include both what the skill\n   does and the phrases or situations that should trigger it. Avoid vague\n   descriptions like \"Helpful notes for X\".\n\n4. Keep the body procedural.\n   Use concise imperative steps. Explain why non-obvious constraints matter.\n   Prefer general principles over overfitted examples. If a skill is getting\n   long, move details into `references/` and point to the relevant file.\n\n5. Add deterministic resources only when they pay for themselves.\n   If the skill repeatedly needs the same script, template, schema, or checklist,\n   place it under `scripts/`, `assets/`, or `references/` rather than making the\n   model regenerate it every time.\n\n6. Verify loading.\n   Run a local loader check for the chosen target. For bundled skills, use:\n\n   ```bash\n   PYTHONPATH=src python - <<'PY'\n   from openharness.skills import load_skill_registry\n   for skill in load_skill_registry().list_skills():\n       print(skill.name, skill.source, skill.path)\n   PY\n   ```\n\n   For ohmo-private skills, include the extra skill directory:\n\n   ```bash\n   PYTHONPATH=src python - <<'PY'\n   from openharness.skills import load_skill_registry\n   from ohmo.workspace import get_skills_dir\n   for skill in load_skill_registry(extra_skill_dirs=[get_skills_dir()]).list_skills():\n       print(skill.name, skill.source, skill.path)\n   PY\n   ```\n\n7. Test behavior.\n   Create two or three realistic prompts that should use the skill. If the skill\n   changes code or files, add regression tests around loading or command\n   behavior. Run targeted tests first, then broader tests if the loader or\n   runtime path changed.\n\n8. Iterate from evidence.\n   If a skill under-triggers, improve the frontmatter description. If it\n   triggers but produces poor work, improve the body workflow. If agents repeat\n   deterministic work, add a helper script or reference file.\n\n## Writing rules\n\n- Preserve the skill name when updating an existing skill unless the user\n  explicitly asks to rename it.\n- Do not overwrite user-created skills without reading the existing `SKILL.md`.\n- Keep `SKILL.md` under roughly 500 lines; use `references/` for long material.\n- Keep trigger guidance in frontmatter `description`, not buried only in the\n  body.\n- Make skills safe and unsurprising. Do not create skills that hide behavior,\n  bypass permissions, exfiltrate data, or encourage unauthorized access.\n- Prefer directory names that are lowercase kebab-case, even if the displayed\n  skill `name` is more human-readable.\n- Mention exact install or verification paths in the final response.\n\n## Test prompts\n\nAfter drafting a skill, propose a small eval set:\n\n```json\n{\n  \"skill_name\": \"my-skill\",\n  \"evals\": [\n    {\n      \"id\": \"happy-path\",\n      \"prompt\": \"A realistic user request that should trigger the skill.\",\n      \"expected_output\": \"What good behavior looks like.\"\n    },\n    {\n      \"id\": \"near-miss\",\n      \"prompt\": \"A similar request that should not trigger this skill.\",\n      \"expected_output\": \"The agent should choose a different path.\"\n    }\n  ]\n}\n```\n\nFor objective skills, add assertions or tests. For subjective skills, collect\nhuman feedback and revise the workflow from concrete complaints rather than\nadding rigid rules.\n\n<!-- File: src/openharness/skills/bundled/content/test.md (Tokens: ~230 | Category: anthropic-skill) -->\n# test\n\nWrite and run tests for code.\n\n## When to use\n\nUse when the user asks to write tests, verify behavior, or improve test coverage.\n\n## Workflow\n\n1. Understand what needs testing:\n   - New feature? Write unit tests for the happy path and edge cases\n   - Bug fix? Write a regression test that would have caught the bug\n   - Refactor? Ensure existing tests still pass\n2. Follow the project's testing patterns:\n   - Check existing tests for framework (pytest, jest, go test, etc.)\n   - Match naming conventions and file organization\n   - Use the same fixtures and helpers\n3. Write tests that are:\n   - Independent: each test can run alone\n   - Deterministic: same result every time\n   - Fast: mock external services, use in-memory databases\n4. Run the tests and verify they pass\n\n## Rules\n\n- Test behavior, not implementation details\n- One assertion per test when possible\n- Use descriptive test names that explain the scenario\n- Don't test framework or library code\n- Mock at system boundaries (external APIs, filesystem, network)\n</yakaai_skills>"}