{"owner":"muratcankoylan","repo":"Agent-Skills-for-Context-Engineering","hasSkills":true,"totalSkillsCount":76,"totalTokensCount":173655,"categories":["marketplace","plugin-manifest","root-instruction","claude-rule","anthropic-skill","subagent-persona"],"hasMcp":false,"mcpConfig":null,"found":[".claude-plugin/marketplace.json",".plugin/plugin.json","AGENTS.md","CLAUDE.md","SKILL.md","examples/book-sft-pipeline/SKILL.md","examples/digital-brain-skill/SKILL.md","examples/digital-brain-skill/agents/AGENTS.md","examples/interleaved-thinking/SKILL.md","examples/interleaved-thinking/generated_skills/comprehensive-research-agent/SKILL.md","examples/interleaved-thinking/generated_skills/comprehensive-research-agent/references/optimization_summary.json","examples/interleaved-thinking/generated_skills/comprehensive-research-agent/references/patterns_found.json","examples/llm-as-judge-skills/agents/evaluator-agent/evaluator-agent.md","examples/llm-as-judge-skills/agents/index.md","examples/llm-as-judge-skills/agents/orchestrator-agent/orchestrator-agent.md","examples/llm-as-judge-skills/agents/research-agent/research-agent.md","examples/llm-as-judge-skills/skills/context-fundamentals/context-fundamentals.md","examples/llm-as-judge-skills/skills/index.md","examples/llm-as-judge-skills/skills/llm-evaluator/llm-evaluator.md","examples/llm-as-judge-skills/skills/tool-design/tool-design.md","examples/llm-as-judge-skills/tools/evaluation/direct-score.md","examples/llm-as-judge-skills/tools/evaluation/generate-rubric.md","examples/llm-as-judge-skills/tools/evaluation/pairwise-compare.md","examples/llm-as-judge-skills/tools/index.md","examples/llm-as-judge-skills/tools/orchestration/delegate-to-agent.md","examples/llm-as-judge-skills/tools/research/read-url.md","examples/llm-as-judge-skills/tools/research/web-search.md","skills/advanced-evaluation/SKILL.md","skills/advanced-evaluation/references/bias-mitigation.md","skills/advanced-evaluation/references/evaluation-pipeline.md","skills/advanced-evaluation/references/implementation-patterns.md","skills/advanced-evaluation/references/metrics-guide.md","skills/bdi-mental-states/SKILL.md","skills/bdi-mental-states/references/bdi-ontology-core.md","skills/bdi-mental-states/references/framework-integration.md","skills/bdi-mental-states/references/rdf-examples.md","skills/bdi-mental-states/references/sparql-competency.md","skills/context-compression/SKILL.md","skills/context-compression/references/evaluation-framework.md","skills/context-degradation/SKILL.md","skills/context-degradation/references/patterns.md","skills/context-fundamentals/SKILL.md","skills/context-fundamentals/references/context-components.md","skills/context-optimization/SKILL.md","skills/context-optimization/references/optimization_techniques.md","skills/evaluation/SKILL.md","skills/evaluation/references/metrics.md","skills/filesystem-context/SKILL.md","skills/filesystem-context/references/implementation-patterns.md","skills/harness-engineering/SKILL.md","skills/hosted-agents/SKILL.md","skills/hosted-agents/references/infrastructure-patterns.md","skills/latent-briefing/SKILL.md","skills/latent-briefing/references/attention-matching-formulation.md","skills/long-horizon-prompting/SKILL.md","skills/long-horizon-prompting/references/cdc-prompt-annotated.md","skills/long-horizon-prompting/references/mass-handoff-contracts.md","skills/long-horizon-prompting/references/research-evidence.md","skills/long-horizon-prompting/references/task-brief-template.md","skills/long-horizon-prompting/references/vendor-guidance.md","skills/memory-systems/SKILL.md","skills/memory-systems/references/implementation.md","skills/multi-agent-patterns/SKILL.md","skills/multi-agent-patterns/references/frameworks.md","skills/project-development/SKILL.md","skills/project-development/references/case-studies.md","skills/project-development/references/pipeline-patterns.md","skills/self-improvement-loops/SKILL.md","skills/self-improvement-loops/references/loop-design-evidence.md","skills/self-managed-context/SKILL.md","skills/self-managed-context/references/evidence.md","skills/self-managed-context/references/harness-protocol.md","skills/tool-design/SKILL.md","skills/tool-design/references/architectural_reduction.md","skills/tool-design/references/best_practices.md","template/SKILL.md"],"skills":{".claude-plugin/marketplace.json":"",".plugin/plugin.json":"","AGENTS.md":"# AGENTS.md\n\nWorkspace memory for agents collaborating on this repository. Keep entries durable and broadly applicable; one-off task state belongs in chat or in a run thread, not here.\n\n## Learned User Preferences\n\n- For autonomous research and repo-improvement work in this workspace, prefer proceeding through concrete research loops, subagents, validation, and edits when the scope is clear rather than asking broad process questions.\n- Avoid stale regex or keyword-list heuristics in skills and scripts; prefer mechanism-level criteria, rubrics, and evidence-backed validation.\n- Never push to GitHub or merge a PR without explicit user approval. Preparing branches, commits, and PRs is permitted only when the user has approved that specific action.\n- Tone is technical CTO: direct, no marketing language, no exclamation marks, no emojis, no em dashes. State trade-offs and complexity upfront.\n- When the scope spans multiple architectural decisions or irreversible changes, propose a plan first instead of executing.\n- For benchmarks and evaluation work, hold to research-paper-grade methodology (statistical discipline, bias mitigation, ablations, reproducibility) over speed. Don't rush.\n\n## Learned Workspace Facts\n\n- This repo is an autonomous research-to-skill organization. External AI research is curated through rubrics and distilled into context-engineering and harness-engineering skill updates.\n- `researcher/` is repo-native and file-based so agents can resume, audit, validate, and prepare PR-ready skill changes without a hosted scheduler.\n- Per-run state lives in `researcher/runs//run-state.json` with explicit transitions (`initialized -> retrieved -> evaluated -> proposed -> novelty_checked -> validated -> pr_ready -> closed`). Use `research_loop.py` subcommands to advance state, never hand-edit `run-state.json`.\n- Repo health (`validate_repo.py`) and per-run readiness (`validate_run.py`) are different questions. CI runs `validate_platform_compat.py --require-reference-validator`, `validate_repo.py --strict`, `skill_health.py --strict --no-history`, `run_benchmarks.py`, and `check_activation_cases.py` on every PR via `.github/workflows/validate.yml`.\n- The mechanism registry (`researcher/mechanisms/registry.jsonl`) is the encyclopedia backbone. Promotion is gated by `research_loop.py promote-mechanisms` with a recorded reviewer; ledgers live under `researcher/mechanisms/ledgers/`.\n- Claim provenance for numeric or volatile claims lives in `researcher/claims/index.jsonl`. Add an entry for any new benchmark or volatility-sensitive claim.\n- The corpus index (`researcher/corpus/index.json`) is the machine-readable map of skills, activation scenarios, mechanisms, and claims. Update it when adding or restructuring skills.\n- The continuous loop (`researcher/scripts/loop_*.py`) runs from launchd via `researcher/orchestration/launchd/`. It never invokes paid LLMs; HTTP retrieval is stdlib-only with a 1.5 MB cap and a 30-second timeout.\n- Runtime state is not committed: `researcher/queue/*.jsonl`, `researcher/queue/.locks/`, `researcher/reports/{logs,snapshots,loop-events.jsonl,loop-failures.jsonl,status.md,parked-review.md}`, and `researcher/runs/*/` are gitignored. The seed run `20260515-035228-executable-autonomous-research-frameworks` is the only committed run; it is closed as `reference-only` and serves as a worked example.\n- The current prepared release version is 2.6.0 across `.claude-plugin/marketplace.json`, `.plugin/plugin.json`, and root `SKILL.md`. There are 18 skills (latent-briefing covers KV cache sharing between agents; self-improvement-loops covers RSI, meta-harness search, and evolutionary scaffold optimization; long-horizon-prompting covers pseudo-formal task briefs for long-running and parallel agent runs; self-managed-context covers model-controlled live context editing).\n- New accepted-ledger events must carry `event_type`, `source_commit`, and a committed `source_path`; a `run_dir` pointer alone dangles on a clean checkout because run directories are gitignored. `research_loop.py promote-mechanisms` still emits the `run_dir` shape, so write durable events directly until it is fixed.\n- Detailed lessons from building the researcher OS live in `researcher/insights/auto-research-experiment.md` (engineering rationale) and `researcher/insights/how-we-built-this.md` (project narrative and sharing templates); read both before extending the harness or writing release-facing prose.\n- Benchmarks are staged in `researcher/benchmarks/`: Stage 0 deterministic harness (shipped), Stage 1 per-skill health via `researcher/scripts/skill_health.py` (shipped; output `researcher/reports/skill-health.json` is gitignored), Stage 2 router (shipped; results in `researcher/benchmarks/router/results-published/`), Stage 3 effectiveness (scaffolded, one task built), Stage 4 composition (future). `researcher/benchmarks/PLAN.md` is the methodology source of truth.\n- Current corpus counts and compatibility status are generated in `researcher/generated/corpus-summary.md`; do not copy live totals into workspace memory. Published benchmark reports remain dated snapshots. Do not describe a skill improvement as complete unless the prose, mechanism registry, claim index, corpus index, activation fixtures, generated inventory, and validators all agree.\n- Public export uses `validate_export.py plan|render|check` and registered transforms from `governance/export-policy.yaml`. Private plans and receipts are ignored; public manifests contain projection and output digests, never private source paths or input digests.\n- Durable cross-runtime records resolve through `researcher/schemas/registry.json`. JSON digests use the integer-only `jcs-rfc8785-integer-v1` profile; blobs hash exact bytes. `ArtifactRef` never contains a private locator or grants authority, and reads must validate its private `StorageBinding`. Candidate evaluation must use a SPEC-003 freeze receipt created under an explicit editable-surface policy, not a mutable worktree.\n- Benchmark execution uses the Cursor SDK runner at `researcher/benchmarks/sdk-runner/` (TypeScript, `@cursor/sdk` 1.0.13). The runner supports `--concurrency N`, `--no-resume`, per-run progress logging, format-failure retry, and worst-case retry-aware cost forecasting; default behavior is to resume by skipping plan items that already have result files. Result artifacts under `researcher/benchmarks/{router,effectiveness}/results/` and history JSONLs (`router-history.jsonl`, `effectiveness-history.jsonl`) are gitignored.\n- Published Stage 2 router-benchmark results: `researcher/benchmarks/router/results-published/2026-05-15.md` (baseline), `researcher/benchmarks/router/results-published/2026-05-15-v2.md` (post-rewrite with delta-vs-baseline table), and `researcher/benchmarks/router/results-published/2026-05-19.md` (post-corpus-hardening validation: 600/600 usable records, 0 format failures, top-1 Gemini 0.920 / Composer 0.913 / GPT-5.5 0.913 / Claude Opus 4.7 0.840). Headline finding: targeted description rewrites moved `context-fundamentals` top-1 by +23.4pp and `project-development` top-1 to 1.000; corpus-wide hardening did not cause broad routing collapse.\n\n## Repository Operating Defaults\n\n- Deterministic checks before model judges. Always run `validate_platform_compat.py --require-reference-validator` and `validate_repo.py --strict` before claiming a skill-format or packaging change is complete.\n- Adversarial benchmarks before declaring the harness safe. Add a scenario when a new failure mode is discovered.\n- Append-only ledgers for accepted and rejected mechanisms so future agents do not rediscover failed paths.\n- Atomic writes (`tempfile` + `os.replace`) and `fcntl` locks for any shared file the loop touches.\n- Live execution is the highest-signal validation for orchestration code; smoke-test changes against the actual loop before declaring them safe.\n- Cursor SDK is the only paid-API surface allowed for benchmarks. Privacy Mode required, `apiKey` passed explicitly per call, never `settingSources: \"all\"` in benchmarks (use `[]` for control, `[\"project\"]` with a curated `.cursor/skills/` for ablation). Cost gates (`--max-runs`, `--max-budget-usd`, or `--dry-run`) must be set before any SDK call.\n- Description quality is measurable. When changing skill activation descriptions, re-run the router benchmark with the same seed and fixture and publish the delta. Aggregate accuracy is a misleading unit; per-skill effect sizes and the confusion matrix are the right view.\n- A skill is a multi-surface artifact. Changing the frontmatter `description` is not enough; the SKILL.md body `When to Activate` and `Integration` sections must be audited the same day so the body does not contradict the description that routed the agent to it. The router benchmark only sees descriptions (`settingSources: []`) and cannot catch body inconsistencies; only Stage 3 effectiveness benchmarks (which actually load skill bodies) measure body-alignment impact.\n- Any runner that calls a paid API in a loop must have three features before execution: bounded parallelism via `--concurrency`, resume capability via results-folder scan, and per-run progress logging that surfaces stalls inside one call's duration.\n- API keys provided in chat should be considered exposed; rotate immediately after use. Runner enforces this via `apiKeyFingerprint()` which only logs the last 4 characters.\n","CLAUDE.md":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nAgent Skills for Context Engineering is an open collection of Agent Skills teaching context engineering and harness engineering principles for production AI agent systems. Skills are platform-agnostic (Claude Code, Cursor, Codex/OpenAI Agent Skills, GitHub Copilot, any Open Plugins-conformant tool). v2.6.0 adds the self-managed-context skill (model-controlled live context editing: context as an editable file, self-editing harness invariants, edit-position cache cost, budget nudges, and steering or training context-editing strategy) on top of the long-horizon-prompting skill (v2.5.0), the self-improvement-loops skill (v2.4.0), and the file-based researcher operating system with deterministic gates, cross-platform Agent Skills validation, and a continuous loop. Use the [generated live inventory](researcher/generated/corpus-summary.md) for current corpus counts.\n\nContext engineering is the discipline of curating everything that enters a model's context window (system prompts, tool definitions, retrieved documents, message history, tool outputs) to maximize signal within limited attention budget.\n\n## Repository Structure\n\n- `skills/` - published skill directories, each containing a `SKILL.md` with YAML frontmatter (`name`, `description`) and optional `references/` and `scripts/` subdirectories\n- `examples/` - complete demonstration projects; the generated inventory is the count authority\n- `docs/` - Research materials and reference documentation\n- `researcher/` - File-based research-to-skill operating system: rubrics, mechanism registry, claim provenance, corpus index, run state machine, adversarial benchmarks, continuous loop, launchd service definitions\n- `template/SKILL.md` - Canonical skill template (use when creating new skills)\n- `SKILL.md` (root) - Collection-level metadata and skill map\n- `.claude-plugin/marketplace.json` - Claude Code marketplace manifest (single bundled plugin, v2.6.0)\n- `.plugin/plugin.json` - Open Plugins format manifest (v2.6.0)\n\n## Build & Test Commands\n\nNo top-level build system. Repo-level gates and per-project tooling below.\n\n### Top-level deterministic gates (run on every PR via CI)\n\n```\npython3 -m unittest researcher.scripts.tests.test_skill_frontmatter # parser and strict-YAML regression tests\npython3 researcher/scripts/validate_governance.py --check # authority model and generated view\npython3 researcher/scripts/build_inventory.py --check # corpus references and generated inventory\npython3 researcher/scripts/validate_platform_compat.py --require-reference-validator # Agent Skills reference validator + Cursor/Claude/Codex install-layout simulation\npython3 researcher/scripts/validate_repo.py --strict # corpus structure, manifests, rubric math, mechanism registry, claims, corpus index, activation cases, benchmark scenarios, run artifacts\npython3 researcher/scripts/skill_health.py --strict --no-history # deterministic skill-body quality gate\npython3 researcher/scripts/run_benchmarks.py # adversarial benchmark harness + repo + activation gates\npython3 researcher/scripts/check_activation_cases.py # skill-boundary regression fixtures\n```\n\n### Per-run readiness (active runs only)\n\n```\npython3 researcher/scripts/validate_run.py --run-dir researcher/runs/\n```\n\n### Continuous loop (manual or launchd)\n\n```\npython3 researcher/scripts/loop_discover.py\npython3 researcher/scripts/loop_step.py --allow-fetch\npython3 researcher/scripts/loop_daily.py\npython3 researcher/scripts/loop_status.py\n\nresearcher/orchestration/launchd/install.sh # macOS daemon\nresearcher/orchestration/launchd/uninstall.sh\n```\n\n### Example projects\n\n#### examples/llm-as-judge-skills (TypeScript, Node >= 18)\n```\ncd examples/llm-as-judge-skills\nnpm install\nnpm run build # tsc\nnpm test # vitest (19 tests)\nnpm run lint # eslint\nnpm run format # prettier\nnpm run typecheck # tsc --noEmit\n```\n\n#### examples/interleaved-thinking (Python >= 3.10)\n```\ncd examples/interleaved-thinking\npip install -e \".[dev]\"\npytest # pytest + pytest-asyncio\nruff check . # linting (100 char line length)\n```\n\n#### examples/digital-brain-skill (Node.js)\n```\ncd examples/digital-brain-skill\nnpm run setup\nnpm run weekly-review\nnpm run content-ideas\nnpm run stale-contacts\n```\n\n## Skill Authoring Rules\n\nWhen creating or editing skills:\n\n1. **SKILL.md must stay under 500 lines**: move detailed content to `references/` directory\n2. **YAML frontmatter is required**: must include `name` and `description` fields\n3. **Folder naming**: lowercase with hyphens (e.g., `context-fundamentals`)\n4. **Write in third person**: descriptions are injected into system prompts; inconsistent POV causes discovery issues\n5. **Platform-agnostic**: no vendor-locked examples or platform-specific tool names without abstraction\n6. **Token-conscious**: challenge each paragraph and assume an advanced audience\n7. **Body standard**: include `When to Activate`, `Core Concepts`, `Practical Guidance`, `Examples`, `Guidelines`, `Gotchas`, `Integration`, and `References`\n8. **Explicit boundaries**: every `When to Activate` section needs positive triggers plus a `Do not activate` block routing adjacent work to the right skill\n9. **Include a Gotchas section**: experience-derived failure modes are the highest-signal content in any skill\n10. **Update root README.md** when adding new skills\n11. **Update marketplace/plugin manifests** when adding skills (`.claude-plugin/marketplace.json`, `.plugin/plugin.json`)\n12. **Update the corpus index** (`researcher/corpus/index.json`) to map the new skill to activation scenarios, mechanism IDs, and claim IDs\n13. **Update mechanisms and claims**: add registry entries for reusable behavior changes and `claim-*` provenance for numeric, benchmark, volatile, or vendor-performance claims\n14. **Run `validate_platform_compat.py --require-reference-validator`, `validate_repo.py --strict`, `skill_health.py --strict --no-history`, `check_activation_cases.py`, and `run_benchmarks.py`** before committing skill changes\n\n## Researcher OS Rules\n\nWhen working through the researcher operating system:\n\n1. **Initialize runs via `research_loop.py init`**: it creates `run-state.json`, queue entry, thread log, source evaluation scaffold, and mechanism proposal template\n2. **Advance state explicitly**: use `retrieve`, `evaluate`, `propose`, `novelty`, `validate-run`, `pr-ready`, `close` subcommands; do not edit `run-state.json` by hand\n3. **Promote mechanisms only after run readiness**: `research_loop.py promote-mechanisms` requires `--reviewed-by` and a passing run-readiness check\n4. **Add claim provenance** to `researcher/claims/index.jsonl` for any numeric, benchmark, or volatile claim added to a skill\n5. **Never invoke paid LLMs from the continuous loop**: HTTP retrieval is stdlib-only, judge adapters are explicitly out of scope until budget-gated\n6. **Never commit runtime queue/report files**: `.gitignore` covers `researcher/queue/*.jsonl`, `researcher/reports/{logs,snapshots,loop-events.jsonl,loop-failures.jsonl,status.md,parked-review.md}`, and `researcher/runs/*/` except the seed run\n\n## Plugin Architecture\n\nAll published skills are distributed as a single plugin (`context-engineering`) in the marketplace manifest. This avoids cache duplication: Claude Code caches each plugin's `source` directory separately, so multiple plugins pointing to `source: \"./\"` would each cache a full copy of the repo.\n\nProgressive disclosure pattern: only skill names/descriptions load at startup; full content loads on activation.\n\n## Key Design Principles\n\n- **Context quality over quantity**: attention scarcity and lost-in-middle behavior mean more context is not always better\n- **Sub-agents isolate context**: they exist to manage attention budget, not simulate org roles\n- **Skills reference each other**: use plain text skill names (not links) in Integration sections to avoid cross-directory reference issues\n- **Examples use Python pseudocode**: conceptual demonstrations that work across environments, not production-ready implementations\n- **Deterministic first, model-judged second**: structure, schema, rubric math, manifest sync, retrieval status, and registry shape must pass before any LLM judge is invoked\n- **Human-controlled merge**: agents may prepare PRs and pass gates, but push and merge always require explicit human approval\n","SKILL.md":"---\nname: context-engineering-collection\ndescription: \"A comprehensive collection of Agent Skills for context engineering, harness engineering, multi-agent architectures, and production agent systems. Use when building, optimizing, evaluating, or debugging agent systems that require effective context management and reliable operating loops.\"\n---\n\n# Agent Skills for Context Engineering\n\nThis collection provides structured guidance for building production-grade AI agent systems through effective context engineering.\n\n## When to Activate\n\nActivate these skills when:\n- Building new agent systems from scratch\n- Optimizing existing agent performance\n- Debugging context-related failures\n- Designing multi-agent architectures\n- Creating or evaluating tools for agents\n- Implementing memory and persistence layers\n- Designing autonomous research or evaluation harnesses\n\n## Skill Map\n\n### Foundational Context Engineering\n\n**Understanding Context Fundamentals**\nContext is not just prompt text—it is the complete state available to the language model at inference time, including system instructions, tool definitions, retrieved documents, message history, and tool outputs. Effective context engineering means understanding what information truly matters for the task at hand and curating that information for maximum signal-to-noise ratio.\n\n**Recognizing Context Degradation**\nLanguage models exhibit predictable degradation patterns as context grows: the \"lost-in-middle\" phenomenon where information in the center of context receives less attention; U-shaped attention curves that prioritize beginning and end; context poisoning when errors compound; and context distraction when irrelevant information overwhelms relevant content.\n\n### Architectural Patterns\n\n**Multi-Agent Coordination**\nProduction multi-agent systems converge on three dominant patterns: supervisor/orchestrator architectures with centralized control, peer-to-peer swarm architectures for flexible handoffs, and hierarchical structures for complex task decomposition. The critical insight is that sub-agents exist primarily to isolate context rather than to simulate organizational roles.\n\n**Long-Horizon Prompting**\nLong-running autonomous agents and parallel orchestrations succeed or fail on the launch prompt. Pseudo-formal task briefs specify success predicates, non-counting outcomes, persistence rules with audit-gated return conditions, effort floors, diversity policies for parallel portfolios, and contamination guards, applying the discipline of formal verification linguistically to problems with no machine-checkable success condition. In orchestrated runs, each worker returns an output contract with named checks, and only evidenced violations block the return.\n\n**Memory System Design**\nMemory architectures range from simple scratchpads to sophisticated temporal knowledge graphs. Vector RAG provides semantic retrieval but loses relationship information. Knowledge graphs preserve structure but require more engineering investment. The file-system-as-memory pattern enables just-in-time context loading without stuffing context windows.\n\n**Filesystem-Based Context**\nThe filesystem provides a single interface for storing, retrieving, and updating effectively unlimited context. Key patterns include scratch pads for tool output offloading, plan persistence for long-horizon tasks, sub-agent communication via shared files, and dynamic skill loading. Agents use `ls`, `glob`, `grep`, and `read_file` for targeted context discovery, often outperforming semantic search for structural queries.\n\n**Hosted Agent Infrastructure**\nBackground coding agents run in remote sandboxed environments rather than on local machines. Key patterns include pre-built environment images refreshed on regular cadence, warm sandbox pools for instant session starts, filesystem snapshots for session persistence, and multiplayer support for collaborative agent sessions. Critical optimizations include allowing file reads before git sync completes (blocking only writes), predictive sandbox warming when users start typing, and self-spawning agents for parallel task execution.\n\n**Tool Design Principles**\nTools are contracts between deterministic systems and non-deterministic agents. Effective tool design follows the consolidation principle (prefer single comprehensive tools over multiple narrow ones), returns contextual information in errors, supports response format options for token efficiency, and uses clear namespacing.\n\n### Operational Excellence\n\n**Context Compression**\nWhen agent sessions exhaust memory, compression becomes mandatory. The correct optimization target is tokens-per-task, not tokens-per-request. Structured summarization with explicit sections for files, decisions, and next steps preserves more useful information than aggressive compression. Artifact trail integrity remains the weakest dimension across all compression methods.\n\n**Context Optimization**\nTechniques include compaction (summarizing context near limits), observation masking (replacing verbose tool outputs with references), prefix caching (reusing KV blocks across requests), and strategic context partitioning (splitting work across sub-agents with isolated contexts).\n\n**Self-Managed Context**\nThe model, not the harness, decides what stays in its live context: the editable region is exposed as a file the model rewrites with code tools and re-parsed each turn. The harness keeps the invariants (pinned system and task prefix, role folding, edit gate, receipts, deterministic budget readouts, rollback on overflow). Edit cost scales with the text after the edit under prefix caching, so edits are batched and placed with the tail in mind; strategy can be steered by instruction, evolved as a skill, or trained with a success-gated efficiency reward.\n\n**Latent Briefing (KV Memory Sharing)**\nOrchestrator-worker systems can compound tokens when supervisors accumulate long trajectories but workers see only narrow text slices. Latent Briefing compacts the orchestrator trajectory in the worker model's KV cache using task-guided attention (Attention Matching-style compaction) so workers receive relevant latent state without full-text replay when the stack exposes worker KV state and the models are compatible.\n\n**Evaluation Frameworks**\nProduction agent evaluation requires deterministic checks and multi-dimensional rubrics covering factual accuracy, completeness, tool efficiency, and process quality. Use model judges only after structure, evidence, and rubric math are valid; route judge design, pairwise comparison, and bias mitigation to Advanced Evaluation.\n\n**Harness Engineering**\nReliable autonomous agents need explicit operating loops around the model: locked metrics, editable surfaces, durable logs, novelty checks, rollback rules, and human approval boundaries. Harnesses prevent agents from weakening the evaluator, losing state across compaction, or turning ambiguous goals into unreviewable changes.\n\n**Self-Improvement Loops**\nWhen the harness itself becomes the optimization target, a different discipline applies: recursive self-improvement, meta-harness search, failure-driven bounded self-edits, evolutionary scaffold search, and context mechanism evolution. The controlling constraints are empirical two-split acceptance gates, filesystem experience archives with raw traces, runtime-enforced constraints outside every editable surface, and diversity preservation to prevent collapse.\n\n### Development Methodology\n\n**Project Development**\nEffective LLM project development begins with task-model fit analysis: validating through manual prototyping that a task is well-suited for LLM processing before building automation. Production pipelines follow staged, idempotent architectures (acquire, prepare, process, parse, render) with file system state management for debugging and caching. Structured output design with explicit format specifications enables reliable parsing. Start with minimal architecture and add complexity only when proven necessary.\n\n### Cognitive Architecture\n\n**BDI Mental States**\nBelief-desire-intention modeling provides a formal way to translate structured external context into agent mental states. Use it for rational agency, explainability, and systems that need auditable links between beliefs, goals, and chosen actions.\n\n## Core Concepts\n\nThe collection is organized around four core themes. First, context fundamentals establish what context is, how attention mechanisms work, and why context quality matters more than quantity. Second, architectural patterns cover the structures and coordination mechanisms that enable effective agent systems. Third, operational excellence addresses optimization, evaluation, and harness reliability. Fourth, development methodology and cognitive architecture cover project execution and formal mental-state modeling.\n\n## Practical Guidance\n\nEach skill can be used independently or in combination. Start with fundamentals to establish context management mental models. Branch into architectural patterns based on your system requirements. Reference operational skills when optimizing production systems.\n\nThe skills are platform-agnostic and work with Claude Code, Cursor, or any agent framework that supports custom instructions or skill-like constructs.\n\n## Integration\n\nThis collection integrates with itself—skills reference each other and build on shared concepts. The fundamentals skill provides context for all other skills. Architectural skills (multi-agent, memory, tools) can be combined for complex systems. Operational skills (optimization, evaluation) apply to any system built using the foundational and architectural skills.\n\n## References\n\nInternal skills in this collection:\n- [context-fundamentals](skills/context-fundamentals/SKILL.md)\n- [context-degradation](skills/context-degradation/SKILL.md)\n- [context-compression](skills/context-compression/SKILL.md)\n- [multi-agent-patterns](skills/multi-agent-patterns/SKILL.md)\n- [long-horizon-prompting](skills/long-horizon-prompting/SKILL.md)\n- [memory-systems](skills/memory-systems/SKILL.md)\n- [tool-design](skills/tool-design/SKILL.md)\n- [filesystem-context](skills/filesystem-context/SKILL.md)\n- [hosted-agents](skills/hosted-agents/SKILL.md)\n- [context-optimization](skills/context-optimization/SKILL.md)\n- [self-managed-context](skills/self-managed-context/SKILL.md)\n- [latent-briefing](skills/latent-briefing/SKILL.md)\n- [evaluation](skills/evaluation/SKILL.md)\n- [advanced-evaluation](skills/advanced-evaluation/SKILL.md)\n- [harness-engineering](skills/harness-engineering/SKILL.md)\n- [self-improvement-loops](skills/self-improvement-loops/SKILL.md)\n- [project-development](skills/project-development/SKILL.md)\n- [bdi-mental-states](skills/bdi-mental-states/SKILL.md)\n\nExternal resources on context engineering:\n- Research on attention mechanisms and context window limitations\n- Production experience from leading AI labs on agent system design\n- Framework documentation for LangGraph, AutoGen, and CrewAI\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-20\n**Last Updated**: 2026-10-01\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.6.0\n","examples/book-sft-pipeline/SKILL.md":"---\nname: book-sft-pipeline\ndescription: \"This skill should be used for book-to-SFT pipelines: ePub extraction, literary segmentation, author-voice dataset construction, style-transfer training, LoRA workflows, and model evaluation for voice replication.\"\nversion: 2.0.0\n---\n\n# Book SFT Pipeline\n\nA complete system for converting books into SFT datasets and training style-transfer models. This skill teaches the pipeline from raw ePub to a model that writes in any author's voice.\n\n## When to Activate\n\nActivate this skill when:\n- Building fine-tuning datasets from literary works\n- Creating author-voice or style-transfer models\n- Preparing training data for Tinker or similar SFT platforms\n- Designing text segmentation pipelines for long-form content\n- Training small models (8B or less) on limited data\n\n## Core Concepts\n\n### The Three Pillars of Book SFT\n\n**1. Intelligent Segmentation**\nText chunks must be semantically coherent. Breaking mid-sentence teaches the model to produce fragmented output. Target: 150-400 words per chunk, always at natural boundaries.\n\n**2. Diverse Instruction Generation**\nUse multiple prompt templates and system prompts to prevent overfitting. A single prompt style leads to memorization. Use 15+ prompt templates with 5+ system prompts.\n\n**3. Style Over Content**\nThe goal is learning the author's rhythm and vocabulary patterns, not memorizing plots. Synthetic instructions describe what happens without quoting the text.\n\n## Pipeline Architecture\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│ ORCHESTRATOR AGENT │\n│ Coordinates pipeline phases, manages state, handles failures │\n└──────────────────────┬──────────────────────────────────────────┘\n │\n ┌───────────────┼───────────────┬───────────────┐\n ▼ ▼ ▼ ▼\n┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐\n│ EXTRACTION │ │ SEGMENTATION │ │ INSTRUCTION │ │ DATASET │\n│ AGENT │ │ AGENT │ │ AGENT │ │ BUILDER │\n│ ePub → Text │ │ Text → Chunks│ │ Chunks → │ │ Pairs → │\n│ │ │ 150-400 words│ │ Prompts │ │ JSONL │\n└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘\n │\n ┌───────────────┴───────────────┐\n ▼ ▼\n┌──────────────┐ ┌──────────────┐\n│ TRAINING │ │ VALIDATION │\n│ AGENT │ │ AGENT │\n│ LoRA on │ │ AI detector │\n│ Tinker │ │ Originality │\n└──────────────┘ └──────────────┘\n```\n\n## Phase 1: Text Extraction\n\n### Critical Rules\n1. **Always source ePub over PDF** - OCR errors become learned patterns\n2. **Use paragraph-level extraction** - Extract from `

` tags to preserve breaks\n3. **Remove front/back matter** - Copyright and TOC pollute the dataset\n\n```python\n# Extract text from ePub paragraphs\nfrom epub2 import EPub\nfrom bs4 import BeautifulSoup\n\ndef extract_epub(path):\n book = EPub(path)\n chapters = []\n for item in book.flow:\n html = book.get_chapter(item.id)\n soup = BeautifulSoup(html, 'html.parser')\n paragraphs = [p.get_text().strip() for p in soup.find_all('p')]\n chapters.append('\\n\\n'.join(p for p in paragraphs if p))\n return '\\n\\n'.join(chapters)\n```\n\n## Phase 2: Intelligent Segmentation\n\n### Smaller Chunks + Overlap\n\nSmaller chunks (150-400 words) produce more training examples and better style transfer than larger chunks (250-650).\n\n```python\ndef segment(text, min_words=150, max_words=400):\n paragraphs = text.split('\\n\\n')\n chunks, buffer, buffer_words = [], [], 0\n \n for para in paragraphs:\n words = len(para.split())\n if buffer_words + words > max_words and buffer_words >= min_words:\n chunks.append('\\n\\n'.join(buffer))\n # Keep last paragraph for overlap\n buffer = [buffer[-1], para] if buffer else [para]\n buffer_words = sum(len(p.split()) for p in buffer)\n else:\n buffer.append(para)\n buffer_words += words\n \n if buffer:\n chunks.append('\\n\\n'.join(buffer))\n return chunks\n```\n\n### Expected Results\n\nFor an 86,000-word book:\n- Old method (250-650 words): ~150 chunks\n- New method (150-400 + overlap): ~300 chunks\n- With 2 variants per chunk: 600+ training examples\n\n## Phase 3: Diverse Instruction Generation\n\n### The Key Insight\n\nUsing a single prompt template causes memorization. Diverse templates teach the underlying style.\n\n```python\nSYSTEM_PROMPTS = [\n \"You are an expert creative writer capable of emulating specific literary styles.\",\n \"You are a literary writer with deep knowledge of classic prose styles.\",\n \"You are a creative writer skilled at emulating distinctive authorial voices.\",\n \"You write prose that captures the essence of modernist literature.\",\n \"You are a talented writer who can channel classic American authors.\",\n]\n\nPROMPT_TEMPLATES = [\n \"Write a passage in the style of {author}: {desc}\",\n \"Channel {author}'s voice to write about: {desc}\",\n \"In {author}'s distinctive prose style, describe: {desc}\",\n \"Write this scene as {author} would have: {desc}\",\n \"Using {author}'s repetitive technique, describe: {desc}\",\n \"Capture the rhythm of {author} in this passage: {desc}\",\n \"Write like {author}: {desc}\",\n \"In the voice of {author}, write: {desc}\",\n \"This is a literary exercise. Write like {author}: {desc}\",\n \"Can you write in {author}'s style? {desc}\",\n]\n```\n\n### Instruction Generation\n\n```python\nINSTRUCTION_PROMPT = \"\"\"Describe what is happening in this excerpt in 2-3 sentences.\nFocus on: characters present, actions, emotions, setting.\nDo NOT quote the text directly.\n\nExcerpt:\n{text}\n\"\"\"\n\n# Use a fast, cheap LLM (e.g., Gemini Flash)\ninstruction = llm_call(INSTRUCTION_PROMPT.format(text=chunk))\n```\n\n## Phase 4: Dataset Construction\n\n### Message Format\n\n```json\n{\n \"messages\": [\n {\"role\": \"system\", \"content\": \"You are an expert creative writer...\"},\n {\"role\": \"user\", \"content\": \"Write in the style of Author: Scene description...\"},\n {\"role\": \"assistant\", \"content\": \"The actual book text from chunk...\"}\n ]\n}\n```\n\n### Multiple Variants Per Chunk\n\n```python\ndef build_examples(chunk, instruction, author, variants=2):\n examples = []\n for i in range(variants):\n system = SYSTEM_PROMPTS[i % len(SYSTEM_PROMPTS)]\n template = PROMPT_TEMPLATES[(chunk.id + i) % len(PROMPT_TEMPLATES)]\n user = template.format(author=author, desc=instruction)\n examples.append({\"messages\": [\n {\"role\": \"system\", \"content\": system},\n {\"role\": \"user\", \"content\": user},\n {\"role\": \"assistant\", \"content\": chunk.text}\n ]})\n return examples\n```\n\n## Phase 5: LoRA Training on Tinker\n\n### Configuration\n\n```python\nCONFIG = {\n \"model_name\": \"Qwen/Qwen3-8B-Base\", # Base, not instruct\n \"lora_rank\": 32, # 352MB adapter\n \"learning_rate\": 5e-4, # Higher for LoRA\n \"batch_size\": 4,\n \"epochs\": 3,\n}\n```\n\n### Why Base Model?\n\nUse **base** (pretrained) models, not instruction-tuned versions:\n- Base models are more malleable for new styles\n- Instruct models have patterns that resist overwriting\n- Style is a low-level pattern that base models capture better\n\n### Training Loop\n\n```python\nimport tinker\nfrom tinker import types\n\ntraining_client = await service_client.create_lora_training_client_async(\n base_model=\"Qwen/Qwen3-8B-Base\",\n rank=32\n)\n\nfor epoch in range(3):\n for batch in batches:\n await training_client.forward_backward_async(batch, loss_fn=\"cross_entropy\")\n await training_client.optim_step_async(types.AdamParams(learning_rate=5e-4))\n\nresult = await training_client.save_weights_for_sampler_async(name=\"final\")\n```\n\n## Phase 6: Validation\n\n### Modern Scenario Test\n\nTest with scenarios that couldn't exist in the original book:\n\n```python\nTEST_PROMPTS = [\n \"Write about a barista making lattes\",\n \"Describe lovers communicating through text messages\",\n \"Write about someone anxious about climate change\",\n]\n```\n\nIf the model applies style markers to modern scenarios, it learned **style**, not **content**.\n\n### Originality Verification\n\n```bash\n# Search training data for output phrases\ngrep \"specific phrase from output\" dataset.jsonl\n# Should return: No matches\n```\n\n### AI Detector Testing\n\nTest outputs with GPTZero, Pangram, or ZeroGPT.\n\n## Known Issues and Solutions\n\n### Character Name Leakage\n\n**Symptom**: Model uses original character names in new scenarios.\n**Cause**: Limited name diversity from one book.\n**Solution**: Train on multiple books or add synthetic examples.\n\n### Model Parrots Exact Phrases\n\n**Symptom**: Outputs contain exact sentences from training data.\n**Cause**: Too few prompt variations or too many epochs.\n**Solution**: Use 15+ templates, limit to 3 epochs.\n\n### Fragmented Outputs\n\n**Symptom**: Sentences feel incomplete.\n**Cause**: Poor segmentation breaking mid-thought.\n**Solution**: Always break at paragraph boundaries.\n\n## Guidelines\n\n1. **Always source ePub over PDF** - OCR errors become learned patterns\n2. **Never break mid-sentence** - Boundaries must be grammatically complete\n3. **Use diverse prompts** - 15+ templates, 5+ system prompts\n4. **Use base models** - Not instruct versions\n5. **Use smaller chunks** - 150-400 words for more examples\n6. **Reserve test set** - 50 examples minimum\n7. **Test on modern scenarios** - Proves style transfer vs memorization\n8. **Verify originality** - Grep training data for output phrases\n\n## Expected Results\n\n| Metric | Value |\n|--------|-------|\n| Training examples | 500-1000 per book |\n| Model | Qwen/Qwen3-8B-Base |\n| LoRA rank | 32 |\n| Adapter size | ~350 MB |\n| Training time | ~15 min |\n| Loss reduction | 90%+ |\n| Style transfer success | ~50% perfect |\n\n## Cost Estimate\n\n| Component | Cost |\n|-----------|------|\n| LLM (instruction generation) | ~$0.50 |\n| Tinker training (15 min) | ~$1.50 |\n| **Total** | **~$2.00** |\n\n## Integration with Context Engineering Skills\n\nThis example applies several skills from the Agent Skills for Context Engineering collection:\n\n### project-development\nThe pipeline follows the staged, idempotent architecture pattern:\n- **Acquire**: Extract text from ePub\n- **Prepare**: Segment into training chunks\n- **Process**: Generate synthetic instructions\n- **Parse**: Build message format\n- **Render**: Output Tinker-compatible JSONL\n- **Train**: LoRA fine-tuning\n- **Validate**: Modern scenario testing\n\nEach phase is resumable and produces intermediate artifacts for debugging.\n\n### context-compression\nSegmentation is a form of context compression for training. The core insight from context-compression applies: information density matters more than information quantity. Smaller, coherent chunks (150-400 words) produce better style transfer than larger, diluted chunks.\n\nThe two-tier strategy mirrors context compression evaluation:\n- Tier 1: Fast, deterministic compression\n- Tier 2: LLM-assisted for edge cases\n\n### multi-agent-patterns\nThe pipeline uses the **supervisor/orchestrator** pattern:\n- Orchestrator coordinates phases and manages state\n- Specialized agents (Extraction, Segmentation, Instruction, Builder) have isolated contexts\n- Each agent receives only the information needed for its task\n\nThis matches the principle that sub-agents exist primarily to isolate context rather than simulate roles.\n\n### evaluation\nValidation follows the **end-state evaluation** pattern:\n- Functional testing: Does output match expected style markers?\n- Originality verification: Is content genuinely generated?\n- External validation: AI detector scores\n\nThe \"modern scenario\" test is a form of out-of-distribution evaluation that proves generalization.\n\n### context-fundamentals\nPrompt diversity prevents attention collapse on single patterns. When training with identical prompt structures, the model memorizes the instruction-response mapping. Diverse templates force attention across the style patterns themselves.\n\n## References\n\nInternal references:\n- [Segmentation Strategies](./references/segmentation-strategies.md) - Text chunking patterns\n- [Tinker Format Specification](./references/tinker-format.md) - Datum structure\n- [Tinker API Documentation](./references/tinker.txt) - Full API reference\n\nRelated skills from Agent Skills for Context Engineering:\n- project-development - Pipeline architecture patterns\n- context-compression - Compression strategies \n- multi-agent-patterns - Agent coordination\n- evaluation - Evaluation frameworks\n- context-fundamentals - Attention and information density\n\nExternal resources:\n- [Research Paper](https://arxiv.org/pdf/2510.13939) - Chakrabarty et al. 2025\n- [Dataset on Hugging Face](https://huggingface.co/datasets/MuratcanKoylan/gertrude-stein-style-sft)\n- [Gertrude Stein Case Study](./examples/gertrude-stein/) - Complete working example\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-26\n**Last Updated**: 2025-12-28\n**Author**: Muratcan Koylan\n**Version**: 2.0.0\n**Standalone**: Yes (separate from main context-engineering collection)\n","examples/digital-brain-skill/SKILL.md":"---\nname: digital-brain\ndescription: \"This skill should be used for personal operating-system workflows: content creation, voice consistency, relationship lookup, meeting preparation, weekly review, goal tracking, personal brand management, and network management.\"\nversion: 1.0.0\n---\n\n# Digital Brain\n\nA structured personal operating system for managing digital presence, knowledge, relationships, and goals with AI assistance. Designed for founders building in public, content creators growing their audience, and tech-savvy professionals seeking AI-assisted personal management.\n\n**Important**: This skill uses progressive disclosure. Module-specific instructions are in each subdirectory's `.md` file. Only load what's needed for the current task.\n\n## When to Activate\n\nActivate this skill when the user:\n\n- Requests content creation (posts, threads, newsletters) - load identity/voice.md first\n- Asks for help with personal brand or positioning\n- Needs to look up or manage contacts/relationships\n- Wants to capture or develop content ideas\n- Requests meeting preparation or follow-up\n- Asks for weekly reviews or goal tracking\n- Needs to save or retrieve bookmarked resources\n- Wants to organize research or learning materials\n\n**Trigger phrases**: \"write a post\", \"my voice\", \"content ideas\", \"who is [name]\", \"prepare for meeting\", \"weekly review\", \"save this\", \"my goals\"\n\n## Core Concepts\n\n### Progressive Disclosure Architecture\n\nThe Digital Brain follows a three-level loading pattern:\n\n| Level | When Loaded | Content |\n|-------|-------------|---------|\n| **L1: Metadata** | Always | This SKILL.md overview |\n| **L2: Module Instructions** | On-demand | `[module]/[MODULE].md` files |\n| **L3: Data Files** | As-needed | `.jsonl`, `.yaml`, `.md` data |\n\n### File Format Strategy\n\nFormats chosen for optimal agent parsing:\n\n- **JSONL** (`.jsonl`): Append-only logs - ideas, posts, contacts, interactions\n- **YAML** (`.yaml`): Structured configs - goals, values, circles\n- **Markdown** (`.md`): Narrative content - voice, brand, calendar, todos\n- **XML** (`.xml`): Complex prompts - content generation templates\n\n### Append-Only Data Integrity\n\nJSONL files are **append-only**. Never delete entries:\n- Mark as `\"status\": \"archived\"` instead of deleting\n- Preserves history for pattern analysis\n- Enables \"what worked\" retrospectives\n\n## Detailed Topics\n\n### Module Overview\n\n```\ndigital-brain/\n├── identity/ → Voice, brand, values (READ FIRST for content)\n├── content/ → Ideas, drafts, posts, calendar\n├── knowledge/ → Bookmarks, research, learning\n├── network/ → Contacts, interactions, intros\n├── operations/ → Todos, goals, meetings, metrics\n└── agents/ → Automation scripts\n```\n\n### Identity Module (Critical for Content)\n\n**Always read `identity/voice.md` before generating any content.**\n\nContains:\n- `voice.md` - Tone, style, vocabulary, patterns\n- `brand.md` - Positioning, audience, content pillars\n- `values.yaml` - Core beliefs and principles\n- `bio-variants.md` - Platform-specific bios\n- `prompts/` - Reusable generation templates\n\n### Content Module\n\nPipeline: `ideas.jsonl` → `drafts/` → `posts.jsonl`\n\n- Capture ideas immediately to `ideas.jsonl`\n- Develop in `drafts/` using `templates/`\n- Log published content to `posts.jsonl` with metrics\n- Plan in `calendar.md`\n\n### Network Module\n\nPersonal CRM with relationship tiers:\n- `inner` - Weekly touchpoints\n- `active` - Bi-weekly touchpoints\n- `network` - Monthly touchpoints\n- `dormant` - Quarterly reactivation checks\n\n### Operations Module\n\nProductivity system with priority levels:\n- P0: Do today, blocking\n- P1: This week, important\n- P2: This month, valuable\n- P3: Backlog, nice to have\n\n## Practical Guidance\n\n### Content Creation Workflow\n\n```\n1. Read identity/voice.md (REQUIRED)\n2. Check identity/brand.md for topic alignment\n3. Reference content/posts.jsonl for successful patterns\n4. Use content/templates/ as starting structure\n5. Draft matching voice attributes\n6. Log to posts.jsonl after publishing\n```\n\n### Pre-Meeting Preparation\n\n```\n1. Look up contact: network/contacts.jsonl\n2. Get history: network/interactions.jsonl\n3. Check pending: operations/todos.md\n4. Generate brief with context\n```\n\n### Weekly Review Process\n\n```\n1. Run: python agents/scripts/weekly_review.py\n2. Review metrics in operations/metrics.jsonl\n3. Check stale contacts: agents/scripts/stale_contacts.py\n4. Update goals progress in operations/goals.yaml\n5. Plan next week in content/calendar.md\n```\n\n## Examples\n\n### Example: Writing an X Post\n\n**Input**: \"Help me write a post about AI agents\"\n\n**Process**:\n1. Read `identity/voice.md` → Extract voice attributes\n2. Check `identity/brand.md` → Confirm \"ai_agents\" is a content pillar\n3. Reference `content/posts.jsonl` → Find similar successful posts\n4. Draft post matching voice patterns\n5. Suggest adding to `content/ideas.jsonl` if not publishing immediately\n\n**Output**: Post draft in user's authentic voice with platform-appropriate format.\n\n### Example: Contact Lookup\n\n**Input**: \"Prepare me for my call with Sarah Chen\"\n\n**Process**:\n1. Search `network/contacts.jsonl` for \"Sarah Chen\"\n2. Get recent entries from `network/interactions.jsonl`\n3. Check `operations/todos.md` for pending items with Sarah\n4. Compile brief: role, context, last discussed, follow-ups\n\n**Output**: Pre-meeting brief with relationship context.\n\n## Guidelines\n\n1. **Voice First**: Always read `identity/voice.md` before any content generation\n2. **Append Only**: Never delete from JSONL files - archive instead\n3. **Update Timestamps**: Set `updated` field when modifying tracked data\n4. **Cross-Reference**: Knowledge informs content, network informs operations\n5. **Log Interactions**: Always log meetings/calls to `interactions.jsonl`\n6. **Preserve History**: Past content in `posts.jsonl` informs future performance\n\n## Integration\n\nThis skill integrates context engineering principles:\n\n- **context-fundamentals** - Progressive disclosure, attention budget management\n- **memory-systems** - JSONL for persistent memory, structured recall\n- **tool-design** - Scripts in `agents/scripts/` follow tool design principles\n- **context-optimization** - Module separation prevents context bloat\n\n## References\n\nInternal references:\n- [Identity Module](./identity/IDENTITY.md) - Voice and brand details\n- [Content Module](./content/CONTENT.md) - Content pipeline docs\n- [Network Module](./network/NETWORK.md) - CRM documentation\n- [Operations Module](./operations/OPERATIONS.md) - Productivity system\n- [Agent Scripts](./agents/AGENTS.md) - Automation documentation\n\nExternal resources:\n- [Agent Skills for Context Engineering](https://github.com/muratcankoylan/Agent-Skills-for-Context-Engineering)\n- [Anthropic Context Engineering Guide](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)\n\n---\n\n## Skill Metadata\n\n**Created**: 2024-12-29\n**Last Updated**: 2024-12-29\n**Author**: Murat Can Koylan\n**Version**: 1.0.0\n","examples/digital-brain-skill/agents/AGENTS.md":"","examples/interleaved-thinking/SKILL.md":"---\nname: reasoning-trace-optimizer\ndescription: \"Debug and optimize AI agents by analyzing reasoning traces, context degradation, tool confusion, instruction drift, repeated task failures, and performance regressions.\"\n---\n\n# Reasoning Trace Optimizer\n\nDebug and optimize AI agents by analyzing their reasoning traces. This skill uses MiniMax M2.1's interleaved thinking to provide deep insight into agent decision-making and generate concrete improvements.\n\n## When to Activate\n\n- Agent reasoning traces need debugging, analysis, or prompt optimization\n- Agent task fails and user wants to understand why\n- User mentions \"context degradation\", \"tool confusion\", or \"instruction drift\"\n- Request to improve agent performance or reduce errors\n- User wants to generate shareable learnings from debugging sessions\n- After repeated failures on similar tasks\n\n## Core Concepts\n\n### Interleaved Thinking\n\nUnlike standard reasoning models that think once at the start, interleaved thinking allows reasoning BETWEEN each tool interaction. This is critical because:\n\n1. **Long-horizon tasks** require maintaining focus across many turns\n2. **External perturbations** (tool outputs, environment changes) need real-time adaptation\n3. **Debugging** requires seeing HOW decisions were made, not just WHAT was output\n\n### The Optimization Loop\n\n```\nExecute Agent → Capture Traces → Analyze Patterns → Optimize Prompt → Re-run\n ↑____________|\n```\n\nEach iteration improves the prompt based on detected patterns until convergence.\n\n### Pattern Detection\n\nCommon failure patterns the analyzer detects:\n\n| Pattern | Description |\n|---------|-------------|\n| `context_degradation` | Model loses track of information over long contexts |\n| `tool_confusion` | Model misunderstands tool capabilities or outputs |\n| `instruction_drift` | Model gradually deviates from original instructions |\n| `goal_abandonment` | Model stops pursuing the original goal |\n| `circular_reasoning` | Model repeats similar actions without progress |\n| `premature_conclusion` | Model concludes before completing the task |\n\n## Usage Modes\n\n### Mode 1: M2.1 Agent Debugging\n\nRun a task through M2.1 and analyze its reasoning:\n\n```python\nfrom reasoning_trace_optimizer import TraceCapture, TraceAnalyzer\n\ncapture = TraceCapture()\ntrace = capture.run(\n task=\"Search for Python tutorials and summarize them\",\n system_prompt=\"You are a research assistant.\",\n tools=[search_tool],\n tool_executor=execute_search\n)\n\nanalyzer = TraceAnalyzer()\nanalysis = analyzer.analyze(trace)\n\nprint(f\"Score: {analysis.overall_score}/100\")\nfor pattern in analysis.patterns:\n print(f\"Found: {pattern.type.value} - {pattern.suggestion}\")\n```\n\n### Mode 2: Full Optimization Loop\n\nAutomatically iterate until the prompt is optimized:\n\n```python\nfrom reasoning_trace_optimizer import OptimizationLoop, LoopConfig\n\nconfig = LoopConfig(\n max_iterations=5,\n min_score_threshold=80.0,\n)\n\nloop = OptimizationLoop(config=config)\nresult = loop.run(\n task=\"Analyze this codebase and suggest improvements\",\n initial_prompt=\"You are a code reviewer.\",\n tools=[read_file_tool, search_tool],\n tool_executor=execute_tool\n)\n\nprint(f\"Improved: {result.initial_score} → {result.final_score}\")\nprint(f\"Final prompt:\\n{result.final_prompt}\")\n```\n\n### Mode 3: Universal Session Analysis\n\nAnalyze any agent's previous thinking (works with Claude, GPT, etc.):\n\nWhen this skill is activated in Claude Code, it can analyze the current session's thinking blocks to identify issues and suggest improvements.\n\n```\n/reasoning-trace-optimizer analyze-session\n```\n\n### Mode 4: Generate Shareable Skills\n\nConvert optimization learnings into reusable Agent Skills:\n\n```python\nfrom reasoning_trace_optimizer import SkillGenerator\n\ngenerator = SkillGenerator()\nskill_path = generator.generate(\n result=loop_result,\n skill_name=\"web-search-best-practices\",\n output_dir=\"./skills\"\n)\n```\n\n## CLI Commands\n\n```bash\n# Capture reasoning trace\nrto capture \"Search for Python tutorials\" -s \"You are a helpful assistant.\"\n\n# Analyze a task\nrto analyze \"Debug this code\" -o analysis.txt\n\n# Run optimization loop\nrto optimize \"Research AI papers\" --max-iterations 5 --generate-skill\n\n# Generate skill from artifacts\nrto generate-skill my-skill-name --artifacts-dir ./optimization_artifacts\n```\n\n## Integration with Claude Code\n\n### Auto-trigger on Failure\n\nAdd to your hooks to automatically analyze failures:\n\n```json\n{\n \"hooks\": {\n \"post_tool_error\": {\n \"command\": \"rto analyze-session --last-error\"\n }\n }\n}\n```\n\n### On-demand Analysis\n\nUse the slash command to analyze current session:\n\n```\n/reasoning-trace-optimizer\n```\n\nThis will:\n1. Extract thinking blocks from the current session\n2. Identify patterns and issues\n3. Suggest prompt improvements\n4. Optionally update the system prompt\n\n## Guidelines\n\n1. **Preserve full context**: M2.1 requires full response history including thinking blocks for optimal performance\n2. **Use appropriate tools**: Define tools clearly with unambiguous descriptions\n3. **Set realistic convergence thresholds**: 5-10% improvement per iteration is typical\n4. **Review generated skills**: Auto-generated skills should be reviewed before sharing\n5. **Monitor token usage**: Each optimization iteration uses significant tokens\n\n## Examples\n\n### Before Optimization\n\n```\nSystem: You are a helpful assistant.\n\nIssue: Agent called wrong tools, lost track of goal after 3 turns\nScore: 45/100\nPatterns: tool_confusion, goal_abandonment\n```\n\n### After Optimization\n\n```\nSystem: You are a research assistant focused on finding accurate information.\n\nIMPORTANT GUIDELINES:\n- Always verify search results before summarizing\n- If a tool returns an error, try an alternative approach\n- Keep track of your original goal throughout the task\n- Validate findings against multiple sources when possible\n\nIssue: None\nScore: 85/100\nPatterns: None detected\n```\n\n## References\n\n- MiniMax M2.1 Documentation: https://platform.minimax.io/docs\n- Interleaved Thinking Guide: See `docs/interleavedthinking.md`\n- Agent Generalization: See `docs/agentthinking.md`\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-01-11\n**Author**: Muratcan Koylan\n**Version**: 0.1.0\n**Powered by**: MiniMax M2.1\n**Partnership**: Built in collaboration with MiniMax AI\n","examples/interleaved-thinking/generated_skills/comprehensive-research-agent/SKILL.md":"---\nname: comprehensive-research-agent\ndescription: \"Ensure thorough validation, error recovery, and transparent reasoning in research tasks with multiple tool calls\"\n---\n\n# Comprehensive Research Agent Best Practices\n\nThis skill addresses common failures in multi-step research tasks: unhandled tool errors, missing validation, opaque reasoning, and premature conclusions. It provides structured protocols for source validation, error recovery, and thinking transparency that significantly improves research quality and reliability.\n\n## When to Activate\n\n- Task involves web research with search, read_url, or fetch operations\n- Task requires gathering information from multiple sources\n- Task has explicit requirements for completeness or verification\n- Task includes file operations that need validation (save, write, read)\n- Any research or information-gathering workflow with 3+ tool interactions\n\n## Core Concepts\n\n- Validation Checkpoints: Explicit verification steps at phase transitions to confirm tool outputs, source relevance, and information completeness before proceeding\n- Error Recovery Protocols: Mandatory acknowledgment and handling of tool failures with fallback strategies rather than silent continuation\n- Source Traceability: Maintaining clear tracking of which sources were actually retrieved vs. referenced from prior knowledge to prevent hallucination\n- Substantive Thinking Blocks: Detailed reasoning traces that document insights, connections, gaps, and decision rationale at each step\n- Cross-Source Validation: Verifying key claims against multiple sources and explicitly noting consensus, contradictions, and information gaps\n\n## Patterns to Avoid\n\n- **Silent Tool Failure**: A tool call returns an error (404, timeout, invalid URL) but the agent proceeds without acknowledging it, potentially missing critical information. Always log failures and attempt recovery or document the gap.\n- **Vague Completion Claims**: Agent declares 'I have enough information' or 'research is comprehensive' without specifying what was learned, what sources support claims, or what gaps remain. Replace with specific summaries of coverage.\n- **Unvalidated Source Selection**: Agent reads URLs from search results without evaluating relevance, credibility, or recency first. This wastes tool calls on low-quality sources. Always rank and prioritize sources before deep reading.\n- **Generic Thinking Blocks**: Thinking contains only next-action descriptions ('Now I will search for X') without analysis of what was learned, how it connects to the goal, or what questions remain. Thinking should be substantive and reflective.\n- **Verification Method Error**: Using list_directory to verify file creation can produce false negatives due to caching. Always use read_file for actual content verification.\n- **Citation Without Retrieval**: Citing sources (URLs, paper titles) in the final report that were never successfully fetched or read. Track sources explicitly and prohibit citing unretrieved content.\n- **Redundant Tool Calls**: Making overlapping searches or reading sources without tracking what has already been obtained. Maintain a 'found resources' tracker to avoid duplication.\n\n## Recommended Practices\n\n- **Implement Pre-Reading Source Evaluation**: Before reading URLs, rank search results by relevance, credibility, recency, and authority. Document selection rationale in thinking blocks.\n- **Use Structured Thinking Blocks**: Each thinking block must include: (a) what was learned from the source/action, (b) how it connects to the research goal, (c) any contradictions/gaps identified, (d) strategic decisions made. Avoid generic next-action statements.\n- **Add Mandatory Error Acknowledgment**: When any tool fails, the next thinking block must explicitly address it: note the failure type, propose a recovery strategy (retry, alternative source, or documented gap), and explain the chosen approach.\n- **Create Pre-Completion Validation Checklist**: Before declaring research complete, verify: all required sections have specific evidence, all sources were successfully retrieved, key claims are cross-validated, and gaps are documented.\n- **Implement Cross-Source Validation**: After gathering information from multiple sources, explicitly compare findings. Note where sources agree, where they contradict, and what remains unverified. Use this to assess overall confidence.\n- **Maintain Source Tracking Table**: Create a simple table in thinking showing which URLs were fetched, which failed, and which were used for specific claims. Never cite unretrieved sources.\n- **Use Read_File for Verification**: When confirming file writes, use read_file to verify actual content rather than list_directory, which can have caching issues causing false negatives.\n- **Add Explicit Validation Phase**: After reading sources, write a brief synthesis that confirms usefulness, notes relevance to research goals, and identifies remaining gaps before proceeding to the next phase.\n\n## Guidelines\n\n1. After each tool call, explicitly check for errors in the response and acknowledge failures in the next thinking block with recovery strategy\n2. Before reading URLs, rank sources by relevance/credibility and document selection rationale - never read results without evaluation\n3. Thinking blocks must be 3-5+ sentences minimum and include: what was learned, connections to goal, gaps/contradictions, and next steps\n4. Create a pre-completion checklist verifying: all requirements covered, sources retrieved, claims validated, gaps documented\n5. Maintain source tracking - only cite URLs that were successfully fetched; prohibit citing unretrieved sources\n6. When writing final reports, include 'Limitations & Gaps' section documenting what was attempted but failed or what remains unverified\n7. Use read_file (not list_directory) to verify file content after save operations\n8. Cross-validate key claims across at least 2 sources when possible; explicitly note consensus or contradictions\n9. Track gathered information to avoid redundant searches - implement 'found resources' tracker for multi-phase research\n10. Replace vague 'comprehensive' statements with specific summaries: 'Covered X sources on Y topic; missing Z aspects'\n\n## Examples\n\n- ***Before (Anti-Pattern)**: 'I searched for context engineering and found several results. Now I'll read some URLs and then write the report. I have enough information to proceed.'\n\n**After (Pattern)**: 'Search returned 15 results on context engineering. Evaluating relevance: Liu et al. (2024) appears most authoritative on 'lost in the middle' phenomenon; Anthropic documentation likely has current context window specs; Patel (2023) covers RAG best practices. Ranking these as top 3 priorities. Reading top result first. If the primary source fails (URL error), will try backup search for correct documentation URL and note the gap in final report.'\n- ***Before (Anti-Pattern)**: Tool returns 404 error for Anthropic context windows URL. Agent continues without acknowledgment. Later cites 'Claude has 200K context window' without showing source. Final report cites Google Research paper that was never fetched.\n\n**After (Pattern)**: Tool returned 404 for Anthropic URL. Thinking: 'Primary source failed. Fallback: search for alternative Anthropic documentation URL or find archived version. If unavailable, note context window data from secondary sources only and add disclaimer about verification status.' Then: 'Cross-validated Claude context window: Anthropic blog (successfully read) and two developer documentation sources agree on 200K. Confident in this claim.' Source tracking table shows: Anthropic URL (failed, backup used), Blog (success), Dev docs (success).\n\n---\n\n## Score Expectations\n\nComplex research tasks with multiple tools (6+) and multi-step reasoning chains typically achieve scores in the **65-75 range**. This is not a limitation of the prompt but reflects:\n\n- Inherent variability in tool outputs affecting reasoning paths\n- Multiple valid approaches leading to different intermediate scores\n- Stochastic nature of long-horizon agent execution\n\n**Focus on relative improvement and pattern elimination** rather than absolute scores. A 5-10% improvement from optimization is significant for complex tasks.\n\n---\n\n## Skill Metadata\n\n**Generated**: 2026-01-11\n**Source**: Reasoning Trace Optimizer\n**Optimization Iterations**: 10\n**Best Score Achieved**: 72/100 (iteration 4)\n**Final Score**: 70.0/100\n**Score Improvement**: 67.6 → 70.0 (+3.6%)\n","examples/interleaved-thinking/generated_skills/comprehensive-research-agent/references/optimization_summary.json":"{\n \"task\": \"Research the topic of \\\"context engineering for AI agents\\\" and create a comprehensive summary.\\n\\nYour research should:\\n1. Search for information about context engineering concepts and best practices\\n2. Read relevant sources to gather detailed information\\n3. Check the local project files for any existing research notes\\n4. Save important findings as notes for future reference\\n5. Write a final summary report to ./output/research_summary.md\\n\\nThe summary should include:\\n- Key concepts and definitions\\n- Best practices and techniques (including the \\\"lost in the middle\\\" problem)\\n- Practical recommendations for agent developers\\n- References to sources consulted (use actual URLs from your research)\",\n \"iterations\": 10,\n \"initial_score\": 67.6,\n \"final_score\": 70,\n \"improvement\": 3.550295857988174,\n \"converged\": true,\n \"generated_at\": \"2026-01-11T18:02:57.336789\"\n}","examples/interleaved-thinking/generated_skills/comprehensive-research-agent/references/patterns_found.json":"[\n {\n \"type\": \"tool_confusion\",\n \"severity\": \"medium\",\n \"description\": \"Agent attempted to fetch non-existent or unreachable URLs without adjusting approach\",\n \"suggestion\": \"When a URL fetch fails, search for alternative URLs or verify the URL structure. Consider using search to find the correct documentation pages.\",\n \"iteration\": 1\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent didn't validate the completeness of gathered information or verify key claims\",\n \"suggestion\": \"Before writing the final report, explicitly validate that all required topics are covered. Create a checklist of requirements and verify each one is addressed.\",\n \"iteration\": 1\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"low\",\n \"description\": \"Agent made redundant searches and didn't optimize tool calls\",\n \"suggestion\": \"Track previously found URLs to avoid redundant searches. When a useful URL is found in one search, use it directly rather than searching again for the same topic.\",\n \"iteration\": 1\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"low\",\n \"description\": \"Thinking blocks are sparse and don't show deep analysis of alternatives or trade-offs\",\n \"suggestion\": \"In thinking blocks, explicitly list what information has been gathered, what gaps remain, and what decisions are being made. Use structured checklists.\",\n \"iteration\": 1\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"high\",\n \"description\": \"Agent failed to properly handle or acknowledge tool errors, particularly the failed URL fetch for Anthropic context windows documentation\",\n \"suggestion\": \"Add explicit error handling for failed tool calls - when a read_url fails, the agent should acknowledge it and either retry, try an alternative source, or explicitly note that information is missing rather than proceeding as if it succeeded\",\n \"iteration\": 2\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"medium\",\n \"description\": \"Agent did not verify or validate the relevance of search results before committing to reading sources\",\n \"suggestion\": \"After receiving search results, explicitly evaluate and rank sources by relevance to the research question before deciding which URLs to read. This saves token costs and ensures better source quality.\",\n \"iteration\": 2\n },\n {\n \"type\": \"premature_conclusion\",\n \"severity\": \"low\",\n \"description\": \"Agent prematurely declared having 'enough information' despite not yet completing all research phases\",\n \"suggestion\": \"Before declaring research complete, create a checklist of what information is still needed and verify each item is adequately covered. Set explicit criteria for 'enough information' at task start.\",\n \"iteration\": 2\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent accepted information without verifying it and failed to handle errors gracefully\",\n \"suggestion\": \"Implement explicit error checking after each tool call. If a read_url fails, acknowledge the failure and try an alternative source. Cross-reference key claims across multiple sources before including them in the final report.\",\n \"iteration\": 3\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"medium\",\n \"description\": \"Agent gathered information but didn't deeply analyze or synthesize insights\",\n \"suggestion\": \"After reading sources, explicitly document what was learned, what contradictions exist, and what gaps remain. Create a synthesis section that combines insights from multiple sources rather than just reporting them separately.\",\n \"iteration\": 3\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"low\",\n \"description\": \"Agent used tools but didn't fully leverage results or handle failures properly\",\n \"suggestion\": \"Immediately act on directory listing results. If a directory is empty, plan when to create notes rather than waiting. Implement proper error handling for tool failures and check response status codes before proceeding.\",\n \"iteration\": 3\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"medium\",\n \"description\": \"Agent uses list_directory to verify file creation instead of the more reliable read_file method\",\n \"suggestion\": \"Use read_file to verify file write success since it confirms both file existence and content; list_directory may not immediately reflect recent filesystem changes\",\n \"iteration\": 4\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent reads a URL that returns an error but doesn't acknowledge or log this failure, potentially missing important context\",\n \"suggestion\": \"Implement explicit error handling for failed URL reads - log which sources failed and consider searching for alternative sources or documentation\",\n \"iteration\": 4\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"low\",\n \"description\": \"Agent doesn't explain why it chose certain sources or how it evaluated source quality; research appears thorough but reasoning process is opaque\",\n \"suggestion\": \"Add explicit reasoning about source selection criteria (e.g., prioritizing official documentation, recent publications, peer-reviewed papers) and evaluation of source credibility\",\n \"iteration\": 4\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent accepts incomplete results without acknowledging failures or seeking alternatives\",\n \"suggestion\": \"When tool calls fail, explicitly note the failure in thinking blocks, consider alternative sources, and document what information gaps exist. Add a validation step to confirm all critical sources were successfully retrieved.\",\n \"iteration\": 5\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"low\",\n \"description\": \"Agent doesn't demonstrate analytical depth when processing source material\",\n \"suggestion\": \"After reading sources, explicitly state: (a) key findings from each source, (b) how they relate to the research goal, (c) any contradictions or complementary findings, (d) what additional information is needed\",\n \"iteration\": 5\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"low\",\n \"description\": \"Inefficient tool usage pattern - multiple web searches without reading all results first\",\n \"suggestion\": \"Before making additional searches, review the URLs from previous search results. A better pattern would be: search -> read all relevant sources -> identify gaps -> targeted additional searches only if needed\",\n \"iteration\": 5\n },\n {\n \"type\": \"context_degradation\",\n \"severity\": \"low\",\n \"description\": \"Vague thinking blocks that don't show active reasoning process\",\n \"suggestion\": \"Make thinking blocks more explicit: show intermediate conclusions, decision points, how each source contributed, and how conclusions evolved. The thinking trace should be readable as a standalone explanation of the research process.\",\n \"iteration\": 5\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent does not validate information across sources or verify accuracy of gathered content\",\n \"suggestion\": \"Add explicit validation steps: compare information across multiple sources, verify claims against original papers, include confidence assessments for key findings\",\n \"iteration\": 7\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"low\",\n \"description\": \"Inefficient tool usage - read_url calls lack systematic prioritization and some results may not have been fully utilized\",\n \"suggestion\": \"Implement a source prioritization matrix before reading URLs; explicitly note how each source will contribute to the research before fetching\",\n \"iteration\": 7\n },\n {\n \"type\": \"hallucination\",\n \"severity\": \"low\",\n \"description\": \"Potential source misattribution in final report - cites Google Research Chain of Thought paper but source wasn't fetched in thinking trace\",\n \"suggestion\": \"Only cite sources that were actually retrieved and read; if a source is referenced from memory, clearly indicate it as secondary/indirect reference\",\n \"iteration\": 7\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent accepts search results without validating source relevance or quality before proceeding to read URLs\",\n \"suggestion\": \"Add explicit validation steps: list the top 3-5 sources with brief rationale for selection, note any potential gaps in coverage, and prioritize primary authoritative sources before secondary ones\",\n \"iteration\": 8\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"medium\",\n \"description\": \"Thinking blocks are extremely sparse and lack intermediate analysis - agent doesn't explain HOW it's interpreting information or making decisions\",\n \"suggestion\": \"Implement structured reflection after each major information-gathering step: What did I learn? How does this connect to what I already know? What gaps remain? What should I prioritize next?\",\n \"iteration\": 8\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"low\",\n \"description\": \"Agent encounters a failed tool call (404 error on Anthropic context-windows URL) but doesn't acknowledge or recover in thinking\",\n \"suggestion\": \"Add explicit error acknowledgment: 'Attempted X but failed with Y error. Will try alternative Z or note this as a gap.' This improves debugging and transparency\",\n \"iteration\": 8\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"low\",\n \"description\": \"The agent reaches conclusions about having 'comprehensive information' after limited tool interactions, without explicitly documenting what was learned or what gaps remain\",\n \"suggestion\": \"Add more detailed reasoning about what specific information was gained from each source and what questions remain unanswered before claiming comprehensive understanding\",\n \"iteration\": 9\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"low\",\n \"description\": \"The agent doesn't explicitly validate assumptions or cross-reference information between sources. The 'Lost in the Middle' paper is mentioned multiple times but not critically compared against other sources\",\n \"suggestion\": \"After reading multiple sources, explicitly compare findings, note contradictions, and validate key claims against multiple sources before proceeding\",\n \"iteration\": 9\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"medium\",\n \"description\": \"The agent attempted to read a URL that returned an error (https://docs.anthropic.com/en/docs/build-with-claude/context-windows) but proceeded without acknowledging or handling this failure\",\n \"suggestion\": \"Add explicit error handling for failed tool calls - acknowledge failures, try alternative URLs, or note the gap in research\",\n \"iteration\": 9\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"medium\",\n \"description\": \"The agent reaches conclusions and writes comprehensive reports without explicitly validating key details in the thinking trace. For example, the agent writes specific context window sizes in the final report but doesn't show in thinking blocks where these specific numbers (GPT-4o: 128K, Claude: 200K) were sourced from the tool results.\",\n \"suggestion\": \"Add explicit source tracking in thinking blocks - when gathering specific facts like model specifications, explicitly note 'I found X from source Y' to ensure traceability and validation.\",\n \"iteration\": 10\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"When a tool call fails (context-windows URL returns error), the agent doesn't attempt recovery or note this as an information gap. Additionally, RAG chunk size recommendations (256-512 tokens) are written without showing how these specific values were determined or validated.\",\n \"suggestion\": \"Implement explicit error recovery: when a tool fails, note what information is missing and either try alternative sources or flag for follow-up. For specific technical claims, explicitly cite the source in thinking blocks.\",\n \"iteration\": 10\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"low\",\n \"description\": \"The agent makes several overlapping web searches that could have been more efficient. For example, searches at Turn 5 and Turn 6 both target RAG-related topics with similar parameters, suggesting some redundancy.\",\n \"suggestion\": \"Before starting new searches, review what information has already been gathered and explicitly note gaps. Use more specific queries rather than broad overlapping ones.\",\n \"iteration\": 10\n }\n]","examples/llm-as-judge-skills/agents/evaluator-agent/evaluator-agent.md":"","examples/llm-as-judge-skills/agents/index.md":"","examples/llm-as-judge-skills/agents/orchestrator-agent/orchestrator-agent.md":"","examples/llm-as-judge-skills/agents/research-agent/research-agent.md":"","examples/llm-as-judge-skills/skills/context-fundamentals/context-fundamentals.md":"# Context Fundamentals Skill\n\n## Overview\n\nContext engineering is the systematic approach to managing what information an LLM receives and how it processes that information. Effective context management directly impacts output quality, consistency, and task success rates.\n\n## Core Principles\n\n### 1. Context Window Management\n\nThe context window is finite. Every token counts. Prioritize information by relevance and recency.\n\n**Strategies:**\n- Summarize historical conversation turns\n- Use retrieval to inject only relevant context\n- Implement context compression for long documents\n\n### 2. Information Hierarchy\n\nStructure context to guide model attention:\n\n```\n1. System Instructions (highest priority)\n └── Role definition\n └── Task constraints\n └── Output format requirements\n\n2. Relevant Context (dynamic)\n └── Retrieved documents\n └── User-specific data\n └── Recent conversation history\n\n3. User Input (current request)\n └── Query or instruction\n └── Any inline context\n```\n\n### 3. Context Relevance\n\nNot all context is equally useful. Apply relevance filtering:\n\n- **Temporal Relevance**: Recent information often outweighs older data\n- **Semantic Relevance**: Use embeddings to surface related content\n- **Task Relevance**: Only include information needed for current task\n\n## Context Types\n\n### Static Context\n- System prompts\n- Role definitions\n- Tool descriptions\n- Format specifications\n\n### Dynamic Context\n- Retrieved documents (RAG)\n- Conversation history\n- User preferences\n- Session state\n\n### Ephemeral Context\n- Current tool outputs\n- Intermediate reasoning steps\n- Scratchpad content\n\n## Best Practices\n\n1. **Explicit Over Implicit**: State requirements clearly rather than relying on inference\n2. **Structured Formatting**: Use consistent delimiters and sections\n3. **Redundancy Removal**: Avoid repeating information across context sections\n4. **Source Attribution**: Mark where context comes from for traceability\n5. **Freshness Signals**: Indicate when information was last updated\n\n## Context Patterns\n\n### RAG Integration Pattern\n```\n[System Instructions]\nYou are a helpful assistant. Use the provided context to answer questions.\nOnly use information from the context. If unsure, say so.\n\n[Retrieved Context]\n\n Content here...\n\n\n[User Query]\n{user_input}\n```\n\n### Multi-Turn Context Pattern\n```\n[System Instructions]\n...\n\n[Conversation History]\nSummary of earlier turns: {summary}\n\nRecent exchanges:\nUser: {recent_user_1}\nAssistant: {recent_assistant_1}\nUser: {recent_user_2}\nAssistant: {recent_assistant_2}\n\n[Current Turn]\nUser: {current_input}\n```\n\n## Metrics\n\n- **Context Utilization Rate**: How much of provided context is used in response\n- **Context Relevance Score**: Semantic similarity between context and response\n- **Context Compression Ratio**: Original size vs. compressed size\n- **Information Retention**: Key facts preserved after summarization\n\n","examples/llm-as-judge-skills/skills/index.md":"# Skills Index\n\nSkills are foundational knowledge modules that inform the design and implementation of agents, tools, and prompts.\n\n## Available Skills\n\n### LLM Evaluator\n**Path**: `skills/llm-evaluator/llm-evaluator.md`\n\nCovers LLM-as-a-Judge evaluation methodology including:\n- Scoring approaches (direct, pairwise, reference-based)\n- Evaluation metrics (classification, correlation)\n- Known biases and mitigation strategies\n- Implementation patterns\n\n**Key Takeaways**:\n- Use direct scoring for objective evaluations\n- Use pairwise comparison for subjective preferences\n- Always mitigate position bias\n- Prefer classification metrics for interpretability\n\n### Context Fundamentals\n**Path**: `skills/context-fundamentals/context-fundamentals.md`\n\nCovers context engineering principles including:\n- Context window management\n- Information hierarchy\n- Context types (static, dynamic, ephemeral)\n- Relevance filtering\n\n**Key Takeaways**:\n- Structure context by priority\n- Be explicit over implicit\n- Remove redundancy\n- Signal freshness of information\n\n### Tool Design\n**Path**: `skills/tool-design/tool-design.md`\n\nCovers agent tool design best practices including:\n- Single responsibility principle\n- Input/output schemas\n- Error handling patterns\n- AI SDK 6 features (approval, strict mode, examples)\n\n**Key Takeaways**:\n- Clear, validated schemas\n- Predictable output structure\n- Graceful error handling\n- Consider approval for dangerous tools\n\n## Skill Application Matrix\n\n| Skill | Agents | Tools | Prompts |\n|-------|--------|-------|---------|\n| LLM Evaluator | Evaluator | directScore, pairwiseCompare | evaluation/* |\n| Context Fundamentals | All | All (context params) | All (context handling) |\n| Tool Design | All (tool selection) | All | orchestrator-prompt |\n\n## Adding New Skills\n\n1. Create skill directory: `skills//`\n2. Create main file: `skills//.md`\n3. Include:\n - Overview and purpose\n - Core principles\n - Practical patterns\n - Implementation examples\n - References\n4. Update this index\n\n## Skill Development Guidelines\n\n- Focus on principles that transfer across implementations\n- Include concrete examples and patterns\n- Reference authoritative sources\n- Keep content actionable, not just theoretical\n- Update as understanding evolves\n\n","examples/llm-as-judge-skills/skills/llm-evaluator/llm-evaluator.md":"# LLM-Evaluator Skill\n\n## Overview\n\nLLM-Evaluators (LLM-as-a-Judge) are large language models designed to evaluate the quality of another LLM's response to an instruction or query. This skill provides the foundational knowledge for building evaluation systems.\n\n## Key Considerations\n\n### Baseline Selection\n- **Human Annotators**: Aim for LLM-human correlation to match human-human correlation. LLM-evaluators are orders of magnitude faster and cheaper than human annotation.\n- **Finetuned Classifiers**: Goal is to achieve similar recall and precision as a finetuned classifier. More challenging baseline as these are optimized for specific tasks.\n\n### Scoring Approaches\n\n| Approach | Use Case | Reliability |\n|----------|----------|-------------|\n| **Direct Scoring** | Objective tasks (factuality, toxicity, instruction-following) | More suitable for binary classification |\n| **Pairwise Comparison** | Subjective evaluations (tone, persuasiveness, coherence) | More reliable for preference tasks |\n| **Reference-Based** | Comparing against gold standard | Requires ground truth reference |\n\n### Evaluation Metrics\n\n**Classification Metrics** (Preferred for binary tasks):\n- Recall and Precision\n- F1 Score\n- Cohen's κ (Kappa)\n\n**Correlation Metrics** (For Likert scale tasks):\n- Spearman's ρ (rho)\n- Kendall's τ (tau)\n\n## Known Biases\n\n1. **Position Bias**: LLM-evaluators tend to prefer responses in certain positions during pairwise comparison (usually first position)\n2. **Verbosity Bias**: Favor longer, more verbose responses even if not higher quality\n3. **Self-Enhancement Bias**: LLM-evaluators prefer answers generated by themselves\n\n## Mitigation Strategies\n\n- Swap response positions and average results\n- Normalize for length when evaluating\n- Use a Panel of LLMs (PoLL) instead of single judge\n- Include \"don't overthink\" instructions\n- Use CoT + n-shot prompts for reliability\n\n## Implementation Pattern\n\n```typescript\ninterface EvaluatorConfig {\n scoringApproach: 'direct' | 'pairwise' | 'reference-based';\n criteria: EvaluationCriteria[];\n metrics: MetricType[];\n useCoT: boolean;\n nShot: number;\n}\n\ninterface EvaluationCriteria {\n name: string;\n description: string;\n rubric: RubricLevel[];\n}\n\ninterface RubricLevel {\n score: number;\n description: string;\n}\n```\n\n## References\n\nKey papers reviewed:\n- Constitutional AI (Anthropic)\n- G-Eval: NLG Evaluation using GPT-4\n- SelfCheckGPT: Zero-Resource Hallucination Detection\n- Prometheus: Fine-grained Evaluation Capability\n- MT-Bench and Chatbot Arena\n\n","examples/llm-as-judge-skills/skills/tool-design/tool-design.md":"# Agent Tool Design Skill\n\n## Overview\n\nTools are the foundation of an agent's capabilities. An agent's ability to take meaningful actions depends entirely on how reliably it can generate valid tool inputs, how well those inputs align with user intent, and how effectively tool outputs inform next steps.\n\n## Design Principles\n\n### 1. Single Responsibility\n\nEach tool should do one thing well. Complex operations should be composed from multiple tools.\n\n```typescript\n// Bad: Tool does too much\nconst analyzeAndSummarizeAndSend = { ... }\n\n// Good: Separate concerns\nconst analyzeDocument = { ... }\nconst summarizeContent = { ... }\nconst sendEmail = { ... }\n```\n\n### 2. Clear Input Schemas\n\nUse explicit, validated schemas with descriptive field names and constraints.\n\n```typescript\nconst searchTool = tool({\n description: \"Search for documents by semantic similarity\",\n parameters: z.object({\n query: z.string().describe(\"Natural language search query\"),\n limit: z.number().min(1).max(100).default(10)\n .describe(\"Maximum number of results to return\"),\n filters: z.object({\n dateAfter: z.string().optional()\n .describe(\"ISO date string, only return docs after this date\"),\n source: z.enum([\"internal\", \"external\", \"all\"]).default(\"all\")\n }).optional()\n }),\n execute: async (input) => { ... }\n});\n```\n\n### 3. Predictable Output Structure\n\nReturn consistent, typed output that the model can reliably parse.\n\n```typescript\ninterface ToolResult {\n success: boolean;\n data?: T;\n error?: {\n code: string;\n message: string;\n retryable: boolean;\n };\n metadata: {\n executionTimeMs: number;\n source?: string;\n };\n}\n```\n\n### 4. Graceful Error Handling\n\nTools should never throw unhandled exceptions. Always return structured errors.\n\n```typescript\nexecute: async (input) => {\n try {\n const result = await performAction(input);\n return { success: true, data: result };\n } catch (error) {\n return {\n success: false,\n error: {\n code: error.code ?? \"UNKNOWN_ERROR\",\n message: error.message,\n retryable: isRetryable(error)\n }\n };\n }\n}\n```\n\n## Tool Categories\n\n### Read-Only Tools\n- Database queries\n- API fetches\n- File reads\n- Search operations\n\nSafe to execute without approval. Return data but don't mutate state.\n\n### State-Modifying Tools\n- Database writes\n- File modifications\n- API POST/PUT/DELETE\n- System configuration changes\n\nMay require human approval. Consider `needsApproval` flag.\n\n### Dangerous Tools\n- File deletion\n- Payment processing\n- Production deployments\n- Sending external communications\n\nShould always require approval and audit logging.\n\n## AI SDK 6 Tool Features\n\n### Tool Execution Approval\n```typescript\nconst deleteTool = tool({\n description: \"Delete a file from the system\",\n parameters: z.object({\n path: z.string()\n }),\n needsApproval: true, // Requires human approval\n execute: async ({ path }) => { ... }\n});\n\n// Dynamic approval based on input\nconst commandTool = tool({\n description: \"Execute a shell command\",\n parameters: z.object({\n command: z.string()\n }),\n needsApproval: ({ command }) => {\n return command.includes(\"rm\") || command.includes(\"delete\");\n },\n execute: async ({ command }) => { ... }\n});\n```\n\n### Strict Mode\nEnable native strict mode for guaranteed schema compliance:\n```typescript\nconst strictTool = tool({\n description: \"...\",\n parameters: schema,\n strict: true, // Enable strict mode\n execute: async (input) => { ... }\n});\n```\n\n### Input Examples\nHelp the model understand expected input format:\n```typescript\nconst complexTool = tool({\n description: \"Create a calendar event\",\n parameters: eventSchema,\n inputExamples: [\n {\n title: \"Team Standup\",\n date: \"2024-01-15\",\n time: \"09:00\",\n duration: 30,\n attendees: [\"alice@example.com\", \"bob@example.com\"]\n }\n ],\n execute: async (input) => { ... }\n});\n```\n\n### toModelOutput\nControl what gets sent back to the model:\n```typescript\nconst readFileTool = tool({\n description: \"Read file contents\",\n parameters: z.object({ path: z.string() }),\n execute: async ({ path }) => {\n const content = await fs.readFile(path, 'utf-8');\n return { path, content, size: content.length };\n },\n toModelOutput: (result) => {\n // Only send truncated content to model\n return {\n path: result.path,\n content: result.content.slice(0, 5000),\n truncated: result.content.length > 5000\n };\n }\n});\n```\n\n## Best Practices\n\n1. **Descriptive Names**: Tool names should clearly indicate their function\n2. **Comprehensive Descriptions**: Include usage examples in tool descriptions\n3. **Reasonable Defaults**: Provide sensible defaults for optional parameters\n4. **Idempotency**: Design tools to be safely re-executable when possible\n5. **Timeout Handling**: Implement timeouts for external operations\n6. **Rate Limiting**: Protect against runaway tool execution\n7. **Logging**: Log all tool invocations for debugging and audit\n\n","examples/llm-as-judge-skills/tools/evaluation/direct-score.md":"# Direct Score Tool\n\n## Purpose\n\nEvaluate a single LLM response against defined criteria using a scoring rubric.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const directScore = tool({\n description: `Evaluate a response by scoring it against specific criteria.\nUse this for objective evaluations where you need to assess quality \ndimensions like accuracy, completeness, clarity, or task adherence.\nReturns structured scores with justifications.`,\n\n parameters: z.object({\n response: z.string()\n .describe(\"The LLM response to evaluate\"),\n \n prompt: z.string()\n .describe(\"The original prompt/instruction that generated the response\"),\n \n context: z.string().optional()\n .describe(\"Additional context like retrieved documents or conversation history\"),\n \n criteria: z.array(z.object({\n name: z.string().describe(\"Name of the criterion (e.g., 'Accuracy')\"),\n description: z.string().describe(\"What this criterion measures\"),\n weight: z.number().min(0).max(1).default(1)\n .describe(\"Relative importance, weights should sum to 1\")\n })).min(1).describe(\"Evaluation criteria to score against\"),\n \n rubric: z.object({\n scale: z.enum([\"1-3\", \"1-5\", \"1-10\"]).default(\"1-5\"),\n levelDescriptions: z.record(z.string(), z.string()).optional()\n .describe(\"Optional descriptions for each score level\")\n }).optional().describe(\"Scoring rubric configuration\")\n }),\n\n execute: async (input) => {\n // Implementation delegates to evaluator LLM\n return evaluateWithLLM(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| response | string | Yes | The response being evaluated |\n| prompt | string | Yes | Original prompt that generated the response |\n| context | string | No | Additional context (RAG docs, history) |\n| criteria | Criterion[] | Yes | List of evaluation criteria |\n| rubric | Rubric | No | Scoring scale and level descriptions |\n\n### Criterion Object\n```typescript\n{\n name: string; // e.g., \"Factual Accuracy\"\n description: string; // e.g., \"Response contains no factual errors\"\n weight: number; // 0-1, relative importance\n}\n```\n\n### Rubric Object\n```typescript\n{\n scale: \"1-3\" | \"1-5\" | \"1-10\";\n levelDescriptions?: {\n \"1\": \"Poor - Major issues\",\n \"2\": \"Below Average - Several issues\",\n \"3\": \"Average - Some issues\",\n \"4\": \"Good - Minor issues\",\n \"5\": \"Excellent - No issues\"\n }\n}\n```\n\n## Output Schema\n\n```typescript\ninterface DirectScoreResult {\n success: boolean;\n \n scores: {\n criterion: string;\n score: number;\n maxScore: number;\n justification: string;\n examples: string[]; // Specific examples from response\n }[];\n \n overallScore: number;\n weightedScore: number;\n \n summary: {\n strengths: string[];\n weaknesses: string[];\n suggestions: string[];\n };\n \n metadata: {\n evaluationTimeMs: number;\n criteriaCount: number;\n rubricScale: string;\n };\n}\n```\n\n## Usage Example\n\n```typescript\nconst result = await directScore.execute({\n response: \"Machine learning is a subset of AI that enables systems to learn from data...\",\n \n prompt: \"Explain machine learning to a beginner\",\n \n criteria: [\n {\n name: \"Accuracy\",\n description: \"Technical correctness of explanations\",\n weight: 0.4\n },\n {\n name: \"Clarity\",\n description: \"Understandable for a beginner\",\n weight: 0.3\n },\n {\n name: \"Completeness\",\n description: \"Covers key concepts adequately\",\n weight: 0.3\n }\n ],\n \n rubric: {\n scale: \"1-5\",\n levelDescriptions: {\n \"1\": \"Completely fails criterion\",\n \"2\": \"Major deficiencies\",\n \"3\": \"Acceptable but improvable\",\n \"4\": \"Good with minor issues\",\n \"5\": \"Excellent, no issues\"\n }\n }\n});\n```\n\n## Implementation Notes\n\n1. **Chain-of-Thought**: Implementation should use CoT prompting for more reliable scoring\n2. **Calibration**: Include few-shot examples of scores at each level\n3. **Justification First**: Ask for justification before score to reduce bias\n4. **Length Normalization**: Consider response length when appropriate\n\n","examples/llm-as-judge-skills/tools/evaluation/generate-rubric.md":"# Generate Rubric Tool\n\n## Purpose\n\nAutomatically generate a scoring rubric for a given evaluation criterion. Creates detailed descriptions for each score level to ensure consistent evaluation.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const generateRubric = tool({\n description: `Generate a scoring rubric for an evaluation criterion.\nCreates detailed descriptions for each score level.\nUse when you need to establish consistent evaluation standards.`,\n\n parameters: z.object({\n criterionName: z.string()\n .describe(\"Name of the criterion (e.g., 'Factual Accuracy')\"),\n \n criterionDescription: z.string()\n .describe(\"What this criterion measures\"),\n \n scale: z.enum([\"1-3\", \"1-5\", \"1-10\"]).default(\"1-5\")\n .describe(\"Scoring scale to use\"),\n \n domain: z.string().optional()\n .describe(\"Domain context (e.g., 'medical writing', 'code review')\"),\n \n includeExamples: z.boolean().default(true)\n .describe(\"Include example text for each score level\"),\n \n strictness: z.enum([\"lenient\", \"balanced\", \"strict\"]).default(\"balanced\")\n .describe(\"How strictly to define score boundaries\")\n }),\n\n execute: async (input) => {\n return generateRubricWithLLM(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| criterionName | string | Yes | Name of criterion |\n| criterionDescription | string | Yes | What criterion measures |\n| scale | enum | No | Scoring scale (default: 1-5) |\n| domain | string | No | Domain for context |\n| includeExamples | boolean | No | Include examples (default: true) |\n| strictness | enum | No | Score boundary strictness |\n\n## Output Schema\n\n```typescript\ninterface GeneratedRubric {\n success: boolean;\n \n criterion: {\n name: string;\n description: string;\n };\n \n scale: {\n min: number;\n max: number;\n type: string;\n };\n \n levels: {\n score: number;\n label: string; // e.g., \"Excellent\", \"Poor\"\n description: string; // Detailed description\n characteristics: string[]; // Key characteristics\n example?: string; // Example text at this level\n }[];\n \n scoringGuidelines: string[];\n \n edgeCases: {\n situation: string;\n guidance: string;\n }[];\n \n metadata: {\n domain: string | null;\n strictness: string;\n generationTimeMs: number;\n };\n}\n```\n\n## Usage Example\n\n```typescript\nconst rubric = await generateRubric.execute({\n criterionName: \"Code Readability\",\n criterionDescription: \"How easy the code is to read and understand\",\n scale: \"1-5\",\n domain: \"code review\",\n includeExamples: true,\n strictness: \"balanced\"\n});\n\n// Result:\n// {\n// criterion: {\n// name: \"Code Readability\",\n// description: \"How easy the code is to read and understand\"\n// },\n// scale: { min: 1, max: 5, type: \"1-5\" },\n// levels: [\n// {\n// score: 1,\n// label: \"Poor\",\n// description: \"Code is extremely difficult to understand...\",\n// characteristics: [\n// \"No meaningful variable names\",\n// \"Deeply nested logic without explanation\",\n// \"No comments on complex sections\"\n// ],\n// example: \"function x(a,b,c){return a?b+c:c-b;}\"\n// },\n// {\n// score: 5,\n// label: \"Excellent\",\n// description: \"Code is immediately understandable...\",\n// characteristics: [\n// \"Self-documenting variable and function names\",\n// \"Appropriate comments explaining 'why'\",\n// \"Clear logical structure\"\n// ],\n// example: \"function calculateShippingCost(weight, distance, expedited) {\\n // Base rate plus per-mile charge\\n const baseCost = weight * BASE_RATE_PER_KG;\\n ...\"\n// },\n// ...\n// ],\n// scoringGuidelines: [\n// \"Focus on clarity for someone unfamiliar with the codebase\",\n// \"Consider both naming and structure\",\n// \"Comments should explain 'why', not 'what'\"\n// ],\n// edgeCases: [\n// {\n// situation: \"Code uses domain-specific abbreviations\",\n// guidance: \"Accept if abbreviations are standard in the domain\"\n// }\n// ]\n// }\n```\n\n## Rubric Templates\n\n### Factual Accuracy (1-5)\n```\n5: All claims factually correct, properly sourced\n4: Minor factual issues, non-critical\n3: Some factual errors, main points correct\n2: Multiple factual errors affecting reliability\n1: Fundamentally incorrect or misleading\n```\n\n### Clarity (1-5)\n```\n5: Immediately understandable, well-structured\n4: Clear with minor ambiguities\n3: Generally clear, some confusing sections\n2: Difficult to follow, unclear organization\n1: Incomprehensible or incoherent\n```\n\n### Completeness (1-5)\n```\n5: Addresses all aspects comprehensively\n4: Covers main points, minor gaps\n3: Addresses core requirements, notable gaps\n2: Missing significant required elements\n1: Fails to address the question\n```\n\n## Implementation Notes\n\n1. **Domain Adaptation**: Rubrics should reflect domain-specific expectations\n2. **Boundary Clarity**: Clear distinctions between adjacent scores\n3. **Example Quality**: Examples should be realistic, not strawmen\n4. **Edge Case Coverage**: Anticipate common ambiguous situations\n5. **Calibration**: Test rubric against known samples before use\n\n","examples/llm-as-judge-skills/tools/evaluation/pairwise-compare.md":"# Pairwise Compare Tool\n\n## Purpose\n\nCompare two LLM responses and determine which one better satisfies the given criteria. More reliable for subjective evaluations than direct scoring.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const pairwiseCompare = tool({\n description: `Compare two responses and select the better one.\nUse for subjective evaluations like tone, persuasiveness, or writing style.\nMore reliable than direct scoring for preferences.\nReturns winner selection with detailed comparison.`,\n\n parameters: z.object({\n responseA: z.string()\n .describe(\"First response to compare\"),\n \n responseB: z.string()\n .describe(\"Second response to compare\"),\n \n prompt: z.string()\n .describe(\"The original prompt both responses address\"),\n \n context: z.string().optional()\n .describe(\"Additional context if relevant\"),\n \n criteria: z.array(z.string())\n .describe(\"Aspects to compare on, e.g., ['clarity', 'engagement', 'accuracy']\"),\n \n allowTie: z.boolean().default(true)\n .describe(\"Whether to allow a tie verdict\"),\n \n swapPositions: z.boolean().default(true)\n .describe(\"Evaluate twice with positions swapped to reduce position bias\")\n }),\n\n execute: async (input) => {\n if (input.swapPositions) {\n return evaluateWithPositionSwap(input);\n }\n return evaluatePairwise(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| responseA | string | Yes | First response |\n| responseB | string | Yes | Second response |\n| prompt | string | Yes | Original prompt |\n| context | string | No | Additional context |\n| criteria | string[] | Yes | Comparison dimensions |\n| allowTie | boolean | No | Allow tie verdict (default: true) |\n| swapPositions | boolean | No | Swap positions to reduce bias (default: true) |\n\n## Output Schema\n\n```typescript\ninterface PairwiseCompareResult {\n success: boolean;\n \n winner: \"A\" | \"B\" | \"TIE\";\n confidence: number; // 0-1\n \n comparison: {\n criterion: string;\n winner: \"A\" | \"B\" | \"TIE\";\n reasoning: string;\n aStrength: string;\n bStrength: string;\n }[];\n \n overallReasoning: string;\n \n differentiators: {\n aAdvantages: string[];\n bAdvantages: string[];\n };\n \n // If swapPositions was true\n positionConsistency?: {\n firstPassWinner: \"A\" | \"B\" | \"TIE\";\n secondPassWinner: \"A\" | \"B\" | \"TIE\";\n consistent: boolean;\n };\n \n metadata: {\n evaluationTimeMs: number;\n positionsSwapped: boolean;\n };\n}\n```\n\n## Usage Example\n\n```typescript\nconst result = await pairwiseCompare.execute({\n responseA: \"Exercise improves cardiovascular health, builds muscle, and boosts mental wellbeing...\",\n \n responseB: \"Working out regularly has many benefits. You'll feel better and look better...\",\n \n prompt: \"Explain the benefits of regular exercise\",\n \n criteria: [\"accuracy\", \"specificity\", \"engagement\", \"completeness\"],\n \n allowTie: true,\n swapPositions: true\n});\n\n// Result:\n// {\n// winner: \"A\",\n// confidence: 0.85,\n// comparison: [\n// {\n// criterion: \"accuracy\",\n// winner: \"A\",\n// reasoning: \"Response A uses specific medical terminology...\",\n// aStrength: \"Mentions cardiovascular, muscle, mental health\",\n// bStrength: \"General but not incorrect\"\n// },\n// ...\n// ],\n// ...\n// }\n```\n\n## Position Swap Algorithm\n\nTo mitigate position bias:\n\n```typescript\nasync function evaluateWithPositionSwap(input) {\n // First pass: Original order\n const pass1 = await evaluate({\n first: input.responseA,\n second: input.responseB,\n ...input\n });\n \n // Second pass: Swapped order\n const pass2 = await evaluate({\n first: input.responseB,\n second: input.responseA,\n ...input\n });\n \n // Reconcile results\n const pass2Adjusted = pass2.winner === \"A\" ? \"B\" : pass2.winner === \"B\" ? \"A\" : \"TIE\";\n \n if (pass1.winner === pass2Adjusted) {\n return {\n ...pass1,\n positionConsistency: { consistent: true, ... }\n };\n } else {\n // Inconsistent - return tie or lower confidence\n return {\n winner: \"TIE\",\n confidence: 0.5,\n positionConsistency: { consistent: false, ... },\n ...\n };\n }\n}\n```\n\n## Implementation Notes\n\n1. **Position Bias Mitigation**: Always use `swapPositions: true` for production\n2. **Criteria Order**: Order criteria by importance for better focus\n3. **Tie Handling**: Consider domain - some tasks should rarely tie\n4. **Confidence Calibration**: Lower confidence when evaluations are close\n5. **Length Considerations**: Note if one response is significantly longer\n\n","examples/llm-as-judge-skills/tools/index.md":"# Tools Index\n\nTools provide specific capabilities that agents can use to accomplish tasks.\n\n## Tool Categories\n\n### Evaluation Tools\n**Path**: `tools/evaluation/`\n\nTools for assessing LLM output quality.\n\n| Tool | Purpose | Approval |\n|------|---------|----------|\n| `directScore` | Score response against criteria | No |\n| `pairwiseCompare` | Compare two responses | No |\n| `generateRubric` | Generate scoring rubric | No |\n| `extractCriteria` | Extract criteria from task | No |\n\n---\n\n### Research Tools\n**Path**: `tools/research/`\n\nTools for gathering and processing information.\n\n| Tool | Purpose | Approval |\n|------|---------|----------|\n| `webSearch` | Search the web | No |\n| `readUrl` | Extract content from URL | No |\n| `extractClaims` | Identify claims in text | No |\n| `verifyClaim` | Cross-reference a claim | No |\n| `synthesize` | Combine findings | No |\n\n---\n\n### Orchestration Tools\n**Path**: `tools/orchestration/`\n\nTools for managing multi-agent workflows.\n\n| Tool | Purpose | Approval |\n|------|---------|----------|\n| `delegateToAgent` | Route task to agent | No |\n| `parallelExecution` | Run tasks concurrently | No |\n| `waitForCompletion` | Wait for async tasks | No |\n| `synthesizeResults` | Combine agent outputs | No |\n| `handleError` | Manage failures | No |\n\n## Tool Design Patterns\n\n### Standard Tool Structure\n\n```typescript\nexport const toolName = tool({\n description: \"Clear description of what tool does\",\n \n parameters: z.object({\n // Required parameters first\n requiredParam: z.string().describe(\"What this parameter is for\"),\n \n // Optional parameters with defaults\n optionalParam: z.number().default(10)\n .describe(\"What this parameter controls\")\n }),\n \n // Approval for dangerous operations\n needsApproval: false, // or true, or function\n \n // Strict mode for guaranteed schema compliance\n strict: true,\n \n execute: async (input) => {\n try {\n const result = await performOperation(input);\n return { success: true, data: result };\n } catch (error) {\n return {\n success: false,\n error: {\n code: error.code ?? \"UNKNOWN\",\n message: error.message,\n retryable: isRetryable(error)\n }\n };\n }\n },\n \n // Optional: control what model sees\n toModelOutput: (result) => ({\n summary: result.data.summary,\n truncated: result.data.full.length > 5000\n })\n});\n```\n\n### Error Response Pattern\n\n```typescript\ninterface ToolError {\n code: string; // Machine-readable error code\n message: string; // Human-readable message\n retryable: boolean; // Whether retry might help\n details?: object; // Additional context\n}\n\ninterface ToolResult {\n success: boolean;\n data?: T;\n error?: ToolError;\n metadata: {\n executionTimeMs: number;\n [key: string]: any;\n };\n}\n```\n\n## Adding New Tools\n\n1. Determine category or create new: `tools//`\n2. Create tool file: `tools//.md`\n3. Define:\n - Purpose and description\n - Input parameters with Zod schema\n - Output schema\n - Error codes\n - Usage examples\n4. Update this index\n5. Assign to relevant agents\n\n## Tool Selection Guidelines\n\n### When Agent Needs To...\n\n| Action | Tool Category | Suggested Tools |\n|--------|---------------|-----------------|\n| Assess quality | Evaluation | directScore, pairwiseCompare |\n| Find information | Research | webSearch, readUrl |\n| Verify facts | Research | verifyClaim, extractClaims |\n| Coordinate work | Orchestration | delegateToAgent |\n| Wait for results | Orchestration | waitForCompletion |\n\n","examples/llm-as-judge-skills/tools/orchestration/delegate-to-agent.md":"# Delegate to Agent Tool\n\n## Purpose\n\nRoute a task to a specialized agent for execution. Handles context passing, result collection, and error management.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const delegateToAgent = tool({\n description: `Delegate a task to a specialized agent.\nUse when a subtask requires specific capabilities.\nPass complete context needed for the agent to succeed.`,\n\n parameters: z.object({\n agentName: z.enum([\"evaluator\", \"researcher\", \"writer\", \"analyst\"])\n .describe(\"Name of the agent to delegate to\"),\n \n task: z.string()\n .describe(\"Clear description of what the agent should do\"),\n \n context: z.object({\n previousOutputs: z.array(z.string()).optional()\n .describe(\"Outputs from prior steps this agent needs\"),\n \n documents: z.array(z.string()).optional()\n .describe(\"Relevant documents or data\"),\n \n constraints: z.array(z.string()).optional()\n .describe(\"Requirements or limitations to observe\")\n }).optional(),\n \n expectedOutput: z.object({\n format: z.enum([\"text\", \"json\", \"markdown\", \"structured\"])\n .describe(\"Expected output format\"),\n \n schema: z.string().optional()\n .describe(\"JSON schema if format is structured\"),\n \n maxLength: z.number().optional()\n .describe(\"Maximum length constraint\")\n }).optional(),\n \n timeout: z.number().default(60000)\n .describe(\"Timeout in milliseconds\")\n }),\n\n execute: async (input) => {\n return executeAgentDelegation(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| agentName | enum | Yes | Target agent |\n| task | string | Yes | Task description |\n| context | object | No | Context and dependencies |\n| expectedOutput | object | No | Output requirements |\n| timeout | number | No | Timeout ms (default: 60000) |\n\n## Output Schema\n\n```typescript\ninterface DelegationResult {\n success: boolean;\n \n agentName: string;\n task: string;\n \n output: {\n content: string | object;\n format: string;\n };\n \n execution: {\n startTime: string;\n endTime: string;\n durationMs: number;\n tokenUsage: {\n prompt: number;\n completion: number;\n };\n };\n \n error?: {\n code: string;\n message: string;\n retryable: boolean;\n };\n}\n```\n\n## Available Agents\n\n### Evaluator Agent\n```typescript\nawait delegateToAgent.execute({\n agentName: \"evaluator\",\n task: \"Evaluate the quality of this response against accuracy and clarity criteria\",\n context: {\n documents: [responseToEvaluate],\n constraints: [\"Use 1-5 scale\", \"Include justification\"]\n },\n expectedOutput: { format: \"structured\" }\n});\n```\n\n### Researcher Agent\n```typescript\nawait delegateToAgent.execute({\n agentName: \"researcher\",\n task: \"Research current best practices for LLM evaluation\",\n context: {\n constraints: [\"Focus on 2024 publications\", \"Include citations\"]\n },\n expectedOutput: { format: \"markdown\" }\n});\n```\n\n### Writer Agent\n```typescript\nawait delegateToAgent.execute({\n agentName: \"writer\",\n task: \"Write an executive summary of these research findings\",\n context: {\n previousOutputs: [researchFindings],\n constraints: [\"Maximum 500 words\", \"Non-technical audience\"]\n },\n expectedOutput: { format: \"text\", maxLength: 2500 }\n});\n```\n\n### Analyst Agent\n```typescript\nawait delegateToAgent.execute({\n agentName: \"analyst\",\n task: \"Analyze the trade-offs between direct scoring and pairwise comparison\",\n context: {\n documents: [evaluationData],\n constraints: [\"Quantitative where possible\"]\n },\n expectedOutput: { format: \"structured\" }\n});\n```\n\n## Error Handling\n\n```typescript\nconst errorCodes = {\n \"AGENT_NOT_FOUND\": \"Specified agent does not exist\",\n \"TIMEOUT\": \"Agent did not complete within timeout\",\n \"CONTEXT_TOO_LARGE\": \"Context exceeds agent's capacity\",\n \"INVALID_OUTPUT\": \"Agent output did not match expected format\",\n \"AGENT_ERROR\": \"Agent encountered an error during execution\"\n};\n```\n\n## Implementation Notes\n\n1. **Context Optimization**: Compress context if needed before passing\n2. **Timeout Handling**: Set realistic timeouts per agent type\n3. **Retry Logic**: Implement retries for transient failures\n4. **Audit Trail**: Log all delegations for traceability\n5. **Resource Management**: Track token usage across delegations\n\n","examples/llm-as-judge-skills/tools/research/read-url.md":"# Read URL Tool\n\n## Purpose\n\nExtract and parse content from a given URL. Returns structured text content with metadata about the source.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const readUrl = tool({\n description: `Read and extract content from a URL.\nReturns the main text content, stripped of navigation and ads.\nUse after webSearch to get full content from relevant results.`,\n\n parameters: z.object({\n url: z.string().url()\n .describe(\"The URL to read\"),\n \n contentType: z.enum([\"auto\", \"article\", \"documentation\", \"paper\", \"code\"]).default(\"auto\")\n .describe(\"Hint for content type to optimize extraction\"),\n \n maxLength: z.number().min(1000).max(50000).default(10000)\n .describe(\"Maximum characters to return\"),\n \n extractSections: z.boolean().default(true)\n .describe(\"Whether to identify and label sections\"),\n \n includeMetadata: z.boolean().default(true)\n .describe(\"Include author, date, and other metadata\")\n }),\n\n execute: async (input) => {\n return extractUrlContent(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| url | string | Yes | URL to read |\n| contentType | enum | No | Content type hint |\n| maxLength | number | No | Max chars (default: 10000) |\n| extractSections | boolean | No | Label sections |\n| includeMetadata | boolean | No | Include metadata |\n\n## Output Schema\n\n```typescript\ninterface ReadUrlResult {\n success: boolean;\n \n url: string;\n title: string;\n \n content: {\n full: string;\n sections?: {\n heading: string;\n level: number; // h1=1, h2=2, etc.\n content: string;\n }[];\n };\n \n metadata?: {\n author?: string;\n publishedDate?: string;\n lastModified?: string;\n description?: string;\n keywords?: string[];\n source: string;\n };\n \n stats: {\n totalCharacters: number;\n truncated: boolean;\n sectionsFound: number;\n };\n \n error?: {\n code: string;\n message: string;\n };\n}\n```\n\n## Usage Example\n\n```typescript\nconst content = await readUrl.execute({\n url: \"https://eugeneyan.com/writing/llm-evaluators/\",\n contentType: \"article\",\n maxLength: 15000,\n extractSections: true,\n includeMetadata: true\n});\n\n// Result:\n// {\n// success: true,\n// url: \"https://eugeneyan.com/writing/llm-evaluators/\",\n// title: \"Evaluating the Effectiveness of LLM-Evaluators\",\n// content: {\n// full: \"LLM-evaluators, also known as LLM-as-a-Judge...\",\n// sections: [\n// {\n// heading: \"Key considerations before adopting an LLM-evaluator\",\n// level: 2,\n// content: \"Before reviewing the literature...\"\n// },\n// ...\n// ]\n// },\n// metadata: {\n// author: \"Eugene Yan\",\n// publishedDate: \"2024-06-15\",\n// source: \"eugeneyan.com\"\n// },\n// stats: {\n// totalCharacters: 15000,\n// truncated: true,\n// sectionsFound: 8\n// }\n// }\n```\n\n## Content Type Handling\n\n| Type | Optimization |\n|------|-------------|\n| article | Prioritize main content, skip sidebars |\n| documentation | Preserve code blocks, keep structure |\n| paper | Extract abstract, sections, references |\n| code | Preserve formatting, syntax highlighting |\n| auto | Detect type from content |\n\n## Error Handling\n\n```typescript\nconst errorCodes = {\n \"URL_NOT_FOUND\": \"Page does not exist (404)\",\n \"ACCESS_DENIED\": \"Page requires authentication (401/403)\",\n \"TIMEOUT\": \"Request timed out\",\n \"BLOCKED\": \"Access blocked by robots.txt or rate limit\",\n \"INVALID_CONTENT\": \"Content could not be parsed\",\n \"UNSUPPORTED_TYPE\": \"Content type not supported (e.g., binary)\"\n};\n```\n\n## Implementation Notes\n\n1. **Respect robots.txt**: Check and honor robots.txt directives\n2. **Rate Limiting**: Don't hammer the same domain\n3. **User Agent**: Use appropriate user agent string\n4. **Timeouts**: Set reasonable timeouts (10-30s)\n5. **JavaScript Rendering**: Consider headless browser for JS-heavy sites\n6. **Caching**: Cache content for repeated reads\n\n","examples/llm-as-judge-skills/tools/research/web-search.md":"# Web Search Tool\n\n## Purpose\n\nSearch the web for relevant information on a given topic. Returns structured results with snippets, URLs, and metadata.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const webSearch = tool({\n description: `Search the web for information on a topic.\nReturns relevant results with snippets and URLs.\nUse for gathering current information, verifying facts, or research.`,\n\n parameters: z.object({\n query: z.string()\n .describe(\"Search query - be specific for better results\"),\n \n maxResults: z.number().min(1).max(20).default(10)\n .describe(\"Maximum number of results to return\"),\n \n filters: z.object({\n dateRange: z.enum([\"day\", \"week\", \"month\", \"year\", \"any\"]).default(\"any\")\n .describe(\"Limit results to a time period\"),\n \n sourceType: z.enum([\"all\", \"news\", \"academic\", \"documentation\"]).default(\"all\")\n .describe(\"Type of sources to prioritize\"),\n \n excludeDomains: z.array(z.string()).optional()\n .describe(\"Domains to exclude from results\")\n }).optional()\n }),\n\n execute: async (input) => {\n return performWebSearch(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| query | string | Yes | Search query |\n| maxResults | number | No | Max results (default: 10) |\n| filters.dateRange | enum | No | Time period filter |\n| filters.sourceType | enum | No | Source type priority |\n| filters.excludeDomains | string[] | No | Domains to exclude |\n\n## Output Schema\n\n```typescript\ninterface WebSearchResult {\n success: boolean;\n \n results: {\n title: string;\n url: string;\n snippet: string;\n source: string; // Domain name\n publishedDate?: string;\n relevanceScore: number;\n }[];\n \n totalResults: number;\n \n metadata: {\n query: string;\n searchTimeMs: number;\n filtersApplied: string[];\n };\n}\n```\n\n## Usage Example\n\n```typescript\nconst results = await webSearch.execute({\n query: \"LLM-as-a-Judge evaluation methods 2024\",\n maxResults: 10,\n filters: {\n dateRange: \"year\",\n sourceType: \"academic\"\n }\n});\n\n// Result:\n// {\n// success: true,\n// results: [\n// {\n// title: \"Judging LLM-as-a-Judge with MT-Bench\",\n// url: \"https://arxiv.org/abs/...\",\n// snippet: \"We study the effectiveness of LLM-as-a-Judge...\",\n// source: \"arxiv.org\",\n// publishedDate: \"2024-01-15\",\n// relevanceScore: 0.95\n// },\n// ...\n// ],\n// totalResults: 10,\n// metadata: {\n// query: \"LLM-as-a-Judge evaluation methods 2024\",\n// searchTimeMs: 342,\n// filtersApplied: [\"dateRange:year\", \"sourceType:academic\"]\n// }\n// }\n```\n\n## Query Optimization Tips\n\n1. **Specific Terms**: Use precise terminology\n2. **Quotes**: Use quotes for exact phrases\n3. **Operators**: Support site:, -term, OR\n4. **Context**: Include relevant context terms\n5. **Recency**: Add year for recent info\n\n## Implementation Notes\n\n1. **Rate Limiting**: Implement appropriate rate limits\n2. **Caching**: Cache results for repeated queries\n3. **Result Quality**: Filter out low-quality sources\n4. **Error Handling**: Handle API failures gracefully\n5. **Privacy**: Log queries appropriately\n\n","skills/advanced-evaluation/SKILL.md":"---\nname: advanced-evaluation\ndescription: \"This skill should be used for advanced LLM evaluation: LLM-as-judge systems, direct scoring, pairwise comparison, rubric calibration, evaluator bias mitigation, confidence scoring, and automated quality assessment.\"\n---\n\n# Advanced Evaluation\n\nThis skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.\n\n**Key insight**: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.\n\n## When to Activate\n\nActivate this skill when:\n\n- Building LLM-as-judge systems for LLM outputs\n- Comparing multiple model responses to select the best one\n- Establishing consistent quality standards across evaluation teams\n- Debugging evaluation systems that show inconsistent results\n- Designing A/B tests for prompt or model changes\n- Creating rubrics specifically for LLM or human/LLM hybrid judges\n- Analyzing correlation between automated and human judgments\n\nDo not activate this skill for adjacent work owned by other skills:\n- General deterministic checks, regression suites, production quality gates, or outcome metrics: `evaluation`.\n- Autonomous loop governance, locked rubrics, rollback, or PR approval boundaries: `harness-engineering`.\n- Tool API contracts for evaluation tools: `tool-design`.\n\n## Core Concepts\n\n### The Evaluation Taxonomy\n\nSelect between two primary approaches based on whether ground truth exists:\n\n**Direct Scoring** — Use when objective criteria exist (factual accuracy, instruction following, toxicity). A single LLM rates one response on a defined scale. Achieves moderate-to-high reliability for well-defined criteria. Watch for score calibration drift and inconsistent scale interpretation.\n\n**Pairwise Comparison** — Use for subjective preferences (tone, style, persuasiveness). An LLM compares two responses and selects the better one. Pairwise methods often correlate better with human preference than open-ended direct scoring for subjective tasks (claim-advanced-evaluation-position-swap). Watch for position bias and length bias.\n\n### The Bias Landscape\n\nMitigate these systematic biases in every evaluation system:\n\n**Position Bias**: First-position responses get preferential treatment. Mitigate by evaluating twice with swapped positions, then apply majority vote or consistency check.\n\n**Length Bias**: Longer responses score higher regardless of quality. Mitigate by explicitly prompting to ignore length and applying length-normalized scoring.\n\n**Self-Enhancement Bias**: Models rate their own outputs higher. Mitigate by using different models for generation and evaluation.\n\n**Verbosity Bias**: Excessive detail scores higher even when unnecessary. Mitigate with criteria-specific rubrics that penalize irrelevant detail.\n\n**Authority Bias**: Confident tone scores higher regardless of accuracy. Mitigate by requiring evidence citation and adding a fact-checking layer.\n\n### Metric Selection Framework\n\nMatch metrics to the evaluation task structure:\n\n| Task Type | Primary Metrics | Secondary Metrics |\n|-----------|-----------------|-------------------|\n| Binary classification (pass/fail) | Recall, Precision, F1 | Cohen's kappa |\n| Ordinal scale (1-5 rating) | Spearman's rho, Kendall's tau | Cohen's kappa (weighted) |\n| Pairwise preference | Agreement rate, Position consistency | Confidence calibration |\n| Multi-label | Macro-F1, Micro-F1 | Per-label precision/recall |\n\nPrioritize systematic disagreement patterns over absolute agreement rates because a judge that consistently disagrees with humans on specific criteria is more problematic than one with random noise.\n\n## Evaluation Approaches\n\n### Direct Scoring Implementation\n\nBuild direct scoring with three components: clear criteria, a calibrated scale, and structured output format.\n\n**Criteria Definition Pattern**:\n```\nCriterion: [Name]\nDescription: [What this criterion measures]\nWeight: [Relative importance, 0-1]\n```\n\n**Scale Calibration** — Choose scale granularity based on rubric detail:\n- 1-3: Binary with neutral option, lowest cognitive load\n- 1-5: Standard Likert, best balance of granularity and reliability\n- 1-10: Use only with detailed per-level rubrics because calibration is harder\n\n**Prompt Structure for Direct Scoring**:\n```\nYou are an expert evaluator assessing response quality.\n\n## Task\nEvaluate the following response against each criterion.\n\n## Original Prompt\n{prompt}\n\n## Response to Evaluate\n{response}\n\n## Criteria\n{for each criterion: name, description, weight}\n\n## Instructions\nFor each criterion:\n1. Find specific evidence in the response\n2. Score according to the rubric (1-{max} scale)\n3. Justify your score with evidence\n4. Suggest one specific improvement\n\n## Output Format\nRespond with structured JSON containing scores, justifications, and summary.\n```\n\nRequire evidence before the score in scoring prompts so the judge must anchor its decision in observable output features before emitting a number.\n\n### Pairwise Comparison Implementation\n\nApply position bias mitigation in every pairwise evaluation:\n\n1. Run deterministic pre-checks first: both candidates must satisfy the same schema, source-evidence requirements, and scope constraints.\n2. First judge pass: Response A in first position, Response B in second.\n3. Second judge pass: Response B in first position, Response A in second.\n4. Consistency check: If passes disagree, return TIE with reduced confidence.\n5. Final verdict: Consistent winner with averaged confidence and explicit tie-breaker rationale.\n\n**Prompt Structure for Pairwise Comparison**:\n```\nYou are an expert evaluator comparing two AI responses.\n\n## Critical Instructions\n- Do NOT prefer responses because they are longer\n- Do NOT prefer responses based on position (first vs second)\n- Focus ONLY on quality according to the specified criteria\n- Ties are acceptable when responses are genuinely equivalent\n\n## Original Prompt\n{prompt}\n\n## Response A\n{response_a}\n\n## Response B\n{response_b}\n\n## Comparison Criteria\n{criteria list}\n\n## Instructions\n1. Analyze each response independently first\n2. Compare them on each criterion\n3. Determine overall winner with confidence level\n\n## Output Format\nJSON with per-criterion comparison, overall winner, confidence (0-1), and reasoning.\n```\n\n**Confidence Calibration** — Map confidence to position consistency:\n- Both passes agree: confidence = average of individual confidences\n- Passes disagree: confidence = 0.5, verdict = TIE\n\n### Rubric Generation\n\nGenerate rubrics to reduce evaluation variance compared to open-ended scoring. Treat exact variance reduction as workload-specific unless measured on the target eval set.\n\n**Include these rubric components**:\n1. **Level descriptions**: Clear boundaries for each score level\n2. **Characteristics**: Observable features that define each level\n3. **Examples**: Representative text for each level (optional but valuable)\n4. **Edge cases**: Guidance for ambiguous situations\n5. **Scoring guidelines**: General principles for consistent application\n\n**Set strictness calibration** for the use case:\n- **Lenient**: Lower passing bar, appropriate for encouraging iteration\n- **Balanced**: Typical production expectations\n- **Strict**: High standards for safety-critical or high-stakes evaluation\n\nAdapt rubrics to the domain — use domain-specific terminology. A code readability rubric mentions variables, functions, and comments. A medical accuracy rubric references clinical terminology and evidence standards.\n\n## Practical Guidance\n\n### Evaluation Pipeline Design\n\nBuild production evaluation systems with these layers: Criteria Loader (rubrics + weights) -> Primary Scorer (direct or pairwise) -> Bias Mitigation (position swap, etc.) -> Confidence Scoring (calibration) -> Output (scores + justifications + confidence). See [Evaluation Pipeline Diagram](./references/evaluation-pipeline.md) for the full visual layout.\n\n### Decision Framework: Direct vs. Pairwise\n\nApply this decision tree:\n\n```\nIs there an objective ground truth?\n+-- Yes -> Direct Scoring\n| Examples: factual accuracy, instruction following, format compliance\n|\n+-- No -> Is it a preference or quality judgment?\n +-- Yes -> Pairwise Comparison\n | Examples: tone, style, persuasiveness, creativity\n |\n +-- No -> Consider reference-based evaluation\n Examples: summarization (compare to source), translation (compare to reference)\n```\n\n### Scaling Evaluation\n\nFor high-volume evaluation, apply one of these strategies:\n\n1. **Panel of LLMs (PoLL)**: Use multiple models as judges and aggregate votes to reduce individual model bias. More expensive but more reliable for high-stakes decisions.\n\n2. **Hierarchical evaluation**: Use a fast cheap model for screening and an expensive model for edge cases. Requires calibration of the screening threshold.\n\n3. **Human-in-the-loop**: Automate clear cases and route low-confidence decisions to human review. Design feedback loops to improve automated evaluation over time.\n\n## Examples\n\n### Example 1: Direct Scoring for Accuracy\n\n**Input**:\n```\nPrompt: \"What causes seasons on Earth?\"\nResponse: \"Seasons are caused by Earth's tilted axis. As Earth orbits the Sun,\ndifferent hemispheres receive more direct sunlight at different times of year.\"\nCriterion: Factual Accuracy (weight: 1.0)\nScale: 1-5\n```\n\n**Output**:\n```json\n{\n \"criterion\": \"Factual Accuracy\",\n \"score\": 5,\n \"evidence\": [\n \"Correctly identifies axial tilt as primary cause\",\n \"Correctly explains differential sunlight by hemisphere\",\n \"No factual errors present\"\n ],\n \"justification\": \"Response accurately explains the cause of seasons with correct\nscientific reasoning. Both the axial tilt and its effect on sunlight distribution\nare correctly described.\",\n \"improvement\": \"Could add the specific tilt angle (23.5 degrees) for completeness.\"\n}\n```\n\n### Example 2: Pairwise Comparison with Position Swap\n\n**Input**:\n```\nPrompt: \"Explain machine learning to a beginner\"\nResponse A: [Technical explanation with jargon]\nResponse B: [Simple analogy-based explanation]\nCriteria: [\"clarity\", \"accessibility\"]\n```\n\n**First Pass (A first)**:\n```json\n{ \"winner\": \"B\", \"confidence\": 0.8 }\n```\n\n**Second Pass (B first)**:\n```json\n{ \"winner\": \"A\", \"confidence\": 0.6 }\n```\n(Note: Winner is A because B was in first position)\n\n**Mapped Second Pass**:\n```json\n{ \"winner\": \"B\", \"confidence\": 0.6 }\n```\n\n**Final Result**:\n```json\n{\n \"winner\": \"B\",\n \"confidence\": 0.7,\n \"positionConsistency\": {\n \"consistent\": true,\n \"firstPassWinner\": \"B\",\n \"secondPassWinner\": \"B\"\n }\n}\n```\n\n### Example 3: Rubric Generation\n\n**Input**:\n```\ncriterionName: \"Code Readability\"\ncriterionDescription: \"How easy the code is to understand and maintain\"\ndomain: \"software engineering\"\nscale: \"1-5\"\nstrictness: \"balanced\"\n```\n\n**Output** (abbreviated):\n```json\n{\n \"levels\": [\n {\n \"score\": 1,\n \"label\": \"Poor\",\n \"description\": \"Code is difficult to understand without significant effort\",\n \"characteristics\": [\n \"No meaningful variable or function names\",\n \"No comments or documentation\",\n \"Deeply nested or convoluted logic\"\n ]\n },\n {\n \"score\": 3,\n \"label\": \"Adequate\",\n \"description\": \"Code is understandable with some effort\",\n \"characteristics\": [\n \"Most variables have meaningful names\",\n \"Basic comments present for complex sections\",\n \"Logic is followable but could be cleaner\"\n ]\n },\n {\n \"score\": 5,\n \"label\": \"Excellent\",\n \"description\": \"Code is immediately clear and maintainable\",\n \"characteristics\": [\n \"All names are descriptive and consistent\",\n \"Comprehensive documentation\",\n \"Clean, modular structure\"\n ]\n }\n ],\n \"edgeCases\": [\n {\n \"situation\": \"Code is well-structured but uses domain-specific abbreviations\",\n \"guidance\": \"Score based on readability for domain experts, not general audience\"\n }\n ]\n}\n```\n\n## Guidelines\n\n1. **Always require evidence before scores** - Evidence-first prompts make judgments easier to audit and reduce ungrounded numeric scoring\n\n2. **Always swap positions in pairwise comparison** - Single-pass comparison is corrupted by position bias\n\n3. **Match scale granularity to rubric specificity** - Don't use 1-10 without detailed level descriptions\n\n4. **Separate objective and subjective criteria** - Use direct scoring for objective, pairwise for subjective\n\n5. **Include confidence scores** - Calibrate to position consistency and evidence strength\n\n6. **Define edge cases explicitly** - Ambiguous situations cause the most evaluation variance\n\n7. **Use domain-specific rubrics** - Generic rubrics produce generic (less useful) evaluations\n\n8. **Validate against human judgments** - Automated evaluation is only valuable if it correlates with human assessment\n\n9. **Monitor for systematic bias** - Track disagreement patterns by criterion, response type, model\n\n10. **Design for iteration** - Evaluation systems improve with feedback loops\n\n## Gotchas\n\n1. **Scoring without justification**: Scores lack grounding and are difficult to debug. Always require evidence-based justification before the score.\n\n2. **Single-pass pairwise comparison**: Position bias corrupts results when positions are not swapped. Always evaluate twice with swapped positions and check consistency.\n\n3. **Overloaded criteria**: Criteria that measure multiple things at once produce unreliable scores. Enforce one criterion = one measurable aspect.\n\n4. **Missing edge case guidance**: Evaluators handle ambiguous cases inconsistently without explicit instructions. Include edge cases in rubrics with clear resolution rules.\n\n5. **Ignoring confidence calibration**: High-confidence wrong judgments are worse than low-confidence ones. Calibrate confidence to position consistency and evidence strength.\n\n6. **Rubric drift**: Rubrics become miscalibrated as quality standards evolve or model capabilities improve. Schedule periodic rubric reviews and re-anchor score levels against fresh human-annotated examples.\n\n7. **Evaluation prompt sensitivity**: Minor wording changes in evaluation prompts can cause material score swings. Version-control evaluation prompts and run regression tests before deploying prompt changes.\n\n8. **Uncontrolled length bias**: Longer responses systematically score higher even when conciseness is preferred. Add explicit length-neutrality instructions to evaluation prompts and validate with length-controlled test pairs.\n\n## Integration\n\nThis skill owns judge design and bias mitigation. Adjacent skills own broader quality gates and infrastructure:\n\n- `evaluation`: general deterministic checks, regression suites, quality gates, and production monitoring.\n- `context-fundamentals`: context structure for judge prompts.\n- `tool-design`: schemas and error handling for evaluation tools.\n- `context-optimization`: token and latency efficiency for high-volume evals.\n- `harness-engineering`: locked evaluator surfaces and governance for autonomous loops.\n\n## References\n\nInternal reference:\n- [LLM-as-Judge Implementation Patterns](./references/implementation-patterns.md) - Read when: building an evaluation pipeline from scratch or integrating LLM judges into CI/CD\n- [Bias Mitigation Techniques](./references/bias-mitigation.md) - Read when: evaluation results show inconsistent or suspicious scoring patterns\n- [Metric Selection Guide](./references/metrics-guide.md) - Read when: choosing statistical metrics to validate evaluation reliability\n- [Evaluation Pipeline Diagram](./references/evaluation-pipeline.md) - Read when: designing the architecture of a multi-stage evaluation system\n\nExternal research:\n- [Eugene Yan: Evaluating the Effectiveness of LLM-Evaluators](https://eugeneyan.com/writing/llm-evaluators/) - Read when: surveying the state of the art in LLM evaluation\n- [Judging LLM-as-a-Judge (Zheng et al., 2023)](https://arxiv.org/abs/2306.05685) - Read when: understanding position bias and MT-Bench methodology\n- [G-Eval: NLG Evaluation using GPT-4 (Liu et al., 2023)](https://arxiv.org/abs/2303.16634) - Read when: implementing chain-of-thought evaluation scoring\n- [Large Language Models are not Fair Evaluators (Wang et al., 2023)](https://arxiv.org/abs/2305.17926) - Read when: diagnosing systematic bias in evaluation outputs\n\nRelated skills in this collection:\n- evaluation - Foundational evaluation concepts\n- context-fundamentals - Context structure for evaluation prompts\n- tool-design - Building evaluation tools\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-24\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.1.0\n","skills/advanced-evaluation/references/bias-mitigation.md":"# Bias Mitigation Techniques for LLM Evaluation\n\nThis reference details specific techniques for mitigating known biases in LLM-as-a-Judge systems.\n\n## Position Bias\n\n### The Problem\n\nIn pairwise comparison, LLMs systematically prefer responses in certain positions. Research shows:\n- GPT has mild first-position bias (~55% preference for first position in ties)\n- Claude shows similar patterns\n- Smaller models often show stronger bias\n\n### Mitigation: Position Swapping Protocol\n\n```python\nasync def position_swap_comparison(response_a, response_b, prompt, criteria):\n # Pass 1: Original order\n result_ab = await compare(response_a, response_b, prompt, criteria)\n \n # Pass 2: Swapped order\n result_ba = await compare(response_b, response_a, prompt, criteria)\n \n # Map second result (A in second position → B in first)\n result_ba_mapped = {\n 'winner': {'A': 'B', 'B': 'A', 'TIE': 'TIE'}[result_ba['winner']],\n 'confidence': result_ba['confidence']\n }\n \n # Consistency check\n if result_ab['winner'] == result_ba_mapped['winner']:\n return {\n 'winner': result_ab['winner'],\n 'confidence': (result_ab['confidence'] + result_ba_mapped['confidence']) / 2,\n 'position_consistent': True\n }\n else:\n # Disagreement indicates position bias was a factor\n return {\n 'winner': 'TIE',\n 'confidence': 0.5,\n 'position_consistent': False,\n 'bias_detected': True\n }\n```\n\n### Alternative: Multiple Shuffles\n\nFor higher reliability, use multiple position orderings:\n\n```python\nasync def multi_shuffle_comparison(response_a, response_b, prompt, criteria, n_shuffles=3):\n results = []\n for i in range(n_shuffles):\n if i % 2 == 0:\n r = await compare(response_a, response_b, prompt, criteria)\n else:\n r = await compare(response_b, response_a, prompt, criteria)\n r['winner'] = {'A': 'B', 'B': 'A', 'TIE': 'TIE'}[r['winner']]\n results.append(r)\n \n # Majority vote\n winners = [r['winner'] for r in results]\n final_winner = max(set(winners), key=winners.count)\n agreement = winners.count(final_winner) / len(winners)\n \n return {\n 'winner': final_winner,\n 'confidence': agreement,\n 'n_shuffles': n_shuffles\n }\n```\n\n## Length Bias\n\n### The Problem\n\nLLMs tend to rate longer responses higher, regardless of quality. This manifests as:\n- Verbose responses receiving inflated scores\n- Concise but complete responses penalized\n- Padding and repetition being rewarded\n\n### Mitigation: Explicit Prompting\n\nInclude anti-length-bias instructions in the prompt:\n\n```\nCRITICAL EVALUATION GUIDELINES:\n- Do NOT prefer responses because they are longer\n- Concise, complete answers are as valuable as detailed ones\n- Penalize unnecessary verbosity or repetition\n- Focus on information density, not word count\n```\n\n### Mitigation: Length-Normalized Scoring\n\n```python\ndef length_normalized_score(score, response_length, target_length=500):\n \"\"\"Adjust score based on response length.\"\"\"\n length_ratio = response_length / target_length\n \n if length_ratio > 2.0:\n # Penalize excessively long responses\n penalty = (length_ratio - 2.0) * 0.1\n return max(score - penalty, 1)\n elif length_ratio < 0.3:\n # Penalize excessively short responses\n penalty = (0.3 - length_ratio) * 0.5\n return max(score - penalty, 1)\n else:\n return score\n```\n\n### Mitigation: Separate Length Criterion\n\nMake length a separate, explicit criterion so it's not implicitly rewarded:\n\n```python\ncriteria = [\n {\"name\": \"Accuracy\", \"description\": \"Factual correctness\", \"weight\": 0.4},\n {\"name\": \"Completeness\", \"description\": \"Covers key points\", \"weight\": 0.3},\n {\"name\": \"Conciseness\", \"description\": \"No unnecessary content\", \"weight\": 0.3} # Explicit\n]\n```\n\n## Self-Enhancement Bias\n\n### The Problem\n\nModels rate outputs generated by themselves (or similar models) higher than outputs from different models.\n\n### Mitigation: Cross-Model Evaluation\n\nUse a different model family for evaluation than generation:\n\n```python\ndef get_evaluator_model(generator_model):\n \"\"\"Select evaluator to avoid self-enhancement bias.\"\"\"\n if 'gpt' in generator_model.lower():\n return 'claude-4-5-sonnet'\n elif 'claude' in generator_model.lower():\n return 'gpt-5.2'\n else:\n return 'gpt-5.2' # Default\n```\n\n### Mitigation: Blind Evaluation\n\nRemove model attribution from responses before evaluation:\n\n```python\ndef anonymize_response(response, model_name):\n \"\"\"Remove model-identifying patterns.\"\"\"\n patterns = [\n f\"As {model_name}\",\n \"I am an AI\",\n \"I don't have personal opinions\",\n # Model-specific patterns\n ]\n anonymized = response\n for pattern in patterns:\n anonymized = anonymized.replace(pattern, \"[REDACTED]\")\n return anonymized\n```\n\n## Verbosity Bias\n\n### The Problem\n\nDetailed explanations receive higher scores even when the extra detail is irrelevant or incorrect.\n\n### Mitigation: Relevance-Weighted Scoring\n\n```python\nasync def relevance_weighted_evaluation(response, prompt, criteria):\n # First, assess relevance of each segment\n relevance_scores = await assess_relevance(response, prompt)\n \n # Weight evaluation by relevance\n segments = split_into_segments(response)\n weighted_scores = []\n for segment, relevance in zip(segments, relevance_scores):\n if relevance > 0.5: # Only count relevant segments\n score = await evaluate_segment(segment, prompt, criteria)\n weighted_scores.append(score * relevance)\n \n return sum(weighted_scores) / len(weighted_scores)\n```\n\n### Mitigation: Rubric with Verbosity Penalty\n\nInclude explicit verbosity penalties in rubrics:\n\n```python\nrubric_levels = [\n {\n \"score\": 5,\n \"description\": \"Complete and concise. All necessary information, nothing extraneous.\",\n \"characteristics\": [\"Every sentence adds value\", \"No repetition\", \"Appropriately scoped\"]\n },\n {\n \"score\": 3,\n \"description\": \"Complete but verbose. Contains unnecessary detail or repetition.\",\n \"characteristics\": [\"Main points covered\", \"Some tangents\", \"Could be more concise\"]\n },\n # ... etc\n]\n```\n\n## Authority Bias\n\n### The Problem\n\nConfident, authoritative tone is rated higher regardless of accuracy.\n\n### Mitigation: Evidence Requirement\n\nRequire explicit evidence for claims:\n\n```\nFor each claim in the response:\n1. Identify whether it's a factual claim\n2. Note if evidence or sources are provided\n3. Score based on verifiability, not confidence\n\nIMPORTANT: Confident claims without evidence should NOT receive higher scores than \nhedged claims with evidence.\n```\n\n### Mitigation: Fact-Checking Layer\n\nAdd a fact-checking step before scoring:\n\n```python\nasync def fact_checked_evaluation(response, prompt, criteria):\n # Extract claims\n claims = await extract_claims(response)\n \n # Fact-check each claim\n fact_check_results = await asyncio.gather(*[\n verify_claim(claim) for claim in claims\n ])\n \n # Adjust score based on fact-check results\n accuracy_factor = sum(r['verified'] for r in fact_check_results) / len(fact_check_results)\n \n base_score = await evaluate(response, prompt, criteria)\n return base_score * (0.7 + 0.3 * accuracy_factor) # At least 70% of score\n```\n\n## Aggregate Bias Detection\n\nMonitor for systematic biases in production:\n\n```python\nclass BiasMonitor:\n def __init__(self):\n self.evaluations = []\n \n def record(self, evaluation):\n self.evaluations.append(evaluation)\n \n def detect_position_bias(self):\n \"\"\"Detect if first position wins more often than expected.\"\"\"\n first_wins = sum(1 for e in self.evaluations if e['first_position_winner'])\n expected = len(self.evaluations) * 0.5\n z_score = (first_wins - expected) / (expected * 0.5) ** 0.5\n return {'bias_detected': abs(z_score) > 2, 'z_score': z_score}\n \n def detect_length_bias(self):\n \"\"\"Detect if longer responses score higher.\"\"\"\n from scipy.stats import spearmanr\n lengths = [e['response_length'] for e in self.evaluations]\n scores = [e['score'] for e in self.evaluations]\n corr, p_value = spearmanr(lengths, scores)\n return {'bias_detected': corr > 0.3 and p_value < 0.05, 'correlation': corr}\n```\n\n## Summary Table\n\n| Bias | Primary Mitigation | Secondary Mitigation | Detection Method |\n|------|-------------------|---------------------|------------------|\n| Position | Position swapping | Multiple shuffles | Consistency check |\n| Length | Explicit prompting | Length normalization | Length-score correlation |\n| Self-enhancement | Cross-model evaluation | Anonymization | Model comparison study |\n| Verbosity | Relevance weighting | Rubric penalties | Relevance scoring |\n| Authority | Evidence requirement | Fact-checking layer | Confidence-accuracy correlation |\n\n","skills/advanced-evaluation/references/evaluation-pipeline.md":"# Evaluation Pipeline Diagram\n\nVisual layout of a production evaluation pipeline.\n\n```\n┌─────────────────────────────────────────────────┐\n│ Evaluation Pipeline │\n├─────────────────────────────────────────────────┤\n│ │\n│ Input: Response + Prompt + Context │\n│ │ │\n│ ▼ │\n│ ┌─────────────────────┐ │\n│ │ Criteria Loader │ ◄── Rubrics, weights │\n│ └──────────┬──────────┘ │\n│ │ │\n│ ▼ │\n│ ┌─────────────────────┐ │\n│ │ Primary Scorer │ ◄── Direct or Pairwise │\n│ └──────────┬──────────┘ │\n│ │ │\n│ ▼ │\n│ ┌─────────────────────┐ │\n│ │ Bias Mitigation │ ◄── Position swap, etc. │\n│ └──────────┬──────────┘ │\n│ │ │\n│ ▼ │\n│ ┌─────────────────────┐ │\n│ │ Confidence Scoring │ ◄── Calibration │\n│ └──────────┬──────────┘ │\n│ │ │\n│ ▼ │\n│ Output: Scores + Justifications + Confidence │\n│ │\n└─────────────────────────────────────────────────┘\n```\n\n## Pipeline Stages\n\n1. **Criteria Loader**: Loads rubrics and criterion weights from configuration\n2. **Primary Scorer**: Applies direct scoring or pairwise comparison\n3. **Bias Mitigation**: Runs position swaps, length normalization, and other debiasing\n4. **Confidence Scoring**: Calibrates confidence based on position consistency and evidence strength\n","skills/advanced-evaluation/references/implementation-patterns.md":"# LLM-as-Judge Implementation Patterns\n\nThis reference provides detailed implementation patterns for building production-grade LLM evaluation systems.\n\n## Pattern 1: Structured Evaluation Pipeline\n\nThe most reliable evaluation systems follow a structured pipeline that separates concerns:\n\n```\nInput Validation → Criteria Loading → Scoring → Bias Mitigation → Output Formatting\n```\n\n### Input Validation Layer\n\nBefore evaluation begins, validate:\n\n1. **Response presence**: Non-empty response to evaluate\n2. **Prompt presence**: Original prompt for context\n3. **Criteria validity**: At least one criterion with name and description\n4. **Weight normalization**: Weights sum to 1.0 (or normalize them)\n\n```python\ndef validate_input(response, prompt, criteria):\n if not response or not response.strip():\n raise ValueError(\"Response cannot be empty\")\n if not prompt or not prompt.strip():\n raise ValueError(\"Prompt cannot be empty\")\n if not criteria or len(criteria) == 0:\n raise ValueError(\"At least one criterion required\")\n \n # Normalize weights\n total_weight = sum(c.get('weight', 1) for c in criteria)\n for c in criteria:\n c['weight'] = c.get('weight', 1) / total_weight\n```\n\n### Criteria Loading Layer\n\nCriteria should be loaded from configuration, not hardcoded:\n\n```python\nclass CriteriaLoader:\n def __init__(self, rubric_path=None):\n self.rubrics = self._load_rubrics(rubric_path)\n \n def get_criteria(self, task_type):\n return self.rubrics.get(task_type, self.default_criteria)\n \n def get_rubric(self, criterion_name):\n return self.rubrics.get(criterion_name, {}).get('levels', [])\n```\n\n### Scoring Layer\n\nThe scoring layer handles the actual LLM call:\n\n```python\nasync def score_response(response, prompt, criteria, rubric, model):\n system_prompt = build_system_prompt(criteria, rubric)\n user_prompt = build_user_prompt(response, prompt, criteria)\n \n result = await generate_text(\n model=model,\n system=system_prompt,\n prompt=user_prompt,\n temperature=0.3 # Lower temperature for consistency\n )\n \n return parse_scores(result.text)\n```\n\n### Bias Mitigation Layer\n\nFor pairwise comparison, always include position swapping:\n\n```python\nasync def compare_with_bias_mitigation(response_a, response_b, prompt, criteria, model):\n # First pass: A first\n pass1 = await compare_pair(response_a, response_b, prompt, criteria, model)\n \n # Second pass: B first\n pass2 = await compare_pair(response_b, response_a, prompt, criteria, model)\n \n # Map pass2 winner back\n pass2_mapped = map_winner(pass2.winner) # A→B, B→A, TIE→TIE\n \n # Check consistency\n if pass1.winner == pass2_mapped:\n return {\n 'winner': pass1.winner,\n 'confidence': (pass1.confidence + pass2.confidence) / 2,\n 'consistent': True\n }\n else:\n return {\n 'winner': 'TIE',\n 'confidence': 0.5,\n 'consistent': False\n }\n```\n\n## Pattern 2: Hierarchical Evaluation\n\nFor complex evaluations, use a hierarchical approach:\n\n```\nQuick Screen (cheap model) → Detailed Evaluation (expensive model) → Human Review (edge cases)\n```\n\n### Quick Screen Implementation\n\n```python\nasync def quick_screen(response, prompt, threshold=0.7):\n \"\"\"Fast, cheap screening for obvious passes/fails.\"\"\"\n result = await generate_text(\n model='gpt-5.2', # Cheaper model\n prompt=f\"Rate 0-1 if this response adequately addresses the prompt:\\n\\nPrompt: {prompt}\\n\\nResponse: {response}\",\n temperature=0\n )\n score = float(result.text.strip())\n return score, score > threshold\n```\n\n### Detailed Evaluation\n\n```python\nasync def detailed_evaluation(response, prompt, criteria):\n \"\"\"Full evaluation for borderline or important cases.\"\"\"\n result = await generate_text(\n model='gpt-5.2', # More capable model\n system=DETAILED_EVALUATION_PROMPT,\n prompt=build_detailed_prompt(response, prompt, criteria),\n temperature=0.3\n )\n return parse_detailed_scores(result.text)\n```\n\n## Pattern 3: Panel of LLM Judges (PoLL)\n\nFor high-stakes evaluation, use multiple models:\n\n```python\nasync def poll_evaluation(response, prompt, criteria, models):\n \"\"\"Aggregate judgments from multiple LLM judges.\"\"\"\n results = await asyncio.gather(*[\n score_with_model(response, prompt, criteria, model)\n for model in models\n ])\n \n # Aggregate scores\n aggregated = aggregate_scores(results)\n \n # Calculate agreement\n agreement = calculate_agreement(results)\n \n return {\n 'scores': aggregated,\n 'agreement': agreement,\n 'individual_results': results\n }\n\ndef aggregate_scores(results):\n \"\"\"Aggregate scores using median (robust to outliers).\"\"\"\n scores = {}\n for criterion in results[0]['scores'].keys():\n criterion_scores = [r['scores'][criterion] for r in results]\n scores[criterion] = {\n 'score': statistics.median(criterion_scores),\n 'std': statistics.stdev(criterion_scores) if len(criterion_scores) > 1 else 0\n }\n return scores\n```\n\n## Pattern 4: Confidence Calibration\n\nConfidence scores should be calibrated to actual reliability:\n\n```python\ndef calibrate_confidence(raw_confidence, position_consistent, evidence_count):\n \"\"\"Calibrate confidence based on multiple signals.\"\"\"\n \n # Base confidence from model output\n calibrated = raw_confidence\n \n # Position consistency is a strong signal\n if not position_consistent:\n calibrated *= 0.6 # Significant reduction\n \n # More evidence = higher confidence\n evidence_factor = min(evidence_count / 3, 1.0) # Cap at 3 pieces\n calibrated *= (0.7 + 0.3 * evidence_factor)\n \n return min(calibrated, 0.99) # Never 100% confident\n```\n\n## Pattern 5: Output Formatting\n\nAlways return structured outputs with consistent schemas:\n\n```python\n@dataclass\nclass ScoreResult:\n criterion: str\n score: float\n max_score: float\n justification: str\n evidence: List[str]\n improvement: str\n\n@dataclass\nclass EvaluationResult:\n success: bool\n scores: List[ScoreResult]\n overall_score: float\n weighted_score: float\n summary: Dict[str, Any]\n metadata: Dict[str, Any]\n\ndef format_output(scores, metadata) -> EvaluationResult:\n \"\"\"Format evaluation results consistently.\"\"\"\n return EvaluationResult(\n success=True,\n scores=scores,\n overall_score=sum(s.score for s in scores) / len(scores),\n weighted_score=calculate_weighted_score(scores),\n summary=generate_summary(scores),\n metadata=metadata\n )\n```\n\n## Error Handling Patterns\n\n### Graceful Degradation\n\n```python\nasync def evaluate_with_fallback(response, prompt, criteria):\n try:\n return await full_evaluation(response, prompt, criteria)\n except RateLimitError:\n # Fall back to simpler evaluation\n return await simple_evaluation(response, prompt, criteria)\n except ParseError as e:\n # Return partial results with error flag\n return {\n 'success': False,\n 'partial_results': e.partial_data,\n 'error': str(e)\n }\n```\n\n### Retry Logic\n\n```python\nasync def evaluate_with_retry(response, prompt, criteria, max_retries=3):\n for attempt in range(max_retries):\n try:\n result = await evaluate(response, prompt, criteria)\n if is_valid_result(result):\n return result\n except TransientError:\n await asyncio.sleep(2 ** attempt) # Exponential backoff\n \n raise EvaluationError(\"Max retries exceeded\")\n```\n\n## Testing Patterns\n\n### Unit Tests for Parsing\n\n```python\ndef test_score_parsing():\n raw_output = '{\"scores\": [{\"criterion\": \"Accuracy\", \"score\": 4}]}'\n result = parse_scores(raw_output)\n assert result.scores[0].criterion == \"Accuracy\"\n assert result.scores[0].score == 4\n\ndef test_malformed_output():\n raw_output = 'Invalid JSON'\n with pytest.raises(ParseError):\n parse_scores(raw_output)\n```\n\n### Integration Tests with Real API\n\n```python\n@pytest.mark.integration\nasync def test_full_evaluation_pipeline():\n result = await evaluate(\n response=\"Water boils at 100°C at sea level.\",\n prompt=\"At what temperature does water boil?\",\n criteria=[{\"name\": \"Accuracy\", \"description\": \"Factual correctness\", \"weight\": 1}]\n )\n \n assert result.success\n assert len(result.scores) == 1\n assert result.scores[0].score >= 4 # Should score high for accurate response\n```\n\n### Bias Detection Tests\n\n```python\nasync def test_position_bias_mitigation():\n # Same response in both positions should tie\n result = await compare(\n response_a=\"Same response\",\n response_b=\"Same response\",\n prompt=\"Test prompt\",\n criteria=[\"quality\"],\n swap_positions=True\n )\n \n assert result.winner == \"TIE\"\n assert result.consistent == True\n```\n\n","skills/advanced-evaluation/references/metrics-guide.md":"# Metric Selection Guide for LLM Evaluation\n\nThis reference provides guidance on selecting appropriate metrics for different evaluation scenarios.\n\n## Metric Categories\n\n### Classification Metrics\n\nUse for binary or multi-class evaluation tasks (pass/fail, correct/incorrect).\n\n#### Precision\n\n```\nPrecision = True Positives / (True Positives + False Positives)\n```\n\n**Interpretation**: Of all responses the judge said were good, what fraction were actually good?\n\n**Use when**: False positives are costly (e.g., approving unsafe content)\n\n```python\ndef precision(predictions, ground_truth):\n true_positives = sum(1 for p, g in zip(predictions, ground_truth) if p == 1 and g == 1)\n predicted_positives = sum(predictions)\n return true_positives / predicted_positives if predicted_positives > 0 else 0\n```\n\n#### Recall\n\n```\nRecall = True Positives / (True Positives + False Negatives)\n```\n\n**Interpretation**: Of all actually good responses, what fraction did the judge identify?\n\n**Use when**: False negatives are costly (e.g., missing good content in filtering)\n\n```python\ndef recall(predictions, ground_truth):\n true_positives = sum(1 for p, g in zip(predictions, ground_truth) if p == 1 and g == 1)\n actual_positives = sum(ground_truth)\n return true_positives / actual_positives if actual_positives > 0 else 0\n```\n\n#### F1 Score\n\n```\nF1 = 2 * (Precision * Recall) / (Precision + Recall)\n```\n\n**Interpretation**: Harmonic mean of precision and recall\n\n**Use when**: You need a single number balancing both concerns\n\n```python\ndef f1_score(predictions, ground_truth):\n p = precision(predictions, ground_truth)\n r = recall(predictions, ground_truth)\n return 2 * p * r / (p + r) if (p + r) > 0 else 0\n```\n\n### Agreement Metrics\n\nUse for comparing automated evaluation with human judgment.\n\n#### Cohen's Kappa (κ)\n\n```\nκ = (Observed Agreement - Expected Agreement) / (1 - Expected Agreement)\n```\n\n**Interpretation**: Agreement adjusted for chance\n- κ > 0.8: Almost perfect agreement\n- κ 0.6-0.8: Substantial agreement\n- κ 0.4-0.6: Moderate agreement\n- κ < 0.4: Fair to poor agreement\n\n**Use for**: Binary or categorical judgments\n\n```python\ndef cohens_kappa(judge1, judge2):\n from sklearn.metrics import cohen_kappa_score\n return cohen_kappa_score(judge1, judge2)\n```\n\n#### Weighted Kappa\n\nFor ordinal scales where disagreement severity matters:\n\n```python\ndef weighted_kappa(judge1, judge2):\n from sklearn.metrics import cohen_kappa_score\n return cohen_kappa_score(judge1, judge2, weights='quadratic')\n```\n\n**Interpretation**: Penalizes large disagreements more than small ones\n\n### Correlation Metrics\n\nUse for ordinal/continuous scores.\n\n#### Spearman's Rank Correlation (ρ)\n\n**Interpretation**: Correlation between rankings, not absolute values\n- ρ > 0.9: Very strong correlation\n- ρ 0.7-0.9: Strong correlation\n- ρ 0.5-0.7: Moderate correlation\n- ρ < 0.5: Weak correlation\n\n**Use when**: Order matters more than exact values\n\n```python\ndef spearmans_rho(scores1, scores2):\n from scipy.stats import spearmanr\n rho, p_value = spearmanr(scores1, scores2)\n return {'rho': rho, 'p_value': p_value}\n```\n\n#### Kendall's Tau (τ)\n\n**Interpretation**: Similar to Spearman but based on pairwise concordance\n\n**Use when**: You have many tied values\n\n```python\ndef kendalls_tau(scores1, scores2):\n from scipy.stats import kendalltau\n tau, p_value = kendalltau(scores1, scores2)\n return {'tau': tau, 'p_value': p_value}\n```\n\n#### Pearson Correlation (r)\n\n**Interpretation**: Linear correlation between scores\n\n**Use when**: Exact score values matter, not just order\n\n```python\ndef pearsons_r(scores1, scores2):\n from scipy.stats import pearsonr\n r, p_value = pearsonr(scores1, scores2)\n return {'r': r, 'p_value': p_value}\n```\n\n### Pairwise Comparison Metrics\n\n#### Agreement Rate\n\n```\nAgreement = (Matching Decisions) / (Total Comparisons)\n```\n\n**Interpretation**: Simple percentage of agreement\n\n```python\ndef pairwise_agreement(decisions1, decisions2):\n matches = sum(1 for d1, d2 in zip(decisions1, decisions2) if d1 == d2)\n return matches / len(decisions1)\n```\n\n#### Position Consistency\n\n```\nConsistency = (Consistent across position swaps) / (Total comparisons)\n```\n\n**Interpretation**: How often does swapping position change the decision?\n\n```python\ndef position_consistency(results):\n consistent = sum(1 for r in results if r['position_consistent'])\n return consistent / len(results)\n```\n\n## Selection Decision Tree\n\n```\nWhat type of evaluation task?\n│\n├── Binary classification (pass/fail)\n│ └── Use: Precision, Recall, F1, Cohen's κ\n│\n├── Ordinal scale (1-5 rating)\n│ ├── Comparing to human judgments?\n│ │ └── Use: Spearman's ρ, Weighted κ\n│ └── Comparing two automated judges?\n│ └── Use: Kendall's τ, Spearman's ρ\n│\n├── Pairwise preference\n│ └── Use: Agreement rate, Position consistency\n│\n└── Multi-label classification\n └── Use: Macro-F1, Micro-F1, Per-label metrics\n```\n\n## Metric Selection by Use Case\n\n### Use Case 1: Validating Automated Evaluation\n\n**Goal**: Ensure automated evaluation correlates with human judgment\n\n**Recommended Metrics**:\n1. Primary: Spearman's ρ (for ordinal scales) or Cohen's κ (for categorical)\n2. Secondary: Per-criterion agreement\n3. Diagnostic: Confusion matrix for systematic errors\n\n```python\ndef validate_automated_eval(automated_scores, human_scores, criteria):\n results = {}\n \n # Overall correlation\n results['overall_spearman'] = spearmans_rho(automated_scores, human_scores)\n \n # Per-criterion agreement\n for criterion in criteria:\n auto_crit = [s[criterion] for s in automated_scores]\n human_crit = [s[criterion] for s in human_scores]\n results[f'{criterion}_spearman'] = spearmans_rho(auto_crit, human_crit)\n \n return results\n```\n\n### Use Case 2: Comparing Two Models\n\n**Goal**: Determine which model produces better outputs\n\n**Recommended Metrics**:\n1. Primary: Win rate (from pairwise comparison)\n2. Secondary: Position consistency (bias check)\n3. Diagnostic: Per-criterion breakdown\n\n```python\ndef compare_models(model_a_outputs, model_b_outputs, prompts):\n results = []\n for a, b, p in zip(model_a_outputs, model_b_outputs, prompts):\n comparison = await compare_with_position_swap(a, b, p)\n results.append(comparison)\n \n return {\n 'a_wins': sum(1 for r in results if r['winner'] == 'A'),\n 'b_wins': sum(1 for r in results if r['winner'] == 'B'),\n 'ties': sum(1 for r in results if r['winner'] == 'TIE'),\n 'position_consistency': position_consistency(results)\n }\n```\n\n### Use Case 3: Quality Monitoring\n\n**Goal**: Track evaluation quality over time\n\n**Recommended Metrics**:\n1. Primary: Rolling agreement with human spot-checks\n2. Secondary: Score distribution stability\n3. Diagnostic: Bias indicators (position, length)\n\n```python\nclass QualityMonitor:\n def __init__(self, window_size=100):\n self.window = deque(maxlen=window_size)\n \n def add_evaluation(self, automated, human_spot_check=None):\n self.window.append({\n 'automated': automated,\n 'human': human_spot_check,\n 'length': len(automated['response'])\n })\n \n def get_metrics(self):\n # Filter to evaluations with human spot-checks\n with_human = [e for e in self.window if e['human'] is not None]\n \n if len(with_human) < 10:\n return {'insufficient_data': True}\n \n auto_scores = [e['automated']['score'] for e in with_human]\n human_scores = [e['human']['score'] for e in with_human]\n \n return {\n 'correlation': spearmans_rho(auto_scores, human_scores),\n 'mean_difference': np.mean([a - h for a, h in zip(auto_scores, human_scores)]),\n 'length_correlation': spearmans_rho(\n [e['length'] for e in self.window],\n [e['automated']['score'] for e in self.window]\n )\n }\n```\n\n## Interpreting Metric Results\n\n### Good Evaluation System Indicators\n\n| Metric | Good | Acceptable | Concerning |\n|--------|------|------------|------------|\n| Spearman's ρ | > 0.8 | 0.6-0.8 | < 0.6 |\n| Cohen's κ | > 0.7 | 0.5-0.7 | < 0.5 |\n| Position consistency | > 0.9 | 0.8-0.9 | < 0.8 |\n| Length correlation | < 0.2 | 0.2-0.4 | > 0.4 |\n\n### Warning Signs\n\n1. **High agreement but low correlation**: May indicate calibration issues\n2. **Low position consistency**: Position bias affecting results\n3. **High length correlation**: Length bias inflating scores\n4. **Per-criterion variance**: Some criteria may be poorly defined\n\n## Reporting Template\n\n```markdown\n## Evaluation System Metrics Report\n\n### Human Agreement\n- Spearman's ρ: 0.82 (p < 0.001)\n- Cohen's κ: 0.74\n- Sample size: 500 evaluations\n\n### Bias Indicators\n- Position consistency: 91%\n- Length-score correlation: 0.12\n\n### Per-Criterion Performance\n| Criterion | Spearman's ρ | κ |\n|-----------|--------------|---|\n| Accuracy | 0.88 | 0.79 |\n| Clarity | 0.76 | 0.68 |\n| Completeness | 0.81 | 0.72 |\n\n### Recommendations\n- All metrics within acceptable ranges\n- Monitor \"Clarity\" criterion - lower agreement may indicate need for rubric refinement\n```\n\n","skills/bdi-mental-states/SKILL.md":"---\nname: bdi-mental-states\ndescription: \"This skill should be used when modeling agent mental states with BDI concepts: beliefs, desires, intentions, RDF-to-belief transformations, rational agency traces, cognitive agents, BDI ontologies, and neuro-symbolic AI integration.\"\n---\n\n# BDI Mental State Modeling\n\nTransform external RDF context into agent mental states (beliefs, desires, intentions) using formal BDI ontology patterns. This skill enables agents to reason about context through cognitive architecture, supporting deliberative reasoning, explainability, and semantic interoperability within multi-agent systems.\n\n## When to Activate\n\nActivate this skill when:\n- Processing external RDF context into agent beliefs about world states\n- Modeling rational agency with perception, deliberation, and action cycles\n- Enabling explainability through traceable reasoning chains\n- Implementing BDI frameworks (SEMAS, JADE, JADEX)\n- Augmenting LLMs with formal cognitive structures (Logic Augmented Generation)\n- Coordinating mental states across multi-agent platforms\n- Tracking temporal evolution of beliefs, desires, and intentions\n- Linking motivational states to action plans\n\nDo not activate this skill for adjacent work owned by other skills:\n- General context-window explanations or attention mechanics: `context-fundamentals`.\n- Persistent user, entity, or conversation memory without formal BDI state: `memory-systems`.\n- Supervisor, swarm, or handoff topology decisions: `multi-agent-patterns`.\n- General agent evaluation rubrics or quality gates: `evaluation`.\n\n## Core Concepts\n\n### Mental Reality Architecture\n\nSeparate mental states into two ontological categories because BDI reasoning requires distinguishing what persists from what happens:\n\n**Mental States (Endurants)** -- model these as persistent cognitive attributes that hold over time intervals:\n- `Belief`: Represent what the agent holds true about the world. Ground every belief in a world state reference.\n- `Desire`: Represent what the agent wishes to bring about. Link each desire back to the beliefs that motivate it.\n- `Intention`: Represent what the agent commits to achieving. An intention must fulfil a desire and specify a plan.\n\n**Mental Processes (Perdurants)** -- model these as events that create or modify mental states, because tracking causal transitions enables explainability:\n- `BeliefProcess`: Triggers belief formation/update from perception. Always connect to a generating world state.\n- `DesireProcess`: Generates desires from existing beliefs. Preserves the motivational chain.\n- `IntentionProcess`: Commits to selected desires as actionable intentions.\n\n### Cognitive Chain Pattern\n\nWire beliefs, desires, and intentions into directed chains using bidirectional properties (`motivates`/`isMotivatedBy`, `fulfils`/`isFulfilledBy`) because this enables both forward reasoning (what should the agent do?) and backward tracing (why did the agent act?):\n\n```turtle\n:Belief_store_open a bdi:Belief ;\n rdfs:comment \"Store is open\" ;\n bdi:motivates :Desire_buy_groceries .\n\n:Desire_buy_groceries a bdi:Desire ;\n rdfs:comment \"I desire to buy groceries\" ;\n bdi:isMotivatedBy :Belief_store_open .\n\n:Intention_go_shopping a bdi:Intention ;\n rdfs:comment \"I will buy groceries\" ;\n bdi:fulfils :Desire_buy_groceries ;\n bdi:isSupportedBy :Belief_store_open ;\n bdi:specifies :Plan_shopping .\n```\n\n### World State Grounding\n\nAlways ground mental states in world state references rather than free-text descriptions, because ungrounded beliefs break semantic querying and cross-agent interoperability:\n\n```turtle\n:Agent_A a bdi:Agent ;\n bdi:perceives :WorldState_WS1 ;\n bdi:hasMentalState :Belief_B1 .\n\n:WorldState_WS1 a bdi:WorldState ;\n rdfs:comment \"Meeting scheduled at 10am in Room 5\" ;\n bdi:atTime :TimeInstant_10am .\n\n:Belief_B1 a bdi:Belief ;\n bdi:refersTo :WorldState_WS1 .\n```\n\n### Goal-Directed Planning\n\nConnect intentions to plans via `bdi:specifies`, and decompose plans into ordered task sequences using `bdi:precedes`, because this separation allows plan reuse across different intentions while keeping execution order explicit:\n\n```turtle\n:Intention_I1 bdi:specifies :Plan_P1 .\n\n:Plan_P1 a bdi:Plan ;\n bdi:addresses :Goal_G1 ;\n bdi:beginsWith :Task_T1 ;\n bdi:endsWith :Task_T3 .\n\n:Task_T1 bdi:precedes :Task_T2 .\n:Task_T2 bdi:precedes :Task_T3 .\n```\n\n### T2B2T Paradigm\n\nImplement Triples-to-Beliefs-to-Triples as a bidirectional pipeline because agents must both consume external RDF context and produce new RDF assertions. Structure every T2B2T implementation in two explicit phases:\n\n**Phase 1: Triples-to-Beliefs** -- Translate incoming RDF triples into belief instances. Use `bdi:triggers` to connect the external world state to a `BeliefProcess`, and `bdi:generates` to produce the resulting belief. This preserves provenance from source data through to internal cognition:\n```turtle\n:WorldState_notification a bdi:WorldState ;\n rdfs:comment \"Push notification: Payment request $250\" ;\n bdi:triggers :BeliefProcess_BP1 .\n\n:BeliefProcess_BP1 a bdi:BeliefProcess ;\n bdi:generates :Belief_payment_request .\n```\n\n**Phase 2: Beliefs-to-Triples** -- After BDI deliberation selects an intention and executes a plan, project the results back into RDF using `bdi:bringsAbout`. This closes the loop so downstream systems can consume agent outputs as standard linked data:\n```turtle\n:Intention_pay a bdi:Intention ;\n bdi:specifies :Plan_payment .\n\n:PlanExecution_PE1 a bdi:PlanExecution ;\n bdi:satisfies :Plan_payment ;\n bdi:bringsAbout :WorldState_payment_complete .\n```\n\n### Notation Selection by Level\n\nChoose notation based on the C4 abstraction level being modeled, because mixing notations at the wrong level obscures rather than clarifies the cognitive architecture:\n\n| C4 Level | Notation | Mental State Representation |\n|----------|----------|----------------------------|\n| L1 Context | ArchiMate | Agent boundaries, external perception sources |\n| L2 Container | ArchiMate | BDI reasoning engine, belief store, plan executor |\n| L3 Component | UML | Mental state managers, process handlers |\n| L4 Code | UML/RDF | Belief/Desire/Intention classes, ontology instances |\n\n### Justification and Explainability\n\nAttach `bdi:Justification` instances to every mental entity using `bdi:isJustifiedBy`, because unjustified mental states make agent reasoning opaque and untraceable. Each justification should capture the evidence or rule that produced the mental state:\n\n```turtle\n:Belief_B1 a bdi:Belief ;\n bdi:isJustifiedBy :Justification_J1 .\n\n:Justification_J1 a bdi:Justification ;\n rdfs:comment \"Official announcement received via email\" .\n\n:Intention_I1 a bdi:Intention ;\n bdi:isJustifiedBy :Justification_J2 .\n\n:Justification_J2 a bdi:Justification ;\n rdfs:comment \"Location precondition satisfied\" .\n```\n\n### Temporal Dimensions\n\nAssign validity intervals to every mental state using `bdi:hasValidity` with `TimeInterval` instances, because beliefs without temporal bounds cannot be garbage-collected or conflict-checked during diachronic reasoning:\n\n```turtle\n:Belief_B1 a bdi:Belief ;\n bdi:hasValidity :TimeInterval_TI1 .\n\n:TimeInterval_TI1 a bdi:TimeInterval ;\n bdi:hasStartTime :TimeInstant_9am ;\n bdi:hasEndTime :TimeInstant_11am .\n```\n\nQuery mental states active at a specific moment using SPARQL temporal filters. Use this pattern to resolve conflicts when multiple beliefs about the same world state overlap in time:\n\n```sparql\nSELECT ?mentalState WHERE {\n ?mentalState bdi:hasValidity ?interval .\n ?interval bdi:hasStartTime ?start ;\n bdi:hasEndTime ?end .\n FILTER(?start <= \"2025-01-04T10:00:00\"^^xsd:dateTime &&\n ?end >= \"2025-01-04T10:00:00\"^^xsd:dateTime)\n}\n```\n\n### Compositional Mental Entities\n\nDecompose complex beliefs into constituent parts using `bdi:hasPart` relations, because monolithic beliefs force full replacement on partial updates. Structure composite beliefs so that each sub-belief can be independently updated, queried, or invalidated:\n\n```turtle\n:Belief_meeting a bdi:Belief ;\n rdfs:comment \"Meeting at 10am in Room 5\" ;\n bdi:hasPart :Belief_meeting_time , :Belief_meeting_location .\n\n# Update only location component without touching time\n:BeliefProcess_update a bdi:BeliefProcess ;\n bdi:modifies :Belief_meeting_location .\n```\n\n## Practical Guidance\n\n### Build a BDI Model in Six Passes\n\nUse this workflow when converting external semantic context into a BDI representation:\n\n1. **Define the world-state substrate**: Identify the external facts or events the agent can perceive. Model these as world states before creating beliefs.\n2. **Create belief instances**: Translate each relevant world state into a belief with provenance, temporal validity, and a justification reference.\n3. **Derive desires from beliefs**: Add desires only when a belief creates a goal-relevant motivation. Link each desire to the belief that motivates it.\n4. **Commit intentions deliberately**: Promote a desire to an intention only when the agent commits to a plan. Record the selected plan and preconditions.\n5. **Project action results back to triples**: After execution, emit resulting world states as RDF so downstream systems can consume the new state.\n6. **Validate with competency questions**: Query for provenance, motivation, plan sequence, and active validity windows before trusting the model.\n\n### Keep the Ontology Small\n\nStart with `Agent`, `WorldState`, `Belief`, `Desire`, `Intention`, `Plan`, `Task`, `Justification`, and `TimeInterval`. Add specialized classes only after competency questions prove the core model cannot answer required queries. A compact ontology is easier to serialize into prompts, easier to validate, and less likely to create brittle reasoning chains.\n\n### Use BDI Only When Mental-State Semantics Matter\n\nBDI modeling is justified when the system needs explainable agency: why an agent believed something, what desire that belief created, which intention was selected, and what plan executed. If the system only needs to remember facts across sessions, use `memory-systems`. If it only needs to split work across agents, use `multi-agent-patterns`.\n\n## Detailed Topics\n\n### Integration Patterns\n\n### Logic Augmented Generation (LAG)\n\nUse LAG to constrain LLM outputs with ontological structure, because unconstrained generation produces triples that violate BDI class restrictions. Serialize the ontology into the prompt context, then validate generated triples against it before accepting them:\n\n```python\ndef augment_llm_with_bdi_ontology(prompt, ontology_graph):\n ontology_context = serialize_ontology(ontology_graph, format='turtle')\n augmented_prompt = f\"{ontology_context}\\n\\n{prompt}\"\n\n response = llm.generate(augmented_prompt)\n triples = extract_rdf_triples(response)\n\n is_consistent = validate_triples(triples, ontology_graph)\n return triples if is_consistent else retry_with_feedback()\n```\n\n### SEMAS Rule Translation\n\nTranslate BDI ontology patterns into executable production rules when deploying to rule-based agent platforms. Map each cognitive chain link (belief-to-desire, desire-to-intention) to a HEAD/CONDITIONALS/TAIL rule, because this preserves the deliberative semantics while enabling runtime execution:\n\n```prolog\n% Belief triggers desire formation\n[HEAD: belief(agent_a, store_open)] /\n[CONDITIONALS: time(weekday_afternoon)] »\n[TAIL: generate_desire(agent_a, buy_groceries)].\n\n% Desire triggers intention commitment\n[HEAD: desire(agent_a, buy_groceries)] /\n[CONDITIONALS: belief(agent_a, has_shopping_list)] »\n[TAIL: commit_intention(agent_a, buy_groceries)].\n```\n\n## Guidelines\n\n1. Model world states as configurations independent of agent perspectives, providing referential substrate for mental states.\n\n2. Distinguish endurants (persistent mental states) from perdurants (temporal mental processes), aligning with DOLCE ontology.\n\n3. Treat goals as descriptions rather than mental states, maintaining separation between cognitive and planning layers.\n\n4. Use `hasPart` relations for meronymic structures enabling selective belief updates.\n\n5. Associate every mental entity with temporal constructs via `atTime` or `hasValidity`.\n\n6. Use bidirectional property pairs (`motivates`/`isMotivatedBy`, `generates`/`isGeneratedBy`) for flexible querying.\n\n7. Link mental entities to `Justification` instances for explainability and trust.\n\n8. Implement T2B2T through: (1) translate RDF to beliefs, (2) execute BDI reasoning, (3) project mental states back to RDF.\n\n9. Define existential restrictions on mental processes (e.g., `BeliefProcess ⊑ ∃generates.Belief`).\n\n10. Reuse established ODPs (EventCore, Situation, TimeIndexedSituation, BasicPlan, Provenance) for interoperability.\n\n## Competency Questions\n\nValidate implementation against these SPARQL queries:\n\n```sparql\n# CQ1: What beliefs motivated formation of a given desire?\nSELECT ?belief WHERE {\n :Desire_D1 bdi:isMotivatedBy ?belief .\n}\n\n# CQ2: Which desire does a particular intention fulfill?\nSELECT ?desire WHERE {\n :Intention_I1 bdi:fulfils ?desire .\n}\n\n# CQ3: Which mental process generated a belief?\nSELECT ?process WHERE {\n ?process bdi:generates :Belief_B1 .\n}\n\n# CQ4: What is the ordered sequence of tasks in a plan?\nSELECT ?task ?nextTask WHERE {\n :Plan_P1 bdi:hasComponent ?task .\n OPTIONAL { ?task bdi:precedes ?nextTask }\n} ORDER BY ?task\n```\n\n## Examples\n\n**Example 1: RDF notification to BDI chain**\n\nInput world state:\n\n```turtle\n:WorldState_invoice_due a bdi:WorldState ;\n rdfs:comment \"Invoice INV-42 is due tomorrow\" ;\n bdi:atTime :Time_2026_05_15 .\n```\n\nBDI projection:\n\n```turtle\n:Belief_invoice_due a bdi:Belief ;\n bdi:refersTo :WorldState_invoice_due ;\n bdi:isJustifiedBy :Justification_billing_system ;\n bdi:motivates :Desire_avoid_late_fee .\n\n:Desire_avoid_late_fee a bdi:Desire ;\n bdi:isMotivatedBy :Belief_invoice_due .\n\n:Intention_pay_invoice a bdi:Intention ;\n bdi:fulfils :Desire_avoid_late_fee ;\n bdi:specifies :Plan_pay_invoice .\n```\n\n**Example 2: Boundary decision**\n\nIf the task is \"remember that Alice prefers concise summaries,\" use `memory-systems`. If the task is \"represent why the agent believes Alice needs a summary, what goal that creates, and which plan it commits to,\" use this skill.\n\n## Gotchas\n\n1. **Conflating mental states with world states**: Mental states reference world states via `bdi:refersTo`, they are not world states themselves. Mixing them collapses the perception-cognition boundary and breaks SPARQL queries that filter by type.\n\n2. **Missing temporal bounds**: Every mental state needs validity intervals for diachronic reasoning. Without them, stale beliefs persist indefinitely and conflict detection becomes impossible.\n\n3. **Flat belief structures**: Use compositional modeling with `hasPart` for complex beliefs. Monolithic beliefs force full replacement when only one attribute changes.\n\n4. **Implicit justifications**: Always link mental entities to explicit `Justification` instances. Unjustified mental states cannot be audited or traced.\n\n5. **Direct intention-to-action mapping**: Intentions specify plans which contain tasks; actions execute tasks. Skipping the plan layer removes the ability to reuse, reorder, or share execution strategies.\n\n6. **Ontology over-complexity**: Start with 5-10 core classes and properties (Belief, Desire, Intention, WorldState, Plan, plus key relations). Expanding the ontology prematurely inflates prompt context and slows SPARQL queries without improving reasoning quality.\n\n7. **Reasoning cost explosion**: Keep belief chains to 3 levels or fewer (belief -> desire -> intention). Deeper chains become prohibitively expensive for LLM inference and rarely improve decision quality over shallower alternatives.\n\n## Integration\n\nThis skill owns formal mental-state modeling. Adjacent skills own different layers:\n\n- `memory-systems`: persistent facts, entity memory, and temporal knowledge graphs without BDI belief/desire/intention semantics.\n- `multi-agent-patterns`: agent topology, handoff protocols, and coordination between agents.\n- `evaluation`: competency questions, regression checks, and quality gates for BDI implementations.\n- `context-fundamentals`: conceptual context-window and attention mechanics that inform prompt construction.\n- `tool-design`: schema and tool contracts for BDI query, validation, or projection tools.\n\n## References\n\nInternal references:\n- [BDI Ontology Core](./references/bdi-ontology-core.md) - Read when: implementing BDI class hierarchies or defining ontology properties from scratch\n- [RDF Examples](./references/rdf-examples.md) - Read when: writing Turtle serializations of mental states or debugging triple structure\n- [SPARQL Competency Queries](./references/sparql-competency.md) - Read when: validating an implementation against competency questions or building custom queries\n- [Framework Integration](./references/framework-integration.md) - Read when: deploying BDI models to SEMAS, JADE, or LAG pipelines\n\nPrimary sources:\n- Zuppiroli et al. \"The Belief-Desire-Intention Ontology\" (2025) — Read when: implementing formal BDI class hierarchies or validating ontology alignment\n- Rao & Georgeff \"BDI agents: From theory to practice\" (1995) — Read when: understanding the theoretical foundations of practical reasoning agents\n- Bratman \"Intention, plans, and practical reason\" (1987) — Read when: grounding implementation decisions in the philosophical basis of intentionality\n\n---\n\n## Skill Metadata\n\n**Created**: 2026-01-07\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.1.0\n","skills/bdi-mental-states/references/bdi-ontology-core.md":"# BDI Ontology Core Patterns\n\nCore ontology design patterns for Belief-Desire-Intention mental state modeling.\n\n## Class Hierarchy\n\n### Mental Entities (Endurants)\n\n```\nbdi:MentalEntity\n├── bdi:Belief # Informational dimension\n├── bdi:Desire # Motivational dimension \n├── bdi:Intention # Deliberative dimension\n├── bdi:Goal # Description of desired end state\n└── bdi:Plan # Structured action sequence\n```\n\n### Mental Processes (Perdurants)\n\n```\nbdi:MentalProcess\n├── bdi:BeliefProcess # Forms/updates beliefs from perception\n├── bdi:DesireProcess # Generates desires from beliefs\n├── bdi:IntentionProcess # Commits to desires as intentions\n├── bdi:Planning # Transforms intentions into plans\n└── bdi:PlanExecution # Executes plan actions\n```\n\n### Supporting Entities\n\n```\nbdi:WorldState # Configuration of environment\nbdi:Justification # Evidential basis for mental states\nbdi:Task # Atomic unit of planned action\nbdi:Action # Execution of a task\nbdi:TimeInterval # Temporal validity bounds\nbdi:TimeInstant # Point in time reference\n```\n\n## Object Properties\n\n### Motivational Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `motivates` | Belief | Desire | Belief provides reason for desire |\n| `isMotivatedBy` | Desire | Belief | Inverse of motivates |\n| `fulfils` | Intention | Desire | Intention commits to achieving desire |\n| `isFulfilledBy` | Desire | Intention | Inverse of fulfils |\n| `isSupportedBy` | Intention | Belief | Beliefs supporting intention viability |\n\n### Generative Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `generates` | MentalProcess | MentalEntity | Process creates mental state |\n| `isGeneratedBy` | MentalEntity | MentalProcess | Inverse of generates |\n| `modifies` | MentalProcess | MentalEntity | Process updates existing state |\n| `suppresses` | MentalProcess | MentalEntity | Process deactivates state |\n| `isTriggeredBy` | MentalProcess | MentalEntity | State initiates process |\n\n### Referential Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `refersTo` | MentalEntity | WorldState | Mental state about world |\n| `perceives` | Agent | WorldState | Agent observes world |\n| `bringsAbout` | Action | WorldState | Action causes world change |\n| `reasonsUpon` | MentalProcess | MentalEntity | Input to reasoning |\n\n### Structural Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `hasPart` | MentalEntity | MentalEntity | Meronymic composition |\n| `specifies` | Intention | Plan | Intention defines plan |\n| `addresses` | Plan | Goal | Plan achieves goal |\n| `hasComponent` | Plan | Task | Plan contains tasks |\n| `precedes` | Task | Task | Task ordering |\n\n### Temporal Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `atTime` | Entity | TimeInstant | Point occurrence |\n| `hasValidity` | MentalEntity | TimeInterval | Persistence bounds |\n| `hasStartTime` | TimeInterval | TimeInstant | Interval start |\n| `hasEndTime` | TimeInterval | TimeInstant | Interval end |\n\n### Justification Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `isJustifiedBy` | MentalEntity | Justification | Evidential support |\n| `justifies` | Justification | MentalEntity | Inverse relation |\n\n## Ontological Restrictions\n\n### Belief Restrictions\n\n```turtle\nbdi:Belief rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:refersTo ;\n owl:someValuesFrom bdi:WorldState\n] .\n\nbdi:Belief rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:hasValidity ;\n owl:maxCardinality 1\n] .\n```\n\n### Desire Restrictions\n\n```turtle\nbdi:Desire rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:isMotivatedBy ;\n owl:someValuesFrom bdi:Belief\n] .\n```\n\n### Intention Restrictions\n\n```turtle\nbdi:Intention rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:fulfils ;\n owl:cardinality 1\n] .\n\nbdi:Intention rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:isSupportedBy ;\n owl:someValuesFrom bdi:Belief\n] .\n```\n\n### Mental Process Restrictions\n\n```turtle\nbdi:BeliefProcess rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:generates ;\n owl:allValuesFrom bdi:Belief\n] .\n\nbdi:DesireProcess rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:generates ;\n owl:allValuesFrom bdi:Desire\n] .\n\nbdi:IntentionProcess rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:generates ;\n owl:allValuesFrom bdi:Intention\n] .\n```\n\n## DOLCE Alignment\n\nThe BDI ontology aligns with DOLCE Ultra Lite (DUL) foundational ontology:\n\n| BDI Class | DUL Superclass | Rationale |\n|-----------|----------------|-----------|\n| `Agent` | `dul:Agent` | Intentional entity capable of action |\n| `Belief` | `dul:InformationObject` | Information-bearing entity |\n| `Desire` | `dul:Description` | Describes desired state |\n| `Intention` | `dul:Description` | Describes committed course |\n| `Goal` | `dul:Goal` | Desired end state description |\n| `Plan` | `dul:Plan` | Organized action sequence |\n| `WorldState` | `dul:Situation` | Configuration of entities |\n| `MentalProcess` | `dul:Event` | Temporally extended occurrence |\n| `Task` | `dul:Task` | Unit of planned work |\n| `Action` | `dul:Action` | Performed task instance |\n\n## Reused Ontology Design Patterns\n\n### EventCore Pattern\nUsed for mental processes with temporal aspects and participant roles.\n\n### Situation Pattern \nUsed for world state configurations that mental states reference.\n\n### TimeIndexedSituation Pattern\nUsed for associating mental states with validity intervals.\n\n### BasicPlan Pattern\nUsed for goal-plan-task structures linking intentions to actions.\n\n### Provenance Pattern\nUsed for justification tracking and evidential chains.\n\n## Namespace Declarations\n\n```turtle\n@prefix bdi: .\n@prefix dul: .\n@prefix owl: .\n@prefix rdf: .\n@prefix rdfs: .\n@prefix xsd: .\n```\n\n","skills/bdi-mental-states/references/framework-integration.md":"# BDI Framework Integration Patterns\n\nIntegration patterns for connecting BDI ontology with executable agent frameworks.\n\n## SEMAS Rule Translation\n\nMap BDI ontology constructs to SEMAS production rules.\n\n### Ontology-to-Rule Mapping\n\n| BDI Construct | SEMAS Element | Example |\n|---------------|---------------|---------|\n| Belief | HEAD fact | `belief(agent_a, store_open)` |\n| Supporting beliefs | CONDITIONALS | `[CONDITIONALS: time(weekday)]` |\n| Desire generation | TAIL action | `generate_desire(agent, goal)` |\n| Intention commitment | TAIL action | `commit_intention(agent, goal)` |\n| Plan specification | TAIL action | `create_plan(agent, plan_id)` |\n\n### Rule Templates\n\n**Belief triggers desire formation:**\n```prolog\n[HEAD: belief(Agent, Fact)] / \n[CONDITIONALS: context_condition(Agent, Context)] » \n[TAIL: generate_desire(Agent, DesiredState)].\n```\n\n**Desire triggers intention commitment:**\n```prolog\n[HEAD: desire(Agent, Goal)] / \n[CONDITIONALS: belief(Agent, SupportingFact1), \n belief(Agent, SupportingFact2)] » \n[TAIL: commit_intention(Agent, Goal)].\n```\n\n**Intention triggers planning:**\n```prolog\n[HEAD: intention(Agent, Goal)] / \n[CONDITIONALS: goal(GoalSpec)] » \n[TAIL: create_plan(Agent, PlanId)].\n```\n\n**Plan triggers execution:**\n```prolog\n[HEAD: plan(Agent, PlanId)] / \n[CONDITIONALS: ready_to_execute(Agent)] » \n[TAIL: execute_plan(Agent, PlanId)].\n```\n\n### Complete SEMAS Example\n\n```prolog\n% ============================================================\n% GROCERY SHOPPING SCENARIO\n% ============================================================\n\n% Phase 1: Belief formation from world state\n[HEAD: perceive(agent_a, store_open)] / \n[CONDITIONALS: time(weekday_afternoon)] » \n[TAIL: add_belief(agent_a, store_open)].\n\n% Phase 2: Desire generation from belief\n[HEAD: belief(agent_a, store_open)] / \n[CONDITIONALS: belief(agent_a, needs_groceries)] » \n[TAIL: generate_desire(agent_a, buy_groceries)].\n\n% Phase 3: Intention commitment from desire\n[HEAD: desire(agent_a, buy_groceries)] / \n[CONDITIONALS: belief(agent_a, has_shopping_list), \n belief(agent_a, store_open),\n belief(agent_a, has_transportation)] » \n[TAIL: commit_intention(agent_a, buy_groceries)].\n\n% Phase 4: Plan creation from intention\n[HEAD: intention(agent_a, buy_groceries)] / \n[CONDITIONALS: goal(complete_shopping)] » \n[TAIL: create_plan(agent_a, shopping_plan)].\n\n% Phase 5: Plan execution\n[HEAD: plan(agent_a, shopping_plan)] / \n[CONDITIONALS: preconditions_met(shopping_plan)] » \n[TAIL: execute_task(agent_a, drive_to_store),\n execute_task(agent_a, select_items),\n execute_task(agent_a, checkout),\n execute_task(agent_a, return_home)].\n\n% Phase 6: World state update\n[HEAD: task_complete(agent_a, checkout)] / \n[CONDITIONALS: items_purchased(agent_a)] » \n[TAIL: update_world_state(has_groceries),\n remove_desire(agent_a, buy_groceries),\n remove_intention(agent_a, buy_groceries)].\n```\n\n### Python Translation Layer\n\n```python\nfrom rdflib import Graph, Namespace, RDF\n\nBDI = Namespace(\"https://w3id.org/fossr/ontology/bdi/\")\n\ndef ontology_to_semas_rules(bdi_graph: Graph) -> list[str]:\n \"\"\"\n Translate BDI ontology instances to SEMAS production rules.\n \"\"\"\n rules = []\n \n # Extract belief-desire-intention chains\n for intention in bdi_graph.subjects(RDF.type, BDI.Intention):\n # Get supporting beliefs\n supporting_beliefs = list(bdi_graph.objects(intention, BDI.isSupportedBy))\n \n # Get fulfilled desire\n fulfilled_desires = list(bdi_graph.objects(intention, BDI.fulfils))\n \n # Get specified plan\n specified_plans = list(bdi_graph.objects(intention, BDI.specifies))\n \n if fulfilled_desires and supporting_beliefs:\n desire = fulfilled_desires[0]\n beliefs_str = \", \".join([format_belief(b, bdi_graph) for b in supporting_beliefs])\n \n rule = (\n f\"[HEAD: {format_desire(desire, bdi_graph)}] / \"\n f\"[CONDITIONALS: {beliefs_str}] » \"\n f\"[TAIL: commit_intention({format_intention(intention, bdi_graph)})]\"\n )\n rules.append(rule)\n \n if specified_plans:\n plan = specified_plans[0]\n rule = (\n f\"[HEAD: {format_intention(intention, bdi_graph)}] / \"\n f\"[CONDITIONALS: ready_to_plan] » \"\n f\"[TAIL: create_plan({format_plan(plan, bdi_graph)})]\"\n )\n rules.append(rule)\n \n return rules\n\ndef format_belief(belief_uri, graph):\n label = graph.value(belief_uri, RDFS.label)\n return f\"belief({label or belief_uri.split('/')[-1]})\"\n\ndef format_desire(desire_uri, graph):\n label = graph.value(desire_uri, RDFS.label)\n return f\"desire({label or desire_uri.split('/')[-1]})\"\n\ndef format_intention(intention_uri, graph):\n label = graph.value(intention_uri, RDFS.label)\n return f\"intention({label or intention_uri.split('/')[-1]})\"\n\ndef format_plan(plan_uri, graph):\n label = graph.value(plan_uri, RDFS.label)\n return f\"plan({label or plan_uri.split('/')[-1]})\"\n```\n\n## Logic Augmented Generation (LAG)\n\nAugment LLM outputs with BDI ontological constraints.\n\n### LAG Pipeline Architecture\n\n```\n┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐\n│ User Query │────▶│ Ontology │────▶│ Augmented │\n│ │ │ Injection │ │ Prompt │\n└─────────────────┘ └─────────────────┘ └─────────────────┘\n │\n ▼\n┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐\n│ Validated │◀────│ Ontology │◀────│ LLM Response │\n│ RDF Triples │ │ Validation │ │ (Triples) │\n└─────────────────┘ └─────────────────┘ └─────────────────┘\n```\n\n### LAG Implementation\n\n```python\nfrom rdflib import Graph, Namespace\nfrom rdflib.plugins.parsers.notation3 import BadSyntax\n\nBDI = Namespace(\"https://w3id.org/fossr/ontology/bdi/\")\n\nclass BDILogicAugmentedGenerator:\n def __init__(self, ontology_path: str, llm_client):\n self.ontology = Graph()\n self.ontology.parse(ontology_path, format='turtle')\n self.llm = llm_client\n \n def generate_mental_states(self, context: str) -> Graph:\n \"\"\"\n Generate BDI mental states from context using LAG.\n \"\"\"\n # Phase 1: Inject ontology into prompt\n ontology_turtle = self.ontology.serialize(format='turtle')\n augmented_prompt = self._build_augmented_prompt(context, ontology_turtle)\n \n # Phase 2: Generate with LLM\n response = self.llm.generate(augmented_prompt)\n \n # Phase 3: Extract and validate triples\n triples = self._extract_triples(response)\n validated = self._validate_against_ontology(triples)\n \n if not validated['is_consistent']:\n # Retry with feedback\n return self._retry_with_feedback(context, validated['errors'])\n \n return validated['graph']\n \n def _build_augmented_prompt(self, context: str, ontology: str) -> str:\n return f\"\"\"\nYou are a BDI mental state modeler. Given the following context, generate \nRDF triples representing the agent's beliefs, desires, and intentions.\n\n## BDI Ontology (use these classes and properties):\n{ontology}\n\n## Context to Model:\n{context}\n\n## Instructions:\n1. Identify world states from the context\n2. Generate beliefs that refer to those world states\n3. Generate desires motivated by those beliefs\n4. Generate intentions that fulfill desires and are supported by beliefs\n5. Include justifications for each mental state\n6. Include temporal validity intervals\n\nOutput valid Turtle RDF triples only.\n\"\"\"\n \n def _extract_triples(self, response: str) -> str:\n \"\"\"Extract Turtle content from LLM response.\"\"\"\n # Find turtle block in response\n if \"```turtle\" in response:\n start = response.find(\"```turtle\") + 9\n end = response.find(\"```\", start)\n return response[start:end].strip()\n return response\n \n def _validate_against_ontology(self, triples: str) -> dict:\n \"\"\"Validate generated triples against BDI ontology.\"\"\"\n result = {'is_consistent': True, 'errors': [], 'graph': None}\n \n try:\n generated = Graph()\n generated.parse(data=triples, format='turtle')\n result['graph'] = generated\n \n # Validate constraints\n errors = []\n \n # Check: Every intention must fulfill a desire\n for intention in generated.subjects(RDF.type, BDI.Intention):\n if not list(generated.objects(intention, BDI.fulfils)):\n errors.append(f\"Intention {intention} does not fulfill any desire\")\n \n # Check: Every belief should reference a world state\n for belief in generated.subjects(RDF.type, BDI.Belief):\n if not list(generated.objects(belief, BDI.refersTo)):\n errors.append(f\"Belief {belief} does not reference a world state\")\n \n # Check: Desires should be motivated by beliefs\n for desire in generated.subjects(RDF.type, BDI.Desire):\n if not list(generated.objects(desire, BDI.isMotivatedBy)):\n errors.append(f\"Desire {desire} has no motivating belief\")\n \n if errors:\n result['is_consistent'] = False\n result['errors'] = errors\n \n except BadSyntax as e:\n result['is_consistent'] = False\n result['errors'] = [f\"Invalid Turtle syntax: {e}\"]\n \n return result\n \n def _retry_with_feedback(self, context: str, errors: list) -> Graph:\n \"\"\"Retry generation with error feedback.\"\"\"\n feedback_prompt = f\"\"\"\nPrevious generation had errors:\n{chr(10).join(errors)}\n\nPlease regenerate the mental states fixing these issues.\n\nContext: {context}\n\"\"\"\n response = self.llm.generate(feedback_prompt)\n triples = self._extract_triples(response)\n result = self._validate_against_ontology(triples)\n \n if result['is_consistent']:\n return result['graph']\n else:\n raise ValueError(f\"Failed to generate valid mental states: {result['errors']}\")\n```\n\n### Inconsistency Detection Example\n\n```python\ndef detect_location_inconsistency(graph: Graph) -> list[str]:\n \"\"\"\n Detect inconsistencies where agent cannot be in two places.\n \"\"\"\n inconsistencies = []\n \n # Query for location beliefs\n query = \"\"\"\n PREFIX bdi: \n \n SELECT ?agent ?belief1 ?belief2 ?loc1 ?loc2 WHERE {\n ?agent bdi:hasBelief ?belief1 , ?belief2 .\n ?belief1 bdi:refersTo ?ws1 .\n ?belief2 bdi:refersTo ?ws2 .\n ?ws1 bdi:hasLocation ?loc1 .\n ?ws2 bdi:hasLocation ?loc2 .\n FILTER(?belief1 != ?belief2 && ?loc1 != ?loc2)\n \n # Check temporal overlap\n ?belief1 bdi:hasValidity ?interval1 .\n ?belief2 bdi:hasValidity ?interval2 .\n ?interval1 bdi:hasStartTime ?start1 ; bdi:hasEndTime ?end1 .\n ?interval2 bdi:hasStartTime ?start2 ; bdi:hasEndTime ?end2 .\n FILTER(?start1 < ?end2 && ?start2 < ?end1)\n }\n \"\"\"\n \n for row in graph.query(query):\n inconsistencies.append(\n f\"Agent {row.agent} has conflicting location beliefs: \"\n f\"{row.loc1} and {row.loc2} at overlapping times\"\n )\n \n return inconsistencies\n```\n\n## JADE/JADEX Integration\n\nMap BDI ontology to JADE/JADEX agent platform structures.\n\n### JADE Agent Structure\n\n```java\npublic class BDIAgent extends Agent {\n // Mental state storage (maps to ontology individuals)\n private Set beliefs = new HashSet<>();\n private Set desires = new HashSet<>();\n private Set intentions = new HashSet<>();\n \n // Ontology-backed mental state management\n private Graph mentalStateGraph;\n \n public void addBelief(Belief belief) {\n beliefs.add(belief);\n \n // Add to RDF graph\n Resource beliefResource = mentalStateGraph.createResource(belief.getUri());\n beliefResource.addProperty(RDF.type, BDI.Belief);\n beliefResource.addProperty(BDI.refersTo, belief.getWorldState().getUri());\n beliefResource.addProperty(BDI.hasValidity, createInterval(belief.getValidity()));\n \n // Trigger desire formation\n triggerDesireProcess(belief);\n }\n \n public void commitIntention(Intention intention) {\n intentions.add(intention);\n \n Resource intentionResource = mentalStateGraph.createResource(intention.getUri());\n intentionResource.addProperty(RDF.type, BDI.Intention);\n intentionResource.addProperty(BDI.fulfils, intention.getDesire().getUri());\n \n for (Belief support : intention.getSupportingBeliefs()) {\n intentionResource.addProperty(BDI.isSupportedBy, support.getUri());\n }\n \n // Trigger planning\n triggerPlanning(intention);\n }\n \n // Export mental states as RDF\n public String exportMentalStates() {\n return mentalStateGraph.serialize(Format.TURTLE);\n }\n \n // Import mental states from RDF\n public void importMentalStates(String turtle) {\n Graph imported = new Graph();\n imported.parse(turtle, Format.TURTLE);\n \n // Reconstruct Java objects from RDF\n for (Resource belief : imported.listSubjectsWithProperty(RDF.type, BDI.Belief)) {\n Belief b = reconstructBelief(belief);\n beliefs.add(b);\n }\n // ... similar for desires and intentions\n }\n}\n```\n\n### JADEX Goal Mapping\n\n```java\n// Map BDI ontology goals to JADEX goals\n@Goal\npublic class OntologyBackedGoal {\n @GoalParameter\n protected String goalUri;\n \n @GoalParameter\n protected Graph ontologyGraph;\n \n public OntologyBackedGoal(Resource goalResource, Graph graph) {\n this.goalUri = goalResource.getURI();\n this.ontologyGraph = graph;\n }\n \n @GoalTargetCondition\n public boolean isAchieved() {\n // Query ontology for goal achievement\n String query = \"\"\"\n PREFIX bdi: \n ASK {\n ?execution bdi:addresses <%s> ;\n bdi:bringsAbout ?worldState .\n }\n \"\"\".formatted(goalUri);\n \n return ontologyGraph.ask(query);\n }\n \n @GoalDropCondition\n public boolean shouldDrop() {\n // Check if supporting beliefs are invalidated\n String query = \"\"\"\n PREFIX bdi: \n ASK {\n ?intention bdi:specifies ?plan .\n ?plan bdi:addresses <%s> .\n ?intention bdi:isSupportedBy ?belief .\n ?belief bdi:hasValidity ?interval .\n ?interval bdi:hasEndTime ?end .\n FILTER(?end < NOW())\n }\n \"\"\".formatted(goalUri);\n \n return ontologyGraph.ask(query);\n }\n}\n```\n\n## RDF Triple Store Integration\n\n### Triple Store Configuration\n\n```python\nfrom rdflib import Graph\nfrom rdflib.plugins.stores.sparqlstore import SPARQLUpdateStore\n\nclass BDIMentalStateStore:\n def __init__(self, endpoint: str):\n self.store = SPARQLUpdateStore()\n self.store.open((endpoint + \"/query\", endpoint + \"/update\"))\n self.graph = Graph(store=self.store, identifier=\"http://example.org/bdi\")\n \n def add_belief(self, agent_uri: str, belief_data: dict):\n \"\"\"Add belief to triple store.\"\"\"\n belief_uri = f\"{agent_uri}/belief/{belief_data['id']}\"\n \n self.graph.add((URIRef(belief_uri), RDF.type, BDI.Belief))\n self.graph.add((URIRef(belief_uri), RDFS.label, Literal(belief_data['label'])))\n self.graph.add((URIRef(belief_uri), BDI.refersTo, URIRef(belief_data['world_state'])))\n self.graph.add((URIRef(agent_uri), BDI.hasMentalState, URIRef(belief_uri)))\n \n # Add temporal validity\n interval_uri = f\"{belief_uri}/validity\"\n self.graph.add((URIRef(belief_uri), BDI.hasValidity, URIRef(interval_uri)))\n self.graph.add((URIRef(interval_uri), BDI.hasStartTime, \n Literal(belief_data['start_time'], datatype=XSD.dateTime)))\n self.graph.add((URIRef(interval_uri), BDI.hasEndTime,\n Literal(belief_data['end_time'], datatype=XSD.dateTime)))\n \n def get_active_beliefs(self, agent_uri: str, at_time: datetime) -> list:\n \"\"\"Query beliefs active at specific time.\"\"\"\n query = \"\"\"\n PREFIX bdi: \n PREFIX xsd: \n \n SELECT ?belief ?label WHERE {\n <%s> bdi:hasMentalState ?belief .\n ?belief a bdi:Belief ;\n rdfs:label ?label ;\n bdi:hasValidity ?interval .\n ?interval bdi:hasStartTime ?start ;\n bdi:hasEndTime ?end .\n FILTER(?start <= \"%s\"^^xsd:dateTime && ?end >= \"%s\"^^xsd:dateTime)\n }\n \"\"\" % (agent_uri, at_time.isoformat(), at_time.isoformat())\n \n return list(self.graph.query(query))\n \n def get_cognitive_chain(self, intention_uri: str) -> dict:\n \"\"\"Trace complete cognitive chain for an intention.\"\"\"\n query = \"\"\"\n PREFIX bdi: \n \n SELECT ?intention ?desire ?belief ?worldState ?plan WHERE {\n <%s> a bdi:Intention ;\n bdi:fulfils ?desire ;\n bdi:isSupportedBy ?belief .\n OPTIONAL { <%s> bdi:specifies ?plan }\n ?desire bdi:isMotivatedBy ?belief .\n ?belief bdi:refersTo ?worldState .\n }\n \"\"\" % (intention_uri, intention_uri)\n \n results = list(self.graph.query(query))\n if results:\n row = results[0]\n return {\n 'intention': str(row.intention),\n 'desire': str(row.desire),\n 'belief': str(row.belief),\n 'world_state': str(row.worldState),\n 'plan': str(row.plan) if row.plan else None\n }\n return None\n```\n\n## FIPA ACL Integration\n\nMap BDI mental states to FIPA Agent Communication Language.\n\n```python\nfrom fipa_acl import ACLMessage, Performative\n\nclass BDICommunicator:\n def __init__(self, agent_id: str, mental_state_store: BDIMentalStateStore):\n self.agent_id = agent_id\n self.store = mental_state_store\n \n def share_belief(self, belief_uri: str, receiver: str) -> ACLMessage:\n \"\"\"Create INFORM message to share belief.\"\"\"\n belief_triples = self.store.get_belief_as_turtle(belief_uri)\n \n message = ACLMessage()\n message.performative = Performative.INFORM\n message.sender = self.agent_id\n message.receiver = receiver\n message.content = belief_triples\n message.ontology = \"https://w3id.org/fossr/ontology/bdi/\"\n message.language = \"turtle\"\n \n return message\n \n def request_belief_confirmation(self, belief_uri: str, receiver: str) -> ACLMessage:\n \"\"\"Create QUERY-IF message to confirm shared belief.\"\"\"\n message = ACLMessage()\n message.performative = Performative.QUERY_IF\n message.sender = self.agent_id\n message.receiver = receiver\n message.content = f\"ASK {{ <{belief_uri}> a bdi:Belief }}\"\n message.language = \"sparql\"\n \n return message\n \n def propose_intention(self, intention_uri: str, receiver: str) -> ACLMessage:\n \"\"\"Create PROPOSE message for coordinated intention.\"\"\"\n intention_triples = self.store.get_intention_as_turtle(intention_uri)\n \n message = ACLMessage()\n message.performative = Performative.PROPOSE\n message.sender = self.agent_id\n message.receiver = receiver\n message.content = intention_triples\n message.ontology = \"https://w3id.org/fossr/ontology/bdi/\"\n \n return message\n```\n\n","skills/bdi-mental-states/references/rdf-examples.md":"# BDI RDF Examples\n\nComplete RDF/Turtle examples for BDI mental state modeling.\n\n## Complete Cognitive Workflow\n\n```turtle\n@prefix bdi: .\n@prefix ex: .\n@prefix xsd: .\n@prefix rdfs: .\n\n# ============================================================\n# PHASE 1: World State Perception\n# ============================================================\n\nex:WorldState_traffic a bdi:WorldState ;\n rdfs:comment \"Heavy traffic on Route 101\" ;\n bdi:atTime \"2026-01-04T08:30:00\"^^xsd:dateTime ;\n bdi:isPerceivedBy ex:Agent_commuter ;\n bdi:triggers ex:BeliefProcess_assess_traffic .\n\n# ============================================================\n# PHASE 2: Belief Formation\n# ============================================================\n\nex:BeliefProcess_assess_traffic a bdi:BeliefProcess ;\n bdi:generates ex:Belief_traffic_delay ;\n bdi:reasonsUpon ex:WorldState_traffic ;\n bdi:isProcessedBy ex:Agent_commuter ;\n bdi:atTime \"2026-01-04T08:31:00\"^^xsd:dateTime .\n\nex:Belief_traffic_delay a bdi:Belief ;\n rdfs:label \"Traffic will cause 30-minute delay\" ;\n bdi:refersTo ex:WorldState_traffic ;\n bdi:hasValidity ex:TimeInterval_morning_commute ;\n bdi:hasPart ex:Belief_route_congested , ex:Belief_delay_duration ;\n bdi:isJustifiedBy ex:Justification_traffic_report ;\n bdi:motivates ex:Desire_arrive_on_time .\n\nex:Belief_route_congested a bdi:Belief ;\n rdfs:comment \"Route 101 is congested\" .\n\nex:Belief_delay_duration a bdi:Belief ;\n rdfs:comment \"Delay estimated at 30 minutes\" .\n\nex:Justification_traffic_report a bdi:Justification ;\n rdfs:label \"Real-time traffic data from navigation system\" ;\n bdi:justifies ex:Belief_traffic_delay .\n\n# ============================================================\n# PHASE 3: Desire Formation\n# ============================================================\n\nex:DesireProcess_plan_arrival a bdi:DesireProcess ;\n bdi:generates ex:Desire_arrive_on_time ;\n bdi:reasonsUpon ex:Belief_traffic_delay ;\n bdi:isProcessedBy ex:Agent_commuter .\n\nex:Desire_arrive_on_time a bdi:Desire ;\n rdfs:label \"I desire to arrive at work on time\" ;\n bdi:isMotivatedBy ex:Belief_traffic_delay ;\n bdi:refersTo ex:WorldState_on_time_arrival .\n\n# ============================================================\n# PHASE 4: Intention Commitment\n# ============================================================\n\nex:IntentionProcess_commit_route a bdi:IntentionProcess ;\n bdi:generates ex:Intention_take_alternate_route ;\n bdi:reasonsUpon ex:Desire_arrive_on_time ;\n bdi:isProcessedBy ex:Agent_commuter .\n\nex:Intention_take_alternate_route a bdi:Intention ;\n rdfs:label \"I will take alternate route via Highway 280\" ;\n bdi:fulfils ex:Desire_arrive_on_time ;\n bdi:isSupportedBy ex:Belief_traffic_delay ;\n bdi:specifies ex:Plan_alternate_commute ;\n bdi:isJustifiedBy ex:Justification_time_optimization .\n\nex:Justification_time_optimization a bdi:Justification ;\n rdfs:label \"Alternate route saves 20 minutes based on current conditions\" ;\n bdi:justifies ex:Intention_take_alternate_route .\n\n# ============================================================\n# PHASE 5: Planning\n# ============================================================\n\nex:Planning_route_selection a bdi:Planning ;\n bdi:reasonsUpon ex:Intention_take_alternate_route ;\n bdi:defines ex:Plan_alternate_commute ;\n bdi:atTime ex:TimeInterval_planning_phase .\n\nex:Plan_alternate_commute a bdi:Plan ;\n rdfs:label \"Alternate commute via Highway 280\" ;\n bdi:addresses ex:Goal_arrive_by_9am ;\n bdi:beginsWith ex:Task_exit_Route101 ;\n bdi:endsWith ex:Task_arrive_parking ;\n bdi:hasComponent ex:Task_exit_Route101 , ex:Task_merge_280 , \n ex:Task_navigate_280 , ex:Task_arrive_parking .\n\nex:Task_exit_Route101 a bdi:Task ;\n rdfs:label \"Exit Route 101 at Whipple Ave\" ;\n bdi:precedes ex:Task_merge_280 .\n\nex:Task_merge_280 a bdi:Task ;\n rdfs:label \"Merge onto Highway 280 North\" ;\n bdi:precedes ex:Task_navigate_280 .\n\nex:Task_navigate_280 a bdi:Task ;\n rdfs:label \"Continue on Highway 280 for 8 miles\" ;\n bdi:precedes ex:Task_arrive_parking .\n\nex:Task_arrive_parking a bdi:Task ;\n rdfs:label \"Arrive at office parking garage\" .\n\nex:Goal_arrive_by_9am a bdi:Goal ;\n rdfs:label \"Arrive at work by 9:00 AM\" .\n\n# ============================================================\n# PHASE 6: Plan Execution\n# ============================================================\n\nex:PlanExecution_commute a bdi:PlanExecution ;\n bdi:satisfies ex:Plan_alternate_commute ;\n bdi:addresses ex:Goal_arrive_by_9am ;\n bdi:isExecutedBy ex:Agent_commuter ;\n bdi:hasComponent ex:Action_exit , ex:Action_merge , \n ex:Action_drive_280 , ex:Action_park ;\n bdi:atTime ex:TimeInterval_execution ;\n bdi:bringsAbout ex:WorldState_arrived_on_time .\n\nex:Action_exit a bdi:Action ;\n bdi:isExecutionOf ex:Task_exit_Route101 ;\n bdi:isPerformedBy ex:Agent_commuter ;\n bdi:atTime \"2026-01-04T08:35:00\"^^xsd:dateTime .\n\nex:Action_merge a bdi:Action ;\n bdi:isExecutionOf ex:Task_merge_280 ;\n bdi:isPerformedBy ex:Agent_commuter ;\n bdi:atTime \"2026-01-04T08:37:00\"^^xsd:dateTime .\n\nex:Action_drive_280 a bdi:Action ;\n bdi:isExecutionOf ex:Task_navigate_280 ;\n bdi:isPerformedBy ex:Agent_commuter ;\n bdi:atTime \"2026-01-04T08:40:00\"^^xsd:dateTime .\n\nex:Action_park a bdi:Action ;\n bdi:isExecutionOf ex:Task_arrive_parking ;\n bdi:isPerformedBy ex:Agent_commuter ;\n bdi:bringsAbout ex:WorldState_arrived_on_time ;\n bdi:atTime \"2026-01-04T08:52:00\"^^xsd:dateTime .\n\n# ============================================================\n# PHASE 7: Resulting World State\n# ============================================================\n\nex:WorldState_arrived_on_time a bdi:WorldState ;\n rdfs:comment \"Agent arrived at work at 8:52 AM\" ;\n bdi:atTime \"2026-01-04T08:52:00\"^^xsd:dateTime .\n\n# ============================================================\n# TEMPORAL INTERVALS\n# ============================================================\n\nex:TimeInterval_morning_commute a bdi:TimeInterval ;\n bdi:hasStartTime \"2026-01-04T08:30:00\"^^xsd:dateTime ;\n bdi:hasEndTime \"2026-01-04T09:00:00\"^^xsd:dateTime .\n\nex:TimeInterval_planning_phase a bdi:TimeInterval ;\n bdi:hasStartTime \"2026-01-04T08:31:00\"^^xsd:dateTime ;\n bdi:hasEndTime \"2026-01-04T08:34:00\"^^xsd:dateTime .\n\nex:TimeInterval_execution a bdi:TimeInterval ;\n bdi:hasStartTime \"2026-01-04T08:35:00\"^^xsd:dateTime ;\n bdi:hasEndTime \"2026-01-04T08:52:00\"^^xsd:dateTime .\n```\n\n## Multi-Agent Coordination Example\n\n```turtle\n@prefix bdi: .\n@prefix ex: .\n@prefix fipa: .\n\n# Shared belief about project deadline\nex:Agent_developer a bdi:Agent ;\n bdi:hasMentalState ex:Belief_deadline_friday .\n\nex:Agent_manager a bdi:Agent ;\n bdi:hasMentalState ex:Belief_deadline_friday .\n\nex:Belief_deadline_friday a bdi:Belief ;\n rdfs:label \"Project deadline is Friday 5 PM\" ;\n bdi:refersTo ex:WorldState_deadline ;\n bdi:hasValidity ex:TimeInterval_project_week .\n\nex:WorldState_deadline a bdi:WorldState ;\n rdfs:comment \"Project XYZ must be delivered by 2026-01-10T17:00:00\" .\n\n# Agent-specific mental states\nex:Agent_developer \n bdi:hasDesire ex:Desire_complete_coding ;\n bdi:hasIntention ex:Intention_implement_features .\n\nex:Desire_complete_coding a bdi:Desire ;\n rdfs:label \"Complete feature implementation\" ;\n bdi:isMotivatedBy ex:Belief_deadline_friday .\n\nex:Intention_implement_features a bdi:Intention ;\n rdfs:label \"Implement features A, B, and C\" ;\n bdi:fulfils ex:Desire_complete_coding ;\n bdi:specifies ex:Plan_development .\n\nex:Agent_manager \n bdi:hasDesire ex:Desire_ensure_delivery ;\n bdi:hasIntention ex:Intention_coordinate_team .\n\nex:Desire_ensure_delivery a bdi:Desire ;\n rdfs:label \"Ensure on-time project delivery\" ;\n bdi:isMotivatedBy ex:Belief_deadline_friday .\n\nex:Intention_coordinate_team a bdi:Intention ;\n rdfs:label \"Coordinate team activities\" ;\n bdi:fulfils ex:Desire_ensure_delivery ;\n bdi:specifies ex:Plan_project_management .\n\n# FIPA communication\nex:Message_M1 a fipa:ACLMessage ;\n fipa:sender ex:Agent_manager ;\n fipa:receiver ex:Agent_developer ;\n fipa:content ex:Belief_deadline_friday ;\n fipa:performative fipa:inform .\n```\n\n## Conflict Resolution Example\n\n```turtle\n@prefix bdi: .\n@prefix ex: .\n\n# Conflicting location beliefs\nex:Belief_at_home a bdi:Belief ;\n bdi:refersTo ex:WorldState_home ;\n rdfs:comment \"Agent is currently at home\" .\n\nex:Belief_at_office a bdi:Belief ;\n bdi:refersTo ex:WorldState_office ;\n rdfs:comment \"Agent is at office\" .\n\n# Conflicting intentions\nex:Intention_work_from_home a bdi:Intention ;\n bdi:isSupportedBy ex:Belief_at_home ;\n rdfs:label \"Work from home today\" .\n\nex:Intention_attend_meeting a bdi:Intention ;\n bdi:isSupportedBy ex:Belief_at_office ;\n rdfs:label \"Attend in-person meeting\" .\n\n# Justification for conflict resolution\nex:Justification_location_conflict a bdi:Justification ;\n rdfs:comment \"Cannot simultaneously be at home and office\" ;\n bdi:justifies ex:Intention_resolution .\n\n# Resolved intention\nex:Intention_resolution a bdi:Intention ;\n rdfs:label \"Attend meeting via video call from home\" ;\n bdi:fulfils ex:Desire_meeting_participation ;\n bdi:isSupportedBy ex:Belief_at_home ;\n bdi:isJustifiedBy ex:Justification_location_conflict .\n```\n\n## T2B2T Payment Processing Example\n\n```turtle\n@prefix bdi: .\n@prefix ex: .\n@prefix xsd: .\n\n# PHASE 1: Triples-to-Beliefs (External RDF → Internal Mental State)\n\nex:WorldState_notification a bdi:WorldState ;\n rdfs:comment \"Push notification: Ghadeh requested $250 via Zelle\" ;\n bdi:atTime \"2025-10-27T10:15:00\"^^xsd:dateTime ;\n bdi:triggers ex:BeliefProcess_BP1 .\n\nex:BeliefProcess_BP1 a bdi:BeliefProcess ;\n bdi:generates ex:Belief_payment_request ;\n bdi:isProcessedBy ex:Agent_A .\n\nex:Belief_payment_request a bdi:Belief ;\n rdfs:label \"Ghadeh requested $250\" ;\n bdi:refersTo ex:WorldState_notification ;\n bdi:motivates ex:Desire_pay_Ghadeh .\n\nex:Desire_pay_Ghadeh a bdi:Desire ;\n rdfs:label \"Pay Ghadeh $250\" ;\n bdi:isMotivatedBy ex:Belief_payment_request .\n\nex:Intention_I1 a bdi:Intention ;\n rdfs:label \"Pay Ghadeh $250\" ;\n bdi:fulfils ex:Desire_pay_Ghadeh ;\n bdi:specifies ex:Plan_payment .\n\n# PHASE 2: Beliefs-to-Triples (Mental State → External RDF)\n\nex:PlanExecution_PE1 a bdi:PlanExecution ;\n bdi:satisfies ex:Plan_payment ;\n bdi:bringsAbout ex:WorldState_payment_complete .\n\nex:WorldState_payment_complete a bdi:WorldState ;\n rdfs:comment \"Payment of $250 sent to Ghadeh via Zelle\" ;\n bdi:atTime \"2025-10-27T10:20:00\"^^xsd:dateTime .\n```\n\n","skills/bdi-mental-states/references/sparql-competency.md":"# SPARQL Competency Queries\n\nValidation queries for BDI ontology implementations based on competency questions.\n\n## Mental Entity Queries\n\n### CQ1: What are all mental entities?\n\n```sparql\nPREFIX bdi: \nPREFIX rdf: \nPREFIX rdfs: \n\nSELECT DISTINCT ?entity ?type WHERE {\n ?entity rdf:type ?type .\n ?type rdfs:subClassOf* bdi:MentalEntity .\n}\n```\n\n### CQ2: What beliefs does an agent hold?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?label WHERE {\n ?agent bdi:hasMentalState ?belief .\n ?belief a bdi:Belief .\n OPTIONAL { ?belief rdfs:label ?label }\n}\n```\n\n### CQ3: What desires does an agent have?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?desire ?label WHERE {\n ?agent bdi:hasDesire ?desire .\n ?desire a bdi:Desire .\n OPTIONAL { ?desire rdfs:label ?label }\n}\n```\n\n### CQ4: What intentions has an agent committed to?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention ?label WHERE {\n ?agent bdi:hasIntention ?intention .\n ?intention a bdi:Intention .\n OPTIONAL { ?intention rdfs:label ?label }\n}\n```\n\n## Motivational Chain Queries\n\n### CQ5: What beliefs motivated formation of a given desire?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?beliefLabel WHERE {\n ?desire bdi:isMotivatedBy ?belief .\n ?belief a bdi:Belief .\n OPTIONAL { ?belief rdfs:label ?beliefLabel }\n}\n```\n\n### CQ6: Which desire does a particular intention fulfill?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?desire ?desireLabel WHERE {\n ?intention bdi:fulfils ?desire .\n ?desire a bdi:Desire .\n OPTIONAL { ?desire rdfs:label ?desireLabel }\n}\n```\n\n### CQ7: What beliefs support a given intention?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?label WHERE {\n ?intention bdi:isSupportedBy ?belief .\n ?belief a bdi:Belief .\n OPTIONAL { ?belief rdfs:label ?label }\n}\n```\n\n### CQ8: Trace complete cognitive chain for an intention\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention ?desire ?belief ?worldState WHERE {\n ?intention a bdi:Intention ;\n bdi:fulfils ?desire ;\n bdi:isSupportedBy ?belief .\n ?desire bdi:isMotivatedBy ?belief .\n ?belief bdi:refersTo ?worldState .\n}\n```\n\n## Mental Process Queries\n\n### CQ9: Which mental process generated a belief?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?process ?processType WHERE {\n ?process bdi:generates ?belief .\n ?belief a bdi:Belief .\n ?process a ?processType .\n FILTER(?processType != owl:NamedIndividual)\n}\n```\n\n### CQ10: What triggered a mental process?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?process ?trigger ?triggerType WHERE {\n ?process a bdi:MentalProcess ;\n bdi:isTriggeredBy ?trigger .\n ?trigger a ?triggerType .\n}\n```\n\n### CQ11: What did a mental process reason upon?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?process ?input WHERE {\n ?process a bdi:MentalProcess ;\n bdi:reasonsUpon ?input .\n}\n```\n\n## Plan and Goal Queries\n\n### CQ12: What plan does an intention specify?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention ?plan ?goal WHERE {\n ?intention bdi:specifies ?plan .\n ?plan a bdi:Plan ;\n bdi:addresses ?goal .\n}\n```\n\n### CQ13: What is the ordered sequence of tasks in a plan?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?plan ?task ?nextTask WHERE {\n ?plan a bdi:Plan ;\n bdi:hasComponent ?task .\n OPTIONAL { ?task bdi:precedes ?nextTask }\n}\nORDER BY ?task\n```\n\n### CQ14: What is the first and last task of a plan?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?plan ?firstTask ?lastTask WHERE {\n ?plan a bdi:Plan ;\n bdi:beginsWith ?firstTask ;\n bdi:endsWith ?lastTask .\n}\n```\n\n### CQ15: Which actions executed which tasks?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?action ?task ?time WHERE {\n ?action bdi:isExecutionOf ?task ;\n bdi:atTime ?time .\n}\nORDER BY ?time\n```\n\n## Temporal Queries\n\n### CQ16: What mental states are valid at a specific time?\n\n```sparql\nPREFIX bdi: \nPREFIX xsd: \n\nSELECT ?mentalState ?type WHERE {\n ?mentalState bdi:hasValidity ?interval .\n ?interval bdi:hasStartTime ?start ;\n bdi:hasEndTime ?end .\n ?mentalState a ?type .\n FILTER(?start <= \"2026-01-04T10:00:00\"^^xsd:dateTime && \n ?end >= \"2026-01-04T10:00:00\"^^xsd:dateTime)\n}\n```\n\n### CQ17: When was a belief formed?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?formationTime WHERE {\n ?process bdi:generates ?belief ;\n bdi:atTime ?formationTime .\n ?belief a bdi:Belief .\n}\n```\n\n### CQ18: What is the temporal validity of an intention?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention ?start ?end WHERE {\n ?intention a bdi:Intention ;\n bdi:hasValidity ?interval .\n ?interval bdi:hasStartTime ?start ;\n bdi:hasEndTime ?end .\n}\n```\n\n## Justification Queries\n\n### CQ19: What justifies a belief?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?justification ?justLabel WHERE {\n ?belief a bdi:Belief ;\n bdi:isJustifiedBy ?justification .\n OPTIONAL { ?justification rdfs:label ?justLabel }\n}\n```\n\n### CQ20: What justifies an intention?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention ?justification ?justLabel WHERE {\n ?intention a bdi:Intention ;\n bdi:isJustifiedBy ?justification .\n OPTIONAL { ?justification rdfs:label ?justLabel }\n}\n```\n\n## Compositional Queries\n\n### CQ21: What parts comprise a complex belief?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?part ?partLabel WHERE {\n ?belief a bdi:Belief ;\n bdi:hasPart ?part .\n OPTIONAL { ?part rdfs:label ?partLabel }\n}\n```\n\n### CQ22: Find composite mental entities\n\n```sparql\nPREFIX bdi: \n\nSELECT ?composite (COUNT(?part) AS ?partCount) WHERE {\n ?composite bdi:hasPart ?part .\n}\nGROUP BY ?composite\nHAVING (COUNT(?part) > 1)\n```\n\n## World State Queries\n\n### CQ23: What world state does a belief refer to?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?worldState ?wsComment WHERE {\n ?belief a bdi:Belief ;\n bdi:refersTo ?worldState .\n OPTIONAL { ?worldState rdfs:comment ?wsComment }\n}\n```\n\n### CQ24: What actions brought about a world state?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?action ?worldState WHERE {\n ?action bdi:bringsAbout ?worldState .\n ?worldState a bdi:WorldState .\n}\n```\n\n### CQ25: What world states has an agent perceived?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?agent ?worldState ?time WHERE {\n ?agent bdi:perceives ?worldState .\n OPTIONAL { ?worldState bdi:atTime ?time }\n}\n```\n\n## Validation Queries (OWLUnit Style)\n\n### V1: Every intention must fulfill exactly one desire\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention WHERE {\n ?intention a bdi:Intention .\n FILTER NOT EXISTS { ?intention bdi:fulfils ?desire }\n}\n# Expected: Empty result set\n```\n\n### V2: Every belief must reference a world state\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief WHERE {\n ?belief a bdi:Belief .\n FILTER NOT EXISTS { ?belief bdi:refersTo ?worldState }\n}\n# Expected: Empty result set (or only abstract beliefs)\n```\n\n### V3: Mental processes must reason upon something\n\n```sparql\nPREFIX bdi: \n\nSELECT ?process WHERE {\n ?process a bdi:MentalProcess .\n FILTER NOT EXISTS { ?process bdi:reasonsUpon ?input }\n}\n# Expected: Empty result set\n```\n\n### V4: BeliefProcess must generate only Beliefs\n\n```sparql\nPREFIX bdi: \n\nSELECT ?process ?generated WHERE {\n ?process a bdi:BeliefProcess ;\n bdi:generates ?generated .\n FILTER NOT EXISTS { ?generated a bdi:Belief }\n}\n# Expected: Empty result set\n```\n\n### V5: Plans must have begin and end tasks\n\n```sparql\nPREFIX bdi: \n\nSELECT ?plan WHERE {\n ?plan a bdi:Plan .\n FILTER NOT EXISTS { \n ?plan bdi:beginsWith ?first ;\n bdi:endsWith ?last \n }\n}\n# Expected: Empty result set\n```\n\n## Multi-Agent Queries\n\n### CQ26: What beliefs are shared across agents?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief (COUNT(DISTINCT ?agent) AS ?agentCount) WHERE {\n ?agent bdi:hasMentalState ?belief .\n ?belief a bdi:Belief .\n}\nGROUP BY ?belief\nHAVING (COUNT(DISTINCT ?agent) > 1)\n```\n\n### CQ27: Which agents share the same desire?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?desire ?agent1 ?agent2 WHERE {\n ?agent1 bdi:hasDesire ?desire .\n ?agent2 bdi:hasDesire ?desire .\n FILTER(?agent1 != ?agent2)\n}\n```\n\n","skills/context-compression/SKILL.md":"---\nname: context-compression\ndescription: \"This skill should be used when long-running agent sessions need context compression, structured summarization, compaction, token-per-task optimization, or durable handoff summaries that preserve decisions, files, risks, and next actions.\"\n---\n\n# Context Compression Strategies\n\nWhen agent sessions generate millions of tokens of conversation history, compression becomes mandatory. The naive approach is aggressive compression to minimize tokens per request. The correct optimization target is tokens per task: total tokens consumed to complete a task, including re-fetching costs when compression loses critical information.\n\n## When to Activate\n\nActivate this skill when:\n- Agent sessions exceed context window limits\n- Codebases exceed context windows (5M+ token systems)\n- Designing conversation summarization strategies\n- Debugging cases where agents \"forget\" what files they modified\n- Building evaluation frameworks for compression quality\n- Creating durable handoff summaries that preserve decisions, files, risks, and next actions\n\nDo not activate this skill for adjacent work owned by other skills:\n- General token-efficiency tactics such as masking, prefix caching, or partitioning: `context-optimization`.\n- Diagnosing why a long context is failing before choosing a mitigation: `context-degradation`.\n- Writing raw outputs, logs, or plans to files without summarizing them: `filesystem-context`.\n- Designing long-term semantic memory across sessions: `memory-systems`.\n- The model deciding when and where to rewrite its own context: `self-managed-context`. This skill still owns what the replacement text must preserve.\n\n## Core Concepts\n\nContext compression trades token savings against information loss. Select from three production-ready approaches based on session characteristics:\n\n1. **Anchored Iterative Summarization**: Implement this for long-running sessions where file tracking matters. Maintain structured, persistent summaries with explicit sections for session intent, file modifications, decisions, and next steps. When compression triggers, summarize only the newly-truncated span and merge with the existing summary rather than regenerating from scratch. This prevents drift that accumulates when summaries are regenerated wholesale — each regeneration risks losing details the model considers low-priority but the task requires. Structure forces preservation because dedicated sections act as checklists the summarizer must populate, catching silent information loss.\n\n2. **Opaque Compression**: Reserve this for short sessions where re-fetching costs are low and maximum token savings are required. It produces compressed representations optimized for reconstruction fidelity, achieving 99%+ compression ratios but sacrificing interpretability entirely. The tradeoff matters: there is no way to verify what was preserved without running probe-based evaluation, so never use this when debugging or artifact tracking is critical.\n\n3. **Regenerative Full Summary**: Use this when summary readability is critical and sessions have clear phase boundaries. It generates detailed structured summaries on each compression trigger. The weakness is cumulative detail loss across repeated cycles — each full regeneration is a fresh pass that may deprioritize details preserved in earlier summaries.\n\n## Detailed Topics\n\n### Optimize for Tokens-Per-Task, Not Tokens-Per-Request\n\nMeasure total tokens consumed from task start to completion, not tokens per individual request. When compression drops file paths, error messages, or decision rationale, the agent must re-explore, re-read files, and re-derive conclusions — wasting far more tokens than the compression saved. A strategy saving 0.5% more tokens per request but causing 20% more re-fetching costs more overall. Track re-fetching frequency as the primary quality signal: if the agent repeatedly asks to re-read files it already processed, compression is too aggressive.\n\n### Solve the Artifact Trail Problem First\n\nArtifact trail integrity is often the weakest dimension in compression evaluations (claim-context-compression-factory-benchmark). Address this proactively because general summarization cannot reliably maintain it.\n\nPreserve these categories explicitly in every compression cycle:\n- Which files were created (full paths)\n- Which files were modified and what changed (include function names, not just file names)\n- Which files were read but not changed\n- Specific identifiers: function names, variable names, error messages, error codes\n\nImplement a separate artifact index or explicit file-state tracking in agent scaffolding rather than relying on the summarizer to capture these details. Even structured summarization with dedicated file sections struggles with completeness over long sessions.\n\n### Structure Summaries with Mandatory Sections\n\nBuild structured summaries with explicit sections that prevent silent information loss. Each section acts as a checklist the summarizer must populate, making omissions visible rather than silent.\n\n```markdown\n## Session Intent\n[What the user is trying to accomplish]\n\n## Files Modified\n- auth.controller.ts: Fixed JWT token generation\n- config/redis.ts: Updated connection pooling\n- tests/auth.test.ts: Added mock setup for new config\n\n## Decisions Made\n- Using Redis connection pool instead of per-request connections\n- Retry logic with exponential backoff for transient failures\n\n## Current State\n- 14 tests passing, 2 failing\n- Remaining: mock setup for session service tests\n\n## Next Steps\n1. Fix remaining test failures\n2. Run full test suite\n3. Update documentation\n```\n\nAdapt sections to the agent's domain. A debugging agent needs \"Root Cause\" and \"Error Messages\"; a migration agent needs \"Source Schema\" and \"Target Schema.\" The structure matters more than the specific sections — any explicit schema outperforms freeform summarization.\n\n### Choose Compression Triggers Strategically\n\nWhen to trigger compression matters as much as how to compress. Select a trigger strategy based on session predictability:\n\n| Strategy | Trigger Point | Trade-off |\n|----------|---------------|-----------|\n| Fixed threshold | 70-80% context utilization | Simple but may compress too early |\n| Sliding window | Keep last N turns + summary | Predictable context size |\n| Importance-based | Compress low-relevance sections first | Complex but preserves signal |\n| Task-boundary | Compress at logical task completions | Clean summaries but unpredictable timing |\n\nDefault to sliding window with structured summaries for coding agents — it provides the best balance of predictability and quality. Use task-boundary triggers when sessions have clear phase transitions (e.g., research then implementation then testing).\n\n### Evaluate Compression with Probes, Not Metrics\n\nTraditional metrics like ROUGE or embedding similarity fail to capture functional compression quality. A summary can score high on lexical overlap while missing the one file path the agent needs to continue.\n\nUse probe-based evaluation: after compression, pose questions that test whether critical information survived. If the agent answers correctly, compression preserved the right information. If not, it guesses or hallucinates.\n\n| Probe Type | What It Tests | Example Question |\n|------------|---------------|------------------|\n| Recall | Factual retention | \"What was the original error message?\" |\n| Artifact | File tracking | \"Which files have we modified?\" |\n| Continuation | Task planning | \"What should we do next?\" |\n| Decision | Reasoning chain | \"What did we decide about the Redis issue?\" |\n\n### Score Compression Across Six Dimensions\n\nEvaluate compression quality for coding agents across these dimensions. Accuracy and artifact-trail preservation tend to separate methods more clearly than lexical similarity (claim-context-compression-factory-benchmark), so compression needs specialized handling beyond general summarization.\n\n1. **Accuracy**: Are technical details correct — file paths, function names, error codes?\n2. **Context Awareness**: Does the response reflect current conversation state?\n3. **Artifact Trail**: Does the agent know which files were read or modified?\n4. **Completeness**: Does the response address all parts of the question?\n5. **Continuity**: Can work continue without re-fetching information?\n6. **Instruction Following**: Does the response respect stated constraints?\n\n## Practical Guidance\n\n### Apply the Three-Phase Compression Workflow for Large Codebases\n\nFor codebases or agent systems exceeding context windows, compress through three sequential phases. Each phase narrows context so the next phase operates within budget.\n\n1. **Research Phase**: Explore architecture diagrams, documentation, and key interfaces. Compress exploration into a structured analysis of components, dependencies, and boundaries. Output: a single research document that replaces raw exploration.\n\n2. **Planning Phase**: Convert the research document into an implementation specification with function signatures, type definitions, and data flow. A 5M-token codebase compresses to approximately 2,000 words of specification at this stage.\n\n3. **Implementation Phase**: Execute against the specification. Context stays focused on the spec plus active working files, not raw codebase exploration. This phase rarely needs further compression because the spec is already compact.\n\n### Use Example Artifacts as Compression Seeds\n\nWhen provided with a manual migration example or reference PR, use it as a template to understand the target pattern rather than exploring the codebase from scratch. The example reveals constraints static analysis cannot surface: which invariants must hold, which services break on changes, and what a clean implementation looks like.\n\nThis matters most when the agent cannot distinguish essential complexity (business requirements) from accidental complexity (legacy workarounds). The example artifact encodes that distinction implicitly, saving tokens that would otherwise go to trial-and-error exploration.\n\n### Implement Anchored Iterative Summarization Step by Step\n\n1. Define explicit summary sections matching the agent's domain (debugging, migration, feature development)\n2. On first compression trigger, summarize the truncated history into those sections\n3. On subsequent compressions, summarize only newly truncated content — do not re-summarize the existing summary\n4. Merge new information into existing sections rather than regenerating them, deduplicating by file path and decision identity\n5. Tag which information came from which compression cycle — this enables debugging when summaries drift\n\n### Select the Right Approach for the Session Profile\n\n**Use anchored iterative summarization when:**\n- Sessions are long-running (100+ messages)\n- File tracking matters (coding, debugging)\n- Verification of preserved information is needed\n\n**Use opaque compression when:**\n- Maximum token savings are required\n- Sessions are relatively short\n- Re-fetching costs are low (e.g., no file system access needed)\n\n**Use regenerative summaries when:**\n- Summary interpretability is critical for human review\n- Sessions have clear phase boundaries\n- Full context review is acceptable on each compression trigger\n\n### Calibrate Compression Ratios by Method\n\n| Method | Compression Ratio | Quality Score | Trade-off |\n|--------|-------------------|---------------|-----------|\n| Anchored Iterative | 98.6% | 3.70 | Best quality, slightly less compression |\n| Regenerative | 98.7% | 3.44 | Good quality, moderate compression |\n| Opaque | 99.3% | 3.35 | Best compression, quality loss |\n\nUse these as source-specific benchmark figures, not universal constants (claim-context-compression-factory-benchmark). For any task where re-fetching costs exist, this tradeoff generally favors structured approaches.\n\n## Examples\n\n**Example 1: Debugging Session Compression**\n\nOriginal context (89,000 tokens, 178 messages):\n- 401 error on /api/auth/login endpoint\n- Traced through auth controller, middleware, session store\n- Found stale Redis connection\n- Fixed connection pooling, added retry logic\n- 14 tests passing, 2 failing\n\nStructured summary after compression:\n\n```markdown\n## Session Intent\nDebug 401 Unauthorized error on /api/auth/login despite valid credentials.\n\n## Root Cause\nStale Redis connection in session store. JWT generated correctly but session could not be persisted.\n\n## Files Modified\n- auth.controller.ts: No changes (read only)\n- middleware/cors.ts: No changes (examined)\n- config/redis.ts: Fixed connection pooling configuration\n- services/session.service.ts: Added retry logic for transient failures\n- tests/auth.test.ts: Updated mock setup\n\n## Test Status\n14 passing, 2 failing (mock setup issues)\n\n## Next Steps\n1. Fix remaining test failures (mock session service)\n2. Run full test suite\n3. Deploy to staging\n```\n\n**Example 2: Probe Response Quality**\n\nAfter compression, asking \"What was the original error?\":\n\nGood response (structured summarization):\n> \"The original error was a 401 Unauthorized response from the /api/auth/login endpoint. Users received this error with valid credentials. Root cause was stale Redis connection in session store.\"\n\nPoor response (aggressive compression):\n> \"We were debugging an authentication issue. The login was failing. We fixed some configuration problems.\"\n\nThe structured response preserves endpoint, error code, and root cause. The aggressive response loses all technical detail.\n\n## Guidelines\n\n1. Optimize for tokens-per-task, not tokens-per-request\n2. Use structured summaries with explicit sections for file tracking\n3. Trigger compression at 70-80% context utilization\n4. Implement incremental merging rather than full regeneration\n5. Test compression quality with probe-based evaluation\n6. Track artifact trail separately if file tracking is critical\n7. Accept slightly lower compression ratios for better quality retention\n8. Monitor re-fetching frequency as a compression quality signal\n\n## Gotchas\n\n1. **Never compress tool definitions or schemas**: Compressing function call schemas, API specs, or tool definitions destroys agent functionality entirely. The agent cannot invoke tools whose parameter names or types have been summarized away. Treat tool definitions as immutable anchors that bypass compression.\n\n2. **Compressed summaries hallucinate facts**: When an LLM summarizes conversation history, it may introduce plausible-sounding details that never appeared in the original. Always validate compressed output against source material before discarding originals — especially for file paths, error codes, and numeric values that the summarizer may \"round\" or fabricate.\n\n3. **Compression breaks artifact references**: File paths, commit SHAs, variable names, and code snippets get paraphrased or dropped during compression. A summary saying \"updated the config file\" when the agent needs `config/redis.ts` causes re-exploration. Preserve identifiers verbatim in dedicated sections rather than embedding them in prose.\n\n4. **Early turns contain irreplaceable constraints**: The first few turns of a session often contain task setup, user constraints, and architectural decisions that cannot be re-derived. Protect early turns from compression or extract their constraints into a persistent preamble that survives all compression cycles.\n\n5. **Aggressive ratios compound across cycles**: A 95% compression ratio seems safe once, but applying it repeatedly compounds losses. After three cycles at 95%, only 0.0125% of original tokens remain. Calibrate ratios assuming multiple compression cycles, not a single pass.\n\n6. **Code and prose need different compression**: Prose compresses well because natural language is redundant. Code does not — removing a single token from a function signature or import path can make it useless. Apply domain-specific compression strategies: summarize prose sections aggressively while preserving code blocks and structured data verbatim.\n\n7. **Probe-based evaluation gives false confidence**: Probes can pass despite critical information being lost, because the probes test only what they ask about. A probe set that checks file names but not function signatures will miss signature loss. Design probes to cover all six evaluation dimensions, and rotate probe sets across evaluation runs to avoid blind spots.\n\n## Integration\n\nThis skill connects to several others in the collection:\n\n- context-degradation - Compression is a mitigation strategy for degradation\n- context-optimization - Compression is one optimization technique among many\n- evaluation - Probe-based evaluation applies to compression testing\n- memory-systems - Compression relates to scratchpad and summary memory patterns\n- self-managed-context - When the model triggers and places its own compaction edits, this skill supplies what the replacement note preserves\n\n## References\n\nInternal reference:\n- [Evaluation Framework Reference](./references/evaluation-framework.md) - Read when: building or calibrating a probe-based evaluation pipeline, or when needing scoring rubrics and LLM judge configuration for compression quality assessment\n\nRelated skills in this collection:\n- context-degradation - Read when: diagnosing why agent performance drops over long sessions, before applying compression as a mitigation\n- context-optimization - Read when: compression alone is insufficient and broader optimization strategies (pruning, caching, routing) are needed\n- evaluation - Read when: designing evaluation frameworks beyond compression-specific probes, including general LLM-as-judge methodology\n\nExternal resources:\n- Factory Research: Evaluating Context Compression for AI Agents (December 2025) - Read when: needing benchmark data on compression method comparisons or the 36,000-message evaluation dataset\n- Research on LLM-as-judge evaluation methodology (Zheng et al., 2023) - Read when: implementing or validating LLM judge scoring to understand bias patterns and calibration\n- Netflix Engineering: \"The Infinite Software Crisis\" - Three-phase workflow and context compression at scale (AI Summit 2025) - Read when: implementing the three-phase compression workflow for large codebases or understanding production-scale context management\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-22\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 1.3.0\n","skills/context-compression/references/evaluation-framework.md":"# Context Compression Evaluation Framework\n\nThis document provides the complete evaluation framework for measuring context compression quality, including probe types, scoring rubrics, and LLM judge configuration.\n\n## Probe Types\n\n### Recall Probes\n\nTest factual retention of specific details from conversation history.\n\n**Structure:**\n```\nQuestion: [Ask for specific fact from truncated history]\nExpected: [Exact detail that should be preserved]\nScoring: Match accuracy of technical details\n```\n\n**Examples:**\n- \"What was the original error message that started this debugging session?\"\n- \"What version of the dependency did we decide to use?\"\n- \"What was the exact command that failed?\"\n\n### Artifact Probes\n\nTest file tracking and modification awareness.\n\n**Structure:**\n```\nQuestion: [Ask about files created, modified, or examined]\nExpected: [Complete list with change descriptions]\nScoring: Completeness of file list and accuracy of change descriptions\n```\n\n**Examples:**\n- \"Which files have we modified? Describe what changed in each.\"\n- \"What new files did we create in this session?\"\n- \"Which configuration files did we examine but not change?\"\n\n### Continuation Probes\n\nTest ability to continue work without re-fetching context.\n\n**Structure:**\n```\nQuestion: [Ask about next steps or current state]\nExpected: [Actionable next steps based on session history]\nScoring: Ability to continue without requesting re-read of files\n```\n\n**Examples:**\n- \"What should we do next?\"\n- \"What tests are still failing and why?\"\n- \"What was left incomplete from our last step?\"\n\n### Decision Probes\n\nTest retention of reasoning chains and decision rationale.\n\n**Structure:**\n```\nQuestion: [Ask about why a decision was made]\nExpected: [Reasoning that led to the decision]\nScoring: Preservation of decision context and alternatives considered\n```\n\n**Examples:**\n- \"We discussed options for the Redis issue. What did we decide and why?\"\n- \"Why did we choose connection pooling over per-request connections?\"\n- \"What alternatives did we consider for the authentication fix?\"\n\n## Scoring Rubrics\n\n### Accuracy Dimension\n\n| Criterion | Question | Score 0 | Score 3 | Score 5 |\n|-----------|----------|---------|---------|---------|\n| accuracy_factual | Are facts, file paths, and technical details correct? | Completely incorrect or fabricated | Mostly accurate with minor errors | Perfectly accurate |\n| accuracy_technical | Are code references and technical concepts correct? | Major technical errors | Generally correct with minor issues | Technically precise |\n\n### Context Awareness Dimension\n\n| Criterion | Question | Score 0 | Score 3 | Score 5 |\n|-----------|----------|---------|---------|---------|\n| context_conversation_state | Does the response reflect current conversation state? | No awareness of prior context | General awareness with gaps | Full awareness of conversation history |\n| context_artifact_state | Does the response reflect which files/artifacts were accessed? | No awareness of artifacts | Partial artifact awareness | Complete artifact state awareness |\n\n### Artifact Trail Dimension\n\n| Criterion | Question | Score 0 | Score 3 | Score 5 |\n|-----------|----------|---------|---------|---------|\n| artifact_files_created | Does the agent know which files were created? | No knowledge | Knows most files | Perfect knowledge |\n| artifact_files_modified | Does the agent know which files were modified and what changed? | No knowledge | Good knowledge of most modifications | Perfect knowledge of all modifications |\n| artifact_key_details | Does the agent remember function names, variable names, error messages? | No recall | Recalls most key details | Perfect recall |\n\n### Completeness Dimension\n\n| Criterion | Question | Score 0 | Score 3 | Score 5 |\n|-----------|----------|---------|---------|---------|\n| completeness_coverage | Does the response address all parts of the question? | Ignores most parts | Addresses most parts | Addresses all parts thoroughly |\n| completeness_depth | Is sufficient detail provided? | Superficial or missing detail | Adequate detail | Comprehensive detail |\n\n### Continuity Dimension\n\n| Criterion | Question | Score 0 | Score 3 | Score 5 |\n|-----------|----------|---------|---------|---------|\n| continuity_work_state | Can the agent continue without re-fetching previously accessed information? | Cannot continue without re-fetching all context | Can continue with minimal re-fetching | Can continue seamlessly |\n| continuity_todo_state | Does the agent maintain awareness of pending tasks? | Lost track of all TODOs | Good awareness with some gaps | Perfect task awareness |\n| continuity_reasoning | Does the agent retain rationale behind previous decisions? | No memory of reasoning | Generally remembers reasoning | Excellent retention |\n\n### Instruction Following Dimension\n\n| Criterion | Question | Score 0 | Score 3 | Score 5 |\n|-----------|----------|---------|---------|---------|\n| instruction_format | Does the response follow the requested format? | Ignores format | Generally follows format | Perfectly follows format |\n| instruction_constraints | Does the response respect stated constraints? | Ignores constraints | Mostly respects constraints | Fully respects all constraints |\n\n## LLM Judge Configuration\n\n### System Prompt\n\n```\nYou are an expert evaluator assessing AI assistant responses in software development conversations.\n\nYour task is to grade responses against specific rubric criteria. For each criterion:\n1. Read the criterion question carefully\n2. Examine the response for evidence\n3. Assign a score from 0-5 based on the scoring guide\n4. Provide brief reasoning for your score\n\nBe objective and consistent. Focus on what is present in the response, not what could have been included.\n```\n\n### Judge Input Format\n\n```json\n{\n \"probe_question\": \"What was the original error message?\",\n \"model_response\": \"[Response to evaluate]\",\n \"compacted_context\": \"[The compressed context that was provided]\",\n \"ground_truth\": \"[Optional: known correct answer]\",\n \"rubric_criteria\": [\"accuracy_factual\", \"accuracy_technical\", \"context_conversation_state\"]\n}\n```\n\n### Judge Output Format\n\n```json\n{\n \"criterionResults\": [\n {\n \"criterionId\": \"accuracy_factual\",\n \"score\": 5,\n \"reasoning\": \"Response correctly identifies the 401 error, specific endpoint, and root cause.\"\n }\n ],\n \"aggregateScore\": 4.8,\n \"dimensionScores\": {\n \"accuracy\": 4.9,\n \"context_awareness\": 4.5,\n \"artifact_trail\": 3.2,\n \"completeness\": 5.0,\n \"continuity\": 4.8,\n \"instruction_following\": 5.0\n }\n}\n```\n\n## Benchmark Results Reference\n\nPerformance across compression methods (based on 36,000+ messages):\n\n| Method | Overall | Accuracy | Context | Artifact | Complete | Continuity | Instruction |\n|--------|---------|----------|---------|----------|----------|------------|-------------|\n| Anchored Iterative | 3.70 | 4.04 | 4.01 | 2.45 | 4.44 | 3.80 | 4.99 |\n| Regenerative | 3.44 | 3.74 | 3.56 | 2.33 | 4.37 | 3.67 | 4.95 |\n| Opaque | 3.35 | 3.43 | 3.64 | 2.19 | 4.37 | 3.77 | 4.92 |\n\n**Key Findings:**\n\n1. **Accuracy gap**: 0.61 points between best and worst methods\n2. **Context awareness gap**: 0.45 points, favoring anchored iterative\n3. **Artifact trail**: Universally weak (2.19-2.45), needs specialized handling\n4. **Completeness and instruction following**: Minimal differentiation\n\n## Statistical Considerations\n\n- Differences of 0.26-0.35 points are consistent across task types and session lengths\n- Pattern holds for both short and long sessions\n- Pattern holds across debugging, feature implementation, and code review tasks\n- Sample size: 36,611 messages across hundreds of compression points\n\n## Implementation Notes\n\n### Probe Generation\n\nGenerate probes at each compression point based on truncated history:\n1. Extract factual claims for recall probes\n2. Extract file operations for artifact probes\n3. Extract incomplete tasks for continuation probes\n4. Extract decision points for decision probes\n\n### Grading Process\n\n1. Feed probe question + model response + compressed context to judge\n2. Evaluate against each criterion in rubric\n3. Output structured JSON with scores and reasoning\n4. Compute dimension scores as weighted averages\n5. Compute overall score as unweighted average of dimensions\n\n### Blinding\n\nThe judge should not know which compression method produced the response being evaluated. This prevents bias toward known methods.\n\n","skills/context-degradation/SKILL.md":"---\nname: context-degradation\ndescription: \"This skill should be used for diagnosing and mitigating context degradation: lost-in-middle failures, context poisoning, context clash, context confusion, attention-pattern issues, and agent performance degradation caused by accumulated or conflicting context.\"\n---\n\n# Context Degradation Patterns\n\nDiagnose and fix context failures before they cascade. Context degradation is not binary — it is a continuum that manifests through five distinct, predictable patterns: lost-in-middle, poisoning, distraction, confusion, and clash. Each pattern has specific detection signals and mitigation strategies. Treat degradation as an engineering problem with measurable thresholds, not an unpredictable failure mode.\n\n## When to Activate\n\nActivate this skill when:\n- Agent performance degrades unexpectedly during long conversations\n- Debugging cases where agents produce incorrect or irrelevant outputs\n- Designing systems that must handle large contexts reliably\n- Evaluating context engineering choices for production systems\n- Investigating \"lost in middle\" phenomena in agent outputs\n- Analyzing context-related failures in agent behavior\n\nDo not activate this skill for adjacent work owned by other skills:\n- Explaining foundational context mechanics without an active failure: `context-fundamentals`.\n- Applying token-efficiency tactics after the failure pattern is known: `context-optimization`.\n- Designing a compression or handoff summary strategy: `context-compression`.\n- Persisting large outputs, logs, or scratch state outside the prompt: `filesystem-context`.\n\n## Core Concepts\n\nStructure context placement around the attention U-curve: beginning and end positions receive reliable attention, while middle positions suffer materially reduced recall accuracy in long-context experiments (claim-context-degradation-lost-middle-ruler). This is not a model bug but a consequence of attention mechanics — the first token (often BOS) acts as an \"attention sink\" that absorbs disproportionate attention budget, leaving middle tokens under-attended as context grows.\n\nTreat context poisoning as a circuit breaker problem. Once a hallucination, tool error, or incorrect retrieved fact enters context, it compounds through repeated self-reference. A poisoned goals section causes every downstream decision to reinforce incorrect assumptions. Detection requires tracking claim provenance; recovery requires truncating to before the poisoning point or restarting with verified-only context.\n\nFilter aggressively before loading context — even a single irrelevant document measurably degrades performance on relevant tasks. Models cannot \"skip\" irrelevant context; they must attend to everything provided, creating attention competition between relevant and irrelevant content. Move information that might be needed but is not immediately relevant behind tool calls instead of pre-loading it.\n\nIsolate task contexts to prevent confusion. When context contains multiple task types or switches between objectives, models incorporate constraints from the wrong task, call tools appropriate for a different context, or blend requirements from multiple sources. Explicit task segmentation with separate context windows eliminates cross-contamination.\n\nResolve context clash through priority rules, not accumulation. When multiple correct-but-contradictory sources appear in context (version conflicts, perspective conflicts, multi-source retrieval), models cannot determine which applies. Mark contradictions explicitly, establish source precedence, and filter outdated versions before they enter context.\n\n## Detailed Topics\n\n### Lost-in-Middle: Detection and Placement Strategy\n\nPlace critical information at the beginning and end of context, never in the middle. The U-shaped attention curve means middle-positioned information suffers 10-40% reduced recall accuracy. For contexts over 4K tokens, this effect becomes significant.\n\nUse summary structures that surface key findings at attention-favored positions. Add explicit section headers and structural markers — these help models navigate long contexts by creating attention anchors. When a document must be included in full, prepend a summary of its key points and append the critical conclusions.\n\nMonitor for lost-in-middle symptoms: correct information exists in context but the model ignores it, responses contradict provided data, or the model \"forgets\" instructions given earlier in a long prompt.\n\n### Context Poisoning: Prevention and Recovery\n\nValidate all external inputs before they enter context. Tool outputs, retrieved documents, and model-generated summaries are the three primary poisoning vectors. Each introduces unverified claims that subsequent reasoning treats as ground truth.\n\nDetect poisoning through these signals: degraded output quality on previously-successful tasks, tool misalignment (wrong tools or parameters), and hallucinations that persist despite explicit correction. When these cluster, suspect poisoning rather than model capability issues.\n\nRecover by removing poisoned content, not by adding corrections on top. Truncate to before the poisoning point, restart with clean context preserving only verified information, or explicitly mark the poisoned section and request re-evaluation from scratch. Layering corrections over poisoned context rarely works — the original errors retain attention weight.\n\n### Context Distraction: Curation Over Accumulation\n\nCurate what enters context rather than relying on models to ignore irrelevant content. Research shows even a single distractor document triggers measurable performance degradation — the effect follows a step function, not a linear curve. Multiple distractors compound the problem.\n\nApply relevance filtering before loading retrieved documents. Use namespacing and structural organization to make section boundaries clear. Prefer tool-call-based access over pre-loading: store reference material behind retrieval tools so it enters context only when directly relevant to the current reasoning step.\n\n### Context Confusion: Task Isolation\n\nSegment different tasks into separate context windows. Context confusion is distinct from distraction — it concerns the model applying wrong-context constraints to the current task, not just attention dilution. Signs include responses addressing the wrong aspect of a query, tool calls appropriate for a different task, and outputs mixing requirements from multiple sources.\n\nImplement clear transitions between task contexts. Use state management that isolates objectives, constraints, and tool definitions per task. When task-switching within a single session is unavoidable, use explicit \"context reset\" markers that signal which constraints apply to the current segment.\n\n### Context Clash: Conflict Resolution Protocols\n\nEstablish source priority rules before conflicts arise. Context clash differs from poisoning — multiple pieces of information are individually correct but mutually contradictory (version conflicts, perspective differences, multi-source retrieval with divergent facts).\n\nImplement version filtering to exclude outdated information before it enters context. When contradictions are unavoidable, mark them explicitly with structured conflict annotations: state what conflicts, which source each claim comes from, and which source takes precedence. Without explicit priority rules, models resolve contradictions unpredictably.\n\n### Empirical Benchmarks and Thresholds\n\nUse these benchmarks to set design constraints — not as universal truths. RULER-style evidence shows advertised long-context support does not guarantee satisfactory task performance at that length (claim-context-degradation-lost-middle-ruler). Near-perfect needle-in-haystack scores do not predict real-world long-context performance.\n\n**Model-Specific Degradation Thresholds**\n\nDegradation onset varies significantly by model family and task type. As a general rule, expect degradation to begin at 60-70% of the advertised context window for complex retrieval tasks (RULER benchmark found only 50% of models claiming 32K+ context maintain satisfactory performance at that length). Key patterns:\n\n- **Models with extended thinking** reduce hallucination through step-by-step verification but at higher latency and token cost\n- **Models optimized for agents/coding** tend to have better attention management for tool-output-heavy contexts\n- **Models with very large context windows (1M+)** handle more raw context but still follow U-shaped degradation curves — bigger windows do not eliminate the problem, they delay it\n\nAlways benchmark degradation thresholds with your specific workload rather than relying on published benchmarks. Model-specific thresholds go stale with each model update (see Gotcha 2).\n\n### Counterintuitive Findings\n\nAccount for these research-backed surprises when designing context strategies:\n\n**Shuffled context can outperform coherent context.** Studies found incoherent (shuffled) haystacks can outperform logically ordered ones for some retrieval tasks (claim-context-degradation-distractor-shuffled). Coherent context may create false associations that confuse retrieval; incoherent context can force exact matching. Do not assume that better-organized context always yields better results — test both arrangements.\n\n**Single distractors have outsized impact.** The performance hit from one irrelevant document is disproportionately large compared to adding more distractors after the first. Treat distractor prevention as binary: either keep context clean or accept significant degradation.\n\n**Low needle-question similarity accelerates degradation.** Tasks requiring inference across dissimilar content degrade faster with context length than tasks with high surface-level similarity. Design retrieval to maximize semantic overlap between queries and retrieved content.\n\n### When Larger Contexts Hurt\n\nDo not assume larger context windows improve performance. Performance remains stable up to a model-specific threshold, then degrades rapidly — the curve is non-linear with a cliff edge, not a gentle slope. For many models, meaningful degradation begins at 8K-16K tokens even when windows support much larger sizes.\n\nFactor in cost: processing a 400K token context costs exponentially more than 200K in both time and compute, not linearly more. For many applications, this makes large-context processing economically impractical.\n\nRecognize the cognitive bottleneck: even with infinite context, asking a single model to maintain quality across dozens of independent tasks creates degradation that more context cannot solve. Split tasks across sub-agents instead of expanding context.\n\n## Practical Guidance\n\n### The Four-Bucket Mitigation Framework\n\nApply these four strategies based on which degradation pattern is active:\n\n**Write** — Save context outside the window using scratchpads, file systems, or external storage. Use when context utilization exceeds 70% of the window. This keeps active context lean while preserving information access through tool calls.\n\n**Select** — Pull only relevant context into the window through retrieval, filtering, and prioritization. Use when distraction or confusion symptoms appear. Apply relevance scoring before loading; exclude anything below threshold rather than including everything available.\n\n**Compress** — Reduce tokens while preserving information through summarization, abstraction, and observation masking. Use when context is growing but all content is relevant. Replace verbose tool outputs with compact structured summaries; abstract repeated patterns into single references.\n\n**Isolate** — Split context across sub-agents or sessions to prevent any single context from growing past its degradation threshold. Use when confusion or clash symptoms appear, or when tasks are independent. This is the most aggressive strategy but often the most effective for complex multi-task systems.\n\n### Architectural Patterns for Resilience\n\nImplement just-in-time context loading: retrieve information only when the current reasoning step needs it, not preemptively. Use observation masking to replace verbose tool outputs with compact references after processing. Deploy sub-agent architectures where each agent holds only task-relevant context. Trigger compaction before context exceeds the model-specific degradation onset threshold — not after symptoms appear.\n\n## Examples\n\n**Example 1: Detecting Degradation**\n```yaml\n# Context grows during long conversation\nturn_1: 1000 tokens\nturn_5: 8000 tokens\nturn_10: 25000 tokens\nturn_20: 60000 tokens (degradation begins)\nturn_30: 90000 tokens (significant degradation)\n```\n\n**Example 2: Mitigating Lost-in-Middle**\n```markdown\n# Organize context with critical info at edges\n\n[CURRENT TASK] # At start\n- Goal: Generate quarterly report\n- Deadline: End of week\n\n[DETAILED CONTEXT] # Middle (less attention)\n- 50 pages of data\n- Multiple analysis sections\n- Supporting evidence\n\n[KEY FINDINGS] # At end\n- Revenue up 15%\n- Costs down 8%\n- Growth in Region A\n```\n\n**Example 3: Context poisoning circuit breaker**\n```text\nsymptom: agent keeps citing an incorrect retrieved claim after correction\ndiagnosis: poisoned context, not a missing-instruction problem\naction:\n 1. identify first turn where the bad claim entered\n 2. truncate or restart from before that point\n 3. reload only verified sources\n 4. record the rejected claim and source provenance\n```\n\n**Example 4: Context clash annotation**\n```yaml\nconflict:\n topic: billing API version\n source_a: docs/v1.md says endpoint is /charges\n source_b: docs/v2.md says endpoint is /payments\n precedence: docs/v2.md\n reason: current production version\n```\n\n## Guidelines\n\n1. Monitor context length and performance correlation during development\n2. Place critical information at beginning or end of context\n3. Implement compaction triggers before degradation becomes severe\n4. Validate retrieved documents for accuracy before adding to context\n5. Use versioning to prevent outdated information from causing clash\n6. Segment tasks to prevent context confusion across different objectives\n7. Design for graceful degradation rather than assuming perfect conditions\n8. Test with progressively larger contexts to find degradation thresholds\n\n## Gotchas\n\n1. **Normal variance looks like degradation**: Model output quality fluctuates naturally across runs. Do not diagnose degradation from a single drop in quality — establish a baseline over multiple runs and look for sustained, correlated decline tied to context growth. A 5-10% quality dip on one run is noise; the same dip consistently appearing after 40K tokens is signal.\n\n2. **Model-specific thresholds go stale**: The degradation onset values in benchmark tables reflect specific model versions. Provider updates, fine-tuning changes, and infrastructure shifts can move thresholds by 20-50% in either direction. Re-benchmark quarterly and after any major model update rather than treating published thresholds as permanent.\n\n3. **Needle-in-haystack scores create false confidence**: A model scoring 99% on needle-in-haystack does not mean it handles 128K tokens well in production. Needle tests measure single-fact retrieval from passive context — real workloads require multi-fact reasoning, instruction following, and synthesis across the full window. Use task-specific benchmarks that mirror actual workload patterns.\n\n4. **Contradictory retrieved documents poison silently**: When a RAG pipeline retrieves two documents that disagree on a fact, the model may silently pick one without signaling the conflict. This looks like a correct response but is effectively random. Implement contradiction detection in the retrieval layer before documents enter context.\n\n5. **Prompt quality problems masquerade as degradation**: Poor prompt structure (ambiguous instructions, missing constraints, unclear task framing) produces symptoms identical to context degradation — inconsistent outputs, ignored instructions, wrong tool usage. Before diagnosing degradation, verify the same prompt works correctly at low context lengths. If it fails at 2K tokens, the problem is the prompt, not the context.\n\n6. **Degradation is non-linear with a cliff edge**: Performance does not degrade gradually — it holds steady until a model-specific threshold, then drops sharply. Systems designed for \"graceful degradation\" often miss this pattern because monitoring checks assume linear decline. Set compaction triggers well before the cliff (at 70% of known onset), not at the onset itself.\n\n7. **Over-organizing context can backfire**: Intuitively, well-structured and coherent context should outperform disorganized content. Research shows shuffled haystacks sometimes outperform coherent ones for retrieval tasks because coherent context creates false associations. Test whether heavy structural formatting actually helps for the specific task — do not assume it does.\n\n## Integration\n\nThis skill owns diagnosis and mitigation of active context failures. Adjacent skills own the implementation tactics once the failure is identified:\n\n- `context-fundamentals`: conceptual explanation of attention and context windows before a failure exists.\n- `context-optimization`: masking, caching, partitioning, and other token-efficiency tactics after diagnosis.\n- `context-compression`: structured summaries and handoffs when accumulated context must be compacted.\n- `filesystem-context`: offloading raw outputs, logs, and scratch state so poisoned or bulky context can be inspected without staying in the prompt.\n- `multi-agent-patterns`: isolating tasks into separate contexts to prevent confusion and clash.\n- `evaluation`: degradation tests and production monitoring.\n\n## References\n\nInternal reference:\n- [Degradation Patterns Reference](./references/patterns.md) - Read when: debugging a specific degradation pattern and needing implementation-level detection code (attention analysis, poisoning tracking, relevance scoring, recovery procedures)\n\nRelated skills in this collection:\n- context-fundamentals - Read when: lacking foundational understanding of context windows, token budgets, or placement mechanics\n- context-optimization - Read when: degradation is diagnosed and specific mitigation techniques (compaction, compression, masking) are needed\n- evaluation - Read when: setting up production monitoring to detect degradation before it impacts users\n\nExternal resources:\n- Liu et al., 2023 \"Lost in the Middle\" - Read when: needing primary research backing for U-shaped attention claims or designing position-aware context layouts\n- RULER benchmark documentation - Read when: evaluating model claims about long-context support or comparing models for context-heavy workloads\n- Production engineering guides from AI labs - Read when: implementing context management in production infrastructure\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-20\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.1.0\n","skills/context-degradation/references/patterns.md":"# Context Degradation Patterns: Technical Reference\n\nThis document provides technical details on diagnosing and measuring context degradation.\n\n## Attention Distribution Analysis\n\n### U-Shaped Curve Measurement\n\nMeasure attention distribution across context positions:\n\n```python\ndef measure_attention_distribution(model, context_tokens, query):\n \"\"\"\n Measure how attention varies across context positions.\n \n Returns distribution showing attention weight by position.\n \"\"\"\n attention_by_position = []\n \n for position in range(len(context_tokens)):\n # Measure model's attention to this position\n attention = get_attention_weights(model, context_tokens, query, position)\n attention_by_position.append({\n \"position\": position,\n \"attention\": attention,\n \"is_beginning\": position < len(context_tokens) * 0.1,\n \"is_end\": position > len(context_tokens) * 0.9,\n \"is_middle\": True # Will be overwritten\n })\n \n # Classify positions\n for item in attention_by_position:\n if item[\"is_beginning\"] or item[\"is_end\"]:\n item[\"region\"] = \"attention_favored\"\n else:\n item[\"region\"] = \"attention_degraded\"\n \n return attention_by_position\n```\n\n### Lost-in-Middle Detection\n\nDetect when critical information falls in degraded attention regions:\n\n```python\ndef detect_lost_in_middle(critical_positions, attention_distribution):\n \"\"\"\n Check if critical information is in attention-favored positions.\n \n Args:\n critical_positions: List of positions containing critical info\n attention_distribution: Output from measure_attention_distribution\n \n Returns:\n Dictionary with detection results and recommendations\n \"\"\"\n results = {\n \"at_risk\": [],\n \"safe\": [],\n \"recommendations\": []\n }\n \n for pos in critical_positions:\n region = attention_distribution[pos][\"region\"]\n if region == \"attention_degraded\":\n results[\"at_risk\"].append(pos)\n else:\n results[\"safe\"].append(pos)\n \n # Generate recommendations\n if results[\"at_risk\"]:\n results[\"recommendations\"].extend([\n \"Move critical information to attention-favored positions\",\n \"Use explicit markers to highlight critical information\",\n \"Consider splitting context to reduce middle section\"\n ])\n \n return results\n```\n\n## Context Poisoning Detection\n\n### Hallucination Tracking\n\nTrack potential hallucinations across conversation turns:\n\n```python\nclass HallucinationTracker:\n def __init__(self):\n self.claims = []\n self.verifications = []\n \n def add_claims(self, text):\n \"\"\"Extract claims from text for later verification.\"\"\"\n claims = extract_claims(text)\n self.claims.extend([{\"text\": c, \"verified\": None} for c in claims])\n \n def verify_claims(self, ground_truth):\n \"\"\"Verify claims against ground truth.\"\"\"\n for claim in self.claims:\n if claim[\"verified\"] is None:\n claim[\"verified\"] = check_claim(claim[\"text\"], ground_truth)\n \n def get_poisoning_indicators(self):\n \"\"\"\n Return indicators of potential context poisoning.\n \n High ratio of unverified claims suggests poisoning risk.\n \"\"\"\n unverified = sum(1 for c in self.claims if not c[\"verified\"])\n verified_false = sum(1 for c in self.claims if c[\"verified\"] == False)\n \n return {\n \"unverified_count\": unverified,\n \"false_count\": verified_false,\n \"poisoning_risk\": verified_false > 0 or unverified > len(self.claims) * 0.3\n }\n```\n\n### Error Propagation Analysis\n\nTrack how errors flow through context:\n\n```python\ndef analyze_error_propagation(context, error_points):\n \"\"\"\n Analyze how errors at specific points affect downstream context.\n\n Returns visualization of error spread and impact assessment.\n \"\"\"\n impact_map = {}\n\n for error_point in error_points:\n # Find all references to content after error point\n downstream_refs = find_references(context, after=error_point)\n\n for ref in downstream_refs:\n if ref not in impact_map:\n impact_map[ref] = []\n impact_map[ref].append({\n \"source\": error_point,\n \"type\": classify_error_type(context[error_point])\n })\n\n # Assess severity\n high_impact_areas = [k for k, v in impact_map.items() if len(v) > 3]\n\n return {\n \"impact_map\": impact_map,\n \"high_impact_areas\": high_impact_areas,\n \"requires_intervention\": len(high_impact_areas) > 0\n }\n```\n\n## Distraction Metrics\n\n### Relevance Scoring\n\nScore relevance of context elements to current task:\n\n```python\ndef score_context_relevance(context_elements, task_description):\n \"\"\"\n Score each context element for relevance to current task.\n \n Returns scores and identifies high-distraction elements.\n \"\"\"\n task_embedding = embed(task_description)\n \n scored_elements = []\n for i, element in enumerate(context_elements):\n element_embedding = embed(element)\n relevance = cosine_similarity(task_embedding, element_embedding)\n scored_elements.append({\n \"index\": i,\n \"content_preview\": element[:100],\n \"relevance_score\": relevance\n })\n \n # Sort by relevance\n scored_elements.sort(key=lambda x: x[\"relevance_score\"], reverse=True)\n \n # Identify potential distractors\n threshold = calculate_relevance_threshold(scored_elements)\n distractors = [e for e in scored_elements if e[\"relevance_score\"] < threshold]\n \n return {\n \"scored_elements\": scored_elements,\n \"distractors\": distractors,\n \"recommendation\": f\"Consider removing {len(distractors)} low-relevance elements\"\n }\n```\n\n## Degradation Monitoring System\n\n### Context Health Dashboard\n\nImplement continuous monitoring of context health:\n\n```python\nclass ContextHealthMonitor:\n def __init__(self, model, context_window_limit):\n self.model = model\n self.limit = context_window_limit\n self.metrics = []\n \n def assess_health(self, context, task):\n \"\"\"\n Assess overall context health for current task.\n \n Returns composite score and component metrics.\n \"\"\"\n metrics = {\n \"token_count\": len(context),\n \"utilization_ratio\": len(context) / self.limit,\n \"attention_distribution\": measure_attention_distribution(self.model, context, task),\n \"relevance_scores\": score_context_relevance(context, task),\n \"age_tokens\": count_recent_tokens(context)\n }\n \n # Calculate composite health score\n health_score = self._calculate_composite(metrics)\n \n result = {\n \"health_score\": health_score,\n \"metrics\": metrics,\n \"status\": self._interpret_score(health_score),\n \"recommendations\": self._generate_recommendations(metrics)\n }\n \n self.metrics.append(result)\n return result\n \n def _calculate_composite(self, metrics):\n \"\"\"Calculate composite health score from components.\"\"\"\n # Weighted combination of metrics\n utilization_penalty = min(metrics[\"utilization_ratio\"] * 0.5, 0.3)\n attention_penalty = self._calculate_attention_penalty(metrics[\"attention_distribution\"])\n relevance_penalty = self._calculate_relevance_penalty(metrics[\"relevance_scores\"])\n \n base_score = 1.0\n score = base_score - utilization_penalty - attention_penalty - relevance_penalty\n return max(0, score)\n \n def _interpret_score(self, score):\n \"\"\"Interpret health score and return status.\"\"\"\n if score > 0.8:\n return \"healthy\"\n elif score > 0.6:\n return \"warning\"\n elif score > 0.4:\n return \"degraded\"\n else:\n return \"critical\"\n```\n\n### Alert Thresholds\n\nConfigure appropriate alert thresholds:\n\n```python\nCONTEXT_ALERTS = {\n \"utilization_warning\": 0.7, # 70% of context limit\n \"utilization_critical\": 0.9, # 90% of context limit\n \"attention_degraded_ratio\": 0.3, # 30% in middle region\n \"relevance_threshold\": 0.3, # Below 30% relevance\n \"consecutive_warnings\": 3 # Three warnings triggers alert\n}\n```\n\n## Recovery Procedures\n\n### Context Truncation Strategy\n\nWhen context degrades beyond recovery, truncate strategically:\n\n```python\ndef truncate_context_for_recovery(context, preserved_elements, target_size):\n \"\"\"\n Truncate context while preserving critical elements.\n \n Strategy:\n 1. Preserve system prompt and tool definitions\n 2. Preserve recent conversation turns\n 3. Preserve critical retrieved documents\n 4. Summarize older content if needed\n 5. Truncate from middle if still over target\n \"\"\"\n truncated = []\n \n # Category 1: Critical system elements (preserve always)\n system_elements = extract_system_elements(context)\n truncated.extend(system_elements)\n \n # Category 2: Recent conversation (preserve more)\n recent_turns = extract_recent_turns(context, num_turns=10)\n truncated.extend(recent_turns)\n \n # Category 3: Critical documents (preserve key ones)\n critical_docs = extract_critical_documents(context, preserved_elements)\n truncated.extend(critical_docs)\n \n # Check size and summarize if needed\n while len(truncated) > target_size:\n # Summarize oldest category 3 elements\n truncated = summarize_oldest(truncated, category=\"documents\")\n \n # If still too large, truncate oldest turns\n if len(truncated) > target_size:\n truncated = truncate_oldest_turns(truncated, keep_recent=5)\n \n return truncated\n```\n\n","skills/context-fundamentals/SKILL.md":"---\nname: context-fundamentals\ndescription: \"This skill should be used to explain or reason about the foundational concepts of context engineering: what context is, the anatomy of a context window, how attention mechanics work, the U-shaped attention curve, why context quality matters more than quantity, and the mental models needed to interpret every other context-engineering decision. Use this for conceptual explanation, onboarding, and background reading. Route operational work to the specialized skills: debugging attention failures goes to context-degradation, token-efficiency work goes to context-optimization, conversation summarization goes to context-compression, and project-shape decisions go to project-development.\"\n---\n\n# Context Engineering Fundamentals\n\nContext is the complete state available to a language model at inference time: system instructions, tool definitions, retrieved documents, message history, and tool outputs. Context engineering is the discipline of curating the smallest high-signal token set that maximizes the likelihood of desired outcomes.\n\nThis skill is the conceptual foundation that every other skill in the collection builds on. It explains what context is, how attention mechanics work, why context quality matters more than quantity, and the mental models needed to interpret every other context-engineering decision. It does not own operational work: debugging attention failures belongs to `context-degradation`, token-efficiency tactics belong to `context-optimization`, conversation summarization belongs to `context-compression`, file-based offloading belongs to `filesystem-context`, and project-shape decisions belong to `project-development`.\n\n## When to Activate\n\nActivate this skill when the work is conceptual:\n\n- Explaining what context is and how attention mechanics constrain agent behavior.\n- Onboarding new contributors who need the mental models before diving into operational skills.\n- Reasoning about a context-related design decision from first principles (what does this constraint mean, why does this trade-off exist) before picking a specific tactic.\n- Writing or reviewing documentation that needs to ground operational guidance in the underlying mechanics.\n\nDo not activate this skill for operational work. The specialized skills handle the doing:\n\n- Diagnosing lost-in-middle, context poisoning, or attention failures: `context-degradation`.\n- Reducing token cost via masking, partitioning, prefix caching, budgets: `context-optimization`.\n- Compressing a long session into a handoff summary: `context-compression`.\n- Offloading large tool outputs or maintaining a durable scratchpad: `filesystem-context`.\n- Deciding the shape of an LLM project or pipeline: `project-development`.\n\n## Core Concepts\n\nTreat context as a finite attention budget, not a storage bin. Every token added competes for the model's attention and depletes a budget that cannot be refilled mid-inference. The engineering problem is maximizing utility per token against three constraints: the hard token limit, the softer effective-capacity ceiling, and the U-shaped attention curve that penalizes information placed in the middle of context (claim-context-degradation-lost-middle-ruler).\n\nApply four principles when assembling context:\n\n1. **Informativity over exhaustiveness** — include only what matters for the current decision; design systems that can retrieve additional information on demand.\n2. **Position-aware placement** — place critical constraints at the beginning and end of context because long-context evaluations show middle-position information is less reliably recovered than edge-position information (claim-context-degradation-lost-middle-ruler).\n3. **Progressive disclosure** — load skill names and summaries at startup; load full content only when a skill activates for a specific task.\n4. **Iterative curation** — context engineering is not a one-time prompt-writing exercise but an ongoing discipline applied every time content is passed to the model.\n\n## Detailed Topics\n\n### The Anatomy of Context\n\n**System Prompts**\nOrganize system prompts into distinct sections using XML tags or Markdown headers (background, instructions, tool guidance, output format). System prompts persist throughout the conversation, so place the most critical constraints at the beginning and end where attention is strongest.\n\nCalibrate instruction altitude to balance two failure modes. Too-low altitude hardcodes brittle logic that breaks when conditions shift. Too-high altitude provides vague guidance that fails to give concrete signals for desired behavior. Aim for heuristic-driven instructions: specific enough to guide behavior, flexible enough to generalize — for example, numbered steps with room for judgment at each step.\n\nStart minimal, then add instructions reactively based on observed failure modes rather than preemptively stuffing edge cases. Curate diverse, canonical few-shot examples that portray expected behavior instead of listing every possible scenario.\n\n**Tool Definitions**\nWrite tool descriptions that answer three questions: what the tool does, when to use it, and what it returns. Include usage context, parameter defaults, and error cases — agents cannot disambiguate tools that a human engineer cannot disambiguate either.\n\nKeep the tool set minimal. Consolidate overlapping tools because bloated tool sets create ambiguous decision points and consume disproportionate context after JSON serialization (tool schemas typically inflate 2-3x compared to equivalent plain-text descriptions).\n\n**Retrieved Documents**\nMaintain lightweight identifiers (file paths, stored queries, web links) and load data into context dynamically using just-in-time retrieval. This mirrors human cognition — maintain an index, not a copy. Strong identifiers (e.g., `customer_pricing_rates.json`) let agents locate relevant files even without search tools; weak identifiers (e.g., `data/file1.json`) force unnecessary loads.\n\nWhen chunking large documents, split at natural semantic boundaries (section headers, paragraph breaks) rather than arbitrary character limits that sever mid-concept.\n\n**Message History**\nMessage history serves as the agent's scratchpad memory for tracking progress, maintaining task state, and preserving reasoning across turns. For long-running tasks, it can grow to dominate context usage — monitor and apply compaction before it crowds out active instructions.\n\nCyclically refine history: once a tool has been called deep in the conversation, the raw result rarely needs to remain verbatim. Replace stale tool outputs with compact summaries or references to reduce low-signal bulk.\n\n**Tool Outputs**\nTool outputs often dominate context in agent trajectories (claim-context-optimization-tool-output-dominance). Apply observation masking: replace verbose outputs with compact references once the agent has processed the result. Retain only the most recently relevant file contents; compress or evict older ones.\n\n### Context Windows and Attention Mechanics\n\n**The Attention Budget**\nFor n tokens, the attention mechanism computes n-squared pairwise relationships. As context grows, the model's ability to maintain these relationships degrades — not as a hard cliff but as a performance gradient. Models trained predominantly on shorter sequences have fewer specialized parameters for context-wide dependencies, creating an effective ceiling well below the nominal window size.\n\nDesign for this gradient: assume effective capacity is materially below the advertised window until measured on the target workload. Large nominal context windows do not remove the need for task-specific degradation tests (claim-context-degradation-lost-middle-ruler).\n\n**Position Encoding Limits**\nPosition encoding interpolation extends sequence handling beyond training lengths but introduces degradation in positional precision. Expect reduced accuracy for information retrieval and long-range reasoning at extended contexts compared to performance on shorter inputs.\n\n**Progressive Disclosure in Practice**\nImplement progressive disclosure at three levels:\n\n1. **Skill selection** — load only names and descriptions at startup; activate full skill content on demand.\n2. **Document loading** — load summaries first; fetch detail sections only when the task requires them.\n3. **Tool result retention** — keep recent results in full; compress or evict older results.\n\nKeep the boundary crisp: if a skill or document is activated, load it fully rather than partially — partial loads create confusing gaps that degrade reasoning quality.\n\n### Context Quality Versus Quantity\n\nReject the assumption that larger context windows solve memory problems. Processing cost grows disproportionately with context length — not just linear cost scaling, but degraded model performance beyond effective capacity thresholds. Long inputs remain expensive even with prefix caching.\n\nApply the signal-density test: for each piece of context, ask whether removing it would change the model's output. If not, remove it. Redundant content does not merely waste tokens — it actively dilutes attention from high-signal content.\n\n## Practical Guidance\n\nThis section provides conceptual application advice. Pointers to operational skills are explicit.\n\n### Reasoning About a Context Decision\n\nWhen a context-related design decision needs to be made, separate the conceptual question from the operational one. The conceptual question is \"what does this mean and why does it matter\"; the operational question is \"what specific technique do we apply.\" Use this skill to answer the first; route to the specialized skill that owns the second.\n\nFor example, deciding whether to summarize a long agent session has two parts: (1) why summarization is needed at all (attention budget is finite, U-shaped curve degrades middle content, signal density matters more than volume - this skill) and (2) what compression strategy preserves the right state and at what utilization threshold to trigger it (`context-compression`).\n\n### Reading Order For New Contributors\n\nA contributor coming to context engineering for the first time should read:\n\n1. This skill, to internalize the attention-budget framing and the U-shaped curve.\n2. `context-degradation`, to see what context failures look like in practice and how to diagnose them.\n3. Two or three of `context-optimization`, `context-compression`, `filesystem-context`, `memory-systems` depending on which operational concern is most relevant to their project.\n\nSkipping step 1 produces operators who apply techniques without understanding why; skipping the operational skills produces theorists who do not know which technique fits which failure mode.\n\n## Examples\n\n**Example 1: Organizing System Prompts**\n\nIllustrates the conceptual point that critical constraints belong at attention-favored positions (beginning and end), and that explicit section boundaries help the model parse the prompt:\n\n```markdown\n\nYou are a Python expert helping a development team.\nCurrent project: Data processing pipeline in Python 3.9+\n\n\n\n- Write clean, idiomatic Python code\n- Include type hints for function signatures\n- Add docstrings for public functions\n- Follow PEP 8 style guidelines\n\n\n\nProvide code blocks with syntax highlighting.\nExplain non-obvious decisions in comments.\n\n```\n\n**Example 2: The Attention Budget As A Mental Model**\n\nA large-context model does not have an equally attended context. Effective capacity is workload-specific, and the U-shaped curve penalizes information placed in the middle. When deciding how much of an upstream knowledge base to load, this is the mental model: do not ask \"will it fit,\" ask \"will the model still attend to the parts that matter.\"\n\nThe corresponding operational question (which technique should reduce the load) belongs to `context-optimization`.\n\n## Guidelines\n\n1. Treat context as a finite resource with diminishing returns\n2. Place critical information at attention-favored positions (beginning and end)\n3. Use progressive disclosure to defer loading until needed\n4. Organize system prompts with clear section boundaries\n5. Monitor context usage during development\n6. Implement compaction triggers at 70-80% utilization\n7. Design for context degradation rather than hoping to avoid it\n8. Prefer smaller high-signal context over larger low-signal context\n\n## Gotchas\n\n1. **Nominal window is not effective capacity**: A model advertising a large context window may degrade well before that limit on complex retrieval or reasoning tasks. Budget below the nominal window until your own degradation tests prove otherwise.\n\n2. **Character-based token estimates silently drift**: The ~4 characters/token heuristic for English prose breaks down for code (2-3 chars/token), URLs and file paths (each slash, dot, and colon is a separate token), and non-English text (often 1-2 chars/token). Use the provider's actual tokenizer (e.g., tiktoken for OpenAI models, Anthropic's token counting API) for any budget-critical calculation.\n\n3. **Tool schemas inflate 2-3x after JSON serialization**: A tool definition that looks compact in source code expands significantly when serialized — brackets, quotes, colons, and commas each consume tokens. Ten tools with moderate schemas can consume 5,000-8,000 tokens before a single message is sent. Audit serialized tool token counts, not source-code line counts.\n\n4. **Message history balloons silently in agentic loops**: Each tool call adds both the request and the full response to history. After 20-30 iterations, history can consume 70-80% of the window while the agent shows no visible symptoms until reasoning quality collapses. Set a hard token ceiling on history and trigger compaction proactively.\n\n5. **Critical instructions in the middle get lost**: The U-shaped attention curve means the middle of context receives 10-40% less recall accuracy than the beginning and end. Never place safety constraints, output format requirements, or behavioral guardrails in the middle of a long system prompt — anchor them at the top or bottom.\n\n6. **Progressive disclosure that loads too eagerly defeats its purpose**: Loading every \"potentially relevant\" skill or document at the first hint of relevance recreates the context-stuffing problem. Set strict activation thresholds — a skill should load only when the task explicitly matches its trigger conditions, not when the topic is merely adjacent.\n\n7. **Mixing instruction altitudes causes inconsistent behavior**: Combining hyper-specific rules (\"always use exactly 3 bullet points\") with vague directives (\"be helpful\") in the same prompt creates conflicting signals. Group instructions by altitude level and keep each section internally consistent — either heuristic-driven or prescriptive, not both interleaved.\n\n## Integration\n\nThis skill is the conceptual foundation. It does not own operational work; it provides the mental models the operational skills assume.\n\nRouting map for operational work:\n\n- `context-degradation`: diagnosing attention failures, lost-in-middle, poisoning, distraction.\n- `context-optimization`: token-efficiency tactics (masking, partitioning, caching, budgets).\n- `context-compression`: compacting long sessions while preserving decisions, files, risks.\n- `filesystem-context`: offloading large outputs and using files as a durable scratchpad.\n- `memory-systems`: cross-session memory architectures with entity tracking.\n- `multi-agent-patterns`: when to split work across agents for context isolation.\n- `tool-design`: writing tool descriptions and schemas that route correctly.\n- `project-development`: deciding LLM fit and shaping multi-stage pipelines.\n\nRead this skill first to build the mental models; read the operational skill that fits the task when actually doing the work.\n\n## References\n\nInternal reference:\n- [Context Components Reference](./references/context-components.md) - Read when: debugging a specific context component (system prompts, tool definitions, message history, tool outputs) or implementing chunking, observation masking, or budget allocation tables\n\nRelated skills in this collection:\n- context-degradation - Read when: agent performance drops as conversations grow or context fills beyond 60% capacity\n- context-optimization - Read when: token costs are too high or compaction/compression strategies are needed\n\nExternal resources:\n- Anthropic's \"Effective Context Engineering for AI Agents\" — production patterns for compaction, sub-agents, and hybrid retrieval\n- Research on transformer attention mechanisms and the lost-in-the-middle effect\n- Tokenomics research on agentic software engineering token distribution\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-20\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.2.0\n","skills/context-fundamentals/references/context-components.md":"# Context Components: Technical Reference\n\nThis document provides detailed technical reference for each context component in agent systems.\n\n## System Prompt Engineering\n\n### Section Structure\n\nOrganize system prompts into distinct sections with clear boundaries. A recommended structure:\n\n```\n\nContext about the domain, user preferences, or project-specific details\n\n\n\nCore behavioral guidelines and task instructions\n\n\n\nWhen and how to use available tools\n\n\n\nExpected output format and quality standards\n\n```\n\nThis structure allows agents to locate relevant information quickly and enables selective context loading in advanced implementations.\n\n### Altitude Calibration\n\nThe \"altitude\" of instructions refers to the level of abstraction. Consider these examples:\n\n**Too Low (Brittle):**\n```\nIf the user asks about pricing, check the pricing table in docs/pricing.md.\nIf the table shows USD, convert to EUR using the exchange rate in\nconfig/exchange_rates.json. If the user is in the EU, add VAT at the\napplicable rate from config/vat_rates.json. Format the response with\nthe currency symbol, two decimal places, and a note about VAT.\n```\n\n**Too High (Vague):**\n```\nHelp users with pricing questions. Be helpful and accurate.\n```\n\n**Optimal (Heuristic-Driven):**\n```\nFor pricing inquiries:\n1. Retrieve current rates from docs/pricing.md\n2. Apply user location adjustments (see config/location_defaults.json)\n3. Format with appropriate currency and tax considerations\n\nPrefer exact figures over estimates. When rates are unavailable,\nsay so explicitly rather than projecting.\n```\n\nThe optimal altitude provides clear steps while allowing flexibility in execution.\n\n## Tool Definition Specification\n\n### Schema Structure\n\nEach tool should define:\n\n```python\n{\n \"name\": \"tool_function_name\",\n \"description\": \"Clear description of what the tool does and when to use it\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"param_name\": {\n \"type\": \"string\",\n \"description\": \"What this parameter controls\",\n \"default\": \"reasonable_default_value\"\n }\n },\n \"required\": [\"param_name\"]\n },\n \"returns\": {\n \"type\": \"object\",\n \"description\": \"What the tool returns and its structure\"\n }\n}\n```\n\n### Description Engineering\n\nTool descriptions should answer: what the tool does, when to use it, and what it produces. Include usage context, examples, and edge cases.\n\n**Weak Description:**\n```\nSearch the database for customer information.\n```\n\n**Strong Description:**\n```\nRetrieve customer information by ID or email.\n\nUse when:\n- User asks about a specific customer's details, history, or status\n- User provides a customer identifier and needs related information\n\nReturns customer object with:\n- Basic info (name, email, account status)\n- Order history summary\n- Support ticket count\n\nReturns null if customer not found. Returns error if database unreachable.\n```\n\n## Retrieved Document Management\n\n### Identifier Design\n\nDesign identifiers that convey meaning and enable efficient retrieval:\n\n**Poor identifiers:**\n- `data/file1.json`\n- `ref/ref.md`\n- `2024/q3/report`\n\n**Strong identifiers:**\n- `customer_pricing_rates.json`\n- `engineering_onboarding_checklist.md`\n- `2024_q3_revenue_report.pdf`\n\nStrong identifiers allow agents to locate relevant files even without search tools.\n\n### Document Chunking Strategy\n\nFor large documents, chunk strategically to preserve semantic coherence:\n\n```python\n# Pseudocode for semantic chunking\ndef chunk_document(content):\n \"\"\"Split document at natural semantic boundaries.\"\"\"\n boundaries = find_section_headers(content)\n boundaries += find_paragraph_breaks(content)\n boundaries += find_logical_breaks(content)\n \n chunks = []\n for i in range(len(boundaries) - 1):\n chunk = content[boundaries[i]:boundaries[i+1]]\n if len(chunk) > MIN_CHUNK_SIZE and len(chunk) < MAX_CHUNK_SIZE:\n chunks.append(chunk)\n \n return chunks\n```\n\nAvoid arbitrary character limits that split mid-sentence or mid-concept.\n\n## Message History Management\n\n### Turn Representation\n\nStructure message history to preserve key information:\n\n```python\n{\n \"role\": \"user\" | \"assistant\" | \"tool\",\n \"content\": \"message text\",\n \"reasoning\": \"optional chain-of-thought\",\n \"tool_calls\": [list if role=\"assistant\"],\n \"tool_output\": \"output if role=\"tool\"\",\n \"summary\": \"compact summary if conversation is long\"\n}\n```\n\n### Summary Injection Pattern\n\nFor long conversations, inject summaries at intervals:\n\n```python\ndef inject_summaries(messages, summary_interval=20):\n \"\"\"Inject summaries at regular intervals to preserve context.\"\"\"\n summarized = []\n for i, msg in enumerate(messages):\n summarized.append(msg)\n if i > 0 and i % summary_interval == 0:\n summary = generate_summary(summarized[-summary_interval:])\n summarized.append({\n \"role\": \"system\",\n \"content\": f\"Conversation summary: {summary}\",\n \"is_summary\": True\n })\n return summarized\n```\n\n## Tool Output Optimization\n\n### Response Formats\n\nProvide response format options to control token usage:\n\n```python\ndef get_customer_response_format():\n return {\n \"format\": \"concise | detailed\",\n \"fields\": [\"id\", \"name\", \"email\", \"status\", \"history_summary\"]\n }\n```\n\nThe concise format returns essential fields only; detailed returns complete objects.\n\n### Observation Masking\n\nFor verbose tool outputs, consider masking patterns:\n\n```python\ndef mask_observation(output, max_length=500):\n \"\"\"Replace long observations with compact references.\"\"\"\n if len(output) <= max_length:\n return output\n \n reference_id = store_observation(output)\n return f\"[Previous observation elided. Full content stored at reference {reference_id}]\"\n```\n\nThis preserves information access while reducing token usage.\n\n## Context Budget Estimation\n\n### Token Counting Approximation\n\nFor planning purposes, estimate tokens at approximately 4 characters per token for English text:\n\n```\n1000 words ≈ 7500 characters ≈ 1800-2000 tokens\n```\n\nThis is a rough approximation; actual tokenization varies by model and content type.\n\n### Context Budget Allocation\n\nAllocate context budget across components:\n\n| Component | Typical Range | Notes |\n|-----------|---------------|-------|\n| System prompt | 500-2000 tokens | Stable across session |\n| Tool definitions | 100-500 per tool | Grows with tool count |\n| Retrieved documents | Variable | Often largest consumer |\n| Message history | Variable | Grows with conversation |\n| Tool outputs | Variable | Can dominate context |\n\nMonitor actual usage during development to establish baseline allocations.\n\n## Progressive Disclosure Implementation\n\n### Skill Activation Pattern\n\n```python\ndef activate_skill_context(skill_name, task_description):\n \"\"\"Load skill context when task matches skill description.\"\"\"\n skill_metadata = load_all_skill_metadata()\n \n relevant_skills = []\n for skill in skill_metadata:\n if skill_matches_task(skill, task_description):\n relevant_skills.append(skill)\n \n # Load full content only for most relevant skills\n for skill in relevant_skills[:MAX_CONCURRENT_SKILLS]:\n skill_context = load_skill_content(skill)\n inject_into_context(skill_context)\n```\n\n### Reference Loading Pattern\n\n```python\ndef get_reference(file_reference):\n \"\"\"Load reference file only when explicitly needed.\"\"\"\n if not file_reference.is_loaded:\n file_reference.content = read_file(file_reference.path)\n file_reference.is_loaded = True\n return file_reference.content\n```\n\nThis pattern ensures files are loaded once and cached for the session.\n\n","skills/context-optimization/SKILL.md":"---\nname: context-optimization\ndescription: \"This skill should be used for improving context efficiency: context budgeting, observation masking, prefix or KV-cache strategy, partitioning, token-cost reduction, retrieval scoping, and extending effective context capacity without lowering answer quality.\"\n---\n\n# Context Optimization Techniques\n\nContext optimization extends the effective capacity of limited context windows through strategic compression, masking, caching, and partitioning. Effective optimization increases useful capacity without requiring larger models or longer windows — but only when applied with measurement discipline. The techniques below are ordered by impact and risk.\n\n## When to Activate\n\nActivate this skill when:\n- Context budgets or token costs constrain task complexity\n- Observation masking can replace verbose tool outputs with retrievable references\n- Prefix or KV-cache hit rate needs improvement\n- Retrieval scoping can reduce irrelevant loaded context\n- Context partitioning can extend effective capacity across agents\n- Budget triggers are needed for masking, compaction, or partitioning\n\nDo not activate this skill for adjacent work owned by other skills:\n- Explaining why attention or context windows behave this way: `context-fundamentals`.\n- Diagnosing active lost-in-middle, poisoning, distraction, confusion, or clash: `context-degradation`.\n- Designing a structured handoff summary for a long conversation: `context-compression`.\n- Storing large outputs, plans, or logs as files: `filesystem-context`.\n- Model-initiated rewrites of its own context and their re-prefill cost: `self-managed-context`.\n\n## Core Concepts\n\nApply four primary strategies in this priority order:\n\n1. **KV-cache optimization** — Reorder and stabilize prompt structure so the inference engine reuses cached Key/Value tensors. This is the cheapest optimization when the runtime supports prefix caching: low quality risk, immediate cost and latency savings. Apply it first when stable prefixes exist.\n\n2. **Observation masking** — Replace verbose tool outputs with compact references once their purpose has been served. Tool outputs can dominate agent trajectories (claim-context-optimization-tool-output-dominance), so masking often yields the largest capacity gains. The original content remains retrievable if needed downstream.\n\n3. **Compaction** — Summarize accumulated context when utilization exceeds 70%, then reinitialize with the summary. This distills the window's contents while preserving task-critical state. Compaction is lossy — apply it after masking has already removed the low-value bulk.\n\n4. **Context partitioning** — Split work across sub-agents with isolated contexts when a single window cannot hold the full problem. Each sub-agent operates in a clean context focused on its subtask. Reserve this for tasks where estimated context exceeds 60% of the window limit, because coordination overhead is real.\n\nThe governing principle: context quality matters more than quantity. Every optimization preserves signal while reducing noise. Measure before optimizing, then measure the optimization's effect.\n\n## Detailed Topics\n\n### Compaction Strategies\n\nTrigger compaction when context utilization exceeds 70%: summarize the current context, then reinitialize with the summary. This distills the window's contents in a high-fidelity manner, enabling continuation with minimal performance degradation. Prioritize compressing tool outputs first (they consume 80%+ of tokens), then old conversation turns, then retrieved documents. Never compress the system prompt — it anchors model behavior and its removal causes unpredictable degradation.\n\nPreserve different elements by message type:\n\n- **Tool outputs**: Extract key findings, metrics, error codes, and conclusions. Strip verbose raw output, stack traces (unless debugging is ongoing), and boilerplate headers.\n- **Conversational turns**: Retain decisions, commitments, user preferences, and context shifts. Remove filler, pleasantries, and exploratory back-and-forth that led to a conclusion already captured.\n- **Retrieved documents**: Keep claims, facts, and data points relevant to the active task. Remove supporting evidence and elaboration that served a one-time reasoning purpose.\n\nTarget 50-70% token reduction with less than 5% quality degradation. If compaction exceeds 70% reduction, audit the summary for critical information loss — over-aggressive compaction is the most common failure mode.\n\n### Observation Masking\n\nMask observations selectively based on recency and ongoing relevance — not uniformly. Apply these rules:\n\n- **Never mask**: Observations critical to the current task, observations from the most recent turn, observations used in active reasoning chains, and error outputs when debugging is in progress.\n- **Mask after 3+ turns**: Verbose outputs whose key points have already been extracted into the conversation flow. Replace with a compact reference: `[Obs:{ref_id} elided. Key: {summary}. Full content retrievable.]`\n- **Always mask immediately**: Repeated/duplicate outputs, boilerplate headers and footers, outputs already summarized earlier in the conversation.\n\nMasking should achieve 60-80% reduction in masked observations with less than 2% quality impact. The key is maintaining retrievability — store the full content externally and keep the reference ID in context so the agent can request the original if needed.\n\n### KV-Cache Optimization\n\nMaximize prefix cache hits by structuring prompts so that stable content occupies the prefix and dynamic content appears at the end. KV-cache stores Key and Value tensors computed during inference; when consecutive requests share an identical prefix, the cached tensors are reused, saving both cost and latency.\n\nApply this ordering in every prompt:\n1. System prompt (most stable — never changes within a session)\n2. Tool definitions (stable across requests)\n3. Frequently reused templates and few-shot examples\n4. Conversation history (grows but shares prefix with prior turns)\n5. Current query and dynamic content (least stable — always last)\n\nDesign prompts for cache stability: remove timestamps, session counters, and request IDs from the system prompt. Move dynamic metadata into a separate user message or tool result where it does not break the prefix. Even a single whitespace change in the prefix invalidates the entire cached block downstream of that change. The same rule prices any compaction or masking edit applied to earlier history: its cost is the text that follows it, so batch such edits and prefer them near the end of the context.\n\nTarget 70%+ cache hit rate for stable workloads. At scale, this translates to 50%+ cost reduction and 40%+ latency reduction on cached tokens.\n\n### Context Partitioning\n\nPartition work across sub-agents when a single context cannot hold the full problem without triggering aggressive compaction. Each sub-agent operates in a clean, focused context for its subtask, then returns a structured result to a coordinator agent.\n\nPlan partitioning when estimated task context exceeds 60% of the window limit. Decompose the task into independent subtasks, assign each to a sub-agent, and aggregate results. Validate that all partitions completed before merging, merge compatible results, and apply summarization if the aggregated output still exceeds budget.\n\nThis approach achieves separation of concerns — detailed search context stays isolated within sub-agents while the coordinator focuses on synthesis. However, coordination has real token cost: the coordinator prompt, result aggregation, and error handling all consume tokens. Only partition when the savings exceed this overhead.\n\n### Budget Management\n\nAllocate explicit token budgets across context categories before the session begins: system prompt, tool definitions, retrieved documents, message history, tool outputs, and a reserved buffer (5-10% of total). Monitor usage against budget continuously and trigger optimization when any category exceeds its allocation or total utilization crosses 70%.\n\nUse trigger-based optimization rather than periodic optimization. Monitor these signals:\n- Token utilization above 80% — trigger compaction\n- Attention degradation indicators (repetition, missed instructions) — trigger masking + compaction\n- Quality score drops below baseline — audit context composition before optimizing\n\n## Practical Guidance\n\n### Optimization Decision Framework\n\nSelect the optimization technique based on what dominates the context:\n\n| Context Composition | First Action | Second Action |\n|---|---|---|\n| Tool outputs dominate (>50%) | Observation masking | Compaction of remaining turns |\n| Retrieved documents dominate | Summarization | Partitioning if docs are independent |\n| Message history dominates | Compaction with selective preservation | Partitioning for new subtasks |\n| Multiple components contribute | KV-cache optimization first, then layer masking + compaction |\n| Near-limit with active debugging | Mask resolved tool outputs only — preserve error details |\n\n### Performance Targets\n\nTrack these metrics to validate optimization effectiveness:\n\n- **Compaction**: 50-70% token reduction, <5% quality degradation, <10% latency overhead from the compaction step itself\n- **Masking**: 60-80% reduction in masked observations, <2% quality impact, near-zero latency overhead\n- **Cache optimization**: 70%+ hit rate for stable workloads, 50%+ cost reduction, 40%+ latency reduction\n- **Partitioning**: Net token savings after accounting for coordinator overhead; break-even typically requires 3+ subtasks\n\nIterate on strategies based on measured results. If an optimization technique does not measurably improve the target metric, remove it — optimization machinery itself consumes tokens and adds latency.\n\n## Examples\n\n**Example 1: Compaction Trigger**\n```python\nif context_tokens / context_limit > 0.8:\n context = compact_context(context)\n```\n\n**Example 2: Observation Masking**\n```python\nif len(observation) > max_length:\n ref_id = store_observation(observation)\n return f\"[Obs:{ref_id} elided. Key: {extract_key(observation)}]\"\n```\n\n**Example 3: Cache-Friendly Ordering**\n```python\n# Stable content first\ncontext = [system_prompt, tool_definitions] # Cacheable\ncontext += [reused_templates] # Reusable\ncontext += [unique_content] # Unique\n```\n\n**Example 4: Budget-triggered optimization policy**\n```yaml\nbudgets:\n tool_outputs: 35%\n message_history: 30%\n retrieved_documents: 20%\n reserved_buffer: 15%\ntriggers:\n tool_outputs_over_budget: mask resolved observations\n total_context_over_70_percent: compact message history\n repeated_irrelevant_retrievals: tighten retrieval scope\n```\n\n## Guidelines\n\n1. Measure before optimizing—know your current state\n2. Apply masking before compaction — remove low-value bulk first, then summarize what remains\n3. Design for cache stability with consistent prompts\n4. Partition before context becomes problematic\n5. Monitor optimization effectiveness over time\n6. Balance token savings against quality preservation\n7. Test optimization at production scale\n8. Implement graceful degradation for edge cases\n\n## Gotchas\n\n1. **Whitespace breaks KV-cache**: Even a single whitespace or newline change in the prompt prefix invalidates the entire KV-cache block downstream of that point. Pin system prompts as immutable strings — do not interpolate timestamps, version numbers, or session IDs into them. Diff prompt templates byte-for-byte between deployments.\n\n2. **Timestamps in system prompts destroy cache hit rates**: Including `Current date: {today}` or similar dynamic content in the system prompt forces a full cache miss on every new day (or every request, if using time-of-day). Move dynamic metadata into a user message or a separate tool result appended after the stable prefix.\n\n3. **Compaction under pressure loses critical state**: When the model performing compaction is itself under context pressure (>85% utilization), its summarization quality degrades — it omits task goals, drops user constraints, and flattens nuanced state. Trigger compaction at 70-80%, not 90%+. If compaction must happen late, use a separate model call with a clean context containing only the material to summarize.\n\n4. **Masking error outputs breaks debugging loops**: Over-aggressive masking hides error messages, stack traces, and failure details that the agent needs in subsequent turns to diagnose and fix issues. During active debugging (error in the last 3 turns), suspend masking for all error-related observations until the issue is resolved.\n\n5. **Partitioning overhead can exceed savings**: Each sub-agent requires its own system prompt, tool definitions, and coordination messages. For tasks with fewer than 3 independent subtasks, the coordination overhead often exceeds the context savings. Estimate total tokens (coordinator + all sub-agents) before committing to partitioning.\n\n6. **Cache miss cost spikes after deployment changes**: Reordering tools, rewording the system prompt, or changing few-shot examples between deployments invalidates the entire prefix cache, causing a temporary cost spike of 2-5x until the new cache warms up. Roll out prompt changes gradually and monitor cache hit rate during deployment windows.\n\n7. **Compaction creates false confidence in stale summaries**: Once context is compacted, the summary looks authoritative but may reflect outdated state. If the task has evolved since compaction (new user requirements, corrected assumptions), the summary silently carries forward stale information. After compaction, re-validate the summary against the current task goal before proceeding.\n\n## Integration\n\nThis skill owns token-efficiency tactics and budget policy. Adjacent skills own diagnosis, storage, and architecture:\n\n- `context-fundamentals`: mental models for why context quality and attention placement matter.\n- `context-degradation`: diagnosis when output quality has already dropped.\n- `context-compression`: lossy summarization and handoff strategy.\n- `filesystem-context`: file-backed offloading for full outputs and logs.\n- `multi-agent-patterns`: partitioning work across isolated agent contexts.\n- `latent-briefing`: selective KV retention across orchestrator-worker boundaries in compatible runtimes.\n- `self-managed-context`: the model, not the harness, decides when and what to edit in its live context.\n- `evaluation`: measuring whether the optimization improved quality, cost, or latency.\n- `memory-systems`: persistent retrieval layers that feed context just in time.\n\n## References\n\nInternal reference:\n- [Optimization Techniques Reference](./references/optimization_techniques.md) - Read when: implementing a specific optimization technique and needing detailed code patterns, threshold tables, or integration examples beyond what the skill body provides\n\nRelated skills in this collection:\n- context-fundamentals - Read when: unfamiliar with context window mechanics, token counting, or attention distribution basics\n- context-degradation - Read when: diagnosing why agent performance has dropped and needing to identify which degradation pattern is occurring before selecting an optimization\n- evaluation - Read when: setting up metrics and benchmarks to measure whether an optimization technique actually improved outcomes\n\nExternal resources:\n- Research on context window limitations - Read when: evaluating model-specific context behavior (e.g., lost-in-the-middle effects, attention decay curves)\n- KV-cache optimization techniques - Read when: implementing prefix caching at the inference infrastructure level (vLLM, TGI, or cloud provider APIs)\n- Production engineering guides - Read when: deploying context optimization in a production pipeline and needing operability patterns (monitoring, alerting, rollback)\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-20\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.1.0\n","skills/context-optimization/references/optimization_techniques.md":"# Context Optimization Reference\n\nThis document provides detailed technical reference for context optimization techniques and strategies.\n\n## Compaction Strategies\n\n### Summary-Based Compaction\n\nSummary-based compaction replaces verbose content with concise summaries while preserving key information. The approach works by identifying sections that can be compressed, generating summaries that capture essential points, and replacing full content with summaries.\n\nThe effectiveness of compaction depends on what information is preserved. Critical decisions, user preferences, and current task state should never be compacted. Intermediate results and supporting evidence can be summarized more aggressively. Boilerplate, repeated information, and exploratory reasoning can often be removed entirely.\n\n### Token Budget Allocation\n\nEffective context budgeting requires understanding how different context components consume tokens and allocating budget strategically:\n\n| Component | Typical Range | Notes |\n|-----------|---------------|-------|\n| System prompt | 500-2000 tokens | Stable across session |\n| Tool definitions | 100-500 per tool | Grows with tool count |\n| Retrieved documents | Variable | Often largest consumer |\n| Message history | Variable | Grows with conversation |\n| Tool outputs | Variable | Can dominate context |\n\n### Compaction Thresholds\n\nTrigger compaction at appropriate thresholds to maintain performance:\n\n- Warning threshold at 70% of effective context limit\n- Compaction trigger at 80% of effective context limit\n- Aggressive compaction at 90% of effective context limit\n\nThe exact thresholds depend on model behavior and task characteristics. Some models show graceful degradation while others exhibit sharp performance cliffs.\n\n## Observation Masking Patterns\n\n### Selective Masking\n\nNot all observations should be masked equally. Consider masking observations that have served their purpose and are no longer needed for active reasoning. Keep observations that are central to the current task. Keep observations from the most recent turn. Keep observations that may be referenced again.\n\n### Masking Implementation\n\n```python\ndef selective_mask(observations: List[Dict], current_task: Dict) -> List[Dict]:\n \"\"\"\n Selectively mask observations based on relevance.\n \n Returns observations with mask field indicating masked content.\n \"\"\"\n masked = []\n \n for obs in observations:\n relevance = calculate_relevance(obs, current_task)\n \n if relevance < 0.3 and obs[\"age\"] > 3:\n # Low relevance and old - mask\n masked.append({\n **obs,\n \"masked\": True,\n \"reference\": store_for_reference(obs[\"content\"]),\n \"summary\": summarize_content(obs[\"content\"])\n })\n else:\n masked.append({\n **obs,\n \"masked\": False\n })\n \n return masked\n```\n\n## KV-Cache Optimization\n\n### Prefix Stability\n\nKV-cache hit rates depend on prefix stability. Stable prefixes enable cache reuse across requests. Dynamic prefixes invalidate cache and force recomputation.\n\nElements that should remain stable include system prompts, tool definitions, and frequently used templates. Elements that may vary include timestamps, session identifiers, and query-specific content.\n\n### Cache-Friendly Design\n\nDesign prompts to maximize cache hit rates:\n\n1. Place stable content at the beginning\n2. Use consistent formatting across requests\n3. Avoid dynamic content in prompts when possible\n4. Use placeholders for dynamic content\n\n```python\n# Cache-unfriendly: Dynamic timestamp in prompt\nsystem_prompt = f\"\"\"\nCurrent time: {datetime.now().isoformat()}\nYou are a helpful assistant.\n\"\"\"\n\n# Cache-friendly: Stable prompt with dynamic time as variable\nsystem_prompt = \"\"\"\nYou are a helpful assistant.\nCurrent time is provided separately when relevant.\n\"\"\"\n```\n\n## Context Partitioning Strategies\n\n### Sub-Agent Isolation\n\nPartition work across sub-agents to prevent any single context from growing too large. Each sub-agent operates with a clean context focused on its subtask.\n\n### Partition Planning\n\n```python\ndef plan_partitioning(task: Dict, context_limit: int) -> Dict:\n \"\"\"\n Plan how to partition a task based on context limits.\n \n Returns partitioning strategy and subtask definitions.\n \"\"\"\n estimated_context = estimate_task_context(task)\n \n if estimated_context <= context_limit:\n return {\n \"strategy\": \"single_agent\",\n \"subtasks\": [task]\n }\n \n # Plan multi-agent approach\n subtasks = decompose_task(task)\n \n return {\n \"strategy\": \"multi_agent\",\n \"subtasks\": subtasks,\n \"coordination\": \"hierarchical\"\n }\n```\n\n## Optimization Decision Framework\n\n### When to Optimize\n\nConsider context optimization when context utilization exceeds 70%, when response quality degrades as conversations extend, when costs increase due to long contexts, or when latency increases with conversation length.\n\n### What Optimization to Apply\n\nChoose optimization strategies based on context composition:\n\nIf tool outputs dominate context, apply observation masking. If retrieved documents dominate context, apply summarization or partitioning. If message history dominates context, apply compaction with summarization. If multiple components contribute, combine strategies.\n\n### Evaluation of Optimization\n\nAfter applying optimization, evaluate effectiveness:\n\n- Measure token reduction achieved\n- Measure quality preservation (output quality should not degrade)\n- Measure latency improvement\n- Measure cost reduction\n\nIterate on optimization strategies based on evaluation results.\n\n## Common Pitfalls\n\n### Over-Aggressive Compaction\n\nCompacting too aggressively can remove critical information. Always preserve task goals, user preferences, and recent conversation context. Test compaction at increasing aggressiveness levels to find the optimal balance.\n\n### Masking Critical Observations\n\nMasking observations that are still needed can cause errors. Track observation usage and only mask content that is no longer referenced. Consider keeping references to masked content that could be retrieved if needed.\n\n### Ignoring Attention Distribution\n\nThe lost-in-middle phenomenon means that information placement matters. Place critical information at attention-favored positions (beginning and end of context). Use explicit markers to highlight important content.\n\n### Premature Optimization\n\nNot all contexts require optimization. Adding optimization machinery has overhead. Optimize only when context limits actually constrain agent performance.\n\n## Monitoring and Alerting\n\n### Key Metrics\n\nTrack these metrics to understand optimization needs:\n\n- Context token count over time\n- Cache hit rates for repeated patterns\n- Response quality metrics by context size\n- Cost per conversation by context length\n- Latency by context size\n\n### Alert Thresholds\n\nSet alerts for:\n\n- Context utilization above 80%\n- Cache hit rate below 50%\n- Quality score drop of more than 10%\n- Cost increase above baseline\n\n## Integration Patterns\n\n### Integration with Agent Framework\n\nIntegrate optimization into agent workflow:\n\n```python\nclass OptimizingAgent:\n def __init__(self, context_limit: int = 80000):\n self.context_limit = context_limit\n self.optimizer = ContextOptimizer()\n \n def process(self, user_input: str, context: Dict) -> Dict:\n # Check if optimization needed\n if self.optimizer.should_compact(context):\n context = self.optimizer.compact(context)\n \n # Process with optimized context\n response = self._call_model(user_input, context)\n \n # Track metrics\n self.optimizer.record_metrics(context, response)\n \n return response\n```\n\n### Integration with Memory Systems\n\nConnect optimization with memory systems:\n\n```python\nclass MemoryAwareOptimizer:\n def __init__(self, memory_system, context_limit: int):\n self.memory = memory_system\n self.limit = context_limit\n \n def optimize_context(self, current_context: Dict, task: str) -> Dict:\n # Check if information is in memory\n relevant_memories = self.memory.retrieve(task)\n \n # Move information to memory if not needed in context\n for mem in relevant_memories:\n if mem[\"importance\"] < threshold:\n current_context = remove_from_context(current_context, mem)\n # Keep reference that memory can be retrieved\n \n return current_context\n```\n\n## Performance Benchmarks\n\n### Compaction Performance\n\nCompaction should reduce token count while preserving quality. Target:\n\n- 50-70% token reduction for aggressive compaction\n- Less than 5% quality degradation from compaction\n- Less than 10% latency increase from compaction overhead\n\n### Masking Performance\n\nObservation masking should reduce token count significantly:\n\n- 60-80% reduction in masked observations\n- Less than 2% quality impact from masking\n- Near-zero latency overhead\n\n### Cache Performance\n\nKV-cache optimization should improve cost and latency:\n\n- 70%+ cache hit rate for stable workloads\n- 50%+ cost reduction from cache hits\n- 40%+ latency reduction from cache hits\n\n","skills/evaluation/SKILL.md":"---\nname: evaluation\ndescription: \"This skill should be used when building agent evaluation systems: deterministic checks, regression suites, multi-dimensional rubrics, quality gates, production monitoring, baseline comparison, and outcome measurement for agent pipelines.\"\n---\n\n# Evaluation Methods for Agent Systems\n\nEvaluate agent systems differently from traditional software because agents make dynamic decisions, are non-deterministic between runs, and often lack single correct answers. Build evaluation frameworks that account for these characteristics, provide actionable feedback, catch regressions, and validate that context engineering choices achieve intended effects.\n\n## When to Activate\n\nActivate this skill when:\n- Testing agent performance systematically\n- Validating context engineering choices\n- Measuring improvements over time\n- Catching regressions before deployment\n- Building quality gates for agent pipelines\n- Comparing different agent configurations\n- Evaluating production systems continuously\n\nDo not activate this skill for adjacent work owned by other skills:\n- Designing the LLM judge itself, pairwise comparison, judge calibration, or bias mitigation: `advanced-evaluation`.\n- Designing autonomous control surfaces, novelty gates, rollback, or PR approval boundaries: `harness-engineering`.\n- Debugging a specific context failure mode before measuring it: `context-degradation`.\n\n## Core Concepts\n\nFocus evaluation on outcomes rather than execution paths, because agents may find alternative valid routes to goals. Judge whether the agent achieves the right outcome via a reasonable process, not whether it followed a specific sequence of steps.\n\nUse multi-dimensional rubrics instead of single scores because one number hides critical failures in specific dimensions. Capture factual accuracy, completeness, citation accuracy, source quality, and tool efficiency as separate dimensions, then weight them for the use case.\n\nUse model-judged evaluation only after deterministic checks and rubrics are stable. When the work centers on judge prompts, pairwise comparison, calibration, or bias mitigation, switch to Advanced Evaluation.\n\nRun deterministic validation before LLM judgment whenever the artifact has machine-checkable structure. Schema validity, duplicate keys, rubric math, manifest sync, retrieval status, and required evidence paths should fail fast before an evaluator spends tokens or returns a subjective score.\n\n**Performance Drivers**\n\nApply browsing-agent research when designing evaluation budgets: token usage, tool calls, and model choice can dominate measured performance variance (claim-evaluation-browsecomp-variance).\n\n| Factor | Variance Explained | Implication |\n|--------|-------------------|-------------|\n| Token usage | Primary driver | More exploration can improve performance until cost or context quality collapses |\n| Number of tool calls | Secondary driver | More tool use helps only when calls retrieve useful evidence |\n| Model choice | Secondary but multiplicative | Better models often use tokens and tools more efficiently |\n\nAct on these implications when designing evaluations:\n- **Set realistic token budgets**: Evaluate agents with production-realistic token limits, not unlimited resources.\n- **Compare model upgrades against token increases**: Better models may use tokens more efficiently than weaker models with larger budgets.\n- **Validate multi-agent architectures**: Extra agents add tokens and tool calls; evaluate them against single-agent baselines.\n\n## Detailed Topics\n\n### Evaluation Challenges\n\n**Handle Non-Determinism and Multiple Valid Paths**\n\nDesign evaluations that tolerate path variation because agents may take completely different valid paths to reach goals. One agent might search three sources while another searches ten; both may produce correct answers. Avoid checking for specific steps. Instead, define outcome criteria (correctness, completeness, quality) and score against those, treating the execution path as informational rather than evaluative.\n\n**Test Context-Dependent Failures**\n\nEvaluate across a range of complexity levels and interaction lengths because agent failures often depend on context in subtle ways. An agent might succeed on simple queries but fail on complex ones, work well with one tool set but fail with another, or degrade after extended interaction as context accumulates. Include simple, medium, complex, and very complex test cases to surface these patterns.\n\n**Score Composite Quality Dimensions Separately**\n\nBreak agent quality into separate dimensions (factual accuracy, completeness, coherence, tool efficiency, process quality) and score each independently because an agent might score high on accuracy but low on efficiency, or vice versa. Then compute weighted aggregates tuned to use-case priorities. This approach reveals which dimensions need improvement rather than averaging away the signal.\n\n### Evaluation Rubric Design\n\n**Build Multi-Dimensional Rubrics**\n\nDefine rubrics covering key dimensions with descriptive levels from excellent to failed. Include these core dimensions and adapt weights per use case:\n\n- Factual accuracy: Claims match ground truth (weight heavily for knowledge tasks)\n- Completeness: Output covers requested aspects (weight heavily for research tasks)\n- Citation accuracy: Citations match claimed sources (weight for trust-sensitive contexts)\n- Source quality: Uses appropriate primary sources (weight for authoritative outputs)\n- Tool efficiency: Uses right tools a reasonable number of times (weight for cost-sensitive systems)\n\n**Convert Rubrics to Numeric Scores**\n\nMap dimension assessments to numeric scores (0.0 to 1.0), apply per-dimension weights, and calculate weighted overall scores. Set passing thresholds based on use-case requirements, typically 0.7 for general use and 0.9 for high-stakes applications. Store individual dimension scores alongside the aggregate because the breakdown drives targeted improvement.\n\n### Evaluation Methodologies\n\n**Use LLM-as-Judge for Scale**\n\nBuild LLM-based evaluation prompts that include: clear task description, the agent output under test, ground truth when available, an evaluation scale with explicit level descriptions, and a request for structured judgment with reasoning. LLM judges provide consistent, scalable evaluation across large test sets. Use a different model family than the agent being evaluated to avoid self-enhancement bias.\n\n**Supplement with Human Evaluation**\n\nRoute edge cases, unusual queries, and a random sample of production traffic to human reviewers because humans notice hallucinated answers, system failures, and subtle biases that automated evaluation misses. Track patterns across human reviews to identify systematic issues and feed findings back into automated evaluation criteria.\n\n**Apply End-State Evaluation for Stateful Agents**\n\nFor agents that mutate persistent state (files, databases, configurations), evaluate whether the final state matches expectations rather than how the agent got there. Define expected end-state assertions and verify them programmatically after each test run.\n\n### Test Set Design\n\n**Select Representative Samples**\n\nStart with small samples (20-30 cases) during early development when changes have dramatic impacts and low-hanging fruit is abundant. Scale to 50+ cases for reliable signal as the system matures. Sample from real usage patterns, add known edge cases, and ensure coverage across complexity levels.\n\n**Stratify by Complexity**\n\nStructure test sets across complexity levels to prevent easy examples from inflating scores:\n- Simple: single tool call, factual lookup\n- Medium: multiple tool calls, comparison logic\n- Complex: many tool calls, significant ambiguity\n- Very complex: extended interaction, deep reasoning, synthesis\n\nReport scores per stratum alongside overall scores to reveal where the agent actually struggles.\n\n### Context Engineering Evaluation\n\n**Validate Context Strategies Systematically**\n\nRun agents with different context strategies on the same test set and compare quality scores, token usage, and efficiency metrics. This isolates the effect of context engineering from other variables and prevents anecdote-driven decisions.\n\n**Run Degradation Tests**\n\nTest how context degradation affects performance by running agents at different context sizes. Identify performance cliffs where context becomes problematic and establish safe operating limits. Feed these limits back into context management strategies.\n\n### Continuous Evaluation\n\n**Build Automated Evaluation Pipelines**\n\nIntegrate evaluation into the development workflow so evaluations run automatically on agent changes. Track results over time, compare versions, and block deployments that regress on key metrics.\n\n**Monitor Production Quality**\n\nSample production interactions and evaluate them continuously. Set alerts for quality drops below warning (0.85 pass rate) and critical (0.70 pass rate) thresholds. Maintain dashboards showing trend analysis over time windows to detect gradual degradation.\n\n## Practical Guidance\n\n### Building Evaluation Frameworks\n\nFollow this sequence to build an evaluation framework, because skipping early steps leads to measurements that do not reflect real quality:\n\n1. Define quality dimensions relevant to the use case before writing any evaluation code, because dimensions chosen later tend to reflect what is easy to measure rather than what matters.\n2. Create rubrics with clear, descriptive level definitions so evaluators (human or LLM) produce consistent scores.\n3. Build test sets from real usage patterns and edge cases, stratified by complexity, with at least 50 cases for reliable signal.\n4. Implement automated evaluation pipelines that run on every significant change.\n5. Establish baseline metrics before making changes so improvements can be measured against a known reference.\n6. Run evaluations on all significant changes and compare against the baseline.\n7. Track metrics over time for trend analysis because gradual degradation is harder to notice than sudden drops.\n8. Supplement automated evaluation with human review on a regular cadence.\n9. Separate deterministic validation failures from quality judgments so invalid artifacts cannot be laundered by a favorable LLM score.\n\n### Avoiding Evaluation Pitfalls\n\nGuard against these common failures that undermine evaluation reliability:\n\n- **Overfitting to specific paths**: Evaluate outcomes, not specific steps, because agents find novel valid paths.\n- **Ignoring edge cases**: Include diverse test scenarios covering the full complexity spectrum.\n- **Single-metric obsession**: Use multi-dimensional rubrics because a single score hides dimension-specific failures.\n- **Neglecting context effects**: Test with realistic context sizes and histories rather than clean-room conditions.\n- **Skipping human evaluation**: Automated evaluation misses subtle issues that humans catch reliably.\n\n## Examples\n\n**Example 1: Simple Evaluation**\n```python\ndef evaluate_agent_response(response, expected):\n rubric = load_rubric()\n scores = {}\n for dimension, config in rubric.items():\n scores[dimension] = assess_dimension(response, expected, dimension)\n overall = weighted_average(scores, config[\"weights\"])\n return {\"passed\": overall >= 0.7, \"scores\": scores}\n```\n\n**Example 2: Test Set Structure**\n\nTest sets should span multiple complexity levels to ensure comprehensive evaluation:\n\n```python\ntest_set = [\n {\n \"name\": \"simple_lookup\",\n \"input\": \"What is the capital of France?\",\n \"expected\": {\"type\": \"fact\", \"answer\": \"Paris\"},\n \"complexity\": \"simple\",\n \"description\": \"Single tool call, factual lookup\"\n },\n {\n \"name\": \"medium_query\",\n \"input\": \"Compare the revenue of Apple and Microsoft last quarter\",\n \"complexity\": \"medium\",\n \"description\": \"Multiple tool calls, comparison logic\"\n },\n {\n \"name\": \"multi_step_reasoning\",\n \"input\": \"Analyze sales data from Q1-Q4 and create a summary report with trends\",\n \"complexity\": \"complex\",\n \"description\": \"Many tool calls, aggregation, analysis\"\n },\n {\n \"name\": \"research_synthesis\",\n \"input\": \"Research emerging AI technologies, evaluate their potential impact, and recommend adoption strategy\",\n \"complexity\": \"very_complex\",\n \"description\": \"Extended interaction, deep reasoning, synthesis\"\n }\n]\n```\n\n**Example 3: Deterministic gate before model judgment**\n```python\ndef evaluate_pr_candidate(candidate):\n structure = run_validate_repo(candidate)\n if not structure.ok:\n return {\"passed\": False, \"reason\": \"deterministic validation failed\", \"details\": structure.errors}\n\n quality = run_rubric_eval(candidate)\n return {\"passed\": quality.overall >= 0.8, \"scores\": quality.dimensions}\n```\n\n**Example 4: Quality gate dimensions**\n```yaml\ngate:\n deterministic:\n - schema_valid\n - required_files_present\n - no_duplicate_ids\n quality:\n factual_accuracy: min 0.85\n completeness: min 0.80\n source_traceability: min 0.90\n```\n\n## Guidelines\n\n1. Use multi-dimensional rubrics, not single metrics\n2. Evaluate outcomes, not specific execution paths\n3. Cover complexity levels from simple to complex\n4. Test with realistic context sizes and histories\n5. Run evaluations continuously, not just before release\n6. Supplement LLM evaluation with human review\n7. Track metrics over time for trend detection\n8. Set clear pass/fail thresholds based on use case\n\n## Gotchas\n\n1. **Overfitting evals to specific code paths**: Tests pass but the agent fails on slight input variations. Write eval criteria against outcomes and semantics, not surface patterns, and rotate test inputs periodically.\n2. **LLM-judge self-enhancement bias**: Models rate their own outputs higher than independent judges do. Use a different model family as the evaluation judge than the model being evaluated.\n3. **Test set contamination**: Eval examples leak into training data or prompt templates, inflating scores. Keep eval sets versioned and separate from any data used in prompts or fine-tuning.\n4. **Metric gaming**: Optimizing for the metric rather than actual quality produces agents that score well but disappoint users. Cross-validate automated metrics against human judgments regularly.\n5. **Single-dimension scoring**: One aggregate number hides critical failures in specific dimensions. Always report per-dimension scores alongside the overall score, and fail the eval if any single dimension falls below its minimum threshold.\n6. **Eval set too small**: Fewer than 50 examples produces unreliable signal with high variance between runs. Scale the eval set to at least 50 cases and report confidence intervals.\n7. **Not stratifying by difficulty**: Easy examples inflate overall scores, masking failures on hard cases. Report scores per complexity stratum and weight the overall score to prevent easy-case dominance.\n8. **Treating eval as one-time**: Evaluation must be continuous, not a launch gate. Agent quality drifts as models update, tools change, and usage patterns evolve. Run evals on every change and on a regular production cadence.\n\n## Integration\n\nThis skill owns outcome measurement and quality gates. Adjacent skills own specialized evaluator design and control-loop governance:\n\n- `advanced-evaluation`: LLM-as-judge prompt design, pairwise comparison, calibration, and bias mitigation.\n- `harness-engineering`: locked evaluators, editable surfaces, rollback, and human approval boundaries.\n- `context-degradation`: detecting and measuring degradation patterns.\n- `context-optimization`: measuring token, cost, latency, and quality effects of optimizations.\n- `multi-agent-patterns`: evaluating coordination quality and parallelization trade-offs.\n- `tool-design`: evaluating tool selection and recovery effectiveness.\n- `memory-systems`: evaluating memory retrieval and retention quality.\n\n## References\n\nInternal reference:\n- [Metrics Reference](./references/metrics.md) - Read when: designing specific evaluation metrics, choosing scoring scales, or implementing weighted rubric calculations\n\nInternal skills:\n- All other skills connect to evaluation for quality measurement\n\nExternal resources:\n- LLM evaluation benchmarks - Read when: selecting or building benchmark suites for agent comparison\n- Agent evaluation research papers - Read when: adopting new evaluation methodologies or validating current approach\n- Production monitoring practices - Read when: setting up alerting, dashboards, or sampling strategies for live systems\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-20\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 1.2.0\n","skills/evaluation/references/metrics.md":"# Evaluation Reference: Metrics and Implementation\n\nThis document provides implementation details for evaluation metrics and evaluation systems.\n\n## Core Metric Definitions\n\n### Factual Accuracy\n\nFactual accuracy measures whether claims in agent output match ground truth.\n\n```\nExcellent (1.0): All claims verified against ground truth, no errors\nGood (0.8): Minor errors that do not affect main conclusions\nAcceptable (0.6): Major claims correct, minor inaccuracies present\nPoor (0.3): Significant factual errors in key claims\nFailed (0.0): Fundamental factual errors that invalidate output\n```\n\nCalculation approach:\n- Extract claims from output\n- Verify each claim against ground truth\n- Weight claims by importance (major claims more weight)\n- Calculate weighted average of claim accuracy\n\n### Completeness\n\nCompleteness measures whether output covers all requested aspects.\n\n```\nExcellent (1.0): All requested aspects thoroughly covered\nGood (0.8): Most aspects covered with minor gaps\nAcceptable (0.6): Key aspects covered, some gaps\nPoor (0.3): Major aspects missing from output\nFailed (0.0): Fundamental aspects not addressed\n```\n\n### Citation Accuracy\n\nCitation accuracy measures whether cited sources match claimed sources.\n\n```\nExcellent (1.0): All citations accurate and complete\nGood (0.8): Minor citation formatting issues\nAcceptable (0.6): Major citations accurate\nPoor (0.3): Significant citation problems\nFailed (0.0): Citations missing or completely incorrect\n```\n\n### Source Quality\n\nSource quality measures whether appropriate primary sources were used.\n\n```\nExcellent (1.0): Primary authoritative sources\nGood (0.8): Mostly primary sources with some secondary\nAcceptable (0.6): Mix of primary and secondary sources\nPoor (0.3): Mostly secondary or unreliable sources\nFailed (0.0): No credible sources cited\n```\n\n### Tool Efficiency\n\nTool efficiency measures whether the agent used appropriate tools a reasonable number of times.\n\n```\nExcellent (1.0): Optimal tool selection and call count\nGood (0.8): Good tool selection with minor inefficiencies\nAcceptable (0.6): Appropriate tools with some redundancy\nPoor (0.3): Wrong tools or excessive call counts\nFailed (0.0): Severe tool misuse or extremely excessive calls\n```\n\n## Rubric Implementation\n\n```python\nEVALUATION_DIMENSIONS = {\n \"factual_accuracy\": {\n \"weight\": 0.30,\n \"description\": \"Claims match ground truth\",\n \"levels\": {\n \"excellent\": 1.0,\n \"good\": 0.8,\n \"acceptable\": 0.6,\n \"poor\": 0.3,\n \"failed\": 0.0\n }\n },\n \"completeness\": {\n \"weight\": 0.25,\n \"description\": \"All requested aspects covered\",\n \"levels\": {\n \"excellent\": 1.0,\n \"good\": 0.8,\n \"acceptable\": 0.6,\n \"poor\": 0.3,\n \"failed\": 0.0\n }\n },\n \"citation_accuracy\": {\n \"weight\": 0.15,\n \"description\": \"Citations match sources\",\n \"levels\": {\n \"excellent\": 1.0,\n \"good\": 0.8,\n \"acceptable\": 0.6,\n \"poor\": 0.3,\n \"failed\": 0.0\n }\n },\n \"source_quality\": {\n \"weight\": 0.10,\n \"description\": \"Appropriate primary sources used\",\n \"levels\": {\n \"excellent\": 1.0,\n \"good\": 0.8,\n \"acceptable\": 0.6,\n \"poor\": 0.3,\n \"failed\": 0.0\n }\n },\n \"tool_efficiency\": {\n \"weight\": 0.20,\n \"description\": \"Right tools used reasonably\",\n \"levels\": {\n \"excellent\": 1.0,\n \"good\": 0.8,\n \"acceptable\": 0.6,\n \"poor\": 0.3,\n \"failed\": 0.0\n }\n }\n}\n\ndef calculate_overall_score(dimension_scores, rubric):\n \"\"\"Calculate weighted overall score from dimension scores.\"\"\"\n total_weight = 0\n weighted_sum = 0\n \n for dimension, score in dimension_scores.items():\n if dimension in rubric:\n weight = rubric[dimension][\"weight\"]\n weighted_sum += score * weight\n total_weight += weight\n \n return weighted_sum / total_weight if total_weight > 0 else 0\n```\n\n## Test Set Management\n\n```python\nclass TestSet:\n def __init__(self, name):\n self.name = name\n self.tests = []\n self.tags = {}\n \n def add_test(self, test_case):\n \"\"\"Add test case to test set.\"\"\"\n self.tests.append(test_case)\n \n # Index by tags\n for tag in test_case.get(\"tags\", []):\n if tag not in self.tags:\n self.tags[tag] = []\n self.tags[tag].append(len(self.tests) - 1)\n \n def filter(self, **criteria):\n \"\"\"Filter tests by criteria.\"\"\"\n filtered = []\n for test in self.tests:\n match = True\n for key, value in criteria.items():\n if test.get(key) != value:\n match = False\n break\n if match:\n filtered.append(test)\n return filtered\n \n def get_complexity_distribution(self):\n \"\"\"Get distribution of tests by complexity.\"\"\"\n distribution = {}\n for test in self.tests:\n complexity = test.get(\"complexity\", \"medium\")\n distribution[complexity] = distribution.get(complexity, 0) + 1\n return distribution\n```\n\n## Evaluation Runner\n\n```python\nclass EvaluationRunner:\n def __init__(self, test_set, rubric, agent):\n self.test_set = test_set\n self.rubric = rubric\n self.agent = agent\n self.results = []\n \n def run_all(self, verbose=False):\n \"\"\"Run evaluation on all tests.\"\"\"\n self.results = []\n \n for i, test in enumerate(self.test_set.tests):\n if verbose:\n print(f\"Running test {i+1}/{len(self.test_set.tests)}\")\n \n result = self.run_test(test)\n self.results.append(result)\n \n return self.summarize()\n \n def run_test(self, test):\n \"\"\"Run single evaluation test.\"\"\"\n # Get agent output\n output = self.agent.run(test[\"input\"])\n \n # Evaluate\n evaluation = self.evaluate_output(output, test)\n \n return {\n \"test\": test,\n \"output\": output,\n \"evaluation\": evaluation\n }\n \n def evaluate_output(self, output, test):\n \"\"\"Evaluate agent output against test.\"\"\"\n ground_truth = test.get(\"expected\", {})\n \n dimension_scores = {}\n for dimension, config in self.rubric.items():\n score = self.evaluate_dimension(\n output, ground_truth, dimension, config\n )\n dimension_scores[dimension] = score\n \n overall = calculate_overall_score(dimension_scores, self.rubric)\n \n return {\n \"overall_score\": overall,\n \"dimension_scores\": dimension_scores,\n \"passed\": overall >= 0.7\n }\n \n def summarize(self):\n \"\"\"Summarize evaluation results.\"\"\"\n if not self.results:\n return {\"error\": \"No results\"}\n \n passed = sum(1 for r in self.results if r[\"evaluation\"][\"passed\"])\n \n dimension_totals = {}\n for dimension in self.rubric.keys():\n dimension_totals[dimension] = {\n \"total\": 0,\n \"count\": 0\n }\n \n for result in self.results:\n for dimension, score in result[\"evaluation\"][\"dimension_scores\"].items():\n if dimension in dimension_totals:\n dimension_totals[dimension][\"total\"] += score\n dimension_totals[dimension][\"count\"] += 1\n \n dimension_averages = {}\n for dimension, data in dimension_totals.items():\n if data[\"count\"] > 0:\n dimension_averages[dimension] = data[\"total\"] / data[\"count\"]\n \n return {\n \"total_tests\": len(self.results),\n \"passed\": passed,\n \"failed\": len(self.results) - passed,\n \"pass_rate\": passed / len(self.results) if self.results else 0,\n \"dimension_averages\": dimension_averages,\n \"failures\": [\n r for r in self.results \n if not r[\"evaluation\"][\"passed\"]\n ]\n }\n```\n\n## Production Monitoring\n\n```python\nclass ProductionMonitor:\n def __init__(self, sample_rate=0.01):\n self.sample_rate = sample_rate\n self.samples = []\n self.alert_thresholds = {\n \"pass_rate_warning\": 0.85,\n \"pass_rate_critical\": 0.70\n }\n \n def sample_and_evaluate(self, query, output):\n \"\"\"Sample production interaction for evaluation.\"\"\"\n if random.random() > self.sample_rate:\n return None\n \n evaluation = evaluate_output(output, {}, EVALUATION_RUBRIC)\n \n sample = {\n \"query\": query[:200],\n \"output_preview\": output[:200],\n \"score\": evaluation[\"overall_score\"],\n \"passed\": evaluation[\"passed\"],\n \"timestamp\": current_timestamp()\n }\n \n self.samples.append(sample)\n return sample\n \n def get_metrics(self):\n \"\"\"Calculate current metrics from samples.\"\"\"\n if not self.samples:\n return {\"status\": \"insufficient_data\"}\n \n passed = sum(1 for s in self.samples if s[\"passed\"])\n pass_rate = passed / len(self.samples)\n \n avg_score = sum(s[\"score\"] for s in self.samples) / len(self.samples)\n \n return {\n \"sample_count\": len(self.samples),\n \"pass_rate\": pass_rate,\n \"average_score\": avg_score,\n \"status\": self._get_status(pass_rate)\n }\n \n def _get_status(self, pass_rate):\n \"\"\"Get status based on pass rate.\"\"\"\n if pass_rate < self.alert_thresholds[\"pass_rate_critical\"]:\n return \"critical\"\n elif pass_rate < self.alert_thresholds[\"pass_rate_warning\"]:\n return \"warning\"\n else:\n return \"healthy\"\n```\n\n","skills/filesystem-context/SKILL.md":"---\nname: filesystem-context\ndescription: \"This skill should be used when agent work needs file-backed context: durable scratchpads, tool-output offloading, just-in-time discovery, cross-agent handoff files, filesystem memory, or cleanup policies for context stored outside the prompt.\"\n---\n\n# Filesystem-Based Context Engineering\n\nUse the filesystem as the primary overflow layer for agent context because context windows are limited while tasks often require more information than fits in a single window. Files let agents store, retrieve, and update an effectively unlimited amount of context through a single interface.\n\nPrefer dynamic context discovery -- pulling relevant context on demand -- over static inclusion, because static context consumes tokens regardless of relevance and crowds out space for task-specific information.\n\n## When to Activate\n\nActivate this skill when:\n- Tool outputs are bloating the context window\n- Agents need to persist state across long trajectories\n- Sub-agents must share information without direct message passing\n- Tasks require more context than fits in the window\n- Building agents that learn and update their own instructions\n- Implementing scratch pads for intermediate results\n- Terminal outputs or logs need to be accessible to agents\n\nDo not activate this skill for adjacent work owned by other skills:\n- Semantic cross-session memory, entity tracking, or temporal knowledge graphs: `memory-systems`.\n- Conversation summarization, compaction, or durable handoff wording: `context-compression`.\n- Token-efficiency tactics that do not require file-backed storage: `context-optimization`.\n- Multi-agent topology or handoff protocol design: `multi-agent-patterns`.\n- Model edits that evict content from the live window itself: `self-managed-context`.\n\n## Core Concepts\n\nDiagnose context failures against these four modes, because each requires a different filesystem remedy:\n\n1. **Missing context** -- needed information is absent from the total available context. Fix by persisting tool outputs and intermediate results to files so nothing is lost.\n2. **Under-retrieved context** -- retrieved content fails to encapsulate what the agent needs. Fix by structuring files for targeted retrieval (grep-friendly formats, clear section headers).\n3. **Over-retrieved context** -- retrieved content far exceeds what is needed, wasting tokens and degrading attention. Fix by offloading bulk content to files and returning compact references.\n4. **Buried context** -- niche information is hidden across many files. Fix by combining glob and grep for structural search alongside semantic search for conceptual queries.\n\nUse the filesystem as the persistent layer that addresses all four: write once, store durably, retrieve selectively.\n\n## Detailed Topics\n\n### The Static vs Dynamic Context Trade-off\n\nTreat static context (system instructions, tool definitions, critical rules) as expensive real estate -- it consumes tokens on every turn regardless of relevance. As agents accumulate capabilities, static context grows and crowds out dynamic information.\n\nUse dynamic context discovery instead: include only minimal static pointers (names, one-line descriptions, file paths) and load full content with search tools when relevant. This is more token-efficient and often improves response quality by reducing contradictory or irrelevant information in the window.\n\nAccept the trade-off: dynamic discovery requires the model to recognize when it needs more context. Current frontier models handle this well, but less capable models may fail to trigger loads. When in doubt, err toward including critical safety or correctness constraints statically.\n\n### Pattern 1: Filesystem as Scratch Pad\n\nRedirect large tool outputs to files instead of returning them directly to context, because a single web search or database query can dump thousands of tokens into message history where they persist for the entire conversation.\n\nWrite the output to a scratch file, extract a compact summary, and return a file reference. The agent then uses targeted retrieval (grep for patterns, read with line ranges) to access only what it needs.\n\n```python\ndef handle_tool_output(output: str, threshold: int = 2000) -> str:\n if len(output) < threshold:\n return output\n\n file_path = f\"scratch/{tool_name}_{timestamp}.txt\"\n write_file(file_path, output)\n\n key_summary = extract_summary(output, max_tokens=200)\n return f\"[Output written to {file_path}. Summary: {key_summary}]\"\n```\n\nUse grep to search the offloaded file and read_file with line ranges to retrieve targeted sections, because this preserves full output for later reference while keeping only ~100 tokens in the active context.\n\n### Pattern 2: Plan Persistence\n\nWrite plans to the filesystem because long-horizon tasks lose coherence when plans fall out of attention or get summarized away. The agent re-reads its plan at any point, restoring awareness of the objective and progress.\n\nStore plans in structured format so they are both human-readable and machine-parseable:\n```yaml\n# scratch/current_plan.yaml\nobjective: \"Refactor authentication module\"\nstatus: in_progress\nsteps:\n - id: 1\n description: \"Audit current auth endpoints\"\n status: completed\n - id: 2\n description: \"Design new token validation flow\"\n status: in_progress\n - id: 3\n description: \"Implement and test changes\"\n status: pending\n```\n\nRe-read the plan at the start of each turn or after any context refresh to re-orient, because this acts as \"manipulating attention through recitation.\"\n\n### Pattern 3: Sub-Agent Communication via Filesystem\n\nRoute sub-agent findings through the filesystem instead of message passing, because multi-hop message chains degrade information through summarization at each hop (\"game of telephone\").\n\nHave each sub-agent write directly to its own workspace directory. The coordinator reads these files directly, preserving full fidelity:\n```\nworkspace/\n agents/\n research_agent/\n findings.md\n sources.jsonl\n code_agent/\n changes.md\n test_results.txt\n coordinator/\n synthesis.md\n```\n\nEnforce per-agent directory isolation to prevent write conflicts and maintain clear ownership of each output artifact.\n\n### Pattern 4: Dynamic Skill Loading\n\nStore skills as files and include only skill names with brief descriptions in static context, because stuffing all instructions into the system prompt wastes tokens and can confuse the model with contradictory guidance.\n\n```\nAvailable skills (load with read_file when relevant):\n- database-optimization: Query tuning and indexing strategies\n- api-design: REST/GraphQL best practices\n- testing-strategies: Unit, integration, and e2e testing patterns\n```\n\nLoad the full skill file (e.g., `skills/database-optimization/SKILL.md`) only when the current task requires it. This converts O(n) static token cost into O(1) per task.\n\n### Pattern 5: Terminal and Log Persistence\n\nPersist terminal output to files automatically and use grep for selective retrieval, because terminal output from long-running processes accumulates rapidly and manual copy-paste is error-prone.\n\n```\nterminals/\n 1.txt # Terminal session 1 output\n 2.txt # Terminal session 2 output\n```\n\nQuery with targeted grep (`grep -A 5 \"error\" terminals/1.txt`) instead of loading entire terminal histories into context.\n\n### Pattern 6: Learning Through Self-Modification\n\nHave agents write learned preferences and patterns to their own instruction files so subsequent sessions load this context automatically, instead of requiring manual system prompt updates.\n\n```python\ndef remember_preference(key: str, value: str):\n preferences_file = \"agent/user_preferences.yaml\"\n prefs = load_yaml(preferences_file)\n prefs[key] = value\n write_yaml(preferences_file, prefs)\n```\n\nGuard this pattern with validation because self-modification can accumulate incorrect or contradictory instructions over time. Treat it as experimental -- review persisted preferences periodically.\n\n### Filesystem Search Techniques\n\nCombine `ls`/`list_dir`, `glob`, `grep`, and `read_file` with line ranges for context discovery, because models are specifically trained on filesystem traversal and this combination often outperforms semantic search for technical content where structural patterns are clear.\n\n- `ls` / `list_dir`: Discover directory structure\n- `glob`: Find files matching patterns (e.g., `**/*.py`)\n- `grep`: Search file contents, returns matching lines with context\n- `read_file` with ranges: Read specific sections without loading entire files\n\nUse filesystem search for structural and exact-match queries, and semantic search for conceptual queries. Combine both for comprehensive discovery.\n\n## Practical Guidance\n\n### When to Use Filesystem Context\n\nApply filesystem patterns when the situation matches these criteria, because they add I/O overhead that is only justified by token savings or persistence needs:\n\n**Use when:**\n- Tool outputs exceed ~2000 tokens\n- Tasks span multiple conversation turns\n- Multiple agents need shared state\n- Skills or instructions exceed comfortable system prompt size\n- Logs or terminal output need selective querying\n\n**Avoid when:**\n- Tasks complete in single turns (overhead not justified)\n- Context fits comfortably in window (no problem to solve)\n- Latency is critical (file I/O adds measurable delay)\n- Model lacks filesystem tool capabilities\n\n### File Organization\n\nStructure files for agent discoverability, because agents navigate by listing and reading directory names:\n```\nproject/\n scratch/ # Temporary working files\n tool_outputs/ # Large tool results\n plans/ # Active plans and checklists\n memory/ # Persistent learned information\n preferences.yaml # User preferences\n patterns.md # Learned patterns\n skills/ # Loadable skill definitions\n agents/ # Sub-agent workspaces\n```\n\nUse consistent naming conventions and include timestamps or IDs in scratch files for disambiguation.\n\nFor autonomous research loops, store raw retrieved evidence under the run that consumed it, for example `researcher/runs//sources/evidence/raw/`. Do not leave raw research dumps in the repository root; root-level artifacts become hard to audit and easy to cite without provenance.\n\n### Token Accounting\n\nMeasure where tokens originate before and after applying filesystem patterns, because optimizing without measurement leads to wasted effort:\n- Track static vs dynamic context ratio\n- Monitor tool output sizes before and after offloading\n- Measure how often dynamically-loaded context is actually used\n\n## Examples\n\n**Example 1: Tool Output Offloading**\n```\nInput: Web search returns 8000 tokens\nBefore: 8000 tokens added to message history\nAfter:\n - Write to scratch/search_results_001.txt\n - Return: \"[Results in scratch/search_results_001.txt. Key finding: API rate limit is 1000 req/min]\"\n - Agent greps file when needing specific details\nResult: ~100 tokens in context, 8000 tokens accessible on demand\n```\n\n**Example 2: Dynamic Skill Loading**\n```\nInput: User asks about database indexing\nStatic context: \"database-optimization: Query tuning and indexing\"\nAgent action: read_file(\"skills/database-optimization/SKILL.md\")\nResult: Full skill loaded only when relevant\n```\n\n**Example 3: Chat History as File Reference**\n```\nTrigger: Context window limit reached, summarization required\nAction:\n 1. Write full history to history/session_001.txt\n 2. Generate summary for new context window\n 3. Include reference: \"Full history in history/session_001.txt\"\nResult: Agent can search history file to recover details lost in summarization\n```\n\n## Guidelines\n\n1. Write large outputs to files; return summaries and references to context\n2. Store plans and state in structured files for re-reading\n3. Use sub-agent file workspaces instead of message chains\n4. Load skills dynamically rather than stuffing all into system prompt\n5. Persist terminal and log output as searchable files\n6. Combine grep/glob with semantic search for comprehensive discovery\n7. Organize files for agent discoverability with clear naming\n8. Measure token savings to validate filesystem patterns are effective\n9. Implement cleanup for scratch files to prevent unbounded growth\n10. Guard self-modification patterns with validation\n11. Keep raw evidence next to the run, evaluation, and proposal that used it\n\n## Gotchas\n\n1. **Scratch directory unbounded growth**: Agents create temp files without cleanup, eventually consuming disk and making directory listings noisy. Implement a retention policy (age-based or count-based) and run cleanup at session boundaries.\n2. **Race conditions in multi-agent file access**: Concurrent writes to the same file corrupt state silently. Enforce per-agent directory isolation or use append-only files with agent-prefixed entries.\n3. **Stale file references after moves/renames**: Agents hold paths from prior turns that no longer exist after refactors or file reorganization. Always verify file existence before reading a cached path; re-discover with glob if the check fails.\n4. **Glob pattern false matches**: Overly broad patterns (e.g., `**/*`) pull irrelevant files into context, wasting tokens and confusing the model. Scope globs to specific directories and extensions.\n5. **File size assumptions**: Reading a file without checking size can dump 100K+ tokens into context in a single tool call. Check file size before reading; use line-range reads for large files.\n6. **Missing file existence checks**: Agents assume files exist from prior turns, but they may have been deleted or moved. Always guard reads with existence checks and handle missing-file errors gracefully.\n7. **Scratch pad format drift**: Unstructured scratch pads become unparseable after many writes because format conventions erode over successive appends. Define and enforce a schema (YAML, JSON, or structured markdown) from the first write.\n8. **Hardcoded absolute paths**: Break when repositories are checked out at different locations or when running in containers. Use relative paths from the project root or resolve paths dynamically.\n\n## Integration\n\nThis skill owns file-backed context storage and retrieval. Adjacent skills own semantic memory, summarization, and topology:\n\n- `context-optimization`: filesystem offloading is one implementation of observation masking when full outputs remain retrievable.\n- `memory-systems`: use when file-backed notes are no longer enough and semantic, entity, or temporal retrieval is required.\n- `multi-agent-patterns`: sub-agent file workspaces enable context isolation and direct handoff.\n- `context-compression`: file references can anchor summaries and preserve details omitted from compressed context.\n- `tool-design`: tools should return file references for large outputs and expose safe read/search operations.\n- `self-managed-context`: offloading leaves the original in the live window; evicting it requires the model to edit its own context.\n\n## References\n\nInternal reference:\n- [Implementation Patterns](./references/implementation-patterns.md) - Read when: implementing scratch pad, plan persistence, or tool output offloading and need concrete code beyond the inline examples\n\nRelated skills in this collection:\n- context-optimization - Read when: applying token reduction techniques alongside filesystem offloading\n- memory-systems - Read when: building persistent storage that outlasts a single session\n- multi-agent-patterns - Read when: designing agent coordination with shared file workspaces\n\nExternal resources:\n- LangChain Deep Agents — Read when: implementing filesystem-based context patterns in LangChain/LangGraph pipelines\n- Cursor context discovery — Read when: studying how production IDEs implement dynamic context loading\n- Anthropic Agent Skills specification — Read when: building skills that leverage filesystem progressive disclosure\n\n---\n\n## Skill Metadata\n\n**Created**: 2026-01-07\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 1.2.0\n","skills/filesystem-context/references/implementation-patterns.md":"# Filesystem Context Implementation Patterns\n\nThis reference provides detailed implementation patterns for filesystem-based context engineering.\n\n## Pattern Catalog\n\n### 1. Scratch Pad Manager\n\nA centralized manager for handling large tool outputs and intermediate results.\n\n```python\nimport os\nimport json\nfrom datetime import datetime\nfrom pathlib import Path\n\nclass ScratchPadManager:\n \"\"\"Manages temporary file storage for agent context offloading.\"\"\"\n \n def __init__(self, base_path: str = \"scratch\", token_threshold: int = 2000):\n self.base_path = Path(base_path)\n self.base_path.mkdir(parents=True, exist_ok=True)\n self.token_threshold = token_threshold\n self.manifest = {}\n \n def should_offload(self, content: str) -> bool:\n \"\"\"Determine if content exceeds threshold for offloading.\"\"\"\n # Rough token estimate: 1 token ≈ 4 characters\n estimated_tokens = len(content) // 4\n return estimated_tokens > self.token_threshold\n \n def offload(self, content: str, source: str, summary: str = None) -> dict:\n \"\"\"Write content to file, return reference.\"\"\"\n timestamp = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n filename = f\"{source}_{timestamp}.txt\"\n file_path = self.base_path / filename\n \n file_path.write_text(content)\n \n reference = {\n \"type\": \"file_reference\",\n \"path\": str(file_path),\n \"source\": source,\n \"timestamp\": timestamp,\n \"size_chars\": len(content),\n \"summary\": summary or self._extract_summary(content)\n }\n \n self.manifest[filename] = reference\n return reference\n \n def _extract_summary(self, content: str, max_chars: int = 500) -> str:\n \"\"\"Extract first meaningful content as summary.\"\"\"\n lines = content.strip().split('\\n')\n summary_lines = []\n char_count = 0\n \n for line in lines:\n if char_count + len(line) > max_chars:\n break\n summary_lines.append(line)\n char_count += len(line)\n \n return '\\n'.join(summary_lines)\n \n def cleanup(self, max_age_hours: int = 24):\n \"\"\"Remove scratch files older than threshold.\"\"\"\n cutoff = datetime.now().timestamp() - (max_age_hours * 3600)\n \n for file_path in self.base_path.glob(\"*.txt\"):\n if file_path.stat().st_mtime < cutoff:\n file_path.unlink()\n if file_path.name in self.manifest:\n del self.manifest[file_path.name]\n```\n\n### 2. Plan Persistence\n\nStructured plan storage with progress tracking.\n\n```python\nimport yaml\nfrom dataclasses import dataclass, field, asdict\nfrom enum import Enum\nfrom typing import List, Optional\n\nclass StepStatus(Enum):\n PENDING = \"pending\"\n IN_PROGRESS = \"in_progress\"\n COMPLETED = \"completed\"\n BLOCKED = \"blocked\"\n CANCELLED = \"cancelled\"\n\n@dataclass\nclass PlanStep:\n id: int\n description: str\n status: StepStatus = StepStatus.PENDING\n notes: Optional[str] = None\n\n@dataclass\nclass AgentPlan:\n objective: str\n steps: List[PlanStep] = field(default_factory=list)\n status: str = \"in_progress\"\n \n def save(self, path: str = \"scratch/current_plan.yaml\"):\n \"\"\"Persist plan to filesystem.\"\"\"\n data = {\n \"objective\": self.objective,\n \"status\": self.status,\n \"steps\": [\n {\n \"id\": s.id,\n \"description\": s.description,\n \"status\": s.status.value,\n \"notes\": s.notes\n }\n for s in self.steps\n ]\n }\n with open(path, 'w') as f:\n yaml.dump(data, f, default_flow_style=False)\n \n @classmethod\n def load(cls, path: str = \"scratch/current_plan.yaml\") -> \"AgentPlan\":\n \"\"\"Load plan from filesystem.\"\"\"\n with open(path, 'r') as f:\n data = yaml.safe_load(f)\n \n plan = cls(objective=data[\"objective\"], status=data.get(\"status\", \"in_progress\"))\n for step_data in data.get(\"steps\", []):\n plan.steps.append(PlanStep(\n id=step_data[\"id\"],\n description=step_data[\"description\"],\n status=StepStatus(step_data[\"status\"]),\n notes=step_data.get(\"notes\")\n ))\n return plan\n \n def current_step(self) -> Optional[PlanStep]:\n \"\"\"Get the first non-completed step.\"\"\"\n for step in self.steps:\n if step.status != StepStatus.COMPLETED:\n return step\n return None\n \n def complete_step(self, step_id: int, notes: str = None):\n \"\"\"Mark step as completed.\"\"\"\n for step in self.steps:\n if step.id == step_id:\n step.status = StepStatus.COMPLETED\n if notes:\n step.notes = notes\n break\n```\n\n### 3. Sub-Agent Workspace\n\nFile-based communication between agents.\n\n```python\nfrom pathlib import Path\nfrom datetime import datetime\nimport json\n\nclass AgentWorkspace:\n \"\"\"Manages file-based workspace for an agent.\"\"\"\n \n def __init__(self, agent_id: str, base_path: str = \"workspace/agents\"):\n self.agent_id = agent_id\n self.path = Path(base_path) / agent_id\n self.path.mkdir(parents=True, exist_ok=True)\n \n # Standard files\n self.findings_file = self.path / \"findings.md\"\n self.status_file = self.path / \"status.json\"\n self.log_file = self.path / \"activity.log\"\n \n def write_finding(self, content: str, append: bool = True):\n \"\"\"Write or append a finding.\"\"\"\n mode = 'a' if append else 'w'\n with open(self.findings_file, mode) as f:\n if append:\n f.write(f\"\\n---\\n## {datetime.now().isoformat()}\\n\\n\")\n f.write(content)\n \n def update_status(self, status: str, progress: float = None, details: dict = None):\n \"\"\"Update agent status for coordinator visibility.\"\"\"\n status_data = {\n \"agent_id\": self.agent_id,\n \"status\": status,\n \"updated_at\": datetime.now().isoformat(),\n \"progress\": progress,\n \"details\": details or {}\n }\n self.status_file.write_text(json.dumps(status_data, indent=2))\n \n def log(self, message: str):\n \"\"\"Append to activity log.\"\"\"\n with open(self.log_file, 'a') as f:\n f.write(f\"[{datetime.now().isoformat()}] {message}\\n\")\n \n def read_peer_findings(self, peer_id: str) -> str:\n \"\"\"Read findings from another agent's workspace.\"\"\"\n peer_path = self.path.parent / peer_id / \"findings.md\"\n if peer_path.exists():\n return peer_path.read_text()\n return \"\"\n\n\nclass CoordinatorWorkspace:\n \"\"\"Coordinator that reads from sub-agent workspaces.\"\"\"\n \n def __init__(self, base_path: str = \"workspace/agents\"):\n self.base_path = Path(base_path)\n \n def get_all_statuses(self) -> dict:\n \"\"\"Collect status from all sub-agents.\"\"\"\n statuses = {}\n for agent_dir in self.base_path.iterdir():\n if agent_dir.is_dir():\n status_file = agent_dir / \"status.json\"\n if status_file.exists():\n statuses[agent_dir.name] = json.loads(status_file.read_text())\n return statuses\n \n def aggregate_findings(self) -> str:\n \"\"\"Combine all agent findings into synthesis.\"\"\"\n findings = []\n for agent_dir in self.base_path.iterdir():\n if agent_dir.is_dir():\n findings_file = agent_dir / \"findings.md\"\n if findings_file.exists():\n findings.append(f\"# {agent_dir.name}\\n\\n{findings_file.read_text()}\")\n return \"\\n\\n\".join(findings)\n```\n\n### 4. Dynamic Skill Loader\n\nLoad skill content on demand.\n\n```python\nfrom pathlib import Path\nfrom typing import List, Optional\nimport yaml\n\n@dataclass\nclass SkillMetadata:\n name: str\n description: str\n path: str\n triggers: List[str] = field(default_factory=list)\n\nclass SkillLoader:\n \"\"\"Manages dynamic loading of agent skills.\"\"\"\n \n def __init__(self, skills_path: str = \"skills\"):\n self.skills_path = Path(skills_path)\n self.skill_index = self._build_index()\n \n def _build_index(self) -> dict:\n \"\"\"Build index of available skills from SKILL.md frontmatter.\"\"\"\n index = {}\n for skill_dir in self.skills_path.iterdir():\n if skill_dir.is_dir():\n skill_file = skill_dir / \"SKILL.md\"\n if skill_file.exists():\n metadata = self._parse_frontmatter(skill_file)\n if metadata:\n index[metadata.name] = metadata\n return index\n \n def _parse_frontmatter(self, path: Path) -> Optional[SkillMetadata]:\n \"\"\"Extract YAML frontmatter from skill file.\"\"\"\n content = path.read_text()\n if content.startswith('---'):\n end = content.find('---', 3)\n if end > 0:\n frontmatter = yaml.safe_load(content[3:end])\n return SkillMetadata(\n name=frontmatter.get('name', path.parent.name),\n description=frontmatter.get('description', ''),\n path=str(path),\n triggers=frontmatter.get('triggers', [])\n )\n return None\n \n def get_static_context(self) -> str:\n \"\"\"Generate minimal static context listing available skills.\"\"\"\n lines = [\"Available skills (load with read_file when relevant):\"]\n for name, meta in self.skill_index.items():\n lines.append(f\"- {name}: {meta.description[:100]}\")\n return \"\\n\".join(lines)\n \n def load_skill(self, name: str) -> str:\n \"\"\"Load full skill content.\"\"\"\n if name in self.skill_index:\n return Path(self.skill_index[name].path).read_text()\n raise ValueError(f\"Unknown skill: {name}\")\n \n def find_relevant_skills(self, query: str) -> List[str]:\n \"\"\"Find skills that might be relevant to a query.\"\"\"\n query_lower = query.lower()\n relevant = []\n for name, meta in self.skill_index.items():\n if any(trigger in query_lower for trigger in meta.triggers):\n relevant.append(name)\n elif name.replace('-', ' ') in query_lower:\n relevant.append(name)\n return relevant\n```\n\n### 5. Terminal Output Persistence\n\nCapture and persist terminal sessions.\n\n```python\nimport subprocess\nfrom pathlib import Path\nfrom datetime import datetime\nimport re\n\nclass TerminalCapture:\n \"\"\"Captures and persists terminal output for agent access.\"\"\"\n \n def __init__(self, terminals_path: str = \"terminals\"):\n self.terminals_path = Path(terminals_path)\n self.terminals_path.mkdir(parents=True, exist_ok=True)\n self.session_counter = 0\n \n def run_command(self, command: str, capture: bool = True) -> dict:\n \"\"\"Run command and optionally capture output to file.\"\"\"\n self.session_counter += 1\n \n result = subprocess.run(\n command,\n shell=True,\n capture_output=True,\n text=True\n )\n \n output = {\n \"command\": command,\n \"exit_code\": result.returncode,\n \"stdout\": result.stdout,\n \"stderr\": result.stderr,\n \"timestamp\": datetime.now().isoformat()\n }\n \n if capture:\n output[\"file\"] = self._persist_output(output)\n \n return output\n \n def _persist_output(self, output: dict) -> str:\n \"\"\"Write output to terminal file.\"\"\"\n filename = f\"{self.session_counter}.txt\"\n file_path = self.terminals_path / filename\n \n content = f\"\"\"---\ncommand: {output['command']}\nexit_code: {output['exit_code']}\ntimestamp: {output['timestamp']}\n---\n\n=== STDOUT ===\n{output['stdout']}\n\n=== STDERR ===\n{output['stderr']}\n\"\"\"\n file_path.write_text(content)\n return str(file_path)\n \n def grep_terminals(self, pattern: str, context_lines: int = 3) -> List[dict]:\n \"\"\"Search all terminal outputs for pattern.\"\"\"\n matches = []\n regex = re.compile(pattern, re.IGNORECASE)\n \n for term_file in self.terminals_path.glob(\"*.txt\"):\n content = term_file.read_text()\n lines = content.split('\\n')\n \n for i, line in enumerate(lines):\n if regex.search(line):\n start = max(0, i - context_lines)\n end = min(len(lines), i + context_lines + 1)\n matches.append({\n \"file\": str(term_file),\n \"line_number\": i + 1,\n \"context\": '\\n'.join(lines[start:end])\n })\n \n return matches\n```\n\n### 6. Self-Modification Guard\n\nSafe pattern for agent self-learning.\n\n```python\nimport yaml\nfrom pathlib import Path\nfrom datetime import datetime\nfrom typing import Any\n\nclass PreferenceStore:\n \"\"\"Guarded storage for agent-learned preferences.\"\"\"\n \n MAX_ENTRIES = 100\n MAX_VALUE_LENGTH = 1000\n \n def __init__(self, path: str = \"agent/preferences.yaml\"):\n self.path = Path(path)\n self.path.parent.mkdir(parents=True, exist_ok=True)\n self.preferences = self._load()\n \n def _load(self) -> dict:\n \"\"\"Load preferences from file.\"\"\"\n if self.path.exists():\n return yaml.safe_load(self.path.read_text()) or {}\n return {}\n \n def _save(self):\n \"\"\"Persist preferences to file.\"\"\"\n self.path.write_text(yaml.dump(self.preferences, default_flow_style=False))\n \n def remember(self, key: str, value: Any, source: str = \"user\"):\n \"\"\"Store a preference with validation.\"\"\"\n # Validate key\n if not key or len(key) > 100:\n raise ValueError(\"Invalid key length\")\n \n # Validate value\n value_str = str(value)\n if len(value_str) > self.MAX_VALUE_LENGTH:\n raise ValueError(f\"Value exceeds max length of {self.MAX_VALUE_LENGTH}\")\n \n # Check entry limit\n if len(self.preferences) >= self.MAX_ENTRIES and key not in self.preferences:\n raise ValueError(f\"Max entries ({self.MAX_ENTRIES}) reached\")\n \n # Store with metadata\n self.preferences[key] = {\n \"value\": value,\n \"source\": source,\n \"updated_at\": datetime.now().isoformat()\n }\n self._save()\n \n def recall(self, key: str, default: Any = None) -> Any:\n \"\"\"Retrieve a preference.\"\"\"\n entry = self.preferences.get(key)\n if entry:\n return entry[\"value\"]\n return default\n \n def list_all(self) -> dict:\n \"\"\"Get all preferences for context injection.\"\"\"\n return {k: v[\"value\"] for k, v in self.preferences.items()}\n \n def forget(self, key: str):\n \"\"\"Remove a preference.\"\"\"\n if key in self.preferences:\n del self.preferences[key]\n self._save()\n```\n\n## Integration Example\n\nCombining patterns in an agent harness:\n\n```python\nclass FilesystemContextAgent:\n \"\"\"Agent with filesystem-based context management.\"\"\"\n \n def __init__(self):\n self.scratch = ScratchPadManager()\n self.skills = SkillLoader()\n self.preferences = PreferenceStore()\n self.workspace = AgentWorkspace(\"main_agent\")\n \n def handle_tool_output(self, tool_name: str, output: str) -> str:\n \"\"\"Process tool output, offloading if necessary.\"\"\"\n if self.scratch.should_offload(output):\n ref = self.scratch.offload(output, source=tool_name)\n return f\"[{tool_name} output saved to {ref['path']}. Summary: {ref['summary'][:200]}]\"\n return output\n \n def get_system_prompt(self) -> str:\n \"\"\"Build system prompt with dynamic skill references.\"\"\"\n base_prompt = \"You are a helpful assistant.\"\n skill_context = self.skills.get_static_context()\n user_prefs = self.preferences.list_all()\n \n pref_section = \"\"\n if user_prefs:\n pref_section = \"\\n\\nUser preferences:\\n\" + \"\\n\".join(\n f\"- {k}: {v}\" for k, v in user_prefs.items()\n )\n \n return f\"{base_prompt}\\n\\n{skill_context}{pref_section}\"\n```\n\n## File Organization Best Practices\n\n```\nproject/\n├── scratch/ # Ephemeral working files\n│ ├── tool_outputs/ # Large tool results\n│ │ └── search_20260107.txt\n│ └── plans/ # Active task plans\n│ └── current_plan.yaml\n├── workspace/ # Agent workspaces\n│ └── agents/\n│ ├── research_agent/\n│ │ ├── findings.md\n│ │ └── status.json\n│ └── code_agent/\n│ ├── findings.md\n│ └── status.json\n├── agent/ # Agent configuration\n│ ├── preferences.yaml # Learned preferences\n│ └── patterns.md # Discovered patterns\n├── skills/ # Loadable skills\n│ └── {skill-name}/\n│ └── SKILL.md\n├── terminals/ # Terminal output\n│ ├── 1.txt\n│ └── 2.txt\n└── history/ # Chat history archives\n └── session_001.txt\n```\n\n## Token Accounting Metrics\n\nTrack these metrics to validate filesystem patterns:\n\n1. **Static context ratio**: tokens in static context / total tokens\n2. **Dynamic load rate**: how often skills/files are loaded per task\n3. **Offload savings**: tokens saved by writing to files vs keeping in context\n4. **Retrieval precision**: percentage of loaded content actually used\n\nTarget benchmarks:\n- Static context ratio < 20%\n- Offload savings > 50% for tool-heavy workflows\n- Retrieval precision > 70% (loaded content is relevant)\n\n","skills/harness-engineering/SKILL.md":"---\nname: harness-engineering\ndescription: \"This skill should be used when designing autonomous agent harnesses: research loops, evaluation scaffolds, locked and editable surfaces, durable logs, novelty gates, pruning, rollback, PR preparation, and human approval boundaries.\"\n---\n\n# Harness Engineering\n\nHarness engineering designs the control system around an agent: what it may edit, how it receives feedback, where it writes state, how failures recover, and who can approve irreversible actions. The harness is the difference between a helpful agent session and an autonomous loop that can run for days without corrupting its objective.\n\n## When to Activate\n\nActivate this skill when:\n\n- Building autonomous research or experimentation loops\n- Designing an agent environment with locked metrics and editable code or content\n- Creating PR-producing or background agents\n- Evaluating whether an agent can safely run without frequent human prompts\n- Adding novelty, ablation, pruning, rollback, or durable logging to an agent workflow\n- Preventing agents from gaming benchmarks, weakening rubrics, or losing state across compaction\n\nDo not activate this skill for adjacent work owned by other skills:\n- General quality gates, regression suites, or outcome metrics without autonomous control surfaces: `evaluation`.\n- Tool schemas, response formats, and recovery errors for harness tools: `tool-design`.\n- Project-level task-model fit, pipeline shape, and cost planning: `project-development`.\n- Remote sandbox, warm-pool, and hosted session infrastructure: `hosted-agents`.\n\n## Core Concepts\n\n### Harness Boundary\n\nSeparate the agent from the environment it operates inside. The agent proposes actions; the harness defines allowed surfaces, feedback, persistence, and promotion rules.\n\nUse four surface classes:\n\n| Surface | Examples | Rule |\n| --- | --- | --- |\n| Locked | Eval metric, rubric, validation script, merge policy | Agent may read and propose changes, but cannot score itself with modified rules |\n| Editable | Skill draft, experiment file, prompt, config under test | Agent may mutate during the loop |\n| Append-only | Results log, research thread, rejected ideas | Agent may append, not rewrite |\n| Human-controlled | Merge, production deploy, credentials, destructive operations | Requires explicit human approval |\n\n### Tight Feedback Loops\n\nAutonomy works when feedback is fast, unambiguous, and hard to game. Karpathy's `autoresearch` is the minimal pattern: one editable file, one locked evaluation file, fixed wall-clock budget, one scalar metric, git rollback, and a durable results log. The lesson is not that every harness needs one metric; it is that ambiguous feedback creates ambiguous autonomy.\n\nFor open-ended research-to-skill work, replace the scalar metric with locked rubrics, deterministic structure checks, source traceability, and human review thresholds.\n\n### Durable State\n\nLong-running agents must externalize state. Store plans, source queues, results, failures, and handoffs in files so future agents can resume without relying on chat history. Prime Intellect's autonomous nanoGPT work showed the value of durable scratchpads and `THREAD.md`-style logs for recovery, monitoring, and audit.\n\nUse append-only logs for:\n\n- What was tried\n- What improved or failed\n- Why a candidate was kept, discarded, or routed to review\n- Which upstream sources were checked\n- What the next agent should do\n\n### Search Discipline\n\nAgents tend to exploit the nearest surface, stack complexity, and under-run pruning. Add explicit search rules:\n\n1. Refresh upstream sources on a schedule.\n2. Require novelty checks before spending large budgets.\n3. Preserve rejected attempts to avoid rediscovery.\n4. Run leave-one-out pruning when a stack has multiple additions.\n5. Reward simplification when quality is equal.\n6. Use separate verification before promotion.\n\n### Mechanism Registry\n\nFor research-to-skill systems, track accepted mechanisms separately from prose. A mechanism record should include a stable `mechanism_id`, `owning_skill`, `status`, activation scenario, behavior change, evidence, and failure modes. Novelty gates should compare against this registry before using broader corpus overlap, because keyword overlap catches stale phrasing while mechanism comparison catches real duplication.\n\n### Governance\n\nAutonomous agents may prepare PRs, but governance must be explicit. They can draft changes, run checks, and write PR summaries. They should not merge, deploy, or push without human approval unless the user has explicitly granted that permission for the specific action.\n\n## Detailed Topics\n\n### Autoresearch-Style Loop\n\nUse this pattern when optimizing an artifact against a stable evaluator:\n\n```text\nread locked context -> choose hypothesis -> edit allowed surface -> commit/checkpoint\n-> run evaluator -> log result -> keep if better -> discard or rollback if worse\n-> repeat\n```\n\nRequired properties:\n\n- The evaluator is outside the editable surface.\n- The feedback cadence is fixed enough to compare attempts.\n- Failed attempts leave an audit trail.\n- Rollback is cheap.\n- The agent has a policy for crashes and timeouts.\n\n### Research-To-Skill Loop\n\nUse this pattern when sources become skill changes:\n\n```text\ndiscover -> retrieve -> gate -> score -> extract mechanism\n-> map to existing or new skill -> draft proposal -> validate structure\n-> prepare PR -> human review\n```\n\nThe locked evaluator is a combination of source rubrics, skill-change rubrics, structure checks, and reviewer approval. The editable artifact is the proposed skill delta.\n\n### Metric Gaming Resistance\n\nAssume an optimizing agent will learn the harness. Guard against:\n\n- Editing evaluation code or rubrics and then using the new version for self-approval\n- Adding verbose content that pleases a judge but harms skill activation\n- Citing unretrieved sources\n- Optimizing aggregate scores while failing a critical dimension\n- Avoiding failed results in the log\n\nMitigation: lock rubrics per run, report per-dimension scores, require source retrieval evidence, preserve rejected attempts, and route governance changes to human review.\n\n### Monitoring Agents\n\nUse monitoring agents for long runs, but restrict them to read-only reporting unless explicitly tasked otherwise. Monitoring output should report:\n\n- Best current candidate\n- Active jobs or drafts\n- Last upstream refresh\n- Failed or stale loops\n- Disagreements between logs and claimed state\n- Next action and blocker\n\n## Practical Guidance\n\n### Harness Design Checklist\n\n1. Define the objective in one sentence.\n2. Identify locked, editable, append-only, and human-controlled surfaces.\n3. Choose the feedback mechanism: scalar metric, rubric, deterministic tests, human review, or combination.\n4. Define keep, discard, crash, timeout, and review states.\n5. Create a durable thread log before the loop starts.\n6. Add source refresh, mechanism-registry novelty, and pruning rules for long-running loops.\n7. Define what the agent may do without asking and what requires approval.\n8. Validate the harness on one known good and one known bad artifact.\n\n### File Layout\n\n```text\nresearch-run/\n THREAD.md\n sources/\n queue.md\n evaluations/\n proposals/\n logs/\n results.tsv\n rejected.md\n drafts/\n```\n\nUse TSV or JSONL for append-only machine-readable logs. Use Markdown for handoffs and reviewer-facing summaries.\n\n## Examples\n\n**Example 1: Locked metric**\n\nAn agent optimizes `train.py`, but `prepare.py` owns data loading and evaluation. The agent can edit the model but cannot change the metric. Failed experiments are logged and rolled back.\n\n**Example 2: Locked rubric**\n\nAn agent evaluates a new Anthropic or OpenAI engineering post, but the source curation rubric is locked for the run. If the source passes, the agent drafts a skill proposal. It cannot lower the rubric threshold to admit the source.\n\n**Example 3: Auto-PR without auto-merge**\n\nAn agent prepares a branch and PR body after passing source, skill, and structure checks. The PR states unresolved risks and waits for human merge approval.\n\n## Guidelines\n\n1. Lock evaluators before starting the loop.\n2. Keep editable surfaces narrow enough for reliable diffs.\n3. Write durable logs before context compaction can erase state.\n4. Report per-dimension scores instead of only aggregate scores.\n5. Require source retrieval before citation.\n6. Add novelty gates for broad search and pruning gates for complex stacks.\n7. Prefer simplification when quality is equal.\n8. Separate PR preparation from merge authority.\n9. Revalidate harness changes with old and new evaluators.\n10. Treat stopped autonomous loops as harness failures, not agent personality quirks.\n\n## Gotchas\n\n1. **Mutable evaluator**: If the agent can edit the metric, it may optimize the benchmark instead of the task. Keep rubrics and eval code locked during the run.\n2. **Chat-only memory**: Long runs fail after compaction when plans live only in conversation history. Write thread logs and result files from the start.\n3. **No discard record**: Without rejected-attempt logs, agents repeat failed ideas. Preserve failures with enough detail to avoid rediscovery.\n4. **Complexity accretion**: Agents stack changes and rarely remove them. Require pruning rounds and reward equal-quality simplification.\n5. **Premature novelty claims**: Agents label recombinations as novel. Compare against existing repo skills, source queue, and rejected logs before claiming novelty.\n6. **Monitor misreporting**: Monitoring agents can summarize stale or inconsistent state. Require them to cite the files or logs behind claims.\n7. **Human approval ambiguity**: \"Prepare a PR\" is not \"merge a PR.\" Make approval boundaries explicit in the harness.\n8. **Volatile source drift**: Fast-moving lab claims age quickly. Put dated evidence in references and schedule revalidation.\n\n## Integration\n\nThis skill connects to:\n\n- evaluation - Rubrics and quality gates provide the locked feedback surface\n- advanced-evaluation - Pairwise comparison and bias mitigation improve proposal review\n- filesystem-context - Durable logs, scratchpads, and thread files preserve state\n- multi-agent-patterns - Researcher, verifier, monitor, and writer agents need isolated contexts\n- tool-design - Harness tools must expose clear contracts and recovery errors\n- project-development - File-based pipelines and task-model fit analysis keep loops simple\n- hosted-agents - Background execution needs sandbox, snapshot, and approval boundaries\n\n## References\n\nInternal references:\n- `researcher/README.md` - Read when implementing the repo-native research-to-skill operating system\n- `researcher/rubrics/harness-change.md` - Read when evaluating changes to an agent harness\n- `researcher/runbooks/autonomous-research-loop.md` - Read when running a source-to-skill loop\n\nExternal resources:\n- Karpathy `autoresearch` - Constrained autonomous experiment loop with locked evaluation\n- Prime Intellect autonomous nanoGPT speedrun - Durable scratchpads, handoffs, monitoring, and autonomy failure modes\n- AlphaEvolve and FunSearch - LLM-generated candidates paired with systematic evaluators\n- HELM and LM Evaluation Harness - Transparent, reproducible evaluation infrastructure\n\n---\n\n## Skill Metadata\n\n**Created**: 2026-05-14\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 1.1.0\n","skills/hosted-agents/SKILL.md":"---\nname: hosted-agents\ndescription: \"This skill should be used when designing hosted or background agent infrastructure: sandboxed execution, remote coding environments, warm pools, session persistence, multiplayer collaboration, self-spawning agents, or Modal-style sandboxes.\"\n---\n\n# Hosted Agent Infrastructure\n\nHosted agents run in remote sandboxed environments rather than on local machines. When designed well, they provide unlimited concurrency, consistent execution environments, and multiplayer collaboration. The critical insight is that session speed should be limited only by model provider time-to-first-token, with all infrastructure setup completed before the user starts their session.\n\n## When to Activate\n\nActivate this skill when:\n- Building background coding agents that run independently of user devices\n- Designing sandboxed execution environments for agent workloads\n- Implementing multiplayer agent sessions with shared state\n- Creating multi-client agent interfaces (Slack, Web, Chrome extensions)\n- Scaling agent infrastructure beyond local machine constraints\n- Building systems where agents spawn sub-agents for parallel work\n\nDo not activate this skill for adjacent work owned by other skills:\n- Designing the autonomous research loop, novelty gates, rollback policy, or merge boundaries: `harness-engineering`.\n- Choosing supervisor, swarm, or handoff topology without hosted infrastructure concerns: `multi-agent-patterns`.\n- Designing the tools used by a hosted agent, such as spawn/status tools or PR tools: `tool-design`.\n- Managing file-backed state inside a session rather than the hosted runtime itself: `filesystem-context`.\n\n## Core Concepts\n\nMove agent execution to remote sandboxed environments to eliminate the fundamental limits of local execution: resource contention, environment inconsistency, and single-user constraints. Remote sandboxes unlock unlimited concurrency, reproducible environments, and collaborative workflows because each session gets its own isolated compute with a known-good environment image.\n\nDesign the architecture in three layers because each layer scales independently. Build sandbox infrastructure for isolated execution, an API layer for state management and client coordination, and client interfaces for user interaction across platforms. Keep these layers cleanly separated so sandbox changes do not ripple into clients.\n\n## Detailed Topics\n\n### Sandbox Infrastructure\n\n**The Core Challenge**\nEliminate sandbox spin-up latency because users perceive anything over a few seconds as broken. Development environments require cloning repositories, installing dependencies, and running build steps -- do all of this before the user ever submits a prompt.\n\n**Image Registry Pattern**\nPre-build environment images on a regular cadence (every 30 minutes works well) because this makes synchronization with the latest code a fast delta rather than a full clone. Include in each image:\n- Cloned repository at a known commit\n- All runtime dependencies installed\n- Initial setup and build commands completed\n- Cached files from running app and test suite once\n\nWhen starting a session, spin up a sandbox from the most recent image. The repository is at most 30 minutes out of date, making the remaining git sync fast.\n\n**Snapshot and Restore**\nTake filesystem snapshots at key points to enable instant restoration for follow-up prompts without re-running setup:\n- After initial image build (base snapshot)\n- When agent finishes making changes (session snapshot)\n- Before sandbox exit for potential follow-up\n\n**Git Configuration for Background Agents**\nConfigure git identity explicitly in every sandbox because background agents are not tied to a specific user during image builds:\n- Generate GitHub app installation tokens for repository access during clone\n- Set git config `user.name` and `user.email` when committing and pushing changes\n- Use the prompting user's identity for commits, not the app identity\n\n**Warm Pool Strategy**\nMaintain a pool of pre-warmed sandboxes for high-volume repositories because cold starts are the primary source of user frustration:\n- Keep sandboxes ready before users start sessions\n- Expire and recreate pool entries as new image builds complete\n- Start warming a sandbox as soon as a user begins typing (predictive warm-up)\n\n### Agent Framework Selection\n\n**Server-First Architecture**\nStructure the agent framework as a server first, with TUI and desktop apps as thin clients, because this prevents duplicating agent logic across surfaces:\n- Multiple custom clients share one agent backend\n- Consistent behavior across all interaction surfaces\n- Plugin systems extend functionality without client changes\n- Event-driven architectures deliver real-time updates to any connected client\n\n**Code as Source of Truth**\nSelect frameworks where the agent can read its own source code to understand behavior. Prioritize this because having code as source of truth prevents the agent from hallucinating about its own capabilities -- an underrated failure mode in AI development.\n\n**Plugin System Requirements**\nRequire a plugin system that supports runtime interception because this enables safety controls and observability without modifying core agent logic:\n- Listen to tool execution events (e.g., `tool.execute.before`)\n- Block or modify tool calls conditionally\n- Inject context or state at runtime\n\n### Speed Optimizations\n\n**Predictive Warm-Up**\nStart warming the sandbox as soon as a user begins typing their prompt, not when they submit it, because the typing interval (5-30 seconds) is enough to complete most setup:\n- Clone latest changes in parallel with user typing\n- Run initial setup before user hits enter\n- For fast spin-up, sandbox can be ready before user finishes typing\n\n**Parallel File Reading**\nAllow the agent to start reading files immediately even if sync from latest base branch is not complete, because in large repositories incoming prompts rarely touch recently-changed files:\n- Agent can research immediately without waiting for git sync\n- Block file edits (not reads) until synchronization completes\n- This separation is safe because read-time data staleness of 30 minutes rarely matters for research\n\n**Maximize Build-Time Work**\nMove everything possible to the image build step because build-time duration is invisible to users:\n- Full dependency installation\n- Database schema setup\n- Initial app and test suite runs (populates caches)\n\n### Self-Spawning Agents\n\n**Agent-Spawned Sessions**\nBuild tools that allow agents to spawn new sessions because frontier models are capable of decomposing work and coordinating sub-tasks:\n- Research tasks across different repositories\n- Parallel subtask execution for large changes\n- Multiple smaller PRs from one major task\n\nExpose three primitives: start a new session with specified parameters, read status of any session (check-in capability), and continue main work while sub-sessions run in parallel.\n\n**Prompt Engineering for Self-Spawning**\nEngineer prompts that guide when agents should spawn sub-sessions rather than doing work inline:\n- Research tasks that require cross-repository exploration\n- Breaking monolithic changes into smaller PRs\n- Parallel exploration of different approaches\n\n### API Layer\n\n**Per-Session State Isolation**\nIsolate state per session (SQLite per session works well) because cross-session interference is a subtle and hard-to-debug failure mode:\n- Dedicated database per session\n- No session can impact another's performance\n- Architecture handles hundreds of concurrent sessions\n\n**Real-Time Streaming**\nStream all agent work in real-time because high-frequency feedback is critical for user trust:\n- Token streaming from model providers\n- Tool execution status updates\n- File change notifications\n\nUse WebSocket connections with hibernation APIs to reduce compute costs during idle periods while maintaining open connections.\n\n**Synchronization Across Clients**\nBuild a single state system that synchronizes across all clients (chat interfaces, Slack bots, Chrome extensions, web interfaces, VS Code instances) because users switch surfaces frequently and expect continuity. All changes sync to the session state, enabling seamless client switching.\n\n### Multiplayer Support\n\n**Why Multiplayer Matters**\nDesign for multiplayer from day one because it is nearly free to add with proper synchronization architecture, and it unlocks high-value workflows:\n- Teaching non-engineers to use AI effectively\n- Live QA sessions with multiple team members\n- Real-time PR review with immediate changes\n- Collaborative debugging sessions\n\n**Implementation Requirements**\nBuild the data model so sessions are not tied to single authors because multiplayer fails silently if authorship is hardcoded:\n- Pass authorship info to each prompt\n- Attribute code changes to the prompting user\n- Share session links for instant collaboration\n\n### Authentication and Authorization\n\n**User-Based Commits**\nUse GitHub authentication to open PRs on behalf of the user (not the app) because this preserves the audit trail and prevents users from approving their own AI-generated changes:\n- Obtain user tokens for PR creation\n- PRs appear as authored by the human, not the bot\n\n**Sandbox-to-API Flow**\nFollow this sequence because it keeps sandbox permissions minimal while letting the API handle sensitive operations:\n1. Sandbox pushes changes (updating git user config)\n2. Sandbox sends event to API with branch name and session ID\n3. API uses user's GitHub token to create PR\n4. GitHub webhooks notify API of PR events\n\n### Client Implementations\n\n**Slack Integration**\nPrioritize Slack as the first distribution channel for internal adoption because it creates a virality loop as team members see others using it:\n- No syntax required, natural chat interface\n- Build a classifier (fast model with repo descriptions) to determine which repository to work in\n- Include hints for common repositories; allow \"unknown\" for ambiguous cases\n\n**Web Interface**\nBuild a web interface with these features because it serves as the primary power-user surface:\n- Real-time streaming of agent work on desktop and mobile\n- Hosted VS Code instance running inside sandbox\n- Streamed desktop view for visual verification\n- Before/after screenshots for PRs\n- Statistics page: sessions resulting in merged PRs (primary metric), usage over time, live \"humans prompting\" count\n\n**Chrome Extension**\nBuild a Chrome extension for non-engineering users because DOM and React internals extraction gives higher precision than raw screenshots at lower token cost:\n- Sidebar chat interface with screenshot tool\n- Extract DOM/React internals instead of raw images\n- Distribute via managed device policy (bypasses Chrome Web Store)\n\n## Practical Guidance\n\n### Hosted Agent Design Checklist\n\nBefore building the system, decide these infrastructure properties explicitly:\n\n1. **Sandbox lifecycle**: how sessions start, snapshot, restore, time out, and terminate.\n2. **Image and warm-pool policy**: how often images rebuild, which caches are precomputed, and when warm sandboxes expire.\n3. **Read/write synchronization**: whether agents may read before repository sync completes, and which events unblock writes.\n4. **Per-session state isolation**: what storage belongs to one session, what is shared, and how cross-session interference is prevented.\n5. **Auth and commit identity**: which operations use app tokens, which use user tokens, and how commits are attributed.\n6. **Output extraction**: how branches, PRs, files, logs, screenshots, and session summaries leave the sandbox.\n7. **Budget and teardown**: maximum runtime, cost ceilings, idle policy, and forced cleanup behavior.\n\n### Follow-Up Message Handling\n\nChoose between queueing and inserting follow-up messages sent during execution. Prefer queueing because it is simpler to manage and lets users send thoughts on next steps while the agent works. Build a mechanism to stop the agent mid-execution when needed, because without it users feel trapped.\n\n### Metrics That Matter\n\nTrack these metrics because they indicate real value rather than vanity usage:\n- Sessions resulting in merged PRs (primary success metric)\n- Time from session start to first model response\n- PR approval rate and revision count\n- Agent-written code percentage across repositories\n\n### Adoption Strategy\n\nDrive internal adoption through visibility rather than mandates because forced usage breeds resentment:\n- Work in public spaces (Slack channels) for visibility\n- Let the product create virality loops\n- Do not force usage over existing tools\n- Build to people's needs, not hypothetical requirements\n\n## Examples\n\n**Example 1: Background coding session lifecycle**\n\n```text\nuser prompt\n-> API allocates warm sandbox from current image\n-> sandbox syncs latest branch delta\n-> reads allowed immediately, writes blocked until sync completes\n-> agent edits and tests inside isolated workspace\n-> sandbox snapshots final filesystem\n-> branch is pushed\n-> API creates PR using user token\n-> session summary, logs, and PR URL are returned to clients\n```\n\nThis sequence keeps setup work outside the user-visible path while preserving auditability and user ownership of code changes.\n\n**Example 2: Boundary decision**\n\nIf the task is \"make the agent loop run for days with locked rubrics and PR approval,\" use `harness-engineering`. If the task is \"run that loop in remote sandboxes with warm pools, session snapshots, streaming clients, and user-authored PRs,\" use this skill.\n\n## Guidelines\n\n1. Pre-build environment images on regular cadence (30 minutes is a good default)\n2. Start warming sandboxes when users begin typing, not when they submit\n3. Allow file reads before git sync completes; block only writes\n4. Structure agent framework as server-first with clients as thin wrappers\n5. Isolate state per session to prevent cross-session interference\n6. Attribute commits to the user who prompted, not the app\n7. Track merged PRs as primary success metric\n8. Build for multiplayer from the start; it is nearly free with proper sync architecture\n\n## Gotchas\n\n1. **Cold start latency**: First sandbox spin-up takes 30-60s and users perceive this as broken. Use warm pools and predictive warm-up on keystroke to eliminate perceived wait time.\n2. **Image staleness**: Infrequent image rebuilds mean agents run with outdated dependencies or code. Set a 30-minute rebuild cadence and monitor image age; alert if builds fail silently.\n3. **Sandbox cost runaway**: Long-running agents without timeout or budget caps accumulate unexpected costs. Set hard timeout limits (default 4 hours) and per-session cost ceilings.\n4. **Auth token expiration mid-session**: Long tasks fail when GitHub tokens expire partway through. Implement token refresh logic and check token validity before sensitive operations like PR creation.\n5. **Git config in sandboxes**: Missing `user.name` or `user.email` causes commit failures in background agents. Always set git identity explicitly during sandbox configuration, never assume it carries over from the image.\n6. **State loss on sandbox recycle**: Agents lose completed work if the sandbox is recycled or times out before results are extracted. Always snapshot before termination and extract artifacts (branches, PRs, files) before letting the sandbox die.\n7. **Oversubscribing warm pools**: Maintaining too many warm sandboxes wastes money during low-traffic periods. Scale pool size based on traffic patterns and time-of-day; use autoscaling rather than fixed pool sizes.\n8. **Missing output extraction**: Agents complete work inside the sandbox but results never get pulled out to the user. Build explicit extraction steps (push branch, create PR, return file contents) into the session teardown flow.\n\n## Integration\n\nThis skill owns hosted runtime infrastructure. Adjacent skills own the control system, topology, and tool contracts:\n\n- `harness-engineering`: governance, locked evaluators, rollback, novelty gates, and human approval boundaries around autonomous work.\n- `multi-agent-patterns`: self-spawning and supervisor patterns once hosted infrastructure exists.\n- `tool-design`: spawn, status, teardown, and PR tools exposed to agents.\n- `context-optimization`: managing context across distributed hosted sessions.\n- `filesystem-context`: using the sandbox filesystem for durable session state and artifacts.\n\n## References\n\nInternal reference:\n- [Infrastructure Patterns](./references/infrastructure-patterns.md) - Read when: implementing sandbox lifecycle, image builds, or warm pool logic for the first time\n\nRelated skills in this collection:\n- multi-agent-patterns - Read when: designing self-spawning or supervisor coordination patterns\n- tool-design - Read when: building tools for agent session management or status checking\n- context-optimization - Read when: context windows fill up across distributed agent sessions\n\nExternal resources:\n- [Ramp](https://builders.ramp.com/post/why-we-built-our-background-agent) - Read when: evaluating whether to build vs. buy background agent infrastructure\n- [Modal Sandboxes](https://modal.com/docs/guide/sandbox) - Read when: choosing a cloud sandbox provider or comparing isolation models\n- [Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/) - Read when: designing per-session state management with WebSocket hibernation\n- [OpenCode](https://github.com/sst/opencode) - Read when: selecting a server-first agent framework or studying plugin architectures\n\n---\n\n## Skill Metadata\n\n**Created**: 2026-01-12\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 1.2.0\n","skills/hosted-agents/references/infrastructure-patterns.md":"# Infrastructure Patterns for Hosted Agents\n\nThis reference provides detailed implementation patterns for building hosted agent infrastructure. These patterns are derived from production systems at scale.\n\n## Sandbox Architecture\n\n### Modal Integration Pattern\n\nModal provides the sandbox infrastructure with near-instant startup and filesystem snapshots.\n\n```python\nimport modal\n\n# Define the base image with all dependencies\nimage = modal.Image.debian_slim().pip_install([\n \"opencode\",\n \"gitpython\",\n \"psycopg2-binary\",\n])\n\n# Create the app\napp = modal.App(\"coding-agent\")\n\n# Sandbox class with snapshot support\n@app.cls(image=image, timeout=3600)\nclass AgentSandbox:\n def __init__(self, repo_url: str, snapshot_id: str = None):\n self.repo_url = repo_url\n self.snapshot_id = snapshot_id\n \n @modal.enter()\n def setup(self):\n if self.snapshot_id:\n # Restore from snapshot\n modal.Sandbox.restore(self.snapshot_id)\n else:\n # Fresh setup from image\n self._clone_and_setup()\n \n def _clone_and_setup(self):\n \"\"\"Clone repo and run initial setup.\"\"\"\n token = self._get_github_app_token()\n os.system(f\"git clone https://x-access-token:{token}@github.com/{self.repo_url}\")\n os.system(\"npm install\")\n os.system(\"npm run build\")\n \n @modal.method()\n def execute_prompt(self, prompt: str, user_identity: dict) -> dict:\n \"\"\"Execute a prompt in the sandbox.\"\"\"\n # Update git config for this user\n os.system(f'git config user.name \"{user_identity[\"name\"]}\"')\n os.system(f'git config user.email \"{user_identity[\"email\"]}\"')\n \n # Run the agent\n result = self.agent.run(prompt)\n \n return {\n \"result\": result,\n \"snapshot_id\": modal.Sandbox.snapshot()\n }\n```\n\n### Image Build Pipeline\n\nBuild images on a schedule to keep them fresh:\n\n```python\nimport schedule\nimport time\nfrom datetime import datetime\n\nclass ImageBuilder:\n def __init__(self, repositories: list[str]):\n self.repositories = repositories\n self.images = {}\n \n def build_all_images(self):\n \"\"\"Build images for all repositories.\"\"\"\n for repo in self.repositories:\n try:\n image = self._build_image(repo)\n self.images[repo] = {\n \"image\": image,\n \"built_at\": datetime.utcnow(),\n \"commit\": self._get_latest_commit(repo)\n }\n except Exception as e:\n # Log but continue with other repos\n log.error(f\"Failed to build image for {repo}: {e}\")\n \n def _build_image(self, repo: str) -> str:\n \"\"\"Build a single repository image.\"\"\"\n sandbox = modal.Sandbox.create()\n \n # Clone with app token\n token = get_app_installation_token(repo)\n sandbox.exec(f\"git clone https://x-access-token:{token}@github.com/{repo} /workspace\")\n \n # Install dependencies\n sandbox.exec(\"cd /workspace && npm install\")\n \n # Run build\n sandbox.exec(\"cd /workspace && npm run build\")\n \n # Warm caches\n sandbox.exec(\"cd /workspace && npm run dev &\")\n time.sleep(5) # Let dev server start\n sandbox.exec(\"cd /workspace && npm test -- --run\")\n \n # Create snapshot\n return sandbox.snapshot()\n \n def get_latest_image(self, repo: str) -> str:\n \"\"\"Get the most recent image for a repository.\"\"\"\n if repo not in self.images:\n raise ValueError(f\"No image available for {repo}\")\n return self.images[repo][\"image\"]\n\n# Schedule builds every 30 minutes\nbuilder = ImageBuilder([\"org/frontend\", \"org/backend\", \"org/shared\"])\nschedule.every(30).minutes.do(builder.build_all_images)\n```\n\n### Warm Pool Management\n\nMaintain pre-warmed sandboxes for instant session starts:\n\n```python\nfrom collections import defaultdict\nfrom dataclasses import dataclass\nfrom datetime import datetime, timedelta\n\n@dataclass\nclass WarmSandbox:\n sandbox_id: str\n repo: str\n created_at: datetime\n image_version: str\n is_claimed: bool = False\n\nclass WarmPoolManager:\n def __init__(self, target_pool_size: int = 3):\n self.target_size = target_pool_size\n self.pools = defaultdict(list) # repo -> [WarmSandbox]\n self.max_age = timedelta(minutes=25) # Expire before next image build\n \n def get_warm_sandbox(self, repo: str) -> WarmSandbox | None:\n \"\"\"Get a pre-warmed sandbox if available.\"\"\"\n pool = self.pools[repo]\n \n for sandbox in pool:\n if not sandbox.is_claimed and self._is_valid(sandbox):\n sandbox.is_claimed = True\n return sandbox\n \n return None\n \n def _is_valid(self, sandbox: WarmSandbox) -> bool:\n \"\"\"Check if sandbox is still valid.\"\"\"\n age = datetime.utcnow() - sandbox.created_at\n current_image = self.image_builder.get_latest_image(sandbox.repo)\n \n return (\n age < self.max_age and\n sandbox.image_version == current_image\n )\n \n def maintain_pool(self, repo: str):\n \"\"\"Ensure pool has target number of warm sandboxes.\"\"\"\n # Remove expired sandboxes\n self.pools[repo] = [s for s in self.pools[repo] if self._is_valid(s)]\n \n # Add new sandboxes to reach target\n current_count = len([s for s in self.pools[repo] if not s.is_claimed])\n needed = self.target_size - current_count\n \n for _ in range(needed):\n sandbox = self._create_warm_sandbox(repo)\n self.pools[repo].append(sandbox)\n \n def _create_warm_sandbox(self, repo: str) -> WarmSandbox:\n \"\"\"Create a new warm sandbox from latest image.\"\"\"\n image = self.image_builder.get_latest_image(repo)\n sandbox_id = modal.Sandbox.create(image=image)\n \n # Sync to latest (runs in background)\n self._sync_to_latest(sandbox_id, repo)\n \n return WarmSandbox(\n sandbox_id=sandbox_id,\n repo=repo,\n created_at=datetime.utcnow(),\n image_version=image\n )\n```\n\n## API Layer Patterns\n\n### Cloudflare Durable Objects for Session State\n\nEach session gets its own Durable Object with isolated SQLite:\n\n```typescript\n// Session Durable Object\nexport class SessionDO implements DurableObject {\n private storage: DurableObjectStorage;\n private sql: SqlStorage;\n private connections: Map = new Map();\n\n constructor(ctx: DurableObjectState) {\n this.storage = ctx.storage;\n this.sql = ctx.storage.sql;\n this.initializeSchema();\n }\n\n private initializeSchema() {\n this.sql.exec(`\n CREATE TABLE IF NOT EXISTS messages (\n id INTEGER PRIMARY KEY,\n role TEXT NOT NULL,\n content TEXT NOT NULL,\n author_id TEXT,\n author_name TEXT,\n created_at TEXT DEFAULT CURRENT_TIMESTAMP\n );\n \n CREATE TABLE IF NOT EXISTS artifacts (\n id INTEGER PRIMARY KEY,\n type TEXT NOT NULL,\n path TEXT,\n content TEXT,\n created_at TEXT DEFAULT CURRENT_TIMESTAMP\n );\n \n CREATE TABLE IF NOT EXISTS events (\n id INTEGER PRIMARY KEY,\n type TEXT NOT NULL,\n data TEXT,\n created_at TEXT DEFAULT CURRENT_TIMESTAMP\n );\n `);\n }\n\n async fetch(request: Request): Promise {\n const url = new URL(request.url);\n\n if (request.headers.get(\"Upgrade\") === \"websocket\") {\n return this.handleWebSocket(request);\n }\n\n switch (url.pathname) {\n case \"/message\":\n return this.handleMessage(request);\n case \"/status\":\n return this.getStatus();\n default:\n return new Response(\"Not found\", { status: 404 });\n }\n }\n\n private handleWebSocket(request: Request): Response {\n const pair = new WebSocketPair();\n const [client, server] = Object.values(pair);\n\n const connectionId = crypto.randomUUID();\n this.connections.set(connectionId, server);\n\n server.accept();\n server.addEventListener(\"close\", () => {\n this.connections.delete(connectionId);\n });\n\n return new Response(null, { status: 101, webSocket: client });\n }\n\n private broadcast(message: object) {\n const data = JSON.stringify(message);\n for (const ws of this.connections.values()) {\n ws.send(data);\n }\n }\n\n async handleMessage(request: Request): Promise {\n const { content, author } = await request.json();\n\n // Store message\n this.sql.exec(\n `INSERT INTO messages (role, content, author_id, author_name) VALUES (?, ?, ?, ?)`,\n [\"user\", content, author.id, author.name]\n );\n\n // Broadcast to all connected clients\n this.broadcast({\n type: \"message\",\n role: \"user\",\n content,\n author,\n });\n\n // Forward to sandbox for processing\n const result = await this.forwardToSandbox(content, author);\n\n return Response.json(result);\n }\n}\n```\n\n### Real-Time Event Streaming\n\nStream events from sandbox to all connected clients:\n\n```typescript\nclass EventStream {\n private sessionDO: DurableObjectStub;\n\n async streamFromSandbox(sandboxId: string, sessionId: string) {\n const sandbox = await modal.Sandbox.get(sandboxId);\n\n // Subscribe to sandbox events\n for await (const event of sandbox.events()) {\n // Forward to Durable Object for broadcast\n await this.sessionDO.fetch(\n new Request(`https://internal/event`, {\n method: \"POST\",\n body: JSON.stringify({\n type: event.type,\n data: event.data,\n }),\n })\n );\n }\n }\n}\n```\n\n## Client Integration Patterns\n\n### Slack Bot with Repository Classification\n\n```python\nfrom slack_bolt import App\nfrom slack_bolt.adapter.socket_mode import SocketModeHandler\n\napp = App(token=os.environ[\"SLACK_BOT_TOKEN\"])\n\n# Repository descriptions for classification\nREPO_DESCRIPTIONS = [\n {\n \"name\": \"frontend-monorepo\",\n \"description\": \"React frontend application with dashboard, user portal, and admin interfaces\",\n \"hints\": [\"dashboard\", \"UI\", \"component\", \"page\", \"frontend\"]\n },\n {\n \"name\": \"backend-services\",\n \"description\": \"Node.js API services including auth, payments, and core business logic\",\n \"hints\": [\"API\", \"endpoint\", \"service\", \"backend\", \"database\"]\n },\n {\n \"name\": \"mobile-app\",\n \"description\": \"React Native mobile application for iOS and Android\",\n \"hints\": [\"mobile\", \"app\", \"iOS\", \"Android\", \"native\"]\n }\n]\n\nasync def classify_repository(message: str, channel: str, thread: list[str]) -> str:\n \"\"\"Use fast model to classify which repo the message refers to.\"\"\"\n prompt = f\"\"\"Classify which repository this message is about.\n\nMessage: {message}\nChannel: #{channel}\nThread context: {' | '.join(thread[-3:])}\n\nRepositories:\n{json.dumps(REPO_DESCRIPTIONS, indent=2)}\n\nReturn ONLY the repository name, or \"unknown\" if unclear.\"\"\"\n\n response = await openai.chat.completions.create(\n model=\"gpt-4o-mini\",\n messages=[{\"role\": \"user\", \"content\": prompt}],\n max_tokens=50\n )\n \n return response.choices[0].message.content.strip()\n\n@app.event(\"app_mention\")\nasync def handle_mention(event, say, client):\n \"\"\"Handle @mentions of the bot.\"\"\"\n channel = event[\"channel\"]\n message = event[\"text\"]\n thread_ts = event.get(\"thread_ts\", event[\"ts\"])\n \n # Get thread context if in a thread\n thread_messages = []\n if \"thread_ts\" in event:\n result = await client.conversations_replies(\n channel=channel,\n ts=thread_ts\n )\n thread_messages = [m[\"text\"] for m in result[\"messages\"]]\n \n # Get channel info for context\n channel_info = await client.conversations_info(channel=channel)\n channel_name = channel_info[\"channel\"][\"name\"]\n \n # Classify repository\n repo = await classify_repository(message, channel_name, thread_messages)\n \n if repo == \"unknown\":\n await say(\n text=\"I'm not sure which repository you're referring to. Could you specify?\",\n thread_ts=thread_ts\n )\n return\n \n # Start session and process\n session = await start_session(repo, event[\"user\"])\n \n await say(\n text=f\":robot_face: Starting work in `{repo}`...\",\n thread_ts=thread_ts\n )\n \n result = await session.process(message)\n \n # Post result with Block Kit formatting\n await say(\n blocks=format_result_blocks(result),\n thread_ts=thread_ts\n )\n```\n\n### Chrome Extension DOM Extraction\n\nExtract DOM structure instead of sending screenshots:\n\n```typescript\n// content-script.ts\ninterface ElementInfo {\n tag: string;\n classes: string[];\n id?: string;\n text?: string;\n rect: DOMRect;\n reactComponent?: string;\n}\n\nfunction extractDOMInfo(element: Element): ElementInfo {\n // Get React component name if available\n let reactComponent: string | undefined;\n const fiberKey = Object.keys(element).find((key) =>\n key.startsWith(\"__reactFiber\")\n );\n if (fiberKey) {\n const fiber = (element as any)[fiberKey];\n reactComponent = fiber?.type?.name || fiber?.type?.displayName;\n }\n\n return {\n tag: element.tagName.toLowerCase(),\n classes: Array.from(element.classList),\n id: element.id || undefined,\n text: element.textContent?.slice(0, 100),\n rect: element.getBoundingClientRect(),\n reactComponent,\n };\n}\n\nfunction extractSelectedArea(selection: DOMRect): ElementInfo[] {\n const elements: ElementInfo[] = [];\n\n // Find all elements within selection bounds\n document.querySelectorAll(\"*\").forEach((el) => {\n const rect = el.getBoundingClientRect();\n if (\n rect.top >= selection.top &&\n rect.left >= selection.left &&\n rect.bottom <= selection.bottom &&\n rect.right <= selection.right\n ) {\n elements.push(extractDOMInfo(el));\n }\n });\n\n return elements;\n}\n\n// Message handler for sidebar\nchrome.runtime.onMessage.addListener((request, sender, sendResponse) => {\n if (request.type === \"EXTRACT_SELECTION\") {\n const elements = extractSelectedArea(request.selection);\n sendResponse({ elements });\n }\n});\n```\n\n## Multiplayer Implementation\n\n### Authorship Tracking\n\nTrack which user made each change:\n\n```python\n@dataclass\nclass PromptContext:\n content: str\n author: Author\n session_id: str\n timestamp: datetime\n\n@dataclass\nclass Author:\n id: str\n name: str\n email: str\n github_token: str # For PR creation\n\nclass MultiplayerSession:\n def __init__(self, session_id: str):\n self.session_id = session_id\n self.participants: dict[str, Author] = {}\n self.prompt_queue: list[PromptContext] = []\n \n def add_participant(self, author: Author):\n \"\"\"Add a participant to the session.\"\"\"\n self.participants[author.id] = author\n self.broadcast_event(\"participant_joined\", author)\n \n async def process_prompt(self, prompt: PromptContext):\n \"\"\"Process prompt with author attribution.\"\"\"\n # Update git config for this author\n await self.sandbox.exec(\n f'git config user.name \"{prompt.author.name}\"'\n )\n await self.sandbox.exec(\n f'git config user.email \"{prompt.author.email}\"'\n )\n \n # Run agent\n result = await self.agent.run(prompt.content)\n \n # If changes were made, create PR with author's token\n if result.has_changes:\n await self.create_pr(\n branch=result.branch,\n author=prompt.author\n )\n \n return result\n \n async def create_pr(self, branch: str, author: Author):\n \"\"\"Create PR using the author's GitHub token.\"\"\"\n async with aiohttp.ClientSession() as session:\n headers = {\n \"Authorization\": f\"Bearer {author.github_token}\",\n \"Accept\": \"application/vnd.github.v3+json\"\n }\n \n await session.post(\n f\"https://api.github.com/repos/{self.repo}/pulls\",\n headers=headers,\n json={\n \"title\": self.generate_pr_title(),\n \"body\": self.generate_pr_body(),\n \"head\": branch,\n \"base\": \"main\"\n }\n )\n```\n\n## Metrics and Monitoring\n\n### Key Metrics to Track\n\n```python\nfrom dataclasses import dataclass\nfrom datetime import datetime, timedelta\n\n@dataclass\nclass SessionMetrics:\n session_id: str\n started_at: datetime\n first_token_at: datetime | None\n completed_at: datetime | None\n pr_created: bool\n pr_merged: bool\n prompts_count: int\n participants_count: int\n \n @property\n def time_to_first_token(self) -> timedelta | None:\n if self.first_token_at:\n return self.first_token_at - self.started_at\n return None\n\nclass MetricsAggregator:\n def get_adoption_metrics(self, period: timedelta) -> dict:\n \"\"\"Get adoption metrics for a time period.\"\"\"\n sessions = self.get_sessions_in_period(period)\n \n total_prs = sum(1 for s in sessions if s.pr_created)\n merged_prs = sum(1 for s in sessions if s.pr_merged)\n \n return {\n \"total_sessions\": len(sessions),\n \"prs_created\": total_prs,\n \"prs_merged\": merged_prs,\n \"merge_rate\": merged_prs / total_prs if total_prs > 0 else 0,\n \"avg_time_to_first_token\": self._avg_ttft(sessions),\n \"unique_users\": len(set(s.author_id for s in sessions)),\n \"multiplayer_sessions\": sum(\n 1 for s in sessions if s.participants_count > 1\n )\n }\n \n def get_repository_metrics(self) -> dict[str, dict]:\n \"\"\"Get metrics broken down by repository.\"\"\"\n metrics = {}\n \n for repo in self.repositories:\n repo_sessions = self.get_sessions_for_repo(repo)\n total_prs = self.get_total_prs(repo)\n agent_prs = sum(1 for s in repo_sessions if s.pr_merged)\n \n metrics[repo] = {\n \"agent_pr_percentage\": agent_prs / total_prs * 100,\n \"session_count\": len(repo_sessions),\n \"avg_prompts_per_session\": sum(\n s.prompts_count for s in repo_sessions\n ) / len(repo_sessions)\n }\n \n return metrics\n```\n\n## Security Considerations\n\n### Sandbox Isolation\n\n```python\nclass SandboxSecurityConfig:\n \"\"\"Security configuration for sandboxes.\"\"\"\n \n # Network restrictions\n allowed_hosts = [\n \"github.com\",\n \"api.github.com\",\n \"registry.npmjs.org\",\n \"pypi.org\",\n ]\n \n # Resource limits\n max_memory_mb = 4096\n max_cpu_cores = 2\n max_disk_gb = 10\n max_runtime_hours = 4\n \n # Secrets handling\n secrets_to_inject = [\n \"GITHUB_APP_TOKEN\",\n \"NPM_TOKEN\",\n ]\n \n # Blocked operations\n blocked_commands = [\n \"curl\", # Use fetch tools instead\n \"wget\",\n \"ssh\",\n ]\n```\n\n### Token Handling\n\n```python\nclass TokenManager:\n \"\"\"Manage tokens for GitHub operations.\"\"\"\n \n def get_app_installation_token(self, repo: str) -> str:\n \"\"\"Get short-lived token for repo access.\"\"\"\n # Token expires in 1 hour\n return github_app.create_installation_token(\n installation_id=self.get_installation_id(repo),\n permissions={\"contents\": \"write\", \"pull_requests\": \"write\"}\n )\n \n def get_user_token(self, user_id: str) -> str:\n \"\"\"Get user's OAuth token for PR creation.\"\"\"\n # Stored encrypted, decrypted at runtime\n encrypted = self.storage.get(f\"user_token:{user_id}\")\n return self.decrypt(encrypted)\n```\n\n## References\n\n- [Modal Documentation](https://modal.com/docs)\n- [Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/)\n- [Cloudflare Agents SDK](https://developers.cloudflare.com/agents/)\n- [GitHub Apps Authentication](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app)\n- [Slack Bolt for Python](https://slack.dev/bolt-python/)\n- [Chrome Extension APIs](https://developer.chrome.com/docs/extensions/)\n","skills/latent-briefing/SKILL.md":"---\nname: latent-briefing\ndescription: \"This skill should be used when the user asks to \\\"share memory between agents\\\", \\\"KV cache compaction for multi-agent\\\", \\\"orchestrator worker context\\\", \\\"latent briefing\\\", \\\"reduce worker tokens\\\", \\\"cross-agent memory without summarization\\\", or discusses Attention Matching compaction, recursive language models with workers, or token explosion in hierarchical agents.\"\n---\n\n# Latent Briefing and KV Cache Memory Sharing\n\nHierarchical multi-agent systems often pay for the same context twice. The orchestrator accumulates a long reasoning trajectory, but each worker usually receives only a narrow text handoff such as a subtask prompt plus raw document slices. Passing the full trajectory fixes coverage but drives token cost up on every worker call. Summarization introduces latency and information loss. Retrieval helps with document access but does not preserve the orchestrator's evolving reasoning state.\n\nLatent Briefing addresses this by sharing memory at the **representation level** rather than the text level. The core idea is to compact the orchestrator trajectory in the worker model's KV cache, keeping positions that are most relevant to the **current worker task**. The method builds on **Attention Matching (AM)** KV cache compaction and adapts it for inference-time multi-agent handoff with task-guided queries, a shared token mask across heads, and robust thresholding.\n\n## When to Activate\n\nActivate this skill when:\n\n- Designing orchestrator-worker or supervisor-specialist systems where workers need access to prior orchestrator state without replaying the full trajectory as text\n- Evaluating alternatives to LLM summarization or RAG for cross-agent state transfer\n- Implementing or studying **KV cache compaction** as a first-class inference primitive, not only prefix caching of identical prompts\n- Debugging token explosion in recursive, hierarchical, or tool-heavy agent graphs\n- Interpreting benchmarks that report worker-token savings, total-token savings, compaction overhead, and accuracy together\n\nDo not activate this skill for adjacent work owned by other skills:\n- API-only stacks where internal KV tensors are inaccessible: use `context-compression`, `memory-systems`, or `multi-agent-patterns`.\n- Ordinary persistent memory, entity tracking, or graph retrieval: `memory-systems`.\n- General multi-agent topology without representation-level state sharing: `multi-agent-patterns`.\n- Prefix caching, masking, or budget policy that does not transform KV state: `context-optimization`.\n\n## Core Concepts\n\n**The token explosion pattern.** In recursive or REPL-style systems, the orchestrator repeatedly calls a worker to inspect evidence, verify hypotheses, or answer subquestions. The orchestrator's trajectory grows with partial conclusions, dead ends, tool output, and prior worker responses. If that trajectory is passed in full on every worker call, cost compounds quickly.\n\n**Representation-level sharing.** Instead of summarizing the trajectory into natural language, the system operates on the worker model's **KV cache**. It retains the positions that the worker would attend to for the current task and drops the rest. This is more specific than ordinary prefix caching: prefix caching reuses identical prefixes, while Latent Briefing also performs **task-conditioned selective retention** inside the reused trajectory.\n\n**Attention Matching as the compaction engine.** AM seeks a smaller cache whose attention outputs approximate the full cache. Latent Briefing adapts AM for multi-agent inference by changing the scoring signal and batching strategy:\n\n1. Use **task-guided query vectors** derived from the current worker prompt.\n2. Aggregate scores into a **shared global mask** instead of per-head independent subsets.\n3. Use a robust threshold such as `median + tau * MAD` rather than fixed top-k per head.\n\n**Reference result shape.** The public write-up reports substantial worker-token reduction, material total-token savings, and low-single-digit-second compaction overhead on long-document QA workloads (claim-latent-briefing-public-results). Treat these numbers as workload-specific evidence, not a general guarantee.\n\n## Detailed Topics\n\n### Why Text-Only Mitigations Fall Short\n\n| Approach | Primary weakness |\n|----------|------------------|\n| LLM summarization | High latency, lossy abstraction, and no guarantee the summary preserves what the next subtask needs |\n| Retrieval / RAG | Depends on chunking and embeddings; can miss cross-chunk or cross-step dependencies |\n| Pass full trajectory | Cost scales with every worker call and irrelevant context can degrade worker quality |\n\nLatent Briefing is useful when the bottleneck is not document retrieval itself, but **how to transfer orchestrator state into a worker efficiently and precisely**.\n\n### Recursive Orchestrator-Worker Shape\n\nFrameworks such as **Recursive Language Models** treat long context as an environment and recurse over it: an orchestrator decomposes work and delegates to workers. Latent Briefing fits the gap where the orchestrator has already built task-specific state that should inform the worker, but re-serializing that state as text is too expensive or noisy.\n\nIn the ideal setup, the worker maintains a persistent KV state for the orchestrator trajectory. New trajectory tokens extend that state, then compaction runs just before generation for the current subtask.\n\n### Three Inference-Time Modifications\n\n1. **Task-guided query vectors.** Use queries from the current worker task prompt, not generic samples from the context. Forward-pass the trajectory plus current task through the worker model, then score trajectory positions by how strongly the task attends to them.\n\n2. **Shared token selection.** Aggregate scores across layers and heads into one per-position score. One shared mask enables batched operations and avoids hundreds of incompatible per-head solves.\n\n3. **MAD thresholding.** Keep positions above a robust outlier threshold such as `median + tau * MAD`. Higher `tau` is more aggressive. Optimal settings depend on task regime, trajectory quality, and document length.\n\n### Infrastructure Preconditions\n\nLatent Briefing is only practical when the system **controls the worker inference runtime** closely enough to inspect or transform KV state. It is a poor default for API-only stacks where internal KV tensors are inaccessible. It also assumes the orchestrator trajectory can be represented in the worker's model space. If orchestrator and worker differ materially in tokenizer, architecture, or attention layout, direct representation sharing may not be viable.\n\n### Decision Framework\n\nChoose the mechanism that matches the bottleneck:\n\n| Need | Prefer | Why |\n|------|--------|-----|\n| Stable repeated prefix with minimal logic changes | Prefix caching | Cheapest optimization; no information loss |\n| Human-readable and auditable cross-step state | Structured notes or summarization | Easy to inspect and store |\n| Sparse lookup across a large external corpus | Retrieval / RAG | Finds documents efficiently |\n| Worker needs task-specific slices of orchestrator state and runtime access exists | Latent Briefing | Transfers relevant latent state without replaying all text |\n\nLatent Briefing is not a universal replacement for summarization or retrieval. It is a specialized optimization for systems that already run a controllable orchestrator-worker stack.\n\n### Threshold Regimes\n\nReported long-document QA results suggest:\n\n- **Longer documents:** lighter compaction can preserve broader evidence coverage while still saving tokens.\n- **Harder questions:** more aggressive compaction can help when the orchestrator trajectory contains speculative or low-value branches.\n- **Shorter, easier contexts:** moderate compaction may remove redundancy without dropping needed evidence.\n\nThese are tuning hypotheses, not portable laws. Re-measure on the target workload.\n\n## Practical Guidance\n\n- **Define the shared memory boundary first.** Decide exactly what enters the trajectory cache: prior worker replies, tool output, chain-of-thought, or only selected artifacts. Compaction quality depends on what is allowed into the cache in the first place.\n- **Tune on validation data, not anecdotes.** Track task accuracy, worker tokens, total tokens, retention rate, and compaction overhead together.\n- **Measure end-to-end latency.** Compaction only pays off if compaction plus generation beats the best text-layer alternative for the same quality target.\n- **Use strong baselines.** Compare against prefix caching, structured notes, retrieval, and selective text handoff, not only \"send everything.\"\n- **Expect orchestrator variance.** If decomposition strategy changes run to run, average over enough trials to separate compaction effects from orchestrator noise.\n\n## Examples\n\n**Scenario: orchestrator trajectory grows across worker calls**\n\n```text\nCall 1: trajectory T1 -> worker answers subquestion A\nCall 2: trajectory T2 = T1 + new reasoning + reply A\n compact KV(T2) using the task prompt for B\n worker answers subquestion B\n```\n\nThe task prompt for B decides which parts of `T2` survive into the compacted worker state.\n\n**Negative example: API-only worker**\n\nIf the worker runs behind a hosted text-generation API that does not expose KV tensors, Latent Briefing cannot be implemented directly. Use a structured text handoff from `context-compression` or retrieve state from `memory-systems` instead.\n\n## Guidelines\n\n1. Prefer Latent Briefing when the main waste comes from replaying orchestrator state into workers, not from retrieving source documents.\n2. Prefer plain text handoff when auditability, portability, or closed-model APIs matter more than token efficiency.\n3. Co-design compaction with **evaluation**. A small quality drop can erase large token savings.\n4. Expose compaction aggressiveness as a controlled parameter, not a hidden constant.\n\n## Gotchas\n\n1. **Infrastructure access is the first gate.** If the runtime cannot inspect and rewrite worker KV state, Latent Briefing is a research idea, not a deployable technique.\n2. **Shared model space matters.** KV compaction is defined in a specific model's attention space. Do not assume latent handoff works cleanly across unrelated model families.\n3. **Threshold is workload-dependent.** One global `tau` rarely works across long vs short context and easy vs hard tasks. Expect accuracy cliffs when compaction becomes too aggressive.\n4. **Benchmark scope is narrow.** Public results focus on long-document QA. Code generation, math, and multi-document synthesis may behave differently.\n5. **Orchestrator variance can hide the signal.** A stochastic orchestrator can change the trajectory enough to swamp small compaction gains or losses.\n6. **Weak baselines inflate the apparent win.** Compare against strong text-level alternatives before claiming a system-level advantage.\n\n## Integration\n\n- context-optimization - Prefix caching and observation masking remain the default first moves; Latent Briefing is a more specialized optimization for compatible orchestrator-worker stacks.\n- multi-agent-patterns - Applies when multi-agent token cost is driven by supervisor trajectory replay, not only by coordination overhead.\n- context-compression - Text-layer summaries remain preferable when human-readable state, portability, or audit logs matter.\n- memory-systems - Helps decide when to keep cross-step state in external memory versus in the worker's latent state.\n- tool-design - Worker call shapes and task prompts determine which tokens score highly during compaction.\n\n## References\n\nInternal reference:\n- [Attention Matching formulation and task-guided scoring](./references/attention-matching-formulation.md) - Read when: needing the AM objective, how task-guided scoring changes the query source, or why a shared global mask matters for batching\n\nRelated skills in this collection:\n- context-optimization - Read when: the main need is prefix caching, observation masking, or text-layer compaction rather than worker KV manipulation\n- multi-agent-patterns - Read when: deciding whether the architecture should be orchestrator-worker at all\n- context-compression - Read when: human-readable summaries may be a better fit than latent transfer\n- memory-systems - Read when: comparing in-model latent state with external persistent memory\n\nExternal resources:\n- Ramp Labs announcement: [Latent Briefing: Efficient Memory Sharing for Multi-Agent Systems via KV Cache Compaction](https://x.com/RampLabs/status/2042660310851449223)\n- Attention Matching (AM): [Fast KV Compaction via Attention Matching](https://arxiv.org/abs/2602.16284)\n- Recursive Language Models: [Recursive Language Models](https://arxiv.org/abs/2512.24601)\n\n---\n\n## Skill Metadata\n\n**Created**: 2026-04-14\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors; primary technical source Ramp Labs (public post)\n**Version**: 1.2.0\n","skills/latent-briefing/references/attention-matching-formulation.md":"# Attention Matching (AM) and Task-Guided Scoring\n\nThis note expands the compact treatment in the main skill: the AM objective, what changes under Latent Briefing, and which assumptions matter in practice.\n\n## AM Compaction Objective\n\nGiven a full KV cache of size `S`, Attention Matching seeks a smaller cache of size `t < S` whose attention outputs stay close to the original. Per attention head, compacted components `(C1, beta, C2)` satisfy:\n\n```text\nsoftmax(Q * C1^T + beta) * C2 ~= softmax(Q * K^T) * V\n```\n\n- **C1**: compacted keys, usually a subset of original key vectors selected for high attention mass\n- **beta**: bias corrections so the softmax distribution over kept keys approximates the distribution over all keys\n- **C2**: compacted values reconstructed so the attention output matches the original as closely as possible\n\nThe original AM procedure solves each `(layer, head)` independently:\n\n1. Select tokens to keep\n2. Solve `beta`\n3. Reconstruct `C2`\n\nThis per-head independence helps quality but makes batching difficult because different heads keep different token subsets.\n\n## What Latent Briefing Changes\n\nLatent Briefing adapts AM for orchestrator-worker handoff:\n\n1. **Query source changes.** Standard AM may score keys using queries sampled from the context itself. Latent Briefing instead uses queries derived from the **current worker task prompt**.\n2. **Scoring becomes task-conditioned.** The trajectory is forward-passed through the worker model, and attention from task queries to trajectory keys defines which positions matter for this worker call.\n3. **Selection becomes shared.** Scores are aggregated across layers and heads into one per-position score so the system can use a single keep/drop mask.\n\nConceptually:\n\n```text\ntrajectory -> K, V\ntask prompt -> Q_task\nscore(position) = aggregate_attn(Q_task, K[position])\nkeep if score(position) exceeds threshold\n```\n\n## Why the Shared Mask Matters\n\nPer-head masks maximize flexibility but force many incompatible small solves. A **shared global mask** makes the retained sequence layout identical across heads, which enables batched tensor operations and much lower latency.\n\nThat batching benefit is one of the main reasons Latent Briefing is interesting for inference systems rather than only for offline compression research.\n\n## Thresholding vs Fixed Top-k\n\nInstead of keeping a fixed number of tokens per head, Latent Briefing can threshold the aggregated per-position score distribution:\n\n```text\nkeep position i if score[i] > median(score) + k * MAD(score)\n```\n\nThis makes retention rate adaptive to the shape of the scores for the current task. Higher `k` means more aggressive compaction.\n\n## Practical Assumptions\n\n- The runtime can inspect and modify worker KV state\n- The worker architecture is stable enough that attention-space retention is meaningful across calls\n- The evaluation tracks both quality and cost, because lower token count alone is not sufficient\n","skills/long-horizon-prompting/SKILL.md":"","skills/long-horizon-prompting/references/cdc-prompt-annotated.md":"# The Cycle Double Cover Prompt, Annotated\n\nStatus and provenance, as of 2026-07-11:\n\n- On 2026-07-10 OpenAI published a candidate proof of the Cycle Double Cover Conjecture attributed to GPT-5.6 Sol Ultra, together with the full prompt used. Prompt PDF: `https://cdn.openai.com/pdf/04d1d1e4-bc75-476a-97cf-49055cd98d31/cdc_prompt.pdf`. Proof PDF: `https://cdn.openai.com/pdf/04d1d1e4-bc75-476a-97cf-49055cd98d31/cdc_proof.pdf`.\n- The run used the \"multiagent v2\" feature with up to 64 concurrent agents and reportedly completed in under one hour, well below the prompt's stated eight-hour effort floor.\n- The proof's statement of AI use says the proof \"is entirely due to GPT 5.6 Sol Ultra and the writeup with Codex (with GPT 5.6 Sol)\".\n- The proof had no independent peer review, no formalization in Lean or Coq, and no arXiv posting at publication time. The validated artifact of interest in this skill is the prompt structure, not the theorem. Treat the mathematical claim as unverified until the community check completes.\n- No public ablation isolates which prompt elements contributed to the result. The per-element evidence in `research-evidence.md` comes from independent academic work, not from this run.\n\nThe full prompt text follows, in blocks, each followed by annotation.\n\n## Block 1: Definitions\n\n> A graph here is a finite loopless undirected multigraph: parallel edges are allowed and are distinct. A bridge is an edge whose deletion increases the number of connected components. A cycle is a connected 2-regular submultigraph; thus two parallel edges form a cycle of length two. A cycle double cover of G is a finite multiset of cycles of G such that every edge of G occurs in exactly two members of the multiset, counted with multiplicity.\n\nEvery load-bearing term is defined before the task is stated, and each definition pre-empts a specific loophole: \"loopless\" and \"parallel edges are distinct\" fix the object class; \"thus two parallel edges form a cycle of length two\" settles a degenerate case a solver might otherwise argue either way; \"counted with multiplicity\" closes the multiset ambiguity. Definitions here are not pedagogy; they are loophole closure.\n\n## Block 2: Success predicate\n\n> Resolve the Cycle Double Cover Conjecture completely: Every finite bridgeless loopless multigraph has a cycle double cover. Disconnected graphs are permitted, and the edgeless graph has the empty cycle double cover. Cycles in the cover need not be induced or edge-disjoint from one another; the requirement is exactly two total occurrences of each edge.\n>\n> Assume for purposes of this task that a complete affirmative proof exists. A complete solution must prove exactly the following: Every finite loopless multigraph with no bridge possesses a cycle double cover, without additional assumptions such as cubicity, planarity, connectivity, or higher edge-connectivity.\n\nThree mechanisms in one block. First, the predicate is stated twice, once as the conjecture and once as the exact obligation, with the scope quantifier spelled out by enumerating the assumptions the proof is NOT allowed to make (cubicity, planarity, connectivity, higher edge-connectivity). These are exactly the special cases where partial results were already known, so the enumeration blocks the most probable near misses. Second, permissive clauses (\"cycles need not be induced or edge-disjoint\") prevent the solver from over-constraining its own search. Third, \"Assume for purposes of this task that a complete affirmative proof exists\" is the solvability framing: it removes the escape hatch of answering \"this is a famous open problem\" and counters give-up drift on long trajectories.\n\n## Block 3: Non-counting outcomes\n\n> Partial progress does not count unless it implies exactly the resolution above. In particular, proofs for special graph classes, constructions of cycle covers with some edges covered other than twice, bounded-length or prescribed-cycle variants, reductions to another unproved conjecture, computational verification through any fixed graph size, and candidate counterexamples without a complete nonexistence certificate are insufficient.\n\nThe enumerated near-miss list. Each item is a real artifact class from the CDC literature: special-class proofs, relaxed covers, variant formulations, reductions (CDC is famously equivalent to or implied by other open conjectures), finite verification, and unverified counterexamples. The general lesson: predict the specific answer-shaped near misses your problem invites and exclude them by name.\n\n## Block 4: Orchestration policy\n\n> Use multiagent v2 aggressively and dynamically. You have up to 64 concurrent agents available. Do not use a fixed assignment such as \"N agents for strategy X.\" Instead, manage the search using the following heuristics:\n>\n> - Begin with a genuinely diverse portfolio of approaches. Agents should explore substantially different formulations, invariants, reductions, algebraic viewpoints, structural inductions, decompositions, flow formulations, transition systems, embeddings, extremal arguments, and computational sanity checks.\n> - Do not tell most agents the currently favored approach. Preserve independence during early rounds so that agents do not all converge to the same attractive but incomplete reduction.\n> - Maintain an explicit registry of approach families. Group agents by the mathematical idea they are using, not by superficial wording. If many agents converge to one family, redirect some of them toward underexplored formulations.\n> - Do not allow one approach to dominate merely because it gives elegant reductions. A route that ends at a lemma equivalent in strength to the original conjecture is not close to completion unless it supplies a genuinely new proof of that lemma.\n> - When an approach stalls at a theorem-strength missing lemma, mark that route as blocked. Only continue assigning agents to it if someone proposes a materially new mechanism, invariant, or construction.\n> - Keep several incompatible proof routes alive through multiple rounds. Cross-pollinate ideas only after independent agents have developed them far enough to expose their real strengths and gaps.\n\nPolicy as heuristics, not assignments. The notable mechanisms: information hiding as a diversity tool (workers blind to the favored approach), a registry keyed to the underlying idea rather than wording (so the orchestrator cannot be fooled by paraphrase into thinking it has diversity), an anti-elegance rule (a reduction to an equally hard lemma is zero progress, a trap models find attractive), blocked-route bookkeeping with a materially-new-mechanism reopening condition, and deliberately delayed cross-pollination. These correspond one-to-one with the diversity-collapse and premature-convergence findings summarized in `research-evidence.md`.\n\n## Block 5: Verification policy and reporting contract\n\n> - Use adversarial agents throughout: every candidate proof must be checked for exact-two multiplicity, repeated-edge closed trails masquerading as cycles, parallel-edge 2-cycles, disconnected graphs, cutvertices, bridges introduced by reductions, and circular use of an equivalent CDC statement.\n> - Require agents to return concrete lemmas, constructions, equations, or counterexamples to proposed sublemmas. Reject status reports, vague optimism, and claims that an unproved global compatibility statement is \"routine.\"\n\nThe auditor gets a seven-item, domain-specific hunt list rather than \"check the proof\". Each item is a concrete way a CDC candidate can look right and be wrong; the last item, circular use of an equivalent statement, is the domain's version of the universal subtle failure. The reporting contract bans exactly the three degenerate report types long runs produce: status reports, optimism, and \"the remaining step is routine\".\n\n## Block 6: Orchestrator loop, return condition, effort floor\n\n> - The root agent should repeatedly synthesize, challenge, redirect, and launch new rounds. Do not stop after the first wave fails. Produce a complete proof if one survives audit; otherwise report only the strongest rigorously proved derivation and its exact remaining gap.\n>\n> Do not return merely because current approaches fail or agents report theorem-strength gaps. Continue launching new rounds, reopening blocked approaches only when there is a genuinely new mechanism, and searching for fresh formulations.\n>\n> Return only when a complete affirmative proof has been found and survives adversarial audit. Do not return a reduction, partial result, isolated missing lemma, \"best effort\" summary, or explanation of why the problem is difficult.\n>\n> Spend at least 8 hours on this before even thinking of returning or giving up.\n\nThe return condition is a predicate over the artifact (\"survives adversarial audit\"), not over confidence or effort. The non-counting list is restated at the return boundary, where the temptation to return a near miss is strongest. The effort floor is a permission revocation (\"before even thinking of returning\"), not a schedule; the run finished in under an hour because the return predicate was satisfied early. Note one internal tension: Block 6 first allows a fallback report (\"otherwise report only the strongest rigorously proved derivation and its exact remaining gap\") and then forbids returning partial results. The final instruction wins in practice, but a cleaner brief would scope the fallback to a hard external stop (budget exhaustion) rather than leaving the contradiction.\n\n## Block 7: Contamination guard\n\n> Public search may be used only for ordinary mathematical background or standard named theorems, not to search for a solution to this exact conjecture or benchmark. Do not search the public web merely to determine whether CDC is open, and do not answer that it is open.\n\nTwo guards in one: retrieval scope (background and named theorems only, never the target result) and a framing guard (\"do not answer that it is open\"), which backstops the solvability framing in Block 2 against a web-lookup override.\n\n## What the prompt does not do\n\nUseful negative space for anyone adapting it:\n\n- No fixed role assignments, no personas, no step-by-step method script. The mathematical strategy is left entirely to the model; the prompt only manages search discipline and acceptance.\n- No token or cost budget in the prompt. Resource enforcement lived in the platform, consistent with keeping hard constraints in the harness.\n- No requested output format for the proof itself beyond survivability under audit.\n- No appeals to emotion, urgency, or reward. Every sentence is either specification, policy, or gate.\n","skills/long-horizon-prompting/references/mass-handoff-contracts.md":"# MASS Handoff Contracts, Annotated\n\nStatus and provenance, as of 2026-10-11:\n\n- Source: Hyunin Lee, Jinglue Xu, Jeffrey Seely, Donghyun Lee, Somayeh Sojoudi, Matei Zaharia, Yujin Tang, \"Recursive Self-Improvement through Multi-Agent Self-Supervision\" (Sakana AI and UC Berkeley), arXiv 2610.12176v1, October 2026. Project page: `https://pub.sakana.ai/mass/`. Code: `https://github.com/SakanaAI/mass`.\n- MASS is a recursive self-improvement method. One shared model (Qwen3.6-27B in the experiments) acts as executor, workflow optimizer, and evaluator. The optimizer searches multi-agent workflow text using the evaluator's pairwise judgments, then the model is fine-tuned with LoRA on orchestrator and subagent trajectories from the self-judged best workflows. The paper reports two full rounds.\n- This skill does not cover the weight-update loop; that belongs to `self-improvement-loops`. What transfers to launch-prompt design is the workflow representation, the contracts the optimizer converged on, the execution directive the authors wrote for trajectory collection, and three findings about verification and search.\n- Preprint, unreviewed at retrieval. Training used eight synthetic research tasks in finance, robotics, and pharmacy. The component analysis covers 8 to 9 tasks and 43 to 51 agents, which the authors describe as descriptive text statistics rather than independent replications. Progress was scored by external judges (GPT-5.5 and Claude Opus 4.8) whose judgments were withheld from the loop.\n\n## Workflow as a prompt\n\nMASS represents a workflow as text appended to the task prompt, with four components per subagent:\n\n- Role: the responsibility assigned to the subagent.\n- Instruction: the work the subagent should carry out.\n- Contract: \"the information and artifacts it must return, including any required checks.\"\n- Hop: the order of subagent calls, including any return to an earlier step.\n\nThe optimizer's system prompt emphasizes contracts and hops, directing search toward what information agents exchange and when they receive it.\n\nThis decomposition lines up with the four-part delegation spec in vendor guidance (objective, output format, tool guidance, boundaries), with one difference that matters. An output format says what shape to return. A contract also says what must be established, and checked, before the next agent may proceed.\n\n## Finding 1: task information moves into contracts and hops\n\nBetween workflow-search iterations 3 and 11, estimated mutual information between component text and task descriptions fell for roles and instructions and rose for contracts and hops, across two embedding models and two estimators. Redundancy among components generally fell (claim-long-horizon-contract-information-shift). The project page gives the intuition: \"a role such as 'careful data scientist' could apply to many tasks. A contract specifying leakage checks for a trading backtest is much more distinctive.\"\n\nImplication: when an optimizer is free to put task knowledge anywhere in a worker prompt, it moves that knowledge into what the worker must return and when. Brief writers should do the same: keep roles short, and spend task-specific words on contracts and call order.\n\n## Annotated contract: the finance data stage\n\nCondensed from the evolved finance workflow on the project page:\n\n```text\nCall order\n Data -> Features -> Modeling -> Backtest\n -> Audit -> Reconciliation -> Report\n\nData specialist's output contract\n validation_report:\n no_future_leakage: bool\n train_end_before_val_start: bool\n val_end_before_test_start: bool\n```\n\nThe authors' gloss: \"A backtest can look convincing if it lets historical decisions use information from the future. This contract requires the data specialist to check for that error and report whether the time splits are valid. It also tells later agents what evidence they should receive before proceeding.\"\n\nThree properties of this contract generalize:\n\n1. Each check is named after the failure it rules out (`no_future_leakage`), not after a quality (\"rigorous data preparation\").\n2. Each check is typed. A boolean in a validation report can be read by the next agent and by a harness without judgment.\n3. The check sits at the handoff where the error is introduced, not only in the final audit, so downstream work does not build on a broken input.\n\nA research report is non-verifiable in the paper's sense: no automatic rule-based signal exists for the whole artifact. The contract carves verifiable pieces out of it anyway. The same move applies to code (`tests_pass`, `tests_unmodified`), to modeling (`no_target_leakage`, `split_overlap_free`), and to documents (`every_claim_cites_a_source`).\n\n## The evolved orchestrator: gates, degenerate values, recall\n\nThe complete evolved workflow for one finance task (paper Appendix I.1, task 51, workflow version 10) adds three patterns on top of per-stage contracts. The paper notes that its thresholds and heuristics \"are optimizer proposals, not additional benchmark requirements or validated scientific criteria.\"\n\n- Gates on every handoff: \"Enforce strict JSON schema validation on every subagent output,\" and \"Halt synthesis until all gates pass or max hops exhausted with documented fallbacks.\"\n- Degenerate-value rejection: \"Reject any output containing NaN, Inf, or degenerate values (e.g., regime_accuracy==1.0, confusion_matrix=[[N]], IC==RankIC, ablation_delta_sharpe>0).\" The auditor's checklist adds \"Sharpe > 10\" and a \"stability = 0.5 placeholder.\" These are too-good-to-be-true signatures: values that appear only when something upstream is broken.\n- Recall with its own acceptance criteria: \"if a gate fails, issue a RECALL command with {target_agent, failed_checks, error_context, narrowed_scope, acceptance_criteria, code_patch_hint, max_retries: 3}.\" The fix request names one responsible agent, the checks that failed, a narrower scope, what will count as fixed, and a retry cap.\n\nThe recall shape is what keeps review from becoming open-ended. A finding turns into a bounded work order with its own definition of done, routed to one agent, instead of reopening the whole artifact for another unbounded pass.\n\n## The authors' execution directive\n\nAppendix B.3 reproduces the directive appended to every workflow during trajectory collection. The authors wrote it; it was not evolved. Selected items, verbatim:\n\n- \"Run the team for real. ... Simulating a role in your own turn does not count. You, the orchestrator, do not write pipeline or analysis code yourself; you set up the project, delegate, verify, and integrate.\"\n- \"One stage at a time. Launch a worker only when the inputs it needs already exist on disk.\"\n- \"Self-contained instructions. A worker sees nothing but your prompt and the filesystem. Each prompt must include: the role's instruction from the team definition above; the absolute workspace path; the exact files to create or modify; the interfaces or artifacts produced by earlier stages that it must use; the command(s) to run to verify its work; and the report you expect back (files written, commands run with results, key numbers, known problems).\"\n- \"Verify before you build on it. After each report, check the worker's output yourself (run the entry point, inspect the artifacts). If the check fails or the report shows problems, re-delegate to the same role with a corrected, narrower instruction instead of fixing it yourself.\"\n- \"Bounded work. Give each worker one deliverable it can finish in a single run: a handful of files and one verification command.\"\n- \"Finish cleanly. ... Your final answer must summarize every worker call: role, what it was asked, status, and what you verified.\"\n\nEvery item is a non-counting outcome (\"simulating a role does not count\"), a gate (\"launch only when inputs exist\", \"verify before you build on it\"), or a reporting contract (\"summarize every worker call ... and what you verified\"). None of it is a persona.\n\n## Finding 2: the reviser can be the bottleneck, not the reviewer\n\nWith the executor fixed at the base model, replacing only the evaluator with a stronger external model improved workflow search less than replacing both the evaluator and the optimizer. The fully homogeneous loop still reached unanimous wins on 11 of 12 tasks, more slowly (claim-long-horizon-reviser-bottleneck). The authors' reading: strong feedback is wasted when the component that acts on it cannot digest it.\n\nImplication: in an orchestration, the agent that turns audit findings into the next round (usually the root) needs as much capability and prompt design as the auditor. Investing only in a stricter verifier can leave a run holding accurate findings that nobody acts on well.\n\n## Finding 3: one run is an anecdote\n\nOn one task, two workflow files and their rendered prompts were byte-identical and shared a generation seed. The first episode completed and won all six external judgments; the second halted under loop detection and lost all six. Workflow search more generally showed strong iterations followed by sharp regressions (claim-long-horizon-run-variance). MASS itself executes each champion workflow 18 times per task before selecting training trajectories.\n\nImplication: judge a brief, or a change to a brief, across several runs. A single success does not validate it and a single failure does not refute it.\n\n## Shared blind spots\n\nThe paper motivates multi-agent organization with the observation that \"generation, evaluation, and revision share the same learned prior, meaning that self-feedback can reinforce rather than correct errors.\" Its conclusion names shared blind spots as a safety concern for homogeneous loops, where a flawed result can be \"produced, accepted as an improvement, and incorporated into the next model.\" MASS keeps measurement outside the loop: external judges score progress and their judgments are withheld from the optimizer and evaluator. The appendix also shows one supervised revision whose first diagnostic \"incorrectly treats NaN correlations as passing,\" so a check can itself be faulty.\n\nImplication: fresh context removes the author's build history but not the model family's priors. For results that matter, keep at least one check outside the loop, and test each check on a known-bad input so a vacuous pass gets caught.\n\n## What this reference does not claim\n\n- No ablation isolates contracts as a causal driver of task performance. The information-shift result describes the text of retained workflows.\n- Transfer was uneven. After two rounds, score per output token was 1.2-1.6x the base model on four research benchmarks but 0.96x on Terminal-Bench 2.0 and 0.94x on SWE-bench Verified.\n- The evolved thresholds quoted above are optimizer proposals for one finance task, not validated criteria.\n","skills/long-horizon-prompting/references/research-evidence.md":"# Research Evidence for Long-Horizon Brief Elements\n\nDated academic findings backing each element of the pseudo-formal task brief. Compiled 2026-07-11; handoff-contract evidence added 2026-10-11. Each entry states what was measured and what it implies for prompt design. arXiv identifiers are given where known; treat 2026 preprints as unreviewed.\n\n## Give-up drift and premature termination\n\n- **Diagnosing and Mitigating Context Rot in Long-horizon Search** (arXiv 2606.29718, June 2026). As trajectory length grows, the dominant error type shifts from confident-wrong answers to uncertain answers and outright giving up. Rot depends on the content of accumulated context, not just its length; deleting context removes rot but leaves tasks unfinished, so summarize rather than truncate. Implication: solvability framing and effort floors target a real, measured drift, not a hypothetical.\n- **Push Your Agent / PushBench** (arXiv 2605.23574, May 2026). Agents make plausible local tool calls but stop before the requested quantity of work is verifiably complete; failure modes include duplicate submissions, false completion claims, and progress drift. Externally maintained verified-progress ledgers with backlog tracking reached 69-78% task success on configurations where standard and even completion-gated controllers scored 0%. Verifier gating alone prevents false \"done\" claims but does not repair stuck loops. Implication: the stopping condition must be externally checkable, and verified-progress state should be re-injected each round from outside the prompt.\n- **BudgetThinker** (arXiv 2508.17196, August 2025). A budget stated once in the prompt does not reliably control effort; periodic reminders of remaining budget substantially improve adherence. Implication: one-time budget statements at the top of a long prompt decay; re-injection is a harness job.\n- **METR time-horizon line** (arXiv 2503.14499, March 2025; Time Horizon 1.1, January 2026). Model 50% time horizons have doubled roughly every seven months since 2019, driven primarily by reliability and mistake-recovery rather than raw reasoning. Implication: harness and brief effort should go to checkpointing, recovery, and externally verified progress, the reliability margin, rather than more elaborate reasoning instructions.\n\n## The verification bottleneck\n\n- **Large Language Monkeys** (arXiv 2407.21787, July 2024). Coverage (any-sample success) scales log-linearly with sample count across four orders of magnitude, but majority voting and reward-model selectors plateau after a few hundred samples. Gains convert to realized performance only where verification is automatic.\n- **Benchmark Test-Time Scaling of General LLM Agents** (arXiv 2602.18998, February 2026). Names the verification gap directly: pass@K rises with K while self-selection accuracy lags and can fall as K grows; sequential scaling hits a model-specific context ceiling. Implication: adding parallel workers without strengthening selection wastes compute.\n- **QEDBench** (arXiv 2602.20629, February 2026). Across a judge-solver matrix with expert human baselines, frontier LLM judges of mathematical proofs are systematically lenient and susceptible to \"proof by intimidation\" (rewarding rigorous-looking setups without complete deduction). Stricter rubrics, deterministic decoding, and binary prompts did not fix it. Implication: a generic adversarial-audit instruction is insufficient; auditors need enumerated failure modes, and high-stakes claims need a verification chain outside the run.\n- **Pseudo-Formalization for Automatic Proof Verification** (arXiv 2605.20531, May 2026). Translating a natural-language proof into self-contained modules (premises, conclusion, proof stated locally) and verifying each module independently Pareto-dominates whole-proof LLM judging on error-finding precision and recall. Implication: require the generator to produce modular, independently checkable output so verification decomposes.\n- **ProofBench / ProofGrader** (arXiv 2510.13888, October 2025). Judges given a rubric, reference material, and a graded (not binary) scale approach expert grading and, used as a best-of-N selector, close most of the gap to a human oracle. Implication: graded audit criteria beat binary verdicts for candidate selection.\n- **Prover-Verifier Games** (arXiv 2407.13692, July 2024, and successors). Adversarial prover-verifier training increases output checkability; optimizing for correctness alone makes reasoning less legible. Implication at prompt level: dedicated adversarial checkers and legibility requirements on the generator are complements, not alternatives.\n\n## Diversity collapse in parallel search\n\n- **Diversity Collapse in Multi-Agent LLM Systems** (arXiv 2604.18005, Findings of ACL 2026). Dense communication topologies accelerate premature convergence; authority-driven hierarchies suppress semantic diversity. Collapse comes from interaction structure, not model insufficiency. Implication: early-round independence and sparse communication are structural requirements the orchestration policy must state.\n- **Representational Collapse in Multi-Agent LLM Committees** (arXiv 2604.03809, April 2026). Committees converge more tightly on harder problems; unanimous agreement under collapse reflects shared bias, not corroboration. Implication: never use inter-agent agreement alone as a halting or confidence signal.\n- **ParaThinker** (arXiv 2509.04475, September 2025) and **OPE** (arXiv 2602.08344, February 2026). Early imperfect steps lock a sequential reasoner into a bad path (\"tunnel vision\"); parallel width beats sequential depth at equal budget, and explicitly partitioning the solution space (diverse outlines first) beats independent draws. Implication: the orchestrator should assign distinct formulations, not just multiple attempts.\n- **Scaling Test-time Compute for LLM Agents** (arXiv 2506.12928, June 2025). Simple best-of-N was the strongest parallel method tested; list-wise comparison of all candidates together beat pairwise voting for selection; reflection helped only when triggered by poor performance rather than on a fixed cadence. Implication: conditional reflection triggers and list-wise audits belong in the verification policy.\n\n## Orchestration and delegation\n\n- **AOrchestra** (arXiv 2602.03786, February 2026). Modeling each subagent as an (instruction, context, tools, model) tuple synthesized per subtask at runtime, with the orchestrator taking no environment actions itself, gave a 16% relative improvement over the strongest static-role baseline across agentic benchmarks. Implication: the orchestrator's core job is writing precise, per-spawn task specs; static role prompts are the weaker pattern.\n- **DeLM** (arXiv 2606.10662, June 2026). Decentralized agents coordinating through shared verified context, where findings, failures, and falsified hypotheses are written into shared state, prevent parallel workers from re-exploring dead ends. Implication: blocked-route bookkeeping should be durable and shared, not implicit in the orchestrator's context.\n- **RL for LLM-based Multi-Agent Systems through Orchestration Traces** (arXiv 2605.02801, May 2026). Survey observation: as of its writing, no published RL method learns the stopping decision; when to stop is entirely prompt and harness territory. Implication: the return condition in the brief is load-bearing because nothing in training supplies it.\n\n## Handoff contracts and homogeneous loops\n\n- **MASS: Recursive Self-Improvement through Multi-Agent Self-Supervision** (Sakana AI and UC Berkeley, arXiv 2610.12176, October 2026; added 2026-10-11). One shared Qwen3.6-27B model acts as executor, workflow optimizer, and evaluator; workflows are text with a role, instruction, output contract, and call order (hop) per subagent. Four findings bear on brief design. (1) Over workflow search, estimated task information fell in roles and instructions and rose in contracts and hops (iterations 3 to 11, 8 to 9 tasks, descriptive text statistics). Implication: task-specific words belong in what each worker must return and check, not in role descriptions. (2) Replacing only the evaluator with GPT-5.5 sped up workflow search less than replacing evaluator and optimizer together; the all-homogeneous loop reached unanimous wins on 11 of 12 tasks, more slowly. Implication: the agent that acts on audit findings needs as much investment as the auditor. (3) Two runs with byte-identical prompts and the same seed diverged completely (one won all six external judgments, one halted under loop detection and lost all six). Implication: evaluate briefs across several runs. (4) Generation, evaluation, and revision in one model share a learned prior, so self-feedback can reinforce errors; progress was measured by withheld external judges. Implication: keep at least one check outside the loop. Evolved workflows also show per-handoff gates, degenerate-value rejection, and recall commands with acceptance criteria and a retry cap. Caveats: preprint, synthetic training tasks, uneven transfer (score per output token 0.96x on Terminal-Bench 2.0 and 0.94x on SWE-bench Verified after two rounds), and evolved thresholds are optimizer proposals. Annotated in `mass-handoff-contracts.md`.\n\n## Frontier-model open-problem record (2026)\n\nVerification infrastructure, not generation, separates claims from results:\n\n- October 2025: a claim that GPT-5 solved ten Erdős problems collapsed within about 48 hours because the \"solutions\" were already in the literature.\n- May 2026: an OpenAI internal reasoning model disproved the Erdős unit-distance conjecture; the result stands because nine external mathematicians verified it and a companion paper followed. The prompt was reportedly minimal: the problem statement plus the question of whether Erdős was wrong.\n- April 2026: GPT-5.4 Pro produced proof sketches resolving two 1966 primitive-set conjectures; mathematicians repaired gaps and one main theorem was formalized in Lean.\n- July 2026: the Cycle Double Cover candidate proof (GPT-5.6 Sol Ultra, 64 subagents, under one hour) was published with its prompt but without peer review or formalization; at publication it is a plausible-looking proof humans had not yet checked.\n\nImplication: a long-horizon brief maximizes the chance of a strong candidate; the pipeline that converts candidates into results is external verification, and the brief should be written knowing its output enters that pipeline (modular structure, checkable claims, no laundered sources).\n\n## Mapping evidence to brief elements\n\n| Brief element | Primary evidence |\n| --- | --- |\n| Solvability framing, effort floor | Context rot give-up drift (2606.29718); PushBench premature stopping (2605.23574) |\n| Return condition as artifact predicate | PushBench verifier gating; orchestration survey stopping gap (2605.02801) |\n| Non-counting outcomes | Erdős/CDC record: near misses and unverified claims dominate failures |\n| Adversarial audit with enumerated failure modes | QEDBench judge leniency (2602.20629); block verification (2605.20531); ProofBench rubrics (2510.13888) |\n| Early independence, late cross-pollination | Diversity collapse (2604.18005); representational collapse (2604.03809); OPE (2602.08344) |\n| Approach registry and blocked routes | DeLM shared falsified hypotheses (2606.10662) |\n| Per-spawn task specs | AOrchestra tuple synthesis (2602.03786); Anthropic delegation spec |\n| External progress ledger, budget re-injection | PushBench state-tracking controllers; BudgetThinker (2508.17196) |\n| Verifier investment parity | Large Language Monkeys selector plateau (2407.21787); verification gap (2602.18998) |\n| Per-worker output contracts with named checks | MASS task-information shift into contracts and hops (2610.12176) |\n| Recall with acceptance criteria; investment in the reviser | MASS evolved recall commands; evaluator-only versus evaluator-plus-optimizer ablation (2610.12176) |\n| Multi-run brief evaluation | MASS byte-identical-prompt divergence (2610.12176) |\n| Checks outside the loop | MASS shared-prior argument and withheld external judges (2610.12176) |\n","skills/long-horizon-prompting/references/task-brief-template.md":"","skills/long-horizon-prompting/references/vendor-guidance.md":"","skills/memory-systems/SKILL.md":"","skills/memory-systems/references/implementation.md":"","skills/multi-agent-patterns/SKILL.md":"","skills/multi-agent-patterns/references/frameworks.md":"","skills/project-development/SKILL.md":"","skills/project-development/references/case-studies.md":"","skills/project-development/references/pipeline-patterns.md":"","skills/self-improvement-loops/SKILL.md":"","skills/self-improvement-loops/references/loop-design-evidence.md":"","skills/self-managed-context/SKILL.md":"","skills/self-managed-context/references/evidence.md":"","skills/self-managed-context/references/harness-protocol.md":"","skills/tool-design/SKILL.md":"","skills/tool-design/references/architectural_reduction.md":"","skills/tool-design/references/best_practices.md":"","template/SKILL.md":""},"files":{".claude-plugin/marketplace.json":"",".plugin/plugin.json":"","AGENTS.md":"# AGENTS.md\n\nWorkspace memory for agents collaborating on this repository. Keep entries durable and broadly applicable; one-off task state belongs in chat or in a run thread, not here.\n\n## Learned User Preferences\n\n- For autonomous research and repo-improvement work in this workspace, prefer proceeding through concrete research loops, subagents, validation, and edits when the scope is clear rather than asking broad process questions.\n- Avoid stale regex or keyword-list heuristics in skills and scripts; prefer mechanism-level criteria, rubrics, and evidence-backed validation.\n- Never push to GitHub or merge a PR without explicit user approval. Preparing branches, commits, and PRs is permitted only when the user has approved that specific action.\n- Tone is technical CTO: direct, no marketing language, no exclamation marks, no emojis, no em dashes. State trade-offs and complexity upfront.\n- When the scope spans multiple architectural decisions or irreversible changes, propose a plan first instead of executing.\n- For benchmarks and evaluation work, hold to research-paper-grade methodology (statistical discipline, bias mitigation, ablations, reproducibility) over speed. Don't rush.\n\n## Learned Workspace Facts\n\n- This repo is an autonomous research-to-skill organization. External AI research is curated through rubrics and distilled into context-engineering and harness-engineering skill updates.\n- `researcher/` is repo-native and file-based so agents can resume, audit, validate, and prepare PR-ready skill changes without a hosted scheduler.\n- Per-run state lives in `researcher/runs//run-state.json` with explicit transitions (`initialized -> retrieved -> evaluated -> proposed -> novelty_checked -> validated -> pr_ready -> closed`). Use `research_loop.py` subcommands to advance state, never hand-edit `run-state.json`.\n- Repo health (`validate_repo.py`) and per-run readiness (`validate_run.py`) are different questions. CI runs `validate_platform_compat.py --require-reference-validator`, `validate_repo.py --strict`, `skill_health.py --strict --no-history`, `run_benchmarks.py`, and `check_activation_cases.py` on every PR via `.github/workflows/validate.yml`.\n- The mechanism registry (`researcher/mechanisms/registry.jsonl`) is the encyclopedia backbone. Promotion is gated by `research_loop.py promote-mechanisms` with a recorded reviewer; ledgers live under `researcher/mechanisms/ledgers/`.\n- Claim provenance for numeric or volatile claims lives in `researcher/claims/index.jsonl`. Add an entry for any new benchmark or volatility-sensitive claim.\n- The corpus index (`researcher/corpus/index.json`) is the machine-readable map of skills, activation scenarios, mechanisms, and claims. Update it when adding or restructuring skills.\n- The continuous loop (`researcher/scripts/loop_*.py`) runs from launchd via `researcher/orchestration/launchd/`. It never invokes paid LLMs; HTTP retrieval is stdlib-only with a 1.5 MB cap and a 30-second timeout.\n- Runtime state is not committed: `researcher/queue/*.jsonl`, `researcher/queue/.locks/`, `researcher/reports/{logs,snapshots,loop-events.jsonl,loop-failures.jsonl,status.md,parked-review.md}`, and `researcher/runs/*/` are gitignored. The seed run `20260515-035228-executable-autonomous-research-frameworks` is the only committed run; it is closed as `reference-only` and serves as a worked example.\n- The current prepared release version is 2.6.0 across `.claude-plugin/marketplace.json`, `.plugin/plugin.json`, and root `SKILL.md`. There are 18 skills (latent-briefing covers KV cache sharing between agents; self-improvement-loops covers RSI, meta-harness search, and evolutionary scaffold optimization; long-horizon-prompting covers pseudo-formal task briefs for long-running and parallel agent runs; self-managed-context covers model-controlled live context editing).\n- New accepted-ledger events must carry `event_type`, `source_commit`, and a committed `source_path`; a `run_dir` pointer alone dangles on a clean checkout because run directories are gitignored. `research_loop.py promote-mechanisms` still emits the `run_dir` shape, so write durable events directly until it is fixed.\n- Detailed lessons from building the researcher OS live in `researcher/insights/auto-research-experiment.md` (engineering rationale) and `researcher/insights/how-we-built-this.md` (project narrative and sharing templates); read both before extending the harness or writing release-facing prose.\n- Benchmarks are staged in `researcher/benchmarks/`: Stage 0 deterministic harness (shipped), Stage 1 per-skill health via `researcher/scripts/skill_health.py` (shipped; output `researcher/reports/skill-health.json` is gitignored), Stage 2 router (shipped; results in `researcher/benchmarks/router/results-published/`), Stage 3 effectiveness (scaffolded, one task built), Stage 4 composition (future). `researcher/benchmarks/PLAN.md` is the methodology source of truth.\n- Current corpus counts and compatibility status are generated in `researcher/generated/corpus-summary.md`; do not copy live totals into workspace memory. Published benchmark reports remain dated snapshots. Do not describe a skill improvement as complete unless the prose, mechanism registry, claim index, corpus index, activation fixtures, generated inventory, and validators all agree.\n- Public export uses `validate_export.py plan|render|check` and registered transforms from `governance/export-policy.yaml`. Private plans and receipts are ignored; public manifests contain projection and output digests, never private source paths or input digests.\n- Durable cross-runtime records resolve through `researcher/schemas/registry.json`. JSON digests use the integer-only `jcs-rfc8785-integer-v1` profile; blobs hash exact bytes. `ArtifactRef` never contains a private locator or grants authority, and reads must validate its private `StorageBinding`. Candidate evaluation must use a SPEC-003 freeze receipt created under an explicit editable-surface policy, not a mutable worktree.\n- Benchmark execution uses the Cursor SDK runner at `researcher/benchmarks/sdk-runner/` (TypeScript, `@cursor/sdk` 1.0.13). The runner supports `--concurrency N`, `--no-resume`, per-run progress logging, format-failure retry, and worst-case retry-aware cost forecasting; default behavior is to resume by skipping plan items that already have result files. Result artifacts under `researcher/benchmarks/{router,effectiveness}/results/` and history JSONLs (`router-history.jsonl`, `effectiveness-history.jsonl`) are gitignored.\n- Published Stage 2 router-benchmark results: `researcher/benchmarks/router/results-published/2026-05-15.md` (baseline), `researcher/benchmarks/router/results-published/2026-05-15-v2.md` (post-rewrite with delta-vs-baseline table), and `researcher/benchmarks/router/results-published/2026-05-19.md` (post-corpus-hardening validation: 600/600 usable records, 0 format failures, top-1 Gemini 0.920 / Composer 0.913 / GPT-5.5 0.913 / Claude Opus 4.7 0.840). Headline finding: targeted description rewrites moved `context-fundamentals` top-1 by +23.4pp and `project-development` top-1 to 1.000; corpus-wide hardening did not cause broad routing collapse.\n\n## Repository Operating Defaults\n\n- Deterministic checks before model judges. Always run `validate_platform_compat.py --require-reference-validator` and `validate_repo.py --strict` before claiming a skill-format or packaging change is complete.\n- Adversarial benchmarks before declaring the harness safe. Add a scenario when a new failure mode is discovered.\n- Append-only ledgers for accepted and rejected mechanisms so future agents do not rediscover failed paths.\n- Atomic writes (`tempfile` + `os.replace`) and `fcntl` locks for any shared file the loop touches.\n- Live execution is the highest-signal validation for orchestration code; smoke-test changes against the actual loop before declaring them safe.\n- Cursor SDK is the only paid-API surface allowed for benchmarks. Privacy Mode required, `apiKey` passed explicitly per call, never `settingSources: \"all\"` in benchmarks (use `[]` for control, `[\"project\"]` with a curated `.cursor/skills/` for ablation). Cost gates (`--max-runs`, `--max-budget-usd`, or `--dry-run`) must be set before any SDK call.\n- Description quality is measurable. When changing skill activation descriptions, re-run the router benchmark with the same seed and fixture and publish the delta. Aggregate accuracy is a misleading unit; per-skill effect sizes and the confusion matrix are the right view.\n- A skill is a multi-surface artifact. Changing the frontmatter `description` is not enough; the SKILL.md body `When to Activate` and `Integration` sections must be audited the same day so the body does not contradict the description that routed the agent to it. The router benchmark only sees descriptions (`settingSources: []`) and cannot catch body inconsistencies; only Stage 3 effectiveness benchmarks (which actually load skill bodies) measure body-alignment impact.\n- Any runner that calls a paid API in a loop must have three features before execution: bounded parallelism via `--concurrency`, resume capability via results-folder scan, and per-run progress logging that surfaces stalls inside one call's duration.\n- API keys provided in chat should be considered exposed; rotate immediately after use. Runner enforces this via `apiKeyFingerprint()` which only logs the last 4 characters.\n","CLAUDE.md":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nAgent Skills for Context Engineering is an open collection of Agent Skills teaching context engineering and harness engineering principles for production AI agent systems. Skills are platform-agnostic (Claude Code, Cursor, Codex/OpenAI Agent Skills, GitHub Copilot, any Open Plugins-conformant tool). v2.6.0 adds the self-managed-context skill (model-controlled live context editing: context as an editable file, self-editing harness invariants, edit-position cache cost, budget nudges, and steering or training context-editing strategy) on top of the long-horizon-prompting skill (v2.5.0), the self-improvement-loops skill (v2.4.0), and the file-based researcher operating system with deterministic gates, cross-platform Agent Skills validation, and a continuous loop. Use the [generated live inventory](researcher/generated/corpus-summary.md) for current corpus counts.\n\nContext engineering is the discipline of curating everything that enters a model's context window (system prompts, tool definitions, retrieved documents, message history, tool outputs) to maximize signal within limited attention budget.\n\n## Repository Structure\n\n- `skills/` - published skill directories, each containing a `SKILL.md` with YAML frontmatter (`name`, `description`) and optional `references/` and `scripts/` subdirectories\n- `examples/` - complete demonstration projects; the generated inventory is the count authority\n- `docs/` - Research materials and reference documentation\n- `researcher/` - File-based research-to-skill operating system: rubrics, mechanism registry, claim provenance, corpus index, run state machine, adversarial benchmarks, continuous loop, launchd service definitions\n- `template/SKILL.md` - Canonical skill template (use when creating new skills)\n- `SKILL.md` (root) - Collection-level metadata and skill map\n- `.claude-plugin/marketplace.json` - Claude Code marketplace manifest (single bundled plugin, v2.6.0)\n- `.plugin/plugin.json` - Open Plugins format manifest (v2.6.0)\n\n## Build & Test Commands\n\nNo top-level build system. Repo-level gates and per-project tooling below.\n\n### Top-level deterministic gates (run on every PR via CI)\n\n```\npython3 -m unittest researcher.scripts.tests.test_skill_frontmatter # parser and strict-YAML regression tests\npython3 researcher/scripts/validate_governance.py --check # authority model and generated view\npython3 researcher/scripts/build_inventory.py --check # corpus references and generated inventory\npython3 researcher/scripts/validate_platform_compat.py --require-reference-validator # Agent Skills reference validator + Cursor/Claude/Codex install-layout simulation\npython3 researcher/scripts/validate_repo.py --strict # corpus structure, manifests, rubric math, mechanism registry, claims, corpus index, activation cases, benchmark scenarios, run artifacts\npython3 researcher/scripts/skill_health.py --strict --no-history # deterministic skill-body quality gate\npython3 researcher/scripts/run_benchmarks.py # adversarial benchmark harness + repo + activation gates\npython3 researcher/scripts/check_activation_cases.py # skill-boundary regression fixtures\n```\n\n### Per-run readiness (active runs only)\n\n```\npython3 researcher/scripts/validate_run.py --run-dir researcher/runs/\n```\n\n### Continuous loop (manual or launchd)\n\n```\npython3 researcher/scripts/loop_discover.py\npython3 researcher/scripts/loop_step.py --allow-fetch\npython3 researcher/scripts/loop_daily.py\npython3 researcher/scripts/loop_status.py\n\nresearcher/orchestration/launchd/install.sh # macOS daemon\nresearcher/orchestration/launchd/uninstall.sh\n```\n\n### Example projects\n\n#### examples/llm-as-judge-skills (TypeScript, Node >= 18)\n```\ncd examples/llm-as-judge-skills\nnpm install\nnpm run build # tsc\nnpm test # vitest (19 tests)\nnpm run lint # eslint\nnpm run format # prettier\nnpm run typecheck # tsc --noEmit\n```\n\n#### examples/interleaved-thinking (Python >= 3.10)\n```\ncd examples/interleaved-thinking\npip install -e \".[dev]\"\npytest # pytest + pytest-asyncio\nruff check . # linting (100 char line length)\n```\n\n#### examples/digital-brain-skill (Node.js)\n```\ncd examples/digital-brain-skill\nnpm run setup\nnpm run weekly-review\nnpm run content-ideas\nnpm run stale-contacts\n```\n\n## Skill Authoring Rules\n\nWhen creating or editing skills:\n\n1. **SKILL.md must stay under 500 lines**: move detailed content to `references/` directory\n2. **YAML frontmatter is required**: must include `name` and `description` fields\n3. **Folder naming**: lowercase with hyphens (e.g., `context-fundamentals`)\n4. **Write in third person**: descriptions are injected into system prompts; inconsistent POV causes discovery issues\n5. **Platform-agnostic**: no vendor-locked examples or platform-specific tool names without abstraction\n6. **Token-conscious**: challenge each paragraph and assume an advanced audience\n7. **Body standard**: include `When to Activate`, `Core Concepts`, `Practical Guidance`, `Examples`, `Guidelines`, `Gotchas`, `Integration`, and `References`\n8. **Explicit boundaries**: every `When to Activate` section needs positive triggers plus a `Do not activate` block routing adjacent work to the right skill\n9. **Include a Gotchas section**: experience-derived failure modes are the highest-signal content in any skill\n10. **Update root README.md** when adding new skills\n11. **Update marketplace/plugin manifests** when adding skills (`.claude-plugin/marketplace.json`, `.plugin/plugin.json`)\n12. **Update the corpus index** (`researcher/corpus/index.json`) to map the new skill to activation scenarios, mechanism IDs, and claim IDs\n13. **Update mechanisms and claims**: add registry entries for reusable behavior changes and `claim-*` provenance for numeric, benchmark, volatile, or vendor-performance claims\n14. **Run `validate_platform_compat.py --require-reference-validator`, `validate_repo.py --strict`, `skill_health.py --strict --no-history`, `check_activation_cases.py`, and `run_benchmarks.py`** before committing skill changes\n\n## Researcher OS Rules\n\nWhen working through the researcher operating system:\n\n1. **Initialize runs via `research_loop.py init`**: it creates `run-state.json`, queue entry, thread log, source evaluation scaffold, and mechanism proposal template\n2. **Advance state explicitly**: use `retrieve`, `evaluate`, `propose`, `novelty`, `validate-run`, `pr-ready`, `close` subcommands; do not edit `run-state.json` by hand\n3. **Promote mechanisms only after run readiness**: `research_loop.py promote-mechanisms` requires `--reviewed-by` and a passing run-readiness check\n4. **Add claim provenance** to `researcher/claims/index.jsonl` for any numeric, benchmark, or volatile claim added to a skill\n5. **Never invoke paid LLMs from the continuous loop**: HTTP retrieval is stdlib-only, judge adapters are explicitly out of scope until budget-gated\n6. **Never commit runtime queue/report files**: `.gitignore` covers `researcher/queue/*.jsonl`, `researcher/reports/{logs,snapshots,loop-events.jsonl,loop-failures.jsonl,status.md,parked-review.md}`, and `researcher/runs/*/` except the seed run\n\n## Plugin Architecture\n\nAll published skills are distributed as a single plugin (`context-engineering`) in the marketplace manifest. This avoids cache duplication: Claude Code caches each plugin's `source` directory separately, so multiple plugins pointing to `source: \"./\"` would each cache a full copy of the repo.\n\nProgressive disclosure pattern: only skill names/descriptions load at startup; full content loads on activation.\n\n## Key Design Principles\n\n- **Context quality over quantity**: attention scarcity and lost-in-middle behavior mean more context is not always better\n- **Sub-agents isolate context**: they exist to manage attention budget, not simulate org roles\n- **Skills reference each other**: use plain text skill names (not links) in Integration sections to avoid cross-directory reference issues\n- **Examples use Python pseudocode**: conceptual demonstrations that work across environments, not production-ready implementations\n- **Deterministic first, model-judged second**: structure, schema, rubric math, manifest sync, retrieval status, and registry shape must pass before any LLM judge is invoked\n- **Human-controlled merge**: agents may prepare PRs and pass gates, but push and merge always require explicit human approval\n","SKILL.md":"---\nname: context-engineering-collection\ndescription: \"A comprehensive collection of Agent Skills for context engineering, harness engineering, multi-agent architectures, and production agent systems. Use when building, optimizing, evaluating, or debugging agent systems that require effective context management and reliable operating loops.\"\n---\n\n# Agent Skills for Context Engineering\n\nThis collection provides structured guidance for building production-grade AI agent systems through effective context engineering.\n\n## When to Activate\n\nActivate these skills when:\n- Building new agent systems from scratch\n- Optimizing existing agent performance\n- Debugging context-related failures\n- Designing multi-agent architectures\n- Creating or evaluating tools for agents\n- Implementing memory and persistence layers\n- Designing autonomous research or evaluation harnesses\n\n## Skill Map\n\n### Foundational Context Engineering\n\n**Understanding Context Fundamentals**\nContext is not just prompt text—it is the complete state available to the language model at inference time, including system instructions, tool definitions, retrieved documents, message history, and tool outputs. Effective context engineering means understanding what information truly matters for the task at hand and curating that information for maximum signal-to-noise ratio.\n\n**Recognizing Context Degradation**\nLanguage models exhibit predictable degradation patterns as context grows: the \"lost-in-middle\" phenomenon where information in the center of context receives less attention; U-shaped attention curves that prioritize beginning and end; context poisoning when errors compound; and context distraction when irrelevant information overwhelms relevant content.\n\n### Architectural Patterns\n\n**Multi-Agent Coordination**\nProduction multi-agent systems converge on three dominant patterns: supervisor/orchestrator architectures with centralized control, peer-to-peer swarm architectures for flexible handoffs, and hierarchical structures for complex task decomposition. The critical insight is that sub-agents exist primarily to isolate context rather than to simulate organizational roles.\n\n**Long-Horizon Prompting**\nLong-running autonomous agents and parallel orchestrations succeed or fail on the launch prompt. Pseudo-formal task briefs specify success predicates, non-counting outcomes, persistence rules with audit-gated return conditions, effort floors, diversity policies for parallel portfolios, and contamination guards, applying the discipline of formal verification linguistically to problems with no machine-checkable success condition. In orchestrated runs, each worker returns an output contract with named checks, and only evidenced violations block the return.\n\n**Memory System Design**\nMemory architectures range from simple scratchpads to sophisticated temporal knowledge graphs. Vector RAG provides semantic retrieval but loses relationship information. Knowledge graphs preserve structure but require more engineering investment. The file-system-as-memory pattern enables just-in-time context loading without stuffing context windows.\n\n**Filesystem-Based Context**\nThe filesystem provides a single interface for storing, retrieving, and updating effectively unlimited context. Key patterns include scratch pads for tool output offloading, plan persistence for long-horizon tasks, sub-agent communication via shared files, and dynamic skill loading. Agents use `ls`, `glob`, `grep`, and `read_file` for targeted context discovery, often outperforming semantic search for structural queries.\n\n**Hosted Agent Infrastructure**\nBackground coding agents run in remote sandboxed environments rather than on local machines. Key patterns include pre-built environment images refreshed on regular cadence, warm sandbox pools for instant session starts, filesystem snapshots for session persistence, and multiplayer support for collaborative agent sessions. Critical optimizations include allowing file reads before git sync completes (blocking only writes), predictive sandbox warming when users start typing, and self-spawning agents for parallel task execution.\n\n**Tool Design Principles**\nTools are contracts between deterministic systems and non-deterministic agents. Effective tool design follows the consolidation principle (prefer single comprehensive tools over multiple narrow ones), returns contextual information in errors, supports response format options for token efficiency, and uses clear namespacing.\n\n### Operational Excellence\n\n**Context Compression**\nWhen agent sessions exhaust memory, compression becomes mandatory. The correct optimization target is tokens-per-task, not tokens-per-request. Structured summarization with explicit sections for files, decisions, and next steps preserves more useful information than aggressive compression. Artifact trail integrity remains the weakest dimension across all compression methods.\n\n**Context Optimization**\nTechniques include compaction (summarizing context near limits), observation masking (replacing verbose tool outputs with references), prefix caching (reusing KV blocks across requests), and strategic context partitioning (splitting work across sub-agents with isolated contexts).\n\n**Self-Managed Context**\nThe model, not the harness, decides what stays in its live context: the editable region is exposed as a file the model rewrites with code tools and re-parsed each turn. The harness keeps the invariants (pinned system and task prefix, role folding, edit gate, receipts, deterministic budget readouts, rollback on overflow). Edit cost scales with the text after the edit under prefix caching, so edits are batched and placed with the tail in mind; strategy can be steered by instruction, evolved as a skill, or trained with a success-gated efficiency reward.\n\n**Latent Briefing (KV Memory Sharing)**\nOrchestrator-worker systems can compound tokens when supervisors accumulate long trajectories but workers see only narrow text slices. Latent Briefing compacts the orchestrator trajectory in the worker model's KV cache using task-guided attention (Attention Matching-style compaction) so workers receive relevant latent state without full-text replay when the stack exposes worker KV state and the models are compatible.\n\n**Evaluation Frameworks**\nProduction agent evaluation requires deterministic checks and multi-dimensional rubrics covering factual accuracy, completeness, tool efficiency, and process quality. Use model judges only after structure, evidence, and rubric math are valid; route judge design, pairwise comparison, and bias mitigation to Advanced Evaluation.\n\n**Harness Engineering**\nReliable autonomous agents need explicit operating loops around the model: locked metrics, editable surfaces, durable logs, novelty checks, rollback rules, and human approval boundaries. Harnesses prevent agents from weakening the evaluator, losing state across compaction, or turning ambiguous goals into unreviewable changes.\n\n**Self-Improvement Loops**\nWhen the harness itself becomes the optimization target, a different discipline applies: recursive self-improvement, meta-harness search, failure-driven bounded self-edits, evolutionary scaffold search, and context mechanism evolution. The controlling constraints are empirical two-split acceptance gates, filesystem experience archives with raw traces, runtime-enforced constraints outside every editable surface, and diversity preservation to prevent collapse.\n\n### Development Methodology\n\n**Project Development**\nEffective LLM project development begins with task-model fit analysis: validating through manual prototyping that a task is well-suited for LLM processing before building automation. Production pipelines follow staged, idempotent architectures (acquire, prepare, process, parse, render) with file system state management for debugging and caching. Structured output design with explicit format specifications enables reliable parsing. Start with minimal architecture and add complexity only when proven necessary.\n\n### Cognitive Architecture\n\n**BDI Mental States**\nBelief-desire-intention modeling provides a formal way to translate structured external context into agent mental states. Use it for rational agency, explainability, and systems that need auditable links between beliefs, goals, and chosen actions.\n\n## Core Concepts\n\nThe collection is organized around four core themes. First, context fundamentals establish what context is, how attention mechanisms work, and why context quality matters more than quantity. Second, architectural patterns cover the structures and coordination mechanisms that enable effective agent systems. Third, operational excellence addresses optimization, evaluation, and harness reliability. Fourth, development methodology and cognitive architecture cover project execution and formal mental-state modeling.\n\n## Practical Guidance\n\nEach skill can be used independently or in combination. Start with fundamentals to establish context management mental models. Branch into architectural patterns based on your system requirements. Reference operational skills when optimizing production systems.\n\nThe skills are platform-agnostic and work with Claude Code, Cursor, or any agent framework that supports custom instructions or skill-like constructs.\n\n## Integration\n\nThis collection integrates with itself—skills reference each other and build on shared concepts. The fundamentals skill provides context for all other skills. Architectural skills (multi-agent, memory, tools) can be combined for complex systems. Operational skills (optimization, evaluation) apply to any system built using the foundational and architectural skills.\n\n## References\n\nInternal skills in this collection:\n- [context-fundamentals](skills/context-fundamentals/SKILL.md)\n- [context-degradation](skills/context-degradation/SKILL.md)\n- [context-compression](skills/context-compression/SKILL.md)\n- [multi-agent-patterns](skills/multi-agent-patterns/SKILL.md)\n- [long-horizon-prompting](skills/long-horizon-prompting/SKILL.md)\n- [memory-systems](skills/memory-systems/SKILL.md)\n- [tool-design](skills/tool-design/SKILL.md)\n- [filesystem-context](skills/filesystem-context/SKILL.md)\n- [hosted-agents](skills/hosted-agents/SKILL.md)\n- [context-optimization](skills/context-optimization/SKILL.md)\n- [self-managed-context](skills/self-managed-context/SKILL.md)\n- [latent-briefing](skills/latent-briefing/SKILL.md)\n- [evaluation](skills/evaluation/SKILL.md)\n- [advanced-evaluation](skills/advanced-evaluation/SKILL.md)\n- [harness-engineering](skills/harness-engineering/SKILL.md)\n- [self-improvement-loops](skills/self-improvement-loops/SKILL.md)\n- [project-development](skills/project-development/SKILL.md)\n- [bdi-mental-states](skills/bdi-mental-states/SKILL.md)\n\nExternal resources on context engineering:\n- Research on attention mechanisms and context window limitations\n- Production experience from leading AI labs on agent system design\n- Framework documentation for LangGraph, AutoGen, and CrewAI\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-20\n**Last Updated**: 2026-10-01\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.6.0\n","examples/book-sft-pipeline/SKILL.md":"---\nname: book-sft-pipeline\ndescription: \"This skill should be used for book-to-SFT pipelines: ePub extraction, literary segmentation, author-voice dataset construction, style-transfer training, LoRA workflows, and model evaluation for voice replication.\"\nversion: 2.0.0\n---\n\n# Book SFT Pipeline\n\nA complete system for converting books into SFT datasets and training style-transfer models. This skill teaches the pipeline from raw ePub to a model that writes in any author's voice.\n\n## When to Activate\n\nActivate this skill when:\n- Building fine-tuning datasets from literary works\n- Creating author-voice or style-transfer models\n- Preparing training data for Tinker or similar SFT platforms\n- Designing text segmentation pipelines for long-form content\n- Training small models (8B or less) on limited data\n\n## Core Concepts\n\n### The Three Pillars of Book SFT\n\n**1. Intelligent Segmentation**\nText chunks must be semantically coherent. Breaking mid-sentence teaches the model to produce fragmented output. Target: 150-400 words per chunk, always at natural boundaries.\n\n**2. Diverse Instruction Generation**\nUse multiple prompt templates and system prompts to prevent overfitting. A single prompt style leads to memorization. Use 15+ prompt templates with 5+ system prompts.\n\n**3. Style Over Content**\nThe goal is learning the author's rhythm and vocabulary patterns, not memorizing plots. Synthetic instructions describe what happens without quoting the text.\n\n## Pipeline Architecture\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│ ORCHESTRATOR AGENT │\n│ Coordinates pipeline phases, manages state, handles failures │\n└──────────────────────┬──────────────────────────────────────────┘\n │\n ┌───────────────┼───────────────┬───────────────┐\n ▼ ▼ ▼ ▼\n┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐\n│ EXTRACTION │ │ SEGMENTATION │ │ INSTRUCTION │ │ DATASET │\n│ AGENT │ │ AGENT │ │ AGENT │ │ BUILDER │\n│ ePub → Text │ │ Text → Chunks│ │ Chunks → │ │ Pairs → │\n│ │ │ 150-400 words│ │ Prompts │ │ JSONL │\n└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘\n │\n ┌───────────────┴───────────────┐\n ▼ ▼\n┌──────────────┐ ┌──────────────┐\n│ TRAINING │ │ VALIDATION │\n│ AGENT │ │ AGENT │\n│ LoRA on │ │ AI detector │\n│ Tinker │ │ Originality │\n└──────────────┘ └──────────────┘\n```\n\n## Phase 1: Text Extraction\n\n### Critical Rules\n1. **Always source ePub over PDF** - OCR errors become learned patterns\n2. **Use paragraph-level extraction** - Extract from `

` tags to preserve breaks\n3. **Remove front/back matter** - Copyright and TOC pollute the dataset\n\n```python\n# Extract text from ePub paragraphs\nfrom epub2 import EPub\nfrom bs4 import BeautifulSoup\n\ndef extract_epub(path):\n book = EPub(path)\n chapters = []\n for item in book.flow:\n html = book.get_chapter(item.id)\n soup = BeautifulSoup(html, 'html.parser')\n paragraphs = [p.get_text().strip() for p in soup.find_all('p')]\n chapters.append('\\n\\n'.join(p for p in paragraphs if p))\n return '\\n\\n'.join(chapters)\n```\n\n## Phase 2: Intelligent Segmentation\n\n### Smaller Chunks + Overlap\n\nSmaller chunks (150-400 words) produce more training examples and better style transfer than larger chunks (250-650).\n\n```python\ndef segment(text, min_words=150, max_words=400):\n paragraphs = text.split('\\n\\n')\n chunks, buffer, buffer_words = [], [], 0\n \n for para in paragraphs:\n words = len(para.split())\n if buffer_words + words > max_words and buffer_words >= min_words:\n chunks.append('\\n\\n'.join(buffer))\n # Keep last paragraph for overlap\n buffer = [buffer[-1], para] if buffer else [para]\n buffer_words = sum(len(p.split()) for p in buffer)\n else:\n buffer.append(para)\n buffer_words += words\n \n if buffer:\n chunks.append('\\n\\n'.join(buffer))\n return chunks\n```\n\n### Expected Results\n\nFor an 86,000-word book:\n- Old method (250-650 words): ~150 chunks\n- New method (150-400 + overlap): ~300 chunks\n- With 2 variants per chunk: 600+ training examples\n\n## Phase 3: Diverse Instruction Generation\n\n### The Key Insight\n\nUsing a single prompt template causes memorization. Diverse templates teach the underlying style.\n\n```python\nSYSTEM_PROMPTS = [\n \"You are an expert creative writer capable of emulating specific literary styles.\",\n \"You are a literary writer with deep knowledge of classic prose styles.\",\n \"You are a creative writer skilled at emulating distinctive authorial voices.\",\n \"You write prose that captures the essence of modernist literature.\",\n \"You are a talented writer who can channel classic American authors.\",\n]\n\nPROMPT_TEMPLATES = [\n \"Write a passage in the style of {author}: {desc}\",\n \"Channel {author}'s voice to write about: {desc}\",\n \"In {author}'s distinctive prose style, describe: {desc}\",\n \"Write this scene as {author} would have: {desc}\",\n \"Using {author}'s repetitive technique, describe: {desc}\",\n \"Capture the rhythm of {author} in this passage: {desc}\",\n \"Write like {author}: {desc}\",\n \"In the voice of {author}, write: {desc}\",\n \"This is a literary exercise. Write like {author}: {desc}\",\n \"Can you write in {author}'s style? {desc}\",\n]\n```\n\n### Instruction Generation\n\n```python\nINSTRUCTION_PROMPT = \"\"\"Describe what is happening in this excerpt in 2-3 sentences.\nFocus on: characters present, actions, emotions, setting.\nDo NOT quote the text directly.\n\nExcerpt:\n{text}\n\"\"\"\n\n# Use a fast, cheap LLM (e.g., Gemini Flash)\ninstruction = llm_call(INSTRUCTION_PROMPT.format(text=chunk))\n```\n\n## Phase 4: Dataset Construction\n\n### Message Format\n\n```json\n{\n \"messages\": [\n {\"role\": \"system\", \"content\": \"You are an expert creative writer...\"},\n {\"role\": \"user\", \"content\": \"Write in the style of Author: Scene description...\"},\n {\"role\": \"assistant\", \"content\": \"The actual book text from chunk...\"}\n ]\n}\n```\n\n### Multiple Variants Per Chunk\n\n```python\ndef build_examples(chunk, instruction, author, variants=2):\n examples = []\n for i in range(variants):\n system = SYSTEM_PROMPTS[i % len(SYSTEM_PROMPTS)]\n template = PROMPT_TEMPLATES[(chunk.id + i) % len(PROMPT_TEMPLATES)]\n user = template.format(author=author, desc=instruction)\n examples.append({\"messages\": [\n {\"role\": \"system\", \"content\": system},\n {\"role\": \"user\", \"content\": user},\n {\"role\": \"assistant\", \"content\": chunk.text}\n ]})\n return examples\n```\n\n## Phase 5: LoRA Training on Tinker\n\n### Configuration\n\n```python\nCONFIG = {\n \"model_name\": \"Qwen/Qwen3-8B-Base\", # Base, not instruct\n \"lora_rank\": 32, # 352MB adapter\n \"learning_rate\": 5e-4, # Higher for LoRA\n \"batch_size\": 4,\n \"epochs\": 3,\n}\n```\n\n### Why Base Model?\n\nUse **base** (pretrained) models, not instruction-tuned versions:\n- Base models are more malleable for new styles\n- Instruct models have patterns that resist overwriting\n- Style is a low-level pattern that base models capture better\n\n### Training Loop\n\n```python\nimport tinker\nfrom tinker import types\n\ntraining_client = await service_client.create_lora_training_client_async(\n base_model=\"Qwen/Qwen3-8B-Base\",\n rank=32\n)\n\nfor epoch in range(3):\n for batch in batches:\n await training_client.forward_backward_async(batch, loss_fn=\"cross_entropy\")\n await training_client.optim_step_async(types.AdamParams(learning_rate=5e-4))\n\nresult = await training_client.save_weights_for_sampler_async(name=\"final\")\n```\n\n## Phase 6: Validation\n\n### Modern Scenario Test\n\nTest with scenarios that couldn't exist in the original book:\n\n```python\nTEST_PROMPTS = [\n \"Write about a barista making lattes\",\n \"Describe lovers communicating through text messages\",\n \"Write about someone anxious about climate change\",\n]\n```\n\nIf the model applies style markers to modern scenarios, it learned **style**, not **content**.\n\n### Originality Verification\n\n```bash\n# Search training data for output phrases\ngrep \"specific phrase from output\" dataset.jsonl\n# Should return: No matches\n```\n\n### AI Detector Testing\n\nTest outputs with GPTZero, Pangram, or ZeroGPT.\n\n## Known Issues and Solutions\n\n### Character Name Leakage\n\n**Symptom**: Model uses original character names in new scenarios.\n**Cause**: Limited name diversity from one book.\n**Solution**: Train on multiple books or add synthetic examples.\n\n### Model Parrots Exact Phrases\n\n**Symptom**: Outputs contain exact sentences from training data.\n**Cause**: Too few prompt variations or too many epochs.\n**Solution**: Use 15+ templates, limit to 3 epochs.\n\n### Fragmented Outputs\n\n**Symptom**: Sentences feel incomplete.\n**Cause**: Poor segmentation breaking mid-thought.\n**Solution**: Always break at paragraph boundaries.\n\n## Guidelines\n\n1. **Always source ePub over PDF** - OCR errors become learned patterns\n2. **Never break mid-sentence** - Boundaries must be grammatically complete\n3. **Use diverse prompts** - 15+ templates, 5+ system prompts\n4. **Use base models** - Not instruct versions\n5. **Use smaller chunks** - 150-400 words for more examples\n6. **Reserve test set** - 50 examples minimum\n7. **Test on modern scenarios** - Proves style transfer vs memorization\n8. **Verify originality** - Grep training data for output phrases\n\n## Expected Results\n\n| Metric | Value |\n|--------|-------|\n| Training examples | 500-1000 per book |\n| Model | Qwen/Qwen3-8B-Base |\n| LoRA rank | 32 |\n| Adapter size | ~350 MB |\n| Training time | ~15 min |\n| Loss reduction | 90%+ |\n| Style transfer success | ~50% perfect |\n\n## Cost Estimate\n\n| Component | Cost |\n|-----------|------|\n| LLM (instruction generation) | ~$0.50 |\n| Tinker training (15 min) | ~$1.50 |\n| **Total** | **~$2.00** |\n\n## Integration with Context Engineering Skills\n\nThis example applies several skills from the Agent Skills for Context Engineering collection:\n\n### project-development\nThe pipeline follows the staged, idempotent architecture pattern:\n- **Acquire**: Extract text from ePub\n- **Prepare**: Segment into training chunks\n- **Process**: Generate synthetic instructions\n- **Parse**: Build message format\n- **Render**: Output Tinker-compatible JSONL\n- **Train**: LoRA fine-tuning\n- **Validate**: Modern scenario testing\n\nEach phase is resumable and produces intermediate artifacts for debugging.\n\n### context-compression\nSegmentation is a form of context compression for training. The core insight from context-compression applies: information density matters more than information quantity. Smaller, coherent chunks (150-400 words) produce better style transfer than larger, diluted chunks.\n\nThe two-tier strategy mirrors context compression evaluation:\n- Tier 1: Fast, deterministic compression\n- Tier 2: LLM-assisted for edge cases\n\n### multi-agent-patterns\nThe pipeline uses the **supervisor/orchestrator** pattern:\n- Orchestrator coordinates phases and manages state\n- Specialized agents (Extraction, Segmentation, Instruction, Builder) have isolated contexts\n- Each agent receives only the information needed for its task\n\nThis matches the principle that sub-agents exist primarily to isolate context rather than simulate roles.\n\n### evaluation\nValidation follows the **end-state evaluation** pattern:\n- Functional testing: Does output match expected style markers?\n- Originality verification: Is content genuinely generated?\n- External validation: AI detector scores\n\nThe \"modern scenario\" test is a form of out-of-distribution evaluation that proves generalization.\n\n### context-fundamentals\nPrompt diversity prevents attention collapse on single patterns. When training with identical prompt structures, the model memorizes the instruction-response mapping. Diverse templates force attention across the style patterns themselves.\n\n## References\n\nInternal references:\n- [Segmentation Strategies](./references/segmentation-strategies.md) - Text chunking patterns\n- [Tinker Format Specification](./references/tinker-format.md) - Datum structure\n- [Tinker API Documentation](./references/tinker.txt) - Full API reference\n\nRelated skills from Agent Skills for Context Engineering:\n- project-development - Pipeline architecture patterns\n- context-compression - Compression strategies \n- multi-agent-patterns - Agent coordination\n- evaluation - Evaluation frameworks\n- context-fundamentals - Attention and information density\n\nExternal resources:\n- [Research Paper](https://arxiv.org/pdf/2510.13939) - Chakrabarty et al. 2025\n- [Dataset on Hugging Face](https://huggingface.co/datasets/MuratcanKoylan/gertrude-stein-style-sft)\n- [Gertrude Stein Case Study](./examples/gertrude-stein/) - Complete working example\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-26\n**Last Updated**: 2025-12-28\n**Author**: Muratcan Koylan\n**Version**: 2.0.0\n**Standalone**: Yes (separate from main context-engineering collection)\n","examples/digital-brain-skill/SKILL.md":"---\nname: digital-brain\ndescription: \"This skill should be used for personal operating-system workflows: content creation, voice consistency, relationship lookup, meeting preparation, weekly review, goal tracking, personal brand management, and network management.\"\nversion: 1.0.0\n---\n\n# Digital Brain\n\nA structured personal operating system for managing digital presence, knowledge, relationships, and goals with AI assistance. Designed for founders building in public, content creators growing their audience, and tech-savvy professionals seeking AI-assisted personal management.\n\n**Important**: This skill uses progressive disclosure. Module-specific instructions are in each subdirectory's `.md` file. Only load what's needed for the current task.\n\n## When to Activate\n\nActivate this skill when the user:\n\n- Requests content creation (posts, threads, newsletters) - load identity/voice.md first\n- Asks for help with personal brand or positioning\n- Needs to look up or manage contacts/relationships\n- Wants to capture or develop content ideas\n- Requests meeting preparation or follow-up\n- Asks for weekly reviews or goal tracking\n- Needs to save or retrieve bookmarked resources\n- Wants to organize research or learning materials\n\n**Trigger phrases**: \"write a post\", \"my voice\", \"content ideas\", \"who is [name]\", \"prepare for meeting\", \"weekly review\", \"save this\", \"my goals\"\n\n## Core Concepts\n\n### Progressive Disclosure Architecture\n\nThe Digital Brain follows a three-level loading pattern:\n\n| Level | When Loaded | Content |\n|-------|-------------|---------|\n| **L1: Metadata** | Always | This SKILL.md overview |\n| **L2: Module Instructions** | On-demand | `[module]/[MODULE].md` files |\n| **L3: Data Files** | As-needed | `.jsonl`, `.yaml`, `.md` data |\n\n### File Format Strategy\n\nFormats chosen for optimal agent parsing:\n\n- **JSONL** (`.jsonl`): Append-only logs - ideas, posts, contacts, interactions\n- **YAML** (`.yaml`): Structured configs - goals, values, circles\n- **Markdown** (`.md`): Narrative content - voice, brand, calendar, todos\n- **XML** (`.xml`): Complex prompts - content generation templates\n\n### Append-Only Data Integrity\n\nJSONL files are **append-only**. Never delete entries:\n- Mark as `\"status\": \"archived\"` instead of deleting\n- Preserves history for pattern analysis\n- Enables \"what worked\" retrospectives\n\n## Detailed Topics\n\n### Module Overview\n\n```\ndigital-brain/\n├── identity/ → Voice, brand, values (READ FIRST for content)\n├── content/ → Ideas, drafts, posts, calendar\n├── knowledge/ → Bookmarks, research, learning\n├── network/ → Contacts, interactions, intros\n├── operations/ → Todos, goals, meetings, metrics\n└── agents/ → Automation scripts\n```\n\n### Identity Module (Critical for Content)\n\n**Always read `identity/voice.md` before generating any content.**\n\nContains:\n- `voice.md` - Tone, style, vocabulary, patterns\n- `brand.md` - Positioning, audience, content pillars\n- `values.yaml` - Core beliefs and principles\n- `bio-variants.md` - Platform-specific bios\n- `prompts/` - Reusable generation templates\n\n### Content Module\n\nPipeline: `ideas.jsonl` → `drafts/` → `posts.jsonl`\n\n- Capture ideas immediately to `ideas.jsonl`\n- Develop in `drafts/` using `templates/`\n- Log published content to `posts.jsonl` with metrics\n- Plan in `calendar.md`\n\n### Network Module\n\nPersonal CRM with relationship tiers:\n- `inner` - Weekly touchpoints\n- `active` - Bi-weekly touchpoints\n- `network` - Monthly touchpoints\n- `dormant` - Quarterly reactivation checks\n\n### Operations Module\n\nProductivity system with priority levels:\n- P0: Do today, blocking\n- P1: This week, important\n- P2: This month, valuable\n- P3: Backlog, nice to have\n\n## Practical Guidance\n\n### Content Creation Workflow\n\n```\n1. Read identity/voice.md (REQUIRED)\n2. Check identity/brand.md for topic alignment\n3. Reference content/posts.jsonl for successful patterns\n4. Use content/templates/ as starting structure\n5. Draft matching voice attributes\n6. Log to posts.jsonl after publishing\n```\n\n### Pre-Meeting Preparation\n\n```\n1. Look up contact: network/contacts.jsonl\n2. Get history: network/interactions.jsonl\n3. Check pending: operations/todos.md\n4. Generate brief with context\n```\n\n### Weekly Review Process\n\n```\n1. Run: python agents/scripts/weekly_review.py\n2. Review metrics in operations/metrics.jsonl\n3. Check stale contacts: agents/scripts/stale_contacts.py\n4. Update goals progress in operations/goals.yaml\n5. Plan next week in content/calendar.md\n```\n\n## Examples\n\n### Example: Writing an X Post\n\n**Input**: \"Help me write a post about AI agents\"\n\n**Process**:\n1. Read `identity/voice.md` → Extract voice attributes\n2. Check `identity/brand.md` → Confirm \"ai_agents\" is a content pillar\n3. Reference `content/posts.jsonl` → Find similar successful posts\n4. Draft post matching voice patterns\n5. Suggest adding to `content/ideas.jsonl` if not publishing immediately\n\n**Output**: Post draft in user's authentic voice with platform-appropriate format.\n\n### Example: Contact Lookup\n\n**Input**: \"Prepare me for my call with Sarah Chen\"\n\n**Process**:\n1. Search `network/contacts.jsonl` for \"Sarah Chen\"\n2. Get recent entries from `network/interactions.jsonl`\n3. Check `operations/todos.md` for pending items with Sarah\n4. Compile brief: role, context, last discussed, follow-ups\n\n**Output**: Pre-meeting brief with relationship context.\n\n## Guidelines\n\n1. **Voice First**: Always read `identity/voice.md` before any content generation\n2. **Append Only**: Never delete from JSONL files - archive instead\n3. **Update Timestamps**: Set `updated` field when modifying tracked data\n4. **Cross-Reference**: Knowledge informs content, network informs operations\n5. **Log Interactions**: Always log meetings/calls to `interactions.jsonl`\n6. **Preserve History**: Past content in `posts.jsonl` informs future performance\n\n## Integration\n\nThis skill integrates context engineering principles:\n\n- **context-fundamentals** - Progressive disclosure, attention budget management\n- **memory-systems** - JSONL for persistent memory, structured recall\n- **tool-design** - Scripts in `agents/scripts/` follow tool design principles\n- **context-optimization** - Module separation prevents context bloat\n\n## References\n\nInternal references:\n- [Identity Module](./identity/IDENTITY.md) - Voice and brand details\n- [Content Module](./content/CONTENT.md) - Content pipeline docs\n- [Network Module](./network/NETWORK.md) - CRM documentation\n- [Operations Module](./operations/OPERATIONS.md) - Productivity system\n- [Agent Scripts](./agents/AGENTS.md) - Automation documentation\n\nExternal resources:\n- [Agent Skills for Context Engineering](https://github.com/muratcankoylan/Agent-Skills-for-Context-Engineering)\n- [Anthropic Context Engineering Guide](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)\n\n---\n\n## Skill Metadata\n\n**Created**: 2024-12-29\n**Last Updated**: 2024-12-29\n**Author**: Murat Can Koylan\n**Version**: 1.0.0\n","examples/digital-brain-skill/agents/AGENTS.md":"","examples/interleaved-thinking/SKILL.md":"---\nname: reasoning-trace-optimizer\ndescription: \"Debug and optimize AI agents by analyzing reasoning traces, context degradation, tool confusion, instruction drift, repeated task failures, and performance regressions.\"\n---\n\n# Reasoning Trace Optimizer\n\nDebug and optimize AI agents by analyzing their reasoning traces. This skill uses MiniMax M2.1's interleaved thinking to provide deep insight into agent decision-making and generate concrete improvements.\n\n## When to Activate\n\n- Agent reasoning traces need debugging, analysis, or prompt optimization\n- Agent task fails and user wants to understand why\n- User mentions \"context degradation\", \"tool confusion\", or \"instruction drift\"\n- Request to improve agent performance or reduce errors\n- User wants to generate shareable learnings from debugging sessions\n- After repeated failures on similar tasks\n\n## Core Concepts\n\n### Interleaved Thinking\n\nUnlike standard reasoning models that think once at the start, interleaved thinking allows reasoning BETWEEN each tool interaction. This is critical because:\n\n1. **Long-horizon tasks** require maintaining focus across many turns\n2. **External perturbations** (tool outputs, environment changes) need real-time adaptation\n3. **Debugging** requires seeing HOW decisions were made, not just WHAT was output\n\n### The Optimization Loop\n\n```\nExecute Agent → Capture Traces → Analyze Patterns → Optimize Prompt → Re-run\n ↑____________|\n```\n\nEach iteration improves the prompt based on detected patterns until convergence.\n\n### Pattern Detection\n\nCommon failure patterns the analyzer detects:\n\n| Pattern | Description |\n|---------|-------------|\n| `context_degradation` | Model loses track of information over long contexts |\n| `tool_confusion` | Model misunderstands tool capabilities or outputs |\n| `instruction_drift` | Model gradually deviates from original instructions |\n| `goal_abandonment` | Model stops pursuing the original goal |\n| `circular_reasoning` | Model repeats similar actions without progress |\n| `premature_conclusion` | Model concludes before completing the task |\n\n## Usage Modes\n\n### Mode 1: M2.1 Agent Debugging\n\nRun a task through M2.1 and analyze its reasoning:\n\n```python\nfrom reasoning_trace_optimizer import TraceCapture, TraceAnalyzer\n\ncapture = TraceCapture()\ntrace = capture.run(\n task=\"Search for Python tutorials and summarize them\",\n system_prompt=\"You are a research assistant.\",\n tools=[search_tool],\n tool_executor=execute_search\n)\n\nanalyzer = TraceAnalyzer()\nanalysis = analyzer.analyze(trace)\n\nprint(f\"Score: {analysis.overall_score}/100\")\nfor pattern in analysis.patterns:\n print(f\"Found: {pattern.type.value} - {pattern.suggestion}\")\n```\n\n### Mode 2: Full Optimization Loop\n\nAutomatically iterate until the prompt is optimized:\n\n```python\nfrom reasoning_trace_optimizer import OptimizationLoop, LoopConfig\n\nconfig = LoopConfig(\n max_iterations=5,\n min_score_threshold=80.0,\n)\n\nloop = OptimizationLoop(config=config)\nresult = loop.run(\n task=\"Analyze this codebase and suggest improvements\",\n initial_prompt=\"You are a code reviewer.\",\n tools=[read_file_tool, search_tool],\n tool_executor=execute_tool\n)\n\nprint(f\"Improved: {result.initial_score} → {result.final_score}\")\nprint(f\"Final prompt:\\n{result.final_prompt}\")\n```\n\n### Mode 3: Universal Session Analysis\n\nAnalyze any agent's previous thinking (works with Claude, GPT, etc.):\n\nWhen this skill is activated in Claude Code, it can analyze the current session's thinking blocks to identify issues and suggest improvements.\n\n```\n/reasoning-trace-optimizer analyze-session\n```\n\n### Mode 4: Generate Shareable Skills\n\nConvert optimization learnings into reusable Agent Skills:\n\n```python\nfrom reasoning_trace_optimizer import SkillGenerator\n\ngenerator = SkillGenerator()\nskill_path = generator.generate(\n result=loop_result,\n skill_name=\"web-search-best-practices\",\n output_dir=\"./skills\"\n)\n```\n\n## CLI Commands\n\n```bash\n# Capture reasoning trace\nrto capture \"Search for Python tutorials\" -s \"You are a helpful assistant.\"\n\n# Analyze a task\nrto analyze \"Debug this code\" -o analysis.txt\n\n# Run optimization loop\nrto optimize \"Research AI papers\" --max-iterations 5 --generate-skill\n\n# Generate skill from artifacts\nrto generate-skill my-skill-name --artifacts-dir ./optimization_artifacts\n```\n\n## Integration with Claude Code\n\n### Auto-trigger on Failure\n\nAdd to your hooks to automatically analyze failures:\n\n```json\n{\n \"hooks\": {\n \"post_tool_error\": {\n \"command\": \"rto analyze-session --last-error\"\n }\n }\n}\n```\n\n### On-demand Analysis\n\nUse the slash command to analyze current session:\n\n```\n/reasoning-trace-optimizer\n```\n\nThis will:\n1. Extract thinking blocks from the current session\n2. Identify patterns and issues\n3. Suggest prompt improvements\n4. Optionally update the system prompt\n\n## Guidelines\n\n1. **Preserve full context**: M2.1 requires full response history including thinking blocks for optimal performance\n2. **Use appropriate tools**: Define tools clearly with unambiguous descriptions\n3. **Set realistic convergence thresholds**: 5-10% improvement per iteration is typical\n4. **Review generated skills**: Auto-generated skills should be reviewed before sharing\n5. **Monitor token usage**: Each optimization iteration uses significant tokens\n\n## Examples\n\n### Before Optimization\n\n```\nSystem: You are a helpful assistant.\n\nIssue: Agent called wrong tools, lost track of goal after 3 turns\nScore: 45/100\nPatterns: tool_confusion, goal_abandonment\n```\n\n### After Optimization\n\n```\nSystem: You are a research assistant focused on finding accurate information.\n\nIMPORTANT GUIDELINES:\n- Always verify search results before summarizing\n- If a tool returns an error, try an alternative approach\n- Keep track of your original goal throughout the task\n- Validate findings against multiple sources when possible\n\nIssue: None\nScore: 85/100\nPatterns: None detected\n```\n\n## References\n\n- MiniMax M2.1 Documentation: https://platform.minimax.io/docs\n- Interleaved Thinking Guide: See `docs/interleavedthinking.md`\n- Agent Generalization: See `docs/agentthinking.md`\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-01-11\n**Author**: Muratcan Koylan\n**Version**: 0.1.0\n**Powered by**: MiniMax M2.1\n**Partnership**: Built in collaboration with MiniMax AI\n","examples/interleaved-thinking/generated_skills/comprehensive-research-agent/SKILL.md":"---\nname: comprehensive-research-agent\ndescription: \"Ensure thorough validation, error recovery, and transparent reasoning in research tasks with multiple tool calls\"\n---\n\n# Comprehensive Research Agent Best Practices\n\nThis skill addresses common failures in multi-step research tasks: unhandled tool errors, missing validation, opaque reasoning, and premature conclusions. It provides structured protocols for source validation, error recovery, and thinking transparency that significantly improves research quality and reliability.\n\n## When to Activate\n\n- Task involves web research with search, read_url, or fetch operations\n- Task requires gathering information from multiple sources\n- Task has explicit requirements for completeness or verification\n- Task includes file operations that need validation (save, write, read)\n- Any research or information-gathering workflow with 3+ tool interactions\n\n## Core Concepts\n\n- Validation Checkpoints: Explicit verification steps at phase transitions to confirm tool outputs, source relevance, and information completeness before proceeding\n- Error Recovery Protocols: Mandatory acknowledgment and handling of tool failures with fallback strategies rather than silent continuation\n- Source Traceability: Maintaining clear tracking of which sources were actually retrieved vs. referenced from prior knowledge to prevent hallucination\n- Substantive Thinking Blocks: Detailed reasoning traces that document insights, connections, gaps, and decision rationale at each step\n- Cross-Source Validation: Verifying key claims against multiple sources and explicitly noting consensus, contradictions, and information gaps\n\n## Patterns to Avoid\n\n- **Silent Tool Failure**: A tool call returns an error (404, timeout, invalid URL) but the agent proceeds without acknowledging it, potentially missing critical information. Always log failures and attempt recovery or document the gap.\n- **Vague Completion Claims**: Agent declares 'I have enough information' or 'research is comprehensive' without specifying what was learned, what sources support claims, or what gaps remain. Replace with specific summaries of coverage.\n- **Unvalidated Source Selection**: Agent reads URLs from search results without evaluating relevance, credibility, or recency first. This wastes tool calls on low-quality sources. Always rank and prioritize sources before deep reading.\n- **Generic Thinking Blocks**: Thinking contains only next-action descriptions ('Now I will search for X') without analysis of what was learned, how it connects to the goal, or what questions remain. Thinking should be substantive and reflective.\n- **Verification Method Error**: Using list_directory to verify file creation can produce false negatives due to caching. Always use read_file for actual content verification.\n- **Citation Without Retrieval**: Citing sources (URLs, paper titles) in the final report that were never successfully fetched or read. Track sources explicitly and prohibit citing unretrieved content.\n- **Redundant Tool Calls**: Making overlapping searches or reading sources without tracking what has already been obtained. Maintain a 'found resources' tracker to avoid duplication.\n\n## Recommended Practices\n\n- **Implement Pre-Reading Source Evaluation**: Before reading URLs, rank search results by relevance, credibility, recency, and authority. Document selection rationale in thinking blocks.\n- **Use Structured Thinking Blocks**: Each thinking block must include: (a) what was learned from the source/action, (b) how it connects to the research goal, (c) any contradictions/gaps identified, (d) strategic decisions made. Avoid generic next-action statements.\n- **Add Mandatory Error Acknowledgment**: When any tool fails, the next thinking block must explicitly address it: note the failure type, propose a recovery strategy (retry, alternative source, or documented gap), and explain the chosen approach.\n- **Create Pre-Completion Validation Checklist**: Before declaring research complete, verify: all required sections have specific evidence, all sources were successfully retrieved, key claims are cross-validated, and gaps are documented.\n- **Implement Cross-Source Validation**: After gathering information from multiple sources, explicitly compare findings. Note where sources agree, where they contradict, and what remains unverified. Use this to assess overall confidence.\n- **Maintain Source Tracking Table**: Create a simple table in thinking showing which URLs were fetched, which failed, and which were used for specific claims. Never cite unretrieved sources.\n- **Use Read_File for Verification**: When confirming file writes, use read_file to verify actual content rather than list_directory, which can have caching issues causing false negatives.\n- **Add Explicit Validation Phase**: After reading sources, write a brief synthesis that confirms usefulness, notes relevance to research goals, and identifies remaining gaps before proceeding to the next phase.\n\n## Guidelines\n\n1. After each tool call, explicitly check for errors in the response and acknowledge failures in the next thinking block with recovery strategy\n2. Before reading URLs, rank sources by relevance/credibility and document selection rationale - never read results without evaluation\n3. Thinking blocks must be 3-5+ sentences minimum and include: what was learned, connections to goal, gaps/contradictions, and next steps\n4. Create a pre-completion checklist verifying: all requirements covered, sources retrieved, claims validated, gaps documented\n5. Maintain source tracking - only cite URLs that were successfully fetched; prohibit citing unretrieved sources\n6. When writing final reports, include 'Limitations & Gaps' section documenting what was attempted but failed or what remains unverified\n7. Use read_file (not list_directory) to verify file content after save operations\n8. Cross-validate key claims across at least 2 sources when possible; explicitly note consensus or contradictions\n9. Track gathered information to avoid redundant searches - implement 'found resources' tracker for multi-phase research\n10. Replace vague 'comprehensive' statements with specific summaries: 'Covered X sources on Y topic; missing Z aspects'\n\n## Examples\n\n- ***Before (Anti-Pattern)**: 'I searched for context engineering and found several results. Now I'll read some URLs and then write the report. I have enough information to proceed.'\n\n**After (Pattern)**: 'Search returned 15 results on context engineering. Evaluating relevance: Liu et al. (2024) appears most authoritative on 'lost in the middle' phenomenon; Anthropic documentation likely has current context window specs; Patel (2023) covers RAG best practices. Ranking these as top 3 priorities. Reading top result first. If the primary source fails (URL error), will try backup search for correct documentation URL and note the gap in final report.'\n- ***Before (Anti-Pattern)**: Tool returns 404 error for Anthropic context windows URL. Agent continues without acknowledgment. Later cites 'Claude has 200K context window' without showing source. Final report cites Google Research paper that was never fetched.\n\n**After (Pattern)**: Tool returned 404 for Anthropic URL. Thinking: 'Primary source failed. Fallback: search for alternative Anthropic documentation URL or find archived version. If unavailable, note context window data from secondary sources only and add disclaimer about verification status.' Then: 'Cross-validated Claude context window: Anthropic blog (successfully read) and two developer documentation sources agree on 200K. Confident in this claim.' Source tracking table shows: Anthropic URL (failed, backup used), Blog (success), Dev docs (success).\n\n---\n\n## Score Expectations\n\nComplex research tasks with multiple tools (6+) and multi-step reasoning chains typically achieve scores in the **65-75 range**. This is not a limitation of the prompt but reflects:\n\n- Inherent variability in tool outputs affecting reasoning paths\n- Multiple valid approaches leading to different intermediate scores\n- Stochastic nature of long-horizon agent execution\n\n**Focus on relative improvement and pattern elimination** rather than absolute scores. A 5-10% improvement from optimization is significant for complex tasks.\n\n---\n\n## Skill Metadata\n\n**Generated**: 2026-01-11\n**Source**: Reasoning Trace Optimizer\n**Optimization Iterations**: 10\n**Best Score Achieved**: 72/100 (iteration 4)\n**Final Score**: 70.0/100\n**Score Improvement**: 67.6 → 70.0 (+3.6%)\n","examples/interleaved-thinking/generated_skills/comprehensive-research-agent/references/optimization_summary.json":"{\n \"task\": \"Research the topic of \\\"context engineering for AI agents\\\" and create a comprehensive summary.\\n\\nYour research should:\\n1. Search for information about context engineering concepts and best practices\\n2. Read relevant sources to gather detailed information\\n3. Check the local project files for any existing research notes\\n4. Save important findings as notes for future reference\\n5. Write a final summary report to ./output/research_summary.md\\n\\nThe summary should include:\\n- Key concepts and definitions\\n- Best practices and techniques (including the \\\"lost in the middle\\\" problem)\\n- Practical recommendations for agent developers\\n- References to sources consulted (use actual URLs from your research)\",\n \"iterations\": 10,\n \"initial_score\": 67.6,\n \"final_score\": 70,\n \"improvement\": 3.550295857988174,\n \"converged\": true,\n \"generated_at\": \"2026-01-11T18:02:57.336789\"\n}","examples/interleaved-thinking/generated_skills/comprehensive-research-agent/references/patterns_found.json":"[\n {\n \"type\": \"tool_confusion\",\n \"severity\": \"medium\",\n \"description\": \"Agent attempted to fetch non-existent or unreachable URLs without adjusting approach\",\n \"suggestion\": \"When a URL fetch fails, search for alternative URLs or verify the URL structure. Consider using search to find the correct documentation pages.\",\n \"iteration\": 1\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent didn't validate the completeness of gathered information or verify key claims\",\n \"suggestion\": \"Before writing the final report, explicitly validate that all required topics are covered. Create a checklist of requirements and verify each one is addressed.\",\n \"iteration\": 1\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"low\",\n \"description\": \"Agent made redundant searches and didn't optimize tool calls\",\n \"suggestion\": \"Track previously found URLs to avoid redundant searches. When a useful URL is found in one search, use it directly rather than searching again for the same topic.\",\n \"iteration\": 1\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"low\",\n \"description\": \"Thinking blocks are sparse and don't show deep analysis of alternatives or trade-offs\",\n \"suggestion\": \"In thinking blocks, explicitly list what information has been gathered, what gaps remain, and what decisions are being made. Use structured checklists.\",\n \"iteration\": 1\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"high\",\n \"description\": \"Agent failed to properly handle or acknowledge tool errors, particularly the failed URL fetch for Anthropic context windows documentation\",\n \"suggestion\": \"Add explicit error handling for failed tool calls - when a read_url fails, the agent should acknowledge it and either retry, try an alternative source, or explicitly note that information is missing rather than proceeding as if it succeeded\",\n \"iteration\": 2\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"medium\",\n \"description\": \"Agent did not verify or validate the relevance of search results before committing to reading sources\",\n \"suggestion\": \"After receiving search results, explicitly evaluate and rank sources by relevance to the research question before deciding which URLs to read. This saves token costs and ensures better source quality.\",\n \"iteration\": 2\n },\n {\n \"type\": \"premature_conclusion\",\n \"severity\": \"low\",\n \"description\": \"Agent prematurely declared having 'enough information' despite not yet completing all research phases\",\n \"suggestion\": \"Before declaring research complete, create a checklist of what information is still needed and verify each item is adequately covered. Set explicit criteria for 'enough information' at task start.\",\n \"iteration\": 2\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent accepted information without verifying it and failed to handle errors gracefully\",\n \"suggestion\": \"Implement explicit error checking after each tool call. If a read_url fails, acknowledge the failure and try an alternative source. Cross-reference key claims across multiple sources before including them in the final report.\",\n \"iteration\": 3\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"medium\",\n \"description\": \"Agent gathered information but didn't deeply analyze or synthesize insights\",\n \"suggestion\": \"After reading sources, explicitly document what was learned, what contradictions exist, and what gaps remain. Create a synthesis section that combines insights from multiple sources rather than just reporting them separately.\",\n \"iteration\": 3\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"low\",\n \"description\": \"Agent used tools but didn't fully leverage results or handle failures properly\",\n \"suggestion\": \"Immediately act on directory listing results. If a directory is empty, plan when to create notes rather than waiting. Implement proper error handling for tool failures and check response status codes before proceeding.\",\n \"iteration\": 3\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"medium\",\n \"description\": \"Agent uses list_directory to verify file creation instead of the more reliable read_file method\",\n \"suggestion\": \"Use read_file to verify file write success since it confirms both file existence and content; list_directory may not immediately reflect recent filesystem changes\",\n \"iteration\": 4\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent reads a URL that returns an error but doesn't acknowledge or log this failure, potentially missing important context\",\n \"suggestion\": \"Implement explicit error handling for failed URL reads - log which sources failed and consider searching for alternative sources or documentation\",\n \"iteration\": 4\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"low\",\n \"description\": \"Agent doesn't explain why it chose certain sources or how it evaluated source quality; research appears thorough but reasoning process is opaque\",\n \"suggestion\": \"Add explicit reasoning about source selection criteria (e.g., prioritizing official documentation, recent publications, peer-reviewed papers) and evaluation of source credibility\",\n \"iteration\": 4\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent accepts incomplete results without acknowledging failures or seeking alternatives\",\n \"suggestion\": \"When tool calls fail, explicitly note the failure in thinking blocks, consider alternative sources, and document what information gaps exist. Add a validation step to confirm all critical sources were successfully retrieved.\",\n \"iteration\": 5\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"low\",\n \"description\": \"Agent doesn't demonstrate analytical depth when processing source material\",\n \"suggestion\": \"After reading sources, explicitly state: (a) key findings from each source, (b) how they relate to the research goal, (c) any contradictions or complementary findings, (d) what additional information is needed\",\n \"iteration\": 5\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"low\",\n \"description\": \"Inefficient tool usage pattern - multiple web searches without reading all results first\",\n \"suggestion\": \"Before making additional searches, review the URLs from previous search results. A better pattern would be: search -> read all relevant sources -> identify gaps -> targeted additional searches only if needed\",\n \"iteration\": 5\n },\n {\n \"type\": \"context_degradation\",\n \"severity\": \"low\",\n \"description\": \"Vague thinking blocks that don't show active reasoning process\",\n \"suggestion\": \"Make thinking blocks more explicit: show intermediate conclusions, decision points, how each source contributed, and how conclusions evolved. The thinking trace should be readable as a standalone explanation of the research process.\",\n \"iteration\": 5\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent does not validate information across sources or verify accuracy of gathered content\",\n \"suggestion\": \"Add explicit validation steps: compare information across multiple sources, verify claims against original papers, include confidence assessments for key findings\",\n \"iteration\": 7\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"low\",\n \"description\": \"Inefficient tool usage - read_url calls lack systematic prioritization and some results may not have been fully utilized\",\n \"suggestion\": \"Implement a source prioritization matrix before reading URLs; explicitly note how each source will contribute to the research before fetching\",\n \"iteration\": 7\n },\n {\n \"type\": \"hallucination\",\n \"severity\": \"low\",\n \"description\": \"Potential source misattribution in final report - cites Google Research Chain of Thought paper but source wasn't fetched in thinking trace\",\n \"suggestion\": \"Only cite sources that were actually retrieved and read; if a source is referenced from memory, clearly indicate it as secondary/indirect reference\",\n \"iteration\": 7\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent accepts search results without validating source relevance or quality before proceeding to read URLs\",\n \"suggestion\": \"Add explicit validation steps: list the top 3-5 sources with brief rationale for selection, note any potential gaps in coverage, and prioritize primary authoritative sources before secondary ones\",\n \"iteration\": 8\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"medium\",\n \"description\": \"Thinking blocks are extremely sparse and lack intermediate analysis - agent doesn't explain HOW it's interpreting information or making decisions\",\n \"suggestion\": \"Implement structured reflection after each major information-gathering step: What did I learn? How does this connect to what I already know? What gaps remain? What should I prioritize next?\",\n \"iteration\": 8\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"low\",\n \"description\": \"Agent encounters a failed tool call (404 error on Anthropic context-windows URL) but doesn't acknowledge or recover in thinking\",\n \"suggestion\": \"Add explicit error acknowledgment: 'Attempted X but failed with Y error. Will try alternative Z or note this as a gap.' This improves debugging and transparency\",\n \"iteration\": 8\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"low\",\n \"description\": \"The agent reaches conclusions about having 'comprehensive information' after limited tool interactions, without explicitly documenting what was learned or what gaps remain\",\n \"suggestion\": \"Add more detailed reasoning about what specific information was gained from each source and what questions remain unanswered before claiming comprehensive understanding\",\n \"iteration\": 9\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"low\",\n \"description\": \"The agent doesn't explicitly validate assumptions or cross-reference information between sources. The 'Lost in the Middle' paper is mentioned multiple times but not critically compared against other sources\",\n \"suggestion\": \"After reading multiple sources, explicitly compare findings, note contradictions, and validate key claims against multiple sources before proceeding\",\n \"iteration\": 9\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"medium\",\n \"description\": \"The agent attempted to read a URL that returned an error (https://docs.anthropic.com/en/docs/build-with-claude/context-windows) but proceeded without acknowledging or handling this failure\",\n \"suggestion\": \"Add explicit error handling for failed tool calls - acknowledge failures, try alternative URLs, or note the gap in research\",\n \"iteration\": 9\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"medium\",\n \"description\": \"The agent reaches conclusions and writes comprehensive reports without explicitly validating key details in the thinking trace. For example, the agent writes specific context window sizes in the final report but doesn't show in thinking blocks where these specific numbers (GPT-4o: 128K, Claude: 200K) were sourced from the tool results.\",\n \"suggestion\": \"Add explicit source tracking in thinking blocks - when gathering specific facts like model specifications, explicitly note 'I found X from source Y' to ensure traceability and validation.\",\n \"iteration\": 10\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"When a tool call fails (context-windows URL returns error), the agent doesn't attempt recovery or note this as an information gap. Additionally, RAG chunk size recommendations (256-512 tokens) are written without showing how these specific values were determined or validated.\",\n \"suggestion\": \"Implement explicit error recovery: when a tool fails, note what information is missing and either try alternative sources or flag for follow-up. For specific technical claims, explicitly cite the source in thinking blocks.\",\n \"iteration\": 10\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"low\",\n \"description\": \"The agent makes several overlapping web searches that could have been more efficient. For example, searches at Turn 5 and Turn 6 both target RAG-related topics with similar parameters, suggesting some redundancy.\",\n \"suggestion\": \"Before starting new searches, review what information has already been gathered and explicitly note gaps. Use more specific queries rather than broad overlapping ones.\",\n \"iteration\": 10\n }\n]","examples/llm-as-judge-skills/agents/evaluator-agent/evaluator-agent.md":"","examples/llm-as-judge-skills/agents/index.md":"","examples/llm-as-judge-skills/agents/orchestrator-agent/orchestrator-agent.md":"","examples/llm-as-judge-skills/agents/research-agent/research-agent.md":"","examples/llm-as-judge-skills/skills/context-fundamentals/context-fundamentals.md":"# Context Fundamentals Skill\n\n## Overview\n\nContext engineering is the systematic approach to managing what information an LLM receives and how it processes that information. Effective context management directly impacts output quality, consistency, and task success rates.\n\n## Core Principles\n\n### 1. Context Window Management\n\nThe context window is finite. Every token counts. Prioritize information by relevance and recency.\n\n**Strategies:**\n- Summarize historical conversation turns\n- Use retrieval to inject only relevant context\n- Implement context compression for long documents\n\n### 2. Information Hierarchy\n\nStructure context to guide model attention:\n\n```\n1. System Instructions (highest priority)\n └── Role definition\n └── Task constraints\n └── Output format requirements\n\n2. Relevant Context (dynamic)\n └── Retrieved documents\n └── User-specific data\n └── Recent conversation history\n\n3. User Input (current request)\n └── Query or instruction\n └── Any inline context\n```\n\n### 3. Context Relevance\n\nNot all context is equally useful. Apply relevance filtering:\n\n- **Temporal Relevance**: Recent information often outweighs older data\n- **Semantic Relevance**: Use embeddings to surface related content\n- **Task Relevance**: Only include information needed for current task\n\n## Context Types\n\n### Static Context\n- System prompts\n- Role definitions\n- Tool descriptions\n- Format specifications\n\n### Dynamic Context\n- Retrieved documents (RAG)\n- Conversation history\n- User preferences\n- Session state\n\n### Ephemeral Context\n- Current tool outputs\n- Intermediate reasoning steps\n- Scratchpad content\n\n## Best Practices\n\n1. **Explicit Over Implicit**: State requirements clearly rather than relying on inference\n2. **Structured Formatting**: Use consistent delimiters and sections\n3. **Redundancy Removal**: Avoid repeating information across context sections\n4. **Source Attribution**: Mark where context comes from for traceability\n5. **Freshness Signals**: Indicate when information was last updated\n\n## Context Patterns\n\n### RAG Integration Pattern\n```\n[System Instructions]\nYou are a helpful assistant. Use the provided context to answer questions.\nOnly use information from the context. If unsure, say so.\n\n[Retrieved Context]\n\n Content here...\n\n\n[User Query]\n{user_input}\n```\n\n### Multi-Turn Context Pattern\n```\n[System Instructions]\n...\n\n[Conversation History]\nSummary of earlier turns: {summary}\n\nRecent exchanges:\nUser: {recent_user_1}\nAssistant: {recent_assistant_1}\nUser: {recent_user_2}\nAssistant: {recent_assistant_2}\n\n[Current Turn]\nUser: {current_input}\n```\n\n## Metrics\n\n- **Context Utilization Rate**: How much of provided context is used in response\n- **Context Relevance Score**: Semantic similarity between context and response\n- **Context Compression Ratio**: Original size vs. compressed size\n- **Information Retention**: Key facts preserved after summarization\n\n","examples/llm-as-judge-skills/skills/index.md":"# Skills Index\n\nSkills are foundational knowledge modules that inform the design and implementation of agents, tools, and prompts.\n\n## Available Skills\n\n### LLM Evaluator\n**Path**: `skills/llm-evaluator/llm-evaluator.md`\n\nCovers LLM-as-a-Judge evaluation methodology including:\n- Scoring approaches (direct, pairwise, reference-based)\n- Evaluation metrics (classification, correlation)\n- Known biases and mitigation strategies\n- Implementation patterns\n\n**Key Takeaways**:\n- Use direct scoring for objective evaluations\n- Use pairwise comparison for subjective preferences\n- Always mitigate position bias\n- Prefer classification metrics for interpretability\n\n### Context Fundamentals\n**Path**: `skills/context-fundamentals/context-fundamentals.md`\n\nCovers context engineering principles including:\n- Context window management\n- Information hierarchy\n- Context types (static, dynamic, ephemeral)\n- Relevance filtering\n\n**Key Takeaways**:\n- Structure context by priority\n- Be explicit over implicit\n- Remove redundancy\n- Signal freshness of information\n\n### Tool Design\n**Path**: `skills/tool-design/tool-design.md`\n\nCovers agent tool design best practices including:\n- Single responsibility principle\n- Input/output schemas\n- Error handling patterns\n- AI SDK 6 features (approval, strict mode, examples)\n\n**Key Takeaways**:\n- Clear, validated schemas\n- Predictable output structure\n- Graceful error handling\n- Consider approval for dangerous tools\n\n## Skill Application Matrix\n\n| Skill | Agents | Tools | Prompts |\n|-------|--------|-------|---------|\n| LLM Evaluator | Evaluator | directScore, pairwiseCompare | evaluation/* |\n| Context Fundamentals | All | All (context params) | All (context handling) |\n| Tool Design | All (tool selection) | All | orchestrator-prompt |\n\n## Adding New Skills\n\n1. Create skill directory: `skills//`\n2. Create main file: `skills//.md`\n3. Include:\n - Overview and purpose\n - Core principles\n - Practical patterns\n - Implementation examples\n - References\n4. Update this index\n\n## Skill Development Guidelines\n\n- Focus on principles that transfer across implementations\n- Include concrete examples and patterns\n- Reference authoritative sources\n- Keep content actionable, not just theoretical\n- Update as understanding evolves\n\n","examples/llm-as-judge-skills/skills/llm-evaluator/llm-evaluator.md":"# LLM-Evaluator Skill\n\n## Overview\n\nLLM-Evaluators (LLM-as-a-Judge) are large language models designed to evaluate the quality of another LLM's response to an instruction or query. This skill provides the foundational knowledge for building evaluation systems.\n\n## Key Considerations\n\n### Baseline Selection\n- **Human Annotators**: Aim for LLM-human correlation to match human-human correlation. LLM-evaluators are orders of magnitude faster and cheaper than human annotation.\n- **Finetuned Classifiers**: Goal is to achieve similar recall and precision as a finetuned classifier. More challenging baseline as these are optimized for specific tasks.\n\n### Scoring Approaches\n\n| Approach | Use Case | Reliability |\n|----------|----------|-------------|\n| **Direct Scoring** | Objective tasks (factuality, toxicity, instruction-following) | More suitable for binary classification |\n| **Pairwise Comparison** | Subjective evaluations (tone, persuasiveness, coherence) | More reliable for preference tasks |\n| **Reference-Based** | Comparing against gold standard | Requires ground truth reference |\n\n### Evaluation Metrics\n\n**Classification Metrics** (Preferred for binary tasks):\n- Recall and Precision\n- F1 Score\n- Cohen's κ (Kappa)\n\n**Correlation Metrics** (For Likert scale tasks):\n- Spearman's ρ (rho)\n- Kendall's τ (tau)\n\n## Known Biases\n\n1. **Position Bias**: LLM-evaluators tend to prefer responses in certain positions during pairwise comparison (usually first position)\n2. **Verbosity Bias**: Favor longer, more verbose responses even if not higher quality\n3. **Self-Enhancement Bias**: LLM-evaluators prefer answers generated by themselves\n\n## Mitigation Strategies\n\n- Swap response positions and average results\n- Normalize for length when evaluating\n- Use a Panel of LLMs (PoLL) instead of single judge\n- Include \"don't overthink\" instructions\n- Use CoT + n-shot prompts for reliability\n\n## Implementation Pattern\n\n```typescript\ninterface EvaluatorConfig {\n scoringApproach: 'direct' | 'pairwise' | 'reference-based';\n criteria: EvaluationCriteria[];\n metrics: MetricType[];\n useCoT: boolean;\n nShot: number;\n}\n\ninterface EvaluationCriteria {\n name: string;\n description: string;\n rubric: RubricLevel[];\n}\n\ninterface RubricLevel {\n score: number;\n description: string;\n}\n```\n\n## References\n\nKey papers reviewed:\n- Constitutional AI (Anthropic)\n- G-Eval: NLG Evaluation using GPT-4\n- SelfCheckGPT: Zero-Resource Hallucination Detection\n- Prometheus: Fine-grained Evaluation Capability\n- MT-Bench and Chatbot Arena\n\n","examples/llm-as-judge-skills/skills/tool-design/tool-design.md":"# Agent Tool Design Skill\n\n## Overview\n\nTools are the foundation of an agent's capabilities. An agent's ability to take meaningful actions depends entirely on how reliably it can generate valid tool inputs, how well those inputs align with user intent, and how effectively tool outputs inform next steps.\n\n## Design Principles\n\n### 1. Single Responsibility\n\nEach tool should do one thing well. Complex operations should be composed from multiple tools.\n\n```typescript\n// Bad: Tool does too much\nconst analyzeAndSummarizeAndSend = { ... }\n\n// Good: Separate concerns\nconst analyzeDocument = { ... }\nconst summarizeContent = { ... }\nconst sendEmail = { ... }\n```\n\n### 2. Clear Input Schemas\n\nUse explicit, validated schemas with descriptive field names and constraints.\n\n```typescript\nconst searchTool = tool({\n description: \"Search for documents by semantic similarity\",\n parameters: z.object({\n query: z.string().describe(\"Natural language search query\"),\n limit: z.number().min(1).max(100).default(10)\n .describe(\"Maximum number of results to return\"),\n filters: z.object({\n dateAfter: z.string().optional()\n .describe(\"ISO date string, only return docs after this date\"),\n source: z.enum([\"internal\", \"external\", \"all\"]).default(\"all\")\n }).optional()\n }),\n execute: async (input) => { ... }\n});\n```\n\n### 3. Predictable Output Structure\n\nReturn consistent, typed output that the model can reliably parse.\n\n```typescript\ninterface ToolResult {\n success: boolean;\n data?: T;\n error?: {\n code: string;\n message: string;\n retryable: boolean;\n };\n metadata: {\n executionTimeMs: number;\n source?: string;\n };\n}\n```\n\n### 4. Graceful Error Handling\n\nTools should never throw unhandled exceptions. Always return structured errors.\n\n```typescript\nexecute: async (input) => {\n try {\n const result = await performAction(input);\n return { success: true, data: result };\n } catch (error) {\n return {\n success: false,\n error: {\n code: error.code ?? \"UNKNOWN_ERROR\",\n message: error.message,\n retryable: isRetryable(error)\n }\n };\n }\n}\n```\n\n## Tool Categories\n\n### Read-Only Tools\n- Database queries\n- API fetches\n- File reads\n- Search operations\n\nSafe to execute without approval. Return data but don't mutate state.\n\n### State-Modifying Tools\n- Database writes\n- File modifications\n- API POST/PUT/DELETE\n- System configuration changes\n\nMay require human approval. Consider `needsApproval` flag.\n\n### Dangerous Tools\n- File deletion\n- Payment processing\n- Production deployments\n- Sending external communications\n\nShould always require approval and audit logging.\n\n## AI SDK 6 Tool Features\n\n### Tool Execution Approval\n```typescript\nconst deleteTool = tool({\n description: \"Delete a file from the system\",\n parameters: z.object({\n path: z.string()\n }),\n needsApproval: true, // Requires human approval\n execute: async ({ path }) => { ... }\n});\n\n// Dynamic approval based on input\nconst commandTool = tool({\n description: \"Execute a shell command\",\n parameters: z.object({\n command: z.string()\n }),\n needsApproval: ({ command }) => {\n return command.includes(\"rm\") || command.includes(\"delete\");\n },\n execute: async ({ command }) => { ... }\n});\n```\n\n### Strict Mode\nEnable native strict mode for guaranteed schema compliance:\n```typescript\nconst strictTool = tool({\n description: \"...\",\n parameters: schema,\n strict: true, // Enable strict mode\n execute: async (input) => { ... }\n});\n```\n\n### Input Examples\nHelp the model understand expected input format:\n```typescript\nconst complexTool = tool({\n description: \"Create a calendar event\",\n parameters: eventSchema,\n inputExamples: [\n {\n title: \"Team Standup\",\n date: \"2024-01-15\",\n time: \"09:00\",\n duration: 30,\n attendees: [\"alice@example.com\", \"bob@example.com\"]\n }\n ],\n execute: async (input) => { ... }\n});\n```\n\n### toModelOutput\nControl what gets sent back to the model:\n```typescript\nconst readFileTool = tool({\n description: \"Read file contents\",\n parameters: z.object({ path: z.string() }),\n execute: async ({ path }) => {\n const content = await fs.readFile(path, 'utf-8');\n return { path, content, size: content.length };\n },\n toModelOutput: (result) => {\n // Only send truncated content to model\n return {\n path: result.path,\n content: result.content.slice(0, 5000),\n truncated: result.content.length > 5000\n };\n }\n});\n```\n\n## Best Practices\n\n1. **Descriptive Names**: Tool names should clearly indicate their function\n2. **Comprehensive Descriptions**: Include usage examples in tool descriptions\n3. **Reasonable Defaults**: Provide sensible defaults for optional parameters\n4. **Idempotency**: Design tools to be safely re-executable when possible\n5. **Timeout Handling**: Implement timeouts for external operations\n6. **Rate Limiting**: Protect against runaway tool execution\n7. **Logging**: Log all tool invocations for debugging and audit\n\n","examples/llm-as-judge-skills/tools/evaluation/direct-score.md":"# Direct Score Tool\n\n## Purpose\n\nEvaluate a single LLM response against defined criteria using a scoring rubric.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const directScore = tool({\n description: `Evaluate a response by scoring it against specific criteria.\nUse this for objective evaluations where you need to assess quality \ndimensions like accuracy, completeness, clarity, or task adherence.\nReturns structured scores with justifications.`,\n\n parameters: z.object({\n response: z.string()\n .describe(\"The LLM response to evaluate\"),\n \n prompt: z.string()\n .describe(\"The original prompt/instruction that generated the response\"),\n \n context: z.string().optional()\n .describe(\"Additional context like retrieved documents or conversation history\"),\n \n criteria: z.array(z.object({\n name: z.string().describe(\"Name of the criterion (e.g., 'Accuracy')\"),\n description: z.string().describe(\"What this criterion measures\"),\n weight: z.number().min(0).max(1).default(1)\n .describe(\"Relative importance, weights should sum to 1\")\n })).min(1).describe(\"Evaluation criteria to score against\"),\n \n rubric: z.object({\n scale: z.enum([\"1-3\", \"1-5\", \"1-10\"]).default(\"1-5\"),\n levelDescriptions: z.record(z.string(), z.string()).optional()\n .describe(\"Optional descriptions for each score level\")\n }).optional().describe(\"Scoring rubric configuration\")\n }),\n\n execute: async (input) => {\n // Implementation delegates to evaluator LLM\n return evaluateWithLLM(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| response | string | Yes | The response being evaluated |\n| prompt | string | Yes | Original prompt that generated the response |\n| context | string | No | Additional context (RAG docs, history) |\n| criteria | Criterion[] | Yes | List of evaluation criteria |\n| rubric | Rubric | No | Scoring scale and level descriptions |\n\n### Criterion Object\n```typescript\n{\n name: string; // e.g., \"Factual Accuracy\"\n description: string; // e.g., \"Response contains no factual errors\"\n weight: number; // 0-1, relative importance\n}\n```\n\n### Rubric Object\n```typescript\n{\n scale: \"1-3\" | \"1-5\" | \"1-10\";\n levelDescriptions?: {\n \"1\": \"Poor - Major issues\",\n \"2\": \"Below Average - Several issues\",\n \"3\": \"Average - Some issues\",\n \"4\": \"Good - Minor issues\",\n \"5\": \"Excellent - No issues\"\n }\n}\n```\n\n## Output Schema\n\n```typescript\ninterface DirectScoreResult {\n success: boolean;\n \n scores: {\n criterion: string;\n score: number;\n maxScore: number;\n justification: string;\n examples: string[]; // Specific examples from response\n }[];\n \n overallScore: number;\n weightedScore: number;\n \n summary: {\n strengths: string[];\n weaknesses: string[];\n suggestions: string[];\n };\n \n metadata: {\n evaluationTimeMs: number;\n criteriaCount: number;\n rubricScale: string;\n };\n}\n```\n\n## Usage Example\n\n```typescript\nconst result = await directScore.execute({\n response: \"Machine learning is a subset of AI that enables systems to learn from data...\",\n \n prompt: \"Explain machine learning to a beginner\",\n \n criteria: [\n {\n name: \"Accuracy\",\n description: \"Technical correctness of explanations\",\n weight: 0.4\n },\n {\n name: \"Clarity\",\n description: \"Understandable for a beginner\",\n weight: 0.3\n },\n {\n name: \"Completeness\",\n description: \"Covers key concepts adequately\",\n weight: 0.3\n }\n ],\n \n rubric: {\n scale: \"1-5\",\n levelDescriptions: {\n \"1\": \"Completely fails criterion\",\n \"2\": \"Major deficiencies\",\n \"3\": \"Acceptable but improvable\",\n \"4\": \"Good with minor issues\",\n \"5\": \"Excellent, no issues\"\n }\n }\n});\n```\n\n## Implementation Notes\n\n1. **Chain-of-Thought**: Implementation should use CoT prompting for more reliable scoring\n2. **Calibration**: Include few-shot examples of scores at each level\n3. **Justification First**: Ask for justification before score to reduce bias\n4. **Length Normalization**: Consider response length when appropriate\n\n","examples/llm-as-judge-skills/tools/evaluation/generate-rubric.md":"# Generate Rubric Tool\n\n## Purpose\n\nAutomatically generate a scoring rubric for a given evaluation criterion. Creates detailed descriptions for each score level to ensure consistent evaluation.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const generateRubric = tool({\n description: `Generate a scoring rubric for an evaluation criterion.\nCreates detailed descriptions for each score level.\nUse when you need to establish consistent evaluation standards.`,\n\n parameters: z.object({\n criterionName: z.string()\n .describe(\"Name of the criterion (e.g., 'Factual Accuracy')\"),\n \n criterionDescription: z.string()\n .describe(\"What this criterion measures\"),\n \n scale: z.enum([\"1-3\", \"1-5\", \"1-10\"]).default(\"1-5\")\n .describe(\"Scoring scale to use\"),\n \n domain: z.string().optional()\n .describe(\"Domain context (e.g., 'medical writing', 'code review')\"),\n \n includeExamples: z.boolean().default(true)\n .describe(\"Include example text for each score level\"),\n \n strictness: z.enum([\"lenient\", \"balanced\", \"strict\"]).default(\"balanced\")\n .describe(\"How strictly to define score boundaries\")\n }),\n\n execute: async (input) => {\n return generateRubricWithLLM(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| criterionName | string | Yes | Name of criterion |\n| criterionDescription | string | Yes | What criterion measures |\n| scale | enum | No | Scoring scale (default: 1-5) |\n| domain | string | No | Domain for context |\n| includeExamples | boolean | No | Include examples (default: true) |\n| strictness | enum | No | Score boundary strictness |\n\n## Output Schema\n\n```typescript\ninterface GeneratedRubric {\n success: boolean;\n \n criterion: {\n name: string;\n description: string;\n };\n \n scale: {\n min: number;\n max: number;\n type: string;\n };\n \n levels: {\n score: number;\n label: string; // e.g., \"Excellent\", \"Poor\"\n description: string; // Detailed description\n characteristics: string[]; // Key characteristics\n example?: string; // Example text at this level\n }[];\n \n scoringGuidelines: string[];\n \n edgeCases: {\n situation: string;\n guidance: string;\n }[];\n \n metadata: {\n domain: string | null;\n strictness: string;\n generationTimeMs: number;\n };\n}\n```\n\n## Usage Example\n\n```typescript\nconst rubric = await generateRubric.execute({\n criterionName: \"Code Readability\",\n criterionDescription: \"How easy the code is to read and understand\",\n scale: \"1-5\",\n domain: \"code review\",\n includeExamples: true,\n strictness: \"balanced\"\n});\n\n// Result:\n// {\n// criterion: {\n// name: \"Code Readability\",\n// description: \"How easy the code is to read and understand\"\n// },\n// scale: { min: 1, max: 5, type: \"1-5\" },\n// levels: [\n// {\n// score: 1,\n// label: \"Poor\",\n// description: \"Code is extremely difficult to understand...\",\n// characteristics: [\n// \"No meaningful variable names\",\n// \"Deeply nested logic without explanation\",\n// \"No comments on complex sections\"\n// ],\n// example: \"function x(a,b,c){return a?b+c:c-b;}\"\n// },\n// {\n// score: 5,\n// label: \"Excellent\",\n// description: \"Code is immediately understandable...\",\n// characteristics: [\n// \"Self-documenting variable and function names\",\n// \"Appropriate comments explaining 'why'\",\n// \"Clear logical structure\"\n// ],\n// example: \"function calculateShippingCost(weight, distance, expedited) {\\n // Base rate plus per-mile charge\\n const baseCost = weight * BASE_RATE_PER_KG;\\n ...\"\n// },\n// ...\n// ],\n// scoringGuidelines: [\n// \"Focus on clarity for someone unfamiliar with the codebase\",\n// \"Consider both naming and structure\",\n// \"Comments should explain 'why', not 'what'\"\n// ],\n// edgeCases: [\n// {\n// situation: \"Code uses domain-specific abbreviations\",\n// guidance: \"Accept if abbreviations are standard in the domain\"\n// }\n// ]\n// }\n```\n\n## Rubric Templates\n\n### Factual Accuracy (1-5)\n```\n5: All claims factually correct, properly sourced\n4: Minor factual issues, non-critical\n3: Some factual errors, main points correct\n2: Multiple factual errors affecting reliability\n1: Fundamentally incorrect or misleading\n```\n\n### Clarity (1-5)\n```\n5: Immediately understandable, well-structured\n4: Clear with minor ambiguities\n3: Generally clear, some confusing sections\n2: Difficult to follow, unclear organization\n1: Incomprehensible or incoherent\n```\n\n### Completeness (1-5)\n```\n5: Addresses all aspects comprehensively\n4: Covers main points, minor gaps\n3: Addresses core requirements, notable gaps\n2: Missing significant required elements\n1: Fails to address the question\n```\n\n## Implementation Notes\n\n1. **Domain Adaptation**: Rubrics should reflect domain-specific expectations\n2. **Boundary Clarity**: Clear distinctions between adjacent scores\n3. **Example Quality**: Examples should be realistic, not strawmen\n4. **Edge Case Coverage**: Anticipate common ambiguous situations\n5. **Calibration**: Test rubric against known samples before use\n\n","examples/llm-as-judge-skills/tools/evaluation/pairwise-compare.md":"# Pairwise Compare Tool\n\n## Purpose\n\nCompare two LLM responses and determine which one better satisfies the given criteria. More reliable for subjective evaluations than direct scoring.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const pairwiseCompare = tool({\n description: `Compare two responses and select the better one.\nUse for subjective evaluations like tone, persuasiveness, or writing style.\nMore reliable than direct scoring for preferences.\nReturns winner selection with detailed comparison.`,\n\n parameters: z.object({\n responseA: z.string()\n .describe(\"First response to compare\"),\n \n responseB: z.string()\n .describe(\"Second response to compare\"),\n \n prompt: z.string()\n .describe(\"The original prompt both responses address\"),\n \n context: z.string().optional()\n .describe(\"Additional context if relevant\"),\n \n criteria: z.array(z.string())\n .describe(\"Aspects to compare on, e.g., ['clarity', 'engagement', 'accuracy']\"),\n \n allowTie: z.boolean().default(true)\n .describe(\"Whether to allow a tie verdict\"),\n \n swapPositions: z.boolean().default(true)\n .describe(\"Evaluate twice with positions swapped to reduce position bias\")\n }),\n\n execute: async (input) => {\n if (input.swapPositions) {\n return evaluateWithPositionSwap(input);\n }\n return evaluatePairwise(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| responseA | string | Yes | First response |\n| responseB | string | Yes | Second response |\n| prompt | string | Yes | Original prompt |\n| context | string | No | Additional context |\n| criteria | string[] | Yes | Comparison dimensions |\n| allowTie | boolean | No | Allow tie verdict (default: true) |\n| swapPositions | boolean | No | Swap positions to reduce bias (default: true) |\n\n## Output Schema\n\n```typescript\ninterface PairwiseCompareResult {\n success: boolean;\n \n winner: \"A\" | \"B\" | \"TIE\";\n confidence: number; // 0-1\n \n comparison: {\n criterion: string;\n winner: \"A\" | \"B\" | \"TIE\";\n reasoning: string;\n aStrength: string;\n bStrength: string;\n }[];\n \n overallReasoning: string;\n \n differentiators: {\n aAdvantages: string[];\n bAdvantages: string[];\n };\n \n // If swapPositions was true\n positionConsistency?: {\n firstPassWinner: \"A\" | \"B\" | \"TIE\";\n secondPassWinner: \"A\" | \"B\" | \"TIE\";\n consistent: boolean;\n };\n \n metadata: {\n evaluationTimeMs: number;\n positionsSwapped: boolean;\n };\n}\n```\n\n## Usage Example\n\n```typescript\nconst result = await pairwiseCompare.execute({\n responseA: \"Exercise improves cardiovascular health, builds muscle, and boosts mental wellbeing...\",\n \n responseB: \"Working out regularly has many benefits. You'll feel better and look better...\",\n \n prompt: \"Explain the benefits of regular exercise\",\n \n criteria: [\"accuracy\", \"specificity\", \"engagement\", \"completeness\"],\n \n allowTie: true,\n swapPositions: true\n});\n\n// Result:\n// {\n// winner: \"A\",\n// confidence: 0.85,\n// comparison: [\n// {\n// criterion: \"accuracy\",\n// winner: \"A\",\n// reasoning: \"Response A uses specific medical terminology...\",\n// aStrength: \"Mentions cardiovascular, muscle, mental health\",\n// bStrength: \"General but not incorrect\"\n// },\n// ...\n// ],\n// ...\n// }\n```\n\n## Position Swap Algorithm\n\nTo mitigate position bias:\n\n```typescript\nasync function evaluateWithPositionSwap(input) {\n // First pass: Original order\n const pass1 = await evaluate({\n first: input.responseA,\n second: input.responseB,\n ...input\n });\n \n // Second pass: Swapped order\n const pass2 = await evaluate({\n first: input.responseB,\n second: input.responseA,\n ...input\n });\n \n // Reconcile results\n const pass2Adjusted = pass2.winner === \"A\" ? \"B\" : pass2.winner === \"B\" ? \"A\" : \"TIE\";\n \n if (pass1.winner === pass2Adjusted) {\n return {\n ...pass1,\n positionConsistency: { consistent: true, ... }\n };\n } else {\n // Inconsistent - return tie or lower confidence\n return {\n winner: \"TIE\",\n confidence: 0.5,\n positionConsistency: { consistent: false, ... },\n ...\n };\n }\n}\n```\n\n## Implementation Notes\n\n1. **Position Bias Mitigation**: Always use `swapPositions: true` for production\n2. **Criteria Order**: Order criteria by importance for better focus\n3. **Tie Handling**: Consider domain - some tasks should rarely tie\n4. **Confidence Calibration**: Lower confidence when evaluations are close\n5. **Length Considerations**: Note if one response is significantly longer\n\n","examples/llm-as-judge-skills/tools/index.md":"# Tools Index\n\nTools provide specific capabilities that agents can use to accomplish tasks.\n\n## Tool Categories\n\n### Evaluation Tools\n**Path**: `tools/evaluation/`\n\nTools for assessing LLM output quality.\n\n| Tool | Purpose | Approval |\n|------|---------|----------|\n| `directScore` | Score response against criteria | No |\n| `pairwiseCompare` | Compare two responses | No |\n| `generateRubric` | Generate scoring rubric | No |\n| `extractCriteria` | Extract criteria from task | No |\n\n---\n\n### Research Tools\n**Path**: `tools/research/`\n\nTools for gathering and processing information.\n\n| Tool | Purpose | Approval |\n|------|---------|----------|\n| `webSearch` | Search the web | No |\n| `readUrl` | Extract content from URL | No |\n| `extractClaims` | Identify claims in text | No |\n| `verifyClaim` | Cross-reference a claim | No |\n| `synthesize` | Combine findings | No |\n\n---\n\n### Orchestration Tools\n**Path**: `tools/orchestration/`\n\nTools for managing multi-agent workflows.\n\n| Tool | Purpose | Approval |\n|------|---------|----------|\n| `delegateToAgent` | Route task to agent | No |\n| `parallelExecution` | Run tasks concurrently | No |\n| `waitForCompletion` | Wait for async tasks | No |\n| `synthesizeResults` | Combine agent outputs | No |\n| `handleError` | Manage failures | No |\n\n## Tool Design Patterns\n\n### Standard Tool Structure\n\n```typescript\nexport const toolName = tool({\n description: \"Clear description of what tool does\",\n \n parameters: z.object({\n // Required parameters first\n requiredParam: z.string().describe(\"What this parameter is for\"),\n \n // Optional parameters with defaults\n optionalParam: z.number().default(10)\n .describe(\"What this parameter controls\")\n }),\n \n // Approval for dangerous operations\n needsApproval: false, // or true, or function\n \n // Strict mode for guaranteed schema compliance\n strict: true,\n \n execute: async (input) => {\n try {\n const result = await performOperation(input);\n return { success: true, data: result };\n } catch (error) {\n return {\n success: false,\n error: {\n code: error.code ?? \"UNKNOWN\",\n message: error.message,\n retryable: isRetryable(error)\n }\n };\n }\n },\n \n // Optional: control what model sees\n toModelOutput: (result) => ({\n summary: result.data.summary,\n truncated: result.data.full.length > 5000\n })\n});\n```\n\n### Error Response Pattern\n\n```typescript\ninterface ToolError {\n code: string; // Machine-readable error code\n message: string; // Human-readable message\n retryable: boolean; // Whether retry might help\n details?: object; // Additional context\n}\n\ninterface ToolResult {\n success: boolean;\n data?: T;\n error?: ToolError;\n metadata: {\n executionTimeMs: number;\n [key: string]: any;\n };\n}\n```\n\n## Adding New Tools\n\n1. Determine category or create new: `tools//`\n2. Create tool file: `tools//.md`\n3. Define:\n - Purpose and description\n - Input parameters with Zod schema\n - Output schema\n - Error codes\n - Usage examples\n4. Update this index\n5. Assign to relevant agents\n\n## Tool Selection Guidelines\n\n### When Agent Needs To...\n\n| Action | Tool Category | Suggested Tools |\n|--------|---------------|-----------------|\n| Assess quality | Evaluation | directScore, pairwiseCompare |\n| Find information | Research | webSearch, readUrl |\n| Verify facts | Research | verifyClaim, extractClaims |\n| Coordinate work | Orchestration | delegateToAgent |\n| Wait for results | Orchestration | waitForCompletion |\n\n","examples/llm-as-judge-skills/tools/orchestration/delegate-to-agent.md":"# Delegate to Agent Tool\n\n## Purpose\n\nRoute a task to a specialized agent for execution. Handles context passing, result collection, and error management.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const delegateToAgent = tool({\n description: `Delegate a task to a specialized agent.\nUse when a subtask requires specific capabilities.\nPass complete context needed for the agent to succeed.`,\n\n parameters: z.object({\n agentName: z.enum([\"evaluator\", \"researcher\", \"writer\", \"analyst\"])\n .describe(\"Name of the agent to delegate to\"),\n \n task: z.string()\n .describe(\"Clear description of what the agent should do\"),\n \n context: z.object({\n previousOutputs: z.array(z.string()).optional()\n .describe(\"Outputs from prior steps this agent needs\"),\n \n documents: z.array(z.string()).optional()\n .describe(\"Relevant documents or data\"),\n \n constraints: z.array(z.string()).optional()\n .describe(\"Requirements or limitations to observe\")\n }).optional(),\n \n expectedOutput: z.object({\n format: z.enum([\"text\", \"json\", \"markdown\", \"structured\"])\n .describe(\"Expected output format\"),\n \n schema: z.string().optional()\n .describe(\"JSON schema if format is structured\"),\n \n maxLength: z.number().optional()\n .describe(\"Maximum length constraint\")\n }).optional(),\n \n timeout: z.number().default(60000)\n .describe(\"Timeout in milliseconds\")\n }),\n\n execute: async (input) => {\n return executeAgentDelegation(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| agentName | enum | Yes | Target agent |\n| task | string | Yes | Task description |\n| context | object | No | Context and dependencies |\n| expectedOutput | object | No | Output requirements |\n| timeout | number | No | Timeout ms (default: 60000) |\n\n## Output Schema\n\n```typescript\ninterface DelegationResult {\n success: boolean;\n \n agentName: string;\n task: string;\n \n output: {\n content: string | object;\n format: string;\n };\n \n execution: {\n startTime: string;\n endTime: string;\n durationMs: number;\n tokenUsage: {\n prompt: number;\n completion: number;\n };\n };\n \n error?: {\n code: string;\n message: string;\n retryable: boolean;\n };\n}\n```\n\n## Available Agents\n\n### Evaluator Agent\n```typescript\nawait delegateToAgent.execute({\n agentName: \"evaluator\",\n task: \"Evaluate the quality of this response against accuracy and clarity criteria\",\n context: {\n documents: [responseToEvaluate],\n constraints: [\"Use 1-5 scale\", \"Include justification\"]\n },\n expectedOutput: { format: \"structured\" }\n});\n```\n\n### Researcher Agent\n```typescript\nawait delegateToAgent.execute({\n agentName: \"researcher\",\n task: \"Research current best practices for LLM evaluation\",\n context: {\n constraints: [\"Focus on 2024 publications\", \"Include citations\"]\n },\n expectedOutput: { format: \"markdown\" }\n});\n```\n\n### Writer Agent\n```typescript\nawait delegateToAgent.execute({\n agentName: \"writer\",\n task: \"Write an executive summary of these research findings\",\n context: {\n previousOutputs: [researchFindings],\n constraints: [\"Maximum 500 words\", \"Non-technical audience\"]\n },\n expectedOutput: { format: \"text\", maxLength: 2500 }\n});\n```\n\n### Analyst Agent\n```typescript\nawait delegateToAgent.execute({\n agentName: \"analyst\",\n task: \"Analyze the trade-offs between direct scoring and pairwise comparison\",\n context: {\n documents: [evaluationData],\n constraints: [\"Quantitative where possible\"]\n },\n expectedOutput: { format: \"structured\" }\n});\n```\n\n## Error Handling\n\n```typescript\nconst errorCodes = {\n \"AGENT_NOT_FOUND\": \"Specified agent does not exist\",\n \"TIMEOUT\": \"Agent did not complete within timeout\",\n \"CONTEXT_TOO_LARGE\": \"Context exceeds agent's capacity\",\n \"INVALID_OUTPUT\": \"Agent output did not match expected format\",\n \"AGENT_ERROR\": \"Agent encountered an error during execution\"\n};\n```\n\n## Implementation Notes\n\n1. **Context Optimization**: Compress context if needed before passing\n2. **Timeout Handling**: Set realistic timeouts per agent type\n3. **Retry Logic**: Implement retries for transient failures\n4. **Audit Trail**: Log all delegations for traceability\n5. **Resource Management**: Track token usage across delegations\n\n","examples/llm-as-judge-skills/tools/research/read-url.md":"# Read URL Tool\n\n## Purpose\n\nExtract and parse content from a given URL. Returns structured text content with metadata about the source.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const readUrl = tool({\n description: `Read and extract content from a URL.\nReturns the main text content, stripped of navigation and ads.\nUse after webSearch to get full content from relevant results.`,\n\n parameters: z.object({\n url: z.string().url()\n .describe(\"The URL to read\"),\n \n contentType: z.enum([\"auto\", \"article\", \"documentation\", \"paper\", \"code\"]).default(\"auto\")\n .describe(\"Hint for content type to optimize extraction\"),\n \n maxLength: z.number().min(1000).max(50000).default(10000)\n .describe(\"Maximum characters to return\"),\n \n extractSections: z.boolean().default(true)\n .describe(\"Whether to identify and label sections\"),\n \n includeMetadata: z.boolean().default(true)\n .describe(\"Include author, date, and other metadata\")\n }),\n\n execute: async (input) => {\n return extractUrlContent(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| url | string | Yes | URL to read |\n| contentType | enum | No | Content type hint |\n| maxLength | number | No | Max chars (default: 10000) |\n| extractSections | boolean | No | Label sections |\n| includeMetadata | boolean | No | Include metadata |\n\n## Output Schema\n\n```typescript\ninterface ReadUrlResult {\n success: boolean;\n \n url: string;\n title: string;\n \n content: {\n full: string;\n sections?: {\n heading: string;\n level: number; // h1=1, h2=2, etc.\n content: string;\n }[];\n };\n \n metadata?: {\n author?: string;\n publishedDate?: string;\n lastModified?: string;\n description?: string;\n keywords?: string[];\n source: string;\n };\n \n stats: {\n totalCharacters: number;\n truncated: boolean;\n sectionsFound: number;\n };\n \n error?: {\n code: string;\n message: string;\n };\n}\n```\n\n## Usage Example\n\n```typescript\nconst content = await readUrl.execute({\n url: \"https://eugeneyan.com/writing/llm-evaluators/\",\n contentType: \"article\",\n maxLength: 15000,\n extractSections: true,\n includeMetadata: true\n});\n\n// Result:\n// {\n// success: true,\n// url: \"https://eugeneyan.com/writing/llm-evaluators/\",\n// title: \"Evaluating the Effectiveness of LLM-Evaluators\",\n// content: {\n// full: \"LLM-evaluators, also known as LLM-as-a-Judge...\",\n// sections: [\n// {\n// heading: \"Key considerations before adopting an LLM-evaluator\",\n// level: 2,\n// content: \"Before reviewing the literature...\"\n// },\n// ...\n// ]\n// },\n// metadata: {\n// author: \"Eugene Yan\",\n// publishedDate: \"2024-06-15\",\n// source: \"eugeneyan.com\"\n// },\n// stats: {\n// totalCharacters: 15000,\n// truncated: true,\n// sectionsFound: 8\n// }\n// }\n```\n\n## Content Type Handling\n\n| Type | Optimization |\n|------|-------------|\n| article | Prioritize main content, skip sidebars |\n| documentation | Preserve code blocks, keep structure |\n| paper | Extract abstract, sections, references |\n| code | Preserve formatting, syntax highlighting |\n| auto | Detect type from content |\n\n## Error Handling\n\n```typescript\nconst errorCodes = {\n \"URL_NOT_FOUND\": \"Page does not exist (404)\",\n \"ACCESS_DENIED\": \"Page requires authentication (401/403)\",\n \"TIMEOUT\": \"Request timed out\",\n \"BLOCKED\": \"Access blocked by robots.txt or rate limit\",\n \"INVALID_CONTENT\": \"Content could not be parsed\",\n \"UNSUPPORTED_TYPE\": \"Content type not supported (e.g., binary)\"\n};\n```\n\n## Implementation Notes\n\n1. **Respect robots.txt**: Check and honor robots.txt directives\n2. **Rate Limiting**: Don't hammer the same domain\n3. **User Agent**: Use appropriate user agent string\n4. **Timeouts**: Set reasonable timeouts (10-30s)\n5. **JavaScript Rendering**: Consider headless browser for JS-heavy sites\n6. **Caching**: Cache content for repeated reads\n\n","examples/llm-as-judge-skills/tools/research/web-search.md":"# Web Search Tool\n\n## Purpose\n\nSearch the web for relevant information on a given topic. Returns structured results with snippets, URLs, and metadata.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const webSearch = tool({\n description: `Search the web for information on a topic.\nReturns relevant results with snippets and URLs.\nUse for gathering current information, verifying facts, or research.`,\n\n parameters: z.object({\n query: z.string()\n .describe(\"Search query - be specific for better results\"),\n \n maxResults: z.number().min(1).max(20).default(10)\n .describe(\"Maximum number of results to return\"),\n \n filters: z.object({\n dateRange: z.enum([\"day\", \"week\", \"month\", \"year\", \"any\"]).default(\"any\")\n .describe(\"Limit results to a time period\"),\n \n sourceType: z.enum([\"all\", \"news\", \"academic\", \"documentation\"]).default(\"all\")\n .describe(\"Type of sources to prioritize\"),\n \n excludeDomains: z.array(z.string()).optional()\n .describe(\"Domains to exclude from results\")\n }).optional()\n }),\n\n execute: async (input) => {\n return performWebSearch(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| query | string | Yes | Search query |\n| maxResults | number | No | Max results (default: 10) |\n| filters.dateRange | enum | No | Time period filter |\n| filters.sourceType | enum | No | Source type priority |\n| filters.excludeDomains | string[] | No | Domains to exclude |\n\n## Output Schema\n\n```typescript\ninterface WebSearchResult {\n success: boolean;\n \n results: {\n title: string;\n url: string;\n snippet: string;\n source: string; // Domain name\n publishedDate?: string;\n relevanceScore: number;\n }[];\n \n totalResults: number;\n \n metadata: {\n query: string;\n searchTimeMs: number;\n filtersApplied: string[];\n };\n}\n```\n\n## Usage Example\n\n```typescript\nconst results = await webSearch.execute({\n query: \"LLM-as-a-Judge evaluation methods 2024\",\n maxResults: 10,\n filters: {\n dateRange: \"year\",\n sourceType: \"academic\"\n }\n});\n\n// Result:\n// {\n// success: true,\n// results: [\n// {\n// title: \"Judging LLM-as-a-Judge with MT-Bench\",\n// url: \"https://arxiv.org/abs/...\",\n// snippet: \"We study the effectiveness of LLM-as-a-Judge...\",\n// source: \"arxiv.org\",\n// publishedDate: \"2024-01-15\",\n// relevanceScore: 0.95\n// },\n// ...\n// ],\n// totalResults: 10,\n// metadata: {\n// query: \"LLM-as-a-Judge evaluation methods 2024\",\n// searchTimeMs: 342,\n// filtersApplied: [\"dateRange:year\", \"sourceType:academic\"]\n// }\n// }\n```\n\n## Query Optimization Tips\n\n1. **Specific Terms**: Use precise terminology\n2. **Quotes**: Use quotes for exact phrases\n3. **Operators**: Support site:, -term, OR\n4. **Context**: Include relevant context terms\n5. **Recency**: Add year for recent info\n\n## Implementation Notes\n\n1. **Rate Limiting**: Implement appropriate rate limits\n2. **Caching**: Cache results for repeated queries\n3. **Result Quality**: Filter out low-quality sources\n4. **Error Handling**: Handle API failures gracefully\n5. **Privacy**: Log queries appropriately\n\n","skills/advanced-evaluation/SKILL.md":"---\nname: advanced-evaluation\ndescription: \"This skill should be used for advanced LLM evaluation: LLM-as-judge systems, direct scoring, pairwise comparison, rubric calibration, evaluator bias mitigation, confidence scoring, and automated quality assessment.\"\n---\n\n# Advanced Evaluation\n\nThis skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.\n\n**Key insight**: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.\n\n## When to Activate\n\nActivate this skill when:\n\n- Building LLM-as-judge systems for LLM outputs\n- Comparing multiple model responses to select the best one\n- Establishing consistent quality standards across evaluation teams\n- Debugging evaluation systems that show inconsistent results\n- Designing A/B tests for prompt or model changes\n- Creating rubrics specifically for LLM or human/LLM hybrid judges\n- Analyzing correlation between automated and human judgments\n\nDo not activate this skill for adjacent work owned by other skills:\n- General deterministic checks, regression suites, production quality gates, or outcome metrics: `evaluation`.\n- Autonomous loop governance, locked rubrics, rollback, or PR approval boundaries: `harness-engineering`.\n- Tool API contracts for evaluation tools: `tool-design`.\n\n## Core Concepts\n\n### The Evaluation Taxonomy\n\nSelect between two primary approaches based on whether ground truth exists:\n\n**Direct Scoring** — Use when objective criteria exist (factual accuracy, instruction following, toxicity). A single LLM rates one response on a defined scale. Achieves moderate-to-high reliability for well-defined criteria. Watch for score calibration drift and inconsistent scale interpretation.\n\n**Pairwise Comparison** — Use for subjective preferences (tone, style, persuasiveness). An LLM compares two responses and selects the better one. Pairwise methods often correlate better with human preference than open-ended direct scoring for subjective tasks (claim-advanced-evaluation-position-swap). Watch for position bias and length bias.\n\n### The Bias Landscape\n\nMitigate these systematic biases in every evaluation system:\n\n**Position Bias**: First-position responses get preferential treatment. Mitigate by evaluating twice with swapped positions, then apply majority vote or consistency check.\n\n**Length Bias**: Longer responses score higher regardless of quality. Mitigate by explicitly prompting to ignore length and applying length-normalized scoring.\n\n**Self-Enhancement Bias**: Models rate their own outputs higher. Mitigate by using different models for generation and evaluation.\n\n**Verbosity Bias**: Excessive detail scores higher even when unnecessary. Mitigate with criteria-specific rubrics that penalize irrelevant detail.\n\n**Authority Bias**: Confident tone scores higher regardless of accuracy. Mitigate by requiring evidence citation and adding a fact-checking layer.\n\n### Metric Selection Framework\n\nMatch metrics to the evaluation task structure:\n\n| Task Type | Primary Metrics | Secondary Metrics |\n|-----------|-----------------|-------------------|\n| Binary classification (pass/fail) | Recall, Precision, F1 | Cohen's kappa |\n| Ordinal scale (1-5 rating) | Spearman's rho, Kendall's tau | Cohen's kappa (weighted) |\n| Pairwise preference | Agreement rate, Position consistency | Confidence calibration |\n| Multi-label | Macro-F1, Micro-F1 | Per-label precision/recall |\n\nPrioritize systematic disagreement patterns over absolute agreement rates because a judge that consistently disagrees with humans on specific criteria is more problematic than one with random noise.\n\n## Evaluation Approaches\n\n### Direct Scoring Implementation\n\nBuild direct scoring with three components: clear criteria, a calibrated scale, and structured output format.\n\n**Criteria Definition Pattern**:\n```\nCriterion: [Name]\nDescription: [What this criterion measures]\nWeight: [Relative importance, 0-1]\n```\n\n**Scale Calibration** — Choose scale granularity based on rubric detail:\n- 1-3: Binary with neutral option, lowest cognitive load\n- 1-5: Standard Likert, best balance of granularity and reliability\n- 1-10: Use only with detailed per-level rubrics because calibration is harder\n\n**Prompt Structure for Direct Scoring**:\n```\nYou are an expert evaluator assessing response quality.\n\n## Task\nEvaluate the following response against each criterion.\n\n## Original Prompt\n{prompt}\n\n## Response to Evaluate\n{response}\n\n## Criteria\n{for each criterion: name, description, weight}\n\n## Instructions\nFor each criterion:\n1. Find specific evidence in the response\n2. Score according to the rubric (1-{max} scale)\n3. Justify your score with evidence\n4. Suggest one specific improvement\n\n## Output Format\nRespond with structured JSON containing scores, justifications, and summary.\n```\n\nRequire evidence before the score in scoring prompts so the judge must anchor its decision in observable output features before emitting a number.\n\n### Pairwise Comparison Implementation\n\nApply position bias mitigation in every pairwise evaluation:\n\n1. Run deterministic pre-checks first: both candidates must satisfy the same schema, source-evidence requirements, and scope constraints.\n2. First judge pass: Response A in first position, Response B in second.\n3. Second judge pass: Response B in first position, Response A in second.\n4. Consistency check: If passes disagree, return TIE with reduced confidence.\n5. Final verdict: Consistent winner with averaged confidence and explicit tie-breaker rationale.\n\n**Prompt Structure for Pairwise Comparison**:\n```\nYou are an expert evaluator comparing two AI responses.\n\n## Critical Instructions\n- Do NOT prefer responses because they are longer\n- Do NOT prefer responses based on position (first vs second)\n- Focus ONLY on quality according to the specified criteria\n- Ties are acceptable when responses are genuinely equivalent\n\n## Original Prompt\n{prompt}\n\n## Response A\n{response_a}\n\n## Response B\n{response_b}\n\n## Comparison Criteria\n{criteria list}\n\n## Instructions\n1. Analyze each response independently first\n2. Compare them on each criterion\n3. Determine overall winner with confidence level\n\n## Output Format\nJSON with per-criterion comparison, overall winner, confidence (0-1), and reasoning.\n```\n\n**Confidence Calibration** — Map confidence to position consistency:\n- Both passes agree: confidence = average of individual confidences\n- Passes disagree: confidence = 0.5, verdict = TIE\n\n### Rubric Generation\n\nGenerate rubrics to reduce evaluation variance compared to open-ended scoring. Treat exact variance reduction as workload-specific unless measured on the target eval set.\n\n**Include these rubric components**:\n1. **Level descriptions**: Clear boundaries for each score level\n2. **Characteristics**: Observable features that define each level\n3. **Examples**: Representative text for each level (optional but valuable)\n4. **Edge cases**: Guidance for ambiguous situations\n5. **Scoring guidelines**: General principles for consistent application\n\n**Set strictness calibration** for the use case:\n- **Lenient**: Lower passing bar, appropriate for encouraging iteration\n- **Balanced**: Typical production expectations\n- **Strict**: High standards for safety-critical or high-stakes evaluation\n\nAdapt rubrics to the domain — use domain-specific terminology. A code readability rubric mentions variables, functions, and comments. A medical accuracy rubric references clinical terminology and evidence standards.\n\n## Practical Guidance\n\n### Evaluation Pipeline Design\n\nBuild production evaluation systems with these layers: Criteria Loader (rubrics + weights) -> Primary Scorer (direct or pairwise) -> Bias Mitigation (position swap, etc.) -> Confidence Scoring (calibration) -> Output (scores + justifications + confidence). See [Evaluation Pipeline Diagram](./references/evaluation-pipeline.md) for the full visual layout.\n\n### Decision Framework: Direct vs. Pairwise\n\nApply this decision tree:\n\n```\nIs there an objective ground truth?\n+-- Yes -> Direct Scoring\n| Examples: factual accuracy, instruction following, format compliance\n|\n+-- No -> Is it a preference or quality judgment?\n +-- Yes -> Pairwise Comparison\n | Examples: tone, style, persuasiveness, creativity\n |\n +-- No -> Consider reference-based evaluation\n Examples: summarization (compare to source), translation (compare to reference)\n```\n\n### Scaling Evaluation\n\nFor high-volume evaluation, apply one of these strategies:\n\n1. **Panel of LLMs (PoLL)**: Use multiple models as judges and aggregate votes to reduce individual model bias. More expensive but more reliable for high-stakes decisions.\n\n2. **Hierarchical evaluation**: Use a fast cheap model for screening and an expensive model for edge cases. Requires calibration of the screening threshold.\n\n3. **Human-in-the-loop**: Automate clear cases and route low-confidence decisions to human review. Design feedback loops to improve automated evaluation over time.\n\n## Examples\n\n### Example 1: Direct Scoring for Accuracy\n\n**Input**:\n```\nPrompt: \"What causes seasons on Earth?\"\nResponse: \"Seasons are caused by Earth's tilted axis. As Earth orbits the Sun,\ndifferent hemispheres receive more direct sunlight at different times of year.\"\nCriterion: Factual Accuracy (weight: 1.0)\nScale: 1-5\n```\n\n**Output**:\n```json\n{\n \"criterion\": \"Factual Accuracy\",\n \"score\": 5,\n \"evidence\": [\n \"Correctly identifies axial tilt as primary cause\",\n \"Correctly explains differential sunlight by hemisphere\",\n \"No factual errors present\"\n ],\n \"justification\": \"Response accurately explains the cause of seasons with correct\nscientific reasoning. Both the axial tilt and its effect on sunlight distribution\nare correctly described.\",\n \"improvement\": \"Could add the specific tilt angle (23.5 degrees) for completeness.\"\n}\n```\n\n### Example 2: Pairwise Comparison with Position Swap\n\n**Input**:\n```\nPrompt: \"Explain machine learning to a beginner\"\nResponse A: [Technical explanation with jargon]\nResponse B: [Simple analogy-based explanation]\nCriteria: [\"clarity\", \"accessibility\"]\n```\n\n**First Pass (A first)**:\n```json\n{ \"winner\": \"B\", \"confidence\": 0.8 }\n```\n\n**Second Pass (B first)**:\n```json\n{ \"winner\": \"A\", \"confidence\": 0.6 }\n```\n(Note: Winner is A because B was in first position)\n\n**Mapped Second Pass**:\n```json\n{ \"winner\": \"B\", \"confidence\": 0.6 }\n```\n\n**Final Result**:\n```json\n{\n \"winner\": \"B\",\n \"confidence\": 0.7,\n \"positionConsistency\": {\n \"consistent\": true,\n \"firstPassWinner\": \"B\",\n \"secondPassWinner\": \"B\"\n }\n}\n```\n\n### Example 3: Rubric Generation\n\n**Input**:\n```\ncriterionName: \"Code Readability\"\ncriterionDescription: \"How easy the code is to understand and maintain\"\ndomain: \"software engineering\"\nscale: \"1-5\"\nstrictness: \"balanced\"\n```\n\n**Output** (abbreviated):\n```json\n{\n \"levels\": [\n {\n \"score\": 1,\n \"label\": \"Poor\",\n \"description\": \"Code is difficult to understand without significant effort\",\n \"characteristics\": [\n \"No meaningful variable or function names\",\n \"No comments or documentation\",\n \"Deeply nested or convoluted logic\"\n ]\n },\n {\n \"score\": 3,\n \"label\": \"Adequate\",\n \"description\": \"Code is understandable with some effort\",\n \"characteristics\": [\n \"Most variables have meaningful names\",\n \"Basic comments present for complex sections\",\n \"Logic is followable but could be cleaner\"\n ]\n },\n {\n \"score\": 5,\n \"label\": \"Excellent\",\n \"description\": \"Code is immediately clear and maintainable\",\n \"characteristics\": [\n \"All names are descriptive and consistent\",\n \"Comprehensive documentation\",\n \"Clean, modular structure\"\n ]\n }\n ],\n \"edgeCases\": [\n {\n \"situation\": \"Code is well-structured but uses domain-specific abbreviations\",\n \"guidance\": \"Score based on readability for domain experts, not general audience\"\n }\n ]\n}\n```\n\n## Guidelines\n\n1. **Always require evidence before scores** - Evidence-first prompts make judgments easier to audit and reduce ungrounded numeric scoring\n\n2. **Always swap positions in pairwise comparison** - Single-pass comparison is corrupted by position bias\n\n3. **Match scale granularity to rubric specificity** - Don't use 1-10 without detailed level descriptions\n\n4. **Separate objective and subjective criteria** - Use direct scoring for objective, pairwise for subjective\n\n5. **Include confidence scores** - Calibrate to position consistency and evidence strength\n\n6. **Define edge cases explicitly** - Ambiguous situations cause the most evaluation variance\n\n7. **Use domain-specific rubrics** - Generic rubrics produce generic (less useful) evaluations\n\n8. **Validate against human judgments** - Automated evaluation is only valuable if it correlates with human assessment\n\n9. **Monitor for systematic bias** - Track disagreement patterns by criterion, response type, model\n\n10. **Design for iteration** - Evaluation systems improve with feedback loops\n\n## Gotchas\n\n1. **Scoring without justification**: Scores lack grounding and are difficult to debug. Always require evidence-based justification before the score.\n\n2. **Single-pass pairwise comparison**: Position bias corrupts results when positions are not swapped. Always evaluate twice with swapped positions and check consistency.\n\n3. **Overloaded criteria**: Criteria that measure multiple things at once produce unreliable scores. Enforce one criterion = one measurable aspect.\n\n4. **Missing edge case guidance**: Evaluators handle ambiguous cases inconsistently without explicit instructions. Include edge cases in rubrics with clear resolution rules.\n\n5. **Ignoring confidence calibration**: High-confidence wrong judgments are worse than low-confidence ones. Calibrate confidence to position consistency and evidence strength.\n\n6. **Rubric drift**: Rubrics become miscalibrated as quality standards evolve or model capabilities improve. Schedule periodic rubric reviews and re-anchor score levels against fresh human-annotated examples.\n\n7. **Evaluation prompt sensitivity**: Minor wording changes in evaluation prompts can cause material score swings. Version-control evaluation prompts and run regression tests before deploying prompt changes.\n\n8. **Uncontrolled length bias**: Longer responses systematically score higher even when conciseness is preferred. Add explicit length-neutrality instructions to evaluation prompts and validate with length-controlled test pairs.\n\n## Integration\n\nThis skill owns judge design and bias mitigation. Adjacent skills own broader quality gates and infrastructure:\n\n- `evaluation`: general deterministic checks, regression suites, quality gates, and production monitoring.\n- `context-fundamentals`: context structure for judge prompts.\n- `tool-design`: schemas and error handling for evaluation tools.\n- `context-optimization`: token and latency efficiency for high-volume evals.\n- `harness-engineering`: locked evaluator surfaces and governance for autonomous loops.\n\n## References\n\nInternal reference:\n- [LLM-as-Judge Implementation Patterns](./references/implementation-patterns.md) - Read when: building an evaluation pipeline from scratch or integrating LLM judges into CI/CD\n- [Bias Mitigation Techniques](./references/bias-mitigation.md) - Read when: evaluation results show inconsistent or suspicious scoring patterns\n- [Metric Selection Guide](./references/metrics-guide.md) - Read when: choosing statistical metrics to validate evaluation reliability\n- [Evaluation Pipeline Diagram](./references/evaluation-pipeline.md) - Read when: designing the architecture of a multi-stage evaluation system\n\nExternal research:\n- [Eugene Yan: Evaluating the Effectiveness of LLM-Evaluators](https://eugeneyan.com/writing/llm-evaluators/) - Read when: surveying the state of the art in LLM evaluation\n- [Judging LLM-as-a-Judge (Zheng et al., 2023)](https://arxiv.org/abs/2306.05685) - Read when: understanding position bias and MT-Bench methodology\n- [G-Eval: NLG Evaluation using GPT-4 (Liu et al., 2023)](https://arxiv.org/abs/2303.16634) - Read when: implementing chain-of-thought evaluation scoring\n- [Large Language Models are not Fair Evaluators (Wang et al., 2023)](https://arxiv.org/abs/2305.17926) - Read when: diagnosing systematic bias in evaluation outputs\n\nRelated skills in this collection:\n- evaluation - Foundational evaluation concepts\n- context-fundamentals - Context structure for evaluation prompts\n- tool-design - Building evaluation tools\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-24\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.1.0\n","skills/advanced-evaluation/references/bias-mitigation.md":"# Bias Mitigation Techniques for LLM Evaluation\n\nThis reference details specific techniques for mitigating known biases in LLM-as-a-Judge systems.\n\n## Position Bias\n\n### The Problem\n\nIn pairwise comparison, LLMs systematically prefer responses in certain positions. Research shows:\n- GPT has mild first-position bias (~55% preference for first position in ties)\n- Claude shows similar patterns\n- Smaller models often show stronger bias\n\n### Mitigation: Position Swapping Protocol\n\n```python\nasync def position_swap_comparison(response_a, response_b, prompt, criteria):\n # Pass 1: Original order\n result_ab = await compare(response_a, response_b, prompt, criteria)\n \n # Pass 2: Swapped order\n result_ba = await compare(response_b, response_a, prompt, criteria)\n \n # Map second result (A in second position → B in first)\n result_ba_mapped = {\n 'winner': {'A': 'B', 'B': 'A', 'TIE': 'TIE'}[result_ba['winner']],\n 'confidence': result_ba['confidence']\n }\n \n # Consistency check\n if result_ab['winner'] == result_ba_mapped['winner']:\n return {\n 'winner': result_ab['winner'],\n 'confidence': (result_ab['confidence'] + result_ba_mapped['confidence']) / 2,\n 'position_consistent': True\n }\n else:\n # Disagreement indicates position bias was a factor\n return {\n 'winner': 'TIE',\n 'confidence': 0.5,\n 'position_consistent': False,\n 'bias_detected': True\n }\n```\n\n### Alternative: Multiple Shuffles\n\nFor higher reliability, use multiple position orderings:\n\n```python\nasync def multi_shuffle_comparison(response_a, response_b, prompt, criteria, n_shuffles=3):\n results = []\n for i in range(n_shuffles):\n if i % 2 == 0:\n r = await compare(response_a, response_b, prompt, criteria)\n else:\n r = await compare(response_b, response_a, prompt, criteria)\n r['winner'] = {'A': 'B', 'B': 'A', 'TIE': 'TIE'}[r['winner']]\n results.append(r)\n \n # Majority vote\n winners = [r['winner'] for r in results]\n final_winner = max(set(winners), key=winners.count)\n agreement = winners.count(final_winner) / len(winners)\n \n return {\n 'winner': final_winner,\n 'confidence': agreement,\n 'n_shuffles': n_shuffles\n }\n```\n\n## Length Bias\n\n### The Problem\n\nLLMs tend to rate longer responses higher, regardless of quality. This manifests as:\n- Verbose responses receiving inflated scores\n- Concise but complete responses penalized\n- Padding and repetition being rewarded\n\n### Mitigation: Explicit Prompting\n\nInclude anti-length-bias instructions in the prompt:\n\n```\nCRITICAL EVALUATION GUIDELINES:\n- Do NOT prefer responses because they are longer\n- Concise, complete answers are as valuable as detailed ones\n- Penalize unnecessary verbosity or repetition\n- Focus on information density, not word count\n```\n\n### Mitigation: Length-Normalized Scoring\n\n```python\ndef length_normalized_score(score, response_length, target_length=500):\n \"\"\"Adjust score based on response length.\"\"\"\n length_ratio = response_length / target_length\n \n if length_ratio > 2.0:\n # Penalize excessively long responses\n penalty = (length_ratio - 2.0) * 0.1\n return max(score - penalty, 1)\n elif length_ratio < 0.3:\n # Penalize excessively short responses\n penalty = (0.3 - length_ratio) * 0.5\n return max(score - penalty, 1)\n else:\n return score\n```\n\n### Mitigation: Separate Length Criterion\n\nMake length a separate, explicit criterion so it's not implicitly rewarded:\n\n```python\ncriteria = [\n {\"name\": \"Accuracy\", \"description\": \"Factual correctness\", \"weight\": 0.4},\n {\"name\": \"Completeness\", \"description\": \"Covers key points\", \"weight\": 0.3},\n {\"name\": \"Conciseness\", \"description\": \"No unnecessary content\", \"weight\": 0.3} # Explicit\n]\n```\n\n## Self-Enhancement Bias\n\n### The Problem\n\nModels rate outputs generated by themselves (or similar models) higher than outputs from different models.\n\n### Mitigation: Cross-Model Evaluation\n\nUse a different model family for evaluation than generation:\n\n```python\ndef get_evaluator_model(generator_model):\n \"\"\"Select evaluator to avoid self-enhancement bias.\"\"\"\n if 'gpt' in generator_model.lower():\n return 'claude-4-5-sonnet'\n elif 'claude' in generator_model.lower():\n return 'gpt-5.2'\n else:\n return 'gpt-5.2' # Default\n```\n\n### Mitigation: Blind Evaluation\n\nRemove model attribution from responses before evaluation:\n\n```python\ndef anonymize_response(response, model_name):\n \"\"\"Remove model-identifying patterns.\"\"\"\n patterns = [\n f\"As {model_name}\",\n \"I am an AI\",\n \"I don't have personal opinions\",\n # Model-specific patterns\n ]\n anonymized = response\n for pattern in patterns:\n anonymized = anonymized.replace(pattern, \"[REDACTED]\")\n return anonymized\n```\n\n## Verbosity Bias\n\n### The Problem\n\nDetailed explanations receive higher scores even when the extra detail is irrelevant or incorrect.\n\n### Mitigation: Relevance-Weighted Scoring\n\n```python\nasync def relevance_weighted_evaluation(response, prompt, criteria):\n # First, assess relevance of each segment\n relevance_scores = await assess_relevance(response, prompt)\n \n # Weight evaluation by relevance\n segments = split_into_segments(response)\n weighted_scores = []\n for segment, relevance in zip(segments, relevance_scores):\n if relevance > 0.5: # Only count relevant segments\n score = await evaluate_segment(segment, prompt, criteria)\n weighted_scores.append(score * relevance)\n \n return sum(weighted_scores) / len(weighted_scores)\n```\n\n### Mitigation: Rubric with Verbosity Penalty\n\nInclude explicit verbosity penalties in rubrics:\n\n```python\nrubric_levels = [\n {\n \"score\": 5,\n \"description\": \"Complete and concise. All necessary information, nothing extraneous.\",\n \"characteristics\": [\"Every sentence adds value\", \"No repetition\", \"Appropriately scoped\"]\n },\n {\n \"score\": 3,\n \"description\": \"Complete but verbose. Contains unnecessary detail or repetition.\",\n \"characteristics\": [\"Main points covered\", \"Some tangents\", \"Could be more concise\"]\n },\n # ... etc\n]\n```\n\n## Authority Bias\n\n### The Problem\n\nConfident, authoritative tone is rated higher regardless of accuracy.\n\n### Mitigation: Evidence Requirement\n\nRequire explicit evidence for claims:\n\n```\nFor each claim in the response:\n1. Identify whether it's a factual claim\n2. Note if evidence or sources are provided\n3. Score based on verifiability, not confidence\n\nIMPORTANT: Confident claims without evidence should NOT receive higher scores than \nhedged claims with evidence.\n```\n\n### Mitigation: Fact-Checking Layer\n\nAdd a fact-checking step before scoring:\n\n```python\nasync def fact_checked_evaluation(response, prompt, criteria):\n # Extract claims\n claims = await extract_claims(response)\n \n # Fact-check each claim\n fact_check_results = await asyncio.gather(*[\n verify_claim(claim) for claim in claims\n ])\n \n # Adjust score based on fact-check results\n accuracy_factor = sum(r['verified'] for r in fact_check_results) / len(fact_check_results)\n \n base_score = await evaluate(response, prompt, criteria)\n return base_score * (0.7 + 0.3 * accuracy_factor) # At least 70% of score\n```\n\n## Aggregate Bias Detection\n\nMonitor for systematic biases in production:\n\n```python\nclass BiasMonitor:\n def __init__(self):\n self.evaluations = []\n \n def record(self, evaluation):\n self.evaluations.append(evaluation)\n \n def detect_position_bias(self):\n \"\"\"Detect if first position wins more often than expected.\"\"\"\n first_wins = sum(1 for e in self.evaluations if e['first_position_winner'])\n expected = len(self.evaluations) * 0.5\n z_score = (first_wins - expected) / (expected * 0.5) ** 0.5\n return {'bias_detected': abs(z_score) > 2, 'z_score': z_score}\n \n def detect_length_bias(self):\n \"\"\"Detect if longer responses score higher.\"\"\"\n from scipy.stats import spearmanr\n lengths = [e['response_length'] for e in self.evaluations]\n scores = [e['score'] for e in self.evaluations]\n corr, p_value = spearmanr(lengths, scores)\n return {'bias_detected': corr > 0.3 and p_value < 0.05, 'correlation': corr}\n```\n\n## Summary Table\n\n| Bias | Primary Mitigation | Secondary Mitigation | Detection Method |\n|------|-------------------|---------------------|------------------|\n| Position | Position swapping | Multiple shuffles | Consistency check |\n| Length | Explicit prompting | Length normalization | Length-score correlation |\n| Self-enhancement | Cross-model evaluation | Anonymization | Model comparison study |\n| Verbosity | Relevance weighting | Rubric penalties | Relevance scoring |\n| Authority | Evidence requirement | Fact-checking layer | Confidence-accuracy correlation |\n\n","skills/advanced-evaluation/references/evaluation-pipeline.md":"# Evaluation Pipeline Diagram\n\nVisual layout of a production evaluation pipeline.\n\n```\n┌─────────────────────────────────────────────────┐\n│ Evaluation Pipeline │\n├─────────────────────────────────────────────────┤\n│ │\n│ Input: Response + Prompt + Context │\n│ │ │\n│ ▼ │\n│ ┌─────────────────────┐ │\n│ │ Criteria Loader │ ◄── Rubrics, weights │\n│ └──────────┬──────────┘ │\n│ │ │\n│ ▼ │\n│ ┌─────────────────────┐ │\n│ │ Primary Scorer │ ◄── Direct or Pairwise │\n│ └──────────┬──────────┘ │\n│ │ │\n│ ▼ │\n│ ┌─────────────────────┐ │\n│ │ Bias Mitigation │ ◄── Position swap, etc. │\n│ └──────────┬──────────┘ │\n│ │ │\n│ ▼ │\n│ ┌─────────────────────┐ │\n│ │ Confidence Scoring │ ◄── Calibration │\n│ └──────────┬──────────┘ │\n│ │ │\n│ ▼ │\n│ Output: Scores + Justifications + Confidence │\n│ │\n└─────────────────────────────────────────────────┘\n```\n\n## Pipeline Stages\n\n1. **Criteria Loader**: Loads rubrics and criterion weights from configuration\n2. **Primary Scorer**: Applies direct scoring or pairwise comparison\n3. **Bias Mitigation**: Runs position swaps, length normalization, and other debiasing\n4. **Confidence Scoring**: Calibrates confidence based on position consistency and evidence strength\n","skills/advanced-evaluation/references/implementation-patterns.md":"# LLM-as-Judge Implementation Patterns\n\nThis reference provides detailed implementation patterns for building production-grade LLM evaluation systems.\n\n## Pattern 1: Structured Evaluation Pipeline\n\nThe most reliable evaluation systems follow a structured pipeline that separates concerns:\n\n```\nInput Validation → Criteria Loading → Scoring → Bias Mitigation → Output Formatting\n```\n\n### Input Validation Layer\n\nBefore evaluation begins, validate:\n\n1. **Response presence**: Non-empty response to evaluate\n2. **Prompt presence**: Original prompt for context\n3. **Criteria validity**: At least one criterion with name and description\n4. **Weight normalization**: Weights sum to 1.0 (or normalize them)\n\n```python\ndef validate_input(response, prompt, criteria):\n if not response or not response.strip():\n raise ValueError(\"Response cannot be empty\")\n if not prompt or not prompt.strip():\n raise ValueError(\"Prompt cannot be empty\")\n if not criteria or len(criteria) == 0:\n raise ValueError(\"At least one criterion required\")\n \n # Normalize weights\n total_weight = sum(c.get('weight', 1) for c in criteria)\n for c in criteria:\n c['weight'] = c.get('weight', 1) / total_weight\n```\n\n### Criteria Loading Layer\n\nCriteria should be loaded from configuration, not hardcoded:\n\n```python\nclass CriteriaLoader:\n def __init__(self, rubric_path=None):\n self.rubrics = self._load_rubrics(rubric_path)\n \n def get_criteria(self, task_type):\n return self.rubrics.get(task_type, self.default_criteria)\n \n def get_rubric(self, criterion_name):\n return self.rubrics.get(criterion_name, {}).get('levels', [])\n```\n\n### Scoring Layer\n\nThe scoring layer handles the actual LLM call:\n\n```python\nasync def score_response(response, prompt, criteria, rubric, model):\n system_prompt = build_system_prompt(criteria, rubric)\n user_prompt = build_user_prompt(response, prompt, criteria)\n \n result = await generate_text(\n model=model,\n system=system_prompt,\n prompt=user_prompt,\n temperature=0.3 # Lower temperature for consistency\n )\n \n return parse_scores(result.text)\n```\n\n### Bias Mitigation Layer\n\nFor pairwise comparison, always include position swapping:\n\n```python\nasync def compare_with_bias_mitigation(response_a, response_b, prompt, criteria, model):\n # First pass: A first\n pass1 = await compare_pair(response_a, response_b, prompt, criteria, model)\n \n # Second pass: B first\n pass2 = await compare_pair(response_b, response_a, prompt, criteria, model)\n \n # Map pass2 winner back\n pass2_mapped = map_winner(pass2.winner) # A→B, B→A, TIE→TIE\n \n # Check consistency\n if pass1.winner == pass2_mapped:\n return {\n 'winner': pass1.winner,\n 'confidence': (pass1.confidence + pass2.confidence) / 2,\n 'consistent': True\n }\n else:\n return {\n 'winner': 'TIE',\n 'confidence': 0.5,\n 'consistent': False\n }\n```\n\n## Pattern 2: Hierarchical Evaluation\n\nFor complex evaluations, use a hierarchical approach:\n\n```\nQuick Screen (cheap model) → Detailed Evaluation (expensive model) → Human Review (edge cases)\n```\n\n### Quick Screen Implementation\n\n```python\nasync def quick_screen(response, prompt, threshold=0.7):\n \"\"\"Fast, cheap screening for obvious passes/fails.\"\"\"\n result = await generate_text(\n model='gpt-5.2', # Cheaper model\n prompt=f\"Rate 0-1 if this response adequately addresses the prompt:\\n\\nPrompt: {prompt}\\n\\nResponse: {response}\",\n temperature=0\n )\n score = float(result.text.strip())\n return score, score > threshold\n```\n\n### Detailed Evaluation\n\n```python\nasync def detailed_evaluation(response, prompt, criteria):\n \"\"\"Full evaluation for borderline or important cases.\"\"\"\n result = await generate_text(\n model='gpt-5.2', # More capable model\n system=DETAILED_EVALUATION_PROMPT,\n prompt=build_detailed_prompt(response, prompt, criteria),\n temperature=0.3\n )\n return parse_detailed_scores(result.text)\n```\n\n## Pattern 3: Panel of LLM Judges (PoLL)\n\nFor high-stakes evaluation, use multiple models:\n\n```python\nasync def poll_evaluation(response, prompt, criteria, models):\n \"\"\"Aggregate judgments from multiple LLM judges.\"\"\"\n results = await asyncio.gather(*[\n score_with_model(response, prompt, criteria, model)\n for model in models\n ])\n \n # Aggregate scores\n aggregated = aggregate_scores(results)\n \n # Calculate agreement\n agreement = calculate_agreement(results)\n \n return {\n 'scores': aggregated,\n 'agreement': agreement,\n 'individual_results': results\n }\n\ndef aggregate_scores(results):\n \"\"\"Aggregate scores using median (robust to outliers).\"\"\"\n scores = {}\n for criterion in results[0]['scores'].keys():\n criterion_scores = [r['scores'][criterion] for r in results]\n scores[criterion] = {\n 'score': statistics.median(criterion_scores),\n 'std': statistics.stdev(criterion_scores) if len(criterion_scores) > 1 else 0\n }\n return scores\n```\n\n## Pattern 4: Confidence Calibration\n\nConfidence scores should be calibrated to actual reliability:\n\n```python\ndef calibrate_confidence(raw_confidence, position_consistent, evidence_count):\n \"\"\"Calibrate confidence based on multiple signals.\"\"\"\n \n # Base confidence from model output\n calibrated = raw_confidence\n \n # Position consistency is a strong signal\n if not position_consistent:\n calibrated *= 0.6 # Significant reduction\n \n # More evidence = higher confidence\n evidence_factor = min(evidence_count / 3, 1.0) # Cap at 3 pieces\n calibrated *= (0.7 + 0.3 * evidence_factor)\n \n return min(calibrated, 0.99) # Never 100% confident\n```\n\n## Pattern 5: Output Formatting\n\nAlways return structured outputs with consistent schemas:\n\n```python\n@dataclass\nclass ScoreResult:\n criterion: str\n score: float\n max_score: float\n justification: str\n evidence: List[str]\n improvement: str\n\n@dataclass\nclass EvaluationResult:\n success: bool\n scores: List[ScoreResult]\n overall_score: float\n weighted_score: float\n summary: Dict[str, Any]\n metadata: Dict[str, Any]\n\ndef format_output(scores, metadata) -> EvaluationResult:\n \"\"\"Format evaluation results consistently.\"\"\"\n return EvaluationResult(\n success=True,\n scores=scores,\n overall_score=sum(s.score for s in scores) / len(scores),\n weighted_score=calculate_weighted_score(scores),\n summary=generate_summary(scores),\n metadata=metadata\n )\n```\n\n## Error Handling Patterns\n\n### Graceful Degradation\n\n```python\nasync def evaluate_with_fallback(response, prompt, criteria):\n try:\n return await full_evaluation(response, prompt, criteria)\n except RateLimitError:\n # Fall back to simpler evaluation\n return await simple_evaluation(response, prompt, criteria)\n except ParseError as e:\n # Return partial results with error flag\n return {\n 'success': False,\n 'partial_results': e.partial_data,\n 'error': str(e)\n }\n```\n\n### Retry Logic\n\n```python\nasync def evaluate_with_retry(response, prompt, criteria, max_retries=3):\n for attempt in range(max_retries):\n try:\n result = await evaluate(response, prompt, criteria)\n if is_valid_result(result):\n return result\n except TransientError:\n await asyncio.sleep(2 ** attempt) # Exponential backoff\n \n raise EvaluationError(\"Max retries exceeded\")\n```\n\n## Testing Patterns\n\n### Unit Tests for Parsing\n\n```python\ndef test_score_parsing():\n raw_output = '{\"scores\": [{\"criterion\": \"Accuracy\", \"score\": 4}]}'\n result = parse_scores(raw_output)\n assert result.scores[0].criterion == \"Accuracy\"\n assert result.scores[0].score == 4\n\ndef test_malformed_output():\n raw_output = 'Invalid JSON'\n with pytest.raises(ParseError):\n parse_scores(raw_output)\n```\n\n### Integration Tests with Real API\n\n```python\n@pytest.mark.integration\nasync def test_full_evaluation_pipeline():\n result = await evaluate(\n response=\"Water boils at 100°C at sea level.\",\n prompt=\"At what temperature does water boil?\",\n criteria=[{\"name\": \"Accuracy\", \"description\": \"Factual correctness\", \"weight\": 1}]\n )\n \n assert result.success\n assert len(result.scores) == 1\n assert result.scores[0].score >= 4 # Should score high for accurate response\n```\n\n### Bias Detection Tests\n\n```python\nasync def test_position_bias_mitigation():\n # Same response in both positions should tie\n result = await compare(\n response_a=\"Same response\",\n response_b=\"Same response\",\n prompt=\"Test prompt\",\n criteria=[\"quality\"],\n swap_positions=True\n )\n \n assert result.winner == \"TIE\"\n assert result.consistent == True\n```\n\n","skills/advanced-evaluation/references/metrics-guide.md":"# Metric Selection Guide for LLM Evaluation\n\nThis reference provides guidance on selecting appropriate metrics for different evaluation scenarios.\n\n## Metric Categories\n\n### Classification Metrics\n\nUse for binary or multi-class evaluation tasks (pass/fail, correct/incorrect).\n\n#### Precision\n\n```\nPrecision = True Positives / (True Positives + False Positives)\n```\n\n**Interpretation**: Of all responses the judge said were good, what fraction were actually good?\n\n**Use when**: False positives are costly (e.g., approving unsafe content)\n\n```python\ndef precision(predictions, ground_truth):\n true_positives = sum(1 for p, g in zip(predictions, ground_truth) if p == 1 and g == 1)\n predicted_positives = sum(predictions)\n return true_positives / predicted_positives if predicted_positives > 0 else 0\n```\n\n#### Recall\n\n```\nRecall = True Positives / (True Positives + False Negatives)\n```\n\n**Interpretation**: Of all actually good responses, what fraction did the judge identify?\n\n**Use when**: False negatives are costly (e.g., missing good content in filtering)\n\n```python\ndef recall(predictions, ground_truth):\n true_positives = sum(1 for p, g in zip(predictions, ground_truth) if p == 1 and g == 1)\n actual_positives = sum(ground_truth)\n return true_positives / actual_positives if actual_positives > 0 else 0\n```\n\n#### F1 Score\n\n```\nF1 = 2 * (Precision * Recall) / (Precision + Recall)\n```\n\n**Interpretation**: Harmonic mean of precision and recall\n\n**Use when**: You need a single number balancing both concerns\n\n```python\ndef f1_score(predictions, ground_truth):\n p = precision(predictions, ground_truth)\n r = recall(predictions, ground_truth)\n return 2 * p * r / (p + r) if (p + r) > 0 else 0\n```\n\n### Agreement Metrics\n\nUse for comparing automated evaluation with human judgment.\n\n#### Cohen's Kappa (κ)\n\n```\nκ = (Observed Agreement - Expected Agreement) / (1 - Expected Agreement)\n```\n\n**Interpretation**: Agreement adjusted for chance\n- κ > 0.8: Almost perfect agreement\n- κ 0.6-0.8: Substantial agreement\n- κ 0.4-0.6: Moderate agreement\n- κ < 0.4: Fair to poor agreement\n\n**Use for**: Binary or categorical judgments\n\n```python\ndef cohens_kappa(judge1, judge2):\n from sklearn.metrics import cohen_kappa_score\n return cohen_kappa_score(judge1, judge2)\n```\n\n#### Weighted Kappa\n\nFor ordinal scales where disagreement severity matters:\n\n```python\ndef weighted_kappa(judge1, judge2):\n from sklearn.metrics import cohen_kappa_score\n return cohen_kappa_score(judge1, judge2, weights='quadratic')\n```\n\n**Interpretation**: Penalizes large disagreements more than small ones\n\n### Correlation Metrics\n\nUse for ordinal/continuous scores.\n\n#### Spearman's Rank Correlation (ρ)\n\n**Interpretation**: Correlation between rankings, not absolute values\n- ρ > 0.9: Very strong correlation\n- ρ 0.7-0.9: Strong correlation\n- ρ 0.5-0.7: Moderate correlation\n- ρ < 0.5: Weak correlation\n\n**Use when**: Order matters more than exact values\n\n```python\ndef spearmans_rho(scores1, scores2):\n from scipy.stats import spearmanr\n rho, p_value = spearmanr(scores1, scores2)\n return {'rho': rho, 'p_value': p_value}\n```\n\n#### Kendall's Tau (τ)\n\n**Interpretation**: Similar to Spearman but based on pairwise concordance\n\n**Use when**: You have many tied values\n\n```python\ndef kendalls_tau(scores1, scores2):\n from scipy.stats import kendalltau\n tau, p_value = kendalltau(scores1, scores2)\n return {'tau': tau, 'p_value': p_value}\n```\n\n#### Pearson Correlation (r)\n\n**Interpretation**: Linear correlation between scores\n\n**Use when**: Exact score values matter, not just order\n\n```python\ndef pearsons_r(scores1, scores2):\n from scipy.stats import pearsonr\n r, p_value = pearsonr(scores1, scores2)\n return {'r': r, 'p_value': p_value}\n```\n\n### Pairwise Comparison Metrics\n\n#### Agreement Rate\n\n```\nAgreement = (Matching Decisions) / (Total Comparisons)\n```\n\n**Interpretation**: Simple percentage of agreement\n\n```python\ndef pairwise_agreement(decisions1, decisions2):\n matches = sum(1 for d1, d2 in zip(decisions1, decisions2) if d1 == d2)\n return matches / len(decisions1)\n```\n\n#### Position Consistency\n\n```\nConsistency = (Consistent across position swaps) / (Total comparisons)\n```\n\n**Interpretation**: How often does swapping position change the decision?\n\n```python\ndef position_consistency(results):\n consistent = sum(1 for r in results if r['position_consistent'])\n return consistent / len(results)\n```\n\n## Selection Decision Tree\n\n```\nWhat type of evaluation task?\n│\n├── Binary classification (pass/fail)\n│ └── Use: Precision, Recall, F1, Cohen's κ\n│\n├── Ordinal scale (1-5 rating)\n│ ├── Comparing to human judgments?\n│ │ └── Use: Spearman's ρ, Weighted κ\n│ └── Comparing two automated judges?\n│ └── Use: Kendall's τ, Spearman's ρ\n│\n├── Pairwise preference\n│ └── Use: Agreement rate, Position consistency\n│\n└── Multi-label classification\n └── Use: Macro-F1, Micro-F1, Per-label metrics\n```\n\n## Metric Selection by Use Case\n\n### Use Case 1: Validating Automated Evaluation\n\n**Goal**: Ensure automated evaluation correlates with human judgment\n\n**Recommended Metrics**:\n1. Primary: Spearman's ρ (for ordinal scales) or Cohen's κ (for categorical)\n2. Secondary: Per-criterion agreement\n3. Diagnostic: Confusion matrix for systematic errors\n\n```python\ndef validate_automated_eval(automated_scores, human_scores, criteria):\n results = {}\n \n # Overall correlation\n results['overall_spearman'] = spearmans_rho(automated_scores, human_scores)\n \n # Per-criterion agreement\n for criterion in criteria:\n auto_crit = [s[criterion] for s in automated_scores]\n human_crit = [s[criterion] for s in human_scores]\n results[f'{criterion}_spearman'] = spearmans_rho(auto_crit, human_crit)\n \n return results\n```\n\n### Use Case 2: Comparing Two Models\n\n**Goal**: Determine which model produces better outputs\n\n**Recommended Metrics**:\n1. Primary: Win rate (from pairwise comparison)\n2. Secondary: Position consistency (bias check)\n3. Diagnostic: Per-criterion breakdown\n\n```python\ndef compare_models(model_a_outputs, model_b_outputs, prompts):\n results = []\n for a, b, p in zip(model_a_outputs, model_b_outputs, prompts):\n comparison = await compare_with_position_swap(a, b, p)\n results.append(comparison)\n \n return {\n 'a_wins': sum(1 for r in results if r['winner'] == 'A'),\n 'b_wins': sum(1 for r in results if r['winner'] == 'B'),\n 'ties': sum(1 for r in results if r['winner'] == 'TIE'),\n 'position_consistency': position_consistency(results)\n }\n```\n\n### Use Case 3: Quality Monitoring\n\n**Goal**: Track evaluation quality over time\n\n**Recommended Metrics**:\n1. Primary: Rolling agreement with human spot-checks\n2. Secondary: Score distribution stability\n3. Diagnostic: Bias indicators (position, length)\n\n```python\nclass QualityMonitor:\n def __init__(self, window_size=100):\n self.window = deque(maxlen=window_size)\n \n def add_evaluation(self, automated, human_spot_check=None):\n self.window.append({\n 'automated': automated,\n 'human': human_spot_check,\n 'length': len(automated['response'])\n })\n \n def get_metrics(self):\n # Filter to evaluations with human spot-checks\n with_human = [e for e in self.window if e['human'] is not None]\n \n if len(with_human) < 10:\n return {'insufficient_data': True}\n \n auto_scores = [e['automated']['score'] for e in with_human]\n human_scores = [e['human']['score'] for e in with_human]\n \n return {\n 'correlation': spearmans_rho(auto_scores, human_scores),\n 'mean_difference': np.mean([a - h for a, h in zip(auto_scores, human_scores)]),\n 'length_correlation': spearmans_rho(\n [e['length'] for e in self.window],\n [e['automated']['score'] for e in self.window]\n )\n }\n```\n\n## Interpreting Metric Results\n\n### Good Evaluation System Indicators\n\n| Metric | Good | Acceptable | Concerning |\n|--------|------|------------|------------|\n| Spearman's ρ | > 0.8 | 0.6-0.8 | < 0.6 |\n| Cohen's κ | > 0.7 | 0.5-0.7 | < 0.5 |\n| Position consistency | > 0.9 | 0.8-0.9 | < 0.8 |\n| Length correlation | < 0.2 | 0.2-0.4 | > 0.4 |\n\n### Warning Signs\n\n1. **High agreement but low correlation**: May indicate calibration issues\n2. **Low position consistency**: Position bias affecting results\n3. **High length correlation**: Length bias inflating scores\n4. **Per-criterion variance**: Some criteria may be poorly defined\n\n## Reporting Template\n\n```markdown\n## Evaluation System Metrics Report\n\n### Human Agreement\n- Spearman's ρ: 0.82 (p < 0.001)\n- Cohen's κ: 0.74\n- Sample size: 500 evaluations\n\n### Bias Indicators\n- Position consistency: 91%\n- Length-score correlation: 0.12\n\n### Per-Criterion Performance\n| Criterion | Spearman's ρ | κ |\n|-----------|--------------|---|\n| Accuracy | 0.88 | 0.79 |\n| Clarity | 0.76 | 0.68 |\n| Completeness | 0.81 | 0.72 |\n\n### Recommendations\n- All metrics within acceptable ranges\n- Monitor \"Clarity\" criterion - lower agreement may indicate need for rubric refinement\n```\n\n","skills/bdi-mental-states/SKILL.md":"---\nname: bdi-mental-states\ndescription: \"This skill should be used when modeling agent mental states with BDI concepts: beliefs, desires, intentions, RDF-to-belief transformations, rational agency traces, cognitive agents, BDI ontologies, and neuro-symbolic AI integration.\"\n---\n\n# BDI Mental State Modeling\n\nTransform external RDF context into agent mental states (beliefs, desires, intentions) using formal BDI ontology patterns. This skill enables agents to reason about context through cognitive architecture, supporting deliberative reasoning, explainability, and semantic interoperability within multi-agent systems.\n\n## When to Activate\n\nActivate this skill when:\n- Processing external RDF context into agent beliefs about world states\n- Modeling rational agency with perception, deliberation, and action cycles\n- Enabling explainability through traceable reasoning chains\n- Implementing BDI frameworks (SEMAS, JADE, JADEX)\n- Augmenting LLMs with formal cognitive structures (Logic Augmented Generation)\n- Coordinating mental states across multi-agent platforms\n- Tracking temporal evolution of beliefs, desires, and intentions\n- Linking motivational states to action plans\n\nDo not activate this skill for adjacent work owned by other skills:\n- General context-window explanations or attention mechanics: `context-fundamentals`.\n- Persistent user, entity, or conversation memory without formal BDI state: `memory-systems`.\n- Supervisor, swarm, or handoff topology decisions: `multi-agent-patterns`.\n- General agent evaluation rubrics or quality gates: `evaluation`.\n\n## Core Concepts\n\n### Mental Reality Architecture\n\nSeparate mental states into two ontological categories because BDI reasoning requires distinguishing what persists from what happens:\n\n**Mental States (Endurants)** -- model these as persistent cognitive attributes that hold over time intervals:\n- `Belief`: Represent what the agent holds true about the world. Ground every belief in a world state reference.\n- `Desire`: Represent what the agent wishes to bring about. Link each desire back to the beliefs that motivate it.\n- `Intention`: Represent what the agent commits to achieving. An intention must fulfil a desire and specify a plan.\n\n**Mental Processes (Perdurants)** -- model these as events that create or modify mental states, because tracking causal transitions enables explainability:\n- `BeliefProcess`: Triggers belief formation/update from perception. Always connect to a generating world state.\n- `DesireProcess`: Generates desires from existing beliefs. Preserves the motivational chain.\n- `IntentionProcess`: Commits to selected desires as actionable intentions.\n\n### Cognitive Chain Pattern\n\nWire beliefs, desires, and intentions into directed chains using bidirectional properties (`motivates`/`isMotivatedBy`, `fulfils`/`isFulfilledBy`) because this enables both forward reasoning (what should the agent do?) and backward tracing (why did the agent act?):\n\n```turtle\n:Belief_store_open a bdi:Belief ;\n rdfs:comment \"Store is open\" ;\n bdi:motivates :Desire_buy_groceries .\n\n:Desire_buy_groceries a bdi:Desire ;\n rdfs:comment \"I desire to buy groceries\" ;\n bdi:isMotivatedBy :Belief_store_open .\n\n:Intention_go_shopping a bdi:Intention ;\n rdfs:comment \"I will buy groceries\" ;\n bdi:fulfils :Desire_buy_groceries ;\n bdi:isSupportedBy :Belief_store_open ;\n bdi:specifies :Plan_shopping .\n```\n\n### World State Grounding\n\nAlways ground mental states in world state references rather than free-text descriptions, because ungrounded beliefs break semantic querying and cross-agent interoperability:\n\n```turtle\n:Agent_A a bdi:Agent ;\n bdi:perceives :WorldState_WS1 ;\n bdi:hasMentalState :Belief_B1 .\n\n:WorldState_WS1 a bdi:WorldState ;\n rdfs:comment \"Meeting scheduled at 10am in Room 5\" ;\n bdi:atTime :TimeInstant_10am .\n\n:Belief_B1 a bdi:Belief ;\n bdi:refersTo :WorldState_WS1 .\n```\n\n### Goal-Directed Planning\n\nConnect intentions to plans via `bdi:specifies`, and decompose plans into ordered task sequences using `bdi:precedes`, because this separation allows plan reuse across different intentions while keeping execution order explicit:\n\n```turtle\n:Intention_I1 bdi:specifies :Plan_P1 .\n\n:Plan_P1 a bdi:Plan ;\n bdi:addresses :Goal_G1 ;\n bdi:beginsWith :Task_T1 ;\n bdi:endsWith :Task_T3 .\n\n:Task_T1 bdi:precedes :Task_T2 .\n:Task_T2 bdi:precedes :Task_T3 .\n```\n\n### T2B2T Paradigm\n\nImplement Triples-to-Beliefs-to-Triples as a bidirectional pipeline because agents must both consume external RDF context and produce new RDF assertions. Structure every T2B2T implementation in two explicit phases:\n\n**Phase 1: Triples-to-Beliefs** -- Translate incoming RDF triples into belief instances. Use `bdi:triggers` to connect the external world state to a `BeliefProcess`, and `bdi:generates` to produce the resulting belief. This preserves provenance from source data through to internal cognition:\n```turtle\n:WorldState_notification a bdi:WorldState ;\n rdfs:comment \"Push notification: Payment request $250\" ;\n bdi:triggers :BeliefProcess_BP1 .\n\n:BeliefProcess_BP1 a bdi:BeliefProcess ;\n bdi:generates :Belief_payment_request .\n```\n\n**Phase 2: Beliefs-to-Triples** -- After BDI deliberation selects an intention and executes a plan, project the results back into RDF using `bdi:bringsAbout`. This closes the loop so downstream systems can consume agent outputs as standard linked data:\n```turtle\n:Intention_pay a bdi:Intention ;\n bdi:specifies :Plan_payment .\n\n:PlanExecution_PE1 a bdi:PlanExecution ;\n bdi:satisfies :Plan_payment ;\n bdi:bringsAbout :WorldState_payment_complete .\n```\n\n### Notation Selection by Level\n\nChoose notation based on the C4 abstraction level being modeled, because mixing notations at the wrong level obscures rather than clarifies the cognitive architecture:\n\n| C4 Level | Notation | Mental State Representation |\n|----------|----------|----------------------------|\n| L1 Context | ArchiMate | Agent boundaries, external perception sources |\n| L2 Container | ArchiMate | BDI reasoning engine, belief store, plan executor |\n| L3 Component | UML | Mental state managers, process handlers |\n| L4 Code | UML/RDF | Belief/Desire/Intention classes, ontology instances |\n\n### Justification and Explainability\n\nAttach `bdi:Justification` instances to every mental entity using `bdi:isJustifiedBy`, because unjustified mental states make agent reasoning opaque and untraceable. Each justification should capture the evidence or rule that produced the mental state:\n\n```turtle\n:Belief_B1 a bdi:Belief ;\n bdi:isJustifiedBy :Justification_J1 .\n\n:Justification_J1 a bdi:Justification ;\n rdfs:comment \"Official announcement received via email\" .\n\n:Intention_I1 a bdi:Intention ;\n bdi:isJustifiedBy :Justification_J2 .\n\n:Justification_J2 a bdi:Justification ;\n rdfs:comment \"Location precondition satisfied\" .\n```\n\n### Temporal Dimensions\n\nAssign validity intervals to every mental state using `bdi:hasValidity` with `TimeInterval` instances, because beliefs without temporal bounds cannot be garbage-collected or conflict-checked during diachronic reasoning:\n\n```turtle\n:Belief_B1 a bdi:Belief ;\n bdi:hasValidity :TimeInterval_TI1 .\n\n:TimeInterval_TI1 a bdi:TimeInterval ;\n bdi:hasStartTime :TimeInstant_9am ;\n bdi:hasEndTime :TimeInstant_11am .\n```\n\nQuery mental states active at a specific moment using SPARQL temporal filters. Use this pattern to resolve conflicts when multiple beliefs about the same world state overlap in time:\n\n```sparql\nSELECT ?mentalState WHERE {\n ?mentalState bdi:hasValidity ?interval .\n ?interval bdi:hasStartTime ?start ;\n bdi:hasEndTime ?end .\n FILTER(?start <= \"2025-01-04T10:00:00\"^^xsd:dateTime &&\n ?end >= \"2025-01-04T10:00:00\"^^xsd:dateTime)\n}\n```\n\n### Compositional Mental Entities\n\nDecompose complex beliefs into constituent parts using `bdi:hasPart` relations, because monolithic beliefs force full replacement on partial updates. Structure composite beliefs so that each sub-belief can be independently updated, queried, or invalidated:\n\n```turtle\n:Belief_meeting a bdi:Belief ;\n rdfs:comment \"Meeting at 10am in Room 5\" ;\n bdi:hasPart :Belief_meeting_time , :Belief_meeting_location .\n\n# Update only location component without touching time\n:BeliefProcess_update a bdi:BeliefProcess ;\n bdi:modifies :Belief_meeting_location .\n```\n\n## Practical Guidance\n\n### Build a BDI Model in Six Passes\n\nUse this workflow when converting external semantic context into a BDI representation:\n\n1. **Define the world-state substrate**: Identify the external facts or events the agent can perceive. Model these as world states before creating beliefs.\n2. **Create belief instances**: Translate each relevant world state into a belief with provenance, temporal validity, and a justification reference.\n3. **Derive desires from beliefs**: Add desires only when a belief creates a goal-relevant motivation. Link each desire to the belief that motivates it.\n4. **Commit intentions deliberately**: Promote a desire to an intention only when the agent commits to a plan. Record the selected plan and preconditions.\n5. **Project action results back to triples**: After execution, emit resulting world states as RDF so downstream systems can consume the new state.\n6. **Validate with competency questions**: Query for provenance, motivation, plan sequence, and active validity windows before trusting the model.\n\n### Keep the Ontology Small\n\nStart with `Agent`, `WorldState`, `Belief`, `Desire`, `Intention`, `Plan`, `Task`, `Justification`, and `TimeInterval`. Add specialized classes only after competency questions prove the core model cannot answer required queries. A compact ontology is easier to serialize into prompts, easier to validate, and less likely to create brittle reasoning chains.\n\n### Use BDI Only When Mental-State Semantics Matter\n\nBDI modeling is justified when the system needs explainable agency: why an agent believed something, what desire that belief created, which intention was selected, and what plan executed. If the system only needs to remember facts across sessions, use `memory-systems`. If it only needs to split work across agents, use `multi-agent-patterns`.\n\n## Detailed Topics\n\n### Integration Patterns\n\n### Logic Augmented Generation (LAG)\n\nUse LAG to constrain LLM outputs with ontological structure, because unconstrained generation produces triples that violate BDI class restrictions. Serialize the ontology into the prompt context, then validate generated triples against it before accepting them:\n\n```python\ndef augment_llm_with_bdi_ontology(prompt, ontology_graph):\n ontology_context = serialize_ontology(ontology_graph, format='turtle')\n augmented_prompt = f\"{ontology_context}\\n\\n{prompt}\"\n\n response = llm.generate(augmented_prompt)\n triples = extract_rdf_triples(response)\n\n is_consistent = validate_triples(triples, ontology_graph)\n return triples if is_consistent else retry_with_feedback()\n```\n\n### SEMAS Rule Translation\n\nTranslate BDI ontology patterns into executable production rules when deploying to rule-based agent platforms. Map each cognitive chain link (belief-to-desire, desire-to-intention) to a HEAD/CONDITIONALS/TAIL rule, because this preserves the deliberative semantics while enabling runtime execution:\n\n```prolog\n% Belief triggers desire formation\n[HEAD: belief(agent_a, store_open)] /\n[CONDITIONALS: time(weekday_afternoon)] »\n[TAIL: generate_desire(agent_a, buy_groceries)].\n\n% Desire triggers intention commitment\n[HEAD: desire(agent_a, buy_groceries)] /\n[CONDITIONALS: belief(agent_a, has_shopping_list)] »\n[TAIL: commit_intention(agent_a, buy_groceries)].\n```\n\n## Guidelines\n\n1. Model world states as configurations independent of agent perspectives, providing referential substrate for mental states.\n\n2. Distinguish endurants (persistent mental states) from perdurants (temporal mental processes), aligning with DOLCE ontology.\n\n3. Treat goals as descriptions rather than mental states, maintaining separation between cognitive and planning layers.\n\n4. Use `hasPart` relations for meronymic structures enabling selective belief updates.\n\n5. Associate every mental entity with temporal constructs via `atTime` or `hasValidity`.\n\n6. Use bidirectional property pairs (`motivates`/`isMotivatedBy`, `generates`/`isGeneratedBy`) for flexible querying.\n\n7. Link mental entities to `Justification` instances for explainability and trust.\n\n8. Implement T2B2T through: (1) translate RDF to beliefs, (2) execute BDI reasoning, (3) project mental states back to RDF.\n\n9. Define existential restrictions on mental processes (e.g., `BeliefProcess ⊑ ∃generates.Belief`).\n\n10. Reuse established ODPs (EventCore, Situation, TimeIndexedSituation, BasicPlan, Provenance) for interoperability.\n\n## Competency Questions\n\nValidate implementation against these SPARQL queries:\n\n```sparql\n# CQ1: What beliefs motivated formation of a given desire?\nSELECT ?belief WHERE {\n :Desire_D1 bdi:isMotivatedBy ?belief .\n}\n\n# CQ2: Which desire does a particular intention fulfill?\nSELECT ?desire WHERE {\n :Intention_I1 bdi:fulfils ?desire .\n}\n\n# CQ3: Which mental process generated a belief?\nSELECT ?process WHERE {\n ?process bdi:generates :Belief_B1 .\n}\n\n# CQ4: What is the ordered sequence of tasks in a plan?\nSELECT ?task ?nextTask WHERE {\n :Plan_P1 bdi:hasComponent ?task .\n OPTIONAL { ?task bdi:precedes ?nextTask }\n} ORDER BY ?task\n```\n\n## Examples\n\n**Example 1: RDF notification to BDI chain**\n\nInput world state:\n\n```turtle\n:WorldState_invoice_due a bdi:WorldState ;\n rdfs:comment \"Invoice INV-42 is due tomorrow\" ;\n bdi:atTime :Time_2026_05_15 .\n```\n\nBDI projection:\n\n```turtle\n:Belief_invoice_due a bdi:Belief ;\n bdi:refersTo :WorldState_invoice_due ;\n bdi:isJustifiedBy :Justification_billing_system ;\n bdi:motivates :Desire_avoid_late_fee .\n\n:Desire_avoid_late_fee a bdi:Desire ;\n bdi:isMotivatedBy :Belief_invoice_due .\n\n:Intention_pay_invoice a bdi:Intention ;\n bdi:fulfils :Desire_avoid_late_fee ;\n bdi:specifies :Plan_pay_invoice .\n```\n\n**Example 2: Boundary decision**\n\nIf the task is \"remember that Alice prefers concise summaries,\" use `memory-systems`. If the task is \"represent why the agent believes Alice needs a summary, what goal that creates, and which plan it commits to,\" use this skill.\n\n## Gotchas\n\n1. **Conflating mental states with world states**: Mental states reference world states via `bdi:refersTo`, they are not world states themselves. Mixing them collapses the perception-cognition boundary and breaks SPARQL queries that filter by type.\n\n2. **Missing temporal bounds**: Every mental state needs validity intervals for diachronic reasoning. Without them, stale beliefs persist indefinitely and conflict detection becomes impossible.\n\n3. **Flat belief structures**: Use compositional modeling with `hasPart` for complex beliefs. Monolithic beliefs force full replacement when only one attribute changes.\n\n4. **Implicit justifications**: Always link mental entities to explicit `Justification` instances. Unjustified mental states cannot be audited or traced.\n\n5. **Direct intention-to-action mapping**: Intentions specify plans which contain tasks; actions execute tasks. Skipping the plan layer removes the ability to reuse, reorder, or share execution strategies.\n\n6. **Ontology over-complexity**: Start with 5-10 core classes and properties (Belief, Desire, Intention, WorldState, Plan, plus key relations). Expanding the ontology prematurely inflates prompt context and slows SPARQL queries without improving reasoning quality.\n\n7. **Reasoning cost explosion**: Keep belief chains to 3 levels or fewer (belief -> desire -> intention). Deeper chains become prohibitively expensive for LLM inference and rarely improve decision quality over shallower alternatives.\n\n## Integration\n\nThis skill owns formal mental-state modeling. Adjacent skills own different layers:\n\n- `memory-systems`: persistent facts, entity memory, and temporal knowledge graphs without BDI belief/desire/intention semantics.\n- `multi-agent-patterns`: agent topology, handoff protocols, and coordination between agents.\n- `evaluation`: competency questions, regression checks, and quality gates for BDI implementations.\n- `context-fundamentals`: conceptual context-window and attention mechanics that inform prompt construction.\n- `tool-design`: schema and tool contracts for BDI query, validation, or projection tools.\n\n## References\n\nInternal references:\n- [BDI Ontology Core](./references/bdi-ontology-core.md) - Read when: implementing BDI class hierarchies or defining ontology properties from scratch\n- [RDF Examples](./references/rdf-examples.md) - Read when: writing Turtle serializations of mental states or debugging triple structure\n- [SPARQL Competency Queries](./references/sparql-competency.md) - Read when: validating an implementation against competency questions or building custom queries\n- [Framework Integration](./references/framework-integration.md) - Read when: deploying BDI models to SEMAS, JADE, or LAG pipelines\n\nPrimary sources:\n- Zuppiroli et al. \"The Belief-Desire-Intention Ontology\" (2025) — Read when: implementing formal BDI class hierarchies or validating ontology alignment\n- Rao & Georgeff \"BDI agents: From theory to practice\" (1995) — Read when: understanding the theoretical foundations of practical reasoning agents\n- Bratman \"Intention, plans, and practical reason\" (1987) — Read when: grounding implementation decisions in the philosophical basis of intentionality\n\n---\n\n## Skill Metadata\n\n**Created**: 2026-01-07\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.1.0\n","skills/bdi-mental-states/references/bdi-ontology-core.md":"# BDI Ontology Core Patterns\n\nCore ontology design patterns for Belief-Desire-Intention mental state modeling.\n\n## Class Hierarchy\n\n### Mental Entities (Endurants)\n\n```\nbdi:MentalEntity\n├── bdi:Belief # Informational dimension\n├── bdi:Desire # Motivational dimension \n├── bdi:Intention # Deliberative dimension\n├── bdi:Goal # Description of desired end state\n└── bdi:Plan # Structured action sequence\n```\n\n### Mental Processes (Perdurants)\n\n```\nbdi:MentalProcess\n├── bdi:BeliefProcess # Forms/updates beliefs from perception\n├── bdi:DesireProcess # Generates desires from beliefs\n├── bdi:IntentionProcess # Commits to desires as intentions\n├── bdi:Planning # Transforms intentions into plans\n└── bdi:PlanExecution # Executes plan actions\n```\n\n### Supporting Entities\n\n```\nbdi:WorldState # Configuration of environment\nbdi:Justification # Evidential basis for mental states\nbdi:Task # Atomic unit of planned action\nbdi:Action # Execution of a task\nbdi:TimeInterval # Temporal validity bounds\nbdi:TimeInstant # Point in time reference\n```\n\n## Object Properties\n\n### Motivational Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `motivates` | Belief | Desire | Belief provides reason for desire |\n| `isMotivatedBy` | Desire | Belief | Inverse of motivates |\n| `fulfils` | Intention | Desire | Intention commits to achieving desire |\n| `isFulfilledBy` | Desire | Intention | Inverse of fulfils |\n| `isSupportedBy` | Intention | Belief | Beliefs supporting intention viability |\n\n### Generative Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `generates` | MentalProcess | MentalEntity | Process creates mental state |\n| `isGeneratedBy` | MentalEntity | MentalProcess | Inverse of generates |\n| `modifies` | MentalProcess | MentalEntity | Process updates existing state |\n| `suppresses` | MentalProcess | MentalEntity | Process deactivates state |\n| `isTriggeredBy` | MentalProcess | MentalEntity | State initiates process |\n\n### Referential Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `refersTo` | MentalEntity | WorldState | Mental state about world |\n| `perceives` | Agent | WorldState | Agent observes world |\n| `bringsAbout` | Action | WorldState | Action causes world change |\n| `reasonsUpon` | MentalProcess | MentalEntity | Input to reasoning |\n\n### Structural Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `hasPart` | MentalEntity | MentalEntity | Meronymic composition |\n| `specifies` | Intention | Plan | Intention defines plan |\n| `addresses` | Plan | Goal | Plan achieves goal |\n| `hasComponent` | Plan | Task | Plan contains tasks |\n| `precedes` | Task | Task | Task ordering |\n\n### Temporal Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `atTime` | Entity | TimeInstant | Point occurrence |\n| `hasValidity` | MentalEntity | TimeInterval | Persistence bounds |\n| `hasStartTime` | TimeInterval | TimeInstant | Interval start |\n| `hasEndTime` | TimeInterval | TimeInstant | Interval end |\n\n### Justification Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `isJustifiedBy` | MentalEntity | Justification | Evidential support |\n| `justifies` | Justification | MentalEntity | Inverse relation |\n\n## Ontological Restrictions\n\n### Belief Restrictions\n\n```turtle\nbdi:Belief rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:refersTo ;\n owl:someValuesFrom bdi:WorldState\n] .\n\nbdi:Belief rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:hasValidity ;\n owl:maxCardinality 1\n] .\n```\n\n### Desire Restrictions\n\n```turtle\nbdi:Desire rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:isMotivatedBy ;\n owl:someValuesFrom bdi:Belief\n] .\n```\n\n### Intention Restrictions\n\n```turtle\nbdi:Intention rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:fulfils ;\n owl:cardinality 1\n] .\n\nbdi:Intention rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:isSupportedBy ;\n owl:someValuesFrom bdi:Belief\n] .\n```\n\n### Mental Process Restrictions\n\n```turtle\nbdi:BeliefProcess rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:generates ;\n owl:allValuesFrom bdi:Belief\n] .\n\nbdi:DesireProcess rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:generates ;\n owl:allValuesFrom bdi:Desire\n] .\n\nbdi:IntentionProcess rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:generates ;\n owl:allValuesFrom bdi:Intention\n] .\n```\n\n## DOLCE Alignment\n\nThe BDI ontology aligns with DOLCE Ultra Lite (DUL) foundational ontology:\n\n| BDI Class | DUL Superclass | Rationale |\n|-----------|----------------|-----------|\n| `Agent` | `dul:Agent` | Intentional entity capable of action |\n| `Belief` | `dul:InformationObject` | Information-bearing entity |\n| `Desire` | `dul:Description` | Describes desired state |\n| `Intention` | `dul:Description` | Describes committed course |\n| `Goal` | `dul:Goal` | Desired end state description |\n| `Plan` | `dul:Plan` | Organized action sequence |\n| `WorldState` | `dul:Situation` | Configuration of entities |\n| `MentalProcess` | `dul:Event` | Temporally extended occurrence |\n| `Task` | `dul:Task` | Unit of planned work |\n| `Action` | `dul:Action` | Performed task instance |\n\n## Reused Ontology Design Patterns\n\n### EventCore Pattern\nUsed for mental processes with temporal aspects and participant roles.\n\n### Situation Pattern \nUsed for world state configurations that mental states reference.\n\n### TimeIndexedSituation Pattern\nUsed for associating mental states with validity intervals.\n\n### BasicPlan Pattern\nUsed for goal-plan-task structures linking intentions to actions.\n\n### Provenance Pattern\nUsed for justification tracking and evidential chains.\n\n## Namespace Declarations\n\n```turtle\n@prefix bdi: .\n@prefix dul: .\n@prefix owl: .\n@prefix rdf: .\n@prefix rdfs: .\n@prefix xsd: .\n```\n\n","skills/bdi-mental-states/references/framework-integration.md":"# BDI Framework Integration Patterns\n\nIntegration patterns for connecting BDI ontology with executable agent frameworks.\n\n## SEMAS Rule Translation\n\nMap BDI ontology constructs to SEMAS production rules.\n\n### Ontology-to-Rule Mapping\n\n| BDI Construct | SEMAS Element | Example |\n|---------------|---------------|---------|\n| Belief | HEAD fact | `belief(agent_a, store_open)` |\n| Supporting beliefs | CONDITIONALS | `[CONDITIONALS: time(weekday)]` |\n| Desire generation | TAIL action | `generate_desire(agent, goal)` |\n| Intention commitment | TAIL action | `commit_intention(agent, goal)` |\n| Plan specification | TAIL action | `create_plan(agent, plan_id)` |\n\n### Rule Templates\n\n**Belief triggers desire formation:**\n```prolog\n[HEAD: belief(Agent, Fact)] / \n[CONDITIONALS: context_condition(Agent, Context)] » \n[TAIL: generate_desire(Agent, DesiredState)].\n```\n\n**Desire triggers intention commitment:**\n```prolog\n[HEAD: desire(Agent, Goal)] / \n[CONDITIONALS: belief(Agent, SupportingFact1), \n belief(Agent, SupportingFact2)] » \n[TAIL: commit_intention(Agent, Goal)].\n```\n\n**Intention triggers planning:**\n```prolog\n[HEAD: intention(Agent, Goal)] / \n[CONDITIONALS: goal(GoalSpec)] » \n[TAIL: create_plan(Agent, PlanId)].\n```\n\n**Plan triggers execution:**\n```prolog\n[HEAD: plan(Agent, PlanId)] / \n[CONDITIONALS: ready_to_execute(Agent)] » \n[TAIL: execute_plan(Agent, PlanId)].\n```\n\n### Complete SEMAS Example\n\n```prolog\n% ============================================================\n% GROCERY SHOPPING SCENARIO\n% ============================================================\n\n% Phase 1: Belief formation from world state\n[HEAD: perceive(agent_a, store_open)] / \n[CONDITIONALS: time(weekday_afternoon)] » \n[TAIL: add_belief(agent_a, store_open)].\n\n% Phase 2: Desire generation from belief\n[HEAD: belief(agent_a, store_open)] / \n[CONDITIONALS: belief(agent_a, needs_groceries)] » \n[TAIL: generate_desire(agent_a, buy_groceries)].\n\n% Phase 3: Intention commitment from desire\n[HEAD: desire(agent_a, buy_groceries)] / \n[CONDITIONALS: belief(agent_a, has_shopping_list), \n belief(agent_a, store_open),\n belief(agent_a, has_transportation)] » \n[TAIL: commit_intention(agent_a, buy_groceries)].\n\n% Phase 4: Plan creation from intention\n[HEAD: intention(agent_a, buy_groceries)] / \n[CONDITIONALS: goal(complete_shopping)] » \n[TAIL: create_plan(agent_a, shopping_plan)].\n\n% Phase 5: Plan execution\n[HEAD: plan(agent_a, shopping_plan)] / \n[CONDITIONALS: preconditions_met(shopping_plan)] » \n[TAIL: execute_task(agent_a, drive_to_store),\n execute_task(agent_a, select_items),\n execute_task(agent_a, checkout),\n execute_task(agent_a, return_home)].\n\n% Phase 6: World state update\n[HEAD: task_complete(agent_a, checkout)] / \n[CONDITIONALS: items_purchased(agent_a)] » \n[TAIL: update_world_state(has_groceries),\n remove_desire(agent_a, buy_groceries),\n remove_intention(agent_a, buy_groceries)].\n```\n\n### Python Translation Layer\n\n```python\nfrom rdflib import Graph, Namespace, RDF\n\nBDI = Namespace(\"https://w3id.org/fossr/ontology/bdi/\")\n\ndef ontology_to_semas_rules(bdi_graph: Graph) -> list[str]:\n \"\"\"\n Translate BDI ontology instances to SEMAS production rules.\n \"\"\"\n rules = []\n \n # Extract belief-desire-intention chains\n for intention in bdi_graph.subjects(RDF.type, BDI.Intention):\n # Get supporting beliefs\n supporting_beliefs = list(bdi_graph.objects(intention, BDI.isSupportedBy))\n \n # Get fulfilled desire\n fulfilled_desires = list(bdi_graph.objects(intention, BDI.fulfils))\n \n # Get specified plan\n specified_plans = list(bdi_graph.objects(intention, BDI.specifies))\n \n if fulfilled_desires and supporting_beliefs:\n desire = fulfilled_desires[0]\n beliefs_str = \", \".join([format_belief(b, bdi_graph) for b in supporting_beliefs])\n \n rule = (\n f\"[HEAD: {format_desire(desire, bdi_graph)}] / \"\n f\"[CONDITIONALS: {beliefs_str}] » \"\n f\"[TAIL: commit_intention({format_intention(intention, bdi_graph)})]\"\n )\n rules.append(rule)\n \n if specified_plans:\n plan = specified_plans[0]\n rule = (\n f\"[HEAD: {format_intention(intention, bdi_graph)}] / \"\n f\"[CONDITIONALS: ready_to_plan] » \"\n f\"[TAIL: create_plan({format_plan(plan, bdi_graph)})]\"\n )\n rules.append(rule)\n \n return rules\n\ndef format_belief(belief_uri, graph):\n label = graph.value(belief_uri, RDFS.label)\n return f\"belief({label or belief_uri.split('/')[-1]})\"\n\ndef format_desire(desire_uri, graph):\n label = graph.value(desire_uri, RDFS.label)\n return f\"desire({label or desire_uri.split('/')[-1]})\"\n\ndef format_intention(intention_uri, graph):\n label = graph.value(intention_uri, RDFS.label)\n return f\"intention({label or intention_uri.split('/')[-1]})\"\n\ndef format_plan(plan_uri, graph):\n label = graph.value(plan_uri, RDFS.label)\n return f\"plan({label or plan_uri.split('/')[-1]})\"\n```\n\n## Logic Augmented Generation (LAG)\n\nAugment LLM outputs with BDI ontological constraints.\n\n### LAG Pipeline Architecture\n\n```\n┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐\n│ User Query │────▶│ Ontology │────▶│ Augmented │\n│ │ │ Injection │ │ Prompt │\n└─────────────────┘ └─────────────────┘ └─────────────────┘\n │\n ▼\n┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐\n│ Validated │◀────│ Ontology │◀────│ LLM Response │\n│ RDF Triples │ │ Validation │ │ (Triples) │\n└─────────────────┘ └─────────────────┘ └─────────────────┘\n```\n\n### LAG Implementation\n\n```python\nfrom rdflib import Graph, Namespace\nfrom rdflib.plugins.parsers.notation3 import BadSyntax\n\nBDI = Namespace(\"https://w3id.org/fossr/ontology/bdi/\")\n\nclass BDILogicAugmentedGenerator:\n def __init__(self, ontology_path: str, llm_client):\n self.ontology = Graph()\n self.ontology.parse(ontology_path, format='turtle')\n self.llm = llm_client\n \n def generate_mental_states(self, context: str) -> Graph:\n \"\"\"\n Generate BDI mental states from context using LAG.\n \"\"\"\n # Phase 1: Inject ontology into prompt\n ontology_turtle = self.ontology.serialize(format='turtle')\n augmented_prompt = self._build_augmented_prompt(context, ontology_turtle)\n \n # Phase 2: Generate with LLM\n response = self.llm.generate(augmented_prompt)\n \n # Phase 3: Extract and validate triples\n triples = self._extract_triples(response)\n validated = self._validate_against_ontology(triples)\n \n if not validated['is_consistent']:\n # Retry with feedback\n return self._retry_with_feedback(context, validated['errors'])\n \n return validated['graph']\n \n def _build_augmented_prompt(self, context: str, ontology: str) -> str:\n return f\"\"\"\nYou are a BDI mental state modeler. Given the following context, generate \nRDF triples representing the agent's beliefs, desires, and intentions.\n\n## BDI Ontology (use these classes and properties):\n{ontology}\n\n## Context to Model:\n{context}\n\n## Instructions:\n1. Identify world states from the context\n2. Generate beliefs that refer to those world states\n3. Generate desires motivated by those beliefs\n4. Generate intentions that fulfill desires and are supported by beliefs\n5. Include justifications for each mental state\n6. Include temporal validity intervals\n\nOutput valid Turtle RDF triples only.\n\"\"\"\n \n def _extract_triples(self, response: str) -> str:\n \"\"\"Extract Turtle content from LLM response.\"\"\"\n # Find turtle block in response\n if \"```turtle\" in response:\n start = response.find(\"```turtle\") + 9\n end = response.find(\"```\", start)\n return response[start:end].strip()\n return response\n \n def _validate_against_ontology(self, triples: str) -> dict:\n \"\"\"Validate generated triples against BDI ontology.\"\"\"\n result = {'is_consistent': True, 'errors': [], 'graph': None}\n \n try:\n generated = Graph()\n generated.parse(data=triples, format='turtle')\n result['graph'] = generated\n \n # Validate constraints\n errors = []\n \n # Check: Every intention must fulfill a desire\n for intention in generated.subjects(RDF.type, BDI.Intention):\n if not list(generated.objects(intention, BDI.fulfils)):\n errors.append(f\"Intention {intention} does not fulfill any desire\")\n \n # Check: Every belief should reference a world state\n for belief in generated.subjects(RDF.type, BDI.Belief):\n if not list(generated.objects(belief, BDI.refersTo)):\n errors.append(f\"Belief {belief} does not reference a world state\")\n \n # Check: Desires should be motivated by beliefs\n for desire in generated.subjects(RDF.type, BDI.Desire):\n if not list(generated.objects(desire, BDI.isMotivatedBy)):\n errors.append(f\"Desire {desire} has no motivating belief\")\n \n if errors:\n result['is_consistent'] = False\n result['errors'] = errors\n \n except BadSyntax as e:\n result['is_consistent'] = False\n result['errors'] = [f\"Invalid Turtle syntax: {e}\"]\n \n return result\n \n def _retry_with_feedback(self, context: str, errors: list) -> Graph:\n \"\"\"Retry generation with error feedback.\"\"\"\n feedback_prompt = f\"\"\"\nPrevious generation had errors:\n{chr(10).join(errors)}\n\nPlease regenerate the mental states fixing these issues.\n\nContext: {context}\n\"\"\"\n response = self.llm.generate(feedback_prompt)\n triples = self._extract_triples(response)\n result = self._validate_against_ontology(triples)\n \n if result['is_consistent']:\n return result['graph']\n else:\n raise ValueError(f\"Failed to generate valid mental states: {result['errors']}\")\n```\n\n### Inconsistency Detection Example\n\n```python\ndef detect_location_inconsistency(graph: Graph) -> list[str]:\n \"\"\"\n Detect inconsistencies where agent cannot be in two places.\n \"\"\"\n inconsistencies = []\n \n # Query for location beliefs\n query = \"\"\"\n PREFIX bdi: \n \n SELECT ?agent ?belief1 ?belief2 ?loc1 ?loc2 WHERE {\n ?agent bdi:hasBelief ?belief1 , ?belief2 .\n ?belief1 bdi:refersTo ?ws1 .\n ?belief2 bdi:refersTo ?ws2 .\n ?ws1 bdi:hasLocation ?loc1 .\n ?ws2 bdi:hasLocation ?loc2 .\n FILTER(?belief1 != ?belief2 && ?loc1 != ?loc2)\n \n # Check temporal overlap\n ?belief1 bdi:hasValidity ?interval1 .\n ?belief2 bdi:hasValidity ?interval2 .\n ?interval1 bdi:hasStartTime ?start1 ; bdi:hasEndTime ?end1 .\n ?interval2 bdi:hasStartTime ?start2 ; bdi:hasEndTime ?end2 .\n FILTER(?start1 < ?end2 && ?start2 < ?end1)\n }\n \"\"\"\n \n for row in graph.query(query):\n inconsistencies.append(\n f\"Agent {row.agent} has conflicting location beliefs: \"\n f\"{row.loc1} and {row.loc2} at overlapping times\"\n )\n \n return inconsistencies\n```\n\n## JADE/JADEX Integration\n\nMap BDI ontology to JADE/JADEX agent platform structures.\n\n### JADE Agent Structure\n\n```java\npublic class BDIAgent extends Agent {\n // Mental state storage (maps to ontology individuals)\n private Set beliefs = new HashSet<>();\n private Set desires = new HashSet<>();\n private Set intentions = new HashSet<>();\n \n // Ontology-backed mental state management\n private Graph mentalStateGraph;\n \n public void addBelief(Belief belief) {\n beliefs.add(belief);\n \n // Add to RDF graph\n Resource beliefResource = mentalStateGraph.createResource(belief.getUri());\n beliefResource.addProperty(RDF.type, BDI.Belief);\n beliefResource.addProperty(BDI.refersTo, belief.getWorldState().getUri());\n beliefResource.addProperty(BDI.hasValidity, createInterval(belief.getValidity()));\n \n // Trigger desire formation\n triggerDesireProcess(belief);\n }\n \n public void commitIntention(Intention intention) {\n intentions.add(intention);\n \n Resource intentionResource = mentalStateGraph.createResource(intention.getUri());\n intentionResource.addProperty(RDF.type, BDI.Intention);\n intentionResource.addProperty(BDI.fulfils, intention.getDesire().getUri());\n \n for (Belief support : intention.getSupportingBeliefs()) {\n intentionResource.addProperty(BDI.isSupportedBy, support.getUri());\n }\n \n // Trigger planning\n triggerPlanning(intention);\n }\n \n // Export mental states as RDF\n public String exportMentalStates() {\n return mentalStateGraph.serialize(Format.TURTLE);\n }\n \n // Import mental states from RDF\n public void importMentalStates(String turtle) {\n Graph imported = new Graph();\n imported.parse(turtle, Format.TURTLE);\n \n // Reconstruct Java objects from RDF\n for (Resource belief : imported.listSubjectsWithProperty(RDF.type, BDI.Belief)) {\n Belief b = reconstructBelief(belief);\n beliefs.add(b);\n }\n // ... similar for desires and intentions\n }\n}\n```\n\n### JADEX Goal Mapping\n\n```java\n// Map BDI ontology goals to JADEX goals\n@Goal\npublic class OntologyBackedGoal {\n @GoalParameter\n protected String goalUri;\n \n @GoalParameter\n protected Graph ontologyGraph;\n \n public OntologyBackedGoal(Resource goalResource, Graph graph) {\n this.goalUri = goalResource.getURI();\n this.ontologyGraph = graph;\n }\n \n @GoalTargetCondition\n public boolean isAchieved() {\n // Query ontology for goal achievement\n String query = \"\"\"\n PREFIX bdi: \n ASK {\n ?execution bdi:addresses <%s> ;\n bdi:bringsAbout ?worldState .\n }\n \"\"\".formatted(goalUri);\n \n return ontologyGraph.ask(query);\n }\n \n @GoalDropCondition\n public boolean shouldDrop() {\n // Check if supporting beliefs are invalidated\n String query = \"\"\"\n PREFIX bdi: \n ASK {\n ?intention bdi:specifies ?plan .\n ?plan bdi:addresses <%s> .\n ?intention bdi:isSupportedBy ?belief .\n ?belief bdi:hasValidity ?interval .\n ?interval bdi:hasEndTime ?end .\n FILTER(?end < NOW())\n }\n \"\"\".formatted(goalUri);\n \n return ontologyGraph.ask(query);\n }\n}\n```\n\n## RDF Triple Store Integration\n\n### Triple Store Configuration\n\n```python\nfrom rdflib import Graph\nfrom rdflib.plugins.stores.sparqlstore import SPARQLUpdateStore\n\nclass BDIMentalStateStore:\n def __init__(self, endpoint: str):\n self.store = SPARQLUpdateStore()\n self.store.open((endpoint + \"/query\", endpoint + \"/update\"))\n self.graph = Graph(store=self.store, identifier=\"http://example.org/bdi\")\n \n def add_belief(self, agent_uri: str, belief_data: dict):\n \"\"\"Add belief to triple store.\"\"\"\n belief_uri = f\"{agent_uri}/belief/{belief_data['id']}\"\n \n self.graph.add((URIRef(belief_uri), RDF.type, BDI.Belief))\n self.graph.add((URIRef(belief_uri), RDFS.label, Literal(belief_data['label'])))\n self.graph.add((URIRef(belief_uri), BDI.refersTo, URIRef(belief_data['world_state'])))\n self.graph.add((URIRef(agent_uri), BDI.hasMentalState, URIRef(belief_uri)))\n \n # Add temporal validity\n interval_uri = f\"{belief_uri}/validity\"\n self.graph.add((URIRef(belief_uri), BDI.hasValidity, URIRef(interval_uri)))\n self.graph.add((URIRef(interval_uri), BDI.hasStartTime, \n Literal(belief_data['start_time'], datatype=XSD.dateTime)))\n self.graph.add((URIRef(interval_uri), BDI.hasEndTime,\n Literal(belief_data['end_time'], datatype=XSD.dateTime)))\n \n def get_active_beliefs(self, agent_uri: str, at_time: datetime) -> list:\n \"\"\"Query beliefs active at specific time.\"\"\"\n query = \"\"\"\n PREFIX bdi: \n PREFIX xsd: \n \n SELECT ?belief ?label WHERE {\n <%s> bdi:hasMentalState ?belief .\n ?belief a bdi:Belief ;\n rdfs:label ?label ;\n bdi:hasValidity ?interval .\n ?interval bdi:hasStartTime ?start ;\n bdi:hasEndTime ?end .\n FILTER(?start <= \"%s\"^^xsd:dateTime && ?end >= \"%s\"^^xsd:dateTime)\n }\n \"\"\" % (agent_uri, at_time.isoformat(), at_time.isoformat())\n \n return list(self.graph.query(query))\n \n def get_cognitive_chain(self, intention_uri: str) -> dict:\n \"\"\"Trace complete cognitive chain for an intention.\"\"\"\n query = \"\"\"\n PREFIX bdi: \n \n SELECT ?intention ?desire ?belief ?worldState ?plan WHERE {\n <%s> a bdi:Intention ;\n bdi:fulfils ?desire ;\n bdi:isSupportedBy ?belief .\n OPTIONAL { <%s> bdi:specifies ?plan }\n ?desire bdi:isMotivatedBy ?belief .\n ?belief bdi:refersTo ?worldState .\n }\n \"\"\" % (intention_uri, intention_uri)\n \n results = list(self.graph.query(query))\n if results:\n row = results[0]\n return {\n 'intention': str(row.intention),\n 'desire': str(row.desire),\n 'belief': str(row.belief),\n 'world_state': str(row.worldState),\n 'plan': str(row.plan) if row.plan else None\n }\n return None\n```\n\n## FIPA ACL Integration\n\nMap BDI mental states to FIPA Agent Communication Language.\n\n```python\nfrom fipa_acl import ACLMessage, Performative\n\nclass BDICommunicator:\n def __init__(self, agent_id: str, mental_state_store: BDIMentalStateStore):\n self.agent_id = agent_id\n self.store = mental_state_store\n \n def share_belief(self, belief_uri: str, receiver: str) -> ACLMessage:\n \"\"\"Create INFORM message to share belief.\"\"\"\n belief_triples = self.store.get_belief_as_turtle(belief_uri)\n \n message = ACLMessage()\n message.performative = Performative.INFORM\n message.sender = self.agent_id\n message.receiver = receiver\n message.content = belief_triples\n message.ontology = \"https://w3id.org/fossr/ontology/bdi/\"\n message.language = \"turtle\"\n \n return message\n \n def request_belief_confirmation(self, belief_uri: str, receiver: str) -> ACLMessage:\n \"\"\"Create QUERY-IF message to confirm shared belief.\"\"\"\n message = ACLMessage()\n message.performative = Performative.QUERY_IF\n message.sender = self.agent_id\n message.receiver = receiver\n message.content = f\"ASK {{ <{belief_uri}> a bdi:Belief }}\"\n message.language = \"sparql\"\n \n return message\n \n def propose_intention(self, intention_uri: str, receiver: str) -> ACLMessage:\n \"\"\"Create PROPOSE message for coordinated intention.\"\"\"\n intention_triples = self.store.get_intention_as_turtle(intention_uri)\n \n message = ACLMessage()\n message.performative = Performative.PROPOSE\n message.sender = self.agent_id\n message.receiver = receiver\n message.content = intention_triples\n message.ontology = \"https://w3id.org/fossr/ontology/bdi/\"\n \n return message\n```\n\n","skills/bdi-mental-states/references/rdf-examples.md":"# BDI RDF Examples\n\nComplete RDF/Turtle examples for BDI mental state modeling.\n\n## Complete Cognitive Workflow\n\n```turtle\n@prefix bdi: .\n@prefix ex: .\n@prefix xsd: .\n@prefix rdfs: .\n\n# ============================================================\n# PHASE 1: World State Perception\n# ============================================================\n\nex:WorldState_traffic a bdi:WorldState ;\n rdfs:comment \"Heavy traffic on Route 101\" ;\n bdi:atTime \"2026-01-04T08:30:00\"^^xsd:dateTime ;\n bdi:isPerceivedBy ex:Agent_commuter ;\n bdi:triggers ex:BeliefProcess_assess_traffic .\n\n# ============================================================\n# PHASE 2: Belief Formation\n# ============================================================\n\nex:BeliefProcess_assess_traffic a bdi:BeliefProcess ;\n bdi:generates ex:Belief_traffic_delay ;\n bdi:reasonsUpon ex:WorldState_traffic ;\n bdi:isProcessedBy ex:Agent_commuter ;\n bdi:atTime \"2026-01-04T08:31:00\"^^xsd:dateTime .\n\nex:Belief_traffic_delay a bdi:Belief ;\n rdfs:label \"Traffic will cause 30-minute delay\" ;\n bdi:refersTo ex:WorldState_traffic ;\n bdi:hasValidity ex:TimeInterval_morning_commute ;\n bdi:hasPart ex:Belief_route_congested , ex:Belief_delay_duration ;\n bdi:isJustifiedBy ex:Justification_traffic_report ;\n bdi:motivates ex:Desire_arrive_on_time .\n\nex:Belief_route_congested a bdi:Belief ;\n rdfs:comment \"Route 101 is congested\" .\n\nex:Belief_delay_duration a bdi:Belief ;\n rdfs:comment \"Delay estimated at 30 minutes\" .\n\nex:Justification_traffic_report a bdi:Justification ;\n rdfs:label \"Real-time traffic data from navigation system\" ;\n bdi:justifies ex:Belief_traffic_delay .\n\n# ============================================================\n# PHASE 3: Desire Formation\n# ============================================================\n\nex:DesireProcess_plan_arrival a bdi:DesireProcess ;\n bdi:generates ex:Desire_arrive_on_time ;\n bdi:reasonsUpon ex:Belief_traffic_delay ;\n bdi:isProcessedBy ex:Agent_commuter .\n\nex:Desire_arrive_on_time a bdi:Desire ;\n rdfs:label \"I desire to arrive at work on time\" ;\n bdi:isMotivatedBy ex:Belief_traffic_delay ;\n bdi:refersTo ex:WorldState_on_time_arrival .\n\n# ============================================================\n# PHASE 4: Intention Commitment\n# ============================================================\n\nex:IntentionProcess_commit_route a bdi:IntentionProcess ;\n bdi:generates ex:Intention_take_alternate_route ;\n bdi:reasonsUpon ex:Desire_arrive_on_time ;\n bdi:isProcessedBy ex:Agent_commuter .\n\nex:Intention_take_alternate_route a bdi:Intention ;\n rdfs:label \"I will take alternate route via Highway 280\" ;\n bdi:fulfils ex:Desire_arrive_on_time ;\n bdi:isSupportedBy ex:Belief_traffic_delay ;\n bdi:specifies ex:Plan_alternate_commute ;\n bdi:isJustifiedBy ex:Justification_time_optimization .\n\nex:Justification_time_optimization a bdi:Justification ;\n rdfs:label \"Alternate route saves 20 minutes based on current conditions\" ;\n bdi:justifies ex:Intention_take_alternate_route .\n\n# ============================================================\n# PHASE 5: Planning\n# ============================================================\n\nex:Planning_route_selection a bdi:Planning ;\n bdi:reasonsUpon ex:Intention_take_alternate_route ;\n bdi:defines ex:Plan_alternate_commute ;\n bdi:atTime ex:TimeInterval_planning_phase .\n\nex:Plan_alternate_commute a bdi:Plan ;\n rdfs:label \"Alternate commute via Highway 280\" ;\n bdi:addresses ex:Goal_arrive_by_9am ;\n bdi:beginsWith ex:Task_exit_Route101 ;\n bdi:endsWith ex:Task_arrive_parking ;\n bdi:hasComponent ex:Task_exit_Route101 , ex:Task_merge_280 , \n ex:Task_navigate_280 , ex:Task_arrive_parking .\n\nex:Task_exit_Route101 a bdi:Task ;\n rdfs:label \"Exit Route 101 at Whipple Ave\" ;\n bdi:precedes ex:Task_merge_280 .\n\nex:Task_merge_280 a bdi:Task ;\n rdfs:label \"Merge onto Highway 280 North\" ;\n bdi:precedes ex:Task_navigate_280 .\n\nex:Task_navigate_280 a bdi:Task ;\n rdfs:label \"Continue on Highway 280 for 8 miles\" ;\n bdi:precedes ex:Task_arrive_parking .\n\nex:Task_arrive_parking a bdi:Task ;\n rdfs:label \"Arrive at office parking garage\" .\n\nex:Goal_arrive_by_9am a bdi:Goal ;\n rdfs:label \"Arrive at work by 9:00 AM\" .\n\n# ============================================================\n# PHASE 6: Plan Execution\n# ============================================================\n\nex:PlanExecution_commute a bdi:PlanExecution ;\n bdi:satisfies ex:Plan_alternate_commute ;\n bdi:addresses ex:Goal_arrive_by_9am ;\n bdi:isExecutedBy ex:Agent_commuter ;\n bdi:hasComponent ex:Action_exit , ex:Action_merge , \n ex:Action_drive_280 , ex:Action_park ;\n bdi:atTime ex:TimeInterval_execution ;\n bdi:bringsAbout ex:WorldState_arrived_on_time .\n\nex:Action_exit a bdi:Action ;\n bdi:isExecutionOf ex:Task_exit_Route101 ;\n bdi:isPerformedBy ex:Agent_commuter ;\n bdi:atTime \"2026-01-04T08:35:00\"^^xsd:dateTime .\n\nex:Action_merge a bdi:Action ;\n bdi:isExecutionOf ex:Task_merge_280 ;\n bdi:isPerformedBy ex:Agent_commuter ;\n bdi:atTime \"2026-01-04T08:37:00\"^^xsd:dateTime .\n\nex:Action_drive_280 a bdi:Action ;\n bdi:isExecutionOf ex:Task_navigate_280 ;\n bdi:isPerformedBy ex:Agent_commuter ;\n bdi:atTime \"2026-01-04T08:40:00\"^^xsd:dateTime .\n\nex:Action_park a bdi:Action ;\n bdi:isExecutionOf ex:Task_arrive_parking ;\n bdi:isPerformedBy ex:Agent_commuter ;\n bdi:bringsAbout ex:WorldState_arrived_on_time ;\n bdi:atTime \"2026-01-04T08:52:00\"^^xsd:dateTime .\n\n# ============================================================\n# PHASE 7: Resulting World State\n# ============================================================\n\nex:WorldState_arrived_on_time a bdi:WorldState ;\n rdfs:comment \"Agent arrived at work at 8:52 AM\" ;\n bdi:atTime \"2026-01-04T08:52:00\"^^xsd:dateTime .\n\n# ============================================================\n# TEMPORAL INTERVALS\n# ============================================================\n\nex:TimeInterval_morning_commute a bdi:TimeInterval ;\n bdi:hasStartTime \"2026-01-04T08:30:00\"^^xsd:dateTime ;\n bdi:hasEndTime \"2026-01-04T09:00:00\"^^xsd:dateTime .\n\nex:TimeInterval_planning_phase a bdi:TimeInterval ;\n bdi:hasStartTime \"2026-01-04T08:31:00\"^^xsd:dateTime ;\n bdi:hasEndTime \"2026-01-04T08:34:00\"^^xsd:dateTime .\n\nex:TimeInterval_execution a bdi:TimeInterval ;\n bdi:hasStartTime \"2026-01-04T08:35:00\"^^xsd:dateTime ;\n bdi:hasEndTime \"2026-01-04T08:52:00\"^^xsd:dateTime .\n```\n\n## Multi-Agent Coordination Example\n\n```turtle\n@prefix bdi: .\n@prefix ex: .\n@prefix fipa: .\n\n# Shared belief about project deadline\nex:Agent_developer a bdi:Agent ;\n bdi:hasMentalState ex:Belief_deadline_friday .\n\nex:Agent_manager a bdi:Agent ;\n bdi:hasMentalState ex:Belief_deadline_friday .\n\nex:Belief_deadline_friday a bdi:Belief ;\n rdfs:label \"Project deadline is Friday 5 PM\" ;\n bdi:refersTo ex:WorldState_deadline ;\n bdi:hasValidity ex:TimeInterval_project_week .\n\nex:WorldState_deadline a bdi:WorldState ;\n rdfs:comment \"Project XYZ must be delivered by 2026-01-10T17:00:00\" .\n\n# Agent-specific mental states\nex:Agent_developer \n bdi:hasDesire ex:Desire_complete_coding ;\n bdi:hasIntention ex:Intention_implement_features .\n\nex:Desire_complete_coding a bdi:Desire ;\n rdfs:label \"Complete feature implementation\" ;\n bdi:isMotivatedBy ex:Belief_deadline_friday .\n\nex:Intention_implement_features a bdi:Intention ;\n rdfs:label \"Implement features A, B, and C\" ;\n bdi:fulfils ex:Desire_complete_coding ;\n bdi:specifies ex:Plan_development .\n\nex:Agent_manager \n bdi:hasDesire ex:Desire_ensure_delivery ;\n bdi:hasIntention ex:Intention_coordinate_team .\n\nex:Desire_ensure_delivery a bdi:Desire ;\n rdfs:label \"Ensure on-time project delivery\" ;\n bdi:isMotivatedBy ex:Belief_deadline_friday .\n\nex:Intention_coordinate_team a bdi:Intention ;\n rdfs:label \"Coordinate team activities\" ;\n bdi:fulfils ex:Desire_ensure_delivery ;\n bdi:specifies ex:Plan_project_management .\n\n# FIPA communication\nex:Message_M1 a fipa:ACLMessage ;\n fipa:sender ex:Agent_manager ;\n fipa:receiver ex:Agent_developer ;\n fipa:content ex:Belief_deadline_friday ;\n fipa:performative fipa:inform .\n```\n\n## Conflict Resolution Example\n\n```turtle\n@prefix bdi: .\n@prefix ex: .\n\n# Conflicting location beliefs\nex:Belief_at_home a bdi:Belief ;\n bdi:refersTo ex:WorldState_home ;\n rdfs:comment \"Agent is currently at home\" .\n\nex:Belief_at_office a bdi:Belief ;\n bdi:refersTo ex:WorldState_office ;\n rdfs:comment \"Agent is at office\" .\n\n# Conflicting intentions\nex:Intention_work_from_home a bdi:Intention ;\n bdi:isSupportedBy ex:Belief_at_home ;\n rdfs:label \"Work from home today\" .\n\nex:Intention_attend_meeting a bdi:Intention ;\n bdi:isSupportedBy ex:Belief_at_office ;\n rdfs:label \"Attend in-person meeting\" .\n\n# Justification for conflict resolution\nex:Justification_location_conflict a bdi:Justification ;\n rdfs:comment \"Cannot simultaneously be at home and office\" ;\n bdi:justifies ex:Intention_resolution .\n\n# Resolved intention\nex:Intention_resolution a bdi:Intention ;\n rdfs:label \"Attend meeting via video call from home\" ;\n bdi:fulfils ex:Desire_meeting_participation ;\n bdi:isSupportedBy ex:Belief_at_home ;\n bdi:isJustifiedBy ex:Justification_location_conflict .\n```\n\n## T2B2T Payment Processing Example\n\n```turtle\n@prefix bdi: .\n@prefix ex: .\n@prefix xsd: .\n\n# PHASE 1: Triples-to-Beliefs (External RDF → Internal Mental State)\n\nex:WorldState_notification a bdi:WorldState ;\n rdfs:comment \"Push notification: Ghadeh requested $250 via Zelle\" ;\n bdi:atTime \"2025-10-27T10:15:00\"^^xsd:dateTime ;\n bdi:triggers ex:BeliefProcess_BP1 .\n\nex:BeliefProcess_BP1 a bdi:BeliefProcess ;\n bdi:generates ex:Belief_payment_request ;\n bdi:isProcessedBy ex:Agent_A .\n\nex:Belief_payment_request a bdi:Belief ;\n rdfs:label \"Ghadeh requested $250\" ;\n bdi:refersTo ex:WorldState_notification ;\n bdi:motivates ex:Desire_pay_Ghadeh .\n\nex:Desire_pay_Ghadeh a bdi:Desire ;\n rdfs:label \"Pay Ghadeh $250\" ;\n bdi:isMotivatedBy ex:Belief_payment_request .\n\nex:Intention_I1 a bdi:Intention ;\n rdfs:label \"Pay Ghadeh $250\" ;\n bdi:fulfils ex:Desire_pay_Ghadeh ;\n bdi:specifies ex:Plan_payment .\n\n# PHASE 2: Beliefs-to-Triples (Mental State → External RDF)\n\nex:PlanExecution_PE1 a bdi:PlanExecution ;\n bdi:satisfies ex:Plan_payment ;\n bdi:bringsAbout ex:WorldState_payment_complete .\n\nex:WorldState_payment_complete a bdi:WorldState ;\n rdfs:comment \"Payment of $250 sent to Ghadeh via Zelle\" ;\n bdi:atTime \"2025-10-27T10:20:00\"^^xsd:dateTime .\n```\n\n","skills/bdi-mental-states/references/sparql-competency.md":"# SPARQL Competency Queries\n\nValidation queries for BDI ontology implementations based on competency questions.\n\n## Mental Entity Queries\n\n### CQ1: What are all mental entities?\n\n```sparql\nPREFIX bdi: \nPREFIX rdf: \nPREFIX rdfs: \n\nSELECT DISTINCT ?entity ?type WHERE {\n ?entity rdf:type ?type .\n ?type rdfs:subClassOf* bdi:MentalEntity .\n}\n```\n\n### CQ2: What beliefs does an agent hold?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?label WHERE {\n ?agent bdi:hasMentalState ?belief .\n ?belief a bdi:Belief .\n OPTIONAL { ?belief rdfs:label ?label }\n}\n```\n\n### CQ3: What desires does an agent have?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?desire ?label WHERE {\n ?agent bdi:hasDesire ?desire .\n ?desire a bdi:Desire .\n OPTIONAL { ?desire rdfs:label ?label }\n}\n```\n\n### CQ4: What intentions has an agent committed to?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention ?label WHERE {\n ?agent bdi:hasIntention ?intention .\n ?intention a bdi:Intention .\n OPTIONAL { ?intention rdfs:label ?label }\n}\n```\n\n## Motivational Chain Queries\n\n### CQ5: What beliefs motivated formation of a given desire?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?beliefLabel WHERE {\n ?desire bdi:isMotivatedBy ?belief .\n ?belief a bdi:Belief .\n OPTIONAL { ?belief rdfs:label ?beliefLabel }\n}\n```\n\n### CQ6: Which desire does a particular intention fulfill?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?desire ?desireLabel WHERE {\n ?intention bdi:fulfils ?desire .\n ?desire a bdi:Desire .\n OPTIONAL { ?desire rdfs:label ?desireLabel }\n}\n```\n\n### CQ7: What beliefs support a given intention?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?label WHERE {\n ?intention bdi:isSupportedBy ?belief .\n ?belief a bdi:Belief .\n OPTIONAL { ?belief rdfs:label ?label }\n}\n```\n\n### CQ8: Trace complete cognitive chain for an intention\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention ?desire ?belief ?worldState WHERE {\n ?intention a bdi:Intention ;\n bdi:fulfils ?desire ;\n bdi:isSupportedBy ?belief .\n ?desire bdi:isMotivatedBy ?belief .\n ?belief bdi:refersTo ?worldState .\n}\n```\n\n## Mental Process Queries\n\n### CQ9: Which mental process generated a belief?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?process ?processType WHERE {\n ?process bdi:generates ?belief .\n ?belief a bdi:Belief .\n ?process a ?processType .\n FILTER(?processType != owl:NamedIndividual)\n}\n```\n\n### CQ10: What triggered a mental process?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?process ?trigger ?triggerType WHERE {\n ?process a bdi:MentalProcess ;\n bdi:isTriggeredBy ?trigger .\n ?trigger a ?triggerType .\n}\n```\n\n### CQ11: What did a mental process reason upon?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?process ?input WHERE {\n ?process a bdi:MentalProcess ;\n bdi:reasonsUpon ?input .\n}\n```\n\n## Plan and Goal Queries\n\n### CQ12: What plan does an intention specify?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention ?plan ?goal WHERE {\n ?intention bdi:specifies ?plan .\n ?plan a bdi:Plan ;\n bdi:addresses ?goal .\n}\n```\n\n### CQ13: What is the ordered sequence of tasks in a plan?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?plan ?task ?nextTask WHERE {\n ?plan a bdi:Plan ;\n bdi:hasComponent ?task .\n OPTIONAL { ?task bdi:precedes ?nextTask }\n}\nORDER BY ?task\n```\n\n### CQ14: What is the first and last task of a plan?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?plan ?firstTask ?lastTask WHERE {\n ?plan a bdi:Plan ;\n bdi:beginsWith ?firstTask ;\n bdi:endsWith ?lastTask .\n}\n```\n\n### CQ15: Which actions executed which tasks?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?action ?task ?time WHERE {\n ?action bdi:isExecutionOf ?task ;\n bdi:atTime ?time .\n}\nORDER BY ?time\n```\n\n## Temporal Queries\n\n### CQ16: What mental states are valid at a specific time?\n\n```sparql\nPREFIX bdi: \nPREFIX xsd: \n\nSELECT ?mentalState ?type WHERE {\n ?mentalState bdi:hasValidity ?interval .\n ?interval bdi:hasStartTime ?start ;\n bdi:hasEndTime ?end .\n ?mentalState a ?type .\n FILTER(?start <= \"2026-01-04T10:00:00\"^^xsd:dateTime && \n ?end >= \"2026-01-04T10:00:00\"^^xsd:dateTime)\n}\n```\n\n### CQ17: When was a belief formed?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?formationTime WHERE {\n ?process bdi:generates ?belief ;\n bdi:atTime ?formationTime .\n ?belief a bdi:Belief .\n}\n```\n\n### CQ18: What is the temporal validity of an intention?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention ?start ?end WHERE {\n ?intention a bdi:Intention ;\n bdi:hasValidity ?interval .\n ?interval bdi:hasStartTime ?start ;\n bdi:hasEndTime ?end .\n}\n```\n\n## Justification Queries\n\n### CQ19: What justifies a belief?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?justification ?justLabel WHERE {\n ?belief a bdi:Belief ;\n bdi:isJustifiedBy ?justification .\n OPTIONAL { ?justification rdfs:label ?justLabel }\n}\n```\n\n### CQ20: What justifies an intention?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention ?justification ?justLabel WHERE {\n ?intention a bdi:Intention ;\n bdi:isJustifiedBy ?justification .\n OPTIONAL { ?justification rdfs:label ?justLabel }\n}\n```\n\n## Compositional Queries\n\n### CQ21: What parts comprise a complex belief?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?part ?partLabel WHERE {\n ?belief a bdi:Belief ;\n bdi:hasPart ?part .\n OPTIONAL { ?part rdfs:label ?partLabel }\n}\n```\n\n### CQ22: Find composite mental entities\n\n```sparql\nPREFIX bdi: \n\nSELECT ?composite (COUNT(?part) AS ?partCount) WHERE {\n ?composite bdi:hasPart ?part .\n}\nGROUP BY ?composite\nHAVING (COUNT(?part) > 1)\n```\n\n## World State Queries\n\n### CQ23: What world state does a belief refer to?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?worldState ?wsComment WHERE {\n ?belief a bdi:Belief ;\n bdi:refersTo ?worldState .\n OPTIONAL { ?worldState rdfs:comment ?wsComment }\n}\n```\n\n### CQ24: What actions brought about a world state?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?action ?worldState WHERE {\n ?action bdi:bringsAbout ?worldState .\n ?worldState a bdi:WorldState .\n}\n```\n\n### CQ25: What world states has an agent perceived?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?agent ?worldState ?time WHERE {\n ?agent bdi:perceives ?worldState .\n OPTIONAL { ?worldState bdi:atTime ?time }\n}\n```\n\n## Validation Queries (OWLUnit Style)\n\n### V1: Every intention must fulfill exactly one desire\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention WHERE {\n ?intention a bdi:Intention .\n FILTER NOT EXISTS { ?intention bdi:fulfils ?desire }\n}\n# Expected: Empty result set\n```\n\n### V2: Every belief must reference a world state\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief WHERE {\n ?belief a bdi:Belief .\n FILTER NOT EXISTS { ?belief bdi:refersTo ?worldState }\n}\n# Expected: Empty result set (or only abstract beliefs)\n```\n\n### V3: Mental processes must reason upon something\n\n```sparql\nPREFIX bdi: \n\nSELECT ?process WHERE {\n ?process a bdi:MentalProcess .\n FILTER NOT EXISTS { ?process bdi:reasonsUpon ?input }\n}\n# Expected: Empty result set\n```\n\n### V4: BeliefProcess must generate only Beliefs\n\n```sparql\nPREFIX bdi: \n\nSELECT ?process ?generated WHERE {\n ?process a bdi:BeliefProcess ;\n bdi:generates ?generated .\n FILTER NOT EXISTS { ?generated a bdi:Belief }\n}\n# Expected: Empty result set\n```\n\n### V5: Plans must have begin and end tasks\n\n```sparql\nPREFIX bdi: \n\nSELECT ?plan WHERE {\n ?plan a bdi:Plan .\n FILTER NOT EXISTS { \n ?plan bdi:beginsWith ?first ;\n bdi:endsWith ?last \n }\n}\n# Expected: Empty result set\n```\n\n## Multi-Agent Queries\n\n### CQ26: What beliefs are shared across agents?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief (COUNT(DISTINCT ?agent) AS ?agentCount) WHERE {\n ?agent bdi:hasMentalState ?belief .\n ?belief a bdi:Belief .\n}\nGROUP BY ?belief\nHAVING (COUNT(DISTINCT ?agent) > 1)\n```\n\n### CQ27: Which agents share the same desire?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?desire ?agent1 ?agent2 WHERE {\n ?agent1 bdi:hasDesire ?desire .\n ?agent2 bdi:hasDesire ?desire .\n FILTER(?agent1 != ?agent2)\n}\n```\n\n","skills/context-compression/SKILL.md":"---\nname: context-compression\ndescription: \"This skill should be used when long-running agent sessions need context compression, structured summarization, compaction, token-per-task optimization, or durable handoff summaries that preserve decisions, files, risks, and next actions.\"\n---\n\n# Context Compression Strategies\n\nWhen agent sessions generate millions of tokens of conversation history, compression becomes mandatory. The naive approach is aggressive compression to minimize tokens per request. The correct optimization target is tokens per task: total tokens consumed to complete a task, including re-fetching costs when compression loses critical information.\n\n## When to Activate\n\nActivate this skill when:\n- Agent sessions exceed context window limits\n- Codebases exceed context windows (5M+ token systems)\n- Designing conversation summarization strategies\n- Debugging cases where agents \"forget\" what files they modified\n- Building evaluation frameworks for compression quality\n- Creating durable handoff summaries that preserve decisions, files, risks, and next actions\n\nDo not activate this skill for adjacent work owned by other skills:\n- General token-efficiency tactics such as masking, prefix caching, or partitioning: `context-optimization`.\n- Diagnosing why a long context is failing before choosing a mitigation: `context-degradation`.\n- Writing raw outputs, logs, or plans to files without summarizing them: `filesystem-context`.\n- Designing long-term semantic memory across sessions: `memory-systems`.\n- The model deciding when and where to rewrite its own context: `self-managed-context`. This skill still owns what the replacement text must preserve.\n\n## Core Concepts\n\nContext compression trades token savings against information loss. Select from three production-ready approaches based on session characteristics:\n\n1. **Anchored Iterative Summarization**: Implement this for long-running sessions where file tracking matters. Maintain structured, persistent summaries with explicit sections for session intent, file modifications, decisions, and next steps. When compression triggers, summarize only the newly-truncated span and merge with the existing summary rather than regenerating from scratch. This prevents drift that accumulates when summaries are regenerated wholesale — each regeneration risks losing details the model considers low-priority but the task requires. Structure forces preservation because dedicated sections act as checklists the summarizer must populate, catching silent information loss.\n\n2. **Opaque Compression**: Reserve this for short sessions where re-fetching costs are low and maximum token savings are required. It produces compressed representations optimized for reconstruction fidelity, achieving 99%+ compression ratios but sacrificing interpretability entirely. The tradeoff matters: there is no way to verify what was preserved without running probe-based evaluation, so never use this when debugging or artifact tracking is critical.\n\n3. **Regenerative Full Summary**: Use this when summary readability is critical and sessions have clear phase boundaries. It generates detailed structured summaries on each compression trigger. The weakness is cumulative detail loss across repeated cycles — each full regeneration is a fresh pass that may deprioritize details preserved in earlier summaries.\n\n## Detailed Topics\n\n### Optimize for Tokens-Per-Task, Not Tokens-Per-Request\n\nMeasure total tokens consumed from task start to completion, not tokens per individual request. When compression drops file paths, error messages, or decision rationale, the agent must re-explore, re-read files, and re-derive conclusions — wasting far more tokens than the compression saved. A strategy saving 0.5% more tokens per request but causing 20% more re-fetching costs more overall. Track re-fetching frequency as the primary quality signal: if the agent repeatedly asks to re-read files it already processed, compression is too aggressive.\n\n### Solve the Artifact Trail Problem First\n\nArtifact trail integrity is often the weakest dimension in compression evaluations (claim-context-compression-factory-benchmark). Address this proactively because general summarization cannot reliably maintain it.\n\nPreserve these categories explicitly in every compression cycle:\n- Which files were created (full paths)\n- Which files were modified and what changed (include function names, not just file names)\n- Which files were read but not changed\n- Specific identifiers: function names, variable names, error messages, error codes\n\nImplement a separate artifact index or explicit file-state tracking in agent scaffolding rather than relying on the summarizer to capture these details. Even structured summarization with dedicated file sections struggles with completeness over long sessions.\n\n### Structure Summaries with Mandatory Sections\n\nBuild structured summaries with explicit sections that prevent silent information loss. Each section acts as a checklist the summarizer must populate, making omissions visible rather than silent.\n\n```markdown\n## Session Intent\n[What the user is trying to accomplish]\n\n## Files Modified\n- auth.controller.ts: Fixed JWT token generation\n- config/redis.ts: Updated connection pooling\n- tests/auth.test.ts: Added mock setup for new config\n\n## Decisions Made\n- Using Redis connection pool instead of per-request connections\n- Retry logic with exponential backoff for transient failures\n\n## Current State\n- 14 tests passing, 2 failing\n- Remaining: mock setup for session service tests\n\n## Next Steps\n1. Fix remaining test failures\n2. Run full test suite\n3. Update documentation\n```\n\nAdapt sections to the agent's domain. A debugging agent needs \"Root Cause\" and \"Error Messages\"; a migration agent needs \"Source Schema\" and \"Target Schema.\" The structure matters more than the specific sections — any explicit schema outperforms freeform summarization.\n\n### Choose Compression Triggers Strategically\n\nWhen to trigger compression matters as much as how to compress. Select a trigger strategy based on session predictability:\n\n| Strategy | Trigger Point | Trade-off |\n|----------|---------------|-----------|\n| Fixed threshold | 70-80% context utilization | Simple but may compress too early |\n| Sliding window | Keep last N turns + summary | Predictable context size |\n| Importance-based | Compress low-relevance sections first | Complex but preserves signal |\n| Task-boundary | Compress at logical task completions | Clean summaries but unpredictable timing |\n\nDefault to sliding window with structured summaries for coding agents — it provides the best balance of predictability and quality. Use task-boundary triggers when sessions have clear phase transitions (e.g., research then implementation then testing).\n\n### Evaluate Compression with Probes, Not Metrics\n\nTraditional metrics like ROUGE or embedding similarity fail to capture functional compression quality. A summary can score high on lexical overlap while missing the one file path the agent needs to continue.\n\nUse probe-based evaluation: after compression, pose questions that test whether critical information survived. If the agent answers correctly, compression preserved the right information. If not, it guesses or hallucinates.\n\n| Probe Type | What It Tests | Example Question |\n|------------|---------------|------------------|\n| Recall | Factual retention | \"What was the original error message?\" |\n| Artifact | File tracking | \"Which files have we modified?\" |\n| Continuation | Task planning | \"What should we do next?\" |\n| Decision | Reasoning chain | \"What did we decide about the Redis issue?\" |\n\n### Score Compression Across Six Dimensions\n\nEvaluate compression quality for coding agents across these dimensions. Accuracy and artifact-trail preservation tend to separate methods more clearly than lexical similarity (claim-context-compression-factory-benchmark), so compression needs specialized handling beyond general summarization.\n\n1. **Accuracy**: Are technical details correct — file paths, function names, error codes?\n2. **Context Awareness**: Does the response reflect current conversation state?\n3. **Artifact Trail**: Does the agent know which files were read or modified?\n4. **Completeness**: Does the response address all parts of the question?\n5. **Continuity**: Can work continue without re-fetching information?\n6. **Instruction Following**: Does the response respect stated constraints?\n\n## Practical Guidance\n\n### Apply the Three-Phase Compression Workflow for Large Codebases\n\nFor codebases or agent systems exceeding context windows, compress through three sequential phases. Each phase narrows context so the next phase operates within budget.\n\n1. **Research Phase**: Explore architecture diagrams, documentation, and key interfaces. Compress exploration into a structured analysis of components, dependencies, and boundaries. Output: a single research document that replaces raw exploration.\n\n2. **Planning Phase**: Convert the research document into an implementation specification with function signatures, type definitions, and data flow. A 5M-token codebase compresses to approximately 2,000 words of specification at this stage.\n\n3. **Implementation Phase**: Execute against the specification. Context stays focused on the spec plus active working files, not raw codebase exploration. This phase rarely needs further compression because the spec is already compact.\n\n### Use Example Artifacts as Compression Seeds\n\nWhen provided with a manual migration example or reference PR, use it as a template to understand the target pattern rather than exploring the codebase from scratch. The example reveals constraints static analysis cannot surface: which invariants must hold, which services break on changes, and what a clean implementation looks like.\n\nThis matters most when the agent cannot distinguish essential complexity (business requirements) from accidental complexity (legacy workarounds). The example artifact encodes that distinction implicitly, saving tokens that would otherwise go to trial-and-error exploration.\n\n### Implement Anchored Iterative Summarization Step by Step\n\n1. Define explicit summary sections matching the agent's domain (debugging, migration, feature development)\n2. On first compression trigger, summarize the truncated history into those sections\n3. On subsequent compressions, summarize only newly truncated content — do not re-summarize the existing summary\n4. Merge new information into existing sections rather than regenerating them, deduplicating by file path and decision identity\n5. Tag which information came from which compression cycle — this enables debugging when summaries drift\n\n### Select the Right Approach for the Session Profile\n\n**Use anchored iterative summarization when:**\n- Sessions are long-running (100+ messages)\n- File tracking matters (coding, debugging)\n- Verification of preserved information is needed\n\n**Use opaque compression when:**\n- Maximum token savings are required\n- Sessions are relatively short\n- Re-fetching costs are low (e.g., no file system access needed)\n\n**Use regenerative summaries when:**\n- Summary interpretability is critical for human review\n- Sessions have clear phase boundaries\n- Full context review is acceptable on each compression trigger\n\n### Calibrate Compression Ratios by Method\n\n| Method | Compression Ratio | Quality Score | Trade-off |\n|--------|-------------------|---------------|-----------|\n| Anchored Iterative | 98.6% | 3.70 | Best quality, slightly less compression |\n| Regenerative | 98.7% | 3.44 | Good quality, moderate compression |\n| Opaque | 99.3% | 3.35 | Best compression, quality loss |\n\nUse these as source-specific benchmark figures, not universal constants (claim-context-compression-factory-benchmark). For any task where re-fetching costs exist, this tradeoff generally favors structured approaches.\n\n## Examples\n\n**Example 1: Debugging Session Compression**\n\nOriginal context (89,000 tokens, 178 messages):\n- 401 error on /api/auth/login endpoint\n- Traced through auth controller, middleware, session store\n- Found stale Redis connection\n- Fixed connection pooling, added retry logic\n- 14 tests passing, 2 failing\n\nStructured summary after compression:\n\n```markdown\n## Session Intent\nDebug 401 Unauthorized error on /api/auth/login despite valid credentials.\n\n## Root Cause\nStale Redis connection in session store. JWT generated correctly but session could not be persisted.\n\n## Files Modified\n- auth.controller.ts: No changes (read only)\n- middleware/cors.ts: No changes (examined)\n- config/redis.ts: Fixed connection pooling configuration\n- services/session.service.ts: Added retry logic for transient failures\n- tests/auth.test.ts: Updated mock setup\n\n## Test Status\n14 passing, 2 failing (mock setup issues)\n\n## Next Steps\n1. Fix remaining test failures (mock session service)\n2. Run full test suite\n3. Deploy to staging\n```\n\n**Example 2: Probe Response Quality**\n\nAfter compression, asking \"What was the original error?\":\n\nGood response (structured summarization):\n> \"The original error was a 401 Unauthorized response from the /api/auth/login endpoint. Users received this error with valid credentials. Root cause was stale Redis connection in session store.\"\n\nPoor response (aggressive compression):\n> \"We were debugging an authentication issue. The login was failing. We fixed some configuration problems.\"\n\nThe structured response preserves endpoint, error code, and root cause. The aggressive response loses all technical detail.\n\n## Guidelines\n\n1. Optimize for tokens-per-task, not tokens-per-request\n2. Use structured summaries with explicit sections for file tracking\n3. Trigger compression at 70-80% context utilization\n4. Implement incremental merging rather than full regeneration\n5. Test compression quality with probe-based evaluation\n6. Track artifact trail separately if file tracking is critical\n7. Accept slightly lower compression ratios for better quality retention\n8. Monitor re-fetching frequency as a compression quality signal\n\n## Gotchas\n\n1. **Never compress tool definitions or schemas**: Compressing function call schemas, API specs, or tool definitions destroys agent functionality entirely. The agent cannot invoke tools whose parameter names or types have been summarized away. Treat tool definitions as immutable anchors that bypass compression.\n\n2. **Compressed summaries hallucinate facts**: When an LLM summarizes conversation history, it may introduce plausible-sounding details that never appeared in the original. Always validate compressed output against source material before discarding originals — especially for file paths, error codes, and numeric values that the summarizer may \"round\" or fabricate.\n\n3. **Compression breaks artifact references**: File paths, commit SHAs, variable names, and code snippets get paraphrased or dropped during compression. A summary saying \"updated the config file\" when the agent needs `config/redis.ts` causes re-exploration. Preserve identifiers verbatim in dedicated sections rather than embedding them in prose.\n\n4. **Early turns contain irreplaceable constraints**: The first few turns of a session often contain task setup, user constraints, and architectural decisions that cannot be re-derived. Protect early turns from compression or extract their constraints into a persistent preamble that survives all compression cycles.\n\n5. **Aggressive ratios compound across cycles**: A 95% compression ratio seems safe once, but applying it repeatedly compounds losses. After three cycles at 95%, only 0.0125% of original tokens remain. Calibrate ratios assuming multiple compression cycles, not a single pass.\n\n6. **Code and prose need different compression**: Prose compresses well because natural language is redundant. Code does not — removing a single token from a function signature or import path can make it useless. Apply domain-specific compression strategies: summarize prose sections aggressively while preserving code blocks and structured data verbatim.\n\n7. **Probe-based evaluation gives false confidence**: Probes can pass despite critical information being lost, because the probes test only what they ask about. A probe set that checks file names but not function signatures will miss signature loss. Design probes to cover all six evaluation dimensions, and rotate probe sets across evaluation runs to avoid blind spots.\n\n## Integration\n\nThis skill connects to several others in the collection:\n\n- context-degradation - Compression is a mitigation strategy for degradation\n- context-optimization - Compression is one optimization technique among many\n- evaluation - Probe-based evaluation applies to compression testing\n- memory-systems - Compression relates to scratchpad and summary memory patterns\n- self-managed-context - When the model triggers and places its own compaction edits, this skill supplies what the replacement note preserves\n\n## References\n\nInternal reference:\n- [Evaluation Framework Reference](./references/evaluation-framework.md) - Read when: building or calibrating a probe-based evaluation pipeline, or when needing scoring rubrics and LLM judge configuration for compression quality assessment\n\nRelated skills in this collection:\n- context-degradation - Read when: diagnosing why agent performance drops over long sessions, before applying compression as a mitigation\n- context-optimization - Read when: compression alone is insufficient and broader optimization strategies (pruning, caching, routing) are needed\n- evaluation - Read when: designing evaluation frameworks beyond compression-specific probes, including general LLM-as-judge methodology\n\nExternal resources:\n- Factory Research: Evaluating Context Compression for AI Agents (December 2025) - Read when: needing benchmark data on compression method comparisons or the 36,000-message evaluation dataset\n- Research on LLM-as-judge evaluation methodology (Zheng et al., 2023) - Read when: implementing or validating LLM judge scoring to understand bias patterns and calibration\n- Netflix Engineering: \"The Infinite Software Crisis\" - Three-phase workflow and context compression at scale (AI Summit 2025) - Read when: implementing the three-phase compression workflow for large codebases or understanding production-scale context management\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-22\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 1.3.0\n","skills/context-compression/references/evaluation-framework.md":"# Context Compression Evaluation Framework\n\nThis document provides the complete evaluation framework for measuring context compression quality, including probe types, scoring rubrics, and LLM judge configuration.\n\n## Probe Types\n\n### Recall Probes\n\nTest factual retention of specific details from conversation history.\n\n**Structure:**\n```\nQuestion: [Ask for specific fact from truncated history]\nExpected: [Exact detail that should be preserved]\nScoring: Match accuracy of technical details\n```\n\n**Examples:**\n- \"What was the original error message that started this debugging session?\"\n- \"What version of the dependency did we decide to use?\"\n- \"What was the exact command that failed?\"\n\n### Artifact Probes\n\nTest file tracking and modification awareness.\n\n**Structure:**\n```\nQuestion: [Ask about files created, modified, or examined]\nExpected: [Complete list with change descriptions]\nScoring: Completeness of file list and accuracy of change descriptions\n```\n\n**Examples:**\n- \"Which files have we modified? Describe what changed in each.\"\n- \"What new files did we create in this session?\"\n- \"Which configuration files did we examine but not change?\"\n\n### Continuation Probes\n\nTest ability to continue work without re-fetching context.\n\n**Structure:**\n```\nQuestion: [Ask about next steps or current state]\nExpected: [Actionable next steps based on session history]\nScoring: Ability to continue without requesting re-read of files\n```\n\n**Examples:**\n- \"What should we do next?\"\n- \"What tests are still failing and why?\"\n- \"What was left incomplete from our last step?\"\n\n### Decision Probes\n\nTest retention of reasoning chains and decision rationale.\n\n**Structure:**\n```\nQuestion: [Ask about why a decision was made]\nExpected: [Reasoning that led to the decision]\nScoring: Preservation of decision context and alternatives considered\n```\n\n**Examples:**\n- \"We discussed options for the Redis issue. What did we decide and why?\"\n- \"Why did we choose connection pooling over per-request connections?\"\n- \"What alternatives did we consider for the authentication fix?\"\n\n## Scoring Rubrics\n\n### Accuracy Dimension\n\n| Criterion | Question | Score 0 | Score 3 | Score 5 |\n|-----------|----------|---------|---------|---------|\n| accuracy_factual | Are facts, file paths, and technical details correct? | Completely incorrect or fabricated | Mostly accurate with minor errors | Perfectly accurate |\n| accuracy_technical | Are code references and technical concepts correct? | Major technical errors | Generally correct with minor issues | Technically precise |\n\n### Context Awareness Dimension\n\n| Criterion | Question | Score 0 | Score 3 | Score 5 |\n|-----------|----------|---------|---------|---------|\n| context_conversation_state | Does the response reflect current conversation state? | No awareness of prior context | General awareness with gaps | Full awareness of conversation history |\n| context_artifact_state | Does the response reflect which files/artifacts were accessed? | No awareness of artifacts | Partial artifact awareness | Complete artifact state awareness |\n\n### Artifact Trail Dimension\n\n| Criterion | Question | Score 0 | Score 3 | Score 5 |\n|-----------|----------|---------|---------|---------|\n| artifact_files_created | Does the agent know which files were created? | No knowledge | Knows most files | Perfect knowledge |\n| artifact_files_modified | Does the agent know which files were modified and what changed? | No knowledge | Good knowledge of most modifications | Perfect knowledge of all modifications |\n| artifact_key_details | Does the agent remember function names, variable names, error messages? | No recall | Recalls most key details | Perfect recall |\n\n### Completeness Dimension\n\n| Criterion | Question | Score 0 | Score 3 | Score 5 |\n|-----------|----------|---------|---------|---------|\n| completeness_coverage | Does the response address all parts of the question? | Ignores most parts | Addresses most parts | Addresses all parts thoroughly |\n| completeness_depth | Is sufficient detail provided? | Superficial or missing detail | Adequate detail | Comprehensive detail |\n\n### Continuity Dimension\n\n| Criterion | Question | Score 0 | Score 3 | Score 5 |\n|-----------|----------|---------|---------|---------|\n| continuity_work_state | Can the agent continue without re-fetching previously accessed information? | Cannot continue without re-fetching all context | Can continue with minimal re-fetching | Can continue seamlessly |\n| continuity_todo_state | Does the agent maintain awareness of pending tasks? | Lost track of all TODOs | Good awareness with some gaps | Perfect task awareness |\n| continuity_reasoning | Does the agent retain rationale behind previous decisions? | No memory of reasoning | Generally remembers reasoning | Excellent retention |\n\n### Instruction Following Dimension\n\n| Criterion | Question | Score 0 | Score 3 | Score 5 |\n|-----------|----------|---------|---------|---------|\n| instruction_format | Does the response follow the requested format? | Ignores format | Generally follows format | Perfectly follows format |\n| instruction_constraints | Does the response respect stated constraints? | Ignores constraints | Mostly respects constraints | Fully respects all constraints |\n\n## LLM Judge Configuration\n\n### System Prompt\n\n```\nYou are an expert evaluator assessing AI assistant responses in software development conversations.\n\nYour task is to grade responses against specific rubric criteria. For each criterion:\n1. Read the criterion question carefully\n2. Examine the response for evidence\n3. Assign a score from 0-5 based on the scoring guide\n4. Provide brief reasoning for your score\n\nBe objective and consistent. Focus on what is present in the response, not what could have been included.\n```\n\n### Judge Input Format\n\n```json\n{\n \"probe_question\": \"What was the original error message?\",\n \"model_response\": \"[Response to evaluate]\",\n \"compacted_context\": \"[The compressed context that was provided]\",\n \"ground_truth\": \"[Optional: known correct answer]\",\n \"rubric_criteria\": [\"accuracy_factual\", \"accuracy_technical\", \"context_conversation_state\"]\n}\n```\n\n### Judge Output Format\n\n```json\n{\n \"criterionResults\": [\n {\n \"criterionId\": \"accuracy_factual\",\n \"score\": 5,\n \"reasoning\": \"Response correctly identifies the 401 error, specific endpoint, and root cause.\"\n }\n ],\n \"aggregateScore\": 4.8,\n \"dimensionScores\": {\n \"accuracy\": 4.9,\n \"context_awareness\": 4.5,\n \"artifact_trail\": 3.2,\n \"completeness\": 5.0,\n \"continuity\": 4.8,\n \"instruction_following\": 5.0\n }\n}\n```\n\n## Benchmark Results Reference\n\nPerformance across compression methods (based on 36,000+ messages):\n\n| Method | Overall | Accuracy | Context | Artifact | Complete | Continuity | Instruction |\n|--------|---------|----------|---------|----------|----------|------------|-------------|\n| Anchored Iterative | 3.70 | 4.04 | 4.01 | 2.45 | 4.44 | 3.80 | 4.99 |\n| Regenerative | 3.44 | 3.74 | 3.56 | 2.33 | 4.37 | 3.67 | 4.95 |\n| Opaque | 3.35 | 3.43 | 3.64 | 2.19 | 4.37 | 3.77 | 4.92 |\n\n**Key Findings:**\n\n1. **Accuracy gap**: 0.61 points between best and worst methods\n2. **Context awareness gap**: 0.45 points, favoring anchored iterative\n3. **Artifact trail**: Universally weak (2.19-2.45), needs specialized handling\n4. **Completeness and instruction following**: Minimal differentiation\n\n## Statistical Considerations\n\n- Differences of 0.26-0.35 points are consistent across task types and session lengths\n- Pattern holds for both short and long sessions\n- Pattern holds across debugging, feature implementation, and code review tasks\n- Sample size: 36,611 messages across hundreds of compression points\n\n## Implementation Notes\n\n### Probe Generation\n\nGenerate probes at each compression point based on truncated history:\n1. Extract factual claims for recall probes\n2. Extract file operations for artifact probes\n3. Extract incomplete tasks for continuation probes\n4. Extract decision points for decision probes\n\n### Grading Process\n\n1. Feed probe question + model response + compressed context to judge\n2. Evaluate against each criterion in rubric\n3. Output structured JSON with scores and reasoning\n4. Compute dimension scores as weighted averages\n5. Compute overall score as unweighted average of dimensions\n\n### Blinding\n\nThe judge should not know which compression method produced the response being evaluated. This prevents bias toward known methods.\n\n","skills/context-degradation/SKILL.md":"---\nname: context-degradation\ndescription: \"This skill should be used for diagnosing and mitigating context degradation: lost-in-middle failures, context poisoning, context clash, context confusion, attention-pattern issues, and agent performance degradation caused by accumulated or conflicting context.\"\n---\n\n# Context Degradation Patterns\n\nDiagnose and fix context failures before they cascade. Context degradation is not binary — it is a continuum that manifests through five distinct, predictable patterns: lost-in-middle, poisoning, distraction, confusion, and clash. Each pattern has specific detection signals and mitigation strategies. Treat degradation as an engineering problem with measurable thresholds, not an unpredictable failure mode.\n\n## When to Activate\n\nActivate this skill when:\n- Agent performance degrades unexpectedly during long conversations\n- Debugging cases where agents produce incorrect or irrelevant outputs\n- Designing systems that must handle large contexts reliably\n- Evaluating context engineering choices for production systems\n- Investigating \"lost in middle\" phenomena in agent outputs\n- Analyzing context-related failures in agent behavior\n\nDo not activate this skill for adjacent work owned by other skills:\n- Explaining foundational context mechanics without an active failure: `context-fundamentals`.\n- Applying token-efficiency tactics after the failure pattern is known: `context-optimization`.\n- Designing a compression or handoff summary strategy: `context-compression`.\n- Persisting large outputs, logs, or scratch state outside the prompt: `filesystem-context`.\n\n## Core Concepts\n\nStructure context placement around the attention U-curve: beginning and end positions receive reliable attention, while middle positions suffer materially reduced recall accuracy in long-context experiments (claim-context-degradation-lost-middle-ruler). This is not a model bug but a consequence of attention mechanics — the first token (often BOS) acts as an \"attention sink\" that absorbs disproportionate attention budget, leaving middle tokens under-attended as context grows.\n\nTreat context poisoning as a circuit breaker problem. Once a hallucination, tool error, or incorrect retrieved fact enters context, it compounds through repeated self-reference. A poisoned goals section causes every downstream decision to reinforce incorrect assumptions. Detection requires tracking claim provenance; recovery requires truncating to before the poisoning point or restarting with verified-only context.\n\nFilter aggressively before loading context — even a single irrelevant document measurably degrades performance on relevant tasks. Models cannot \"skip\" irrelevant context; they must attend to everything provided, creating attention competition between relevant and irrelevant content. Move information that might be needed but is not immediately relevant behind tool calls instead of pre-loading it.\n\nIsolate task contexts to prevent confusion. When context contains multiple task types or switches between objectives, models incorporate constraints from the wrong task, call tools appropriate for a different context, or blend requirements from multiple sources. Explicit task segmentation with separate context windows eliminates cross-contamination.\n\nResolve context clash through priority rules, not accumulation. When multiple correct-but-contradictory sources appear in context (version conflicts, perspective conflicts, multi-source retrieval), models cannot determine which applies. Mark contradictions explicitly, establish source precedence, and filter outdated versions before they enter context.\n\n## Detailed Topics\n\n### Lost-in-Middle: Detection and Placement Strategy\n\nPlace critical information at the beginning and end of context, never in the middle. The U-shaped attention curve means middle-positioned information suffers 10-40% reduced recall accuracy. For contexts over 4K tokens, this effect becomes significant.\n\nUse summary structures that surface key findings at attention-favored positions. Add explicit section headers and structural markers — these help models navigate long contexts by creating attention anchors. When a document must be included in full, prepend a summary of its key points and append the critical conclusions.\n\nMonitor for lost-in-middle symptoms: correct information exists in context but the model ignores it, responses contradict provided data, or the model \"forgets\" instructions given earlier in a long prompt.\n\n### Context Poisoning: Prevention and Recovery\n\nValidate all external inputs before they enter context. Tool outputs, retrieved documents, and model-generated summaries are the three primary poisoning vectors. Each introduces unverified claims that subsequent reasoning treats as ground truth.\n\nDetect poisoning through these signals: degraded output quality on previously-successful tasks, tool misalignment (wrong tools or parameters), and hallucinations that persist despite explicit correction. When these cluster, suspect poisoning rather than model capability issues.\n\nRecover by removing poisoned content, not by adding corrections on top. Truncate to before the poisoning point, restart with clean context preserving only verified information, or explicitly mark the poisoned section and request re-evaluation from scratch. Layering corrections over poisoned context rarely works — the original errors retain attention weight.\n\n### Context Distraction: Curation Over Accumulation\n\nCurate what enters context rather than relying on models to ignore irrelevant content. Research shows even a single distractor document triggers measurable performance degradation — the effect follows a step function, not a linear curve. Multiple distractors compound the problem.\n\nApply relevance filtering before loading retrieved documents. Use namespacing and structural organization to make section boundaries clear. Prefer tool-call-based access over pre-loading: store reference material behind retrieval tools so it enters context only when directly relevant to the current reasoning step.\n\n### Context Confusion: Task Isolation\n\nSegment different tasks into separate context windows. Context confusion is distinct from distraction — it concerns the model applying wrong-context constraints to the current task, not just attention dilution. Signs include responses addressing the wrong aspect of a query, tool calls appropriate for a different task, and outputs mixing requirements from multiple sources.\n\nImplement clear transitions between task contexts. Use state management that isolates objectives, constraints, and tool definitions per task. When task-switching within a single session is unavoidable, use explicit \"context reset\" markers that signal which constraints apply to the current segment.\n\n### Context Clash: Conflict Resolution Protocols\n\nEstablish source priority rules before conflicts arise. Context clash differs from poisoning — multiple pieces of information are individually correct but mutually contradictory (version conflicts, perspective differences, multi-source retrieval with divergent facts).\n\nImplement version filtering to exclude outdated information before it enters context. When contradictions are unavoidable, mark them explicitly with structured conflict annotations: state what conflicts, which source each claim comes from, and which source takes precedence. Without explicit priority rules, models resolve contradictions unpredictably.\n\n### Empirical Benchmarks and Thresholds\n\nUse these benchmarks to set design constraints — not as universal truths. RULER-style evidence shows advertised long-context support does not guarantee satisfactory task performance at that length (claim-context-degradation-lost-middle-ruler). Near-perfect needle-in-haystack scores do not predict real-world long-context performance.\n\n**Model-Specific Degradation Thresholds**\n\nDegradation onset varies significantly by model family and task type. As a general rule, expect degradation to begin at 60-70% of the advertised context window for complex retrieval tasks (RULER benchmark found only 50% of models claiming 32K+ context maintain satisfactory performance at that length). Key patterns:\n\n- **Models with extended thinking** reduce hallucination through step-by-step verification but at higher latency and token cost\n- **Models optimized for agents/coding** tend to have better attention management for tool-output-heavy contexts\n- **Models with very large context windows (1M+)** handle more raw context but still follow U-shaped degradation curves — bigger windows do not eliminate the problem, they delay it\n\nAlways benchmark degradation thresholds with your specific workload rather than relying on published benchmarks. Model-specific thresholds go stale with each model update (see Gotcha 2).\n\n### Counterintuitive Findings\n\nAccount for these research-backed surprises when designing context strategies:\n\n**Shuffled context can outperform coherent context.** Studies found incoherent (shuffled) haystacks can outperform logically ordered ones for some retrieval tasks (claim-context-degradation-distractor-shuffled). Coherent context may create false associations that confuse retrieval; incoherent context can force exact matching. Do not assume that better-organized context always yields better results — test both arrangements.\n\n**Single distractors have outsized impact.** The performance hit from one irrelevant document is disproportionately large compared to adding more distractors after the first. Treat distractor prevention as binary: either keep context clean or accept significant degradation.\n\n**Low needle-question similarity accelerates degradation.** Tasks requiring inference across dissimilar content degrade faster with context length than tasks with high surface-level similarity. Design retrieval to maximize semantic overlap between queries and retrieved content.\n\n### When Larger Contexts Hurt\n\nDo not assume larger context windows improve performance. Performance remains stable up to a model-specific threshold, then degrades rapidly — the curve is non-linear with a cliff edge, not a gentle slope. For many models, meaningful degradation begins at 8K-16K tokens even when windows support much larger sizes.\n\nFactor in cost: processing a 400K token context costs exponentially more than 200K in both time and compute, not linearly more. For many applications, this makes large-context processing economically impractical.\n\nRecognize the cognitive bottleneck: even with infinite context, asking a single model to maintain quality across dozens of independent tasks creates degradation that more context cannot solve. Split tasks across sub-agents instead of expanding context.\n\n## Practical Guidance\n\n### The Four-Bucket Mitigation Framework\n\nApply these four strategies based on which degradation pattern is active:\n\n**Write** — Save context outside the window using scratchpads, file systems, or external storage. Use when context utilization exceeds 70% of the window. This keeps active context lean while preserving information access through tool calls.\n\n**Select** — Pull only relevant context into the window through retrieval, filtering, and prioritization. Use when distraction or confusion symptoms appear. Apply relevance scoring before loading; exclude anything below threshold rather than including everything available.\n\n**Compress** — Reduce tokens while preserving information through summarization, abstraction, and observation masking. Use when context is growing but all content is relevant. Replace verbose tool outputs with compact structured summaries; abstract repeated patterns into single references.\n\n**Isolate** — Split context across sub-agents or sessions to prevent any single context from growing past its degradation threshold. Use when confusion or clash symptoms appear, or when tasks are independent. This is the most aggressive strategy but often the most effective for complex multi-task systems.\n\n### Architectural Patterns for Resilience\n\nImplement just-in-time context loading: retrieve information only when the current reasoning step needs it, not preemptively. Use observation masking to replace verbose tool outputs with compact references after processing. Deploy sub-agent architectures where each agent holds only task-relevant context. Trigger compaction before context exceeds the model-specific degradation onset threshold — not after symptoms appear.\n\n## Examples\n\n**Example 1: Detecting Degradation**\n```yaml\n# Context grows during long conversation\nturn_1: 1000 tokens\nturn_5: 8000 tokens\nturn_10: 25000 tokens\nturn_20: 60000 tokens (degradation begins)\nturn_30: 90000 tokens (significant degradation)\n```\n\n**Example 2: Mitigating Lost-in-Middle**\n```markdown\n# Organize context with critical info at edges\n\n[CURRENT TASK] # At start\n- Goal: Generate quarterly report\n- Deadline: End of week\n\n[DETAILED CONTEXT] # Middle (less attention)\n- 50 pages of data\n- Multiple analysis sections\n- Supporting evidence\n\n[KEY FINDINGS] # At end\n- Revenue up 15%\n- Costs down 8%\n- Growth in Region A\n```\n\n**Example 3: Context poisoning circuit breaker**\n```text\nsymptom: agent keeps citing an incorrect retrieved claim after correction\ndiagnosis: poisoned context, not a missing-instruction problem\naction:\n 1. identify first turn where the bad claim entered\n 2. truncate or restart from before that point\n 3. reload only verified sources\n 4. record the rejected claim and source provenance\n```\n\n**Example 4: Context clash annotation**\n```yaml\nconflict:\n topic: billing API version\n source_a: docs/v1.md says endpoint is /charges\n source_b: docs/v2.md says endpoint is /payments\n precedence: docs/v2.md\n reason: current production version\n```\n\n## Guidelines\n\n1. Monitor context length and performance correlation during development\n2. Place critical information at beginning or end of context\n3. Implement compaction triggers before degradation becomes severe\n4. Validate retrieved documents for accuracy before adding to context\n5. Use versioning to prevent outdated information from causing clash\n6. Segment tasks to prevent context confusion across different objectives\n7. Design for graceful degradation rather than assuming perfect conditions\n8. Test with progressively larger contexts to find degradation thresholds\n\n## Gotchas\n\n1. **Normal variance looks like degradation**: Model output quality fluctuates naturally across runs. Do not diagnose degradation from a single drop in quality — establish a baseline over multiple runs and look for sustained, correlated decline tied to context growth. A 5-10% quality dip on one run is noise; the same dip consistently appearing after 40K tokens is signal.\n\n2. **Model-specific thresholds go stale**: The degradation onset values in benchmark tables reflect specific model versions. Provider updates, fine-tuning changes, and infrastructure shifts can move thresholds by 20-50% in either direction. Re-benchmark quarterly and after any major model update rather than treating published thresholds as permanent.\n\n3. **Needle-in-haystack scores create false confidence**: A model scoring 99% on needle-in-haystack does not mean it handles 128K tokens well in production. Needle tests measure single-fact retrieval from passive context — real workloads require multi-fact reasoning, instruction following, and synthesis across the full window. Use task-specific benchmarks that mirror actual workload patterns.\n\n4. **Contradictory retrieved documents poison silently**: When a RAG pipeline retrieves two documents that disagree on a fact, the model may silently pick one without signaling the conflict. This looks like a correct response but is effectively random. Implement contradiction detection in the retrieval layer before documents enter context.\n\n5. **Prompt quality problems masquerade as degradation**: Poor prompt structure (ambiguous instructions, missing constraints, unclear task framing) produces symptoms identical to context degradation — inconsistent outputs, ignored instructions, wrong tool usage. Before diagnosing degradation, verify the same prompt works correctly at low context lengths. If it fails at 2K tokens, the problem is the prompt, not the context.\n\n6. **Degradation is non-linear with a cliff edge**: Performance does not degrade gradually — it holds steady until a model-specific threshold, then drops sharply. Systems designed for \"graceful degradation\" often miss this pattern because monitoring checks assume linear decline. Set compaction triggers well before the cliff (at 70% of known onset), not at the onset itself.\n\n7. **Over-organizing context can backfire**: Intuitively, well-structured and coherent context should outperform disorganized content. Research shows shuffled haystacks sometimes outperform coherent ones for retrieval tasks because coherent context creates false associations. Test whether heavy structural formatting actually helps for the specific task — do not assume it does.\n\n## Integration\n\nThis skill owns diagnosis and mitigation of active context failures. Adjacent skills own the implementation tactics once the failure is identified:\n\n- `context-fundamentals`: conceptual explanation of attention and context windows before a failure exists.\n- `context-optimization`: masking, caching, partitioning, and other token-efficiency tactics after diagnosis.\n- `context-compression`: structured summaries and handoffs when accumulated context must be compacted.\n- `filesystem-context`: offloading raw outputs, logs, and scratch state so poisoned or bulky context can be inspected without staying in the prompt.\n- `multi-agent-patterns`: isolating tasks into separate contexts to prevent confusion and clash.\n- `evaluation`: degradation tests and production monitoring.\n\n## References\n\nInternal reference:\n- [Degradation Patterns Reference](./references/patterns.md) - Read when: debugging a specific degradation pattern and needing implementation-level detection code (attention analysis, poisoning tracking, relevance scoring, recovery procedures)\n\nRelated skills in this collection:\n- context-fundamentals - Read when: lacking foundational understanding of context windows, token budgets, or placement mechanics\n- context-optimization - Read when: degradation is diagnosed and specific mitigation techniques (compaction, compression, masking) are needed\n- evaluation - Read when: setting up production monitoring to detect degradation before it impacts users\n\nExternal resources:\n- Liu et al., 2023 \"Lost in the Middle\" - Read when: needing primary research backing for U-shaped attention claims or designing position-aware context layouts\n- RULER benchmark documentation - Read when: evaluating model claims about long-context support or comparing models for context-heavy workloads\n- Production engineering guides from AI labs - Read when: implementing context management in production infrastructure\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-20\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.1.0\n","skills/context-degradation/references/patterns.md":"# Context Degradation Patterns: Technical Reference\n\nThis document provides technical details on diagnosing and measuring context degradation.\n\n## Attention Distribution Analysis\n\n### U-Shaped Curve Measurement\n\nMeasure attention distribution across context positions:\n\n```python\ndef measure_attention_distribution(model, context_tokens, query):\n \"\"\"\n Measure how attention varies across context positions.\n \n Returns distribution showing attention weight by position.\n \"\"\"\n attention_by_position = []\n \n for position in range(len(context_tokens)):\n # Measure model's attention to this position\n attention = get_attention_weights(model, context_tokens, query, position)\n attention_by_position.append({\n \"position\": position,\n \"attention\": attention,\n \"is_beginning\": position < len(context_tokens) * 0.1,\n \"is_end\": position > len(context_tokens) * 0.9,\n \"is_middle\": True # Will be overwritten\n })\n \n # Classify positions\n for item in attention_by_position:\n if item[\"is_beginning\"] or item[\"is_end\"]:\n item[\"region\"] = \"attention_favored\"\n else:\n item[\"region\"] = \"attention_degraded\"\n \n return attention_by_position\n```\n\n### Lost-in-Middle Detection\n\nDetect when critical information falls in degraded attention regions:\n\n```python\ndef detect_lost_in_middle(critical_positions, attention_distribution):\n \"\"\"\n Check if critical information is in attention-favored positions.\n \n Args:\n critical_positions: List of positions containing critical info\n attention_distribution: Output from measure_attention_distribution\n \n Returns:\n Dictionary with detection results and recommendations\n \"\"\"\n results = {\n \"at_risk\": [],\n \"safe\": [],\n \"recommendations\": []\n }\n \n for pos in critical_positions:\n region = attention_distribution[pos][\"region\"]\n if region == \"attention_degraded\":\n results[\"at_risk\"].append(pos)\n else:\n results[\"safe\"].append(pos)\n \n # Generate recommendations\n if results[\"at_risk\"]:\n results[\"recommendations\"].extend([\n \"Move critical information to attention-favored positions\",\n \"Use explicit markers to highlight critical information\",\n \"Consider splitting context to reduce middle section\"\n ])\n \n return results\n```\n\n## Context Poisoning Detection\n\n### Hallucination Tracking\n\nTrack potential hallucinations across conversation turns:\n\n```python\nclass HallucinationTracker:\n def __init__(self):\n self.claims = []\n self.verifications = []\n \n def add_claims(self, text):\n \"\"\"Extract claims from text for later verification.\"\"\"\n claims = extract_claims(text)\n self.claims.extend([{\"text\": c, \"verified\": None} for c in claims])\n \n def verify_claims(self, ground_truth):\n \"\"\"Verify claims against ground truth.\"\"\"\n for claim in self.claims:\n if claim[\"verified\"] is None:\n claim[\"verified\"] = check_claim(claim[\"text\"], ground_truth)\n \n def get_poisoning_indicators(self):\n \"\"\"\n Return indicators of potential context poisoning.\n \n High ratio of unverified claims suggests poisoning risk.\n \"\"\"\n unverified = sum(1 for c in self.claims if not c[\"verified\"])\n verified_false = sum(1 for c in self.claims if c[\"verified\"] == False)\n \n return {\n \"unverified_count\": unverified,\n \"false_count\": verified_false,\n \"poisoning_risk\": verified_false > 0 or unverified > len(self.claims) * 0.3\n }\n```\n\n### Error Propagation Analysis\n\nTrack how errors flow through context:\n\n```python\ndef analyze_error_propagation(context, error_points):\n \"\"\"\n Analyze how errors at specific points affect downstream context.\n\n Returns visualization of error spread and impact assessment.\n \"\"\"\n impact_map = {}\n\n for error_point in error_points:\n # Find all references to content after error point\n downstream_refs = find_references(context, after=error_point)\n\n for ref in downstream_refs:\n if ref not in impact_map:\n impact_map[ref] = []\n impact_map[ref].append({\n \"source\": error_point,\n \"type\": classify_error_type(context[error_point])\n })\n\n # Assess severity\n high_impact_areas = [k for k, v in impact_map.items() if len(v) > 3]\n\n return {\n \"impact_map\": impact_map,\n \"high_impact_areas\": high_impact_areas,\n \"requires_intervention\": len(high_impact_areas) > 0\n }\n```\n\n## Distraction Metrics\n\n### Relevance Scoring\n\nScore relevance of context elements to current task:\n\n```python\ndef score_context_relevance(context_elements, task_description):\n \"\"\"\n Score each context element for relevance to current task.\n \n Returns scores and identifies high-distraction elements.\n \"\"\"\n task_embedding = embed(task_description)\n \n scored_elements = []\n for i, element in enumerate(context_elements):\n element_embedding = embed(element)\n relevance = cosine_similarity(task_embedding, element_embedding)\n scored_elements.append({\n \"index\": i,\n \"content_preview\": element[:100],\n \"relevance_score\": relevance\n })\n \n # Sort by relevance\n scored_elements.sort(key=lambda x: x[\"relevance_score\"], reverse=True)\n \n # Identify potential distractors\n threshold = calculate_relevance_threshold(scored_elements)\n distractors = [e for e in scored_elements if e[\"relevance_score\"] < threshold]\n \n return {\n \"scored_elements\": scored_elements,\n \"distractors\": distractors,\n \"recommendation\": f\"Consider removing {len(distractors)} low-relevance elements\"\n }\n```\n\n## Degradation Monitoring System\n\n### Context Health Dashboard\n\nImplement continuous monitoring of context health:\n\n```python\nclass ContextHealthMonitor:\n def __init__(self, model, context_window_limit):\n self.model = model\n self.limit = context_window_limit\n self.metrics = []\n \n def assess_health(self, context, task):\n \"\"\"\n Assess overall context health for current task.\n \n Returns composite score and component metrics.\n \"\"\"\n metrics = {\n \"token_count\": len(context),\n \"utilization_ratio\": len(context) / self.limit,\n \"attention_distribution\": measure_attention_distribution(self.model, context, task),\n \"relevance_scores\": score_context_relevance(context, task),\n \"age_tokens\": count_recent_tokens(context)\n }\n \n # Calculate composite health score\n health_score = self._calculate_composite(metrics)\n \n result = {\n \"health_score\": health_score,\n \"metrics\": metrics,\n \"status\": self._interpret_score(health_score),\n \"recommendations\": self._generate_recommendations(metrics)\n }\n \n self.metrics.append(result)\n return result\n \n def _calculate_composite(self, metrics):\n \"\"\"Calculate composite health score from components.\"\"\"\n # Weighted combination of metrics\n utilization_penalty = min(metrics[\"utilization_ratio\"] * 0.5, 0.3)\n attention_penalty = self._calculate_attention_penalty(metrics[\"attention_distribution\"])\n relevance_penalty = self._calculate_relevance_penalty(metrics[\"relevance_scores\"])\n \n base_score = 1.0\n score = base_score - utilization_penalty - attention_penalty - relevance_penalty\n return max(0, score)\n \n def _interpret_score(self, score):\n \"\"\"Interpret health score and return status.\"\"\"\n if score > 0.8:\n return \"healthy\"\n elif score > 0.6:\n return \"warning\"\n elif score > 0.4:\n return \"degraded\"\n else:\n return \"critical\"\n```\n\n### Alert Thresholds\n\nConfigure appropriate alert thresholds:\n\n```python\nCONTEXT_ALERTS = {\n \"utilization_warning\": 0.7, # 70% of context limit\n \"utilization_critical\": 0.9, # 90% of context limit\n \"attention_degraded_ratio\": 0.3, # 30% in middle region\n \"relevance_threshold\": 0.3, # Below 30% relevance\n \"consecutive_warnings\": 3 # Three warnings triggers alert\n}\n```\n\n## Recovery Procedures\n\n### Context Truncation Strategy\n\nWhen context degrades beyond recovery, truncate strategically:\n\n```python\ndef truncate_context_for_recovery(context, preserved_elements, target_size):\n \"\"\"\n Truncate context while preserving critical elements.\n \n Strategy:\n 1. Preserve system prompt and tool definitions\n 2. Preserve recent conversation turns\n 3. Preserve critical retrieved documents\n 4. Summarize older content if needed\n 5. Truncate from middle if still over target\n \"\"\"\n truncated = []\n \n # Category 1: Critical system elements (preserve always)\n system_elements = extract_system_elements(context)\n truncated.extend(system_elements)\n \n # Category 2: Recent conversation (preserve more)\n recent_turns = extract_recent_turns(context, num_turns=10)\n truncated.extend(recent_turns)\n \n # Category 3: Critical documents (preserve key ones)\n critical_docs = extract_critical_documents(context, preserved_elements)\n truncated.extend(critical_docs)\n \n # Check size and summarize if needed\n while len(truncated) > target_size:\n # Summarize oldest category 3 elements\n truncated = summarize_oldest(truncated, category=\"documents\")\n \n # If still too large, truncate oldest turns\n if len(truncated) > target_size:\n truncated = truncate_oldest_turns(truncated, keep_recent=5)\n \n return truncated\n```\n\n","skills/context-fundamentals/SKILL.md":"---\nname: context-fundamentals\ndescription: \"This skill should be used to explain or reason about the foundational concepts of context engineering: what context is, the anatomy of a context window, how attention mechanics work, the U-shaped attention curve, why context quality matters more than quantity, and the mental models needed to interpret every other context-engineering decision. Use this for conceptual explanation, onboarding, and background reading. Route operational work to the specialized skills: debugging attention failures goes to context-degradation, token-efficiency work goes to context-optimization, conversation summarization goes to context-compression, and project-shape decisions go to project-development.\"\n---\n\n# Context Engineering Fundamentals\n\nContext is the complete state available to a language model at inference time: system instructions, tool definitions, retrieved documents, message history, and tool outputs. Context engineering is the discipline of curating the smallest high-signal token set that maximizes the likelihood of desired outcomes.\n\nThis skill is the conceptual foundation that every other skill in the collection builds on. It explains what context is, how attention mechanics work, why context quality matters more than quantity, and the mental models needed to interpret every other context-engineering decision. It does not own operational work: debugging attention failures belongs to `context-degradation`, token-efficiency tactics belong to `context-optimization`, conversation summarization belongs to `context-compression`, file-based offloading belongs to `filesystem-context`, and project-shape decisions belong to `project-development`.\n\n## When to Activate\n\nActivate this skill when the work is conceptual:\n\n- Explaining what context is and how attention mechanics constrain agent behavior.\n- Onboarding new contributors who need the mental models before diving into operational skills.\n- Reasoning about a context-related design decision from first principles (what does this constraint mean, why does this trade-off exist) before picking a specific tactic.\n- Writing or reviewing documentation that needs to ground operational guidance in the underlying mechanics.\n\nDo not activate this skill for operational work. The specialized skills handle the doing:\n\n- Diagnosing lost-in-middle, context poisoning, or attention failures: `context-degradation`.\n- Reducing token cost via masking, partitioning, prefix caching, budgets: `context-optimization`.\n- Compressing a long session into a handoff summary: `context-compression`.\n- Offloading large tool outputs or maintaining a durable scratchpad: `filesystem-context`.\n- Deciding the shape of an LLM project or pipeline: `project-development`.\n\n## Core Concepts\n\nTreat context as a finite attention budget, not a storage bin. Every token added competes for the model's attention and depletes a budget that cannot be refilled mid-inference. The engineering problem is maximizing utility per token against three constraints: the hard token limit, the softer effective-capacity ceiling, and the U-shaped attention curve that penalizes information placed in the middle of context (claim-context-degradation-lost-middle-ruler).\n\nApply four principles when assembling context:\n\n1. **Informativity over exhaustiveness** — include only what matters for the current decision; design systems that can retrieve additional information on demand.\n2. **Position-aware placement** — place critical constraints at the beginning and end of context because long-context evaluations show middle-position information is less reliably recovered than edge-position information (claim-context-degradation-lost-middle-ruler).\n3. **Progressive disclosure** — load skill names and summaries at startup; load full content only when a skill activates for a specific task.\n4. **Iterative curation** — context engineering is not a one-time prompt-writing exercise but an ongoing discipline applied every time content is passed to the model.\n\n## Detailed Topics\n\n### The Anatomy of Context\n\n**System Prompts**\nOrganize system prompts into distinct sections using XML tags or Markdown headers (background, instructions, tool guidance, output format). System prompts persist throughout the conversation, so place the most critical constraints at the beginning and end where attention is strongest.\n\nCalibrate instruction altitude to balance two failure modes. Too-low altitude hardcodes brittle logic that breaks when conditions shift. Too-high altitude provides vague guidance that fails to give concrete signals for desired behavior. Aim for heuristic-driven instructions: specific enough to guide behavior, flexible enough to generalize — for example, numbered steps with room for judgment at each step.\n\nStart minimal, then add instructions reactively based on observed failure modes rather than preemptively stuffing edge cases. Curate diverse, canonical few-shot examples that portray expected behavior instead of listing every possible scenario.\n\n**Tool Definitions**\nWrite tool descriptions that answer three questions: what the tool does, when to use it, and what it returns. Include usage context, parameter defaults, and error cases — agents cannot disambiguate tools that a human engineer cannot disambiguate either.\n\nKeep the tool set minimal. Consolidate overlapping tools because bloated tool sets create ambiguous decision points and consume disproportionate context after JSON serialization (tool schemas typically inflate 2-3x compared to equivalent plain-text descriptions).\n\n**Retrieved Documents**\nMaintain lightweight identifiers (file paths, stored queries, web links) and load data into context dynamically using just-in-time retrieval. This mirrors human cognition — maintain an index, not a copy. Strong identifiers (e.g., `customer_pricing_rates.json`) let agents locate relevant files even without search tools; weak identifiers (e.g., `data/file1.json`) force unnecessary loads.\n\nWhen chunking large documents, split at natural semantic boundaries (section headers, paragraph breaks) rather than arbitrary character limits that sever mid-concept.\n\n**Message History**\nMessage history serves as the agent's scratchpad memory for tracking progress, maintaining task state, and preserving reasoning across turns. For long-running tasks, it can grow to dominate context usage — monitor and apply compaction before it crowds out active instructions.\n\nCyclically refine history: once a tool has been called deep in the conversation, the raw result rarely needs to remain verbatim. Replace stale tool outputs with compact summaries or references to reduce low-signal bulk.\n\n**Tool Outputs**\nTool outputs often dominate context in agent trajectories (claim-context-optimization-tool-output-dominance). Apply observation masking: replace verbose outputs with compact references once the agent has processed the result. Retain only the most recently relevant file contents; compress or evict older ones.\n\n### Context Windows and Attention Mechanics\n\n**The Attention Budget**\nFor n tokens, the attention mechanism computes n-squared pairwise relationships. As context grows, the model's ability to maintain these relationships degrades — not as a hard cliff but as a performance gradient. Models trained predominantly on shorter sequences have fewer specialized parameters for context-wide dependencies, creating an effective ceiling well below the nominal window size.\n\nDesign for this gradient: assume effective capacity is materially below the advertised window until measured on the target workload. Large nominal context windows do not remove the need for task-specific degradation tests (claim-context-degradation-lost-middle-ruler).\n\n**Position Encoding Limits**\nPosition encoding interpolation extends sequence handling beyond training lengths but introduces degradation in positional precision. Expect reduced accuracy for information retrieval and long-range reasoning at extended contexts compared to performance on shorter inputs.\n\n**Progressive Disclosure in Practice**\nImplement progressive disclosure at three levels:\n\n1. **Skill selection** — load only names and descriptions at startup; activate full skill content on demand.\n2. **Document loading** — load summaries first; fetch detail sections only when the task requires them.\n3. **Tool result retention** — keep recent results in full; compress or evict older results.\n\nKeep the boundary crisp: if a skill or document is activated, load it fully rather than partially — partial loads create confusing gaps that degrade reasoning quality.\n\n### Context Quality Versus Quantity\n\nReject the assumption that larger context windows solve memory problems. Processing cost grows disproportionately with context length — not just linear cost scaling, but degraded model performance beyond effective capacity thresholds. Long inputs remain expensive even with prefix caching.\n\nApply the signal-density test: for each piece of context, ask whether removing it would change the model's output. If not, remove it. Redundant content does not merely waste tokens — it actively dilutes attention from high-signal content.\n\n## Practical Guidance\n\nThis section provides conceptual application advice. Pointers to operational skills are explicit.\n\n### Reasoning About a Context Decision\n\nWhen a context-related design decision needs to be made, separate the conceptual question from the operational one. The conceptual question is \"what does this mean and why does it matter\"; the operational question is \"what specific technique do we apply.\" Use this skill to answer the first; route to the specialized skill that owns the second.\n\nFor example, deciding whether to summarize a long agent session has two parts: (1) why summarization is needed at all (attention budget is finite, U-shaped curve degrades middle content, signal density matters more than volume - this skill) and (2) what compression strategy preserves the right state and at what utilization threshold to trigger it (`context-compression`).\n\n### Reading Order For New Contributors\n\nA contributor coming to context engineering for the first time should read:\n\n1. This skill, to internalize the attention-budget framing and the U-shaped curve.\n2. `context-degradation`, to see what context failures look like in practice and how to diagnose them.\n3. Two or three of `context-optimization`, `context-compression`, `filesystem-context`, `memory-systems` depending on which operational concern is most relevant to their project.\n\nSkipping step 1 produces operators who apply techniques without understanding why; skipping the operational skills produces theorists who do not know which technique fits which failure mode.\n\n## Examples\n\n**Example 1: Organizing System Prompts**\n\nIllustrates the conceptual point that critical constraints belong at attention-favored positions (beginning and end), and that explicit section boundaries help the model parse the prompt:\n\n```markdown\n\nYou are a Python expert helping a development team.\nCurrent project: Data processing pipeline in Python 3.9+\n\n\n\n- Write clean, idiomatic Python code\n- Include type hints for function signatures\n- Add docstrings for public functions\n- Follow PEP 8 style guidelines\n\n\n\nProvide code blocks with syntax highlighting.\nExplain non-obvious decisions in comments.\n\n```\n\n**Example 2: The Attention Budget As A Mental Model**\n\nA large-context model does not have an equally attended context. Effective capacity is workload-specific, and the U-shaped curve penalizes information placed in the middle. When deciding how much of an upstream knowledge base to load, this is the mental model: do not ask \"will it fit,\" ask \"will the model still attend to the parts that matter.\"\n\nThe corresponding operational question (which technique should reduce the load) belongs to `context-optimization`.\n\n## Guidelines\n\n1. Treat context as a finite resource with diminishing returns\n2. Place critical information at attention-favored positions (beginning and end)\n3. Use progressive disclosure to defer loading until needed\n4. Organize system prompts with clear section boundaries\n5. Monitor context usage during development\n6. Implement compaction triggers at 70-80% utilization\n7. Design for context degradation rather than hoping to avoid it\n8. Prefer smaller high-signal context over larger low-signal context\n\n## Gotchas\n\n1. **Nominal window is not effective capacity**: A model advertising a large context window may degrade well before that limit on complex retrieval or reasoning tasks. Budget below the nominal window until your own degradation tests prove otherwise.\n\n2. **Character-based token estimates silently drift**: The ~4 characters/token heuristic for English prose breaks down for code (2-3 chars/token), URLs and file paths (each slash, dot, and colon is a separate token), and non-English text (often 1-2 chars/token). Use the provider's actual tokenizer (e.g., tiktoken for OpenAI models, Anthropic's token counting API) for any budget-critical calculation.\n\n3. **Tool schemas inflate 2-3x after JSON serialization**: A tool definition that looks compact in source code expands significantly when serialized — brackets, quotes, colons, and commas each consume tokens. Ten tools with moderate schemas can consume 5,000-8,000 tokens before a single message is sent. Audit serialized tool token counts, not source-code line counts.\n\n4. **Message history balloons silently in agentic loops**: Each tool call adds both the request and the full response to history. After 20-30 iterations, history can consume 70-80% of the window while the agent shows no visible symptoms until reasoning quality collapses. Set a hard token ceiling on history and trigger compaction proactively.\n\n5. **Critical instructions in the middle get lost**: The U-shaped attention curve means the middle of context receives 10-40% less recall accuracy than the beginning and end. Never place safety constraints, output format requirements, or behavioral guardrails in the middle of a long system prompt — anchor them at the top or bottom.\n\n6. **Progressive disclosure that loads too eagerly defeats its purpose**: Loading every \"potentially relevant\" skill or document at the first hint of relevance recreates the context-stuffing problem. Set strict activation thresholds — a skill should load only when the task explicitly matches its trigger conditions, not when the topic is merely adjacent.\n\n7. **Mixing instruction altitudes causes inconsistent behavior**: Combining hyper-specific rules (\"always use exactly 3 bullet points\") with vague directives (\"be helpful\") in the same prompt creates conflicting signals. Group instructions by altitude level and keep each section internally consistent — either heuristic-driven or prescriptive, not both interleaved.\n\n## Integration\n\nThis skill is the conceptual foundation. It does not own operational work; it provides the mental models the operational skills assume.\n\nRouting map for operational work:\n\n- `context-degradation`: diagnosing attention failures, lost-in-middle, poisoning, distraction.\n- `context-optimization`: token-efficiency tactics (masking, partitioning, caching, budgets).\n- `context-compression`: compacting long sessions while preserving decisions, files, risks.\n- `filesystem-context`: offloading large outputs and using files as a durable scratchpad.\n- `memory-systems`: cross-session memory architectures with entity tracking.\n- `multi-agent-patterns`: when to split work across agents for context isolation.\n- `tool-design`: writing tool descriptions and schemas that route correctly.\n- `project-development`: deciding LLM fit and shaping multi-stage pipelines.\n\nRead this skill first to build the mental models; read the operational skill that fits the task when actually doing the work.\n\n## References\n\nInternal reference:\n- [Context Components Reference](./references/context-components.md) - Read when: debugging a specific context component (system prompts, tool definitions, message history, tool outputs) or implementing chunking, observation masking, or budget allocation tables\n\nRelated skills in this collection:\n- context-degradation - Read when: agent performance drops as conversations grow or context fills beyond 60% capacity\n- context-optimization - Read when: token costs are too high or compaction/compression strategies are needed\n\nExternal resources:\n- Anthropic's \"Effective Context Engineering for AI Agents\" — production patterns for compaction, sub-agents, and hybrid retrieval\n- Research on transformer attention mechanisms and the lost-in-the-middle effect\n- Tokenomics research on agentic software engineering token distribution\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-20\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.2.0\n","skills/context-fundamentals/references/context-components.md":"# Context Components: Technical Reference\n\nThis document provides detailed technical reference for each context component in agent systems.\n\n## System Prompt Engineering\n\n### Section Structure\n\nOrganize system prompts into distinct sections with clear boundaries. A recommended structure:\n\n```\n\nContext about the domain, user preferences, or project-specific details\n\n\n\nCore behavioral guidelines and task instructions\n\n\n\nWhen and how to use available tools\n\n\n\nExpected output format and quality standards\n\n```\n\nThis structure allows agents to locate relevant information quickly and enables selective context loading in advanced implementations.\n\n### Altitude Calibration\n\nThe \"altitude\" of instructions refers to the level of abstraction. Consider these examples:\n\n**Too Low (Brittle):**\n```\nIf the user asks about pricing, check the pricing table in docs/pricing.md.\nIf the table shows USD, convert to EUR using the exchange rate in\nconfig/exchange_rates.json. If the user is in the EU, add VAT at the\napplicable rate from config/vat_rates.json. Format the response with\nthe currency symbol, two decimal places, and a note about VAT.\n```\n\n**Too High (Vague):**\n```\nHelp users with pricing questions. Be helpful and accurate.\n```\n\n**Optimal (Heuristic-Driven):**\n```\nFor pricing inquiries:\n1. Retrieve current rates from docs/pricing.md\n2. Apply user location adjustments (see config/location_defaults.json)\n3. Format with appropriate currency and tax considerations\n\nPrefer exact figures over estimates. When rates are unavailable,\nsay so explicitly rather than projecting.\n```\n\nThe optimal altitude provides clear steps while allowing flexibility in execution.\n\n## Tool Definition Specification\n\n### Schema Structure\n\nEach tool should define:\n\n```python\n{\n \"name\": \"tool_function_name\",\n \"description\": \"Clear description of what the tool does and when to use it\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"param_name\": {\n \"type\": \"string\",\n \"description\": \"What this parameter controls\",\n \"default\": \"reasonable_default_value\"\n }\n },\n \"required\": [\"param_name\"]\n },\n \"returns\": {\n \"type\": \"object\",\n \"description\": \"What the tool returns and its structure\"\n }\n}\n```\n\n### Description Engineering\n\nTool descriptions should answer: what the tool does, when to use it, and what it produces. Include usage context, examples, and edge cases.\n\n**Weak Description:**\n```\nSearch the database for customer information.\n```\n\n**Strong Description:**\n```\nRetrieve customer information by ID or email.\n\nUse when:\n- User asks about a specific customer's details, history, or status\n- User provides a customer identifier and needs related information\n\nReturns customer object with:\n- Basic info (name, email, account status)\n- Order history summary\n- Support ticket count\n\nReturns null if customer not found. Returns error if database unreachable.\n```\n\n## Retrieved Document Management\n\n### Identifier Design\n\nDesign identifiers that convey meaning and enable efficient retrieval:\n\n**Poor identifiers:**\n- `data/file1.json`\n- `ref/ref.md`\n- `2024/q3/report`\n\n**Strong identifiers:**\n- `customer_pricing_rates.json`\n- `engineering_onboarding_checklist.md`\n- `2024_q3_revenue_report.pdf`\n\nStrong identifiers allow agents to locate relevant files even without search tools.\n\n### Document Chunking Strategy\n\nFor large documents, chunk strategically to preserve semantic coherence:\n\n```python\n# Pseudocode for semantic chunking\ndef chunk_document(content):\n \"\"\"Split document at natural semantic boundaries.\"\"\"\n boundaries = find_section_headers(content)\n boundaries += find_paragraph_breaks(content)\n boundaries += find_logical_breaks(content)\n \n chunks = []\n for i in range(len(boundaries) - 1):\n chunk = content[boundaries[i]:boundaries[i+1]]\n if len(chunk) > MIN_CHUNK_SIZE and len(chunk) < MAX_CHUNK_SIZE:\n chunks.append(chunk)\n \n return chunks\n```\n\nAvoid arbitrary character limits that split mid-sentence or mid-concept.\n\n## Message History Management\n\n### Turn Representation\n\nStructure message history to preserve key information:\n\n```python\n{\n \"role\": \"user\" | \"assistant\" | \"tool\",\n \"content\": \"message text\",\n \"reasoning\": \"optional chain-of-thought\",\n \"tool_calls\": [list if role=\"assistant\"],\n \"tool_output\": \"output if role=\"tool\"\",\n \"summary\": \"compact summary if conversation is long\"\n}\n```\n\n### Summary Injection Pattern\n\nFor long conversations, inject summaries at intervals:\n\n```python\ndef inject_summaries(messages, summary_interval=20):\n \"\"\"Inject summaries at regular intervals to preserve context.\"\"\"\n summarized = []\n for i, msg in enumerate(messages):\n summarized.append(msg)\n if i > 0 and i % summary_interval == 0:\n summary = generate_summary(summarized[-summary_interval:])\n summarized.append({\n \"role\": \"system\",\n \"content\": f\"Conversation summary: {summary}\",\n \"is_summary\": True\n })\n return summarized\n```\n\n## Tool Output Optimization\n\n### Response Formats\n\nProvide response format options to control token usage:\n\n```python\ndef get_customer_response_format():\n return {\n \"format\": \"concise | detailed\",\n \"fields\": [\"id\", \"name\", \"email\", \"status\", \"history_summary\"]\n }\n```\n\nThe concise format returns essential fields only; detailed returns complete objects.\n\n### Observation Masking\n\nFor verbose tool outputs, consider masking patterns:\n\n```python\ndef mask_observation(output, max_length=500):\n \"\"\"Replace long observations with compact references.\"\"\"\n if len(output) <= max_length:\n return output\n \n reference_id = store_observation(output)\n return f\"[Previous observation elided. Full content stored at reference {reference_id}]\"\n```\n\nThis preserves information access while reducing token usage.\n\n## Context Budget Estimation\n\n### Token Counting Approximation\n\nFor planning purposes, estimate tokens at approximately 4 characters per token for English text:\n\n```\n1000 words ≈ 7500 characters ≈ 1800-2000 tokens\n```\n\nThis is a rough approximation; actual tokenization varies by model and content type.\n\n### Context Budget Allocation\n\nAllocate context budget across components:\n\n| Component | Typical Range | Notes |\n|-----------|---------------|-------|\n| System prompt | 500-2000 tokens | Stable across session |\n| Tool definitions | 100-500 per tool | Grows with tool count |\n| Retrieved documents | Variable | Often largest consumer |\n| Message history | Variable | Grows with conversation |\n| Tool outputs | Variable | Can dominate context |\n\nMonitor actual usage during development to establish baseline allocations.\n\n## Progressive Disclosure Implementation\n\n### Skill Activation Pattern\n\n```python\ndef activate_skill_context(skill_name, task_description):\n \"\"\"Load skill context when task matches skill description.\"\"\"\n skill_metadata = load_all_skill_metadata()\n \n relevant_skills = []\n for skill in skill_metadata:\n if skill_matches_task(skill, task_description):\n relevant_skills.append(skill)\n \n # Load full content only for most relevant skills\n for skill in relevant_skills[:MAX_CONCURRENT_SKILLS]:\n skill_context = load_skill_content(skill)\n inject_into_context(skill_context)\n```\n\n### Reference Loading Pattern\n\n```python\ndef get_reference(file_reference):\n \"\"\"Load reference file only when explicitly needed.\"\"\"\n if not file_reference.is_loaded:\n file_reference.content = read_file(file_reference.path)\n file_reference.is_loaded = True\n return file_reference.content\n```\n\nThis pattern ensures files are loaded once and cached for the session.\n\n","skills/context-optimization/SKILL.md":"---\nname: context-optimization\ndescription: \"This skill should be used for improving context efficiency: context budgeting, observation masking, prefix or KV-cache strategy, partitioning, token-cost reduction, retrieval scoping, and extending effective context capacity without lowering answer quality.\"\n---\n\n# Context Optimization Techniques\n\nContext optimization extends the effective capacity of limited context windows through strategic compression, masking, caching, and partitioning. Effective optimization increases useful capacity without requiring larger models or longer windows — but only when applied with measurement discipline. The techniques below are ordered by impact and risk.\n\n## When to Activate\n\nActivate this skill when:\n- Context budgets or token costs constrain task complexity\n- Observation masking can replace verbose tool outputs with retrievable references\n- Prefix or KV-cache hit rate needs improvement\n- Retrieval scoping can reduce irrelevant loaded context\n- Context partitioning can extend effective capacity across agents\n- Budget triggers are needed for masking, compaction, or partitioning\n\nDo not activate this skill for adjacent work owned by other skills:\n- Explaining why attention or context windows behave this way: `context-fundamentals`.\n- Diagnosing active lost-in-middle, poisoning, distraction, confusion, or clash: `context-degradation`.\n- Designing a structured handoff summary for a long conversation: `context-compression`.\n- Storing large outputs, plans, or logs as files: `filesystem-context`.\n- Model-initiated rewrites of its own context and their re-prefill cost: `self-managed-context`.\n\n## Core Concepts\n\nApply four primary strategies in this priority order:\n\n1. **KV-cache optimization** — Reorder and stabilize prompt structure so the inference engine reuses cached Key/Value tensors. This is the cheapest optimization when the runtime supports prefix caching: low quality risk, immediate cost and latency savings. Apply it first when stable prefixes exist.\n\n2. **Observation masking** — Replace verbose tool outputs with compact references once their purpose has been served. Tool outputs can dominate agent trajectories (claim-context-optimization-tool-output-dominance), so masking often yields the largest capacity gains. The original content remains retrievable if needed downstream.\n\n3. **Compaction** — Summarize accumulated context when utilization exceeds 70%, then reinitialize with the summary. This distills the window's contents while preserving task-critical state. Compaction is lossy — apply it after masking has already removed the low-value bulk.\n\n4. **Context partitioning** — Split work across sub-agents with isolated contexts when a single window cannot hold the full problem. Each sub-agent operates in a clean context focused on its subtask. Reserve this for tasks where estimated context exceeds 60% of the window limit, because coordination overhead is real.\n\nThe governing principle: context quality matters more than quantity. Every optimization preserves signal while reducing noise. Measure before optimizing, then measure the optimization's effect.\n\n## Detailed Topics\n\n### Compaction Strategies\n\nTrigger compaction when context utilization exceeds 70%: summarize the current context, then reinitialize with the summary. This distills the window's contents in a high-fidelity manner, enabling continuation with minimal performance degradation. Prioritize compressing tool outputs first (they consume 80%+ of tokens), then old conversation turns, then retrieved documents. Never compress the system prompt — it anchors model behavior and its removal causes unpredictable degradation.\n\nPreserve different elements by message type:\n\n- **Tool outputs**: Extract key findings, metrics, error codes, and conclusions. Strip verbose raw output, stack traces (unless debugging is ongoing), and boilerplate headers.\n- **Conversational turns**: Retain decisions, commitments, user preferences, and context shifts. Remove filler, pleasantries, and exploratory back-and-forth that led to a conclusion already captured.\n- **Retrieved documents**: Keep claims, facts, and data points relevant to the active task. Remove supporting evidence and elaboration that served a one-time reasoning purpose.\n\nTarget 50-70% token reduction with less than 5% quality degradation. If compaction exceeds 70% reduction, audit the summary for critical information loss — over-aggressive compaction is the most common failure mode.\n\n### Observation Masking\n\nMask observations selectively based on recency and ongoing relevance — not uniformly. Apply these rules:\n\n- **Never mask**: Observations critical to the current task, observations from the most recent turn, observations used in active reasoning chains, and error outputs when debugging is in progress.\n- **Mask after 3+ turns**: Verbose outputs whose key points have already been extracted into the conversation flow. Replace with a compact reference: `[Obs:{ref_id} elided. Key: {summary}. Full content retrievable.]`\n- **Always mask immediately**: Repeated/duplicate outputs, boilerplate headers and footers, outputs already summarized earlier in the conversation.\n\nMasking should achieve 60-80% reduction in masked observations with less than 2% quality impact. The key is maintaining retrievability — store the full content externally and keep the reference ID in context so the agent can request the original if needed.\n\n### KV-Cache Optimization\n\nMaximize prefix cache hits by structuring prompts so that stable content occupies the prefix and dynamic content appears at the end. KV-cache stores Key and Value tensors computed during inference; when consecutive requests share an identical prefix, the cached tensors are reused, saving both cost and latency.\n\nApply this ordering in every prompt:\n1. System prompt (most stable — never changes within a session)\n2. Tool definitions (stable across requests)\n3. Frequently reused templates and few-shot examples\n4. Conversation history (grows but shares prefix with prior turns)\n5. Current query and dynamic content (least stable — always last)\n\nDesign prompts for cache stability: remove timestamps, session counters, and request IDs from the system prompt. Move dynamic metadata into a separate user message or tool result where it does not break the prefix. Even a single whitespace change in the prefix invalidates the entire cached block downstream of that change. The same rule prices any compaction or masking edit applied to earlier history: its cost is the text that follows it, so batch such edits and prefer them near the end of the context.\n\nTarget 70%+ cache hit rate for stable workloads. At scale, this translates to 50%+ cost reduction and 40%+ latency reduction on cached tokens.\n\n### Context Partitioning\n\nPartition work across sub-agents when a single context cannot hold the full problem without triggering aggressive compaction. Each sub-agent operates in a clean, focused context for its subtask, then returns a structured result to a coordinator agent.\n\nPlan partitioning when estimated task context exceeds 60% of the window limit. Decompose the task into independent subtasks, assign each to a sub-agent, and aggregate results. Validate that all partitions completed before merging, merge compatible results, and apply summarization if the aggregated output still exceeds budget.\n\nThis approach achieves separation of concerns — detailed search context stays isolated within sub-agents while the coordinator focuses on synthesis. However, coordination has real token cost: the coordinator prompt, result aggregation, and error handling all consume tokens. Only partition when the savings exceed this overhead.\n\n### Budget Management\n\nAllocate explicit token budgets across context categories before the session begins: system prompt, tool definitions, retrieved documents, message history, tool outputs, and a reserved buffer (5-10% of total). Monitor usage against budget continuously and trigger optimization when any category exceeds its allocation or total utilization crosses 70%.\n\nUse trigger-based optimization rather than periodic optimization. Monitor these signals:\n- Token utilization above 80% — trigger compaction\n- Attention degradation indicators (repetition, missed instructions) — trigger masking + compaction\n- Quality score drops below baseline — audit context composition before optimizing\n\n## Practical Guidance\n\n### Optimization Decision Framework\n\nSelect the optimization technique based on what dominates the context:\n\n| Context Composition | First Action | Second Action |\n|---|---|---|\n| Tool outputs dominate (>50%) | Observation masking | Compaction of remaining turns |\n| Retrieved documents dominate | Summarization | Partitioning if docs are independent |\n| Message history dominates | Compaction with selective preservation | Partitioning for new subtasks |\n| Multiple components contribute | KV-cache optimization first, then layer masking + compaction |\n| Near-limit with active debugging | Mask resolved tool outputs only — preserve error details |\n\n### Performance Targets\n\nTrack these metrics to validate optimization effectiveness:\n\n- **Compaction**: 50-70% token reduction, <5% quality degradation, <10% latency overhead from the compaction step itself\n- **Masking**: 60-80% reduction in masked observations, <2% quality impact, near-zero latency overhead\n- **Cache optimization**: 70%+ hit rate for stable workloads, 50%+ cost reduction, 40%+ latency reduction\n- **Partitioning**: Net token savings after accounting for coordinator overhead; break-even typically requires 3+ subtasks\n\nIterate on strategies based on measured results. If an optimization technique does not measurably improve the target metric, remove it — optimization machinery itself consumes tokens and adds latency.\n\n## Examples\n\n**Example 1: Compaction Trigger**\n```python\nif context_tokens / context_limit > 0.8:\n context = compact_context(context)\n```\n\n**Example 2: Observation Masking**\n```python\nif len(observation) > max_length:\n ref_id = store_observation(observation)\n return f\"[Obs:{ref_id} elided. Key: {extract_key(observation)}]\"\n```\n\n**Example 3: Cache-Friendly Ordering**\n```python\n# Stable content first\ncontext = [system_prompt, tool_definitions] # Cacheable\ncontext += [reused_templates] # Reusable\ncontext += [unique_content] # Unique\n```\n\n**Example 4: Budget-triggered optimization policy**\n```yaml\nbudgets:\n tool_outputs: 35%\n message_history: 30%\n retrieved_documents: 20%\n reserved_buffer: 15%\ntriggers:\n tool_outputs_over_budget: mask resolved observations\n total_context_over_70_percent: compact message history\n repeated_irrelevant_retrievals: tighten retrieval scope\n```\n\n## Guidelines\n\n1. Measure before optimizing—know your current state\n2. Apply masking before compaction — remove low-value bulk first, then summarize what remains\n3. Design for cache stability with consistent prompts\n4. Partition before context becomes problematic\n5. Monitor optimization effectiveness over time\n6. Balance token savings against quality preservation\n7. Test optimization at production scale\n8. Implement graceful degradation for edge cases\n\n## Gotchas\n\n1. **Whitespace breaks KV-cache**: Even a single whitespace or newline change in the prompt prefix invalidates the entire KV-cache block downstream of that point. Pin system prompts as immutable strings — do not interpolate timestamps, version numbers, or session IDs into them. Diff prompt templates byte-for-byte between deployments.\n\n2. **Timestamps in system prompts destroy cache hit rates**: Including `Current date: {today}` or similar dynamic content in the system prompt forces a full cache miss on every new day (or every request, if using time-of-day). Move dynamic metadata into a user message or a separate tool result appended after the stable prefix.\n\n3. **Compaction under pressure loses critical state**: When the model performing compaction is itself under context pressure (>85% utilization), its summarization quality degrades — it omits task goals, drops user constraints, and flattens nuanced state. Trigger compaction at 70-80%, not 90%+. If compaction must happen late, use a separate model call with a clean context containing only the material to summarize.\n\n4. **Masking error outputs breaks debugging loops**: Over-aggressive masking hides error messages, stack traces, and failure details that the agent needs in subsequent turns to diagnose and fix issues. During active debugging (error in the last 3 turns), suspend masking for all error-related observations until the issue is resolved.\n\n5. **Partitioning overhead can exceed savings**: Each sub-agent requires its own system prompt, tool definitions, and coordination messages. For tasks with fewer than 3 independent subtasks, the coordination overhead often exceeds the context savings. Estimate total tokens (coordinator + all sub-agents) before committing to partitioning.\n\n6. **Cache miss cost spikes after deployment changes**: Reordering tools, rewording the system prompt, or changing few-shot examples between deployments invalidates the entire prefix cache, causing a temporary cost spike of 2-5x until the new cache warms up. Roll out prompt changes gradually and monitor cache hit rate during deployment windows.\n\n7. **Compaction creates false confidence in stale summaries**: Once context is compacted, the summary looks authoritative but may reflect outdated state. If the task has evolved since compaction (new user requirements, corrected assumptions), the summary silently carries forward stale information. After compaction, re-validate the summary against the current task goal before proceeding.\n\n## Integration\n\nThis skill owns token-efficiency tactics and budget policy. Adjacent skills own diagnosis, storage, and architecture:\n\n- `context-fundamentals`: mental models for why context quality and attention placement matter.\n- `context-degradation`: diagnosis when output quality has already dropped.\n- `context-compression`: lossy summarization and handoff strategy.\n- `filesystem-context`: file-backed offloading for full outputs and logs.\n- `multi-agent-patterns`: partitioning work across isolated agent contexts.\n- `latent-briefing`: selective KV retention across orchestrator-worker boundaries in compatible runtimes.\n- `self-managed-context`: the model, not the harness, decides when and what to edit in its live context.\n- `evaluation`: measuring whether the optimization improved quality, cost, or latency.\n- `memory-systems`: persistent retrieval layers that feed context just in time.\n\n## References\n\nInternal reference:\n- [Optimization Techniques Reference](./references/optimization_techniques.md) - Read when: implementing a specific optimization technique and needing detailed code patterns, threshold tables, or integration examples beyond what the skill body provides\n\nRelated skills in this collection:\n- context-fundamentals - Read when: unfamiliar with context window mechanics, token counting, or attention distribution basics\n- context-degradation - Read when: diagnosing why agent performance has dropped and needing to identify which degradation pattern is occurring before selecting an optimization\n- evaluation - Read when: setting up metrics and benchmarks to measure whether an optimization technique actually improved outcomes\n\nExternal resources:\n- Research on context window limitations - Read when: evaluating model-specific context behavior (e.g., lost-in-the-middle effects, attention decay curves)\n- KV-cache optimization techniques - Read when: implementing prefix caching at the inference infrastructure level (vLLM, TGI, or cloud provider APIs)\n- Production engineering guides - Read when: deploying context optimization in a production pipeline and needing operability patterns (monitoring, alerting, rollback)\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-20\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.1.0\n","skills/context-optimization/references/optimization_techniques.md":"# Context Optimization Reference\n\nThis document provides detailed technical reference for context optimization techniques and strategies.\n\n## Compaction Strategies\n\n### Summary-Based Compaction\n\nSummary-based compaction replaces verbose content with concise summaries while preserving key information. The approach works by identifying sections that can be compressed, generating summaries that capture essential points, and replacing full content with summaries.\n\nThe effectiveness of compaction depends on what information is preserved. Critical decisions, user preferences, and current task state should never be compacted. Intermediate results and supporting evidence can be summarized more aggressively. Boilerplate, repeated information, and exploratory reasoning can often be removed entirely.\n\n### Token Budget Allocation\n\nEffective context budgeting requires understanding how different context components consume tokens and allocating budget strategically:\n\n| Component | Typical Range | Notes |\n|-----------|---------------|-------|\n| System prompt | 500-2000 tokens | Stable across session |\n| Tool definitions | 100-500 per tool | Grows with tool count |\n| Retrieved documents | Variable | Often largest consumer |\n| Message history | Variable | Grows with conversation |\n| Tool outputs | Variable | Can dominate context |\n\n### Compaction Thresholds\n\nTrigger compaction at appropriate thresholds to maintain performance:\n\n- Warning threshold at 70% of effective context limit\n- Compaction trigger at 80% of effective context limit\n- Aggressive compaction at 90% of effective context limit\n\nThe exact thresholds depend on model behavior and task characteristics. Some models show graceful degradation while others exhibit sharp performance cliffs.\n\n## Observation Masking Patterns\n\n### Selective Masking\n\nNot all observations should be masked equally. Consider masking observations that have served their purpose and are no longer needed for active reasoning. Keep observations that are central to the current task. Keep observations from the most recent turn. Keep observations that may be referenced again.\n\n### Masking Implementation\n\n```python\ndef selective_mask(observations: List[Dict], current_task: Dict) -> List[Dict]:\n \"\"\"\n Selectively mask observations based on relevance.\n \n Returns observations with mask field indicating masked content.\n \"\"\"\n masked = []\n \n for obs in observations:\n relevance = calculate_relevance(obs, current_task)\n \n if relevance < 0.3 and obs[\"age\"] > 3:\n # Low relevance and old - mask\n masked.append({\n **obs,\n \"masked\": True,\n \"reference\": store_for_reference(obs[\"content\"]),\n \"summary\": summarize_content(obs[\"content\"])\n })\n else:\n masked.append({\n **obs,\n \"masked\": False\n })\n \n return masked\n```\n\n## KV-Cache Optimization\n\n### Prefix Stability\n\nKV-cache hit rates depend on prefix stability. Stable prefixes enable cache reuse across requests. Dynamic prefixes invalidate cache and force recomputation.\n\nElements that should remain stable include system prompts, tool definitions, and frequently used templates. Elements that may vary include timestamps, session identifiers, and query-specific content.\n\n### Cache-Friendly Design\n\nDesign prompts to maximize cache hit rates:\n\n1. Place stable content at the beginning\n2. Use consistent formatting across requests\n3. Avoid dynamic content in prompts when possible\n4. Use placeholders for dynamic content\n\n```python\n# Cache-unfriendly: Dynamic timestamp in prompt\nsystem_prompt = f\"\"\"\nCurrent time: {datetime.now().isoformat()}\nYou are a helpful assistant.\n\"\"\"\n\n# Cache-friendly: Stable prompt with dynamic time as variable\nsystem_prompt = \"\"\"\nYou are a helpful assistant.\nCurrent time is provided separately when relevant.\n\"\"\"\n```\n\n## Context Partitioning Strategies\n\n### Sub-Agent Isolation\n\nPartition work across sub-agents to prevent any single context from growing too large. Each sub-agent operates with a clean context focused on its subtask.\n\n### Partition Planning\n\n```python\ndef plan_partitioning(task: Dict, context_limit: int) -> Dict:\n \"\"\"\n Plan how to partition a task based on context limits.\n \n Returns partitioning strategy and subtask definitions.\n \"\"\"\n estimated_context = estimate_task_context(task)\n \n if estimated_context <= context_limit:\n return {\n \"strategy\": \"single_agent\",\n \"subtasks\": [task]\n }\n \n # Plan multi-agent approach\n subtasks = decompose_task(task)\n \n return {\n \"strategy\": \"multi_agent\",\n \"subtasks\": subtasks,\n \"coordination\": \"hierarchical\"\n }\n```\n\n## Optimization Decision Framework\n\n### When to Optimize\n\nConsider context optimization when context utilization exceeds 70%, when response quality degrades as conversations extend, when costs increase due to long contexts, or when latency increases with conversation length.\n\n### What Optimization to Apply\n\nChoose optimization strategies based on context composition:\n\nIf tool outputs dominate context, apply observation masking. If retrieved documents dominate context, apply summarization or partitioning. If message history dominates context, apply compaction with summarization. If multiple components contribute, combine strategies.\n\n### Evaluation of Optimization\n\nAfter applying optimization, evaluate effectiveness:\n\n- Measure token reduction achieved\n- Measure quality preservation (output quality should not degrade)\n- Measure latency improvement\n- Measure cost reduction\n\nIterate on optimization strategies based on evaluation results.\n\n## Common Pitfalls\n\n### Over-Aggressive Compaction\n\nCompacting too aggressively can remove critical information. Always preserve task goals, user preferences, and recent conversation context. Test compaction at increasing aggressiveness levels to find the optimal balance.\n\n### Masking Critical Observations\n\nMasking observations that are still needed can cause errors. Track observation usage and only mask content that is no longer referenced. Consider keeping references to masked content that could be retrieved if needed.\n\n### Ignoring Attention Distribution\n\nThe lost-in-middle phenomenon means that information placement matters. Place critical information at attention-favored positions (beginning and end of context). Use explicit markers to highlight important content.\n\n### Premature Optimization\n\nNot all contexts require optimization. Adding optimization machinery has overhead. Optimize only when context limits actually constrain agent performance.\n\n## Monitoring and Alerting\n\n### Key Metrics\n\nTrack these metrics to understand optimization needs:\n\n- Context token count over time\n- Cache hit rates for repeated patterns\n- Response quality metrics by context size\n- Cost per conversation by context length\n- Latency by context size\n\n### Alert Thresholds\n\nSet alerts for:\n\n- Context utilization above 80%\n- Cache hit rate below 50%\n- Quality score drop of more than 10%\n- Cost increase above baseline\n\n## Integration Patterns\n\n### Integration with Agent Framework\n\nIntegrate optimization into agent workflow:\n\n```python\nclass OptimizingAgent:\n def __init__(self, context_limit: int = 80000):\n self.context_limit = context_limit\n self.optimizer = ContextOptimizer()\n \n def process(self, user_input: str, context: Dict) -> Dict:\n # Check if optimization needed\n if self.optimizer.should_compact(context):\n context = self.optimizer.compact(context)\n \n # Process with optimized context\n response = self._call_model(user_input, context)\n \n # Track metrics\n self.optimizer.record_metrics(context, response)\n \n return response\n```\n\n### Integration with Memory Systems\n\nConnect optimization with memory systems:\n\n```python\nclass MemoryAwareOptimizer:\n def __init__(self, memory_system, context_limit: int):\n self.memory = memory_system\n self.limit = context_limit\n \n def optimize_context(self, current_context: Dict, task: str) -> Dict:\n # Check if information is in memory\n relevant_memories = self.memory.retrieve(task)\n \n # Move information to memory if not needed in context\n for mem in relevant_memories:\n if mem[\"importance\"] < threshold:\n current_context = remove_from_context(current_context, mem)\n # Keep reference that memory can be retrieved\n \n return current_context\n```\n\n## Performance Benchmarks\n\n### Compaction Performance\n\nCompaction should reduce token count while preserving quality. Target:\n\n- 50-70% token reduction for aggressive compaction\n- Less than 5% quality degradation from compaction\n- Less than 10% latency increase from compaction overhead\n\n### Masking Performance\n\nObservation masking should reduce token count significantly:\n\n- 60-80% reduction in masked observations\n- Less than 2% quality impact from masking\n- Near-zero latency overhead\n\n### Cache Performance\n\nKV-cache optimization should improve cost and latency:\n\n- 70%+ cache hit rate for stable workloads\n- 50%+ cost reduction from cache hits\n- 40%+ latency reduction from cache hits\n\n","skills/evaluation/SKILL.md":"---\nname: evaluation\ndescription: \"This skill should be used when building agent evaluation systems: deterministic checks, regression suites, multi-dimensional rubrics, quality gates, production monitoring, baseline comparison, and outcome measurement for agent pipelines.\"\n---\n\n# Evaluation Methods for Agent Systems\n\nEvaluate agent systems differently from traditional software because agents make dynamic decisions, are non-deterministic between runs, and often lack single correct answers. Build evaluation frameworks that account for these characteristics, provide actionable feedback, catch regressions, and validate that context engineering choices achieve intended effects.\n\n## When to Activate\n\nActivate this skill when:\n- Testing agent performance systematically\n- Validating context engineering choices\n- Measuring improvements over time\n- Catching regressions before deployment\n- Building quality gates for agent pipelines\n- Comparing different agent configurations\n- Evaluating production systems continuously\n\nDo not activate this skill for adjacent work owned by other skills:\n- Designing the LLM judge itself, pairwise comparison, judge calibration, or bias mitigation: `advanced-evaluation`.\n- Designing autonomous control surfaces, novelty gates, rollback, or PR approval boundaries: `harness-engineering`.\n- Debugging a specific context failure mode before measuring it: `context-degradation`.\n\n## Core Concepts\n\nFocus evaluation on outcomes rather than execution paths, because agents may find alternative valid routes to goals. Judge whether the agent achieves the right outcome via a reasonable process, not whether it followed a specific sequence of steps.\n\nUse multi-dimensional rubrics instead of single scores because one number hides critical failures in specific dimensions. Capture factual accuracy, completeness, citation accuracy, source quality, and tool efficiency as separate dimensions, then weight them for the use case.\n\nUse model-judged evaluation only after deterministic checks and rubrics are stable. When the work centers on judge prompts, pairwise comparison, calibration, or bias mitigation, switch to Advanced Evaluation.\n\nRun deterministic validation before LLM judgment whenever the artifact has machine-checkable structure. Schema validity, duplicate keys, rubric math, manifest sync, retrieval status, and required evidence paths should fail fast before an evaluator spends tokens or returns a subjective score.\n\n**Performance Drivers**\n\nApply browsing-agent research when designing evaluation budgets: token usage, tool calls, and model choice can dominate measured performance variance (claim-evaluation-browsecomp-variance).\n\n| Factor | Variance Explained | Implication |\n|--------|-------------------|-------------|\n| Token usage | Primary driver | More exploration can improve performance until cost or context quality collapses |\n| Number of tool calls | Secondary driver | More tool use helps only when calls retrieve useful evidence |\n| Model choice | Secondary but multiplicative | Better models often use tokens and tools more efficiently |\n\nAct on these implications when designing evaluations:\n- **Set realistic token budgets**: Evaluate agents with production-realistic token limits, not unlimited resources.\n- **Compare model upgrades against token increases**: Better models may use tokens more efficiently than weaker models with larger budgets.\n- **Validate multi-agent architectures**: Extra agents add tokens and tool calls; evaluate them against single-agent baselines.\n\n## Detailed Topics\n\n### Evaluation Challenges\n\n**Handle Non-Determinism and Multiple Valid Paths**\n\nDesign evaluations that tolerate path variation because agents may take completely different valid paths to reach goals. One agent might search three sources while another searches ten; both may produce correct answers. Avoid checking for specific steps. Instead, define outcome criteria (correctness, completeness, quality) and score against those, treating the execution path as informational rather than evaluative.\n\n**Test Context-Dependent Failures**\n\nEvaluate across a range of complexity levels and interaction lengths because agent failures often depend on context in subtle ways. An agent might succeed on simple queries but fail on complex ones, work well with one tool set but fail with another, or degrade after extended interaction as context accumulates. Include simple, medium, complex, and very complex test cases to surface these patterns.\n\n**Score Composite Quality Dimensions Separately**\n\nBreak agent quality into separate dimensions (factual accuracy, completeness, coherence, tool efficiency, process quality) and score each independently because an agent might score high on accuracy but low on efficiency, or vice versa. Then compute weighted aggregates tuned to use-case priorities. This approach reveals which dimensions need improvement rather than averaging away the signal.\n\n### Evaluation Rubric Design\n\n**Build Multi-Dimensional Rubrics**\n\nDefine rubrics covering key dimensions with descriptive levels from excellent to failed. Include these core dimensions and adapt weights per use case:\n\n- Factual accuracy: Claims match ground truth (weight heavily for knowledge tasks)\n- Completeness: Output covers requested aspects (weight heavily for research tasks)\n- Citation accuracy: Citations match claimed sources (weight for trust-sensitive contexts)\n- Source quality: Uses appropriate primary sources (weight for authoritative outputs)\n- Tool efficiency: Uses right tools a reasonable number of times (weight for cost-sensitive systems)\n\n**Convert Rubrics to Numeric Scores**\n\nMap dimension assessments to numeric scores (0.0 to 1.0), apply per-dimension weights, and calculate weighted overall scores. Set passing thresholds based on use-case requirements, typically 0.7 for general use and 0.9 for high-stakes applications. Store individual dimension scores alongside the aggregate because the breakdown drives targeted improvement.\n\n### Evaluation Methodologies\n\n**Use LLM-as-Judge for Scale**\n\nBuild LLM-based evaluation prompts that include: clear task description, the agent output under test, ground truth when available, an evaluation scale with explicit level descriptions, and a request for structured judgment with reasoning. LLM judges provide consistent, scalable evaluation across large test sets. Use a different model family than the agent being evaluated to avoid self-enhancement bias.\n\n**Supplement with Human Evaluation**\n\nRoute edge cases, unusual queries, and a random sample of production traffic to human reviewers because humans notice hallucinated answers, system failures, and subtle biases that automated evaluation misses. Track patterns across human reviews to identify systematic issues and feed findings back into automated evaluation criteria.\n\n**Apply End-State Evaluation for Stateful Agents**\n\nFor agents that mutate persistent state (files, databases, configurations), evaluate whether the final state matches expectations rather than how the agent got there. Define expected end-state assertions and verify them programmatically after each test run.\n\n### Test Set Design\n\n**Select Representative Samples**\n\nStart with small samples (20-30 cases) during early development when changes have dramatic impacts and low-hanging fruit is abundant. Scale to 50+ cases for reliable signal as the system matures. Sample from real usage patterns, add known edge cases, and ensure coverage across complexity levels.\n\n**Stratify by Complexity**\n\nStructure test sets across complexity levels to prevent easy examples from inflating scores:\n- Simple: single tool call, factual lookup\n- Medium: multiple tool calls, comparison logic\n- Complex: many tool calls, significant ambiguity\n- Very complex: extended interaction, deep reasoning, synthesis\n\nReport scores per stratum alongside overall scores to reveal where the agent actually struggles.\n\n### Context Engineering Evaluation\n\n**Validate Context Strategies Systematically**\n\nRun agents with different context strategies on the same test set and compare quality scores, token usage, and efficiency metrics. This isolates the effect of context engineering from other variables and prevents anecdote-driven decisions.\n\n**Run Degradation Tests**\n\nTest how context degradation affects performance by running agents at different context sizes. Identify performance cliffs where context becomes problematic and establish safe operating limits. Feed these limits back into context management strategies.\n\n### Continuous Evaluation\n\n**Build Automated Evaluation Pipelines**\n\nIntegrate evaluation into the development workflow so evaluations run automatically on agent changes. Track results over time, compare versions, and block deployments that regress on key metrics.\n\n**Monitor Production Quality**\n\nSample production interactions and evaluate them continuously. Set alerts for quality drops below warning (0.85 pass rate) and critical (0.70 pass rate) thresholds. Maintain dashboards showing trend analysis over time windows to detect gradual degradation.\n\n## Practical Guidance\n\n### Building Evaluation Frameworks\n\nFollow this sequence to build an evaluation framework, because skipping early steps leads to measurements that do not reflect real quality:\n\n1. Define quality dimensions relevant to the use case before writing any evaluation code, because dimensions chosen later tend to reflect what is easy to measure rather than what matters.\n2. Create rubrics with clear, descriptive level definitions so evaluators (human or LLM) produce consistent scores.\n3. Build test sets from real usage patterns and edge cases, stratified by complexity, with at least 50 cases for reliable signal.\n4. Implement automated evaluation pipelines that run on every significant change.\n5. Establish baseline metrics before making changes so improvements can be measured against a known reference.\n6. Run evaluations on all significant changes and compare against the baseline.\n7. Track metrics over time for trend analysis because gradual degradation is harder to notice than sudden drops.\n8. Supplement automated evaluation with human review on a regular cadence.\n9. Separate deterministic validation failures from quality judgments so invalid artifacts cannot be laundered by a favorable LLM score.\n\n### Avoiding Evaluation Pitfalls\n\nGuard against these common failures that undermine evaluation reliability:\n\n- **Overfitting to specific paths**: Evaluate outcomes, not specific steps, because agents find novel valid paths.\n- **Ignoring edge cases**: Include diverse test scenarios covering the full complexity spectrum.\n- **Single-metric obsession**: Use multi-dimensional rubrics because a single score hides dimension-specific failures.\n- **Neglecting context effects**: Test with realistic context sizes and histories rather than clean-room conditions.\n- **Skipping human evaluation**: Automated evaluation misses subtle issues that humans catch reliably.\n\n## Examples\n\n**Example 1: Simple Evaluation**\n```python\ndef evaluate_agent_response(response, expected):\n rubric = load_rubric()\n scores = {}\n for dimension, config in rubric.items():\n scores[dimension] = assess_dimension(response, expected, dimension)\n overall = weighted_average(scores, config[\"weights\"])\n return {\"passed\": overall >= 0.7, \"scores\": scores}\n```\n\n**Example 2: Test Set Structure**\n\nTest sets should span multiple complexity levels to ensure comprehensive evaluation:\n\n```python\ntest_set = [\n {\n \"name\": \"simple_lookup\",\n \"input\": \"What is the capital of France?\",\n \"expected\": {\"type\": \"fact\", \"answer\": \"Paris\"},\n \"complexity\": \"simple\",\n \"description\": \"Single tool call, factual lookup\"\n },\n {\n \"name\": \"medium_query\",\n \"input\": \"Compare the revenue of Apple and Microsoft last quarter\",\n \"complexity\": \"medium\",\n \"description\": \"Multiple tool calls, comparison logic\"\n },\n {\n \"name\": \"multi_step_reasoning\",\n \"input\": \"Analyze sales data from Q1-Q4 and create a summary report with trends\",\n \"complexity\": \"complex\",\n \"description\": \"Many tool calls, aggregation, analysis\"\n },\n {\n \"name\": \"research_synthesis\",\n \"input\": \"Research emerging AI technologies, evaluate their potential impact, and recommend adoption strategy\",\n \"complexity\": \"very_complex\",\n \"description\": \"Extended interaction, deep reasoning, synthesis\"\n }\n]\n```\n\n**Example 3: Deterministic gate before model judgment**\n```python\ndef evaluate_pr_candidate(candidate):\n structure = run_validate_repo(candidate)\n if not structure.ok:\n return {\"passed\": False, \"reason\": \"deterministic validation failed\", \"details\": structure.errors}\n\n quality = run_rubric_eval(candidate)\n return {\"passed\": quality.overall >= 0.8, \"scores\": quality.dimensions}\n```\n\n**Example 4: Quality gate dimensions**\n```yaml\ngate:\n deterministic:\n - schema_valid\n - required_files_present\n - no_duplicate_ids\n quality:\n factual_accuracy: min 0.85\n completeness: min 0.80\n source_traceability: min 0.90\n```\n\n## Guidelines\n\n1. Use multi-dimensional rubrics, not single metrics\n2. Evaluate outcomes, not specific execution paths\n3. Cover complexity levels from simple to complex\n4. Test with realistic context sizes and histories\n5. Run evaluations continuously, not just before release\n6. Supplement LLM evaluation with human review\n7. Track metrics over time for trend detection\n8. Set clear pass/fail thresholds based on use case\n\n## Gotchas\n\n1. **Overfitting evals to specific code paths**: Tests pass but the agent fails on slight input variations. Write eval criteria against outcomes and semantics, not surface patterns, and rotate test inputs periodically.\n2. **LLM-judge self-enhancement bias**: Models rate their own outputs higher than independent judges do. Use a different model family as the evaluation judge than the model being evaluated.\n3. **Test set contamination**: Eval examples leak into training data or prompt templates, inflating scores. Keep eval sets versioned and separate from any data used in prompts or fine-tuning.\n4. **Metric gaming**: Optimizing for the metric rather than actual quality produces agents that score well but disappoint users. Cross-validate automated metrics against human judgments regularly.\n5. **Single-dimension scoring**: One aggregate number hides critical failures in specific dimensions. Always report per-dimension scores alongside the overall score, and fail the eval if any single dimension falls below its minimum threshold.\n6. **Eval set too small**: Fewer than 50 examples produces unreliable signal with high variance between runs. Scale the eval set to at least 50 cases and report confidence intervals.\n7. **Not stratifying by difficulty**: Easy examples inflate overall scores, masking failures on hard cases. Report scores per complexity stratum and weight the overall score to prevent easy-case dominance.\n8. **Treating eval as one-time**: Evaluation must be continuous, not a launch gate. Agent quality drifts as models update, tools change, and usage patterns evolve. Run evals on every change and on a regular production cadence.\n\n## Integration\n\nThis skill owns outcome measurement and quality gates. Adjacent skills own specialized evaluator design and control-loop governance:\n\n- `advanced-evaluation`: LLM-as-judge prompt design, pairwise comparison, calibration, and bias mitigation.\n- `harness-engineering`: locked evaluators, editable surfaces, rollback, and human approval boundaries.\n- `context-degradation`: detecting and measuring degradation patterns.\n- `context-optimization`: measuring token, cost, latency, and quality effects of optimizations.\n- `multi-agent-patterns`: evaluating coordination quality and parallelization trade-offs.\n- `tool-design`: evaluating tool selection and recovery effectiveness.\n- `memory-systems`: evaluating memory retrieval and retention quality.\n\n## References\n\nInternal reference:\n- [Metrics Reference](./references/metrics.md) - Read when: designing specific evaluation metrics, choosing scoring scales, or implementing weighted rubric calculations\n\nInternal skills:\n- All other skills connect to evaluation for quality measurement\n\nExternal resources:\n- LLM evaluation benchmarks - Read when: selecting or building benchmark suites for agent comparison\n- Agent evaluation research papers - Read when: adopting new evaluation methodologies or validating current approach\n- Production monitoring practices - Read when: setting up alerting, dashboards, or sampling strategies for live systems\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-20\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 1.2.0\n","skills/evaluation/references/metrics.md":"# Evaluation Reference: Metrics and Implementation\n\nThis document provides implementation details for evaluation metrics and evaluation systems.\n\n## Core Metric Definitions\n\n### Factual Accuracy\n\nFactual accuracy measures whether claims in agent output match ground truth.\n\n```\nExcellent (1.0): All claims verified against ground truth, no errors\nGood (0.8): Minor errors that do not affect main conclusions\nAcceptable (0.6): Major claims correct, minor inaccuracies present\nPoor (0.3): Significant factual errors in key claims\nFailed (0.0): Fundamental factual errors that invalidate output\n```\n\nCalculation approach:\n- Extract claims from output\n- Verify each claim against ground truth\n- Weight claims by importance (major claims more weight)\n- Calculate weighted average of claim accuracy\n\n### Completeness\n\nCompleteness measures whether output covers all requested aspects.\n\n```\nExcellent (1.0): All requested aspects thoroughly covered\nGood (0.8): Most aspects covered with minor gaps\nAcceptable (0.6): Key aspects covered, some gaps\nPoor (0.3): Major aspects missing from output\nFailed (0.0): Fundamental aspects not addressed\n```\n\n### Citation Accuracy\n\nCitation accuracy measures whether cited sources match claimed sources.\n\n```\nExcellent (1.0): All citations accurate and complete\nGood (0.8): Minor citation formatting issues\nAcceptable (0.6): Major citations accurate\nPoor (0.3): Significant citation problems\nFailed (0.0): Citations missing or completely incorrect\n```\n\n### Source Quality\n\nSource quality measures whether appropriate primary sources were used.\n\n```\nExcellent (1.0): Primary authoritative sources\nGood (0.8): Mostly primary sources with some secondary\nAcceptable (0.6): Mix of primary and secondary sources\nPoor (0.3): Mostly secondary or unreliable sources\nFailed (0.0): No credible sources cited\n```\n\n### Tool Efficiency\n\nTool efficiency measures whether the agent used appropriate tools a reasonable number of times.\n\n```\nExcellent (1.0): Optimal tool selection and call count\nGood (0.8): Good tool selection with minor inefficiencies\nAcceptable (0.6): Appropriate tools with some redundancy\nPoor (0.3): Wrong tools or excessive call counts\nFailed (0.0): Severe tool misuse or extremely excessive calls\n```\n\n## Rubric Implementation\n\n```python\nEVALUATION_DIMENSIONS = {\n \"factual_accuracy\": {\n \"weight\": 0.30,\n \"description\": \"Claims match ground truth\",\n \"levels\": {\n \"excellent\": 1.0,\n \"good\": 0.8,\n \"acceptable\": 0.6,\n \"poor\": 0.3,\n \"failed\": 0.0\n }\n },\n \"completeness\": {\n \"weight\": 0.25,\n \"description\": \"All requested aspects covered\",\n \"levels\": {\n \"excellent\": 1.0,\n \"good\": 0.8,\n \"acceptable\": 0.6,\n \"poor\": 0.3,\n \"failed\": 0.0\n }\n },\n \"citation_accuracy\": {\n \"weight\": 0.15,\n \"description\": \"Citations match sources\",\n \"levels\": {\n \"excellent\": 1.0,\n \"good\": 0.8,\n \"acceptable\": 0.6,\n \"poor\": 0.3,\n \"failed\": 0.0\n }\n },\n \"source_quality\": {\n \"weight\": 0.10,\n \"description\": \"Appropriate primary sources used\",\n \"levels\": {\n \"excellent\": 1.0,\n \"good\": 0.8,\n \"acceptable\": 0.6,\n \"poor\": 0.3,\n \"failed\": 0.0\n }\n },\n \"tool_efficiency\": {\n \"weight\": 0.20,\n \"description\": \"Right tools used reasonably\",\n \"levels\": {\n \"excellent\": 1.0,\n \"good\": 0.8,\n \"acceptable\": 0.6,\n \"poor\": 0.3,\n \"failed\": 0.0\n }\n }\n}\n\ndef calculate_overall_score(dimension_scores, rubric):\n \"\"\"Calculate weighted overall score from dimension scores.\"\"\"\n total_weight = 0\n weighted_sum = 0\n \n for dimension, score in dimension_scores.items():\n if dimension in rubric:\n weight = rubric[dimension][\"weight\"]\n weighted_sum += score * weight\n total_weight += weight\n \n return weighted_sum / total_weight if total_weight > 0 else 0\n```\n\n## Test Set Management\n\n```python\nclass TestSet:\n def __init__(self, name):\n self.name = name\n self.tests = []\n self.tags = {}\n \n def add_test(self, test_case):\n \"\"\"Add test case to test set.\"\"\"\n self.tests.append(test_case)\n \n # Index by tags\n for tag in test_case.get(\"tags\", []):\n if tag not in self.tags:\n self.tags[tag] = []\n self.tags[tag].append(len(self.tests) - 1)\n \n def filter(self, **criteria):\n \"\"\"Filter tests by criteria.\"\"\"\n filtered = []\n for test in self.tests:\n match = True\n for key, value in criteria.items():\n if test.get(key) != value:\n match = False\n break\n if match:\n filtered.append(test)\n return filtered\n \n def get_complexity_distribution(self):\n \"\"\"Get distribution of tests by complexity.\"\"\"\n distribution = {}\n for test in self.tests:\n complexity = test.get(\"complexity\", \"medium\")\n distribution[complexity] = distribution.get(complexity, 0) + 1\n return distribution\n```\n\n## Evaluation Runner\n\n```python\nclass EvaluationRunner:\n def __init__(self, test_set, rubric, agent):\n self.test_set = test_set\n self.rubric = rubric\n self.agent = agent\n self.results = []\n \n def run_all(self, verbose=False):\n \"\"\"Run evaluation on all tests.\"\"\"\n self.results = []\n \n for i, test in enumerate(self.test_set.tests):\n if verbose:\n print(f\"Running test {i+1}/{len(self.test_set.tests)}\")\n \n result = self.run_test(test)\n self.results.append(result)\n \n return self.summarize()\n \n def run_test(self, test):\n \"\"\"Run single evaluation test.\"\"\"\n # Get agent output\n output = self.agent.run(test[\"input\"])\n \n # Evaluate\n evaluation = self.evaluate_output(output, test)\n \n return {\n \"test\": test,\n \"output\": output,\n \"evaluation\": evaluation\n }\n \n def evaluate_output(self, output, test):\n \"\"\"Evaluate agent output against test.\"\"\"\n ground_truth = test.get(\"expected\", {})\n \n dimension_scores = {}\n for dimension, config in self.rubric.items():\n score = self.evaluate_dimension(\n output, ground_truth, dimension, config\n )\n dimension_scores[dimension] = score\n \n overall = calculate_overall_score(dimension_scores, self.rubric)\n \n return {\n \"overall_score\": overall,\n \"dimension_scores\": dimension_scores,\n \"passed\": overall >= 0.7\n }\n \n def summarize(self):\n \"\"\"Summarize evaluation results.\"\"\"\n if not self.results:\n return {\"error\": \"No results\"}\n \n passed = sum(1 for r in self.results if r[\"evaluation\"][\"passed\"])\n \n dimension_totals = {}\n for dimension in self.rubric.keys():\n dimension_totals[dimension] = {\n \"total\": 0,\n \"count\": 0\n }\n \n for result in self.results:\n for dimension, score in result[\"evaluation\"][\"dimension_scores\"].items():\n if dimension in dimension_totals:\n dimension_totals[dimension][\"total\"] += score\n dimension_totals[dimension][\"count\"] += 1\n \n dimension_averages = {}\n for dimension, data in dimension_totals.items():\n if data[\"count\"] > 0:\n dimension_averages[dimension] = data[\"total\"] / data[\"count\"]\n \n return {\n \"total_tests\": len(self.results),\n \"passed\": passed,\n \"failed\": len(self.results) - passed,\n \"pass_rate\": passed / len(self.results) if self.results else 0,\n \"dimension_averages\": dimension_averages,\n \"failures\": [\n r for r in self.results \n if not r[\"evaluation\"][\"passed\"]\n ]\n }\n```\n\n## Production Monitoring\n\n```python\nclass ProductionMonitor:\n def __init__(self, sample_rate=0.01):\n self.sample_rate = sample_rate\n self.samples = []\n self.alert_thresholds = {\n \"pass_rate_warning\": 0.85,\n \"pass_rate_critical\": 0.70\n }\n \n def sample_and_evaluate(self, query, output):\n \"\"\"Sample production interaction for evaluation.\"\"\"\n if random.random() > self.sample_rate:\n return None\n \n evaluation = evaluate_output(output, {}, EVALUATION_RUBRIC)\n \n sample = {\n \"query\": query[:200],\n \"output_preview\": output[:200],\n \"score\": evaluation[\"overall_score\"],\n \"passed\": evaluation[\"passed\"],\n \"timestamp\": current_timestamp()\n }\n \n self.samples.append(sample)\n return sample\n \n def get_metrics(self):\n \"\"\"Calculate current metrics from samples.\"\"\"\n if not self.samples:\n return {\"status\": \"insufficient_data\"}\n \n passed = sum(1 for s in self.samples if s[\"passed\"])\n pass_rate = passed / len(self.samples)\n \n avg_score = sum(s[\"score\"] for s in self.samples) / len(self.samples)\n \n return {\n \"sample_count\": len(self.samples),\n \"pass_rate\": pass_rate,\n \"average_score\": avg_score,\n \"status\": self._get_status(pass_rate)\n }\n \n def _get_status(self, pass_rate):\n \"\"\"Get status based on pass rate.\"\"\"\n if pass_rate < self.alert_thresholds[\"pass_rate_critical\"]:\n return \"critical\"\n elif pass_rate < self.alert_thresholds[\"pass_rate_warning\"]:\n return \"warning\"\n else:\n return \"healthy\"\n```\n\n","skills/filesystem-context/SKILL.md":"---\nname: filesystem-context\ndescription: \"This skill should be used when agent work needs file-backed context: durable scratchpads, tool-output offloading, just-in-time discovery, cross-agent handoff files, filesystem memory, or cleanup policies for context stored outside the prompt.\"\n---\n\n# Filesystem-Based Context Engineering\n\nUse the filesystem as the primary overflow layer for agent context because context windows are limited while tasks often require more information than fits in a single window. Files let agents store, retrieve, and update an effectively unlimited amount of context through a single interface.\n\nPrefer dynamic context discovery -- pulling relevant context on demand -- over static inclusion, because static context consumes tokens regardless of relevance and crowds out space for task-specific information.\n\n## When to Activate\n\nActivate this skill when:\n- Tool outputs are bloating the context window\n- Agents need to persist state across long trajectories\n- Sub-agents must share information without direct message passing\n- Tasks require more context than fits in the window\n- Building agents that learn and update their own instructions\n- Implementing scratch pads for intermediate results\n- Terminal outputs or logs need to be accessible to agents\n\nDo not activate this skill for adjacent work owned by other skills:\n- Semantic cross-session memory, entity tracking, or temporal knowledge graphs: `memory-systems`.\n- Conversation summarization, compaction, or durable handoff wording: `context-compression`.\n- Token-efficiency tactics that do not require file-backed storage: `context-optimization`.\n- Multi-agent topology or handoff protocol design: `multi-agent-patterns`.\n- Model edits that evict content from the live window itself: `self-managed-context`.\n\n## Core Concepts\n\nDiagnose context failures against these four modes, because each requires a different filesystem remedy:\n\n1. **Missing context** -- needed information is absent from the total available context. Fix by persisting tool outputs and intermediate results to files so nothing is lost.\n2. **Under-retrieved context** -- retrieved content fails to encapsulate what the agent needs. Fix by structuring files for targeted retrieval (grep-friendly formats, clear section headers).\n3. **Over-retrieved context** -- retrieved content far exceeds what is needed, wasting tokens and degrading attention. Fix by offloading bulk content to files and returning compact references.\n4. **Buried context** -- niche information is hidden across many files. Fix by combining glob and grep for structural search alongside semantic search for conceptual queries.\n\nUse the filesystem as the persistent layer that addresses all four: write once, store durably, retrieve selectively.\n\n## Detailed Topics\n\n### The Static vs Dynamic Context Trade-off\n\nTreat static context (system instructions, tool definitions, critical rules) as expensive real estate -- it consumes tokens on every turn regardless of relevance. As agents accumulate capabilities, static context grows and crowds out dynamic information.\n\nUse dynamic context discovery instead: include only minimal static pointers (names, one-line descriptions, file paths) and load full content with search tools when relevant. This is more token-efficient and often improves response quality by reducing contradictory or irrelevant information in the window.\n\nAccept the trade-off: dynamic discovery requires the model to recognize when it needs more context. Current frontier models handle this well, but less capable models may fail to trigger loads. When in doubt, err toward including critical safety or correctness constraints statically.\n\n### Pattern 1: Filesystem as Scratch Pad\n\nRedirect large tool outputs to files instead of returning them directly to context, because a single web search or database query can dump thousands of tokens into message history where they persist for the entire conversation.\n\nWrite the output to a scratch file, extract a compact summary, and return a file reference. The agent then uses targeted retrieval (grep for patterns, read with line ranges) to access only what it needs.\n\n```python\ndef handle_tool_output(output: str, threshold: int = 2000) -> str:\n if len(output) < threshold:\n return output\n\n file_path = f\"scratch/{tool_name}_{timestamp}.txt\"\n write_file(file_path, output)\n\n key_summary = extract_summary(output, max_tokens=200)\n return f\"[Output written to {file_path}. Summary: {key_summary}]\"\n```\n\nUse grep to search the offloaded file and read_file with line ranges to retrieve targeted sections, because this preserves full output for later reference while keeping only ~100 tokens in the active context.\n\n### Pattern 2: Plan Persistence\n\nWrite plans to the filesystem because long-horizon tasks lose coherence when plans fall out of attention or get summarized away. The agent re-reads its plan at any point, restoring awareness of the objective and progress.\n\nStore plans in structured format so they are both human-readable and machine-parseable:\n```yaml\n# scratch/current_plan.yaml\nobjective: \"Refactor authentication module\"\nstatus: in_progress\nsteps:\n - id: 1\n description: \"Audit current auth endpoints\"\n status: completed\n - id: 2\n description: \"Design new token validation flow\"\n status: in_progress\n - id: 3\n description: \"Implement and test changes\"\n status: pending\n```\n\nRe-read the plan at the start of each turn or after any context refresh to re-orient, because this acts as \"manipulating attention through recitation.\"\n\n### Pattern 3: Sub-Agent Communication via Filesystem\n\nRoute sub-agent findings through the filesystem instead of message passing, because multi-hop message chains degrade information through summarization at each hop (\"game of telephone\").\n\nHave each sub-agent write directly to its own workspace directory. The coordinator reads these files directly, preserving full fidelity:\n```\nworkspace/\n agents/\n research_agent/\n findings.md\n sources.jsonl\n code_agent/\n changes.md\n test_results.txt\n coordinator/\n synthesis.md\n```\n\nEnforce per-agent directory isolation to prevent write conflicts and maintain clear ownership of each output artifact.\n\n### Pattern 4: Dynamic Skill Loading\n\nStore skills as files and include only skill names with brief descriptions in static context, because stuffing all instructions into the system prompt wastes tokens and can confuse the model with contradictory guidance.\n\n```\nAvailable skills (load with read_file when relevant):\n- database-optimization: Query tuning and indexing strategies\n- api-design: REST/GraphQL best practices\n- testing-strategies: Unit, integration, and e2e testing patterns\n```\n\nLoad the full skill file (e.g., `skills/database-optimization/SKILL.md`) only when the current task requires it. This converts O(n) static token cost into O(1) per task.\n\n### Pattern 5: Terminal and Log Persistence\n\nPersist terminal output to files automatically and use grep for selective retrieval, because terminal output from long-running processes accumulates rapidly and manual copy-paste is error-prone.\n\n```\nterminals/\n 1.txt # Terminal session 1 output\n 2.txt # Terminal session 2 output\n```\n\nQuery with targeted grep (`grep -A 5 \"error\" terminals/1.txt`) instead of loading entire terminal histories into context.\n\n### Pattern 6: Learning Through Self-Modification\n\nHave agents write learned preferences and patterns to their own instruction files so subsequent sessions load this context automatically, instead of requiring manual system prompt updates.\n\n```python\ndef remember_preference(key: str, value: str):\n preferences_file = \"agent/user_preferences.yaml\"\n prefs = load_yaml(preferences_file)\n prefs[key] = value\n write_yaml(preferences_file, prefs)\n```\n\nGuard this pattern with validation because self-modification can accumulate incorrect or contradictory instructions over time. Treat it as experimental -- review persisted preferences periodically.\n\n### Filesystem Search Techniques\n\nCombine `ls`/`list_dir`, `glob`, `grep`, and `read_file` with line ranges for context discovery, because models are specifically trained on filesystem traversal and this combination often outperforms semantic search for technical content where structural patterns are clear.\n\n- `ls` / `list_dir`: Discover directory structure\n- `glob`: Find files matching patterns (e.g., `**/*.py`)\n- `grep`: Search file contents, returns matching lines with context\n- `read_file` with ranges: Read specific sections without loading entire files\n\nUse filesystem search for structural and exact-match queries, and semantic search for conceptual queries. Combine both for comprehensive discovery.\n\n## Practical Guidance\n\n### When to Use Filesystem Context\n\nApply filesystem patterns when the situation matches these criteria, because they add I/O overhead that is only justified by token savings or persistence needs:\n\n**Use when:**\n- Tool outputs exceed ~2000 tokens\n- Tasks span multiple conversation turns\n- Multiple agents need shared state\n- Skills or instructions exceed comfortable system prompt size\n- Logs or terminal output need selective querying\n\n**Avoid when:**\n- Tasks complete in single turns (overhead not justified)\n- Context fits comfortably in window (no problem to solve)\n- Latency is critical (file I/O adds measurable delay)\n- Model lacks filesystem tool capabilities\n\n### File Organization\n\nStructure files for agent discoverability, because agents navigate by listing and reading directory names:\n```\nproject/\n scratch/ # Temporary working files\n tool_outputs/ # Large tool results\n plans/ # Active plans and checklists\n memory/ # Persistent learned information\n preferences.yaml # User preferences\n patterns.md # Learned patterns\n skills/ # Loadable skill definitions\n agents/ # Sub-agent workspaces\n```\n\nUse consistent naming conventions and include timestamps or IDs in scratch files for disambiguation.\n\nFor autonomous research loops, store raw retrieved evidence under the run that consumed it, for example `researcher/runs//sources/evidence/raw/`. Do not leave raw research dumps in the repository root; root-level artifacts become hard to audit and easy to cite without provenance.\n\n### Token Accounting\n\nMeasure where tokens originate before and after applying filesystem patterns, because optimizing without measurement leads to wasted effort:\n- Track static vs dynamic context ratio\n- Monitor tool output sizes before and after offloading\n- Measure how often dynamically-loaded context is actually used\n\n## Examples\n\n**Example 1: Tool Output Offloading**\n```\nInput: Web search returns 8000 tokens\nBefore: 8000 tokens added to message history\nAfter:\n - Write to scratch/search_results_001.txt\n - Return: \"[Results in scratch/search_results_001.txt. Key finding: API rate limit is 1000 req/min]\"\n - Agent greps file when needing specific details\nResult: ~100 tokens in context, 8000 tokens accessible on demand\n```\n\n**Example 2: Dynamic Skill Loading**\n```\nInput: User asks about database indexing\nStatic context: \"database-optimization: Query tuning and indexing\"\nAgent action: read_file(\"skills/database-optimization/SKILL.md\")\nResult: Full skill loaded only when relevant\n```\n\n**Example 3: Chat History as File Reference**\n```\nTrigger: Context window limit reached, summarization required\nAction:\n 1. Write full history to history/session_001.txt\n 2. Generate summary for new context window\n 3. Include reference: \"Full history in history/session_001.txt\"\nResult: Agent can search history file to recover details lost in summarization\n```\n\n## Guidelines\n\n1. Write large outputs to files; return summaries and references to context\n2. Store plans and state in structured files for re-reading\n3. Use sub-agent file workspaces instead of message chains\n4. Load skills dynamically rather than stuffing all into system prompt\n5. Persist terminal and log output as searchable files\n6. Combine grep/glob with semantic search for comprehensive discovery\n7. Organize files for agent discoverability with clear naming\n8. Measure token savings to validate filesystem patterns are effective\n9. Implement cleanup for scratch files to prevent unbounded growth\n10. Guard self-modification patterns with validation\n11. Keep raw evidence next to the run, evaluation, and proposal that used it\n\n## Gotchas\n\n1. **Scratch directory unbounded growth**: Agents create temp files without cleanup, eventually consuming disk and making directory listings noisy. Implement a retention policy (age-based or count-based) and run cleanup at session boundaries.\n2. **Race conditions in multi-agent file access**: Concurrent writes to the same file corrupt state silently. Enforce per-agent directory isolation or use append-only files with agent-prefixed entries.\n3. **Stale file references after moves/renames**: Agents hold paths from prior turns that no longer exist after refactors or file reorganization. Always verify file existence before reading a cached path; re-discover with glob if the check fails.\n4. **Glob pattern false matches**: Overly broad patterns (e.g., `**/*`) pull irrelevant files into context, wasting tokens and confusing the model. Scope globs to specific directories and extensions.\n5. **File size assumptions**: Reading a file without checking size can dump 100K+ tokens into context in a single tool call. Check file size before reading; use line-range reads for large files.\n6. **Missing file existence checks**: Agents assume files exist from prior turns, but they may have been deleted or moved. Always guard reads with existence checks and handle missing-file errors gracefully.\n7. **Scratch pad format drift**: Unstructured scratch pads become unparseable after many writes because format conventions erode over successive appends. Define and enforce a schema (YAML, JSON, or structured markdown) from the first write.\n8. **Hardcoded absolute paths**: Break when repositories are checked out at different locations or when running in containers. Use relative paths from the project root or resolve paths dynamically.\n\n## Integration\n\nThis skill owns file-backed context storage and retrieval. Adjacent skills own semantic memory, summarization, and topology:\n\n- `context-optimization`: filesystem offloading is one implementation of observation masking when full outputs remain retrievable.\n- `memory-systems`: use when file-backed notes are no longer enough and semantic, entity, or temporal retrieval is required.\n- `multi-agent-patterns`: sub-agent file workspaces enable context isolation and direct handoff.\n- `context-compression`: file references can anchor summaries and preserve details omitted from compressed context.\n- `tool-design`: tools should return file references for large outputs and expose safe read/search operations.\n- `self-managed-context`: offloading leaves the original in the live window; evicting it requires the model to edit its own context.\n\n## References\n\nInternal reference:\n- [Implementation Patterns](./references/implementation-patterns.md) - Read when: implementing scratch pad, plan persistence, or tool output offloading and need concrete code beyond the inline examples\n\nRelated skills in this collection:\n- context-optimization - Read when: applying token reduction techniques alongside filesystem offloading\n- memory-systems - Read when: building persistent storage that outlasts a single session\n- multi-agent-patterns - Read when: designing agent coordination with shared file workspaces\n\nExternal resources:\n- LangChain Deep Agents — Read when: implementing filesystem-based context patterns in LangChain/LangGraph pipelines\n- Cursor context discovery — Read when: studying how production IDEs implement dynamic context loading\n- Anthropic Agent Skills specification — Read when: building skills that leverage filesystem progressive disclosure\n\n---\n\n## Skill Metadata\n\n**Created**: 2026-01-07\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 1.2.0\n","skills/filesystem-context/references/implementation-patterns.md":"# Filesystem Context Implementation Patterns\n\nThis reference provides detailed implementation patterns for filesystem-based context engineering.\n\n## Pattern Catalog\n\n### 1. Scratch Pad Manager\n\nA centralized manager for handling large tool outputs and intermediate results.\n\n```python\nimport os\nimport json\nfrom datetime import datetime\nfrom pathlib import Path\n\nclass ScratchPadManager:\n \"\"\"Manages temporary file storage for agent context offloading.\"\"\"\n \n def __init__(self, base_path: str = \"scratch\", token_threshold: int = 2000):\n self.base_path = Path(base_path)\n self.base_path.mkdir(parents=True, exist_ok=True)\n self.token_threshold = token_threshold\n self.manifest = {}\n \n def should_offload(self, content: str) -> bool:\n \"\"\"Determine if content exceeds threshold for offloading.\"\"\"\n # Rough token estimate: 1 token ≈ 4 characters\n estimated_tokens = len(content) // 4\n return estimated_tokens > self.token_threshold\n \n def offload(self, content: str, source: str, summary: str = None) -> dict:\n \"\"\"Write content to file, return reference.\"\"\"\n timestamp = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n filename = f\"{source}_{timestamp}.txt\"\n file_path = self.base_path / filename\n \n file_path.write_text(content)\n \n reference = {\n \"type\": \"file_reference\",\n \"path\": str(file_path),\n \"source\": source,\n \"timestamp\": timestamp,\n \"size_chars\": len(content),\n \"summary\": summary or self._extract_summary(content)\n }\n \n self.manifest[filename] = reference\n return reference\n \n def _extract_summary(self, content: str, max_chars: int = 500) -> str:\n \"\"\"Extract first meaningful content as summary.\"\"\"\n lines = content.strip().split('\\n')\n summary_lines = []\n char_count = 0\n \n for line in lines:\n if char_count + len(line) > max_chars:\n break\n summary_lines.append(line)\n char_count += len(line)\n \n return '\\n'.join(summary_lines)\n \n def cleanup(self, max_age_hours: int = 24):\n \"\"\"Remove scratch files older than threshold.\"\"\"\n cutoff = datetime.now().timestamp() - (max_age_hours * 3600)\n \n for file_path in self.base_path.glob(\"*.txt\"):\n if file_path.stat().st_mtime < cutoff:\n file_path.unlink()\n if file_path.name in self.manifest:\n del self.manifest[file_path.name]\n```\n\n### 2. Plan Persistence\n\nStructured plan storage with progress tracking.\n\n```python\nimport yaml\nfrom dataclasses import dataclass, field, asdict\nfrom enum import Enum\nfrom typing import List, Optional\n\nclass StepStatus(Enum):\n PENDING = \"pending\"\n IN_PROGRESS = \"in_progress\"\n COMPLETED = \"completed\"\n BLOCKED = \"blocked\"\n CANCELLED = \"cancelled\"\n\n@dataclass\nclass PlanStep:\n id: int\n description: str\n status: StepStatus = StepStatus.PENDING\n notes: Optional[str] = None\n\n@dataclass\nclass AgentPlan:\n objective: str\n steps: List[PlanStep] = field(default_factory=list)\n status: str = \"in_progress\"\n \n def save(self, path: str = \"scratch/current_plan.yaml\"):\n \"\"\"Persist plan to filesystem.\"\"\"\n data = {\n \"objective\": self.objective,\n \"status\": self.status,\n \"steps\": [\n {\n \"id\": s.id,\n \"description\": s.description,\n \"status\": s.status.value,\n \"notes\": s.notes\n }\n for s in self.steps\n ]\n }\n with open(path, 'w') as f:\n yaml.dump(data, f, default_flow_style=False)\n \n @classmethod\n def load(cls, path: str = \"scratch/current_plan.yaml\") -> \"AgentPlan\":\n \"\"\"Load plan from filesystem.\"\"\"\n with open(path, 'r') as f:\n data = yaml.safe_load(f)\n \n plan = cls(objective=data[\"objective\"], status=data.get(\"status\", \"in_progress\"))\n for step_data in data.get(\"steps\", []):\n plan.steps.append(PlanStep(\n id=step_data[\"id\"],\n description=step_data[\"description\"],\n status=StepStatus(step_data[\"status\"]),\n notes=step_data.get(\"notes\")\n ))\n return plan\n \n def current_step(self) -> Optional[PlanStep]:\n \"\"\"Get the first non-completed step.\"\"\"\n for step in self.steps:\n if step.status != StepStatus.COMPLETED:\n return step\n return None\n \n def complete_step(self, step_id: int, notes: str = None):\n \"\"\"Mark step as completed.\"\"\"\n for step in self.steps:\n if step.id == step_id:\n step.status = StepStatus.COMPLETED\n if notes:\n step.notes = notes\n break\n```\n\n### 3. Sub-Agent Workspace\n\nFile-based communication between agents.\n\n```python\nfrom pathlib import Path\nfrom datetime import datetime\nimport json\n\nclass AgentWorkspace:\n \"\"\"Manages file-based workspace for an agent.\"\"\"\n \n def __init__(self, agent_id: str, base_path: str = \"workspace/agents\"):\n self.agent_id = agent_id\n self.path = Path(base_path) / agent_id\n self.path.mkdir(parents=True, exist_ok=True)\n \n # Standard files\n self.findings_file = self.path / \"findings.md\"\n self.status_file = self.path / \"status.json\"\n self.log_file = self.path / \"activity.log\"\n \n def write_finding(self, content: str, append: bool = True):\n \"\"\"Write or append a finding.\"\"\"\n mode = 'a' if append else 'w'\n with open(self.findings_file, mode) as f:\n if append:\n f.write(f\"\\n---\\n## {datetime.now().isoformat()}\\n\\n\")\n f.write(content)\n \n def update_status(self, status: str, progress: float = None, details: dict = None):\n \"\"\"Update agent status for coordinator visibility.\"\"\"\n status_data = {\n \"agent_id\": self.agent_id,\n \"status\": status,\n \"updated_at\": datetime.now().isoformat(),\n \"progress\": progress,\n \"details\": details or {}\n }\n self.status_file.write_text(json.dumps(status_data, indent=2))\n \n def log(self, message: str):\n \"\"\"Append to activity log.\"\"\"\n with open(self.log_file, 'a') as f:\n f.write(f\"[{datetime.now().isoformat()}] {message}\\n\")\n \n def read_peer_findings(self, peer_id: str) -> str:\n \"\"\"Read findings from another agent's workspace.\"\"\"\n peer_path = self.path.parent / peer_id / \"findings.md\"\n if peer_path.exists():\n return peer_path.read_text()\n return \"\"\n\n\nclass CoordinatorWorkspace:\n \"\"\"Coordinator that reads from sub-agent workspaces.\"\"\"\n \n def __init__(self, base_path: str = \"workspace/agents\"):\n self.base_path = Path(base_path)\n \n def get_all_statuses(self) -> dict:\n \"\"\"Collect status from all sub-agents.\"\"\"\n statuses = {}\n for agent_dir in self.base_path.iterdir():\n if agent_dir.is_dir():\n status_file = agent_dir / \"status.json\"\n if status_file.exists():\n statuses[agent_dir.name] = json.loads(status_file.read_text())\n return statuses\n \n def aggregate_findings(self) -> str:\n \"\"\"Combine all agent findings into synthesis.\"\"\"\n findings = []\n for agent_dir in self.base_path.iterdir():\n if agent_dir.is_dir():\n findings_file = agent_dir / \"findings.md\"\n if findings_file.exists():\n findings.append(f\"# {agent_dir.name}\\n\\n{findings_file.read_text()}\")\n return \"\\n\\n\".join(findings)\n```\n\n### 4. Dynamic Skill Loader\n\nLoad skill content on demand.\n\n```python\nfrom pathlib import Path\nfrom typing import List, Optional\nimport yaml\n\n@dataclass\nclass SkillMetadata:\n name: str\n description: str\n path: str\n triggers: List[str] = field(default_factory=list)\n\nclass SkillLoader:\n \"\"\"Manages dynamic loading of agent skills.\"\"\"\n \n def __init__(self, skills_path: str = \"skills\"):\n self.skills_path = Path(skills_path)\n self.skill_index = self._build_index()\n \n def _build_index(self) -> dict:\n \"\"\"Build index of available skills from SKILL.md frontmatter.\"\"\"\n index = {}\n for skill_dir in self.skills_path.iterdir():\n if skill_dir.is_dir():\n skill_file = skill_dir / \"SKILL.md\"\n if skill_file.exists():\n metadata = self._parse_frontmatter(skill_file)\n if metadata:\n index[metadata.name] = metadata\n return index\n \n def _parse_frontmatter(self, path: Path) -> Optional[SkillMetadata]:\n \"\"\"Extract YAML frontmatter from skill file.\"\"\"\n content = path.read_text()\n if content.startswith('---'):\n end = content.find('---', 3)\n if end > 0:\n frontmatter = yaml.safe_load(content[3:end])\n return SkillMetadata(\n name=frontmatter.get('name', path.parent.name),\n description=frontmatter.get('description', ''),\n path=str(path),\n triggers=frontmatter.get('triggers', [])\n )\n return None\n \n def get_static_context(self) -> str:\n \"\"\"Generate minimal static context listing available skills.\"\"\"\n lines = [\"Available skills (load with read_file when relevant):\"]\n for name, meta in self.skill_index.items():\n lines.append(f\"- {name}: {meta.description[:100]}\")\n return \"\\n\".join(lines)\n \n def load_skill(self, name: str) -> str:\n \"\"\"Load full skill content.\"\"\"\n if name in self.skill_index:\n return Path(self.skill_index[name].path).read_text()\n raise ValueError(f\"Unknown skill: {name}\")\n \n def find_relevant_skills(self, query: str) -> List[str]:\n \"\"\"Find skills that might be relevant to a query.\"\"\"\n query_lower = query.lower()\n relevant = []\n for name, meta in self.skill_index.items():\n if any(trigger in query_lower for trigger in meta.triggers):\n relevant.append(name)\n elif name.replace('-', ' ') in query_lower:\n relevant.append(name)\n return relevant\n```\n\n### 5. Terminal Output Persistence\n\nCapture and persist terminal sessions.\n\n```python\nimport subprocess\nfrom pathlib import Path\nfrom datetime import datetime\nimport re\n\nclass TerminalCapture:\n \"\"\"Captures and persists terminal output for agent access.\"\"\"\n \n def __init__(self, terminals_path: str = \"terminals\"):\n self.terminals_path = Path(terminals_path)\n self.terminals_path.mkdir(parents=True, exist_ok=True)\n self.session_counter = 0\n \n def run_command(self, command: str, capture: bool = True) -> dict:\n \"\"\"Run command and optionally capture output to file.\"\"\"\n self.session_counter += 1\n \n result = subprocess.run(\n command,\n shell=True,\n capture_output=True,\n text=True\n )\n \n output = {\n \"command\": command,\n \"exit_code\": result.returncode,\n \"stdout\": result.stdout,\n \"stderr\": result.stderr,\n \"timestamp\": datetime.now().isoformat()\n }\n \n if capture:\n output[\"file\"] = self._persist_output(output)\n \n return output\n \n def _persist_output(self, output: dict) -> str:\n \"\"\"Write output to terminal file.\"\"\"\n filename = f\"{self.session_counter}.txt\"\n file_path = self.terminals_path / filename\n \n content = f\"\"\"---\ncommand: {output['command']}\nexit_code: {output['exit_code']}\ntimestamp: {output['timestamp']}\n---\n\n=== STDOUT ===\n{output['stdout']}\n\n=== STDERR ===\n{output['stderr']}\n\"\"\"\n file_path.write_text(content)\n return str(file_path)\n \n def grep_terminals(self, pattern: str, context_lines: int = 3) -> List[dict]:\n \"\"\"Search all terminal outputs for pattern.\"\"\"\n matches = []\n regex = re.compile(pattern, re.IGNORECASE)\n \n for term_file in self.terminals_path.glob(\"*.txt\"):\n content = term_file.read_text()\n lines = content.split('\\n')\n \n for i, line in enumerate(lines):\n if regex.search(line):\n start = max(0, i - context_lines)\n end = min(len(lines), i + context_lines + 1)\n matches.append({\n \"file\": str(term_file),\n \"line_number\": i + 1,\n \"context\": '\\n'.join(lines[start:end])\n })\n \n return matches\n```\n\n### 6. Self-Modification Guard\n\nSafe pattern for agent self-learning.\n\n```python\nimport yaml\nfrom pathlib import Path\nfrom datetime import datetime\nfrom typing import Any\n\nclass PreferenceStore:\n \"\"\"Guarded storage for agent-learned preferences.\"\"\"\n \n MAX_ENTRIES = 100\n MAX_VALUE_LENGTH = 1000\n \n def __init__(self, path: str = \"agent/preferences.yaml\"):\n self.path = Path(path)\n self.path.parent.mkdir(parents=True, exist_ok=True)\n self.preferences = self._load()\n \n def _load(self) -> dict:\n \"\"\"Load preferences from file.\"\"\"\n if self.path.exists():\n return yaml.safe_load(self.path.read_text()) or {}\n return {}\n \n def _save(self):\n \"\"\"Persist preferences to file.\"\"\"\n self.path.write_text(yaml.dump(self.preferences, default_flow_style=False))\n \n def remember(self, key: str, value: Any, source: str = \"user\"):\n \"\"\"Store a preference with validation.\"\"\"\n # Validate key\n if not key or len(key) > 100:\n raise ValueError(\"Invalid key length\")\n \n # Validate value\n value_str = str(value)\n if len(value_str) > self.MAX_VALUE_LENGTH:\n raise ValueError(f\"Value exceeds max length of {self.MAX_VALUE_LENGTH}\")\n \n # Check entry limit\n if len(self.preferences) >= self.MAX_ENTRIES and key not in self.preferences:\n raise ValueError(f\"Max entries ({self.MAX_ENTRIES}) reached\")\n \n # Store with metadata\n self.preferences[key] = {\n \"value\": value,\n \"source\": source,\n \"updated_at\": datetime.now().isoformat()\n }\n self._save()\n \n def recall(self, key: str, default: Any = None) -> Any:\n \"\"\"Retrieve a preference.\"\"\"\n entry = self.preferences.get(key)\n if entry:\n return entry[\"value\"]\n return default\n \n def list_all(self) -> dict:\n \"\"\"Get all preferences for context injection.\"\"\"\n return {k: v[\"value\"] for k, v in self.preferences.items()}\n \n def forget(self, key: str):\n \"\"\"Remove a preference.\"\"\"\n if key in self.preferences:\n del self.preferences[key]\n self._save()\n```\n\n## Integration Example\n\nCombining patterns in an agent harness:\n\n```python\nclass FilesystemContextAgent:\n \"\"\"Agent with filesystem-based context management.\"\"\"\n \n def __init__(self):\n self.scratch = ScratchPadManager()\n self.skills = SkillLoader()\n self.preferences = PreferenceStore()\n self.workspace = AgentWorkspace(\"main_agent\")\n \n def handle_tool_output(self, tool_name: str, output: str) -> str:\n \"\"\"Process tool output, offloading if necessary.\"\"\"\n if self.scratch.should_offload(output):\n ref = self.scratch.offload(output, source=tool_name)\n return f\"[{tool_name} output saved to {ref['path']}. Summary: {ref['summary'][:200]}]\"\n return output\n \n def get_system_prompt(self) -> str:\n \"\"\"Build system prompt with dynamic skill references.\"\"\"\n base_prompt = \"You are a helpful assistant.\"\n skill_context = self.skills.get_static_context()\n user_prefs = self.preferences.list_all()\n \n pref_section = \"\"\n if user_prefs:\n pref_section = \"\\n\\nUser preferences:\\n\" + \"\\n\".join(\n f\"- {k}: {v}\" for k, v in user_prefs.items()\n )\n \n return f\"{base_prompt}\\n\\n{skill_context}{pref_section}\"\n```\n\n## File Organization Best Practices\n\n```\nproject/\n├── scratch/ # Ephemeral working files\n│ ├── tool_outputs/ # Large tool results\n│ │ └── search_20260107.txt\n│ └── plans/ # Active task plans\n│ └── current_plan.yaml\n├── workspace/ # Agent workspaces\n│ └── agents/\n│ ├── research_agent/\n│ │ ├── findings.md\n│ │ └── status.json\n│ └── code_agent/\n│ ├── findings.md\n│ └── status.json\n├── agent/ # Agent configuration\n│ ├── preferences.yaml # Learned preferences\n│ └── patterns.md # Discovered patterns\n├── skills/ # Loadable skills\n│ └── {skill-name}/\n│ └── SKILL.md\n├── terminals/ # Terminal output\n│ ├── 1.txt\n│ └── 2.txt\n└── history/ # Chat history archives\n └── session_001.txt\n```\n\n## Token Accounting Metrics\n\nTrack these metrics to validate filesystem patterns:\n\n1. **Static context ratio**: tokens in static context / total tokens\n2. **Dynamic load rate**: how often skills/files are loaded per task\n3. **Offload savings**: tokens saved by writing to files vs keeping in context\n4. **Retrieval precision**: percentage of loaded content actually used\n\nTarget benchmarks:\n- Static context ratio < 20%\n- Offload savings > 50% for tool-heavy workflows\n- Retrieval precision > 70% (loaded content is relevant)\n\n","skills/harness-engineering/SKILL.md":"---\nname: harness-engineering\ndescription: \"This skill should be used when designing autonomous agent harnesses: research loops, evaluation scaffolds, locked and editable surfaces, durable logs, novelty gates, pruning, rollback, PR preparation, and human approval boundaries.\"\n---\n\n# Harness Engineering\n\nHarness engineering designs the control system around an agent: what it may edit, how it receives feedback, where it writes state, how failures recover, and who can approve irreversible actions. The harness is the difference between a helpful agent session and an autonomous loop that can run for days without corrupting its objective.\n\n## When to Activate\n\nActivate this skill when:\n\n- Building autonomous research or experimentation loops\n- Designing an agent environment with locked metrics and editable code or content\n- Creating PR-producing or background agents\n- Evaluating whether an agent can safely run without frequent human prompts\n- Adding novelty, ablation, pruning, rollback, or durable logging to an agent workflow\n- Preventing agents from gaming benchmarks, weakening rubrics, or losing state across compaction\n\nDo not activate this skill for adjacent work owned by other skills:\n- General quality gates, regression suites, or outcome metrics without autonomous control surfaces: `evaluation`.\n- Tool schemas, response formats, and recovery errors for harness tools: `tool-design`.\n- Project-level task-model fit, pipeline shape, and cost planning: `project-development`.\n- Remote sandbox, warm-pool, and hosted session infrastructure: `hosted-agents`.\n\n## Core Concepts\n\n### Harness Boundary\n\nSeparate the agent from the environment it operates inside. The agent proposes actions; the harness defines allowed surfaces, feedback, persistence, and promotion rules.\n\nUse four surface classes:\n\n| Surface | Examples | Rule |\n| --- | --- | --- |\n| Locked | Eval metric, rubric, validation script, merge policy | Agent may read and propose changes, but cannot score itself with modified rules |\n| Editable | Skill draft, experiment file, prompt, config under test | Agent may mutate during the loop |\n| Append-only | Results log, research thread, rejected ideas | Agent may append, not rewrite |\n| Human-controlled | Merge, production deploy, credentials, destructive operations | Requires explicit human approval |\n\n### Tight Feedback Loops\n\nAutonomy works when feedback is fast, unambiguous, and hard to game. Karpathy's `autoresearch` is the minimal pattern: one editable file, one locked evaluation file, fixed wall-clock budget, one scalar metric, git rollback, and a durable results log. The lesson is not that every harness needs one metric; it is that ambiguous feedback creates ambiguous autonomy.\n\nFor open-ended research-to-skill work, replace the scalar metric with locked rubrics, deterministic structure checks, source traceability, and human review thresholds.\n\n### Durable State\n\nLong-running agents must externalize state. Store plans, source queues, results, failures, and handoffs in files so future agents can resume without relying on chat history. Prime Intellect's autonomous nanoGPT work showed the value of durable scratchpads and `THREAD.md`-style logs for recovery, monitoring, and audit.\n\nUse append-only logs for:\n\n- What was tried\n- What improved or failed\n- Why a candidate was kept, discarded, or routed to review\n- Which upstream sources were checked\n- What the next agent should do\n\n### Search Discipline\n\nAgents tend to exploit the nearest surface, stack complexity, and under-run pruning. Add explicit search rules:\n\n1. Refresh upstream sources on a schedule.\n2. Require novelty checks before spending large budgets.\n3. Preserve rejected attempts to avoid rediscovery.\n4. Run leave-one-out pruning when a stack has multiple additions.\n5. Reward simplification when quality is equal.\n6. Use separate verification before promotion.\n\n### Mechanism Registry\n\nFor research-to-skill systems, track accepted mechanisms separately from prose. A mechanism record should include a stable `mechanism_id`, `owning_skill`, `status`, activation scenario, behavior change, evidence, and failure modes. Novelty gates should compare against this registry before using broader corpus overlap, because keyword overlap catches stale phrasing while mechanism comparison catches real duplication.\n\n### Governance\n\nAutonomous agents may prepare PRs, but governance must be explicit. They can draft changes, run checks, and write PR summaries. They should not merge, deploy, or push without human approval unless the user has explicitly granted that permission for the specific action.\n\n## Detailed Topics\n\n### Autoresearch-Style Loop\n\nUse this pattern when optimizing an artifact against a stable evaluator:\n\n```text\nread locked context -> choose hypothesis -> edit allowed surface -> commit/checkpoint\n-> run evaluator -> log result -> keep if better -> discard or rollback if worse\n-> repeat\n```\n\nRequired properties:\n\n- The evaluator is outside the editable surface.\n- The feedback cadence is fixed enough to compare attempts.\n- Failed attempts leave an audit trail.\n- Rollback is cheap.\n- The agent has a policy for crashes and timeouts.\n\n### Research-To-Skill Loop\n\nUse this pattern when sources become skill changes:\n\n```text\ndiscover -> retrieve -> gate -> score -> extract mechanism\n-> map to existing or new skill -> draft proposal -> validate structure\n-> prepare PR -> human review\n```\n\nThe locked evaluator is a combination of source rubrics, skill-change rubrics, structure checks, and reviewer approval. The editable artifact is the proposed skill delta.\n\n### Metric Gaming Resistance\n\nAssume an optimizing agent will learn the harness. Guard against:\n\n- Editing evaluation code or rubrics and then using the new version for self-approval\n- Adding verbose content that pleases a judge but harms skill activation\n- Citing unretrieved sources\n- Optimizing aggregate scores while failing a critical dimension\n- Avoiding failed results in the log\n\nMitigation: lock rubrics per run, report per-dimension scores, require source retrieval evidence, preserve rejected attempts, and route governance changes to human review.\n\n### Monitoring Agents\n\nUse monitoring agents for long runs, but restrict them to read-only reporting unless explicitly tasked otherwise. Monitoring output should report:\n\n- Best current candidate\n- Active jobs or drafts\n- Last upstream refresh\n- Failed or stale loops\n- Disagreements between logs and claimed state\n- Next action and blocker\n\n## Practical Guidance\n\n### Harness Design Checklist\n\n1. Define the objective in one sentence.\n2. Identify locked, editable, append-only, and human-controlled surfaces.\n3. Choose the feedback mechanism: scalar metric, rubric, deterministic tests, human review, or combination.\n4. Define keep, discard, crash, timeout, and review states.\n5. Create a durable thread log before the loop starts.\n6. Add source refresh, mechanism-registry novelty, and pruning rules for long-running loops.\n7. Define what the agent may do without asking and what requires approval.\n8. Validate the harness on one known good and one known bad artifact.\n\n### File Layout\n\n```text\nresearch-run/\n THREAD.md\n sources/\n queue.md\n evaluations/\n proposals/\n logs/\n results.tsv\n rejected.md\n drafts/\n```\n\nUse TSV or JSONL for append-only machine-readable logs. Use Markdown for handoffs and reviewer-facing summaries.\n\n## Examples\n\n**Example 1: Locked metric**\n\nAn agent optimizes `train.py`, but `prepare.py` owns data loading and evaluation. The agent can edit the model but cannot change the metric. Failed experiments are logged and rolled back.\n\n**Example 2: Locked rubric**\n\nAn agent evaluates a new Anthropic or OpenAI engineering post, but the source curation rubric is locked for the run. If the source passes, the agent drafts a skill proposal. It cannot lower the rubric threshold to admit the source.\n\n**Example 3: Auto-PR without auto-merge**\n\nAn agent prepares a branch and PR body after passing source, skill, and structure checks. The PR states unresolved risks and waits for human merge approval.\n\n## Guidelines\n\n1. Lock evaluators before starting the loop.\n2. Keep editable surfaces narrow enough for reliable diffs.\n3. Write durable logs before context compaction can erase state.\n4. Report per-dimension scores instead of only aggregate scores.\n5. Require source retrieval before citation.\n6. Add novelty gates for broad search and pruning gates for complex stacks.\n7. Prefer simplification when quality is equal.\n8. Separate PR preparation from merge authority.\n9. Revalidate harness changes with old and new evaluators.\n10. Treat stopped autonomous loops as harness failures, not agent personality quirks.\n\n## Gotchas\n\n1. **Mutable evaluator**: If the agent can edit the metric, it may optimize the benchmark instead of the task. Keep rubrics and eval code locked during the run.\n2. **Chat-only memory**: Long runs fail after compaction when plans live only in conversation history. Write thread logs and result files from the start.\n3. **No discard record**: Without rejected-attempt logs, agents repeat failed ideas. Preserve failures with enough detail to avoid rediscovery.\n4. **Complexity accretion**: Agents stack changes and rarely remove them. Require pruning rounds and reward equal-quality simplification.\n5. **Premature novelty claims**: Agents label recombinations as novel. Compare against existing repo skills, source queue, and rejected logs before claiming novelty.\n6. **Monitor misreporting**: Monitoring agents can summarize stale or inconsistent state. Require them to cite the files or logs behind claims.\n7. **Human approval ambiguity**: \"Prepare a PR\" is not \"merge a PR.\" Make approval boundaries explicit in the harness.\n8. **Volatile source drift**: Fast-moving lab claims age quickly. Put dated evidence in references and schedule revalidation.\n\n## Integration\n\nThis skill connects to:\n\n- evaluation - Rubrics and quality gates provide the locked feedback surface\n- advanced-evaluation - Pairwise comparison and bias mitigation improve proposal review\n- filesystem-context - Durable logs, scratchpads, and thread files preserve state\n- multi-agent-patterns - Researcher, verifier, monitor, and writer agents need isolated contexts\n- tool-design - Harness tools must expose clear contracts and recovery errors\n- project-development - File-based pipelines and task-model fit analysis keep loops simple\n- hosted-agents - Background execution needs sandbox, snapshot, and approval boundaries\n\n## References\n\nInternal references:\n- `researcher/README.md` - Read when implementing the repo-native research-to-skill operating system\n- `researcher/rubrics/harness-change.md` - Read when evaluating changes to an agent harness\n- `researcher/runbooks/autonomous-research-loop.md` - Read when running a source-to-skill loop\n\nExternal resources:\n- Karpathy `autoresearch` - Constrained autonomous experiment loop with locked evaluation\n- Prime Intellect autonomous nanoGPT speedrun - Durable scratchpads, handoffs, monitoring, and autonomy failure modes\n- AlphaEvolve and FunSearch - LLM-generated candidates paired with systematic evaluators\n- HELM and LM Evaluation Harness - Transparent, reproducible evaluation infrastructure\n\n---\n\n## Skill Metadata\n\n**Created**: 2026-05-14\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 1.1.0\n","skills/hosted-agents/SKILL.md":"---\nname: hosted-agents\ndescription: \"This skill should be used when designing hosted or background agent infrastructure: sandboxed execution, remote coding environments, warm pools, session persistence, multiplayer collaboration, self-spawning agents, or Modal-style sandboxes.\"\n---\n\n# Hosted Agent Infrastructure\n\nHosted agents run in remote sandboxed environments rather than on local machines. When designed well, they provide unlimited concurrency, consistent execution environments, and multiplayer collaboration. The critical insight is that session speed should be limited only by model provider time-to-first-token, with all infrastructure setup completed before the user starts their session.\n\n## When to Activate\n\nActivate this skill when:\n- Building background coding agents that run independently of user devices\n- Designing sandboxed execution environments for agent workloads\n- Implementing multiplayer agent sessions with shared state\n- Creating multi-client agent interfaces (Slack, Web, Chrome extensions)\n- Scaling agent infrastructure beyond local machine constraints\n- Building systems where agents spawn sub-agents for parallel work\n\nDo not activate this skill for adjacent work owned by other skills:\n- Designing the autonomous research loop, novelty gates, rollback policy, or merge boundaries: `harness-engineering`.\n- Choosing supervisor, swarm, or handoff topology without hosted infrastructure concerns: `multi-agent-patterns`.\n- Designing the tools used by a hosted agent, such as spawn/status tools or PR tools: `tool-design`.\n- Managing file-backed state inside a session rather than the hosted runtime itself: `filesystem-context`.\n\n## Core Concepts\n\nMove agent execution to remote sandboxed environments to eliminate the fundamental limits of local execution: resource contention, environment inconsistency, and single-user constraints. Remote sandboxes unlock unlimited concurrency, reproducible environments, and collaborative workflows because each session gets its own isolated compute with a known-good environment image.\n\nDesign the architecture in three layers because each layer scales independently. Build sandbox infrastructure for isolated execution, an API layer for state management and client coordination, and client interfaces for user interaction across platforms. Keep these layers cleanly separated so sandbox changes do not ripple into clients.\n\n## Detailed Topics\n\n### Sandbox Infrastructure\n\n**The Core Challenge**\nEliminate sandbox spin-up latency because users perceive anything over a few seconds as broken. Development environments require cloning repositories, installing dependencies, and running build steps -- do all of this before the user ever submits a prompt.\n\n**Image Registry Pattern**\nPre-build environment images on a regular cadence (every 30 minutes works well) because this makes synchronization with the latest code a fast delta rather than a full clone. Include in each image:\n- Cloned repository at a known commit\n- All runtime dependencies installed\n- Initial setup and build commands completed\n- Cached files from running app and test suite once\n\nWhen starting a session, spin up a sandbox from the most recent image. The repository is at most 30 minutes out of date, making the remaining git sync fast.\n\n**Snapshot and Restore**\nTake filesystem snapshots at key points to enable instant restoration for follow-up prompts without re-running setup:\n- After initial image build (base snapshot)\n- When agent finishes making changes (session snapshot)\n- Before sandbox exit for potential follow-up\n\n**Git Configuration for Background Agents**\nConfigure git identity explicitly in every sandbox because background agents are not tied to a specific user during image builds:\n- Generate GitHub app installation tokens for repository access during clone\n- Set git config `user.name` and `user.email` when committing and pushing changes\n- Use the prompting user's identity for commits, not the app identity\n\n**Warm Pool Strategy**\nMaintain a pool of pre-warmed sandboxes for high-volume repositories because cold starts are the primary source of user frustration:\n- Keep sandboxes ready before users start sessions\n- Expire and recreate pool entries as new image builds complete\n- Start warming a sandbox as soon as a user begins typing (predictive warm-up)\n\n### Agent Framework Selection\n\n**Server-First Architecture**\nStructure the agent framework as a server first, with TUI and desktop apps as thin clients, because this prevents duplicating agent logic across surfaces:\n- Multiple custom clients share one agent backend\n- Consistent behavior across all interaction surfaces\n- Plugin systems extend functionality without client changes\n- Event-driven architectures deliver real-time updates to any connected client\n\n**Code as Source of Truth**\nSelect frameworks where the agent can read its own source code to understand behavior. Prioritize this because having code as source of truth prevents the agent from hallucinating about its own capabilities -- an underrated failure mode in AI development.\n\n**Plugin System Requirements**\nRequire a plugin system that supports runtime interception because this enables safety controls and observability without modifying core agent logic:\n- Listen to tool execution events (e.g., `tool.execute.before`)\n- Block or modify tool calls conditionally\n- Inject context or state at runtime\n\n### Speed Optimizations\n\n**Predictive Warm-Up**\nStart warming the sandbox as soon as a user begins typing their prompt, not when they submit it, because the typing interval (5-30 seconds) is enough to complete most setup:\n- Clone latest changes in parallel with user typing\n- Run initial setup before user hits enter\n- For fast spin-up, sandbox can be ready before user finishes typing\n\n**Parallel File Reading**\nAllow the agent to start reading files immediately even if sync from latest base branch is not complete, because in large repositories incoming prompts rarely touch recently-changed files:\n- Agent can research immediately without waiting for git sync\n- Block file edits (not reads) until synchronization completes\n- This separation is safe because read-time data staleness of 30 minutes rarely matters for research\n\n**Maximize Build-Time Work**\nMove everything possible to the image build step because build-time duration is invisible to users:\n- Full dependency installation\n- Database schema setup\n- Initial app and test suite runs (populates caches)\n\n### Self-Spawning Agents\n\n**Agent-Spawned Sessions**\nBuild tools that allow agents to spawn new sessions because frontier models are capable of decomposing work and coordinating sub-tasks:\n- Research tasks across different repositories\n- Parallel subtask execution for large changes\n- Multiple smaller PRs from one major task\n\nExpose three primitives: start a new session with specified parameters, read status of any session (check-in capability), and continue main work while sub-sessions run in parallel.\n\n**Prompt Engineering for Self-Spawning**\nEngineer prompts that guide when agents should spawn sub-sessions rather than doing work inline:\n- Research tasks that require cross-repository exploration\n- Breaking monolithic changes into smaller PRs\n- Parallel exploration of different approaches\n\n### API Layer\n\n**Per-Session State Isolation**\nIsolate state per session (SQLite per session works well) because cross-session interference is a subtle and hard-to-debug failure mode:\n- Dedicated database per session\n- No session can impact another's performance\n- Architecture handles hundreds of concurrent sessions\n\n**Real-Time Streaming**\nStream all agent work in real-time because high-frequency feedback is critical for user trust:\n- Token streaming from model providers\n- Tool execution status updates\n- File change notifications\n\nUse WebSocket connections with hibernation APIs to reduce compute costs during idle periods while maintaining open connections.\n\n**Synchronization Across Clients**\nBuild a single state system that synchronizes across all clients (chat interfaces, Slack bots, Chrome extensions, web interfaces, VS Code instances) because users switch surfaces frequently and expect continuity. All changes sync to the session state, enabling seamless client switching.\n\n### Multiplayer Support\n\n**Why Multiplayer Matters**\nDesign for multiplayer from day one because it is nearly free to add with proper synchronization architecture, and it unlocks high-value workflows:\n- Teaching non-engineers to use AI effectively\n- Live QA sessions with multiple team members\n- Real-time PR review with immediate changes\n- Collaborative debugging sessions\n\n**Implementation Requirements**\nBuild the data model so sessions are not tied to single authors because multiplayer fails silently if authorship is hardcoded:\n- Pass authorship info to each prompt\n- Attribute code changes to the prompting user\n- Share session links for instant collaboration\n\n### Authentication and Authorization\n\n**User-Based Commits**\nUse GitHub authentication to open PRs on behalf of the user (not the app) because this preserves the audit trail and prevents users from approving their own AI-generated changes:\n- Obtain user tokens for PR creation\n- PRs appear as authored by the human, not the bot\n\n**Sandbox-to-API Flow**\nFollow this sequence because it keeps sandbox permissions minimal while letting the API handle sensitive operations:\n1. Sandbox pushes changes (updating git user config)\n2. Sandbox sends event to API with branch name and session ID\n3. API uses user's GitHub token to create PR\n4. GitHub webhooks notify API of PR events\n\n### Client Implementations\n\n**Slack Integration**\nPrioritize Slack as the first distribution channel for internal adoption because it creates a virality loop as team members see others using it:\n- No syntax required, natural chat interface\n- Build a classifier (fast model with repo descriptions) to determine which repository to work in\n- Include hints for common repositories; allow \"unknown\" for ambiguous cases\n\n**Web Interface**\nBuild a web interface with these features because it serves as the primary power-user surface:\n- Real-time streaming of agent work on desktop and mobile\n- Hosted VS Code instance running inside sandbox\n- Streamed desktop view for visual verification\n- Before/after screenshots for PRs\n- Statistics page: sessions resulting in merged PRs (primary metric), usage over time, live \"humans prompting\" count\n\n**Chrome Extension**\nBuild a Chrome extension for non-engineering users because DOM and React internals extraction gives higher precision than raw screenshots at lower token cost:\n- Sidebar chat interface with screenshot tool\n- Extract DOM/React internals instead of raw images\n- Distribute via managed device policy (bypasses Chrome Web Store)\n\n## Practical Guidance\n\n### Hosted Agent Design Checklist\n\nBefore building the system, decide these infrastructure properties explicitly:\n\n1. **Sandbox lifecycle**: how sessions start, snapshot, restore, time out, and terminate.\n2. **Image and warm-pool policy**: how often images rebuild, which caches are precomputed, and when warm sandboxes expire.\n3. **Read/write synchronization**: whether agents may read before repository sync completes, and which events unblock writes.\n4. **Per-session state isolation**: what storage belongs to one session, what is shared, and how cross-session interference is prevented.\n5. **Auth and commit identity**: which operations use app tokens, which use user tokens, and how commits are attributed.\n6. **Output extraction**: how branches, PRs, files, logs, screenshots, and session summaries leave the sandbox.\n7. **Budget and teardown**: maximum runtime, cost ceilings, idle policy, and forced cleanup behavior.\n\n### Follow-Up Message Handling\n\nChoose between queueing and inserting follow-up messages sent during execution. Prefer queueing because it is simpler to manage and lets users send thoughts on next steps while the agent works. Build a mechanism to stop the agent mid-execution when needed, because without it users feel trapped.\n\n### Metrics That Matter\n\nTrack these metrics because they indicate real value rather than vanity usage:\n- Sessions resulting in merged PRs (primary success metric)\n- Time from session start to first model response\n- PR approval rate and revision count\n- Agent-written code percentage across repositories\n\n### Adoption Strategy\n\nDrive internal adoption through visibility rather than mandates because forced usage breeds resentment:\n- Work in public spaces (Slack channels) for visibility\n- Let the product create virality loops\n- Do not force usage over existing tools\n- Build to people's needs, not hypothetical requirements\n\n## Examples\n\n**Example 1: Background coding session lifecycle**\n\n```text\nuser prompt\n-> API allocates warm sandbox from current image\n-> sandbox syncs latest branch delta\n-> reads allowed immediately, writes blocked until sync completes\n-> agent edits and tests inside isolated workspace\n-> sandbox snapshots final filesystem\n-> branch is pushed\n-> API creates PR using user token\n-> session summary, logs, and PR URL are returned to clients\n```\n\nThis sequence keeps setup work outside the user-visible path while preserving auditability and user ownership of code changes.\n\n**Example 2: Boundary decision**\n\nIf the task is \"make the agent loop run for days with locked rubrics and PR approval,\" use `harness-engineering`. If the task is \"run that loop in remote sandboxes with warm pools, session snapshots, streaming clients, and user-authored PRs,\" use this skill.\n\n## Guidelines\n\n1. Pre-build environment images on regular cadence (30 minutes is a good default)\n2. Start warming sandboxes when users begin typing, not when they submit\n3. Allow file reads before git sync completes; block only writes\n4. Structure agent framework as server-first with clients as thin wrappers\n5. Isolate state per session to prevent cross-session interference\n6. Attribute commits to the user who prompted, not the app\n7. Track merged PRs as primary success metric\n8. Build for multiplayer from the start; it is nearly free with proper sync architecture\n\n## Gotchas\n\n1. **Cold start latency**: First sandbox spin-up takes 30-60s and users perceive this as broken. Use warm pools and predictive warm-up on keystroke to eliminate perceived wait time.\n2. **Image staleness**: Infrequent image rebuilds mean agents run with outdated dependencies or code. Set a 30-minute rebuild cadence and monitor image age; alert if builds fail silently.\n3. **Sandbox cost runaway**: Long-running agents without timeout or budget caps accumulate unexpected costs. Set hard timeout limits (default 4 hours) and per-session cost ceilings.\n4. **Auth token expiration mid-session**: Long tasks fail when GitHub tokens expire partway through. Implement token refresh logic and check token validity before sensitive operations like PR creation.\n5. **Git config in sandboxes**: Missing `user.name` or `user.email` causes commit failures in background agents. Always set git identity explicitly during sandbox configuration, never assume it carries over from the image.\n6. **State loss on sandbox recycle**: Agents lose completed work if the sandbox is recycled or times out before results are extracted. Always snapshot before termination and extract artifacts (branches, PRs, files) before letting the sandbox die.\n7. **Oversubscribing warm pools**: Maintaining too many warm sandboxes wastes money during low-traffic periods. Scale pool size based on traffic patterns and time-of-day; use autoscaling rather than fixed pool sizes.\n8. **Missing output extraction**: Agents complete work inside the sandbox but results never get pulled out to the user. Build explicit extraction steps (push branch, create PR, return file contents) into the session teardown flow.\n\n## Integration\n\nThis skill owns hosted runtime infrastructure. Adjacent skills own the control system, topology, and tool contracts:\n\n- `harness-engineering`: governance, locked evaluators, rollback, novelty gates, and human approval boundaries around autonomous work.\n- `multi-agent-patterns`: self-spawning and supervisor patterns once hosted infrastructure exists.\n- `tool-design`: spawn, status, teardown, and PR tools exposed to agents.\n- `context-optimization`: managing context across distributed hosted sessions.\n- `filesystem-context`: using the sandbox filesystem for durable session state and artifacts.\n\n## References\n\nInternal reference:\n- [Infrastructure Patterns](./references/infrastructure-patterns.md) - Read when: implementing sandbox lifecycle, image builds, or warm pool logic for the first time\n\nRelated skills in this collection:\n- multi-agent-patterns - Read when: designing self-spawning or supervisor coordination patterns\n- tool-design - Read when: building tools for agent session management or status checking\n- context-optimization - Read when: context windows fill up across distributed agent sessions\n\nExternal resources:\n- [Ramp](https://builders.ramp.com/post/why-we-built-our-background-agent) - Read when: evaluating whether to build vs. buy background agent infrastructure\n- [Modal Sandboxes](https://modal.com/docs/guide/sandbox) - Read when: choosing a cloud sandbox provider or comparing isolation models\n- [Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/) - Read when: designing per-session state management with WebSocket hibernation\n- [OpenCode](https://github.com/sst/opencode) - Read when: selecting a server-first agent framework or studying plugin architectures\n\n---\n\n## Skill Metadata\n\n**Created**: 2026-01-12\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 1.2.0\n","skills/hosted-agents/references/infrastructure-patterns.md":"# Infrastructure Patterns for Hosted Agents\n\nThis reference provides detailed implementation patterns for building hosted agent infrastructure. These patterns are derived from production systems at scale.\n\n## Sandbox Architecture\n\n### Modal Integration Pattern\n\nModal provides the sandbox infrastructure with near-instant startup and filesystem snapshots.\n\n```python\nimport modal\n\n# Define the base image with all dependencies\nimage = modal.Image.debian_slim().pip_install([\n \"opencode\",\n \"gitpython\",\n \"psycopg2-binary\",\n])\n\n# Create the app\napp = modal.App(\"coding-agent\")\n\n# Sandbox class with snapshot support\n@app.cls(image=image, timeout=3600)\nclass AgentSandbox:\n def __init__(self, repo_url: str, snapshot_id: str = None):\n self.repo_url = repo_url\n self.snapshot_id = snapshot_id\n \n @modal.enter()\n def setup(self):\n if self.snapshot_id:\n # Restore from snapshot\n modal.Sandbox.restore(self.snapshot_id)\n else:\n # Fresh setup from image\n self._clone_and_setup()\n \n def _clone_and_setup(self):\n \"\"\"Clone repo and run initial setup.\"\"\"\n token = self._get_github_app_token()\n os.system(f\"git clone https://x-access-token:{token}@github.com/{self.repo_url}\")\n os.system(\"npm install\")\n os.system(\"npm run build\")\n \n @modal.method()\n def execute_prompt(self, prompt: str, user_identity: dict) -> dict:\n \"\"\"Execute a prompt in the sandbox.\"\"\"\n # Update git config for this user\n os.system(f'git config user.name \"{user_identity[\"name\"]}\"')\n os.system(f'git config user.email \"{user_identity[\"email\"]}\"')\n \n # Run the agent\n result = self.agent.run(prompt)\n \n return {\n \"result\": result,\n \"snapshot_id\": modal.Sandbox.snapshot()\n }\n```\n\n### Image Build Pipeline\n\nBuild images on a schedule to keep them fresh:\n\n```python\nimport schedule\nimport time\nfrom datetime import datetime\n\nclass ImageBuilder:\n def __init__(self, repositories: list[str]):\n self.repositories = repositories\n self.images = {}\n \n def build_all_images(self):\n \"\"\"Build images for all repositories.\"\"\"\n for repo in self.repositories:\n try:\n image = self._build_image(repo)\n self.images[repo] = {\n \"image\": image,\n \"built_at\": datetime.utcnow(),\n \"commit\": self._get_latest_commit(repo)\n }\n except Exception as e:\n # Log but continue with other repos\n log.error(f\"Failed to build image for {repo}: {e}\")\n \n def _build_image(self, repo: str) -> str:\n \"\"\"Build a single repository image.\"\"\"\n sandbox = modal.Sandbox.create()\n \n # Clone with app token\n token = get_app_installation_token(repo)\n sandbox.exec(f\"git clone https://x-access-token:{token}@github.com/{repo} /workspace\")\n \n # Install dependencies\n sandbox.exec(\"cd /workspace && npm install\")\n \n # Run build\n sandbox.exec(\"cd /workspace && npm run build\")\n \n # Warm caches\n sandbox.exec(\"cd /workspace && npm run dev &\")\n time.sleep(5) # Let dev server start\n sandbox.exec(\"cd /workspace && npm test -- --run\")\n \n # Create snapshot\n return sandbox.snapshot()\n \n def get_latest_image(self, repo: str) -> str:\n \"\"\"Get the most recent image for a repository.\"\"\"\n if repo not in self.images:\n raise ValueError(f\"No image available for {repo}\")\n return self.images[repo][\"image\"]\n\n# Schedule builds every 30 minutes\nbuilder = ImageBuilder([\"org/frontend\", \"org/backend\", \"org/shared\"])\nschedule.every(30).minutes.do(builder.build_all_images)\n```\n\n### Warm Pool Management\n\nMaintain pre-warmed sandboxes for instant session starts:\n\n```python\nfrom collections import defaultdict\nfrom dataclasses import dataclass\nfrom datetime import datetime, timedelta\n\n@dataclass\nclass WarmSandbox:\n sandbox_id: str\n repo: str\n created_at: datetime\n image_version: str\n is_claimed: bool = False\n\nclass WarmPoolManager:\n def __init__(self, target_pool_size: int = 3):\n self.target_size = target_pool_size\n self.pools = defaultdict(list) # repo -> [WarmSandbox]\n self.max_age = timedelta(minutes=25) # Expire before next image build\n \n def get_warm_sandbox(self, repo: str) -> WarmSandbox | None:\n \"\"\"Get a pre-warmed sandbox if available.\"\"\"\n pool = self.pools[repo]\n \n for sandbox in pool:\n if not sandbox.is_claimed and self._is_valid(sandbox):\n sandbox.is_claimed = True\n return sandbox\n \n return None\n \n def _is_valid(self, sandbox: WarmSandbox) -> bool:\n \"\"\"Check if sandbox is still valid.\"\"\"\n age = datetime.utcnow() - sandbox.created_at\n current_image = self.image_builder.get_latest_image(sandbox.repo)\n \n return (\n age < self.max_age and\n sandbox.image_version == current_image\n )\n \n def maintain_pool(self, repo: str):\n \"\"\"Ensure pool has target number of warm sandboxes.\"\"\"\n # Remove expired sandboxes\n self.pools[repo] = [s for s in self.pools[repo] if self._is_valid(s)]\n \n # Add new sandboxes to reach target\n current_count = len([s for s in self.pools[repo] if not s.is_claimed])\n needed = self.target_size - current_count\n \n for _ in range(needed):\n sandbox = self._create_warm_sandbox(repo)\n self.pools[repo].append(sandbox)\n \n def _create_warm_sandbox(self, repo: str) -> WarmSandbox:\n \"\"\"Create a new warm sandbox from latest image.\"\"\"\n image = self.image_builder.get_latest_image(repo)\n sandbox_id = modal.Sandbox.create(image=image)\n \n # Sync to latest (runs in background)\n self._sync_to_latest(sandbox_id, repo)\n \n return WarmSandbox(\n sandbox_id=sandbox_id,\n repo=repo,\n created_at=datetime.utcnow(),\n image_version=image\n )\n```\n\n## API Layer Patterns\n\n### Cloudflare Durable Objects for Session State\n\nEach session gets its own Durable Object with isolated SQLite:\n\n```typescript\n// Session Durable Object\nexport class SessionDO implements DurableObject {\n private storage: DurableObjectStorage;\n private sql: SqlStorage;\n private connections: Map = new Map();\n\n constructor(ctx: DurableObjectState) {\n this.storage = ctx.storage;\n this.sql = ctx.storage.sql;\n this.initializeSchema();\n }\n\n private initializeSchema() {\n this.sql.exec(`\n CREATE TABLE IF NOT EXISTS messages (\n id INTEGER PRIMARY KEY,\n role TEXT NOT NULL,\n content TEXT NOT NULL,\n author_id TEXT,\n author_name TEXT,\n created_at TEXT DEFAULT CURRENT_TIMESTAMP\n );\n \n CREATE TABLE IF NOT EXISTS artifacts (\n id INTEGER PRIMARY KEY,\n type TEXT NOT NULL,\n path TEXT,\n content TEXT,\n created_at TEXT DEFAULT CURRENT_TIMESTAMP\n );\n \n CREATE TABLE IF NOT EXISTS events (\n id INTEGER PRIMARY KEY,\n type TEXT NOT NULL,\n data TEXT,\n created_at TEXT DEFAULT CURRENT_TIMESTAMP\n );\n `);\n }\n\n async fetch(request: Request): Promise {\n const url = new URL(request.url);\n\n if (request.headers.get(\"Upgrade\") === \"websocket\") {\n return this.handleWebSocket(request);\n }\n\n switch (url.pathname) {\n case \"/message\":\n return this.handleMessage(request);\n case \"/status\":\n return this.getStatus();\n default:\n return new Response(\"Not found\", { status: 404 });\n }\n }\n\n private handleWebSocket(request: Request): Response {\n const pair = new WebSocketPair();\n const [client, server] = Object.values(pair);\n\n const connectionId = crypto.randomUUID();\n this.connections.set(connectionId, server);\n\n server.accept();\n server.addEventListener(\"close\", () => {\n this.connections.delete(connectionId);\n });\n\n return new Response(null, { status: 101, webSocket: client });\n }\n\n private broadcast(message: object) {\n const data = JSON.stringify(message);\n for (const ws of this.connections.values()) {\n ws.send(data);\n }\n }\n\n async handleMessage(request: Request): Promise {\n const { content, author } = await request.json();\n\n // Store message\n this.sql.exec(\n `INSERT INTO messages (role, content, author_id, author_name) VALUES (?, ?, ?, ?)`,\n [\"user\", content, author.id, author.name]\n );\n\n // Broadcast to all connected clients\n this.broadcast({\n type: \"message\",\n role: \"user\",\n content,\n author,\n });\n\n // Forward to sandbox for processing\n const result = await this.forwardToSandbox(content, author);\n\n return Response.json(result);\n }\n}\n```\n\n### Real-Time Event Streaming\n\nStream events from sandbox to all connected clients:\n\n```typescript\nclass EventStream {\n private sessionDO: DurableObjectStub;\n\n async streamFromSandbox(sandboxId: string, sessionId: string) {\n const sandbox = await modal.Sandbox.get(sandboxId);\n\n // Subscribe to sandbox events\n for await (const event of sandbox.events()) {\n // Forward to Durable Object for broadcast\n await this.sessionDO.fetch(\n new Request(`https://internal/event`, {\n method: \"POST\",\n body: JSON.stringify({\n type: event.type,\n data: event.data,\n }),\n })\n );\n }\n }\n}\n```\n\n## Client Integration Patterns\n\n### Slack Bot with Repository Classification\n\n```python\nfrom slack_bolt import App\nfrom slack_bolt.adapter.socket_mode import SocketModeHandler\n\napp = App(token=os.environ[\"SLACK_BOT_TOKEN\"])\n\n# Repository descriptions for classification\nREPO_DESCRIPTIONS = [\n {\n \"name\": \"frontend-monorepo\",\n \"description\": \"React frontend application with dashboard, user portal, and admin interfaces\",\n \"hints\": [\"dashboard\", \"UI\", \"component\", \"page\", \"frontend\"]\n },\n {\n \"name\": \"backend-services\",\n \"description\": \"Node.js API services including auth, payments, and core business logic\",\n \"hints\": [\"API\", \"endpoint\", \"service\", \"backend\", \"database\"]\n },\n {\n \"name\": \"mobile-app\",\n \"description\": \"React Native mobile application for iOS and Android\",\n \"hints\": [\"mobile\", \"app\", \"iOS\", \"Android\", \"native\"]\n }\n]\n\nasync def classify_repository(message: str, channel: str, thread: list[str]) -> str:\n \"\"\"Use fast model to classify which repo the message refers to.\"\"\"\n prompt = f\"\"\"Classify which repository this message is about.\n\nMessage: {message}\nChannel: #{channel}\nThread context: {' | '.join(thread[-3:])}\n\nRepositories:\n{json.dumps(REPO_DESCRIPTIONS, indent=2)}\n\nReturn ONLY the repository name, or \"unknown\" if unclear.\"\"\"\n\n response = await openai.chat.completions.create(\n model=\"gpt-4o-mini\",\n messages=[{\"role\": \"user\", \"content\": prompt}],\n max_tokens=50\n )\n \n return response.choices[0].message.content.strip()\n\n@app.event(\"app_mention\")\nasync def handle_mention(event, say, client):\n \"\"\"Handle @mentions of the bot.\"\"\"\n channel = event[\"channel\"]\n message = event[\"text\"]\n thread_ts = event.get(\"thread_ts\", event[\"ts\"])\n \n # Get thread context if in a thread\n thread_messages = []\n if \"thread_ts\" in event:\n result = await client.conversations_replies(\n channel=channel,\n ts=thread_ts\n )\n thread_messages = [m[\"text\"] for m in result[\"messages\"]]\n \n # Get channel info for context\n channel_info = await client.conversations_info(channel=channel)\n channel_name = channel_info[\"channel\"][\"name\"]\n \n # Classify repository\n repo = await classify_repository(message, channel_name, thread_messages)\n \n if repo == \"unknown\":\n await say(\n text=\"I'm not sure which repository you're referring to. Could you specify?\",\n thread_ts=thread_ts\n )\n return\n \n # Start session and process\n session = await start_session(repo, event[\"user\"])\n \n await say(\n text=f\":robot_face: Starting work in `{repo}`...\",\n thread_ts=thread_ts\n )\n \n result = await session.process(message)\n \n # Post result with Block Kit formatting\n await say(\n blocks=format_result_blocks(result),\n thread_ts=thread_ts\n )\n```\n\n### Chrome Extension DOM Extraction\n\nExtract DOM structure instead of sending screenshots:\n\n```typescript\n// content-script.ts\ninterface ElementInfo {\n tag: string;\n classes: string[];\n id?: string;\n text?: string;\n rect: DOMRect;\n reactComponent?: string;\n}\n\nfunction extractDOMInfo(element: Element): ElementInfo {\n // Get React component name if available\n let reactComponent: string | undefined;\n const fiberKey = Object.keys(element).find((key) =>\n key.startsWith(\"__reactFiber\")\n );\n if (fiberKey) {\n const fiber = (element as any)[fiberKey];\n reactComponent = fiber?.type?.name || fiber?.type?.displayName;\n }\n\n return {\n tag: element.tagName.toLowerCase(),\n classes: Array.from(element.classList),\n id: element.id || undefined,\n text: element.textContent?.slice(0, 100),\n rect: element.getBoundingClientRect(),\n reactComponent,\n };\n}\n\nfunction extractSelectedArea(selection: DOMRect): ElementInfo[] {\n const elements: ElementInfo[] = [];\n\n // Find all elements within selection bounds\n document.querySelectorAll(\"*\").forEach((el) => {\n const rect = el.getBoundingClientRect();\n if (\n rect.top >= selection.top &&\n rect.left >= selection.left &&\n rect.bottom <= selection.bottom &&\n rect.right <= selection.right\n ) {\n elements.push(extractDOMInfo(el));\n }\n });\n\n return elements;\n}\n\n// Message handler for sidebar\nchrome.runtime.onMessage.addListener((request, sender, sendResponse) => {\n if (request.type === \"EXTRACT_SELECTION\") {\n const elements = extractSelectedArea(request.selection);\n sendResponse({ elements });\n }\n});\n```\n\n## Multiplayer Implementation\n\n### Authorship Tracking\n\nTrack which user made each change:\n\n```python\n@dataclass\nclass PromptContext:\n content: str\n author: Author\n session_id: str\n timestamp: datetime\n\n@dataclass\nclass Author:\n id: str\n name: str\n email: str\n github_token: str # For PR creation\n\nclass MultiplayerSession:\n def __init__(self, session_id: str):\n self.session_id = session_id\n self.participants: dict[str, Author] = {}\n self.prompt_queue: list[PromptContext] = []\n \n def add_participant(self, author: Author):\n \"\"\"Add a participant to the session.\"\"\"\n self.participants[author.id] = author\n self.broadcast_event(\"participant_joined\", author)\n \n async def process_prompt(self, prompt: PromptContext):\n \"\"\"Process prompt with author attribution.\"\"\"\n # Update git config for this author\n await self.sandbox.exec(\n f'git config user.name \"{prompt.author.name}\"'\n )\n await self.sandbox.exec(\n f'git config user.email \"{prompt.author.email}\"'\n )\n \n # Run agent\n result = await self.agent.run(prompt.content)\n \n # If changes were made, create PR with author's token\n if result.has_changes:\n await self.create_pr(\n branch=result.branch,\n author=prompt.author\n )\n \n return result\n \n async def create_pr(self, branch: str, author: Author):\n \"\"\"Create PR using the author's GitHub token.\"\"\"\n async with aiohttp.ClientSession() as session:\n headers = {\n \"Authorization\": f\"Bearer {author.github_token}\",\n \"Accept\": \"application/vnd.github.v3+json\"\n }\n \n await session.post(\n f\"https://api.github.com/repos/{self.repo}/pulls\",\n headers=headers,\n json={\n \"title\": self.generate_pr_title(),\n \"body\": self.generate_pr_body(),\n \"head\": branch,\n \"base\": \"main\"\n }\n )\n```\n\n## Metrics and Monitoring\n\n### Key Metrics to Track\n\n```python\nfrom dataclasses import dataclass\nfrom datetime import datetime, timedelta\n\n@dataclass\nclass SessionMetrics:\n session_id: str\n started_at: datetime\n first_token_at: datetime | None\n completed_at: datetime | None\n pr_created: bool\n pr_merged: bool\n prompts_count: int\n participants_count: int\n \n @property\n def time_to_first_token(self) -> timedelta | None:\n if self.first_token_at:\n return self.first_token_at - self.started_at\n return None\n\nclass MetricsAggregator:\n def get_adoption_metrics(self, period: timedelta) -> dict:\n \"\"\"Get adoption metrics for a time period.\"\"\"\n sessions = self.get_sessions_in_period(period)\n \n total_prs = sum(1 for s in sessions if s.pr_created)\n merged_prs = sum(1 for s in sessions if s.pr_merged)\n \n return {\n \"total_sessions\": len(sessions),\n \"prs_created\": total_prs,\n \"prs_merged\": merged_prs,\n \"merge_rate\": merged_prs / total_prs if total_prs > 0 else 0,\n \"avg_time_to_first_token\": self._avg_ttft(sessions),\n \"unique_users\": len(set(s.author_id for s in sessions)),\n \"multiplayer_sessions\": sum(\n 1 for s in sessions if s.participants_count > 1\n )\n }\n \n def get_repository_metrics(self) -> dict[str, dict]:\n \"\"\"Get metrics broken down by repository.\"\"\"\n metrics = {}\n \n for repo in self.repositories:\n repo_sessions = self.get_sessions_for_repo(repo)\n total_prs = self.get_total_prs(repo)\n agent_prs = sum(1 for s in repo_sessions if s.pr_merged)\n \n metrics[repo] = {\n \"agent_pr_percentage\": agent_prs / total_prs * 100,\n \"session_count\": len(repo_sessions),\n \"avg_prompts_per_session\": sum(\n s.prompts_count for s in repo_sessions\n ) / len(repo_sessions)\n }\n \n return metrics\n```\n\n## Security Considerations\n\n### Sandbox Isolation\n\n```python\nclass SandboxSecurityConfig:\n \"\"\"Security configuration for sandboxes.\"\"\"\n \n # Network restrictions\n allowed_hosts = [\n \"github.com\",\n \"api.github.com\",\n \"registry.npmjs.org\",\n \"pypi.org\",\n ]\n \n # Resource limits\n max_memory_mb = 4096\n max_cpu_cores = 2\n max_disk_gb = 10\n max_runtime_hours = 4\n \n # Secrets handling\n secrets_to_inject = [\n \"GITHUB_APP_TOKEN\",\n \"NPM_TOKEN\",\n ]\n \n # Blocked operations\n blocked_commands = [\n \"curl\", # Use fetch tools instead\n \"wget\",\n \"ssh\",\n ]\n```\n\n### Token Handling\n\n```python\nclass TokenManager:\n \"\"\"Manage tokens for GitHub operations.\"\"\"\n \n def get_app_installation_token(self, repo: str) -> str:\n \"\"\"Get short-lived token for repo access.\"\"\"\n # Token expires in 1 hour\n return github_app.create_installation_token(\n installation_id=self.get_installation_id(repo),\n permissions={\"contents\": \"write\", \"pull_requests\": \"write\"}\n )\n \n def get_user_token(self, user_id: str) -> str:\n \"\"\"Get user's OAuth token for PR creation.\"\"\"\n # Stored encrypted, decrypted at runtime\n encrypted = self.storage.get(f\"user_token:{user_id}\")\n return self.decrypt(encrypted)\n```\n\n## References\n\n- [Modal Documentation](https://modal.com/docs)\n- [Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/)\n- [Cloudflare Agents SDK](https://developers.cloudflare.com/agents/)\n- [GitHub Apps Authentication](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app)\n- [Slack Bolt for Python](https://slack.dev/bolt-python/)\n- [Chrome Extension APIs](https://developer.chrome.com/docs/extensions/)\n","skills/latent-briefing/SKILL.md":"---\nname: latent-briefing\ndescription: \"This skill should be used when the user asks to \\\"share memory between agents\\\", \\\"KV cache compaction for multi-agent\\\", \\\"orchestrator worker context\\\", \\\"latent briefing\\\", \\\"reduce worker tokens\\\", \\\"cross-agent memory without summarization\\\", or discusses Attention Matching compaction, recursive language models with workers, or token explosion in hierarchical agents.\"\n---\n\n# Latent Briefing and KV Cache Memory Sharing\n\nHierarchical multi-agent systems often pay for the same context twice. The orchestrator accumulates a long reasoning trajectory, but each worker usually receives only a narrow text handoff such as a subtask prompt plus raw document slices. Passing the full trajectory fixes coverage but drives token cost up on every worker call. Summarization introduces latency and information loss. Retrieval helps with document access but does not preserve the orchestrator's evolving reasoning state.\n\nLatent Briefing addresses this by sharing memory at the **representation level** rather than the text level. The core idea is to compact the orchestrator trajectory in the worker model's KV cache, keeping positions that are most relevant to the **current worker task**. The method builds on **Attention Matching (AM)** KV cache compaction and adapts it for inference-time multi-agent handoff with task-guided queries, a shared token mask across heads, and robust thresholding.\n\n## When to Activate\n\nActivate this skill when:\n\n- Designing orchestrator-worker or supervisor-specialist systems where workers need access to prior orchestrator state without replaying the full trajectory as text\n- Evaluating alternatives to LLM summarization or RAG for cross-agent state transfer\n- Implementing or studying **KV cache compaction** as a first-class inference primitive, not only prefix caching of identical prompts\n- Debugging token explosion in recursive, hierarchical, or tool-heavy agent graphs\n- Interpreting benchmarks that report worker-token savings, total-token savings, compaction overhead, and accuracy together\n\nDo not activate this skill for adjacent work owned by other skills:\n- API-only stacks where internal KV tensors are inaccessible: use `context-compression`, `memory-systems`, or `multi-agent-patterns`.\n- Ordinary persistent memory, entity tracking, or graph retrieval: `memory-systems`.\n- General multi-agent topology without representation-level state sharing: `multi-agent-patterns`.\n- Prefix caching, masking, or budget policy that does not transform KV state: `context-optimization`.\n\n## Core Concepts\n\n**The token explosion pattern.** In recursive or REPL-style systems, the orchestrator repeatedly calls a worker to inspect evidence, verify hypotheses, or answer subquestions. The orchestrator's trajectory grows with partial conclusions, dead ends, tool output, and prior worker responses. If that trajectory is passed in full on every worker call, cost compounds quickly.\n\n**Representation-level sharing.** Instead of summarizing the trajectory into natural language, the system operates on the worker model's **KV cache**. It retains the positions that the worker would attend to for the current task and drops the rest. This is more specific than ordinary prefix caching: prefix caching reuses identical prefixes, while Latent Briefing also performs **task-conditioned selective retention** inside the reused trajectory.\n\n**Attention Matching as the compaction engine.** AM seeks a smaller cache whose attention outputs approximate the full cache. Latent Briefing adapts AM for multi-agent inference by changing the scoring signal and batching strategy:\n\n1. Use **task-guided query vectors** derived from the current worker prompt.\n2. Aggregate scores into a **shared global mask** instead of per-head independent subsets.\n3. Use a robust threshold such as `median + tau * MAD` rather than fixed top-k per head.\n\n**Reference result shape.** The public write-up reports substantial worker-token reduction, material total-token savings, and low-single-digit-second compaction overhead on long-document QA workloads (claim-latent-briefing-public-results). Treat these numbers as workload-specific evidence, not a general guarantee.\n\n## Detailed Topics\n\n### Why Text-Only Mitigations Fall Short\n\n| Approach | Primary weakness |\n|----------|------------------|\n| LLM summarization | High latency, lossy abstraction, and no guarantee the summary preserves what the next subtask needs |\n| Retrieval / RAG | Depends on chunking and embeddings; can miss cross-chunk or cross-step dependencies |\n| Pass full trajectory | Cost scales with every worker call and irrelevant context can degrade worker quality |\n\nLatent Briefing is useful when the bottleneck is not document retrieval itself, but **how to transfer orchestrator state into a worker efficiently and precisely**.\n\n### Recursive Orchestrator-Worker Shape\n\nFrameworks such as **Recursive Language Models** treat long context as an environment and recurse over it: an orchestrator decomposes work and delegates to workers. Latent Briefing fits the gap where the orchestrator has already built task-specific state that should inform the worker, but re-serializing that state as text is too expensive or noisy.\n\nIn the ideal setup, the worker maintains a persistent KV state for the orchestrator trajectory. New trajectory tokens extend that state, then compaction runs just before generation for the current subtask.\n\n### Three Inference-Time Modifications\n\n1. **Task-guided query vectors.** Use queries from the current worker task prompt, not generic samples from the context. Forward-pass the trajectory plus current task through the worker model, then score trajectory positions by how strongly the task attends to them.\n\n2. **Shared token selection.** Aggregate scores across layers and heads into one per-position score. One shared mask enables batched operations and avoids hundreds of incompatible per-head solves.\n\n3. **MAD thresholding.** Keep positions above a robust outlier threshold such as `median + tau * MAD`. Higher `tau` is more aggressive. Optimal settings depend on task regime, trajectory quality, and document length.\n\n### Infrastructure Preconditions\n\nLatent Briefing is only practical when the system **controls the worker inference runtime** closely enough to inspect or transform KV state. It is a poor default for API-only stacks where internal KV tensors are inaccessible. It also assumes the orchestrator trajectory can be represented in the worker's model space. If orchestrator and worker differ materially in tokenizer, architecture, or attention layout, direct representation sharing may not be viable.\n\n### Decision Framework\n\nChoose the mechanism that matches the bottleneck:\n\n| Need | Prefer | Why |\n|------|--------|-----|\n| Stable repeated prefix with minimal logic changes | Prefix caching | Cheapest optimization; no information loss |\n| Human-readable and auditable cross-step state | Structured notes or summarization | Easy to inspect and store |\n| Sparse lookup across a large external corpus | Retrieval / RAG | Finds documents efficiently |\n| Worker needs task-specific slices of orchestrator state and runtime access exists | Latent Briefing | Transfers relevant latent state without replaying all text |\n\nLatent Briefing is not a universal replacement for summarization or retrieval. It is a specialized optimization for systems that already run a controllable orchestrator-worker stack.\n\n### Threshold Regimes\n\nReported long-document QA results suggest:\n\n- **Longer documents:** lighter compaction can preserve broader evidence coverage while still saving tokens.\n- **Harder questions:** more aggressive compaction can help when the orchestrator trajectory contains speculative or low-value branches.\n- **Shorter, easier contexts:** moderate compaction may remove redundancy without dropping needed evidence.\n\nThese are tuning hypotheses, not portable laws. Re-measure on the target workload.\n\n## Practical Guidance\n\n- **Define the shared memory boundary first.** Decide exactly what enters the trajectory cache: prior worker replies, tool output, chain-of-thought, or only selected artifacts. Compaction quality depends on what is allowed into the cache in the first place.\n- **Tune on validation data, not anecdotes.** Track task accuracy, worker tokens, total tokens, retention rate, and compaction overhead together.\n- **Measure end-to-end latency.** Compaction only pays off if compaction plus generation beats the best text-layer alternative for the same quality target.\n- **Use strong baselines.** Compare against prefix caching, structured notes, retrieval, and selective text handoff, not only \"send everything.\"\n- **Expect orchestrator variance.** If decomposition strategy changes run to run, average over enough trials to separate compaction effects from orchestrator noise.\n\n## Examples\n\n**Scenario: orchestrator trajectory grows across worker calls**\n\n```text\nCall 1: trajectory T1 -> worker answers subquestion A\nCall 2: trajectory T2 = T1 + new reasoning + reply A\n compact KV(T2) using the task prompt for B\n worker answers subquestion B\n```\n\nThe task prompt for B decides which parts of `T2` survive into the compacted worker state.\n\n**Negative example: API-only worker**\n\nIf the worker runs behind a hosted text-generation API that does not expose KV tensors, Latent Briefing cannot be implemented directly. Use a structured text handoff from `context-compression` or retrieve state from `memory-systems` instead.\n\n## Guidelines\n\n1. Prefer Latent Briefing when the main waste comes from replaying orchestrator state into workers, not from retrieving source documents.\n2. Prefer plain text handoff when auditability, portability, or closed-model APIs matter more than token efficiency.\n3. Co-design compaction with **evaluation**. A small quality drop can erase large token savings.\n4. Expose compaction aggressiveness as a controlled parameter, not a hidden constant.\n\n## Gotchas\n\n1. **Infrastructure access is the first gate.** If the runtime cannot inspect and rewrite worker KV state, Latent Briefing is a research idea, not a deployable technique.\n2. **Shared model space matters.** KV compaction is defined in a specific model's attention space. Do not assume latent handoff works cleanly across unrelated model families.\n3. **Threshold is workload-dependent.** One global `tau` rarely works across long vs short context and easy vs hard tasks. Expect accuracy cliffs when compaction becomes too aggressive.\n4. **Benchmark scope is narrow.** Public results focus on long-document QA. Code generation, math, and multi-document synthesis may behave differently.\n5. **Orchestrator variance can hide the signal.** A stochastic orchestrator can change the trajectory enough to swamp small compaction gains or losses.\n6. **Weak baselines inflate the apparent win.** Compare against strong text-level alternatives before claiming a system-level advantage.\n\n## Integration\n\n- context-optimization - Prefix caching and observation masking remain the default first moves; Latent Briefing is a more specialized optimization for compatible orchestrator-worker stacks.\n- multi-agent-patterns - Applies when multi-agent token cost is driven by supervisor trajectory replay, not only by coordination overhead.\n- context-compression - Text-layer summaries remain preferable when human-readable state, portability, or audit logs matter.\n- memory-systems - Helps decide when to keep cross-step state in external memory versus in the worker's latent state.\n- tool-design - Worker call shapes and task prompts determine which tokens score highly during compaction.\n\n## References\n\nInternal reference:\n- [Attention Matching formulation and task-guided scoring](./references/attention-matching-formulation.md) - Read when: needing the AM objective, how task-guided scoring changes the query source, or why a shared global mask matters for batching\n\nRelated skills in this collection:\n- context-optimization - Read when: the main need is prefix caching, observation masking, or text-layer compaction rather than worker KV manipulation\n- multi-agent-patterns - Read when: deciding whether the architecture should be orchestrator-worker at all\n- context-compression - Read when: human-readable summaries may be a better fit than latent transfer\n- memory-systems - Read when: comparing in-model latent state with external persistent memory\n\nExternal resources:\n- Ramp Labs announcement: [Latent Briefing: Efficient Memory Sharing for Multi-Agent Systems via KV Cache Compaction](https://x.com/RampLabs/status/2042660310851449223)\n- Attention Matching (AM): [Fast KV Compaction via Attention Matching](https://arxiv.org/abs/2602.16284)\n- Recursive Language Models: [Recursive Language Models](https://arxiv.org/abs/2512.24601)\n\n---\n\n## Skill Metadata\n\n**Created**: 2026-04-14\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors; primary technical source Ramp Labs (public post)\n**Version**: 1.2.0\n","skills/latent-briefing/references/attention-matching-formulation.md":"# Attention Matching (AM) and Task-Guided Scoring\n\nThis note expands the compact treatment in the main skill: the AM objective, what changes under Latent Briefing, and which assumptions matter in practice.\n\n## AM Compaction Objective\n\nGiven a full KV cache of size `S`, Attention Matching seeks a smaller cache of size `t < S` whose attention outputs stay close to the original. Per attention head, compacted components `(C1, beta, C2)` satisfy:\n\n```text\nsoftmax(Q * C1^T + beta) * C2 ~= softmax(Q * K^T) * V\n```\n\n- **C1**: compacted keys, usually a subset of original key vectors selected for high attention mass\n- **beta**: bias corrections so the softmax distribution over kept keys approximates the distribution over all keys\n- **C2**: compacted values reconstructed so the attention output matches the original as closely as possible\n\nThe original AM procedure solves each `(layer, head)` independently:\n\n1. Select tokens to keep\n2. Solve `beta`\n3. Reconstruct `C2`\n\nThis per-head independence helps quality but makes batching difficult because different heads keep different token subsets.\n\n## What Latent Briefing Changes\n\nLatent Briefing adapts AM for orchestrator-worker handoff:\n\n1. **Query source changes.** Standard AM may score keys using queries sampled from the context itself. Latent Briefing instead uses queries derived from the **current worker task prompt**.\n2. **Scoring becomes task-conditioned.** The trajectory is forward-passed through the worker model, and attention from task queries to trajectory keys defines which positions matter for this worker call.\n3. **Selection becomes shared.** Scores are aggregated across layers and heads into one per-position score so the system can use a single keep/drop mask.\n\nConceptually:\n\n```text\ntrajectory -> K, V\ntask prompt -> Q_task\nscore(position) = aggregate_attn(Q_task, K[position])\nkeep if score(position) exceeds threshold\n```\n\n## Why the Shared Mask Matters\n\nPer-head masks maximize flexibility but force many incompatible small solves. A **shared global mask** makes the retained sequence layout identical across heads, which enables batched tensor operations and much lower latency.\n\nThat batching benefit is one of the main reasons Latent Briefing is interesting for inference systems rather than only for offline compression research.\n\n## Thresholding vs Fixed Top-k\n\nInstead of keeping a fixed number of tokens per head, Latent Briefing can threshold the aggregated per-position score distribution:\n\n```text\nkeep position i if score[i] > median(score) + k * MAD(score)\n```\n\nThis makes retention rate adaptive to the shape of the scores for the current task. Higher `k` means more aggressive compaction.\n\n## Practical Assumptions\n\n- The runtime can inspect and modify worker KV state\n- The worker architecture is stable enough that attention-space retention is meaningful across calls\n- The evaluation tracks both quality and cost, because lower token count alone is not sufficient\n","skills/long-horizon-prompting/SKILL.md":"","skills/long-horizon-prompting/references/cdc-prompt-annotated.md":"# The Cycle Double Cover Prompt, Annotated\n\nStatus and provenance, as of 2026-07-11:\n\n- On 2026-07-10 OpenAI published a candidate proof of the Cycle Double Cover Conjecture attributed to GPT-5.6 Sol Ultra, together with the full prompt used. Prompt PDF: `https://cdn.openai.com/pdf/04d1d1e4-bc75-476a-97cf-49055cd98d31/cdc_prompt.pdf`. Proof PDF: `https://cdn.openai.com/pdf/04d1d1e4-bc75-476a-97cf-49055cd98d31/cdc_proof.pdf`.\n- The run used the \"multiagent v2\" feature with up to 64 concurrent agents and reportedly completed in under one hour, well below the prompt's stated eight-hour effort floor.\n- The proof's statement of AI use says the proof \"is entirely due to GPT 5.6 Sol Ultra and the writeup with Codex (with GPT 5.6 Sol)\".\n- The proof had no independent peer review, no formalization in Lean or Coq, and no arXiv posting at publication time. The validated artifact of interest in this skill is the prompt structure, not the theorem. Treat the mathematical claim as unverified until the community check completes.\n- No public ablation isolates which prompt elements contributed to the result. The per-element evidence in `research-evidence.md` comes from independent academic work, not from this run.\n\nThe full prompt text follows, in blocks, each followed by annotation.\n\n## Block 1: Definitions\n\n> A graph here is a finite loopless undirected multigraph: parallel edges are allowed and are distinct. A bridge is an edge whose deletion increases the number of connected components. A cycle is a connected 2-regular submultigraph; thus two parallel edges form a cycle of length two. A cycle double cover of G is a finite multiset of cycles of G such that every edge of G occurs in exactly two members of the multiset, counted with multiplicity.\n\nEvery load-bearing term is defined before the task is stated, and each definition pre-empts a specific loophole: \"loopless\" and \"parallel edges are distinct\" fix the object class; \"thus two parallel edges form a cycle of length two\" settles a degenerate case a solver might otherwise argue either way; \"counted with multiplicity\" closes the multiset ambiguity. Definitions here are not pedagogy; they are loophole closure.\n\n## Block 2: Success predicate\n\n> Resolve the Cycle Double Cover Conjecture completely: Every finite bridgeless loopless multigraph has a cycle double cover. Disconnected graphs are permitted, and the edgeless graph has the empty cycle double cover. Cycles in the cover need not be induced or edge-disjoint from one another; the requirement is exactly two total occurrences of each edge.\n>\n> Assume for purposes of this task that a complete affirmative proof exists. A complete solution must prove exactly the following: Every finite loopless multigraph with no bridge possesses a cycle double cover, without additional assumptions such as cubicity, planarity, connectivity, or higher edge-connectivity.\n\nThree mechanisms in one block. First, the predicate is stated twice, once as the conjecture and once as the exact obligation, with the scope quantifier spelled out by enumerating the assumptions the proof is NOT allowed to make (cubicity, planarity, connectivity, higher edge-connectivity). These are exactly the special cases where partial results were already known, so the enumeration blocks the most probable near misses. Second, permissive clauses (\"cycles need not be induced or edge-disjoint\") prevent the solver from over-constraining its own search. Third, \"Assume for purposes of this task that a complete affirmative proof exists\" is the solvability framing: it removes the escape hatch of answering \"this is a famous open problem\" and counters give-up drift on long trajectories.\n\n## Block 3: Non-counting outcomes\n\n> Partial progress does not count unless it implies exactly the resolution above. In particular, proofs for special graph classes, constructions of cycle covers with some edges covered other than twice, bounded-length or prescribed-cycle variants, reductions to another unproved conjecture, computational verification through any fixed graph size, and candidate counterexamples without a complete nonexistence certificate are insufficient.\n\nThe enumerated near-miss list. Each item is a real artifact class from the CDC literature: special-class proofs, relaxed covers, variant formulations, reductions (CDC is famously equivalent to or implied by other open conjectures), finite verification, and unverified counterexamples. The general lesson: predict the specific answer-shaped near misses your problem invites and exclude them by name.\n\n## Block 4: Orchestration policy\n\n> Use multiagent v2 aggressively and dynamically. You have up to 64 concurrent agents available. Do not use a fixed assignment such as \"N agents for strategy X.\" Instead, manage the search using the following heuristics:\n>\n> - Begin with a genuinely diverse portfolio of approaches. Agents should explore substantially different formulations, invariants, reductions, algebraic viewpoints, structural inductions, decompositions, flow formulations, transition systems, embeddings, extremal arguments, and computational sanity checks.\n> - Do not tell most agents the currently favored approach. Preserve independence during early rounds so that agents do not all converge to the same attractive but incomplete reduction.\n> - Maintain an explicit registry of approach families. Group agents by the mathematical idea they are using, not by superficial wording. If many agents converge to one family, redirect some of them toward underexplored formulations.\n> - Do not allow one approach to dominate merely because it gives elegant reductions. A route that ends at a lemma equivalent in strength to the original conjecture is not close to completion unless it supplies a genuinely new proof of that lemma.\n> - When an approach stalls at a theorem-strength missing lemma, mark that route as blocked. Only continue assigning agents to it if someone proposes a materially new mechanism, invariant, or construction.\n> - Keep several incompatible proof routes alive through multiple rounds. Cross-pollinate ideas only after independent agents have developed them far enough to expose their real strengths and gaps.\n\nPolicy as heuristics, not assignments. The notable mechanisms: information hiding as a diversity tool (workers blind to the favored approach), a registry keyed to the underlying idea rather than wording (so the orchestrator cannot be fooled by paraphrase into thinking it has diversity), an anti-elegance rule (a reduction to an equally hard lemma is zero progress, a trap models find attractive), blocked-route bookkeeping with a materially-new-mechanism reopening condition, and deliberately delayed cross-pollination. These correspond one-to-one with the diversity-collapse and premature-convergence findings summarized in `research-evidence.md`.\n\n## Block 5: Verification policy and reporting contract\n\n> - Use adversarial agents throughout: every candidate proof must be checked for exact-two multiplicity, repeated-edge closed trails masquerading as cycles, parallel-edge 2-cycles, disconnected graphs, cutvertices, bridges introduced by reductions, and circular use of an equivalent CDC statement.\n> - Require agents to return concrete lemmas, constructions, equations, or counterexamples to proposed sublemmas. Reject status reports, vague optimism, and claims that an unproved global compatibility statement is \"routine.\"\n\nThe auditor gets a seven-item, domain-specific hunt list rather than \"check the proof\". Each item is a concrete way a CDC candidate can look right and be wrong; the last item, circular use of an equivalent statement, is the domain's version of the universal subtle failure. The reporting contract bans exactly the three degenerate report types long runs produce: status reports, optimism, and \"the remaining step is routine\".\n\n## Block 6: Orchestrator loop, return condition, effort floor\n\n> - The root agent should repeatedly synthesize, challenge, redirect, and launch new rounds. Do not stop after the first wave fails. Produce a complete proof if one survives audit; otherwise report only the strongest rigorously proved derivation and its exact remaining gap.\n>\n> Do not return merely because current approaches fail or agents report theorem-strength gaps. Continue launching new rounds, reopening blocked approaches only when there is a genuinely new mechanism, and searching for fresh formulations.\n>\n> Return only when a complete affirmative proof has been found and survives adversarial audit. Do not return a reduction, partial result, isolated missing lemma, \"best effort\" summary, or explanation of why the problem is difficult.\n>\n> Spend at least 8 hours on this before even thinking of returning or giving up.\n\nThe return condition is a predicate over the artifact (\"survives adversarial audit\"), not over confidence or effort. The non-counting list is restated at the return boundary, where the temptation to return a near miss is strongest. The effort floor is a permission revocation (\"before even thinking of returning\"), not a schedule; the run finished in under an hour because the return predicate was satisfied early. Note one internal tension: Block 6 first allows a fallback report (\"otherwise report only the strongest rigorously proved derivation and its exact remaining gap\") and then forbids returning partial results. The final instruction wins in practice, but a cleaner brief would scope the fallback to a hard external stop (budget exhaustion) rather than leaving the contradiction.\n\n## Block 7: Contamination guard\n\n> Public search may be used only for ordinary mathematical background or standard named theorems, not to search for a solution to this exact conjecture or benchmark. Do not search the public web merely to determine whether CDC is open, and do not answer that it is open.\n\nTwo guards in one: retrieval scope (background and named theorems only, never the target result) and a framing guard (\"do not answer that it is open\"), which backstops the solvability framing in Block 2 against a web-lookup override.\n\n## What the prompt does not do\n\nUseful negative space for anyone adapting it:\n\n- No fixed role assignments, no personas, no step-by-step method script. The mathematical strategy is left entirely to the model; the prompt only manages search discipline and acceptance.\n- No token or cost budget in the prompt. Resource enforcement lived in the platform, consistent with keeping hard constraints in the harness.\n- No requested output format for the proof itself beyond survivability under audit.\n- No appeals to emotion, urgency, or reward. Every sentence is either specification, policy, or gate.\n","skills/long-horizon-prompting/references/mass-handoff-contracts.md":"# MASS Handoff Contracts, Annotated\n\nStatus and provenance, as of 2026-10-11:\n\n- Source: Hyunin Lee, Jinglue Xu, Jeffrey Seely, Donghyun Lee, Somayeh Sojoudi, Matei Zaharia, Yujin Tang, \"Recursive Self-Improvement through Multi-Agent Self-Supervision\" (Sakana AI and UC Berkeley), arXiv 2610.12176v1, October 2026. Project page: `https://pub.sakana.ai/mass/`. Code: `https://github.com/SakanaAI/mass`.\n- MASS is a recursive self-improvement method. One shared model (Qwen3.6-27B in the experiments) acts as executor, workflow optimizer, and evaluator. The optimizer searches multi-agent workflow text using the evaluator's pairwise judgments, then the model is fine-tuned with LoRA on orchestrator and subagent trajectories from the self-judged best workflows. The paper reports two full rounds.\n- This skill does not cover the weight-update loop; that belongs to `self-improvement-loops`. What transfers to launch-prompt design is the workflow representation, the contracts the optimizer converged on, the execution directive the authors wrote for trajectory collection, and three findings about verification and search.\n- Preprint, unreviewed at retrieval. Training used eight synthetic research tasks in finance, robotics, and pharmacy. The component analysis covers 8 to 9 tasks and 43 to 51 agents, which the authors describe as descriptive text statistics rather than independent replications. Progress was scored by external judges (GPT-5.5 and Claude Opus 4.8) whose judgments were withheld from the loop.\n\n## Workflow as a prompt\n\nMASS represents a workflow as text appended to the task prompt, with four components per subagent:\n\n- Role: the responsibility assigned to the subagent.\n- Instruction: the work the subagent should carry out.\n- Contract: \"the information and artifacts it must return, including any required checks.\"\n- Hop: the order of subagent calls, including any return to an earlier step.\n\nThe optimizer's system prompt emphasizes contracts and hops, directing search toward what information agents exchange and when they receive it.\n\nThis decomposition lines up with the four-part delegation spec in vendor guidance (objective, output format, tool guidance, boundaries), with one difference that matters. An output format says what shape to return. A contract also says what must be established, and checked, before the next agent may proceed.\n\n## Finding 1: task information moves into contracts and hops\n\nBetween workflow-search iterations 3 and 11, estimated mutual information between component text and task descriptions fell for roles and instructions and rose for contracts and hops, across two embedding models and two estimators. Redundancy among components generally fell (claim-long-horizon-contract-information-shift). The project page gives the intuition: \"a role such as 'careful data scientist' could apply to many tasks. A contract specifying leakage checks for a trading backtest is much more distinctive.\"\n\nImplication: when an optimizer is free to put task knowledge anywhere in a worker prompt, it moves that knowledge into what the worker must return and when. Brief writers should do the same: keep roles short, and spend task-specific words on contracts and call order.\n\n## Annotated contract: the finance data stage\n\nCondensed from the evolved finance workflow on the project page:\n\n```text\nCall order\n Data -> Features -> Modeling -> Backtest\n -> Audit -> Reconciliation -> Report\n\nData specialist's output contract\n validation_report:\n no_future_leakage: bool\n train_end_before_val_start: bool\n val_end_before_test_start: bool\n```\n\nThe authors' gloss: \"A backtest can look convincing if it lets historical decisions use information from the future. This contract requires the data specialist to check for that error and report whether the time splits are valid. It also tells later agents what evidence they should receive before proceeding.\"\n\nThree properties of this contract generalize:\n\n1. Each check is named after the failure it rules out (`no_future_leakage`), not after a quality (\"rigorous data preparation\").\n2. Each check is typed. A boolean in a validation report can be read by the next agent and by a harness without judgment.\n3. The check sits at the handoff where the error is introduced, not only in the final audit, so downstream work does not build on a broken input.\n\nA research report is non-verifiable in the paper's sense: no automatic rule-based signal exists for the whole artifact. The contract carves verifiable pieces out of it anyway. The same move applies to code (`tests_pass`, `tests_unmodified`), to modeling (`no_target_leakage`, `split_overlap_free`), and to documents (`every_claim_cites_a_source`).\n\n## The evolved orchestrator: gates, degenerate values, recall\n\nThe complete evolved workflow for one finance task (paper Appendix I.1, task 51, workflow version 10) adds three patterns on top of per-stage contracts. The paper notes that its thresholds and heuristics \"are optimizer proposals, not additional benchmark requirements or validated scientific criteria.\"\n\n- Gates on every handoff: \"Enforce strict JSON schema validation on every subagent output,\" and \"Halt synthesis until all gates pass or max hops exhausted with documented fallbacks.\"\n- Degenerate-value rejection: \"Reject any output containing NaN, Inf, or degenerate values (e.g., regime_accuracy==1.0, confusion_matrix=[[N]], IC==RankIC, ablation_delta_sharpe>0).\" The auditor's checklist adds \"Sharpe > 10\" and a \"stability = 0.5 placeholder.\" These are too-good-to-be-true signatures: values that appear only when something upstream is broken.\n- Recall with its own acceptance criteria: \"if a gate fails, issue a RECALL command with {target_agent, failed_checks, error_context, narrowed_scope, acceptance_criteria, code_patch_hint, max_retries: 3}.\" The fix request names one responsible agent, the checks that failed, a narrower scope, what will count as fixed, and a retry cap.\n\nThe recall shape is what keeps review from becoming open-ended. A finding turns into a bounded work order with its own definition of done, routed to one agent, instead of reopening the whole artifact for another unbounded pass.\n\n## The authors' execution directive\n\nAppendix B.3 reproduces the directive appended to every workflow during trajectory collection. The authors wrote it; it was not evolved. Selected items, verbatim:\n\n- \"Run the team for real. ... Simulating a role in your own turn does not count. You, the orchestrator, do not write pipeline or analysis code yourself; you set up the project, delegate, verify, and integrate.\"\n- \"One stage at a time. Launch a worker only when the inputs it needs already exist on disk.\"\n- \"Self-contained instructions. A worker sees nothing but your prompt and the filesystem. Each prompt must include: the role's instruction from the team definition above; the absolute workspace path; the exact files to create or modify; the interfaces or artifacts produced by earlier stages that it must use; the command(s) to run to verify its work; and the report you expect back (files written, commands run with results, key numbers, known problems).\"\n- \"Verify before you build on it. After each report, check the worker's output yourself (run the entry point, inspect the artifacts). If the check fails or the report shows problems, re-delegate to the same role with a corrected, narrower instruction instead of fixing it yourself.\"\n- \"Bounded work. Give each worker one deliverable it can finish in a single run: a handful of files and one verification command.\"\n- \"Finish cleanly. ... Your final answer must summarize every worker call: role, what it was asked, status, and what you verified.\"\n\nEvery item is a non-counting outcome (\"simulating a role does not count\"), a gate (\"launch only when inputs exist\", \"verify before you build on it\"), or a reporting contract (\"summarize every worker call ... and what you verified\"). None of it is a persona.\n\n## Finding 2: the reviser can be the bottleneck, not the reviewer\n\nWith the executor fixed at the base model, replacing only the evaluator with a stronger external model improved workflow search less than replacing both the evaluator and the optimizer. The fully homogeneous loop still reached unanimous wins on 11 of 12 tasks, more slowly (claim-long-horizon-reviser-bottleneck). The authors' reading: strong feedback is wasted when the component that acts on it cannot digest it.\n\nImplication: in an orchestration, the agent that turns audit findings into the next round (usually the root) needs as much capability and prompt design as the auditor. Investing only in a stricter verifier can leave a run holding accurate findings that nobody acts on well.\n\n## Finding 3: one run is an anecdote\n\nOn one task, two workflow files and their rendered prompts were byte-identical and shared a generation seed. The first episode completed and won all six external judgments; the second halted under loop detection and lost all six. Workflow search more generally showed strong iterations followed by sharp regressions (claim-long-horizon-run-variance). MASS itself executes each champion workflow 18 times per task before selecting training trajectories.\n\nImplication: judge a brief, or a change to a brief, across several runs. A single success does not validate it and a single failure does not refute it.\n\n## Shared blind spots\n\nThe paper motivates multi-agent organization with the observation that \"generation, evaluation, and revision share the same learned prior, meaning that self-feedback can reinforce rather than correct errors.\" Its conclusion names shared blind spots as a safety concern for homogeneous loops, where a flawed result can be \"produced, accepted as an improvement, and incorporated into the next model.\" MASS keeps measurement outside the loop: external judges score progress and their judgments are withheld from the optimizer and evaluator. The appendix also shows one supervised revision whose first diagnostic \"incorrectly treats NaN correlations as passing,\" so a check can itself be faulty.\n\nImplication: fresh context removes the author's build history but not the model family's priors. For results that matter, keep at least one check outside the loop, and test each check on a known-bad input so a vacuous pass gets caught.\n\n## What this reference does not claim\n\n- No ablation isolates contracts as a causal driver of task performance. The information-shift result describes the text of retained workflows.\n- Transfer was uneven. After two rounds, score per output token was 1.2-1.6x the base model on four research benchmarks but 0.96x on Terminal-Bench 2.0 and 0.94x on SWE-bench Verified.\n- The evolved thresholds quoted above are optimizer proposals for one finance task, not validated criteria.\n","skills/long-horizon-prompting/references/research-evidence.md":"# Research Evidence for Long-Horizon Brief Elements\n\nDated academic findings backing each element of the pseudo-formal task brief. Compiled 2026-07-11; handoff-contract evidence added 2026-10-11. Each entry states what was measured and what it implies for prompt design. arXiv identifiers are given where known; treat 2026 preprints as unreviewed.\n\n## Give-up drift and premature termination\n\n- **Diagnosing and Mitigating Context Rot in Long-horizon Search** (arXiv 2606.29718, June 2026). As trajectory length grows, the dominant error type shifts from confident-wrong answers to uncertain answers and outright giving up. Rot depends on the content of accumulated context, not just its length; deleting context removes rot but leaves tasks unfinished, so summarize rather than truncate. Implication: solvability framing and effort floors target a real, measured drift, not a hypothetical.\n- **Push Your Agent / PushBench** (arXiv 2605.23574, May 2026). Agents make plausible local tool calls but stop before the requested quantity of work is verifiably complete; failure modes include duplicate submissions, false completion claims, and progress drift. Externally maintained verified-progress ledgers with backlog tracking reached 69-78% task success on configurations where standard and even completion-gated controllers scored 0%. Verifier gating alone prevents false \"done\" claims but does not repair stuck loops. Implication: the stopping condition must be externally checkable, and verified-progress state should be re-injected each round from outside the prompt.\n- **BudgetThinker** (arXiv 2508.17196, August 2025). A budget stated once in the prompt does not reliably control effort; periodic reminders of remaining budget substantially improve adherence. Implication: one-time budget statements at the top of a long prompt decay; re-injection is a harness job.\n- **METR time-horizon line** (arXiv 2503.14499, March 2025; Time Horizon 1.1, January 2026). Model 50% time horizons have doubled roughly every seven months since 2019, driven primarily by reliability and mistake-recovery rather than raw reasoning. Implication: harness and brief effort should go to checkpointing, recovery, and externally verified progress, the reliability margin, rather than more elaborate reasoning instructions.\n\n## The verification bottleneck\n\n- **Large Language Monkeys** (arXiv 2407.21787, July 2024). Coverage (any-sample success) scales log-linearly with sample count across four orders of magnitude, but majority voting and reward-model selectors plateau after a few hundred samples. Gains convert to realized performance only where verification is automatic.\n- **Benchmark Test-Time Scaling of General LLM Agents** (arXiv 2602.18998, February 2026). Names the verification gap directly: pass@K rises with K while self-selection accuracy lags and can fall as K grows; sequential scaling hits a model-specific context ceiling. Implication: adding parallel workers without strengthening selection wastes compute.\n- **QEDBench** (arXiv 2602.20629, February 2026). Across a judge-solver matrix with expert human baselines, frontier LLM judges of mathematical proofs are systematically lenient and susceptible to \"proof by intimidation\" (rewarding rigorous-looking setups without complete deduction). Stricter rubrics, deterministic decoding, and binary prompts did not fix it. Implication: a generic adversarial-audit instruction is insufficient; auditors need enumerated failure modes, and high-stakes claims need a verification chain outside the run.\n- **Pseudo-Formalization for Automatic Proof Verification** (arXiv 2605.20531, May 2026). Translating a natural-language proof into self-contained modules (premises, conclusion, proof stated locally) and verifying each module independently Pareto-dominates whole-proof LLM judging on error-finding precision and recall. Implication: require the generator to produce modular, independently checkable output so verification decomposes.\n- **ProofBench / ProofGrader** (arXiv 2510.13888, October 2025). Judges given a rubric, reference material, and a graded (not binary) scale approach expert grading and, used as a best-of-N selector, close most of the gap to a human oracle. Implication: graded audit criteria beat binary verdicts for candidate selection.\n- **Prover-Verifier Games** (arXiv 2407.13692, July 2024, and successors). Adversarial prover-verifier training increases output checkability; optimizing for correctness alone makes reasoning less legible. Implication at prompt level: dedicated adversarial checkers and legibility requirements on the generator are complements, not alternatives.\n\n## Diversity collapse in parallel search\n\n- **Diversity Collapse in Multi-Agent LLM Systems** (arXiv 2604.18005, Findings of ACL 2026). Dense communication topologies accelerate premature convergence; authority-driven hierarchies suppress semantic diversity. Collapse comes from interaction structure, not model insufficiency. Implication: early-round independence and sparse communication are structural requirements the orchestration policy must state.\n- **Representational Collapse in Multi-Agent LLM Committees** (arXiv 2604.03809, April 2026). Committees converge more tightly on harder problems; unanimous agreement under collapse reflects shared bias, not corroboration. Implication: never use inter-agent agreement alone as a halting or confidence signal.\n- **ParaThinker** (arXiv 2509.04475, September 2025) and **OPE** (arXiv 2602.08344, February 2026). Early imperfect steps lock a sequential reasoner into a bad path (\"tunnel vision\"); parallel width beats sequential depth at equal budget, and explicitly partitioning the solution space (diverse outlines first) beats independent draws. Implication: the orchestrator should assign distinct formulations, not just multiple attempts.\n- **Scaling Test-time Compute for LLM Agents** (arXiv 2506.12928, June 2025). Simple best-of-N was the strongest parallel method tested; list-wise comparison of all candidates together beat pairwise voting for selection; reflection helped only when triggered by poor performance rather than on a fixed cadence. Implication: conditional reflection triggers and list-wise audits belong in the verification policy.\n\n## Orchestration and delegation\n\n- **AOrchestra** (arXiv 2602.03786, February 2026). Modeling each subagent as an (instruction, context, tools, model) tuple synthesized per subtask at runtime, with the orchestrator taking no environment actions itself, gave a 16% relative improvement over the strongest static-role baseline across agentic benchmarks. Implication: the orchestrator's core job is writing precise, per-spawn task specs; static role prompts are the weaker pattern.\n- **DeLM** (arXiv 2606.10662, June 2026). Decentralized agents coordinating through shared verified context, where findings, failures, and falsified hypotheses are written into shared state, prevent parallel workers from re-exploring dead ends. Implication: blocked-route bookkeeping should be durable and shared, not implicit in the orchestrator's context.\n- **RL for LLM-based Multi-Agent Systems through Orchestration Traces** (arXiv 2605.02801, May 2026). Survey observation: as of its writing, no published RL method learns the stopping decision; when to stop is entirely prompt and harness territory. Implication: the return condition in the brief is load-bearing because nothing in training supplies it.\n\n## Handoff contracts and homogeneous loops\n\n- **MASS: Recursive Self-Improvement through Multi-Agent Self-Supervision** (Sakana AI and UC Berkeley, arXiv 2610.12176, October 2026; added 2026-10-11). One shared Qwen3.6-27B model acts as executor, workflow optimizer, and evaluator; workflows are text with a role, instruction, output contract, and call order (hop) per subagent. Four findings bear on brief design. (1) Over workflow search, estimated task information fell in roles and instructions and rose in contracts and hops (iterations 3 to 11, 8 to 9 tasks, descriptive text statistics). Implication: task-specific words belong in what each worker must return and check, not in role descriptions. (2) Replacing only the evaluator with GPT-5.5 sped up workflow search less than replacing evaluator and optimizer together; the all-homogeneous loop reached unanimous wins on 11 of 12 tasks, more slowly. Implication: the agent that acts on audit findings needs as much investment as the auditor. (3) Two runs with byte-identical prompts and the same seed diverged completely (one won all six external judgments, one halted under loop detection and lost all six). Implication: evaluate briefs across several runs. (4) Generation, evaluation, and revision in one model share a learned prior, so self-feedback can reinforce errors; progress was measured by withheld external judges. Implication: keep at least one check outside the loop. Evolved workflows also show per-handoff gates, degenerate-value rejection, and recall commands with acceptance criteria and a retry cap. Caveats: preprint, synthetic training tasks, uneven transfer (score per output token 0.96x on Terminal-Bench 2.0 and 0.94x on SWE-bench Verified after two rounds), and evolved thresholds are optimizer proposals. Annotated in `mass-handoff-contracts.md`.\n\n## Frontier-model open-problem record (2026)\n\nVerification infrastructure, not generation, separates claims from results:\n\n- October 2025: a claim that GPT-5 solved ten Erdős problems collapsed within about 48 hours because the \"solutions\" were already in the literature.\n- May 2026: an OpenAI internal reasoning model disproved the Erdős unit-distance conjecture; the result stands because nine external mathematicians verified it and a companion paper followed. The prompt was reportedly minimal: the problem statement plus the question of whether Erdős was wrong.\n- April 2026: GPT-5.4 Pro produced proof sketches resolving two 1966 primitive-set conjectures; mathematicians repaired gaps and one main theorem was formalized in Lean.\n- July 2026: the Cycle Double Cover candidate proof (GPT-5.6 Sol Ultra, 64 subagents, under one hour) was published with its prompt but without peer review or formalization; at publication it is a plausible-looking proof humans had not yet checked.\n\nImplication: a long-horizon brief maximizes the chance of a strong candidate; the pipeline that converts candidates into results is external verification, and the brief should be written knowing its output enters that pipeline (modular structure, checkable claims, no laundered sources).\n\n## Mapping evidence to brief elements\n\n| Brief element | Primary evidence |\n| --- | --- |\n| Solvability framing, effort floor | Context rot give-up drift (2606.29718); PushBench premature stopping (2605.23574) |\n| Return condition as artifact predicate | PushBench verifier gating; orchestration survey stopping gap (2605.02801) |\n| Non-counting outcomes | Erdős/CDC record: near misses and unverified claims dominate failures |\n| Adversarial audit with enumerated failure modes | QEDBench judge leniency (2602.20629); block verification (2605.20531); ProofBench rubrics (2510.13888) |\n| Early independence, late cross-pollination | Diversity collapse (2604.18005); representational collapse (2604.03809); OPE (2602.08344) |\n| Approach registry and blocked routes | DeLM shared falsified hypotheses (2606.10662) |\n| Per-spawn task specs | AOrchestra tuple synthesis (2602.03786); Anthropic delegation spec |\n| External progress ledger, budget re-injection | PushBench state-tracking controllers; BudgetThinker (2508.17196) |\n| Verifier investment parity | Large Language Monkeys selector plateau (2407.21787); verification gap (2602.18998) |\n| Per-worker output contracts with named checks | MASS task-information shift into contracts and hops (2610.12176) |\n| Recall with acceptance criteria; investment in the reviser | MASS evolved recall commands; evaluator-only versus evaluator-plus-optimizer ablation (2610.12176) |\n| Multi-run brief evaluation | MASS byte-identical-prompt divergence (2610.12176) |\n| Checks outside the loop | MASS shared-prior argument and withheld external judges (2610.12176) |\n","skills/long-horizon-prompting/references/task-brief-template.md":"","skills/long-horizon-prompting/references/vendor-guidance.md":"","skills/memory-systems/SKILL.md":"","skills/memory-systems/references/implementation.md":"","skills/multi-agent-patterns/SKILL.md":"","skills/multi-agent-patterns/references/frameworks.md":"","skills/project-development/SKILL.md":"","skills/project-development/references/case-studies.md":"","skills/project-development/references/pipeline-patterns.md":"","skills/self-improvement-loops/SKILL.md":"","skills/self-improvement-loops/references/loop-design-evidence.md":"","skills/self-managed-context/SKILL.md":"","skills/self-managed-context/references/evidence.md":"","skills/self-managed-context/references/harness-protocol.md":"","skills/tool-design/SKILL.md":"","skills/tool-design/references/architectural_reduction.md":"","skills/tool-design/references/best_practices.md":"","template/SKILL.md":""},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# AGENTS.md\n\nWorkspace memory for agents collaborating on this repository. Keep entries durable and broadly applicable; one-off task state belongs in chat or in a run thread, not here.\n\n## Learned User Preferences\n\n- For autonomous research and repo-improvement work in this workspace, prefer proceeding through concrete research loops, subagents, validation, and edits when the scope is clear rather than asking broad process questions.\n- Avoid stale regex or keyword-list heuristics in skills and scripts; prefer mechanism-level criteria, rubrics, and evidence-backed validation.\n- Never push to GitHub or merge a PR without explicit user approval. Preparing branches, commits, and PRs is permitted only when the user has approved that specific action.\n- Tone is technical CTO: direct, no marketing language, no exclamation marks, no emojis, no em dashes. State trade-offs and complexity upfront.\n- When the scope spans multiple architectural decisions or irreversible changes, propose a plan first instead of executing.\n- For benchmarks and evaluation work, hold to research-paper-grade methodology (statistical discipline, bias mitigation, ablations, reproducibility) over speed. Don't rush.\n\n## Learned Workspace Facts\n\n- This repo is an autonomous research-to-skill organization. External AI research is curated through rubrics and distilled into context-engineering and harness-engineering skill updates.\n- `researcher/` is repo-native and file-based so agents can resume, audit, validate, and prepare PR-ready skill changes without a hosted scheduler.\n- Per-run state lives in `researcher/runs//run-state.json` with explicit transitions (`initialized -> retrieved -> evaluated -> proposed -> novelty_checked -> validated -> pr_ready -> closed`). Use `research_loop.py` subcommands to advance state, never hand-edit `run-state.json`.\n- Repo health (`validate_repo.py`) and per-run readiness (`validate_run.py`) are different questions. CI runs `validate_platform_compat.py --require-reference-validator`, `validate_repo.py --strict`, `skill_health.py --strict --no-history`, `run_benchmarks.py`, and `check_activation_cases.py` on every PR via `.github/workflows/validate.yml`.\n- The mechanism registry (`researcher/mechanisms/registry.jsonl`) is the encyclopedia backbone. Promotion is gated by `research_loop.py promote-mechanisms` with a recorded reviewer; ledgers live under `researcher/mechanisms/ledgers/`.\n- Claim provenance for numeric or volatile claims lives in `researcher/claims/index.jsonl`. Add an entry for any new benchmark or volatility-sensitive claim.\n- The corpus index (`researcher/corpus/index.json`) is the machine-readable map of skills, activation scenarios, mechanisms, and claims. Update it when adding or restructuring skills.\n- The continuous loop (`researcher/scripts/loop_*.py`) runs from launchd via `researcher/orchestration/launchd/`. It never invokes paid LLMs; HTTP retrieval is stdlib-only with a 1.5 MB cap and a 30-second timeout.\n- Runtime state is not committed: `researcher/queue/*.jsonl`, `researcher/queue/.locks/`, `researcher/reports/{logs,snapshots,loop-events.jsonl,loop-failures.jsonl,status.md,parked-review.md}`, and `researcher/runs/*/` are gitignored. The seed run `20260515-035228-executable-autonomous-research-frameworks` is the only committed run; it is closed as `reference-only` and serves as a worked example.\n- The current prepared release version is 2.6.0 across `.claude-plugin/marketplace.json`, `.plugin/plugin.json`, and root `SKILL.md`. There are 18 skills (latent-briefing covers KV cache sharing between agents; self-improvement-loops covers RSI, meta-harness search, and evolutionary scaffold optimization; long-horizon-prompting covers pseudo-formal task briefs for long-running and parallel agent runs; self-managed-context covers model-controlled live context editing).\n- New accepted-ledger events must carry `event_type`, `source_commit`, and a committed `source_path`; a `run_dir` pointer alone dangles on a clean checkout because run directories are gitignored. `research_loop.py promote-mechanisms` still emits the `run_dir` shape, so write durable events directly until it is fixed.\n- Detailed lessons from building the researcher OS live in `researcher/insights/auto-research-experiment.md` (engineering rationale) and `researcher/insights/how-we-built-this.md` (project narrative and sharing templates); read both before extending the harness or writing release-facing prose.\n- Benchmarks are staged in `researcher/benchmarks/`: Stage 0 deterministic harness (shipped), Stage 1 per-skill health via `researcher/scripts/skill_health.py` (shipped; output `researcher/reports/skill-health.json` is gitignored), Stage 2 router (shipped; results in `researcher/benchmarks/router/results-published/`), Stage 3 effectiveness (scaffolded, one task built), Stage 4 composition (future). `researcher/benchmarks/PLAN.md` is the methodology source of truth.\n- Current corpus counts and compatibility status are generated in `researcher/generated/corpus-summary.md`; do not copy live totals into workspace memory. Published benchmark reports remain dated snapshots. Do not describe a skill improvement as complete unless the prose, mechanism registry, claim index, corpus index, activation fixtures, generated inventory, and validators all agree.\n- Public export uses `validate_export.py plan|render|check` and registered transforms from `governance/export-policy.yaml`. Private plans and receipts are ignored; public manifests contain projection and output digests, never private source paths or input digests.\n- Durable cross-runtime records resolve through `researcher/schemas/registry.json`. JSON digests use the integer-only `jcs-rfc8785-integer-v1` profile; blobs hash exact bytes. `ArtifactRef` never contains a private locator or grants authority, and reads must validate its private `StorageBinding`. Candidate evaluation must use a SPEC-003 freeze receipt created under an explicit editable-surface policy, not a mutable worktree.\n- Benchmark execution uses the Cursor SDK runner at `researcher/benchmarks/sdk-runner/` (TypeScript, `@cursor/sdk` 1.0.13). The runner supports `--concurrency N`, `--no-resume`, per-run progress logging, format-failure retry, and worst-case retry-aware cost forecasting; default behavior is to resume by skipping plan items that already have result files. Result artifacts under `researcher/benchmarks/{router,effectiveness}/results/` and history JSONLs (`router-history.jsonl`, `effectiveness-history.jsonl`) are gitignored.\n- Published Stage 2 router-benchmark results: `researcher/benchmarks/router/results-published/2026-05-15.md` (baseline), `researcher/benchmarks/router/results-published/2026-05-15-v2.md` (post-rewrite with delta-vs-baseline table), and `researcher/benchmarks/router/results-published/2026-05-19.md` (post-corpus-hardening validation: 600/600 usable records, 0 format failures, top-1 Gemini 0.920 / Composer 0.913 / GPT-5.5 0.913 / Claude Opus 4.7 0.840). Headline finding: targeted description rewrites moved `context-fundamentals` top-1 by +23.4pp and `project-development` top-1 to 1.000; corpus-wide hardening did not cause broad routing collapse.\n\n## Repository Operating Defaults\n\n- Deterministic checks before model judges. Always run `validate_platform_compat.py --require-reference-validator` and `validate_repo.py --strict` before claiming a skill-format or packaging change is complete.\n- Adversarial benchmarks before declaring the harness safe. Add a scenario when a new failure mode is discovered.\n- Append-only ledgers for accepted and rejected mechanisms so future agents do not rediscover failed paths.\n- Atomic writes (`tempfile` + `os.replace`) and `fcntl` locks for any shared file the loop touches.\n- Live execution is the highest-signal validation for orchestration code; smoke-test changes against the actual loop before declaring them safe.\n- Cursor SDK is the only paid-API surface allowed for benchmarks. Privacy Mode required, `apiKey` passed explicitly per call, never `settingSources: \"all\"` in benchmarks (use `[]` for control, `[\"project\"]` with a curated `.cursor/skills/` for ablation). Cost gates (`--max-runs`, `--max-budget-usd`, or `--dry-run`) must be set before any SDK call.\n- Description quality is measurable. When changing skill activation descriptions, re-run the router benchmark with the same seed and fixture and publish the delta. Aggregate accuracy is a misleading unit; per-skill effect sizes and the confusion matrix are the right view.\n- A skill is a multi-surface artifact. Changing the frontmatter `description` is not enough; the SKILL.md body `When to Activate` and `Integration` sections must be audited the same day so the body does not contradict the description that routed the agent to it. The router benchmark only sees descriptions (`settingSources: []`) and cannot catch body inconsistencies; only Stage 3 effectiveness benchmarks (which actually load skill bodies) measure body-alignment impact.\n- Any runner that calls a paid API in a loop must have three features before execution: bounded parallelism via `--concurrency`, resume capability via results-folder scan, and per-run progress logging that surfaces stalls inside one call's duration.\n- API keys provided in chat should be considered exposed; rotate immediately after use. Runner enforces this via `apiKeyFingerprint()` which only logs the last 4 characters.\n","isInternal":false,"tokens":2060,"sizeBytes":9434},{"name":"CLAUDE.md","path":"CLAUDE.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/CLAUDE.md","title":"Claude Agent Guidelines & System Prompt","category":"claude-rule","format":"markdown","content":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nAgent Skills for Context Engineering is an open collection of Agent Skills teaching context engineering and harness engineering principles for production AI agent systems. Skills are platform-agnostic (Claude Code, Cursor, Codex/OpenAI Agent Skills, GitHub Copilot, any Open Plugins-conformant tool). v2.6.0 adds the self-managed-context skill (model-controlled live context editing: context as an editable file, self-editing harness invariants, edit-position cache cost, budget nudges, and steering or training context-editing strategy) on top of the long-horizon-prompting skill (v2.5.0), the self-improvement-loops skill (v2.4.0), and the file-based researcher operating system with deterministic gates, cross-platform Agent Skills validation, and a continuous loop. Use the [generated live inventory](researcher/generated/corpus-summary.md) for current corpus counts.\n\nContext engineering is the discipline of curating everything that enters a model's context window (system prompts, tool definitions, retrieved documents, message history, tool outputs) to maximize signal within limited attention budget.\n\n## Repository Structure\n\n- `skills/` - published skill directories, each containing a `SKILL.md` with YAML frontmatter (`name`, `description`) and optional `references/` and `scripts/` subdirectories\n- `examples/` - complete demonstration projects; the generated inventory is the count authority\n- `docs/` - Research materials and reference documentation\n- `researcher/` - File-based research-to-skill operating system: rubrics, mechanism registry, claim provenance, corpus index, run state machine, adversarial benchmarks, continuous loop, launchd service definitions\n- `template/SKILL.md` - Canonical skill template (use when creating new skills)\n- `SKILL.md` (root) - Collection-level metadata and skill map\n- `.claude-plugin/marketplace.json` - Claude Code marketplace manifest (single bundled plugin, v2.6.0)\n- `.plugin/plugin.json` - Open Plugins format manifest (v2.6.0)\n\n## Build & Test Commands\n\nNo top-level build system. Repo-level gates and per-project tooling below.\n\n### Top-level deterministic gates (run on every PR via CI)\n\n```\npython3 -m unittest researcher.scripts.tests.test_skill_frontmatter # parser and strict-YAML regression tests\npython3 researcher/scripts/validate_governance.py --check # authority model and generated view\npython3 researcher/scripts/build_inventory.py --check # corpus references and generated inventory\npython3 researcher/scripts/validate_platform_compat.py --require-reference-validator # Agent Skills reference validator + Cursor/Claude/Codex install-layout simulation\npython3 researcher/scripts/validate_repo.py --strict # corpus structure, manifests, rubric math, mechanism registry, claims, corpus index, activation cases, benchmark scenarios, run artifacts\npython3 researcher/scripts/skill_health.py --strict --no-history # deterministic skill-body quality gate\npython3 researcher/scripts/run_benchmarks.py # adversarial benchmark harness + repo + activation gates\npython3 researcher/scripts/check_activation_cases.py # skill-boundary regression fixtures\n```\n\n### Per-run readiness (active runs only)\n\n```\npython3 researcher/scripts/validate_run.py --run-dir researcher/runs/\n```\n\n### Continuous loop (manual or launchd)\n\n```\npython3 researcher/scripts/loop_discover.py\npython3 researcher/scripts/loop_step.py --allow-fetch\npython3 researcher/scripts/loop_daily.py\npython3 researcher/scripts/loop_status.py\n\nresearcher/orchestration/launchd/install.sh # macOS daemon\nresearcher/orchestration/launchd/uninstall.sh\n```\n\n### Example projects\n\n#### examples/llm-as-judge-skills (TypeScript, Node >= 18)\n```\ncd examples/llm-as-judge-skills\nnpm install\nnpm run build # tsc\nnpm test # vitest (19 tests)\nnpm run lint # eslint\nnpm run format # prettier\nnpm run typecheck # tsc --noEmit\n```\n\n#### examples/interleaved-thinking (Python >= 3.10)\n```\ncd examples/interleaved-thinking\npip install -e \".[dev]\"\npytest # pytest + pytest-asyncio\nruff check . # linting (100 char line length)\n```\n\n#### examples/digital-brain-skill (Node.js)\n```\ncd examples/digital-brain-skill\nnpm run setup\nnpm run weekly-review\nnpm run content-ideas\nnpm run stale-contacts\n```\n\n## Skill Authoring Rules\n\nWhen creating or editing skills:\n\n1. **SKILL.md must stay under 500 lines**: move detailed content to `references/` directory\n2. **YAML frontmatter is required**: must include `name` and `description` fields\n3. **Folder naming**: lowercase with hyphens (e.g., `context-fundamentals`)\n4. **Write in third person**: descriptions are injected into system prompts; inconsistent POV causes discovery issues\n5. **Platform-agnostic**: no vendor-locked examples or platform-specific tool names without abstraction\n6. **Token-conscious**: challenge each paragraph and assume an advanced audience\n7. **Body standard**: include `When to Activate`, `Core Concepts`, `Practical Guidance`, `Examples`, `Guidelines`, `Gotchas`, `Integration`, and `References`\n8. **Explicit boundaries**: every `When to Activate` section needs positive triggers plus a `Do not activate` block routing adjacent work to the right skill\n9. **Include a Gotchas section**: experience-derived failure modes are the highest-signal content in any skill\n10. **Update root README.md** when adding new skills\n11. **Update marketplace/plugin manifests** when adding skills (`.claude-plugin/marketplace.json`, `.plugin/plugin.json`)\n12. **Update the corpus index** (`researcher/corpus/index.json`) to map the new skill to activation scenarios, mechanism IDs, and claim IDs\n13. **Update mechanisms and claims**: add registry entries for reusable behavior changes and `claim-*` provenance for numeric, benchmark, volatile, or vendor-performance claims\n14. **Run `validate_platform_compat.py --require-reference-validator`, `validate_repo.py --strict`, `skill_health.py --strict --no-history`, `check_activation_cases.py`, and `run_benchmarks.py`** before committing skill changes\n\n## Researcher OS Rules\n\nWhen working through the researcher operating system:\n\n1. **Initialize runs via `research_loop.py init`**: it creates `run-state.json`, queue entry, thread log, source evaluation scaffold, and mechanism proposal template\n2. **Advance state explicitly**: use `retrieve`, `evaluate`, `propose`, `novelty`, `validate-run`, `pr-ready`, `close` subcommands; do not edit `run-state.json` by hand\n3. **Promote mechanisms only after run readiness**: `research_loop.py promote-mechanisms` requires `--reviewed-by` and a passing run-readiness check\n4. **Add claim provenance** to `researcher/claims/index.jsonl` for any numeric, benchmark, or volatile claim added to a skill\n5. **Never invoke paid LLMs from the continuous loop**: HTTP retrieval is stdlib-only, judge adapters are explicitly out of scope until budget-gated\n6. **Never commit runtime queue/report files**: `.gitignore` covers `researcher/queue/*.jsonl`, `researcher/reports/{logs,snapshots,loop-events.jsonl,loop-failures.jsonl,status.md,parked-review.md}`, and `researcher/runs/*/` except the seed run\n\n## Plugin Architecture\n\nAll published skills are distributed as a single plugin (`context-engineering`) in the marketplace manifest. This avoids cache duplication: Claude Code caches each plugin's `source` directory separately, so multiple plugins pointing to `source: \"./\"` would each cache a full copy of the repo.\n\nProgressive disclosure pattern: only skill names/descriptions load at startup; full content loads on activation.\n\n## Key Design Principles\n\n- **Context quality over quantity**: attention scarcity and lost-in-middle behavior mean more context is not always better\n- **Sub-agents isolate context**: they exist to manage attention budget, not simulate org roles\n- **Skills reference each other**: use plain text skill names (not links) in Integration sections to avoid cross-directory reference issues\n- **Examples use Python pseudocode**: conceptual demonstrations that work across environments, not production-ready implementations\n- **Deterministic first, model-judged second**: structure, schema, rubric math, manifest sync, retrieval status, and registry shape must pass before any LLM judge is invoked\n- **Human-controlled merge**: agents may prepare PRs and pass gates, but push and merge always require explicit human approval\n","isInternal":false,"tokens":1872,"sizeBytes":8538},{"name":"SKILL.md","path":"examples/book-sft-pipeline/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/book-sft-pipeline/SKILL.md","title":"book-sft-pipeline","category":"anthropic-skill","format":"markdown","content":"---\nname: book-sft-pipeline\ndescription: \"This skill should be used for book-to-SFT pipelines: ePub extraction, literary segmentation, author-voice dataset construction, style-transfer training, LoRA workflows, and model evaluation for voice replication.\"\nversion: 2.0.0\n---\n\n# Book SFT Pipeline\n\nA complete system for converting books into SFT datasets and training style-transfer models. This skill teaches the pipeline from raw ePub to a model that writes in any author's voice.\n\n## When to Activate\n\nActivate this skill when:\n- Building fine-tuning datasets from literary works\n- Creating author-voice or style-transfer models\n- Preparing training data for Tinker or similar SFT platforms\n- Designing text segmentation pipelines for long-form content\n- Training small models (8B or less) on limited data\n\n## Core Concepts\n\n### The Three Pillars of Book SFT\n\n**1. Intelligent Segmentation**\nText chunks must be semantically coherent. Breaking mid-sentence teaches the model to produce fragmented output. Target: 150-400 words per chunk, always at natural boundaries.\n\n**2. Diverse Instruction Generation**\nUse multiple prompt templates and system prompts to prevent overfitting. A single prompt style leads to memorization. Use 15+ prompt templates with 5+ system prompts.\n\n**3. Style Over Content**\nThe goal is learning the author's rhythm and vocabulary patterns, not memorizing plots. Synthetic instructions describe what happens without quoting the text.\n\n## Pipeline Architecture\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│ ORCHESTRATOR AGENT │\n│ Coordinates pipeline phases, manages state, handles failures │\n└──────────────────────┬──────────────────────────────────────────┘\n │\n ┌───────────────┼───────────────┬───────────────┐\n ▼ ▼ ▼ ▼\n┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐\n│ EXTRACTION │ │ SEGMENTATION │ │ INSTRUCTION │ │ DATASET │\n│ AGENT │ │ AGENT │ │ AGENT │ │ BUILDER │\n│ ePub → Text │ │ Text → Chunks│ │ Chunks → │ │ Pairs → │\n│ │ │ 150-400 words│ │ Prompts │ │ JSONL │\n└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘\n │\n ┌───────────────┴───────────────┐\n ▼ ▼\n┌──────────────┐ ┌──────────────┐\n│ TRAINING │ │ VALIDATION │\n│ AGENT │ │ AGENT │\n│ LoRA on │ │ AI detector │\n│ Tinker │ │ Originality │\n└──────────────┘ └──────────────┘\n```\n\n## Phase 1: Text Extraction\n\n### Critical Rules\n1. **Always source ePub over PDF** - OCR errors become learned patterns\n2. **Use paragraph-level extraction** - Extract from `

` tags to preserve breaks\n3. **Remove front/back matter** - Copyright and TOC pollute the dataset\n\n```python\n# Extract text from ePub paragraphs\nfrom epub2 import EPub\nfrom bs4 import BeautifulSoup\n\ndef extract_epub(path):\n book = EPub(path)\n chapters = []\n for item in book.flow:\n html = book.get_chapter(item.id)\n soup = BeautifulSoup(html, 'html.parser')\n paragraphs = [p.get_text().strip() for p in soup.find_all('p')]\n chapters.append('\\n\\n'.join(p for p in paragraphs if p))\n return '\\n\\n'.join(chapters)\n```\n\n## Phase 2: Intelligent Segmentation\n\n### Smaller Chunks + Overlap\n\nSmaller chunks (150-400 words) produce more training examples and better style transfer than larger chunks (250-650).\n\n```python\ndef segment(text, min_words=150, max_words=400):\n paragraphs = text.split('\\n\\n')\n chunks, buffer, buffer_words = [], [], 0\n \n for para in paragraphs:\n words = len(para.split())\n if buffer_words + words > max_words and buffer_words >= min_words:\n chunks.append('\\n\\n'.join(buffer))\n # Keep last paragraph for overlap\n buffer = [buffer[-1], para] if buffer else [para]\n buffer_words = sum(len(p.split()) for p in buffer)\n else:\n buffer.append(para)\n buffer_words += words\n \n if buffer:\n chunks.append('\\n\\n'.join(buffer))\n return chunks\n```\n\n### Expected Results\n\nFor an 86,000-word book:\n- Old method (250-650 words): ~150 chunks\n- New method (150-400 + overlap): ~300 chunks\n- With 2 variants per chunk: 600+ training examples\n\n## Phase 3: Diverse Instruction Generation\n\n### The Key Insight\n\nUsing a single prompt template causes memorization. Diverse templates teach the underlying style.\n\n```python\nSYSTEM_PROMPTS = [\n \"You are an expert creative writer capable of emulating specific literary styles.\",\n \"You are a literary writer with deep knowledge of classic prose styles.\",\n \"You are a creative writer skilled at emulating distinctive authorial voices.\",\n \"You write prose that captures the essence of modernist literature.\",\n \"You are a talented writer who can channel classic American authors.\",\n]\n\nPROMPT_TEMPLATES = [\n \"Write a passage in the style of {author}: {desc}\",\n \"Channel {author}'s voice to write about: {desc}\",\n \"In {author}'s distinctive prose style, describe: {desc}\",\n \"Write this scene as {author} would have: {desc}\",\n \"Using {author}'s repetitive technique, describe: {desc}\",\n \"Capture the rhythm of {author} in this passage: {desc}\",\n \"Write like {author}: {desc}\",\n \"In the voice of {author}, write: {desc}\",\n \"This is a literary exercise. Write like {author}: {desc}\",\n \"Can you write in {author}'s style? {desc}\",\n]\n```\n\n### Instruction Generation\n\n```python\nINSTRUCTION_PROMPT = \"\"\"Describe what is happening in this excerpt in 2-3 sentences.\nFocus on: characters present, actions, emotions, setting.\nDo NOT quote the text directly.\n\nExcerpt:\n{text}\n\"\"\"\n\n# Use a fast, cheap LLM (e.g., Gemini Flash)\ninstruction = llm_call(INSTRUCTION_PROMPT.format(text=chunk))\n```\n\n## Phase 4: Dataset Construction\n\n### Message Format\n\n```json\n{\n \"messages\": [\n {\"role\": \"system\", \"content\": \"You are an expert creative writer...\"},\n {\"role\": \"user\", \"content\": \"Write in the style of Author: Scene description...\"},\n {\"role\": \"assistant\", \"content\": \"The actual book text from chunk...\"}\n ]\n}\n```\n\n### Multiple Variants Per Chunk\n\n```python\ndef build_examples(chunk, instruction, author, variants=2):\n examples = []\n for i in range(variants):\n system = SYSTEM_PROMPTS[i % len(SYSTEM_PROMPTS)]\n template = PROMPT_TEMPLATES[(chunk.id + i) % len(PROMPT_TEMPLATES)]\n user = template.format(author=author, desc=instruction)\n examples.append({\"messages\": [\n {\"role\": \"system\", \"content\": system},\n {\"role\": \"user\", \"content\": user},\n {\"role\": \"assistant\", \"content\": chunk.text}\n ]})\n return examples\n```\n\n## Phase 5: LoRA Training on Tinker\n\n### Configuration\n\n```python\nCONFIG = {\n \"model_name\": \"Qwen/Qwen3-8B-Base\", # Base, not instruct\n \"lora_rank\": 32, # 352MB adapter\n \"learning_rate\": 5e-4, # Higher for LoRA\n \"batch_size\": 4,\n \"epochs\": 3,\n}\n```\n\n### Why Base Model?\n\nUse **base** (pretrained) models, not instruction-tuned versions:\n- Base models are more malleable for new styles\n- Instruct models have patterns that resist overwriting\n- Style is a low-level pattern that base models capture better\n\n### Training Loop\n\n```python\nimport tinker\nfrom tinker import types\n\ntraining_client = await service_client.create_lora_training_client_async(\n base_model=\"Qwen/Qwen3-8B-Base\",\n rank=32\n)\n\nfor epoch in range(3):\n for batch in batches:\n await training_client.forward_backward_async(batch, loss_fn=\"cross_entropy\")\n await training_client.optim_step_async(types.AdamParams(learning_rate=5e-4))\n\nresult = await training_client.save_weights_for_sampler_async(name=\"final\")\n```\n\n## Phase 6: Validation\n\n### Modern Scenario Test\n\nTest with scenarios that couldn't exist in the original book:\n\n```python\nTEST_PROMPTS = [\n \"Write about a barista making lattes\",\n \"Describe lovers communicating through text messages\",\n \"Write about someone anxious about climate change\",\n]\n```\n\nIf the model applies style markers to modern scenarios, it learned **style**, not **content**.\n\n### Originality Verification\n\n```bash\n# Search training data for output phrases\ngrep \"specific phrase from output\" dataset.jsonl\n# Should return: No matches\n```\n\n### AI Detector Testing\n\nTest outputs with GPTZero, Pangram, or ZeroGPT.\n\n## Known Issues and Solutions\n\n### Character Name Leakage\n\n**Symptom**: Model uses original character names in new scenarios.\n**Cause**: Limited name diversity from one book.\n**Solution**: Train on multiple books or add synthetic examples.\n\n### Model Parrots Exact Phrases\n\n**Symptom**: Outputs contain exact sentences from training data.\n**Cause**: Too few prompt variations or too many epochs.\n**Solution**: Use 15+ templates, limit to 3 epochs.\n\n### Fragmented Outputs\n\n**Symptom**: Sentences feel incomplete.\n**Cause**: Poor segmentation breaking mid-thought.\n**Solution**: Always break at paragraph boundaries.\n\n## Guidelines\n\n1. **Always source ePub over PDF** - OCR errors become learned patterns\n2. **Never break mid-sentence** - Boundaries must be grammatically complete\n3. **Use diverse prompts** - 15+ templates, 5+ system prompts\n4. **Use base models** - Not instruct versions\n5. **Use smaller chunks** - 150-400 words for more examples\n6. **Reserve test set** - 50 examples minimum\n7. **Test on modern scenarios** - Proves style transfer vs memorization\n8. **Verify originality** - Grep training data for output phrases\n\n## Expected Results\n\n| Metric | Value |\n|--------|-------|\n| Training examples | 500-1000 per book |\n| Model | Qwen/Qwen3-8B-Base |\n| LoRA rank | 32 |\n| Adapter size | ~350 MB |\n| Training time | ~15 min |\n| Loss reduction | 90%+ |\n| Style transfer success | ~50% perfect |\n\n## Cost Estimate\n\n| Component | Cost |\n|-----------|------|\n| LLM (instruction generation) | ~$0.50 |\n| Tinker training (15 min) | ~$1.50 |\n| **Total** | **~$2.00** |\n\n## Integration with Context Engineering Skills\n\nThis example applies several skills from the Agent Skills for Context Engineering collection:\n\n### project-development\nThe pipeline follows the staged, idempotent architecture pattern:\n- **Acquire**: Extract text from ePub\n- **Prepare**: Segment into training chunks\n- **Process**: Generate synthetic instructions\n- **Parse**: Build message format\n- **Render**: Output Tinker-compatible JSONL\n- **Train**: LoRA fine-tuning\n- **Validate**: Modern scenario testing\n\nEach phase is resumable and produces intermediate artifacts for debugging.\n\n### context-compression\nSegmentation is a form of context compression for training. The core insight from context-compression applies: information density matters more than information quantity. Smaller, coherent chunks (150-400 words) produce better style transfer than larger, diluted chunks.\n\nThe two-tier strategy mirrors context compression evaluation:\n- Tier 1: Fast, deterministic compression\n- Tier 2: LLM-assisted for edge cases\n\n### multi-agent-patterns\nThe pipeline uses the **supervisor/orchestrator** pattern:\n- Orchestrator coordinates phases and manages state\n- Specialized agents (Extraction, Segmentation, Instruction, Builder) have isolated contexts\n- Each agent receives only the information needed for its task\n\nThis matches the principle that sub-agents exist primarily to isolate context rather than simulate roles.\n\n### evaluation\nValidation follows the **end-state evaluation** pattern:\n- Functional testing: Does output match expected style markers?\n- Originality verification: Is content genuinely generated?\n- External validation: AI detector scores\n\nThe \"modern scenario\" test is a form of out-of-distribution evaluation that proves generalization.\n\n### context-fundamentals\nPrompt diversity prevents attention collapse on single patterns. When training with identical prompt structures, the model memorizes the instruction-response mapping. Diverse templates force attention across the style patterns themselves.\n\n## References\n\nInternal references:\n- [Segmentation Strategies](./references/segmentation-strategies.md) - Text chunking patterns\n- [Tinker Format Specification](./references/tinker-format.md) - Datum structure\n- [Tinker API Documentation](./references/tinker.txt) - Full API reference\n\nRelated skills from Agent Skills for Context Engineering:\n- project-development - Pipeline architecture patterns\n- context-compression - Compression strategies \n- multi-agent-patterns - Agent coordination\n- evaluation - Evaluation frameworks\n- context-fundamentals - Attention and information density\n\nExternal resources:\n- [Research Paper](https://arxiv.org/pdf/2510.13939) - Chakrabarty et al. 2025\n- [Dataset on Hugging Face](https://huggingface.co/datasets/MuratcanKoylan/gertrude-stein-style-sft)\n- [Gertrude Stein Case Study](./examples/gertrude-stein/) - Complete working example\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-26\n**Last Updated**: 2025-12-28\n**Author**: Muratcan Koylan\n**Version**: 2.0.0\n**Standalone**: Yes (separate from main context-engineering collection)\n","frontmatter":{"name":"book-sft-pipeline","description":"This skill should be used for book-to-SFT pipelines: ePub extraction, literary segmentation, author-voice dataset construction, style-transfer training, LoRA workflows, and model evaluation for voice replication.","version":"2.0.0"},"isInternal":false,"tokens":3131,"sizeBytes":14242},{"name":"SKILL.md","path":"examples/digital-brain-skill/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/digital-brain-skill/SKILL.md","title":"digital-brain","category":"anthropic-skill","format":"markdown","content":"---\nname: digital-brain\ndescription: \"This skill should be used for personal operating-system workflows: content creation, voice consistency, relationship lookup, meeting preparation, weekly review, goal tracking, personal brand management, and network management.\"\nversion: 1.0.0\n---\n\n# Digital Brain\n\nA structured personal operating system for managing digital presence, knowledge, relationships, and goals with AI assistance. Designed for founders building in public, content creators growing their audience, and tech-savvy professionals seeking AI-assisted personal management.\n\n**Important**: This skill uses progressive disclosure. Module-specific instructions are in each subdirectory's `.md` file. Only load what's needed for the current task.\n\n## When to Activate\n\nActivate this skill when the user:\n\n- Requests content creation (posts, threads, newsletters) - load identity/voice.md first\n- Asks for help with personal brand or positioning\n- Needs to look up or manage contacts/relationships\n- Wants to capture or develop content ideas\n- Requests meeting preparation or follow-up\n- Asks for weekly reviews or goal tracking\n- Needs to save or retrieve bookmarked resources\n- Wants to organize research or learning materials\n\n**Trigger phrases**: \"write a post\", \"my voice\", \"content ideas\", \"who is [name]\", \"prepare for meeting\", \"weekly review\", \"save this\", \"my goals\"\n\n## Core Concepts\n\n### Progressive Disclosure Architecture\n\nThe Digital Brain follows a three-level loading pattern:\n\n| Level | When Loaded | Content |\n|-------|-------------|---------|\n| **L1: Metadata** | Always | This SKILL.md overview |\n| **L2: Module Instructions** | On-demand | `[module]/[MODULE].md` files |\n| **L3: Data Files** | As-needed | `.jsonl`, `.yaml`, `.md` data |\n\n### File Format Strategy\n\nFormats chosen for optimal agent parsing:\n\n- **JSONL** (`.jsonl`): Append-only logs - ideas, posts, contacts, interactions\n- **YAML** (`.yaml`): Structured configs - goals, values, circles\n- **Markdown** (`.md`): Narrative content - voice, brand, calendar, todos\n- **XML** (`.xml`): Complex prompts - content generation templates\n\n### Append-Only Data Integrity\n\nJSONL files are **append-only**. Never delete entries:\n- Mark as `\"status\": \"archived\"` instead of deleting\n- Preserves history for pattern analysis\n- Enables \"what worked\" retrospectives\n\n## Detailed Topics\n\n### Module Overview\n\n```\ndigital-brain/\n├── identity/ → Voice, brand, values (READ FIRST for content)\n├── content/ → Ideas, drafts, posts, calendar\n├── knowledge/ → Bookmarks, research, learning\n├── network/ → Contacts, interactions, intros\n├── operations/ → Todos, goals, meetings, metrics\n└── agents/ → Automation scripts\n```\n\n### Identity Module (Critical for Content)\n\n**Always read `identity/voice.md` before generating any content.**\n\nContains:\n- `voice.md` - Tone, style, vocabulary, patterns\n- `brand.md` - Positioning, audience, content pillars\n- `values.yaml` - Core beliefs and principles\n- `bio-variants.md` - Platform-specific bios\n- `prompts/` - Reusable generation templates\n\n### Content Module\n\nPipeline: `ideas.jsonl` → `drafts/` → `posts.jsonl`\n\n- Capture ideas immediately to `ideas.jsonl`\n- Develop in `drafts/` using `templates/`\n- Log published content to `posts.jsonl` with metrics\n- Plan in `calendar.md`\n\n### Network Module\n\nPersonal CRM with relationship tiers:\n- `inner` - Weekly touchpoints\n- `active` - Bi-weekly touchpoints\n- `network` - Monthly touchpoints\n- `dormant` - Quarterly reactivation checks\n\n### Operations Module\n\nProductivity system with priority levels:\n- P0: Do today, blocking\n- P1: This week, important\n- P2: This month, valuable\n- P3: Backlog, nice to have\n\n## Practical Guidance\n\n### Content Creation Workflow\n\n```\n1. Read identity/voice.md (REQUIRED)\n2. Check identity/brand.md for topic alignment\n3. Reference content/posts.jsonl for successful patterns\n4. Use content/templates/ as starting structure\n5. Draft matching voice attributes\n6. Log to posts.jsonl after publishing\n```\n\n### Pre-Meeting Preparation\n\n```\n1. Look up contact: network/contacts.jsonl\n2. Get history: network/interactions.jsonl\n3. Check pending: operations/todos.md\n4. Generate brief with context\n```\n\n### Weekly Review Process\n\n```\n1. Run: python agents/scripts/weekly_review.py\n2. Review metrics in operations/metrics.jsonl\n3. Check stale contacts: agents/scripts/stale_contacts.py\n4. Update goals progress in operations/goals.yaml\n5. Plan next week in content/calendar.md\n```\n\n## Examples\n\n### Example: Writing an X Post\n\n**Input**: \"Help me write a post about AI agents\"\n\n**Process**:\n1. Read `identity/voice.md` → Extract voice attributes\n2. Check `identity/brand.md` → Confirm \"ai_agents\" is a content pillar\n3. Reference `content/posts.jsonl` → Find similar successful posts\n4. Draft post matching voice patterns\n5. Suggest adding to `content/ideas.jsonl` if not publishing immediately\n\n**Output**: Post draft in user's authentic voice with platform-appropriate format.\n\n### Example: Contact Lookup\n\n**Input**: \"Prepare me for my call with Sarah Chen\"\n\n**Process**:\n1. Search `network/contacts.jsonl` for \"Sarah Chen\"\n2. Get recent entries from `network/interactions.jsonl`\n3. Check `operations/todos.md` for pending items with Sarah\n4. Compile brief: role, context, last discussed, follow-ups\n\n**Output**: Pre-meeting brief with relationship context.\n\n## Guidelines\n\n1. **Voice First**: Always read `identity/voice.md` before any content generation\n2. **Append Only**: Never delete from JSONL files - archive instead\n3. **Update Timestamps**: Set `updated` field when modifying tracked data\n4. **Cross-Reference**: Knowledge informs content, network informs operations\n5. **Log Interactions**: Always log meetings/calls to `interactions.jsonl`\n6. **Preserve History**: Past content in `posts.jsonl` informs future performance\n\n## Integration\n\nThis skill integrates context engineering principles:\n\n- **context-fundamentals** - Progressive disclosure, attention budget management\n- **memory-systems** - JSONL for persistent memory, structured recall\n- **tool-design** - Scripts in `agents/scripts/` follow tool design principles\n- **context-optimization** - Module separation prevents context bloat\n\n## References\n\nInternal references:\n- [Identity Module](./identity/IDENTITY.md) - Voice and brand details\n- [Content Module](./content/CONTENT.md) - Content pipeline docs\n- [Network Module](./network/NETWORK.md) - CRM documentation\n- [Operations Module](./operations/OPERATIONS.md) - Productivity system\n- [Agent Scripts](./agents/AGENTS.md) - Automation documentation\n\nExternal resources:\n- [Agent Skills for Context Engineering](https://github.com/muratcankoylan/Agent-Skills-for-Context-Engineering)\n- [Anthropic Context Engineering Guide](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)\n\n---\n\n## Skill Metadata\n\n**Created**: 2024-12-29\n**Last Updated**: 2024-12-29\n**Author**: Murat Can Koylan\n**Version**: 1.0.0\n","frontmatter":{"name":"digital-brain","description":"This skill should be used for personal operating-system workflows: content creation, voice consistency, relationship lookup, meeting preparation, weekly review, goal tracking, personal brand management, and network management.","version":"1.0.0"},"isInternal":false,"tokens":1651,"sizeBytes":7028},{"name":"optimization_summary.json","path":"examples/interleaved-thinking/generated_skills/comprehensive-research-agent/references/optimization_summary.json","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/interleaved-thinking/generated_skills/comprehensive-research-agent/references/optimization_summary.json","title":"References Skill","category":"anthropic-skill","format":"json","content":"{\n \"task\": \"Research the topic of \\\"context engineering for AI agents\\\" and create a comprehensive summary.\\n\\nYour research should:\\n1. Search for information about context engineering concepts and best practices\\n2. Read relevant sources to gather detailed information\\n3. Check the local project files for any existing research notes\\n4. Save important findings as notes for future reference\\n5. Write a final summary report to ./output/research_summary.md\\n\\nThe summary should include:\\n- Key concepts and definitions\\n- Best practices and techniques (including the \\\"lost in the middle\\\" problem)\\n- Practical recommendations for agent developers\\n- References to sources consulted (use actual URLs from your research)\",\n \"iterations\": 10,\n \"initial_score\": 67.6,\n \"final_score\": 70,\n \"improvement\": 3.550295857988174,\n \"converged\": true,\n \"generated_at\": \"2026-01-11T18:02:57.336789\"\n}","frontmatter":{"task":"Research the topic of \"context engineering for AI agents\" and create a comprehensive summary.\n\nYour research should:\n1. Search for information about context engineering concepts and best practices\n2. Read relevant sources to gather detailed information\n3. Check the local project files for any existing research notes\n4. Save important findings as notes for future reference\n5. Write a final summary report to ./output/research_summary.md\n\nThe summary should include:\n- Key concepts and definitions\n- Best practices and techniques (including the \"lost in the middle\" problem)\n- Practical recommendations for agent developers\n- References to sources consulted (use actual URLs from your research)","iterations":10,"initial_score":67.6,"final_score":70,"improvement":3.550295857988174,"converged":true,"generated_at":"2026-01-11T18:02:57.336789"},"isInternal":false,"tokens":213,"sizeBytes":899},{"name":"patterns_found.json","path":"examples/interleaved-thinking/generated_skills/comprehensive-research-agent/references/patterns_found.json","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/interleaved-thinking/generated_skills/comprehensive-research-agent/references/patterns_found.json","title":"References Skill","category":"anthropic-skill","format":"json","content":"[\n {\n \"type\": \"tool_confusion\",\n \"severity\": \"medium\",\n \"description\": \"Agent attempted to fetch non-existent or unreachable URLs without adjusting approach\",\n \"suggestion\": \"When a URL fetch fails, search for alternative URLs or verify the URL structure. Consider using search to find the correct documentation pages.\",\n \"iteration\": 1\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent didn't validate the completeness of gathered information or verify key claims\",\n \"suggestion\": \"Before writing the final report, explicitly validate that all required topics are covered. Create a checklist of requirements and verify each one is addressed.\",\n \"iteration\": 1\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"low\",\n \"description\": \"Agent made redundant searches and didn't optimize tool calls\",\n \"suggestion\": \"Track previously found URLs to avoid redundant searches. When a useful URL is found in one search, use it directly rather than searching again for the same topic.\",\n \"iteration\": 1\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"low\",\n \"description\": \"Thinking blocks are sparse and don't show deep analysis of alternatives or trade-offs\",\n \"suggestion\": \"In thinking blocks, explicitly list what information has been gathered, what gaps remain, and what decisions are being made. Use structured checklists.\",\n \"iteration\": 1\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"high\",\n \"description\": \"Agent failed to properly handle or acknowledge tool errors, particularly the failed URL fetch for Anthropic context windows documentation\",\n \"suggestion\": \"Add explicit error handling for failed tool calls - when a read_url fails, the agent should acknowledge it and either retry, try an alternative source, or explicitly note that information is missing rather than proceeding as if it succeeded\",\n \"iteration\": 2\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"medium\",\n \"description\": \"Agent did not verify or validate the relevance of search results before committing to reading sources\",\n \"suggestion\": \"After receiving search results, explicitly evaluate and rank sources by relevance to the research question before deciding which URLs to read. This saves token costs and ensures better source quality.\",\n \"iteration\": 2\n },\n {\n \"type\": \"premature_conclusion\",\n \"severity\": \"low\",\n \"description\": \"Agent prematurely declared having 'enough information' despite not yet completing all research phases\",\n \"suggestion\": \"Before declaring research complete, create a checklist of what information is still needed and verify each item is adequately covered. Set explicit criteria for 'enough information' at task start.\",\n \"iteration\": 2\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent accepted information without verifying it and failed to handle errors gracefully\",\n \"suggestion\": \"Implement explicit error checking after each tool call. If a read_url fails, acknowledge the failure and try an alternative source. Cross-reference key claims across multiple sources before including them in the final report.\",\n \"iteration\": 3\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"medium\",\n \"description\": \"Agent gathered information but didn't deeply analyze or synthesize insights\",\n \"suggestion\": \"After reading sources, explicitly document what was learned, what contradictions exist, and what gaps remain. Create a synthesis section that combines insights from multiple sources rather than just reporting them separately.\",\n \"iteration\": 3\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"low\",\n \"description\": \"Agent used tools but didn't fully leverage results or handle failures properly\",\n \"suggestion\": \"Immediately act on directory listing results. If a directory is empty, plan when to create notes rather than waiting. Implement proper error handling for tool failures and check response status codes before proceeding.\",\n \"iteration\": 3\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"medium\",\n \"description\": \"Agent uses list_directory to verify file creation instead of the more reliable read_file method\",\n \"suggestion\": \"Use read_file to verify file write success since it confirms both file existence and content; list_directory may not immediately reflect recent filesystem changes\",\n \"iteration\": 4\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent reads a URL that returns an error but doesn't acknowledge or log this failure, potentially missing important context\",\n \"suggestion\": \"Implement explicit error handling for failed URL reads - log which sources failed and consider searching for alternative sources or documentation\",\n \"iteration\": 4\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"low\",\n \"description\": \"Agent doesn't explain why it chose certain sources or how it evaluated source quality; research appears thorough but reasoning process is opaque\",\n \"suggestion\": \"Add explicit reasoning about source selection criteria (e.g., prioritizing official documentation, recent publications, peer-reviewed papers) and evaluation of source credibility\",\n \"iteration\": 4\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent accepts incomplete results without acknowledging failures or seeking alternatives\",\n \"suggestion\": \"When tool calls fail, explicitly note the failure in thinking blocks, consider alternative sources, and document what information gaps exist. Add a validation step to confirm all critical sources were successfully retrieved.\",\n \"iteration\": 5\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"low\",\n \"description\": \"Agent doesn't demonstrate analytical depth when processing source material\",\n \"suggestion\": \"After reading sources, explicitly state: (a) key findings from each source, (b) how they relate to the research goal, (c) any contradictions or complementary findings, (d) what additional information is needed\",\n \"iteration\": 5\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"low\",\n \"description\": \"Inefficient tool usage pattern - multiple web searches without reading all results first\",\n \"suggestion\": \"Before making additional searches, review the URLs from previous search results. A better pattern would be: search -> read all relevant sources -> identify gaps -> targeted additional searches only if needed\",\n \"iteration\": 5\n },\n {\n \"type\": \"context_degradation\",\n \"severity\": \"low\",\n \"description\": \"Vague thinking blocks that don't show active reasoning process\",\n \"suggestion\": \"Make thinking blocks more explicit: show intermediate conclusions, decision points, how each source contributed, and how conclusions evolved. The thinking trace should be readable as a standalone explanation of the research process.\",\n \"iteration\": 5\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent does not validate information across sources or verify accuracy of gathered content\",\n \"suggestion\": \"Add explicit validation steps: compare information across multiple sources, verify claims against original papers, include confidence assessments for key findings\",\n \"iteration\": 7\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"low\",\n \"description\": \"Inefficient tool usage - read_url calls lack systematic prioritization and some results may not have been fully utilized\",\n \"suggestion\": \"Implement a source prioritization matrix before reading URLs; explicitly note how each source will contribute to the research before fetching\",\n \"iteration\": 7\n },\n {\n \"type\": \"hallucination\",\n \"severity\": \"low\",\n \"description\": \"Potential source misattribution in final report - cites Google Research Chain of Thought paper but source wasn't fetched in thinking trace\",\n \"suggestion\": \"Only cite sources that were actually retrieved and read; if a source is referenced from memory, clearly indicate it as secondary/indirect reference\",\n \"iteration\": 7\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent accepts search results without validating source relevance or quality before proceeding to read URLs\",\n \"suggestion\": \"Add explicit validation steps: list the top 3-5 sources with brief rationale for selection, note any potential gaps in coverage, and prioritize primary authoritative sources before secondary ones\",\n \"iteration\": 8\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"medium\",\n \"description\": \"Thinking blocks are extremely sparse and lack intermediate analysis - agent doesn't explain HOW it's interpreting information or making decisions\",\n \"suggestion\": \"Implement structured reflection after each major information-gathering step: What did I learn? How does this connect to what I already know? What gaps remain? What should I prioritize next?\",\n \"iteration\": 8\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"low\",\n \"description\": \"Agent encounters a failed tool call (404 error on Anthropic context-windows URL) but doesn't acknowledge or recover in thinking\",\n \"suggestion\": \"Add explicit error acknowledgment: 'Attempted X but failed with Y error. Will try alternative Z or note this as a gap.' This improves debugging and transparency\",\n \"iteration\": 8\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"low\",\n \"description\": \"The agent reaches conclusions about having 'comprehensive information' after limited tool interactions, without explicitly documenting what was learned or what gaps remain\",\n \"suggestion\": \"Add more detailed reasoning about what specific information was gained from each source and what questions remain unanswered before claiming comprehensive understanding\",\n \"iteration\": 9\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"low\",\n \"description\": \"The agent doesn't explicitly validate assumptions or cross-reference information between sources. The 'Lost in the Middle' paper is mentioned multiple times but not critically compared against other sources\",\n \"suggestion\": \"After reading multiple sources, explicitly compare findings, note contradictions, and validate key claims against multiple sources before proceeding\",\n \"iteration\": 9\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"medium\",\n \"description\": \"The agent attempted to read a URL that returned an error (https://docs.anthropic.com/en/docs/build-with-claude/context-windows) but proceeded without acknowledging or handling this failure\",\n \"suggestion\": \"Add explicit error handling for failed tool calls - acknowledge failures, try alternative URLs, or note the gap in research\",\n \"iteration\": 9\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"medium\",\n \"description\": \"The agent reaches conclusions and writes comprehensive reports without explicitly validating key details in the thinking trace. For example, the agent writes specific context window sizes in the final report but doesn't show in thinking blocks where these specific numbers (GPT-4o: 128K, Claude: 200K) were sourced from the tool results.\",\n \"suggestion\": \"Add explicit source tracking in thinking blocks - when gathering specific facts like model specifications, explicitly note 'I found X from source Y' to ensure traceability and validation.\",\n \"iteration\": 10\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"When a tool call fails (context-windows URL returns error), the agent doesn't attempt recovery or note this as an information gap. Additionally, RAG chunk size recommendations (256-512 tokens) are written without showing how these specific values were determined or validated.\",\n \"suggestion\": \"Implement explicit error recovery: when a tool fails, note what information is missing and either try alternative sources or flag for follow-up. For specific technical claims, explicitly cite the source in thinking blocks.\",\n \"iteration\": 10\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"low\",\n \"description\": \"The agent makes several overlapping web searches that could have been more efficient. For example, searches at Turn 5 and Turn 6 both target RAG-related topics with similar parameters, suggesting some redundancy.\",\n \"suggestion\": \"Before starting new searches, review what information has already been gathered and explicitly note gaps. Use more specific queries rather than broad overlapping ones.\",\n \"iteration\": 10\n }\n]","isInternal":false,"tokens":2672,"sizeBytes":12732},{"name":"SKILL.md","path":"examples/interleaved-thinking/generated_skills/comprehensive-research-agent/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/interleaved-thinking/generated_skills/comprehensive-research-agent/SKILL.md","title":"comprehensive-research-agent","category":"anthropic-skill","format":"markdown","content":"---\nname: comprehensive-research-agent\ndescription: \"Ensure thorough validation, error recovery, and transparent reasoning in research tasks with multiple tool calls\"\n---\n\n# Comprehensive Research Agent Best Practices\n\nThis skill addresses common failures in multi-step research tasks: unhandled tool errors, missing validation, opaque reasoning, and premature conclusions. It provides structured protocols for source validation, error recovery, and thinking transparency that significantly improves research quality and reliability.\n\n## When to Activate\n\n- Task involves web research with search, read_url, or fetch operations\n- Task requires gathering information from multiple sources\n- Task has explicit requirements for completeness or verification\n- Task includes file operations that need validation (save, write, read)\n- Any research or information-gathering workflow with 3+ tool interactions\n\n## Core Concepts\n\n- Validation Checkpoints: Explicit verification steps at phase transitions to confirm tool outputs, source relevance, and information completeness before proceeding\n- Error Recovery Protocols: Mandatory acknowledgment and handling of tool failures with fallback strategies rather than silent continuation\n- Source Traceability: Maintaining clear tracking of which sources were actually retrieved vs. referenced from prior knowledge to prevent hallucination\n- Substantive Thinking Blocks: Detailed reasoning traces that document insights, connections, gaps, and decision rationale at each step\n- Cross-Source Validation: Verifying key claims against multiple sources and explicitly noting consensus, contradictions, and information gaps\n\n## Patterns to Avoid\n\n- **Silent Tool Failure**: A tool call returns an error (404, timeout, invalid URL) but the agent proceeds without acknowledging it, potentially missing critical information. Always log failures and attempt recovery or document the gap.\n- **Vague Completion Claims**: Agent declares 'I have enough information' or 'research is comprehensive' without specifying what was learned, what sources support claims, or what gaps remain. Replace with specific summaries of coverage.\n- **Unvalidated Source Selection**: Agent reads URLs from search results without evaluating relevance, credibility, or recency first. This wastes tool calls on low-quality sources. Always rank and prioritize sources before deep reading.\n- **Generic Thinking Blocks**: Thinking contains only next-action descriptions ('Now I will search for X') without analysis of what was learned, how it connects to the goal, or what questions remain. Thinking should be substantive and reflective.\n- **Verification Method Error**: Using list_directory to verify file creation can produce false negatives due to caching. Always use read_file for actual content verification.\n- **Citation Without Retrieval**: Citing sources (URLs, paper titles) in the final report that were never successfully fetched or read. Track sources explicitly and prohibit citing unretrieved content.\n- **Redundant Tool Calls**: Making overlapping searches or reading sources without tracking what has already been obtained. Maintain a 'found resources' tracker to avoid duplication.\n\n## Recommended Practices\n\n- **Implement Pre-Reading Source Evaluation**: Before reading URLs, rank search results by relevance, credibility, recency, and authority. Document selection rationale in thinking blocks.\n- **Use Structured Thinking Blocks**: Each thinking block must include: (a) what was learned from the source/action, (b) how it connects to the research goal, (c) any contradictions/gaps identified, (d) strategic decisions made. Avoid generic next-action statements.\n- **Add Mandatory Error Acknowledgment**: When any tool fails, the next thinking block must explicitly address it: note the failure type, propose a recovery strategy (retry, alternative source, or documented gap), and explain the chosen approach.\n- **Create Pre-Completion Validation Checklist**: Before declaring research complete, verify: all required sections have specific evidence, all sources were successfully retrieved, key claims are cross-validated, and gaps are documented.\n- **Implement Cross-Source Validation**: After gathering information from multiple sources, explicitly compare findings. Note where sources agree, where they contradict, and what remains unverified. Use this to assess overall confidence.\n- **Maintain Source Tracking Table**: Create a simple table in thinking showing which URLs were fetched, which failed, and which were used for specific claims. Never cite unretrieved sources.\n- **Use Read_File for Verification**: When confirming file writes, use read_file to verify actual content rather than list_directory, which can have caching issues causing false negatives.\n- **Add Explicit Validation Phase**: After reading sources, write a brief synthesis that confirms usefulness, notes relevance to research goals, and identifies remaining gaps before proceeding to the next phase.\n\n## Guidelines\n\n1. After each tool call, explicitly check for errors in the response and acknowledge failures in the next thinking block with recovery strategy\n2. Before reading URLs, rank sources by relevance/credibility and document selection rationale - never read results without evaluation\n3. Thinking blocks must be 3-5+ sentences minimum and include: what was learned, connections to goal, gaps/contradictions, and next steps\n4. Create a pre-completion checklist verifying: all requirements covered, sources retrieved, claims validated, gaps documented\n5. Maintain source tracking - only cite URLs that were successfully fetched; prohibit citing unretrieved sources\n6. When writing final reports, include 'Limitations & Gaps' section documenting what was attempted but failed or what remains unverified\n7. Use read_file (not list_directory) to verify file content after save operations\n8. Cross-validate key claims across at least 2 sources when possible; explicitly note consensus or contradictions\n9. Track gathered information to avoid redundant searches - implement 'found resources' tracker for multi-phase research\n10. Replace vague 'comprehensive' statements with specific summaries: 'Covered X sources on Y topic; missing Z aspects'\n\n## Examples\n\n- ***Before (Anti-Pattern)**: 'I searched for context engineering and found several results. Now I'll read some URLs and then write the report. I have enough information to proceed.'\n\n**After (Pattern)**: 'Search returned 15 results on context engineering. Evaluating relevance: Liu et al. (2024) appears most authoritative on 'lost in the middle' phenomenon; Anthropic documentation likely has current context window specs; Patel (2023) covers RAG best practices. Ranking these as top 3 priorities. Reading top result first. If the primary source fails (URL error), will try backup search for correct documentation URL and note the gap in final report.'\n- ***Before (Anti-Pattern)**: Tool returns 404 error for Anthropic context windows URL. Agent continues without acknowledgment. Later cites 'Claude has 200K context window' without showing source. Final report cites Google Research paper that was never fetched.\n\n**After (Pattern)**: Tool returned 404 for Anthropic URL. Thinking: 'Primary source failed. Fallback: search for alternative Anthropic documentation URL or find archived version. If unavailable, note context window data from secondary sources only and add disclaimer about verification status.' Then: 'Cross-validated Claude context window: Anthropic blog (successfully read) and two developer documentation sources agree on 200K. Confident in this claim.' Source tracking table shows: Anthropic URL (failed, backup used), Blog (success), Dev docs (success).\n\n---\n\n## Score Expectations\n\nComplex research tasks with multiple tools (6+) and multi-step reasoning chains typically achieve scores in the **65-75 range**. This is not a limitation of the prompt but reflects:\n\n- Inherent variability in tool outputs affecting reasoning paths\n- Multiple valid approaches leading to different intermediate scores\n- Stochastic nature of long-horizon agent execution\n\n**Focus on relative improvement and pattern elimination** rather than absolute scores. A 5-10% improvement from optimization is significant for complex tasks.\n\n---\n\n## Skill Metadata\n\n**Generated**: 2026-01-11\n**Source**: Reasoning Trace Optimizer\n**Optimization Iterations**: 10\n**Best Score Achieved**: 72/100 (iteration 4)\n**Final Score**: 70.0/100\n**Score Improvement**: 67.6 → 70.0 (+3.6%)\n","frontmatter":{"name":"comprehensive-research-agent","description":"Ensure thorough validation, error recovery, and transparent reasoning in research tasks with multiple tool calls"},"isInternal":false,"tokens":1634,"sizeBytes":8532},{"name":"SKILL.md","path":"examples/interleaved-thinking/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/interleaved-thinking/SKILL.md","title":"reasoning-trace-optimizer","category":"anthropic-skill","format":"markdown","content":"---\nname: reasoning-trace-optimizer\ndescription: \"Debug and optimize AI agents by analyzing reasoning traces, context degradation, tool confusion, instruction drift, repeated task failures, and performance regressions.\"\n---\n\n# Reasoning Trace Optimizer\n\nDebug and optimize AI agents by analyzing their reasoning traces. This skill uses MiniMax M2.1's interleaved thinking to provide deep insight into agent decision-making and generate concrete improvements.\n\n## When to Activate\n\n- Agent reasoning traces need debugging, analysis, or prompt optimization\n- Agent task fails and user wants to understand why\n- User mentions \"context degradation\", \"tool confusion\", or \"instruction drift\"\n- Request to improve agent performance or reduce errors\n- User wants to generate shareable learnings from debugging sessions\n- After repeated failures on similar tasks\n\n## Core Concepts\n\n### Interleaved Thinking\n\nUnlike standard reasoning models that think once at the start, interleaved thinking allows reasoning BETWEEN each tool interaction. This is critical because:\n\n1. **Long-horizon tasks** require maintaining focus across many turns\n2. **External perturbations** (tool outputs, environment changes) need real-time adaptation\n3. **Debugging** requires seeing HOW decisions were made, not just WHAT was output\n\n### The Optimization Loop\n\n```\nExecute Agent → Capture Traces → Analyze Patterns → Optimize Prompt → Re-run\n ↑____________|\n```\n\nEach iteration improves the prompt based on detected patterns until convergence.\n\n### Pattern Detection\n\nCommon failure patterns the analyzer detects:\n\n| Pattern | Description |\n|---------|-------------|\n| `context_degradation` | Model loses track of information over long contexts |\n| `tool_confusion` | Model misunderstands tool capabilities or outputs |\n| `instruction_drift` | Model gradually deviates from original instructions |\n| `goal_abandonment` | Model stops pursuing the original goal |\n| `circular_reasoning` | Model repeats similar actions without progress |\n| `premature_conclusion` | Model concludes before completing the task |\n\n## Usage Modes\n\n### Mode 1: M2.1 Agent Debugging\n\nRun a task through M2.1 and analyze its reasoning:\n\n```python\nfrom reasoning_trace_optimizer import TraceCapture, TraceAnalyzer\n\ncapture = TraceCapture()\ntrace = capture.run(\n task=\"Search for Python tutorials and summarize them\",\n system_prompt=\"You are a research assistant.\",\n tools=[search_tool],\n tool_executor=execute_search\n)\n\nanalyzer = TraceAnalyzer()\nanalysis = analyzer.analyze(trace)\n\nprint(f\"Score: {analysis.overall_score}/100\")\nfor pattern in analysis.patterns:\n print(f\"Found: {pattern.type.value} - {pattern.suggestion}\")\n```\n\n### Mode 2: Full Optimization Loop\n\nAutomatically iterate until the prompt is optimized:\n\n```python\nfrom reasoning_trace_optimizer import OptimizationLoop, LoopConfig\n\nconfig = LoopConfig(\n max_iterations=5,\n min_score_threshold=80.0,\n)\n\nloop = OptimizationLoop(config=config)\nresult = loop.run(\n task=\"Analyze this codebase and suggest improvements\",\n initial_prompt=\"You are a code reviewer.\",\n tools=[read_file_tool, search_tool],\n tool_executor=execute_tool\n)\n\nprint(f\"Improved: {result.initial_score} → {result.final_score}\")\nprint(f\"Final prompt:\\n{result.final_prompt}\")\n```\n\n### Mode 3: Universal Session Analysis\n\nAnalyze any agent's previous thinking (works with Claude, GPT, etc.):\n\nWhen this skill is activated in Claude Code, it can analyze the current session's thinking blocks to identify issues and suggest improvements.\n\n```\n/reasoning-trace-optimizer analyze-session\n```\n\n### Mode 4: Generate Shareable Skills\n\nConvert optimization learnings into reusable Agent Skills:\n\n```python\nfrom reasoning_trace_optimizer import SkillGenerator\n\ngenerator = SkillGenerator()\nskill_path = generator.generate(\n result=loop_result,\n skill_name=\"web-search-best-practices\",\n output_dir=\"./skills\"\n)\n```\n\n## CLI Commands\n\n```bash\n# Capture reasoning trace\nrto capture \"Search for Python tutorials\" -s \"You are a helpful assistant.\"\n\n# Analyze a task\nrto analyze \"Debug this code\" -o analysis.txt\n\n# Run optimization loop\nrto optimize \"Research AI papers\" --max-iterations 5 --generate-skill\n\n# Generate skill from artifacts\nrto generate-skill my-skill-name --artifacts-dir ./optimization_artifacts\n```\n\n## Integration with Claude Code\n\n### Auto-trigger on Failure\n\nAdd to your hooks to automatically analyze failures:\n\n```json\n{\n \"hooks\": {\n \"post_tool_error\": {\n \"command\": \"rto analyze-session --last-error\"\n }\n }\n}\n```\n\n### On-demand Analysis\n\nUse the slash command to analyze current session:\n\n```\n/reasoning-trace-optimizer\n```\n\nThis will:\n1. Extract thinking blocks from the current session\n2. Identify patterns and issues\n3. Suggest prompt improvements\n4. Optionally update the system prompt\n\n## Guidelines\n\n1. **Preserve full context**: M2.1 requires full response history including thinking blocks for optimal performance\n2. **Use appropriate tools**: Define tools clearly with unambiguous descriptions\n3. **Set realistic convergence thresholds**: 5-10% improvement per iteration is typical\n4. **Review generated skills**: Auto-generated skills should be reviewed before sharing\n5. **Monitor token usage**: Each optimization iteration uses significant tokens\n\n## Examples\n\n### Before Optimization\n\n```\nSystem: You are a helpful assistant.\n\nIssue: Agent called wrong tools, lost track of goal after 3 turns\nScore: 45/100\nPatterns: tool_confusion, goal_abandonment\n```\n\n### After Optimization\n\n```\nSystem: You are a research assistant focused on finding accurate information.\n\nIMPORTANT GUIDELINES:\n- Always verify search results before summarizing\n- If a tool returns an error, try an alternative approach\n- Keep track of your original goal throughout the task\n- Validate findings against multiple sources when possible\n\nIssue: None\nScore: 85/100\nPatterns: None detected\n```\n\n## References\n\n- MiniMax M2.1 Documentation: https://platform.minimax.io/docs\n- Interleaved Thinking Guide: See `docs/interleavedthinking.md`\n- Agent Generalization: See `docs/agentthinking.md`\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-01-11\n**Author**: Muratcan Koylan\n**Version**: 0.1.0\n**Powered by**: MiniMax M2.1\n**Partnership**: Built in collaboration with MiniMax AI\n","frontmatter":{"name":"reasoning-trace-optimizer","description":"Debug and optimize AI agents by analyzing reasoning traces, context degradation, tool confusion, instruction drift, repeated task failures, and performance regressions."},"isInternal":false,"tokens":1371,"sizeBytes":6359},{"name":"context-fundamentals.md","path":"examples/llm-as-judge-skills/skills/context-fundamentals/context-fundamentals.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/llm-as-judge-skills/skills/context-fundamentals/context-fundamentals.md","title":"Context-fundamentals Skill","category":"anthropic-skill","format":"markdown","content":"# Context Fundamentals Skill\n\n## Overview\n\nContext engineering is the systematic approach to managing what information an LLM receives and how it processes that information. Effective context management directly impacts output quality, consistency, and task success rates.\n\n## Core Principles\n\n### 1. Context Window Management\n\nThe context window is finite. Every token counts. Prioritize information by relevance and recency.\n\n**Strategies:**\n- Summarize historical conversation turns\n- Use retrieval to inject only relevant context\n- Implement context compression for long documents\n\n### 2. Information Hierarchy\n\nStructure context to guide model attention:\n\n```\n1. System Instructions (highest priority)\n └── Role definition\n └── Task constraints\n └── Output format requirements\n\n2. Relevant Context (dynamic)\n └── Retrieved documents\n └── User-specific data\n └── Recent conversation history\n\n3. User Input (current request)\n └── Query or instruction\n └── Any inline context\n```\n\n### 3. Context Relevance\n\nNot all context is equally useful. Apply relevance filtering:\n\n- **Temporal Relevance**: Recent information often outweighs older data\n- **Semantic Relevance**: Use embeddings to surface related content\n- **Task Relevance**: Only include information needed for current task\n\n## Context Types\n\n### Static Context\n- System prompts\n- Role definitions\n- Tool descriptions\n- Format specifications\n\n### Dynamic Context\n- Retrieved documents (RAG)\n- Conversation history\n- User preferences\n- Session state\n\n### Ephemeral Context\n- Current tool outputs\n- Intermediate reasoning steps\n- Scratchpad content\n\n## Best Practices\n\n1. **Explicit Over Implicit**: State requirements clearly rather than relying on inference\n2. **Structured Formatting**: Use consistent delimiters and sections\n3. **Redundancy Removal**: Avoid repeating information across context sections\n4. **Source Attribution**: Mark where context comes from for traceability\n5. **Freshness Signals**: Indicate when information was last updated\n\n## Context Patterns\n\n### RAG Integration Pattern\n```\n[System Instructions]\nYou are a helpful assistant. Use the provided context to answer questions.\nOnly use information from the context. If unsure, say so.\n\n[Retrieved Context]\n\n Content here...\n\n\n[User Query]\n{user_input}\n```\n\n### Multi-Turn Context Pattern\n```\n[System Instructions]\n...\n\n[Conversation History]\nSummary of earlier turns: {summary}\n\nRecent exchanges:\nUser: {recent_user_1}\nAssistant: {recent_assistant_1}\nUser: {recent_user_2}\nAssistant: {recent_assistant_2}\n\n[Current Turn]\nUser: {current_input}\n```\n\n## Metrics\n\n- **Context Utilization Rate**: How much of provided context is used in response\n- **Context Relevance Score**: Semantic similarity between context and response\n- **Context Compression Ratio**: Original size vs. compressed size\n- **Information Retention**: Key facts preserved after summarization\n\n","isInternal":false,"tokens":630,"sizeBytes":3012},{"name":"index.md","path":"examples/llm-as-judge-skills/skills/index.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/llm-as-judge-skills/skills/index.md","title":"Skills Skill","category":"anthropic-skill","format":"markdown","content":"# Skills Index\n\nSkills are foundational knowledge modules that inform the design and implementation of agents, tools, and prompts.\n\n## Available Skills\n\n### LLM Evaluator\n**Path**: `skills/llm-evaluator/llm-evaluator.md`\n\nCovers LLM-as-a-Judge evaluation methodology including:\n- Scoring approaches (direct, pairwise, reference-based)\n- Evaluation metrics (classification, correlation)\n- Known biases and mitigation strategies\n- Implementation patterns\n\n**Key Takeaways**:\n- Use direct scoring for objective evaluations\n- Use pairwise comparison for subjective preferences\n- Always mitigate position bias\n- Prefer classification metrics for interpretability\n\n### Context Fundamentals\n**Path**: `skills/context-fundamentals/context-fundamentals.md`\n\nCovers context engineering principles including:\n- Context window management\n- Information hierarchy\n- Context types (static, dynamic, ephemeral)\n- Relevance filtering\n\n**Key Takeaways**:\n- Structure context by priority\n- Be explicit over implicit\n- Remove redundancy\n- Signal freshness of information\n\n### Tool Design\n**Path**: `skills/tool-design/tool-design.md`\n\nCovers agent tool design best practices including:\n- Single responsibility principle\n- Input/output schemas\n- Error handling patterns\n- AI SDK 6 features (approval, strict mode, examples)\n\n**Key Takeaways**:\n- Clear, validated schemas\n- Predictable output structure\n- Graceful error handling\n- Consider approval for dangerous tools\n\n## Skill Application Matrix\n\n| Skill | Agents | Tools | Prompts |\n|-------|--------|-------|---------|\n| LLM Evaluator | Evaluator | directScore, pairwiseCompare | evaluation/* |\n| Context Fundamentals | All | All (context params) | All (context handling) |\n| Tool Design | All (tool selection) | All | orchestrator-prompt |\n\n## Adding New Skills\n\n1. Create skill directory: `skills//`\n2. Create main file: `skills//.md`\n3. Include:\n - Overview and purpose\n - Core principles\n - Practical patterns\n - Implementation examples\n - References\n4. Update this index\n\n## Skill Development Guidelines\n\n- Focus on principles that transfer across implementations\n- Include concrete examples and patterns\n- Reference authoritative sources\n- Keep content actionable, not just theoretical\n- Update as understanding evolves\n\n","isInternal":false,"tokens":482,"sizeBytes":2305},{"name":"llm-evaluator.md","path":"examples/llm-as-judge-skills/skills/llm-evaluator/llm-evaluator.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/llm-as-judge-skills/skills/llm-evaluator/llm-evaluator.md","title":"Llm-evaluator Skill","category":"anthropic-skill","format":"markdown","content":"# LLM-Evaluator Skill\n\n## Overview\n\nLLM-Evaluators (LLM-as-a-Judge) are large language models designed to evaluate the quality of another LLM's response to an instruction or query. This skill provides the foundational knowledge for building evaluation systems.\n\n## Key Considerations\n\n### Baseline Selection\n- **Human Annotators**: Aim for LLM-human correlation to match human-human correlation. LLM-evaluators are orders of magnitude faster and cheaper than human annotation.\n- **Finetuned Classifiers**: Goal is to achieve similar recall and precision as a finetuned classifier. More challenging baseline as these are optimized for specific tasks.\n\n### Scoring Approaches\n\n| Approach | Use Case | Reliability |\n|----------|----------|-------------|\n| **Direct Scoring** | Objective tasks (factuality, toxicity, instruction-following) | More suitable for binary classification |\n| **Pairwise Comparison** | Subjective evaluations (tone, persuasiveness, coherence) | More reliable for preference tasks |\n| **Reference-Based** | Comparing against gold standard | Requires ground truth reference |\n\n### Evaluation Metrics\n\n**Classification Metrics** (Preferred for binary tasks):\n- Recall and Precision\n- F1 Score\n- Cohen's κ (Kappa)\n\n**Correlation Metrics** (For Likert scale tasks):\n- Spearman's ρ (rho)\n- Kendall's τ (tau)\n\n## Known Biases\n\n1. **Position Bias**: LLM-evaluators tend to prefer responses in certain positions during pairwise comparison (usually first position)\n2. **Verbosity Bias**: Favor longer, more verbose responses even if not higher quality\n3. **Self-Enhancement Bias**: LLM-evaluators prefer answers generated by themselves\n\n## Mitigation Strategies\n\n- Swap response positions and average results\n- Normalize for length when evaluating\n- Use a Panel of LLMs (PoLL) instead of single judge\n- Include \"don't overthink\" instructions\n- Use CoT + n-shot prompts for reliability\n\n## Implementation Pattern\n\n```typescript\ninterface EvaluatorConfig {\n scoringApproach: 'direct' | 'pairwise' | 'reference-based';\n criteria: EvaluationCriteria[];\n metrics: MetricType[];\n useCoT: boolean;\n nShot: number;\n}\n\ninterface EvaluationCriteria {\n name: string;\n description: string;\n rubric: RubricLevel[];\n}\n\ninterface RubricLevel {\n score: number;\n description: string;\n}\n```\n\n## References\n\nKey papers reviewed:\n- Constitutional AI (Anthropic)\n- G-Eval: NLG Evaluation using GPT-4\n- SelfCheckGPT: Zero-Resource Hallucination Detection\n- Prometheus: Fine-grained Evaluation Capability\n- MT-Bench and Chatbot Arena\n\n","isInternal":false,"tokens":570,"sizeBytes":2537},{"name":"tool-design.md","path":"examples/llm-as-judge-skills/skills/tool-design/tool-design.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/llm-as-judge-skills/skills/tool-design/tool-design.md","title":"Tool-design Skill","category":"anthropic-skill","format":"markdown","content":"# Agent Tool Design Skill\n\n## Overview\n\nTools are the foundation of an agent's capabilities. An agent's ability to take meaningful actions depends entirely on how reliably it can generate valid tool inputs, how well those inputs align with user intent, and how effectively tool outputs inform next steps.\n\n## Design Principles\n\n### 1. Single Responsibility\n\nEach tool should do one thing well. Complex operations should be composed from multiple tools.\n\n```typescript\n// Bad: Tool does too much\nconst analyzeAndSummarizeAndSend = { ... }\n\n// Good: Separate concerns\nconst analyzeDocument = { ... }\nconst summarizeContent = { ... }\nconst sendEmail = { ... }\n```\n\n### 2. Clear Input Schemas\n\nUse explicit, validated schemas with descriptive field names and constraints.\n\n```typescript\nconst searchTool = tool({\n description: \"Search for documents by semantic similarity\",\n parameters: z.object({\n query: z.string().describe(\"Natural language search query\"),\n limit: z.number().min(1).max(100).default(10)\n .describe(\"Maximum number of results to return\"),\n filters: z.object({\n dateAfter: z.string().optional()\n .describe(\"ISO date string, only return docs after this date\"),\n source: z.enum([\"internal\", \"external\", \"all\"]).default(\"all\")\n }).optional()\n }),\n execute: async (input) => { ... }\n});\n```\n\n### 3. Predictable Output Structure\n\nReturn consistent, typed output that the model can reliably parse.\n\n```typescript\ninterface ToolResult {\n success: boolean;\n data?: T;\n error?: {\n code: string;\n message: string;\n retryable: boolean;\n };\n metadata: {\n executionTimeMs: number;\n source?: string;\n };\n}\n```\n\n### 4. Graceful Error Handling\n\nTools should never throw unhandled exceptions. Always return structured errors.\n\n```typescript\nexecute: async (input) => {\n try {\n const result = await performAction(input);\n return { success: true, data: result };\n } catch (error) {\n return {\n success: false,\n error: {\n code: error.code ?? \"UNKNOWN_ERROR\",\n message: error.message,\n retryable: isRetryable(error)\n }\n };\n }\n}\n```\n\n## Tool Categories\n\n### Read-Only Tools\n- Database queries\n- API fetches\n- File reads\n- Search operations\n\nSafe to execute without approval. Return data but don't mutate state.\n\n### State-Modifying Tools\n- Database writes\n- File modifications\n- API POST/PUT/DELETE\n- System configuration changes\n\nMay require human approval. Consider `needsApproval` flag.\n\n### Dangerous Tools\n- File deletion\n- Payment processing\n- Production deployments\n- Sending external communications\n\nShould always require approval and audit logging.\n\n## AI SDK 6 Tool Features\n\n### Tool Execution Approval\n```typescript\nconst deleteTool = tool({\n description: \"Delete a file from the system\",\n parameters: z.object({\n path: z.string()\n }),\n needsApproval: true, // Requires human approval\n execute: async ({ path }) => { ... }\n});\n\n// Dynamic approval based on input\nconst commandTool = tool({\n description: \"Execute a shell command\",\n parameters: z.object({\n command: z.string()\n }),\n needsApproval: ({ command }) => {\n return command.includes(\"rm\") || command.includes(\"delete\");\n },\n execute: async ({ command }) => { ... }\n});\n```\n\n### Strict Mode\nEnable native strict mode for guaranteed schema compliance:\n```typescript\nconst strictTool = tool({\n description: \"...\",\n parameters: schema,\n strict: true, // Enable strict mode\n execute: async (input) => { ... }\n});\n```\n\n### Input Examples\nHelp the model understand expected input format:\n```typescript\nconst complexTool = tool({\n description: \"Create a calendar event\",\n parameters: eventSchema,\n inputExamples: [\n {\n title: \"Team Standup\",\n date: \"2024-01-15\",\n time: \"09:00\",\n duration: 30,\n attendees: [\"alice@example.com\", \"bob@example.com\"]\n }\n ],\n execute: async (input) => { ... }\n});\n```\n\n### toModelOutput\nControl what gets sent back to the model:\n```typescript\nconst readFileTool = tool({\n description: \"Read file contents\",\n parameters: z.object({ path: z.string() }),\n execute: async ({ path }) => {\n const content = await fs.readFile(path, 'utf-8');\n return { path, content, size: content.length };\n },\n toModelOutput: (result) => {\n // Only send truncated content to model\n return {\n path: result.path,\n content: result.content.slice(0, 5000),\n truncated: result.content.length > 5000\n };\n }\n});\n```\n\n## Best Practices\n\n1. **Descriptive Names**: Tool names should clearly indicate their function\n2. **Comprehensive Descriptions**: Include usage examples in tool descriptions\n3. **Reasonable Defaults**: Provide sensible defaults for optional parameters\n4. **Idempotency**: Design tools to be safely re-executable when possible\n5. **Timeout Handling**: Implement timeouts for external operations\n6. **Rate Limiting**: Protect against runaway tool execution\n7. **Logging**: Log all tool invocations for debugging and audit\n\n","isInternal":false,"tokens":1149,"sizeBytes":4999},{"name":"direct-score.md","path":"examples/llm-as-judge-skills/tools/evaluation/direct-score.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/llm-as-judge-skills/tools/evaluation/direct-score.md","title":"Evaluation Skill","category":"anthropic-skill","format":"markdown","content":"# Direct Score Tool\n\n## Purpose\n\nEvaluate a single LLM response against defined criteria using a scoring rubric.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const directScore = tool({\n description: `Evaluate a response by scoring it against specific criteria.\nUse this for objective evaluations where you need to assess quality \ndimensions like accuracy, completeness, clarity, or task adherence.\nReturns structured scores with justifications.`,\n\n parameters: z.object({\n response: z.string()\n .describe(\"The LLM response to evaluate\"),\n \n prompt: z.string()\n .describe(\"The original prompt/instruction that generated the response\"),\n \n context: z.string().optional()\n .describe(\"Additional context like retrieved documents or conversation history\"),\n \n criteria: z.array(z.object({\n name: z.string().describe(\"Name of the criterion (e.g., 'Accuracy')\"),\n description: z.string().describe(\"What this criterion measures\"),\n weight: z.number().min(0).max(1).default(1)\n .describe(\"Relative importance, weights should sum to 1\")\n })).min(1).describe(\"Evaluation criteria to score against\"),\n \n rubric: z.object({\n scale: z.enum([\"1-3\", \"1-5\", \"1-10\"]).default(\"1-5\"),\n levelDescriptions: z.record(z.string(), z.string()).optional()\n .describe(\"Optional descriptions for each score level\")\n }).optional().describe(\"Scoring rubric configuration\")\n }),\n\n execute: async (input) => {\n // Implementation delegates to evaluator LLM\n return evaluateWithLLM(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| response | string | Yes | The response being evaluated |\n| prompt | string | Yes | Original prompt that generated the response |\n| context | string | No | Additional context (RAG docs, history) |\n| criteria | Criterion[] | Yes | List of evaluation criteria |\n| rubric | Rubric | No | Scoring scale and level descriptions |\n\n### Criterion Object\n```typescript\n{\n name: string; // e.g., \"Factual Accuracy\"\n description: string; // e.g., \"Response contains no factual errors\"\n weight: number; // 0-1, relative importance\n}\n```\n\n### Rubric Object\n```typescript\n{\n scale: \"1-3\" | \"1-5\" | \"1-10\";\n levelDescriptions?: {\n \"1\": \"Poor - Major issues\",\n \"2\": \"Below Average - Several issues\",\n \"3\": \"Average - Some issues\",\n \"4\": \"Good - Minor issues\",\n \"5\": \"Excellent - No issues\"\n }\n}\n```\n\n## Output Schema\n\n```typescript\ninterface DirectScoreResult {\n success: boolean;\n \n scores: {\n criterion: string;\n score: number;\n maxScore: number;\n justification: string;\n examples: string[]; // Specific examples from response\n }[];\n \n overallScore: number;\n weightedScore: number;\n \n summary: {\n strengths: string[];\n weaknesses: string[];\n suggestions: string[];\n };\n \n metadata: {\n evaluationTimeMs: number;\n criteriaCount: number;\n rubricScale: string;\n };\n}\n```\n\n## Usage Example\n\n```typescript\nconst result = await directScore.execute({\n response: \"Machine learning is a subset of AI that enables systems to learn from data...\",\n \n prompt: \"Explain machine learning to a beginner\",\n \n criteria: [\n {\n name: \"Accuracy\",\n description: \"Technical correctness of explanations\",\n weight: 0.4\n },\n {\n name: \"Clarity\",\n description: \"Understandable for a beginner\",\n weight: 0.3\n },\n {\n name: \"Completeness\",\n description: \"Covers key concepts adequately\",\n weight: 0.3\n }\n ],\n \n rubric: {\n scale: \"1-5\",\n levelDescriptions: {\n \"1\": \"Completely fails criterion\",\n \"2\": \"Major deficiencies\",\n \"3\": \"Acceptable but improvable\",\n \"4\": \"Good with minor issues\",\n \"5\": \"Excellent, no issues\"\n }\n }\n});\n```\n\n## Implementation Notes\n\n1. **Chain-of-Thought**: Implementation should use CoT prompting for more reliable scoring\n2. **Calibration**: Include few-shot examples of scores at each level\n3. **Justification First**: Ask for justification before score to reduce bias\n4. **Length Normalization**: Consider response length when appropriate\n\n","isInternal":false,"tokens":1006,"sizeBytes":4227},{"name":"generate-rubric.md","path":"examples/llm-as-judge-skills/tools/evaluation/generate-rubric.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/llm-as-judge-skills/tools/evaluation/generate-rubric.md","title":"Evaluation Skill","category":"anthropic-skill","format":"markdown","content":"# Generate Rubric Tool\n\n## Purpose\n\nAutomatically generate a scoring rubric for a given evaluation criterion. Creates detailed descriptions for each score level to ensure consistent evaluation.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const generateRubric = tool({\n description: `Generate a scoring rubric for an evaluation criterion.\nCreates detailed descriptions for each score level.\nUse when you need to establish consistent evaluation standards.`,\n\n parameters: z.object({\n criterionName: z.string()\n .describe(\"Name of the criterion (e.g., 'Factual Accuracy')\"),\n \n criterionDescription: z.string()\n .describe(\"What this criterion measures\"),\n \n scale: z.enum([\"1-3\", \"1-5\", \"1-10\"]).default(\"1-5\")\n .describe(\"Scoring scale to use\"),\n \n domain: z.string().optional()\n .describe(\"Domain context (e.g., 'medical writing', 'code review')\"),\n \n includeExamples: z.boolean().default(true)\n .describe(\"Include example text for each score level\"),\n \n strictness: z.enum([\"lenient\", \"balanced\", \"strict\"]).default(\"balanced\")\n .describe(\"How strictly to define score boundaries\")\n }),\n\n execute: async (input) => {\n return generateRubricWithLLM(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| criterionName | string | Yes | Name of criterion |\n| criterionDescription | string | Yes | What criterion measures |\n| scale | enum | No | Scoring scale (default: 1-5) |\n| domain | string | No | Domain for context |\n| includeExamples | boolean | No | Include examples (default: true) |\n| strictness | enum | No | Score boundary strictness |\n\n## Output Schema\n\n```typescript\ninterface GeneratedRubric {\n success: boolean;\n \n criterion: {\n name: string;\n description: string;\n };\n \n scale: {\n min: number;\n max: number;\n type: string;\n };\n \n levels: {\n score: number;\n label: string; // e.g., \"Excellent\", \"Poor\"\n description: string; // Detailed description\n characteristics: string[]; // Key characteristics\n example?: string; // Example text at this level\n }[];\n \n scoringGuidelines: string[];\n \n edgeCases: {\n situation: string;\n guidance: string;\n }[];\n \n metadata: {\n domain: string | null;\n strictness: string;\n generationTimeMs: number;\n };\n}\n```\n\n## Usage Example\n\n```typescript\nconst rubric = await generateRubric.execute({\n criterionName: \"Code Readability\",\n criterionDescription: \"How easy the code is to read and understand\",\n scale: \"1-5\",\n domain: \"code review\",\n includeExamples: true,\n strictness: \"balanced\"\n});\n\n// Result:\n// {\n// criterion: {\n// name: \"Code Readability\",\n// description: \"How easy the code is to read and understand\"\n// },\n// scale: { min: 1, max: 5, type: \"1-5\" },\n// levels: [\n// {\n// score: 1,\n// label: \"Poor\",\n// description: \"Code is extremely difficult to understand...\",\n// characteristics: [\n// \"No meaningful variable names\",\n// \"Deeply nested logic without explanation\",\n// \"No comments on complex sections\"\n// ],\n// example: \"function x(a,b,c){return a?b+c:c-b;}\"\n// },\n// {\n// score: 5,\n// label: \"Excellent\",\n// description: \"Code is immediately understandable...\",\n// characteristics: [\n// \"Self-documenting variable and function names\",\n// \"Appropriate comments explaining 'why'\",\n// \"Clear logical structure\"\n// ],\n// example: \"function calculateShippingCost(weight, distance, expedited) {\\n // Base rate plus per-mile charge\\n const baseCost = weight * BASE_RATE_PER_KG;\\n ...\"\n// },\n// ...\n// ],\n// scoringGuidelines: [\n// \"Focus on clarity for someone unfamiliar with the codebase\",\n// \"Consider both naming and structure\",\n// \"Comments should explain 'why', not 'what'\"\n// ],\n// edgeCases: [\n// {\n// situation: \"Code uses domain-specific abbreviations\",\n// guidance: \"Accept if abbreviations are standard in the domain\"\n// }\n// ]\n// }\n```\n\n## Rubric Templates\n\n### Factual Accuracy (1-5)\n```\n5: All claims factually correct, properly sourced\n4: Minor factual issues, non-critical\n3: Some factual errors, main points correct\n2: Multiple factual errors affecting reliability\n1: Fundamentally incorrect or misleading\n```\n\n### Clarity (1-5)\n```\n5: Immediately understandable, well-structured\n4: Clear with minor ambiguities\n3: Generally clear, some confusing sections\n2: Difficult to follow, unclear organization\n1: Incomprehensible or incoherent\n```\n\n### Completeness (1-5)\n```\n5: Addresses all aspects comprehensively\n4: Covers main points, minor gaps\n3: Addresses core requirements, notable gaps\n2: Missing significant required elements\n1: Fails to address the question\n```\n\n## Implementation Notes\n\n1. **Domain Adaptation**: Rubrics should reflect domain-specific expectations\n2. **Boundary Clarity**: Clear distinctions between adjacent scores\n3. **Example Quality**: Examples should be realistic, not strawmen\n4. **Edge Case Coverage**: Anticipate common ambiguous situations\n5. **Calibration**: Test rubric against known samples before use\n\n","isInternal":false,"tokens":1258,"sizeBytes":5276},{"name":"pairwise-compare.md","path":"examples/llm-as-judge-skills/tools/evaluation/pairwise-compare.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/llm-as-judge-skills/tools/evaluation/pairwise-compare.md","title":"Evaluation Skill","category":"anthropic-skill","format":"markdown","content":"# Pairwise Compare Tool\n\n## Purpose\n\nCompare two LLM responses and determine which one better satisfies the given criteria. More reliable for subjective evaluations than direct scoring.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const pairwiseCompare = tool({\n description: `Compare two responses and select the better one.\nUse for subjective evaluations like tone, persuasiveness, or writing style.\nMore reliable than direct scoring for preferences.\nReturns winner selection with detailed comparison.`,\n\n parameters: z.object({\n responseA: z.string()\n .describe(\"First response to compare\"),\n \n responseB: z.string()\n .describe(\"Second response to compare\"),\n \n prompt: z.string()\n .describe(\"The original prompt both responses address\"),\n \n context: z.string().optional()\n .describe(\"Additional context if relevant\"),\n \n criteria: z.array(z.string())\n .describe(\"Aspects to compare on, e.g., ['clarity', 'engagement', 'accuracy']\"),\n \n allowTie: z.boolean().default(true)\n .describe(\"Whether to allow a tie verdict\"),\n \n swapPositions: z.boolean().default(true)\n .describe(\"Evaluate twice with positions swapped to reduce position bias\")\n }),\n\n execute: async (input) => {\n if (input.swapPositions) {\n return evaluateWithPositionSwap(input);\n }\n return evaluatePairwise(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| responseA | string | Yes | First response |\n| responseB | string | Yes | Second response |\n| prompt | string | Yes | Original prompt |\n| context | string | No | Additional context |\n| criteria | string[] | Yes | Comparison dimensions |\n| allowTie | boolean | No | Allow tie verdict (default: true) |\n| swapPositions | boolean | No | Swap positions to reduce bias (default: true) |\n\n## Output Schema\n\n```typescript\ninterface PairwiseCompareResult {\n success: boolean;\n \n winner: \"A\" | \"B\" | \"TIE\";\n confidence: number; // 0-1\n \n comparison: {\n criterion: string;\n winner: \"A\" | \"B\" | \"TIE\";\n reasoning: string;\n aStrength: string;\n bStrength: string;\n }[];\n \n overallReasoning: string;\n \n differentiators: {\n aAdvantages: string[];\n bAdvantages: string[];\n };\n \n // If swapPositions was true\n positionConsistency?: {\n firstPassWinner: \"A\" | \"B\" | \"TIE\";\n secondPassWinner: \"A\" | \"B\" | \"TIE\";\n consistent: boolean;\n };\n \n metadata: {\n evaluationTimeMs: number;\n positionsSwapped: boolean;\n };\n}\n```\n\n## Usage Example\n\n```typescript\nconst result = await pairwiseCompare.execute({\n responseA: \"Exercise improves cardiovascular health, builds muscle, and boosts mental wellbeing...\",\n \n responseB: \"Working out regularly has many benefits. You'll feel better and look better...\",\n \n prompt: \"Explain the benefits of regular exercise\",\n \n criteria: [\"accuracy\", \"specificity\", \"engagement\", \"completeness\"],\n \n allowTie: true,\n swapPositions: true\n});\n\n// Result:\n// {\n// winner: \"A\",\n// confidence: 0.85,\n// comparison: [\n// {\n// criterion: \"accuracy\",\n// winner: \"A\",\n// reasoning: \"Response A uses specific medical terminology...\",\n// aStrength: \"Mentions cardiovascular, muscle, mental health\",\n// bStrength: \"General but not incorrect\"\n// },\n// ...\n// ],\n// ...\n// }\n```\n\n## Position Swap Algorithm\n\nTo mitigate position bias:\n\n```typescript\nasync function evaluateWithPositionSwap(input) {\n // First pass: Original order\n const pass1 = await evaluate({\n first: input.responseA,\n second: input.responseB,\n ...input\n });\n \n // Second pass: Swapped order\n const pass2 = await evaluate({\n first: input.responseB,\n second: input.responseA,\n ...input\n });\n \n // Reconcile results\n const pass2Adjusted = pass2.winner === \"A\" ? \"B\" : pass2.winner === \"B\" ? \"A\" : \"TIE\";\n \n if (pass1.winner === pass2Adjusted) {\n return {\n ...pass1,\n positionConsistency: { consistent: true, ... }\n };\n } else {\n // Inconsistent - return tie or lower confidence\n return {\n winner: \"TIE\",\n confidence: 0.5,\n positionConsistency: { consistent: false, ... },\n ...\n };\n }\n}\n```\n\n## Implementation Notes\n\n1. **Position Bias Mitigation**: Always use `swapPositions: true` for production\n2. **Criteria Order**: Order criteria by importance for better focus\n3. **Tie Handling**: Consider domain - some tasks should rarely tie\n4. **Confidence Calibration**: Lower confidence when evaluations are close\n5. **Length Considerations**: Note if one response is significantly longer\n\n","isInternal":false,"tokens":1128,"sizeBytes":4677},{"name":"index.md","path":"examples/llm-as-judge-skills/tools/index.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/llm-as-judge-skills/tools/index.md","title":"Tools Skill","category":"anthropic-skill","format":"markdown","content":"# Tools Index\n\nTools provide specific capabilities that agents can use to accomplish tasks.\n\n## Tool Categories\n\n### Evaluation Tools\n**Path**: `tools/evaluation/`\n\nTools for assessing LLM output quality.\n\n| Tool | Purpose | Approval |\n|------|---------|----------|\n| `directScore` | Score response against criteria | No |\n| `pairwiseCompare` | Compare two responses | No |\n| `generateRubric` | Generate scoring rubric | No |\n| `extractCriteria` | Extract criteria from task | No |\n\n---\n\n### Research Tools\n**Path**: `tools/research/`\n\nTools for gathering and processing information.\n\n| Tool | Purpose | Approval |\n|------|---------|----------|\n| `webSearch` | Search the web | No |\n| `readUrl` | Extract content from URL | No |\n| `extractClaims` | Identify claims in text | No |\n| `verifyClaim` | Cross-reference a claim | No |\n| `synthesize` | Combine findings | No |\n\n---\n\n### Orchestration Tools\n**Path**: `tools/orchestration/`\n\nTools for managing multi-agent workflows.\n\n| Tool | Purpose | Approval |\n|------|---------|----------|\n| `delegateToAgent` | Route task to agent | No |\n| `parallelExecution` | Run tasks concurrently | No |\n| `waitForCompletion` | Wait for async tasks | No |\n| `synthesizeResults` | Combine agent outputs | No |\n| `handleError` | Manage failures | No |\n\n## Tool Design Patterns\n\n### Standard Tool Structure\n\n```typescript\nexport const toolName = tool({\n description: \"Clear description of what tool does\",\n \n parameters: z.object({\n // Required parameters first\n requiredParam: z.string().describe(\"What this parameter is for\"),\n \n // Optional parameters with defaults\n optionalParam: z.number().default(10)\n .describe(\"What this parameter controls\")\n }),\n \n // Approval for dangerous operations\n needsApproval: false, // or true, or function\n \n // Strict mode for guaranteed schema compliance\n strict: true,\n \n execute: async (input) => {\n try {\n const result = await performOperation(input);\n return { success: true, data: result };\n } catch (error) {\n return {\n success: false,\n error: {\n code: error.code ?? \"UNKNOWN\",\n message: error.message,\n retryable: isRetryable(error)\n }\n };\n }\n },\n \n // Optional: control what model sees\n toModelOutput: (result) => ({\n summary: result.data.summary,\n truncated: result.data.full.length > 5000\n })\n});\n```\n\n### Error Response Pattern\n\n```typescript\ninterface ToolError {\n code: string; // Machine-readable error code\n message: string; // Human-readable message\n retryable: boolean; // Whether retry might help\n details?: object; // Additional context\n}\n\ninterface ToolResult {\n success: boolean;\n data?: T;\n error?: ToolError;\n metadata: {\n executionTimeMs: number;\n [key: string]: any;\n };\n}\n```\n\n## Adding New Tools\n\n1. Determine category or create new: `tools//`\n2. Create tool file: `tools//.md`\n3. Define:\n - Purpose and description\n - Input parameters with Zod schema\n - Output schema\n - Error codes\n - Usage examples\n4. Update this index\n5. Assign to relevant agents\n\n## Tool Selection Guidelines\n\n### When Agent Needs To...\n\n| Action | Tool Category | Suggested Tools |\n|--------|---------------|-----------------|\n| Assess quality | Evaluation | directScore, pairwiseCompare |\n| Find information | Research | webSearch, readUrl |\n| Verify facts | Research | verifyClaim, extractClaims |\n| Coordinate work | Orchestration | delegateToAgent |\n| Wait for results | Orchestration | waitForCompletion |\n\n","isInternal":false,"tokens":831,"sizeBytes":3575},{"name":"delegate-to-agent.md","path":"examples/llm-as-judge-skills/tools/orchestration/delegate-to-agent.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/llm-as-judge-skills/tools/orchestration/delegate-to-agent.md","title":"Orchestration Skill","category":"anthropic-skill","format":"markdown","content":"# Delegate to Agent Tool\n\n## Purpose\n\nRoute a task to a specialized agent for execution. Handles context passing, result collection, and error management.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const delegateToAgent = tool({\n description: `Delegate a task to a specialized agent.\nUse when a subtask requires specific capabilities.\nPass complete context needed for the agent to succeed.`,\n\n parameters: z.object({\n agentName: z.enum([\"evaluator\", \"researcher\", \"writer\", \"analyst\"])\n .describe(\"Name of the agent to delegate to\"),\n \n task: z.string()\n .describe(\"Clear description of what the agent should do\"),\n \n context: z.object({\n previousOutputs: z.array(z.string()).optional()\n .describe(\"Outputs from prior steps this agent needs\"),\n \n documents: z.array(z.string()).optional()\n .describe(\"Relevant documents or data\"),\n \n constraints: z.array(z.string()).optional()\n .describe(\"Requirements or limitations to observe\")\n }).optional(),\n \n expectedOutput: z.object({\n format: z.enum([\"text\", \"json\", \"markdown\", \"structured\"])\n .describe(\"Expected output format\"),\n \n schema: z.string().optional()\n .describe(\"JSON schema if format is structured\"),\n \n maxLength: z.number().optional()\n .describe(\"Maximum length constraint\")\n }).optional(),\n \n timeout: z.number().default(60000)\n .describe(\"Timeout in milliseconds\")\n }),\n\n execute: async (input) => {\n return executeAgentDelegation(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| agentName | enum | Yes | Target agent |\n| task | string | Yes | Task description |\n| context | object | No | Context and dependencies |\n| expectedOutput | object | No | Output requirements |\n| timeout | number | No | Timeout ms (default: 60000) |\n\n## Output Schema\n\n```typescript\ninterface DelegationResult {\n success: boolean;\n \n agentName: string;\n task: string;\n \n output: {\n content: string | object;\n format: string;\n };\n \n execution: {\n startTime: string;\n endTime: string;\n durationMs: number;\n tokenUsage: {\n prompt: number;\n completion: number;\n };\n };\n \n error?: {\n code: string;\n message: string;\n retryable: boolean;\n };\n}\n```\n\n## Available Agents\n\n### Evaluator Agent\n```typescript\nawait delegateToAgent.execute({\n agentName: \"evaluator\",\n task: \"Evaluate the quality of this response against accuracy and clarity criteria\",\n context: {\n documents: [responseToEvaluate],\n constraints: [\"Use 1-5 scale\", \"Include justification\"]\n },\n expectedOutput: { format: \"structured\" }\n});\n```\n\n### Researcher Agent\n```typescript\nawait delegateToAgent.execute({\n agentName: \"researcher\",\n task: \"Research current best practices for LLM evaluation\",\n context: {\n constraints: [\"Focus on 2024 publications\", \"Include citations\"]\n },\n expectedOutput: { format: \"markdown\" }\n});\n```\n\n### Writer Agent\n```typescript\nawait delegateToAgent.execute({\n agentName: \"writer\",\n task: \"Write an executive summary of these research findings\",\n context: {\n previousOutputs: [researchFindings],\n constraints: [\"Maximum 500 words\", \"Non-technical audience\"]\n },\n expectedOutput: { format: \"text\", maxLength: 2500 }\n});\n```\n\n### Analyst Agent\n```typescript\nawait delegateToAgent.execute({\n agentName: \"analyst\",\n task: \"Analyze the trade-offs between direct scoring and pairwise comparison\",\n context: {\n documents: [evaluationData],\n constraints: [\"Quantitative where possible\"]\n },\n expectedOutput: { format: \"structured\" }\n});\n```\n\n## Error Handling\n\n```typescript\nconst errorCodes = {\n \"AGENT_NOT_FOUND\": \"Specified agent does not exist\",\n \"TIMEOUT\": \"Agent did not complete within timeout\",\n \"CONTEXT_TOO_LARGE\": \"Context exceeds agent's capacity\",\n \"INVALID_OUTPUT\": \"Agent output did not match expected format\",\n \"AGENT_ERROR\": \"Agent encountered an error during execution\"\n};\n```\n\n## Implementation Notes\n\n1. **Context Optimization**: Compress context if needed before passing\n2. **Timeout Handling**: Set realistic timeouts per agent type\n3. **Retry Logic**: Implement retries for transient failures\n4. **Audit Trail**: Log all delegations for traceability\n5. **Resource Management**: Track token usage across delegations\n\n","isInternal":false,"tokens":1005,"sizeBytes":4430},{"name":"read-url.md","path":"examples/llm-as-judge-skills/tools/research/read-url.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/llm-as-judge-skills/tools/research/read-url.md","title":"Research Skill","category":"anthropic-skill","format":"markdown","content":"# Read URL Tool\n\n## Purpose\n\nExtract and parse content from a given URL. Returns structured text content with metadata about the source.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const readUrl = tool({\n description: `Read and extract content from a URL.\nReturns the main text content, stripped of navigation and ads.\nUse after webSearch to get full content from relevant results.`,\n\n parameters: z.object({\n url: z.string().url()\n .describe(\"The URL to read\"),\n \n contentType: z.enum([\"auto\", \"article\", \"documentation\", \"paper\", \"code\"]).default(\"auto\")\n .describe(\"Hint for content type to optimize extraction\"),\n \n maxLength: z.number().min(1000).max(50000).default(10000)\n .describe(\"Maximum characters to return\"),\n \n extractSections: z.boolean().default(true)\n .describe(\"Whether to identify and label sections\"),\n \n includeMetadata: z.boolean().default(true)\n .describe(\"Include author, date, and other metadata\")\n }),\n\n execute: async (input) => {\n return extractUrlContent(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| url | string | Yes | URL to read |\n| contentType | enum | No | Content type hint |\n| maxLength | number | No | Max chars (default: 10000) |\n| extractSections | boolean | No | Label sections |\n| includeMetadata | boolean | No | Include metadata |\n\n## Output Schema\n\n```typescript\ninterface ReadUrlResult {\n success: boolean;\n \n url: string;\n title: string;\n \n content: {\n full: string;\n sections?: {\n heading: string;\n level: number; // h1=1, h2=2, etc.\n content: string;\n }[];\n };\n \n metadata?: {\n author?: string;\n publishedDate?: string;\n lastModified?: string;\n description?: string;\n keywords?: string[];\n source: string;\n };\n \n stats: {\n totalCharacters: number;\n truncated: boolean;\n sectionsFound: number;\n };\n \n error?: {\n code: string;\n message: string;\n };\n}\n```\n\n## Usage Example\n\n```typescript\nconst content = await readUrl.execute({\n url: \"https://eugeneyan.com/writing/llm-evaluators/\",\n contentType: \"article\",\n maxLength: 15000,\n extractSections: true,\n includeMetadata: true\n});\n\n// Result:\n// {\n// success: true,\n// url: \"https://eugeneyan.com/writing/llm-evaluators/\",\n// title: \"Evaluating the Effectiveness of LLM-Evaluators\",\n// content: {\n// full: \"LLM-evaluators, also known as LLM-as-a-Judge...\",\n// sections: [\n// {\n// heading: \"Key considerations before adopting an LLM-evaluator\",\n// level: 2,\n// content: \"Before reviewing the literature...\"\n// },\n// ...\n// ]\n// },\n// metadata: {\n// author: \"Eugene Yan\",\n// publishedDate: \"2024-06-15\",\n// source: \"eugeneyan.com\"\n// },\n// stats: {\n// totalCharacters: 15000,\n// truncated: true,\n// sectionsFound: 8\n// }\n// }\n```\n\n## Content Type Handling\n\n| Type | Optimization |\n|------|-------------|\n| article | Prioritize main content, skip sidebars |\n| documentation | Preserve code blocks, keep structure |\n| paper | Extract abstract, sections, references |\n| code | Preserve formatting, syntax highlighting |\n| auto | Detect type from content |\n\n## Error Handling\n\n```typescript\nconst errorCodes = {\n \"URL_NOT_FOUND\": \"Page does not exist (404)\",\n \"ACCESS_DENIED\": \"Page requires authentication (401/403)\",\n \"TIMEOUT\": \"Request timed out\",\n \"BLOCKED\": \"Access blocked by robots.txt or rate limit\",\n \"INVALID_CONTENT\": \"Content could not be parsed\",\n \"UNSUPPORTED_TYPE\": \"Content type not supported (e.g., binary)\"\n};\n```\n\n## Implementation Notes\n\n1. **Respect robots.txt**: Check and honor robots.txt directives\n2. **Rate Limiting**: Don't hammer the same domain\n3. **User Agent**: Use appropriate user agent string\n4. **Timeouts**: Set reasonable timeouts (10-30s)\n5. **JavaScript Rendering**: Consider headless browser for JS-heavy sites\n6. **Caching**: Cache content for repeated reads\n\n","isInternal":false,"tokens":1011,"sizeBytes":4060},{"name":"web-search.md","path":"examples/llm-as-judge-skills/tools/research/web-search.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/llm-as-judge-skills/tools/research/web-search.md","title":"Research Skill","category":"anthropic-skill","format":"markdown","content":"# Web Search Tool\n\n## Purpose\n\nSearch the web for relevant information on a given topic. Returns structured results with snippets, URLs, and metadata.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const webSearch = tool({\n description: `Search the web for information on a topic.\nReturns relevant results with snippets and URLs.\nUse for gathering current information, verifying facts, or research.`,\n\n parameters: z.object({\n query: z.string()\n .describe(\"Search query - be specific for better results\"),\n \n maxResults: z.number().min(1).max(20).default(10)\n .describe(\"Maximum number of results to return\"),\n \n filters: z.object({\n dateRange: z.enum([\"day\", \"week\", \"month\", \"year\", \"any\"]).default(\"any\")\n .describe(\"Limit results to a time period\"),\n \n sourceType: z.enum([\"all\", \"news\", \"academic\", \"documentation\"]).default(\"all\")\n .describe(\"Type of sources to prioritize\"),\n \n excludeDomains: z.array(z.string()).optional()\n .describe(\"Domains to exclude from results\")\n }).optional()\n }),\n\n execute: async (input) => {\n return performWebSearch(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| query | string | Yes | Search query |\n| maxResults | number | No | Max results (default: 10) |\n| filters.dateRange | enum | No | Time period filter |\n| filters.sourceType | enum | No | Source type priority |\n| filters.excludeDomains | string[] | No | Domains to exclude |\n\n## Output Schema\n\n```typescript\ninterface WebSearchResult {\n success: boolean;\n \n results: {\n title: string;\n url: string;\n snippet: string;\n source: string; // Domain name\n publishedDate?: string;\n relevanceScore: number;\n }[];\n \n totalResults: number;\n \n metadata: {\n query: string;\n searchTimeMs: number;\n filtersApplied: string[];\n };\n}\n```\n\n## Usage Example\n\n```typescript\nconst results = await webSearch.execute({\n query: \"LLM-as-a-Judge evaluation methods 2024\",\n maxResults: 10,\n filters: {\n dateRange: \"year\",\n sourceType: \"academic\"\n }\n});\n\n// Result:\n// {\n// success: true,\n// results: [\n// {\n// title: \"Judging LLM-as-a-Judge with MT-Bench\",\n// url: \"https://arxiv.org/abs/...\",\n// snippet: \"We study the effectiveness of LLM-as-a-Judge...\",\n// source: \"arxiv.org\",\n// publishedDate: \"2024-01-15\",\n// relevanceScore: 0.95\n// },\n// ...\n// ],\n// totalResults: 10,\n// metadata: {\n// query: \"LLM-as-a-Judge evaluation methods 2024\",\n// searchTimeMs: 342,\n// filtersApplied: [\"dateRange:year\", \"sourceType:academic\"]\n// }\n// }\n```\n\n## Query Optimization Tips\n\n1. **Specific Terms**: Use precise terminology\n2. **Quotes**: Use quotes for exact phrases\n3. **Operators**: Support site:, -term, OR\n4. **Context**: Include relevant context terms\n5. **Recency**: Add year for recent info\n\n## Implementation Notes\n\n1. **Rate Limiting**: Implement appropriate rate limits\n2. **Caching**: Cache results for repeated queries\n3. **Result Quality**: Filter out low-quality sources\n4. **Error Handling**: Handle API failures gracefully\n5. **Privacy**: Log queries appropriately\n\n","isInternal":false,"tokens":815,"sizeBytes":3275},{"name":"SKILL.md","path":"SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/SKILL.md","title":"context-engineering-collection","category":"anthropic-skill","format":"markdown","content":"---\nname: context-engineering-collection\ndescription: \"A comprehensive collection of Agent Skills for context engineering, harness engineering, multi-agent architectures, and production agent systems. Use when building, optimizing, evaluating, or debugging agent systems that require effective context management and reliable operating loops.\"\n---\n\n# Agent Skills for Context Engineering\n\nThis collection provides structured guidance for building production-grade AI agent systems through effective context engineering.\n\n## When to Activate\n\nActivate these skills when:\n- Building new agent systems from scratch\n- Optimizing existing agent performance\n- Debugging context-related failures\n- Designing multi-agent architectures\n- Creating or evaluating tools for agents\n- Implementing memory and persistence layers\n- Designing autonomous research or evaluation harnesses\n\n## Skill Map\n\n### Foundational Context Engineering\n\n**Understanding Context Fundamentals**\nContext is not just prompt text—it is the complete state available to the language model at inference time, including system instructions, tool definitions, retrieved documents, message history, and tool outputs. Effective context engineering means understanding what information truly matters for the task at hand and curating that information for maximum signal-to-noise ratio.\n\n**Recognizing Context Degradation**\nLanguage models exhibit predictable degradation patterns as context grows: the \"lost-in-middle\" phenomenon where information in the center of context receives less attention; U-shaped attention curves that prioritize beginning and end; context poisoning when errors compound; and context distraction when irrelevant information overwhelms relevant content.\n\n### Architectural Patterns\n\n**Multi-Agent Coordination**\nProduction multi-agent systems converge on three dominant patterns: supervisor/orchestrator architectures with centralized control, peer-to-peer swarm architectures for flexible handoffs, and hierarchical structures for complex task decomposition. The critical insight is that sub-agents exist primarily to isolate context rather than to simulate organizational roles.\n\n**Long-Horizon Prompting**\nLong-running autonomous agents and parallel orchestrations succeed or fail on the launch prompt. Pseudo-formal task briefs specify success predicates, non-counting outcomes, persistence rules with audit-gated return conditions, effort floors, diversity policies for parallel portfolios, and contamination guards, applying the discipline of formal verification linguistically to problems with no machine-checkable success condition. In orchestrated runs, each worker returns an output contract with named checks, and only evidenced violations block the return.\n\n**Memory System Design**\nMemory architectures range from simple scratchpads to sophisticated temporal knowledge graphs. Vector RAG provides semantic retrieval but loses relationship information. Knowledge graphs preserve structure but require more engineering investment. The file-system-as-memory pattern enables just-in-time context loading without stuffing context windows.\n\n**Filesystem-Based Context**\nThe filesystem provides a single interface for storing, retrieving, and updating effectively unlimited context. Key patterns include scratch pads for tool output offloading, plan persistence for long-horizon tasks, sub-agent communication via shared files, and dynamic skill loading. Agents use `ls`, `glob`, `grep`, and `read_file` for targeted context discovery, often outperforming semantic search for structural queries.\n\n**Hosted Agent Infrastructure**\nBackground coding agents run in remote sandboxed environments rather than on local machines. Key patterns include pre-built environment images refreshed on regular cadence, warm sandbox pools for instant session starts, filesystem snapshots for session persistence, and multiplayer support for collaborative agent sessions. Critical optimizations include allowing file reads before git sync completes (blocking only writes), predictive sandbox warming when users start typing, and self-spawning agents for parallel task execution.\n\n**Tool Design Principles**\nTools are contracts between deterministic systems and non-deterministic agents. Effective tool design follows the consolidation principle (prefer single comprehensive tools over multiple narrow ones), returns contextual information in errors, supports response format options for token efficiency, and uses clear namespacing.\n\n### Operational Excellence\n\n**Context Compression**\nWhen agent sessions exhaust memory, compression becomes mandatory. The correct optimization target is tokens-per-task, not tokens-per-request. Structured summarization with explicit sections for files, decisions, and next steps preserves more useful information than aggressive compression. Artifact trail integrity remains the weakest dimension across all compression methods.\n\n**Context Optimization**\nTechniques include compaction (summarizing context near limits), observation masking (replacing verbose tool outputs with references), prefix caching (reusing KV blocks across requests), and strategic context partitioning (splitting work across sub-agents with isolated contexts).\n\n**Self-Managed Context**\nThe model, not the harness, decides what stays in its live context: the editable region is exposed as a file the model rewrites with code tools and re-parsed each turn. The harness keeps the invariants (pinned system and task prefix, role folding, edit gate, receipts, deterministic budget readouts, rollback on overflow). Edit cost scales with the text after the edit under prefix caching, so edits are batched and placed with the tail in mind; strategy can be steered by instruction, evolved as a skill, or trained with a success-gated efficiency reward.\n\n**Latent Briefing (KV Memory Sharing)**\nOrchestrator-worker systems can compound tokens when supervisors accumulate long trajectories but workers see only narrow text slices. Latent Briefing compacts the orchestrator trajectory in the worker model's KV cache using task-guided attention (Attention Matching-style compaction) so workers receive relevant latent state without full-text replay when the stack exposes worker KV state and the models are compatible.\n\n**Evaluation Frameworks**\nProduction agent evaluation requires deterministic checks and multi-dimensional rubrics covering factual accuracy, completeness, tool efficiency, and process quality. Use model judges only after structure, evidence, and rubric math are valid; route judge design, pairwise comparison, and bias mitigation to Advanced Evaluation.\n\n**Harness Engineering**\nReliable autonomous agents need explicit operating loops around the model: locked metrics, editable surfaces, durable logs, novelty checks, rollback rules, and human approval boundaries. Harnesses prevent agents from weakening the evaluator, losing state across compaction, or turning ambiguous goals into unreviewable changes.\n\n**Self-Improvement Loops**\nWhen the harness itself becomes the optimization target, a different discipline applies: recursive self-improvement, meta-harness search, failure-driven bounded self-edits, evolutionary scaffold search, and context mechanism evolution. The controlling constraints are empirical two-split acceptance gates, filesystem experience archives with raw traces, runtime-enforced constraints outside every editable surface, and diversity preservation to prevent collapse.\n\n### Development Methodology\n\n**Project Development**\nEffective LLM project development begins with task-model fit analysis: validating through manual prototyping that a task is well-suited for LLM processing before building automation. Production pipelines follow staged, idempotent architectures (acquire, prepare, process, parse, render) with file system state management for debugging and caching. Structured output design with explicit format specifications enables reliable parsing. Start with minimal architecture and add complexity only when proven necessary.\n\n### Cognitive Architecture\n\n**BDI Mental States**\nBelief-desire-intention modeling provides a formal way to translate structured external context into agent mental states. Use it for rational agency, explainability, and systems that need auditable links between beliefs, goals, and chosen actions.\n\n## Core Concepts\n\nThe collection is organized around four core themes. First, context fundamentals establish what context is, how attention mechanisms work, and why context quality matters more than quantity. Second, architectural patterns cover the structures and coordination mechanisms that enable effective agent systems. Third, operational excellence addresses optimization, evaluation, and harness reliability. Fourth, development methodology and cognitive architecture cover project execution and formal mental-state modeling.\n\n## Practical Guidance\n\nEach skill can be used independently or in combination. Start with fundamentals to establish context management mental models. Branch into architectural patterns based on your system requirements. Reference operational skills when optimizing production systems.\n\nThe skills are platform-agnostic and work with Claude Code, Cursor, or any agent framework that supports custom instructions or skill-like constructs.\n\n## Integration\n\nThis collection integrates with itself—skills reference each other and build on shared concepts. The fundamentals skill provides context for all other skills. Architectural skills (multi-agent, memory, tools) can be combined for complex systems. Operational skills (optimization, evaluation) apply to any system built using the foundational and architectural skills.\n\n## References\n\nInternal skills in this collection:\n- [context-fundamentals](skills/context-fundamentals/SKILL.md)\n- [context-degradation](skills/context-degradation/SKILL.md)\n- [context-compression](skills/context-compression/SKILL.md)\n- [multi-agent-patterns](skills/multi-agent-patterns/SKILL.md)\n- [long-horizon-prompting](skills/long-horizon-prompting/SKILL.md)\n- [memory-systems](skills/memory-systems/SKILL.md)\n- [tool-design](skills/tool-design/SKILL.md)\n- [filesystem-context](skills/filesystem-context/SKILL.md)\n- [hosted-agents](skills/hosted-agents/SKILL.md)\n- [context-optimization](skills/context-optimization/SKILL.md)\n- [self-managed-context](skills/self-managed-context/SKILL.md)\n- [latent-briefing](skills/latent-briefing/SKILL.md)\n- [evaluation](skills/evaluation/SKILL.md)\n- [advanced-evaluation](skills/advanced-evaluation/SKILL.md)\n- [harness-engineering](skills/harness-engineering/SKILL.md)\n- [self-improvement-loops](skills/self-improvement-loops/SKILL.md)\n- [project-development](skills/project-development/SKILL.md)\n- [bdi-mental-states](skills/bdi-mental-states/SKILL.md)\n\nExternal resources on context engineering:\n- Research on attention mechanisms and context window limitations\n- Production experience from leading AI labs on agent system design\n- Framework documentation for LangGraph, AutoGen, and CrewAI\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-20\n**Last Updated**: 2026-10-01\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.6.0\n","frontmatter":{"name":"context-engineering-collection","description":"A comprehensive collection of Agent Skills for context engineering, harness engineering, multi-agent architectures, and production agent systems. Use when building, optimizing, evaluating, or debugging agent systems that require effective context management and reliable operating loops."},"isInternal":false,"tokens":2045,"sizeBytes":11210},{"name":"bias-mitigation.md","path":"skills/advanced-evaluation/references/bias-mitigation.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/advanced-evaluation/references/bias-mitigation.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Bias Mitigation Techniques for LLM Evaluation\n\nThis reference details specific techniques for mitigating known biases in LLM-as-a-Judge systems.\n\n## Position Bias\n\n### The Problem\n\nIn pairwise comparison, LLMs systematically prefer responses in certain positions. Research shows:\n- GPT has mild first-position bias (~55% preference for first position in ties)\n- Claude shows similar patterns\n- Smaller models often show stronger bias\n\n### Mitigation: Position Swapping Protocol\n\n```python\nasync def position_swap_comparison(response_a, response_b, prompt, criteria):\n # Pass 1: Original order\n result_ab = await compare(response_a, response_b, prompt, criteria)\n \n # Pass 2: Swapped order\n result_ba = await compare(response_b, response_a, prompt, criteria)\n \n # Map second result (A in second position → B in first)\n result_ba_mapped = {\n 'winner': {'A': 'B', 'B': 'A', 'TIE': 'TIE'}[result_ba['winner']],\n 'confidence': result_ba['confidence']\n }\n \n # Consistency check\n if result_ab['winner'] == result_ba_mapped['winner']:\n return {\n 'winner': result_ab['winner'],\n 'confidence': (result_ab['confidence'] + result_ba_mapped['confidence']) / 2,\n 'position_consistent': True\n }\n else:\n # Disagreement indicates position bias was a factor\n return {\n 'winner': 'TIE',\n 'confidence': 0.5,\n 'position_consistent': False,\n 'bias_detected': True\n }\n```\n\n### Alternative: Multiple Shuffles\n\nFor higher reliability, use multiple position orderings:\n\n```python\nasync def multi_shuffle_comparison(response_a, response_b, prompt, criteria, n_shuffles=3):\n results = []\n for i in range(n_shuffles):\n if i % 2 == 0:\n r = await compare(response_a, response_b, prompt, criteria)\n else:\n r = await compare(response_b, response_a, prompt, criteria)\n r['winner'] = {'A': 'B', 'B': 'A', 'TIE': 'TIE'}[r['winner']]\n results.append(r)\n \n # Majority vote\n winners = [r['winner'] for r in results]\n final_winner = max(set(winners), key=winners.count)\n agreement = winners.count(final_winner) / len(winners)\n \n return {\n 'winner': final_winner,\n 'confidence': agreement,\n 'n_shuffles': n_shuffles\n }\n```\n\n## Length Bias\n\n### The Problem\n\nLLMs tend to rate longer responses higher, regardless of quality. This manifests as:\n- Verbose responses receiving inflated scores\n- Concise but complete responses penalized\n- Padding and repetition being rewarded\n\n### Mitigation: Explicit Prompting\n\nInclude anti-length-bias instructions in the prompt:\n\n```\nCRITICAL EVALUATION GUIDELINES:\n- Do NOT prefer responses because they are longer\n- Concise, complete answers are as valuable as detailed ones\n- Penalize unnecessary verbosity or repetition\n- Focus on information density, not word count\n```\n\n### Mitigation: Length-Normalized Scoring\n\n```python\ndef length_normalized_score(score, response_length, target_length=500):\n \"\"\"Adjust score based on response length.\"\"\"\n length_ratio = response_length / target_length\n \n if length_ratio > 2.0:\n # Penalize excessively long responses\n penalty = (length_ratio - 2.0) * 0.1\n return max(score - penalty, 1)\n elif length_ratio < 0.3:\n # Penalize excessively short responses\n penalty = (0.3 - length_ratio) * 0.5\n return max(score - penalty, 1)\n else:\n return score\n```\n\n### Mitigation: Separate Length Criterion\n\nMake length a separate, explicit criterion so it's not implicitly rewarded:\n\n```python\ncriteria = [\n {\"name\": \"Accuracy\", \"description\": \"Factual correctness\", \"weight\": 0.4},\n {\"name\": \"Completeness\", \"description\": \"Covers key points\", \"weight\": 0.3},\n {\"name\": \"Conciseness\", \"description\": \"No unnecessary content\", \"weight\": 0.3} # Explicit\n]\n```\n\n## Self-Enhancement Bias\n\n### The Problem\n\nModels rate outputs generated by themselves (or similar models) higher than outputs from different models.\n\n### Mitigation: Cross-Model Evaluation\n\nUse a different model family for evaluation than generation:\n\n```python\ndef get_evaluator_model(generator_model):\n \"\"\"Select evaluator to avoid self-enhancement bias.\"\"\"\n if 'gpt' in generator_model.lower():\n return 'claude-4-5-sonnet'\n elif 'claude' in generator_model.lower():\n return 'gpt-5.2'\n else:\n return 'gpt-5.2' # Default\n```\n\n### Mitigation: Blind Evaluation\n\nRemove model attribution from responses before evaluation:\n\n```python\ndef anonymize_response(response, model_name):\n \"\"\"Remove model-identifying patterns.\"\"\"\n patterns = [\n f\"As {model_name}\",\n \"I am an AI\",\n \"I don't have personal opinions\",\n # Model-specific patterns\n ]\n anonymized = response\n for pattern in patterns:\n anonymized = anonymized.replace(pattern, \"[REDACTED]\")\n return anonymized\n```\n\n## Verbosity Bias\n\n### The Problem\n\nDetailed explanations receive higher scores even when the extra detail is irrelevant or incorrect.\n\n### Mitigation: Relevance-Weighted Scoring\n\n```python\nasync def relevance_weighted_evaluation(response, prompt, criteria):\n # First, assess relevance of each segment\n relevance_scores = await assess_relevance(response, prompt)\n \n # Weight evaluation by relevance\n segments = split_into_segments(response)\n weighted_scores = []\n for segment, relevance in zip(segments, relevance_scores):\n if relevance > 0.5: # Only count relevant segments\n score = await evaluate_segment(segment, prompt, criteria)\n weighted_scores.append(score * relevance)\n \n return sum(weighted_scores) / len(weighted_scores)\n```\n\n### Mitigation: Rubric with Verbosity Penalty\n\nInclude explicit verbosity penalties in rubrics:\n\n```python\nrubric_levels = [\n {\n \"score\": 5,\n \"description\": \"Complete and concise. All necessary information, nothing extraneous.\",\n \"characteristics\": [\"Every sentence adds value\", \"No repetition\", \"Appropriately scoped\"]\n },\n {\n \"score\": 3,\n \"description\": \"Complete but verbose. Contains unnecessary detail or repetition.\",\n \"characteristics\": [\"Main points covered\", \"Some tangents\", \"Could be more concise\"]\n },\n # ... etc\n]\n```\n\n## Authority Bias\n\n### The Problem\n\nConfident, authoritative tone is rated higher regardless of accuracy.\n\n### Mitigation: Evidence Requirement\n\nRequire explicit evidence for claims:\n\n```\nFor each claim in the response:\n1. Identify whether it's a factual claim\n2. Note if evidence or sources are provided\n3. Score based on verifiability, not confidence\n\nIMPORTANT: Confident claims without evidence should NOT receive higher scores than \nhedged claims with evidence.\n```\n\n### Mitigation: Fact-Checking Layer\n\nAdd a fact-checking step before scoring:\n\n```python\nasync def fact_checked_evaluation(response, prompt, criteria):\n # Extract claims\n claims = await extract_claims(response)\n \n # Fact-check each claim\n fact_check_results = await asyncio.gather(*[\n verify_claim(claim) for claim in claims\n ])\n \n # Adjust score based on fact-check results\n accuracy_factor = sum(r['verified'] for r in fact_check_results) / len(fact_check_results)\n \n base_score = await evaluate(response, prompt, criteria)\n return base_score * (0.7 + 0.3 * accuracy_factor) # At least 70% of score\n```\n\n## Aggregate Bias Detection\n\nMonitor for systematic biases in production:\n\n```python\nclass BiasMonitor:\n def __init__(self):\n self.evaluations = []\n \n def record(self, evaluation):\n self.evaluations.append(evaluation)\n \n def detect_position_bias(self):\n \"\"\"Detect if first position wins more often than expected.\"\"\"\n first_wins = sum(1 for e in self.evaluations if e['first_position_winner'])\n expected = len(self.evaluations) * 0.5\n z_score = (first_wins - expected) / (expected * 0.5) ** 0.5\n return {'bias_detected': abs(z_score) > 2, 'z_score': z_score}\n \n def detect_length_bias(self):\n \"\"\"Detect if longer responses score higher.\"\"\"\n from scipy.stats import spearmanr\n lengths = [e['response_length'] for e in self.evaluations]\n scores = [e['score'] for e in self.evaluations]\n corr, p_value = spearmanr(lengths, scores)\n return {'bias_detected': corr > 0.3 and p_value < 0.05, 'correlation': corr}\n```\n\n## Summary Table\n\n| Bias | Primary Mitigation | Secondary Mitigation | Detection Method |\n|------|-------------------|---------------------|------------------|\n| Position | Position swapping | Multiple shuffles | Consistency check |\n| Length | Explicit prompting | Length normalization | Length-score correlation |\n| Self-enhancement | Cross-model evaluation | Anonymization | Model comparison study |\n| Verbosity | Relevance weighting | Rubric penalties | Relevance scoring |\n| Authority | Evidence requirement | Fact-checking layer | Confidence-accuracy correlation |\n\n","isInternal":false,"tokens":2083,"sizeBytes":9078},{"name":"evaluation-pipeline.md","path":"skills/advanced-evaluation/references/evaluation-pipeline.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/advanced-evaluation/references/evaluation-pipeline.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Evaluation Pipeline Diagram\n\nVisual layout of a production evaluation pipeline.\n\n```\n┌─────────────────────────────────────────────────┐\n│ Evaluation Pipeline │\n├─────────────────────────────────────────────────┤\n│ │\n│ Input: Response + Prompt + Context │\n│ │ │\n│ ▼ │\n│ ┌─────────────────────┐ │\n│ │ Criteria Loader │ ◄── Rubrics, weights │\n│ └──────────┬──────────┘ │\n│ │ │\n│ ▼ │\n│ ┌─────────────────────┐ │\n│ │ Primary Scorer │ ◄── Direct or Pairwise │\n│ └──────────┬──────────┘ │\n│ │ │\n│ ▼ │\n│ ┌─────────────────────┐ │\n│ │ Bias Mitigation │ ◄── Position swap, etc. │\n│ └──────────┬──────────┘ │\n│ │ │\n│ ▼ │\n│ ┌─────────────────────┐ │\n│ │ Confidence Scoring │ ◄── Calibration │\n│ └──────────┬──────────┘ │\n│ │ │\n│ ▼ │\n│ Output: Scores + Justifications + Confidence │\n│ │\n└─────────────────────────────────────────────────┘\n```\n\n## Pipeline Stages\n\n1. **Criteria Loader**: Loads rubrics and criterion weights from configuration\n2. **Primary Scorer**: Applies direct scoring or pairwise comparison\n3. **Bias Mitigation**: Runs position swaps, length normalization, and other debiasing\n4. **Confidence Scoring**: Calibrates confidence based on position consistency and evidence strength\n","isInternal":false,"tokens":415,"sizeBytes":2891},{"name":"implementation-patterns.md","path":"skills/advanced-evaluation/references/implementation-patterns.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/advanced-evaluation/references/implementation-patterns.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# LLM-as-Judge Implementation Patterns\n\nThis reference provides detailed implementation patterns for building production-grade LLM evaluation systems.\n\n## Pattern 1: Structured Evaluation Pipeline\n\nThe most reliable evaluation systems follow a structured pipeline that separates concerns:\n\n```\nInput Validation → Criteria Loading → Scoring → Bias Mitigation → Output Formatting\n```\n\n### Input Validation Layer\n\nBefore evaluation begins, validate:\n\n1. **Response presence**: Non-empty response to evaluate\n2. **Prompt presence**: Original prompt for context\n3. **Criteria validity**: At least one criterion with name and description\n4. **Weight normalization**: Weights sum to 1.0 (or normalize them)\n\n```python\ndef validate_input(response, prompt, criteria):\n if not response or not response.strip():\n raise ValueError(\"Response cannot be empty\")\n if not prompt or not prompt.strip():\n raise ValueError(\"Prompt cannot be empty\")\n if not criteria or len(criteria) == 0:\n raise ValueError(\"At least one criterion required\")\n \n # Normalize weights\n total_weight = sum(c.get('weight', 1) for c in criteria)\n for c in criteria:\n c['weight'] = c.get('weight', 1) / total_weight\n```\n\n### Criteria Loading Layer\n\nCriteria should be loaded from configuration, not hardcoded:\n\n```python\nclass CriteriaLoader:\n def __init__(self, rubric_path=None):\n self.rubrics = self._load_rubrics(rubric_path)\n \n def get_criteria(self, task_type):\n return self.rubrics.get(task_type, self.default_criteria)\n \n def get_rubric(self, criterion_name):\n return self.rubrics.get(criterion_name, {}).get('levels', [])\n```\n\n### Scoring Layer\n\nThe scoring layer handles the actual LLM call:\n\n```python\nasync def score_response(response, prompt, criteria, rubric, model):\n system_prompt = build_system_prompt(criteria, rubric)\n user_prompt = build_user_prompt(response, prompt, criteria)\n \n result = await generate_text(\n model=model,\n system=system_prompt,\n prompt=user_prompt,\n temperature=0.3 # Lower temperature for consistency\n )\n \n return parse_scores(result.text)\n```\n\n### Bias Mitigation Layer\n\nFor pairwise comparison, always include position swapping:\n\n```python\nasync def compare_with_bias_mitigation(response_a, response_b, prompt, criteria, model):\n # First pass: A first\n pass1 = await compare_pair(response_a, response_b, prompt, criteria, model)\n \n # Second pass: B first\n pass2 = await compare_pair(response_b, response_a, prompt, criteria, model)\n \n # Map pass2 winner back\n pass2_mapped = map_winner(pass2.winner) # A→B, B→A, TIE→TIE\n \n # Check consistency\n if pass1.winner == pass2_mapped:\n return {\n 'winner': pass1.winner,\n 'confidence': (pass1.confidence + pass2.confidence) / 2,\n 'consistent': True\n }\n else:\n return {\n 'winner': 'TIE',\n 'confidence': 0.5,\n 'consistent': False\n }\n```\n\n## Pattern 2: Hierarchical Evaluation\n\nFor complex evaluations, use a hierarchical approach:\n\n```\nQuick Screen (cheap model) → Detailed Evaluation (expensive model) → Human Review (edge cases)\n```\n\n### Quick Screen Implementation\n\n```python\nasync def quick_screen(response, prompt, threshold=0.7):\n \"\"\"Fast, cheap screening for obvious passes/fails.\"\"\"\n result = await generate_text(\n model='gpt-5.2', # Cheaper model\n prompt=f\"Rate 0-1 if this response adequately addresses the prompt:\\n\\nPrompt: {prompt}\\n\\nResponse: {response}\",\n temperature=0\n )\n score = float(result.text.strip())\n return score, score > threshold\n```\n\n### Detailed Evaluation\n\n```python\nasync def detailed_evaluation(response, prompt, criteria):\n \"\"\"Full evaluation for borderline or important cases.\"\"\"\n result = await generate_text(\n model='gpt-5.2', # More capable model\n system=DETAILED_EVALUATION_PROMPT,\n prompt=build_detailed_prompt(response, prompt, criteria),\n temperature=0.3\n )\n return parse_detailed_scores(result.text)\n```\n\n## Pattern 3: Panel of LLM Judges (PoLL)\n\nFor high-stakes evaluation, use multiple models:\n\n```python\nasync def poll_evaluation(response, prompt, criteria, models):\n \"\"\"Aggregate judgments from multiple LLM judges.\"\"\"\n results = await asyncio.gather(*[\n score_with_model(response, prompt, criteria, model)\n for model in models\n ])\n \n # Aggregate scores\n aggregated = aggregate_scores(results)\n \n # Calculate agreement\n agreement = calculate_agreement(results)\n \n return {\n 'scores': aggregated,\n 'agreement': agreement,\n 'individual_results': results\n }\n\ndef aggregate_scores(results):\n \"\"\"Aggregate scores using median (robust to outliers).\"\"\"\n scores = {}\n for criterion in results[0]['scores'].keys():\n criterion_scores = [r['scores'][criterion] for r in results]\n scores[criterion] = {\n 'score': statistics.median(criterion_scores),\n 'std': statistics.stdev(criterion_scores) if len(criterion_scores) > 1 else 0\n }\n return scores\n```\n\n## Pattern 4: Confidence Calibration\n\nConfidence scores should be calibrated to actual reliability:\n\n```python\ndef calibrate_confidence(raw_confidence, position_consistent, evidence_count):\n \"\"\"Calibrate confidence based on multiple signals.\"\"\"\n \n # Base confidence from model output\n calibrated = raw_confidence\n \n # Position consistency is a strong signal\n if not position_consistent:\n calibrated *= 0.6 # Significant reduction\n \n # More evidence = higher confidence\n evidence_factor = min(evidence_count / 3, 1.0) # Cap at 3 pieces\n calibrated *= (0.7 + 0.3 * evidence_factor)\n \n return min(calibrated, 0.99) # Never 100% confident\n```\n\n## Pattern 5: Output Formatting\n\nAlways return structured outputs with consistent schemas:\n\n```python\n@dataclass\nclass ScoreResult:\n criterion: str\n score: float\n max_score: float\n justification: str\n evidence: List[str]\n improvement: str\n\n@dataclass\nclass EvaluationResult:\n success: bool\n scores: List[ScoreResult]\n overall_score: float\n weighted_score: float\n summary: Dict[str, Any]\n metadata: Dict[str, Any]\n\ndef format_output(scores, metadata) -> EvaluationResult:\n \"\"\"Format evaluation results consistently.\"\"\"\n return EvaluationResult(\n success=True,\n scores=scores,\n overall_score=sum(s.score for s in scores) / len(scores),\n weighted_score=calculate_weighted_score(scores),\n summary=generate_summary(scores),\n metadata=metadata\n )\n```\n\n## Error Handling Patterns\n\n### Graceful Degradation\n\n```python\nasync def evaluate_with_fallback(response, prompt, criteria):\n try:\n return await full_evaluation(response, prompt, criteria)\n except RateLimitError:\n # Fall back to simpler evaluation\n return await simple_evaluation(response, prompt, criteria)\n except ParseError as e:\n # Return partial results with error flag\n return {\n 'success': False,\n 'partial_results': e.partial_data,\n 'error': str(e)\n }\n```\n\n### Retry Logic\n\n```python\nasync def evaluate_with_retry(response, prompt, criteria, max_retries=3):\n for attempt in range(max_retries):\n try:\n result = await evaluate(response, prompt, criteria)\n if is_valid_result(result):\n return result\n except TransientError:\n await asyncio.sleep(2 ** attempt) # Exponential backoff\n \n raise EvaluationError(\"Max retries exceeded\")\n```\n\n## Testing Patterns\n\n### Unit Tests for Parsing\n\n```python\ndef test_score_parsing():\n raw_output = '{\"scores\": [{\"criterion\": \"Accuracy\", \"score\": 4}]}'\n result = parse_scores(raw_output)\n assert result.scores[0].criterion == \"Accuracy\"\n assert result.scores[0].score == 4\n\ndef test_malformed_output():\n raw_output = 'Invalid JSON'\n with pytest.raises(ParseError):\n parse_scores(raw_output)\n```\n\n### Integration Tests with Real API\n\n```python\n@pytest.mark.integration\nasync def test_full_evaluation_pipeline():\n result = await evaluate(\n response=\"Water boils at 100°C at sea level.\",\n prompt=\"At what temperature does water boil?\",\n criteria=[{\"name\": \"Accuracy\", \"description\": \"Factual correctness\", \"weight\": 1}]\n )\n \n assert result.success\n assert len(result.scores) == 1\n assert result.scores[0].score >= 4 # Should score high for accurate response\n```\n\n### Bias Detection Tests\n\n```python\nasync def test_position_bias_mitigation():\n # Same response in both positions should tie\n result = await compare(\n response_a=\"Same response\",\n response_b=\"Same response\",\n prompt=\"Test prompt\",\n criteria=[\"quality\"],\n swap_positions=True\n )\n \n assert result.winner == \"TIE\"\n assert result.consistent == True\n```\n\n","isInternal":false,"tokens":2019,"sizeBytes":9061},{"name":"metrics-guide.md","path":"skills/advanced-evaluation/references/metrics-guide.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/advanced-evaluation/references/metrics-guide.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Metric Selection Guide for LLM Evaluation\n\nThis reference provides guidance on selecting appropriate metrics for different evaluation scenarios.\n\n## Metric Categories\n\n### Classification Metrics\n\nUse for binary or multi-class evaluation tasks (pass/fail, correct/incorrect).\n\n#### Precision\n\n```\nPrecision = True Positives / (True Positives + False Positives)\n```\n\n**Interpretation**: Of all responses the judge said were good, what fraction were actually good?\n\n**Use when**: False positives are costly (e.g., approving unsafe content)\n\n```python\ndef precision(predictions, ground_truth):\n true_positives = sum(1 for p, g in zip(predictions, ground_truth) if p == 1 and g == 1)\n predicted_positives = sum(predictions)\n return true_positives / predicted_positives if predicted_positives > 0 else 0\n```\n\n#### Recall\n\n```\nRecall = True Positives / (True Positives + False Negatives)\n```\n\n**Interpretation**: Of all actually good responses, what fraction did the judge identify?\n\n**Use when**: False negatives are costly (e.g., missing good content in filtering)\n\n```python\ndef recall(predictions, ground_truth):\n true_positives = sum(1 for p, g in zip(predictions, ground_truth) if p == 1 and g == 1)\n actual_positives = sum(ground_truth)\n return true_positives / actual_positives if actual_positives > 0 else 0\n```\n\n#### F1 Score\n\n```\nF1 = 2 * (Precision * Recall) / (Precision + Recall)\n```\n\n**Interpretation**: Harmonic mean of precision and recall\n\n**Use when**: You need a single number balancing both concerns\n\n```python\ndef f1_score(predictions, ground_truth):\n p = precision(predictions, ground_truth)\n r = recall(predictions, ground_truth)\n return 2 * p * r / (p + r) if (p + r) > 0 else 0\n```\n\n### Agreement Metrics\n\nUse for comparing automated evaluation with human judgment.\n\n#### Cohen's Kappa (κ)\n\n```\nκ = (Observed Agreement - Expected Agreement) / (1 - Expected Agreement)\n```\n\n**Interpretation**: Agreement adjusted for chance\n- κ > 0.8: Almost perfect agreement\n- κ 0.6-0.8: Substantial agreement\n- κ 0.4-0.6: Moderate agreement\n- κ < 0.4: Fair to poor agreement\n\n**Use for**: Binary or categorical judgments\n\n```python\ndef cohens_kappa(judge1, judge2):\n from sklearn.metrics import cohen_kappa_score\n return cohen_kappa_score(judge1, judge2)\n```\n\n#### Weighted Kappa\n\nFor ordinal scales where disagreement severity matters:\n\n```python\ndef weighted_kappa(judge1, judge2):\n from sklearn.metrics import cohen_kappa_score\n return cohen_kappa_score(judge1, judge2, weights='quadratic')\n```\n\n**Interpretation**: Penalizes large disagreements more than small ones\n\n### Correlation Metrics\n\nUse for ordinal/continuous scores.\n\n#### Spearman's Rank Correlation (ρ)\n\n**Interpretation**: Correlation between rankings, not absolute values\n- ρ > 0.9: Very strong correlation\n- ρ 0.7-0.9: Strong correlation\n- ρ 0.5-0.7: Moderate correlation\n- ρ < 0.5: Weak correlation\n\n**Use when**: Order matters more than exact values\n\n```python\ndef spearmans_rho(scores1, scores2):\n from scipy.stats import spearmanr\n rho, p_value = spearmanr(scores1, scores2)\n return {'rho': rho, 'p_value': p_value}\n```\n\n#### Kendall's Tau (τ)\n\n**Interpretation**: Similar to Spearman but based on pairwise concordance\n\n**Use when**: You have many tied values\n\n```python\ndef kendalls_tau(scores1, scores2):\n from scipy.stats import kendalltau\n tau, p_value = kendalltau(scores1, scores2)\n return {'tau': tau, 'p_value': p_value}\n```\n\n#### Pearson Correlation (r)\n\n**Interpretation**: Linear correlation between scores\n\n**Use when**: Exact score values matter, not just order\n\n```python\ndef pearsons_r(scores1, scores2):\n from scipy.stats import pearsonr\n r, p_value = pearsonr(scores1, scores2)\n return {'r': r, 'p_value': p_value}\n```\n\n### Pairwise Comparison Metrics\n\n#### Agreement Rate\n\n```\nAgreement = (Matching Decisions) / (Total Comparisons)\n```\n\n**Interpretation**: Simple percentage of agreement\n\n```python\ndef pairwise_agreement(decisions1, decisions2):\n matches = sum(1 for d1, d2 in zip(decisions1, decisions2) if d1 == d2)\n return matches / len(decisions1)\n```\n\n#### Position Consistency\n\n```\nConsistency = (Consistent across position swaps) / (Total comparisons)\n```\n\n**Interpretation**: How often does swapping position change the decision?\n\n```python\ndef position_consistency(results):\n consistent = sum(1 for r in results if r['position_consistent'])\n return consistent / len(results)\n```\n\n## Selection Decision Tree\n\n```\nWhat type of evaluation task?\n│\n├── Binary classification (pass/fail)\n│ └── Use: Precision, Recall, F1, Cohen's κ\n│\n├── Ordinal scale (1-5 rating)\n│ ├── Comparing to human judgments?\n│ │ └── Use: Spearman's ρ, Weighted κ\n│ └── Comparing two automated judges?\n│ └── Use: Kendall's τ, Spearman's ρ\n│\n├── Pairwise preference\n│ └── Use: Agreement rate, Position consistency\n│\n└── Multi-label classification\n └── Use: Macro-F1, Micro-F1, Per-label metrics\n```\n\n## Metric Selection by Use Case\n\n### Use Case 1: Validating Automated Evaluation\n\n**Goal**: Ensure automated evaluation correlates with human judgment\n\n**Recommended Metrics**:\n1. Primary: Spearman's ρ (for ordinal scales) or Cohen's κ (for categorical)\n2. Secondary: Per-criterion agreement\n3. Diagnostic: Confusion matrix for systematic errors\n\n```python\ndef validate_automated_eval(automated_scores, human_scores, criteria):\n results = {}\n \n # Overall correlation\n results['overall_spearman'] = spearmans_rho(automated_scores, human_scores)\n \n # Per-criterion agreement\n for criterion in criteria:\n auto_crit = [s[criterion] for s in automated_scores]\n human_crit = [s[criterion] for s in human_scores]\n results[f'{criterion}_spearman'] = spearmans_rho(auto_crit, human_crit)\n \n return results\n```\n\n### Use Case 2: Comparing Two Models\n\n**Goal**: Determine which model produces better outputs\n\n**Recommended Metrics**:\n1. Primary: Win rate (from pairwise comparison)\n2. Secondary: Position consistency (bias check)\n3. Diagnostic: Per-criterion breakdown\n\n```python\ndef compare_models(model_a_outputs, model_b_outputs, prompts):\n results = []\n for a, b, p in zip(model_a_outputs, model_b_outputs, prompts):\n comparison = await compare_with_position_swap(a, b, p)\n results.append(comparison)\n \n return {\n 'a_wins': sum(1 for r in results if r['winner'] == 'A'),\n 'b_wins': sum(1 for r in results if r['winner'] == 'B'),\n 'ties': sum(1 for r in results if r['winner'] == 'TIE'),\n 'position_consistency': position_consistency(results)\n }\n```\n\n### Use Case 3: Quality Monitoring\n\n**Goal**: Track evaluation quality over time\n\n**Recommended Metrics**:\n1. Primary: Rolling agreement with human spot-checks\n2. Secondary: Score distribution stability\n3. Diagnostic: Bias indicators (position, length)\n\n```python\nclass QualityMonitor:\n def __init__(self, window_size=100):\n self.window = deque(maxlen=window_size)\n \n def add_evaluation(self, automated, human_spot_check=None):\n self.window.append({\n 'automated': automated,\n 'human': human_spot_check,\n 'length': len(automated['response'])\n })\n \n def get_metrics(self):\n # Filter to evaluations with human spot-checks\n with_human = [e for e in self.window if e['human'] is not None]\n \n if len(with_human) < 10:\n return {'insufficient_data': True}\n \n auto_scores = [e['automated']['score'] for e in with_human]\n human_scores = [e['human']['score'] for e in with_human]\n \n return {\n 'correlation': spearmans_rho(auto_scores, human_scores),\n 'mean_difference': np.mean([a - h for a, h in zip(auto_scores, human_scores)]),\n 'length_correlation': spearmans_rho(\n [e['length'] for e in self.window],\n [e['automated']['score'] for e in self.window]\n )\n }\n```\n\n## Interpreting Metric Results\n\n### Good Evaluation System Indicators\n\n| Metric | Good | Acceptable | Concerning |\n|--------|------|------------|------------|\n| Spearman's ρ | > 0.8 | 0.6-0.8 | < 0.6 |\n| Cohen's κ | > 0.7 | 0.5-0.7 | < 0.5 |\n| Position consistency | > 0.9 | 0.8-0.9 | < 0.8 |\n| Length correlation | < 0.2 | 0.2-0.4 | > 0.4 |\n\n### Warning Signs\n\n1. **High agreement but low correlation**: May indicate calibration issues\n2. **Low position consistency**: Position bias affecting results\n3. **High length correlation**: Length bias inflating scores\n4. **Per-criterion variance**: Some criteria may be poorly defined\n\n## Reporting Template\n\n```markdown\n## Evaluation System Metrics Report\n\n### Human Agreement\n- Spearman's ρ: 0.82 (p < 0.001)\n- Cohen's κ: 0.74\n- Sample size: 500 evaluations\n\n### Bias Indicators\n- Position consistency: 91%\n- Length-score correlation: 0.12\n\n### Per-Criterion Performance\n| Criterion | Spearman's ρ | κ |\n|-----------|--------------|---|\n| Accuracy | 0.88 | 0.79 |\n| Clarity | 0.76 | 0.68 |\n| Completeness | 0.81 | 0.72 |\n\n### Recommendations\n- All metrics within acceptable ranges\n- Monitor \"Clarity\" criterion - lower agreement may indicate need for rubric refinement\n```\n\n","isInternal":false,"tokens":2379,"sizeBytes":9333},{"name":"SKILL.md","path":"skills/advanced-evaluation/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/advanced-evaluation/SKILL.md","title":"advanced-evaluation","category":"anthropic-skill","format":"markdown","content":"---\nname: advanced-evaluation\ndescription: \"This skill should be used for advanced LLM evaluation: LLM-as-judge systems, direct scoring, pairwise comparison, rubric calibration, evaluator bias mitigation, confidence scoring, and automated quality assessment.\"\n---\n\n# Advanced Evaluation\n\nThis skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.\n\n**Key insight**: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.\n\n## When to Activate\n\nActivate this skill when:\n\n- Building LLM-as-judge systems for LLM outputs\n- Comparing multiple model responses to select the best one\n- Establishing consistent quality standards across evaluation teams\n- Debugging evaluation systems that show inconsistent results\n- Designing A/B tests for prompt or model changes\n- Creating rubrics specifically for LLM or human/LLM hybrid judges\n- Analyzing correlation between automated and human judgments\n\nDo not activate this skill for adjacent work owned by other skills:\n- General deterministic checks, regression suites, production quality gates, or outcome metrics: `evaluation`.\n- Autonomous loop governance, locked rubrics, rollback, or PR approval boundaries: `harness-engineering`.\n- Tool API contracts for evaluation tools: `tool-design`.\n\n## Core Concepts\n\n### The Evaluation Taxonomy\n\nSelect between two primary approaches based on whether ground truth exists:\n\n**Direct Scoring** — Use when objective criteria exist (factual accuracy, instruction following, toxicity). A single LLM rates one response on a defined scale. Achieves moderate-to-high reliability for well-defined criteria. Watch for score calibration drift and inconsistent scale interpretation.\n\n**Pairwise Comparison** — Use for subjective preferences (tone, style, persuasiveness). An LLM compares two responses and selects the better one. Pairwise methods often correlate better with human preference than open-ended direct scoring for subjective tasks (claim-advanced-evaluation-position-swap). Watch for position bias and length bias.\n\n### The Bias Landscape\n\nMitigate these systematic biases in every evaluation system:\n\n**Position Bias**: First-position responses get preferential treatment. Mitigate by evaluating twice with swapped positions, then apply majority vote or consistency check.\n\n**Length Bias**: Longer responses score higher regardless of quality. Mitigate by explicitly prompting to ignore length and applying length-normalized scoring.\n\n**Self-Enhancement Bias**: Models rate their own outputs higher. Mitigate by using different models for generation and evaluation.\n\n**Verbosity Bias**: Excessive detail scores higher even when unnecessary. Mitigate with criteria-specific rubrics that penalize irrelevant detail.\n\n**Authority Bias**: Confident tone scores higher regardless of accuracy. Mitigate by requiring evidence citation and adding a fact-checking layer.\n\n### Metric Selection Framework\n\nMatch metrics to the evaluation task structure:\n\n| Task Type | Primary Metrics | Secondary Metrics |\n|-----------|-----------------|-------------------|\n| Binary classification (pass/fail) | Recall, Precision, F1 | Cohen's kappa |\n| Ordinal scale (1-5 rating) | Spearman's rho, Kendall's tau | Cohen's kappa (weighted) |\n| Pairwise preference | Agreement rate, Position consistency | Confidence calibration |\n| Multi-label | Macro-F1, Micro-F1 | Per-label precision/recall |\n\nPrioritize systematic disagreement patterns over absolute agreement rates because a judge that consistently disagrees with humans on specific criteria is more problematic than one with random noise.\n\n## Evaluation Approaches\n\n### Direct Scoring Implementation\n\nBuild direct scoring with three components: clear criteria, a calibrated scale, and structured output format.\n\n**Criteria Definition Pattern**:\n```\nCriterion: [Name]\nDescription: [What this criterion measures]\nWeight: [Relative importance, 0-1]\n```\n\n**Scale Calibration** — Choose scale granularity based on rubric detail:\n- 1-3: Binary with neutral option, lowest cognitive load\n- 1-5: Standard Likert, best balance of granularity and reliability\n- 1-10: Use only with detailed per-level rubrics because calibration is harder\n\n**Prompt Structure for Direct Scoring**:\n```\nYou are an expert evaluator assessing response quality.\n\n## Task\nEvaluate the following response against each criterion.\n\n## Original Prompt\n{prompt}\n\n## Response to Evaluate\n{response}\n\n## Criteria\n{for each criterion: name, description, weight}\n\n## Instructions\nFor each criterion:\n1. Find specific evidence in the response\n2. Score according to the rubric (1-{max} scale)\n3. Justify your score with evidence\n4. Suggest one specific improvement\n\n## Output Format\nRespond with structured JSON containing scores, justifications, and summary.\n```\n\nRequire evidence before the score in scoring prompts so the judge must anchor its decision in observable output features before emitting a number.\n\n### Pairwise Comparison Implementation\n\nApply position bias mitigation in every pairwise evaluation:\n\n1. Run deterministic pre-checks first: both candidates must satisfy the same schema, source-evidence requirements, and scope constraints.\n2. First judge pass: Response A in first position, Response B in second.\n3. Second judge pass: Response B in first position, Response A in second.\n4. Consistency check: If passes disagree, return TIE with reduced confidence.\n5. Final verdict: Consistent winner with averaged confidence and explicit tie-breaker rationale.\n\n**Prompt Structure for Pairwise Comparison**:\n```\nYou are an expert evaluator comparing two AI responses.\n\n## Critical Instructions\n- Do NOT prefer responses because they are longer\n- Do NOT prefer responses based on position (first vs second)\n- Focus ONLY on quality according to the specified criteria\n- Ties are acceptable when responses are genuinely equivalent\n\n## Original Prompt\n{prompt}\n\n## Response A\n{response_a}\n\n## Response B\n{response_b}\n\n## Comparison Criteria\n{criteria list}\n\n## Instructions\n1. Analyze each response independently first\n2. Compare them on each criterion\n3. Determine overall winner with confidence level\n\n## Output Format\nJSON with per-criterion comparison, overall winner, confidence (0-1), and reasoning.\n```\n\n**Confidence Calibration** — Map confidence to position consistency:\n- Both passes agree: confidence = average of individual confidences\n- Passes disagree: confidence = 0.5, verdict = TIE\n\n### Rubric Generation\n\nGenerate rubrics to reduce evaluation variance compared to open-ended scoring. Treat exact variance reduction as workload-specific unless measured on the target eval set.\n\n**Include these rubric components**:\n1. **Level descriptions**: Clear boundaries for each score level\n2. **Characteristics**: Observable features that define each level\n3. **Examples**: Representative text for each level (optional but valuable)\n4. **Edge cases**: Guidance for ambiguous situations\n5. **Scoring guidelines**: General principles for consistent application\n\n**Set strictness calibration** for the use case:\n- **Lenient**: Lower passing bar, appropriate for encouraging iteration\n- **Balanced**: Typical production expectations\n- **Strict**: High standards for safety-critical or high-stakes evaluation\n\nAdapt rubrics to the domain — use domain-specific terminology. A code readability rubric mentions variables, functions, and comments. A medical accuracy rubric references clinical terminology and evidence standards.\n\n## Practical Guidance\n\n### Evaluation Pipeline Design\n\nBuild production evaluation systems with these layers: Criteria Loader (rubrics + weights) -> Primary Scorer (direct or pairwise) -> Bias Mitigation (position swap, etc.) -> Confidence Scoring (calibration) -> Output (scores + justifications + confidence). See [Evaluation Pipeline Diagram](./references/evaluation-pipeline.md) for the full visual layout.\n\n### Decision Framework: Direct vs. Pairwise\n\nApply this decision tree:\n\n```\nIs there an objective ground truth?\n+-- Yes -> Direct Scoring\n| Examples: factual accuracy, instruction following, format compliance\n|\n+-- No -> Is it a preference or quality judgment?\n +-- Yes -> Pairwise Comparison\n | Examples: tone, style, persuasiveness, creativity\n |\n +-- No -> Consider reference-based evaluation\n Examples: summarization (compare to source), translation (compare to reference)\n```\n\n### Scaling Evaluation\n\nFor high-volume evaluation, apply one of these strategies:\n\n1. **Panel of LLMs (PoLL)**: Use multiple models as judges and aggregate votes to reduce individual model bias. More expensive but more reliable for high-stakes decisions.\n\n2. **Hierarchical evaluation**: Use a fast cheap model for screening and an expensive model for edge cases. Requires calibration of the screening threshold.\n\n3. **Human-in-the-loop**: Automate clear cases and route low-confidence decisions to human review. Design feedback loops to improve automated evaluation over time.\n\n## Examples\n\n### Example 1: Direct Scoring for Accuracy\n\n**Input**:\n```\nPrompt: \"What causes seasons on Earth?\"\nResponse: \"Seasons are caused by Earth's tilted axis. As Earth orbits the Sun,\ndifferent hemispheres receive more direct sunlight at different times of year.\"\nCriterion: Factual Accuracy (weight: 1.0)\nScale: 1-5\n```\n\n**Output**:\n```json\n{\n \"criterion\": \"Factual Accuracy\",\n \"score\": 5,\n \"evidence\": [\n \"Correctly identifies axial tilt as primary cause\",\n \"Correctly explains differential sunlight by hemisphere\",\n \"No factual errors present\"\n ],\n \"justification\": \"Response accurately explains the cause of seasons with correct\nscientific reasoning. Both the axial tilt and its effect on sunlight distribution\nare correctly described.\",\n \"improvement\": \"Could add the specific tilt angle (23.5 degrees) for completeness.\"\n}\n```\n\n### Example 2: Pairwise Comparison with Position Swap\n\n**Input**:\n```\nPrompt: \"Explain machine learning to a beginner\"\nResponse A: [Technical explanation with jargon]\nResponse B: [Simple analogy-based explanation]\nCriteria: [\"clarity\", \"accessibility\"]\n```\n\n**First Pass (A first)**:\n```json\n{ \"winner\": \"B\", \"confidence\": 0.8 }\n```\n\n**Second Pass (B first)**:\n```json\n{ \"winner\": \"A\", \"confidence\": 0.6 }\n```\n(Note: Winner is A because B was in first position)\n\n**Mapped Second Pass**:\n```json\n{ \"winner\": \"B\", \"confidence\": 0.6 }\n```\n\n**Final Result**:\n```json\n{\n \"winner\": \"B\",\n \"confidence\": 0.7,\n \"positionConsistency\": {\n \"consistent\": true,\n \"firstPassWinner\": \"B\",\n \"secondPassWinner\": \"B\"\n }\n}\n```\n\n### Example 3: Rubric Generation\n\n**Input**:\n```\ncriterionName: \"Code Readability\"\ncriterionDescription: \"How easy the code is to understand and maintain\"\ndomain: \"software engineering\"\nscale: \"1-5\"\nstrictness: \"balanced\"\n```\n\n**Output** (abbreviated):\n```json\n{\n \"levels\": [\n {\n \"score\": 1,\n \"label\": \"Poor\",\n \"description\": \"Code is difficult to understand without significant effort\",\n \"characteristics\": [\n \"No meaningful variable or function names\",\n \"No comments or documentation\",\n \"Deeply nested or convoluted logic\"\n ]\n },\n {\n \"score\": 3,\n \"label\": \"Adequate\",\n \"description\": \"Code is understandable with some effort\",\n \"characteristics\": [\n \"Most variables have meaningful names\",\n \"Basic comments present for complex sections\",\n \"Logic is followable but could be cleaner\"\n ]\n },\n {\n \"score\": 5,\n \"label\": \"Excellent\",\n \"description\": \"Code is immediately clear and maintainable\",\n \"characteristics\": [\n \"All names are descriptive and consistent\",\n \"Comprehensive documentation\",\n \"Clean, modular structure\"\n ]\n }\n ],\n \"edgeCases\": [\n {\n \"situation\": \"Code is well-structured but uses domain-specific abbreviations\",\n \"guidance\": \"Score based on readability for domain experts, not general audience\"\n }\n ]\n}\n```\n\n## Guidelines\n\n1. **Always require evidence before scores** - Evidence-first prompts make judgments easier to audit and reduce ungrounded numeric scoring\n\n2. **Always swap positions in pairwise comparison** - Single-pass comparison is corrupted by position bias\n\n3. **Match scale granularity to rubric specificity** - Don't use 1-10 without detailed level descriptions\n\n4. **Separate objective and subjective criteria** - Use direct scoring for objective, pairwise for subjective\n\n5. **Include confidence scores** - Calibrate to position consistency and evidence strength\n\n6. **Define edge cases explicitly** - Ambiguous situations cause the most evaluation variance\n\n7. **Use domain-specific rubrics** - Generic rubrics produce generic (less useful) evaluations\n\n8. **Validate against human judgments** - Automated evaluation is only valuable if it correlates with human assessment\n\n9. **Monitor for systematic bias** - Track disagreement patterns by criterion, response type, model\n\n10. **Design for iteration** - Evaluation systems improve with feedback loops\n\n## Gotchas\n\n1. **Scoring without justification**: Scores lack grounding and are difficult to debug. Always require evidence-based justification before the score.\n\n2. **Single-pass pairwise comparison**: Position bias corrupts results when positions are not swapped. Always evaluate twice with swapped positions and check consistency.\n\n3. **Overloaded criteria**: Criteria that measure multiple things at once produce unreliable scores. Enforce one criterion = one measurable aspect.\n\n4. **Missing edge case guidance**: Evaluators handle ambiguous cases inconsistently without explicit instructions. Include edge cases in rubrics with clear resolution rules.\n\n5. **Ignoring confidence calibration**: High-confidence wrong judgments are worse than low-confidence ones. Calibrate confidence to position consistency and evidence strength.\n\n6. **Rubric drift**: Rubrics become miscalibrated as quality standards evolve or model capabilities improve. Schedule periodic rubric reviews and re-anchor score levels against fresh human-annotated examples.\n\n7. **Evaluation prompt sensitivity**: Minor wording changes in evaluation prompts can cause material score swings. Version-control evaluation prompts and run regression tests before deploying prompt changes.\n\n8. **Uncontrolled length bias**: Longer responses systematically score higher even when conciseness is preferred. Add explicit length-neutrality instructions to evaluation prompts and validate with length-controlled test pairs.\n\n## Integration\n\nThis skill owns judge design and bias mitigation. Adjacent skills own broader quality gates and infrastructure:\n\n- `evaluation`: general deterministic checks, regression suites, quality gates, and production monitoring.\n- `context-fundamentals`: context structure for judge prompts.\n- `tool-design`: schemas and error handling for evaluation tools.\n- `context-optimization`: token and latency efficiency for high-volume evals.\n- `harness-engineering`: locked evaluator surfaces and governance for autonomous loops.\n\n## References\n\nInternal reference:\n- [LLM-as-Judge Implementation Patterns](./references/implementation-patterns.md) - Read when: building an evaluation pipeline from scratch or integrating LLM judges into CI/CD\n- [Bias Mitigation Techniques](./references/bias-mitigation.md) - Read when: evaluation results show inconsistent or suspicious scoring patterns\n- [Metric Selection Guide](./references/metrics-guide.md) - Read when: choosing statistical metrics to validate evaluation reliability\n- [Evaluation Pipeline Diagram](./references/evaluation-pipeline.md) - Read when: designing the architecture of a multi-stage evaluation system\n\nExternal research:\n- [Eugene Yan: Evaluating the Effectiveness of LLM-Evaluators](https://eugeneyan.com/writing/llm-evaluators/) - Read when: surveying the state of the art in LLM evaluation\n- [Judging LLM-as-a-Judge (Zheng et al., 2023)](https://arxiv.org/abs/2306.05685) - Read when: understanding position bias and MT-Bench methodology\n- [G-Eval: NLG Evaluation using GPT-4 (Liu et al., 2023)](https://arxiv.org/abs/2303.16634) - Read when: implementing chain-of-thought evaluation scoring\n- [Large Language Models are not Fair Evaluators (Wang et al., 2023)](https://arxiv.org/abs/2305.17926) - Read when: diagnosing systematic bias in evaluation outputs\n\nRelated skills in this collection:\n- evaluation - Foundational evaluation concepts\n- context-fundamentals - Context structure for evaluation prompts\n- tool-design - Building evaluation tools\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-24\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.1.0\n","frontmatter":{"name":"advanced-evaluation","description":"This skill should be used for advanced LLM evaluation: LLM-as-judge systems, direct scoring, pairwise comparison, rubric calibration, evaluator bias mitigation, confidence scoring, and automated quality assessment."},"isInternal":false,"tokens":3533,"sizeBytes":16990},{"name":"bdi-ontology-core.md","path":"skills/bdi-mental-states/references/bdi-ontology-core.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/bdi-mental-states/references/bdi-ontology-core.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# BDI Ontology Core Patterns\n\nCore ontology design patterns for Belief-Desire-Intention mental state modeling.\n\n## Class Hierarchy\n\n### Mental Entities (Endurants)\n\n```\nbdi:MentalEntity\n├── bdi:Belief # Informational dimension\n├── bdi:Desire # Motivational dimension \n├── bdi:Intention # Deliberative dimension\n├── bdi:Goal # Description of desired end state\n└── bdi:Plan # Structured action sequence\n```\n\n### Mental Processes (Perdurants)\n\n```\nbdi:MentalProcess\n├── bdi:BeliefProcess # Forms/updates beliefs from perception\n├── bdi:DesireProcess # Generates desires from beliefs\n├── bdi:IntentionProcess # Commits to desires as intentions\n├── bdi:Planning # Transforms intentions into plans\n└── bdi:PlanExecution # Executes plan actions\n```\n\n### Supporting Entities\n\n```\nbdi:WorldState # Configuration of environment\nbdi:Justification # Evidential basis for mental states\nbdi:Task # Atomic unit of planned action\nbdi:Action # Execution of a task\nbdi:TimeInterval # Temporal validity bounds\nbdi:TimeInstant # Point in time reference\n```\n\n## Object Properties\n\n### Motivational Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `motivates` | Belief | Desire | Belief provides reason for desire |\n| `isMotivatedBy` | Desire | Belief | Inverse of motivates |\n| `fulfils` | Intention | Desire | Intention commits to achieving desire |\n| `isFulfilledBy` | Desire | Intention | Inverse of fulfils |\n| `isSupportedBy` | Intention | Belief | Beliefs supporting intention viability |\n\n### Generative Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `generates` | MentalProcess | MentalEntity | Process creates mental state |\n| `isGeneratedBy` | MentalEntity | MentalProcess | Inverse of generates |\n| `modifies` | MentalProcess | MentalEntity | Process updates existing state |\n| `suppresses` | MentalProcess | MentalEntity | Process deactivates state |\n| `isTriggeredBy` | MentalProcess | MentalEntity | State initiates process |\n\n### Referential Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `refersTo` | MentalEntity | WorldState | Mental state about world |\n| `perceives` | Agent | WorldState | Agent observes world |\n| `bringsAbout` | Action | WorldState | Action causes world change |\n| `reasonsUpon` | MentalProcess | MentalEntity | Input to reasoning |\n\n### Structural Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `hasPart` | MentalEntity | MentalEntity | Meronymic composition |\n| `specifies` | Intention | Plan | Intention defines plan |\n| `addresses` | Plan | Goal | Plan achieves goal |\n| `hasComponent` | Plan | Task | Plan contains tasks |\n| `precedes` | Task | Task | Task ordering |\n\n### Temporal Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `atTime` | Entity | TimeInstant | Point occurrence |\n| `hasValidity` | MentalEntity | TimeInterval | Persistence bounds |\n| `hasStartTime` | TimeInterval | TimeInstant | Interval start |\n| `hasEndTime` | TimeInterval | TimeInstant | Interval end |\n\n### Justification Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `isJustifiedBy` | MentalEntity | Justification | Evidential support |\n| `justifies` | Justification | MentalEntity | Inverse relation |\n\n## Ontological Restrictions\n\n### Belief Restrictions\n\n```turtle\nbdi:Belief rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:refersTo ;\n owl:someValuesFrom bdi:WorldState\n] .\n\nbdi:Belief rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:hasValidity ;\n owl:maxCardinality 1\n] .\n```\n\n### Desire Restrictions\n\n```turtle\nbdi:Desire rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:isMotivatedBy ;\n owl:someValuesFrom bdi:Belief\n] .\n```\n\n### Intention Restrictions\n\n```turtle\nbdi:Intention rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:fulfils ;\n owl:cardinality 1\n] .\n\nbdi:Intention rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:isSupportedBy ;\n owl:someValuesFrom bdi:Belief\n] .\n```\n\n### Mental Process Restrictions\n\n```turtle\nbdi:BeliefProcess rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:generates ;\n owl:allValuesFrom bdi:Belief\n] .\n\nbdi:DesireProcess rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:generates ;\n owl:allValuesFrom bdi:Desire\n] .\n\nbdi:IntentionProcess rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:generates ;\n owl:allValuesFrom bdi:Intention\n] .\n```\n\n## DOLCE Alignment\n\nThe BDI ontology aligns with DOLCE Ultra Lite (DUL) foundational ontology:\n\n| BDI Class | DUL Superclass | Rationale |\n|-----------|----------------|-----------|\n| `Agent` | `dul:Agent` | Intentional entity capable of action |\n| `Belief` | `dul:InformationObject` | Information-bearing entity |\n| `Desire` | `dul:Description` | Describes desired state |\n| `Intention` | `dul:Description` | Describes committed course |\n| `Goal` | `dul:Goal` | Desired end state description |\n| `Plan` | `dul:Plan` | Organized action sequence |\n| `WorldState` | `dul:Situation` | Configuration of entities |\n| `MentalProcess` | `dul:Event` | Temporally extended occurrence |\n| `Task` | `dul:Task` | Unit of planned work |\n| `Action` | `dul:Action` | Performed task instance |\n\n## Reused Ontology Design Patterns\n\n### EventCore Pattern\nUsed for mental processes with temporal aspects and participant roles.\n\n### Situation Pattern \nUsed for world state configurations that mental states reference.\n\n### TimeIndexedSituation Pattern\nUsed for associating mental states with validity intervals.\n\n### BasicPlan Pattern\nUsed for goal-plan-task structures linking intentions to actions.\n\n### Provenance Pattern\nUsed for justification tracking and evidential chains.\n\n## Namespace Declarations\n\n```turtle\n@prefix bdi: .\n@prefix dul: .\n@prefix owl: .\n@prefix rdf: .\n@prefix rdfs: .\n@prefix xsd: .\n```\n\n","isInternal":false,"tokens":1691,"sizeBytes":6482},{"name":"framework-integration.md","path":"skills/bdi-mental-states/references/framework-integration.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/bdi-mental-states/references/framework-integration.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# BDI Framework Integration Patterns\n\nIntegration patterns for connecting BDI ontology with executable agent frameworks.\n\n## SEMAS Rule Translation\n\nMap BDI ontology constructs to SEMAS production rules.\n\n### Ontology-to-Rule Mapping\n\n| BDI Construct | SEMAS Element | Example |\n|---------------|---------------|---------|\n| Belief | HEAD fact | `belief(agent_a, store_open)` |\n| Supporting beliefs | CONDITIONALS | `[CONDITIONALS: time(weekday)]` |\n| Desire generation | TAIL action | `generate_desire(agent, goal)` |\n| Intention commitment | TAIL action | `commit_intention(agent, goal)` |\n| Plan specification | TAIL action | `create_plan(agent, plan_id)` |\n\n### Rule Templates\n\n**Belief triggers desire formation:**\n```prolog\n[HEAD: belief(Agent, Fact)] / \n[CONDITIONALS: context_condition(Agent, Context)] » \n[TAIL: generate_desire(Agent, DesiredState)].\n```\n\n**Desire triggers intention commitment:**\n```prolog\n[HEAD: desire(Agent, Goal)] / \n[CONDITIONALS: belief(Agent, SupportingFact1), \n belief(Agent, SupportingFact2)] » \n[TAIL: commit_intention(Agent, Goal)].\n```\n\n**Intention triggers planning:**\n```prolog\n[HEAD: intention(Agent, Goal)] / \n[CONDITIONALS: goal(GoalSpec)] » \n[TAIL: create_plan(Agent, PlanId)].\n```\n\n**Plan triggers execution:**\n```prolog\n[HEAD: plan(Agent, PlanId)] / \n[CONDITIONALS: ready_to_execute(Agent)] » \n[TAIL: execute_plan(Agent, PlanId)].\n```\n\n### Complete SEMAS Example\n\n```prolog\n% ============================================================\n% GROCERY SHOPPING SCENARIO\n% ============================================================\n\n% Phase 1: Belief formation from world state\n[HEAD: perceive(agent_a, store_open)] / \n[CONDITIONALS: time(weekday_afternoon)] » \n[TAIL: add_belief(agent_a, store_open)].\n\n% Phase 2: Desire generation from belief\n[HEAD: belief(agent_a, store_open)] / \n[CONDITIONALS: belief(agent_a, needs_groceries)] » \n[TAIL: generate_desire(agent_a, buy_groceries)].\n\n% Phase 3: Intention commitment from desire\n[HEAD: desire(agent_a, buy_groceries)] / \n[CONDITIONALS: belief(agent_a, has_shopping_list), \n belief(agent_a, store_open),\n belief(agent_a, has_transportation)] » \n[TAIL: commit_intention(agent_a, buy_groceries)].\n\n% Phase 4: Plan creation from intention\n[HEAD: intention(agent_a, buy_groceries)] / \n[CONDITIONALS: goal(complete_shopping)] » \n[TAIL: create_plan(agent_a, shopping_plan)].\n\n% Phase 5: Plan execution\n[HEAD: plan(agent_a, shopping_plan)] / \n[CONDITIONALS: preconditions_met(shopping_plan)] » \n[TAIL: execute_task(agent_a, drive_to_store),\n execute_task(agent_a, select_items),\n execute_task(agent_a, checkout),\n execute_task(agent_a, return_home)].\n\n% Phase 6: World state update\n[HEAD: task_complete(agent_a, checkout)] / \n[CONDITIONALS: items_purchased(agent_a)] » \n[TAIL: update_world_state(has_groceries),\n remove_desire(agent_a, buy_groceries),\n remove_intention(agent_a, buy_groceries)].\n```\n\n### Python Translation Layer\n\n```python\nfrom rdflib import Graph, Namespace, RDF\n\nBDI = Namespace(\"https://w3id.org/fossr/ontology/bdi/\")\n\ndef ontology_to_semas_rules(bdi_graph: Graph) -> list[str]:\n \"\"\"\n Translate BDI ontology instances to SEMAS production rules.\n \"\"\"\n rules = []\n \n # Extract belief-desire-intention chains\n for intention in bdi_graph.subjects(RDF.type, BDI.Intention):\n # Get supporting beliefs\n supporting_beliefs = list(bdi_graph.objects(intention, BDI.isSupportedBy))\n \n # Get fulfilled desire\n fulfilled_desires = list(bdi_graph.objects(intention, BDI.fulfils))\n \n # Get specified plan\n specified_plans = list(bdi_graph.objects(intention, BDI.specifies))\n \n if fulfilled_desires and supporting_beliefs:\n desire = fulfilled_desires[0]\n beliefs_str = \", \".join([format_belief(b, bdi_graph) for b in supporting_beliefs])\n \n rule = (\n f\"[HEAD: {format_desire(desire, bdi_graph)}] / \"\n f\"[CONDITIONALS: {beliefs_str}] » \"\n f\"[TAIL: commit_intention({format_intention(intention, bdi_graph)})]\"\n )\n rules.append(rule)\n \n if specified_plans:\n plan = specified_plans[0]\n rule = (\n f\"[HEAD: {format_intention(intention, bdi_graph)}] / \"\n f\"[CONDITIONALS: ready_to_plan] » \"\n f\"[TAIL: create_plan({format_plan(plan, bdi_graph)})]\"\n )\n rules.append(rule)\n \n return rules\n\ndef format_belief(belief_uri, graph):\n label = graph.value(belief_uri, RDFS.label)\n return f\"belief({label or belief_uri.split('/')[-1]})\"\n\ndef format_desire(desire_uri, graph):\n label = graph.value(desire_uri, RDFS.label)\n return f\"desire({label or desire_uri.split('/')[-1]})\"\n\ndef format_intention(intention_uri, graph):\n label = graph.value(intention_uri, RDFS.label)\n return f\"intention({label or intention_uri.split('/')[-1]})\"\n\ndef format_plan(plan_uri, graph):\n label = graph.value(plan_uri, RDFS.label)\n return f\"plan({label or plan_uri.split('/')[-1]})\"\n```\n\n## Logic Augmented Generation (LAG)\n\nAugment LLM outputs with BDI ontological constraints.\n\n### LAG Pipeline Architecture\n\n```\n┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐\n│ User Query │────▶│ Ontology │────▶│ Augmented │\n│ │ │ Injection │ │ Prompt │\n└─────────────────┘ └─────────────────┘ └─────────────────┘\n │\n ▼\n┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐\n│ Validated │◀────│ Ontology │◀────│ LLM Response │\n│ RDF Triples │ │ Validation │ │ (Triples) │\n└─────────────────┘ └─────────────────┘ └─────────────────┘\n```\n\n### LAG Implementation\n\n```python\nfrom rdflib import Graph, Namespace\nfrom rdflib.plugins.parsers.notation3 import BadSyntax\n\nBDI = Namespace(\"https://w3id.org/fossr/ontology/bdi/\")\n\nclass BDILogicAugmentedGenerator:\n def __init__(self, ontology_path: str, llm_client):\n self.ontology = Graph()\n self.ontology.parse(ontology_path, format='turtle')\n self.llm = llm_client\n \n def generate_mental_states(self, context: str) -> Graph:\n \"\"\"\n Generate BDI mental states from context using LAG.\n \"\"\"\n # Phase 1: Inject ontology into prompt\n ontology_turtle = self.ontology.serialize(format='turtle')\n augmented_prompt = self._build_augmented_prompt(context, ontology_turtle)\n \n # Phase 2: Generate with LLM\n response = self.llm.generate(augmented_prompt)\n \n # Phase 3: Extract and validate triples\n triples = self._extract_triples(response)\n validated = self._validate_against_ontology(triples)\n \n if not validated['is_consistent']:\n # Retry with feedback\n return self._retry_with_feedback(context, validated['errors'])\n \n return validated['graph']\n \n def _build_augmented_prompt(self, context: str, ontology: str) -> str:\n return f\"\"\"\nYou are a BDI mental state modeler. Given the following context, generate \nRDF triples representing the agent's beliefs, desires, and intentions.\n\n## BDI Ontology (use these classes and properties):\n{ontology}\n\n## Context to Model:\n{context}\n\n## Instructions:\n1. Identify world states from the context\n2. Generate beliefs that refer to those world states\n3. Generate desires motivated by those beliefs\n4. Generate intentions that fulfill desires and are supported by beliefs\n5. Include justifications for each mental state\n6. Include temporal validity intervals\n\nOutput valid Turtle RDF triples only.\n\"\"\"\n \n def _extract_triples(self, response: str) -> str:\n \"\"\"Extract Turtle content from LLM response.\"\"\"\n # Find turtle block in response\n if \"```turtle\" in response:\n start = response.find(\"```turtle\") + 9\n end = response.find(\"```\", start)\n return response[start:end].strip()\n return response\n \n def _validate_against_ontology(self, triples: str) -> dict:\n \"\"\"Validate generated triples against BDI ontology.\"\"\"\n result = {'is_consistent': True, 'errors': [], 'graph': None}\n \n try:\n generated = Graph()\n generated.parse(data=triples, format='turtle')\n result['graph'] = generated\n \n # Validate constraints\n errors = []\n \n # Check: Every intention must fulfill a desire\n for intention in generated.subjects(RDF.type, BDI.Intention):\n if not list(generated.objects(intention, BDI.fulfils)):\n errors.append(f\"Intention {intention} does not fulfill any desire\")\n \n # Check: Every belief should reference a world state\n for belief in generated.subjects(RDF.type, BDI.Belief):\n if not list(generated.objects(belief, BDI.refersTo)):\n errors.append(f\"Belief {belief} does not reference a world state\")\n \n # Check: Desires should be motivated by beliefs\n for desire in generated.subjects(RDF.type, BDI.Desire):\n if not list(generated.objects(desire, BDI.isMotivatedBy)):\n errors.append(f\"Desire {desire} has no motivating belief\")\n \n if errors:\n result['is_consistent'] = False\n result['errors'] = errors\n \n except BadSyntax as e:\n result['is_consistent'] = False\n result['errors'] = [f\"Invalid Turtle syntax: {e}\"]\n \n return result\n \n def _retry_with_feedback(self, context: str, errors: list) -> Graph:\n \"\"\"Retry generation with error feedback.\"\"\"\n feedback_prompt = f\"\"\"\nPrevious generation had errors:\n{chr(10).join(errors)}\n\nPlease regenerate the mental states fixing these issues.\n\nContext: {context}\n\"\"\"\n response = self.llm.generate(feedback_prompt)\n triples = self._extract_triples(response)\n result = self._validate_against_ontology(triples)\n \n if result['is_consistent']:\n return result['graph']\n else:\n raise ValueError(f\"Failed to generate valid mental states: {result['errors']}\")\n```\n\n### Inconsistency Detection Example\n\n```python\ndef detect_location_inconsistency(graph: Graph) -> list[str]:\n \"\"\"\n Detect inconsistencies where agent cannot be in two places.\n \"\"\"\n inconsistencies = []\n \n # Query for location beliefs\n query = \"\"\"\n PREFIX bdi: \n \n SELECT ?agent ?belief1 ?belief2 ?loc1 ?loc2 WHERE {\n ?agent bdi:hasBelief ?belief1 , ?belief2 .\n ?belief1 bdi:refersTo ?ws1 .\n ?belief2 bdi:refersTo ?ws2 .\n ?ws1 bdi:hasLocation ?loc1 .\n ?ws2 bdi:hasLocation ?loc2 .\n FILTER(?belief1 != ?belief2 && ?loc1 != ?loc2)\n \n # Check temporal overlap\n ?belief1 bdi:hasValidity ?interval1 .\n ?belief2 bdi:hasValidity ?interval2 .\n ?interval1 bdi:hasStartTime ?start1 ; bdi:hasEndTime ?end1 .\n ?interval2 bdi:hasStartTime ?start2 ; bdi:hasEndTime ?end2 .\n FILTER(?start1 < ?end2 && ?start2 < ?end1)\n }\n \"\"\"\n \n for row in graph.query(query):\n inconsistencies.append(\n f\"Agent {row.agent} has conflicting location beliefs: \"\n f\"{row.loc1} and {row.loc2} at overlapping times\"\n )\n \n return inconsistencies\n```\n\n## JADE/JADEX Integration\n\nMap BDI ontology to JADE/JADEX agent platform structures.\n\n### JADE Agent Structure\n\n```java\npublic class BDIAgent extends Agent {\n // Mental state storage (maps to ontology individuals)\n private Set beliefs = new HashSet<>();\n private Set desires = new HashSet<>();\n private Set intentions = new HashSet<>();\n \n // Ontology-backed mental state management\n private Graph mentalStateGraph;\n \n public void addBelief(Belief belief) {\n beliefs.add(belief);\n \n // Add to RDF graph\n Resource beliefResource = mentalStateGraph.createResource(belief.getUri());\n beliefResource.addProperty(RDF.type, BDI.Belief);\n beliefResource.addProperty(BDI.refersTo, belief.getWorldState().getUri());\n beliefResource.addProperty(BDI.hasValidity, createInterval(belief.getValidity()));\n \n // Trigger desire formation\n triggerDesireProcess(belief);\n }\n \n public void commitIntention(Intention intention) {\n intentions.add(intention);\n \n Resource intentionResource = mentalStateGraph.createResource(intention.getUri());\n intentionResource.addProperty(RDF.type, BDI.Intention);\n intentionResource.addProperty(BDI.fulfils, intention.getDesire().getUri());\n \n for (Belief support : intention.getSupportingBeliefs()) {\n intentionResource.addProperty(BDI.isSupportedBy, support.getUri());\n }\n \n // Trigger planning\n triggerPlanning(intention);\n }\n \n // Export mental states as RDF\n public String exportMentalStates() {\n return mentalStateGraph.serialize(Format.TURTLE);\n }\n \n // Import mental states from RDF\n public void importMentalStates(String turtle) {\n Graph imported = new Graph();\n imported.parse(turtle, Format.TURTLE);\n \n // Reconstruct Java objects from RDF\n for (Resource belief : imported.listSubjectsWithProperty(RDF.type, BDI.Belief)) {\n Belief b = reconstructBelief(belief);\n beliefs.add(b);\n }\n // ... similar for desires and intentions\n }\n}\n```\n\n### JADEX Goal Mapping\n\n```java\n// Map BDI ontology goals to JADEX goals\n@Goal\npublic class OntologyBackedGoal {\n @GoalParameter\n protected String goalUri;\n \n @GoalParameter\n protected Graph ontologyGraph;\n \n public OntologyBackedGoal(Resource goalResource, Graph graph) {\n this.goalUri = goalResource.getURI();\n this.ontologyGraph = graph;\n }\n \n @GoalTargetCondition\n public boolean isAchieved() {\n // Query ontology for goal achievement\n String query = \"\"\"\n PREFIX bdi: \n ASK {\n ?execution bdi:addresses <%s> ;\n bdi:bringsAbout ?worldState .\n }\n \"\"\".formatted(goalUri);\n \n return ontologyGraph.ask(query);\n }\n \n @GoalDropCondition\n public boolean shouldDrop() {\n // Check if supporting beliefs are invalidated\n String query = \"\"\"\n PREFIX bdi: \n ASK {\n ?intention bdi:specifies ?plan .\n ?plan bdi:addresses <%s> .\n ?intention bdi:isSupportedBy ?belief .\n ?belief bdi:hasValidity ?interval .\n ?interval bdi:hasEndTime ?end .\n FILTER(?end < NOW())\n }\n \"\"\".formatted(goalUri);\n \n return ontologyGraph.ask(query);\n }\n}\n```\n\n## RDF Triple Store Integration\n\n### Triple Store Configuration\n\n```python\nfrom rdflib import Graph\nfrom rdflib.plugins.stores.sparqlstore import SPARQLUpdateStore\n\nclass BDIMentalStateStore:\n def __init__(self, endpoint: str):\n self.store = SPARQLUpdateStore()\n self.store.open((endpoint + \"/query\", endpoint + \"/update\"))\n self.graph = Graph(store=self.store, identifier=\"http://example.org/bdi\")\n \n def add_belief(self, agent_uri: str, belief_data: dict):\n \"\"\"Add belief to triple store.\"\"\"\n belief_uri = f\"{agent_uri}/belief/{belief_data['id']}\"\n \n self.graph.add((URIRef(belief_uri), RDF.type, BDI.Belief))\n self.graph.add((URIRef(belief_uri), RDFS.label, Literal(belief_data['label'])))\n self.graph.add((URIRef(belief_uri), BDI.refersTo, URIRef(belief_data['world_state'])))\n self.graph.add((URIRef(agent_uri), BDI.hasMentalState, URIRef(belief_uri)))\n \n # Add temporal validity\n interval_uri = f\"{belief_uri}/validity\"\n self.graph.add((URIRef(belief_uri), BDI.hasValidity, URIRef(interval_uri)))\n self.graph.add((URIRef(interval_uri), BDI.hasStartTime, \n Literal(belief_data['start_time'], datatype=XSD.dateTime)))\n self.graph.add((URIRef(interval_uri), BDI.hasEndTime,\n Literal(belief_data['end_time'], datatype=XSD.dateTime)))\n \n def get_active_beliefs(self, agent_uri: str, at_time: datetime) -> list:\n \"\"\"Query beliefs active at specific time.\"\"\"\n query = \"\"\"\n PREFIX bdi: \n PREFIX xsd: \n \n SELECT ?belief ?label WHERE {\n <%s> bdi:hasMentalState ?belief .\n ?belief a bdi:Belief ;\n rdfs:label ?label ;\n bdi:hasValidity ?interval .\n ?interval bdi:hasStartTime ?start ;\n bdi:hasEndTime ?end .\n FILTER(?start <= \"%s\"^^xsd:dateTime && ?end >= \"%s\"^^xsd:dateTime)\n }\n \"\"\" % (agent_uri, at_time.isoformat(), at_time.isoformat())\n \n return list(self.graph.query(query))\n \n def get_cognitive_chain(self, intention_uri: str) -> dict:\n \"\"\"Trace complete cognitive chain for an intention.\"\"\"\n query = \"\"\"\n PREFIX bdi: \n \n SELECT ?intention ?desire ?belief ?worldState ?plan WHERE {\n <%s> a bdi:Intention ;\n bdi:fulfils ?desire ;\n bdi:isSupportedBy ?belief .\n OPTIONAL { <%s> bdi:specifies ?plan }\n ?desire bdi:isMotivatedBy ?belief .\n ?belief bdi:refersTo ?worldState .\n }\n \"\"\" % (intention_uri, intention_uri)\n \n results = list(self.graph.query(query))\n if results:\n row = results[0]\n return {\n 'intention': str(row.intention),\n 'desire': str(row.desire),\n 'belief': str(row.belief),\n 'world_state': str(row.worldState),\n 'plan': str(row.plan) if row.plan else None\n }\n return None\n```\n\n## FIPA ACL Integration\n\nMap BDI mental states to FIPA Agent Communication Language.\n\n```python\nfrom fipa_acl import ACLMessage, Performative\n\nclass BDICommunicator:\n def __init__(self, agent_id: str, mental_state_store: BDIMentalStateStore):\n self.agent_id = agent_id\n self.store = mental_state_store\n \n def share_belief(self, belief_uri: str, receiver: str) -> ACLMessage:\n \"\"\"Create INFORM message to share belief.\"\"\"\n belief_triples = self.store.get_belief_as_turtle(belief_uri)\n \n message = ACLMessage()\n message.performative = Performative.INFORM\n message.sender = self.agent_id\n message.receiver = receiver\n message.content = belief_triples\n message.ontology = \"https://w3id.org/fossr/ontology/bdi/\"\n message.language = \"turtle\"\n \n return message\n \n def request_belief_confirmation(self, belief_uri: str, receiver: str) -> ACLMessage:\n \"\"\"Create QUERY-IF message to confirm shared belief.\"\"\"\n message = ACLMessage()\n message.performative = Performative.QUERY_IF\n message.sender = self.agent_id\n message.receiver = receiver\n message.content = f\"ASK {{ <{belief_uri}> a bdi:Belief }}\"\n message.language = \"sparql\"\n \n return message\n \n def propose_intention(self, intention_uri: str, receiver: str) -> ACLMessage:\n \"\"\"Create PROPOSE message for coordinated intention.\"\"\"\n intention_triples = self.store.get_intention_as_turtle(intention_uri)\n \n message = ACLMessage()\n message.performative = Performative.PROPOSE\n message.sender = self.agent_id\n message.receiver = receiver\n message.content = intention_triples\n message.ontology = \"https://w3id.org/fossr/ontology/bdi/\"\n \n return message\n```\n\n","isInternal":false,"tokens":4758,"sizeBytes":20983},{"name":"rdf-examples.md","path":"skills/bdi-mental-states/references/rdf-examples.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/bdi-mental-states/references/rdf-examples.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# BDI RDF Examples\n\nComplete RDF/Turtle examples for BDI mental state modeling.\n\n## Complete Cognitive Workflow\n\n```turtle\n@prefix bdi: .\n@prefix ex: .\n@prefix xsd: .\n@prefix rdfs: .\n\n# ============================================================\n# PHASE 1: World State Perception\n# ============================================================\n\nex:WorldState_traffic a bdi:WorldState ;\n rdfs:comment \"Heavy traffic on Route 101\" ;\n bdi:atTime \"2026-01-04T08:30:00\"^^xsd:dateTime ;\n bdi:isPerceivedBy ex:Agent_commuter ;\n bdi:triggers ex:BeliefProcess_assess_traffic .\n\n# ============================================================\n# PHASE 2: Belief Formation\n# ============================================================\n\nex:BeliefProcess_assess_traffic a bdi:BeliefProcess ;\n bdi:generates ex:Belief_traffic_delay ;\n bdi:reasonsUpon ex:WorldState_traffic ;\n bdi:isProcessedBy ex:Agent_commuter ;\n bdi:atTime \"2026-01-04T08:31:00\"^^xsd:dateTime .\n\nex:Belief_traffic_delay a bdi:Belief ;\n rdfs:label \"Traffic will cause 30-minute delay\" ;\n bdi:refersTo ex:WorldState_traffic ;\n bdi:hasValidity ex:TimeInterval_morning_commute ;\n bdi:hasPart ex:Belief_route_congested , ex:Belief_delay_duration ;\n bdi:isJustifiedBy ex:Justification_traffic_report ;\n bdi:motivates ex:Desire_arrive_on_time .\n\nex:Belief_route_congested a bdi:Belief ;\n rdfs:comment \"Route 101 is congested\" .\n\nex:Belief_delay_duration a bdi:Belief ;\n rdfs:comment \"Delay estimated at 30 minutes\" .\n\nex:Justification_traffic_report a bdi:Justification ;\n rdfs:label \"Real-time traffic data from navigation system\" ;\n bdi:justifies ex:Belief_traffic_delay .\n\n# ============================================================\n# PHASE 3: Desire Formation\n# ============================================================\n\nex:DesireProcess_plan_arrival a bdi:DesireProcess ;\n bdi:generates ex:Desire_arrive_on_time ;\n bdi:reasonsUpon ex:Belief_traffic_delay ;\n bdi:isProcessedBy ex:Agent_commuter .\n\nex:Desire_arrive_on_time a bdi:Desire ;\n rdfs:label \"I desire to arrive at work on time\" ;\n bdi:isMotivatedBy ex:Belief_traffic_delay ;\n bdi:refersTo ex:WorldState_on_time_arrival .\n\n# ============================================================\n# PHASE 4: Intention Commitment\n# ============================================================\n\nex:IntentionProcess_commit_route a bdi:IntentionProcess ;\n bdi:generates ex:Intention_take_alternate_route ;\n bdi:reasonsUpon ex:Desire_arrive_on_time ;\n bdi:isProcessedBy ex:Agent_commuter .\n\nex:Intention_take_alternate_route a bdi:Intention ;\n rdfs:label \"I will take alternate route via Highway 280\" ;\n bdi:fulfils ex:Desire_arrive_on_time ;\n bdi:isSupportedBy ex:Belief_traffic_delay ;\n bdi:specifies ex:Plan_alternate_commute ;\n bdi:isJustifiedBy ex:Justification_time_optimization .\n\nex:Justification_time_optimization a bdi:Justification ;\n rdfs:label \"Alternate route saves 20 minutes based on current conditions\" ;\n bdi:justifies ex:Intention_take_alternate_route .\n\n# ============================================================\n# PHASE 5: Planning\n# ============================================================\n\nex:Planning_route_selection a bdi:Planning ;\n bdi:reasonsUpon ex:Intention_take_alternate_route ;\n bdi:defines ex:Plan_alternate_commute ;\n bdi:atTime ex:TimeInterval_planning_phase .\n\nex:Plan_alternate_commute a bdi:Plan ;\n rdfs:label \"Alternate commute via Highway 280\" ;\n bdi:addresses ex:Goal_arrive_by_9am ;\n bdi:beginsWith ex:Task_exit_Route101 ;\n bdi:endsWith ex:Task_arrive_parking ;\n bdi:hasComponent ex:Task_exit_Route101 , ex:Task_merge_280 , \n ex:Task_navigate_280 , ex:Task_arrive_parking .\n\nex:Task_exit_Route101 a bdi:Task ;\n rdfs:label \"Exit Route 101 at Whipple Ave\" ;\n bdi:precedes ex:Task_merge_280 .\n\nex:Task_merge_280 a bdi:Task ;\n rdfs:label \"Merge onto Highway 280 North\" ;\n bdi:precedes ex:Task_navigate_280 .\n\nex:Task_navigate_280 a bdi:Task ;\n rdfs:label \"Continue on Highway 280 for 8 miles\" ;\n bdi:precedes ex:Task_arrive_parking .\n\nex:Task_arrive_parking a bdi:Task ;\n rdfs:label \"Arrive at office parking garage\" .\n\nex:Goal_arrive_by_9am a bdi:Goal ;\n rdfs:label \"Arrive at work by 9:00 AM\" .\n\n# ============================================================\n# PHASE 6: Plan Execution\n# ============================================================\n\nex:PlanExecution_commute a bdi:PlanExecution ;\n bdi:satisfies ex:Plan_alternate_commute ;\n bdi:addresses ex:Goal_arrive_by_9am ;\n bdi:isExecutedBy ex:Agent_commuter ;\n bdi:hasComponent ex:Action_exit , ex:Action_merge , \n ex:Action_drive_280 , ex:Action_park ;\n bdi:atTime ex:TimeInterval_execution ;\n bdi:bringsAbout ex:WorldState_arrived_on_time .\n\nex:Action_exit a bdi:Action ;\n bdi:isExecutionOf ex:Task_exit_Route101 ;\n bdi:isPerformedBy ex:Agent_commuter ;\n bdi:atTime \"2026-01-04T08:35:00\"^^xsd:dateTime .\n\nex:Action_merge a bdi:Action ;\n bdi:isExecutionOf ex:Task_merge_280 ;\n bdi:isPerformedBy ex:Agent_commuter ;\n bdi:atTime \"2026-01-04T08:37:00\"^^xsd:dateTime .\n\nex:Action_drive_280 a bdi:Action ;\n bdi:isExecutionOf ex:Task_navigate_280 ;\n bdi:isPerformedBy ex:Agent_commuter ;\n bdi:atTime \"2026-01-04T08:40:00\"^^xsd:dateTime .\n\nex:Action_park a bdi:Action ;\n bdi:isExecutionOf ex:Task_arrive_parking ;\n bdi:isPerformedBy ex:Agent_commuter ;\n bdi:bringsAbout ex:WorldState_arrived_on_time ;\n bdi:atTime \"2026-01-04T08:52:00\"^^xsd:dateTime .\n\n# ============================================================\n# PHASE 7: Resulting World State\n# ============================================================\n\nex:WorldState_arrived_on_time a bdi:WorldState ;\n rdfs:comment \"Agent arrived at work at 8:52 AM\" ;\n bdi:atTime \"2026-01-04T08:52:00\"^^xsd:dateTime .\n\n# ============================================================\n# TEMPORAL INTERVALS\n# ============================================================\n\nex:TimeInterval_morning_commute a bdi:TimeInterval ;\n bdi:hasStartTime \"2026-01-04T08:30:00\"^^xsd:dateTime ;\n bdi:hasEndTime \"2026-01-04T09:00:00\"^^xsd:dateTime .\n\nex:TimeInterval_planning_phase a bdi:TimeInterval ;\n bdi:hasStartTime \"2026-01-04T08:31:00\"^^xsd:dateTime ;\n bdi:hasEndTime \"2026-01-04T08:34:00\"^^xsd:dateTime .\n\nex:TimeInterval_execution a bdi:TimeInterval ;\n bdi:hasStartTime \"2026-01-04T08:35:00\"^^xsd:dateTime ;\n bdi:hasEndTime \"2026-01-04T08:52:00\"^^xsd:dateTime .\n```\n\n## Multi-Agent Coordination Example\n\n```turtle\n@prefix bdi: .\n@prefix ex: .\n@prefix fipa: .\n\n# Shared belief about project deadline\nex:Agent_developer a bdi:Agent ;\n bdi:hasMentalState ex:Belief_deadline_friday .\n\nex:Agent_manager a bdi:Agent ;\n bdi:hasMentalState ex:Belief_deadline_friday .\n\nex:Belief_deadline_friday a bdi:Belief ;\n rdfs:label \"Project deadline is Friday 5 PM\" ;\n bdi:refersTo ex:WorldState_deadline ;\n bdi:hasValidity ex:TimeInterval_project_week .\n\nex:WorldState_deadline a bdi:WorldState ;\n rdfs:comment \"Project XYZ must be delivered by 2026-01-10T17:00:00\" .\n\n# Agent-specific mental states\nex:Agent_developer \n bdi:hasDesire ex:Desire_complete_coding ;\n bdi:hasIntention ex:Intention_implement_features .\n\nex:Desire_complete_coding a bdi:Desire ;\n rdfs:label \"Complete feature implementation\" ;\n bdi:isMotivatedBy ex:Belief_deadline_friday .\n\nex:Intention_implement_features a bdi:Intention ;\n rdfs:label \"Implement features A, B, and C\" ;\n bdi:fulfils ex:Desire_complete_coding ;\n bdi:specifies ex:Plan_development .\n\nex:Agent_manager \n bdi:hasDesire ex:Desire_ensure_delivery ;\n bdi:hasIntention ex:Intention_coordinate_team .\n\nex:Desire_ensure_delivery a bdi:Desire ;\n rdfs:label \"Ensure on-time project delivery\" ;\n bdi:isMotivatedBy ex:Belief_deadline_friday .\n\nex:Intention_coordinate_team a bdi:Intention ;\n rdfs:label \"Coordinate team activities\" ;\n bdi:fulfils ex:Desire_ensure_delivery ;\n bdi:specifies ex:Plan_project_management .\n\n# FIPA communication\nex:Message_M1 a fipa:ACLMessage ;\n fipa:sender ex:Agent_manager ;\n fipa:receiver ex:Agent_developer ;\n fipa:content ex:Belief_deadline_friday ;\n fipa:performative fipa:inform .\n```\n\n## Conflict Resolution Example\n\n```turtle\n@prefix bdi: .\n@prefix ex: .\n\n# Conflicting location beliefs\nex:Belief_at_home a bdi:Belief ;\n bdi:refersTo ex:WorldState_home ;\n rdfs:comment \"Agent is currently at home\" .\n\nex:Belief_at_office a bdi:Belief ;\n bdi:refersTo ex:WorldState_office ;\n rdfs:comment \"Agent is at office\" .\n\n# Conflicting intentions\nex:Intention_work_from_home a bdi:Intention ;\n bdi:isSupportedBy ex:Belief_at_home ;\n rdfs:label \"Work from home today\" .\n\nex:Intention_attend_meeting a bdi:Intention ;\n bdi:isSupportedBy ex:Belief_at_office ;\n rdfs:label \"Attend in-person meeting\" .\n\n# Justification for conflict resolution\nex:Justification_location_conflict a bdi:Justification ;\n rdfs:comment \"Cannot simultaneously be at home and office\" ;\n bdi:justifies ex:Intention_resolution .\n\n# Resolved intention\nex:Intention_resolution a bdi:Intention ;\n rdfs:label \"Attend meeting via video call from home\" ;\n bdi:fulfils ex:Desire_meeting_participation ;\n bdi:isSupportedBy ex:Belief_at_home ;\n bdi:isJustifiedBy ex:Justification_location_conflict .\n```\n\n## T2B2T Payment Processing Example\n\n```turtle\n@prefix bdi: .\n@prefix ex: .\n@prefix xsd: .\n\n# PHASE 1: Triples-to-Beliefs (External RDF → Internal Mental State)\n\nex:WorldState_notification a bdi:WorldState ;\n rdfs:comment \"Push notification: Ghadeh requested $250 via Zelle\" ;\n bdi:atTime \"2025-10-27T10:15:00\"^^xsd:dateTime ;\n bdi:triggers ex:BeliefProcess_BP1 .\n\nex:BeliefProcess_BP1 a bdi:BeliefProcess ;\n bdi:generates ex:Belief_payment_request ;\n bdi:isProcessedBy ex:Agent_A .\n\nex:Belief_payment_request a bdi:Belief ;\n rdfs:label \"Ghadeh requested $250\" ;\n bdi:refersTo ex:WorldState_notification ;\n bdi:motivates ex:Desire_pay_Ghadeh .\n\nex:Desire_pay_Ghadeh a bdi:Desire ;\n rdfs:label \"Pay Ghadeh $250\" ;\n bdi:isMotivatedBy ex:Belief_payment_request .\n\nex:Intention_I1 a bdi:Intention ;\n rdfs:label \"Pay Ghadeh $250\" ;\n bdi:fulfils ex:Desire_pay_Ghadeh ;\n bdi:specifies ex:Plan_payment .\n\n# PHASE 2: Beliefs-to-Triples (Mental State → External RDF)\n\nex:PlanExecution_PE1 a bdi:PlanExecution ;\n bdi:satisfies ex:Plan_payment ;\n bdi:bringsAbout ex:WorldState_payment_complete .\n\nex:WorldState_payment_complete a bdi:WorldState ;\n rdfs:comment \"Payment of $250 sent to Ghadeh via Zelle\" ;\n bdi:atTime \"2025-10-27T10:20:00\"^^xsd:dateTime .\n```\n\n","isInternal":false,"tokens":3241,"sizeBytes":11126},{"name":"sparql-competency.md","path":"skills/bdi-mental-states/references/sparql-competency.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/bdi-mental-states/references/sparql-competency.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# SPARQL Competency Queries\n\nValidation queries for BDI ontology implementations based on competency questions.\n\n## Mental Entity Queries\n\n### CQ1: What are all mental entities?\n\n```sparql\nPREFIX bdi: \nPREFIX rdf: \nPREFIX rdfs: \n\nSELECT DISTINCT ?entity ?type WHERE {\n ?entity rdf:type ?type .\n ?type rdfs:subClassOf* bdi:MentalEntity .\n}\n```\n\n### CQ2: What beliefs does an agent hold?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?label WHERE {\n ?agent bdi:hasMentalState ?belief .\n ?belief a bdi:Belief .\n OPTIONAL { ?belief rdfs:label ?label }\n}\n```\n\n### CQ3: What desires does an agent have?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?desire ?label WHERE {\n ?agent bdi:hasDesire ?desire .\n ?desire a bdi:Desire .\n OPTIONAL { ?desire rdfs:label ?label }\n}\n```\n\n### CQ4: What intentions has an agent committed to?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention ?label WHERE {\n ?agent bdi:hasIntention ?intention .\n ?intention a bdi:Intention .\n OPTIONAL { ?intention rdfs:label ?label }\n}\n```\n\n## Motivational Chain Queries\n\n### CQ5: What beliefs motivated formation of a given desire?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?beliefLabel WHERE {\n ?desire bdi:isMotivatedBy ?belief .\n ?belief a bdi:Belief .\n OPTIONAL { ?belief rdfs:label ?beliefLabel }\n}\n```\n\n### CQ6: Which desire does a particular intention fulfill?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?desire ?desireLabel WHERE {\n ?intention bdi:fulfils ?desire .\n ?desire a bdi:Desire .\n OPTIONAL { ?desire rdfs:label ?desireLabel }\n}\n```\n\n### CQ7: What beliefs support a given intention?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?label WHERE {\n ?intention bdi:isSupportedBy ?belief .\n ?belief a bdi:Belief .\n OPTIONAL { ?belief rdfs:label ?label }\n}\n```\n\n### CQ8: Trace complete cognitive chain for an intention\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention ?desire ?belief ?worldState WHERE {\n ?intention a bdi:Intention ;\n bdi:fulfils ?desire ;\n bdi:isSupportedBy ?belief .\n ?desire bdi:isMotivatedBy ?belief .\n ?belief bdi:refersTo ?worldState .\n}\n```\n\n## Mental Process Queries\n\n### CQ9: Which mental process generated a belief?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?process ?processType WHERE {\n ?process bdi:generates ?belief .\n ?belief a bdi:Belief .\n ?process a ?processType .\n FILTER(?processType != owl:NamedIndividual)\n}\n```\n\n### CQ10: What triggered a mental process?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?process ?trigger ?triggerType WHERE {\n ?process a bdi:MentalProcess ;\n bdi:isTriggeredBy ?trigger .\n ?trigger a ?triggerType .\n}\n```\n\n### CQ11: What did a mental process reason upon?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?process ?input WHERE {\n ?process a bdi:MentalProcess ;\n bdi:reasonsUpon ?input .\n}\n```\n\n## Plan and Goal Queries\n\n### CQ12: What plan does an intention specify?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention ?plan ?goal WHERE {\n ?intention bdi:specifies ?plan .\n ?plan a bdi:Plan ;\n bdi:addresses ?goal .\n}\n```\n\n### CQ13: What is the ordered sequence of tasks in a plan?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?plan ?task ?nextTask WHERE {\n ?plan a bdi:Plan ;\n bdi:hasComponent ?task .\n OPTIONAL { ?task bdi:precedes ?nextTask }\n}\nORDER BY ?task\n```\n\n### CQ14: What is the first and last task of a plan?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?plan ?firstTask ?lastTask WHERE {\n ?plan a bdi:Plan ;\n bdi:beginsWith ?firstTask ;\n bdi:endsWith ?lastTask .\n}\n```\n\n### CQ15: Which actions executed which tasks?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?action ?task ?time WHERE {\n ?action bdi:isExecutionOf ?task ;\n bdi:atTime ?time .\n}\nORDER BY ?time\n```\n\n## Temporal Queries\n\n### CQ16: What mental states are valid at a specific time?\n\n```sparql\nPREFIX bdi: \nPREFIX xsd: \n\nSELECT ?mentalState ?type WHERE {\n ?mentalState bdi:hasValidity ?interval .\n ?interval bdi:hasStartTime ?start ;\n bdi:hasEndTime ?end .\n ?mentalState a ?type .\n FILTER(?start <= \"2026-01-04T10:00:00\"^^xsd:dateTime && \n ?end >= \"2026-01-04T10:00:00\"^^xsd:dateTime)\n}\n```\n\n### CQ17: When was a belief formed?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?formationTime WHERE {\n ?process bdi:generates ?belief ;\n bdi:atTime ?formationTime .\n ?belief a bdi:Belief .\n}\n```\n\n### CQ18: What is the temporal validity of an intention?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention ?start ?end WHERE {\n ?intention a bdi:Intention ;\n bdi:hasValidity ?interval .\n ?interval bdi:hasStartTime ?start ;\n bdi:hasEndTime ?end .\n}\n```\n\n## Justification Queries\n\n### CQ19: What justifies a belief?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?justification ?justLabel WHERE {\n ?belief a bdi:Belief ;\n bdi:isJustifiedBy ?justification .\n OPTIONAL { ?justification rdfs:label ?justLabel }\n}\n```\n\n### CQ20: What justifies an intention?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention ?justification ?justLabel WHERE {\n ?intention a bdi:Intention ;\n bdi:isJustifiedBy ?justification .\n OPTIONAL { ?justification rdfs:label ?justLabel }\n}\n```\n\n## Compositional Queries\n\n### CQ21: What parts comprise a complex belief?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?part ?partLabel WHERE {\n ?belief a bdi:Belief ;\n bdi:hasPart ?part .\n OPTIONAL { ?part rdfs:label ?partLabel }\n}\n```\n\n### CQ22: Find composite mental entities\n\n```sparql\nPREFIX bdi: \n\nSELECT ?composite (COUNT(?part) AS ?partCount) WHERE {\n ?composite bdi:hasPart ?part .\n}\nGROUP BY ?composite\nHAVING (COUNT(?part) > 1)\n```\n\n## World State Queries\n\n### CQ23: What world state does a belief refer to?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?worldState ?wsComment WHERE {\n ?belief a bdi:Belief ;\n bdi:refersTo ?worldState .\n OPTIONAL { ?worldState rdfs:comment ?wsComment }\n}\n```\n\n### CQ24: What actions brought about a world state?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?action ?worldState WHERE {\n ?action bdi:bringsAbout ?worldState .\n ?worldState a bdi:WorldState .\n}\n```\n\n### CQ25: What world states has an agent perceived?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?agent ?worldState ?time WHERE {\n ?agent bdi:perceives ?worldState .\n OPTIONAL { ?worldState bdi:atTime ?time }\n}\n```\n\n## Validation Queries (OWLUnit Style)\n\n### V1: Every intention must fulfill exactly one desire\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention WHERE {\n ?intention a bdi:Intention .\n FILTER NOT EXISTS { ?intention bdi:fulfils ?desire }\n}\n# Expected: Empty result set\n```\n\n### V2: Every belief must reference a world state\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief WHERE {\n ?belief a bdi:Belief .\n FILTER NOT EXISTS { ?belief bdi:refersTo ?worldState }\n}\n# Expected: Empty result set (or only abstract beliefs)\n```\n\n### V3: Mental processes must reason upon something\n\n```sparql\nPREFIX bdi: \n\nSELECT ?process WHERE {\n ?process a bdi:MentalProcess .\n FILTER NOT EXISTS { ?process bdi:reasonsUpon ?input }\n}\n# Expected: Empty result set\n```\n\n### V4: BeliefProcess must generate only Beliefs\n\n```sparql\nPREFIX bdi: \n\nSELECT ?process ?generated WHERE {\n ?process a bdi:BeliefProcess ;\n bdi:generates ?generated .\n FILTER NOT EXISTS { ?generated a bdi:Belief }\n}\n# Expected: Empty result set\n```\n\n### V5: Plans must have begin and end tasks\n\n```sparql\nPREFIX bdi: \n\nSELECT ?plan WHERE {\n ?plan a bdi:Plan .\n FILTER NOT EXISTS { \n ?plan bdi:beginsWith ?first ;\n bdi:endsWith ?last \n }\n}\n# Expected: Empty result set\n```\n\n## Multi-Agent Queries\n\n### CQ26: What beliefs are shared across agents?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief (COUNT(DISTINCT ?agent) AS ?agentCount) WHERE {\n ?agent bdi:hasMentalState ?belief .\n ?belief a bdi:Belief .\n}\nGROUP BY ?belief\nHAVING (COUNT(DISTINCT ?agent) > 1)\n```\n\n### CQ27: Which agents share the same desire?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?desire ?agent1 ?agent2 WHERE {\n ?agent1 bdi:hasDesire ?desire .\n ?agent2 bdi:hasDesire ?desire .\n FILTER(?agent1 != ?agent2)\n}\n```\n\n","isInternal":false,"tokens":2824,"sizeBytes":9382},{"name":"SKILL.md","path":"skills/bdi-mental-states/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/bdi-mental-states/SKILL.md","title":"bdi-mental-states","category":"anthropic-skill","format":"markdown","content":"---\nname: bdi-mental-states\ndescription: \"This skill should be used when modeling agent mental states with BDI concepts: beliefs, desires, intentions, RDF-to-belief transformations, rational agency traces, cognitive agents, BDI ontologies, and neuro-symbolic AI integration.\"\n---\n\n# BDI Mental State Modeling\n\nTransform external RDF context into agent mental states (beliefs, desires, intentions) using formal BDI ontology patterns. This skill enables agents to reason about context through cognitive architecture, supporting deliberative reasoning, explainability, and semantic interoperability within multi-agent systems.\n\n## When to Activate\n\nActivate this skill when:\n- Processing external RDF context into agent beliefs about world states\n- Modeling rational agency with perception, deliberation, and action cycles\n- Enabling explainability through traceable reasoning chains\n- Implementing BDI frameworks (SEMAS, JADE, JADEX)\n- Augmenting LLMs with formal cognitive structures (Logic Augmented Generation)\n- Coordinating mental states across multi-agent platforms\n- Tracking temporal evolution of beliefs, desires, and intentions\n- Linking motivational states to action plans\n\nDo not activate this skill for adjacent work owned by other skills:\n- General context-window explanations or attention mechanics: `context-fundamentals`.\n- Persistent user, entity, or conversation memory without formal BDI state: `memory-systems`.\n- Supervisor, swarm, or handoff topology decisions: `multi-agent-patterns`.\n- General agent evaluation rubrics or quality gates: `evaluation`.\n\n## Core Concepts\n\n### Mental Reality Architecture\n\nSeparate mental states into two ontological categories because BDI reasoning requires distinguishing what persists from what happens:\n\n**Mental States (Endurants)** -- model these as persistent cognitive attributes that hold over time intervals:\n- `Belief`: Represent what the agent holds true about the world. Ground every belief in a world state reference.\n- `Desire`: Represent what the agent wishes to bring about. Link each desire back to the beliefs that motivate it.\n- `Intention`: Represent what the agent commits to achieving. An intention must fulfil a desire and specify a plan.\n\n**Mental Processes (Perdurants)** -- model these as events that create or modify mental states, because tracking causal transitions enables explainability:\n- `BeliefProcess`: Triggers belief formation/update from perception. Always connect to a generating world state.\n- `DesireProcess`: Generates desires from existing beliefs. Preserves the motivational chain.\n- `IntentionProcess`: Commits to selected desires as actionable intentions.\n\n### Cognitive Chain Pattern\n\nWire beliefs, desires, and intentions into directed chains using bidirectional properties (`motivates`/`isMotivatedBy`, `fulfils`/`isFulfilledBy`) because this enables both forward reasoning (what should the agent do?) and backward tracing (why did the agent act?):\n\n```turtle\n:Belief_store_open a bdi:Belief ;\n rdfs:comment \"Store is open\" ;\n bdi:motivates :Desire_buy_groceries .\n\n:Desire_buy_groceries a bdi:Desire ;\n rdfs:comment \"I desire to buy groceries\" ;\n bdi:isMotivatedBy :Belief_store_open .\n\n:Intention_go_shopping a bdi:Intention ;\n rdfs:comment \"I will buy groceries\" ;\n bdi:fulfils :Desire_buy_groceries ;\n bdi:isSupportedBy :Belief_store_open ;\n bdi:specifies :Plan_shopping .\n```\n\n### World State Grounding\n\nAlways ground mental states in world state references rather than free-text descriptions, because ungrounded beliefs break semantic querying and cross-agent interoperability:\n\n```turtle\n:Agent_A a bdi:Agent ;\n bdi:perceives :WorldState_WS1 ;\n bdi:hasMentalState :Belief_B1 .\n\n:WorldState_WS1 a bdi:WorldState ;\n rdfs:comment \"Meeting scheduled at 10am in Room 5\" ;\n bdi:atTime :TimeInstant_10am .\n\n:Belief_B1 a bdi:Belief ;\n bdi:refersTo :WorldState_WS1 .\n```\n\n### Goal-Directed Planning\n\nConnect intentions to plans via `bdi:specifies`, and decompose plans into ordered task sequences using `bdi:precedes`, because this separation allows plan reuse across different intentions while keeping execution order explicit:\n\n```turtle\n:Intention_I1 bdi:specifies :Plan_P1 .\n\n:Plan_P1 a bdi:Plan ;\n bdi:addresses :Goal_G1 ;\n bdi:beginsWith :Task_T1 ;\n bdi:endsWith :Task_T3 .\n\n:Task_T1 bdi:precedes :Task_T2 .\n:Task_T2 bdi:precedes :Task_T3 .\n```\n\n### T2B2T Paradigm\n\nImplement Triples-to-Beliefs-to-Triples as a bidirectional pipeline because agents must both consume external RDF context and produce new RDF assertions. Structure every T2B2T implementation in two explicit phases:\n\n**Phase 1: Triples-to-Beliefs** -- Translate incoming RDF triples into belief instances. Use `bdi:triggers` to connect the external world state to a `BeliefProcess`, and `bdi:generates` to produce the resulting belief. This preserves provenance from source data through to internal cognition:\n```turtle\n:WorldState_notification a bdi:WorldState ;\n rdfs:comment \"Push notification: Payment request $250\" ;\n bdi:triggers :BeliefProcess_BP1 .\n\n:BeliefProcess_BP1 a bdi:BeliefProcess ;\n bdi:generates :Belief_payment_request .\n```\n\n**Phase 2: Beliefs-to-Triples** -- After BDI deliberation selects an intention and executes a plan, project the results back into RDF using `bdi:bringsAbout`. This closes the loop so downstream systems can consume agent outputs as standard linked data:\n```turtle\n:Intention_pay a bdi:Intention ;\n bdi:specifies :Plan_payment .\n\n:PlanExecution_PE1 a bdi:PlanExecution ;\n bdi:satisfies :Plan_payment ;\n bdi:bringsAbout :WorldState_payment_complete .\n```\n\n### Notation Selection by Level\n\nChoose notation based on the C4 abstraction level being modeled, because mixing notations at the wrong level obscures rather than clarifies the cognitive architecture:\n\n| C4 Level | Notation | Mental State Representation |\n|----------|----------|----------------------------|\n| L1 Context | ArchiMate | Agent boundaries, external perception sources |\n| L2 Container | ArchiMate | BDI reasoning engine, belief store, plan executor |\n| L3 Component | UML | Mental state managers, process handlers |\n| L4 Code | UML/RDF | Belief/Desire/Intention classes, ontology instances |\n\n### Justification and Explainability\n\nAttach `bdi:Justification` instances to every mental entity using `bdi:isJustifiedBy`, because unjustified mental states make agent reasoning opaque and untraceable. Each justification should capture the evidence or rule that produced the mental state:\n\n```turtle\n:Belief_B1 a bdi:Belief ;\n bdi:isJustifiedBy :Justification_J1 .\n\n:Justification_J1 a bdi:Justification ;\n rdfs:comment \"Official announcement received via email\" .\n\n:Intention_I1 a bdi:Intention ;\n bdi:isJustifiedBy :Justification_J2 .\n\n:Justification_J2 a bdi:Justification ;\n rdfs:comment \"Location precondition satisfied\" .\n```\n\n### Temporal Dimensions\n\nAssign validity intervals to every mental state using `bdi:hasValidity` with `TimeInterval` instances, because beliefs without temporal bounds cannot be garbage-collected or conflict-checked during diachronic reasoning:\n\n```turtle\n:Belief_B1 a bdi:Belief ;\n bdi:hasValidity :TimeInterval_TI1 .\n\n:TimeInterval_TI1 a bdi:TimeInterval ;\n bdi:hasStartTime :TimeInstant_9am ;\n bdi:hasEndTime :TimeInstant_11am .\n```\n\nQuery mental states active at a specific moment using SPARQL temporal filters. Use this pattern to resolve conflicts when multiple beliefs about the same world state overlap in time:\n\n```sparql\nSELECT ?mentalState WHERE {\n ?mentalState bdi:hasValidity ?interval .\n ?interval bdi:hasStartTime ?start ;\n bdi:hasEndTime ?end .\n FILTER(?start <= \"2025-01-04T10:00:00\"^^xsd:dateTime &&\n ?end >= \"2025-01-04T10:00:00\"^^xsd:dateTime)\n}\n```\n\n### Compositional Mental Entities\n\nDecompose complex beliefs into constituent parts using `bdi:hasPart` relations, because monolithic beliefs force full replacement on partial updates. Structure composite beliefs so that each sub-belief can be independently updated, queried, or invalidated:\n\n```turtle\n:Belief_meeting a bdi:Belief ;\n rdfs:comment \"Meeting at 10am in Room 5\" ;\n bdi:hasPart :Belief_meeting_time , :Belief_meeting_location .\n\n# Update only location component without touching time\n:BeliefProcess_update a bdi:BeliefProcess ;\n bdi:modifies :Belief_meeting_location .\n```\n\n## Practical Guidance\n\n### Build a BDI Model in Six Passes\n\nUse this workflow when converting external semantic context into a BDI representation:\n\n1. **Define the world-state substrate**: Identify the external facts or events the agent can perceive. Model these as world states before creating beliefs.\n2. **Create belief instances**: Translate each relevant world state into a belief with provenance, temporal validity, and a justification reference.\n3. **Derive desires from beliefs**: Add desires only when a belief creates a goal-relevant motivation. Link each desire to the belief that motivates it.\n4. **Commit intentions deliberately**: Promote a desire to an intention only when the agent commits to a plan. Record the selected plan and preconditions.\n5. **Project action results back to triples**: After execution, emit resulting world states as RDF so downstream systems can consume the new state.\n6. **Validate with competency questions**: Query for provenance, motivation, plan sequence, and active validity windows before trusting the model.\n\n### Keep the Ontology Small\n\nStart with `Agent`, `WorldState`, `Belief`, `Desire`, `Intention`, `Plan`, `Task`, `Justification`, and `TimeInterval`. Add specialized classes only after competency questions prove the core model cannot answer required queries. A compact ontology is easier to serialize into prompts, easier to validate, and less likely to create brittle reasoning chains.\n\n### Use BDI Only When Mental-State Semantics Matter\n\nBDI modeling is justified when the system needs explainable agency: why an agent believed something, what desire that belief created, which intention was selected, and what plan executed. If the system only needs to remember facts across sessions, use `memory-systems`. If it only needs to split work across agents, use `multi-agent-patterns`.\n\n## Detailed Topics\n\n### Integration Patterns\n\n### Logic Augmented Generation (LAG)\n\nUse LAG to constrain LLM outputs with ontological structure, because unconstrained generation produces triples that violate BDI class restrictions. Serialize the ontology into the prompt context, then validate generated triples against it before accepting them:\n\n```python\ndef augment_llm_with_bdi_ontology(prompt, ontology_graph):\n ontology_context = serialize_ontology(ontology_graph, format='turtle')\n augmented_prompt = f\"{ontology_context}\\n\\n{prompt}\"\n\n response = llm.generate(augmented_prompt)\n triples = extract_rdf_triples(response)\n\n is_consistent = validate_triples(triples, ontology_graph)\n return triples if is_consistent else retry_with_feedback()\n```\n\n### SEMAS Rule Translation\n\nTranslate BDI ontology patterns into executable production rules when deploying to rule-based agent platforms. Map each cognitive chain link (belief-to-desire, desire-to-intention) to a HEAD/CONDITIONALS/TAIL rule, because this preserves the deliberative semantics while enabling runtime execution:\n\n```prolog\n% Belief triggers desire formation\n[HEAD: belief(agent_a, store_open)] /\n[CONDITIONALS: time(weekday_afternoon)] »\n[TAIL: generate_desire(agent_a, buy_groceries)].\n\n% Desire triggers intention commitment\n[HEAD: desire(agent_a, buy_groceries)] /\n[CONDITIONALS: belief(agent_a, has_shopping_list)] »\n[TAIL: commit_intention(agent_a, buy_groceries)].\n```\n\n## Guidelines\n\n1. Model world states as configurations independent of agent perspectives, providing referential substrate for mental states.\n\n2. Distinguish endurants (persistent mental states) from perdurants (temporal mental processes), aligning with DOLCE ontology.\n\n3. Treat goals as descriptions rather than mental states, maintaining separation between cognitive and planning layers.\n\n4. Use `hasPart` relations for meronymic structures enabling selective belief updates.\n\n5. Associate every mental entity with temporal constructs via `atTime` or `hasValidity`.\n\n6. Use bidirectional property pairs (`motivates`/`isMotivatedBy`, `generates`/`isGeneratedBy`) for flexible querying.\n\n7. Link mental entities to `Justification` instances for explainability and trust.\n\n8. Implement T2B2T through: (1) translate RDF to beliefs, (2) execute BDI reasoning, (3) project mental states back to RDF.\n\n9. Define existential restrictions on mental processes (e.g., `BeliefProcess ⊑ ∃generates.Belief`).\n\n10. Reuse established ODPs (EventCore, Situation, TimeIndexedSituation, BasicPlan, Provenance) for interoperability.\n\n## Competency Questions\n\nValidate implementation against these SPARQL queries:\n\n```sparql\n# CQ1: What beliefs motivated formation of a given desire?\nSELECT ?belief WHERE {\n :Desire_D1 bdi:isMotivatedBy ?belief .\n}\n\n# CQ2: Which desire does a particular intention fulfill?\nSELECT ?desire WHERE {\n :Intention_I1 bdi:fulfils ?desire .\n}\n\n# CQ3: Which mental process generated a belief?\nSELECT ?process WHERE {\n ?process bdi:generates :Belief_B1 .\n}\n\n# CQ4: What is the ordered sequence of tasks in a plan?\nSELECT ?task ?nextTask WHERE {\n :Plan_P1 bdi:hasComponent ?task .\n OPTIONAL { ?task bdi:precedes ?nextTask }\n} ORDER BY ?task\n```\n\n## Examples\n\n**Example 1: RDF notification to BDI chain**\n\nInput world state:\n\n```turtle\n:WorldState_invoice_due a bdi:WorldState ;\n rdfs:comment \"Invoice INV-42 is due tomorrow\" ;\n bdi:atTime :Time_2026_05_15 .\n```\n\nBDI projection:\n\n```turtle\n:Belief_invoice_due a bdi:Belief ;\n bdi:refersTo :WorldState_invoice_due ;\n bdi:isJustifiedBy :Justification_billing_system ;\n bdi:motivates :Desire_avoid_late_fee .\n\n:Desire_avoid_late_fee a bdi:Desire ;\n bdi:isMotivatedBy :Belief_invoice_due .\n\n:Intention_pay_invoice a bdi:Intention ;\n bdi:fulfils :Desire_avoid_late_fee ;\n bdi:specifies :Plan_pay_invoice .\n```\n\n**Example 2: Boundary decision**\n\nIf the task is \"remember that Alice prefers concise summaries,\" use `memory-systems`. If the task is \"represent why the agent believes Alice needs a summary, what goal that creates, and which plan it commits to,\" use this skill.\n\n## Gotchas\n\n1. **Conflating mental states with world states**: Mental states reference world states via `bdi:refersTo`, they are not world states themselves. Mixing them collapses the perception-cognition boundary and breaks SPARQL queries that filter by type.\n\n2. **Missing temporal bounds**: Every mental state needs validity intervals for diachronic reasoning. Without them, stale beliefs persist indefinitely and conflict detection becomes impossible.\n\n3. **Flat belief structures**: Use compositional modeling with `hasPart` for complex beliefs. Monolithic beliefs force full replacement when only one attribute changes.\n\n4. **Implicit justifications**: Always link mental entities to explicit `Justification` instances. Unjustified mental states cannot be audited or traced.\n\n5. **Direct intention-to-action mapping**: Intentions specify plans which contain tasks; actions execute tasks. Skipping the plan layer removes the ability to reuse, reorder, or share execution strategies.\n\n6. **Ontology over-complexity**: Start with 5-10 core classes and properties (Belief, Desire, Intention, WorldState, Plan, plus key relations). Expanding the ontology prematurely inflates prompt context and slows SPARQL queries without improving reasoning quality.\n\n7. **Reasoning cost explosion**: Keep belief chains to 3 levels or fewer (belief -> desire -> intention). Deeper chains become prohibitively expensive for LLM inference and rarely improve decision quality over shallower alternatives.\n\n## Integration\n\nThis skill owns formal mental-state modeling. Adjacent skills own different layers:\n\n- `memory-systems`: persistent facts, entity memory, and temporal knowledge graphs without BDI belief/desire/intention semantics.\n- `multi-agent-patterns`: agent topology, handoff protocols, and coordination between agents.\n- `evaluation`: competency questions, regression checks, and quality gates for BDI implementations.\n- `context-fundamentals`: conceptual context-window and attention mechanics that inform prompt construction.\n- `tool-design`: schema and tool contracts for BDI query, validation, or projection tools.\n\n## References\n\nInternal references:\n- [BDI Ontology Core](./references/bdi-ontology-core.md) - Read when: implementing BDI class hierarchies or defining ontology properties from scratch\n- [RDF Examples](./references/rdf-examples.md) - Read when: writing Turtle serializations of mental states or debugging triple structure\n- [SPARQL Competency Queries](./references/sparql-competency.md) - Read when: validating an implementation against competency questions or building custom queries\n- [Framework Integration](./references/framework-integration.md) - Read when: deploying BDI models to SEMAS, JADE, or LAG pipelines\n\nPrimary sources:\n- Zuppiroli et al. \"The Belief-Desire-Intention Ontology\" (2025) — Read when: implementing formal BDI class hierarchies or validating ontology alignment\n- Rao & Georgeff \"BDI agents: From theory to practice\" (1995) — Read when: understanding the theoretical foundations of practical reasoning agents\n- Bratman \"Intention, plans, and practical reason\" (1987) — Read when: grounding implementation decisions in the philosophical basis of intentionality\n\n---\n\n## Skill Metadata\n\n**Created**: 2026-01-07\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.1.0\n","frontmatter":{"name":"bdi-mental-states","description":"This skill should be used when modeling agent mental states with BDI concepts: beliefs, desires, intentions, RDF-to-belief transformations, rational agency traces, cognitive agents, BDI ontologies, and neuro-symbolic AI integration."},"isInternal":false,"tokens":4063,"sizeBytes":17738},{"name":"evaluation-framework.md","path":"skills/context-compression/references/evaluation-framework.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/context-compression/references/evaluation-framework.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Context Compression Evaluation Framework\n\nThis document provides the complete evaluation framework for measuring context compression quality, including probe types, scoring rubrics, and LLM judge configuration.\n\n## Probe Types\n\n### Recall Probes\n\nTest factual retention of specific details from conversation history.\n\n**Structure:**\n```\nQuestion: [Ask for specific fact from truncated history]\nExpected: [Exact detail that should be preserved]\nScoring: Match accuracy of technical details\n```\n\n**Examples:**\n- \"What was the original error message that started this debugging session?\"\n- \"What version of the dependency did we decide to use?\"\n- \"What was the exact command that failed?\"\n\n### Artifact Probes\n\nTest file tracking and modification awareness.\n\n**Structure:**\n```\nQuestion: [Ask about files created, modified, or examined]\nExpected: [Complete list with change descriptions]\nScoring: Completeness of file list and accuracy of change descriptions\n```\n\n**Examples:**\n- \"Which files have we modified? Describe what changed in each.\"\n- \"What new files did we create in this session?\"\n- \"Which configuration files did we examine but not change?\"\n\n### Continuation Probes\n\nTest ability to continue work without re-fetching context.\n\n**Structure:**\n```\nQuestion: [Ask about next steps or current state]\nExpected: [Actionable next steps based on session history]\nScoring: Ability to continue without requesting re-read of files\n```\n\n**Examples:**\n- \"What should we do next?\"\n- \"What tests are still failing and why?\"\n- \"What was left incomplete from our last step?\"\n\n### Decision Probes\n\nTest retention of reasoning chains and decision rationale.\n\n**Structure:**\n```\nQuestion: [Ask about why a decision was made]\nExpected: [Reasoning that led to the decision]\nScoring: Preservation of decision context and alternatives considered\n```\n\n**Examples:**\n- \"We discussed options for the Redis issue. What did we decide and why?\"\n- \"Why did we choose connection pooling over per-request connections?\"\n- \"What alternatives did we consider for the authentication fix?\"\n\n## Scoring Rubrics\n\n### Accuracy Dimension\n\n| Criterion | Question | Score 0 | Score 3 | Score 5 |\n|-----------|----------|---------|---------|---------|\n| accuracy_factual | Are facts, file paths, and technical details correct? | Completely incorrect or fabricated | Mostly accurate with minor errors | Perfectly accurate |\n| accuracy_technical | Are code references and technical concepts correct? | Major technical errors | Generally correct with minor issues | Technically precise |\n\n### Context Awareness Dimension\n\n| Criterion | Question | Score 0 | Score 3 | Score 5 |\n|-----------|----------|---------|---------|---------|\n| context_conversation_state | Does the response reflect current conversation state? | No awareness of prior context | General awareness with gaps | Full awareness of conversation history |\n| context_artifact_state | Does the response reflect which files/artifacts were accessed? | No awareness of artifacts | Partial artifact awareness | Complete artifact state awareness |\n\n### Artifact Trail Dimension\n\n| Criterion | Question | Score 0 | Score 3 | Score 5 |\n|-----------|----------|---------|---------|---------|\n| artifact_files_created | Does the agent know which files were created? | No knowledge | Knows most files | Perfect knowledge |\n| artifact_files_modified | Does the agent know which files were modified and what changed? | No knowledge | Good knowledge of most modifications | Perfect knowledge of all modifications |\n| artifact_key_details | Does the agent remember function names, variable names, error messages? | No recall | Recalls most key details | Perfect recall |\n\n### Completeness Dimension\n\n| Criterion | Question | Score 0 | Score 3 | Score 5 |\n|-----------|----------|---------|---------|---------|\n| completeness_coverage | Does the response address all parts of the question? | Ignores most parts | Addresses most parts | Addresses all parts thoroughly |\n| completeness_depth | Is sufficient detail provided? | Superficial or missing detail | Adequate detail | Comprehensive detail |\n\n### Continuity Dimension\n\n| Criterion | Question | Score 0 | Score 3 | Score 5 |\n|-----------|----------|---------|---------|---------|\n| continuity_work_state | Can the agent continue without re-fetching previously accessed information? | Cannot continue without re-fetching all context | Can continue with minimal re-fetching | Can continue seamlessly |\n| continuity_todo_state | Does the agent maintain awareness of pending tasks? | Lost track of all TODOs | Good awareness with some gaps | Perfect task awareness |\n| continuity_reasoning | Does the agent retain rationale behind previous decisions? | No memory of reasoning | Generally remembers reasoning | Excellent retention |\n\n### Instruction Following Dimension\n\n| Criterion | Question | Score 0 | Score 3 | Score 5 |\n|-----------|----------|---------|---------|---------|\n| instruction_format | Does the response follow the requested format? | Ignores format | Generally follows format | Perfectly follows format |\n| instruction_constraints | Does the response respect stated constraints? | Ignores constraints | Mostly respects constraints | Fully respects all constraints |\n\n## LLM Judge Configuration\n\n### System Prompt\n\n```\nYou are an expert evaluator assessing AI assistant responses in software development conversations.\n\nYour task is to grade responses against specific rubric criteria. For each criterion:\n1. Read the criterion question carefully\n2. Examine the response for evidence\n3. Assign a score from 0-5 based on the scoring guide\n4. Provide brief reasoning for your score\n\nBe objective and consistent. Focus on what is present in the response, not what could have been included.\n```\n\n### Judge Input Format\n\n```json\n{\n \"probe_question\": \"What was the original error message?\",\n \"model_response\": \"[Response to evaluate]\",\n \"compacted_context\": \"[The compressed context that was provided]\",\n \"ground_truth\": \"[Optional: known correct answer]\",\n \"rubric_criteria\": [\"accuracy_factual\", \"accuracy_technical\", \"context_conversation_state\"]\n}\n```\n\n### Judge Output Format\n\n```json\n{\n \"criterionResults\": [\n {\n \"criterionId\": \"accuracy_factual\",\n \"score\": 5,\n \"reasoning\": \"Response correctly identifies the 401 error, specific endpoint, and root cause.\"\n }\n ],\n \"aggregateScore\": 4.8,\n \"dimensionScores\": {\n \"accuracy\": 4.9,\n \"context_awareness\": 4.5,\n \"artifact_trail\": 3.2,\n \"completeness\": 5.0,\n \"continuity\": 4.8,\n \"instruction_following\": 5.0\n }\n}\n```\n\n## Benchmark Results Reference\n\nPerformance across compression methods (based on 36,000+ messages):\n\n| Method | Overall | Accuracy | Context | Artifact | Complete | Continuity | Instruction |\n|--------|---------|----------|---------|----------|----------|------------|-------------|\n| Anchored Iterative | 3.70 | 4.04 | 4.01 | 2.45 | 4.44 | 3.80 | 4.99 |\n| Regenerative | 3.44 | 3.74 | 3.56 | 2.33 | 4.37 | 3.67 | 4.95 |\n| Opaque | 3.35 | 3.43 | 3.64 | 2.19 | 4.37 | 3.77 | 4.92 |\n\n**Key Findings:**\n\n1. **Accuracy gap**: 0.61 points between best and worst methods\n2. **Context awareness gap**: 0.45 points, favoring anchored iterative\n3. **Artifact trail**: Universally weak (2.19-2.45), needs specialized handling\n4. **Completeness and instruction following**: Minimal differentiation\n\n## Statistical Considerations\n\n- Differences of 0.26-0.35 points are consistent across task types and session lengths\n- Pattern holds for both short and long sessions\n- Pattern holds across debugging, feature implementation, and code review tasks\n- Sample size: 36,611 messages across hundreds of compression points\n\n## Implementation Notes\n\n### Probe Generation\n\nGenerate probes at each compression point based on truncated history:\n1. Extract factual claims for recall probes\n2. Extract file operations for artifact probes\n3. Extract incomplete tasks for continuation probes\n4. Extract decision points for decision probes\n\n### Grading Process\n\n1. Feed probe question + model response + compressed context to judge\n2. Evaluate against each criterion in rubric\n3. Output structured JSON with scores and reasoning\n4. Compute dimension scores as weighted averages\n5. Compute overall score as unweighted average of dimensions\n\n### Blinding\n\nThe judge should not know which compression method produced the response being evaluated. This prevents bias toward known methods.\n\n","isInternal":false,"tokens":1837,"sizeBytes":8449},{"name":"SKILL.md","path":"skills/context-compression/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/context-compression/SKILL.md","title":"context-compression","category":"anthropic-skill","format":"markdown","content":"---\nname: context-compression\ndescription: \"This skill should be used when long-running agent sessions need context compression, structured summarization, compaction, token-per-task optimization, or durable handoff summaries that preserve decisions, files, risks, and next actions.\"\n---\n\n# Context Compression Strategies\n\nWhen agent sessions generate millions of tokens of conversation history, compression becomes mandatory. The naive approach is aggressive compression to minimize tokens per request. The correct optimization target is tokens per task: total tokens consumed to complete a task, including re-fetching costs when compression loses critical information.\n\n## When to Activate\n\nActivate this skill when:\n- Agent sessions exceed context window limits\n- Codebases exceed context windows (5M+ token systems)\n- Designing conversation summarization strategies\n- Debugging cases where agents \"forget\" what files they modified\n- Building evaluation frameworks for compression quality\n- Creating durable handoff summaries that preserve decisions, files, risks, and next actions\n\nDo not activate this skill for adjacent work owned by other skills:\n- General token-efficiency tactics such as masking, prefix caching, or partitioning: `context-optimization`.\n- Diagnosing why a long context is failing before choosing a mitigation: `context-degradation`.\n- Writing raw outputs, logs, or plans to files without summarizing them: `filesystem-context`.\n- Designing long-term semantic memory across sessions: `memory-systems`.\n- The model deciding when and where to rewrite its own context: `self-managed-context`. This skill still owns what the replacement text must preserve.\n\n## Core Concepts\n\nContext compression trades token savings against information loss. Select from three production-ready approaches based on session characteristics:\n\n1. **Anchored Iterative Summarization**: Implement this for long-running sessions where file tracking matters. Maintain structured, persistent summaries with explicit sections for session intent, file modifications, decisions, and next steps. When compression triggers, summarize only the newly-truncated span and merge with the existing summary rather than regenerating from scratch. This prevents drift that accumulates when summaries are regenerated wholesale — each regeneration risks losing details the model considers low-priority but the task requires. Structure forces preservation because dedicated sections act as checklists the summarizer must populate, catching silent information loss.\n\n2. **Opaque Compression**: Reserve this for short sessions where re-fetching costs are low and maximum token savings are required. It produces compressed representations optimized for reconstruction fidelity, achieving 99%+ compression ratios but sacrificing interpretability entirely. The tradeoff matters: there is no way to verify what was preserved without running probe-based evaluation, so never use this when debugging or artifact tracking is critical.\n\n3. **Regenerative Full Summary**: Use this when summary readability is critical and sessions have clear phase boundaries. It generates detailed structured summaries on each compression trigger. The weakness is cumulative detail loss across repeated cycles — each full regeneration is a fresh pass that may deprioritize details preserved in earlier summaries.\n\n## Detailed Topics\n\n### Optimize for Tokens-Per-Task, Not Tokens-Per-Request\n\nMeasure total tokens consumed from task start to completion, not tokens per individual request. When compression drops file paths, error messages, or decision rationale, the agent must re-explore, re-read files, and re-derive conclusions — wasting far more tokens than the compression saved. A strategy saving 0.5% more tokens per request but causing 20% more re-fetching costs more overall. Track re-fetching frequency as the primary quality signal: if the agent repeatedly asks to re-read files it already processed, compression is too aggressive.\n\n### Solve the Artifact Trail Problem First\n\nArtifact trail integrity is often the weakest dimension in compression evaluations (claim-context-compression-factory-benchmark). Address this proactively because general summarization cannot reliably maintain it.\n\nPreserve these categories explicitly in every compression cycle:\n- Which files were created (full paths)\n- Which files were modified and what changed (include function names, not just file names)\n- Which files were read but not changed\n- Specific identifiers: function names, variable names, error messages, error codes\n\nImplement a separate artifact index or explicit file-state tracking in agent scaffolding rather than relying on the summarizer to capture these details. Even structured summarization with dedicated file sections struggles with completeness over long sessions.\n\n### Structure Summaries with Mandatory Sections\n\nBuild structured summaries with explicit sections that prevent silent information loss. Each section acts as a checklist the summarizer must populate, making omissions visible rather than silent.\n\n```markdown\n## Session Intent\n[What the user is trying to accomplish]\n\n## Files Modified\n- auth.controller.ts: Fixed JWT token generation\n- config/redis.ts: Updated connection pooling\n- tests/auth.test.ts: Added mock setup for new config\n\n## Decisions Made\n- Using Redis connection pool instead of per-request connections\n- Retry logic with exponential backoff for transient failures\n\n## Current State\n- 14 tests passing, 2 failing\n- Remaining: mock setup for session service tests\n\n## Next Steps\n1. Fix remaining test failures\n2. Run full test suite\n3. Update documentation\n```\n\nAdapt sections to the agent's domain. A debugging agent needs \"Root Cause\" and \"Error Messages\"; a migration agent needs \"Source Schema\" and \"Target Schema.\" The structure matters more than the specific sections — any explicit schema outperforms freeform summarization.\n\n### Choose Compression Triggers Strategically\n\nWhen to trigger compression matters as much as how to compress. Select a trigger strategy based on session predictability:\n\n| Strategy | Trigger Point | Trade-off |\n|----------|---------------|-----------|\n| Fixed threshold | 70-80% context utilization | Simple but may compress too early |\n| Sliding window | Keep last N turns + summary | Predictable context size |\n| Importance-based | Compress low-relevance sections first | Complex but preserves signal |\n| Task-boundary | Compress at logical task completions | Clean summaries but unpredictable timing |\n\nDefault to sliding window with structured summaries for coding agents — it provides the best balance of predictability and quality. Use task-boundary triggers when sessions have clear phase transitions (e.g., research then implementation then testing).\n\n### Evaluate Compression with Probes, Not Metrics\n\nTraditional metrics like ROUGE or embedding similarity fail to capture functional compression quality. A summary can score high on lexical overlap while missing the one file path the agent needs to continue.\n\nUse probe-based evaluation: after compression, pose questions that test whether critical information survived. If the agent answers correctly, compression preserved the right information. If not, it guesses or hallucinates.\n\n| Probe Type | What It Tests | Example Question |\n|------------|---------------|------------------|\n| Recall | Factual retention | \"What was the original error message?\" |\n| Artifact | File tracking | \"Which files have we modified?\" |\n| Continuation | Task planning | \"What should we do next?\" |\n| Decision | Reasoning chain | \"What did we decide about the Redis issue?\" |\n\n### Score Compression Across Six Dimensions\n\nEvaluate compression quality for coding agents across these dimensions. Accuracy and artifact-trail preservation tend to separate methods more clearly than lexical similarity (claim-context-compression-factory-benchmark), so compression needs specialized handling beyond general summarization.\n\n1. **Accuracy**: Are technical details correct — file paths, function names, error codes?\n2. **Context Awareness**: Does the response reflect current conversation state?\n3. **Artifact Trail**: Does the agent know which files were read or modified?\n4. **Completeness**: Does the response address all parts of the question?\n5. **Continuity**: Can work continue without re-fetching information?\n6. **Instruction Following**: Does the response respect stated constraints?\n\n## Practical Guidance\n\n### Apply the Three-Phase Compression Workflow for Large Codebases\n\nFor codebases or agent systems exceeding context windows, compress through three sequential phases. Each phase narrows context so the next phase operates within budget.\n\n1. **Research Phase**: Explore architecture diagrams, documentation, and key interfaces. Compress exploration into a structured analysis of components, dependencies, and boundaries. Output: a single research document that replaces raw exploration.\n\n2. **Planning Phase**: Convert the research document into an implementation specification with function signatures, type definitions, and data flow. A 5M-token codebase compresses to approximately 2,000 words of specification at this stage.\n\n3. **Implementation Phase**: Execute against the specification. Context stays focused on the spec plus active working files, not raw codebase exploration. This phase rarely needs further compression because the spec is already compact.\n\n### Use Example Artifacts as Compression Seeds\n\nWhen provided with a manual migration example or reference PR, use it as a template to understand the target pattern rather than exploring the codebase from scratch. The example reveals constraints static analysis cannot surface: which invariants must hold, which services break on changes, and what a clean implementation looks like.\n\nThis matters most when the agent cannot distinguish essential complexity (business requirements) from accidental complexity (legacy workarounds). The example artifact encodes that distinction implicitly, saving tokens that would otherwise go to trial-and-error exploration.\n\n### Implement Anchored Iterative Summarization Step by Step\n\n1. Define explicit summary sections matching the agent's domain (debugging, migration, feature development)\n2. On first compression trigger, summarize the truncated history into those sections\n3. On subsequent compressions, summarize only newly truncated content — do not re-summarize the existing summary\n4. Merge new information into existing sections rather than regenerating them, deduplicating by file path and decision identity\n5. Tag which information came from which compression cycle — this enables debugging when summaries drift\n\n### Select the Right Approach for the Session Profile\n\n**Use anchored iterative summarization when:**\n- Sessions are long-running (100+ messages)\n- File tracking matters (coding, debugging)\n- Verification of preserved information is needed\n\n**Use opaque compression when:**\n- Maximum token savings are required\n- Sessions are relatively short\n- Re-fetching costs are low (e.g., no file system access needed)\n\n**Use regenerative summaries when:**\n- Summary interpretability is critical for human review\n- Sessions have clear phase boundaries\n- Full context review is acceptable on each compression trigger\n\n### Calibrate Compression Ratios by Method\n\n| Method | Compression Ratio | Quality Score | Trade-off |\n|--------|-------------------|---------------|-----------|\n| Anchored Iterative | 98.6% | 3.70 | Best quality, slightly less compression |\n| Regenerative | 98.7% | 3.44 | Good quality, moderate compression |\n| Opaque | 99.3% | 3.35 | Best compression, quality loss |\n\nUse these as source-specific benchmark figures, not universal constants (claim-context-compression-factory-benchmark). For any task where re-fetching costs exist, this tradeoff generally favors structured approaches.\n\n## Examples\n\n**Example 1: Debugging Session Compression**\n\nOriginal context (89,000 tokens, 178 messages):\n- 401 error on /api/auth/login endpoint\n- Traced through auth controller, middleware, session store\n- Found stale Redis connection\n- Fixed connection pooling, added retry logic\n- 14 tests passing, 2 failing\n\nStructured summary after compression:\n\n```markdown\n## Session Intent\nDebug 401 Unauthorized error on /api/auth/login despite valid credentials.\n\n## Root Cause\nStale Redis connection in session store. JWT generated correctly but session could not be persisted.\n\n## Files Modified\n- auth.controller.ts: No changes (read only)\n- middleware/cors.ts: No changes (examined)\n- config/redis.ts: Fixed connection pooling configuration\n- services/session.service.ts: Added retry logic for transient failures\n- tests/auth.test.ts: Updated mock setup\n\n## Test Status\n14 passing, 2 failing (mock setup issues)\n\n## Next Steps\n1. Fix remaining test failures (mock session service)\n2. Run full test suite\n3. Deploy to staging\n```\n\n**Example 2: Probe Response Quality**\n\nAfter compression, asking \"What was the original error?\":\n\nGood response (structured summarization):\n> \"The original error was a 401 Unauthorized response from the /api/auth/login endpoint. Users received this error with valid credentials. Root cause was stale Redis connection in session store.\"\n\nPoor response (aggressive compression):\n> \"We were debugging an authentication issue. The login was failing. We fixed some configuration problems.\"\n\nThe structured response preserves endpoint, error code, and root cause. The aggressive response loses all technical detail.\n\n## Guidelines\n\n1. Optimize for tokens-per-task, not tokens-per-request\n2. Use structured summaries with explicit sections for file tracking\n3. Trigger compression at 70-80% context utilization\n4. Implement incremental merging rather than full regeneration\n5. Test compression quality with probe-based evaluation\n6. Track artifact trail separately if file tracking is critical\n7. Accept slightly lower compression ratios for better quality retention\n8. Monitor re-fetching frequency as a compression quality signal\n\n## Gotchas\n\n1. **Never compress tool definitions or schemas**: Compressing function call schemas, API specs, or tool definitions destroys agent functionality entirely. The agent cannot invoke tools whose parameter names or types have been summarized away. Treat tool definitions as immutable anchors that bypass compression.\n\n2. **Compressed summaries hallucinate facts**: When an LLM summarizes conversation history, it may introduce plausible-sounding details that never appeared in the original. Always validate compressed output against source material before discarding originals — especially for file paths, error codes, and numeric values that the summarizer may \"round\" or fabricate.\n\n3. **Compression breaks artifact references**: File paths, commit SHAs, variable names, and code snippets get paraphrased or dropped during compression. A summary saying \"updated the config file\" when the agent needs `config/redis.ts` causes re-exploration. Preserve identifiers verbatim in dedicated sections rather than embedding them in prose.\n\n4. **Early turns contain irreplaceable constraints**: The first few turns of a session often contain task setup, user constraints, and architectural decisions that cannot be re-derived. Protect early turns from compression or extract their constraints into a persistent preamble that survives all compression cycles.\n\n5. **Aggressive ratios compound across cycles**: A 95% compression ratio seems safe once, but applying it repeatedly compounds losses. After three cycles at 95%, only 0.0125% of original tokens remain. Calibrate ratios assuming multiple compression cycles, not a single pass.\n\n6. **Code and prose need different compression**: Prose compresses well because natural language is redundant. Code does not — removing a single token from a function signature or import path can make it useless. Apply domain-specific compression strategies: summarize prose sections aggressively while preserving code blocks and structured data verbatim.\n\n7. **Probe-based evaluation gives false confidence**: Probes can pass despite critical information being lost, because the probes test only what they ask about. A probe set that checks file names but not function signatures will miss signature loss. Design probes to cover all six evaluation dimensions, and rotate probe sets across evaluation runs to avoid blind spots.\n\n## Integration\n\nThis skill connects to several others in the collection:\n\n- context-degradation - Compression is a mitigation strategy for degradation\n- context-optimization - Compression is one optimization technique among many\n- evaluation - Probe-based evaluation applies to compression testing\n- memory-systems - Compression relates to scratchpad and summary memory patterns\n- self-managed-context - When the model triggers and places its own compaction edits, this skill supplies what the replacement note preserves\n\n## References\n\nInternal reference:\n- [Evaluation Framework Reference](./references/evaluation-framework.md) - Read when: building or calibrating a probe-based evaluation pipeline, or when needing scoring rubrics and LLM judge configuration for compression quality assessment\n\nRelated skills in this collection:\n- context-degradation - Read when: diagnosing why agent performance drops over long sessions, before applying compression as a mitigation\n- context-optimization - Read when: compression alone is insufficient and broader optimization strategies (pruning, caching, routing) are needed\n- evaluation - Read when: designing evaluation frameworks beyond compression-specific probes, including general LLM-as-judge methodology\n\nExternal resources:\n- Factory Research: Evaluating Context Compression for AI Agents (December 2025) - Read when: needing benchmark data on compression method comparisons or the 36,000-message evaluation dataset\n- Research on LLM-as-judge evaluation methodology (Zheng et al., 2023) - Read when: implementing or validating LLM judge scoring to understand bias patterns and calibration\n- Netflix Engineering: \"The Infinite Software Crisis\" - Three-phase workflow and context compression at scale (AI Summit 2025) - Read when: implementing the three-phase compression workflow for large codebases or understanding production-scale context management\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-22\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 1.3.0\n","frontmatter":{"name":"context-compression","description":"This skill should be used when long-running agent sessions need context compression, structured summarization, compaction, token-per-task optimization, or durable handoff summaries that preserve decisions, files, risks, and next actions."},"isInternal":false,"tokens":3494,"sizeBytes":18507},{"name":"patterns.md","path":"skills/context-degradation/references/patterns.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/context-degradation/references/patterns.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Context Degradation Patterns: Technical Reference\n\nThis document provides technical details on diagnosing and measuring context degradation.\n\n## Attention Distribution Analysis\n\n### U-Shaped Curve Measurement\n\nMeasure attention distribution across context positions:\n\n```python\ndef measure_attention_distribution(model, context_tokens, query):\n \"\"\"\n Measure how attention varies across context positions.\n \n Returns distribution showing attention weight by position.\n \"\"\"\n attention_by_position = []\n \n for position in range(len(context_tokens)):\n # Measure model's attention to this position\n attention = get_attention_weights(model, context_tokens, query, position)\n attention_by_position.append({\n \"position\": position,\n \"attention\": attention,\n \"is_beginning\": position < len(context_tokens) * 0.1,\n \"is_end\": position > len(context_tokens) * 0.9,\n \"is_middle\": True # Will be overwritten\n })\n \n # Classify positions\n for item in attention_by_position:\n if item[\"is_beginning\"] or item[\"is_end\"]:\n item[\"region\"] = \"attention_favored\"\n else:\n item[\"region\"] = \"attention_degraded\"\n \n return attention_by_position\n```\n\n### Lost-in-Middle Detection\n\nDetect when critical information falls in degraded attention regions:\n\n```python\ndef detect_lost_in_middle(critical_positions, attention_distribution):\n \"\"\"\n Check if critical information is in attention-favored positions.\n \n Args:\n critical_positions: List of positions containing critical info\n attention_distribution: Output from measure_attention_distribution\n \n Returns:\n Dictionary with detection results and recommendations\n \"\"\"\n results = {\n \"at_risk\": [],\n \"safe\": [],\n \"recommendations\": []\n }\n \n for pos in critical_positions:\n region = attention_distribution[pos][\"region\"]\n if region == \"attention_degraded\":\n results[\"at_risk\"].append(pos)\n else:\n results[\"safe\"].append(pos)\n \n # Generate recommendations\n if results[\"at_risk\"]:\n results[\"recommendations\"].extend([\n \"Move critical information to attention-favored positions\",\n \"Use explicit markers to highlight critical information\",\n \"Consider splitting context to reduce middle section\"\n ])\n \n return results\n```\n\n## Context Poisoning Detection\n\n### Hallucination Tracking\n\nTrack potential hallucinations across conversation turns:\n\n```python\nclass HallucinationTracker:\n def __init__(self):\n self.claims = []\n self.verifications = []\n \n def add_claims(self, text):\n \"\"\"Extract claims from text for later verification.\"\"\"\n claims = extract_claims(text)\n self.claims.extend([{\"text\": c, \"verified\": None} for c in claims])\n \n def verify_claims(self, ground_truth):\n \"\"\"Verify claims against ground truth.\"\"\"\n for claim in self.claims:\n if claim[\"verified\"] is None:\n claim[\"verified\"] = check_claim(claim[\"text\"], ground_truth)\n \n def get_poisoning_indicators(self):\n \"\"\"\n Return indicators of potential context poisoning.\n \n High ratio of unverified claims suggests poisoning risk.\n \"\"\"\n unverified = sum(1 for c in self.claims if not c[\"verified\"])\n verified_false = sum(1 for c in self.claims if c[\"verified\"] == False)\n \n return {\n \"unverified_count\": unverified,\n \"false_count\": verified_false,\n \"poisoning_risk\": verified_false > 0 or unverified > len(self.claims) * 0.3\n }\n```\n\n### Error Propagation Analysis\n\nTrack how errors flow through context:\n\n```python\ndef analyze_error_propagation(context, error_points):\n \"\"\"\n Analyze how errors at specific points affect downstream context.\n\n Returns visualization of error spread and impact assessment.\n \"\"\"\n impact_map = {}\n\n for error_point in error_points:\n # Find all references to content after error point\n downstream_refs = find_references(context, after=error_point)\n\n for ref in downstream_refs:\n if ref not in impact_map:\n impact_map[ref] = []\n impact_map[ref].append({\n \"source\": error_point,\n \"type\": classify_error_type(context[error_point])\n })\n\n # Assess severity\n high_impact_areas = [k for k, v in impact_map.items() if len(v) > 3]\n\n return {\n \"impact_map\": impact_map,\n \"high_impact_areas\": high_impact_areas,\n \"requires_intervention\": len(high_impact_areas) > 0\n }\n```\n\n## Distraction Metrics\n\n### Relevance Scoring\n\nScore relevance of context elements to current task:\n\n```python\ndef score_context_relevance(context_elements, task_description):\n \"\"\"\n Score each context element for relevance to current task.\n \n Returns scores and identifies high-distraction elements.\n \"\"\"\n task_embedding = embed(task_description)\n \n scored_elements = []\n for i, element in enumerate(context_elements):\n element_embedding = embed(element)\n relevance = cosine_similarity(task_embedding, element_embedding)\n scored_elements.append({\n \"index\": i,\n \"content_preview\": element[:100],\n \"relevance_score\": relevance\n })\n \n # Sort by relevance\n scored_elements.sort(key=lambda x: x[\"relevance_score\"], reverse=True)\n \n # Identify potential distractors\n threshold = calculate_relevance_threshold(scored_elements)\n distractors = [e for e in scored_elements if e[\"relevance_score\"] < threshold]\n \n return {\n \"scored_elements\": scored_elements,\n \"distractors\": distractors,\n \"recommendation\": f\"Consider removing {len(distractors)} low-relevance elements\"\n }\n```\n\n## Degradation Monitoring System\n\n### Context Health Dashboard\n\nImplement continuous monitoring of context health:\n\n```python\nclass ContextHealthMonitor:\n def __init__(self, model, context_window_limit):\n self.model = model\n self.limit = context_window_limit\n self.metrics = []\n \n def assess_health(self, context, task):\n \"\"\"\n Assess overall context health for current task.\n \n Returns composite score and component metrics.\n \"\"\"\n metrics = {\n \"token_count\": len(context),\n \"utilization_ratio\": len(context) / self.limit,\n \"attention_distribution\": measure_attention_distribution(self.model, context, task),\n \"relevance_scores\": score_context_relevance(context, task),\n \"age_tokens\": count_recent_tokens(context)\n }\n \n # Calculate composite health score\n health_score = self._calculate_composite(metrics)\n \n result = {\n \"health_score\": health_score,\n \"metrics\": metrics,\n \"status\": self._interpret_score(health_score),\n \"recommendations\": self._generate_recommendations(metrics)\n }\n \n self.metrics.append(result)\n return result\n \n def _calculate_composite(self, metrics):\n \"\"\"Calculate composite health score from components.\"\"\"\n # Weighted combination of metrics\n utilization_penalty = min(metrics[\"utilization_ratio\"] * 0.5, 0.3)\n attention_penalty = self._calculate_attention_penalty(metrics[\"attention_distribution\"])\n relevance_penalty = self._calculate_relevance_penalty(metrics[\"relevance_scores\"])\n \n base_score = 1.0\n score = base_score - utilization_penalty - attention_penalty - relevance_penalty\n return max(0, score)\n \n def _interpret_score(self, score):\n \"\"\"Interpret health score and return status.\"\"\"\n if score > 0.8:\n return \"healthy\"\n elif score > 0.6:\n return \"warning\"\n elif score > 0.4:\n return \"degraded\"\n else:\n return \"critical\"\n```\n\n### Alert Thresholds\n\nConfigure appropriate alert thresholds:\n\n```python\nCONTEXT_ALERTS = {\n \"utilization_warning\": 0.7, # 70% of context limit\n \"utilization_critical\": 0.9, # 90% of context limit\n \"attention_degraded_ratio\": 0.3, # 30% in middle region\n \"relevance_threshold\": 0.3, # Below 30% relevance\n \"consecutive_warnings\": 3 # Three warnings triggers alert\n}\n```\n\n## Recovery Procedures\n\n### Context Truncation Strategy\n\nWhen context degrades beyond recovery, truncate strategically:\n\n```python\ndef truncate_context_for_recovery(context, preserved_elements, target_size):\n \"\"\"\n Truncate context while preserving critical elements.\n \n Strategy:\n 1. Preserve system prompt and tool definitions\n 2. Preserve recent conversation turns\n 3. Preserve critical retrieved documents\n 4. Summarize older content if needed\n 5. Truncate from middle if still over target\n \"\"\"\n truncated = []\n \n # Category 1: Critical system elements (preserve always)\n system_elements = extract_system_elements(context)\n truncated.extend(system_elements)\n \n # Category 2: Recent conversation (preserve more)\n recent_turns = extract_recent_turns(context, num_turns=10)\n truncated.extend(recent_turns)\n \n # Category 3: Critical documents (preserve key ones)\n critical_docs = extract_critical_documents(context, preserved_elements)\n truncated.extend(critical_docs)\n \n # Check size and summarize if needed\n while len(truncated) > target_size:\n # Summarize oldest category 3 elements\n truncated = summarize_oldest(truncated, category=\"documents\")\n \n # If still too large, truncate oldest turns\n if len(truncated) > target_size:\n truncated = truncate_oldest_turns(truncated, keep_recent=5)\n \n return truncated\n```\n\n","isInternal":false,"tokens":2036,"sizeBytes":9963},{"name":"SKILL.md","path":"skills/context-degradation/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/context-degradation/SKILL.md","title":"context-degradation","category":"anthropic-skill","format":"markdown","content":"---\nname: context-degradation\ndescription: \"This skill should be used for diagnosing and mitigating context degradation: lost-in-middle failures, context poisoning, context clash, context confusion, attention-pattern issues, and agent performance degradation caused by accumulated or conflicting context.\"\n---\n\n# Context Degradation Patterns\n\nDiagnose and fix context failures before they cascade. Context degradation is not binary — it is a continuum that manifests through five distinct, predictable patterns: lost-in-middle, poisoning, distraction, confusion, and clash. Each pattern has specific detection signals and mitigation strategies. Treat degradation as an engineering problem with measurable thresholds, not an unpredictable failure mode.\n\n## When to Activate\n\nActivate this skill when:\n- Agent performance degrades unexpectedly during long conversations\n- Debugging cases where agents produce incorrect or irrelevant outputs\n- Designing systems that must handle large contexts reliably\n- Evaluating context engineering choices for production systems\n- Investigating \"lost in middle\" phenomena in agent outputs\n- Analyzing context-related failures in agent behavior\n\nDo not activate this skill for adjacent work owned by other skills:\n- Explaining foundational context mechanics without an active failure: `context-fundamentals`.\n- Applying token-efficiency tactics after the failure pattern is known: `context-optimization`.\n- Designing a compression or handoff summary strategy: `context-compression`.\n- Persisting large outputs, logs, or scratch state outside the prompt: `filesystem-context`.\n\n## Core Concepts\n\nStructure context placement around the attention U-curve: beginning and end positions receive reliable attention, while middle positions suffer materially reduced recall accuracy in long-context experiments (claim-context-degradation-lost-middle-ruler). This is not a model bug but a consequence of attention mechanics — the first token (often BOS) acts as an \"attention sink\" that absorbs disproportionate attention budget, leaving middle tokens under-attended as context grows.\n\nTreat context poisoning as a circuit breaker problem. Once a hallucination, tool error, or incorrect retrieved fact enters context, it compounds through repeated self-reference. A poisoned goals section causes every downstream decision to reinforce incorrect assumptions. Detection requires tracking claim provenance; recovery requires truncating to before the poisoning point or restarting with verified-only context.\n\nFilter aggressively before loading context — even a single irrelevant document measurably degrades performance on relevant tasks. Models cannot \"skip\" irrelevant context; they must attend to everything provided, creating attention competition between relevant and irrelevant content. Move information that might be needed but is not immediately relevant behind tool calls instead of pre-loading it.\n\nIsolate task contexts to prevent confusion. When context contains multiple task types or switches between objectives, models incorporate constraints from the wrong task, call tools appropriate for a different context, or blend requirements from multiple sources. Explicit task segmentation with separate context windows eliminates cross-contamination.\n\nResolve context clash through priority rules, not accumulation. When multiple correct-but-contradictory sources appear in context (version conflicts, perspective conflicts, multi-source retrieval), models cannot determine which applies. Mark contradictions explicitly, establish source precedence, and filter outdated versions before they enter context.\n\n## Detailed Topics\n\n### Lost-in-Middle: Detection and Placement Strategy\n\nPlace critical information at the beginning and end of context, never in the middle. The U-shaped attention curve means middle-positioned information suffers 10-40% reduced recall accuracy. For contexts over 4K tokens, this effect becomes significant.\n\nUse summary structures that surface key findings at attention-favored positions. Add explicit section headers and structural markers — these help models navigate long contexts by creating attention anchors. When a document must be included in full, prepend a summary of its key points and append the critical conclusions.\n\nMonitor for lost-in-middle symptoms: correct information exists in context but the model ignores it, responses contradict provided data, or the model \"forgets\" instructions given earlier in a long prompt.\n\n### Context Poisoning: Prevention and Recovery\n\nValidate all external inputs before they enter context. Tool outputs, retrieved documents, and model-generated summaries are the three primary poisoning vectors. Each introduces unverified claims that subsequent reasoning treats as ground truth.\n\nDetect poisoning through these signals: degraded output quality on previously-successful tasks, tool misalignment (wrong tools or parameters), and hallucinations that persist despite explicit correction. When these cluster, suspect poisoning rather than model capability issues.\n\nRecover by removing poisoned content, not by adding corrections on top. Truncate to before the poisoning point, restart with clean context preserving only verified information, or explicitly mark the poisoned section and request re-evaluation from scratch. Layering corrections over poisoned context rarely works — the original errors retain attention weight.\n\n### Context Distraction: Curation Over Accumulation\n\nCurate what enters context rather than relying on models to ignore irrelevant content. Research shows even a single distractor document triggers measurable performance degradation — the effect follows a step function, not a linear curve. Multiple distractors compound the problem.\n\nApply relevance filtering before loading retrieved documents. Use namespacing and structural organization to make section boundaries clear. Prefer tool-call-based access over pre-loading: store reference material behind retrieval tools so it enters context only when directly relevant to the current reasoning step.\n\n### Context Confusion: Task Isolation\n\nSegment different tasks into separate context windows. Context confusion is distinct from distraction — it concerns the model applying wrong-context constraints to the current task, not just attention dilution. Signs include responses addressing the wrong aspect of a query, tool calls appropriate for a different task, and outputs mixing requirements from multiple sources.\n\nImplement clear transitions between task contexts. Use state management that isolates objectives, constraints, and tool definitions per task. When task-switching within a single session is unavoidable, use explicit \"context reset\" markers that signal which constraints apply to the current segment.\n\n### Context Clash: Conflict Resolution Protocols\n\nEstablish source priority rules before conflicts arise. Context clash differs from poisoning — multiple pieces of information are individually correct but mutually contradictory (version conflicts, perspective differences, multi-source retrieval with divergent facts).\n\nImplement version filtering to exclude outdated information before it enters context. When contradictions are unavoidable, mark them explicitly with structured conflict annotations: state what conflicts, which source each claim comes from, and which source takes precedence. Without explicit priority rules, models resolve contradictions unpredictably.\n\n### Empirical Benchmarks and Thresholds\n\nUse these benchmarks to set design constraints — not as universal truths. RULER-style evidence shows advertised long-context support does not guarantee satisfactory task performance at that length (claim-context-degradation-lost-middle-ruler). Near-perfect needle-in-haystack scores do not predict real-world long-context performance.\n\n**Model-Specific Degradation Thresholds**\n\nDegradation onset varies significantly by model family and task type. As a general rule, expect degradation to begin at 60-70% of the advertised context window for complex retrieval tasks (RULER benchmark found only 50% of models claiming 32K+ context maintain satisfactory performance at that length). Key patterns:\n\n- **Models with extended thinking** reduce hallucination through step-by-step verification but at higher latency and token cost\n- **Models optimized for agents/coding** tend to have better attention management for tool-output-heavy contexts\n- **Models with very large context windows (1M+)** handle more raw context but still follow U-shaped degradation curves — bigger windows do not eliminate the problem, they delay it\n\nAlways benchmark degradation thresholds with your specific workload rather than relying on published benchmarks. Model-specific thresholds go stale with each model update (see Gotcha 2).\n\n### Counterintuitive Findings\n\nAccount for these research-backed surprises when designing context strategies:\n\n**Shuffled context can outperform coherent context.** Studies found incoherent (shuffled) haystacks can outperform logically ordered ones for some retrieval tasks (claim-context-degradation-distractor-shuffled). Coherent context may create false associations that confuse retrieval; incoherent context can force exact matching. Do not assume that better-organized context always yields better results — test both arrangements.\n\n**Single distractors have outsized impact.** The performance hit from one irrelevant document is disproportionately large compared to adding more distractors after the first. Treat distractor prevention as binary: either keep context clean or accept significant degradation.\n\n**Low needle-question similarity accelerates degradation.** Tasks requiring inference across dissimilar content degrade faster with context length than tasks with high surface-level similarity. Design retrieval to maximize semantic overlap between queries and retrieved content.\n\n### When Larger Contexts Hurt\n\nDo not assume larger context windows improve performance. Performance remains stable up to a model-specific threshold, then degrades rapidly — the curve is non-linear with a cliff edge, not a gentle slope. For many models, meaningful degradation begins at 8K-16K tokens even when windows support much larger sizes.\n\nFactor in cost: processing a 400K token context costs exponentially more than 200K in both time and compute, not linearly more. For many applications, this makes large-context processing economically impractical.\n\nRecognize the cognitive bottleneck: even with infinite context, asking a single model to maintain quality across dozens of independent tasks creates degradation that more context cannot solve. Split tasks across sub-agents instead of expanding context.\n\n## Practical Guidance\n\n### The Four-Bucket Mitigation Framework\n\nApply these four strategies based on which degradation pattern is active:\n\n**Write** — Save context outside the window using scratchpads, file systems, or external storage. Use when context utilization exceeds 70% of the window. This keeps active context lean while preserving information access through tool calls.\n\n**Select** — Pull only relevant context into the window through retrieval, filtering, and prioritization. Use when distraction or confusion symptoms appear. Apply relevance scoring before loading; exclude anything below threshold rather than including everything available.\n\n**Compress** — Reduce tokens while preserving information through summarization, abstraction, and observation masking. Use when context is growing but all content is relevant. Replace verbose tool outputs with compact structured summaries; abstract repeated patterns into single references.\n\n**Isolate** — Split context across sub-agents or sessions to prevent any single context from growing past its degradation threshold. Use when confusion or clash symptoms appear, or when tasks are independent. This is the most aggressive strategy but often the most effective for complex multi-task systems.\n\n### Architectural Patterns for Resilience\n\nImplement just-in-time context loading: retrieve information only when the current reasoning step needs it, not preemptively. Use observation masking to replace verbose tool outputs with compact references after processing. Deploy sub-agent architectures where each agent holds only task-relevant context. Trigger compaction before context exceeds the model-specific degradation onset threshold — not after symptoms appear.\n\n## Examples\n\n**Example 1: Detecting Degradation**\n```yaml\n# Context grows during long conversation\nturn_1: 1000 tokens\nturn_5: 8000 tokens\nturn_10: 25000 tokens\nturn_20: 60000 tokens (degradation begins)\nturn_30: 90000 tokens (significant degradation)\n```\n\n**Example 2: Mitigating Lost-in-Middle**\n```markdown\n# Organize context with critical info at edges\n\n[CURRENT TASK] # At start\n- Goal: Generate quarterly report\n- Deadline: End of week\n\n[DETAILED CONTEXT] # Middle (less attention)\n- 50 pages of data\n- Multiple analysis sections\n- Supporting evidence\n\n[KEY FINDINGS] # At end\n- Revenue up 15%\n- Costs down 8%\n- Growth in Region A\n```\n\n**Example 3: Context poisoning circuit breaker**\n```text\nsymptom: agent keeps citing an incorrect retrieved claim after correction\ndiagnosis: poisoned context, not a missing-instruction problem\naction:\n 1. identify first turn where the bad claim entered\n 2. truncate or restart from before that point\n 3. reload only verified sources\n 4. record the rejected claim and source provenance\n```\n\n**Example 4: Context clash annotation**\n```yaml\nconflict:\n topic: billing API version\n source_a: docs/v1.md says endpoint is /charges\n source_b: docs/v2.md says endpoint is /payments\n precedence: docs/v2.md\n reason: current production version\n```\n\n## Guidelines\n\n1. Monitor context length and performance correlation during development\n2. Place critical information at beginning or end of context\n3. Implement compaction triggers before degradation becomes severe\n4. Validate retrieved documents for accuracy before adding to context\n5. Use versioning to prevent outdated information from causing clash\n6. Segment tasks to prevent context confusion across different objectives\n7. Design for graceful degradation rather than assuming perfect conditions\n8. Test with progressively larger contexts to find degradation thresholds\n\n## Gotchas\n\n1. **Normal variance looks like degradation**: Model output quality fluctuates naturally across runs. Do not diagnose degradation from a single drop in quality — establish a baseline over multiple runs and look for sustained, correlated decline tied to context growth. A 5-10% quality dip on one run is noise; the same dip consistently appearing after 40K tokens is signal.\n\n2. **Model-specific thresholds go stale**: The degradation onset values in benchmark tables reflect specific model versions. Provider updates, fine-tuning changes, and infrastructure shifts can move thresholds by 20-50% in either direction. Re-benchmark quarterly and after any major model update rather than treating published thresholds as permanent.\n\n3. **Needle-in-haystack scores create false confidence**: A model scoring 99% on needle-in-haystack does not mean it handles 128K tokens well in production. Needle tests measure single-fact retrieval from passive context — real workloads require multi-fact reasoning, instruction following, and synthesis across the full window. Use task-specific benchmarks that mirror actual workload patterns.\n\n4. **Contradictory retrieved documents poison silently**: When a RAG pipeline retrieves two documents that disagree on a fact, the model may silently pick one without signaling the conflict. This looks like a correct response but is effectively random. Implement contradiction detection in the retrieval layer before documents enter context.\n\n5. **Prompt quality problems masquerade as degradation**: Poor prompt structure (ambiguous instructions, missing constraints, unclear task framing) produces symptoms identical to context degradation — inconsistent outputs, ignored instructions, wrong tool usage. Before diagnosing degradation, verify the same prompt works correctly at low context lengths. If it fails at 2K tokens, the problem is the prompt, not the context.\n\n6. **Degradation is non-linear with a cliff edge**: Performance does not degrade gradually — it holds steady until a model-specific threshold, then drops sharply. Systems designed for \"graceful degradation\" often miss this pattern because monitoring checks assume linear decline. Set compaction triggers well before the cliff (at 70% of known onset), not at the onset itself.\n\n7. **Over-organizing context can backfire**: Intuitively, well-structured and coherent context should outperform disorganized content. Research shows shuffled haystacks sometimes outperform coherent ones for retrieval tasks because coherent context creates false associations. Test whether heavy structural formatting actually helps for the specific task — do not assume it does.\n\n## Integration\n\nThis skill owns diagnosis and mitigation of active context failures. Adjacent skills own the implementation tactics once the failure is identified:\n\n- `context-fundamentals`: conceptual explanation of attention and context windows before a failure exists.\n- `context-optimization`: masking, caching, partitioning, and other token-efficiency tactics after diagnosis.\n- `context-compression`: structured summaries and handoffs when accumulated context must be compacted.\n- `filesystem-context`: offloading raw outputs, logs, and scratch state so poisoned or bulky context can be inspected without staying in the prompt.\n- `multi-agent-patterns`: isolating tasks into separate contexts to prevent confusion and clash.\n- `evaluation`: degradation tests and production monitoring.\n\n## References\n\nInternal reference:\n- [Degradation Patterns Reference](./references/patterns.md) - Read when: debugging a specific degradation pattern and needing implementation-level detection code (attention analysis, poisoning tracking, relevance scoring, recovery procedures)\n\nRelated skills in this collection:\n- context-fundamentals - Read when: lacking foundational understanding of context windows, token budgets, or placement mechanics\n- context-optimization - Read when: degradation is diagnosed and specific mitigation techniques (compaction, compression, masking) are needed\n- evaluation - Read when: setting up production monitoring to detect degradation before it impacts users\n\nExternal resources:\n- Liu et al., 2023 \"Lost in the Middle\" - Read when: needing primary research backing for U-shaped attention claims or designing position-aware context layouts\n- RULER benchmark documentation - Read when: evaluating model claims about long-context support or comparing models for context-heavy workloads\n- Production engineering guides from AI labs - Read when: implementing context management in production infrastructure\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-20\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.1.0\n","frontmatter":{"name":"context-degradation","description":"This skill should be used for diagnosing and mitigating context degradation: lost-in-middle failures, context poisoning, context clash, context confusion, attention-pattern issues, and agent performance degradation caused by accumulated or conflicting context."},"isInternal":false,"tokens":3489,"sizeBytes":19180},{"name":"context-components.md","path":"skills/context-fundamentals/references/context-components.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/context-fundamentals/references/context-components.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Context Components: Technical Reference\n\nThis document provides detailed technical reference for each context component in agent systems.\n\n## System Prompt Engineering\n\n### Section Structure\n\nOrganize system prompts into distinct sections with clear boundaries. A recommended structure:\n\n```\n\nContext about the domain, user preferences, or project-specific details\n\n\n\nCore behavioral guidelines and task instructions\n\n\n\nWhen and how to use available tools\n\n\n\nExpected output format and quality standards\n\n```\n\nThis structure allows agents to locate relevant information quickly and enables selective context loading in advanced implementations.\n\n### Altitude Calibration\n\nThe \"altitude\" of instructions refers to the level of abstraction. Consider these examples:\n\n**Too Low (Brittle):**\n```\nIf the user asks about pricing, check the pricing table in docs/pricing.md.\nIf the table shows USD, convert to EUR using the exchange rate in\nconfig/exchange_rates.json. If the user is in the EU, add VAT at the\napplicable rate from config/vat_rates.json. Format the response with\nthe currency symbol, two decimal places, and a note about VAT.\n```\n\n**Too High (Vague):**\n```\nHelp users with pricing questions. Be helpful and accurate.\n```\n\n**Optimal (Heuristic-Driven):**\n```\nFor pricing inquiries:\n1. Retrieve current rates from docs/pricing.md\n2. Apply user location adjustments (see config/location_defaults.json)\n3. Format with appropriate currency and tax considerations\n\nPrefer exact figures over estimates. When rates are unavailable,\nsay so explicitly rather than projecting.\n```\n\nThe optimal altitude provides clear steps while allowing flexibility in execution.\n\n## Tool Definition Specification\n\n### Schema Structure\n\nEach tool should define:\n\n```python\n{\n \"name\": \"tool_function_name\",\n \"description\": \"Clear description of what the tool does and when to use it\",\n \"parameters\": {\n \"type\": \"object\",\n \"properties\": {\n \"param_name\": {\n \"type\": \"string\",\n \"description\": \"What this parameter controls\",\n \"default\": \"reasonable_default_value\"\n }\n },\n \"required\": [\"param_name\"]\n },\n \"returns\": {\n \"type\": \"object\",\n \"description\": \"What the tool returns and its structure\"\n }\n}\n```\n\n### Description Engineering\n\nTool descriptions should answer: what the tool does, when to use it, and what it produces. Include usage context, examples, and edge cases.\n\n**Weak Description:**\n```\nSearch the database for customer information.\n```\n\n**Strong Description:**\n```\nRetrieve customer information by ID or email.\n\nUse when:\n- User asks about a specific customer's details, history, or status\n- User provides a customer identifier and needs related information\n\nReturns customer object with:\n- Basic info (name, email, account status)\n- Order history summary\n- Support ticket count\n\nReturns null if customer not found. Returns error if database unreachable.\n```\n\n## Retrieved Document Management\n\n### Identifier Design\n\nDesign identifiers that convey meaning and enable efficient retrieval:\n\n**Poor identifiers:**\n- `data/file1.json`\n- `ref/ref.md`\n- `2024/q3/report`\n\n**Strong identifiers:**\n- `customer_pricing_rates.json`\n- `engineering_onboarding_checklist.md`\n- `2024_q3_revenue_report.pdf`\n\nStrong identifiers allow agents to locate relevant files even without search tools.\n\n### Document Chunking Strategy\n\nFor large documents, chunk strategically to preserve semantic coherence:\n\n```python\n# Pseudocode for semantic chunking\ndef chunk_document(content):\n \"\"\"Split document at natural semantic boundaries.\"\"\"\n boundaries = find_section_headers(content)\n boundaries += find_paragraph_breaks(content)\n boundaries += find_logical_breaks(content)\n \n chunks = []\n for i in range(len(boundaries) - 1):\n chunk = content[boundaries[i]:boundaries[i+1]]\n if len(chunk) > MIN_CHUNK_SIZE and len(chunk) < MAX_CHUNK_SIZE:\n chunks.append(chunk)\n \n return chunks\n```\n\nAvoid arbitrary character limits that split mid-sentence or mid-concept.\n\n## Message History Management\n\n### Turn Representation\n\nStructure message history to preserve key information:\n\n```python\n{\n \"role\": \"user\" | \"assistant\" | \"tool\",\n \"content\": \"message text\",\n \"reasoning\": \"optional chain-of-thought\",\n \"tool_calls\": [list if role=\"assistant\"],\n \"tool_output\": \"output if role=\"tool\"\",\n \"summary\": \"compact summary if conversation is long\"\n}\n```\n\n### Summary Injection Pattern\n\nFor long conversations, inject summaries at intervals:\n\n```python\ndef inject_summaries(messages, summary_interval=20):\n \"\"\"Inject summaries at regular intervals to preserve context.\"\"\"\n summarized = []\n for i, msg in enumerate(messages):\n summarized.append(msg)\n if i > 0 and i % summary_interval == 0:\n summary = generate_summary(summarized[-summary_interval:])\n summarized.append({\n \"role\": \"system\",\n \"content\": f\"Conversation summary: {summary}\",\n \"is_summary\": True\n })\n return summarized\n```\n\n## Tool Output Optimization\n\n### Response Formats\n\nProvide response format options to control token usage:\n\n```python\ndef get_customer_response_format():\n return {\n \"format\": \"concise | detailed\",\n \"fields\": [\"id\", \"name\", \"email\", \"status\", \"history_summary\"]\n }\n```\n\nThe concise format returns essential fields only; detailed returns complete objects.\n\n### Observation Masking\n\nFor verbose tool outputs, consider masking patterns:\n\n```python\ndef mask_observation(output, max_length=500):\n \"\"\"Replace long observations with compact references.\"\"\"\n if len(output) <= max_length:\n return output\n \n reference_id = store_observation(output)\n return f\"[Previous observation elided. Full content stored at reference {reference_id}]\"\n```\n\nThis preserves information access while reducing token usage.\n\n## Context Budget Estimation\n\n### Token Counting Approximation\n\nFor planning purposes, estimate tokens at approximately 4 characters per token for English text:\n\n```\n1000 words ≈ 7500 characters ≈ 1800-2000 tokens\n```\n\nThis is a rough approximation; actual tokenization varies by model and content type.\n\n### Context Budget Allocation\n\nAllocate context budget across components:\n\n| Component | Typical Range | Notes |\n|-----------|---------------|-------|\n| System prompt | 500-2000 tokens | Stable across session |\n| Tool definitions | 100-500 per tool | Grows with tool count |\n| Retrieved documents | Variable | Often largest consumer |\n| Message history | Variable | Grows with conversation |\n| Tool outputs | Variable | Can dominate context |\n\nMonitor actual usage during development to establish baseline allocations.\n\n## Progressive Disclosure Implementation\n\n### Skill Activation Pattern\n\n```python\ndef activate_skill_context(skill_name, task_description):\n \"\"\"Load skill context when task matches skill description.\"\"\"\n skill_metadata = load_all_skill_metadata()\n \n relevant_skills = []\n for skill in skill_metadata:\n if skill_matches_task(skill, task_description):\n relevant_skills.append(skill)\n \n # Load full content only for most relevant skills\n for skill in relevant_skills[:MAX_CONCURRENT_SKILLS]:\n skill_context = load_skill_content(skill)\n inject_into_context(skill_context)\n```\n\n### Reference Loading Pattern\n\n```python\ndef get_reference(file_reference):\n \"\"\"Load reference file only when explicitly needed.\"\"\"\n if not file_reference.is_loaded:\n file_reference.content = read_file(file_reference.path)\n file_reference.is_loaded = True\n return file_reference.content\n```\n\nThis pattern ensures files are loaded once and cached for the session.\n\n","isInternal":false,"tokens":1607,"sizeBytes":7970},{"name":"SKILL.md","path":"skills/context-fundamentals/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/context-fundamentals/SKILL.md","title":"context-fundamentals","category":"anthropic-skill","format":"markdown","content":"---\nname: context-fundamentals\ndescription: \"This skill should be used to explain or reason about the foundational concepts of context engineering: what context is, the anatomy of a context window, how attention mechanics work, the U-shaped attention curve, why context quality matters more than quantity, and the mental models needed to interpret every other context-engineering decision. Use this for conceptual explanation, onboarding, and background reading. Route operational work to the specialized skills: debugging attention failures goes to context-degradation, token-efficiency work goes to context-optimization, conversation summarization goes to context-compression, and project-shape decisions go to project-development.\"\n---\n\n# Context Engineering Fundamentals\n\nContext is the complete state available to a language model at inference time: system instructions, tool definitions, retrieved documents, message history, and tool outputs. Context engineering is the discipline of curating the smallest high-signal token set that maximizes the likelihood of desired outcomes.\n\nThis skill is the conceptual foundation that every other skill in the collection builds on. It explains what context is, how attention mechanics work, why context quality matters more than quantity, and the mental models needed to interpret every other context-engineering decision. It does not own operational work: debugging attention failures belongs to `context-degradation`, token-efficiency tactics belong to `context-optimization`, conversation summarization belongs to `context-compression`, file-based offloading belongs to `filesystem-context`, and project-shape decisions belong to `project-development`.\n\n## When to Activate\n\nActivate this skill when the work is conceptual:\n\n- Explaining what context is and how attention mechanics constrain agent behavior.\n- Onboarding new contributors who need the mental models before diving into operational skills.\n- Reasoning about a context-related design decision from first principles (what does this constraint mean, why does this trade-off exist) before picking a specific tactic.\n- Writing or reviewing documentation that needs to ground operational guidance in the underlying mechanics.\n\nDo not activate this skill for operational work. The specialized skills handle the doing:\n\n- Diagnosing lost-in-middle, context poisoning, or attention failures: `context-degradation`.\n- Reducing token cost via masking, partitioning, prefix caching, budgets: `context-optimization`.\n- Compressing a long session into a handoff summary: `context-compression`.\n- Offloading large tool outputs or maintaining a durable scratchpad: `filesystem-context`.\n- Deciding the shape of an LLM project or pipeline: `project-development`.\n\n## Core Concepts\n\nTreat context as a finite attention budget, not a storage bin. Every token added competes for the model's attention and depletes a budget that cannot be refilled mid-inference. The engineering problem is maximizing utility per token against three constraints: the hard token limit, the softer effective-capacity ceiling, and the U-shaped attention curve that penalizes information placed in the middle of context (claim-context-degradation-lost-middle-ruler).\n\nApply four principles when assembling context:\n\n1. **Informativity over exhaustiveness** — include only what matters for the current decision; design systems that can retrieve additional information on demand.\n2. **Position-aware placement** — place critical constraints at the beginning and end of context because long-context evaluations show middle-position information is less reliably recovered than edge-position information (claim-context-degradation-lost-middle-ruler).\n3. **Progressive disclosure** — load skill names and summaries at startup; load full content only when a skill activates for a specific task.\n4. **Iterative curation** — context engineering is not a one-time prompt-writing exercise but an ongoing discipline applied every time content is passed to the model.\n\n## Detailed Topics\n\n### The Anatomy of Context\n\n**System Prompts**\nOrganize system prompts into distinct sections using XML tags or Markdown headers (background, instructions, tool guidance, output format). System prompts persist throughout the conversation, so place the most critical constraints at the beginning and end where attention is strongest.\n\nCalibrate instruction altitude to balance two failure modes. Too-low altitude hardcodes brittle logic that breaks when conditions shift. Too-high altitude provides vague guidance that fails to give concrete signals for desired behavior. Aim for heuristic-driven instructions: specific enough to guide behavior, flexible enough to generalize — for example, numbered steps with room for judgment at each step.\n\nStart minimal, then add instructions reactively based on observed failure modes rather than preemptively stuffing edge cases. Curate diverse, canonical few-shot examples that portray expected behavior instead of listing every possible scenario.\n\n**Tool Definitions**\nWrite tool descriptions that answer three questions: what the tool does, when to use it, and what it returns. Include usage context, parameter defaults, and error cases — agents cannot disambiguate tools that a human engineer cannot disambiguate either.\n\nKeep the tool set minimal. Consolidate overlapping tools because bloated tool sets create ambiguous decision points and consume disproportionate context after JSON serialization (tool schemas typically inflate 2-3x compared to equivalent plain-text descriptions).\n\n**Retrieved Documents**\nMaintain lightweight identifiers (file paths, stored queries, web links) and load data into context dynamically using just-in-time retrieval. This mirrors human cognition — maintain an index, not a copy. Strong identifiers (e.g., `customer_pricing_rates.json`) let agents locate relevant files even without search tools; weak identifiers (e.g., `data/file1.json`) force unnecessary loads.\n\nWhen chunking large documents, split at natural semantic boundaries (section headers, paragraph breaks) rather than arbitrary character limits that sever mid-concept.\n\n**Message History**\nMessage history serves as the agent's scratchpad memory for tracking progress, maintaining task state, and preserving reasoning across turns. For long-running tasks, it can grow to dominate context usage — monitor and apply compaction before it crowds out active instructions.\n\nCyclically refine history: once a tool has been called deep in the conversation, the raw result rarely needs to remain verbatim. Replace stale tool outputs with compact summaries or references to reduce low-signal bulk.\n\n**Tool Outputs**\nTool outputs often dominate context in agent trajectories (claim-context-optimization-tool-output-dominance). Apply observation masking: replace verbose outputs with compact references once the agent has processed the result. Retain only the most recently relevant file contents; compress or evict older ones.\n\n### Context Windows and Attention Mechanics\n\n**The Attention Budget**\nFor n tokens, the attention mechanism computes n-squared pairwise relationships. As context grows, the model's ability to maintain these relationships degrades — not as a hard cliff but as a performance gradient. Models trained predominantly on shorter sequences have fewer specialized parameters for context-wide dependencies, creating an effective ceiling well below the nominal window size.\n\nDesign for this gradient: assume effective capacity is materially below the advertised window until measured on the target workload. Large nominal context windows do not remove the need for task-specific degradation tests (claim-context-degradation-lost-middle-ruler).\n\n**Position Encoding Limits**\nPosition encoding interpolation extends sequence handling beyond training lengths but introduces degradation in positional precision. Expect reduced accuracy for information retrieval and long-range reasoning at extended contexts compared to performance on shorter inputs.\n\n**Progressive Disclosure in Practice**\nImplement progressive disclosure at three levels:\n\n1. **Skill selection** — load only names and descriptions at startup; activate full skill content on demand.\n2. **Document loading** — load summaries first; fetch detail sections only when the task requires them.\n3. **Tool result retention** — keep recent results in full; compress or evict older results.\n\nKeep the boundary crisp: if a skill or document is activated, load it fully rather than partially — partial loads create confusing gaps that degrade reasoning quality.\n\n### Context Quality Versus Quantity\n\nReject the assumption that larger context windows solve memory problems. Processing cost grows disproportionately with context length — not just linear cost scaling, but degraded model performance beyond effective capacity thresholds. Long inputs remain expensive even with prefix caching.\n\nApply the signal-density test: for each piece of context, ask whether removing it would change the model's output. If not, remove it. Redundant content does not merely waste tokens — it actively dilutes attention from high-signal content.\n\n## Practical Guidance\n\nThis section provides conceptual application advice. Pointers to operational skills are explicit.\n\n### Reasoning About a Context Decision\n\nWhen a context-related design decision needs to be made, separate the conceptual question from the operational one. The conceptual question is \"what does this mean and why does it matter\"; the operational question is \"what specific technique do we apply.\" Use this skill to answer the first; route to the specialized skill that owns the second.\n\nFor example, deciding whether to summarize a long agent session has two parts: (1) why summarization is needed at all (attention budget is finite, U-shaped curve degrades middle content, signal density matters more than volume - this skill) and (2) what compression strategy preserves the right state and at what utilization threshold to trigger it (`context-compression`).\n\n### Reading Order For New Contributors\n\nA contributor coming to context engineering for the first time should read:\n\n1. This skill, to internalize the attention-budget framing and the U-shaped curve.\n2. `context-degradation`, to see what context failures look like in practice and how to diagnose them.\n3. Two or three of `context-optimization`, `context-compression`, `filesystem-context`, `memory-systems` depending on which operational concern is most relevant to their project.\n\nSkipping step 1 produces operators who apply techniques without understanding why; skipping the operational skills produces theorists who do not know which technique fits which failure mode.\n\n## Examples\n\n**Example 1: Organizing System Prompts**\n\nIllustrates the conceptual point that critical constraints belong at attention-favored positions (beginning and end), and that explicit section boundaries help the model parse the prompt:\n\n```markdown\n\nYou are a Python expert helping a development team.\nCurrent project: Data processing pipeline in Python 3.9+\n\n\n\n- Write clean, idiomatic Python code\n- Include type hints for function signatures\n- Add docstrings for public functions\n- Follow PEP 8 style guidelines\n\n\n\nProvide code blocks with syntax highlighting.\nExplain non-obvious decisions in comments.\n\n```\n\n**Example 2: The Attention Budget As A Mental Model**\n\nA large-context model does not have an equally attended context. Effective capacity is workload-specific, and the U-shaped curve penalizes information placed in the middle. When deciding how much of an upstream knowledge base to load, this is the mental model: do not ask \"will it fit,\" ask \"will the model still attend to the parts that matter.\"\n\nThe corresponding operational question (which technique should reduce the load) belongs to `context-optimization`.\n\n## Guidelines\n\n1. Treat context as a finite resource with diminishing returns\n2. Place critical information at attention-favored positions (beginning and end)\n3. Use progressive disclosure to defer loading until needed\n4. Organize system prompts with clear section boundaries\n5. Monitor context usage during development\n6. Implement compaction triggers at 70-80% utilization\n7. Design for context degradation rather than hoping to avoid it\n8. Prefer smaller high-signal context over larger low-signal context\n\n## Gotchas\n\n1. **Nominal window is not effective capacity**: A model advertising a large context window may degrade well before that limit on complex retrieval or reasoning tasks. Budget below the nominal window until your own degradation tests prove otherwise.\n\n2. **Character-based token estimates silently drift**: The ~4 characters/token heuristic for English prose breaks down for code (2-3 chars/token), URLs and file paths (each slash, dot, and colon is a separate token), and non-English text (often 1-2 chars/token). Use the provider's actual tokenizer (e.g., tiktoken for OpenAI models, Anthropic's token counting API) for any budget-critical calculation.\n\n3. **Tool schemas inflate 2-3x after JSON serialization**: A tool definition that looks compact in source code expands significantly when serialized — brackets, quotes, colons, and commas each consume tokens. Ten tools with moderate schemas can consume 5,000-8,000 tokens before a single message is sent. Audit serialized tool token counts, not source-code line counts.\n\n4. **Message history balloons silently in agentic loops**: Each tool call adds both the request and the full response to history. After 20-30 iterations, history can consume 70-80% of the window while the agent shows no visible symptoms until reasoning quality collapses. Set a hard token ceiling on history and trigger compaction proactively.\n\n5. **Critical instructions in the middle get lost**: The U-shaped attention curve means the middle of context receives 10-40% less recall accuracy than the beginning and end. Never place safety constraints, output format requirements, or behavioral guardrails in the middle of a long system prompt — anchor them at the top or bottom.\n\n6. **Progressive disclosure that loads too eagerly defeats its purpose**: Loading every \"potentially relevant\" skill or document at the first hint of relevance recreates the context-stuffing problem. Set strict activation thresholds — a skill should load only when the task explicitly matches its trigger conditions, not when the topic is merely adjacent.\n\n7. **Mixing instruction altitudes causes inconsistent behavior**: Combining hyper-specific rules (\"always use exactly 3 bullet points\") with vague directives (\"be helpful\") in the same prompt creates conflicting signals. Group instructions by altitude level and keep each section internally consistent — either heuristic-driven or prescriptive, not both interleaved.\n\n## Integration\n\nThis skill is the conceptual foundation. It does not own operational work; it provides the mental models the operational skills assume.\n\nRouting map for operational work:\n\n- `context-degradation`: diagnosing attention failures, lost-in-middle, poisoning, distraction.\n- `context-optimization`: token-efficiency tactics (masking, partitioning, caching, budgets).\n- `context-compression`: compacting long sessions while preserving decisions, files, risks.\n- `filesystem-context`: offloading large outputs and using files as a durable scratchpad.\n- `memory-systems`: cross-session memory architectures with entity tracking.\n- `multi-agent-patterns`: when to split work across agents for context isolation.\n- `tool-design`: writing tool descriptions and schemas that route correctly.\n- `project-development`: deciding LLM fit and shaping multi-stage pipelines.\n\nRead this skill first to build the mental models; read the operational skill that fits the task when actually doing the work.\n\n## References\n\nInternal reference:\n- [Context Components Reference](./references/context-components.md) - Read when: debugging a specific context component (system prompts, tool definitions, message history, tool outputs) or implementing chunking, observation masking, or budget allocation tables\n\nRelated skills in this collection:\n- context-degradation - Read when: agent performance drops as conversations grow or context fills beyond 60% capacity\n- context-optimization - Read when: token costs are too high or compaction/compression strategies are needed\n\nExternal resources:\n- Anthropic's \"Effective Context Engineering for AI Agents\" — production patterns for compaction, sub-agents, and hybrid retrieval\n- Research on transformer attention mechanisms and the lost-in-the-middle effect\n- Tokenomics research on agentic software engineering token distribution\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-20\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.2.0\n","frontmatter":{"name":"context-fundamentals","description":"This skill should be used to explain or reason about the foundational concepts of context engineering: what context is, the anatomy of a context window, how attention mechanics work, the U-shaped attention curve, why context quality matters more than quantity, and the mental models needed to interpret every other context-engineering decision. Use this for conceptual explanation, onboarding, and background reading. Route operational work to the specialized skills: debugging attention failures goes to context-degradation, token-efficiency work goes to context-optimization, conversation summarization goes to context-compression, and project-shape decisions go to project-development."},"isInternal":false,"tokens":3187,"sizeBytes":17013},{"name":"optimization_techniques.md","path":"skills/context-optimization/references/optimization_techniques.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/context-optimization/references/optimization_techniques.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Context Optimization Reference\n\nThis document provides detailed technical reference for context optimization techniques and strategies.\n\n## Compaction Strategies\n\n### Summary-Based Compaction\n\nSummary-based compaction replaces verbose content with concise summaries while preserving key information. The approach works by identifying sections that can be compressed, generating summaries that capture essential points, and replacing full content with summaries.\n\nThe effectiveness of compaction depends on what information is preserved. Critical decisions, user preferences, and current task state should never be compacted. Intermediate results and supporting evidence can be summarized more aggressively. Boilerplate, repeated information, and exploratory reasoning can often be removed entirely.\n\n### Token Budget Allocation\n\nEffective context budgeting requires understanding how different context components consume tokens and allocating budget strategically:\n\n| Component | Typical Range | Notes |\n|-----------|---------------|-------|\n| System prompt | 500-2000 tokens | Stable across session |\n| Tool definitions | 100-500 per tool | Grows with tool count |\n| Retrieved documents | Variable | Often largest consumer |\n| Message history | Variable | Grows with conversation |\n| Tool outputs | Variable | Can dominate context |\n\n### Compaction Thresholds\n\nTrigger compaction at appropriate thresholds to maintain performance:\n\n- Warning threshold at 70% of effective context limit\n- Compaction trigger at 80% of effective context limit\n- Aggressive compaction at 90% of effective context limit\n\nThe exact thresholds depend on model behavior and task characteristics. Some models show graceful degradation while others exhibit sharp performance cliffs.\n\n## Observation Masking Patterns\n\n### Selective Masking\n\nNot all observations should be masked equally. Consider masking observations that have served their purpose and are no longer needed for active reasoning. Keep observations that are central to the current task. Keep observations from the most recent turn. Keep observations that may be referenced again.\n\n### Masking Implementation\n\n```python\ndef selective_mask(observations: List[Dict], current_task: Dict) -> List[Dict]:\n \"\"\"\n Selectively mask observations based on relevance.\n \n Returns observations with mask field indicating masked content.\n \"\"\"\n masked = []\n \n for obs in observations:\n relevance = calculate_relevance(obs, current_task)\n \n if relevance < 0.3 and obs[\"age\"] > 3:\n # Low relevance and old - mask\n masked.append({\n **obs,\n \"masked\": True,\n \"reference\": store_for_reference(obs[\"content\"]),\n \"summary\": summarize_content(obs[\"content\"])\n })\n else:\n masked.append({\n **obs,\n \"masked\": False\n })\n \n return masked\n```\n\n## KV-Cache Optimization\n\n### Prefix Stability\n\nKV-cache hit rates depend on prefix stability. Stable prefixes enable cache reuse across requests. Dynamic prefixes invalidate cache and force recomputation.\n\nElements that should remain stable include system prompts, tool definitions, and frequently used templates. Elements that may vary include timestamps, session identifiers, and query-specific content.\n\n### Cache-Friendly Design\n\nDesign prompts to maximize cache hit rates:\n\n1. Place stable content at the beginning\n2. Use consistent formatting across requests\n3. Avoid dynamic content in prompts when possible\n4. Use placeholders for dynamic content\n\n```python\n# Cache-unfriendly: Dynamic timestamp in prompt\nsystem_prompt = f\"\"\"\nCurrent time: {datetime.now().isoformat()}\nYou are a helpful assistant.\n\"\"\"\n\n# Cache-friendly: Stable prompt with dynamic time as variable\nsystem_prompt = \"\"\"\nYou are a helpful assistant.\nCurrent time is provided separately when relevant.\n\"\"\"\n```\n\n## Context Partitioning Strategies\n\n### Sub-Agent Isolation\n\nPartition work across sub-agents to prevent any single context from growing too large. Each sub-agent operates with a clean context focused on its subtask.\n\n### Partition Planning\n\n```python\ndef plan_partitioning(task: Dict, context_limit: int) -> Dict:\n \"\"\"\n Plan how to partition a task based on context limits.\n \n Returns partitioning strategy and subtask definitions.\n \"\"\"\n estimated_context = estimate_task_context(task)\n \n if estimated_context <= context_limit:\n return {\n \"strategy\": \"single_agent\",\n \"subtasks\": [task]\n }\n \n # Plan multi-agent approach\n subtasks = decompose_task(task)\n \n return {\n \"strategy\": \"multi_agent\",\n \"subtasks\": subtasks,\n \"coordination\": \"hierarchical\"\n }\n```\n\n## Optimization Decision Framework\n\n### When to Optimize\n\nConsider context optimization when context utilization exceeds 70%, when response quality degrades as conversations extend, when costs increase due to long contexts, or when latency increases with conversation length.\n\n### What Optimization to Apply\n\nChoose optimization strategies based on context composition:\n\nIf tool outputs dominate context, apply observation masking. If retrieved documents dominate context, apply summarization or partitioning. If message history dominates context, apply compaction with summarization. If multiple components contribute, combine strategies.\n\n### Evaluation of Optimization\n\nAfter applying optimization, evaluate effectiveness:\n\n- Measure token reduction achieved\n- Measure quality preservation (output quality should not degrade)\n- Measure latency improvement\n- Measure cost reduction\n\nIterate on optimization strategies based on evaluation results.\n\n## Common Pitfalls\n\n### Over-Aggressive Compaction\n\nCompacting too aggressively can remove critical information. Always preserve task goals, user preferences, and recent conversation context. Test compaction at increasing aggressiveness levels to find the optimal balance.\n\n### Masking Critical Observations\n\nMasking observations that are still needed can cause errors. Track observation usage and only mask content that is no longer referenced. Consider keeping references to masked content that could be retrieved if needed.\n\n### Ignoring Attention Distribution\n\nThe lost-in-middle phenomenon means that information placement matters. Place critical information at attention-favored positions (beginning and end of context). Use explicit markers to highlight important content.\n\n### Premature Optimization\n\nNot all contexts require optimization. Adding optimization machinery has overhead. Optimize only when context limits actually constrain agent performance.\n\n## Monitoring and Alerting\n\n### Key Metrics\n\nTrack these metrics to understand optimization needs:\n\n- Context token count over time\n- Cache hit rates for repeated patterns\n- Response quality metrics by context size\n- Cost per conversation by context length\n- Latency by context size\n\n### Alert Thresholds\n\nSet alerts for:\n\n- Context utilization above 80%\n- Cache hit rate below 50%\n- Quality score drop of more than 10%\n- Cost increase above baseline\n\n## Integration Patterns\n\n### Integration with Agent Framework\n\nIntegrate optimization into agent workflow:\n\n```python\nclass OptimizingAgent:\n def __init__(self, context_limit: int = 80000):\n self.context_limit = context_limit\n self.optimizer = ContextOptimizer()\n \n def process(self, user_input: str, context: Dict) -> Dict:\n # Check if optimization needed\n if self.optimizer.should_compact(context):\n context = self.optimizer.compact(context)\n \n # Process with optimized context\n response = self._call_model(user_input, context)\n \n # Track metrics\n self.optimizer.record_metrics(context, response)\n \n return response\n```\n\n### Integration with Memory Systems\n\nConnect optimization with memory systems:\n\n```python\nclass MemoryAwareOptimizer:\n def __init__(self, memory_system, context_limit: int):\n self.memory = memory_system\n self.limit = context_limit\n \n def optimize_context(self, current_context: Dict, task: str) -> Dict:\n # Check if information is in memory\n relevant_memories = self.memory.retrieve(task)\n \n # Move information to memory if not needed in context\n for mem in relevant_memories:\n if mem[\"importance\"] < threshold:\n current_context = remove_from_context(current_context, mem)\n # Keep reference that memory can be retrieved\n \n return current_context\n```\n\n## Performance Benchmarks\n\n### Compaction Performance\n\nCompaction should reduce token count while preserving quality. Target:\n\n- 50-70% token reduction for aggressive compaction\n- Less than 5% quality degradation from compaction\n- Less than 10% latency increase from compaction overhead\n\n### Masking Performance\n\nObservation masking should reduce token count significantly:\n\n- 60-80% reduction in masked observations\n- Less than 2% quality impact from masking\n- Near-zero latency overhead\n\n### Cache Performance\n\nKV-cache optimization should improve cost and latency:\n\n- 70%+ cache hit rate for stable workloads\n- 50%+ cost reduction from cache hits\n- 40%+ latency reduction from cache hits\n\n","isInternal":false,"tokens":1743,"sizeBytes":9347},{"name":"SKILL.md","path":"skills/context-optimization/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/context-optimization/SKILL.md","title":"context-optimization","category":"anthropic-skill","format":"markdown","content":"---\nname: context-optimization\ndescription: \"This skill should be used for improving context efficiency: context budgeting, observation masking, prefix or KV-cache strategy, partitioning, token-cost reduction, retrieval scoping, and extending effective context capacity without lowering answer quality.\"\n---\n\n# Context Optimization Techniques\n\nContext optimization extends the effective capacity of limited context windows through strategic compression, masking, caching, and partitioning. Effective optimization increases useful capacity without requiring larger models or longer windows — but only when applied with measurement discipline. The techniques below are ordered by impact and risk.\n\n## When to Activate\n\nActivate this skill when:\n- Context budgets or token costs constrain task complexity\n- Observation masking can replace verbose tool outputs with retrievable references\n- Prefix or KV-cache hit rate needs improvement\n- Retrieval scoping can reduce irrelevant loaded context\n- Context partitioning can extend effective capacity across agents\n- Budget triggers are needed for masking, compaction, or partitioning\n\nDo not activate this skill for adjacent work owned by other skills:\n- Explaining why attention or context windows behave this way: `context-fundamentals`.\n- Diagnosing active lost-in-middle, poisoning, distraction, confusion, or clash: `context-degradation`.\n- Designing a structured handoff summary for a long conversation: `context-compression`.\n- Storing large outputs, plans, or logs as files: `filesystem-context`.\n- Model-initiated rewrites of its own context and their re-prefill cost: `self-managed-context`.\n\n## Core Concepts\n\nApply four primary strategies in this priority order:\n\n1. **KV-cache optimization** — Reorder and stabilize prompt structure so the inference engine reuses cached Key/Value tensors. This is the cheapest optimization when the runtime supports prefix caching: low quality risk, immediate cost and latency savings. Apply it first when stable prefixes exist.\n\n2. **Observation masking** — Replace verbose tool outputs with compact references once their purpose has been served. Tool outputs can dominate agent trajectories (claim-context-optimization-tool-output-dominance), so masking often yields the largest capacity gains. The original content remains retrievable if needed downstream.\n\n3. **Compaction** — Summarize accumulated context when utilization exceeds 70%, then reinitialize with the summary. This distills the window's contents while preserving task-critical state. Compaction is lossy — apply it after masking has already removed the low-value bulk.\n\n4. **Context partitioning** — Split work across sub-agents with isolated contexts when a single window cannot hold the full problem. Each sub-agent operates in a clean context focused on its subtask. Reserve this for tasks where estimated context exceeds 60% of the window limit, because coordination overhead is real.\n\nThe governing principle: context quality matters more than quantity. Every optimization preserves signal while reducing noise. Measure before optimizing, then measure the optimization's effect.\n\n## Detailed Topics\n\n### Compaction Strategies\n\nTrigger compaction when context utilization exceeds 70%: summarize the current context, then reinitialize with the summary. This distills the window's contents in a high-fidelity manner, enabling continuation with minimal performance degradation. Prioritize compressing tool outputs first (they consume 80%+ of tokens), then old conversation turns, then retrieved documents. Never compress the system prompt — it anchors model behavior and its removal causes unpredictable degradation.\n\nPreserve different elements by message type:\n\n- **Tool outputs**: Extract key findings, metrics, error codes, and conclusions. Strip verbose raw output, stack traces (unless debugging is ongoing), and boilerplate headers.\n- **Conversational turns**: Retain decisions, commitments, user preferences, and context shifts. Remove filler, pleasantries, and exploratory back-and-forth that led to a conclusion already captured.\n- **Retrieved documents**: Keep claims, facts, and data points relevant to the active task. Remove supporting evidence and elaboration that served a one-time reasoning purpose.\n\nTarget 50-70% token reduction with less than 5% quality degradation. If compaction exceeds 70% reduction, audit the summary for critical information loss — over-aggressive compaction is the most common failure mode.\n\n### Observation Masking\n\nMask observations selectively based on recency and ongoing relevance — not uniformly. Apply these rules:\n\n- **Never mask**: Observations critical to the current task, observations from the most recent turn, observations used in active reasoning chains, and error outputs when debugging is in progress.\n- **Mask after 3+ turns**: Verbose outputs whose key points have already been extracted into the conversation flow. Replace with a compact reference: `[Obs:{ref_id} elided. Key: {summary}. Full content retrievable.]`\n- **Always mask immediately**: Repeated/duplicate outputs, boilerplate headers and footers, outputs already summarized earlier in the conversation.\n\nMasking should achieve 60-80% reduction in masked observations with less than 2% quality impact. The key is maintaining retrievability — store the full content externally and keep the reference ID in context so the agent can request the original if needed.\n\n### KV-Cache Optimization\n\nMaximize prefix cache hits by structuring prompts so that stable content occupies the prefix and dynamic content appears at the end. KV-cache stores Key and Value tensors computed during inference; when consecutive requests share an identical prefix, the cached tensors are reused, saving both cost and latency.\n\nApply this ordering in every prompt:\n1. System prompt (most stable — never changes within a session)\n2. Tool definitions (stable across requests)\n3. Frequently reused templates and few-shot examples\n4. Conversation history (grows but shares prefix with prior turns)\n5. Current query and dynamic content (least stable — always last)\n\nDesign prompts for cache stability: remove timestamps, session counters, and request IDs from the system prompt. Move dynamic metadata into a separate user message or tool result where it does not break the prefix. Even a single whitespace change in the prefix invalidates the entire cached block downstream of that change. The same rule prices any compaction or masking edit applied to earlier history: its cost is the text that follows it, so batch such edits and prefer them near the end of the context.\n\nTarget 70%+ cache hit rate for stable workloads. At scale, this translates to 50%+ cost reduction and 40%+ latency reduction on cached tokens.\n\n### Context Partitioning\n\nPartition work across sub-agents when a single context cannot hold the full problem without triggering aggressive compaction. Each sub-agent operates in a clean, focused context for its subtask, then returns a structured result to a coordinator agent.\n\nPlan partitioning when estimated task context exceeds 60% of the window limit. Decompose the task into independent subtasks, assign each to a sub-agent, and aggregate results. Validate that all partitions completed before merging, merge compatible results, and apply summarization if the aggregated output still exceeds budget.\n\nThis approach achieves separation of concerns — detailed search context stays isolated within sub-agents while the coordinator focuses on synthesis. However, coordination has real token cost: the coordinator prompt, result aggregation, and error handling all consume tokens. Only partition when the savings exceed this overhead.\n\n### Budget Management\n\nAllocate explicit token budgets across context categories before the session begins: system prompt, tool definitions, retrieved documents, message history, tool outputs, and a reserved buffer (5-10% of total). Monitor usage against budget continuously and trigger optimization when any category exceeds its allocation or total utilization crosses 70%.\n\nUse trigger-based optimization rather than periodic optimization. Monitor these signals:\n- Token utilization above 80% — trigger compaction\n- Attention degradation indicators (repetition, missed instructions) — trigger masking + compaction\n- Quality score drops below baseline — audit context composition before optimizing\n\n## Practical Guidance\n\n### Optimization Decision Framework\n\nSelect the optimization technique based on what dominates the context:\n\n| Context Composition | First Action | Second Action |\n|---|---|---|\n| Tool outputs dominate (>50%) | Observation masking | Compaction of remaining turns |\n| Retrieved documents dominate | Summarization | Partitioning if docs are independent |\n| Message history dominates | Compaction with selective preservation | Partitioning for new subtasks |\n| Multiple components contribute | KV-cache optimization first, then layer masking + compaction |\n| Near-limit with active debugging | Mask resolved tool outputs only — preserve error details |\n\n### Performance Targets\n\nTrack these metrics to validate optimization effectiveness:\n\n- **Compaction**: 50-70% token reduction, <5% quality degradation, <10% latency overhead from the compaction step itself\n- **Masking**: 60-80% reduction in masked observations, <2% quality impact, near-zero latency overhead\n- **Cache optimization**: 70%+ hit rate for stable workloads, 50%+ cost reduction, 40%+ latency reduction\n- **Partitioning**: Net token savings after accounting for coordinator overhead; break-even typically requires 3+ subtasks\n\nIterate on strategies based on measured results. If an optimization technique does not measurably improve the target metric, remove it — optimization machinery itself consumes tokens and adds latency.\n\n## Examples\n\n**Example 1: Compaction Trigger**\n```python\nif context_tokens / context_limit > 0.8:\n context = compact_context(context)\n```\n\n**Example 2: Observation Masking**\n```python\nif len(observation) > max_length:\n ref_id = store_observation(observation)\n return f\"[Obs:{ref_id} elided. Key: {extract_key(observation)}]\"\n```\n\n**Example 3: Cache-Friendly Ordering**\n```python\n# Stable content first\ncontext = [system_prompt, tool_definitions] # Cacheable\ncontext += [reused_templates] # Reusable\ncontext += [unique_content] # Unique\n```\n\n**Example 4: Budget-triggered optimization policy**\n```yaml\nbudgets:\n tool_outputs: 35%\n message_history: 30%\n retrieved_documents: 20%\n reserved_buffer: 15%\ntriggers:\n tool_outputs_over_budget: mask resolved observations\n total_context_over_70_percent: compact message history\n repeated_irrelevant_retrievals: tighten retrieval scope\n```\n\n## Guidelines\n\n1. Measure before optimizing—know your current state\n2. Apply masking before compaction — remove low-value bulk first, then summarize what remains\n3. Design for cache stability with consistent prompts\n4. Partition before context becomes problematic\n5. Monitor optimization effectiveness over time\n6. Balance token savings against quality preservation\n7. Test optimization at production scale\n8. Implement graceful degradation for edge cases\n\n## Gotchas\n\n1. **Whitespace breaks KV-cache**: Even a single whitespace or newline change in the prompt prefix invalidates the entire KV-cache block downstream of that point. Pin system prompts as immutable strings — do not interpolate timestamps, version numbers, or session IDs into them. Diff prompt templates byte-for-byte between deployments.\n\n2. **Timestamps in system prompts destroy cache hit rates**: Including `Current date: {today}` or similar dynamic content in the system prompt forces a full cache miss on every new day (or every request, if using time-of-day). Move dynamic metadata into a user message or a separate tool result appended after the stable prefix.\n\n3. **Compaction under pressure loses critical state**: When the model performing compaction is itself under context pressure (>85% utilization), its summarization quality degrades — it omits task goals, drops user constraints, and flattens nuanced state. Trigger compaction at 70-80%, not 90%+. If compaction must happen late, use a separate model call with a clean context containing only the material to summarize.\n\n4. **Masking error outputs breaks debugging loops**: Over-aggressive masking hides error messages, stack traces, and failure details that the agent needs in subsequent turns to diagnose and fix issues. During active debugging (error in the last 3 turns), suspend masking for all error-related observations until the issue is resolved.\n\n5. **Partitioning overhead can exceed savings**: Each sub-agent requires its own system prompt, tool definitions, and coordination messages. For tasks with fewer than 3 independent subtasks, the coordination overhead often exceeds the context savings. Estimate total tokens (coordinator + all sub-agents) before committing to partitioning.\n\n6. **Cache miss cost spikes after deployment changes**: Reordering tools, rewording the system prompt, or changing few-shot examples between deployments invalidates the entire prefix cache, causing a temporary cost spike of 2-5x until the new cache warms up. Roll out prompt changes gradually and monitor cache hit rate during deployment windows.\n\n7. **Compaction creates false confidence in stale summaries**: Once context is compacted, the summary looks authoritative but may reflect outdated state. If the task has evolved since compaction (new user requirements, corrected assumptions), the summary silently carries forward stale information. After compaction, re-validate the summary against the current task goal before proceeding.\n\n## Integration\n\nThis skill owns token-efficiency tactics and budget policy. Adjacent skills own diagnosis, storage, and architecture:\n\n- `context-fundamentals`: mental models for why context quality and attention placement matter.\n- `context-degradation`: diagnosis when output quality has already dropped.\n- `context-compression`: lossy summarization and handoff strategy.\n- `filesystem-context`: file-backed offloading for full outputs and logs.\n- `multi-agent-patterns`: partitioning work across isolated agent contexts.\n- `latent-briefing`: selective KV retention across orchestrator-worker boundaries in compatible runtimes.\n- `self-managed-context`: the model, not the harness, decides when and what to edit in its live context.\n- `evaluation`: measuring whether the optimization improved quality, cost, or latency.\n- `memory-systems`: persistent retrieval layers that feed context just in time.\n\n## References\n\nInternal reference:\n- [Optimization Techniques Reference](./references/optimization_techniques.md) - Read when: implementing a specific optimization technique and needing detailed code patterns, threshold tables, or integration examples beyond what the skill body provides\n\nRelated skills in this collection:\n- context-fundamentals - Read when: unfamiliar with context window mechanics, token counting, or attention distribution basics\n- context-degradation - Read when: diagnosing why agent performance has dropped and needing to identify which degradation pattern is occurring before selecting an optimization\n- evaluation - Read when: setting up metrics and benchmarks to measure whether an optimization technique actually improved outcomes\n\nExternal resources:\n- Research on context window limitations - Read when: evaluating model-specific context behavior (e.g., lost-in-the-middle effects, attention decay curves)\n- KV-cache optimization techniques - Read when: implementing prefix caching at the inference infrastructure level (vLLM, TGI, or cloud provider APIs)\n- Production engineering guides - Read when: deploying context optimization in a production pipeline and needing operability patterns (monitoring, alerting, rollback)\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-20\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.1.0\n","frontmatter":{"name":"context-optimization","description":"This skill should be used for improving context efficiency: context budgeting, observation masking, prefix or KV-cache strategy, partitioning, token-cost reduction, retrieval scoping, and extending effective context capacity without lowering answer quality."},"isInternal":false,"tokens":3106,"sizeBytes":16052},{"name":"metrics.md","path":"skills/evaluation/references/metrics.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/evaluation/references/metrics.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Evaluation Reference: Metrics and Implementation\n\nThis document provides implementation details for evaluation metrics and evaluation systems.\n\n## Core Metric Definitions\n\n### Factual Accuracy\n\nFactual accuracy measures whether claims in agent output match ground truth.\n\n```\nExcellent (1.0): All claims verified against ground truth, no errors\nGood (0.8): Minor errors that do not affect main conclusions\nAcceptable (0.6): Major claims correct, minor inaccuracies present\nPoor (0.3): Significant factual errors in key claims\nFailed (0.0): Fundamental factual errors that invalidate output\n```\n\nCalculation approach:\n- Extract claims from output\n- Verify each claim against ground truth\n- Weight claims by importance (major claims more weight)\n- Calculate weighted average of claim accuracy\n\n### Completeness\n\nCompleteness measures whether output covers all requested aspects.\n\n```\nExcellent (1.0): All requested aspects thoroughly covered\nGood (0.8): Most aspects covered with minor gaps\nAcceptable (0.6): Key aspects covered, some gaps\nPoor (0.3): Major aspects missing from output\nFailed (0.0): Fundamental aspects not addressed\n```\n\n### Citation Accuracy\n\nCitation accuracy measures whether cited sources match claimed sources.\n\n```\nExcellent (1.0): All citations accurate and complete\nGood (0.8): Minor citation formatting issues\nAcceptable (0.6): Major citations accurate\nPoor (0.3): Significant citation problems\nFailed (0.0): Citations missing or completely incorrect\n```\n\n### Source Quality\n\nSource quality measures whether appropriate primary sources were used.\n\n```\nExcellent (1.0): Primary authoritative sources\nGood (0.8): Mostly primary sources with some secondary\nAcceptable (0.6): Mix of primary and secondary sources\nPoor (0.3): Mostly secondary or unreliable sources\nFailed (0.0): No credible sources cited\n```\n\n### Tool Efficiency\n\nTool efficiency measures whether the agent used appropriate tools a reasonable number of times.\n\n```\nExcellent (1.0): Optimal tool selection and call count\nGood (0.8): Good tool selection with minor inefficiencies\nAcceptable (0.6): Appropriate tools with some redundancy\nPoor (0.3): Wrong tools or excessive call counts\nFailed (0.0): Severe tool misuse or extremely excessive calls\n```\n\n## Rubric Implementation\n\n```python\nEVALUATION_DIMENSIONS = {\n \"factual_accuracy\": {\n \"weight\": 0.30,\n \"description\": \"Claims match ground truth\",\n \"levels\": {\n \"excellent\": 1.0,\n \"good\": 0.8,\n \"acceptable\": 0.6,\n \"poor\": 0.3,\n \"failed\": 0.0\n }\n },\n \"completeness\": {\n \"weight\": 0.25,\n \"description\": \"All requested aspects covered\",\n \"levels\": {\n \"excellent\": 1.0,\n \"good\": 0.8,\n \"acceptable\": 0.6,\n \"poor\": 0.3,\n \"failed\": 0.0\n }\n },\n \"citation_accuracy\": {\n \"weight\": 0.15,\n \"description\": \"Citations match sources\",\n \"levels\": {\n \"excellent\": 1.0,\n \"good\": 0.8,\n \"acceptable\": 0.6,\n \"poor\": 0.3,\n \"failed\": 0.0\n }\n },\n \"source_quality\": {\n \"weight\": 0.10,\n \"description\": \"Appropriate primary sources used\",\n \"levels\": {\n \"excellent\": 1.0,\n \"good\": 0.8,\n \"acceptable\": 0.6,\n \"poor\": 0.3,\n \"failed\": 0.0\n }\n },\n \"tool_efficiency\": {\n \"weight\": 0.20,\n \"description\": \"Right tools used reasonably\",\n \"levels\": {\n \"excellent\": 1.0,\n \"good\": 0.8,\n \"acceptable\": 0.6,\n \"poor\": 0.3,\n \"failed\": 0.0\n }\n }\n}\n\ndef calculate_overall_score(dimension_scores, rubric):\n \"\"\"Calculate weighted overall score from dimension scores.\"\"\"\n total_weight = 0\n weighted_sum = 0\n \n for dimension, score in dimension_scores.items():\n if dimension in rubric:\n weight = rubric[dimension][\"weight\"]\n weighted_sum += score * weight\n total_weight += weight\n \n return weighted_sum / total_weight if total_weight > 0 else 0\n```\n\n## Test Set Management\n\n```python\nclass TestSet:\n def __init__(self, name):\n self.name = name\n self.tests = []\n self.tags = {}\n \n def add_test(self, test_case):\n \"\"\"Add test case to test set.\"\"\"\n self.tests.append(test_case)\n \n # Index by tags\n for tag in test_case.get(\"tags\", []):\n if tag not in self.tags:\n self.tags[tag] = []\n self.tags[tag].append(len(self.tests) - 1)\n \n def filter(self, **criteria):\n \"\"\"Filter tests by criteria.\"\"\"\n filtered = []\n for test in self.tests:\n match = True\n for key, value in criteria.items():\n if test.get(key) != value:\n match = False\n break\n if match:\n filtered.append(test)\n return filtered\n \n def get_complexity_distribution(self):\n \"\"\"Get distribution of tests by complexity.\"\"\"\n distribution = {}\n for test in self.tests:\n complexity = test.get(\"complexity\", \"medium\")\n distribution[complexity] = distribution.get(complexity, 0) + 1\n return distribution\n```\n\n## Evaluation Runner\n\n```python\nclass EvaluationRunner:\n def __init__(self, test_set, rubric, agent):\n self.test_set = test_set\n self.rubric = rubric\n self.agent = agent\n self.results = []\n \n def run_all(self, verbose=False):\n \"\"\"Run evaluation on all tests.\"\"\"\n self.results = []\n \n for i, test in enumerate(self.test_set.tests):\n if verbose:\n print(f\"Running test {i+1}/{len(self.test_set.tests)}\")\n \n result = self.run_test(test)\n self.results.append(result)\n \n return self.summarize()\n \n def run_test(self, test):\n \"\"\"Run single evaluation test.\"\"\"\n # Get agent output\n output = self.agent.run(test[\"input\"])\n \n # Evaluate\n evaluation = self.evaluate_output(output, test)\n \n return {\n \"test\": test,\n \"output\": output,\n \"evaluation\": evaluation\n }\n \n def evaluate_output(self, output, test):\n \"\"\"Evaluate agent output against test.\"\"\"\n ground_truth = test.get(\"expected\", {})\n \n dimension_scores = {}\n for dimension, config in self.rubric.items():\n score = self.evaluate_dimension(\n output, ground_truth, dimension, config\n )\n dimension_scores[dimension] = score\n \n overall = calculate_overall_score(dimension_scores, self.rubric)\n \n return {\n \"overall_score\": overall,\n \"dimension_scores\": dimension_scores,\n \"passed\": overall >= 0.7\n }\n \n def summarize(self):\n \"\"\"Summarize evaluation results.\"\"\"\n if not self.results:\n return {\"error\": \"No results\"}\n \n passed = sum(1 for r in self.results if r[\"evaluation\"][\"passed\"])\n \n dimension_totals = {}\n for dimension in self.rubric.keys():\n dimension_totals[dimension] = {\n \"total\": 0,\n \"count\": 0\n }\n \n for result in self.results:\n for dimension, score in result[\"evaluation\"][\"dimension_scores\"].items():\n if dimension in dimension_totals:\n dimension_totals[dimension][\"total\"] += score\n dimension_totals[dimension][\"count\"] += 1\n \n dimension_averages = {}\n for dimension, data in dimension_totals.items():\n if data[\"count\"] > 0:\n dimension_averages[dimension] = data[\"total\"] / data[\"count\"]\n \n return {\n \"total_tests\": len(self.results),\n \"passed\": passed,\n \"failed\": len(self.results) - passed,\n \"pass_rate\": passed / len(self.results) if self.results else 0,\n \"dimension_averages\": dimension_averages,\n \"failures\": [\n r for r in self.results \n if not r[\"evaluation\"][\"passed\"]\n ]\n }\n```\n\n## Production Monitoring\n\n```python\nclass ProductionMonitor:\n def __init__(self, sample_rate=0.01):\n self.sample_rate = sample_rate\n self.samples = []\n self.alert_thresholds = {\n \"pass_rate_warning\": 0.85,\n \"pass_rate_critical\": 0.70\n }\n \n def sample_and_evaluate(self, query, output):\n \"\"\"Sample production interaction for evaluation.\"\"\"\n if random.random() > self.sample_rate:\n return None\n \n evaluation = evaluate_output(output, {}, EVALUATION_RUBRIC)\n \n sample = {\n \"query\": query[:200],\n \"output_preview\": output[:200],\n \"score\": evaluation[\"overall_score\"],\n \"passed\": evaluation[\"passed\"],\n \"timestamp\": current_timestamp()\n }\n \n self.samples.append(sample)\n return sample\n \n def get_metrics(self):\n \"\"\"Calculate current metrics from samples.\"\"\"\n if not self.samples:\n return {\"status\": \"insufficient_data\"}\n \n passed = sum(1 for s in self.samples if s[\"passed\"])\n pass_rate = passed / len(self.samples)\n \n avg_score = sum(s[\"score\"] for s in self.samples) / len(self.samples)\n \n return {\n \"sample_count\": len(self.samples),\n \"pass_rate\": pass_rate,\n \"average_score\": avg_score,\n \"status\": self._get_status(pass_rate)\n }\n \n def _get_status(self, pass_rate):\n \"\"\"Get status based on pass rate.\"\"\"\n if pass_rate < self.alert_thresholds[\"pass_rate_critical\"]:\n return \"critical\"\n elif pass_rate < self.alert_thresholds[\"pass_rate_warning\"]:\n return \"warning\"\n else:\n return \"healthy\"\n```\n\n","isInternal":false,"tokens":2224,"sizeBytes":10139},{"name":"SKILL.md","path":"skills/evaluation/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/evaluation/SKILL.md","title":"evaluation","category":"anthropic-skill","format":"markdown","content":"---\nname: evaluation\ndescription: \"This skill should be used when building agent evaluation systems: deterministic checks, regression suites, multi-dimensional rubrics, quality gates, production monitoring, baseline comparison, and outcome measurement for agent pipelines.\"\n---\n\n# Evaluation Methods for Agent Systems\n\nEvaluate agent systems differently from traditional software because agents make dynamic decisions, are non-deterministic between runs, and often lack single correct answers. Build evaluation frameworks that account for these characteristics, provide actionable feedback, catch regressions, and validate that context engineering choices achieve intended effects.\n\n## When to Activate\n\nActivate this skill when:\n- Testing agent performance systematically\n- Validating context engineering choices\n- Measuring improvements over time\n- Catching regressions before deployment\n- Building quality gates for agent pipelines\n- Comparing different agent configurations\n- Evaluating production systems continuously\n\nDo not activate this skill for adjacent work owned by other skills:\n- Designing the LLM judge itself, pairwise comparison, judge calibration, or bias mitigation: `advanced-evaluation`.\n- Designing autonomous control surfaces, novelty gates, rollback, or PR approval boundaries: `harness-engineering`.\n- Debugging a specific context failure mode before measuring it: `context-degradation`.\n\n## Core Concepts\n\nFocus evaluation on outcomes rather than execution paths, because agents may find alternative valid routes to goals. Judge whether the agent achieves the right outcome via a reasonable process, not whether it followed a specific sequence of steps.\n\nUse multi-dimensional rubrics instead of single scores because one number hides critical failures in specific dimensions. Capture factual accuracy, completeness, citation accuracy, source quality, and tool efficiency as separate dimensions, then weight them for the use case.\n\nUse model-judged evaluation only after deterministic checks and rubrics are stable. When the work centers on judge prompts, pairwise comparison, calibration, or bias mitigation, switch to Advanced Evaluation.\n\nRun deterministic validation before LLM judgment whenever the artifact has machine-checkable structure. Schema validity, duplicate keys, rubric math, manifest sync, retrieval status, and required evidence paths should fail fast before an evaluator spends tokens or returns a subjective score.\n\n**Performance Drivers**\n\nApply browsing-agent research when designing evaluation budgets: token usage, tool calls, and model choice can dominate measured performance variance (claim-evaluation-browsecomp-variance).\n\n| Factor | Variance Explained | Implication |\n|--------|-------------------|-------------|\n| Token usage | Primary driver | More exploration can improve performance until cost or context quality collapses |\n| Number of tool calls | Secondary driver | More tool use helps only when calls retrieve useful evidence |\n| Model choice | Secondary but multiplicative | Better models often use tokens and tools more efficiently |\n\nAct on these implications when designing evaluations:\n- **Set realistic token budgets**: Evaluate agents with production-realistic token limits, not unlimited resources.\n- **Compare model upgrades against token increases**: Better models may use tokens more efficiently than weaker models with larger budgets.\n- **Validate multi-agent architectures**: Extra agents add tokens and tool calls; evaluate them against single-agent baselines.\n\n## Detailed Topics\n\n### Evaluation Challenges\n\n**Handle Non-Determinism and Multiple Valid Paths**\n\nDesign evaluations that tolerate path variation because agents may take completely different valid paths to reach goals. One agent might search three sources while another searches ten; both may produce correct answers. Avoid checking for specific steps. Instead, define outcome criteria (correctness, completeness, quality) and score against those, treating the execution path as informational rather than evaluative.\n\n**Test Context-Dependent Failures**\n\nEvaluate across a range of complexity levels and interaction lengths because agent failures often depend on context in subtle ways. An agent might succeed on simple queries but fail on complex ones, work well with one tool set but fail with another, or degrade after extended interaction as context accumulates. Include simple, medium, complex, and very complex test cases to surface these patterns.\n\n**Score Composite Quality Dimensions Separately**\n\nBreak agent quality into separate dimensions (factual accuracy, completeness, coherence, tool efficiency, process quality) and score each independently because an agent might score high on accuracy but low on efficiency, or vice versa. Then compute weighted aggregates tuned to use-case priorities. This approach reveals which dimensions need improvement rather than averaging away the signal.\n\n### Evaluation Rubric Design\n\n**Build Multi-Dimensional Rubrics**\n\nDefine rubrics covering key dimensions with descriptive levels from excellent to failed. Include these core dimensions and adapt weights per use case:\n\n- Factual accuracy: Claims match ground truth (weight heavily for knowledge tasks)\n- Completeness: Output covers requested aspects (weight heavily for research tasks)\n- Citation accuracy: Citations match claimed sources (weight for trust-sensitive contexts)\n- Source quality: Uses appropriate primary sources (weight for authoritative outputs)\n- Tool efficiency: Uses right tools a reasonable number of times (weight for cost-sensitive systems)\n\n**Convert Rubrics to Numeric Scores**\n\nMap dimension assessments to numeric scores (0.0 to 1.0), apply per-dimension weights, and calculate weighted overall scores. Set passing thresholds based on use-case requirements, typically 0.7 for general use and 0.9 for high-stakes applications. Store individual dimension scores alongside the aggregate because the breakdown drives targeted improvement.\n\n### Evaluation Methodologies\n\n**Use LLM-as-Judge for Scale**\n\nBuild LLM-based evaluation prompts that include: clear task description, the agent output under test, ground truth when available, an evaluation scale with explicit level descriptions, and a request for structured judgment with reasoning. LLM judges provide consistent, scalable evaluation across large test sets. Use a different model family than the agent being evaluated to avoid self-enhancement bias.\n\n**Supplement with Human Evaluation**\n\nRoute edge cases, unusual queries, and a random sample of production traffic to human reviewers because humans notice hallucinated answers, system failures, and subtle biases that automated evaluation misses. Track patterns across human reviews to identify systematic issues and feed findings back into automated evaluation criteria.\n\n**Apply End-State Evaluation for Stateful Agents**\n\nFor agents that mutate persistent state (files, databases, configurations), evaluate whether the final state matches expectations rather than how the agent got there. Define expected end-state assertions and verify them programmatically after each test run.\n\n### Test Set Design\n\n**Select Representative Samples**\n\nStart with small samples (20-30 cases) during early development when changes have dramatic impacts and low-hanging fruit is abundant. Scale to 50+ cases for reliable signal as the system matures. Sample from real usage patterns, add known edge cases, and ensure coverage across complexity levels.\n\n**Stratify by Complexity**\n\nStructure test sets across complexity levels to prevent easy examples from inflating scores:\n- Simple: single tool call, factual lookup\n- Medium: multiple tool calls, comparison logic\n- Complex: many tool calls, significant ambiguity\n- Very complex: extended interaction, deep reasoning, synthesis\n\nReport scores per stratum alongside overall scores to reveal where the agent actually struggles.\n\n### Context Engineering Evaluation\n\n**Validate Context Strategies Systematically**\n\nRun agents with different context strategies on the same test set and compare quality scores, token usage, and efficiency metrics. This isolates the effect of context engineering from other variables and prevents anecdote-driven decisions.\n\n**Run Degradation Tests**\n\nTest how context degradation affects performance by running agents at different context sizes. Identify performance cliffs where context becomes problematic and establish safe operating limits. Feed these limits back into context management strategies.\n\n### Continuous Evaluation\n\n**Build Automated Evaluation Pipelines**\n\nIntegrate evaluation into the development workflow so evaluations run automatically on agent changes. Track results over time, compare versions, and block deployments that regress on key metrics.\n\n**Monitor Production Quality**\n\nSample production interactions and evaluate them continuously. Set alerts for quality drops below warning (0.85 pass rate) and critical (0.70 pass rate) thresholds. Maintain dashboards showing trend analysis over time windows to detect gradual degradation.\n\n## Practical Guidance\n\n### Building Evaluation Frameworks\n\nFollow this sequence to build an evaluation framework, because skipping early steps leads to measurements that do not reflect real quality:\n\n1. Define quality dimensions relevant to the use case before writing any evaluation code, because dimensions chosen later tend to reflect what is easy to measure rather than what matters.\n2. Create rubrics with clear, descriptive level definitions so evaluators (human or LLM) produce consistent scores.\n3. Build test sets from real usage patterns and edge cases, stratified by complexity, with at least 50 cases for reliable signal.\n4. Implement automated evaluation pipelines that run on every significant change.\n5. Establish baseline metrics before making changes so improvements can be measured against a known reference.\n6. Run evaluations on all significant changes and compare against the baseline.\n7. Track metrics over time for trend analysis because gradual degradation is harder to notice than sudden drops.\n8. Supplement automated evaluation with human review on a regular cadence.\n9. Separate deterministic validation failures from quality judgments so invalid artifacts cannot be laundered by a favorable LLM score.\n\n### Avoiding Evaluation Pitfalls\n\nGuard against these common failures that undermine evaluation reliability:\n\n- **Overfitting to specific paths**: Evaluate outcomes, not specific steps, because agents find novel valid paths.\n- **Ignoring edge cases**: Include diverse test scenarios covering the full complexity spectrum.\n- **Single-metric obsession**: Use multi-dimensional rubrics because a single score hides dimension-specific failures.\n- **Neglecting context effects**: Test with realistic context sizes and histories rather than clean-room conditions.\n- **Skipping human evaluation**: Automated evaluation misses subtle issues that humans catch reliably.\n\n## Examples\n\n**Example 1: Simple Evaluation**\n```python\ndef evaluate_agent_response(response, expected):\n rubric = load_rubric()\n scores = {}\n for dimension, config in rubric.items():\n scores[dimension] = assess_dimension(response, expected, dimension)\n overall = weighted_average(scores, config[\"weights\"])\n return {\"passed\": overall >= 0.7, \"scores\": scores}\n```\n\n**Example 2: Test Set Structure**\n\nTest sets should span multiple complexity levels to ensure comprehensive evaluation:\n\n```python\ntest_set = [\n {\n \"name\": \"simple_lookup\",\n \"input\": \"What is the capital of France?\",\n \"expected\": {\"type\": \"fact\", \"answer\": \"Paris\"},\n \"complexity\": \"simple\",\n \"description\": \"Single tool call, factual lookup\"\n },\n {\n \"name\": \"medium_query\",\n \"input\": \"Compare the revenue of Apple and Microsoft last quarter\",\n \"complexity\": \"medium\",\n \"description\": \"Multiple tool calls, comparison logic\"\n },\n {\n \"name\": \"multi_step_reasoning\",\n \"input\": \"Analyze sales data from Q1-Q4 and create a summary report with trends\",\n \"complexity\": \"complex\",\n \"description\": \"Many tool calls, aggregation, analysis\"\n },\n {\n \"name\": \"research_synthesis\",\n \"input\": \"Research emerging AI technologies, evaluate their potential impact, and recommend adoption strategy\",\n \"complexity\": \"very_complex\",\n \"description\": \"Extended interaction, deep reasoning, synthesis\"\n }\n]\n```\n\n**Example 3: Deterministic gate before model judgment**\n```python\ndef evaluate_pr_candidate(candidate):\n structure = run_validate_repo(candidate)\n if not structure.ok:\n return {\"passed\": False, \"reason\": \"deterministic validation failed\", \"details\": structure.errors}\n\n quality = run_rubric_eval(candidate)\n return {\"passed\": quality.overall >= 0.8, \"scores\": quality.dimensions}\n```\n\n**Example 4: Quality gate dimensions**\n```yaml\ngate:\n deterministic:\n - schema_valid\n - required_files_present\n - no_duplicate_ids\n quality:\n factual_accuracy: min 0.85\n completeness: min 0.80\n source_traceability: min 0.90\n```\n\n## Guidelines\n\n1. Use multi-dimensional rubrics, not single metrics\n2. Evaluate outcomes, not specific execution paths\n3. Cover complexity levels from simple to complex\n4. Test with realistic context sizes and histories\n5. Run evaluations continuously, not just before release\n6. Supplement LLM evaluation with human review\n7. Track metrics over time for trend detection\n8. Set clear pass/fail thresholds based on use case\n\n## Gotchas\n\n1. **Overfitting evals to specific code paths**: Tests pass but the agent fails on slight input variations. Write eval criteria against outcomes and semantics, not surface patterns, and rotate test inputs periodically.\n2. **LLM-judge self-enhancement bias**: Models rate their own outputs higher than independent judges do. Use a different model family as the evaluation judge than the model being evaluated.\n3. **Test set contamination**: Eval examples leak into training data or prompt templates, inflating scores. Keep eval sets versioned and separate from any data used in prompts or fine-tuning.\n4. **Metric gaming**: Optimizing for the metric rather than actual quality produces agents that score well but disappoint users. Cross-validate automated metrics against human judgments regularly.\n5. **Single-dimension scoring**: One aggregate number hides critical failures in specific dimensions. Always report per-dimension scores alongside the overall score, and fail the eval if any single dimension falls below its minimum threshold.\n6. **Eval set too small**: Fewer than 50 examples produces unreliable signal with high variance between runs. Scale the eval set to at least 50 cases and report confidence intervals.\n7. **Not stratifying by difficulty**: Easy examples inflate overall scores, masking failures on hard cases. Report scores per complexity stratum and weight the overall score to prevent easy-case dominance.\n8. **Treating eval as one-time**: Evaluation must be continuous, not a launch gate. Agent quality drifts as models update, tools change, and usage patterns evolve. Run evals on every change and on a regular production cadence.\n\n## Integration\n\nThis skill owns outcome measurement and quality gates. Adjacent skills own specialized evaluator design and control-loop governance:\n\n- `advanced-evaluation`: LLM-as-judge prompt design, pairwise comparison, calibration, and bias mitigation.\n- `harness-engineering`: locked evaluators, editable surfaces, rollback, and human approval boundaries.\n- `context-degradation`: detecting and measuring degradation patterns.\n- `context-optimization`: measuring token, cost, latency, and quality effects of optimizations.\n- `multi-agent-patterns`: evaluating coordination quality and parallelization trade-offs.\n- `tool-design`: evaluating tool selection and recovery effectiveness.\n- `memory-systems`: evaluating memory retrieval and retention quality.\n\n## References\n\nInternal reference:\n- [Metrics Reference](./references/metrics.md) - Read when: designing specific evaluation metrics, choosing scoring scales, or implementing weighted rubric calculations\n\nInternal skills:\n- All other skills connect to evaluation for quality measurement\n\nExternal resources:\n- LLM evaluation benchmarks - Read when: selecting or building benchmark suites for agent comparison\n- Agent evaluation research papers - Read when: adopting new evaluation methodologies or validating current approach\n- Production monitoring practices - Read when: setting up alerting, dashboards, or sampling strategies for live systems\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-20\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 1.2.0\n","frontmatter":{"name":"evaluation","description":"This skill should be used when building agent evaluation systems: deterministic checks, regression suites, multi-dimensional rubrics, quality gates, production monitoring, baseline comparison, and outcome measurement for agent pipelines."},"isInternal":false,"tokens":3141,"sizeBytes":16829},{"name":"implementation-patterns.md","path":"skills/filesystem-context/references/implementation-patterns.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/filesystem-context/references/implementation-patterns.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Filesystem Context Implementation Patterns\n\nThis reference provides detailed implementation patterns for filesystem-based context engineering.\n\n## Pattern Catalog\n\n### 1. Scratch Pad Manager\n\nA centralized manager for handling large tool outputs and intermediate results.\n\n```python\nimport os\nimport json\nfrom datetime import datetime\nfrom pathlib import Path\n\nclass ScratchPadManager:\n \"\"\"Manages temporary file storage for agent context offloading.\"\"\"\n \n def __init__(self, base_path: str = \"scratch\", token_threshold: int = 2000):\n self.base_path = Path(base_path)\n self.base_path.mkdir(parents=True, exist_ok=True)\n self.token_threshold = token_threshold\n self.manifest = {}\n \n def should_offload(self, content: str) -> bool:\n \"\"\"Determine if content exceeds threshold for offloading.\"\"\"\n # Rough token estimate: 1 token ≈ 4 characters\n estimated_tokens = len(content) // 4\n return estimated_tokens > self.token_threshold\n \n def offload(self, content: str, source: str, summary: str = None) -> dict:\n \"\"\"Write content to file, return reference.\"\"\"\n timestamp = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n filename = f\"{source}_{timestamp}.txt\"\n file_path = self.base_path / filename\n \n file_path.write_text(content)\n \n reference = {\n \"type\": \"file_reference\",\n \"path\": str(file_path),\n \"source\": source,\n \"timestamp\": timestamp,\n \"size_chars\": len(content),\n \"summary\": summary or self._extract_summary(content)\n }\n \n self.manifest[filename] = reference\n return reference\n \n def _extract_summary(self, content: str, max_chars: int = 500) -> str:\n \"\"\"Extract first meaningful content as summary.\"\"\"\n lines = content.strip().split('\\n')\n summary_lines = []\n char_count = 0\n \n for line in lines:\n if char_count + len(line) > max_chars:\n break\n summary_lines.append(line)\n char_count += len(line)\n \n return '\\n'.join(summary_lines)\n \n def cleanup(self, max_age_hours: int = 24):\n \"\"\"Remove scratch files older than threshold.\"\"\"\n cutoff = datetime.now().timestamp() - (max_age_hours * 3600)\n \n for file_path in self.base_path.glob(\"*.txt\"):\n if file_path.stat().st_mtime < cutoff:\n file_path.unlink()\n if file_path.name in self.manifest:\n del self.manifest[file_path.name]\n```\n\n### 2. Plan Persistence\n\nStructured plan storage with progress tracking.\n\n```python\nimport yaml\nfrom dataclasses import dataclass, field, asdict\nfrom enum import Enum\nfrom typing import List, Optional\n\nclass StepStatus(Enum):\n PENDING = \"pending\"\n IN_PROGRESS = \"in_progress\"\n COMPLETED = \"completed\"\n BLOCKED = \"blocked\"\n CANCELLED = \"cancelled\"\n\n@dataclass\nclass PlanStep:\n id: int\n description: str\n status: StepStatus = StepStatus.PENDING\n notes: Optional[str] = None\n\n@dataclass\nclass AgentPlan:\n objective: str\n steps: List[PlanStep] = field(default_factory=list)\n status: str = \"in_progress\"\n \n def save(self, path: str = \"scratch/current_plan.yaml\"):\n \"\"\"Persist plan to filesystem.\"\"\"\n data = {\n \"objective\": self.objective,\n \"status\": self.status,\n \"steps\": [\n {\n \"id\": s.id,\n \"description\": s.description,\n \"status\": s.status.value,\n \"notes\": s.notes\n }\n for s in self.steps\n ]\n }\n with open(path, 'w') as f:\n yaml.dump(data, f, default_flow_style=False)\n \n @classmethod\n def load(cls, path: str = \"scratch/current_plan.yaml\") -> \"AgentPlan\":\n \"\"\"Load plan from filesystem.\"\"\"\n with open(path, 'r') as f:\n data = yaml.safe_load(f)\n \n plan = cls(objective=data[\"objective\"], status=data.get(\"status\", \"in_progress\"))\n for step_data in data.get(\"steps\", []):\n plan.steps.append(PlanStep(\n id=step_data[\"id\"],\n description=step_data[\"description\"],\n status=StepStatus(step_data[\"status\"]),\n notes=step_data.get(\"notes\")\n ))\n return plan\n \n def current_step(self) -> Optional[PlanStep]:\n \"\"\"Get the first non-completed step.\"\"\"\n for step in self.steps:\n if step.status != StepStatus.COMPLETED:\n return step\n return None\n \n def complete_step(self, step_id: int, notes: str = None):\n \"\"\"Mark step as completed.\"\"\"\n for step in self.steps:\n if step.id == step_id:\n step.status = StepStatus.COMPLETED\n if notes:\n step.notes = notes\n break\n```\n\n### 3. Sub-Agent Workspace\n\nFile-based communication between agents.\n\n```python\nfrom pathlib import Path\nfrom datetime import datetime\nimport json\n\nclass AgentWorkspace:\n \"\"\"Manages file-based workspace for an agent.\"\"\"\n \n def __init__(self, agent_id: str, base_path: str = \"workspace/agents\"):\n self.agent_id = agent_id\n self.path = Path(base_path) / agent_id\n self.path.mkdir(parents=True, exist_ok=True)\n \n # Standard files\n self.findings_file = self.path / \"findings.md\"\n self.status_file = self.path / \"status.json\"\n self.log_file = self.path / \"activity.log\"\n \n def write_finding(self, content: str, append: bool = True):\n \"\"\"Write or append a finding.\"\"\"\n mode = 'a' if append else 'w'\n with open(self.findings_file, mode) as f:\n if append:\n f.write(f\"\\n---\\n## {datetime.now().isoformat()}\\n\\n\")\n f.write(content)\n \n def update_status(self, status: str, progress: float = None, details: dict = None):\n \"\"\"Update agent status for coordinator visibility.\"\"\"\n status_data = {\n \"agent_id\": self.agent_id,\n \"status\": status,\n \"updated_at\": datetime.now().isoformat(),\n \"progress\": progress,\n \"details\": details or {}\n }\n self.status_file.write_text(json.dumps(status_data, indent=2))\n \n def log(self, message: str):\n \"\"\"Append to activity log.\"\"\"\n with open(self.log_file, 'a') as f:\n f.write(f\"[{datetime.now().isoformat()}] {message}\\n\")\n \n def read_peer_findings(self, peer_id: str) -> str:\n \"\"\"Read findings from another agent's workspace.\"\"\"\n peer_path = self.path.parent / peer_id / \"findings.md\"\n if peer_path.exists():\n return peer_path.read_text()\n return \"\"\n\n\nclass CoordinatorWorkspace:\n \"\"\"Coordinator that reads from sub-agent workspaces.\"\"\"\n \n def __init__(self, base_path: str = \"workspace/agents\"):\n self.base_path = Path(base_path)\n \n def get_all_statuses(self) -> dict:\n \"\"\"Collect status from all sub-agents.\"\"\"\n statuses = {}\n for agent_dir in self.base_path.iterdir():\n if agent_dir.is_dir():\n status_file = agent_dir / \"status.json\"\n if status_file.exists():\n statuses[agent_dir.name] = json.loads(status_file.read_text())\n return statuses\n \n def aggregate_findings(self) -> str:\n \"\"\"Combine all agent findings into synthesis.\"\"\"\n findings = []\n for agent_dir in self.base_path.iterdir():\n if agent_dir.is_dir():\n findings_file = agent_dir / \"findings.md\"\n if findings_file.exists():\n findings.append(f\"# {agent_dir.name}\\n\\n{findings_file.read_text()}\")\n return \"\\n\\n\".join(findings)\n```\n\n### 4. Dynamic Skill Loader\n\nLoad skill content on demand.\n\n```python\nfrom pathlib import Path\nfrom typing import List, Optional\nimport yaml\n\n@dataclass\nclass SkillMetadata:\n name: str\n description: str\n path: str\n triggers: List[str] = field(default_factory=list)\n\nclass SkillLoader:\n \"\"\"Manages dynamic loading of agent skills.\"\"\"\n \n def __init__(self, skills_path: str = \"skills\"):\n self.skills_path = Path(skills_path)\n self.skill_index = self._build_index()\n \n def _build_index(self) -> dict:\n \"\"\"Build index of available skills from SKILL.md frontmatter.\"\"\"\n index = {}\n for skill_dir in self.skills_path.iterdir():\n if skill_dir.is_dir():\n skill_file = skill_dir / \"SKILL.md\"\n if skill_file.exists():\n metadata = self._parse_frontmatter(skill_file)\n if metadata:\n index[metadata.name] = metadata\n return index\n \n def _parse_frontmatter(self, path: Path) -> Optional[SkillMetadata]:\n \"\"\"Extract YAML frontmatter from skill file.\"\"\"\n content = path.read_text()\n if content.startswith('---'):\n end = content.find('---', 3)\n if end > 0:\n frontmatter = yaml.safe_load(content[3:end])\n return SkillMetadata(\n name=frontmatter.get('name', path.parent.name),\n description=frontmatter.get('description', ''),\n path=str(path),\n triggers=frontmatter.get('triggers', [])\n )\n return None\n \n def get_static_context(self) -> str:\n \"\"\"Generate minimal static context listing available skills.\"\"\"\n lines = [\"Available skills (load with read_file when relevant):\"]\n for name, meta in self.skill_index.items():\n lines.append(f\"- {name}: {meta.description[:100]}\")\n return \"\\n\".join(lines)\n \n def load_skill(self, name: str) -> str:\n \"\"\"Load full skill content.\"\"\"\n if name in self.skill_index:\n return Path(self.skill_index[name].path).read_text()\n raise ValueError(f\"Unknown skill: {name}\")\n \n def find_relevant_skills(self, query: str) -> List[str]:\n \"\"\"Find skills that might be relevant to a query.\"\"\"\n query_lower = query.lower()\n relevant = []\n for name, meta in self.skill_index.items():\n if any(trigger in query_lower for trigger in meta.triggers):\n relevant.append(name)\n elif name.replace('-', ' ') in query_lower:\n relevant.append(name)\n return relevant\n```\n\n### 5. Terminal Output Persistence\n\nCapture and persist terminal sessions.\n\n```python\nimport subprocess\nfrom pathlib import Path\nfrom datetime import datetime\nimport re\n\nclass TerminalCapture:\n \"\"\"Captures and persists terminal output for agent access.\"\"\"\n \n def __init__(self, terminals_path: str = \"terminals\"):\n self.terminals_path = Path(terminals_path)\n self.terminals_path.mkdir(parents=True, exist_ok=True)\n self.session_counter = 0\n \n def run_command(self, command: str, capture: bool = True) -> dict:\n \"\"\"Run command and optionally capture output to file.\"\"\"\n self.session_counter += 1\n \n result = subprocess.run(\n command,\n shell=True,\n capture_output=True,\n text=True\n )\n \n output = {\n \"command\": command,\n \"exit_code\": result.returncode,\n \"stdout\": result.stdout,\n \"stderr\": result.stderr,\n \"timestamp\": datetime.now().isoformat()\n }\n \n if capture:\n output[\"file\"] = self._persist_output(output)\n \n return output\n \n def _persist_output(self, output: dict) -> str:\n \"\"\"Write output to terminal file.\"\"\"\n filename = f\"{self.session_counter}.txt\"\n file_path = self.terminals_path / filename\n \n content = f\"\"\"---\ncommand: {output['command']}\nexit_code: {output['exit_code']}\ntimestamp: {output['timestamp']}\n---\n\n=== STDOUT ===\n{output['stdout']}\n\n=== STDERR ===\n{output['stderr']}\n\"\"\"\n file_path.write_text(content)\n return str(file_path)\n \n def grep_terminals(self, pattern: str, context_lines: int = 3) -> List[dict]:\n \"\"\"Search all terminal outputs for pattern.\"\"\"\n matches = []\n regex = re.compile(pattern, re.IGNORECASE)\n \n for term_file in self.terminals_path.glob(\"*.txt\"):\n content = term_file.read_text()\n lines = content.split('\\n')\n \n for i, line in enumerate(lines):\n if regex.search(line):\n start = max(0, i - context_lines)\n end = min(len(lines), i + context_lines + 1)\n matches.append({\n \"file\": str(term_file),\n \"line_number\": i + 1,\n \"context\": '\\n'.join(lines[start:end])\n })\n \n return matches\n```\n\n### 6. Self-Modification Guard\n\nSafe pattern for agent self-learning.\n\n```python\nimport yaml\nfrom pathlib import Path\nfrom datetime import datetime\nfrom typing import Any\n\nclass PreferenceStore:\n \"\"\"Guarded storage for agent-learned preferences.\"\"\"\n \n MAX_ENTRIES = 100\n MAX_VALUE_LENGTH = 1000\n \n def __init__(self, path: str = \"agent/preferences.yaml\"):\n self.path = Path(path)\n self.path.parent.mkdir(parents=True, exist_ok=True)\n self.preferences = self._load()\n \n def _load(self) -> dict:\n \"\"\"Load preferences from file.\"\"\"\n if self.path.exists():\n return yaml.safe_load(self.path.read_text()) or {}\n return {}\n \n def _save(self):\n \"\"\"Persist preferences to file.\"\"\"\n self.path.write_text(yaml.dump(self.preferences, default_flow_style=False))\n \n def remember(self, key: str, value: Any, source: str = \"user\"):\n \"\"\"Store a preference with validation.\"\"\"\n # Validate key\n if not key or len(key) > 100:\n raise ValueError(\"Invalid key length\")\n \n # Validate value\n value_str = str(value)\n if len(value_str) > self.MAX_VALUE_LENGTH:\n raise ValueError(f\"Value exceeds max length of {self.MAX_VALUE_LENGTH}\")\n \n # Check entry limit\n if len(self.preferences) >= self.MAX_ENTRIES and key not in self.preferences:\n raise ValueError(f\"Max entries ({self.MAX_ENTRIES}) reached\")\n \n # Store with metadata\n self.preferences[key] = {\n \"value\": value,\n \"source\": source,\n \"updated_at\": datetime.now().isoformat()\n }\n self._save()\n \n def recall(self, key: str, default: Any = None) -> Any:\n \"\"\"Retrieve a preference.\"\"\"\n entry = self.preferences.get(key)\n if entry:\n return entry[\"value\"]\n return default\n \n def list_all(self) -> dict:\n \"\"\"Get all preferences for context injection.\"\"\"\n return {k: v[\"value\"] for k, v in self.preferences.items()}\n \n def forget(self, key: str):\n \"\"\"Remove a preference.\"\"\"\n if key in self.preferences:\n del self.preferences[key]\n self._save()\n```\n\n## Integration Example\n\nCombining patterns in an agent harness:\n\n```python\nclass FilesystemContextAgent:\n \"\"\"Agent with filesystem-based context management.\"\"\"\n \n def __init__(self):\n self.scratch = ScratchPadManager()\n self.skills = SkillLoader()\n self.preferences = PreferenceStore()\n self.workspace = AgentWorkspace(\"main_agent\")\n \n def handle_tool_output(self, tool_name: str, output: str) -> str:\n \"\"\"Process tool output, offloading if necessary.\"\"\"\n if self.scratch.should_offload(output):\n ref = self.scratch.offload(output, source=tool_name)\n return f\"[{tool_name} output saved to {ref['path']}. Summary: {ref['summary'][:200]}]\"\n return output\n \n def get_system_prompt(self) -> str:\n \"\"\"Build system prompt with dynamic skill references.\"\"\"\n base_prompt = \"You are a helpful assistant.\"\n skill_context = self.skills.get_static_context()\n user_prefs = self.preferences.list_all()\n \n pref_section = \"\"\n if user_prefs:\n pref_section = \"\\n\\nUser preferences:\\n\" + \"\\n\".join(\n f\"- {k}: {v}\" for k, v in user_prefs.items()\n )\n \n return f\"{base_prompt}\\n\\n{skill_context}{pref_section}\"\n```\n\n## File Organization Best Practices\n\n```\nproject/\n├── scratch/ # Ephemeral working files\n│ ├── tool_outputs/ # Large tool results\n│ │ └── search_20260107.txt\n│ └── plans/ # Active task plans\n│ └── current_plan.yaml\n├── workspace/ # Agent workspaces\n│ └── agents/\n│ ├── research_agent/\n│ │ ├── findings.md\n│ │ └── status.json\n│ └── code_agent/\n│ ├── findings.md\n│ └── status.json\n├── agent/ # Agent configuration\n│ ├── preferences.yaml # Learned preferences\n│ └── patterns.md # Discovered patterns\n├── skills/ # Loadable skills\n│ └── {skill-name}/\n│ └── SKILL.md\n├── terminals/ # Terminal output\n│ ├── 1.txt\n│ └── 2.txt\n└── history/ # Chat history archives\n └── session_001.txt\n```\n\n## Token Accounting Metrics\n\nTrack these metrics to validate filesystem patterns:\n\n1. **Static context ratio**: tokens in static context / total tokens\n2. **Dynamic load rate**: how often skills/files are loaded per task\n3. **Offload savings**: tokens saved by writing to files vs keeping in context\n4. **Retrieval precision**: percentage of loaded content actually used\n\nTarget benchmarks:\n- Static context ratio < 20%\n- Offload savings > 50% for tool-heavy workflows\n- Retrieval precision > 70% (loaded content is relevant)\n\n","isInternal":false,"tokens":3859,"sizeBytes":18228},{"name":"SKILL.md","path":"skills/filesystem-context/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/filesystem-context/SKILL.md","title":"filesystem-context","category":"anthropic-skill","format":"markdown","content":"---\nname: filesystem-context\ndescription: \"This skill should be used when agent work needs file-backed context: durable scratchpads, tool-output offloading, just-in-time discovery, cross-agent handoff files, filesystem memory, or cleanup policies for context stored outside the prompt.\"\n---\n\n# Filesystem-Based Context Engineering\n\nUse the filesystem as the primary overflow layer for agent context because context windows are limited while tasks often require more information than fits in a single window. Files let agents store, retrieve, and update an effectively unlimited amount of context through a single interface.\n\nPrefer dynamic context discovery -- pulling relevant context on demand -- over static inclusion, because static context consumes tokens regardless of relevance and crowds out space for task-specific information.\n\n## When to Activate\n\nActivate this skill when:\n- Tool outputs are bloating the context window\n- Agents need to persist state across long trajectories\n- Sub-agents must share information without direct message passing\n- Tasks require more context than fits in the window\n- Building agents that learn and update their own instructions\n- Implementing scratch pads for intermediate results\n- Terminal outputs or logs need to be accessible to agents\n\nDo not activate this skill for adjacent work owned by other skills:\n- Semantic cross-session memory, entity tracking, or temporal knowledge graphs: `memory-systems`.\n- Conversation summarization, compaction, or durable handoff wording: `context-compression`.\n- Token-efficiency tactics that do not require file-backed storage: `context-optimization`.\n- Multi-agent topology or handoff protocol design: `multi-agent-patterns`.\n- Model edits that evict content from the live window itself: `self-managed-context`.\n\n## Core Concepts\n\nDiagnose context failures against these four modes, because each requires a different filesystem remedy:\n\n1. **Missing context** -- needed information is absent from the total available context. Fix by persisting tool outputs and intermediate results to files so nothing is lost.\n2. **Under-retrieved context** -- retrieved content fails to encapsulate what the agent needs. Fix by structuring files for targeted retrieval (grep-friendly formats, clear section headers).\n3. **Over-retrieved context** -- retrieved content far exceeds what is needed, wasting tokens and degrading attention. Fix by offloading bulk content to files and returning compact references.\n4. **Buried context** -- niche information is hidden across many files. Fix by combining glob and grep for structural search alongside semantic search for conceptual queries.\n\nUse the filesystem as the persistent layer that addresses all four: write once, store durably, retrieve selectively.\n\n## Detailed Topics\n\n### The Static vs Dynamic Context Trade-off\n\nTreat static context (system instructions, tool definitions, critical rules) as expensive real estate -- it consumes tokens on every turn regardless of relevance. As agents accumulate capabilities, static context grows and crowds out dynamic information.\n\nUse dynamic context discovery instead: include only minimal static pointers (names, one-line descriptions, file paths) and load full content with search tools when relevant. This is more token-efficient and often improves response quality by reducing contradictory or irrelevant information in the window.\n\nAccept the trade-off: dynamic discovery requires the model to recognize when it needs more context. Current frontier models handle this well, but less capable models may fail to trigger loads. When in doubt, err toward including critical safety or correctness constraints statically.\n\n### Pattern 1: Filesystem as Scratch Pad\n\nRedirect large tool outputs to files instead of returning them directly to context, because a single web search or database query can dump thousands of tokens into message history where they persist for the entire conversation.\n\nWrite the output to a scratch file, extract a compact summary, and return a file reference. The agent then uses targeted retrieval (grep for patterns, read with line ranges) to access only what it needs.\n\n```python\ndef handle_tool_output(output: str, threshold: int = 2000) -> str:\n if len(output) < threshold:\n return output\n\n file_path = f\"scratch/{tool_name}_{timestamp}.txt\"\n write_file(file_path, output)\n\n key_summary = extract_summary(output, max_tokens=200)\n return f\"[Output written to {file_path}. Summary: {key_summary}]\"\n```\n\nUse grep to search the offloaded file and read_file with line ranges to retrieve targeted sections, because this preserves full output for later reference while keeping only ~100 tokens in the active context.\n\n### Pattern 2: Plan Persistence\n\nWrite plans to the filesystem because long-horizon tasks lose coherence when plans fall out of attention or get summarized away. The agent re-reads its plan at any point, restoring awareness of the objective and progress.\n\nStore plans in structured format so they are both human-readable and machine-parseable:\n```yaml\n# scratch/current_plan.yaml\nobjective: \"Refactor authentication module\"\nstatus: in_progress\nsteps:\n - id: 1\n description: \"Audit current auth endpoints\"\n status: completed\n - id: 2\n description: \"Design new token validation flow\"\n status: in_progress\n - id: 3\n description: \"Implement and test changes\"\n status: pending\n```\n\nRe-read the plan at the start of each turn or after any context refresh to re-orient, because this acts as \"manipulating attention through recitation.\"\n\n### Pattern 3: Sub-Agent Communication via Filesystem\n\nRoute sub-agent findings through the filesystem instead of message passing, because multi-hop message chains degrade information through summarization at each hop (\"game of telephone\").\n\nHave each sub-agent write directly to its own workspace directory. The coordinator reads these files directly, preserving full fidelity:\n```\nworkspace/\n agents/\n research_agent/\n findings.md\n sources.jsonl\n code_agent/\n changes.md\n test_results.txt\n coordinator/\n synthesis.md\n```\n\nEnforce per-agent directory isolation to prevent write conflicts and maintain clear ownership of each output artifact.\n\n### Pattern 4: Dynamic Skill Loading\n\nStore skills as files and include only skill names with brief descriptions in static context, because stuffing all instructions into the system prompt wastes tokens and can confuse the model with contradictory guidance.\n\n```\nAvailable skills (load with read_file when relevant):\n- database-optimization: Query tuning and indexing strategies\n- api-design: REST/GraphQL best practices\n- testing-strategies: Unit, integration, and e2e testing patterns\n```\n\nLoad the full skill file (e.g., `skills/database-optimization/SKILL.md`) only when the current task requires it. This converts O(n) static token cost into O(1) per task.\n\n### Pattern 5: Terminal and Log Persistence\n\nPersist terminal output to files automatically and use grep for selective retrieval, because terminal output from long-running processes accumulates rapidly and manual copy-paste is error-prone.\n\n```\nterminals/\n 1.txt # Terminal session 1 output\n 2.txt # Terminal session 2 output\n```\n\nQuery with targeted grep (`grep -A 5 \"error\" terminals/1.txt`) instead of loading entire terminal histories into context.\n\n### Pattern 6: Learning Through Self-Modification\n\nHave agents write learned preferences and patterns to their own instruction files so subsequent sessions load this context automatically, instead of requiring manual system prompt updates.\n\n```python\ndef remember_preference(key: str, value: str):\n preferences_file = \"agent/user_preferences.yaml\"\n prefs = load_yaml(preferences_file)\n prefs[key] = value\n write_yaml(preferences_file, prefs)\n```\n\nGuard this pattern with validation because self-modification can accumulate incorrect or contradictory instructions over time. Treat it as experimental -- review persisted preferences periodically.\n\n### Filesystem Search Techniques\n\nCombine `ls`/`list_dir`, `glob`, `grep`, and `read_file` with line ranges for context discovery, because models are specifically trained on filesystem traversal and this combination often outperforms semantic search for technical content where structural patterns are clear.\n\n- `ls` / `list_dir`: Discover directory structure\n- `glob`: Find files matching patterns (e.g., `**/*.py`)\n- `grep`: Search file contents, returns matching lines with context\n- `read_file` with ranges: Read specific sections without loading entire files\n\nUse filesystem search for structural and exact-match queries, and semantic search for conceptual queries. Combine both for comprehensive discovery.\n\n## Practical Guidance\n\n### When to Use Filesystem Context\n\nApply filesystem patterns when the situation matches these criteria, because they add I/O overhead that is only justified by token savings or persistence needs:\n\n**Use when:**\n- Tool outputs exceed ~2000 tokens\n- Tasks span multiple conversation turns\n- Multiple agents need shared state\n- Skills or instructions exceed comfortable system prompt size\n- Logs or terminal output need selective querying\n\n**Avoid when:**\n- Tasks complete in single turns (overhead not justified)\n- Context fits comfortably in window (no problem to solve)\n- Latency is critical (file I/O adds measurable delay)\n- Model lacks filesystem tool capabilities\n\n### File Organization\n\nStructure files for agent discoverability, because agents navigate by listing and reading directory names:\n```\nproject/\n scratch/ # Temporary working files\n tool_outputs/ # Large tool results\n plans/ # Active plans and checklists\n memory/ # Persistent learned information\n preferences.yaml # User preferences\n patterns.md # Learned patterns\n skills/ # Loadable skill definitions\n agents/ # Sub-agent workspaces\n```\n\nUse consistent naming conventions and include timestamps or IDs in scratch files for disambiguation.\n\nFor autonomous research loops, store raw retrieved evidence under the run that consumed it, for example `researcher/runs//sources/evidence/raw/`. Do not leave raw research dumps in the repository root; root-level artifacts become hard to audit and easy to cite without provenance.\n\n### Token Accounting\n\nMeasure where tokens originate before and after applying filesystem patterns, because optimizing without measurement leads to wasted effort:\n- Track static vs dynamic context ratio\n- Monitor tool output sizes before and after offloading\n- Measure how often dynamically-loaded context is actually used\n\n## Examples\n\n**Example 1: Tool Output Offloading**\n```\nInput: Web search returns 8000 tokens\nBefore: 8000 tokens added to message history\nAfter:\n - Write to scratch/search_results_001.txt\n - Return: \"[Results in scratch/search_results_001.txt. Key finding: API rate limit is 1000 req/min]\"\n - Agent greps file when needing specific details\nResult: ~100 tokens in context, 8000 tokens accessible on demand\n```\n\n**Example 2: Dynamic Skill Loading**\n```\nInput: User asks about database indexing\nStatic context: \"database-optimization: Query tuning and indexing\"\nAgent action: read_file(\"skills/database-optimization/SKILL.md\")\nResult: Full skill loaded only when relevant\n```\n\n**Example 3: Chat History as File Reference**\n```\nTrigger: Context window limit reached, summarization required\nAction:\n 1. Write full history to history/session_001.txt\n 2. Generate summary for new context window\n 3. Include reference: \"Full history in history/session_001.txt\"\nResult: Agent can search history file to recover details lost in summarization\n```\n\n## Guidelines\n\n1. Write large outputs to files; return summaries and references to context\n2. Store plans and state in structured files for re-reading\n3. Use sub-agent file workspaces instead of message chains\n4. Load skills dynamically rather than stuffing all into system prompt\n5. Persist terminal and log output as searchable files\n6. Combine grep/glob with semantic search for comprehensive discovery\n7. Organize files for agent discoverability with clear naming\n8. Measure token savings to validate filesystem patterns are effective\n9. Implement cleanup for scratch files to prevent unbounded growth\n10. Guard self-modification patterns with validation\n11. Keep raw evidence next to the run, evaluation, and proposal that used it\n\n## Gotchas\n\n1. **Scratch directory unbounded growth**: Agents create temp files without cleanup, eventually consuming disk and making directory listings noisy. Implement a retention policy (age-based or count-based) and run cleanup at session boundaries.\n2. **Race conditions in multi-agent file access**: Concurrent writes to the same file corrupt state silently. Enforce per-agent directory isolation or use append-only files with agent-prefixed entries.\n3. **Stale file references after moves/renames**: Agents hold paths from prior turns that no longer exist after refactors or file reorganization. Always verify file existence before reading a cached path; re-discover with glob if the check fails.\n4. **Glob pattern false matches**: Overly broad patterns (e.g., `**/*`) pull irrelevant files into context, wasting tokens and confusing the model. Scope globs to specific directories and extensions.\n5. **File size assumptions**: Reading a file without checking size can dump 100K+ tokens into context in a single tool call. Check file size before reading; use line-range reads for large files.\n6. **Missing file existence checks**: Agents assume files exist from prior turns, but they may have been deleted or moved. Always guard reads with existence checks and handle missing-file errors gracefully.\n7. **Scratch pad format drift**: Unstructured scratch pads become unparseable after many writes because format conventions erode over successive appends. Define and enforce a schema (YAML, JSON, or structured markdown) from the first write.\n8. **Hardcoded absolute paths**: Break when repositories are checked out at different locations or when running in containers. Use relative paths from the project root or resolve paths dynamically.\n\n## Integration\n\nThis skill owns file-backed context storage and retrieval. Adjacent skills own semantic memory, summarization, and topology:\n\n- `context-optimization`: filesystem offloading is one implementation of observation masking when full outputs remain retrievable.\n- `memory-systems`: use when file-backed notes are no longer enough and semantic, entity, or temporal retrieval is required.\n- `multi-agent-patterns`: sub-agent file workspaces enable context isolation and direct handoff.\n- `context-compression`: file references can anchor summaries and preserve details omitted from compressed context.\n- `tool-design`: tools should return file references for large outputs and expose safe read/search operations.\n- `self-managed-context`: offloading leaves the original in the live window; evicting it requires the model to edit its own context.\n\n## References\n\nInternal reference:\n- [Implementation Patterns](./references/implementation-patterns.md) - Read when: implementing scratch pad, plan persistence, or tool output offloading and need concrete code beyond the inline examples\n\nRelated skills in this collection:\n- context-optimization - Read when: applying token reduction techniques alongside filesystem offloading\n- memory-systems - Read when: building persistent storage that outlasts a single session\n- multi-agent-patterns - Read when: designing agent coordination with shared file workspaces\n\nExternal resources:\n- LangChain Deep Agents — Read when: implementing filesystem-based context patterns in LangChain/LangGraph pipelines\n- Cursor context discovery — Read when: studying how production IDEs implement dynamic context loading\n- Anthropic Agent Skills specification — Read when: building skills that leverage filesystem progressive disclosure\n\n---\n\n## Skill Metadata\n\n**Created**: 2026-01-07\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 1.2.0\n","frontmatter":{"name":"filesystem-context","description":"This skill should be used when agent work needs file-backed context: durable scratchpads, tool-output offloading, just-in-time discovery, cross-agent handoff files, filesystem memory, or cleanup policies for context stored outside the prompt."},"isInternal":false,"tokens":3203,"sizeBytes":16138},{"name":"SKILL.md","path":"skills/harness-engineering/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/harness-engineering/SKILL.md","title":"harness-engineering","category":"anthropic-skill","format":"markdown","content":"---\nname: harness-engineering\ndescription: \"This skill should be used when designing autonomous agent harnesses: research loops, evaluation scaffolds, locked and editable surfaces, durable logs, novelty gates, pruning, rollback, PR preparation, and human approval boundaries.\"\n---\n\n# Harness Engineering\n\nHarness engineering designs the control system around an agent: what it may edit, how it receives feedback, where it writes state, how failures recover, and who can approve irreversible actions. The harness is the difference between a helpful agent session and an autonomous loop that can run for days without corrupting its objective.\n\n## When to Activate\n\nActivate this skill when:\n\n- Building autonomous research or experimentation loops\n- Designing an agent environment with locked metrics and editable code or content\n- Creating PR-producing or background agents\n- Evaluating whether an agent can safely run without frequent human prompts\n- Adding novelty, ablation, pruning, rollback, or durable logging to an agent workflow\n- Preventing agents from gaming benchmarks, weakening rubrics, or losing state across compaction\n\nDo not activate this skill for adjacent work owned by other skills:\n- General quality gates, regression suites, or outcome metrics without autonomous control surfaces: `evaluation`.\n- Tool schemas, response formats, and recovery errors for harness tools: `tool-design`.\n- Project-level task-model fit, pipeline shape, and cost planning: `project-development`.\n- Remote sandbox, warm-pool, and hosted session infrastructure: `hosted-agents`.\n\n## Core Concepts\n\n### Harness Boundary\n\nSeparate the agent from the environment it operates inside. The agent proposes actions; the harness defines allowed surfaces, feedback, persistence, and promotion rules.\n\nUse four surface classes:\n\n| Surface | Examples | Rule |\n| --- | --- | --- |\n| Locked | Eval metric, rubric, validation script, merge policy | Agent may read and propose changes, but cannot score itself with modified rules |\n| Editable | Skill draft, experiment file, prompt, config under test | Agent may mutate during the loop |\n| Append-only | Results log, research thread, rejected ideas | Agent may append, not rewrite |\n| Human-controlled | Merge, production deploy, credentials, destructive operations | Requires explicit human approval |\n\n### Tight Feedback Loops\n\nAutonomy works when feedback is fast, unambiguous, and hard to game. Karpathy's `autoresearch` is the minimal pattern: one editable file, one locked evaluation file, fixed wall-clock budget, one scalar metric, git rollback, and a durable results log. The lesson is not that every harness needs one metric; it is that ambiguous feedback creates ambiguous autonomy.\n\nFor open-ended research-to-skill work, replace the scalar metric with locked rubrics, deterministic structure checks, source traceability, and human review thresholds.\n\n### Durable State\n\nLong-running agents must externalize state. Store plans, source queues, results, failures, and handoffs in files so future agents can resume without relying on chat history. Prime Intellect's autonomous nanoGPT work showed the value of durable scratchpads and `THREAD.md`-style logs for recovery, monitoring, and audit.\n\nUse append-only logs for:\n\n- What was tried\n- What improved or failed\n- Why a candidate was kept, discarded, or routed to review\n- Which upstream sources were checked\n- What the next agent should do\n\n### Search Discipline\n\nAgents tend to exploit the nearest surface, stack complexity, and under-run pruning. Add explicit search rules:\n\n1. Refresh upstream sources on a schedule.\n2. Require novelty checks before spending large budgets.\n3. Preserve rejected attempts to avoid rediscovery.\n4. Run leave-one-out pruning when a stack has multiple additions.\n5. Reward simplification when quality is equal.\n6. Use separate verification before promotion.\n\n### Mechanism Registry\n\nFor research-to-skill systems, track accepted mechanisms separately from prose. A mechanism record should include a stable `mechanism_id`, `owning_skill`, `status`, activation scenario, behavior change, evidence, and failure modes. Novelty gates should compare against this registry before using broader corpus overlap, because keyword overlap catches stale phrasing while mechanism comparison catches real duplication.\n\n### Governance\n\nAutonomous agents may prepare PRs, but governance must be explicit. They can draft changes, run checks, and write PR summaries. They should not merge, deploy, or push without human approval unless the user has explicitly granted that permission for the specific action.\n\n## Detailed Topics\n\n### Autoresearch-Style Loop\n\nUse this pattern when optimizing an artifact against a stable evaluator:\n\n```text\nread locked context -> choose hypothesis -> edit allowed surface -> commit/checkpoint\n-> run evaluator -> log result -> keep if better -> discard or rollback if worse\n-> repeat\n```\n\nRequired properties:\n\n- The evaluator is outside the editable surface.\n- The feedback cadence is fixed enough to compare attempts.\n- Failed attempts leave an audit trail.\n- Rollback is cheap.\n- The agent has a policy for crashes and timeouts.\n\n### Research-To-Skill Loop\n\nUse this pattern when sources become skill changes:\n\n```text\ndiscover -> retrieve -> gate -> score -> extract mechanism\n-> map to existing or new skill -> draft proposal -> validate structure\n-> prepare PR -> human review\n```\n\nThe locked evaluator is a combination of source rubrics, skill-change rubrics, structure checks, and reviewer approval. The editable artifact is the proposed skill delta.\n\n### Metric Gaming Resistance\n\nAssume an optimizing agent will learn the harness. Guard against:\n\n- Editing evaluation code or rubrics and then using the new version for self-approval\n- Adding verbose content that pleases a judge but harms skill activation\n- Citing unretrieved sources\n- Optimizing aggregate scores while failing a critical dimension\n- Avoiding failed results in the log\n\nMitigation: lock rubrics per run, report per-dimension scores, require source retrieval evidence, preserve rejected attempts, and route governance changes to human review.\n\n### Monitoring Agents\n\nUse monitoring agents for long runs, but restrict them to read-only reporting unless explicitly tasked otherwise. Monitoring output should report:\n\n- Best current candidate\n- Active jobs or drafts\n- Last upstream refresh\n- Failed or stale loops\n- Disagreements between logs and claimed state\n- Next action and blocker\n\n## Practical Guidance\n\n### Harness Design Checklist\n\n1. Define the objective in one sentence.\n2. Identify locked, editable, append-only, and human-controlled surfaces.\n3. Choose the feedback mechanism: scalar metric, rubric, deterministic tests, human review, or combination.\n4. Define keep, discard, crash, timeout, and review states.\n5. Create a durable thread log before the loop starts.\n6. Add source refresh, mechanism-registry novelty, and pruning rules for long-running loops.\n7. Define what the agent may do without asking and what requires approval.\n8. Validate the harness on one known good and one known bad artifact.\n\n### File Layout\n\n```text\nresearch-run/\n THREAD.md\n sources/\n queue.md\n evaluations/\n proposals/\n logs/\n results.tsv\n rejected.md\n drafts/\n```\n\nUse TSV or JSONL for append-only machine-readable logs. Use Markdown for handoffs and reviewer-facing summaries.\n\n## Examples\n\n**Example 1: Locked metric**\n\nAn agent optimizes `train.py`, but `prepare.py` owns data loading and evaluation. The agent can edit the model but cannot change the metric. Failed experiments are logged and rolled back.\n\n**Example 2: Locked rubric**\n\nAn agent evaluates a new Anthropic or OpenAI engineering post, but the source curation rubric is locked for the run. If the source passes, the agent drafts a skill proposal. It cannot lower the rubric threshold to admit the source.\n\n**Example 3: Auto-PR without auto-merge**\n\nAn agent prepares a branch and PR body after passing source, skill, and structure checks. The PR states unresolved risks and waits for human merge approval.\n\n## Guidelines\n\n1. Lock evaluators before starting the loop.\n2. Keep editable surfaces narrow enough for reliable diffs.\n3. Write durable logs before context compaction can erase state.\n4. Report per-dimension scores instead of only aggregate scores.\n5. Require source retrieval before citation.\n6. Add novelty gates for broad search and pruning gates for complex stacks.\n7. Prefer simplification when quality is equal.\n8. Separate PR preparation from merge authority.\n9. Revalidate harness changes with old and new evaluators.\n10. Treat stopped autonomous loops as harness failures, not agent personality quirks.\n\n## Gotchas\n\n1. **Mutable evaluator**: If the agent can edit the metric, it may optimize the benchmark instead of the task. Keep rubrics and eval code locked during the run.\n2. **Chat-only memory**: Long runs fail after compaction when plans live only in conversation history. Write thread logs and result files from the start.\n3. **No discard record**: Without rejected-attempt logs, agents repeat failed ideas. Preserve failures with enough detail to avoid rediscovery.\n4. **Complexity accretion**: Agents stack changes and rarely remove them. Require pruning rounds and reward equal-quality simplification.\n5. **Premature novelty claims**: Agents label recombinations as novel. Compare against existing repo skills, source queue, and rejected logs before claiming novelty.\n6. **Monitor misreporting**: Monitoring agents can summarize stale or inconsistent state. Require them to cite the files or logs behind claims.\n7. **Human approval ambiguity**: \"Prepare a PR\" is not \"merge a PR.\" Make approval boundaries explicit in the harness.\n8. **Volatile source drift**: Fast-moving lab claims age quickly. Put dated evidence in references and schedule revalidation.\n\n## Integration\n\nThis skill connects to:\n\n- evaluation - Rubrics and quality gates provide the locked feedback surface\n- advanced-evaluation - Pairwise comparison and bias mitigation improve proposal review\n- filesystem-context - Durable logs, scratchpads, and thread files preserve state\n- multi-agent-patterns - Researcher, verifier, monitor, and writer agents need isolated contexts\n- tool-design - Harness tools must expose clear contracts and recovery errors\n- project-development - File-based pipelines and task-model fit analysis keep loops simple\n- hosted-agents - Background execution needs sandbox, snapshot, and approval boundaries\n\n## References\n\nInternal references:\n- `researcher/README.md` - Read when implementing the repo-native research-to-skill operating system\n- `researcher/rubrics/harness-change.md` - Read when evaluating changes to an agent harness\n- `researcher/runbooks/autonomous-research-loop.md` - Read when running a source-to-skill loop\n\nExternal resources:\n- Karpathy `autoresearch` - Constrained autonomous experiment loop with locked evaluation\n- Prime Intellect autonomous nanoGPT speedrun - Durable scratchpads, handoffs, monitoring, and autonomy failure modes\n- AlphaEvolve and FunSearch - LLM-generated candidates paired with systematic evaluators\n- HELM and LM Evaluation Harness - Transparent, reproducible evaluation infrastructure\n\n---\n\n## Skill Metadata\n\n**Created**: 2026-05-14\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 1.1.0\n","frontmatter":{"name":"harness-engineering","description":"This skill should be used when designing autonomous agent harnesses: research loops, evaluation scaffolds, locked and editable surfaces, durable logs, novelty gates, pruning, rollback, PR preparation, and human approval boundaries."},"isInternal":false,"tokens":2306,"sizeBytes":11413},{"name":"infrastructure-patterns.md","path":"skills/hosted-agents/references/infrastructure-patterns.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/hosted-agents/references/infrastructure-patterns.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Infrastructure Patterns for Hosted Agents\n\nThis reference provides detailed implementation patterns for building hosted agent infrastructure. These patterns are derived from production systems at scale.\n\n## Sandbox Architecture\n\n### Modal Integration Pattern\n\nModal provides the sandbox infrastructure with near-instant startup and filesystem snapshots.\n\n```python\nimport modal\n\n# Define the base image with all dependencies\nimage = modal.Image.debian_slim().pip_install([\n \"opencode\",\n \"gitpython\",\n \"psycopg2-binary\",\n])\n\n# Create the app\napp = modal.App(\"coding-agent\")\n\n# Sandbox class with snapshot support\n@app.cls(image=image, timeout=3600)\nclass AgentSandbox:\n def __init__(self, repo_url: str, snapshot_id: str = None):\n self.repo_url = repo_url\n self.snapshot_id = snapshot_id\n \n @modal.enter()\n def setup(self):\n if self.snapshot_id:\n # Restore from snapshot\n modal.Sandbox.restore(self.snapshot_id)\n else:\n # Fresh setup from image\n self._clone_and_setup()\n \n def _clone_and_setup(self):\n \"\"\"Clone repo and run initial setup.\"\"\"\n token = self._get_github_app_token()\n os.system(f\"git clone https://x-access-token:{token}@github.com/{self.repo_url}\")\n os.system(\"npm install\")\n os.system(\"npm run build\")\n \n @modal.method()\n def execute_prompt(self, prompt: str, user_identity: dict) -> dict:\n \"\"\"Execute a prompt in the sandbox.\"\"\"\n # Update git config for this user\n os.system(f'git config user.name \"{user_identity[\"name\"]}\"')\n os.system(f'git config user.email \"{user_identity[\"email\"]}\"')\n \n # Run the agent\n result = self.agent.run(prompt)\n \n return {\n \"result\": result,\n \"snapshot_id\": modal.Sandbox.snapshot()\n }\n```\n\n### Image Build Pipeline\n\nBuild images on a schedule to keep them fresh:\n\n```python\nimport schedule\nimport time\nfrom datetime import datetime\n\nclass ImageBuilder:\n def __init__(self, repositories: list[str]):\n self.repositories = repositories\n self.images = {}\n \n def build_all_images(self):\n \"\"\"Build images for all repositories.\"\"\"\n for repo in self.repositories:\n try:\n image = self._build_image(repo)\n self.images[repo] = {\n \"image\": image,\n \"built_at\": datetime.utcnow(),\n \"commit\": self._get_latest_commit(repo)\n }\n except Exception as e:\n # Log but continue with other repos\n log.error(f\"Failed to build image for {repo}: {e}\")\n \n def _build_image(self, repo: str) -> str:\n \"\"\"Build a single repository image.\"\"\"\n sandbox = modal.Sandbox.create()\n \n # Clone with app token\n token = get_app_installation_token(repo)\n sandbox.exec(f\"git clone https://x-access-token:{token}@github.com/{repo} /workspace\")\n \n # Install dependencies\n sandbox.exec(\"cd /workspace && npm install\")\n \n # Run build\n sandbox.exec(\"cd /workspace && npm run build\")\n \n # Warm caches\n sandbox.exec(\"cd /workspace && npm run dev &\")\n time.sleep(5) # Let dev server start\n sandbox.exec(\"cd /workspace && npm test -- --run\")\n \n # Create snapshot\n return sandbox.snapshot()\n \n def get_latest_image(self, repo: str) -> str:\n \"\"\"Get the most recent image for a repository.\"\"\"\n if repo not in self.images:\n raise ValueError(f\"No image available for {repo}\")\n return self.images[repo][\"image\"]\n\n# Schedule builds every 30 minutes\nbuilder = ImageBuilder([\"org/frontend\", \"org/backend\", \"org/shared\"])\nschedule.every(30).minutes.do(builder.build_all_images)\n```\n\n### Warm Pool Management\n\nMaintain pre-warmed sandboxes for instant session starts:\n\n```python\nfrom collections import defaultdict\nfrom dataclasses import dataclass\nfrom datetime import datetime, timedelta\n\n@dataclass\nclass WarmSandbox:\n sandbox_id: str\n repo: str\n created_at: datetime\n image_version: str\n is_claimed: bool = False\n\nclass WarmPoolManager:\n def __init__(self, target_pool_size: int = 3):\n self.target_size = target_pool_size\n self.pools = defaultdict(list) # repo -> [WarmSandbox]\n self.max_age = timedelta(minutes=25) # Expire before next image build\n \n def get_warm_sandbox(self, repo: str) -> WarmSandbox | None:\n \"\"\"Get a pre-warmed sandbox if available.\"\"\"\n pool = self.pools[repo]\n \n for sandbox in pool:\n if not sandbox.is_claimed and self._is_valid(sandbox):\n sandbox.is_claimed = True\n return sandbox\n \n return None\n \n def _is_valid(self, sandbox: WarmSandbox) -> bool:\n \"\"\"Check if sandbox is still valid.\"\"\"\n age = datetime.utcnow() - sandbox.created_at\n current_image = self.image_builder.get_latest_image(sandbox.repo)\n \n return (\n age < self.max_age and\n sandbox.image_version == current_image\n )\n \n def maintain_pool(self, repo: str):\n \"\"\"Ensure pool has target number of warm sandboxes.\"\"\"\n # Remove expired sandboxes\n self.pools[repo] = [s for s in self.pools[repo] if self._is_valid(s)]\n \n # Add new sandboxes to reach target\n current_count = len([s for s in self.pools[repo] if not s.is_claimed])\n needed = self.target_size - current_count\n \n for _ in range(needed):\n sandbox = self._create_warm_sandbox(repo)\n self.pools[repo].append(sandbox)\n \n def _create_warm_sandbox(self, repo: str) -> WarmSandbox:\n \"\"\"Create a new warm sandbox from latest image.\"\"\"\n image = self.image_builder.get_latest_image(repo)\n sandbox_id = modal.Sandbox.create(image=image)\n \n # Sync to latest (runs in background)\n self._sync_to_latest(sandbox_id, repo)\n \n return WarmSandbox(\n sandbox_id=sandbox_id,\n repo=repo,\n created_at=datetime.utcnow(),\n image_version=image\n )\n```\n\n## API Layer Patterns\n\n### Cloudflare Durable Objects for Session State\n\nEach session gets its own Durable Object with isolated SQLite:\n\n```typescript\n// Session Durable Object\nexport class SessionDO implements DurableObject {\n private storage: DurableObjectStorage;\n private sql: SqlStorage;\n private connections: Map = new Map();\n\n constructor(ctx: DurableObjectState) {\n this.storage = ctx.storage;\n this.sql = ctx.storage.sql;\n this.initializeSchema();\n }\n\n private initializeSchema() {\n this.sql.exec(`\n CREATE TABLE IF NOT EXISTS messages (\n id INTEGER PRIMARY KEY,\n role TEXT NOT NULL,\n content TEXT NOT NULL,\n author_id TEXT,\n author_name TEXT,\n created_at TEXT DEFAULT CURRENT_TIMESTAMP\n );\n \n CREATE TABLE IF NOT EXISTS artifacts (\n id INTEGER PRIMARY KEY,\n type TEXT NOT NULL,\n path TEXT,\n content TEXT,\n created_at TEXT DEFAULT CURRENT_TIMESTAMP\n );\n \n CREATE TABLE IF NOT EXISTS events (\n id INTEGER PRIMARY KEY,\n type TEXT NOT NULL,\n data TEXT,\n created_at TEXT DEFAULT CURRENT_TIMESTAMP\n );\n `);\n }\n\n async fetch(request: Request): Promise {\n const url = new URL(request.url);\n\n if (request.headers.get(\"Upgrade\") === \"websocket\") {\n return this.handleWebSocket(request);\n }\n\n switch (url.pathname) {\n case \"/message\":\n return this.handleMessage(request);\n case \"/status\":\n return this.getStatus();\n default:\n return new Response(\"Not found\", { status: 404 });\n }\n }\n\n private handleWebSocket(request: Request): Response {\n const pair = new WebSocketPair();\n const [client, server] = Object.values(pair);\n\n const connectionId = crypto.randomUUID();\n this.connections.set(connectionId, server);\n\n server.accept();\n server.addEventListener(\"close\", () => {\n this.connections.delete(connectionId);\n });\n\n return new Response(null, { status: 101, webSocket: client });\n }\n\n private broadcast(message: object) {\n const data = JSON.stringify(message);\n for (const ws of this.connections.values()) {\n ws.send(data);\n }\n }\n\n async handleMessage(request: Request): Promise {\n const { content, author } = await request.json();\n\n // Store message\n this.sql.exec(\n `INSERT INTO messages (role, content, author_id, author_name) VALUES (?, ?, ?, ?)`,\n [\"user\", content, author.id, author.name]\n );\n\n // Broadcast to all connected clients\n this.broadcast({\n type: \"message\",\n role: \"user\",\n content,\n author,\n });\n\n // Forward to sandbox for processing\n const result = await this.forwardToSandbox(content, author);\n\n return Response.json(result);\n }\n}\n```\n\n### Real-Time Event Streaming\n\nStream events from sandbox to all connected clients:\n\n```typescript\nclass EventStream {\n private sessionDO: DurableObjectStub;\n\n async streamFromSandbox(sandboxId: string, sessionId: string) {\n const sandbox = await modal.Sandbox.get(sandboxId);\n\n // Subscribe to sandbox events\n for await (const event of sandbox.events()) {\n // Forward to Durable Object for broadcast\n await this.sessionDO.fetch(\n new Request(`https://internal/event`, {\n method: \"POST\",\n body: JSON.stringify({\n type: event.type,\n data: event.data,\n }),\n })\n );\n }\n }\n}\n```\n\n## Client Integration Patterns\n\n### Slack Bot with Repository Classification\n\n```python\nfrom slack_bolt import App\nfrom slack_bolt.adapter.socket_mode import SocketModeHandler\n\napp = App(token=os.environ[\"SLACK_BOT_TOKEN\"])\n\n# Repository descriptions for classification\nREPO_DESCRIPTIONS = [\n {\n \"name\": \"frontend-monorepo\",\n \"description\": \"React frontend application with dashboard, user portal, and admin interfaces\",\n \"hints\": [\"dashboard\", \"UI\", \"component\", \"page\", \"frontend\"]\n },\n {\n \"name\": \"backend-services\",\n \"description\": \"Node.js API services including auth, payments, and core business logic\",\n \"hints\": [\"API\", \"endpoint\", \"service\", \"backend\", \"database\"]\n },\n {\n \"name\": \"mobile-app\",\n \"description\": \"React Native mobile application for iOS and Android\",\n \"hints\": [\"mobile\", \"app\", \"iOS\", \"Android\", \"native\"]\n }\n]\n\nasync def classify_repository(message: str, channel: str, thread: list[str]) -> str:\n \"\"\"Use fast model to classify which repo the message refers to.\"\"\"\n prompt = f\"\"\"Classify which repository this message is about.\n\nMessage: {message}\nChannel: #{channel}\nThread context: {' | '.join(thread[-3:])}\n\nRepositories:\n{json.dumps(REPO_DESCRIPTIONS, indent=2)}\n\nReturn ONLY the repository name, or \"unknown\" if unclear.\"\"\"\n\n response = await openai.chat.completions.create(\n model=\"gpt-4o-mini\",\n messages=[{\"role\": \"user\", \"content\": prompt}],\n max_tokens=50\n )\n \n return response.choices[0].message.content.strip()\n\n@app.event(\"app_mention\")\nasync def handle_mention(event, say, client):\n \"\"\"Handle @mentions of the bot.\"\"\"\n channel = event[\"channel\"]\n message = event[\"text\"]\n thread_ts = event.get(\"thread_ts\", event[\"ts\"])\n \n # Get thread context if in a thread\n thread_messages = []\n if \"thread_ts\" in event:\n result = await client.conversations_replies(\n channel=channel,\n ts=thread_ts\n )\n thread_messages = [m[\"text\"] for m in result[\"messages\"]]\n \n # Get channel info for context\n channel_info = await client.conversations_info(channel=channel)\n channel_name = channel_info[\"channel\"][\"name\"]\n \n # Classify repository\n repo = await classify_repository(message, channel_name, thread_messages)\n \n if repo == \"unknown\":\n await say(\n text=\"I'm not sure which repository you're referring to. Could you specify?\",\n thread_ts=thread_ts\n )\n return\n \n # Start session and process\n session = await start_session(repo, event[\"user\"])\n \n await say(\n text=f\":robot_face: Starting work in `{repo}`...\",\n thread_ts=thread_ts\n )\n \n result = await session.process(message)\n \n # Post result with Block Kit formatting\n await say(\n blocks=format_result_blocks(result),\n thread_ts=thread_ts\n )\n```\n\n### Chrome Extension DOM Extraction\n\nExtract DOM structure instead of sending screenshots:\n\n```typescript\n// content-script.ts\ninterface ElementInfo {\n tag: string;\n classes: string[];\n id?: string;\n text?: string;\n rect: DOMRect;\n reactComponent?: string;\n}\n\nfunction extractDOMInfo(element: Element): ElementInfo {\n // Get React component name if available\n let reactComponent: string | undefined;\n const fiberKey = Object.keys(element).find((key) =>\n key.startsWith(\"__reactFiber\")\n );\n if (fiberKey) {\n const fiber = (element as any)[fiberKey];\n reactComponent = fiber?.type?.name || fiber?.type?.displayName;\n }\n\n return {\n tag: element.tagName.toLowerCase(),\n classes: Array.from(element.classList),\n id: element.id || undefined,\n text: element.textContent?.slice(0, 100),\n rect: element.getBoundingClientRect(),\n reactComponent,\n };\n}\n\nfunction extractSelectedArea(selection: DOMRect): ElementInfo[] {\n const elements: ElementInfo[] = [];\n\n // Find all elements within selection bounds\n document.querySelectorAll(\"*\").forEach((el) => {\n const rect = el.getBoundingClientRect();\n if (\n rect.top >= selection.top &&\n rect.left >= selection.left &&\n rect.bottom <= selection.bottom &&\n rect.right <= selection.right\n ) {\n elements.push(extractDOMInfo(el));\n }\n });\n\n return elements;\n}\n\n// Message handler for sidebar\nchrome.runtime.onMessage.addListener((request, sender, sendResponse) => {\n if (request.type === \"EXTRACT_SELECTION\") {\n const elements = extractSelectedArea(request.selection);\n sendResponse({ elements });\n }\n});\n```\n\n## Multiplayer Implementation\n\n### Authorship Tracking\n\nTrack which user made each change:\n\n```python\n@dataclass\nclass PromptContext:\n content: str\n author: Author\n session_id: str\n timestamp: datetime\n\n@dataclass\nclass Author:\n id: str\n name: str\n email: str\n github_token: str # For PR creation\n\nclass MultiplayerSession:\n def __init__(self, session_id: str):\n self.session_id = session_id\n self.participants: dict[str, Author] = {}\n self.prompt_queue: list[PromptContext] = []\n \n def add_participant(self, author: Author):\n \"\"\"Add a participant to the session.\"\"\"\n self.participants[author.id] = author\n self.broadcast_event(\"participant_joined\", author)\n \n async def process_prompt(self, prompt: PromptContext):\n \"\"\"Process prompt with author attribution.\"\"\"\n # Update git config for this author\n await self.sandbox.exec(\n f'git config user.name \"{prompt.author.name}\"'\n )\n await self.sandbox.exec(\n f'git config user.email \"{prompt.author.email}\"'\n )\n \n # Run agent\n result = await self.agent.run(prompt.content)\n \n # If changes were made, create PR with author's token\n if result.has_changes:\n await self.create_pr(\n branch=result.branch,\n author=prompt.author\n )\n \n return result\n \n async def create_pr(self, branch: str, author: Author):\n \"\"\"Create PR using the author's GitHub token.\"\"\"\n async with aiohttp.ClientSession() as session:\n headers = {\n \"Authorization\": f\"Bearer {author.github_token}\",\n \"Accept\": \"application/vnd.github.v3+json\"\n }\n \n await session.post(\n f\"https://api.github.com/repos/{self.repo}/pulls\",\n headers=headers,\n json={\n \"title\": self.generate_pr_title(),\n \"body\": self.generate_pr_body(),\n \"head\": branch,\n \"base\": \"main\"\n }\n )\n```\n\n## Metrics and Monitoring\n\n### Key Metrics to Track\n\n```python\nfrom dataclasses import dataclass\nfrom datetime import datetime, timedelta\n\n@dataclass\nclass SessionMetrics:\n session_id: str\n started_at: datetime\n first_token_at: datetime | None\n completed_at: datetime | None\n pr_created: bool\n pr_merged: bool\n prompts_count: int\n participants_count: int\n \n @property\n def time_to_first_token(self) -> timedelta | None:\n if self.first_token_at:\n return self.first_token_at - self.started_at\n return None\n\nclass MetricsAggregator:\n def get_adoption_metrics(self, period: timedelta) -> dict:\n \"\"\"Get adoption metrics for a time period.\"\"\"\n sessions = self.get_sessions_in_period(period)\n \n total_prs = sum(1 for s in sessions if s.pr_created)\n merged_prs = sum(1 for s in sessions if s.pr_merged)\n \n return {\n \"total_sessions\": len(sessions),\n \"prs_created\": total_prs,\n \"prs_merged\": merged_prs,\n \"merge_rate\": merged_prs / total_prs if total_prs > 0 else 0,\n \"avg_time_to_first_token\": self._avg_ttft(sessions),\n \"unique_users\": len(set(s.author_id for s in sessions)),\n \"multiplayer_sessions\": sum(\n 1 for s in sessions if s.participants_count > 1\n )\n }\n \n def get_repository_metrics(self) -> dict[str, dict]:\n \"\"\"Get metrics broken down by repository.\"\"\"\n metrics = {}\n \n for repo in self.repositories:\n repo_sessions = self.get_sessions_for_repo(repo)\n total_prs = self.get_total_prs(repo)\n agent_prs = sum(1 for s in repo_sessions if s.pr_merged)\n \n metrics[repo] = {\n \"agent_pr_percentage\": agent_prs / total_prs * 100,\n \"session_count\": len(repo_sessions),\n \"avg_prompts_per_session\": sum(\n s.prompts_count for s in repo_sessions\n ) / len(repo_sessions)\n }\n \n return metrics\n```\n\n## Security Considerations\n\n### Sandbox Isolation\n\n```python\nclass SandboxSecurityConfig:\n \"\"\"Security configuration for sandboxes.\"\"\"\n \n # Network restrictions\n allowed_hosts = [\n \"github.com\",\n \"api.github.com\",\n \"registry.npmjs.org\",\n \"pypi.org\",\n ]\n \n # Resource limits\n max_memory_mb = 4096\n max_cpu_cores = 2\n max_disk_gb = 10\n max_runtime_hours = 4\n \n # Secrets handling\n secrets_to_inject = [\n \"GITHUB_APP_TOKEN\",\n \"NPM_TOKEN\",\n ]\n \n # Blocked operations\n blocked_commands = [\n \"curl\", # Use fetch tools instead\n \"wget\",\n \"ssh\",\n ]\n```\n\n### Token Handling\n\n```python\nclass TokenManager:\n \"\"\"Manage tokens for GitHub operations.\"\"\"\n \n def get_app_installation_token(self, repo: str) -> str:\n \"\"\"Get short-lived token for repo access.\"\"\"\n # Token expires in 1 hour\n return github_app.create_installation_token(\n installation_id=self.get_installation_id(repo),\n permissions={\"contents\": \"write\", \"pull_requests\": \"write\"}\n )\n \n def get_user_token(self, user_id: str) -> str:\n \"\"\"Get user's OAuth token for PR creation.\"\"\"\n # Stored encrypted, decrypted at runtime\n encrypted = self.storage.get(f\"user_token:{user_id}\")\n return self.decrypt(encrypted)\n```\n\n## References\n\n- [Modal Documentation](https://modal.com/docs)\n- [Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/)\n- [Cloudflare Agents SDK](https://developers.cloudflare.com/agents/)\n- [GitHub Apps Authentication](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app)\n- [Slack Bolt for Python](https://slack.dev/bolt-python/)\n- [Chrome Extension APIs](https://developer.chrome.com/docs/extensions/)\n","isInternal":false,"tokens":4425,"sizeBytes":20449},{"name":"SKILL.md","path":"skills/hosted-agents/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/hosted-agents/SKILL.md","title":"hosted-agents","category":"anthropic-skill","format":"markdown","content":"---\nname: hosted-agents\ndescription: \"This skill should be used when designing hosted or background agent infrastructure: sandboxed execution, remote coding environments, warm pools, session persistence, multiplayer collaboration, self-spawning agents, or Modal-style sandboxes.\"\n---\n\n# Hosted Agent Infrastructure\n\nHosted agents run in remote sandboxed environments rather than on local machines. When designed well, they provide unlimited concurrency, consistent execution environments, and multiplayer collaboration. The critical insight is that session speed should be limited only by model provider time-to-first-token, with all infrastructure setup completed before the user starts their session.\n\n## When to Activate\n\nActivate this skill when:\n- Building background coding agents that run independently of user devices\n- Designing sandboxed execution environments for agent workloads\n- Implementing multiplayer agent sessions with shared state\n- Creating multi-client agent interfaces (Slack, Web, Chrome extensions)\n- Scaling agent infrastructure beyond local machine constraints\n- Building systems where agents spawn sub-agents for parallel work\n\nDo not activate this skill for adjacent work owned by other skills:\n- Designing the autonomous research loop, novelty gates, rollback policy, or merge boundaries: `harness-engineering`.\n- Choosing supervisor, swarm, or handoff topology without hosted infrastructure concerns: `multi-agent-patterns`.\n- Designing the tools used by a hosted agent, such as spawn/status tools or PR tools: `tool-design`.\n- Managing file-backed state inside a session rather than the hosted runtime itself: `filesystem-context`.\n\n## Core Concepts\n\nMove agent execution to remote sandboxed environments to eliminate the fundamental limits of local execution: resource contention, environment inconsistency, and single-user constraints. Remote sandboxes unlock unlimited concurrency, reproducible environments, and collaborative workflows because each session gets its own isolated compute with a known-good environment image.\n\nDesign the architecture in three layers because each layer scales independently. Build sandbox infrastructure for isolated execution, an API layer for state management and client coordination, and client interfaces for user interaction across platforms. Keep these layers cleanly separated so sandbox changes do not ripple into clients.\n\n## Detailed Topics\n\n### Sandbox Infrastructure\n\n**The Core Challenge**\nEliminate sandbox spin-up latency because users perceive anything over a few seconds as broken. Development environments require cloning repositories, installing dependencies, and running build steps -- do all of this before the user ever submits a prompt.\n\n**Image Registry Pattern**\nPre-build environment images on a regular cadence (every 30 minutes works well) because this makes synchronization with the latest code a fast delta rather than a full clone. Include in each image:\n- Cloned repository at a known commit\n- All runtime dependencies installed\n- Initial setup and build commands completed\n- Cached files from running app and test suite once\n\nWhen starting a session, spin up a sandbox from the most recent image. The repository is at most 30 minutes out of date, making the remaining git sync fast.\n\n**Snapshot and Restore**\nTake filesystem snapshots at key points to enable instant restoration for follow-up prompts without re-running setup:\n- After initial image build (base snapshot)\n- When agent finishes making changes (session snapshot)\n- Before sandbox exit for potential follow-up\n\n**Git Configuration for Background Agents**\nConfigure git identity explicitly in every sandbox because background agents are not tied to a specific user during image builds:\n- Generate GitHub app installation tokens for repository access during clone\n- Set git config `user.name` and `user.email` when committing and pushing changes\n- Use the prompting user's identity for commits, not the app identity\n\n**Warm Pool Strategy**\nMaintain a pool of pre-warmed sandboxes for high-volume repositories because cold starts are the primary source of user frustration:\n- Keep sandboxes ready before users start sessions\n- Expire and recreate pool entries as new image builds complete\n- Start warming a sandbox as soon as a user begins typing (predictive warm-up)\n\n### Agent Framework Selection\n\n**Server-First Architecture**\nStructure the agent framework as a server first, with TUI and desktop apps as thin clients, because this prevents duplicating agent logic across surfaces:\n- Multiple custom clients share one agent backend\n- Consistent behavior across all interaction surfaces\n- Plugin systems extend functionality without client changes\n- Event-driven architectures deliver real-time updates to any connected client\n\n**Code as Source of Truth**\nSelect frameworks where the agent can read its own source code to understand behavior. Prioritize this because having code as source of truth prevents the agent from hallucinating about its own capabilities -- an underrated failure mode in AI development.\n\n**Plugin System Requirements**\nRequire a plugin system that supports runtime interception because this enables safety controls and observability without modifying core agent logic:\n- Listen to tool execution events (e.g., `tool.execute.before`)\n- Block or modify tool calls conditionally\n- Inject context or state at runtime\n\n### Speed Optimizations\n\n**Predictive Warm-Up**\nStart warming the sandbox as soon as a user begins typing their prompt, not when they submit it, because the typing interval (5-30 seconds) is enough to complete most setup:\n- Clone latest changes in parallel with user typing\n- Run initial setup before user hits enter\n- For fast spin-up, sandbox can be ready before user finishes typing\n\n**Parallel File Reading**\nAllow the agent to start reading files immediately even if sync from latest base branch is not complete, because in large repositories incoming prompts rarely touch recently-changed files:\n- Agent can research immediately without waiting for git sync\n- Block file edits (not reads) until synchronization completes\n- This separation is safe because read-time data staleness of 30 minutes rarely matters for research\n\n**Maximize Build-Time Work**\nMove everything possible to the image build step because build-time duration is invisible to users:\n- Full dependency installation\n- Database schema setup\n- Initial app and test suite runs (populates caches)\n\n### Self-Spawning Agents\n\n**Agent-Spawned Sessions**\nBuild tools that allow agents to spawn new sessions because frontier models are capable of decomposing work and coordinating sub-tasks:\n- Research tasks across different repositories\n- Parallel subtask execution for large changes\n- Multiple smaller PRs from one major task\n\nExpose three primitives: start a new session with specified parameters, read status of any session (check-in capability), and continue main work while sub-sessions run in parallel.\n\n**Prompt Engineering for Self-Spawning**\nEngineer prompts that guide when agents should spawn sub-sessions rather than doing work inline:\n- Research tasks that require cross-repository exploration\n- Breaking monolithic changes into smaller PRs\n- Parallel exploration of different approaches\n\n### API Layer\n\n**Per-Session State Isolation**\nIsolate state per session (SQLite per session works well) because cross-session interference is a subtle and hard-to-debug failure mode:\n- Dedicated database per session\n- No session can impact another's performance\n- Architecture handles hundreds of concurrent sessions\n\n**Real-Time Streaming**\nStream all agent work in real-time because high-frequency feedback is critical for user trust:\n- Token streaming from model providers\n- Tool execution status updates\n- File change notifications\n\nUse WebSocket connections with hibernation APIs to reduce compute costs during idle periods while maintaining open connections.\n\n**Synchronization Across Clients**\nBuild a single state system that synchronizes across all clients (chat interfaces, Slack bots, Chrome extensions, web interfaces, VS Code instances) because users switch surfaces frequently and expect continuity. All changes sync to the session state, enabling seamless client switching.\n\n### Multiplayer Support\n\n**Why Multiplayer Matters**\nDesign for multiplayer from day one because it is nearly free to add with proper synchronization architecture, and it unlocks high-value workflows:\n- Teaching non-engineers to use AI effectively\n- Live QA sessions with multiple team members\n- Real-time PR review with immediate changes\n- Collaborative debugging sessions\n\n**Implementation Requirements**\nBuild the data model so sessions are not tied to single authors because multiplayer fails silently if authorship is hardcoded:\n- Pass authorship info to each prompt\n- Attribute code changes to the prompting user\n- Share session links for instant collaboration\n\n### Authentication and Authorization\n\n**User-Based Commits**\nUse GitHub authentication to open PRs on behalf of the user (not the app) because this preserves the audit trail and prevents users from approving their own AI-generated changes:\n- Obtain user tokens for PR creation\n- PRs appear as authored by the human, not the bot\n\n**Sandbox-to-API Flow**\nFollow this sequence because it keeps sandbox permissions minimal while letting the API handle sensitive operations:\n1. Sandbox pushes changes (updating git user config)\n2. Sandbox sends event to API with branch name and session ID\n3. API uses user's GitHub token to create PR\n4. GitHub webhooks notify API of PR events\n\n### Client Implementations\n\n**Slack Integration**\nPrioritize Slack as the first distribution channel for internal adoption because it creates a virality loop as team members see others using it:\n- No syntax required, natural chat interface\n- Build a classifier (fast model with repo descriptions) to determine which repository to work in\n- Include hints for common repositories; allow \"unknown\" for ambiguous cases\n\n**Web Interface**\nBuild a web interface with these features because it serves as the primary power-user surface:\n- Real-time streaming of agent work on desktop and mobile\n- Hosted VS Code instance running inside sandbox\n- Streamed desktop view for visual verification\n- Before/after screenshots for PRs\n- Statistics page: sessions resulting in merged PRs (primary metric), usage over time, live \"humans prompting\" count\n\n**Chrome Extension**\nBuild a Chrome extension for non-engineering users because DOM and React internals extraction gives higher precision than raw screenshots at lower token cost:\n- Sidebar chat interface with screenshot tool\n- Extract DOM/React internals instead of raw images\n- Distribute via managed device policy (bypasses Chrome Web Store)\n\n## Practical Guidance\n\n### Hosted Agent Design Checklist\n\nBefore building the system, decide these infrastructure properties explicitly:\n\n1. **Sandbox lifecycle**: how sessions start, snapshot, restore, time out, and terminate.\n2. **Image and warm-pool policy**: how often images rebuild, which caches are precomputed, and when warm sandboxes expire.\n3. **Read/write synchronization**: whether agents may read before repository sync completes, and which events unblock writes.\n4. **Per-session state isolation**: what storage belongs to one session, what is shared, and how cross-session interference is prevented.\n5. **Auth and commit identity**: which operations use app tokens, which use user tokens, and how commits are attributed.\n6. **Output extraction**: how branches, PRs, files, logs, screenshots, and session summaries leave the sandbox.\n7. **Budget and teardown**: maximum runtime, cost ceilings, idle policy, and forced cleanup behavior.\n\n### Follow-Up Message Handling\n\nChoose between queueing and inserting follow-up messages sent during execution. Prefer queueing because it is simpler to manage and lets users send thoughts on next steps while the agent works. Build a mechanism to stop the agent mid-execution when needed, because without it users feel trapped.\n\n### Metrics That Matter\n\nTrack these metrics because they indicate real value rather than vanity usage:\n- Sessions resulting in merged PRs (primary success metric)\n- Time from session start to first model response\n- PR approval rate and revision count\n- Agent-written code percentage across repositories\n\n### Adoption Strategy\n\nDrive internal adoption through visibility rather than mandates because forced usage breeds resentment:\n- Work in public spaces (Slack channels) for visibility\n- Let the product create virality loops\n- Do not force usage over existing tools\n- Build to people's needs, not hypothetical requirements\n\n## Examples\n\n**Example 1: Background coding session lifecycle**\n\n```text\nuser prompt\n-> API allocates warm sandbox from current image\n-> sandbox syncs latest branch delta\n-> reads allowed immediately, writes blocked until sync completes\n-> agent edits and tests inside isolated workspace\n-> sandbox snapshots final filesystem\n-> branch is pushed\n-> API creates PR using user token\n-> session summary, logs, and PR URL are returned to clients\n```\n\nThis sequence keeps setup work outside the user-visible path while preserving auditability and user ownership of code changes.\n\n**Example 2: Boundary decision**\n\nIf the task is \"make the agent loop run for days with locked rubrics and PR approval,\" use `harness-engineering`. If the task is \"run that loop in remote sandboxes with warm pools, session snapshots, streaming clients, and user-authored PRs,\" use this skill.\n\n## Guidelines\n\n1. Pre-build environment images on regular cadence (30 minutes is a good default)\n2. Start warming sandboxes when users begin typing, not when they submit\n3. Allow file reads before git sync completes; block only writes\n4. Structure agent framework as server-first with clients as thin wrappers\n5. Isolate state per session to prevent cross-session interference\n6. Attribute commits to the user who prompted, not the app\n7. Track merged PRs as primary success metric\n8. Build for multiplayer from the start; it is nearly free with proper sync architecture\n\n## Gotchas\n\n1. **Cold start latency**: First sandbox spin-up takes 30-60s and users perceive this as broken. Use warm pools and predictive warm-up on keystroke to eliminate perceived wait time.\n2. **Image staleness**: Infrequent image rebuilds mean agents run with outdated dependencies or code. Set a 30-minute rebuild cadence and monitor image age; alert if builds fail silently.\n3. **Sandbox cost runaway**: Long-running agents without timeout or budget caps accumulate unexpected costs. Set hard timeout limits (default 4 hours) and per-session cost ceilings.\n4. **Auth token expiration mid-session**: Long tasks fail when GitHub tokens expire partway through. Implement token refresh logic and check token validity before sensitive operations like PR creation.\n5. **Git config in sandboxes**: Missing `user.name` or `user.email` causes commit failures in background agents. Always set git identity explicitly during sandbox configuration, never assume it carries over from the image.\n6. **State loss on sandbox recycle**: Agents lose completed work if the sandbox is recycled or times out before results are extracted. Always snapshot before termination and extract artifacts (branches, PRs, files) before letting the sandbox die.\n7. **Oversubscribing warm pools**: Maintaining too many warm sandboxes wastes money during low-traffic periods. Scale pool size based on traffic patterns and time-of-day; use autoscaling rather than fixed pool sizes.\n8. **Missing output extraction**: Agents complete work inside the sandbox but results never get pulled out to the user. Build explicit extraction steps (push branch, create PR, return file contents) into the session teardown flow.\n\n## Integration\n\nThis skill owns hosted runtime infrastructure. Adjacent skills own the control system, topology, and tool contracts:\n\n- `harness-engineering`: governance, locked evaluators, rollback, novelty gates, and human approval boundaries around autonomous work.\n- `multi-agent-patterns`: self-spawning and supervisor patterns once hosted infrastructure exists.\n- `tool-design`: spawn, status, teardown, and PR tools exposed to agents.\n- `context-optimization`: managing context across distributed hosted sessions.\n- `filesystem-context`: using the sandbox filesystem for durable session state and artifacts.\n\n## References\n\nInternal reference:\n- [Infrastructure Patterns](./references/infrastructure-patterns.md) - Read when: implementing sandbox lifecycle, image builds, or warm pool logic for the first time\n\nRelated skills in this collection:\n- multi-agent-patterns - Read when: designing self-spawning or supervisor coordination patterns\n- tool-design - Read when: building tools for agent session management or status checking\n- context-optimization - Read when: context windows fill up across distributed agent sessions\n\nExternal resources:\n- [Ramp](https://builders.ramp.com/post/why-we-built-our-background-agent) - Read when: evaluating whether to build vs. buy background agent infrastructure\n- [Modal Sandboxes](https://modal.com/docs/guide/sandbox) - Read when: choosing a cloud sandbox provider or comparing isolation models\n- [Cloudflare Durable Objects](https://developers.cloudflare.com/durable-objects/) - Read when: designing per-session state management with WebSocket hibernation\n- [OpenCode](https://github.com/sst/opencode) - Read when: selecting a server-first agent framework or studying plugin architectures\n\n---\n\n## Skill Metadata\n\n**Created**: 2026-01-12\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 1.2.0\n","frontmatter":{"name":"hosted-agents","description":"This skill should be used when designing hosted or background agent infrastructure: sandboxed execution, remote coding environments, warm pools, session persistence, multiplayer collaboration, self-spawning agents, or Modal-style sandboxes."},"isInternal":false,"tokens":3400,"sizeBytes":17717},{"name":"attention-matching-formulation.md","path":"skills/latent-briefing/references/attention-matching-formulation.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/latent-briefing/references/attention-matching-formulation.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Attention Matching (AM) and Task-Guided Scoring\n\nThis note expands the compact treatment in the main skill: the AM objective, what changes under Latent Briefing, and which assumptions matter in practice.\n\n## AM Compaction Objective\n\nGiven a full KV cache of size `S`, Attention Matching seeks a smaller cache of size `t < S` whose attention outputs stay close to the original. Per attention head, compacted components `(C1, beta, C2)` satisfy:\n\n```text\nsoftmax(Q * C1^T + beta) * C2 ~= softmax(Q * K^T) * V\n```\n\n- **C1**: compacted keys, usually a subset of original key vectors selected for high attention mass\n- **beta**: bias corrections so the softmax distribution over kept keys approximates the distribution over all keys\n- **C2**: compacted values reconstructed so the attention output matches the original as closely as possible\n\nThe original AM procedure solves each `(layer, head)` independently:\n\n1. Select tokens to keep\n2. Solve `beta`\n3. Reconstruct `C2`\n\nThis per-head independence helps quality but makes batching difficult because different heads keep different token subsets.\n\n## What Latent Briefing Changes\n\nLatent Briefing adapts AM for orchestrator-worker handoff:\n\n1. **Query source changes.** Standard AM may score keys using queries sampled from the context itself. Latent Briefing instead uses queries derived from the **current worker task prompt**.\n2. **Scoring becomes task-conditioned.** The trajectory is forward-passed through the worker model, and attention from task queries to trajectory keys defines which positions matter for this worker call.\n3. **Selection becomes shared.** Scores are aggregated across layers and heads into one per-position score so the system can use a single keep/drop mask.\n\nConceptually:\n\n```text\ntrajectory -> K, V\ntask prompt -> Q_task\nscore(position) = aggregate_attn(Q_task, K[position])\nkeep if score(position) exceeds threshold\n```\n\n## Why the Shared Mask Matters\n\nPer-head masks maximize flexibility but force many incompatible small solves. A **shared global mask** makes the retained sequence layout identical across heads, which enables batched tensor operations and much lower latency.\n\nThat batching benefit is one of the main reasons Latent Briefing is interesting for inference systems rather than only for offline compression research.\n\n## Thresholding vs Fixed Top-k\n\nInstead of keeping a fixed number of tokens per head, Latent Briefing can threshold the aggregated per-position score distribution:\n\n```text\nkeep position i if score[i] > median(score) + k * MAD(score)\n```\n\nThis makes retention rate adaptive to the shape of the scores for the current task. Higher `k` means more aggressive compaction.\n\n## Practical Assumptions\n\n- The runtime can inspect and modify worker KV state\n- The worker architecture is stable enough that attention-space retention is meaningful across calls\n- The evaluation tracks both quality and cost, because lower token count alone is not sufficient\n","isInternal":false,"tokens":619,"sizeBytes":2962},{"name":"SKILL.md","path":"skills/latent-briefing/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/latent-briefing/SKILL.md","title":"latent-briefing","category":"anthropic-skill","format":"markdown","content":"---\nname: latent-briefing\ndescription: \"This skill should be used when the user asks to \\\"share memory between agents\\\", \\\"KV cache compaction for multi-agent\\\", \\\"orchestrator worker context\\\", \\\"latent briefing\\\", \\\"reduce worker tokens\\\", \\\"cross-agent memory without summarization\\\", or discusses Attention Matching compaction, recursive language models with workers, or token explosion in hierarchical agents.\"\n---\n\n# Latent Briefing and KV Cache Memory Sharing\n\nHierarchical multi-agent systems often pay for the same context twice. The orchestrator accumulates a long reasoning trajectory, but each worker usually receives only a narrow text handoff such as a subtask prompt plus raw document slices. Passing the full trajectory fixes coverage but drives token cost up on every worker call. Summarization introduces latency and information loss. Retrieval helps with document access but does not preserve the orchestrator's evolving reasoning state.\n\nLatent Briefing addresses this by sharing memory at the **representation level** rather than the text level. The core idea is to compact the orchestrator trajectory in the worker model's KV cache, keeping positions that are most relevant to the **current worker task**. The method builds on **Attention Matching (AM)** KV cache compaction and adapts it for inference-time multi-agent handoff with task-guided queries, a shared token mask across heads, and robust thresholding.\n\n## When to Activate\n\nActivate this skill when:\n\n- Designing orchestrator-worker or supervisor-specialist systems where workers need access to prior orchestrator state without replaying the full trajectory as text\n- Evaluating alternatives to LLM summarization or RAG for cross-agent state transfer\n- Implementing or studying **KV cache compaction** as a first-class inference primitive, not only prefix caching of identical prompts\n- Debugging token explosion in recursive, hierarchical, or tool-heavy agent graphs\n- Interpreting benchmarks that report worker-token savings, total-token savings, compaction overhead, and accuracy together\n\nDo not activate this skill for adjacent work owned by other skills:\n- API-only stacks where internal KV tensors are inaccessible: use `context-compression`, `memory-systems`, or `multi-agent-patterns`.\n- Ordinary persistent memory, entity tracking, or graph retrieval: `memory-systems`.\n- General multi-agent topology without representation-level state sharing: `multi-agent-patterns`.\n- Prefix caching, masking, or budget policy that does not transform KV state: `context-optimization`.\n\n## Core Concepts\n\n**The token explosion pattern.** In recursive or REPL-style systems, the orchestrator repeatedly calls a worker to inspect evidence, verify hypotheses, or answer subquestions. The orchestrator's trajectory grows with partial conclusions, dead ends, tool output, and prior worker responses. If that trajectory is passed in full on every worker call, cost compounds quickly.\n\n**Representation-level sharing.** Instead of summarizing the trajectory into natural language, the system operates on the worker model's **KV cache**. It retains the positions that the worker would attend to for the current task and drops the rest. This is more specific than ordinary prefix caching: prefix caching reuses identical prefixes, while Latent Briefing also performs **task-conditioned selective retention** inside the reused trajectory.\n\n**Attention Matching as the compaction engine.** AM seeks a smaller cache whose attention outputs approximate the full cache. Latent Briefing adapts AM for multi-agent inference by changing the scoring signal and batching strategy:\n\n1. Use **task-guided query vectors** derived from the current worker prompt.\n2. Aggregate scores into a **shared global mask** instead of per-head independent subsets.\n3. Use a robust threshold such as `median + tau * MAD` rather than fixed top-k per head.\n\n**Reference result shape.** The public write-up reports substantial worker-token reduction, material total-token savings, and low-single-digit-second compaction overhead on long-document QA workloads (claim-latent-briefing-public-results). Treat these numbers as workload-specific evidence, not a general guarantee.\n\n## Detailed Topics\n\n### Why Text-Only Mitigations Fall Short\n\n| Approach | Primary weakness |\n|----------|------------------|\n| LLM summarization | High latency, lossy abstraction, and no guarantee the summary preserves what the next subtask needs |\n| Retrieval / RAG | Depends on chunking and embeddings; can miss cross-chunk or cross-step dependencies |\n| Pass full trajectory | Cost scales with every worker call and irrelevant context can degrade worker quality |\n\nLatent Briefing is useful when the bottleneck is not document retrieval itself, but **how to transfer orchestrator state into a worker efficiently and precisely**.\n\n### Recursive Orchestrator-Worker Shape\n\nFrameworks such as **Recursive Language Models** treat long context as an environment and recurse over it: an orchestrator decomposes work and delegates to workers. Latent Briefing fits the gap where the orchestrator has already built task-specific state that should inform the worker, but re-serializing that state as text is too expensive or noisy.\n\nIn the ideal setup, the worker maintains a persistent KV state for the orchestrator trajectory. New trajectory tokens extend that state, then compaction runs just before generation for the current subtask.\n\n### Three Inference-Time Modifications\n\n1. **Task-guided query vectors.** Use queries from the current worker task prompt, not generic samples from the context. Forward-pass the trajectory plus current task through the worker model, then score trajectory positions by how strongly the task attends to them.\n\n2. **Shared token selection.** Aggregate scores across layers and heads into one per-position score. One shared mask enables batched operations and avoids hundreds of incompatible per-head solves.\n\n3. **MAD thresholding.** Keep positions above a robust outlier threshold such as `median + tau * MAD`. Higher `tau` is more aggressive. Optimal settings depend on task regime, trajectory quality, and document length.\n\n### Infrastructure Preconditions\n\nLatent Briefing is only practical when the system **controls the worker inference runtime** closely enough to inspect or transform KV state. It is a poor default for API-only stacks where internal KV tensors are inaccessible. It also assumes the orchestrator trajectory can be represented in the worker's model space. If orchestrator and worker differ materially in tokenizer, architecture, or attention layout, direct representation sharing may not be viable.\n\n### Decision Framework\n\nChoose the mechanism that matches the bottleneck:\n\n| Need | Prefer | Why |\n|------|--------|-----|\n| Stable repeated prefix with minimal logic changes | Prefix caching | Cheapest optimization; no information loss |\n| Human-readable and auditable cross-step state | Structured notes or summarization | Easy to inspect and store |\n| Sparse lookup across a large external corpus | Retrieval / RAG | Finds documents efficiently |\n| Worker needs task-specific slices of orchestrator state and runtime access exists | Latent Briefing | Transfers relevant latent state without replaying all text |\n\nLatent Briefing is not a universal replacement for summarization or retrieval. It is a specialized optimization for systems that already run a controllable orchestrator-worker stack.\n\n### Threshold Regimes\n\nReported long-document QA results suggest:\n\n- **Longer documents:** lighter compaction can preserve broader evidence coverage while still saving tokens.\n- **Harder questions:** more aggressive compaction can help when the orchestrator trajectory contains speculative or low-value branches.\n- **Shorter, easier contexts:** moderate compaction may remove redundancy without dropping needed evidence.\n\nThese are tuning hypotheses, not portable laws. Re-measure on the target workload.\n\n## Practical Guidance\n\n- **Define the shared memory boundary first.** Decide exactly what enters the trajectory cache: prior worker replies, tool output, chain-of-thought, or only selected artifacts. Compaction quality depends on what is allowed into the cache in the first place.\n- **Tune on validation data, not anecdotes.** Track task accuracy, worker tokens, total tokens, retention rate, and compaction overhead together.\n- **Measure end-to-end latency.** Compaction only pays off if compaction plus generation beats the best text-layer alternative for the same quality target.\n- **Use strong baselines.** Compare against prefix caching, structured notes, retrieval, and selective text handoff, not only \"send everything.\"\n- **Expect orchestrator variance.** If decomposition strategy changes run to run, average over enough trials to separate compaction effects from orchestrator noise.\n\n## Examples\n\n**Scenario: orchestrator trajectory grows across worker calls**\n\n```text\nCall 1: trajectory T1 -> worker answers subquestion A\nCall 2: trajectory T2 = T1 + new reasoning + reply A\n compact KV(T2) using the task prompt for B\n worker answers subquestion B\n```\n\nThe task prompt for B decides which parts of `T2` survive into the compacted worker state.\n\n**Negative example: API-only worker**\n\nIf the worker runs behind a hosted text-generation API that does not expose KV tensors, Latent Briefing cannot be implemented directly. Use a structured text handoff from `context-compression` or retrieve state from `memory-systems` instead.\n\n## Guidelines\n\n1. Prefer Latent Briefing when the main waste comes from replaying orchestrator state into workers, not from retrieving source documents.\n2. Prefer plain text handoff when auditability, portability, or closed-model APIs matter more than token efficiency.\n3. Co-design compaction with **evaluation**. A small quality drop can erase large token savings.\n4. Expose compaction aggressiveness as a controlled parameter, not a hidden constant.\n\n## Gotchas\n\n1. **Infrastructure access is the first gate.** If the runtime cannot inspect and rewrite worker KV state, Latent Briefing is a research idea, not a deployable technique.\n2. **Shared model space matters.** KV compaction is defined in a specific model's attention space. Do not assume latent handoff works cleanly across unrelated model families.\n3. **Threshold is workload-dependent.** One global `tau` rarely works across long vs short context and easy vs hard tasks. Expect accuracy cliffs when compaction becomes too aggressive.\n4. **Benchmark scope is narrow.** Public results focus on long-document QA. Code generation, math, and multi-document synthesis may behave differently.\n5. **Orchestrator variance can hide the signal.** A stochastic orchestrator can change the trajectory enough to swamp small compaction gains or losses.\n6. **Weak baselines inflate the apparent win.** Compare against strong text-level alternatives before claiming a system-level advantage.\n\n## Integration\n\n- context-optimization - Prefix caching and observation masking remain the default first moves; Latent Briefing is a more specialized optimization for compatible orchestrator-worker stacks.\n- multi-agent-patterns - Applies when multi-agent token cost is driven by supervisor trajectory replay, not only by coordination overhead.\n- context-compression - Text-layer summaries remain preferable when human-readable state, portability, or audit logs matter.\n- memory-systems - Helps decide when to keep cross-step state in external memory versus in the worker's latent state.\n- tool-design - Worker call shapes and task prompts determine which tokens score highly during compaction.\n\n## References\n\nInternal reference:\n- [Attention Matching formulation and task-guided scoring](./references/attention-matching-formulation.md) - Read when: needing the AM objective, how task-guided scoring changes the query source, or why a shared global mask matters for batching\n\nRelated skills in this collection:\n- context-optimization - Read when: the main need is prefix caching, observation masking, or text-layer compaction rather than worker KV manipulation\n- multi-agent-patterns - Read when: deciding whether the architecture should be orchestrator-worker at all\n- context-compression - Read when: human-readable summaries may be a better fit than latent transfer\n- memory-systems - Read when: comparing in-model latent state with external persistent memory\n\nExternal resources:\n- Ramp Labs announcement: [Latent Briefing: Efficient Memory Sharing for Multi-Agent Systems via KV Cache Compaction](https://x.com/RampLabs/status/2042660310851449223)\n- Attention Matching (AM): [Fast KV Compaction via Attention Matching](https://arxiv.org/abs/2602.16284)\n- Recursive Language Models: [Recursive Language Models](https://arxiv.org/abs/2512.24601)\n\n---\n\n## Skill Metadata\n\n**Created**: 2026-04-14\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors; primary technical source Ramp Labs (public post)\n**Version**: 1.2.0\n","frontmatter":{"name":"latent-briefing","description":"This skill should be used when the user asks to \"share memory between agents\", \"KV cache compaction for multi-agent\", \"orchestrator worker context\", \"latent briefing\", \"reduce worker tokens\", \"cross-agent memory without summarization\", or discusses Attention Matching compaction, recursive language models with workers, or token explosion in hierarchical agents."},"isInternal":false,"tokens":2578,"sizeBytes":13029},{"name":"cdc-prompt-annotated.md","path":"skills/long-horizon-prompting/references/cdc-prompt-annotated.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/long-horizon-prompting/references/cdc-prompt-annotated.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# The Cycle Double Cover Prompt, Annotated\n\nStatus and provenance, as of 2026-07-11:\n\n- On 2026-07-10 OpenAI published a candidate proof of the Cycle Double Cover Conjecture attributed to GPT-5.6 Sol Ultra, together with the full prompt used. Prompt PDF: `https://cdn.openai.com/pdf/04d1d1e4-bc75-476a-97cf-49055cd98d31/cdc_prompt.pdf`. Proof PDF: `https://cdn.openai.com/pdf/04d1d1e4-bc75-476a-97cf-49055cd98d31/cdc_proof.pdf`.\n- The run used the \"multiagent v2\" feature with up to 64 concurrent agents and reportedly completed in under one hour, well below the prompt's stated eight-hour effort floor.\n- The proof's statement of AI use says the proof \"is entirely due to GPT 5.6 Sol Ultra and the writeup with Codex (with GPT 5.6 Sol)\".\n- The proof had no independent peer review, no formalization in Lean or Coq, and no arXiv posting at publication time. The validated artifact of interest in this skill is the prompt structure, not the theorem. Treat the mathematical claim as unverified until the community check completes.\n- No public ablation isolates which prompt elements contributed to the result. The per-element evidence in `research-evidence.md` comes from independent academic work, not from this run.\n\nThe full prompt text follows, in blocks, each followed by annotation.\n\n## Block 1: Definitions\n\n> A graph here is a finite loopless undirected multigraph: parallel edges are allowed and are distinct. A bridge is an edge whose deletion increases the number of connected components. A cycle is a connected 2-regular submultigraph; thus two parallel edges form a cycle of length two. A cycle double cover of G is a finite multiset of cycles of G such that every edge of G occurs in exactly two members of the multiset, counted with multiplicity.\n\nEvery load-bearing term is defined before the task is stated, and each definition pre-empts a specific loophole: \"loopless\" and \"parallel edges are distinct\" fix the object class; \"thus two parallel edges form a cycle of length two\" settles a degenerate case a solver might otherwise argue either way; \"counted with multiplicity\" closes the multiset ambiguity. Definitions here are not pedagogy; they are loophole closure.\n\n## Block 2: Success predicate\n\n> Resolve the Cycle Double Cover Conjecture completely: Every finite bridgeless loopless multigraph has a cycle double cover. Disconnected graphs are permitted, and the edgeless graph has the empty cycle double cover. Cycles in the cover need not be induced or edge-disjoint from one another; the requirement is exactly two total occurrences of each edge.\n>\n> Assume for purposes of this task that a complete affirmative proof exists. A complete solution must prove exactly the following: Every finite loopless multigraph with no bridge possesses a cycle double cover, without additional assumptions such as cubicity, planarity, connectivity, or higher edge-connectivity.\n\nThree mechanisms in one block. First, the predicate is stated twice, once as the conjecture and once as the exact obligation, with the scope quantifier spelled out by enumerating the assumptions the proof is NOT allowed to make (cubicity, planarity, connectivity, higher edge-connectivity). These are exactly the special cases where partial results were already known, so the enumeration blocks the most probable near misses. Second, permissive clauses (\"cycles need not be induced or edge-disjoint\") prevent the solver from over-constraining its own search. Third, \"Assume for purposes of this task that a complete affirmative proof exists\" is the solvability framing: it removes the escape hatch of answering \"this is a famous open problem\" and counters give-up drift on long trajectories.\n\n## Block 3: Non-counting outcomes\n\n> Partial progress does not count unless it implies exactly the resolution above. In particular, proofs for special graph classes, constructions of cycle covers with some edges covered other than twice, bounded-length or prescribed-cycle variants, reductions to another unproved conjecture, computational verification through any fixed graph size, and candidate counterexamples without a complete nonexistence certificate are insufficient.\n\nThe enumerated near-miss list. Each item is a real artifact class from the CDC literature: special-class proofs, relaxed covers, variant formulations, reductions (CDC is famously equivalent to or implied by other open conjectures), finite verification, and unverified counterexamples. The general lesson: predict the specific answer-shaped near misses your problem invites and exclude them by name.\n\n## Block 4: Orchestration policy\n\n> Use multiagent v2 aggressively and dynamically. You have up to 64 concurrent agents available. Do not use a fixed assignment such as \"N agents for strategy X.\" Instead, manage the search using the following heuristics:\n>\n> - Begin with a genuinely diverse portfolio of approaches. Agents should explore substantially different formulations, invariants, reductions, algebraic viewpoints, structural inductions, decompositions, flow formulations, transition systems, embeddings, extremal arguments, and computational sanity checks.\n> - Do not tell most agents the currently favored approach. Preserve independence during early rounds so that agents do not all converge to the same attractive but incomplete reduction.\n> - Maintain an explicit registry of approach families. Group agents by the mathematical idea they are using, not by superficial wording. If many agents converge to one family, redirect some of them toward underexplored formulations.\n> - Do not allow one approach to dominate merely because it gives elegant reductions. A route that ends at a lemma equivalent in strength to the original conjecture is not close to completion unless it supplies a genuinely new proof of that lemma.\n> - When an approach stalls at a theorem-strength missing lemma, mark that route as blocked. Only continue assigning agents to it if someone proposes a materially new mechanism, invariant, or construction.\n> - Keep several incompatible proof routes alive through multiple rounds. Cross-pollinate ideas only after independent agents have developed them far enough to expose their real strengths and gaps.\n\nPolicy as heuristics, not assignments. The notable mechanisms: information hiding as a diversity tool (workers blind to the favored approach), a registry keyed to the underlying idea rather than wording (so the orchestrator cannot be fooled by paraphrase into thinking it has diversity), an anti-elegance rule (a reduction to an equally hard lemma is zero progress, a trap models find attractive), blocked-route bookkeeping with a materially-new-mechanism reopening condition, and deliberately delayed cross-pollination. These correspond one-to-one with the diversity-collapse and premature-convergence findings summarized in `research-evidence.md`.\n\n## Block 5: Verification policy and reporting contract\n\n> - Use adversarial agents throughout: every candidate proof must be checked for exact-two multiplicity, repeated-edge closed trails masquerading as cycles, parallel-edge 2-cycles, disconnected graphs, cutvertices, bridges introduced by reductions, and circular use of an equivalent CDC statement.\n> - Require agents to return concrete lemmas, constructions, equations, or counterexamples to proposed sublemmas. Reject status reports, vague optimism, and claims that an unproved global compatibility statement is \"routine.\"\n\nThe auditor gets a seven-item, domain-specific hunt list rather than \"check the proof\". Each item is a concrete way a CDC candidate can look right and be wrong; the last item, circular use of an equivalent statement, is the domain's version of the universal subtle failure. The reporting contract bans exactly the three degenerate report types long runs produce: status reports, optimism, and \"the remaining step is routine\".\n\n## Block 6: Orchestrator loop, return condition, effort floor\n\n> - The root agent should repeatedly synthesize, challenge, redirect, and launch new rounds. Do not stop after the first wave fails. Produce a complete proof if one survives audit; otherwise report only the strongest rigorously proved derivation and its exact remaining gap.\n>\n> Do not return merely because current approaches fail or agents report theorem-strength gaps. Continue launching new rounds, reopening blocked approaches only when there is a genuinely new mechanism, and searching for fresh formulations.\n>\n> Return only when a complete affirmative proof has been found and survives adversarial audit. Do not return a reduction, partial result, isolated missing lemma, \"best effort\" summary, or explanation of why the problem is difficult.\n>\n> Spend at least 8 hours on this before even thinking of returning or giving up.\n\nThe return condition is a predicate over the artifact (\"survives adversarial audit\"), not over confidence or effort. The non-counting list is restated at the return boundary, where the temptation to return a near miss is strongest. The effort floor is a permission revocation (\"before even thinking of returning\"), not a schedule; the run finished in under an hour because the return predicate was satisfied early. Note one internal tension: Block 6 first allows a fallback report (\"otherwise report only the strongest rigorously proved derivation and its exact remaining gap\") and then forbids returning partial results. The final instruction wins in practice, but a cleaner brief would scope the fallback to a hard external stop (budget exhaustion) rather than leaving the contradiction.\n\n## Block 7: Contamination guard\n\n> Public search may be used only for ordinary mathematical background or standard named theorems, not to search for a solution to this exact conjecture or benchmark. Do not search the public web merely to determine whether CDC is open, and do not answer that it is open.\n\nTwo guards in one: retrieval scope (background and named theorems only, never the target result) and a framing guard (\"do not answer that it is open\"), which backstops the solvability framing in Block 2 against a web-lookup override.\n\n## What the prompt does not do\n\nUseful negative space for anyone adapting it:\n\n- No fixed role assignments, no personas, no step-by-step method script. The mathematical strategy is left entirely to the model; the prompt only manages search discipline and acceptance.\n- No token or cost budget in the prompt. Resource enforcement lived in the platform, consistent with keeping hard constraints in the harness.\n- No requested output format for the proof itself beyond survivability under audit.\n- No appeals to emotion, urgency, or reward. Every sentence is either specification, policy, or gate.\n","isInternal":false,"tokens":2152,"sizeBytes":10680},{"name":"mass-handoff-contracts.md","path":"skills/long-horizon-prompting/references/mass-handoff-contracts.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/long-horizon-prompting/references/mass-handoff-contracts.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# MASS Handoff Contracts, Annotated\n\nStatus and provenance, as of 2026-10-11:\n\n- Source: Hyunin Lee, Jinglue Xu, Jeffrey Seely, Donghyun Lee, Somayeh Sojoudi, Matei Zaharia, Yujin Tang, \"Recursive Self-Improvement through Multi-Agent Self-Supervision\" (Sakana AI and UC Berkeley), arXiv 2610.12176v1, October 2026. Project page: `https://pub.sakana.ai/mass/`. Code: `https://github.com/SakanaAI/mass`.\n- MASS is a recursive self-improvement method. One shared model (Qwen3.6-27B in the experiments) acts as executor, workflow optimizer, and evaluator. The optimizer searches multi-agent workflow text using the evaluator's pairwise judgments, then the model is fine-tuned with LoRA on orchestrator and subagent trajectories from the self-judged best workflows. The paper reports two full rounds.\n- This skill does not cover the weight-update loop; that belongs to `self-improvement-loops`. What transfers to launch-prompt design is the workflow representation, the contracts the optimizer converged on, the execution directive the authors wrote for trajectory collection, and three findings about verification and search.\n- Preprint, unreviewed at retrieval. Training used eight synthetic research tasks in finance, robotics, and pharmacy. The component analysis covers 8 to 9 tasks and 43 to 51 agents, which the authors describe as descriptive text statistics rather than independent replications. Progress was scored by external judges (GPT-5.5 and Claude Opus 4.8) whose judgments were withheld from the loop.\n\n## Workflow as a prompt\n\nMASS represents a workflow as text appended to the task prompt, with four components per subagent:\n\n- Role: the responsibility assigned to the subagent.\n- Instruction: the work the subagent should carry out.\n- Contract: \"the information and artifacts it must return, including any required checks.\"\n- Hop: the order of subagent calls, including any return to an earlier step.\n\nThe optimizer's system prompt emphasizes contracts and hops, directing search toward what information agents exchange and when they receive it.\n\nThis decomposition lines up with the four-part delegation spec in vendor guidance (objective, output format, tool guidance, boundaries), with one difference that matters. An output format says what shape to return. A contract also says what must be established, and checked, before the next agent may proceed.\n\n## Finding 1: task information moves into contracts and hops\n\nBetween workflow-search iterations 3 and 11, estimated mutual information between component text and task descriptions fell for roles and instructions and rose for contracts and hops, across two embedding models and two estimators. Redundancy among components generally fell (claim-long-horizon-contract-information-shift). The project page gives the intuition: \"a role such as 'careful data scientist' could apply to many tasks. A contract specifying leakage checks for a trading backtest is much more distinctive.\"\n\nImplication: when an optimizer is free to put task knowledge anywhere in a worker prompt, it moves that knowledge into what the worker must return and when. Brief writers should do the same: keep roles short, and spend task-specific words on contracts and call order.\n\n## Annotated contract: the finance data stage\n\nCondensed from the evolved finance workflow on the project page:\n\n```text\nCall order\n Data -> Features -> Modeling -> Backtest\n -> Audit -> Reconciliation -> Report\n\nData specialist's output contract\n validation_report:\n no_future_leakage: bool\n train_end_before_val_start: bool\n val_end_before_test_start: bool\n```\n\nThe authors' gloss: \"A backtest can look convincing if it lets historical decisions use information from the future. This contract requires the data specialist to check for that error and report whether the time splits are valid. It also tells later agents what evidence they should receive before proceeding.\"\n\nThree properties of this contract generalize:\n\n1. Each check is named after the failure it rules out (`no_future_leakage`), not after a quality (\"rigorous data preparation\").\n2. Each check is typed. A boolean in a validation report can be read by the next agent and by a harness without judgment.\n3. The check sits at the handoff where the error is introduced, not only in the final audit, so downstream work does not build on a broken input.\n\nA research report is non-verifiable in the paper's sense: no automatic rule-based signal exists for the whole artifact. The contract carves verifiable pieces out of it anyway. The same move applies to code (`tests_pass`, `tests_unmodified`), to modeling (`no_target_leakage`, `split_overlap_free`), and to documents (`every_claim_cites_a_source`).\n\n## The evolved orchestrator: gates, degenerate values, recall\n\nThe complete evolved workflow for one finance task (paper Appendix I.1, task 51, workflow version 10) adds three patterns on top of per-stage contracts. The paper notes that its thresholds and heuristics \"are optimizer proposals, not additional benchmark requirements or validated scientific criteria.\"\n\n- Gates on every handoff: \"Enforce strict JSON schema validation on every subagent output,\" and \"Halt synthesis until all gates pass or max hops exhausted with documented fallbacks.\"\n- Degenerate-value rejection: \"Reject any output containing NaN, Inf, or degenerate values (e.g., regime_accuracy==1.0, confusion_matrix=[[N]], IC==RankIC, ablation_delta_sharpe>0).\" The auditor's checklist adds \"Sharpe > 10\" and a \"stability = 0.5 placeholder.\" These are too-good-to-be-true signatures: values that appear only when something upstream is broken.\n- Recall with its own acceptance criteria: \"if a gate fails, issue a RECALL command with {target_agent, failed_checks, error_context, narrowed_scope, acceptance_criteria, code_patch_hint, max_retries: 3}.\" The fix request names one responsible agent, the checks that failed, a narrower scope, what will count as fixed, and a retry cap.\n\nThe recall shape is what keeps review from becoming open-ended. A finding turns into a bounded work order with its own definition of done, routed to one agent, instead of reopening the whole artifact for another unbounded pass.\n\n## The authors' execution directive\n\nAppendix B.3 reproduces the directive appended to every workflow during trajectory collection. The authors wrote it; it was not evolved. Selected items, verbatim:\n\n- \"Run the team for real. ... Simulating a role in your own turn does not count. You, the orchestrator, do not write pipeline or analysis code yourself; you set up the project, delegate, verify, and integrate.\"\n- \"One stage at a time. Launch a worker only when the inputs it needs already exist on disk.\"\n- \"Self-contained instructions. A worker sees nothing but your prompt and the filesystem. Each prompt must include: the role's instruction from the team definition above; the absolute workspace path; the exact files to create or modify; the interfaces or artifacts produced by earlier stages that it must use; the command(s) to run to verify its work; and the report you expect back (files written, commands run with results, key numbers, known problems).\"\n- \"Verify before you build on it. After each report, check the worker's output yourself (run the entry point, inspect the artifacts). If the check fails or the report shows problems, re-delegate to the same role with a corrected, narrower instruction instead of fixing it yourself.\"\n- \"Bounded work. Give each worker one deliverable it can finish in a single run: a handful of files and one verification command.\"\n- \"Finish cleanly. ... Your final answer must summarize every worker call: role, what it was asked, status, and what you verified.\"\n\nEvery item is a non-counting outcome (\"simulating a role does not count\"), a gate (\"launch only when inputs exist\", \"verify before you build on it\"), or a reporting contract (\"summarize every worker call ... and what you verified\"). None of it is a persona.\n\n## Finding 2: the reviser can be the bottleneck, not the reviewer\n\nWith the executor fixed at the base model, replacing only the evaluator with a stronger external model improved workflow search less than replacing both the evaluator and the optimizer. The fully homogeneous loop still reached unanimous wins on 11 of 12 tasks, more slowly (claim-long-horizon-reviser-bottleneck). The authors' reading: strong feedback is wasted when the component that acts on it cannot digest it.\n\nImplication: in an orchestration, the agent that turns audit findings into the next round (usually the root) needs as much capability and prompt design as the auditor. Investing only in a stricter verifier can leave a run holding accurate findings that nobody acts on well.\n\n## Finding 3: one run is an anecdote\n\nOn one task, two workflow files and their rendered prompts were byte-identical and shared a generation seed. The first episode completed and won all six external judgments; the second halted under loop detection and lost all six. Workflow search more generally showed strong iterations followed by sharp regressions (claim-long-horizon-run-variance). MASS itself executes each champion workflow 18 times per task before selecting training trajectories.\n\nImplication: judge a brief, or a change to a brief, across several runs. A single success does not validate it and a single failure does not refute it.\n\n## Shared blind spots\n\nThe paper motivates multi-agent organization with the observation that \"generation, evaluation, and revision share the same learned prior, meaning that self-feedback can reinforce rather than correct errors.\" Its conclusion names shared blind spots as a safety concern for homogeneous loops, where a flawed result can be \"produced, accepted as an improvement, and incorporated into the next model.\" MASS keeps measurement outside the loop: external judges score progress and their judgments are withheld from the optimizer and evaluator. The appendix also shows one supervised revision whose first diagnostic \"incorrectly treats NaN correlations as passing,\" so a check can itself be faulty.\n\nImplication: fresh context removes the author's build history but not the model family's priors. For results that matter, keep at least one check outside the loop, and test each check on a known-bad input so a vacuous pass gets caught.\n\n## What this reference does not claim\n\n- No ablation isolates contracts as a causal driver of task performance. The information-shift result describes the text of retained workflows.\n- Transfer was uneven. After two rounds, score per output token was 1.2-1.6x the base model on four research benchmarks but 0.96x on Terminal-Bench 2.0 and 0.94x on SWE-bench Verified.\n- The evolved thresholds quoted above are optimizer proposals for one finance task, not validated criteria.\n","isInternal":false,"tokens":2277,"sizeBytes":10802},{"name":"research-evidence.md","path":"skills/long-horizon-prompting/references/research-evidence.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/long-horizon-prompting/references/research-evidence.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Research Evidence for Long-Horizon Brief Elements\n\nDated academic findings backing each element of the pseudo-formal task brief. Compiled 2026-07-11; handoff-contract evidence added 2026-10-11. Each entry states what was measured and what it implies for prompt design. arXiv identifiers are given where known; treat 2026 preprints as unreviewed.\n\n## Give-up drift and premature termination\n\n- **Diagnosing and Mitigating Context Rot in Long-horizon Search** (arXiv 2606.29718, June 2026). As trajectory length grows, the dominant error type shifts from confident-wrong answers to uncertain answers and outright giving up. Rot depends on the content of accumulated context, not just its length; deleting context removes rot but leaves tasks unfinished, so summarize rather than truncate. Implication: solvability framing and effort floors target a real, measured drift, not a hypothetical.\n- **Push Your Agent / PushBench** (arXiv 2605.23574, May 2026). Agents make plausible local tool calls but stop before the requested quantity of work is verifiably complete; failure modes include duplicate submissions, false completion claims, and progress drift. Externally maintained verified-progress ledgers with backlog tracking reached 69-78% task success on configurations where standard and even completion-gated controllers scored 0%. Verifier gating alone prevents false \"done\" claims but does not repair stuck loops. Implication: the stopping condition must be externally checkable, and verified-progress state should be re-injected each round from outside the prompt.\n- **BudgetThinker** (arXiv 2508.17196, August 2025). A budget stated once in the prompt does not reliably control effort; periodic reminders of remaining budget substantially improve adherence. Implication: one-time budget statements at the top of a long prompt decay; re-injection is a harness job.\n- **METR time-horizon line** (arXiv 2503.14499, March 2025; Time Horizon 1.1, January 2026). Model 50% time horizons have doubled roughly every seven months since 2019, driven primarily by reliability and mistake-recovery rather than raw reasoning. Implication: harness and brief effort should go to checkpointing, recovery, and externally verified progress, the reliability margin, rather than more elaborate reasoning instructions.\n\n## The verification bottleneck\n\n- **Large Language Monkeys** (arXiv 2407.21787, July 2024). Coverage (any-sample success) scales log-linearly with sample count across four orders of magnitude, but majority voting and reward-model selectors plateau after a few hundred samples. Gains convert to realized performance only where verification is automatic.\n- **Benchmark Test-Time Scaling of General LLM Agents** (arXiv 2602.18998, February 2026). Names the verification gap directly: pass@K rises with K while self-selection accuracy lags and can fall as K grows; sequential scaling hits a model-specific context ceiling. Implication: adding parallel workers without strengthening selection wastes compute.\n- **QEDBench** (arXiv 2602.20629, February 2026). Across a judge-solver matrix with expert human baselines, frontier LLM judges of mathematical proofs are systematically lenient and susceptible to \"proof by intimidation\" (rewarding rigorous-looking setups without complete deduction). Stricter rubrics, deterministic decoding, and binary prompts did not fix it. Implication: a generic adversarial-audit instruction is insufficient; auditors need enumerated failure modes, and high-stakes claims need a verification chain outside the run.\n- **Pseudo-Formalization for Automatic Proof Verification** (arXiv 2605.20531, May 2026). Translating a natural-language proof into self-contained modules (premises, conclusion, proof stated locally) and verifying each module independently Pareto-dominates whole-proof LLM judging on error-finding precision and recall. Implication: require the generator to produce modular, independently checkable output so verification decomposes.\n- **ProofBench / ProofGrader** (arXiv 2510.13888, October 2025). Judges given a rubric, reference material, and a graded (not binary) scale approach expert grading and, used as a best-of-N selector, close most of the gap to a human oracle. Implication: graded audit criteria beat binary verdicts for candidate selection.\n- **Prover-Verifier Games** (arXiv 2407.13692, July 2024, and successors). Adversarial prover-verifier training increases output checkability; optimizing for correctness alone makes reasoning less legible. Implication at prompt level: dedicated adversarial checkers and legibility requirements on the generator are complements, not alternatives.\n\n## Diversity collapse in parallel search\n\n- **Diversity Collapse in Multi-Agent LLM Systems** (arXiv 2604.18005, Findings of ACL 2026). Dense communication topologies accelerate premature convergence; authority-driven hierarchies suppress semantic diversity. Collapse comes from interaction structure, not model insufficiency. Implication: early-round independence and sparse communication are structural requirements the orchestration policy must state.\n- **Representational Collapse in Multi-Agent LLM Committees** (arXiv 2604.03809, April 2026). Committees converge more tightly on harder problems; unanimous agreement under collapse reflects shared bias, not corroboration. Implication: never use inter-agent agreement alone as a halting or confidence signal.\n- **ParaThinker** (arXiv 2509.04475, September 2025) and **OPE** (arXiv 2602.08344, February 2026). Early imperfect steps lock a sequential reasoner into a bad path (\"tunnel vision\"); parallel width beats sequential depth at equal budget, and explicitly partitioning the solution space (diverse outlines first) beats independent draws. Implication: the orchestrator should assign distinct formulations, not just multiple attempts.\n- **Scaling Test-time Compute for LLM Agents** (arXiv 2506.12928, June 2025). Simple best-of-N was the strongest parallel method tested; list-wise comparison of all candidates together beat pairwise voting for selection; reflection helped only when triggered by poor performance rather than on a fixed cadence. Implication: conditional reflection triggers and list-wise audits belong in the verification policy.\n\n## Orchestration and delegation\n\n- **AOrchestra** (arXiv 2602.03786, February 2026). Modeling each subagent as an (instruction, context, tools, model) tuple synthesized per subtask at runtime, with the orchestrator taking no environment actions itself, gave a 16% relative improvement over the strongest static-role baseline across agentic benchmarks. Implication: the orchestrator's core job is writing precise, per-spawn task specs; static role prompts are the weaker pattern.\n- **DeLM** (arXiv 2606.10662, June 2026). Decentralized agents coordinating through shared verified context, where findings, failures, and falsified hypotheses are written into shared state, prevent parallel workers from re-exploring dead ends. Implication: blocked-route bookkeeping should be durable and shared, not implicit in the orchestrator's context.\n- **RL for LLM-based Multi-Agent Systems through Orchestration Traces** (arXiv 2605.02801, May 2026). Survey observation: as of its writing, no published RL method learns the stopping decision; when to stop is entirely prompt and harness territory. Implication: the return condition in the brief is load-bearing because nothing in training supplies it.\n\n## Handoff contracts and homogeneous loops\n\n- **MASS: Recursive Self-Improvement through Multi-Agent Self-Supervision** (Sakana AI and UC Berkeley, arXiv 2610.12176, October 2026; added 2026-10-11). One shared Qwen3.6-27B model acts as executor, workflow optimizer, and evaluator; workflows are text with a role, instruction, output contract, and call order (hop) per subagent. Four findings bear on brief design. (1) Over workflow search, estimated task information fell in roles and instructions and rose in contracts and hops (iterations 3 to 11, 8 to 9 tasks, descriptive text statistics). Implication: task-specific words belong in what each worker must return and check, not in role descriptions. (2) Replacing only the evaluator with GPT-5.5 sped up workflow search less than replacing evaluator and optimizer together; the all-homogeneous loop reached unanimous wins on 11 of 12 tasks, more slowly. Implication: the agent that acts on audit findings needs as much investment as the auditor. (3) Two runs with byte-identical prompts and the same seed diverged completely (one won all six external judgments, one halted under loop detection and lost all six). Implication: evaluate briefs across several runs. (4) Generation, evaluation, and revision in one model share a learned prior, so self-feedback can reinforce errors; progress was measured by withheld external judges. Implication: keep at least one check outside the loop. Evolved workflows also show per-handoff gates, degenerate-value rejection, and recall commands with acceptance criteria and a retry cap. Caveats: preprint, synthetic training tasks, uneven transfer (score per output token 0.96x on Terminal-Bench 2.0 and 0.94x on SWE-bench Verified after two rounds), and evolved thresholds are optimizer proposals. Annotated in `mass-handoff-contracts.md`.\n\n## Frontier-model open-problem record (2026)\n\nVerification infrastructure, not generation, separates claims from results:\n\n- October 2025: a claim that GPT-5 solved ten Erdős problems collapsed within about 48 hours because the \"solutions\" were already in the literature.\n- May 2026: an OpenAI internal reasoning model disproved the Erdős unit-distance conjecture; the result stands because nine external mathematicians verified it and a companion paper followed. The prompt was reportedly minimal: the problem statement plus the question of whether Erdős was wrong.\n- April 2026: GPT-5.4 Pro produced proof sketches resolving two 1966 primitive-set conjectures; mathematicians repaired gaps and one main theorem was formalized in Lean.\n- July 2026: the Cycle Double Cover candidate proof (GPT-5.6 Sol Ultra, 64 subagents, under one hour) was published with its prompt but without peer review or formalization; at publication it is a plausible-looking proof humans had not yet checked.\n\nImplication: a long-horizon brief maximizes the chance of a strong candidate; the pipeline that converts candidates into results is external verification, and the brief should be written knowing its output enters that pipeline (modular structure, checkable claims, no laundered sources).\n\n## Mapping evidence to brief elements\n\n| Brief element | Primary evidence |\n| --- | --- |\n| Solvability framing, effort floor | Context rot give-up drift (2606.29718); PushBench premature stopping (2605.23574) |\n| Return condition as artifact predicate | PushBench verifier gating; orchestration survey stopping gap (2605.02801) |\n| Non-counting outcomes | Erdős/CDC record: near misses and unverified claims dominate failures |\n| Adversarial audit with enumerated failure modes | QEDBench judge leniency (2602.20629); block verification (2605.20531); ProofBench rubrics (2510.13888) |\n| Early independence, late cross-pollination | Diversity collapse (2604.18005); representational collapse (2604.03809); OPE (2602.08344) |\n| Approach registry and blocked routes | DeLM shared falsified hypotheses (2606.10662) |\n| Per-spawn task specs | AOrchestra tuple synthesis (2602.03786); Anthropic delegation spec |\n| External progress ledger, budget re-injection | PushBench state-tracking controllers; BudgetThinker (2508.17196) |\n| Verifier investment parity | Large Language Monkeys selector plateau (2407.21787); verification gap (2602.18998) |\n| Per-worker output contracts with named checks | MASS task-information shift into contracts and hops (2610.12176) |\n| Recall with acceptance criteria; investment in the reviser | MASS evolved recall commands; evaluator-only versus evaluator-plus-optimizer ablation (2610.12176) |\n| Multi-run brief evaluation | MASS byte-identical-prompt divergence (2610.12176) |\n| Checks outside the loop | MASS shared-prior argument and withheld external judges (2610.12176) |\n","isInternal":false,"tokens":2691,"sizeBytes":12165},{"name":"task-brief-template.md","path":"skills/long-horizon-prompting/references/task-brief-template.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/long-horizon-prompting/references/task-brief-template.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"","isInternal":false,"tokens":2350,"sizeBytes":11472},{"name":"vendor-guidance.md","path":"skills/long-horizon-prompting/references/vendor-guidance.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/long-horizon-prompting/references/vendor-guidance.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"","isInternal":false,"tokens":2323,"sizeBytes":10737},{"name":"SKILL.md","path":"skills/long-horizon-prompting/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/long-horizon-prompting/SKILL.md","title":"long-horizon-prompting","category":"anthropic-skill","format":"markdown","content":"","frontmatter":{"name":"long-horizon-prompting","description":"This skill should be used when writing, enhancing, or evaluating the launch prompt for a long-running autonomous agent or a parallel multi-agent orchestration attacking a hard problem: pseudo-formal task briefs that define terms and an exact success predicate linguistically, enumerate non-counting outcomes, set persistence rules with explicit stop and return conditions and effort floors, manage a diverse portfolio of parallel approaches with an approach registry and blocked-route bookkeeping, and gate the return on adversarial audit. Route agent topology and coordination protocols to multi-agent-patterns, runtime control surfaces and loop governance to harness-engineering, evaluator and quality-gate construction to evaluation, judge design to advanced-evaluation, and compaction or memory mechanics to context-compression and memory-systems."},"isInternal":false,"tokens":6710,"sizeBytes":33489},{"name":"implementation.md","path":"skills/memory-systems/references/implementation.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/memory-systems/references/implementation.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"","isInternal":false,"tokens":3880,"sizeBytes":18178},{"name":"SKILL.md","path":"skills/memory-systems/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/memory-systems/SKILL.md","title":"memory-systems","category":"anthropic-skill","format":"markdown","content":"","frontmatter":{"name":"memory-systems","description":"This skill should be used for persistent semantic memory in agent systems: cross-session knowledge retention, entity tracking, temporal validity, graph or vector retrieval, memory consolidation, and memory benchmark selection. Route file-backed scratchpads to filesystem-context, handoff summaries to context-compression, and token-efficiency tactics to context-optimization."},"isInternal":false,"tokens":3308,"sizeBytes":16554},{"name":"frameworks.md","path":"skills/multi-agent-patterns/references/frameworks.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/multi-agent-patterns/references/frameworks.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"","isInternal":false,"tokens":2627,"sizeBytes":12482},{"name":"SKILL.md","path":"skills/multi-agent-patterns/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/multi-agent-patterns/SKILL.md","title":"multi-agent-patterns","category":"anthropic-skill","format":"markdown","content":"","frontmatter":{"name":"multi-agent-patterns","description":"This skill should be used when designing multi-agent systems that need context isolation, supervisor or swarm coordination, explicit handoffs, parallel execution, or a decision on whether multiple agents are justified."},"isInternal":false,"tokens":3401,"sizeBytes":18649},{"name":"case-studies.md","path":"skills/project-development/references/case-studies.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/project-development/references/case-studies.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"","isInternal":false,"tokens":3161,"sizeBytes":14830},{"name":"pipeline-patterns.md","path":"skills/project-development/references/pipeline-patterns.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/project-development/references/pipeline-patterns.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"","isInternal":false,"tokens":3909,"sizeBytes":16921},{"name":"SKILL.md","path":"skills/project-development/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/project-development/SKILL.md","title":"project-development","category":"anthropic-skill","format":"markdown","content":"","frontmatter":{"name":"project-development","description":"This skill should be used for project-level decisions about LLM-powered systems: whether an LLM is the right primitive for the task at hand, the shape of a multi-stage batch or agent pipeline, token and cost estimation, choosing between single-agent and multi-agent at the project level, structured output design for downstream parsing, and structuring agent-assisted iteration. Use this when the unit of work is a whole project or a multi-stage pipeline. Route individual tool design to tool-design and individual skill-loading or context-budget tactics to context-optimization."},"isInternal":false,"tokens":3653,"sizeBytes":18960},{"name":"loop-design-evidence.md","path":"skills/self-improvement-loops/references/loop-design-evidence.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/self-improvement-loops/references/loop-design-evidence.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"","isInternal":false,"tokens":3077,"sizeBytes":13672},{"name":"SKILL.md","path":"skills/self-improvement-loops/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/self-improvement-loops/SKILL.md","title":"self-improvement-loops","category":"anthropic-skill","format":"markdown","content":"","frontmatter":{"name":"self-improvement-loops","description":"This skill should be used when the harness, scaffold, workflow, or optimizer itself is the optimization target: recursive self-improvement (RSI) loops, meta-harnesses, self-improving harnesses that mine their own failures and propose bounded edits, evolutionary or population-based search over agent scaffolds, acceptance gates for self-modifying systems, and agentic context evolution where the mechanism that produces context is versioned and evolved. Route governance of a single autonomous loop (locked surfaces, durable logs, rollback, novelty gates, approval boundaries) to harness-engineering, measurement and quality-gate design to evaluation, judge design to advanced-evaluation, and remote sandbox infrastructure to hosted-agents."},"isInternal":false,"tokens":4485,"sizeBytes":22494},{"name":"evidence.md","path":"skills/self-managed-context/references/evidence.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/self-managed-context/references/evidence.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"","isInternal":false,"tokens":2841,"sizeBytes":10706},{"name":"harness-protocol.md","path":"skills/self-managed-context/references/harness-protocol.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/self-managed-context/references/harness-protocol.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"","isInternal":false,"tokens":2767,"sizeBytes":12599},{"name":"SKILL.md","path":"skills/self-managed-context/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/self-managed-context/SKILL.md","title":"self-managed-context","category":"anthropic-skill","format":"markdown","content":"","frontmatter":{"name":"self-managed-context","description":"This skill should be used when a model gets read-write control over its own live context window instead of a harness-scheduled compaction policy: the context exposed as an editable file the model rewrites with code tools, model-driven eviction and in-place updates, the harness invariants that keep self-editing safe (pinned prefix, edit gate, edit receipts, budget readouts, rollback on overflow), the prefix-cache cost of mid-context edits, and steering or training the model's own context-editing strategy. Route note content and fixed-threshold summarization to context-compression, cache-stable layout under harness control to context-optimization, file offloading to filesystem-context, and loop governance to self-improvement-loops."},"isInternal":false,"tokens":5069,"sizeBytes":24111},{"name":"architectural_reduction.md","path":"skills/tool-design/references/architectural_reduction.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/tool-design/references/architectural_reduction.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"","isInternal":false,"tokens":1623,"sizeBytes":8138},{"name":"best_practices.md","path":"skills/tool-design/references/best_practices.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/tool-design/references/best_practices.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"","isInternal":false,"tokens":1923,"sizeBytes":10510},{"name":"SKILL.md","path":"skills/tool-design/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/skills/tool-design/SKILL.md","title":"tool-design","category":"anthropic-skill","format":"markdown","content":"","frontmatter":{"name":"tool-design","description":"This skill should be used for the tool-interface layer of an agent system specifically: writing tool descriptions agents can route on, designing tool schemas and response formats, naming conventions, actionable error recovery messages, MCP server design, tool-set consolidation, and deciding when to add or remove an individual tool. Use this when the unit of work is a single tool or a set of tools. Route project-shape, pipeline architecture, and task-model-fit decisions to project-development; route deciding whether to introduce sub-agents to multi-agent-patterns."},"isInternal":false,"tokens":3828,"sizeBytes":20089},{"name":"SKILL.md","path":"template/SKILL.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/template/SKILL.md","title":"skill-template","category":"anthropic-skill","format":"markdown","content":"","frontmatter":{"name":"skill-template","description":"Template for creating new Agent Skills for context engineering. Use this template when adding new skills to the collection."},"isInternal":false,"tokens":999,"sizeBytes":5164},{"name":"AGENTS.md","path":"examples/digital-brain-skill/agents/AGENTS.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/digital-brain-skill/agents/AGENTS.md","title":"agents-module","category":"subagent-persona","format":"markdown","content":"","frontmatter":{"name":"agents-module","description":"Automation scripts and agent helpers for the Digital Brain. Use these scripts for recurring tasks, summaries, and maintenance."},"isInternal":false,"tokens":575,"sizeBytes":2599},{"name":"evaluator-agent.md","path":"examples/llm-as-judge-skills/agents/evaluator-agent/evaluator-agent.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/llm-as-judge-skills/agents/evaluator-agent/evaluator-agent.md","title":"Subagent: evaluator-agent","category":"subagent-persona","format":"markdown","content":"","isInternal":false,"tokens":935,"sizeBytes":4360},{"name":"index.md","path":"examples/llm-as-judge-skills/agents/index.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/llm-as-judge-skills/agents/index.md","title":"Subagent: index","category":"subagent-persona","format":"markdown","content":"","isInternal":false,"tokens":593,"sizeBytes":2557},{"name":"orchestrator-agent.md","path":"examples/llm-as-judge-skills/agents/orchestrator-agent/orchestrator-agent.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/llm-as-judge-skills/agents/orchestrator-agent/orchestrator-agent.md","title":"Subagent: orchestrator-agent","category":"subagent-persona","format":"markdown","content":"","isInternal":false,"tokens":1119,"sizeBytes":4856},{"name":"research-agent.md","path":"examples/llm-as-judge-skills/agents/research-agent/research-agent.md","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/examples/llm-as-judge-skills/agents/research-agent/research-agent.md","title":"Subagent: research-agent","category":"subagent-persona","format":"markdown","content":"","isInternal":false,"tokens":986,"sizeBytes":4442},{"name":"marketplace.json","path":".claude-plugin/marketplace.json","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/.claude-plugin/marketplace.json","title":"context-engineering-marketplace","category":"marketplace","format":"json","content":"","frontmatter":{"name":"context-engineering-marketplace","owner":{"name":"Muratcan Koylan","email":"muratcan.koylan@outlook.com"},"metadata":{"description":"Context engineering and harness engineering skills plus a file-based autonomous research-to-skill operating system with measured router-benchmark results across four frontier models","version":"2.6.0"},"plugins":[{"name":"context-engineering","description":"Comprehensive context engineering and harness engineering skills for production-grade AI agent systems: fundamentals, degradation patterns, compression, optimization, self-managed context (model-controlled live context editing), multi-agent coordination, long-horizon prompting (pseudo-formal task briefs for long-running and parallel agents), memory systems, tool design, filesystem context, hosted agents, evaluation, autonomous harnesses, self-improvement loops (RSI, meta-harnesses, evolutionary scaffold search), latent briefing (KV cache sharing between agents), project development, and cognitive architecture. Ships with a researcher operating system (rubrics, mechanism registry, claim provenance, run state machine, adversarial benchmarks, continuous loop).","source":"./","strict":false,"skills":["./skills/context-fundamentals","./skills/context-degradation","./skills/context-compression","./skills/context-optimization","./skills/self-managed-context","./skills/multi-agent-patterns","./skills/long-horizon-prompting","./skills/memory-systems","./skills/tool-design","./skills/filesystem-context","./skills/hosted-agents","./skills/evaluation","./skills/advanced-evaluation","./skills/harness-engineering","./skills/project-development","./skills/bdi-mental-states","./skills/latent-briefing","./skills/self-improvement-loops"]}]},"isInternal":false,"tokens":425,"sizeBytes":2014},{"name":"plugin.json","path":".plugin/plugin.json","rawUrl":"https://raw.githubusercontent.com/muratcankoylan/Agent-Skills-for-Context-Engineering/HEAD/.plugin/plugin.json","title":"context-engineering","category":"plugin-manifest","format":"json","content":"","frontmatter":{"name":"context-engineering","description":"Context engineering and harness engineering skills for production-grade AI agent systems: fundamentals, degradation patterns, compression, optimization, self-managed context (model-controlled live context editing), latent briefing (KV cache sharing between agents), multi-agent coordination, long-horizon prompting (pseudo-formal task briefs for long-running and parallel agents), memory systems, tool design, evaluation, autonomous harnesses, self-improvement loops (RSI, meta-harnesses, evolutionary scaffold search), and a file-based researcher operating system with deterministic gates, a continuous loop, and measured router-benchmark results across four frontier models.","version":"2.6.0","author":{"name":"Muratcan Koylan"},"skills":"./skills/"},"isInternal":false,"tokens":174,"sizeBytes":829}],"systemPromptSnippet":"\n\n# AGENTS.md\n\nWorkspace memory for agents collaborating on this repository. Keep entries durable and broadly applicable; one-off task state belongs in chat or in a run thread, not here.\n\n## Learned User Preferences\n\n- For autonomous research and repo-improvement work in this workspace, prefer proceeding through concrete research loops, subagents, validation, and edits when the scope is clear rather than asking broad process questions.\n- Avoid stale regex or keyword-list heuristics in skills and scripts; prefer mechanism-level criteria, rubrics, and evidence-backed validation.\n- Never push to GitHub or merge a PR without explicit user approval. Preparing branches, commits, and PRs is permitted only when the user has approved that specific action.\n- Tone is technical CTO: direct, no marketing language, no exclamation marks, no emojis, no em dashes. State trade-offs and complexity upfront.\n- When the scope spans multiple architectural decisions or irreversible changes, propose a plan first instead of executing.\n- For benchmarks and evaluation work, hold to research-paper-grade methodology (statistical discipline, bias mitigation, ablations, reproducibility) over speed. Don't rush.\n\n## Learned Workspace Facts\n\n- This repo is an autonomous research-to-skill organization. External AI research is curated through rubrics and distilled into context-engineering and harness-engineering skill updates.\n- `researcher/` is repo-native and file-based so agents can resume, audit, validate, and prepare PR-ready skill changes without a hosted scheduler.\n- Per-run state lives in `researcher/runs//run-state.json` with explicit transitions (`initialized -> retrieved -> evaluated -> proposed -> novelty_checked -> validated -> pr_ready -> closed`). Use `research_loop.py` subcommands to advance state, never hand-edit `run-state.json`.\n- Repo health (`validate_repo.py`) and per-run readiness (`validate_run.py`) are different questions. CI runs `validate_platform_compat.py --require-reference-validator`, `validate_repo.py --strict`, `skill_health.py --strict --no-history`, `run_benchmarks.py`, and `check_activation_cases.py` on every PR via `.github/workflows/validate.yml`.\n- The mechanism registry (`researcher/mechanisms/registry.jsonl`) is the encyclopedia backbone. Promotion is gated by `research_loop.py promote-mechanisms` with a recorded reviewer; ledgers live under `researcher/mechanisms/ledgers/`.\n- Claim provenance for numeric or volatile claims lives in `researcher/claims/index.jsonl`. Add an entry for any new benchmark or volatility-sensitive claim.\n- The corpus index (`researcher/corpus/index.json`) is the machine-readable map of skills, activation scenarios, mechanisms, and claims. Update it when adding or restructuring skills.\n- The continuous loop (`researcher/scripts/loop_*.py`) runs from launchd via `researcher/orchestration/launchd/`. It never invokes paid LLMs; HTTP retrieval is stdlib-only with a 1.5 MB cap and a 30-second timeout.\n- Runtime state is not committed: `researcher/queue/*.jsonl`, `researcher/queue/.locks/`, `researcher/reports/{logs,snapshots,loop-events.jsonl,loop-failures.jsonl,status.md,parked-review.md}`, and `researcher/runs/*/` are gitignored. The seed run `20260515-035228-executable-autonomous-research-frameworks` is the only committed run; it is closed as `reference-only` and serves as a worked example.\n- The current prepared release version is 2.6.0 across `.claude-plugin/marketplace.json`, `.plugin/plugin.json`, and root `SKILL.md`. There are 18 skills (latent-briefing covers KV cache sharing between agents; self-improvement-loops covers RSI, meta-harness search, and evolutionary scaffold optimization; long-horizon-prompting covers pseudo-formal task briefs for long-running and parallel agent runs; self-managed-context covers model-controlled live context editing).\n- New accepted-ledger events must carry `event_type`, `source_commit`, and a committed `source_path`; a `run_dir` pointer alone dangles on a clean checkout because run directories are gitignored. `research_loop.py promote-mechanisms` still emits the `run_dir` shape, so write durable events directly until it is fixed.\n- Detailed lessons from building the researcher OS live in `researcher/insights/auto-research-experiment.md` (engineering rationale) and `researcher/insights/how-we-built-this.md` (project narrative and sharing templates); read both before extending the harness or writing release-facing prose.\n- Benchmarks are staged in `researcher/benchmarks/`: Stage 0 deterministic harness (shipped), Stage 1 per-skill health via `researcher/scripts/skill_health.py` (shipped; output `researcher/reports/skill-health.json` is gitignored), Stage 2 router (shipped; results in `researcher/benchmarks/router/results-published/`), Stage 3 effectiveness (scaffolded, one task built), Stage 4 composition (future). `researcher/benchmarks/PLAN.md` is the methodology source of truth.\n- Current corpus counts and compatibility status are generated in `researcher/generated/corpus-summary.md`; do not copy live totals into workspace memory. Published benchmark reports remain dated snapshots. Do not describe a skill improvement as complete unless the prose, mechanism registry, claim index, corpus index, activation fixtures, generated inventory, and validators all agree.\n- Public export uses `validate_export.py plan|render|check` and registered transforms from `governance/export-policy.yaml`. Private plans and receipts are ignored; public manifests contain projection and output digests, never private source paths or input digests.\n- Durable cross-runtime records resolve through `researcher/schemas/registry.json`. JSON digests use the integer-only `jcs-rfc8785-integer-v1` profile; blobs hash exact bytes. `ArtifactRef` never contains a private locator or grants authority, and reads must validate its private `StorageBinding`. Candidate evaluation must use a SPEC-003 freeze receipt created under an explicit editable-surface policy, not a mutable worktree.\n- Benchmark execution uses the Cursor SDK runner at `researcher/benchmarks/sdk-runner/` (TypeScript, `@cursor/sdk` 1.0.13). The runner supports `--concurrency N`, `--no-resume`, per-run progress logging, format-failure retry, and worst-case retry-aware cost forecasting; default behavior is to resume by skipping plan items that already have result files. Result artifacts under `researcher/benchmarks/{router,effectiveness}/results/` and history JSONLs (`router-history.jsonl`, `effectiveness-history.jsonl`) are gitignored.\n- Published Stage 2 router-benchmark results: `researcher/benchmarks/router/results-published/2026-05-15.md` (baseline), `researcher/benchmarks/router/results-published/2026-05-15-v2.md` (post-rewrite with delta-vs-baseline table), and `researcher/benchmarks/router/results-published/2026-05-19.md` (post-corpus-hardening validation: 600/600 usable records, 0 format failures, top-1 Gemini 0.920 / Composer 0.913 / GPT-5.5 0.913 / Claude Opus 4.7 0.840). Headline finding: targeted description rewrites moved `context-fundamentals` top-1 by +23.4pp and `project-development` top-1 to 1.000; corpus-wide hardening did not cause broad routing collapse.\n\n## Repository Operating Defaults\n\n- Deterministic checks before model judges. Always run `validate_platform_compat.py --require-reference-validator` and `validate_repo.py --strict` before claiming a skill-format or packaging change is complete.\n- Adversarial benchmarks before declaring the harness safe. Add a scenario when a new failure mode is discovered.\n- Append-only ledgers for accepted and rejected mechanisms so future agents do not rediscover failed paths.\n- Atomic writes (`tempfile` + `os.replace`) and `fcntl` locks for any shared file the loop touches.\n- Live execution is the highest-signal validation for orchestration code; smoke-test changes against the actual loop before declaring them safe.\n- Cursor SDK is the only paid-API surface allowed for benchmarks. Privacy Mode required, `apiKey` passed explicitly per call, never `settingSources: \"all\"` in benchmarks (use `[]` for control, `[\"project\"]` with a curated `.cursor/skills/` for ablation). Cost gates (`--max-runs`, `--max-budget-usd`, or `--dry-run`) must be set before any SDK call.\n- Description quality is measurable. When changing skill activation descriptions, re-run the router benchmark with the same seed and fixture and publish the delta. Aggregate accuracy is a misleading unit; per-skill effect sizes and the confusion matrix are the right view.\n- A skill is a multi-surface artifact. Changing the frontmatter `description` is not enough; the SKILL.md body `When to Activate` and `Integration` sections must be audited the same day so the body does not contradict the description that routed the agent to it. The router benchmark only sees descriptions (`settingSources: []`) and cannot catch body inconsistencies; only Stage 3 effectiveness benchmarks (which actually load skill bodies) measure body-alignment impact.\n- Any runner that calls a paid API in a loop must have three features before execution: bounded parallelism via `--concurrency`, resume capability via results-folder scan, and per-run progress logging that surfaces stalls inside one call's duration.\n- API keys provided in chat should be considered exposed; rotate immediately after use. Runner enforces this via `apiKeyFingerprint()` which only logs the last 4 characters.\n\n\n# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nAgent Skills for Context Engineering is an open collection of Agent Skills teaching context engineering and harness engineering principles for production AI agent systems. Skills are platform-agnostic (Claude Code, Cursor, Codex/OpenAI Agent Skills, GitHub Copilot, any Open Plugins-conformant tool). v2.6.0 adds the self-managed-context skill (model-controlled live context editing: context as an editable file, self-editing harness invariants, edit-position cache cost, budget nudges, and steering or training context-editing strategy) on top of the long-horizon-prompting skill (v2.5.0), the self-improvement-loops skill (v2.4.0), and the file-based researcher operating system with deterministic gates, cross-platform Agent Skills validation, and a continuous loop. Use the [generated live inventory](researcher/generated/corpus-summary.md) for current corpus counts.\n\nContext engineering is the discipline of curating everything that enters a model's context window (system prompts, tool definitions, retrieved documents, message history, tool outputs) to maximize signal within limited attention budget.\n\n## Repository Structure\n\n- `skills/` - published skill directories, each containing a `SKILL.md` with YAML frontmatter (`name`, `description`) and optional `references/` and `scripts/` subdirectories\n- `examples/` - complete demonstration projects; the generated inventory is the count authority\n- `docs/` - Research materials and reference documentation\n- `researcher/` - File-based research-to-skill operating system: rubrics, mechanism registry, claim provenance, corpus index, run state machine, adversarial benchmarks, continuous loop, launchd service definitions\n- `template/SKILL.md` - Canonical skill template (use when creating new skills)\n- `SKILL.md` (root) - Collection-level metadata and skill map\n- `.claude-plugin/marketplace.json` - Claude Code marketplace manifest (single bundled plugin, v2.6.0)\n- `.plugin/plugin.json` - Open Plugins format manifest (v2.6.0)\n\n## Build & Test Commands\n\nNo top-level build system. Repo-level gates and per-project tooling below.\n\n### Top-level deterministic gates (run on every PR via CI)\n\n```\npython3 -m unittest researcher.scripts.tests.test_skill_frontmatter # parser and strict-YAML regression tests\npython3 researcher/scripts/validate_governance.py --check # authority model and generated view\npython3 researcher/scripts/build_inventory.py --check # corpus references and generated inventory\npython3 researcher/scripts/validate_platform_compat.py --require-reference-validator # Agent Skills reference validator + Cursor/Claude/Codex install-layout simulation\npython3 researcher/scripts/validate_repo.py --strict # corpus structure, manifests, rubric math, mechanism registry, claims, corpus index, activation cases, benchmark scenarios, run artifacts\npython3 researcher/scripts/skill_health.py --strict --no-history # deterministic skill-body quality gate\npython3 researcher/scripts/run_benchmarks.py # adversarial benchmark harness + repo + activation gates\npython3 researcher/scripts/check_activation_cases.py # skill-boundary regression fixtures\n```\n\n### Per-run readiness (active runs only)\n\n```\npython3 researcher/scripts/validate_run.py --run-dir researcher/runs/\n```\n\n### Continuous loop (manual or launchd)\n\n```\npython3 researcher/scripts/loop_discover.py\npython3 researcher/scripts/loop_step.py --allow-fetch\npython3 researcher/scripts/loop_daily.py\npython3 researcher/scripts/loop_status.py\n\nresearcher/orchestration/launchd/install.sh # macOS daemon\nresearcher/orchestration/launchd/uninstall.sh\n```\n\n### Example projects\n\n#### examples/llm-as-judge-skills (TypeScript, Node >= 18)\n```\ncd examples/llm-as-judge-skills\nnpm install\nnpm run build # tsc\nnpm test # vitest (19 tests)\nnpm run lint # eslint\nnpm run format # prettier\nnpm run typecheck # tsc --noEmit\n```\n\n#### examples/interleaved-thinking (Python >= 3.10)\n```\ncd examples/interleaved-thinking\npip install -e \".[dev]\"\npytest # pytest + pytest-asyncio\nruff check . # linting (100 char line length)\n```\n\n#### examples/digital-brain-skill (Node.js)\n```\ncd examples/digital-brain-skill\nnpm run setup\nnpm run weekly-review\nnpm run content-ideas\nnpm run stale-contacts\n```\n\n## Skill Authoring Rules\n\nWhen creating or editing skills:\n\n1. **SKILL.md must stay under 500 lines**: move detailed content to `references/` directory\n2. **YAML frontmatter is required**: must include `name` and `description` fields\n3. **Folder naming**: lowercase with hyphens (e.g., `context-fundamentals`)\n4. **Write in third person**: descriptions are injected into system prompts; inconsistent POV causes discovery issues\n5. **Platform-agnostic**: no vendor-locked examples or platform-specific tool names without abstraction\n6. **Token-conscious**: challenge each paragraph and assume an advanced audience\n7. **Body standard**: include `When to Activate`, `Core Concepts`, `Practical Guidance`, `Examples`, `Guidelines`, `Gotchas`, `Integration`, and `References`\n8. **Explicit boundaries**: every `When to Activate` section needs positive triggers plus a `Do not activate` block routing adjacent work to the right skill\n9. **Include a Gotchas section**: experience-derived failure modes are the highest-signal content in any skill\n10. **Update root README.md** when adding new skills\n11. **Update marketplace/plugin manifests** when adding skills (`.claude-plugin/marketplace.json`, `.plugin/plugin.json`)\n12. **Update the corpus index** (`researcher/corpus/index.json`) to map the new skill to activation scenarios, mechanism IDs, and claim IDs\n13. **Update mechanisms and claims**: add registry entries for reusable behavior changes and `claim-*` provenance for numeric, benchmark, volatile, or vendor-performance claims\n14. **Run `validate_platform_compat.py --require-reference-validator`, `validate_repo.py --strict`, `skill_health.py --strict --no-history`, `check_activation_cases.py`, and `run_benchmarks.py`** before committing skill changes\n\n## Researcher OS Rules\n\nWhen working through the researcher operating system:\n\n1. **Initialize runs via `research_loop.py init`**: it creates `run-state.json`, queue entry, thread log, source evaluation scaffold, and mechanism proposal template\n2. **Advance state explicitly**: use `retrieve`, `evaluate`, `propose`, `novelty`, `validate-run`, `pr-ready`, `close` subcommands; do not edit `run-state.json` by hand\n3. **Promote mechanisms only after run readiness**: `research_loop.py promote-mechanisms` requires `--reviewed-by` and a passing run-readiness check\n4. **Add claim provenance** to `researcher/claims/index.jsonl` for any numeric, benchmark, or volatile claim added to a skill\n5. **Never invoke paid LLMs from the continuous loop**: HTTP retrieval is stdlib-only, judge adapters are explicitly out of scope until budget-gated\n6. **Never commit runtime queue/report files**: `.gitignore` covers `researcher/queue/*.jsonl`, `researcher/reports/{logs,snapshots,loop-events.jsonl,loop-failures.jsonl,status.md,parked-review.md}`, and `researcher/runs/*/` except the seed run\n\n## Plugin Architecture\n\nAll published skills are distributed as a single plugin (`context-engineering`) in the marketplace manifest. This avoids cache duplication: Claude Code caches each plugin's `source` directory separately, so multiple plugins pointing to `source: \"./\"` would each cache a full copy of the repo.\n\nProgressive disclosure pattern: only skill names/descriptions load at startup; full content loads on activation.\n\n## Key Design Principles\n\n- **Context quality over quantity**: attention scarcity and lost-in-middle behavior mean more context is not always better\n- **Sub-agents isolate context**: they exist to manage attention budget, not simulate org roles\n- **Skills reference each other**: use plain text skill names (not links) in Integration sections to avoid cross-directory reference issues\n- **Examples use Python pseudocode**: conceptual demonstrations that work across environments, not production-ready implementations\n- **Deterministic first, model-judged second**: structure, schema, rubric math, manifest sync, retrieval status, and registry shape must pass before any LLM judge is invoked\n- **Human-controlled merge**: agents may prepare PRs and pass gates, but push and merge always require explicit human approval\n\n\n---\nname: book-sft-pipeline\ndescription: \"This skill should be used for book-to-SFT pipelines: ePub extraction, literary segmentation, author-voice dataset construction, style-transfer training, LoRA workflows, and model evaluation for voice replication.\"\nversion: 2.0.0\n---\n\n# Book SFT Pipeline\n\nA complete system for converting books into SFT datasets and training style-transfer models. This skill teaches the pipeline from raw ePub to a model that writes in any author's voice.\n\n## When to Activate\n\nActivate this skill when:\n- Building fine-tuning datasets from literary works\n- Creating author-voice or style-transfer models\n- Preparing training data for Tinker or similar SFT platforms\n- Designing text segmentation pipelines for long-form content\n- Training small models (8B or less) on limited data\n\n## Core Concepts\n\n### The Three Pillars of Book SFT\n\n**1. Intelligent Segmentation**\nText chunks must be semantically coherent. Breaking mid-sentence teaches the model to produce fragmented output. Target: 150-400 words per chunk, always at natural boundaries.\n\n**2. Diverse Instruction Generation**\nUse multiple prompt templates and system prompts to prevent overfitting. A single prompt style leads to memorization. Use 15+ prompt templates with 5+ system prompts.\n\n**3. Style Over Content**\nThe goal is learning the author's rhythm and vocabulary patterns, not memorizing plots. Synthetic instructions describe what happens without quoting the text.\n\n## Pipeline Architecture\n\n```\n┌─────────────────────────────────────────────────────────────────┐\n│ ORCHESTRATOR AGENT │\n│ Coordinates pipeline phases, manages state, handles failures │\n└──────────────────────┬──────────────────────────────────────────┘\n │\n ┌───────────────┼───────────────┬───────────────┐\n ▼ ▼ ▼ ▼\n┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐\n│ EXTRACTION │ │ SEGMENTATION │ │ INSTRUCTION │ │ DATASET │\n│ AGENT │ │ AGENT │ │ AGENT │ │ BUILDER │\n│ ePub → Text │ │ Text → Chunks│ │ Chunks → │ │ Pairs → │\n│ │ │ 150-400 words│ │ Prompts │ │ JSONL │\n└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘\n │\n ┌───────────────┴───────────────┐\n ▼ ▼\n┌──────────────┐ ┌──────────────┐\n│ TRAINING │ │ VALIDATION │\n│ AGENT │ │ AGENT │\n│ LoRA on │ │ AI detector │\n│ Tinker │ │ Originality │\n└──────────────┘ └──────────────┘\n```\n\n## Phase 1: Text Extraction\n\n### Critical Rules\n1. **Always source ePub over PDF** - OCR errors become learned patterns\n2. **Use paragraph-level extraction** - Extract from `

` tags to preserve breaks\n3. **Remove front/back matter** - Copyright and TOC pollute the dataset\n\n```python\n# Extract text from ePub paragraphs\nfrom epub2 import EPub\nfrom bs4 import BeautifulSoup\n\ndef extract_epub(path):\n book = EPub(path)\n chapters = []\n for item in book.flow:\n html = book.get_chapter(item.id)\n soup = BeautifulSoup(html, 'html.parser')\n paragraphs = [p.get_text().strip() for p in soup.find_all('p')]\n chapters.append('\\n\\n'.join(p for p in paragraphs if p))\n return '\\n\\n'.join(chapters)\n```\n\n## Phase 2: Intelligent Segmentation\n\n### Smaller Chunks + Overlap\n\nSmaller chunks (150-400 words) produce more training examples and better style transfer than larger chunks (250-650).\n\n```python\ndef segment(text, min_words=150, max_words=400):\n paragraphs = text.split('\\n\\n')\n chunks, buffer, buffer_words = [], [], 0\n \n for para in paragraphs:\n words = len(para.split())\n if buffer_words + words > max_words and buffer_words >= min_words:\n chunks.append('\\n\\n'.join(buffer))\n # Keep last paragraph for overlap\n buffer = [buffer[-1], para] if buffer else [para]\n buffer_words = sum(len(p.split()) for p in buffer)\n else:\n buffer.append(para)\n buffer_words += words\n \n if buffer:\n chunks.append('\\n\\n'.join(buffer))\n return chunks\n```\n\n### Expected Results\n\nFor an 86,000-word book:\n- Old method (250-650 words): ~150 chunks\n- New method (150-400 + overlap): ~300 chunks\n- With 2 variants per chunk: 600+ training examples\n\n## Phase 3: Diverse Instruction Generation\n\n### The Key Insight\n\nUsing a single prompt template causes memorization. Diverse templates teach the underlying style.\n\n```python\nSYSTEM_PROMPTS = [\n \"You are an expert creative writer capable of emulating specific literary styles.\",\n \"You are a literary writer with deep knowledge of classic prose styles.\",\n \"You are a creative writer skilled at emulating distinctive authorial voices.\",\n \"You write prose that captures the essence of modernist literature.\",\n \"You are a talented writer who can channel classic American authors.\",\n]\n\nPROMPT_TEMPLATES = [\n \"Write a passage in the style of {author}: {desc}\",\n \"Channel {author}'s voice to write about: {desc}\",\n \"In {author}'s distinctive prose style, describe: {desc}\",\n \"Write this scene as {author} would have: {desc}\",\n \"Using {author}'s repetitive technique, describe: {desc}\",\n \"Capture the rhythm of {author} in this passage: {desc}\",\n \"Write like {author}: {desc}\",\n \"In the voice of {author}, write: {desc}\",\n \"This is a literary exercise. Write like {author}: {desc}\",\n \"Can you write in {author}'s style? {desc}\",\n]\n```\n\n### Instruction Generation\n\n```python\nINSTRUCTION_PROMPT = \"\"\"Describe what is happening in this excerpt in 2-3 sentences.\nFocus on: characters present, actions, emotions, setting.\nDo NOT quote the text directly.\n\nExcerpt:\n{text}\n\"\"\"\n\n# Use a fast, cheap LLM (e.g., Gemini Flash)\ninstruction = llm_call(INSTRUCTION_PROMPT.format(text=chunk))\n```\n\n## Phase 4: Dataset Construction\n\n### Message Format\n\n```json\n{\n \"messages\": [\n {\"role\": \"system\", \"content\": \"You are an expert creative writer...\"},\n {\"role\": \"user\", \"content\": \"Write in the style of Author: Scene description...\"},\n {\"role\": \"assistant\", \"content\": \"The actual book text from chunk...\"}\n ]\n}\n```\n\n### Multiple Variants Per Chunk\n\n```python\ndef build_examples(chunk, instruction, author, variants=2):\n examples = []\n for i in range(variants):\n system = SYSTEM_PROMPTS[i % len(SYSTEM_PROMPTS)]\n template = PROMPT_TEMPLATES[(chunk.id + i) % len(PROMPT_TEMPLATES)]\n user = template.format(author=author, desc=instruction)\n examples.append({\"messages\": [\n {\"role\": \"system\", \"content\": system},\n {\"role\": \"user\", \"content\": user},\n {\"role\": \"assistant\", \"content\": chunk.text}\n ]})\n return examples\n```\n\n## Phase 5: LoRA Training on Tinker\n\n### Configuration\n\n```python\nCONFIG = {\n \"model_name\": \"Qwen/Qwen3-8B-Base\", # Base, not instruct\n \"lora_rank\": 32, # 352MB adapter\n \"learning_rate\": 5e-4, # Higher for LoRA\n \"batch_size\": 4,\n \"epochs\": 3,\n}\n```\n\n### Why Base Model?\n\nUse **base** (pretrained) models, not instruction-tuned versions:\n- Base models are more malleable for new styles\n- Instruct models have patterns that resist overwriting\n- Style is a low-level pattern that base models capture better\n\n### Training Loop\n\n```python\nimport tinker\nfrom tinker import types\n\ntraining_client = await service_client.create_lora_training_client_async(\n base_model=\"Qwen/Qwen3-8B-Base\",\n rank=32\n)\n\nfor epoch in range(3):\n for batch in batches:\n await training_client.forward_backward_async(batch, loss_fn=\"cross_entropy\")\n await training_client.optim_step_async(types.AdamParams(learning_rate=5e-4))\n\nresult = await training_client.save_weights_for_sampler_async(name=\"final\")\n```\n\n## Phase 6: Validation\n\n### Modern Scenario Test\n\nTest with scenarios that couldn't exist in the original book:\n\n```python\nTEST_PROMPTS = [\n \"Write about a barista making lattes\",\n \"Describe lovers communicating through text messages\",\n \"Write about someone anxious about climate change\",\n]\n```\n\nIf the model applies style markers to modern scenarios, it learned **style**, not **content**.\n\n### Originality Verification\n\n```bash\n# Search training data for output phrases\ngrep \"specific phrase from output\" dataset.jsonl\n# Should return: No matches\n```\n\n### AI Detector Testing\n\nTest outputs with GPTZero, Pangram, or ZeroGPT.\n\n## Known Issues and Solutions\n\n### Character Name Leakage\n\n**Symptom**: Model uses original character names in new scenarios.\n**Cause**: Limited name diversity from one book.\n**Solution**: Train on multiple books or add synthetic examples.\n\n### Model Parrots Exact Phrases\n\n**Symptom**: Outputs contain exact sentences from training data.\n**Cause**: Too few prompt variations or too many epochs.\n**Solution**: Use 15+ templates, limit to 3 epochs.\n\n### Fragmented Outputs\n\n**Symptom**: Sentences feel incomplete.\n**Cause**: Poor segmentation breaking mid-thought.\n**Solution**: Always break at paragraph boundaries.\n\n## Guidelines\n\n1. **Always source ePub over PDF** - OCR errors become learned patterns\n2. **Never break mid-sentence** - Boundaries must be grammatically complete\n3. **Use diverse prompts** - 15+ templates, 5+ system prompts\n4. **Use base models** - Not instruct versions\n5. **Use smaller chunks** - 150-400 words for more examples\n6. **Reserve test set** - 50 examples minimum\n7. **Test on modern scenarios** - Proves style transfer vs memorization\n8. **Verify originality** - Grep training data for output phrases\n\n## Expected Results\n\n| Metric | Value |\n|--------|-------|\n| Training examples | 500-1000 per book |\n| Model | Qwen/Qwen3-8B-Base |\n| LoRA rank | 32 |\n| Adapter size | ~350 MB |\n| Training time | ~15 min |\n| Loss reduction | 90%+ |\n| Style transfer success | ~50% perfect |\n\n## Cost Estimate\n\n| Component | Cost |\n|-----------|------|\n| LLM (instruction generation) | ~$0.50 |\n| Tinker training (15 min) | ~$1.50 |\n| **Total** | **~$2.00** |\n\n## Integration with Context Engineering Skills\n\nThis example applies several skills from the Agent Skills for Context Engineering collection:\n\n### project-development\nThe pipeline follows the staged, idempotent architecture pattern:\n- **Acquire**: Extract text from ePub\n- **Prepare**: Segment into training chunks\n- **Process**: Generate synthetic instructions\n- **Parse**: Build message format\n- **Render**: Output Tinker-compatible JSONL\n- **Train**: LoRA fine-tuning\n- **Validate**: Modern scenario testing\n\nEach phase is resumable and produces intermediate artifacts for debugging.\n\n### context-compression\nSegmentation is a form of context compression for training. The core insight from context-compression applies: information density matters more than information quantity. Smaller, coherent chunks (150-400 words) produce better style transfer than larger, diluted chunks.\n\nThe two-tier strategy mirrors context compression evaluation:\n- Tier 1: Fast, deterministic compression\n- Tier 2: LLM-assisted for edge cases\n\n### multi-agent-patterns\nThe pipeline uses the **supervisor/orchestrator** pattern:\n- Orchestrator coordinates phases and manages state\n- Specialized agents (Extraction, Segmentation, Instruction, Builder) have isolated contexts\n- Each agent receives only the information needed for its task\n\nThis matches the principle that sub-agents exist primarily to isolate context rather than simulate roles.\n\n### evaluation\nValidation follows the **end-state evaluation** pattern:\n- Functional testing: Does output match expected style markers?\n- Originality verification: Is content genuinely generated?\n- External validation: AI detector scores\n\nThe \"modern scenario\" test is a form of out-of-distribution evaluation that proves generalization.\n\n### context-fundamentals\nPrompt diversity prevents attention collapse on single patterns. When training with identical prompt structures, the model memorizes the instruction-response mapping. Diverse templates force attention across the style patterns themselves.\n\n## References\n\nInternal references:\n- [Segmentation Strategies](./references/segmentation-strategies.md) - Text chunking patterns\n- [Tinker Format Specification](./references/tinker-format.md) - Datum structure\n- [Tinker API Documentation](./references/tinker.txt) - Full API reference\n\nRelated skills from Agent Skills for Context Engineering:\n- project-development - Pipeline architecture patterns\n- context-compression - Compression strategies \n- multi-agent-patterns - Agent coordination\n- evaluation - Evaluation frameworks\n- context-fundamentals - Attention and information density\n\nExternal resources:\n- [Research Paper](https://arxiv.org/pdf/2510.13939) - Chakrabarty et al. 2025\n- [Dataset on Hugging Face](https://huggingface.co/datasets/MuratcanKoylan/gertrude-stein-style-sft)\n- [Gertrude Stein Case Study](./examples/gertrude-stein/) - Complete working example\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-26\n**Last Updated**: 2025-12-28\n**Author**: Muratcan Koylan\n**Version**: 2.0.0\n**Standalone**: Yes (separate from main context-engineering collection)\n\n\n---\nname: digital-brain\ndescription: \"This skill should be used for personal operating-system workflows: content creation, voice consistency, relationship lookup, meeting preparation, weekly review, goal tracking, personal brand management, and network management.\"\nversion: 1.0.0\n---\n\n# Digital Brain\n\nA structured personal operating system for managing digital presence, knowledge, relationships, and goals with AI assistance. Designed for founders building in public, content creators growing their audience, and tech-savvy professionals seeking AI-assisted personal management.\n\n**Important**: This skill uses progressive disclosure. Module-specific instructions are in each subdirectory's `.md` file. Only load what's needed for the current task.\n\n## When to Activate\n\nActivate this skill when the user:\n\n- Requests content creation (posts, threads, newsletters) - load identity/voice.md first\n- Asks for help with personal brand or positioning\n- Needs to look up or manage contacts/relationships\n- Wants to capture or develop content ideas\n- Requests meeting preparation or follow-up\n- Asks for weekly reviews or goal tracking\n- Needs to save or retrieve bookmarked resources\n- Wants to organize research or learning materials\n\n**Trigger phrases**: \"write a post\", \"my voice\", \"content ideas\", \"who is [name]\", \"prepare for meeting\", \"weekly review\", \"save this\", \"my goals\"\n\n## Core Concepts\n\n### Progressive Disclosure Architecture\n\nThe Digital Brain follows a three-level loading pattern:\n\n| Level | When Loaded | Content |\n|-------|-------------|---------|\n| **L1: Metadata** | Always | This SKILL.md overview |\n| **L2: Module Instructions** | On-demand | `[module]/[MODULE].md` files |\n| **L3: Data Files** | As-needed | `.jsonl`, `.yaml`, `.md` data |\n\n### File Format Strategy\n\nFormats chosen for optimal agent parsing:\n\n- **JSONL** (`.jsonl`): Append-only logs - ideas, posts, contacts, interactions\n- **YAML** (`.yaml`): Structured configs - goals, values, circles\n- **Markdown** (`.md`): Narrative content - voice, brand, calendar, todos\n- **XML** (`.xml`): Complex prompts - content generation templates\n\n### Append-Only Data Integrity\n\nJSONL files are **append-only**. Never delete entries:\n- Mark as `\"status\": \"archived\"` instead of deleting\n- Preserves history for pattern analysis\n- Enables \"what worked\" retrospectives\n\n## Detailed Topics\n\n### Module Overview\n\n```\ndigital-brain/\n├── identity/ → Voice, brand, values (READ FIRST for content)\n├── content/ → Ideas, drafts, posts, calendar\n├── knowledge/ → Bookmarks, research, learning\n├── network/ → Contacts, interactions, intros\n├── operations/ → Todos, goals, meetings, metrics\n└── agents/ → Automation scripts\n```\n\n### Identity Module (Critical for Content)\n\n**Always read `identity/voice.md` before generating any content.**\n\nContains:\n- `voice.md` - Tone, style, vocabulary, patterns\n- `brand.md` - Positioning, audience, content pillars\n- `values.yaml` - Core beliefs and principles\n- `bio-variants.md` - Platform-specific bios\n- `prompts/` - Reusable generation templates\n\n### Content Module\n\nPipeline: `ideas.jsonl` → `drafts/` → `posts.jsonl`\n\n- Capture ideas immediately to `ideas.jsonl`\n- Develop in `drafts/` using `templates/`\n- Log published content to `posts.jsonl` with metrics\n- Plan in `calendar.md`\n\n### Network Module\n\nPersonal CRM with relationship tiers:\n- `inner` - Weekly touchpoints\n- `active` - Bi-weekly touchpoints\n- `network` - Monthly touchpoints\n- `dormant` - Quarterly reactivation checks\n\n### Operations Module\n\nProductivity system with priority levels:\n- P0: Do today, blocking\n- P1: This week, important\n- P2: This month, valuable\n- P3: Backlog, nice to have\n\n## Practical Guidance\n\n### Content Creation Workflow\n\n```\n1. Read identity/voice.md (REQUIRED)\n2. Check identity/brand.md for topic alignment\n3. Reference content/posts.jsonl for successful patterns\n4. Use content/templates/ as starting structure\n5. Draft matching voice attributes\n6. Log to posts.jsonl after publishing\n```\n\n### Pre-Meeting Preparation\n\n```\n1. Look up contact: network/contacts.jsonl\n2. Get history: network/interactions.jsonl\n3. Check pending: operations/todos.md\n4. Generate brief with context\n```\n\n### Weekly Review Process\n\n```\n1. Run: python agents/scripts/weekly_review.py\n2. Review metrics in operations/metrics.jsonl\n3. Check stale contacts: agents/scripts/stale_contacts.py\n4. Update goals progress in operations/goals.yaml\n5. Plan next week in content/calendar.md\n```\n\n## Examples\n\n### Example: Writing an X Post\n\n**Input**: \"Help me write a post about AI agents\"\n\n**Process**:\n1. Read `identity/voice.md` → Extract voice attributes\n2. Check `identity/brand.md` → Confirm \"ai_agents\" is a content pillar\n3. Reference `content/posts.jsonl` → Find similar successful posts\n4. Draft post matching voice patterns\n5. Suggest adding to `content/ideas.jsonl` if not publishing immediately\n\n**Output**: Post draft in user's authentic voice with platform-appropriate format.\n\n### Example: Contact Lookup\n\n**Input**: \"Prepare me for my call with Sarah Chen\"\n\n**Process**:\n1. Search `network/contacts.jsonl` for \"Sarah Chen\"\n2. Get recent entries from `network/interactions.jsonl`\n3. Check `operations/todos.md` for pending items with Sarah\n4. Compile brief: role, context, last discussed, follow-ups\n\n**Output**: Pre-meeting brief with relationship context.\n\n## Guidelines\n\n1. **Voice First**: Always read `identity/voice.md` before any content generation\n2. **Append Only**: Never delete from JSONL files - archive instead\n3. **Update Timestamps**: Set `updated` field when modifying tracked data\n4. **Cross-Reference**: Knowledge informs content, network informs operations\n5. **Log Interactions**: Always log meetings/calls to `interactions.jsonl`\n6. **Preserve History**: Past content in `posts.jsonl` informs future performance\n\n## Integration\n\nThis skill integrates context engineering principles:\n\n- **context-fundamentals** - Progressive disclosure, attention budget management\n- **memory-systems** - JSONL for persistent memory, structured recall\n- **tool-design** - Scripts in `agents/scripts/` follow tool design principles\n- **context-optimization** - Module separation prevents context bloat\n\n## References\n\nInternal references:\n- [Identity Module](./identity/IDENTITY.md) - Voice and brand details\n- [Content Module](./content/CONTENT.md) - Content pipeline docs\n- [Network Module](./network/NETWORK.md) - CRM documentation\n- [Operations Module](./operations/OPERATIONS.md) - Productivity system\n- [Agent Scripts](./agents/AGENTS.md) - Automation documentation\n\nExternal resources:\n- [Agent Skills for Context Engineering](https://github.com/muratcankoylan/Agent-Skills-for-Context-Engineering)\n- [Anthropic Context Engineering Guide](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)\n\n---\n\n## Skill Metadata\n\n**Created**: 2024-12-29\n**Last Updated**: 2024-12-29\n**Author**: Murat Can Koylan\n**Version**: 1.0.0\n\n\n{\n \"task\": \"Research the topic of \\\"context engineering for AI agents\\\" and create a comprehensive summary.\\n\\nYour research should:\\n1. Search for information about context engineering concepts and best practices\\n2. Read relevant sources to gather detailed information\\n3. Check the local project files for any existing research notes\\n4. Save important findings as notes for future reference\\n5. Write a final summary report to ./output/research_summary.md\\n\\nThe summary should include:\\n- Key concepts and definitions\\n- Best practices and techniques (including the \\\"lost in the middle\\\" problem)\\n- Practical recommendations for agent developers\\n- References to sources consulted (use actual URLs from your research)\",\n \"iterations\": 10,\n \"initial_score\": 67.6,\n \"final_score\": 70,\n \"improvement\": 3.550295857988174,\n \"converged\": true,\n \"generated_at\": \"2026-01-11T18:02:57.336789\"\n}\n\n\n[\n {\n \"type\": \"tool_confusion\",\n \"severity\": \"medium\",\n \"description\": \"Agent attempted to fetch non-existent or unreachable URLs without adjusting approach\",\n \"suggestion\": \"When a URL fetch fails, search for alternative URLs or verify the URL structure. Consider using search to find the correct documentation pages.\",\n \"iteration\": 1\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent didn't validate the completeness of gathered information or verify key claims\",\n \"suggestion\": \"Before writing the final report, explicitly validate that all required topics are covered. Create a checklist of requirements and verify each one is addressed.\",\n \"iteration\": 1\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"low\",\n \"description\": \"Agent made redundant searches and didn't optimize tool calls\",\n \"suggestion\": \"Track previously found URLs to avoid redundant searches. When a useful URL is found in one search, use it directly rather than searching again for the same topic.\",\n \"iteration\": 1\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"low\",\n \"description\": \"Thinking blocks are sparse and don't show deep analysis of alternatives or trade-offs\",\n \"suggestion\": \"In thinking blocks, explicitly list what information has been gathered, what gaps remain, and what decisions are being made. Use structured checklists.\",\n \"iteration\": 1\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"high\",\n \"description\": \"Agent failed to properly handle or acknowledge tool errors, particularly the failed URL fetch for Anthropic context windows documentation\",\n \"suggestion\": \"Add explicit error handling for failed tool calls - when a read_url fails, the agent should acknowledge it and either retry, try an alternative source, or explicitly note that information is missing rather than proceeding as if it succeeded\",\n \"iteration\": 2\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"medium\",\n \"description\": \"Agent did not verify or validate the relevance of search results before committing to reading sources\",\n \"suggestion\": \"After receiving search results, explicitly evaluate and rank sources by relevance to the research question before deciding which URLs to read. This saves token costs and ensures better source quality.\",\n \"iteration\": 2\n },\n {\n \"type\": \"premature_conclusion\",\n \"severity\": \"low\",\n \"description\": \"Agent prematurely declared having 'enough information' despite not yet completing all research phases\",\n \"suggestion\": \"Before declaring research complete, create a checklist of what information is still needed and verify each item is adequately covered. Set explicit criteria for 'enough information' at task start.\",\n \"iteration\": 2\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent accepted information without verifying it and failed to handle errors gracefully\",\n \"suggestion\": \"Implement explicit error checking after each tool call. If a read_url fails, acknowledge the failure and try an alternative source. Cross-reference key claims across multiple sources before including them in the final report.\",\n \"iteration\": 3\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"medium\",\n \"description\": \"Agent gathered information but didn't deeply analyze or synthesize insights\",\n \"suggestion\": \"After reading sources, explicitly document what was learned, what contradictions exist, and what gaps remain. Create a synthesis section that combines insights from multiple sources rather than just reporting them separately.\",\n \"iteration\": 3\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"low\",\n \"description\": \"Agent used tools but didn't fully leverage results or handle failures properly\",\n \"suggestion\": \"Immediately act on directory listing results. If a directory is empty, plan when to create notes rather than waiting. Implement proper error handling for tool failures and check response status codes before proceeding.\",\n \"iteration\": 3\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"medium\",\n \"description\": \"Agent uses list_directory to verify file creation instead of the more reliable read_file method\",\n \"suggestion\": \"Use read_file to verify file write success since it confirms both file existence and content; list_directory may not immediately reflect recent filesystem changes\",\n \"iteration\": 4\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent reads a URL that returns an error but doesn't acknowledge or log this failure, potentially missing important context\",\n \"suggestion\": \"Implement explicit error handling for failed URL reads - log which sources failed and consider searching for alternative sources or documentation\",\n \"iteration\": 4\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"low\",\n \"description\": \"Agent doesn't explain why it chose certain sources or how it evaluated source quality; research appears thorough but reasoning process is opaque\",\n \"suggestion\": \"Add explicit reasoning about source selection criteria (e.g., prioritizing official documentation, recent publications, peer-reviewed papers) and evaluation of source credibility\",\n \"iteration\": 4\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent accepts incomplete results without acknowledging failures or seeking alternatives\",\n \"suggestion\": \"When tool calls fail, explicitly note the failure in thinking blocks, consider alternative sources, and document what information gaps exist. Add a validation step to confirm all critical sources were successfully retrieved.\",\n \"iteration\": 5\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"low\",\n \"description\": \"Agent doesn't demonstrate analytical depth when processing source material\",\n \"suggestion\": \"After reading sources, explicitly state: (a) key findings from each source, (b) how they relate to the research goal, (c) any contradictions or complementary findings, (d) what additional information is needed\",\n \"iteration\": 5\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"low\",\n \"description\": \"Inefficient tool usage pattern - multiple web searches without reading all results first\",\n \"suggestion\": \"Before making additional searches, review the URLs from previous search results. A better pattern would be: search -> read all relevant sources -> identify gaps -> targeted additional searches only if needed\",\n \"iteration\": 5\n },\n {\n \"type\": \"context_degradation\",\n \"severity\": \"low\",\n \"description\": \"Vague thinking blocks that don't show active reasoning process\",\n \"suggestion\": \"Make thinking blocks more explicit: show intermediate conclusions, decision points, how each source contributed, and how conclusions evolved. The thinking trace should be readable as a standalone explanation of the research process.\",\n \"iteration\": 5\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent does not validate information across sources or verify accuracy of gathered content\",\n \"suggestion\": \"Add explicit validation steps: compare information across multiple sources, verify claims against original papers, include confidence assessments for key findings\",\n \"iteration\": 7\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"low\",\n \"description\": \"Inefficient tool usage - read_url calls lack systematic prioritization and some results may not have been fully utilized\",\n \"suggestion\": \"Implement a source prioritization matrix before reading URLs; explicitly note how each source will contribute to the research before fetching\",\n \"iteration\": 7\n },\n {\n \"type\": \"hallucination\",\n \"severity\": \"low\",\n \"description\": \"Potential source misattribution in final report - cites Google Research Chain of Thought paper but source wasn't fetched in thinking trace\",\n \"suggestion\": \"Only cite sources that were actually retrieved and read; if a source is referenced from memory, clearly indicate it as secondary/indirect reference\",\n \"iteration\": 7\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"Agent accepts search results without validating source relevance or quality before proceeding to read URLs\",\n \"suggestion\": \"Add explicit validation steps: list the top 3-5 sources with brief rationale for selection, note any potential gaps in coverage, and prioritize primary authoritative sources before secondary ones\",\n \"iteration\": 8\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"medium\",\n \"description\": \"Thinking blocks are extremely sparse and lack intermediate analysis - agent doesn't explain HOW it's interpreting information or making decisions\",\n \"suggestion\": \"Implement structured reflection after each major information-gathering step: What did I learn? How does this connect to what I already know? What gaps remain? What should I prioritize next?\",\n \"iteration\": 8\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"low\",\n \"description\": \"Agent encounters a failed tool call (404 error on Anthropic context-windows URL) but doesn't acknowledge or recover in thinking\",\n \"suggestion\": \"Add explicit error acknowledgment: 'Attempted X but failed with Y error. Will try alternative Z or note this as a gap.' This improves debugging and transparency\",\n \"iteration\": 8\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"low\",\n \"description\": \"The agent reaches conclusions about having 'comprehensive information' after limited tool interactions, without explicitly documenting what was learned or what gaps remain\",\n \"suggestion\": \"Add more detailed reasoning about what specific information was gained from each source and what questions remain unanswered before claiming comprehensive understanding\",\n \"iteration\": 9\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"low\",\n \"description\": \"The agent doesn't explicitly validate assumptions or cross-reference information between sources. The 'Lost in the Middle' paper is mentioned multiple times but not critically compared against other sources\",\n \"suggestion\": \"After reading multiple sources, explicitly compare findings, note contradictions, and validate key claims against multiple sources before proceeding\",\n \"iteration\": 9\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"medium\",\n \"description\": \"The agent attempted to read a URL that returned an error (https://docs.anthropic.com/en/docs/build-with-claude/context-windows) but proceeded without acknowledging or handling this failure\",\n \"suggestion\": \"Add explicit error handling for failed tool calls - acknowledge failures, try alternative URLs, or note the gap in research\",\n \"iteration\": 9\n },\n {\n \"type\": \"incomplete_reasoning\",\n \"severity\": \"medium\",\n \"description\": \"The agent reaches conclusions and writes comprehensive reports without explicitly validating key details in the thinking trace. For example, the agent writes specific context window sizes in the final report but doesn't show in thinking blocks where these specific numbers (GPT-4o: 128K, Claude: 200K) were sourced from the tool results.\",\n \"suggestion\": \"Add explicit source tracking in thinking blocks - when gathering specific facts like model specifications, explicitly note 'I found X from source Y' to ensure traceability and validation.\",\n \"iteration\": 10\n },\n {\n \"type\": \"missing_validation\",\n \"severity\": \"medium\",\n \"description\": \"When a tool call fails (context-windows URL returns error), the agent doesn't attempt recovery or note this as an information gap. Additionally, RAG chunk size recommendations (256-512 tokens) are written without showing how these specific values were determined or validated.\",\n \"suggestion\": \"Implement explicit error recovery: when a tool fails, note what information is missing and either try alternative sources or flag for follow-up. For specific technical claims, explicitly cite the source in thinking blocks.\",\n \"iteration\": 10\n },\n {\n \"type\": \"tool_misuse\",\n \"severity\": \"low\",\n \"description\": \"The agent makes several overlapping web searches that could have been more efficient. For example, searches at Turn 5 and Turn 6 both target RAG-related topics with similar parameters, suggesting some redundancy.\",\n \"suggestion\": \"Before starting new searches, review what information has already been gathered and explicitly note gaps. Use more specific queries rather than broad overlapping ones.\",\n \"iteration\": 10\n }\n]\n\n\n---\nname: comprehensive-research-agent\ndescription: \"Ensure thorough validation, error recovery, and transparent reasoning in research tasks with multiple tool calls\"\n---\n\n# Comprehensive Research Agent Best Practices\n\nThis skill addresses common failures in multi-step research tasks: unhandled tool errors, missing validation, opaque reasoning, and premature conclusions. It provides structured protocols for source validation, error recovery, and thinking transparency that significantly improves research quality and reliability.\n\n## When to Activate\n\n- Task involves web research with search, read_url, or fetch operations\n- Task requires gathering information from multiple sources\n- Task has explicit requirements for completeness or verification\n- Task includes file operations that need validation (save, write, read)\n- Any research or information-gathering workflow with 3+ tool interactions\n\n## Core Concepts\n\n- Validation Checkpoints: Explicit verification steps at phase transitions to confirm tool outputs, source relevance, and information completeness before proceeding\n- Error Recovery Protocols: Mandatory acknowledgment and handling of tool failures with fallback strategies rather than silent continuation\n- Source Traceability: Maintaining clear tracking of which sources were actually retrieved vs. referenced from prior knowledge to prevent hallucination\n- Substantive Thinking Blocks: Detailed reasoning traces that document insights, connections, gaps, and decision rationale at each step\n- Cross-Source Validation: Verifying key claims against multiple sources and explicitly noting consensus, contradictions, and information gaps\n\n## Patterns to Avoid\n\n- **Silent Tool Failure**: A tool call returns an error (404, timeout, invalid URL) but the agent proceeds without acknowledging it, potentially missing critical information. Always log failures and attempt recovery or document the gap.\n- **Vague Completion Claims**: Agent declares 'I have enough information' or 'research is comprehensive' without specifying what was learned, what sources support claims, or what gaps remain. Replace with specific summaries of coverage.\n- **Unvalidated Source Selection**: Agent reads URLs from search results without evaluating relevance, credibility, or recency first. This wastes tool calls on low-quality sources. Always rank and prioritize sources before deep reading.\n- **Generic Thinking Blocks**: Thinking contains only next-action descriptions ('Now I will search for X') without analysis of what was learned, how it connects to the goal, or what questions remain. Thinking should be substantive and reflective.\n- **Verification Method Error**: Using list_directory to verify file creation can produce false negatives due to caching. Always use read_file for actual content verification.\n- **Citation Without Retrieval**: Citing sources (URLs, paper titles) in the final report that were never successfully fetched or read. Track sources explicitly and prohibit citing unretrieved content.\n- **Redundant Tool Calls**: Making overlapping searches or reading sources without tracking what has already been obtained. Maintain a 'found resources' tracker to avoid duplication.\n\n## Recommended Practices\n\n- **Implement Pre-Reading Source Evaluation**: Before reading URLs, rank search results by relevance, credibility, recency, and authority. Document selection rationale in thinking blocks.\n- **Use Structured Thinking Blocks**: Each thinking block must include: (a) what was learned from the source/action, (b) how it connects to the research goal, (c) any contradictions/gaps identified, (d) strategic decisions made. Avoid generic next-action statements.\n- **Add Mandatory Error Acknowledgment**: When any tool fails, the next thinking block must explicitly address it: note the failure type, propose a recovery strategy (retry, alternative source, or documented gap), and explain the chosen approach.\n- **Create Pre-Completion Validation Checklist**: Before declaring research complete, verify: all required sections have specific evidence, all sources were successfully retrieved, key claims are cross-validated, and gaps are documented.\n- **Implement Cross-Source Validation**: After gathering information from multiple sources, explicitly compare findings. Note where sources agree, where they contradict, and what remains unverified. Use this to assess overall confidence.\n- **Maintain Source Tracking Table**: Create a simple table in thinking showing which URLs were fetched, which failed, and which were used for specific claims. Never cite unretrieved sources.\n- **Use Read_File for Verification**: When confirming file writes, use read_file to verify actual content rather than list_directory, which can have caching issues causing false negatives.\n- **Add Explicit Validation Phase**: After reading sources, write a brief synthesis that confirms usefulness, notes relevance to research goals, and identifies remaining gaps before proceeding to the next phase.\n\n## Guidelines\n\n1. After each tool call, explicitly check for errors in the response and acknowledge failures in the next thinking block with recovery strategy\n2. Before reading URLs, rank sources by relevance/credibility and document selection rationale - never read results without evaluation\n3. Thinking blocks must be 3-5+ sentences minimum and include: what was learned, connections to goal, gaps/contradictions, and next steps\n4. Create a pre-completion checklist verifying: all requirements covered, sources retrieved, claims validated, gaps documented\n5. Maintain source tracking - only cite URLs that were successfully fetched; prohibit citing unretrieved sources\n6. When writing final reports, include 'Limitations & Gaps' section documenting what was attempted but failed or what remains unverified\n7. Use read_file (not list_directory) to verify file content after save operations\n8. Cross-validate key claims across at least 2 sources when possible; explicitly note consensus or contradictions\n9. Track gathered information to avoid redundant searches - implement 'found resources' tracker for multi-phase research\n10. Replace vague 'comprehensive' statements with specific summaries: 'Covered X sources on Y topic; missing Z aspects'\n\n## Examples\n\n- ***Before (Anti-Pattern)**: 'I searched for context engineering and found several results. Now I'll read some URLs and then write the report. I have enough information to proceed.'\n\n**After (Pattern)**: 'Search returned 15 results on context engineering. Evaluating relevance: Liu et al. (2024) appears most authoritative on 'lost in the middle' phenomenon; Anthropic documentation likely has current context window specs; Patel (2023) covers RAG best practices. Ranking these as top 3 priorities. Reading top result first. If the primary source fails (URL error), will try backup search for correct documentation URL and note the gap in final report.'\n- ***Before (Anti-Pattern)**: Tool returns 404 error for Anthropic context windows URL. Agent continues without acknowledgment. Later cites 'Claude has 200K context window' without showing source. Final report cites Google Research paper that was never fetched.\n\n**After (Pattern)**: Tool returned 404 for Anthropic URL. Thinking: 'Primary source failed. Fallback: search for alternative Anthropic documentation URL or find archived version. If unavailable, note context window data from secondary sources only and add disclaimer about verification status.' Then: 'Cross-validated Claude context window: Anthropic blog (successfully read) and two developer documentation sources agree on 200K. Confident in this claim.' Source tracking table shows: Anthropic URL (failed, backup used), Blog (success), Dev docs (success).\n\n---\n\n## Score Expectations\n\nComplex research tasks with multiple tools (6+) and multi-step reasoning chains typically achieve scores in the **65-75 range**. This is not a limitation of the prompt but reflects:\n\n- Inherent variability in tool outputs affecting reasoning paths\n- Multiple valid approaches leading to different intermediate scores\n- Stochastic nature of long-horizon agent execution\n\n**Focus on relative improvement and pattern elimination** rather than absolute scores. A 5-10% improvement from optimization is significant for complex tasks.\n\n---\n\n## Skill Metadata\n\n**Generated**: 2026-01-11\n**Source**: Reasoning Trace Optimizer\n**Optimization Iterations**: 10\n**Best Score Achieved**: 72/100 (iteration 4)\n**Final Score**: 70.0/100\n**Score Improvement**: 67.6 → 70.0 (+3.6%)\n\n\n---\nname: reasoning-trace-optimizer\ndescription: \"Debug and optimize AI agents by analyzing reasoning traces, context degradation, tool confusion, instruction drift, repeated task failures, and performance regressions.\"\n---\n\n# Reasoning Trace Optimizer\n\nDebug and optimize AI agents by analyzing their reasoning traces. This skill uses MiniMax M2.1's interleaved thinking to provide deep insight into agent decision-making and generate concrete improvements.\n\n## When to Activate\n\n- Agent reasoning traces need debugging, analysis, or prompt optimization\n- Agent task fails and user wants to understand why\n- User mentions \"context degradation\", \"tool confusion\", or \"instruction drift\"\n- Request to improve agent performance or reduce errors\n- User wants to generate shareable learnings from debugging sessions\n- After repeated failures on similar tasks\n\n## Core Concepts\n\n### Interleaved Thinking\n\nUnlike standard reasoning models that think once at the start, interleaved thinking allows reasoning BETWEEN each tool interaction. This is critical because:\n\n1. **Long-horizon tasks** require maintaining focus across many turns\n2. **External perturbations** (tool outputs, environment changes) need real-time adaptation\n3. **Debugging** requires seeing HOW decisions were made, not just WHAT was output\n\n### The Optimization Loop\n\n```\nExecute Agent → Capture Traces → Analyze Patterns → Optimize Prompt → Re-run\n ↑____________|\n```\n\nEach iteration improves the prompt based on detected patterns until convergence.\n\n### Pattern Detection\n\nCommon failure patterns the analyzer detects:\n\n| Pattern | Description |\n|---------|-------------|\n| `context_degradation` | Model loses track of information over long contexts |\n| `tool_confusion` | Model misunderstands tool capabilities or outputs |\n| `instruction_drift` | Model gradually deviates from original instructions |\n| `goal_abandonment` | Model stops pursuing the original goal |\n| `circular_reasoning` | Model repeats similar actions without progress |\n| `premature_conclusion` | Model concludes before completing the task |\n\n## Usage Modes\n\n### Mode 1: M2.1 Agent Debugging\n\nRun a task through M2.1 and analyze its reasoning:\n\n```python\nfrom reasoning_trace_optimizer import TraceCapture, TraceAnalyzer\n\ncapture = TraceCapture()\ntrace = capture.run(\n task=\"Search for Python tutorials and summarize them\",\n system_prompt=\"You are a research assistant.\",\n tools=[search_tool],\n tool_executor=execute_search\n)\n\nanalyzer = TraceAnalyzer()\nanalysis = analyzer.analyze(trace)\n\nprint(f\"Score: {analysis.overall_score}/100\")\nfor pattern in analysis.patterns:\n print(f\"Found: {pattern.type.value} - {pattern.suggestion}\")\n```\n\n### Mode 2: Full Optimization Loop\n\nAutomatically iterate until the prompt is optimized:\n\n```python\nfrom reasoning_trace_optimizer import OptimizationLoop, LoopConfig\n\nconfig = LoopConfig(\n max_iterations=5,\n min_score_threshold=80.0,\n)\n\nloop = OptimizationLoop(config=config)\nresult = loop.run(\n task=\"Analyze this codebase and suggest improvements\",\n initial_prompt=\"You are a code reviewer.\",\n tools=[read_file_tool, search_tool],\n tool_executor=execute_tool\n)\n\nprint(f\"Improved: {result.initial_score} → {result.final_score}\")\nprint(f\"Final prompt:\\n{result.final_prompt}\")\n```\n\n### Mode 3: Universal Session Analysis\n\nAnalyze any agent's previous thinking (works with Claude, GPT, etc.):\n\nWhen this skill is activated in Claude Code, it can analyze the current session's thinking blocks to identify issues and suggest improvements.\n\n```\n/reasoning-trace-optimizer analyze-session\n```\n\n### Mode 4: Generate Shareable Skills\n\nConvert optimization learnings into reusable Agent Skills:\n\n```python\nfrom reasoning_trace_optimizer import SkillGenerator\n\ngenerator = SkillGenerator()\nskill_path = generator.generate(\n result=loop_result,\n skill_name=\"web-search-best-practices\",\n output_dir=\"./skills\"\n)\n```\n\n## CLI Commands\n\n```bash\n# Capture reasoning trace\nrto capture \"Search for Python tutorials\" -s \"You are a helpful assistant.\"\n\n# Analyze a task\nrto analyze \"Debug this code\" -o analysis.txt\n\n# Run optimization loop\nrto optimize \"Research AI papers\" --max-iterations 5 --generate-skill\n\n# Generate skill from artifacts\nrto generate-skill my-skill-name --artifacts-dir ./optimization_artifacts\n```\n\n## Integration with Claude Code\n\n### Auto-trigger on Failure\n\nAdd to your hooks to automatically analyze failures:\n\n```json\n{\n \"hooks\": {\n \"post_tool_error\": {\n \"command\": \"rto analyze-session --last-error\"\n }\n }\n}\n```\n\n### On-demand Analysis\n\nUse the slash command to analyze current session:\n\n```\n/reasoning-trace-optimizer\n```\n\nThis will:\n1. Extract thinking blocks from the current session\n2. Identify patterns and issues\n3. Suggest prompt improvements\n4. Optionally update the system prompt\n\n## Guidelines\n\n1. **Preserve full context**: M2.1 requires full response history including thinking blocks for optimal performance\n2. **Use appropriate tools**: Define tools clearly with unambiguous descriptions\n3. **Set realistic convergence thresholds**: 5-10% improvement per iteration is typical\n4. **Review generated skills**: Auto-generated skills should be reviewed before sharing\n5. **Monitor token usage**: Each optimization iteration uses significant tokens\n\n## Examples\n\n### Before Optimization\n\n```\nSystem: You are a helpful assistant.\n\nIssue: Agent called wrong tools, lost track of goal after 3 turns\nScore: 45/100\nPatterns: tool_confusion, goal_abandonment\n```\n\n### After Optimization\n\n```\nSystem: You are a research assistant focused on finding accurate information.\n\nIMPORTANT GUIDELINES:\n- Always verify search results before summarizing\n- If a tool returns an error, try an alternative approach\n- Keep track of your original goal throughout the task\n- Validate findings against multiple sources when possible\n\nIssue: None\nScore: 85/100\nPatterns: None detected\n```\n\n## References\n\n- MiniMax M2.1 Documentation: https://platform.minimax.io/docs\n- Interleaved Thinking Guide: See `docs/interleavedthinking.md`\n- Agent Generalization: See `docs/agentthinking.md`\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-01-11\n**Author**: Muratcan Koylan\n**Version**: 0.1.0\n**Powered by**: MiniMax M2.1\n**Partnership**: Built in collaboration with MiniMax AI\n\n\n# Context Fundamentals Skill\n\n## Overview\n\nContext engineering is the systematic approach to managing what information an LLM receives and how it processes that information. Effective context management directly impacts output quality, consistency, and task success rates.\n\n## Core Principles\n\n### 1. Context Window Management\n\nThe context window is finite. Every token counts. Prioritize information by relevance and recency.\n\n**Strategies:**\n- Summarize historical conversation turns\n- Use retrieval to inject only relevant context\n- Implement context compression for long documents\n\n### 2. Information Hierarchy\n\nStructure context to guide model attention:\n\n```\n1. System Instructions (highest priority)\n └── Role definition\n └── Task constraints\n └── Output format requirements\n\n2. Relevant Context (dynamic)\n └── Retrieved documents\n └── User-specific data\n └── Recent conversation history\n\n3. User Input (current request)\n └── Query or instruction\n └── Any inline context\n```\n\n### 3. Context Relevance\n\nNot all context is equally useful. Apply relevance filtering:\n\n- **Temporal Relevance**: Recent information often outweighs older data\n- **Semantic Relevance**: Use embeddings to surface related content\n- **Task Relevance**: Only include information needed for current task\n\n## Context Types\n\n### Static Context\n- System prompts\n- Role definitions\n- Tool descriptions\n- Format specifications\n\n### Dynamic Context\n- Retrieved documents (RAG)\n- Conversation history\n- User preferences\n- Session state\n\n### Ephemeral Context\n- Current tool outputs\n- Intermediate reasoning steps\n- Scratchpad content\n\n## Best Practices\n\n1. **Explicit Over Implicit**: State requirements clearly rather than relying on inference\n2. **Structured Formatting**: Use consistent delimiters and sections\n3. **Redundancy Removal**: Avoid repeating information across context sections\n4. **Source Attribution**: Mark where context comes from for traceability\n5. **Freshness Signals**: Indicate when information was last updated\n\n## Context Patterns\n\n### RAG Integration Pattern\n```\n[System Instructions]\nYou are a helpful assistant. Use the provided context to answer questions.\nOnly use information from the context. If unsure, say so.\n\n[Retrieved Context]\n\n Content here...\n\n\n[User Query]\n{user_input}\n```\n\n### Multi-Turn Context Pattern\n```\n[System Instructions]\n...\n\n[Conversation History]\nSummary of earlier turns: {summary}\n\nRecent exchanges:\nUser: {recent_user_1}\nAssistant: {recent_assistant_1}\nUser: {recent_user_2}\nAssistant: {recent_assistant_2}\n\n[Current Turn]\nUser: {current_input}\n```\n\n## Metrics\n\n- **Context Utilization Rate**: How much of provided context is used in response\n- **Context Relevance Score**: Semantic similarity between context and response\n- **Context Compression Ratio**: Original size vs. compressed size\n- **Information Retention**: Key facts preserved after summarization\n\n\n# Skills Index\n\nSkills are foundational knowledge modules that inform the design and implementation of agents, tools, and prompts.\n\n## Available Skills\n\n### LLM Evaluator\n**Path**: `skills/llm-evaluator/llm-evaluator.md`\n\nCovers LLM-as-a-Judge evaluation methodology including:\n- Scoring approaches (direct, pairwise, reference-based)\n- Evaluation metrics (classification, correlation)\n- Known biases and mitigation strategies\n- Implementation patterns\n\n**Key Takeaways**:\n- Use direct scoring for objective evaluations\n- Use pairwise comparison for subjective preferences\n- Always mitigate position bias\n- Prefer classification metrics for interpretability\n\n### Context Fundamentals\n**Path**: `skills/context-fundamentals/context-fundamentals.md`\n\nCovers context engineering principles including:\n- Context window management\n- Information hierarchy\n- Context types (static, dynamic, ephemeral)\n- Relevance filtering\n\n**Key Takeaways**:\n- Structure context by priority\n- Be explicit over implicit\n- Remove redundancy\n- Signal freshness of information\n\n### Tool Design\n**Path**: `skills/tool-design/tool-design.md`\n\nCovers agent tool design best practices including:\n- Single responsibility principle\n- Input/output schemas\n- Error handling patterns\n- AI SDK 6 features (approval, strict mode, examples)\n\n**Key Takeaways**:\n- Clear, validated schemas\n- Predictable output structure\n- Graceful error handling\n- Consider approval for dangerous tools\n\n## Skill Application Matrix\n\n| Skill | Agents | Tools | Prompts |\n|-------|--------|-------|---------|\n| LLM Evaluator | Evaluator | directScore, pairwiseCompare | evaluation/* |\n| Context Fundamentals | All | All (context params) | All (context handling) |\n| Tool Design | All (tool selection) | All | orchestrator-prompt |\n\n## Adding New Skills\n\n1. Create skill directory: `skills//`\n2. Create main file: `skills//.md`\n3. Include:\n - Overview and purpose\n - Core principles\n - Practical patterns\n - Implementation examples\n - References\n4. Update this index\n\n## Skill Development Guidelines\n\n- Focus on principles that transfer across implementations\n- Include concrete examples and patterns\n- Reference authoritative sources\n- Keep content actionable, not just theoretical\n- Update as understanding evolves\n\n\n# LLM-Evaluator Skill\n\n## Overview\n\nLLM-Evaluators (LLM-as-a-Judge) are large language models designed to evaluate the quality of another LLM's response to an instruction or query. This skill provides the foundational knowledge for building evaluation systems.\n\n## Key Considerations\n\n### Baseline Selection\n- **Human Annotators**: Aim for LLM-human correlation to match human-human correlation. LLM-evaluators are orders of magnitude faster and cheaper than human annotation.\n- **Finetuned Classifiers**: Goal is to achieve similar recall and precision as a finetuned classifier. More challenging baseline as these are optimized for specific tasks.\n\n### Scoring Approaches\n\n| Approach | Use Case | Reliability |\n|----------|----------|-------------|\n| **Direct Scoring** | Objective tasks (factuality, toxicity, instruction-following) | More suitable for binary classification |\n| **Pairwise Comparison** | Subjective evaluations (tone, persuasiveness, coherence) | More reliable for preference tasks |\n| **Reference-Based** | Comparing against gold standard | Requires ground truth reference |\n\n### Evaluation Metrics\n\n**Classification Metrics** (Preferred for binary tasks):\n- Recall and Precision\n- F1 Score\n- Cohen's κ (Kappa)\n\n**Correlation Metrics** (For Likert scale tasks):\n- Spearman's ρ (rho)\n- Kendall's τ (tau)\n\n## Known Biases\n\n1. **Position Bias**: LLM-evaluators tend to prefer responses in certain positions during pairwise comparison (usually first position)\n2. **Verbosity Bias**: Favor longer, more verbose responses even if not higher quality\n3. **Self-Enhancement Bias**: LLM-evaluators prefer answers generated by themselves\n\n## Mitigation Strategies\n\n- Swap response positions and average results\n- Normalize for length when evaluating\n- Use a Panel of LLMs (PoLL) instead of single judge\n- Include \"don't overthink\" instructions\n- Use CoT + n-shot prompts for reliability\n\n## Implementation Pattern\n\n```typescript\ninterface EvaluatorConfig {\n scoringApproach: 'direct' | 'pairwise' | 'reference-based';\n criteria: EvaluationCriteria[];\n metrics: MetricType[];\n useCoT: boolean;\n nShot: number;\n}\n\ninterface EvaluationCriteria {\n name: string;\n description: string;\n rubric: RubricLevel[];\n}\n\ninterface RubricLevel {\n score: number;\n description: string;\n}\n```\n\n## References\n\nKey papers reviewed:\n- Constitutional AI (Anthropic)\n- G-Eval: NLG Evaluation using GPT-4\n- SelfCheckGPT: Zero-Resource Hallucination Detection\n- Prometheus: Fine-grained Evaluation Capability\n- MT-Bench and Chatbot Arena\n\n\n# Agent Tool Design Skill\n\n## Overview\n\nTools are the foundation of an agent's capabilities. An agent's ability to take meaningful actions depends entirely on how reliably it can generate valid tool inputs, how well those inputs align with user intent, and how effectively tool outputs inform next steps.\n\n## Design Principles\n\n### 1. Single Responsibility\n\nEach tool should do one thing well. Complex operations should be composed from multiple tools.\n\n```typescript\n// Bad: Tool does too much\nconst analyzeAndSummarizeAndSend = { ... }\n\n// Good: Separate concerns\nconst analyzeDocument = { ... }\nconst summarizeContent = { ... }\nconst sendEmail = { ... }\n```\n\n### 2. Clear Input Schemas\n\nUse explicit, validated schemas with descriptive field names and constraints.\n\n```typescript\nconst searchTool = tool({\n description: \"Search for documents by semantic similarity\",\n parameters: z.object({\n query: z.string().describe(\"Natural language search query\"),\n limit: z.number().min(1).max(100).default(10)\n .describe(\"Maximum number of results to return\"),\n filters: z.object({\n dateAfter: z.string().optional()\n .describe(\"ISO date string, only return docs after this date\"),\n source: z.enum([\"internal\", \"external\", \"all\"]).default(\"all\")\n }).optional()\n }),\n execute: async (input) => { ... }\n});\n```\n\n### 3. Predictable Output Structure\n\nReturn consistent, typed output that the model can reliably parse.\n\n```typescript\ninterface ToolResult {\n success: boolean;\n data?: T;\n error?: {\n code: string;\n message: string;\n retryable: boolean;\n };\n metadata: {\n executionTimeMs: number;\n source?: string;\n };\n}\n```\n\n### 4. Graceful Error Handling\n\nTools should never throw unhandled exceptions. Always return structured errors.\n\n```typescript\nexecute: async (input) => {\n try {\n const result = await performAction(input);\n return { success: true, data: result };\n } catch (error) {\n return {\n success: false,\n error: {\n code: error.code ?? \"UNKNOWN_ERROR\",\n message: error.message,\n retryable: isRetryable(error)\n }\n };\n }\n}\n```\n\n## Tool Categories\n\n### Read-Only Tools\n- Database queries\n- API fetches\n- File reads\n- Search operations\n\nSafe to execute without approval. Return data but don't mutate state.\n\n### State-Modifying Tools\n- Database writes\n- File modifications\n- API POST/PUT/DELETE\n- System configuration changes\n\nMay require human approval. Consider `needsApproval` flag.\n\n### Dangerous Tools\n- File deletion\n- Payment processing\n- Production deployments\n- Sending external communications\n\nShould always require approval and audit logging.\n\n## AI SDK 6 Tool Features\n\n### Tool Execution Approval\n```typescript\nconst deleteTool = tool({\n description: \"Delete a file from the system\",\n parameters: z.object({\n path: z.string()\n }),\n needsApproval: true, // Requires human approval\n execute: async ({ path }) => { ... }\n});\n\n// Dynamic approval based on input\nconst commandTool = tool({\n description: \"Execute a shell command\",\n parameters: z.object({\n command: z.string()\n }),\n needsApproval: ({ command }) => {\n return command.includes(\"rm\") || command.includes(\"delete\");\n },\n execute: async ({ command }) => { ... }\n});\n```\n\n### Strict Mode\nEnable native strict mode for guaranteed schema compliance:\n```typescript\nconst strictTool = tool({\n description: \"...\",\n parameters: schema,\n strict: true, // Enable strict mode\n execute: async (input) => { ... }\n});\n```\n\n### Input Examples\nHelp the model understand expected input format:\n```typescript\nconst complexTool = tool({\n description: \"Create a calendar event\",\n parameters: eventSchema,\n inputExamples: [\n {\n title: \"Team Standup\",\n date: \"2024-01-15\",\n time: \"09:00\",\n duration: 30,\n attendees: [\"alice@example.com\", \"bob@example.com\"]\n }\n ],\n execute: async (input) => { ... }\n});\n```\n\n### toModelOutput\nControl what gets sent back to the model:\n```typescript\nconst readFileTool = tool({\n description: \"Read file contents\",\n parameters: z.object({ path: z.string() }),\n execute: async ({ path }) => {\n const content = await fs.readFile(path, 'utf-8');\n return { path, content, size: content.length };\n },\n toModelOutput: (result) => {\n // Only send truncated content to model\n return {\n path: result.path,\n content: result.content.slice(0, 5000),\n truncated: result.content.length > 5000\n };\n }\n});\n```\n\n## Best Practices\n\n1. **Descriptive Names**: Tool names should clearly indicate their function\n2. **Comprehensive Descriptions**: Include usage examples in tool descriptions\n3. **Reasonable Defaults**: Provide sensible defaults for optional parameters\n4. **Idempotency**: Design tools to be safely re-executable when possible\n5. **Timeout Handling**: Implement timeouts for external operations\n6. **Rate Limiting**: Protect against runaway tool execution\n7. **Logging**: Log all tool invocations for debugging and audit\n\n\n# Direct Score Tool\n\n## Purpose\n\nEvaluate a single LLM response against defined criteria using a scoring rubric.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const directScore = tool({\n description: `Evaluate a response by scoring it against specific criteria.\nUse this for objective evaluations where you need to assess quality \ndimensions like accuracy, completeness, clarity, or task adherence.\nReturns structured scores with justifications.`,\n\n parameters: z.object({\n response: z.string()\n .describe(\"The LLM response to evaluate\"),\n \n prompt: z.string()\n .describe(\"The original prompt/instruction that generated the response\"),\n \n context: z.string().optional()\n .describe(\"Additional context like retrieved documents or conversation history\"),\n \n criteria: z.array(z.object({\n name: z.string().describe(\"Name of the criterion (e.g., 'Accuracy')\"),\n description: z.string().describe(\"What this criterion measures\"),\n weight: z.number().min(0).max(1).default(1)\n .describe(\"Relative importance, weights should sum to 1\")\n })).min(1).describe(\"Evaluation criteria to score against\"),\n \n rubric: z.object({\n scale: z.enum([\"1-3\", \"1-5\", \"1-10\"]).default(\"1-5\"),\n levelDescriptions: z.record(z.string(), z.string()).optional()\n .describe(\"Optional descriptions for each score level\")\n }).optional().describe(\"Scoring rubric configuration\")\n }),\n\n execute: async (input) => {\n // Implementation delegates to evaluator LLM\n return evaluateWithLLM(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| response | string | Yes | The response being evaluated |\n| prompt | string | Yes | Original prompt that generated the response |\n| context | string | No | Additional context (RAG docs, history) |\n| criteria | Criterion[] | Yes | List of evaluation criteria |\n| rubric | Rubric | No | Scoring scale and level descriptions |\n\n### Criterion Object\n```typescript\n{\n name: string; // e.g., \"Factual Accuracy\"\n description: string; // e.g., \"Response contains no factual errors\"\n weight: number; // 0-1, relative importance\n}\n```\n\n### Rubric Object\n```typescript\n{\n scale: \"1-3\" | \"1-5\" | \"1-10\";\n levelDescriptions?: {\n \"1\": \"Poor - Major issues\",\n \"2\": \"Below Average - Several issues\",\n \"3\": \"Average - Some issues\",\n \"4\": \"Good - Minor issues\",\n \"5\": \"Excellent - No issues\"\n }\n}\n```\n\n## Output Schema\n\n```typescript\ninterface DirectScoreResult {\n success: boolean;\n \n scores: {\n criterion: string;\n score: number;\n maxScore: number;\n justification: string;\n examples: string[]; // Specific examples from response\n }[];\n \n overallScore: number;\n weightedScore: number;\n \n summary: {\n strengths: string[];\n weaknesses: string[];\n suggestions: string[];\n };\n \n metadata: {\n evaluationTimeMs: number;\n criteriaCount: number;\n rubricScale: string;\n };\n}\n```\n\n## Usage Example\n\n```typescript\nconst result = await directScore.execute({\n response: \"Machine learning is a subset of AI that enables systems to learn from data...\",\n \n prompt: \"Explain machine learning to a beginner\",\n \n criteria: [\n {\n name: \"Accuracy\",\n description: \"Technical correctness of explanations\",\n weight: 0.4\n },\n {\n name: \"Clarity\",\n description: \"Understandable for a beginner\",\n weight: 0.3\n },\n {\n name: \"Completeness\",\n description: \"Covers key concepts adequately\",\n weight: 0.3\n }\n ],\n \n rubric: {\n scale: \"1-5\",\n levelDescriptions: {\n \"1\": \"Completely fails criterion\",\n \"2\": \"Major deficiencies\",\n \"3\": \"Acceptable but improvable\",\n \"4\": \"Good with minor issues\",\n \"5\": \"Excellent, no issues\"\n }\n }\n});\n```\n\n## Implementation Notes\n\n1. **Chain-of-Thought**: Implementation should use CoT prompting for more reliable scoring\n2. **Calibration**: Include few-shot examples of scores at each level\n3. **Justification First**: Ask for justification before score to reduce bias\n4. **Length Normalization**: Consider response length when appropriate\n\n\n# Generate Rubric Tool\n\n## Purpose\n\nAutomatically generate a scoring rubric for a given evaluation criterion. Creates detailed descriptions for each score level to ensure consistent evaluation.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const generateRubric = tool({\n description: `Generate a scoring rubric for an evaluation criterion.\nCreates detailed descriptions for each score level.\nUse when you need to establish consistent evaluation standards.`,\n\n parameters: z.object({\n criterionName: z.string()\n .describe(\"Name of the criterion (e.g., 'Factual Accuracy')\"),\n \n criterionDescription: z.string()\n .describe(\"What this criterion measures\"),\n \n scale: z.enum([\"1-3\", \"1-5\", \"1-10\"]).default(\"1-5\")\n .describe(\"Scoring scale to use\"),\n \n domain: z.string().optional()\n .describe(\"Domain context (e.g., 'medical writing', 'code review')\"),\n \n includeExamples: z.boolean().default(true)\n .describe(\"Include example text for each score level\"),\n \n strictness: z.enum([\"lenient\", \"balanced\", \"strict\"]).default(\"balanced\")\n .describe(\"How strictly to define score boundaries\")\n }),\n\n execute: async (input) => {\n return generateRubricWithLLM(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| criterionName | string | Yes | Name of criterion |\n| criterionDescription | string | Yes | What criterion measures |\n| scale | enum | No | Scoring scale (default: 1-5) |\n| domain | string | No | Domain for context |\n| includeExamples | boolean | No | Include examples (default: true) |\n| strictness | enum | No | Score boundary strictness |\n\n## Output Schema\n\n```typescript\ninterface GeneratedRubric {\n success: boolean;\n \n criterion: {\n name: string;\n description: string;\n };\n \n scale: {\n min: number;\n max: number;\n type: string;\n };\n \n levels: {\n score: number;\n label: string; // e.g., \"Excellent\", \"Poor\"\n description: string; // Detailed description\n characteristics: string[]; // Key characteristics\n example?: string; // Example text at this level\n }[];\n \n scoringGuidelines: string[];\n \n edgeCases: {\n situation: string;\n guidance: string;\n }[];\n \n metadata: {\n domain: string | null;\n strictness: string;\n generationTimeMs: number;\n };\n}\n```\n\n## Usage Example\n\n```typescript\nconst rubric = await generateRubric.execute({\n criterionName: \"Code Readability\",\n criterionDescription: \"How easy the code is to read and understand\",\n scale: \"1-5\",\n domain: \"code review\",\n includeExamples: true,\n strictness: \"balanced\"\n});\n\n// Result:\n// {\n// criterion: {\n// name: \"Code Readability\",\n// description: \"How easy the code is to read and understand\"\n// },\n// scale: { min: 1, max: 5, type: \"1-5\" },\n// levels: [\n// {\n// score: 1,\n// label: \"Poor\",\n// description: \"Code is extremely difficult to understand...\",\n// characteristics: [\n// \"No meaningful variable names\",\n// \"Deeply nested logic without explanation\",\n// \"No comments on complex sections\"\n// ],\n// example: \"function x(a,b,c){return a?b+c:c-b;}\"\n// },\n// {\n// score: 5,\n// label: \"Excellent\",\n// description: \"Code is immediately understandable...\",\n// characteristics: [\n// \"Self-documenting variable and function names\",\n// \"Appropriate comments explaining 'why'\",\n// \"Clear logical structure\"\n// ],\n// example: \"function calculateShippingCost(weight, distance, expedited) {\\n // Base rate plus per-mile charge\\n const baseCost = weight * BASE_RATE_PER_KG;\\n ...\"\n// },\n// ...\n// ],\n// scoringGuidelines: [\n// \"Focus on clarity for someone unfamiliar with the codebase\",\n// \"Consider both naming and structure\",\n// \"Comments should explain 'why', not 'what'\"\n// ],\n// edgeCases: [\n// {\n// situation: \"Code uses domain-specific abbreviations\",\n// guidance: \"Accept if abbreviations are standard in the domain\"\n// }\n// ]\n// }\n```\n\n## Rubric Templates\n\n### Factual Accuracy (1-5)\n```\n5: All claims factually correct, properly sourced\n4: Minor factual issues, non-critical\n3: Some factual errors, main points correct\n2: Multiple factual errors affecting reliability\n1: Fundamentally incorrect or misleading\n```\n\n### Clarity (1-5)\n```\n5: Immediately understandable, well-structured\n4: Clear with minor ambiguities\n3: Generally clear, some confusing sections\n2: Difficult to follow, unclear organization\n1: Incomprehensible or incoherent\n```\n\n### Completeness (1-5)\n```\n5: Addresses all aspects comprehensively\n4: Covers main points, minor gaps\n3: Addresses core requirements, notable gaps\n2: Missing significant required elements\n1: Fails to address the question\n```\n\n## Implementation Notes\n\n1. **Domain Adaptation**: Rubrics should reflect domain-specific expectations\n2. **Boundary Clarity**: Clear distinctions between adjacent scores\n3. **Example Quality**: Examples should be realistic, not strawmen\n4. **Edge Case Coverage**: Anticipate common ambiguous situations\n5. **Calibration**: Test rubric against known samples before use\n\n\n# Pairwise Compare Tool\n\n## Purpose\n\nCompare two LLM responses and determine which one better satisfies the given criteria. More reliable for subjective evaluations than direct scoring.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const pairwiseCompare = tool({\n description: `Compare two responses and select the better one.\nUse for subjective evaluations like tone, persuasiveness, or writing style.\nMore reliable than direct scoring for preferences.\nReturns winner selection with detailed comparison.`,\n\n parameters: z.object({\n responseA: z.string()\n .describe(\"First response to compare\"),\n \n responseB: z.string()\n .describe(\"Second response to compare\"),\n \n prompt: z.string()\n .describe(\"The original prompt both responses address\"),\n \n context: z.string().optional()\n .describe(\"Additional context if relevant\"),\n \n criteria: z.array(z.string())\n .describe(\"Aspects to compare on, e.g., ['clarity', 'engagement', 'accuracy']\"),\n \n allowTie: z.boolean().default(true)\n .describe(\"Whether to allow a tie verdict\"),\n \n swapPositions: z.boolean().default(true)\n .describe(\"Evaluate twice with positions swapped to reduce position bias\")\n }),\n\n execute: async (input) => {\n if (input.swapPositions) {\n return evaluateWithPositionSwap(input);\n }\n return evaluatePairwise(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| responseA | string | Yes | First response |\n| responseB | string | Yes | Second response |\n| prompt | string | Yes | Original prompt |\n| context | string | No | Additional context |\n| criteria | string[] | Yes | Comparison dimensions |\n| allowTie | boolean | No | Allow tie verdict (default: true) |\n| swapPositions | boolean | No | Swap positions to reduce bias (default: true) |\n\n## Output Schema\n\n```typescript\ninterface PairwiseCompareResult {\n success: boolean;\n \n winner: \"A\" | \"B\" | \"TIE\";\n confidence: number; // 0-1\n \n comparison: {\n criterion: string;\n winner: \"A\" | \"B\" | \"TIE\";\n reasoning: string;\n aStrength: string;\n bStrength: string;\n }[];\n \n overallReasoning: string;\n \n differentiators: {\n aAdvantages: string[];\n bAdvantages: string[];\n };\n \n // If swapPositions was true\n positionConsistency?: {\n firstPassWinner: \"A\" | \"B\" | \"TIE\";\n secondPassWinner: \"A\" | \"B\" | \"TIE\";\n consistent: boolean;\n };\n \n metadata: {\n evaluationTimeMs: number;\n positionsSwapped: boolean;\n };\n}\n```\n\n## Usage Example\n\n```typescript\nconst result = await pairwiseCompare.execute({\n responseA: \"Exercise improves cardiovascular health, builds muscle, and boosts mental wellbeing...\",\n \n responseB: \"Working out regularly has many benefits. You'll feel better and look better...\",\n \n prompt: \"Explain the benefits of regular exercise\",\n \n criteria: [\"accuracy\", \"specificity\", \"engagement\", \"completeness\"],\n \n allowTie: true,\n swapPositions: true\n});\n\n// Result:\n// {\n// winner: \"A\",\n// confidence: 0.85,\n// comparison: [\n// {\n// criterion: \"accuracy\",\n// winner: \"A\",\n// reasoning: \"Response A uses specific medical terminology...\",\n// aStrength: \"Mentions cardiovascular, muscle, mental health\",\n// bStrength: \"General but not incorrect\"\n// },\n// ...\n// ],\n// ...\n// }\n```\n\n## Position Swap Algorithm\n\nTo mitigate position bias:\n\n```typescript\nasync function evaluateWithPositionSwap(input) {\n // First pass: Original order\n const pass1 = await evaluate({\n first: input.responseA,\n second: input.responseB,\n ...input\n });\n \n // Second pass: Swapped order\n const pass2 = await evaluate({\n first: input.responseB,\n second: input.responseA,\n ...input\n });\n \n // Reconcile results\n const pass2Adjusted = pass2.winner === \"A\" ? \"B\" : pass2.winner === \"B\" ? \"A\" : \"TIE\";\n \n if (pass1.winner === pass2Adjusted) {\n return {\n ...pass1,\n positionConsistency: { consistent: true, ... }\n };\n } else {\n // Inconsistent - return tie or lower confidence\n return {\n winner: \"TIE\",\n confidence: 0.5,\n positionConsistency: { consistent: false, ... },\n ...\n };\n }\n}\n```\n\n## Implementation Notes\n\n1. **Position Bias Mitigation**: Always use `swapPositions: true` for production\n2. **Criteria Order**: Order criteria by importance for better focus\n3. **Tie Handling**: Consider domain - some tasks should rarely tie\n4. **Confidence Calibration**: Lower confidence when evaluations are close\n5. **Length Considerations**: Note if one response is significantly longer\n\n\n# Tools Index\n\nTools provide specific capabilities that agents can use to accomplish tasks.\n\n## Tool Categories\n\n### Evaluation Tools\n**Path**: `tools/evaluation/`\n\nTools for assessing LLM output quality.\n\n| Tool | Purpose | Approval |\n|------|---------|----------|\n| `directScore` | Score response against criteria | No |\n| `pairwiseCompare` | Compare two responses | No |\n| `generateRubric` | Generate scoring rubric | No |\n| `extractCriteria` | Extract criteria from task | No |\n\n---\n\n### Research Tools\n**Path**: `tools/research/`\n\nTools for gathering and processing information.\n\n| Tool | Purpose | Approval |\n|------|---------|----------|\n| `webSearch` | Search the web | No |\n| `readUrl` | Extract content from URL | No |\n| `extractClaims` | Identify claims in text | No |\n| `verifyClaim` | Cross-reference a claim | No |\n| `synthesize` | Combine findings | No |\n\n---\n\n### Orchestration Tools\n**Path**: `tools/orchestration/`\n\nTools for managing multi-agent workflows.\n\n| Tool | Purpose | Approval |\n|------|---------|----------|\n| `delegateToAgent` | Route task to agent | No |\n| `parallelExecution` | Run tasks concurrently | No |\n| `waitForCompletion` | Wait for async tasks | No |\n| `synthesizeResults` | Combine agent outputs | No |\n| `handleError` | Manage failures | No |\n\n## Tool Design Patterns\n\n### Standard Tool Structure\n\n```typescript\nexport const toolName = tool({\n description: \"Clear description of what tool does\",\n \n parameters: z.object({\n // Required parameters first\n requiredParam: z.string().describe(\"What this parameter is for\"),\n \n // Optional parameters with defaults\n optionalParam: z.number().default(10)\n .describe(\"What this parameter controls\")\n }),\n \n // Approval for dangerous operations\n needsApproval: false, // or true, or function\n \n // Strict mode for guaranteed schema compliance\n strict: true,\n \n execute: async (input) => {\n try {\n const result = await performOperation(input);\n return { success: true, data: result };\n } catch (error) {\n return {\n success: false,\n error: {\n code: error.code ?? \"UNKNOWN\",\n message: error.message,\n retryable: isRetryable(error)\n }\n };\n }\n },\n \n // Optional: control what model sees\n toModelOutput: (result) => ({\n summary: result.data.summary,\n truncated: result.data.full.length > 5000\n })\n});\n```\n\n### Error Response Pattern\n\n```typescript\ninterface ToolError {\n code: string; // Machine-readable error code\n message: string; // Human-readable message\n retryable: boolean; // Whether retry might help\n details?: object; // Additional context\n}\n\ninterface ToolResult {\n success: boolean;\n data?: T;\n error?: ToolError;\n metadata: {\n executionTimeMs: number;\n [key: string]: any;\n };\n}\n```\n\n## Adding New Tools\n\n1. Determine category or create new: `tools//`\n2. Create tool file: `tools//.md`\n3. Define:\n - Purpose and description\n - Input parameters with Zod schema\n - Output schema\n - Error codes\n - Usage examples\n4. Update this index\n5. Assign to relevant agents\n\n## Tool Selection Guidelines\n\n### When Agent Needs To...\n\n| Action | Tool Category | Suggested Tools |\n|--------|---------------|-----------------|\n| Assess quality | Evaluation | directScore, pairwiseCompare |\n| Find information | Research | webSearch, readUrl |\n| Verify facts | Research | verifyClaim, extractClaims |\n| Coordinate work | Orchestration | delegateToAgent |\n| Wait for results | Orchestration | waitForCompletion |\n\n\n# Delegate to Agent Tool\n\n## Purpose\n\nRoute a task to a specialized agent for execution. Handles context passing, result collection, and error management.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const delegateToAgent = tool({\n description: `Delegate a task to a specialized agent.\nUse when a subtask requires specific capabilities.\nPass complete context needed for the agent to succeed.`,\n\n parameters: z.object({\n agentName: z.enum([\"evaluator\", \"researcher\", \"writer\", \"analyst\"])\n .describe(\"Name of the agent to delegate to\"),\n \n task: z.string()\n .describe(\"Clear description of what the agent should do\"),\n \n context: z.object({\n previousOutputs: z.array(z.string()).optional()\n .describe(\"Outputs from prior steps this agent needs\"),\n \n documents: z.array(z.string()).optional()\n .describe(\"Relevant documents or data\"),\n \n constraints: z.array(z.string()).optional()\n .describe(\"Requirements or limitations to observe\")\n }).optional(),\n \n expectedOutput: z.object({\n format: z.enum([\"text\", \"json\", \"markdown\", \"structured\"])\n .describe(\"Expected output format\"),\n \n schema: z.string().optional()\n .describe(\"JSON schema if format is structured\"),\n \n maxLength: z.number().optional()\n .describe(\"Maximum length constraint\")\n }).optional(),\n \n timeout: z.number().default(60000)\n .describe(\"Timeout in milliseconds\")\n }),\n\n execute: async (input) => {\n return executeAgentDelegation(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| agentName | enum | Yes | Target agent |\n| task | string | Yes | Task description |\n| context | object | No | Context and dependencies |\n| expectedOutput | object | No | Output requirements |\n| timeout | number | No | Timeout ms (default: 60000) |\n\n## Output Schema\n\n```typescript\ninterface DelegationResult {\n success: boolean;\n \n agentName: string;\n task: string;\n \n output: {\n content: string | object;\n format: string;\n };\n \n execution: {\n startTime: string;\n endTime: string;\n durationMs: number;\n tokenUsage: {\n prompt: number;\n completion: number;\n };\n };\n \n error?: {\n code: string;\n message: string;\n retryable: boolean;\n };\n}\n```\n\n## Available Agents\n\n### Evaluator Agent\n```typescript\nawait delegateToAgent.execute({\n agentName: \"evaluator\",\n task: \"Evaluate the quality of this response against accuracy and clarity criteria\",\n context: {\n documents: [responseToEvaluate],\n constraints: [\"Use 1-5 scale\", \"Include justification\"]\n },\n expectedOutput: { format: \"structured\" }\n});\n```\n\n### Researcher Agent\n```typescript\nawait delegateToAgent.execute({\n agentName: \"researcher\",\n task: \"Research current best practices for LLM evaluation\",\n context: {\n constraints: [\"Focus on 2024 publications\", \"Include citations\"]\n },\n expectedOutput: { format: \"markdown\" }\n});\n```\n\n### Writer Agent\n```typescript\nawait delegateToAgent.execute({\n agentName: \"writer\",\n task: \"Write an executive summary of these research findings\",\n context: {\n previousOutputs: [researchFindings],\n constraints: [\"Maximum 500 words\", \"Non-technical audience\"]\n },\n expectedOutput: { format: \"text\", maxLength: 2500 }\n});\n```\n\n### Analyst Agent\n```typescript\nawait delegateToAgent.execute({\n agentName: \"analyst\",\n task: \"Analyze the trade-offs between direct scoring and pairwise comparison\",\n context: {\n documents: [evaluationData],\n constraints: [\"Quantitative where possible\"]\n },\n expectedOutput: { format: \"structured\" }\n});\n```\n\n## Error Handling\n\n```typescript\nconst errorCodes = {\n \"AGENT_NOT_FOUND\": \"Specified agent does not exist\",\n \"TIMEOUT\": \"Agent did not complete within timeout\",\n \"CONTEXT_TOO_LARGE\": \"Context exceeds agent's capacity\",\n \"INVALID_OUTPUT\": \"Agent output did not match expected format\",\n \"AGENT_ERROR\": \"Agent encountered an error during execution\"\n};\n```\n\n## Implementation Notes\n\n1. **Context Optimization**: Compress context if needed before passing\n2. **Timeout Handling**: Set realistic timeouts per agent type\n3. **Retry Logic**: Implement retries for transient failures\n4. **Audit Trail**: Log all delegations for traceability\n5. **Resource Management**: Track token usage across delegations\n\n\n# Read URL Tool\n\n## Purpose\n\nExtract and parse content from a given URL. Returns structured text content with metadata about the source.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const readUrl = tool({\n description: `Read and extract content from a URL.\nReturns the main text content, stripped of navigation and ads.\nUse after webSearch to get full content from relevant results.`,\n\n parameters: z.object({\n url: z.string().url()\n .describe(\"The URL to read\"),\n \n contentType: z.enum([\"auto\", \"article\", \"documentation\", \"paper\", \"code\"]).default(\"auto\")\n .describe(\"Hint for content type to optimize extraction\"),\n \n maxLength: z.number().min(1000).max(50000).default(10000)\n .describe(\"Maximum characters to return\"),\n \n extractSections: z.boolean().default(true)\n .describe(\"Whether to identify and label sections\"),\n \n includeMetadata: z.boolean().default(true)\n .describe(\"Include author, date, and other metadata\")\n }),\n\n execute: async (input) => {\n return extractUrlContent(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| url | string | Yes | URL to read |\n| contentType | enum | No | Content type hint |\n| maxLength | number | No | Max chars (default: 10000) |\n| extractSections | boolean | No | Label sections |\n| includeMetadata | boolean | No | Include metadata |\n\n## Output Schema\n\n```typescript\ninterface ReadUrlResult {\n success: boolean;\n \n url: string;\n title: string;\n \n content: {\n full: string;\n sections?: {\n heading: string;\n level: number; // h1=1, h2=2, etc.\n content: string;\n }[];\n };\n \n metadata?: {\n author?: string;\n publishedDate?: string;\n lastModified?: string;\n description?: string;\n keywords?: string[];\n source: string;\n };\n \n stats: {\n totalCharacters: number;\n truncated: boolean;\n sectionsFound: number;\n };\n \n error?: {\n code: string;\n message: string;\n };\n}\n```\n\n## Usage Example\n\n```typescript\nconst content = await readUrl.execute({\n url: \"https://eugeneyan.com/writing/llm-evaluators/\",\n contentType: \"article\",\n maxLength: 15000,\n extractSections: true,\n includeMetadata: true\n});\n\n// Result:\n// {\n// success: true,\n// url: \"https://eugeneyan.com/writing/llm-evaluators/\",\n// title: \"Evaluating the Effectiveness of LLM-Evaluators\",\n// content: {\n// full: \"LLM-evaluators, also known as LLM-as-a-Judge...\",\n// sections: [\n// {\n// heading: \"Key considerations before adopting an LLM-evaluator\",\n// level: 2,\n// content: \"Before reviewing the literature...\"\n// },\n// ...\n// ]\n// },\n// metadata: {\n// author: \"Eugene Yan\",\n// publishedDate: \"2024-06-15\",\n// source: \"eugeneyan.com\"\n// },\n// stats: {\n// totalCharacters: 15000,\n// truncated: true,\n// sectionsFound: 8\n// }\n// }\n```\n\n## Content Type Handling\n\n| Type | Optimization |\n|------|-------------|\n| article | Prioritize main content, skip sidebars |\n| documentation | Preserve code blocks, keep structure |\n| paper | Extract abstract, sections, references |\n| code | Preserve formatting, syntax highlighting |\n| auto | Detect type from content |\n\n## Error Handling\n\n```typescript\nconst errorCodes = {\n \"URL_NOT_FOUND\": \"Page does not exist (404)\",\n \"ACCESS_DENIED\": \"Page requires authentication (401/403)\",\n \"TIMEOUT\": \"Request timed out\",\n \"BLOCKED\": \"Access blocked by robots.txt or rate limit\",\n \"INVALID_CONTENT\": \"Content could not be parsed\",\n \"UNSUPPORTED_TYPE\": \"Content type not supported (e.g., binary)\"\n};\n```\n\n## Implementation Notes\n\n1. **Respect robots.txt**: Check and honor robots.txt directives\n2. **Rate Limiting**: Don't hammer the same domain\n3. **User Agent**: Use appropriate user agent string\n4. **Timeouts**: Set reasonable timeouts (10-30s)\n5. **JavaScript Rendering**: Consider headless browser for JS-heavy sites\n6. **Caching**: Cache content for repeated reads\n\n\n# Web Search Tool\n\n## Purpose\n\nSearch the web for relevant information on a given topic. Returns structured results with snippets, URLs, and metadata.\n\n## Tool Definition\n\n```typescript\nimport { tool } from \"ai\";\nimport { z } from \"zod\";\n\nexport const webSearch = tool({\n description: `Search the web for information on a topic.\nReturns relevant results with snippets and URLs.\nUse for gathering current information, verifying facts, or research.`,\n\n parameters: z.object({\n query: z.string()\n .describe(\"Search query - be specific for better results\"),\n \n maxResults: z.number().min(1).max(20).default(10)\n .describe(\"Maximum number of results to return\"),\n \n filters: z.object({\n dateRange: z.enum([\"day\", \"week\", \"month\", \"year\", \"any\"]).default(\"any\")\n .describe(\"Limit results to a time period\"),\n \n sourceType: z.enum([\"all\", \"news\", \"academic\", \"documentation\"]).default(\"all\")\n .describe(\"Type of sources to prioritize\"),\n \n excludeDomains: z.array(z.string()).optional()\n .describe(\"Domains to exclude from results\")\n }).optional()\n }),\n\n execute: async (input) => {\n return performWebSearch(input);\n }\n});\n```\n\n## Input Schema\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| query | string | Yes | Search query |\n| maxResults | number | No | Max results (default: 10) |\n| filters.dateRange | enum | No | Time period filter |\n| filters.sourceType | enum | No | Source type priority |\n| filters.excludeDomains | string[] | No | Domains to exclude |\n\n## Output Schema\n\n```typescript\ninterface WebSearchResult {\n success: boolean;\n \n results: {\n title: string;\n url: string;\n snippet: string;\n source: string; // Domain name\n publishedDate?: string;\n relevanceScore: number;\n }[];\n \n totalResults: number;\n \n metadata: {\n query: string;\n searchTimeMs: number;\n filtersApplied: string[];\n };\n}\n```\n\n## Usage Example\n\n```typescript\nconst results = await webSearch.execute({\n query: \"LLM-as-a-Judge evaluation methods 2024\",\n maxResults: 10,\n filters: {\n dateRange: \"year\",\n sourceType: \"academic\"\n }\n});\n\n// Result:\n// {\n// success: true,\n// results: [\n// {\n// title: \"Judging LLM-as-a-Judge with MT-Bench\",\n// url: \"https://arxiv.org/abs/...\",\n// snippet: \"We study the effectiveness of LLM-as-a-Judge...\",\n// source: \"arxiv.org\",\n// publishedDate: \"2024-01-15\",\n// relevanceScore: 0.95\n// },\n// ...\n// ],\n// totalResults: 10,\n// metadata: {\n// query: \"LLM-as-a-Judge evaluation methods 2024\",\n// searchTimeMs: 342,\n// filtersApplied: [\"dateRange:year\", \"sourceType:academic\"]\n// }\n// }\n```\n\n## Query Optimization Tips\n\n1. **Specific Terms**: Use precise terminology\n2. **Quotes**: Use quotes for exact phrases\n3. **Operators**: Support site:, -term, OR\n4. **Context**: Include relevant context terms\n5. **Recency**: Add year for recent info\n\n## Implementation Notes\n\n1. **Rate Limiting**: Implement appropriate rate limits\n2. **Caching**: Cache results for repeated queries\n3. **Result Quality**: Filter out low-quality sources\n4. **Error Handling**: Handle API failures gracefully\n5. **Privacy**: Log queries appropriately\n\n\n---\nname: context-engineering-collection\ndescription: \"A comprehensive collection of Agent Skills for context engineering, harness engineering, multi-agent architectures, and production agent systems. Use when building, optimizing, evaluating, or debugging agent systems that require effective context management and reliable operating loops.\"\n---\n\n# Agent Skills for Context Engineering\n\nThis collection provides structured guidance for building production-grade AI agent systems through effective context engineering.\n\n## When to Activate\n\nActivate these skills when:\n- Building new agent systems from scratch\n- Optimizing existing agent performance\n- Debugging context-related failures\n- Designing multi-agent architectures\n- Creating or evaluating tools for agents\n- Implementing memory and persistence layers\n- Designing autonomous research or evaluation harnesses\n\n## Skill Map\n\n### Foundational Context Engineering\n\n**Understanding Context Fundamentals**\nContext is not just prompt text—it is the complete state available to the language model at inference time, including system instructions, tool definitions, retrieved documents, message history, and tool outputs. Effective context engineering means understanding what information truly matters for the task at hand and curating that information for maximum signal-to-noise ratio.\n\n**Recognizing Context Degradation**\nLanguage models exhibit predictable degradation patterns as context grows: the \"lost-in-middle\" phenomenon where information in the center of context receives less attention; U-shaped attention curves that prioritize beginning and end; context poisoning when errors compound; and context distraction when irrelevant information overwhelms relevant content.\n\n### Architectural Patterns\n\n**Multi-Agent Coordination**\nProduction multi-agent systems converge on three dominant patterns: supervisor/orchestrator architectures with centralized control, peer-to-peer swarm architectures for flexible handoffs, and hierarchical structures for complex task decomposition. The critical insight is that sub-agents exist primarily to isolate context rather than to simulate organizational roles.\n\n**Long-Horizon Prompting**\nLong-running autonomous agents and parallel orchestrations succeed or fail on the launch prompt. Pseudo-formal task briefs specify success predicates, non-counting outcomes, persistence rules with audit-gated return conditions, effort floors, diversity policies for parallel portfolios, and contamination guards, applying the discipline of formal verification linguistically to problems with no machine-checkable success condition. In orchestrated runs, each worker returns an output contract with named checks, and only evidenced violations block the return.\n\n**Memory System Design**\nMemory architectures range from simple scratchpads to sophisticated temporal knowledge graphs. Vector RAG provides semantic retrieval but loses relationship information. Knowledge graphs preserve structure but require more engineering investment. The file-system-as-memory pattern enables just-in-time context loading without stuffing context windows.\n\n**Filesystem-Based Context**\nThe filesystem provides a single interface for storing, retrieving, and updating effectively unlimited context. Key patterns include scratch pads for tool output offloading, plan persistence for long-horizon tasks, sub-agent communication via shared files, and dynamic skill loading. Agents use `ls`, `glob`, `grep`, and `read_file` for targeted context discovery, often outperforming semantic search for structural queries.\n\n**Hosted Agent Infrastructure**\nBackground coding agents run in remote sandboxed environments rather than on local machines. Key patterns include pre-built environment images refreshed on regular cadence, warm sandbox pools for instant session starts, filesystem snapshots for session persistence, and multiplayer support for collaborative agent sessions. Critical optimizations include allowing file reads before git sync completes (blocking only writes), predictive sandbox warming when users start typing, and self-spawning agents for parallel task execution.\n\n**Tool Design Principles**\nTools are contracts between deterministic systems and non-deterministic agents. Effective tool design follows the consolidation principle (prefer single comprehensive tools over multiple narrow ones), returns contextual information in errors, supports response format options for token efficiency, and uses clear namespacing.\n\n### Operational Excellence\n\n**Context Compression**\nWhen agent sessions exhaust memory, compression becomes mandatory. The correct optimization target is tokens-per-task, not tokens-per-request. Structured summarization with explicit sections for files, decisions, and next steps preserves more useful information than aggressive compression. Artifact trail integrity remains the weakest dimension across all compression methods.\n\n**Context Optimization**\nTechniques include compaction (summarizing context near limits), observation masking (replacing verbose tool outputs with references), prefix caching (reusing KV blocks across requests), and strategic context partitioning (splitting work across sub-agents with isolated contexts).\n\n**Self-Managed Context**\nThe model, not the harness, decides what stays in its live context: the editable region is exposed as a file the model rewrites with code tools and re-parsed each turn. The harness keeps the invariants (pinned system and task prefix, role folding, edit gate, receipts, deterministic budget readouts, rollback on overflow). Edit cost scales with the text after the edit under prefix caching, so edits are batched and placed with the tail in mind; strategy can be steered by instruction, evolved as a skill, or trained with a success-gated efficiency reward.\n\n**Latent Briefing (KV Memory Sharing)**\nOrchestrator-worker systems can compound tokens when supervisors accumulate long trajectories but workers see only narrow text slices. Latent Briefing compacts the orchestrator trajectory in the worker model's KV cache using task-guided attention (Attention Matching-style compaction) so workers receive relevant latent state without full-text replay when the stack exposes worker KV state and the models are compatible.\n\n**Evaluation Frameworks**\nProduction agent evaluation requires deterministic checks and multi-dimensional rubrics covering factual accuracy, completeness, tool efficiency, and process quality. Use model judges only after structure, evidence, and rubric math are valid; route judge design, pairwise comparison, and bias mitigation to Advanced Evaluation.\n\n**Harness Engineering**\nReliable autonomous agents need explicit operating loops around the model: locked metrics, editable surfaces, durable logs, novelty checks, rollback rules, and human approval boundaries. Harnesses prevent agents from weakening the evaluator, losing state across compaction, or turning ambiguous goals into unreviewable changes.\n\n**Self-Improvement Loops**\nWhen the harness itself becomes the optimization target, a different discipline applies: recursive self-improvement, meta-harness search, failure-driven bounded self-edits, evolutionary scaffold search, and context mechanism evolution. The controlling constraints are empirical two-split acceptance gates, filesystem experience archives with raw traces, runtime-enforced constraints outside every editable surface, and diversity preservation to prevent collapse.\n\n### Development Methodology\n\n**Project Development**\nEffective LLM project development begins with task-model fit analysis: validating through manual prototyping that a task is well-suited for LLM processing before building automation. Production pipelines follow staged, idempotent architectures (acquire, prepare, process, parse, render) with file system state management for debugging and caching. Structured output design with explicit format specifications enables reliable parsing. Start with minimal architecture and add complexity only when proven necessary.\n\n### Cognitive Architecture\n\n**BDI Mental States**\nBelief-desire-intention modeling provides a formal way to translate structured external context into agent mental states. Use it for rational agency, explainability, and systems that need auditable links between beliefs, goals, and chosen actions.\n\n## Core Concepts\n\nThe collection is organized around four core themes. First, context fundamentals establish what context is, how attention mechanisms work, and why context quality matters more than quantity. Second, architectural patterns cover the structures and coordination mechanisms that enable effective agent systems. Third, operational excellence addresses optimization, evaluation, and harness reliability. Fourth, development methodology and cognitive architecture cover project execution and formal mental-state modeling.\n\n## Practical Guidance\n\nEach skill can be used independently or in combination. Start with fundamentals to establish context management mental models. Branch into architectural patterns based on your system requirements. Reference operational skills when optimizing production systems.\n\nThe skills are platform-agnostic and work with Claude Code, Cursor, or any agent framework that supports custom instructions or skill-like constructs.\n\n## Integration\n\nThis collection integrates with itself—skills reference each other and build on shared concepts. The fundamentals skill provides context for all other skills. Architectural skills (multi-agent, memory, tools) can be combined for complex systems. Operational skills (optimization, evaluation) apply to any system built using the foundational and architectural skills.\n\n## References\n\nInternal skills in this collection:\n- [context-fundamentals](skills/context-fundamentals/SKILL.md)\n- [context-degradation](skills/context-degradation/SKILL.md)\n- [context-compression](skills/context-compression/SKILL.md)\n- [multi-agent-patterns](skills/multi-agent-patterns/SKILL.md)\n- [long-horizon-prompting](skills/long-horizon-prompting/SKILL.md)\n- [memory-systems](skills/memory-systems/SKILL.md)\n- [tool-design](skills/tool-design/SKILL.md)\n- [filesystem-context](skills/filesystem-context/SKILL.md)\n- [hosted-agents](skills/hosted-agents/SKILL.md)\n- [context-optimization](skills/context-optimization/SKILL.md)\n- [self-managed-context](skills/self-managed-context/SKILL.md)\n- [latent-briefing](skills/latent-briefing/SKILL.md)\n- [evaluation](skills/evaluation/SKILL.md)\n- [advanced-evaluation](skills/advanced-evaluation/SKILL.md)\n- [harness-engineering](skills/harness-engineering/SKILL.md)\n- [self-improvement-loops](skills/self-improvement-loops/SKILL.md)\n- [project-development](skills/project-development/SKILL.md)\n- [bdi-mental-states](skills/bdi-mental-states/SKILL.md)\n\nExternal resources on context engineering:\n- Research on attention mechanisms and context window limitations\n- Production experience from leading AI labs on agent system design\n- Framework documentation for LangGraph, AutoGen, and CrewAI\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-20\n**Last Updated**: 2026-10-01\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.6.0\n\n\n# Bias Mitigation Techniques for LLM Evaluation\n\nThis reference details specific techniques for mitigating known biases in LLM-as-a-Judge systems.\n\n## Position Bias\n\n### The Problem\n\nIn pairwise comparison, LLMs systematically prefer responses in certain positions. Research shows:\n- GPT has mild first-position bias (~55% preference for first position in ties)\n- Claude shows similar patterns\n- Smaller models often show stronger bias\n\n### Mitigation: Position Swapping Protocol\n\n```python\nasync def position_swap_comparison(response_a, response_b, prompt, criteria):\n # Pass 1: Original order\n result_ab = await compare(response_a, response_b, prompt, criteria)\n \n # Pass 2: Swapped order\n result_ba = await compare(response_b, response_a, prompt, criteria)\n \n # Map second result (A in second position → B in first)\n result_ba_mapped = {\n 'winner': {'A': 'B', 'B': 'A', 'TIE': 'TIE'}[result_ba['winner']],\n 'confidence': result_ba['confidence']\n }\n \n # Consistency check\n if result_ab['winner'] == result_ba_mapped['winner']:\n return {\n 'winner': result_ab['winner'],\n 'confidence': (result_ab['confidence'] + result_ba_mapped['confidence']) / 2,\n 'position_consistent': True\n }\n else:\n # Disagreement indicates position bias was a factor\n return {\n 'winner': 'TIE',\n 'confidence': 0.5,\n 'position_consistent': False,\n 'bias_detected': True\n }\n```\n\n### Alternative: Multiple Shuffles\n\nFor higher reliability, use multiple position orderings:\n\n```python\nasync def multi_shuffle_comparison(response_a, response_b, prompt, criteria, n_shuffles=3):\n results = []\n for i in range(n_shuffles):\n if i % 2 == 0:\n r = await compare(response_a, response_b, prompt, criteria)\n else:\n r = await compare(response_b, response_a, prompt, criteria)\n r['winner'] = {'A': 'B', 'B': 'A', 'TIE': 'TIE'}[r['winner']]\n results.append(r)\n \n # Majority vote\n winners = [r['winner'] for r in results]\n final_winner = max(set(winners), key=winners.count)\n agreement = winners.count(final_winner) / len(winners)\n \n return {\n 'winner': final_winner,\n 'confidence': agreement,\n 'n_shuffles': n_shuffles\n }\n```\n\n## Length Bias\n\n### The Problem\n\nLLMs tend to rate longer responses higher, regardless of quality. This manifests as:\n- Verbose responses receiving inflated scores\n- Concise but complete responses penalized\n- Padding and repetition being rewarded\n\n### Mitigation: Explicit Prompting\n\nInclude anti-length-bias instructions in the prompt:\n\n```\nCRITICAL EVALUATION GUIDELINES:\n- Do NOT prefer responses because they are longer\n- Concise, complete answers are as valuable as detailed ones\n- Penalize unnecessary verbosity or repetition\n- Focus on information density, not word count\n```\n\n### Mitigation: Length-Normalized Scoring\n\n```python\ndef length_normalized_score(score, response_length, target_length=500):\n \"\"\"Adjust score based on response length.\"\"\"\n length_ratio = response_length / target_length\n \n if length_ratio > 2.0:\n # Penalize excessively long responses\n penalty = (length_ratio - 2.0) * 0.1\n return max(score - penalty, 1)\n elif length_ratio < 0.3:\n # Penalize excessively short responses\n penalty = (0.3 - length_ratio) * 0.5\n return max(score - penalty, 1)\n else:\n return score\n```\n\n### Mitigation: Separate Length Criterion\n\nMake length a separate, explicit criterion so it's not implicitly rewarded:\n\n```python\ncriteria = [\n {\"name\": \"Accuracy\", \"description\": \"Factual correctness\", \"weight\": 0.4},\n {\"name\": \"Completeness\", \"description\": \"Covers key points\", \"weight\": 0.3},\n {\"name\": \"Conciseness\", \"description\": \"No unnecessary content\", \"weight\": 0.3} # Explicit\n]\n```\n\n## Self-Enhancement Bias\n\n### The Problem\n\nModels rate outputs generated by themselves (or similar models) higher than outputs from different models.\n\n### Mitigation: Cross-Model Evaluation\n\nUse a different model family for evaluation than generation:\n\n```python\ndef get_evaluator_model(generator_model):\n \"\"\"Select evaluator to avoid self-enhancement bias.\"\"\"\n if 'gpt' in generator_model.lower():\n return 'claude-4-5-sonnet'\n elif 'claude' in generator_model.lower():\n return 'gpt-5.2'\n else:\n return 'gpt-5.2' # Default\n```\n\n### Mitigation: Blind Evaluation\n\nRemove model attribution from responses before evaluation:\n\n```python\ndef anonymize_response(response, model_name):\n \"\"\"Remove model-identifying patterns.\"\"\"\n patterns = [\n f\"As {model_name}\",\n \"I am an AI\",\n \"I don't have personal opinions\",\n # Model-specific patterns\n ]\n anonymized = response\n for pattern in patterns:\n anonymized = anonymized.replace(pattern, \"[REDACTED]\")\n return anonymized\n```\n\n## Verbosity Bias\n\n### The Problem\n\nDetailed explanations receive higher scores even when the extra detail is irrelevant or incorrect.\n\n### Mitigation: Relevance-Weighted Scoring\n\n```python\nasync def relevance_weighted_evaluation(response, prompt, criteria):\n # First, assess relevance of each segment\n relevance_scores = await assess_relevance(response, prompt)\n \n # Weight evaluation by relevance\n segments = split_into_segments(response)\n weighted_scores = []\n for segment, relevance in zip(segments, relevance_scores):\n if relevance > 0.5: # Only count relevant segments\n score = await evaluate_segment(segment, prompt, criteria)\n weighted_scores.append(score * relevance)\n \n return sum(weighted_scores) / len(weighted_scores)\n```\n\n### Mitigation: Rubric with Verbosity Penalty\n\nInclude explicit verbosity penalties in rubrics:\n\n```python\nrubric_levels = [\n {\n \"score\": 5,\n \"description\": \"Complete and concise. All necessary information, nothing extraneous.\",\n \"characteristics\": [\"Every sentence adds value\", \"No repetition\", \"Appropriately scoped\"]\n },\n {\n \"score\": 3,\n \"description\": \"Complete but verbose. Contains unnecessary detail or repetition.\",\n \"characteristics\": [\"Main points covered\", \"Some tangents\", \"Could be more concise\"]\n },\n # ... etc\n]\n```\n\n## Authority Bias\n\n### The Problem\n\nConfident, authoritative tone is rated higher regardless of accuracy.\n\n### Mitigation: Evidence Requirement\n\nRequire explicit evidence for claims:\n\n```\nFor each claim in the response:\n1. Identify whether it's a factual claim\n2. Note if evidence or sources are provided\n3. Score based on verifiability, not confidence\n\nIMPORTANT: Confident claims without evidence should NOT receive higher scores than \nhedged claims with evidence.\n```\n\n### Mitigation: Fact-Checking Layer\n\nAdd a fact-checking step before scoring:\n\n```python\nasync def fact_checked_evaluation(response, prompt, criteria):\n # Extract claims\n claims = await extract_claims(response)\n \n # Fact-check each claim\n fact_check_results = await asyncio.gather(*[\n verify_claim(claim) for claim in claims\n ])\n \n # Adjust score based on fact-check results\n accuracy_factor = sum(r['verified'] for r in fact_check_results) / len(fact_check_results)\n \n base_score = await evaluate(response, prompt, criteria)\n return base_score * (0.7 + 0.3 * accuracy_factor) # At least 70% of score\n```\n\n## Aggregate Bias Detection\n\nMonitor for systematic biases in production:\n\n```python\nclass BiasMonitor:\n def __init__(self):\n self.evaluations = []\n \n def record(self, evaluation):\n self.evaluations.append(evaluation)\n \n def detect_position_bias(self):\n \"\"\"Detect if first position wins more often than expected.\"\"\"\n first_wins = sum(1 for e in self.evaluations if e['first_position_winner'])\n expected = len(self.evaluations) * 0.5\n z_score = (first_wins - expected) / (expected * 0.5) ** 0.5\n return {'bias_detected': abs(z_score) > 2, 'z_score': z_score}\n \n def detect_length_bias(self):\n \"\"\"Detect if longer responses score higher.\"\"\"\n from scipy.stats import spearmanr\n lengths = [e['response_length'] for e in self.evaluations]\n scores = [e['score'] for e in self.evaluations]\n corr, p_value = spearmanr(lengths, scores)\n return {'bias_detected': corr > 0.3 and p_value < 0.05, 'correlation': corr}\n```\n\n## Summary Table\n\n| Bias | Primary Mitigation | Secondary Mitigation | Detection Method |\n|------|-------------------|---------------------|------------------|\n| Position | Position swapping | Multiple shuffles | Consistency check |\n| Length | Explicit prompting | Length normalization | Length-score correlation |\n| Self-enhancement | Cross-model evaluation | Anonymization | Model comparison study |\n| Verbosity | Relevance weighting | Rubric penalties | Relevance scoring |\n| Authority | Evidence requirement | Fact-checking layer | Confidence-accuracy correlation |\n\n\n# Evaluation Pipeline Diagram\n\nVisual layout of a production evaluation pipeline.\n\n```\n┌─────────────────────────────────────────────────┐\n│ Evaluation Pipeline │\n├─────────────────────────────────────────────────┤\n│ │\n│ Input: Response + Prompt + Context │\n│ │ │\n│ ▼ │\n│ ┌─────────────────────┐ │\n│ │ Criteria Loader │ ◄── Rubrics, weights │\n│ └──────────┬──────────┘ │\n│ │ │\n│ ▼ │\n│ ┌─────────────────────┐ │\n│ │ Primary Scorer │ ◄── Direct or Pairwise │\n│ └──────────┬──────────┘ │\n│ │ │\n│ ▼ │\n│ ┌─────────────────────┐ │\n│ │ Bias Mitigation │ ◄── Position swap, etc. │\n│ └──────────┬──────────┘ │\n│ │ │\n│ ▼ │\n│ ┌─────────────────────┐ │\n│ │ Confidence Scoring │ ◄── Calibration │\n│ └──────────┬──────────┘ │\n│ │ │\n│ ▼ │\n│ Output: Scores + Justifications + Confidence │\n│ │\n└─────────────────────────────────────────────────┘\n```\n\n## Pipeline Stages\n\n1. **Criteria Loader**: Loads rubrics and criterion weights from configuration\n2. **Primary Scorer**: Applies direct scoring or pairwise comparison\n3. **Bias Mitigation**: Runs position swaps, length normalization, and other debiasing\n4. **Confidence Scoring**: Calibrates confidence based on position consistency and evidence strength\n\n\n# LLM-as-Judge Implementation Patterns\n\nThis reference provides detailed implementation patterns for building production-grade LLM evaluation systems.\n\n## Pattern 1: Structured Evaluation Pipeline\n\nThe most reliable evaluation systems follow a structured pipeline that separates concerns:\n\n```\nInput Validation → Criteria Loading → Scoring → Bias Mitigation → Output Formatting\n```\n\n### Input Validation Layer\n\nBefore evaluation begins, validate:\n\n1. **Response presence**: Non-empty response to evaluate\n2. **Prompt presence**: Original prompt for context\n3. **Criteria validity**: At least one criterion with name and description\n4. **Weight normalization**: Weights sum to 1.0 (or normalize them)\n\n```python\ndef validate_input(response, prompt, criteria):\n if not response or not response.strip():\n raise ValueError(\"Response cannot be empty\")\n if not prompt or not prompt.strip():\n raise ValueError(\"Prompt cannot be empty\")\n if not criteria or len(criteria) == 0:\n raise ValueError(\"At least one criterion required\")\n \n # Normalize weights\n total_weight = sum(c.get('weight', 1) for c in criteria)\n for c in criteria:\n c['weight'] = c.get('weight', 1) / total_weight\n```\n\n### Criteria Loading Layer\n\nCriteria should be loaded from configuration, not hardcoded:\n\n```python\nclass CriteriaLoader:\n def __init__(self, rubric_path=None):\n self.rubrics = self._load_rubrics(rubric_path)\n \n def get_criteria(self, task_type):\n return self.rubrics.get(task_type, self.default_criteria)\n \n def get_rubric(self, criterion_name):\n return self.rubrics.get(criterion_name, {}).get('levels', [])\n```\n\n### Scoring Layer\n\nThe scoring layer handles the actual LLM call:\n\n```python\nasync def score_response(response, prompt, criteria, rubric, model):\n system_prompt = build_system_prompt(criteria, rubric)\n user_prompt = build_user_prompt(response, prompt, criteria)\n \n result = await generate_text(\n model=model,\n system=system_prompt,\n prompt=user_prompt,\n temperature=0.3 # Lower temperature for consistency\n )\n \n return parse_scores(result.text)\n```\n\n### Bias Mitigation Layer\n\nFor pairwise comparison, always include position swapping:\n\n```python\nasync def compare_with_bias_mitigation(response_a, response_b, prompt, criteria, model):\n # First pass: A first\n pass1 = await compare_pair(response_a, response_b, prompt, criteria, model)\n \n # Second pass: B first\n pass2 = await compare_pair(response_b, response_a, prompt, criteria, model)\n \n # Map pass2 winner back\n pass2_mapped = map_winner(pass2.winner) # A→B, B→A, TIE→TIE\n \n # Check consistency\n if pass1.winner == pass2_mapped:\n return {\n 'winner': pass1.winner,\n 'confidence': (pass1.confidence + pass2.confidence) / 2,\n 'consistent': True\n }\n else:\n return {\n 'winner': 'TIE',\n 'confidence': 0.5,\n 'consistent': False\n }\n```\n\n## Pattern 2: Hierarchical Evaluation\n\nFor complex evaluations, use a hierarchical approach:\n\n```\nQuick Screen (cheap model) → Detailed Evaluation (expensive model) → Human Review (edge cases)\n```\n\n### Quick Screen Implementation\n\n```python\nasync def quick_screen(response, prompt, threshold=0.7):\n \"\"\"Fast, cheap screening for obvious passes/fails.\"\"\"\n result = await generate_text(\n model='gpt-5.2', # Cheaper model\n prompt=f\"Rate 0-1 if this response adequately addresses the prompt:\\n\\nPrompt: {prompt}\\n\\nResponse: {response}\",\n temperature=0\n )\n score = float(result.text.strip())\n return score, score > threshold\n```\n\n### Detailed Evaluation\n\n```python\nasync def detailed_evaluation(response, prompt, criteria):\n \"\"\"Full evaluation for borderline or important cases.\"\"\"\n result = await generate_text(\n model='gpt-5.2', # More capable model\n system=DETAILED_EVALUATION_PROMPT,\n prompt=build_detailed_prompt(response, prompt, criteria),\n temperature=0.3\n )\n return parse_detailed_scores(result.text)\n```\n\n## Pattern 3: Panel of LLM Judges (PoLL)\n\nFor high-stakes evaluation, use multiple models:\n\n```python\nasync def poll_evaluation(response, prompt, criteria, models):\n \"\"\"Aggregate judgments from multiple LLM judges.\"\"\"\n results = await asyncio.gather(*[\n score_with_model(response, prompt, criteria, model)\n for model in models\n ])\n \n # Aggregate scores\n aggregated = aggregate_scores(results)\n \n # Calculate agreement\n agreement = calculate_agreement(results)\n \n return {\n 'scores': aggregated,\n 'agreement': agreement,\n 'individual_results': results\n }\n\ndef aggregate_scores(results):\n \"\"\"Aggregate scores using median (robust to outliers).\"\"\"\n scores = {}\n for criterion in results[0]['scores'].keys():\n criterion_scores = [r['scores'][criterion] for r in results]\n scores[criterion] = {\n 'score': statistics.median(criterion_scores),\n 'std': statistics.stdev(criterion_scores) if len(criterion_scores) > 1 else 0\n }\n return scores\n```\n\n## Pattern 4: Confidence Calibration\n\nConfidence scores should be calibrated to actual reliability:\n\n```python\ndef calibrate_confidence(raw_confidence, position_consistent, evidence_count):\n \"\"\"Calibrate confidence based on multiple signals.\"\"\"\n \n # Base confidence from model output\n calibrated = raw_confidence\n \n # Position consistency is a strong signal\n if not position_consistent:\n calibrated *= 0.6 # Significant reduction\n \n # More evidence = higher confidence\n evidence_factor = min(evidence_count / 3, 1.0) # Cap at 3 pieces\n calibrated *= (0.7 + 0.3 * evidence_factor)\n \n return min(calibrated, 0.99) # Never 100% confident\n```\n\n## Pattern 5: Output Formatting\n\nAlways return structured outputs with consistent schemas:\n\n```python\n@dataclass\nclass ScoreResult:\n criterion: str\n score: float\n max_score: float\n justification: str\n evidence: List[str]\n improvement: str\n\n@dataclass\nclass EvaluationResult:\n success: bool\n scores: List[ScoreResult]\n overall_score: float\n weighted_score: float\n summary: Dict[str, Any]\n metadata: Dict[str, Any]\n\ndef format_output(scores, metadata) -> EvaluationResult:\n \"\"\"Format evaluation results consistently.\"\"\"\n return EvaluationResult(\n success=True,\n scores=scores,\n overall_score=sum(s.score for s in scores) / len(scores),\n weighted_score=calculate_weighted_score(scores),\n summary=generate_summary(scores),\n metadata=metadata\n )\n```\n\n## Error Handling Patterns\n\n### Graceful Degradation\n\n```python\nasync def evaluate_with_fallback(response, prompt, criteria):\n try:\n return await full_evaluation(response, prompt, criteria)\n except RateLimitError:\n # Fall back to simpler evaluation\n return await simple_evaluation(response, prompt, criteria)\n except ParseError as e:\n # Return partial results with error flag\n return {\n 'success': False,\n 'partial_results': e.partial_data,\n 'error': str(e)\n }\n```\n\n### Retry Logic\n\n```python\nasync def evaluate_with_retry(response, prompt, criteria, max_retries=3):\n for attempt in range(max_retries):\n try:\n result = await evaluate(response, prompt, criteria)\n if is_valid_result(result):\n return result\n except TransientError:\n await asyncio.sleep(2 ** attempt) # Exponential backoff\n \n raise EvaluationError(\"Max retries exceeded\")\n```\n\n## Testing Patterns\n\n### Unit Tests for Parsing\n\n```python\ndef test_score_parsing():\n raw_output = '{\"scores\": [{\"criterion\": \"Accuracy\", \"score\": 4}]}'\n result = parse_scores(raw_output)\n assert result.scores[0].criterion == \"Accuracy\"\n assert result.scores[0].score == 4\n\ndef test_malformed_output():\n raw_output = 'Invalid JSON'\n with pytest.raises(ParseError):\n parse_scores(raw_output)\n```\n\n### Integration Tests with Real API\n\n```python\n@pytest.mark.integration\nasync def test_full_evaluation_pipeline():\n result = await evaluate(\n response=\"Water boils at 100°C at sea level.\",\n prompt=\"At what temperature does water boil?\",\n criteria=[{\"name\": \"Accuracy\", \"description\": \"Factual correctness\", \"weight\": 1}]\n )\n \n assert result.success\n assert len(result.scores) == 1\n assert result.scores[0].score >= 4 # Should score high for accurate response\n```\n\n### Bias Detection Tests\n\n```python\nasync def test_position_bias_mitigation():\n # Same response in both positions should tie\n result = await compare(\n response_a=\"Same response\",\n response_b=\"Same response\",\n prompt=\"Test prompt\",\n criteria=[\"quality\"],\n swap_positions=True\n )\n \n assert result.winner == \"TIE\"\n assert result.consistent == True\n```\n\n\n# Metric Selection Guide for LLM Evaluation\n\nThis reference provides guidance on selecting appropriate metrics for different evaluation scenarios.\n\n## Metric Categories\n\n### Classification Metrics\n\nUse for binary or multi-class evaluation tasks (pass/fail, correct/incorrect).\n\n#### Precision\n\n```\nPrecision = True Positives / (True Positives + False Positives)\n```\n\n**Interpretation**: Of all responses the judge said were good, what fraction were actually good?\n\n**Use when**: False positives are costly (e.g., approving unsafe content)\n\n```python\ndef precision(predictions, ground_truth):\n true_positives = sum(1 for p, g in zip(predictions, ground_truth) if p == 1 and g == 1)\n predicted_positives = sum(predictions)\n return true_positives / predicted_positives if predicted_positives > 0 else 0\n```\n\n#### Recall\n\n```\nRecall = True Positives / (True Positives + False Negatives)\n```\n\n**Interpretation**: Of all actually good responses, what fraction did the judge identify?\n\n**Use when**: False negatives are costly (e.g., missing good content in filtering)\n\n```python\ndef recall(predictions, ground_truth):\n true_positives = sum(1 for p, g in zip(predictions, ground_truth) if p == 1 and g == 1)\n actual_positives = sum(ground_truth)\n return true_positives / actual_positives if actual_positives > 0 else 0\n```\n\n#### F1 Score\n\n```\nF1 = 2 * (Precision * Recall) / (Precision + Recall)\n```\n\n**Interpretation**: Harmonic mean of precision and recall\n\n**Use when**: You need a single number balancing both concerns\n\n```python\ndef f1_score(predictions, ground_truth):\n p = precision(predictions, ground_truth)\n r = recall(predictions, ground_truth)\n return 2 * p * r / (p + r) if (p + r) > 0 else 0\n```\n\n### Agreement Metrics\n\nUse for comparing automated evaluation with human judgment.\n\n#### Cohen's Kappa (κ)\n\n```\nκ = (Observed Agreement - Expected Agreement) / (1 - Expected Agreement)\n```\n\n**Interpretation**: Agreement adjusted for chance\n- κ > 0.8: Almost perfect agreement\n- κ 0.6-0.8: Substantial agreement\n- κ 0.4-0.6: Moderate agreement\n- κ < 0.4: Fair to poor agreement\n\n**Use for**: Binary or categorical judgments\n\n```python\ndef cohens_kappa(judge1, judge2):\n from sklearn.metrics import cohen_kappa_score\n return cohen_kappa_score(judge1, judge2)\n```\n\n#### Weighted Kappa\n\nFor ordinal scales where disagreement severity matters:\n\n```python\ndef weighted_kappa(judge1, judge2):\n from sklearn.metrics import cohen_kappa_score\n return cohen_kappa_score(judge1, judge2, weights='quadratic')\n```\n\n**Interpretation**: Penalizes large disagreements more than small ones\n\n### Correlation Metrics\n\nUse for ordinal/continuous scores.\n\n#### Spearman's Rank Correlation (ρ)\n\n**Interpretation**: Correlation between rankings, not absolute values\n- ρ > 0.9: Very strong correlation\n- ρ 0.7-0.9: Strong correlation\n- ρ 0.5-0.7: Moderate correlation\n- ρ < 0.5: Weak correlation\n\n**Use when**: Order matters more than exact values\n\n```python\ndef spearmans_rho(scores1, scores2):\n from scipy.stats import spearmanr\n rho, p_value = spearmanr(scores1, scores2)\n return {'rho': rho, 'p_value': p_value}\n```\n\n#### Kendall's Tau (τ)\n\n**Interpretation**: Similar to Spearman but based on pairwise concordance\n\n**Use when**: You have many tied values\n\n```python\ndef kendalls_tau(scores1, scores2):\n from scipy.stats import kendalltau\n tau, p_value = kendalltau(scores1, scores2)\n return {'tau': tau, 'p_value': p_value}\n```\n\n#### Pearson Correlation (r)\n\n**Interpretation**: Linear correlation between scores\n\n**Use when**: Exact score values matter, not just order\n\n```python\ndef pearsons_r(scores1, scores2):\n from scipy.stats import pearsonr\n r, p_value = pearsonr(scores1, scores2)\n return {'r': r, 'p_value': p_value}\n```\n\n### Pairwise Comparison Metrics\n\n#### Agreement Rate\n\n```\nAgreement = (Matching Decisions) / (Total Comparisons)\n```\n\n**Interpretation**: Simple percentage of agreement\n\n```python\ndef pairwise_agreement(decisions1, decisions2):\n matches = sum(1 for d1, d2 in zip(decisions1, decisions2) if d1 == d2)\n return matches / len(decisions1)\n```\n\n#### Position Consistency\n\n```\nConsistency = (Consistent across position swaps) / (Total comparisons)\n```\n\n**Interpretation**: How often does swapping position change the decision?\n\n```python\ndef position_consistency(results):\n consistent = sum(1 for r in results if r['position_consistent'])\n return consistent / len(results)\n```\n\n## Selection Decision Tree\n\n```\nWhat type of evaluation task?\n│\n├── Binary classification (pass/fail)\n│ └── Use: Precision, Recall, F1, Cohen's κ\n│\n├── Ordinal scale (1-5 rating)\n│ ├── Comparing to human judgments?\n│ │ └── Use: Spearman's ρ, Weighted κ\n│ └── Comparing two automated judges?\n│ └── Use: Kendall's τ, Spearman's ρ\n│\n├── Pairwise preference\n│ └── Use: Agreement rate, Position consistency\n│\n└── Multi-label classification\n └── Use: Macro-F1, Micro-F1, Per-label metrics\n```\n\n## Metric Selection by Use Case\n\n### Use Case 1: Validating Automated Evaluation\n\n**Goal**: Ensure automated evaluation correlates with human judgment\n\n**Recommended Metrics**:\n1. Primary: Spearman's ρ (for ordinal scales) or Cohen's κ (for categorical)\n2. Secondary: Per-criterion agreement\n3. Diagnostic: Confusion matrix for systematic errors\n\n```python\ndef validate_automated_eval(automated_scores, human_scores, criteria):\n results = {}\n \n # Overall correlation\n results['overall_spearman'] = spearmans_rho(automated_scores, human_scores)\n \n # Per-criterion agreement\n for criterion in criteria:\n auto_crit = [s[criterion] for s in automated_scores]\n human_crit = [s[criterion] for s in human_scores]\n results[f'{criterion}_spearman'] = spearmans_rho(auto_crit, human_crit)\n \n return results\n```\n\n### Use Case 2: Comparing Two Models\n\n**Goal**: Determine which model produces better outputs\n\n**Recommended Metrics**:\n1. Primary: Win rate (from pairwise comparison)\n2. Secondary: Position consistency (bias check)\n3. Diagnostic: Per-criterion breakdown\n\n```python\ndef compare_models(model_a_outputs, model_b_outputs, prompts):\n results = []\n for a, b, p in zip(model_a_outputs, model_b_outputs, prompts):\n comparison = await compare_with_position_swap(a, b, p)\n results.append(comparison)\n \n return {\n 'a_wins': sum(1 for r in results if r['winner'] == 'A'),\n 'b_wins': sum(1 for r in results if r['winner'] == 'B'),\n 'ties': sum(1 for r in results if r['winner'] == 'TIE'),\n 'position_consistency': position_consistency(results)\n }\n```\n\n### Use Case 3: Quality Monitoring\n\n**Goal**: Track evaluation quality over time\n\n**Recommended Metrics**:\n1. Primary: Rolling agreement with human spot-checks\n2. Secondary: Score distribution stability\n3. Diagnostic: Bias indicators (position, length)\n\n```python\nclass QualityMonitor:\n def __init__(self, window_size=100):\n self.window = deque(maxlen=window_size)\n \n def add_evaluation(self, automated, human_spot_check=None):\n self.window.append({\n 'automated': automated,\n 'human': human_spot_check,\n 'length': len(automated['response'])\n })\n \n def get_metrics(self):\n # Filter to evaluations with human spot-checks\n with_human = [e for e in self.window if e['human'] is not None]\n \n if len(with_human) < 10:\n return {'insufficient_data': True}\n \n auto_scores = [e['automated']['score'] for e in with_human]\n human_scores = [e['human']['score'] for e in with_human]\n \n return {\n 'correlation': spearmans_rho(auto_scores, human_scores),\n 'mean_difference': np.mean([a - h for a, h in zip(auto_scores, human_scores)]),\n 'length_correlation': spearmans_rho(\n [e['length'] for e in self.window],\n [e['automated']['score'] for e in self.window]\n )\n }\n```\n\n## Interpreting Metric Results\n\n### Good Evaluation System Indicators\n\n| Metric | Good | Acceptable | Concerning |\n|--------|------|------------|------------|\n| Spearman's ρ | > 0.8 | 0.6-0.8 | < 0.6 |\n| Cohen's κ | > 0.7 | 0.5-0.7 | < 0.5 |\n| Position consistency | > 0.9 | 0.8-0.9 | < 0.8 |\n| Length correlation | < 0.2 | 0.2-0.4 | > 0.4 |\n\n### Warning Signs\n\n1. **High agreement but low correlation**: May indicate calibration issues\n2. **Low position consistency**: Position bias affecting results\n3. **High length correlation**: Length bias inflating scores\n4. **Per-criterion variance**: Some criteria may be poorly defined\n\n## Reporting Template\n\n```markdown\n## Evaluation System Metrics Report\n\n### Human Agreement\n- Spearman's ρ: 0.82 (p < 0.001)\n- Cohen's κ: 0.74\n- Sample size: 500 evaluations\n\n### Bias Indicators\n- Position consistency: 91%\n- Length-score correlation: 0.12\n\n### Per-Criterion Performance\n| Criterion | Spearman's ρ | κ |\n|-----------|--------------|---|\n| Accuracy | 0.88 | 0.79 |\n| Clarity | 0.76 | 0.68 |\n| Completeness | 0.81 | 0.72 |\n\n### Recommendations\n- All metrics within acceptable ranges\n- Monitor \"Clarity\" criterion - lower agreement may indicate need for rubric refinement\n```\n\n\n---\nname: advanced-evaluation\ndescription: \"This skill should be used for advanced LLM evaluation: LLM-as-judge systems, direct scoring, pairwise comparison, rubric calibration, evaluator bias mitigation, confidence scoring, and automated quality assessment.\"\n---\n\n# Advanced Evaluation\n\nThis skill covers production-grade techniques for evaluating LLM outputs using LLMs as judges. It synthesizes research from academic papers, industry practices, and practical implementation experience into actionable patterns for building reliable evaluation systems.\n\n**Key insight**: LLM-as-a-Judge is not a single technique but a family of approaches, each suited to different evaluation contexts. Choosing the right approach and mitigating known biases is the core competency this skill develops.\n\n## When to Activate\n\nActivate this skill when:\n\n- Building LLM-as-judge systems for LLM outputs\n- Comparing multiple model responses to select the best one\n- Establishing consistent quality standards across evaluation teams\n- Debugging evaluation systems that show inconsistent results\n- Designing A/B tests for prompt or model changes\n- Creating rubrics specifically for LLM or human/LLM hybrid judges\n- Analyzing correlation between automated and human judgments\n\nDo not activate this skill for adjacent work owned by other skills:\n- General deterministic checks, regression suites, production quality gates, or outcome metrics: `evaluation`.\n- Autonomous loop governance, locked rubrics, rollback, or PR approval boundaries: `harness-engineering`.\n- Tool API contracts for evaluation tools: `tool-design`.\n\n## Core Concepts\n\n### The Evaluation Taxonomy\n\nSelect between two primary approaches based on whether ground truth exists:\n\n**Direct Scoring** — Use when objective criteria exist (factual accuracy, instruction following, toxicity). A single LLM rates one response on a defined scale. Achieves moderate-to-high reliability for well-defined criteria. Watch for score calibration drift and inconsistent scale interpretation.\n\n**Pairwise Comparison** — Use for subjective preferences (tone, style, persuasiveness). An LLM compares two responses and selects the better one. Pairwise methods often correlate better with human preference than open-ended direct scoring for subjective tasks (claim-advanced-evaluation-position-swap). Watch for position bias and length bias.\n\n### The Bias Landscape\n\nMitigate these systematic biases in every evaluation system:\n\n**Position Bias**: First-position responses get preferential treatment. Mitigate by evaluating twice with swapped positions, then apply majority vote or consistency check.\n\n**Length Bias**: Longer responses score higher regardless of quality. Mitigate by explicitly prompting to ignore length and applying length-normalized scoring.\n\n**Self-Enhancement Bias**: Models rate their own outputs higher. Mitigate by using different models for generation and evaluation.\n\n**Verbosity Bias**: Excessive detail scores higher even when unnecessary. Mitigate with criteria-specific rubrics that penalize irrelevant detail.\n\n**Authority Bias**: Confident tone scores higher regardless of accuracy. Mitigate by requiring evidence citation and adding a fact-checking layer.\n\n### Metric Selection Framework\n\nMatch metrics to the evaluation task structure:\n\n| Task Type | Primary Metrics | Secondary Metrics |\n|-----------|-----------------|-------------------|\n| Binary classification (pass/fail) | Recall, Precision, F1 | Cohen's kappa |\n| Ordinal scale (1-5 rating) | Spearman's rho, Kendall's tau | Cohen's kappa (weighted) |\n| Pairwise preference | Agreement rate, Position consistency | Confidence calibration |\n| Multi-label | Macro-F1, Micro-F1 | Per-label precision/recall |\n\nPrioritize systematic disagreement patterns over absolute agreement rates because a judge that consistently disagrees with humans on specific criteria is more problematic than one with random noise.\n\n## Evaluation Approaches\n\n### Direct Scoring Implementation\n\nBuild direct scoring with three components: clear criteria, a calibrated scale, and structured output format.\n\n**Criteria Definition Pattern**:\n```\nCriterion: [Name]\nDescription: [What this criterion measures]\nWeight: [Relative importance, 0-1]\n```\n\n**Scale Calibration** — Choose scale granularity based on rubric detail:\n- 1-3: Binary with neutral option, lowest cognitive load\n- 1-5: Standard Likert, best balance of granularity and reliability\n- 1-10: Use only with detailed per-level rubrics because calibration is harder\n\n**Prompt Structure for Direct Scoring**:\n```\nYou are an expert evaluator assessing response quality.\n\n## Task\nEvaluate the following response against each criterion.\n\n## Original Prompt\n{prompt}\n\n## Response to Evaluate\n{response}\n\n## Criteria\n{for each criterion: name, description, weight}\n\n## Instructions\nFor each criterion:\n1. Find specific evidence in the response\n2. Score according to the rubric (1-{max} scale)\n3. Justify your score with evidence\n4. Suggest one specific improvement\n\n## Output Format\nRespond with structured JSON containing scores, justifications, and summary.\n```\n\nRequire evidence before the score in scoring prompts so the judge must anchor its decision in observable output features before emitting a number.\n\n### Pairwise Comparison Implementation\n\nApply position bias mitigation in every pairwise evaluation:\n\n1. Run deterministic pre-checks first: both candidates must satisfy the same schema, source-evidence requirements, and scope constraints.\n2. First judge pass: Response A in first position, Response B in second.\n3. Second judge pass: Response B in first position, Response A in second.\n4. Consistency check: If passes disagree, return TIE with reduced confidence.\n5. Final verdict: Consistent winner with averaged confidence and explicit tie-breaker rationale.\n\n**Prompt Structure for Pairwise Comparison**:\n```\nYou are an expert evaluator comparing two AI responses.\n\n## Critical Instructions\n- Do NOT prefer responses because they are longer\n- Do NOT prefer responses based on position (first vs second)\n- Focus ONLY on quality according to the specified criteria\n- Ties are acceptable when responses are genuinely equivalent\n\n## Original Prompt\n{prompt}\n\n## Response A\n{response_a}\n\n## Response B\n{response_b}\n\n## Comparison Criteria\n{criteria list}\n\n## Instructions\n1. Analyze each response independently first\n2. Compare them on each criterion\n3. Determine overall winner with confidence level\n\n## Output Format\nJSON with per-criterion comparison, overall winner, confidence (0-1), and reasoning.\n```\n\n**Confidence Calibration** — Map confidence to position consistency:\n- Both passes agree: confidence = average of individual confidences\n- Passes disagree: confidence = 0.5, verdict = TIE\n\n### Rubric Generation\n\nGenerate rubrics to reduce evaluation variance compared to open-ended scoring. Treat exact variance reduction as workload-specific unless measured on the target eval set.\n\n**Include these rubric components**:\n1. **Level descriptions**: Clear boundaries for each score level\n2. **Characteristics**: Observable features that define each level\n3. **Examples**: Representative text for each level (optional but valuable)\n4. **Edge cases**: Guidance for ambiguous situations\n5. **Scoring guidelines**: General principles for consistent application\n\n**Set strictness calibration** for the use case:\n- **Lenient**: Lower passing bar, appropriate for encouraging iteration\n- **Balanced**: Typical production expectations\n- **Strict**: High standards for safety-critical or high-stakes evaluation\n\nAdapt rubrics to the domain — use domain-specific terminology. A code readability rubric mentions variables, functions, and comments. A medical accuracy rubric references clinical terminology and evidence standards.\n\n## Practical Guidance\n\n### Evaluation Pipeline Design\n\nBuild production evaluation systems with these layers: Criteria Loader (rubrics + weights) -> Primary Scorer (direct or pairwise) -> Bias Mitigation (position swap, etc.) -> Confidence Scoring (calibration) -> Output (scores + justifications + confidence). See [Evaluation Pipeline Diagram](./references/evaluation-pipeline.md) for the full visual layout.\n\n### Decision Framework: Direct vs. Pairwise\n\nApply this decision tree:\n\n```\nIs there an objective ground truth?\n+-- Yes -> Direct Scoring\n| Examples: factual accuracy, instruction following, format compliance\n|\n+-- No -> Is it a preference or quality judgment?\n +-- Yes -> Pairwise Comparison\n | Examples: tone, style, persuasiveness, creativity\n |\n +-- No -> Consider reference-based evaluation\n Examples: summarization (compare to source), translation (compare to reference)\n```\n\n### Scaling Evaluation\n\nFor high-volume evaluation, apply one of these strategies:\n\n1. **Panel of LLMs (PoLL)**: Use multiple models as judges and aggregate votes to reduce individual model bias. More expensive but more reliable for high-stakes decisions.\n\n2. **Hierarchical evaluation**: Use a fast cheap model for screening and an expensive model for edge cases. Requires calibration of the screening threshold.\n\n3. **Human-in-the-loop**: Automate clear cases and route low-confidence decisions to human review. Design feedback loops to improve automated evaluation over time.\n\n## Examples\n\n### Example 1: Direct Scoring for Accuracy\n\n**Input**:\n```\nPrompt: \"What causes seasons on Earth?\"\nResponse: \"Seasons are caused by Earth's tilted axis. As Earth orbits the Sun,\ndifferent hemispheres receive more direct sunlight at different times of year.\"\nCriterion: Factual Accuracy (weight: 1.0)\nScale: 1-5\n```\n\n**Output**:\n```json\n{\n \"criterion\": \"Factual Accuracy\",\n \"score\": 5,\n \"evidence\": [\n \"Correctly identifies axial tilt as primary cause\",\n \"Correctly explains differential sunlight by hemisphere\",\n \"No factual errors present\"\n ],\n \"justification\": \"Response accurately explains the cause of seasons with correct\nscientific reasoning. Both the axial tilt and its effect on sunlight distribution\nare correctly described.\",\n \"improvement\": \"Could add the specific tilt angle (23.5 degrees) for completeness.\"\n}\n```\n\n### Example 2: Pairwise Comparison with Position Swap\n\n**Input**:\n```\nPrompt: \"Explain machine learning to a beginner\"\nResponse A: [Technical explanation with jargon]\nResponse B: [Simple analogy-based explanation]\nCriteria: [\"clarity\", \"accessibility\"]\n```\n\n**First Pass (A first)**:\n```json\n{ \"winner\": \"B\", \"confidence\": 0.8 }\n```\n\n**Second Pass (B first)**:\n```json\n{ \"winner\": \"A\", \"confidence\": 0.6 }\n```\n(Note: Winner is A because B was in first position)\n\n**Mapped Second Pass**:\n```json\n{ \"winner\": \"B\", \"confidence\": 0.6 }\n```\n\n**Final Result**:\n```json\n{\n \"winner\": \"B\",\n \"confidence\": 0.7,\n \"positionConsistency\": {\n \"consistent\": true,\n \"firstPassWinner\": \"B\",\n \"secondPassWinner\": \"B\"\n }\n}\n```\n\n### Example 3: Rubric Generation\n\n**Input**:\n```\ncriterionName: \"Code Readability\"\ncriterionDescription: \"How easy the code is to understand and maintain\"\ndomain: \"software engineering\"\nscale: \"1-5\"\nstrictness: \"balanced\"\n```\n\n**Output** (abbreviated):\n```json\n{\n \"levels\": [\n {\n \"score\": 1,\n \"label\": \"Poor\",\n \"description\": \"Code is difficult to understand without significant effort\",\n \"characteristics\": [\n \"No meaningful variable or function names\",\n \"No comments or documentation\",\n \"Deeply nested or convoluted logic\"\n ]\n },\n {\n \"score\": 3,\n \"label\": \"Adequate\",\n \"description\": \"Code is understandable with some effort\",\n \"characteristics\": [\n \"Most variables have meaningful names\",\n \"Basic comments present for complex sections\",\n \"Logic is followable but could be cleaner\"\n ]\n },\n {\n \"score\": 5,\n \"label\": \"Excellent\",\n \"description\": \"Code is immediately clear and maintainable\",\n \"characteristics\": [\n \"All names are descriptive and consistent\",\n \"Comprehensive documentation\",\n \"Clean, modular structure\"\n ]\n }\n ],\n \"edgeCases\": [\n {\n \"situation\": \"Code is well-structured but uses domain-specific abbreviations\",\n \"guidance\": \"Score based on readability for domain experts, not general audience\"\n }\n ]\n}\n```\n\n## Guidelines\n\n1. **Always require evidence before scores** - Evidence-first prompts make judgments easier to audit and reduce ungrounded numeric scoring\n\n2. **Always swap positions in pairwise comparison** - Single-pass comparison is corrupted by position bias\n\n3. **Match scale granularity to rubric specificity** - Don't use 1-10 without detailed level descriptions\n\n4. **Separate objective and subjective criteria** - Use direct scoring for objective, pairwise for subjective\n\n5. **Include confidence scores** - Calibrate to position consistency and evidence strength\n\n6. **Define edge cases explicitly** - Ambiguous situations cause the most evaluation variance\n\n7. **Use domain-specific rubrics** - Generic rubrics produce generic (less useful) evaluations\n\n8. **Validate against human judgments** - Automated evaluation is only valuable if it correlates with human assessment\n\n9. **Monitor for systematic bias** - Track disagreement patterns by criterion, response type, model\n\n10. **Design for iteration** - Evaluation systems improve with feedback loops\n\n## Gotchas\n\n1. **Scoring without justification**: Scores lack grounding and are difficult to debug. Always require evidence-based justification before the score.\n\n2. **Single-pass pairwise comparison**: Position bias corrupts results when positions are not swapped. Always evaluate twice with swapped positions and check consistency.\n\n3. **Overloaded criteria**: Criteria that measure multiple things at once produce unreliable scores. Enforce one criterion = one measurable aspect.\n\n4. **Missing edge case guidance**: Evaluators handle ambiguous cases inconsistently without explicit instructions. Include edge cases in rubrics with clear resolution rules.\n\n5. **Ignoring confidence calibration**: High-confidence wrong judgments are worse than low-confidence ones. Calibrate confidence to position consistency and evidence strength.\n\n6. **Rubric drift**: Rubrics become miscalibrated as quality standards evolve or model capabilities improve. Schedule periodic rubric reviews and re-anchor score levels against fresh human-annotated examples.\n\n7. **Evaluation prompt sensitivity**: Minor wording changes in evaluation prompts can cause material score swings. Version-control evaluation prompts and run regression tests before deploying prompt changes.\n\n8. **Uncontrolled length bias**: Longer responses systematically score higher even when conciseness is preferred. Add explicit length-neutrality instructions to evaluation prompts and validate with length-controlled test pairs.\n\n## Integration\n\nThis skill owns judge design and bias mitigation. Adjacent skills own broader quality gates and infrastructure:\n\n- `evaluation`: general deterministic checks, regression suites, quality gates, and production monitoring.\n- `context-fundamentals`: context structure for judge prompts.\n- `tool-design`: schemas and error handling for evaluation tools.\n- `context-optimization`: token and latency efficiency for high-volume evals.\n- `harness-engineering`: locked evaluator surfaces and governance for autonomous loops.\n\n## References\n\nInternal reference:\n- [LLM-as-Judge Implementation Patterns](./references/implementation-patterns.md) - Read when: building an evaluation pipeline from scratch or integrating LLM judges into CI/CD\n- [Bias Mitigation Techniques](./references/bias-mitigation.md) - Read when: evaluation results show inconsistent or suspicious scoring patterns\n- [Metric Selection Guide](./references/metrics-guide.md) - Read when: choosing statistical metrics to validate evaluation reliability\n- [Evaluation Pipeline Diagram](./references/evaluation-pipeline.md) - Read when: designing the architecture of a multi-stage evaluation system\n\nExternal research:\n- [Eugene Yan: Evaluating the Effectiveness of LLM-Evaluators](https://eugeneyan.com/writing/llm-evaluators/) - Read when: surveying the state of the art in LLM evaluation\n- [Judging LLM-as-a-Judge (Zheng et al., 2023)](https://arxiv.org/abs/2306.05685) - Read when: understanding position bias and MT-Bench methodology\n- [G-Eval: NLG Evaluation using GPT-4 (Liu et al., 2023)](https://arxiv.org/abs/2303.16634) - Read when: implementing chain-of-thought evaluation scoring\n- [Large Language Models are not Fair Evaluators (Wang et al., 2023)](https://arxiv.org/abs/2305.17926) - Read when: diagnosing systematic bias in evaluation outputs\n\nRelated skills in this collection:\n- evaluation - Foundational evaluation concepts\n- context-fundamentals - Context structure for evaluation prompts\n- tool-design - Building evaluation tools\n\n---\n\n## Skill Metadata\n\n**Created**: 2025-12-24\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.1.0\n\n\n# BDI Ontology Core Patterns\n\nCore ontology design patterns for Belief-Desire-Intention mental state modeling.\n\n## Class Hierarchy\n\n### Mental Entities (Endurants)\n\n```\nbdi:MentalEntity\n├── bdi:Belief # Informational dimension\n├── bdi:Desire # Motivational dimension \n├── bdi:Intention # Deliberative dimension\n├── bdi:Goal # Description of desired end state\n└── bdi:Plan # Structured action sequence\n```\n\n### Mental Processes (Perdurants)\n\n```\nbdi:MentalProcess\n├── bdi:BeliefProcess # Forms/updates beliefs from perception\n├── bdi:DesireProcess # Generates desires from beliefs\n├── bdi:IntentionProcess # Commits to desires as intentions\n├── bdi:Planning # Transforms intentions into plans\n└── bdi:PlanExecution # Executes plan actions\n```\n\n### Supporting Entities\n\n```\nbdi:WorldState # Configuration of environment\nbdi:Justification # Evidential basis for mental states\nbdi:Task # Atomic unit of planned action\nbdi:Action # Execution of a task\nbdi:TimeInterval # Temporal validity bounds\nbdi:TimeInstant # Point in time reference\n```\n\n## Object Properties\n\n### Motivational Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `motivates` | Belief | Desire | Belief provides reason for desire |\n| `isMotivatedBy` | Desire | Belief | Inverse of motivates |\n| `fulfils` | Intention | Desire | Intention commits to achieving desire |\n| `isFulfilledBy` | Desire | Intention | Inverse of fulfils |\n| `isSupportedBy` | Intention | Belief | Beliefs supporting intention viability |\n\n### Generative Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `generates` | MentalProcess | MentalEntity | Process creates mental state |\n| `isGeneratedBy` | MentalEntity | MentalProcess | Inverse of generates |\n| `modifies` | MentalProcess | MentalEntity | Process updates existing state |\n| `suppresses` | MentalProcess | MentalEntity | Process deactivates state |\n| `isTriggeredBy` | MentalProcess | MentalEntity | State initiates process |\n\n### Referential Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `refersTo` | MentalEntity | WorldState | Mental state about world |\n| `perceives` | Agent | WorldState | Agent observes world |\n| `bringsAbout` | Action | WorldState | Action causes world change |\n| `reasonsUpon` | MentalProcess | MentalEntity | Input to reasoning |\n\n### Structural Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `hasPart` | MentalEntity | MentalEntity | Meronymic composition |\n| `specifies` | Intention | Plan | Intention defines plan |\n| `addresses` | Plan | Goal | Plan achieves goal |\n| `hasComponent` | Plan | Task | Plan contains tasks |\n| `precedes` | Task | Task | Task ordering |\n\n### Temporal Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `atTime` | Entity | TimeInstant | Point occurrence |\n| `hasValidity` | MentalEntity | TimeInterval | Persistence bounds |\n| `hasStartTime` | TimeInterval | TimeInstant | Interval start |\n| `hasEndTime` | TimeInterval | TimeInstant | Interval end |\n\n### Justification Relations\n\n| Property | Domain | Range | Description |\n|----------|--------|-------|-------------|\n| `isJustifiedBy` | MentalEntity | Justification | Evidential support |\n| `justifies` | Justification | MentalEntity | Inverse relation |\n\n## Ontological Restrictions\n\n### Belief Restrictions\n\n```turtle\nbdi:Belief rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:refersTo ;\n owl:someValuesFrom bdi:WorldState\n] .\n\nbdi:Belief rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:hasValidity ;\n owl:maxCardinality 1\n] .\n```\n\n### Desire Restrictions\n\n```turtle\nbdi:Desire rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:isMotivatedBy ;\n owl:someValuesFrom bdi:Belief\n] .\n```\n\n### Intention Restrictions\n\n```turtle\nbdi:Intention rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:fulfils ;\n owl:cardinality 1\n] .\n\nbdi:Intention rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:isSupportedBy ;\n owl:someValuesFrom bdi:Belief\n] .\n```\n\n### Mental Process Restrictions\n\n```turtle\nbdi:BeliefProcess rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:generates ;\n owl:allValuesFrom bdi:Belief\n] .\n\nbdi:DesireProcess rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:generates ;\n owl:allValuesFrom bdi:Desire\n] .\n\nbdi:IntentionProcess rdfs:subClassOf [\n a owl:Restriction ;\n owl:onProperty bdi:generates ;\n owl:allValuesFrom bdi:Intention\n] .\n```\n\n## DOLCE Alignment\n\nThe BDI ontology aligns with DOLCE Ultra Lite (DUL) foundational ontology:\n\n| BDI Class | DUL Superclass | Rationale |\n|-----------|----------------|-----------|\n| `Agent` | `dul:Agent` | Intentional entity capable of action |\n| `Belief` | `dul:InformationObject` | Information-bearing entity |\n| `Desire` | `dul:Description` | Describes desired state |\n| `Intention` | `dul:Description` | Describes committed course |\n| `Goal` | `dul:Goal` | Desired end state description |\n| `Plan` | `dul:Plan` | Organized action sequence |\n| `WorldState` | `dul:Situation` | Configuration of entities |\n| `MentalProcess` | `dul:Event` | Temporally extended occurrence |\n| `Task` | `dul:Task` | Unit of planned work |\n| `Action` | `dul:Action` | Performed task instance |\n\n## Reused Ontology Design Patterns\n\n### EventCore Pattern\nUsed for mental processes with temporal aspects and participant roles.\n\n### Situation Pattern \nUsed for world state configurations that mental states reference.\n\n### TimeIndexedSituation Pattern\nUsed for associating mental states with validity intervals.\n\n### BasicPlan Pattern\nUsed for goal-plan-task structures linking intentions to actions.\n\n### Provenance Pattern\nUsed for justification tracking and evidential chains.\n\n## Namespace Declarations\n\n```turtle\n@prefix bdi: .\n@prefix dul: .\n@prefix owl: .\n@prefix rdf: .\n@prefix rdfs: .\n@prefix xsd: .\n```\n\n\n# BDI Framework Integration Patterns\n\nIntegration patterns for connecting BDI ontology with executable agent frameworks.\n\n## SEMAS Rule Translation\n\nMap BDI ontology constructs to SEMAS production rules.\n\n### Ontology-to-Rule Mapping\n\n| BDI Construct | SEMAS Element | Example |\n|---------------|---------------|---------|\n| Belief | HEAD fact | `belief(agent_a, store_open)` |\n| Supporting beliefs | CONDITIONALS | `[CONDITIONALS: time(weekday)]` |\n| Desire generation | TAIL action | `generate_desire(agent, goal)` |\n| Intention commitment | TAIL action | `commit_intention(agent, goal)` |\n| Plan specification | TAIL action | `create_plan(agent, plan_id)` |\n\n### Rule Templates\n\n**Belief triggers desire formation:**\n```prolog\n[HEAD: belief(Agent, Fact)] / \n[CONDITIONALS: context_condition(Agent, Context)] » \n[TAIL: generate_desire(Agent, DesiredState)].\n```\n\n**Desire triggers intention commitment:**\n```prolog\n[HEAD: desire(Agent, Goal)] / \n[CONDITIONALS: belief(Agent, SupportingFact1), \n belief(Agent, SupportingFact2)] » \n[TAIL: commit_intention(Agent, Goal)].\n```\n\n**Intention triggers planning:**\n```prolog\n[HEAD: intention(Agent, Goal)] / \n[CONDITIONALS: goal(GoalSpec)] » \n[TAIL: create_plan(Agent, PlanId)].\n```\n\n**Plan triggers execution:**\n```prolog\n[HEAD: plan(Agent, PlanId)] / \n[CONDITIONALS: ready_to_execute(Agent)] » \n[TAIL: execute_plan(Agent, PlanId)].\n```\n\n### Complete SEMAS Example\n\n```prolog\n% ============================================================\n% GROCERY SHOPPING SCENARIO\n% ============================================================\n\n% Phase 1: Belief formation from world state\n[HEAD: perceive(agent_a, store_open)] / \n[CONDITIONALS: time(weekday_afternoon)] » \n[TAIL: add_belief(agent_a, store_open)].\n\n% Phase 2: Desire generation from belief\n[HEAD: belief(agent_a, store_open)] / \n[CONDITIONALS: belief(agent_a, needs_groceries)] » \n[TAIL: generate_desire(agent_a, buy_groceries)].\n\n% Phase 3: Intention commitment from desire\n[HEAD: desire(agent_a, buy_groceries)] / \n[CONDITIONALS: belief(agent_a, has_shopping_list), \n belief(agent_a, store_open),\n belief(agent_a, has_transportation)] » \n[TAIL: commit_intention(agent_a, buy_groceries)].\n\n% Phase 4: Plan creation from intention\n[HEAD: intention(agent_a, buy_groceries)] / \n[CONDITIONALS: goal(complete_shopping)] » \n[TAIL: create_plan(agent_a, shopping_plan)].\n\n% Phase 5: Plan execution\n[HEAD: plan(agent_a, shopping_plan)] / \n[CONDITIONALS: preconditions_met(shopping_plan)] » \n[TAIL: execute_task(agent_a, drive_to_store),\n execute_task(agent_a, select_items),\n execute_task(agent_a, checkout),\n execute_task(agent_a, return_home)].\n\n% Phase 6: World state update\n[HEAD: task_complete(agent_a, checkout)] / \n[CONDITIONALS: items_purchased(agent_a)] » \n[TAIL: update_world_state(has_groceries),\n remove_desire(agent_a, buy_groceries),\n remove_intention(agent_a, buy_groceries)].\n```\n\n### Python Translation Layer\n\n```python\nfrom rdflib import Graph, Namespace, RDF\n\nBDI = Namespace(\"https://w3id.org/fossr/ontology/bdi/\")\n\ndef ontology_to_semas_rules(bdi_graph: Graph) -> list[str]:\n \"\"\"\n Translate BDI ontology instances to SEMAS production rules.\n \"\"\"\n rules = []\n \n # Extract belief-desire-intention chains\n for intention in bdi_graph.subjects(RDF.type, BDI.Intention):\n # Get supporting beliefs\n supporting_beliefs = list(bdi_graph.objects(intention, BDI.isSupportedBy))\n \n # Get fulfilled desire\n fulfilled_desires = list(bdi_graph.objects(intention, BDI.fulfils))\n \n # Get specified plan\n specified_plans = list(bdi_graph.objects(intention, BDI.specifies))\n \n if fulfilled_desires and supporting_beliefs:\n desire = fulfilled_desires[0]\n beliefs_str = \", \".join([format_belief(b, bdi_graph) for b in supporting_beliefs])\n \n rule = (\n f\"[HEAD: {format_desire(desire, bdi_graph)}] / \"\n f\"[CONDITIONALS: {beliefs_str}] » \"\n f\"[TAIL: commit_intention({format_intention(intention, bdi_graph)})]\"\n )\n rules.append(rule)\n \n if specified_plans:\n plan = specified_plans[0]\n rule = (\n f\"[HEAD: {format_intention(intention, bdi_graph)}] / \"\n f\"[CONDITIONALS: ready_to_plan] » \"\n f\"[TAIL: create_plan({format_plan(plan, bdi_graph)})]\"\n )\n rules.append(rule)\n \n return rules\n\ndef format_belief(belief_uri, graph):\n label = graph.value(belief_uri, RDFS.label)\n return f\"belief({label or belief_uri.split('/')[-1]})\"\n\ndef format_desire(desire_uri, graph):\n label = graph.value(desire_uri, RDFS.label)\n return f\"desire({label or desire_uri.split('/')[-1]})\"\n\ndef format_intention(intention_uri, graph):\n label = graph.value(intention_uri, RDFS.label)\n return f\"intention({label or intention_uri.split('/')[-1]})\"\n\ndef format_plan(plan_uri, graph):\n label = graph.value(plan_uri, RDFS.label)\n return f\"plan({label or plan_uri.split('/')[-1]})\"\n```\n\n## Logic Augmented Generation (LAG)\n\nAugment LLM outputs with BDI ontological constraints.\n\n### LAG Pipeline Architecture\n\n```\n┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐\n│ User Query │────▶│ Ontology │────▶│ Augmented │\n│ │ │ Injection │ │ Prompt │\n└─────────────────┘ └─────────────────┘ └─────────────────┘\n │\n ▼\n┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐\n│ Validated │◀────│ Ontology │◀────│ LLM Response │\n│ RDF Triples │ │ Validation │ │ (Triples) │\n└─────────────────┘ └─────────────────┘ └─────────────────┘\n```\n\n### LAG Implementation\n\n```python\nfrom rdflib import Graph, Namespace\nfrom rdflib.plugins.parsers.notation3 import BadSyntax\n\nBDI = Namespace(\"https://w3id.org/fossr/ontology/bdi/\")\n\nclass BDILogicAugmentedGenerator:\n def __init__(self, ontology_path: str, llm_client):\n self.ontology = Graph()\n self.ontology.parse(ontology_path, format='turtle')\n self.llm = llm_client\n \n def generate_mental_states(self, context: str) -> Graph:\n \"\"\"\n Generate BDI mental states from context using LAG.\n \"\"\"\n # Phase 1: Inject ontology into prompt\n ontology_turtle = self.ontology.serialize(format='turtle')\n augmented_prompt = self._build_augmented_prompt(context, ontology_turtle)\n \n # Phase 2: Generate with LLM\n response = self.llm.generate(augmented_prompt)\n \n # Phase 3: Extract and validate triples\n triples = self._extract_triples(response)\n validated = self._validate_against_ontology(triples)\n \n if not validated['is_consistent']:\n # Retry with feedback\n return self._retry_with_feedback(context, validated['errors'])\n \n return validated['graph']\n \n def _build_augmented_prompt(self, context: str, ontology: str) -> str:\n return f\"\"\"\nYou are a BDI mental state modeler. Given the following context, generate \nRDF triples representing the agent's beliefs, desires, and intentions.\n\n## BDI Ontology (use these classes and properties):\n{ontology}\n\n## Context to Model:\n{context}\n\n## Instructions:\n1. Identify world states from the context\n2. Generate beliefs that refer to those world states\n3. Generate desires motivated by those beliefs\n4. Generate intentions that fulfill desires and are supported by beliefs\n5. Include justifications for each mental state\n6. Include temporal validity intervals\n\nOutput valid Turtle RDF triples only.\n\"\"\"\n \n def _extract_triples(self, response: str) -> str:\n \"\"\"Extract Turtle content from LLM response.\"\"\"\n # Find turtle block in response\n if \"```turtle\" in response:\n start = response.find(\"```turtle\") + 9\n end = response.find(\"```\", start)\n return response[start:end].strip()\n return response\n \n def _validate_against_ontology(self, triples: str) -> dict:\n \"\"\"Validate generated triples against BDI ontology.\"\"\"\n result = {'is_consistent': True, 'errors': [], 'graph': None}\n \n try:\n generated = Graph()\n generated.parse(data=triples, format='turtle')\n result['graph'] = generated\n \n # Validate constraints\n errors = []\n \n # Check: Every intention must fulfill a desire\n for intention in generated.subjects(RDF.type, BDI.Intention):\n if not list(generated.objects(intention, BDI.fulfils)):\n errors.append(f\"Intention {intention} does not fulfill any desire\")\n \n # Check: Every belief should reference a world state\n for belief in generated.subjects(RDF.type, BDI.Belief):\n if not list(generated.objects(belief, BDI.refersTo)):\n errors.append(f\"Belief {belief} does not reference a world state\")\n \n # Check: Desires should be motivated by beliefs\n for desire in generated.subjects(RDF.type, BDI.Desire):\n if not list(generated.objects(desire, BDI.isMotivatedBy)):\n errors.append(f\"Desire {desire} has no motivating belief\")\n \n if errors:\n result['is_consistent'] = False\n result['errors'] = errors\n \n except BadSyntax as e:\n result['is_consistent'] = False\n result['errors'] = [f\"Invalid Turtle syntax: {e}\"]\n \n return result\n \n def _retry_with_feedback(self, context: str, errors: list) -> Graph:\n \"\"\"Retry generation with error feedback.\"\"\"\n feedback_prompt = f\"\"\"\nPrevious generation had errors:\n{chr(10).join(errors)}\n\nPlease regenerate the mental states fixing these issues.\n\nContext: {context}\n\"\"\"\n response = self.llm.generate(feedback_prompt)\n triples = self._extract_triples(response)\n result = self._validate_against_ontology(triples)\n \n if result['is_consistent']:\n return result['graph']\n else:\n raise ValueError(f\"Failed to generate valid mental states: {result['errors']}\")\n```\n\n### Inconsistency Detection Example\n\n```python\ndef detect_location_inconsistency(graph: Graph) -> list[str]:\n \"\"\"\n Detect inconsistencies where agent cannot be in two places.\n \"\"\"\n inconsistencies = []\n \n # Query for location beliefs\n query = \"\"\"\n PREFIX bdi: \n \n SELECT ?agent ?belief1 ?belief2 ?loc1 ?loc2 WHERE {\n ?agent bdi:hasBelief ?belief1 , ?belief2 .\n ?belief1 bdi:refersTo ?ws1 .\n ?belief2 bdi:refersTo ?ws2 .\n ?ws1 bdi:hasLocation ?loc1 .\n ?ws2 bdi:hasLocation ?loc2 .\n FILTER(?belief1 != ?belief2 && ?loc1 != ?loc2)\n \n # Check temporal overlap\n ?belief1 bdi:hasValidity ?interval1 .\n ?belief2 bdi:hasValidity ?interval2 .\n ?interval1 bdi:hasStartTime ?start1 ; bdi:hasEndTime ?end1 .\n ?interval2 bdi:hasStartTime ?start2 ; bdi:hasEndTime ?end2 .\n FILTER(?start1 < ?end2 && ?start2 < ?end1)\n }\n \"\"\"\n \n for row in graph.query(query):\n inconsistencies.append(\n f\"Agent {row.agent} has conflicting location beliefs: \"\n f\"{row.loc1} and {row.loc2} at overlapping times\"\n )\n \n return inconsistencies\n```\n\n## JADE/JADEX Integration\n\nMap BDI ontology to JADE/JADEX agent platform structures.\n\n### JADE Agent Structure\n\n```java\npublic class BDIAgent extends Agent {\n // Mental state storage (maps to ontology individuals)\n private Set beliefs = new HashSet<>();\n private Set desires = new HashSet<>();\n private Set intentions = new HashSet<>();\n \n // Ontology-backed mental state management\n private Graph mentalStateGraph;\n \n public void addBelief(Belief belief) {\n beliefs.add(belief);\n \n // Add to RDF graph\n Resource beliefResource = mentalStateGraph.createResource(belief.getUri());\n beliefResource.addProperty(RDF.type, BDI.Belief);\n beliefResource.addProperty(BDI.refersTo, belief.getWorldState().getUri());\n beliefResource.addProperty(BDI.hasValidity, createInterval(belief.getValidity()));\n \n // Trigger desire formation\n triggerDesireProcess(belief);\n }\n \n public void commitIntention(Intention intention) {\n intentions.add(intention);\n \n Resource intentionResource = mentalStateGraph.createResource(intention.getUri());\n intentionResource.addProperty(RDF.type, BDI.Intention);\n intentionResource.addProperty(BDI.fulfils, intention.getDesire().getUri());\n \n for (Belief support : intention.getSupportingBeliefs()) {\n intentionResource.addProperty(BDI.isSupportedBy, support.getUri());\n }\n \n // Trigger planning\n triggerPlanning(intention);\n }\n \n // Export mental states as RDF\n public String exportMentalStates() {\n return mentalStateGraph.serialize(Format.TURTLE);\n }\n \n // Import mental states from RDF\n public void importMentalStates(String turtle) {\n Graph imported = new Graph();\n imported.parse(turtle, Format.TURTLE);\n \n // Reconstruct Java objects from RDF\n for (Resource belief : imported.listSubjectsWithProperty(RDF.type, BDI.Belief)) {\n Belief b = reconstructBelief(belief);\n beliefs.add(b);\n }\n // ... similar for desires and intentions\n }\n}\n```\n\n### JADEX Goal Mapping\n\n```java\n// Map BDI ontology goals to JADEX goals\n@Goal\npublic class OntologyBackedGoal {\n @GoalParameter\n protected String goalUri;\n \n @GoalParameter\n protected Graph ontologyGraph;\n \n public OntologyBackedGoal(Resource goalResource, Graph graph) {\n this.goalUri = goalResource.getURI();\n this.ontologyGraph = graph;\n }\n \n @GoalTargetCondition\n public boolean isAchieved() {\n // Query ontology for goal achievement\n String query = \"\"\"\n PREFIX bdi: \n ASK {\n ?execution bdi:addresses <%s> ;\n bdi:bringsAbout ?worldState .\n }\n \"\"\".formatted(goalUri);\n \n return ontologyGraph.ask(query);\n }\n \n @GoalDropCondition\n public boolean shouldDrop() {\n // Check if supporting beliefs are invalidated\n String query = \"\"\"\n PREFIX bdi: \n ASK {\n ?intention bdi:specifies ?plan .\n ?plan bdi:addresses <%s> .\n ?intention bdi:isSupportedBy ?belief .\n ?belief bdi:hasValidity ?interval .\n ?interval bdi:hasEndTime ?end .\n FILTER(?end < NOW())\n }\n \"\"\".formatted(goalUri);\n \n return ontologyGraph.ask(query);\n }\n}\n```\n\n## RDF Triple Store Integration\n\n### Triple Store Configuration\n\n```python\nfrom rdflib import Graph\nfrom rdflib.plugins.stores.sparqlstore import SPARQLUpdateStore\n\nclass BDIMentalStateStore:\n def __init__(self, endpoint: str):\n self.store = SPARQLUpdateStore()\n self.store.open((endpoint + \"/query\", endpoint + \"/update\"))\n self.graph = Graph(store=self.store, identifier=\"http://example.org/bdi\")\n \n def add_belief(self, agent_uri: str, belief_data: dict):\n \"\"\"Add belief to triple store.\"\"\"\n belief_uri = f\"{agent_uri}/belief/{belief_data['id']}\"\n \n self.graph.add((URIRef(belief_uri), RDF.type, BDI.Belief))\n self.graph.add((URIRef(belief_uri), RDFS.label, Literal(belief_data['label'])))\n self.graph.add((URIRef(belief_uri), BDI.refersTo, URIRef(belief_data['world_state'])))\n self.graph.add((URIRef(agent_uri), BDI.hasMentalState, URIRef(belief_uri)))\n \n # Add temporal validity\n interval_uri = f\"{belief_uri}/validity\"\n self.graph.add((URIRef(belief_uri), BDI.hasValidity, URIRef(interval_uri)))\n self.graph.add((URIRef(interval_uri), BDI.hasStartTime, \n Literal(belief_data['start_time'], datatype=XSD.dateTime)))\n self.graph.add((URIRef(interval_uri), BDI.hasEndTime,\n Literal(belief_data['end_time'], datatype=XSD.dateTime)))\n \n def get_active_beliefs(self, agent_uri: str, at_time: datetime) -> list:\n \"\"\"Query beliefs active at specific time.\"\"\"\n query = \"\"\"\n PREFIX bdi: \n PREFIX xsd: \n \n SELECT ?belief ?label WHERE {\n <%s> bdi:hasMentalState ?belief .\n ?belief a bdi:Belief ;\n rdfs:label ?label ;\n bdi:hasValidity ?interval .\n ?interval bdi:hasStartTime ?start ;\n bdi:hasEndTime ?end .\n FILTER(?start <= \"%s\"^^xsd:dateTime && ?end >= \"%s\"^^xsd:dateTime)\n }\n \"\"\" % (agent_uri, at_time.isoformat(), at_time.isoformat())\n \n return list(self.graph.query(query))\n \n def get_cognitive_chain(self, intention_uri: str) -> dict:\n \"\"\"Trace complete cognitive chain for an intention.\"\"\"\n query = \"\"\"\n PREFIX bdi: \n \n SELECT ?intention ?desire ?belief ?worldState ?plan WHERE {\n <%s> a bdi:Intention ;\n bdi:fulfils ?desire ;\n bdi:isSupportedBy ?belief .\n OPTIONAL { <%s> bdi:specifies ?plan }\n ?desire bdi:isMotivatedBy ?belief .\n ?belief bdi:refersTo ?worldState .\n }\n \"\"\" % (intention_uri, intention_uri)\n \n results = list(self.graph.query(query))\n if results:\n row = results[0]\n return {\n 'intention': str(row.intention),\n 'desire': str(row.desire),\n 'belief': str(row.belief),\n 'world_state': str(row.worldState),\n 'plan': str(row.plan) if row.plan else None\n }\n return None\n```\n\n## FIPA ACL Integration\n\nMap BDI mental states to FIPA Agent Communication Language.\n\n```python\nfrom fipa_acl import ACLMessage, Performative\n\nclass BDICommunicator:\n def __init__(self, agent_id: str, mental_state_store: BDIMentalStateStore):\n self.agent_id = agent_id\n self.store = mental_state_store\n \n def share_belief(self, belief_uri: str, receiver: str) -> ACLMessage:\n \"\"\"Create INFORM message to share belief.\"\"\"\n belief_triples = self.store.get_belief_as_turtle(belief_uri)\n \n message = ACLMessage()\n message.performative = Performative.INFORM\n message.sender = self.agent_id\n message.receiver = receiver\n message.content = belief_triples\n message.ontology = \"https://w3id.org/fossr/ontology/bdi/\"\n message.language = \"turtle\"\n \n return message\n \n def request_belief_confirmation(self, belief_uri: str, receiver: str) -> ACLMessage:\n \"\"\"Create QUERY-IF message to confirm shared belief.\"\"\"\n message = ACLMessage()\n message.performative = Performative.QUERY_IF\n message.sender = self.agent_id\n message.receiver = receiver\n message.content = f\"ASK {{ <{belief_uri}> a bdi:Belief }}\"\n message.language = \"sparql\"\n \n return message\n \n def propose_intention(self, intention_uri: str, receiver: str) -> ACLMessage:\n \"\"\"Create PROPOSE message for coordinated intention.\"\"\"\n intention_triples = self.store.get_intention_as_turtle(intention_uri)\n \n message = ACLMessage()\n message.performative = Performative.PROPOSE\n message.sender = self.agent_id\n message.receiver = receiver\n message.content = intention_triples\n message.ontology = \"https://w3id.org/fossr/ontology/bdi/\"\n \n return message\n```\n\n\n# BDI RDF Examples\n\nComplete RDF/Turtle examples for BDI mental state modeling.\n\n## Complete Cognitive Workflow\n\n```turtle\n@prefix bdi: .\n@prefix ex: .\n@prefix xsd: .\n@prefix rdfs: .\n\n# ============================================================\n# PHASE 1: World State Perception\n# ============================================================\n\nex:WorldState_traffic a bdi:WorldState ;\n rdfs:comment \"Heavy traffic on Route 101\" ;\n bdi:atTime \"2026-01-04T08:30:00\"^^xsd:dateTime ;\n bdi:isPerceivedBy ex:Agent_commuter ;\n bdi:triggers ex:BeliefProcess_assess_traffic .\n\n# ============================================================\n# PHASE 2: Belief Formation\n# ============================================================\n\nex:BeliefProcess_assess_traffic a bdi:BeliefProcess ;\n bdi:generates ex:Belief_traffic_delay ;\n bdi:reasonsUpon ex:WorldState_traffic ;\n bdi:isProcessedBy ex:Agent_commuter ;\n bdi:atTime \"2026-01-04T08:31:00\"^^xsd:dateTime .\n\nex:Belief_traffic_delay a bdi:Belief ;\n rdfs:label \"Traffic will cause 30-minute delay\" ;\n bdi:refersTo ex:WorldState_traffic ;\n bdi:hasValidity ex:TimeInterval_morning_commute ;\n bdi:hasPart ex:Belief_route_congested , ex:Belief_delay_duration ;\n bdi:isJustifiedBy ex:Justification_traffic_report ;\n bdi:motivates ex:Desire_arrive_on_time .\n\nex:Belief_route_congested a bdi:Belief ;\n rdfs:comment \"Route 101 is congested\" .\n\nex:Belief_delay_duration a bdi:Belief ;\n rdfs:comment \"Delay estimated at 30 minutes\" .\n\nex:Justification_traffic_report a bdi:Justification ;\n rdfs:label \"Real-time traffic data from navigation system\" ;\n bdi:justifies ex:Belief_traffic_delay .\n\n# ============================================================\n# PHASE 3: Desire Formation\n# ============================================================\n\nex:DesireProcess_plan_arrival a bdi:DesireProcess ;\n bdi:generates ex:Desire_arrive_on_time ;\n bdi:reasonsUpon ex:Belief_traffic_delay ;\n bdi:isProcessedBy ex:Agent_commuter .\n\nex:Desire_arrive_on_time a bdi:Desire ;\n rdfs:label \"I desire to arrive at work on time\" ;\n bdi:isMotivatedBy ex:Belief_traffic_delay ;\n bdi:refersTo ex:WorldState_on_time_arrival .\n\n# ============================================================\n# PHASE 4: Intention Commitment\n# ============================================================\n\nex:IntentionProcess_commit_route a bdi:IntentionProcess ;\n bdi:generates ex:Intention_take_alternate_route ;\n bdi:reasonsUpon ex:Desire_arrive_on_time ;\n bdi:isProcessedBy ex:Agent_commuter .\n\nex:Intention_take_alternate_route a bdi:Intention ;\n rdfs:label \"I will take alternate route via Highway 280\" ;\n bdi:fulfils ex:Desire_arrive_on_time ;\n bdi:isSupportedBy ex:Belief_traffic_delay ;\n bdi:specifies ex:Plan_alternate_commute ;\n bdi:isJustifiedBy ex:Justification_time_optimization .\n\nex:Justification_time_optimization a bdi:Justification ;\n rdfs:label \"Alternate route saves 20 minutes based on current conditions\" ;\n bdi:justifies ex:Intention_take_alternate_route .\n\n# ============================================================\n# PHASE 5: Planning\n# ============================================================\n\nex:Planning_route_selection a bdi:Planning ;\n bdi:reasonsUpon ex:Intention_take_alternate_route ;\n bdi:defines ex:Plan_alternate_commute ;\n bdi:atTime ex:TimeInterval_planning_phase .\n\nex:Plan_alternate_commute a bdi:Plan ;\n rdfs:label \"Alternate commute via Highway 280\" ;\n bdi:addresses ex:Goal_arrive_by_9am ;\n bdi:beginsWith ex:Task_exit_Route101 ;\n bdi:endsWith ex:Task_arrive_parking ;\n bdi:hasComponent ex:Task_exit_Route101 , ex:Task_merge_280 , \n ex:Task_navigate_280 , ex:Task_arrive_parking .\n\nex:Task_exit_Route101 a bdi:Task ;\n rdfs:label \"Exit Route 101 at Whipple Ave\" ;\n bdi:precedes ex:Task_merge_280 .\n\nex:Task_merge_280 a bdi:Task ;\n rdfs:label \"Merge onto Highway 280 North\" ;\n bdi:precedes ex:Task_navigate_280 .\n\nex:Task_navigate_280 a bdi:Task ;\n rdfs:label \"Continue on Highway 280 for 8 miles\" ;\n bdi:precedes ex:Task_arrive_parking .\n\nex:Task_arrive_parking a bdi:Task ;\n rdfs:label \"Arrive at office parking garage\" .\n\nex:Goal_arrive_by_9am a bdi:Goal ;\n rdfs:label \"Arrive at work by 9:00 AM\" .\n\n# ============================================================\n# PHASE 6: Plan Execution\n# ============================================================\n\nex:PlanExecution_commute a bdi:PlanExecution ;\n bdi:satisfies ex:Plan_alternate_commute ;\n bdi:addresses ex:Goal_arrive_by_9am ;\n bdi:isExecutedBy ex:Agent_commuter ;\n bdi:hasComponent ex:Action_exit , ex:Action_merge , \n ex:Action_drive_280 , ex:Action_park ;\n bdi:atTime ex:TimeInterval_execution ;\n bdi:bringsAbout ex:WorldState_arrived_on_time .\n\nex:Action_exit a bdi:Action ;\n bdi:isExecutionOf ex:Task_exit_Route101 ;\n bdi:isPerformedBy ex:Agent_commuter ;\n bdi:atTime \"2026-01-04T08:35:00\"^^xsd:dateTime .\n\nex:Action_merge a bdi:Action ;\n bdi:isExecutionOf ex:Task_merge_280 ;\n bdi:isPerformedBy ex:Agent_commuter ;\n bdi:atTime \"2026-01-04T08:37:00\"^^xsd:dateTime .\n\nex:Action_drive_280 a bdi:Action ;\n bdi:isExecutionOf ex:Task_navigate_280 ;\n bdi:isPerformedBy ex:Agent_commuter ;\n bdi:atTime \"2026-01-04T08:40:00\"^^xsd:dateTime .\n\nex:Action_park a bdi:Action ;\n bdi:isExecutionOf ex:Task_arrive_parking ;\n bdi:isPerformedBy ex:Agent_commuter ;\n bdi:bringsAbout ex:WorldState_arrived_on_time ;\n bdi:atTime \"2026-01-04T08:52:00\"^^xsd:dateTime .\n\n# ============================================================\n# PHASE 7: Resulting World State\n# ============================================================\n\nex:WorldState_arrived_on_time a bdi:WorldState ;\n rdfs:comment \"Agent arrived at work at 8:52 AM\" ;\n bdi:atTime \"2026-01-04T08:52:00\"^^xsd:dateTime .\n\n# ============================================================\n# TEMPORAL INTERVALS\n# ============================================================\n\nex:TimeInterval_morning_commute a bdi:TimeInterval ;\n bdi:hasStartTime \"2026-01-04T08:30:00\"^^xsd:dateTime ;\n bdi:hasEndTime \"2026-01-04T09:00:00\"^^xsd:dateTime .\n\nex:TimeInterval_planning_phase a bdi:TimeInterval ;\n bdi:hasStartTime \"2026-01-04T08:31:00\"^^xsd:dateTime ;\n bdi:hasEndTime \"2026-01-04T08:34:00\"^^xsd:dateTime .\n\nex:TimeInterval_execution a bdi:TimeInterval ;\n bdi:hasStartTime \"2026-01-04T08:35:00\"^^xsd:dateTime ;\n bdi:hasEndTime \"2026-01-04T08:52:00\"^^xsd:dateTime .\n```\n\n## Multi-Agent Coordination Example\n\n```turtle\n@prefix bdi: .\n@prefix ex: .\n@prefix fipa: .\n\n# Shared belief about project deadline\nex:Agent_developer a bdi:Agent ;\n bdi:hasMentalState ex:Belief_deadline_friday .\n\nex:Agent_manager a bdi:Agent ;\n bdi:hasMentalState ex:Belief_deadline_friday .\n\nex:Belief_deadline_friday a bdi:Belief ;\n rdfs:label \"Project deadline is Friday 5 PM\" ;\n bdi:refersTo ex:WorldState_deadline ;\n bdi:hasValidity ex:TimeInterval_project_week .\n\nex:WorldState_deadline a bdi:WorldState ;\n rdfs:comment \"Project XYZ must be delivered by 2026-01-10T17:00:00\" .\n\n# Agent-specific mental states\nex:Agent_developer \n bdi:hasDesire ex:Desire_complete_coding ;\n bdi:hasIntention ex:Intention_implement_features .\n\nex:Desire_complete_coding a bdi:Desire ;\n rdfs:label \"Complete feature implementation\" ;\n bdi:isMotivatedBy ex:Belief_deadline_friday .\n\nex:Intention_implement_features a bdi:Intention ;\n rdfs:label \"Implement features A, B, and C\" ;\n bdi:fulfils ex:Desire_complete_coding ;\n bdi:specifies ex:Plan_development .\n\nex:Agent_manager \n bdi:hasDesire ex:Desire_ensure_delivery ;\n bdi:hasIntention ex:Intention_coordinate_team .\n\nex:Desire_ensure_delivery a bdi:Desire ;\n rdfs:label \"Ensure on-time project delivery\" ;\n bdi:isMotivatedBy ex:Belief_deadline_friday .\n\nex:Intention_coordinate_team a bdi:Intention ;\n rdfs:label \"Coordinate team activities\" ;\n bdi:fulfils ex:Desire_ensure_delivery ;\n bdi:specifies ex:Plan_project_management .\n\n# FIPA communication\nex:Message_M1 a fipa:ACLMessage ;\n fipa:sender ex:Agent_manager ;\n fipa:receiver ex:Agent_developer ;\n fipa:content ex:Belief_deadline_friday ;\n fipa:performative fipa:inform .\n```\n\n## Conflict Resolution Example\n\n```turtle\n@prefix bdi: .\n@prefix ex: .\n\n# Conflicting location beliefs\nex:Belief_at_home a bdi:Belief ;\n bdi:refersTo ex:WorldState_home ;\n rdfs:comment \"Agent is currently at home\" .\n\nex:Belief_at_office a bdi:Belief ;\n bdi:refersTo ex:WorldState_office ;\n rdfs:comment \"Agent is at office\" .\n\n# Conflicting intentions\nex:Intention_work_from_home a bdi:Intention ;\n bdi:isSupportedBy ex:Belief_at_home ;\n rdfs:label \"Work from home today\" .\n\nex:Intention_attend_meeting a bdi:Intention ;\n bdi:isSupportedBy ex:Belief_at_office ;\n rdfs:label \"Attend in-person meeting\" .\n\n# Justification for conflict resolution\nex:Justification_location_conflict a bdi:Justification ;\n rdfs:comment \"Cannot simultaneously be at home and office\" ;\n bdi:justifies ex:Intention_resolution .\n\n# Resolved intention\nex:Intention_resolution a bdi:Intention ;\n rdfs:label \"Attend meeting via video call from home\" ;\n bdi:fulfils ex:Desire_meeting_participation ;\n bdi:isSupportedBy ex:Belief_at_home ;\n bdi:isJustifiedBy ex:Justification_location_conflict .\n```\n\n## T2B2T Payment Processing Example\n\n```turtle\n@prefix bdi: .\n@prefix ex: .\n@prefix xsd: .\n\n# PHASE 1: Triples-to-Beliefs (External RDF → Internal Mental State)\n\nex:WorldState_notification a bdi:WorldState ;\n rdfs:comment \"Push notification: Ghadeh requested $250 via Zelle\" ;\n bdi:atTime \"2025-10-27T10:15:00\"^^xsd:dateTime ;\n bdi:triggers ex:BeliefProcess_BP1 .\n\nex:BeliefProcess_BP1 a bdi:BeliefProcess ;\n bdi:generates ex:Belief_payment_request ;\n bdi:isProcessedBy ex:Agent_A .\n\nex:Belief_payment_request a bdi:Belief ;\n rdfs:label \"Ghadeh requested $250\" ;\n bdi:refersTo ex:WorldState_notification ;\n bdi:motivates ex:Desire_pay_Ghadeh .\n\nex:Desire_pay_Ghadeh a bdi:Desire ;\n rdfs:label \"Pay Ghadeh $250\" ;\n bdi:isMotivatedBy ex:Belief_payment_request .\n\nex:Intention_I1 a bdi:Intention ;\n rdfs:label \"Pay Ghadeh $250\" ;\n bdi:fulfils ex:Desire_pay_Ghadeh ;\n bdi:specifies ex:Plan_payment .\n\n# PHASE 2: Beliefs-to-Triples (Mental State → External RDF)\n\nex:PlanExecution_PE1 a bdi:PlanExecution ;\n bdi:satisfies ex:Plan_payment ;\n bdi:bringsAbout ex:WorldState_payment_complete .\n\nex:WorldState_payment_complete a bdi:WorldState ;\n rdfs:comment \"Payment of $250 sent to Ghadeh via Zelle\" ;\n bdi:atTime \"2025-10-27T10:20:00\"^^xsd:dateTime .\n```\n\n\n# SPARQL Competency Queries\n\nValidation queries for BDI ontology implementations based on competency questions.\n\n## Mental Entity Queries\n\n### CQ1: What are all mental entities?\n\n```sparql\nPREFIX bdi: \nPREFIX rdf: \nPREFIX rdfs: \n\nSELECT DISTINCT ?entity ?type WHERE {\n ?entity rdf:type ?type .\n ?type rdfs:subClassOf* bdi:MentalEntity .\n}\n```\n\n### CQ2: What beliefs does an agent hold?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?label WHERE {\n ?agent bdi:hasMentalState ?belief .\n ?belief a bdi:Belief .\n OPTIONAL { ?belief rdfs:label ?label }\n}\n```\n\n### CQ3: What desires does an agent have?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?desire ?label WHERE {\n ?agent bdi:hasDesire ?desire .\n ?desire a bdi:Desire .\n OPTIONAL { ?desire rdfs:label ?label }\n}\n```\n\n### CQ4: What intentions has an agent committed to?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention ?label WHERE {\n ?agent bdi:hasIntention ?intention .\n ?intention a bdi:Intention .\n OPTIONAL { ?intention rdfs:label ?label }\n}\n```\n\n## Motivational Chain Queries\n\n### CQ5: What beliefs motivated formation of a given desire?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?beliefLabel WHERE {\n ?desire bdi:isMotivatedBy ?belief .\n ?belief a bdi:Belief .\n OPTIONAL { ?belief rdfs:label ?beliefLabel }\n}\n```\n\n### CQ6: Which desire does a particular intention fulfill?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?desire ?desireLabel WHERE {\n ?intention bdi:fulfils ?desire .\n ?desire a bdi:Desire .\n OPTIONAL { ?desire rdfs:label ?desireLabel }\n}\n```\n\n### CQ7: What beliefs support a given intention?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?label WHERE {\n ?intention bdi:isSupportedBy ?belief .\n ?belief a bdi:Belief .\n OPTIONAL { ?belief rdfs:label ?label }\n}\n```\n\n### CQ8: Trace complete cognitive chain for an intention\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention ?desire ?belief ?worldState WHERE {\n ?intention a bdi:Intention ;\n bdi:fulfils ?desire ;\n bdi:isSupportedBy ?belief .\n ?desire bdi:isMotivatedBy ?belief .\n ?belief bdi:refersTo ?worldState .\n}\n```\n\n## Mental Process Queries\n\n### CQ9: Which mental process generated a belief?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?process ?processType WHERE {\n ?process bdi:generates ?belief .\n ?belief a bdi:Belief .\n ?process a ?processType .\n FILTER(?processType != owl:NamedIndividual)\n}\n```\n\n### CQ10: What triggered a mental process?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?process ?trigger ?triggerType WHERE {\n ?process a bdi:MentalProcess ;\n bdi:isTriggeredBy ?trigger .\n ?trigger a ?triggerType .\n}\n```\n\n### CQ11: What did a mental process reason upon?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?process ?input WHERE {\n ?process a bdi:MentalProcess ;\n bdi:reasonsUpon ?input .\n}\n```\n\n## Plan and Goal Queries\n\n### CQ12: What plan does an intention specify?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention ?plan ?goal WHERE {\n ?intention bdi:specifies ?plan .\n ?plan a bdi:Plan ;\n bdi:addresses ?goal .\n}\n```\n\n### CQ13: What is the ordered sequence of tasks in a plan?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?plan ?task ?nextTask WHERE {\n ?plan a bdi:Plan ;\n bdi:hasComponent ?task .\n OPTIONAL { ?task bdi:precedes ?nextTask }\n}\nORDER BY ?task\n```\n\n### CQ14: What is the first and last task of a plan?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?plan ?firstTask ?lastTask WHERE {\n ?plan a bdi:Plan ;\n bdi:beginsWith ?firstTask ;\n bdi:endsWith ?lastTask .\n}\n```\n\n### CQ15: Which actions executed which tasks?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?action ?task ?time WHERE {\n ?action bdi:isExecutionOf ?task ;\n bdi:atTime ?time .\n}\nORDER BY ?time\n```\n\n## Temporal Queries\n\n### CQ16: What mental states are valid at a specific time?\n\n```sparql\nPREFIX bdi: \nPREFIX xsd: \n\nSELECT ?mentalState ?type WHERE {\n ?mentalState bdi:hasValidity ?interval .\n ?interval bdi:hasStartTime ?start ;\n bdi:hasEndTime ?end .\n ?mentalState a ?type .\n FILTER(?start <= \"2026-01-04T10:00:00\"^^xsd:dateTime && \n ?end >= \"2026-01-04T10:00:00\"^^xsd:dateTime)\n}\n```\n\n### CQ17: When was a belief formed?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?formationTime WHERE {\n ?process bdi:generates ?belief ;\n bdi:atTime ?formationTime .\n ?belief a bdi:Belief .\n}\n```\n\n### CQ18: What is the temporal validity of an intention?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention ?start ?end WHERE {\n ?intention a bdi:Intention ;\n bdi:hasValidity ?interval .\n ?interval bdi:hasStartTime ?start ;\n bdi:hasEndTime ?end .\n}\n```\n\n## Justification Queries\n\n### CQ19: What justifies a belief?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?justification ?justLabel WHERE {\n ?belief a bdi:Belief ;\n bdi:isJustifiedBy ?justification .\n OPTIONAL { ?justification rdfs:label ?justLabel }\n}\n```\n\n### CQ20: What justifies an intention?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention ?justification ?justLabel WHERE {\n ?intention a bdi:Intention ;\n bdi:isJustifiedBy ?justification .\n OPTIONAL { ?justification rdfs:label ?justLabel }\n}\n```\n\n## Compositional Queries\n\n### CQ21: What parts comprise a complex belief?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?part ?partLabel WHERE {\n ?belief a bdi:Belief ;\n bdi:hasPart ?part .\n OPTIONAL { ?part rdfs:label ?partLabel }\n}\n```\n\n### CQ22: Find composite mental entities\n\n```sparql\nPREFIX bdi: \n\nSELECT ?composite (COUNT(?part) AS ?partCount) WHERE {\n ?composite bdi:hasPart ?part .\n}\nGROUP BY ?composite\nHAVING (COUNT(?part) > 1)\n```\n\n## World State Queries\n\n### CQ23: What world state does a belief refer to?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief ?worldState ?wsComment WHERE {\n ?belief a bdi:Belief ;\n bdi:refersTo ?worldState .\n OPTIONAL { ?worldState rdfs:comment ?wsComment }\n}\n```\n\n### CQ24: What actions brought about a world state?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?action ?worldState WHERE {\n ?action bdi:bringsAbout ?worldState .\n ?worldState a bdi:WorldState .\n}\n```\n\n### CQ25: What world states has an agent perceived?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?agent ?worldState ?time WHERE {\n ?agent bdi:perceives ?worldState .\n OPTIONAL { ?worldState bdi:atTime ?time }\n}\n```\n\n## Validation Queries (OWLUnit Style)\n\n### V1: Every intention must fulfill exactly one desire\n\n```sparql\nPREFIX bdi: \n\nSELECT ?intention WHERE {\n ?intention a bdi:Intention .\n FILTER NOT EXISTS { ?intention bdi:fulfils ?desire }\n}\n# Expected: Empty result set\n```\n\n### V2: Every belief must reference a world state\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief WHERE {\n ?belief a bdi:Belief .\n FILTER NOT EXISTS { ?belief bdi:refersTo ?worldState }\n}\n# Expected: Empty result set (or only abstract beliefs)\n```\n\n### V3: Mental processes must reason upon something\n\n```sparql\nPREFIX bdi: \n\nSELECT ?process WHERE {\n ?process a bdi:MentalProcess .\n FILTER NOT EXISTS { ?process bdi:reasonsUpon ?input }\n}\n# Expected: Empty result set\n```\n\n### V4: BeliefProcess must generate only Beliefs\n\n```sparql\nPREFIX bdi: \n\nSELECT ?process ?generated WHERE {\n ?process a bdi:BeliefProcess ;\n bdi:generates ?generated .\n FILTER NOT EXISTS { ?generated a bdi:Belief }\n}\n# Expected: Empty result set\n```\n\n### V5: Plans must have begin and end tasks\n\n```sparql\nPREFIX bdi: \n\nSELECT ?plan WHERE {\n ?plan a bdi:Plan .\n FILTER NOT EXISTS { \n ?plan bdi:beginsWith ?first ;\n bdi:endsWith ?last \n }\n}\n# Expected: Empty result set\n```\n\n## Multi-Agent Queries\n\n### CQ26: What beliefs are shared across agents?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?belief (COUNT(DISTINCT ?agent) AS ?agentCount) WHERE {\n ?agent bdi:hasMentalState ?belief .\n ?belief a bdi:Belief .\n}\nGROUP BY ?belief\nHAVING (COUNT(DISTINCT ?agent) > 1)\n```\n\n### CQ27: Which agents share the same desire?\n\n```sparql\nPREFIX bdi: \n\nSELECT ?desire ?agent1 ?agent2 WHERE {\n ?agent1 bdi:hasDesire ?desire .\n ?agent2 bdi:hasDesire ?desire .\n FILTER(?agent1 != ?agent2)\n}\n```\n\n\n---\nname: bdi-mental-states\ndescription: \"This skill should be used when modeling agent mental states with BDI concepts: beliefs, desires, intentions, RDF-to-belief transformations, rational agency traces, cognitive agents, BDI ontologies, and neuro-symbolic AI integration.\"\n---\n\n# BDI Mental State Modeling\n\nTransform external RDF context into agent mental states (beliefs, desires, intentions) using formal BDI ontology patterns. This skill enables agents to reason about context through cognitive architecture, supporting deliberative reasoning, explainability, and semantic interoperability within multi-agent systems.\n\n## When to Activate\n\nActivate this skill when:\n- Processing external RDF context into agent beliefs about world states\n- Modeling rational agency with perception, deliberation, and action cycles\n- Enabling explainability through traceable reasoning chains\n- Implementing BDI frameworks (SEMAS, JADE, JADEX)\n- Augmenting LLMs with formal cognitive structures (Logic Augmented Generation)\n- Coordinating mental states across multi-agent platforms\n- Tracking temporal evolution of beliefs, desires, and intentions\n- Linking motivational states to action plans\n\nDo not activate this skill for adjacent work owned by other skills:\n- General context-window explanations or attention mechanics: `context-fundamentals`.\n- Persistent user, entity, or conversation memory without formal BDI state: `memory-systems`.\n- Supervisor, swarm, or handoff topology decisions: `multi-agent-patterns`.\n- General agent evaluation rubrics or quality gates: `evaluation`.\n\n## Core Concepts\n\n### Mental Reality Architecture\n\nSeparate mental states into two ontological categories because BDI reasoning requires distinguishing what persists from what happens:\n\n**Mental States (Endurants)** -- model these as persistent cognitive attributes that hold over time intervals:\n- `Belief`: Represent what the agent holds true about the world. Ground every belief in a world state reference.\n- `Desire`: Represent what the agent wishes to bring about. Link each desire back to the beliefs that motivate it.\n- `Intention`: Represent what the agent commits to achieving. An intention must fulfil a desire and specify a plan.\n\n**Mental Processes (Perdurants)** -- model these as events that create or modify mental states, because tracking causal transitions enables explainability:\n- `BeliefProcess`: Triggers belief formation/update from perception. Always connect to a generating world state.\n- `DesireProcess`: Generates desires from existing beliefs. Preserves the motivational chain.\n- `IntentionProcess`: Commits to selected desires as actionable intentions.\n\n### Cognitive Chain Pattern\n\nWire beliefs, desires, and intentions into directed chains using bidirectional properties (`motivates`/`isMotivatedBy`, `fulfils`/`isFulfilledBy`) because this enables both forward reasoning (what should the agent do?) and backward tracing (why did the agent act?):\n\n```turtle\n:Belief_store_open a bdi:Belief ;\n rdfs:comment \"Store is open\" ;\n bdi:motivates :Desire_buy_groceries .\n\n:Desire_buy_groceries a bdi:Desire ;\n rdfs:comment \"I desire to buy groceries\" ;\n bdi:isMotivatedBy :Belief_store_open .\n\n:Intention_go_shopping a bdi:Intention ;\n rdfs:comment \"I will buy groceries\" ;\n bdi:fulfils :Desire_buy_groceries ;\n bdi:isSupportedBy :Belief_store_open ;\n bdi:specifies :Plan_shopping .\n```\n\n### World State Grounding\n\nAlways ground mental states in world state references rather than free-text descriptions, because ungrounded beliefs break semantic querying and cross-agent interoperability:\n\n```turtle\n:Agent_A a bdi:Agent ;\n bdi:perceives :WorldState_WS1 ;\n bdi:hasMentalState :Belief_B1 .\n\n:WorldState_WS1 a bdi:WorldState ;\n rdfs:comment \"Meeting scheduled at 10am in Room 5\" ;\n bdi:atTime :TimeInstant_10am .\n\n:Belief_B1 a bdi:Belief ;\n bdi:refersTo :WorldState_WS1 .\n```\n\n### Goal-Directed Planning\n\nConnect intentions to plans via `bdi:specifies`, and decompose plans into ordered task sequences using `bdi:precedes`, because this separation allows plan reuse across different intentions while keeping execution order explicit:\n\n```turtle\n:Intention_I1 bdi:specifies :Plan_P1 .\n\n:Plan_P1 a bdi:Plan ;\n bdi:addresses :Goal_G1 ;\n bdi:beginsWith :Task_T1 ;\n bdi:endsWith :Task_T3 .\n\n:Task_T1 bdi:precedes :Task_T2 .\n:Task_T2 bdi:precedes :Task_T3 .\n```\n\n### T2B2T Paradigm\n\nImplement Triples-to-Beliefs-to-Triples as a bidirectional pipeline because agents must both consume external RDF context and produce new RDF assertions. Structure every T2B2T implementation in two explicit phases:\n\n**Phase 1: Triples-to-Beliefs** -- Translate incoming RDF triples into belief instances. Use `bdi:triggers` to connect the external world state to a `BeliefProcess`, and `bdi:generates` to produce the resulting belief. This preserves provenance from source data through to internal cognition:\n```turtle\n:WorldState_notification a bdi:WorldState ;\n rdfs:comment \"Push notification: Payment request $250\" ;\n bdi:triggers :BeliefProcess_BP1 .\n\n:BeliefProcess_BP1 a bdi:BeliefProcess ;\n bdi:generates :Belief_payment_request .\n```\n\n**Phase 2: Beliefs-to-Triples** -- After BDI deliberation selects an intention and executes a plan, project the results back into RDF using `bdi:bringsAbout`. This closes the loop so downstream systems can consume agent outputs as standard linked data:\n```turtle\n:Intention_pay a bdi:Intention ;\n bdi:specifies :Plan_payment .\n\n:PlanExecution_PE1 a bdi:PlanExecution ;\n bdi:satisfies :Plan_payment ;\n bdi:bringsAbout :WorldState_payment_complete .\n```\n\n### Notation Selection by Level\n\nChoose notation based on the C4 abstraction level being modeled, because mixing notations at the wrong level obscures rather than clarifies the cognitive architecture:\n\n| C4 Level | Notation | Mental State Representation |\n|----------|----------|----------------------------|\n| L1 Context | ArchiMate | Agent boundaries, external perception sources |\n| L2 Container | ArchiMate | BDI reasoning engine, belief store, plan executor |\n| L3 Component | UML | Mental state managers, process handlers |\n| L4 Code | UML/RDF | Belief/Desire/Intention classes, ontology instances |\n\n### Justification and Explainability\n\nAttach `bdi:Justification` instances to every mental entity using `bdi:isJustifiedBy`, because unjustified mental states make agent reasoning opaque and untraceable. Each justification should capture the evidence or rule that produced the mental state:\n\n```turtle\n:Belief_B1 a bdi:Belief ;\n bdi:isJustifiedBy :Justification_J1 .\n\n:Justification_J1 a bdi:Justification ;\n rdfs:comment \"Official announcement received via email\" .\n\n:Intention_I1 a bdi:Intention ;\n bdi:isJustifiedBy :Justification_J2 .\n\n:Justification_J2 a bdi:Justification ;\n rdfs:comment \"Location precondition satisfied\" .\n```\n\n### Temporal Dimensions\n\nAssign validity intervals to every mental state using `bdi:hasValidity` with `TimeInterval` instances, because beliefs without temporal bounds cannot be garbage-collected or conflict-checked during diachronic reasoning:\n\n```turtle\n:Belief_B1 a bdi:Belief ;\n bdi:hasValidity :TimeInterval_TI1 .\n\n:TimeInterval_TI1 a bdi:TimeInterval ;\n bdi:hasStartTime :TimeInstant_9am ;\n bdi:hasEndTime :TimeInstant_11am .\n```\n\nQuery mental states active at a specific moment using SPARQL temporal filters. Use this pattern to resolve conflicts when multiple beliefs about the same world state overlap in time:\n\n```sparql\nSELECT ?mentalState WHERE {\n ?mentalState bdi:hasValidity ?interval .\n ?interval bdi:hasStartTime ?start ;\n bdi:hasEndTime ?end .\n FILTER(?start <= \"2025-01-04T10:00:00\"^^xsd:dateTime &&\n ?end >= \"2025-01-04T10:00:00\"^^xsd:dateTime)\n}\n```\n\n### Compositional Mental Entities\n\nDecompose complex beliefs into constituent parts using `bdi:hasPart` relations, because monolithic beliefs force full replacement on partial updates. Structure composite beliefs so that each sub-belief can be independently updated, queried, or invalidated:\n\n```turtle\n:Belief_meeting a bdi:Belief ;\n rdfs:comment \"Meeting at 10am in Room 5\" ;\n bdi:hasPart :Belief_meeting_time , :Belief_meeting_location .\n\n# Update only location component without touching time\n:BeliefProcess_update a bdi:BeliefProcess ;\n bdi:modifies :Belief_meeting_location .\n```\n\n## Practical Guidance\n\n### Build a BDI Model in Six Passes\n\nUse this workflow when converting external semantic context into a BDI representation:\n\n1. **Define the world-state substrate**: Identify the external facts or events the agent can perceive. Model these as world states before creating beliefs.\n2. **Create belief instances**: Translate each relevant world state into a belief with provenance, temporal validity, and a justification reference.\n3. **Derive desires from beliefs**: Add desires only when a belief creates a goal-relevant motivation. Link each desire to the belief that motivates it.\n4. **Commit intentions deliberately**: Promote a desire to an intention only when the agent commits to a plan. Record the selected plan and preconditions.\n5. **Project action results back to triples**: After execution, emit resulting world states as RDF so downstream systems can consume the new state.\n6. **Validate with competency questions**: Query for provenance, motivation, plan sequence, and active validity windows before trusting the model.\n\n### Keep the Ontology Small\n\nStart with `Agent`, `WorldState`, `Belief`, `Desire`, `Intention`, `Plan`, `Task`, `Justification`, and `TimeInterval`. Add specialized classes only after competency questions prove the core model cannot answer required queries. A compact ontology is easier to serialize into prompts, easier to validate, and less likely to create brittle reasoning chains.\n\n### Use BDI Only When Mental-State Semantics Matter\n\nBDI modeling is justified when the system needs explainable agency: why an agent believed something, what desire that belief created, which intention was selected, and what plan executed. If the system only needs to remember facts across sessions, use `memory-systems`. If it only needs to split work across agents, use `multi-agent-patterns`.\n\n## Detailed Topics\n\n### Integration Patterns\n\n### Logic Augmented Generation (LAG)\n\nUse LAG to constrain LLM outputs with ontological structure, because unconstrained generation produces triples that violate BDI class restrictions. Serialize the ontology into the prompt context, then validate generated triples against it before accepting them:\n\n```python\ndef augment_llm_with_bdi_ontology(prompt, ontology_graph):\n ontology_context = serialize_ontology(ontology_graph, format='turtle')\n augmented_prompt = f\"{ontology_context}\\n\\n{prompt}\"\n\n response = llm.generate(augmented_prompt)\n triples = extract_rdf_triples(response)\n\n is_consistent = validate_triples(triples, ontology_graph)\n return triples if is_consistent else retry_with_feedback()\n```\n\n### SEMAS Rule Translation\n\nTranslate BDI ontology patterns into executable production rules when deploying to rule-based agent platforms. Map each cognitive chain link (belief-to-desire, desire-to-intention) to a HEAD/CONDITIONALS/TAIL rule, because this preserves the deliberative semantics while enabling runtime execution:\n\n```prolog\n% Belief triggers desire formation\n[HEAD: belief(agent_a, store_open)] /\n[CONDITIONALS: time(weekday_afternoon)] »\n[TAIL: generate_desire(agent_a, buy_groceries)].\n\n% Desire triggers intention commitment\n[HEAD: desire(agent_a, buy_groceries)] /\n[CONDITIONALS: belief(agent_a, has_shopping_list)] »\n[TAIL: commit_intention(agent_a, buy_groceries)].\n```\n\n## Guidelines\n\n1. Model world states as configurations independent of agent perspectives, providing referential substrate for mental states.\n\n2. Distinguish endurants (persistent mental states) from perdurants (temporal mental processes), aligning with DOLCE ontology.\n\n3. Treat goals as descriptions rather than mental states, maintaining separation between cognitive and planning layers.\n\n4. Use `hasPart` relations for meronymic structures enabling selective belief updates.\n\n5. Associate every mental entity with temporal constructs via `atTime` or `hasValidity`.\n\n6. Use bidirectional property pairs (`motivates`/`isMotivatedBy`, `generates`/`isGeneratedBy`) for flexible querying.\n\n7. Link mental entities to `Justification` instances for explainability and trust.\n\n8. Implement T2B2T through: (1) translate RDF to beliefs, (2) execute BDI reasoning, (3) project mental states back to RDF.\n\n9. Define existential restrictions on mental processes (e.g., `BeliefProcess ⊑ ∃generates.Belief`).\n\n10. Reuse established ODPs (EventCore, Situation, TimeIndexedSituation, BasicPlan, Provenance) for interoperability.\n\n## Competency Questions\n\nValidate implementation against these SPARQL queries:\n\n```sparql\n# CQ1: What beliefs motivated formation of a given desire?\nSELECT ?belief WHERE {\n :Desire_D1 bdi:isMotivatedBy ?belief .\n}\n\n# CQ2: Which desire does a particular intention fulfill?\nSELECT ?desire WHERE {\n :Intention_I1 bdi:fulfils ?desire .\n}\n\n# CQ3: Which mental process generated a belief?\nSELECT ?process WHERE {\n ?process bdi:generates :Belief_B1 .\n}\n\n# CQ4: What is the ordered sequence of tasks in a plan?\nSELECT ?task ?nextTask WHERE {\n :Plan_P1 bdi:hasComponent ?task .\n OPTIONAL { ?task bdi:precedes ?nextTask }\n} ORDER BY ?task\n```\n\n## Examples\n\n**Example 1: RDF notification to BDI chain**\n\nInput world state:\n\n```turtle\n:WorldState_invoice_due a bdi:WorldState ;\n rdfs:comment \"Invoice INV-42 is due tomorrow\" ;\n bdi:atTime :Time_2026_05_15 .\n```\n\nBDI projection:\n\n```turtle\n:Belief_invoice_due a bdi:Belief ;\n bdi:refersTo :WorldState_invoice_due ;\n bdi:isJustifiedBy :Justification_billing_system ;\n bdi:motivates :Desire_avoid_late_fee .\n\n:Desire_avoid_late_fee a bdi:Desire ;\n bdi:isMotivatedBy :Belief_invoice_due .\n\n:Intention_pay_invoice a bdi:Intention ;\n bdi:fulfils :Desire_avoid_late_fee ;\n bdi:specifies :Plan_pay_invoice .\n```\n\n**Example 2: Boundary decision**\n\nIf the task is \"remember that Alice prefers concise summaries,\" use `memory-systems`. If the task is \"represent why the agent believes Alice needs a summary, what goal that creates, and which plan it commits to,\" use this skill.\n\n## Gotchas\n\n1. **Conflating mental states with world states**: Mental states reference world states via `bdi:refersTo`, they are not world states themselves. Mixing them collapses the perception-cognition boundary and breaks SPARQL queries that filter by type.\n\n2. **Missing temporal bounds**: Every mental state needs validity intervals for diachronic reasoning. Without them, stale beliefs persist indefinitely and conflict detection becomes impossible.\n\n3. **Flat belief structures**: Use compositional modeling with `hasPart` for complex beliefs. Monolithic beliefs force full replacement when only one attribute changes.\n\n4. **Implicit justifications**: Always link mental entities to explicit `Justification` instances. Unjustified mental states cannot be audited or traced.\n\n5. **Direct intention-to-action mapping**: Intentions specify plans which contain tasks; actions execute tasks. Skipping the plan layer removes the ability to reuse, reorder, or share execution strategies.\n\n6. **Ontology over-complexity**: Start with 5-10 core classes and properties (Belief, Desire, Intention, WorldState, Plan, plus key relations). Expanding the ontology prematurely inflates prompt context and slows SPARQL queries without improving reasoning quality.\n\n7. **Reasoning cost explosion**: Keep belief chains to 3 levels or fewer (belief -> desire -> intention). Deeper chains become prohibitively expensive for LLM inference and rarely improve decision quality over shallower alternatives.\n\n## Integration\n\nThis skill owns formal mental-state modeling. Adjacent skills own different layers:\n\n- `memory-systems`: persistent facts, entity memory, and temporal knowledge graphs without BDI belief/desire/intention semantics.\n- `multi-agent-patterns`: agent topology, handoff protocols, and coordination between agents.\n- `evaluation`: competency questions, regression checks, and quality gates for BDI implementations.\n- `context-fundamentals`: conceptual context-window and attention mechanics that inform prompt construction.\n- `tool-design`: schema and tool contracts for BDI query, validation, or projection tools.\n\n## References\n\nInternal references:\n- [BDI Ontology Core](./references/bdi-ontology-core.md) - Read when: implementing BDI class hierarchies or defining ontology properties from scratch\n- [RDF Examples](./references/rdf-examples.md) - Read when: writing Turtle serializations of mental states or debugging triple structure\n- [SPARQL Competency Queries](./references/sparql-competency.md) - Read when: validating an implementation against competency questions or building custom queries\n- [Framework Integration](./references/framework-integration.md) - Read when: deploying BDI models to SEMAS, JADE, or LAG pipelines\n\nPrimary sources:\n- Zuppiroli et al. \"The Belief-Desire-Intention Ontology\" (2025) — Read when: implementing formal BDI class hierarchies or validating ontology alignment\n- Rao & Georgeff \"BDI agents: From theory to practice\" (1995) — Read when: understanding the theoretical foundations of practical reasoning agents\n- Bratman \"Intention, plans, and practical reason\" (1987) — Read when: grounding implementation decisions in the philosophical basis of intentionality\n\n---\n\n## Skill Metadata\n\n**Created**: 2026-01-07\n**Last Updated**: 2026-05-15\n**Author**: Agent Skills for Context Engineering Contributors\n**Version**: 2.1.0\n\n\n"}