{"owner":"Onelevenvy","repo":"flock","hasSkills":true,"totalSkillsCount":19,"totalTokensCount":32186,"categories":["root-instruction","claude-rule","anthropic-skill","subagent-persona"],"hasMcp":false,"mcpConfig":null,"found":["AGENTS.md","CLAUDE.md","flock-data/skills/frontend-design/SKILL.md","flock-data/skills/karpathy/SKILL.md","flock-data/skills/karpathy/metadata.json","flock-data/skills/planning-with-files/SKILL.md","flock-data/skills/planning-with-files/examples.md","flock-data/skills/planning-with-files/reference.md","flock-data/skills/planning-with-files/templates/findings.md","flock-data/skills/planning-with-files/templates/progress.md","flock-data/skills/planning-with-files/templates/task_plan.md","flock-data/skills/skill-creator/SKILL.md","flock-data/skills/skill-creator/agents/analyzer.md","flock-data/skills/skill-creator/agents/comparator.md","flock-data/skills/skill-creator/agents/grader.md","flock-data/skills/skill-creator/references/schemas.md","flock-data/skills/using-superpowers/SKILL.md","flock-data/skills/using-superpowers/references/codex-tools.md","flock-data/skills/using-superpowers/references/gemini-tools.md"],"skills":{"AGENTS.md":"# AGENTS.md\n\nThis file provides guidance to CodeX when working with code in this repository.\n\n## Project Overview\n\nFlock is a multi-provider AI agent desktop application with tool orchestration, sandbox execution, visual workflow, and browser/computer-use capabilities. It's a Rust workspace with a Tauri-based desktop UI (React + TypeScript + Mantine).\n\n## Build & Development Commands\n\n### Rust Backend\n```bash\n# Build the CLI\ncargo build --release\n\n# Run in single-shot mode\ncargo run -- \"your question\"\n\n# Run interactive REPL\ncargo run\n\n# Run with specific provider/profile\ncargo run -- --profile deepseek \"your question\"\n\n# Run tests (workspace-wide)\ncargo test\n\n# Run a single crate's tests\ncargo test -p flock-core\ncargo test -p flock-agent\ncargo test -p flock-tools\ncargo test -p flock-skills\n\n# Run a specific test\ncargo test -p flock-core test_name\n\n# Lint\ncargo clippy --workspace\n```\n\n### Tauri Desktop UI\n```bash\ncd flock-ui\n\n# Install dependencies\nnpm install\n\n# Development server\nnpm run dev\n# or\nnpm run tauri dev\n\n# Build for production\nnpm run build\n# or\nnpm run tauri build\n\n# Lint frontend\nnpm run lint\n```\n\n## Architecture\n\n### Workspace Crates\n\n| Crate | Purpose |\n|-------|---------|\n| `flock-core` | Core types, config, database, IPC interface, crypto, model factory |\n| `flock-agent` | Agent engine, session management, tool execution, memory, graph orchestration, workflow engine |\n| `flock-tools` | Tool registry, built-in tools (Read/Write/Edit/Bash/Grep/Glob), MCP integration, sandbox tools (CodeExecution, SandboxExec, Browser, ComputerUse) |\n| `flock-skills` | Skill discovery, loading, frontmatter parsing, hooks, permissions, bundled skills |\n| `flock-ui/src-tauri` | Tauri desktop app backend (commands, agent state, workspace management) |\n| `workspace-hack` | cargo-hakari build optimization |\n\n### Key Dependencies\n\n- **LangGraph**: Rust reimplementation ([langgraph-rust](https://github.com/Onelevenvy/langgraph-rust)) for agent graph orchestration\n- **LLM Providers**: OpenAI-compatible (default), Anthropic, AWS Bedrock, Google Vertex\n- **Database**: SQLite via sqlx for sessions, conversations, providers, tools, MCP servers, workflows\n- **Async Runtime**: Tokio\n- **Sandbox**: Daytona cloud container runtime for isolated code execution\n- **Browser Automation**: Playwright (Python, runs inside sandbox via CDP)\n\n### Core Architecture Patterns\n\n**Agent Engine** (`flock-agent/src/engine.rs`):\n- `AgentEngine` orchestrates the LLM interaction loop\n- Uses LangGraph `CompiledStateGraph` for execution flow\n- Manages conversation history via SQLite checkpointer\n- Handles tool approval, context compression, plan mode\n\n**Tool System** (`flock-tools/src/lib.rs`):\n- `Tool` trait defines the interface: `name()`, `description()`, `input_schema()`, `execute()`\n- `ToolRegistry` manages all registered tools\n- Built-in tools: Read, Write, Edit, Bash, Grep, Glob\n- Sandbox tools: CodeExecution, SandboxExec, Browser, ComputerUse, RequestHumanAssistance\n- Extensible via MCP servers and custom tool providers\n\n**Workflow Engine** (`flock-agent/src/workflow_graph/`):\n- Visual workflow builder with ReactFlow canvas\n- 10 node types: start, llm, agent, classifier, ifelse, answer, code, human, plugin, parameter_extractor\n- `build_workflow_graph()` compiles ReactFlow JSON into LangGraph `StateGraph`\n- Streaming execution with human-in-the-loop interrupts\n\n**Sandbox System** (`flock-tools/src/daytona/`):\n- Cloud-based Daytona container runtime for isolated execution\n- VNC desktop provisioning (Xvfb, fluxbox, x11vnc, websockify)\n- Lifecycle management: create, destroy, set-public, snapshot\n- Human takeover via VNC with approval flow\n\n**Skills System** (`flock-skills/`):\n- Markdown files with YAML frontmatter in `.flock/skills/`\n- Supports `$ARGUMENTS` substitution, shell execution, conditional activation\n- Two execution contexts: `inline` (same agent) and `fork` (sub-agent)\n- Hot-reload via file watcher\n\n**Memory System** (`flock-agent/src/memory/`):\n- Four types: `user`, `feedback`, `project`, `reference`\n- Stored as Markdown files with frontmatter in `/flock/projects//memory/`\n- `MEMORY.md` index file auto-loaded into system prompt\n\n**IPC Interface** (`flock-core/src/ipc_interface/`):\n- JSON-based protocol for host ↔ agent communication\n- Events (Agent → Host): `ready`, `stream_start`, `text_delta`, `tool_request`, `tool_result`, etc.\n- Commands (Host → Agent): `message`, `stop`, `tool_approve`, `tool_deny`, `set_mode`, `set_config`\n\n**Configuration** (`flock-core/src/config/settings.rs`):\n- Config priority: CLI args/env > project `flock.toml` > global config\n- Supports profiles with inheritance (`extends`)\n- Provider compatibility settings (`ProviderCompat`) for API differences\n\n### Tauri Desktop UI\n\n**Frontend Stack**:\n- React 18 + TypeScript\n- Mantine UI v8 (components, notifications)\n- Zustand (state management)\n- React Query (server state)\n- react-markdown + react-syntax-highlighter (message rendering)\n- i18next (internationalization)\n- ReactFlow (workflow visual editor)\n\n**State Stores** (`flock-ui/src/store/`):\n- `agentStore`: Agent connection status, messages, pending approvals\n- `uiStore`: Theme, sidebar state, active view, file tree\n- `workspaceStore`: Active workspace/conversation, persisted to localStorage\n- `workflowStore`: ReactFlow nodes/edges, execution state, dirty tracking\n\n**Key Views**:\n- `HomeView`: Welcome screen, assistant selection, workspace picker\n- `WorkspaceView`: Chat panel + file tree + optional preview panel (VNC, sandbox runner)\n- `WorkflowEditor`: Visual workflow builder with ReactFlow canvas, node palette, properties panel, execution panel\n- `AssistantPage`: CRUD for custom assistants\n- `SkillsPage`: Tools, MCP servers, and skills management\n- `SettingsModal`: Model providers, basic settings, sandbox settings, system settings\n\n**Tauri Commands** (`flock-ui/src-tauri/src/commands/`):\n- Agent control: `start_agent`, `stop_agent`, `send_message`, `approve_tool`, `deny_tool`\n- Workspace: `list_workspaces`, `create_workspace`, `delete_workspace`\n- Conversations: `list_conversations`, `create_conversation`, `load_conversation_history`\n- Database: `list_providers`, `upsert_provider`, `list_models`, `upsert_model`\n- MCP: `list_mcp_servers`, `upsert_mcp_server`, `test_mcp_server`\n- Skills: `list_skills`, `get_extra_skill_dirs`, `add_extra_skill_dir`\n- Workflow: `list_workflows`, `get_workflow`, `create_workflow`, `update_workflow`, `delete_workflow`, `run_workflow`, `stop_workflow`\n\n### Data Flow\n\n1. User sends message via UI → Tauri command `send_message`\n2. `AgentState` forwards to `AgentEngine::run()`\n3. Engine builds LangGraph, invokes LLM with tools\n4. LLM response streamed as `ProtocolEvent`s via `TauriProtocolEmitter`\n5. Frontend `useEventStream` hook listens to Tauri events, updates `agentStore`\n6. Tool calls require approval (unless auto-approved) → `ToolApprovalInline` component\n7. Approved tools execute → results fed back to LLM for next turn\n\n## Code Style\n\n- Rust edition 2024\n- Workspace lints: `unused = \"allow\"`, `unused_imports = \"allow\"`\n- Do not treat repository-wide `cargo fmt --check` diffs or warnings such as unused imports/variables as blockers unless they are introduced by the current change or explicitly requested. Focus verification on build/test failures and behavior regressions.\n- Error handling: `anyhow` for applications, `thiserror` for libraries\n- Async: Tokio runtime throughout\n- Frontend: ESLint with TypeScript rules, React hooks plugin\n\n## Global Guidelines & Core Principles\n\nWhen modifying the codebase or interacting with the user, please strictly adhere to the following rules:\n\n1. **Git Commit Format**: Upon completing a task, always output a single-line English `git commit` message beginning with standard prefixes (`feat:`, `fix:`, `chore:`, etc.). Provide the message text clearly, but **DO NOT** execute git commit commands or auto-submit.\n2. **State & Logic Abstraction**: Always separate state management and command logic from UI components. Place custom Hooks in `src/hooks` and Zustand stores in `src/store` rather than defining them inside component files.\n3. **Strict i18n Internationalization**: All newly introduced UI components must leverage `i18next` for user-facing texts (e.g. tooltips, titles, toast notifications, confirm boxes). Do not hardcode Chinese or English strings inside JSX.\n4. **Component Granularity**: Strive for modularity. Always split large layout or page files into focused, localized sub-components (placed in a nested `/components` folder). Avoid keeping all view code inside single massive `index.tsx` files.\n5. **UI Specification Compliance**: Ensure any new styling aligns with the design tokens defined in `docs/resources/ui_specification.md` (e.g. cold gray colors, `--flock-accent` solid blue branding, standard monospace typographies, and hover-card transitions) in both dark and light modes.\n","CLAUDE.md":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nFlock is a multi-provider AI agent desktop application with tool orchestration, sandbox execution, visual workflow, and browser/computer-use capabilities. It's a Rust workspace with a Tauri-based desktop UI (React + TypeScript + Mantine).\n\n## Build & Development Commands\n\n### Rust Backend\n```bash\n# Build the CLI\ncargo build --release\n\n# Run in single-shot mode\ncargo run -- \"your question\"\n\n# Run interactive REPL\ncargo run\n\n# Run with specific provider/profile\ncargo run -- --profile deepseek \"your question\"\n\n# Run tests (workspace-wide)\ncargo test\n\n# Run a single crate's tests\ncargo test -p flock-core\ncargo test -p flock-agent\ncargo test -p flock-tools\ncargo test -p flock-skills\n\n# Run a specific test\ncargo test -p flock-core test_name\n\n# Lint\ncargo clippy --workspace\n```\n\n### Tauri Desktop UI\n```bash\ncd flock-ui\n\n# Install dependencies\nnpm install\n\n# Development server\nnpm run dev\n# or\nnpm run tauri dev\n\n# Build for production\nnpm run build\n# or\nnpm run tauri build\n\n# Lint frontend\nnpm run lint\n```\n\n## Architecture\n\n### Workspace Crates\n\n| Crate | Purpose |\n|-------|---------|\n| `flock-core` | Core types, config, database, IPC interface, crypto, model factory |\n| `flock-agent` | Agent engine, session management, tool execution, memory, graph orchestration, workflow engine |\n| `flock-tools` | Tool registry, built-in tools (Read/Write/Edit/Bash/Grep/Glob), MCP integration, sandbox tools (CodeExecution, SandboxExec, Browser, ComputerUse) |\n| `flock-skills` | Skill discovery, loading, frontmatter parsing, hooks, permissions, bundled skills |\n| `flock-ui/src-tauri` | Tauri desktop app backend (commands, agent state, workspace management) |\n| `workspace-hack` | cargo-hakari build optimization |\n\n### Key Dependencies\n\n- **LangGraph**: Rust reimplementation ([langgraph-rust](https://github.com/Onelevenvy/langgraph-rust)) for agent graph orchestration\n- **LLM Providers**: OpenAI-compatible (default), Anthropic, AWS Bedrock, Google Vertex\n- **Database**: SQLite via sqlx for sessions, conversations, providers, tools, MCP servers, workflows\n- **Async Runtime**: Tokio\n- **Sandbox**: Daytona cloud container runtime for isolated code execution\n- **Browser Automation**: Playwright (Python, runs inside sandbox via CDP)\n\n### Core Architecture Patterns\n\n**Agent Engine** (`flock-agent/src/engine.rs`):\n- `AgentEngine` orchestrates the LLM interaction loop\n- Uses LangGraph `CompiledStateGraph` for execution flow\n- Manages conversation history via SQLite checkpointer\n- Handles tool approval, context compression, plan mode\n\n**Tool System** (`flock-tools/src/lib.rs`):\n- `Tool` trait defines the interface: `name()`, `description()`, `input_schema()`, `execute()`\n- `ToolRegistry` manages all registered tools\n- Built-in tools: Read, Write, Edit, Bash, Grep, Glob\n- Sandbox tools: CodeExecution, SandboxExec, Browser, ComputerUse, RequestHumanAssistance\n- Extensible via MCP servers and custom tool providers\n\n**Workflow Engine** (`flock-agent/src/workflow_graph/`):\n- Visual workflow builder with ReactFlow canvas\n- 10 node types: start, llm, agent, classifier, ifelse, answer, code, human, plugin, parameter_extractor\n- `build_workflow_graph()` compiles ReactFlow JSON into LangGraph `StateGraph`\n- Streaming execution with human-in-the-loop interrupts\n\n**Sandbox System** (`flock-tools/src/daytona/`):\n- Cloud-based Daytona container runtime for isolated execution\n- VNC desktop provisioning (Xvfb, fluxbox, x11vnc, websockify)\n- Lifecycle management: create, destroy, set-public, snapshot\n- Human takeover via VNC with approval flow\n\n**Skills System** (`flock-skills/`):\n- Markdown files with YAML frontmatter in `.flock/skills/`\n- Supports `$ARGUMENTS` substitution, shell execution, conditional activation\n- Two execution contexts: `inline` (same agent) and `fork` (sub-agent)\n- Hot-reload via file watcher\n\n**Memory System** (`flock-agent/src/memory/`):\n- Four types: `user`, `feedback`, `project`, `reference`\n- Stored as Markdown files with frontmatter in `/flock/projects//memory/`\n- `MEMORY.md` index file auto-loaded into system prompt\n\n**IPC Interface** (`flock-core/src/ipc_interface/`):\n- JSON-based protocol for host ↔ agent communication\n- Events (Agent → Host): `ready`, `stream_start`, `text_delta`, `tool_request`, `tool_result`, etc.\n- Commands (Host → Agent): `message`, `stop`, `tool_approve`, `tool_deny`, `set_mode`, `set_config`\n\n**Configuration** (`flock-core/src/config/settings.rs`):\n- Config priority: CLI args/env > project `flock.toml` > global config\n- Supports profiles with inheritance (`extends`)\n- Provider compatibility settings (`ProviderCompat`) for API differences\n\n### Tauri Desktop UI\n\n**Frontend Stack**:\n- React 18 + TypeScript\n- Mantine UI v8 (components, notifications)\n- Zustand (state management)\n- React Query (server state)\n- react-markdown + react-syntax-highlighter (message rendering)\n- i18next (internationalization)\n- ReactFlow (workflow visual editor)\n\n**State Stores** (`flock-ui/src/store/`):\n- `agentStore`: Agent connection status, messages, pending approvals\n- `uiStore`: Theme, sidebar state, active view, file tree\n- `workspaceStore`: Active workspace/conversation, persisted to localStorage\n- `workflowStore`: ReactFlow nodes/edges, execution state, dirty tracking\n\n**Key Views**:\n- `HomeView`: Welcome screen, assistant selection, workspace picker\n- `WorkspaceView`: Chat panel + file tree + optional preview panel (VNC, sandbox runner)\n- `WorkflowEditor`: Visual workflow builder with ReactFlow canvas, node palette, properties panel, execution panel\n- `AssistantPage`: CRUD for custom assistants\n- `SkillsPage`: Tools, MCP servers, and skills management\n- `SettingsModal`: Model providers, basic settings, sandbox settings, system settings\n\n**Tauri Commands** (`flock-ui/src-tauri/src/commands/`):\n- Agent control: `start_agent`, `stop_agent`, `send_message`, `approve_tool`, `deny_tool`\n- Workspace: `list_workspaces`, `create_workspace`, `delete_workspace`\n- Conversations: `list_conversations`, `create_conversation`, `load_conversation_history`\n- Database: `list_providers`, `upsert_provider`, `list_models`, `upsert_model`\n- MCP: `list_mcp_servers`, `upsert_mcp_server`, `test_mcp_server`\n- Skills: `list_skills`, `get_extra_skill_dirs`, `add_extra_skill_dir`\n- Workflow: `list_workflows`, `get_workflow`, `create_workflow`, `update_workflow`, `delete_workflow`, `run_workflow`, `stop_workflow`\n\n### Data Flow\n\n1. User sends message via UI → Tauri command `send_message`\n2. `AgentState` forwards to `AgentEngine::run()`\n3. Engine builds LangGraph, invokes LLM with tools\n4. LLM response streamed as `ProtocolEvent`s via `TauriProtocolEmitter`\n5. Frontend `useEventStream` hook listens to Tauri events, updates `agentStore`\n6. Tool calls require approval (unless auto-approved) → `ToolApprovalInline` component\n7. Approved tools execute → results fed back to LLM for next turn\n\n## Code Style\n\n- Rust edition 2024\n- Workspace lints: `unused = \"allow\"`, `unused_imports = \"allow\"`\n- Do not treat repository-wide `cargo fmt --check` diffs or warnings such as unused imports/variables as blockers unless they are introduced by the current change or explicitly requested. Focus verification on build/test failures and behavior regressions.\n- Error handling: `anyhow` for applications, `thiserror` for libraries\n- Async: Tokio runtime throughout\n- Frontend: ESLint with TypeScript rules, React hooks plugin\n\n\n## Global Guidelines & Core Principles\n\nWhen modifying the codebase or interacting with the user, please strictly adhere to the following rules:\n\n1. **Git Commit Format**: Upon completing a task, always output a single-line English `git commit` message beginning with standard prefixes (`feat:`, `fix:`, `chore:`, etc.). Provide the message text clearly, but **DO NOT** execute git commit commands or auto-submit.\n2. **State & Logic Abstraction**: Always separate state management and command logic from UI components. Place custom Hooks in `src/hooks` and Zustand stores in `src/store` rather than defining them inside component files.\n3. **Strict i18n Internationalization**: All newly introduced UI components must leverage `i18next` for user-facing texts (e.g. tooltips, titles, toast notifications, confirm boxes). Do not hardcode Chinese or English strings inside JSX.\n4. **Component Granularity**: Strive for modularity. Always split large layout or page files into focused, localized sub-components (placed in a nested `/components` folder). Avoid keeping all view code inside single massive `index.tsx` files.\n5. **UI Specification Compliance**: Ensure any new styling aligns with the design tokens defined in `docs/resources/ui_specification.md` (e.g. cold gray colors, `--flock-accent` solid blue branding, standard monospace typographies, and hover-card transitions) in both dark and light modes.\n","flock-data/skills/frontend-design/SKILL.md":"---\nname: frontend-design\ndescription: Create distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, or applications. Generates creative, polished code that avoids generic AI aesthetics.\nlicense: Complete terms in LICENSE.txt\n---\n\nThis skill guides creation of distinctive, production-grade frontend interfaces that avoid generic \"AI slop\" aesthetics. Implement real working code with exceptional attention to aesthetic details and creative choices.\n\nThe user provides frontend requirements: a component, page, application, or interface to build. They may include context about the purpose, audience, or technical constraints.\n\n## Design Thinking\n\nBefore coding, understand the context and commit to a BOLD aesthetic direction:\n- **Purpose**: What problem does this interface solve? Who uses it?\n- **Tone**: Pick an extreme: brutally minimal, maximalist chaos, retro-futuristic, organic/natural, luxury/refined, playful/toy-like, editorial/magazine, brutalist/raw, art deco/geometric, soft/pastel, industrial/utilitarian, etc. There are so many flavors to choose from. Use these for inspiration but design one that is true to the aesthetic direction.\n- **Constraints**: Technical requirements (framework, performance, accessibility).\n- **Differentiation**: What makes this UNFORGETTABLE? What's the one thing someone will remember?\n\n**CRITICAL**: Choose a clear conceptual direction and execute it with precision. Bold maximalism and refined minimalism both work - the key is intentionality, not intensity.\n\nThen implement working code (HTML/CSS/JS, React, Vue, etc.) that is:\n- Production-grade and functional\n- Visually striking and memorable\n- Cohesive with a clear aesthetic point-of-view\n- Meticulously refined in every detail\n\n## Frontend Aesthetics Guidelines\n\nFocus on:\n- **Typography**: Choose fonts that are beautiful, unique, and interesting. Avoid generic fonts like Arial and Inter; opt instead for distinctive choices that elevate the frontend's aesthetics; unexpected, characterful font choices. Pair a distinctive display font with a refined body font.\n- **Color & Theme**: Commit to a cohesive aesthetic. Use CSS variables for consistency. Dominant colors with sharp accents outperform timid, evenly-distributed palettes.\n- **Motion**: Use animations for effects and micro-interactions. Prioritize CSS-only solutions for HTML. Use Motion library for React when available. Focus on high-impact moments: one well-orchestrated page load with staggered reveals (animation-delay) creates more delight than scattered micro-interactions. Use scroll-triggering and hover states that surprise.\n- **Spatial Composition**: Unexpected layouts. Asymmetry. Overlap. Diagonal flow. Grid-breaking elements. Generous negative space OR controlled density.\n- **Backgrounds & Visual Details**: Create atmosphere and depth rather than defaulting to solid colors. Add contextual effects and textures that match the overall aesthetic. Apply creative forms like gradient meshes, noise textures, geometric patterns, layered transparencies, dramatic shadows, decorative borders, custom cursors, and grain overlays.\n\nNEVER use generic AI-generated aesthetics like overused font families (Inter, Roboto, Arial, system fonts), cliched color schemes (particularly purple gradients on white backgrounds), predictable layouts and component patterns, and cookie-cutter design that lacks context-specific character.\n\nInterpret creatively and make unexpected choices that feel genuinely designed for the context. No design should be the same. Vary between light and dark themes, different fonts, different aesthetics. NEVER converge on common choices (Space Grotesk, for example) across generations.\n\n**IMPORTANT**: Match implementation complexity to the aesthetic vision. Maximalist designs need elaborate code with extensive animations and effects. Minimalist or refined designs need restraint, precision, and careful attention to spacing, typography, and subtle details. Elegance comes from executing the vision well.\n\nRemember: Claude is capable of extraordinary creative work. Don't hold back, show what can truly be created when thinking outside the box and committing fully to a distinctive vision.","flock-data/skills/karpathy/SKILL.md":"---\nname: karpathy\ndescription: 'Coding principles for simplicity, clarity, and surgical precision. Use when you want Claude to think before coding, keep solutions minimal, make surgical changes, and define verifiable success criteria. Triggers on: karpathy mode, think before coding, simplicity first, surgical changes.'\n---\n\n# Karpathy Coding Principles\n\nPrinciples for writing clean, minimal, and purposeful code.\n\n---\n\n## 1. Think Before Coding\n\n**Don't assume. Don't hide confusion. Surface tradeoffs.**\n\nBefore implementing:\n\n- State your assumptions explicitly. If uncertain, ask.\n- If multiple interpretations exist, present them - don't pick silently.\n- If a simpler approach exists, say so. Push back when warranted.\n- If something is unclear, stop. Name what's confusing. Ask.\n\n---\n\n## 2. Simplicity First\n\n**Minimum code that solves the problem. Nothing speculative.**\n\n- No features beyond what was asked.\n- No abstractions for single-use code.\n- No \"flexibility\" or \"configurability\" that wasn't requested.\n- No error handling for impossible scenarios.\n- If you write 200 lines and it could be 50, rewrite it.\n\nAsk yourself: \"Would a senior engineer say this is overcomplicated?\" If yes, simplify.\n\n---\n\n## 3. Surgical Changes\n\n**Touch only what you must. Clean up only your own mess.**\n\nWhen editing existing code:\n\n- Don't \"improve\" adjacent code, comments, or formatting.\n- Don't refactor things that aren't broken.\n- Match existing style, even if you'd do it differently.\n- If you notice unrelated dead code, mention it - don't delete it.\n\nWhen your changes create orphans:\n\n- Remove imports/variables/functions that YOUR changes made unused.\n- Don't remove pre-existing dead code unless asked.\n\nThe test: Every changed line should trace directly to the user's request.\n\n---\n\n## 4. Goal-Driven Execution\n\n**Define success criteria. Loop until verified.**\n\nTransform tasks into verifiable goals:\n\n- \"Add validation\" -> \"Write tests for invalid inputs, then make them pass\"\n- \"Fix the bug\" -> \"Write a test that reproduces it, then make it pass\"\n- \"Refactor X\" -> \"Ensure tests pass before and after\"\n\nFor multi-step tasks, state a brief plan:\n\n```\n1. [Step] -> verify: [check]\n2. [Step] -> verify: [check]\n3. [Step] -> verify: [check]\n```\n\nStrong success criteria let you loop independently. Weak criteria (\"make it work\") require constant clarification.\n","flock-data/skills/karpathy/metadata.json":"{\n \"name\": \"karpathy\",\n \"description\": \"Coding principles for simplicity, clarity, and surgical precision. Use when you want Claude to think before coding, keep solutions minimal, make surgical changes, and define verifiable success criteria. Triggers on: karpathy mode, think before coding, simplicity first, surgical changes.\",\n \"category\": \"devops\",\n \"canonical_category\": \"devops\",\n \"repository\": \"majiayu000/claude-skill-registry\",\n \"repository_url\": \"https://github.com/majiayu000/claude-skill-registry\",\n \"author\": \"majiayu000\",\n \"author_avatar\": \"https://github.com/majiayu000.png\",\n \"file_path\": \"skills/other/karpathy/SKILL.md\",\n \"source\": \"github_code_search\",\n \"stars\": 0,\n \"quality_score\": 45,\n \"best_practices_score\": 55,\n \"skill_level\": 2,\n \"skill_level_label\": \"instructions\",\n \"has_scripts\": false,\n \"has_extra_files\": true,\n \"downloads\": 0,\n \"content_hash\": \"ec43cf3b4ae428984a1ad36d4814bf6f95e590ada0c997fab9f1ecb4c89b2886\",\n \"indexed_at\": \"2026-02-25T19:40:17.443Z\",\n \"synced_at\": \"2026-02-28T04:09:42.439Z\",\n \"omni_registry_url\": \"https://omni-skill-registry.omniroute.online/#/skill/27fec53067f2a7ff89a65128ed8a407f2e5f7a042d2891790054748048e9f33b\",\n \"install_command\": \"mkdir -p .claude/skills/karpathy && curl -sL \\\"https://raw.githubusercontent.com/majiayu000/claude-skill-registry/main/skills/other/karpathy/SKILL.md\\\" > .claude/skills/karpathy/SKILL.md\",\n \"raw_url\": \"https://raw.githubusercontent.com/majiayu000/claude-skill-registry/main/skills/other/karpathy/SKILL.md\"\n}\n","flock-data/skills/planning-with-files/SKILL.md":"---\nname: planning-with-files\nversion: \"2.10.0\"\ndescription: Implements Manus-style file-based planning for complex tasks. Creates task_plan.md, findings.md, and progress.md. Use when starting complex multi-step tasks, research projects, or any task requiring >5 tool calls. Now with automatic session recovery after /clear.\nuser-invocable: true\nallowed-tools:\n - Read\n - Write\n - Edit\n - Bash\n - Glob\n - Grep\n - WebFetch\n - WebSearch\nhooks:\n PreToolUse:\n - matcher: \"Write|Edit|Bash|Read|Glob|Grep\"\n hooks:\n - type: command\n command: \"cat task_plan.md 2>/dev/null | head -30 || true\"\n PostToolUse:\n - matcher: \"Write|Edit\"\n hooks:\n - type: command\n command: \"echo '[planning-with-files] File updated. If this completes a phase, update task_plan.md status.'\"\n Stop:\n - hooks:\n - type: command\n command: |\n SCRIPT_DIR=\"${CLAUDE_PLUGIN_ROOT:-$HOME/.claude/plugins/planning-with-files}/scripts\"\n\n IS_WINDOWS=0\n if [ \"${OS-}\" = \"Windows_NT\" ]; then\n IS_WINDOWS=1\n else\n UNAME_S=\"$(uname -s 2>/dev/null || echo '')\"\n case \"$UNAME_S\" in\n CYGWIN*|MINGW*|MSYS*) IS_WINDOWS=1 ;;\n esac\n fi\n\n if [ \"$IS_WINDOWS\" -eq 1 ]; then\n if command -v pwsh >/dev/null 2>&1; then\n pwsh -ExecutionPolicy Bypass -File \"$SCRIPT_DIR/check-complete.ps1\" 2>/dev/null ||\n powershell -ExecutionPolicy Bypass -File \"$SCRIPT_DIR/check-complete.ps1\" 2>/dev/null ||\n sh \"$SCRIPT_DIR/check-complete.sh\"\n else\n powershell -ExecutionPolicy Bypass -File \"$SCRIPT_DIR/check-complete.ps1\" 2>/dev/null ||\n sh \"$SCRIPT_DIR/check-complete.sh\"\n fi\n else\n sh \"$SCRIPT_DIR/check-complete.sh\"\n fi\n---\n\n# Planning with Files\n\nWork like Manus: Use persistent markdown files as your \"working memory on disk.\"\n\n## FIRST: Check for Previous Session (v2.2.0)\n\n**Before starting work**, check for unsynced context from a previous session:\n\n```bash\n# Linux/macOS\n$(command -v python3 || command -v python) ${CLAUDE_PLUGIN_ROOT}/scripts/session-catchup.py \"$(pwd)\"\n```\n\n```powershell\n# Windows PowerShell\n& (Get-Command python -ErrorAction SilentlyContinue).Source \"$env:USERPROFILE\\.claude\\skills\\planning-with-files\\scripts\\session-catchup.py\" (Get-Location)\n```\n\nIf catchup report shows unsynced context:\n1. Run `git diff --stat` to see actual code changes\n2. Read current planning files\n3. Update planning files based on catchup + git diff\n4. Then proceed with task\n\n## Important: Where Files Go\n\n- **Templates** are in `${CLAUDE_PLUGIN_ROOT}/templates/`\n- **Your planning files** go in **your project directory**\n\n| Location | What Goes There |\n|----------|-----------------|\n| Skill directory (`${CLAUDE_PLUGIN_ROOT}/`) | Templates, scripts, reference docs |\n| Your project directory | `task_plan.md`, `findings.md`, `progress.md` |\n\n## Quick Start\n\nBefore ANY complex task:\n\n1. **Create `task_plan.md`** — Use [templates/task_plan.md](templates/task_plan.md) as reference\n2. **Create `findings.md`** — Use [templates/findings.md](templates/findings.md) as reference\n3. **Create `progress.md`** — Use [templates/progress.md](templates/progress.md) as reference\n4. **Re-read plan before decisions** — Refreshes goals in attention window\n5. **Update after each phase** — Mark complete, log errors\n\n> **Note:** Planning files go in your project root, not the skill installation folder.\n\n## The Core Pattern\n\n```\nContext Window = RAM (volatile, limited)\nFilesystem = Disk (persistent, unlimited)\n\n→ Anything important gets written to disk.\n```\n\n## File Purposes\n\n| File | Purpose | When to Update |\n|------|---------|----------------|\n| `task_plan.md` | Phases, progress, decisions | After each phase |\n| `findings.md` | Research, discoveries | After ANY discovery |\n| `progress.md` | Session log, test results | Throughout session |\n\n## Critical Rules\n\n### 1. Create Plan First\nNever start a complex task without `task_plan.md`. Non-negotiable.\n\n### 2. The 2-Action Rule\n> \"After every 2 view/browser/search operations, IMMEDIATELY save key findings to text files.\"\n\nThis prevents visual/multimodal information from being lost.\n\n### 3. Read Before Decide\nBefore major decisions, read the plan file. This keeps goals in your attention window.\n\n### 4. Update After Act\nAfter completing any phase:\n- Mark phase status: `in_progress` → `complete`\n- Log any errors encountered\n- Note files created/modified\n\n### 5. Log ALL Errors\nEvery error goes in the plan file. This builds knowledge and prevents repetition.\n\n```markdown\n## Errors Encountered\n| Error | Attempt | Resolution |\n|-------|---------|------------|\n| FileNotFoundError | 1 | Created default config |\n| API timeout | 2 | Added retry logic |\n```\n\n### 6. Never Repeat Failures\n```\nif action_failed:\n next_action != same_action\n```\nTrack what you tried. Mutate the approach.\n\n## The 3-Strike Error Protocol\n\n```\nATTEMPT 1: Diagnose & Fix\n → Read error carefully\n → Identify root cause\n → Apply targeted fix\n\nATTEMPT 2: Alternative Approach\n → Same error? Try different method\n → Different tool? Different library?\n → NEVER repeat exact same failing action\n\nATTEMPT 3: Broader Rethink\n → Question assumptions\n → Search for solutions\n → Consider updating the plan\n\nAFTER 3 FAILURES: Escalate to User\n → Explain what you tried\n → Share the specific error\n → Ask for guidance\n```\n\n## Read vs Write Decision Matrix\n\n| Situation | Action | Reason |\n|-----------|--------|--------|\n| Just wrote a file | DON'T read | Content still in context |\n| Viewed image/PDF | Write findings NOW | Multimodal → text before lost |\n| Browser returned data | Write to file | Screenshots don't persist |\n| Starting new phase | Read plan/findings | Re-orient if context stale |\n| Error occurred | Read relevant file | Need current state to fix |\n| Resuming after gap | Read all planning files | Recover state |\n\n## The 5-Question Reboot Test\n\nIf you can answer these, your context management is solid:\n\n| Question | Answer Source |\n|----------|---------------|\n| Where am I? | Current phase in task_plan.md |\n| Where am I going? | Remaining phases |\n| What's the goal? | Goal statement in plan |\n| What have I learned? | findings.md |\n| What have I done? | progress.md |\n\n## When to Use This Pattern\n\n**Use for:**\n- Multi-step tasks (3+ steps)\n- Research tasks\n- Building/creating projects\n- Tasks spanning many tool calls\n- Anything requiring organization\n\n**Skip for:**\n- Simple questions\n- Single-file edits\n- Quick lookups\n\n## Templates\n\nCopy these templates to start:\n\n- [templates/task_plan.md](templates/task_plan.md) — Phase tracking\n- [templates/findings.md](templates/findings.md) — Research storage\n- [templates/progress.md](templates/progress.md) — Session logging\n\n## Scripts\n\nHelper scripts for automation:\n\n- `scripts/init-session.sh` — Initialize all planning files\n- `scripts/check-complete.sh` — Verify all phases complete\n- `scripts/session-catchup.py` — Recover context from previous session (v2.2.0)\n\n## Advanced Topics\n\n- **Manus Principles:** See [reference.md](reference.md)\n- **Real Examples:** See [examples.md](examples.md)\n\n## Anti-Patterns\n\n| Don't | Do Instead |\n|-------|------------|\n| Use TodoWrite for persistence | Create task_plan.md file |\n| State goals once and forget | Re-read plan before decisions |\n| Hide errors and retry silently | Log errors to plan file |\n| Stuff everything in context | Store large content in files |\n| Start executing immediately | Create plan file FIRST |\n| Repeat failed actions | Track attempts, mutate approach |\n| Create files in skill directory | Create files in your project |\n","flock-data/skills/planning-with-files/examples.md":"# Examples: Planning with Files in Action\n\n## Example 1: Research Task\n\n**User Request:** \"Research the benefits of morning exercise and write a summary\"\n\n### Loop 1: Create Plan\n```bash\nWrite task_plan.md\n```\n\n```markdown\n# Task Plan: Morning Exercise Benefits Research\n\n## Goal\nCreate a research summary on the benefits of morning exercise.\n\n## Phases\n- [ ] Phase 1: Create this plan ✓\n- [ ] Phase 2: Search and gather sources\n- [ ] Phase 3: Synthesize findings\n- [ ] Phase 4: Deliver summary\n\n## Key Questions\n1. What are the physical health benefits?\n2. What are the mental health benefits?\n3. What scientific studies support this?\n\n## Status\n**Currently in Phase 1** - Creating plan\n```\n\n### Loop 2: Research\n```bash\nRead task_plan.md # Refresh goals\nWebSearch \"morning exercise benefits\"\nWrite notes.md # Store findings\nEdit task_plan.md # Mark Phase 2 complete\n```\n\n### Loop 3: Synthesize\n```bash\nRead task_plan.md # Refresh goals\nRead notes.md # Get findings\nWrite morning_exercise_summary.md\nEdit task_plan.md # Mark Phase 3 complete\n```\n\n### Loop 4: Deliver\n```bash\nRead task_plan.md # Verify complete\nDeliver morning_exercise_summary.md\n```\n\n---\n\n## Example 2: Bug Fix Task\n\n**User Request:** \"Fix the login bug in the authentication module\"\n\n### task_plan.md\n```markdown\n# Task Plan: Fix Login Bug\n\n## Goal\nIdentify and fix the bug preventing successful login.\n\n## Phases\n- [x] Phase 1: Understand the bug report ✓\n- [x] Phase 2: Locate relevant code ✓\n- [ ] Phase 3: Identify root cause (CURRENT)\n- [ ] Phase 4: Implement fix\n- [ ] Phase 5: Test and verify\n\n## Key Questions\n1. What error message appears?\n2. Which file handles authentication?\n3. What changed recently?\n\n## Decisions Made\n- Auth handler is in src/auth/login.ts\n- Error occurs in validateToken() function\n\n## Errors Encountered\n- [Initial] TypeError: Cannot read property 'token' of undefined\n → Root cause: user object not awaited properly\n\n## Status\n**Currently in Phase 3** - Found root cause, preparing fix\n```\n\n---\n\n## Example 3: Feature Development\n\n**User Request:** \"Add a dark mode toggle to the settings page\"\n\n### The 3-File Pattern in Action\n\n**task_plan.md:**\n```markdown\n# Task Plan: Dark Mode Toggle\n\n## Goal\nAdd functional dark mode toggle to settings.\n\n## Phases\n- [x] Phase 1: Research existing theme system ✓\n- [x] Phase 2: Design implementation approach ✓\n- [ ] Phase 3: Implement toggle component (CURRENT)\n- [ ] Phase 4: Add theme switching logic\n- [ ] Phase 5: Test and polish\n\n## Decisions Made\n- Using CSS custom properties for theme\n- Storing preference in localStorage\n- Toggle component in SettingsPage.tsx\n\n## Status\n**Currently in Phase 3** - Building toggle component\n```\n\n**notes.md:**\n```markdown\n# Notes: Dark Mode Implementation\n\n## Existing Theme System\n- Located in: src/styles/theme.ts\n- Uses: CSS custom properties\n- Current themes: light only\n\n## Files to Modify\n1. src/styles/theme.ts - Add dark theme colors\n2. src/components/SettingsPage.tsx - Add toggle\n3. src/hooks/useTheme.ts - Create new hook\n4. src/App.tsx - Wrap with ThemeProvider\n\n## Color Decisions\n- Dark background: #1a1a2e\n- Dark surface: #16213e\n- Dark text: #eaeaea\n```\n\n**dark_mode_implementation.md:** (deliverable)\n```markdown\n# Dark Mode Implementation\n\n## Changes Made\n\n### 1. Added dark theme colors\nFile: src/styles/theme.ts\n...\n\n### 2. Created useTheme hook\nFile: src/hooks/useTheme.ts\n...\n```\n\n---\n\n## Example 4: Error Recovery Pattern\n\nWhen something fails, DON'T hide it:\n\n### Before (Wrong)\n```\nAction: Read config.json\nError: File not found\nAction: Read config.json # Silent retry\nAction: Read config.json # Another retry\n```\n\n### After (Correct)\n```\nAction: Read config.json\nError: File not found\n\n# Update task_plan.md:\n## Errors Encountered\n- config.json not found → Will create default config\n\nAction: Write config.json (default config)\nAction: Read config.json\nSuccess!\n```\n\n---\n\n## The Read-Before-Decide Pattern\n\n**Always read your plan before major decisions:**\n\n```\n[Many tool calls have happened...]\n[Context is getting long...]\n[Original goal might be forgotten...]\n\n→ Read task_plan.md # This brings goals back into attention!\n→ Now make the decision # Goals are fresh in context\n```\n\nThis is why Manus can handle ~50 tool calls without losing track. The plan file acts as a \"goal refresh\" mechanism.\n","flock-data/skills/planning-with-files/reference.md":"# Reference: Manus Context Engineering Principles\n\nThis skill is based on context engineering principles from Manus, the AI agent company acquired by Meta for $2 billion in December 2025.\n\n## The 6 Manus Principles\n\n### Principle 1: Design Around KV-Cache\n\n> \"KV-cache hit rate is THE single most important metric for production AI agents.\"\n\n**Statistics:**\n- ~100:1 input-to-output token ratio\n- Cached tokens: $0.30/MTok vs Uncached: $3/MTok\n- 10x cost difference!\n\n**Implementation:**\n- Keep prompt prefixes STABLE (single-token change invalidates cache)\n- NO timestamps in system prompts\n- Make context APPEND-ONLY with deterministic serialization\n\n### Principle 2: Mask, Don't Remove\n\nDon't dynamically remove tools (breaks KV-cache). Use logit masking instead.\n\n**Best Practice:** Use consistent action prefixes (e.g., `browser_`, `shell_`, `file_`) for easier masking.\n\n### Principle 3: Filesystem as External Memory\n\n> \"Markdown is my 'working memory' on disk.\"\n\n**The Formula:**\n```\nContext Window = RAM (volatile, limited)\nFilesystem = Disk (persistent, unlimited)\n```\n\n**Compression Must Be Restorable:**\n- Keep URLs even if web content is dropped\n- Keep file paths when dropping document contents\n- Never lose the pointer to full data\n\n### Principle 4: Manipulate Attention Through Recitation\n\n> \"Creates and updates todo.md throughout tasks to push global plan into model's recent attention span.\"\n\n**Problem:** After ~50 tool calls, models forget original goals (\"lost in the middle\" effect).\n\n**Solution:** Re-read `task_plan.md` before each decision. Goals appear in the attention window.\n\n```\nStart of context: [Original goal - far away, forgotten]\n...many tool calls...\nEnd of context: [Recently read task_plan.md - gets ATTENTION!]\n```\n\n### Principle 5: Keep the Wrong Stuff In\n\n> \"Leave the wrong turns in the context.\"\n\n**Why:**\n- Failed actions with stack traces let model implicitly update beliefs\n- Reduces mistake repetition\n- Error recovery is \"one of the clearest signals of TRUE agentic behavior\"\n\n### Principle 6: Don't Get Few-Shotted\n\n> \"Uniformity breeds fragility.\"\n\n**Problem:** Repetitive action-observation pairs cause drift and hallucination.\n\n**Solution:** Introduce controlled variation:\n- Vary phrasings slightly\n- Don't copy-paste patterns blindly\n- Recalibrate on repetitive tasks\n\n---\n\n## The 3 Context Engineering Strategies\n\nBased on Lance Martin's analysis of Manus architecture.\n\n### Strategy 1: Context Reduction\n\n**Compaction:**\n```\nTool calls have TWO representations:\n├── FULL: Raw tool content (stored in filesystem)\n└── COMPACT: Reference/file path only\n\nRULES:\n- Apply compaction to STALE (older) tool results\n- Keep RECENT results FULL (to guide next decision)\n```\n\n**Summarization:**\n- Applied when compaction reaches diminishing returns\n- Generated using full tool results\n- Creates standardized summary objects\n\n### Strategy 2: Context Isolation (Multi-Agent)\n\n**Architecture:**\n```\n┌─────────────────────────────────┐\n│ PLANNER AGENT │\n│ └─ Assigns tasks to sub-agents │\n├─────────────────────────────────┤\n│ KNOWLEDGE MANAGER │\n│ └─ Reviews conversations │\n│ └─ Determines filesystem store │\n├─────────────────────────────────┤\n│ EXECUTOR SUB-AGENTS │\n│ └─ Perform assigned tasks │\n│ └─ Have own context windows │\n└─────────────────────────────────┘\n```\n\n**Key Insight:** Manus originally used `todo.md` for task planning but found ~33% of actions were spent updating it. Shifted to dedicated planner agent calling executor sub-agents.\n\n### Strategy 3: Context Offloading\n\n**Tool Design:**\n- Use <20 atomic functions total\n- Store full results in filesystem, not context\n- Use `glob` and `grep` for searching\n- Progressive disclosure: load information only as needed\n\n---\n\n## The Agent Loop\n\nManus operates in a continuous 7-step loop:\n\n```\n┌─────────────────────────────────────────┐\n│ 1. ANALYZE CONTEXT │\n│ - Understand user intent │\n│ - Assess current state │\n│ - Review recent observations │\n├─────────────────────────────────────────┤\n│ 2. THINK │\n│ - Should I update the plan? │\n│ - What's the next logical action? │\n│ - Are there blockers? │\n├─────────────────────────────────────────┤\n│ 3. SELECT TOOL │\n│ - Choose ONE tool │\n│ - Ensure parameters available │\n├─────────────────────────────────────────┤\n│ 4. EXECUTE ACTION │\n│ - Tool runs in sandbox │\n├─────────────────────────────────────────┤\n│ 5. RECEIVE OBSERVATION │\n│ - Result appended to context │\n├─────────────────────────────────────────┤\n│ 6. ITERATE │\n│ - Return to step 1 │\n│ - Continue until complete │\n├─────────────────────────────────────────┤\n│ 7. DELIVER OUTCOME │\n│ - Send results to user │\n│ - Attach all relevant files │\n└─────────────────────────────────────────┘\n```\n\n---\n\n## File Types Manus Creates\n\n| File | Purpose | When Created | When Updated |\n|------|---------|--------------|--------------|\n| `task_plan.md` | Phase tracking, progress | Task start | After completing phases |\n| `findings.md` | Discoveries, decisions | After ANY discovery | After viewing images/PDFs |\n| `progress.md` | Session log, what's done | At breakpoints | Throughout session |\n| Code files | Implementation | Before execution | After errors |\n\n---\n\n## Critical Constraints\n\n- **Single-Action Execution:** ONE tool call per turn. No parallel execution.\n- **Plan is Required:** Agent must ALWAYS know: goal, current phase, remaining phases\n- **Files are Memory:** Context = volatile. Filesystem = persistent.\n- **Never Repeat Failures:** If action failed, next action MUST be different\n- **Communication is a Tool:** Message types: `info` (progress), `ask` (blocking), `result` (terminal)\n\n---\n\n## Manus Statistics\n\n| Metric | Value |\n|--------|-------|\n| Average tool calls per task | ~50 |\n| Input-to-output token ratio | 100:1 |\n| Acquisition price | $2 billion |\n| Time to $100M revenue | 8 months |\n| Framework refactors since launch | 5 times |\n\n---\n\n## Key Quotes\n\n> \"Context window = RAM (volatile, limited). Filesystem = Disk (persistent, unlimited). Anything important gets written to disk.\"\n\n> \"if action_failed: next_action != same_action. Track what you tried. Mutate the approach.\"\n\n> \"Error recovery is one of the clearest signals of TRUE agentic behavior.\"\n\n> \"KV-cache hit rate is the single most important metric for a production-stage AI agent.\"\n\n> \"Leave the wrong turns in the context.\"\n\n---\n\n## Source\n\nBased on Manus's official context engineering documentation:\nhttps://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus\n","flock-data/skills/planning-with-files/templates/findings.md":"# Findings & Decisions\n\n\n## Requirements\n\n\n-\n\n## Research Findings\n\n\n-\n\n## Technical Decisions\n\n\n| Decision | Rationale |\n|----------|-----------|\n| | |\n\n## Issues Encountered\n\n\n| Issue | Resolution |\n|-------|------------|\n| | |\n\n## Resources\n\n\n-\n\n## Visual/Browser Findings\n\n\n\n-\n\n---\n\n*Update this file after every 2 view/browser/search operations*\n*This prevents visual information from being lost*\n","flock-data/skills/planning-with-files/templates/progress.md":"# Progress Log\n\n\n## Session: [DATE]\n\n\n### Phase 1: [Title]\n\n- **Status:** in_progress\n- **Started:** [timestamp]\n\n- Actions taken:\n \n -\n- Files created/modified:\n \n -\n\n### Phase 2: [Title]\n\n- **Status:** pending\n- Actions taken:\n -\n- Files created/modified:\n -\n\n## Test Results\n\n| Test | Input | Expected | Actual | Status |\n|------|-------|----------|--------|--------|\n| | | | | |\n\n## Error Log\n\n\n| Timestamp | Error | Attempt | Resolution |\n|-----------|-------|---------|------------|\n| | | 1 | |\n\n## 5-Question Reboot Check\n\n\n| Question | Answer |\n|----------|--------|\n| Where am I? | Phase X |\n| Where am I going? | Remaining phases |\n| What's the goal? | [goal statement] |\n| What have I learned? | See findings.md |\n| What have I done? | See above |\n\n---\n\n*Update after completing each phase or encountering errors*\n","flock-data/skills/planning-with-files/templates/task_plan.md":"# Task Plan: [Brief Description]\n\n\n## Goal\n\n[One sentence describing the end state]\n\n## Current Phase\n\nPhase 1\n\n## Phases\n\n\n### Phase 1: Requirements & Discovery\n\n- [ ] Understand user intent\n- [ ] Identify constraints and requirements\n- [ ] Document findings in findings.md\n- **Status:** in_progress\n\n\n### Phase 2: Planning & Structure\n\n- [ ] Define technical approach\n- [ ] Create project structure if needed\n- [ ] Document decisions with rationale\n- **Status:** pending\n\n### Phase 3: Implementation\n\n- [ ] Execute the plan step by step\n- [ ] Write code to files before executing\n- [ ] Test incrementally\n- **Status:** pending\n\n### Phase 4: Testing & Verification\n\n- [ ] Verify all requirements met\n- [ ] Document test results in progress.md\n- [ ] Fix any issues found\n- **Status:** pending\n\n### Phase 5: Delivery\n\n- [ ] Review all output files\n- [ ] Ensure deliverables are complete\n- [ ] Deliver to user\n- **Status:** pending\n\n## Key Questions\n\n1. [Question to answer]\n2. [Question to answer]\n\n## Decisions Made\n\n| Decision | Rationale |\n|----------|-----------|\n| | |\n\n## Errors Encountered\n\n| Error | Attempt | Resolution |\n|-------|---------|------------|\n| | 1 | |\n\n## Notes\n\n- Update phase status as you progress: pending → in_progress → complete\n- Re-read this plan before major decisions (attention manipulation)\n- Log ALL errors - they help avoid repetition\n","flock-data/skills/skill-creator/SKILL.md":"---\nname: skill-creator\ndescription: Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.\n---\n\n# Skill Creator\n\nA skill for creating new skills and iteratively improving them.\n\nAt a high level, the process of creating a skill goes like this:\n\n- Decide what you want the skill to do and roughly how it should do it\n- Write a draft of the skill\n- Create a few test prompts and run claude-with-access-to-the-skill on them\n- Help the user evaluate the results both qualitatively and quantitatively\n - While the runs happen in the background, draft some quantitative evals if there aren't any (if there are some, you can either use as is or modify if you feel something needs to change about them). Then explain them to the user (or if they already existed, explain the ones that already exist)\n - Use the `eval-viewer/generate_review.py` script to show the user the results for them to look at, and also let them look at the quantitative metrics\n- Rewrite the skill based on feedback from the user's evaluation of the results (and also if there are any glaring flaws that become apparent from the quantitative benchmarks)\n- Repeat until you're satisfied\n- Expand the test set and try again at larger scale\n\nYour job when using this skill is to figure out where the user is in this process and then jump in and help them progress through these stages. So for instance, maybe they're like \"I want to make a skill for X\". You can help narrow down what they mean, write a draft, write the test cases, figure out how they want to evaluate, run all the prompts, and repeat.\n\nOn the other hand, maybe they already have a draft of the skill. In this case you can go straight to the eval/iterate part of the loop.\n\nOf course, you should always be flexible and if the user is like \"I don't need to run a bunch of evaluations, just vibe with me\", you can do that instead.\n\nThen after the skill is done (but again, the order is flexible), you can also run the skill description improver, which we have a whole separate script for, to optimize the triggering of the skill.\n\nCool? Cool.\n\n## Communicating with the user\n\nThe skill creator is liable to be used by people across a wide range of familiarity with coding jargon. If you haven't heard (and how could you, it's only very recently that it started), there's a trend now where the power of Claude is inspiring plumbers to open up their terminals, parents and grandparents to google \"how to install npm\". On the other hand, the bulk of users are probably fairly computer-literate.\n\nSo please pay attention to context cues to understand how to phrase your communication! In the default case, just to give you some idea:\n\n- \"evaluation\" and \"benchmark\" are borderline, but OK\n- for \"JSON\" and \"assertion\" you want to see serious cues from the user that they know what those things are before using them without explaining them\n\nIt's OK to briefly explain terms if you're in doubt, and feel free to clarify terms with a short definition if you're unsure if the user will get it.\n\n---\n\n## Creating a skill\n\n### Capture Intent\n\nStart by understanding the user's intent. The current conversation might already contain a workflow the user wants to capture (e.g., they say \"turn this into a skill\"). If so, extract answers from the conversation history first — the tools used, the sequence of steps, corrections the user made, input/output formats observed. The user may need to fill the gaps, and should confirm before proceeding to the next step.\n\n1. What should this skill enable Claude to do?\n2. When should this skill trigger? (what user phrases/contexts)\n3. What's the expected output format?\n4. Should we set up test cases to verify the skill works? Skills with objectively verifiable outputs (file transforms, data extraction, code generation, fixed workflow steps) benefit from test cases. Skills with subjective outputs (writing style, art) often don't need them. Suggest the appropriate default based on the skill type, but let the user decide.\n\n### Interview and Research\n\nProactively ask questions about edge cases, input/output formats, example files, success criteria, and dependencies. Wait to write test prompts until you've got this part ironed out.\n\nCheck available MCPs - if useful for research (searching docs, finding similar skills, looking up best practices), research in parallel via subagents if available, otherwise inline. Come prepared with context to reduce burden on the user.\n\n### Write the SKILL.md\n\nBased on the user interview, fill in these components:\n\n- **name**: Skill identifier\n- **description**: When to trigger, what it does. This is the primary triggering mechanism - include both what the skill does AND specific contexts for when to use it. All \"when to use\" info goes here, not in the body. Note: currently Claude has a tendency to \"undertrigger\" skills -- to not use them when they'd be useful. To combat this, please make the skill descriptions a little bit \"pushy\". So for instance, instead of \"How to build a simple fast dashboard to display internal Anthropic data.\", you might write \"How to build a simple fast dashboard to display internal Anthropic data. Make sure to use this skill whenever the user mentions dashboards, data visualization, internal metrics, or wants to display any kind of company data, even if they don't explicitly ask for a 'dashboard.'\"\n- **compatibility**: Required tools, dependencies (optional, rarely needed)\n- **the rest of the skill :)**\n\n### Skill Writing Guide\n\n#### Anatomy of a Skill\n\n```\nskill-name/\n├── SKILL.md (required)\n│ ├── YAML frontmatter (name, description required)\n│ └── Markdown instructions\n└── Bundled Resources (optional)\n ├── scripts/ - Executable code for deterministic/repetitive tasks\n ├── references/ - Docs loaded into context as needed\n └── assets/ - Files used in output (templates, icons, fonts)\n```\n\n#### Progressive Disclosure\n\nSkills use a three-level loading system:\n1. **Metadata** (name + description) - Always in context (~100 words)\n2. **SKILL.md body** - In context whenever skill triggers (<500 lines ideal)\n3. **Bundled resources** - As needed (unlimited, scripts can execute without loading)\n\nThese word counts are approximate and you can feel free to go longer if needed.\n\n**Key patterns:**\n- Keep SKILL.md under 500 lines; if you're approaching this limit, add an additional layer of hierarchy along with clear pointers about where the model using the skill should go next to follow up.\n- Reference files clearly from SKILL.md with guidance on when to read them\n- For large reference files (>300 lines), include a table of contents\n\n**Domain organization**: When a skill supports multiple domains/frameworks, organize by variant:\n```\ncloud-deploy/\n├── SKILL.md (workflow + selection)\n└── references/\n ├── aws.md\n ├── gcp.md\n └── azure.md\n```\nClaude reads only the relevant reference file.\n\n#### Principle of Lack of Surprise\n\nThis goes without saying, but skills must not contain malware, exploit code, or any content that could compromise system security. A skill's contents should not surprise the user in their intent if described. Don't go along with requests to create misleading skills or skills designed to facilitate unauthorized access, data exfiltration, or other malicious activities. Things like a \"roleplay as an XYZ\" are OK though.\n\n#### Writing Patterns\n\nPrefer using the imperative form in instructions.\n\n**Defining output formats** - You can do it like this:\n```markdown\n## Report structure\nALWAYS use this exact template:\n# [Title]\n## Executive summary\n## Key findings\n## Recommendations\n```\n\n**Examples pattern** - It's useful to include examples. You can format them like this (but if \"Input\" and \"Output\" are in the examples you might want to deviate a little):\n```markdown\n## Commit message format\n**Example 1:**\nInput: Added user authentication with JWT tokens\nOutput: feat(auth): implement JWT-based authentication\n```\n\n### Writing Style\n\nTry to explain to the model why things are important in lieu of heavy-handed musty MUSTs. Use theory of mind and try to make the skill general and not super-narrow to specific examples. Start by writing a draft and then look at it with fresh eyes and improve it.\n\n### Test Cases\n\nAfter writing the skill draft, come up with 2-3 realistic test prompts — the kind of thing a real user would actually say. Share them with the user: [you don't have to use this exact language] \"Here are a few test cases I'd like to try. Do these look right, or do you want to add more?\" Then run them.\n\nSave test cases to `evals/evals.json`. Don't write assertions yet — just the prompts. You'll draft assertions in the next step while the runs are in progress.\n\n```json\n{\n \"skill_name\": \"example-skill\",\n \"evals\": [\n {\n \"id\": 1,\n \"prompt\": \"User's task prompt\",\n \"expected_output\": \"Description of expected result\",\n \"files\": []\n }\n ]\n}\n```\n\nSee `references/schemas.md` for the full schema (including the `assertions` field, which you'll add later).\n\n## Running and evaluating test cases\n\nThis section is one continuous sequence — don't stop partway through. Do NOT use `/skill-test` or any other testing skill.\n\nPut results in `-workspace/` as a sibling to the skill directory. Within the workspace, organize results by iteration (`iteration-1/`, `iteration-2/`, etc.) and within that, each test case gets a directory (`eval-0/`, `eval-1/`, etc.). Don't create all of this upfront — just create directories as you go.\n\n### Step 1: Spawn all runs (with-skill AND baseline) in the same turn\n\nFor each test case, spawn two subagents in the same turn — one with the skill, one without. This is important: don't spawn the with-skill runs first and then come back for baselines later. Launch everything at once so it all finishes around the same time.\n\n**With-skill run:**\n\n```\nExecute this task:\n- Skill path: \n- Task: \n- Input files: \n- Save outputs to: /iteration-/eval-/with_skill/outputs/\n- Outputs to save: \n```\n\n**Baseline run** (same prompt, but the baseline depends on context):\n- **Creating a new skill**: no skill at all. Same prompt, no skill path, save to `without_skill/outputs/`.\n- **Improving an existing skill**: the old version. Before editing, snapshot the skill (`cp -r /skill-snapshot/`), then point the baseline subagent at the snapshot. Save to `old_skill/outputs/`.\n\nWrite an `eval_metadata.json` for each test case (assertions can be empty for now). Give each eval a descriptive name based on what it's testing — not just \"eval-0\". Use this name for the directory too. If this iteration uses new or modified eval prompts, create these files for each new eval directory — don't assume they carry over from previous iterations.\n\n```json\n{\n \"eval_id\": 0,\n \"eval_name\": \"descriptive-name-here\",\n \"prompt\": \"The user's task prompt\",\n \"assertions\": []\n}\n```\n\n### Step 2: While runs are in progress, draft assertions\n\nDon't just wait for the runs to finish — you can use this time productively. Draft quantitative assertions for each test case and explain them to the user. If assertions already exist in `evals/evals.json`, review them and explain what they check.\n\nGood assertions are objectively verifiable and have descriptive names — they should read clearly in the benchmark viewer so someone glancing at the results immediately understands what each one checks. Subjective skills (writing style, design quality) are better evaluated qualitatively — don't force assertions onto things that need human judgment.\n\nUpdate the `eval_metadata.json` files and `evals/evals.json` with the assertions once drafted. Also explain to the user what they'll see in the viewer — both the qualitative outputs and the quantitative benchmark.\n\n### Step 3: As runs complete, capture timing data\n\nWhen each subagent task completes, you receive a notification containing `total_tokens` and `duration_ms`. Save this data immediately to `timing.json` in the run directory:\n\n```json\n{\n \"total_tokens\": 84852,\n \"duration_ms\": 23332,\n \"total_duration_seconds\": 23.3\n}\n```\n\nThis is the only opportunity to capture this data — it comes through the task notification and isn't persisted elsewhere. Process each notification as it arrives rather than trying to batch them.\n\n### Step 4: Grade, aggregate, and launch the viewer\n\nOnce all runs are done:\n\n1. **Grade each run** — spawn a grader subagent (or grade inline) that reads `agents/grader.md` and evaluates each assertion against the outputs. Save results to `grading.json` in each run directory. The grading.json expectations array must use the fields `text`, `passed`, and `evidence` (not `name`/`met`/`details` or other variants) — the viewer depends on these exact field names. For assertions that can be checked programmatically, write and run a script rather than eyeballing it — scripts are faster, more reliable, and can be reused across iterations.\n\n2. **Aggregate into benchmark** — run the aggregation script from the skill-creator directory:\n ```bash\n python -m scripts.aggregate_benchmark /iteration-N --skill-name \n ```\n This produces `benchmark.json` and `benchmark.md` with pass_rate, time, and tokens for each configuration, with mean ± stddev and the delta. If generating benchmark.json manually, see `references/schemas.md` for the exact schema the viewer expects.\nPut each with_skill version before its baseline counterpart.\n\n3. **Do an analyst pass** — read the benchmark data and surface patterns the aggregate stats might hide. See `agents/analyzer.md` (the \"Analyzing Benchmark Results\" section) for what to look for — things like assertions that always pass regardless of skill (non-discriminating), high-variance evals (possibly flaky), and time/token tradeoffs.\n\n4. **Launch the viewer** with both qualitative outputs and quantitative data:\n ```bash\n nohup python /eval-viewer/generate_review.py \\\n /iteration-N \\\n --skill-name \"my-skill\" \\\n --benchmark /iteration-N/benchmark.json \\\n > /dev/null 2>&1 &\n VIEWER_PID=$!\n ```\n For iteration 2+, also pass `--previous-workspace /iteration-`.\n\n **Cowork / headless environments:** If `webbrowser.open()` is not available or the environment has no display, use `--static ` to write a standalone HTML file instead of starting a server. Feedback will be downloaded as a `feedback.json` file when the user clicks \"Submit All Reviews\". After download, copy `feedback.json` into the workspace directory for the next iteration to pick up.\n\nNote: please use generate_review.py to create the viewer; there's no need to write custom HTML.\n\n5. **Tell the user** something like: \"I've opened the results in your browser. There are two tabs — 'Outputs' lets you click through each test case and leave feedback, 'Benchmark' shows the quantitative comparison. When you're done, come back here and let me know.\"\n\n### What the user sees in the viewer\n\nThe \"Outputs\" tab shows one test case at a time:\n- **Prompt**: the task that was given\n- **Output**: the files the skill produced, rendered inline where possible\n- **Previous Output** (iteration 2+): collapsed section showing last iteration's output\n- **Formal Grades** (if grading was run): collapsed section showing assertion pass/fail\n- **Feedback**: a textbox that auto-saves as they type\n- **Previous Feedback** (iteration 2+): their comments from last time, shown below the textbox\n\nThe \"Benchmark\" tab shows the stats summary: pass rates, timing, and token usage for each configuration, with per-eval breakdowns and analyst observations.\n\nNavigation is via prev/next buttons or arrow keys. When done, they click \"Submit All Reviews\" which saves all feedback to `feedback.json`.\n\n### Step 5: Read the feedback\n\nWhen the user tells you they're done, read `feedback.json`:\n\n```json\n{\n \"reviews\": [\n {\"run_id\": \"eval-0-with_skill\", \"feedback\": \"the chart is missing axis labels\", \"timestamp\": \"...\"},\n {\"run_id\": \"eval-1-with_skill\", \"feedback\": \"\", \"timestamp\": \"...\"},\n {\"run_id\": \"eval-2-with_skill\", \"feedback\": \"perfect, love this\", \"timestamp\": \"...\"}\n ],\n \"status\": \"complete\"\n}\n```\n\nEmpty feedback means the user thought it was fine. Focus your improvements on the test cases where the user had specific complaints.\n\nKill the viewer server when you're done with it:\n\n```bash\nkill $VIEWER_PID 2>/dev/null\n```\n\n---\n\n## Improving the skill\n\nThis is the heart of the loop. You've run the test cases, the user has reviewed the results, and now you need to make the skill better based on their feedback.\n\n### How to think about improvements\n\n1. **Generalize from the feedback.** The big picture thing that's happening here is that we're trying to create skills that can be used a million times (maybe literally, maybe even more who knows) across many different prompts. Here you and the user are iterating on only a few examples over and over again because it helps move faster. The user knows these examples in and out and it's quick for them to assess new outputs. But if the skill you and the user are codeveloping works only for those examples, it's useless. Rather than put in fiddly overfitty changes, or oppressively constrictive MUSTs, if there's some stubborn issue, you might try branching out and using different metaphors, or recommending different patterns of working. It's relatively cheap to try and maybe you'll land on something great.\n\n2. **Keep the prompt lean.** Remove things that aren't pulling their weight. Make sure to read the transcripts, not just the final outputs — if it looks like the skill is making the model waste a bunch of time doing things that are unproductive, you can try getting rid of the parts of the skill that are making it do that and seeing what happens.\n\n3. **Explain the why.** Try hard to explain the **why** behind everything you're asking the model to do. Today's LLMs are *smart*. They have good theory of mind and when given a good harness can go beyond rote instructions and really make things happen. Even if the feedback from the user is terse or frustrated, try to actually understand the task and why the user is writing what they wrote, and what they actually wrote, and then transmit this understanding into the instructions. If you find yourself writing ALWAYS or NEVER in all caps, or using super rigid structures, that's a yellow flag — if possible, reframe and explain the reasoning so that the model understands why the thing you're asking for is important. That's a more humane, powerful, and effective approach.\n\n4. **Look for repeated work across test cases.** Read the transcripts from the test runs and notice if the subagents all independently wrote similar helper scripts or took the same multi-step approach to something. If all 3 test cases resulted in the subagent writing a `create_docx.py` or a `build_chart.py`, that's a strong signal the skill should bundle that script. Write it once, put it in `scripts/`, and tell the skill to use it. This saves every future invocation from reinventing the wheel.\n\nThis task is pretty important (we are trying to create billions a year in economic value here!) and your thinking time is not the blocker; take your time and really mull things over. I'd suggest writing a draft revision and then looking at it anew and making improvements. Really do your best to get into the head of the user and understand what they want and need.\n\n### The iteration loop\n\nAfter improving the skill:\n\n1. Apply your improvements to the skill\n2. Rerun all test cases into a new `iteration-/` directory, including baseline runs. If you're creating a new skill, the baseline is always `without_skill` (no skill) — that stays the same across iterations. If you're improving an existing skill, use your judgment on what makes sense as the baseline: the original version the user came in with, or the previous iteration.\n3. Launch the reviewer with `--previous-workspace` pointing at the previous iteration\n4. Wait for the user to review and tell you they're done\n5. Read the new feedback, improve again, repeat\n\nKeep going until:\n- The user says they're happy\n- The feedback is all empty (everything looks good)\n- You're not making meaningful progress\n\n---\n\n## Advanced: Blind comparison\n\nFor situations where you want a more rigorous comparison between two versions of a skill (e.g., the user asks \"is the new version actually better?\"), there's a blind comparison system. Read `agents/comparator.md` and `agents/analyzer.md` for the details. The basic idea is: give two outputs to an independent agent without telling it which is which, and let it judge quality. Then analyze why the winner won.\n\nThis is optional, requires subagents, and most users won't need it. The human review loop is usually sufficient.\n\n---\n\n## Description Optimization\n\nThe description field in SKILL.md frontmatter is the primary mechanism that determines whether Claude invokes a skill. After creating or improving a skill, offer to optimize the description for better triggering accuracy.\n\n### Step 1: Generate trigger eval queries\n\nCreate 20 eval queries — a mix of should-trigger and should-not-trigger. Save as JSON:\n\n```json\n[\n {\"query\": \"the user prompt\", \"should_trigger\": true},\n {\"query\": \"another prompt\", \"should_trigger\": false}\n]\n```\n\nThe queries must be realistic and something a Claude Code or Claude.ai user would actually type. Not abstract requests, but requests that are concrete and specific and have a good amount of detail. For instance, file paths, personal context about the user's job or situation, column names and values, company names, URLs. A little bit of backstory. Some might be in lowercase or contain abbreviations or typos or casual speech. Use a mix of different lengths, and focus on edge cases rather than making them clear-cut (the user will get a chance to sign off on them).\n\nBad: `\"Format this data\"`, `\"Extract text from PDF\"`, `\"Create a chart\"`\n\nGood: `\"ok so my boss just sent me this xlsx file (its in my downloads, called something like 'Q4 sales final FINAL v2.xlsx') and she wants me to add a column that shows the profit margin as a percentage. The revenue is in column C and costs are in column D i think\"`\n\nFor the **should-trigger** queries (8-10), think about coverage. You want different phrasings of the same intent — some formal, some casual. Include cases where the user doesn't explicitly name the skill or file type but clearly needs it. Throw in some uncommon use cases and cases where this skill competes with another but should win.\n\nFor the **should-not-trigger** queries (8-10), the most valuable ones are the near-misses — queries that share keywords or concepts with the skill but actually need something different. Think adjacent domains, ambiguous phrasing where a naive keyword match would trigger but shouldn't, and cases where the query touches on something the skill does but in a context where another tool is more appropriate.\n\nThe key thing to avoid: don't make should-not-trigger queries obviously irrelevant. \"Write a fibonacci function\" as a negative test for a PDF skill is too easy — it doesn't test anything. The negative cases should be genuinely tricky.\n\n### Step 2: Review with user\n\nPresent the eval set to the user for review using the HTML template:\n\n1. Read the template from `assets/eval_review.html`\n2. Replace the placeholders:\n - `__EVAL_DATA_PLACEHOLDER__` → the JSON array of eval items (no quotes around it — it's a JS variable assignment)\n - `__SKILL_NAME_PLACEHOLDER__` → the skill's name\n - `__SKILL_DESCRIPTION_PLACEHOLDER__` → the skill's current description\n3. Write to a temp file (e.g., `/tmp/eval_review_.html`) and open it: `open /tmp/eval_review_.html`\n4. The user can edit queries, toggle should-trigger, add/remove entries, then click \"Export Eval Set\"\n5. The file downloads to `~/Downloads/eval_set.json` — check the Downloads folder for the most recent version in case there are multiple (e.g., `eval_set (1).json`)\n\nThis step matters — bad eval queries lead to bad descriptions.\n\n### Step 3: Run the optimization loop\n\nTell the user: \"This will take some time — I'll run the optimization loop in the background and check on it periodically.\"\n\nSave the eval set to the workspace, then run in the background:\n\n```bash\npython -m scripts.run_loop \\\n --eval-set \\\n --skill-path \\\n --model \\\n --max-iterations 5 \\\n --verbose\n```\n\nUse the model ID from your system prompt (the one powering the current session) so the triggering test matches what the user actually experiences.\n\nWhile it runs, periodically tail the output to give the user updates on which iteration it's on and what the scores look like.\n\nThis handles the full optimization loop automatically. It splits the eval set into 60% train and 40% held-out test, evaluates the current description (running each query 3 times to get a reliable trigger rate), then calls Claude to propose improvements based on what failed. It re-evaluates each new description on both train and test, iterating up to 5 times. When it's done, it opens an HTML report in the browser showing the results per iteration and returns JSON with `best_description` — selected by test score rather than train score to avoid overfitting.\n\n### How skill triggering works\n\nUnderstanding the triggering mechanism helps design better eval queries. Skills appear in Claude's `available_skills` list with their name + description, and Claude decides whether to consult a skill based on that description. The important thing to know is that Claude only consults skills for tasks it can't easily handle on its own — simple, one-step queries like \"read this PDF\" may not trigger a skill even if the description matches perfectly, because Claude can handle them directly with basic tools. Complex, multi-step, or specialized queries reliably trigger skills when the description matches.\n\nThis means your eval queries should be substantive enough that Claude would actually benefit from consulting a skill. Simple queries like \"read file X\" are poor test cases — they won't trigger skills regardless of description quality.\n\n### Step 4: Apply the result\n\nTake `best_description` from the JSON output and update the skill's SKILL.md frontmatter. Show the user before/after and report the scores.\n\n---\n\n### Package and Present (only if `present_files` tool is available)\n\nCheck whether you have access to the `present_files` tool. If you don't, skip this step. If you do, package the skill and present the .skill file to the user:\n\n```bash\npython -m scripts.package_skill \n```\n\nAfter packaging, direct the user to the resulting `.skill` file path so they can install it.\n\n---\n\n## Claude.ai-specific instructions\n\nIn Claude.ai, the core workflow is the same (draft → test → review → improve → repeat), but because Claude.ai doesn't have subagents, some mechanics change. Here's what to adapt:\n\n**Running test cases**: No subagents means no parallel execution. For each test case, read the skill's SKILL.md, then follow its instructions to accomplish the test prompt yourself. Do them one at a time. This is less rigorous than independent subagents (you wrote the skill and you're also running it, so you have full context), but it's a useful sanity check — and the human review step compensates. Skip the baseline runs — just use the skill to complete the task as requested.\n\n**Reviewing results**: If you can't open a browser (e.g., Claude.ai's VM has no display, or you're on a remote server), skip the browser reviewer entirely. Instead, present results directly in the conversation. For each test case, show the prompt and the output. If the output is a file the user needs to see (like a .docx or .xlsx), save it to the filesystem and tell them where it is so they can download and inspect it. Ask for feedback inline: \"How does this look? Anything you'd change?\"\n\n**Benchmarking**: Skip the quantitative benchmarking — it relies on baseline comparisons which aren't meaningful without subagents. Focus on qualitative feedback from the user.\n\n**The iteration loop**: Same as before — improve the skill, rerun the test cases, ask for feedback — just without the browser reviewer in the middle. You can still organize results into iteration directories on the filesystem if you have one.\n\n**Description optimization**: This section requires the `claude` CLI tool (specifically `claude -p`) which is only available in Claude Code. Skip it if you're on Claude.ai.\n\n**Blind comparison**: Requires subagents. Skip it.\n\n**Packaging**: The `package_skill.py` script works anywhere with Python and a filesystem. On Claude.ai, you can run it and the user can download the resulting `.skill` file.\n\n**Updating an existing skill**: The user might be asking you to update an existing skill, not create a new one. In this case:\n- **Preserve the original name.** Note the skill's directory name and `name` frontmatter field -- use them unchanged. E.g., if the installed skill is `research-helper`, output `research-helper.skill` (not `research-helper-v2`).\n- **Copy to a writeable location before editing.** The installed skill path may be read-only. Copy to `/tmp/skill-name/`, edit there, and package from the copy.\n- **If packaging manually, stage in `/tmp/` first**, then copy to the output directory -- direct writes may fail due to permissions.\n\n---\n\n## Cowork-Specific Instructions\n\nIf you're in Cowork, the main things to know are:\n\n- You have subagents, so the main workflow (spawn test cases in parallel, run baselines, grade, etc.) all works. (However, if you run into severe problems with timeouts, it's OK to run the test prompts in series rather than parallel.)\n- You don't have a browser or display, so when generating the eval viewer, use `--static ` to write a standalone HTML file instead of starting a server. Then proffer a link that the user can click to open the HTML in their browser.\n- For whatever reason, the Cowork setup seems to disincline Claude from generating the eval viewer after running the tests, so just to reiterate: whether you're in Cowork or in Claude Code, after running tests, you should always generate the eval viewer for the human to look at examples before revising the skill yourself and trying to make corrections, using `generate_review.py` (not writing your own boutique html code). Sorry in advance but I'm gonna go all caps here: GENERATE THE EVAL VIEWER *BEFORE* evaluating inputs yourself. You want to get them in front of the human ASAP!\n- Feedback works differently: since there's no running server, the viewer's \"Submit All Reviews\" button will download `feedback.json` as a file. You can then read it from there (you may have to request access first).\n- Packaging works — `package_skill.py` just needs Python and a filesystem.\n- Description optimization (`run_loop.py` / `run_eval.py`) should work in Cowork just fine since it uses `claude -p` via subprocess, not a browser, but please save it until you've fully finished making the skill and the user agrees it's in good shape.\n- **Updating an existing skill**: The user might be asking you to update an existing skill, not create a new one. Follow the update guidance in the claude.ai section above.\n\n---\n\n## Reference files\n\nThe agents/ directory contains instructions for specialized subagents. Read them when you need to spawn the relevant subagent.\n\n- `agents/grader.md` — How to evaluate assertions against outputs\n- `agents/comparator.md` — How to do blind A/B comparison between two outputs\n- `agents/analyzer.md` — How to analyze why one version beat another\n\nThe references/ directory has additional documentation:\n- `references/schemas.md` — JSON structures for evals.json, grading.json, etc.\n\n---\n\nRepeating one more time the core loop here for emphasis:\n\n- Figure out what the skill is about\n- Draft or edit the skill\n- Run claude-with-access-to-the-skill on test prompts\n- With the user, evaluate the outputs:\n - Create benchmark.json and run `eval-viewer/generate_review.py` to help the user review them\n - Run quantitative evals\n- Repeat until you and the user are satisfied\n- Package the final skill and return it to the user.\n\nPlease add steps to your TodoList, if you have such a thing, to make sure you don't forget. If you're in Cowork, please specifically put \"Create evals JSON and run `eval-viewer/generate_review.py` so human can review test cases\" in your TodoList to make sure it happens.\n\nGood luck!\n","flock-data/skills/skill-creator/agents/analyzer.md":"# Post-hoc Analyzer Agent\n\nAnalyze blind comparison results to understand WHY the winner won and generate improvement suggestions.\n\n## Role\n\nAfter the blind comparator determines a winner, the Post-hoc Analyzer \"unblids\" the results by examining the skills and transcripts. The goal is to extract actionable insights: what made the winner better, and how can the loser be improved?\n\n## Inputs\n\nYou receive these parameters in your prompt:\n\n- **winner**: \"A\" or \"B\" (from blind comparison)\n- **winner_skill_path**: Path to the skill that produced the winning output\n- **winner_transcript_path**: Path to the execution transcript for the winner\n- **loser_skill_path**: Path to the skill that produced the losing output\n- **loser_transcript_path**: Path to the execution transcript for the loser\n- **comparison_result_path**: Path to the blind comparator's output JSON\n- **output_path**: Where to save the analysis results\n\n## Process\n\n### Step 1: Read Comparison Result\n\n1. Read the blind comparator's output at comparison_result_path\n2. Note the winning side (A or B), the reasoning, and any scores\n3. Understand what the comparator valued in the winning output\n\n### Step 2: Read Both Skills\n\n1. Read the winner skill's SKILL.md and key referenced files\n2. Read the loser skill's SKILL.md and key referenced files\n3. Identify structural differences:\n - Instructions clarity and specificity\n - Script/tool usage patterns\n - Example coverage\n - Edge case handling\n\n### Step 3: Read Both Transcripts\n\n1. Read the winner's transcript\n2. Read the loser's transcript\n3. Compare execution patterns:\n - How closely did each follow their skill's instructions?\n - What tools were used differently?\n - Where did the loser diverge from optimal behavior?\n - Did either encounter errors or make recovery attempts?\n\n### Step 4: Analyze Instruction Following\n\nFor each transcript, evaluate:\n- Did the agent follow the skill's explicit instructions?\n- Did the agent use the skill's provided tools/scripts?\n- Were there missed opportunities to leverage skill content?\n- Did the agent add unnecessary steps not in the skill?\n\nScore instruction following 1-10 and note specific issues.\n\n### Step 5: Identify Winner Strengths\n\nDetermine what made the winner better:\n- Clearer instructions that led to better behavior?\n- Better scripts/tools that produced better output?\n- More comprehensive examples that guided edge cases?\n- Better error handling guidance?\n\nBe specific. Quote from skills/transcripts where relevant.\n\n### Step 6: Identify Loser Weaknesses\n\nDetermine what held the loser back:\n- Ambiguous instructions that led to suboptimal choices?\n- Missing tools/scripts that forced workarounds?\n- Gaps in edge case coverage?\n- Poor error handling that caused failures?\n\n### Step 7: Generate Improvement Suggestions\n\nBased on the analysis, produce actionable suggestions for improving the loser skill:\n- Specific instruction changes to make\n- Tools/scripts to add or modify\n- Examples to include\n- Edge cases to address\n\nPrioritize by impact. Focus on changes that would have changed the outcome.\n\n### Step 8: Write Analysis Results\n\nSave structured analysis to `{output_path}`.\n\n## Output Format\n\nWrite a JSON file with this structure:\n\n```json\n{\n \"comparison_summary\": {\n \"winner\": \"A\",\n \"winner_skill\": \"path/to/winner/skill\",\n \"loser_skill\": \"path/to/loser/skill\",\n \"comparator_reasoning\": \"Brief summary of why comparator chose winner\"\n },\n \"winner_strengths\": [\n \"Clear step-by-step instructions for handling multi-page documents\",\n \"Included validation script that caught formatting errors\",\n \"Explicit guidance on fallback behavior when OCR fails\"\n ],\n \"loser_weaknesses\": [\n \"Vague instruction 'process the document appropriately' led to inconsistent behavior\",\n \"No script for validation, agent had to improvise and made errors\",\n \"No guidance on OCR failure, agent gave up instead of trying alternatives\"\n ],\n \"instruction_following\": {\n \"winner\": {\n \"score\": 9,\n \"issues\": [\n \"Minor: skipped optional logging step\"\n ]\n },\n \"loser\": {\n \"score\": 6,\n \"issues\": [\n \"Did not use the skill's formatting template\",\n \"Invented own approach instead of following step 3\",\n \"Missed the 'always validate output' instruction\"\n ]\n }\n },\n \"improvement_suggestions\": [\n {\n \"priority\": \"high\",\n \"category\": \"instructions\",\n \"suggestion\": \"Replace 'process the document appropriately' with explicit steps: 1) Extract text, 2) Identify sections, 3) Format per template\",\n \"expected_impact\": \"Would eliminate ambiguity that caused inconsistent behavior\"\n },\n {\n \"priority\": \"high\",\n \"category\": \"tools\",\n \"suggestion\": \"Add validate_output.py script similar to winner skill's validation approach\",\n \"expected_impact\": \"Would catch formatting errors before final output\"\n },\n {\n \"priority\": \"medium\",\n \"category\": \"error_handling\",\n \"suggestion\": \"Add fallback instructions: 'If OCR fails, try: 1) different resolution, 2) image preprocessing, 3) manual extraction'\",\n \"expected_impact\": \"Would prevent early failure on difficult documents\"\n }\n ],\n \"transcript_insights\": {\n \"winner_execution_pattern\": \"Read skill -> Followed 5-step process -> Used validation script -> Fixed 2 issues -> Produced output\",\n \"loser_execution_pattern\": \"Read skill -> Unclear on approach -> Tried 3 different methods -> No validation -> Output had errors\"\n }\n}\n```\n\n## Guidelines\n\n- **Be specific**: Quote from skills and transcripts, don't just say \"instructions were unclear\"\n- **Be actionable**: Suggestions should be concrete changes, not vague advice\n- **Focus on skill improvements**: The goal is to improve the losing skill, not critique the agent\n- **Prioritize by impact**: Which changes would most likely have changed the outcome?\n- **Consider causation**: Did the skill weakness actually cause the worse output, or is it incidental?\n- **Stay objective**: Analyze what happened, don't editorialize\n- **Think about generalization**: Would this improvement help on other evals too?\n\n## Categories for Suggestions\n\nUse these categories to organize improvement suggestions:\n\n| Category | Description |\n|----------|-------------|\n| `instructions` | Changes to the skill's prose instructions |\n| `tools` | Scripts, templates, or utilities to add/modify |\n| `examples` | Example inputs/outputs to include |\n| `error_handling` | Guidance for handling failures |\n| `structure` | Reorganization of skill content |\n| `references` | External docs or resources to add |\n\n## Priority Levels\n\n- **high**: Would likely change the outcome of this comparison\n- **medium**: Would improve quality but may not change win/loss\n- **low**: Nice to have, marginal improvement\n\n---\n\n# Analyzing Benchmark Results\n\nWhen analyzing benchmark results, the analyzer's purpose is to **surface patterns and anomalies** across multiple runs, not suggest skill improvements.\n\n## Role\n\nReview all benchmark run results and generate freeform notes that help the user understand skill performance. Focus on patterns that wouldn't be visible from aggregate metrics alone.\n\n## Inputs\n\nYou receive these parameters in your prompt:\n\n- **benchmark_data_path**: Path to the in-progress benchmark.json with all run results\n- **skill_path**: Path to the skill being benchmarked\n- **output_path**: Where to save the notes (as JSON array of strings)\n\n## Process\n\n### Step 1: Read Benchmark Data\n\n1. Read the benchmark.json containing all run results\n2. Note the configurations tested (with_skill, without_skill)\n3. Understand the run_summary aggregates already calculated\n\n### Step 2: Analyze Per-Assertion Patterns\n\nFor each expectation across all runs:\n- Does it **always pass** in both configurations? (may not differentiate skill value)\n- Does it **always fail** in both configurations? (may be broken or beyond capability)\n- Does it **always pass with skill but fail without**? (skill clearly adds value here)\n- Does it **always fail with skill but pass without**? (skill may be hurting)\n- Is it **highly variable**? (flaky expectation or non-deterministic behavior)\n\n### Step 3: Analyze Cross-Eval Patterns\n\nLook for patterns across evals:\n- Are certain eval types consistently harder/easier?\n- Do some evals show high variance while others are stable?\n- Are there surprising results that contradict expectations?\n\n### Step 4: Analyze Metrics Patterns\n\nLook at time_seconds, tokens, tool_calls:\n- Does the skill significantly increase execution time?\n- Is there high variance in resource usage?\n- Are there outlier runs that skew the aggregates?\n\n### Step 5: Generate Notes\n\nWrite freeform observations as a list of strings. Each note should:\n- State a specific observation\n- Be grounded in the data (not speculation)\n- Help the user understand something the aggregate metrics don't show\n\nExamples:\n- \"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value\"\n- \"Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure that may be flaky\"\n- \"Without-skill runs consistently fail on table extraction expectations (0% pass rate)\"\n- \"Skill adds 13s average execution time but improves pass rate by 50%\"\n- \"Token usage is 80% higher with skill, primarily due to script output parsing\"\n- \"All 3 without-skill runs for eval 1 produced empty output\"\n\n### Step 6: Write Notes\n\nSave notes to `{output_path}` as a JSON array of strings:\n\n```json\n[\n \"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value\",\n \"Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure\",\n \"Without-skill runs consistently fail on table extraction expectations\",\n \"Skill adds 13s average execution time but improves pass rate by 50%\"\n]\n```\n\n## Guidelines\n\n**DO:**\n- Report what you observe in the data\n- Be specific about which evals, expectations, or runs you're referring to\n- Note patterns that aggregate metrics would hide\n- Provide context that helps interpret the numbers\n\n**DO NOT:**\n- Suggest improvements to the skill (that's for the improvement step, not benchmarking)\n- Make subjective quality judgments (\"the output was good/bad\")\n- Speculate about causes without evidence\n- Repeat information already in the run_summary aggregates\n","flock-data/skills/skill-creator/agents/comparator.md":"# Blind Comparator Agent\n\nCompare two outputs WITHOUT knowing which skill produced them.\n\n## Role\n\nThe Blind Comparator judges which output better accomplishes the eval task. You receive two outputs labeled A and B, but you do NOT know which skill produced which. This prevents bias toward a particular skill or approach.\n\nYour judgment is based purely on output quality and task completion.\n\n## Inputs\n\nYou receive these parameters in your prompt:\n\n- **output_a_path**: Path to the first output file or directory\n- **output_b_path**: Path to the second output file or directory\n- **eval_prompt**: The original task/prompt that was executed\n- **expectations**: List of expectations to check (optional - may be empty)\n\n## Process\n\n### Step 1: Read Both Outputs\n\n1. Examine output A (file or directory)\n2. Examine output B (file or directory)\n3. Note the type, structure, and content of each\n4. If outputs are directories, examine all relevant files inside\n\n### Step 2: Understand the Task\n\n1. Read the eval_prompt carefully\n2. Identify what the task requires:\n - What should be produced?\n - What qualities matter (accuracy, completeness, format)?\n - What would distinguish a good output from a poor one?\n\n### Step 3: Generate Evaluation Rubric\n\nBased on the task, generate a rubric with two dimensions:\n\n**Content Rubric** (what the output contains):\n| Criterion | 1 (Poor) | 3 (Acceptable) | 5 (Excellent) |\n|-----------|----------|----------------|---------------|\n| Correctness | Major errors | Minor errors | Fully correct |\n| Completeness | Missing key elements | Mostly complete | All elements present |\n| Accuracy | Significant inaccuracies | Minor inaccuracies | Accurate throughout |\n\n**Structure Rubric** (how the output is organized):\n| Criterion | 1 (Poor) | 3 (Acceptable) | 5 (Excellent) |\n|-----------|----------|----------------|---------------|\n| Organization | Disorganized | Reasonably organized | Clear, logical structure |\n| Formatting | Inconsistent/broken | Mostly consistent | Professional, polished |\n| Usability | Difficult to use | Usable with effort | Easy to use |\n\nAdapt criteria to the specific task. For example:\n- PDF form → \"Field alignment\", \"Text readability\", \"Data placement\"\n- Document → \"Section structure\", \"Heading hierarchy\", \"Paragraph flow\"\n- Data output → \"Schema correctness\", \"Data types\", \"Completeness\"\n\n### Step 4: Evaluate Each Output Against the Rubric\n\nFor each output (A and B):\n\n1. **Score each criterion** on the rubric (1-5 scale)\n2. **Calculate dimension totals**: Content score, Structure score\n3. **Calculate overall score**: Average of dimension scores, scaled to 1-10\n\n### Step 5: Check Assertions (if provided)\n\nIf expectations are provided:\n\n1. Check each expectation against output A\n2. Check each expectation against output B\n3. Count pass rates for each output\n4. Use expectation scores as secondary evidence (not the primary decision factor)\n\n### Step 6: Determine the Winner\n\nCompare A and B based on (in priority order):\n\n1. **Primary**: Overall rubric score (content + structure)\n2. **Secondary**: Assertion pass rates (if applicable)\n3. **Tiebreaker**: If truly equal, declare a TIE\n\nBe decisive - ties should be rare. One output is usually better, even if marginally.\n\n### Step 7: Write Comparison Results\n\nSave results to a JSON file at the path specified (or `comparison.json` if not specified).\n\n## Output Format\n\nWrite a JSON file with this structure:\n\n```json\n{\n \"winner\": \"A\",\n \"reasoning\": \"Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.\",\n \"rubric\": {\n \"A\": {\n \"content\": {\n \"correctness\": 5,\n \"completeness\": 5,\n \"accuracy\": 4\n },\n \"structure\": {\n \"organization\": 4,\n \"formatting\": 5,\n \"usability\": 4\n },\n \"content_score\": 4.7,\n \"structure_score\": 4.3,\n \"overall_score\": 9.0\n },\n \"B\": {\n \"content\": {\n \"correctness\": 3,\n \"completeness\": 2,\n \"accuracy\": 3\n },\n \"structure\": {\n \"organization\": 3,\n \"formatting\": 2,\n \"usability\": 3\n },\n \"content_score\": 2.7,\n \"structure_score\": 2.7,\n \"overall_score\": 5.4\n }\n },\n \"output_quality\": {\n \"A\": {\n \"score\": 9,\n \"strengths\": [\"Complete solution\", \"Well-formatted\", \"All fields present\"],\n \"weaknesses\": [\"Minor style inconsistency in header\"]\n },\n \"B\": {\n \"score\": 5,\n \"strengths\": [\"Readable output\", \"Correct basic structure\"],\n \"weaknesses\": [\"Missing date field\", \"Formatting inconsistencies\", \"Partial data extraction\"]\n }\n },\n \"expectation_results\": {\n \"A\": {\n \"passed\": 4,\n \"total\": 5,\n \"pass_rate\": 0.80,\n \"details\": [\n {\"text\": \"Output includes name\", \"passed\": true},\n {\"text\": \"Output includes date\", \"passed\": true},\n {\"text\": \"Format is PDF\", \"passed\": true},\n {\"text\": \"Contains signature\", \"passed\": false},\n {\"text\": \"Readable text\", \"passed\": true}\n ]\n },\n \"B\": {\n \"passed\": 3,\n \"total\": 5,\n \"pass_rate\": 0.60,\n \"details\": [\n {\"text\": \"Output includes name\", \"passed\": true},\n {\"text\": \"Output includes date\", \"passed\": false},\n {\"text\": \"Format is PDF\", \"passed\": true},\n {\"text\": \"Contains signature\", \"passed\": false},\n {\"text\": \"Readable text\", \"passed\": true}\n ]\n }\n }\n}\n```\n\nIf no expectations were provided, omit the `expectation_results` field entirely.\n\n## Field Descriptions\n\n- **winner**: \"A\", \"B\", or \"TIE\"\n- **reasoning**: Clear explanation of why the winner was chosen (or why it's a tie)\n- **rubric**: Structured rubric evaluation for each output\n - **content**: Scores for content criteria (correctness, completeness, accuracy)\n - **structure**: Scores for structure criteria (organization, formatting, usability)\n - **content_score**: Average of content criteria (1-5)\n - **structure_score**: Average of structure criteria (1-5)\n - **overall_score**: Combined score scaled to 1-10\n- **output_quality**: Summary quality assessment\n - **score**: 1-10 rating (should match rubric overall_score)\n - **strengths**: List of positive aspects\n - **weaknesses**: List of issues or shortcomings\n- **expectation_results**: (Only if expectations provided)\n - **passed**: Number of expectations that passed\n - **total**: Total number of expectations\n - **pass_rate**: Fraction passed (0.0 to 1.0)\n - **details**: Individual expectation results\n\n## Guidelines\n\n- **Stay blind**: DO NOT try to infer which skill produced which output. Judge purely on output quality.\n- **Be specific**: Cite specific examples when explaining strengths and weaknesses.\n- **Be decisive**: Choose a winner unless outputs are genuinely equivalent.\n- **Output quality first**: Assertion scores are secondary to overall task completion.\n- **Be objective**: Don't favor outputs based on style preferences; focus on correctness and completeness.\n- **Explain your reasoning**: The reasoning field should make it clear why you chose the winner.\n- **Handle edge cases**: If both outputs fail, pick the one that fails less badly. If both are excellent, pick the one that's marginally better.\n","flock-data/skills/skill-creator/agents/grader.md":"# Grader Agent\n\nEvaluate expectations against an execution transcript and outputs.\n\n## Role\n\nThe Grader reviews a transcript and output files, then determines whether each expectation passes or fails. Provide clear evidence for each judgment.\n\nYou have two jobs: grade the outputs, and critique the evals themselves. A passing grade on a weak assertion is worse than useless — it creates false confidence. When you notice an assertion that's trivially satisfied, or an important outcome that no assertion checks, say so.\n\n## Inputs\n\nYou receive these parameters in your prompt:\n\n- **expectations**: List of expectations to evaluate (strings)\n- **transcript_path**: Path to the execution transcript (markdown file)\n- **outputs_dir**: Directory containing output files from execution\n\n## Process\n\n### Step 1: Read the Transcript\n\n1. Read the transcript file completely\n2. Note the eval prompt, execution steps, and final result\n3. Identify any issues or errors documented\n\n### Step 2: Examine Output Files\n\n1. List files in outputs_dir\n2. Read/examine each file relevant to the expectations. If outputs aren't plain text, use the inspection tools provided in your prompt — don't rely solely on what the transcript says the executor produced.\n3. Note contents, structure, and quality\n\n### Step 3: Evaluate Each Assertion\n\nFor each expectation:\n\n1. **Search for evidence** in the transcript and outputs\n2. **Determine verdict**:\n - **PASS**: Clear evidence the expectation is true AND the evidence reflects genuine task completion, not just surface-level compliance\n - **FAIL**: No evidence, or evidence contradicts the expectation, or the evidence is superficial (e.g., correct filename but empty/wrong content)\n3. **Cite the evidence**: Quote the specific text or describe what you found\n\n### Step 4: Extract and Verify Claims\n\nBeyond the predefined expectations, extract implicit claims from the outputs and verify them:\n\n1. **Extract claims** from the transcript and outputs:\n - Factual statements (\"The form has 12 fields\")\n - Process claims (\"Used pypdf to fill the form\")\n - Quality claims (\"All fields were filled correctly\")\n\n2. **Verify each claim**:\n - **Factual claims**: Can be checked against the outputs or external sources\n - **Process claims**: Can be verified from the transcript\n - **Quality claims**: Evaluate whether the claim is justified\n\n3. **Flag unverifiable claims**: Note claims that cannot be verified with available information\n\nThis catches issues that predefined expectations might miss.\n\n### Step 5: Read User Notes\n\nIf `{outputs_dir}/user_notes.md` exists:\n1. Read it and note any uncertainties or issues flagged by the executor\n2. Include relevant concerns in the grading output\n3. These may reveal problems even when expectations pass\n\n### Step 6: Critique the Evals\n\nAfter grading, consider whether the evals themselves could be improved. Only surface suggestions when there's a clear gap.\n\nGood suggestions test meaningful outcomes — assertions that are hard to satisfy without actually doing the work correctly. Think about what makes an assertion *discriminating*: it passes when the skill genuinely succeeds and fails when it doesn't.\n\nSuggestions worth raising:\n- An assertion that passed but would also pass for a clearly wrong output (e.g., checking filename existence but not file content)\n- An important outcome you observed — good or bad — that no assertion covers at all\n- An assertion that can't actually be verified from the available outputs\n\nKeep the bar high. The goal is to flag things the eval author would say \"good catch\" about, not to nitpick every assertion.\n\n### Step 7: Write Grading Results\n\nSave results to `{outputs_dir}/../grading.json` (sibling to outputs_dir).\n\n## Grading Criteria\n\n**PASS when**:\n- The transcript or outputs clearly demonstrate the expectation is true\n- Specific evidence can be cited\n- The evidence reflects genuine substance, not just surface compliance (e.g., a file exists AND contains correct content, not just the right filename)\n\n**FAIL when**:\n- No evidence found for the expectation\n- Evidence contradicts the expectation\n- The expectation cannot be verified from available information\n- The evidence is superficial — the assertion is technically satisfied but the underlying task outcome is wrong or incomplete\n- The output appears to meet the assertion by coincidence rather than by actually doing the work\n\n**When uncertain**: The burden of proof to pass is on the expectation.\n\n### Step 8: Read Executor Metrics and Timing\n\n1. If `{outputs_dir}/metrics.json` exists, read it and include in grading output\n2. If `{outputs_dir}/../timing.json` exists, read it and include timing data\n\n## Output Format\n\nWrite a JSON file with this structure:\n\n```json\n{\n \"expectations\": [\n {\n \"text\": \"The output includes the name 'John Smith'\",\n \"passed\": true,\n \"evidence\": \"Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'\"\n },\n {\n \"text\": \"The spreadsheet has a SUM formula in cell B10\",\n \"passed\": false,\n \"evidence\": \"No spreadsheet was created. The output was a text file.\"\n },\n {\n \"text\": \"The assistant used the skill's OCR script\",\n \"passed\": true,\n \"evidence\": \"Transcript Step 2 shows: 'Tool: Bash - python ocr_script.py image.png'\"\n }\n ],\n \"summary\": {\n \"passed\": 2,\n \"failed\": 1,\n \"total\": 3,\n \"pass_rate\": 0.67\n },\n \"execution_metrics\": {\n \"tool_calls\": {\n \"Read\": 5,\n \"Write\": 2,\n \"Bash\": 8\n },\n \"total_tool_calls\": 15,\n \"total_steps\": 6,\n \"errors_encountered\": 0,\n \"output_chars\": 12450,\n \"transcript_chars\": 3200\n },\n \"timing\": {\n \"executor_duration_seconds\": 165.0,\n \"grader_duration_seconds\": 26.0,\n \"total_duration_seconds\": 191.0\n },\n \"claims\": [\n {\n \"claim\": \"The form has 12 fillable fields\",\n \"type\": \"factual\",\n \"verified\": true,\n \"evidence\": \"Counted 12 fields in field_info.json\"\n },\n {\n \"claim\": \"All required fields were populated\",\n \"type\": \"quality\",\n \"verified\": false,\n \"evidence\": \"Reference section was left blank despite data being available\"\n }\n ],\n \"user_notes_summary\": {\n \"uncertainties\": [\"Used 2023 data, may be stale\"],\n \"needs_review\": [],\n \"workarounds\": [\"Fell back to text overlay for non-fillable fields\"]\n },\n \"eval_feedback\": {\n \"suggestions\": [\n {\n \"assertion\": \"The output includes the name 'John Smith'\",\n \"reason\": \"A hallucinated document that mentions the name would also pass — consider checking it appears as the primary contact with matching phone and email from the input\"\n },\n {\n \"reason\": \"No assertion checks whether the extracted phone numbers match the input — I observed incorrect numbers in the output that went uncaught\"\n }\n ],\n \"overall\": \"Assertions check presence but not correctness. Consider adding content verification.\"\n }\n}\n```\n\n## Field Descriptions\n\n- **expectations**: Array of graded expectations\n - **text**: The original expectation text\n - **passed**: Boolean - true if expectation passes\n - **evidence**: Specific quote or description supporting the verdict\n- **summary**: Aggregate statistics\n - **passed**: Count of passed expectations\n - **failed**: Count of failed expectations\n - **total**: Total expectations evaluated\n - **pass_rate**: Fraction passed (0.0 to 1.0)\n- **execution_metrics**: Copied from executor's metrics.json (if available)\n - **output_chars**: Total character count of output files (proxy for tokens)\n - **transcript_chars**: Character count of transcript\n- **timing**: Wall clock timing from timing.json (if available)\n - **executor_duration_seconds**: Time spent in executor subagent\n - **total_duration_seconds**: Total elapsed time for the run\n- **claims**: Extracted and verified claims from the output\n - **claim**: The statement being verified\n - **type**: \"factual\", \"process\", or \"quality\"\n - **verified**: Boolean - whether the claim holds\n - **evidence**: Supporting or contradicting evidence\n- **user_notes_summary**: Issues flagged by the executor\n - **uncertainties**: Things the executor wasn't sure about\n - **needs_review**: Items requiring human attention\n - **workarounds**: Places where the skill didn't work as expected\n- **eval_feedback**: Improvement suggestions for the evals (only when warranted)\n - **suggestions**: List of concrete suggestions, each with a `reason` and optionally an `assertion` it relates to\n - **overall**: Brief assessment — can be \"No suggestions, evals look solid\" if nothing to flag\n\n## Guidelines\n\n- **Be objective**: Base verdicts on evidence, not assumptions\n- **Be specific**: Quote the exact text that supports your verdict\n- **Be thorough**: Check both transcript and output files\n- **Be consistent**: Apply the same standard to each expectation\n- **Explain failures**: Make it clear why evidence was insufficient\n- **No partial credit**: Each expectation is pass or fail, not partial\n","flock-data/skills/skill-creator/references/schemas.md":"# JSON Schemas\n\nThis document defines the JSON schemas used by skill-creator.\n\n---\n\n## evals.json\n\nDefines the evals for a skill. Located at `evals/evals.json` within the skill directory.\n\n```json\n{\n \"skill_name\": \"example-skill\",\n \"evals\": [\n {\n \"id\": 1,\n \"prompt\": \"User's example prompt\",\n \"expected_output\": \"Description of expected result\",\n \"files\": [\"evals/files/sample1.pdf\"],\n \"expectations\": [\n \"The output includes X\",\n \"The skill used script Y\"\n ]\n }\n ]\n}\n```\n\n**Fields:**\n- `skill_name`: Name matching the skill's frontmatter\n- `evals[].id`: Unique integer identifier\n- `evals[].prompt`: The task to execute\n- `evals[].expected_output`: Human-readable description of success\n- `evals[].files`: Optional list of input file paths (relative to skill root)\n- `evals[].expectations`: List of verifiable statements\n\n---\n\n## history.json\n\nTracks version progression in Improve mode. Located at workspace root.\n\n```json\n{\n \"started_at\": \"2026-01-15T10:30:00Z\",\n \"skill_name\": \"pdf\",\n \"current_best\": \"v2\",\n \"iterations\": [\n {\n \"version\": \"v0\",\n \"parent\": null,\n \"expectation_pass_rate\": 0.65,\n \"grading_result\": \"baseline\",\n \"is_current_best\": false\n },\n {\n \"version\": \"v1\",\n \"parent\": \"v0\",\n \"expectation_pass_rate\": 0.75,\n \"grading_result\": \"won\",\n \"is_current_best\": false\n },\n {\n \"version\": \"v2\",\n \"parent\": \"v1\",\n \"expectation_pass_rate\": 0.85,\n \"grading_result\": \"won\",\n \"is_current_best\": true\n }\n ]\n}\n```\n\n**Fields:**\n- `started_at`: ISO timestamp of when improvement started\n- `skill_name`: Name of the skill being improved\n- `current_best`: Version identifier of the best performer\n- `iterations[].version`: Version identifier (v0, v1, ...)\n- `iterations[].parent`: Parent version this was derived from\n- `iterations[].expectation_pass_rate`: Pass rate from grading\n- `iterations[].grading_result`: \"baseline\", \"won\", \"lost\", or \"tie\"\n- `iterations[].is_current_best`: Whether this is the current best version\n\n---\n\n## grading.json\n\nOutput from the grader agent. Located at `/grading.json`.\n\n```json\n{\n \"expectations\": [\n {\n \"text\": \"The output includes the name 'John Smith'\",\n \"passed\": true,\n \"evidence\": \"Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'\"\n },\n {\n \"text\": \"The spreadsheet has a SUM formula in cell B10\",\n \"passed\": false,\n \"evidence\": \"No spreadsheet was created. The output was a text file.\"\n }\n ],\n \"summary\": {\n \"passed\": 2,\n \"failed\": 1,\n \"total\": 3,\n \"pass_rate\": 0.67\n },\n \"execution_metrics\": {\n \"tool_calls\": {\n \"Read\": 5,\n \"Write\": 2,\n \"Bash\": 8\n },\n \"total_tool_calls\": 15,\n \"total_steps\": 6,\n \"errors_encountered\": 0,\n \"output_chars\": 12450,\n \"transcript_chars\": 3200\n },\n \"timing\": {\n \"executor_duration_seconds\": 165.0,\n \"grader_duration_seconds\": 26.0,\n \"total_duration_seconds\": 191.0\n },\n \"claims\": [\n {\n \"claim\": \"The form has 12 fillable fields\",\n \"type\": \"factual\",\n \"verified\": true,\n \"evidence\": \"Counted 12 fields in field_info.json\"\n }\n ],\n \"user_notes_summary\": {\n \"uncertainties\": [\"Used 2023 data, may be stale\"],\n \"needs_review\": [],\n \"workarounds\": [\"Fell back to text overlay for non-fillable fields\"]\n },\n \"eval_feedback\": {\n \"suggestions\": [\n {\n \"assertion\": \"The output includes the name 'John Smith'\",\n \"reason\": \"A hallucinated document that mentions the name would also pass\"\n }\n ],\n \"overall\": \"Assertions check presence but not correctness.\"\n }\n}\n```\n\n**Fields:**\n- `expectations[]`: Graded expectations with evidence\n- `summary`: Aggregate pass/fail counts\n- `execution_metrics`: Tool usage and output size (from executor's metrics.json)\n- `timing`: Wall clock timing (from timing.json)\n- `claims`: Extracted and verified claims from the output\n- `user_notes_summary`: Issues flagged by the executor\n- `eval_feedback`: (optional) Improvement suggestions for the evals, only present when the grader identifies issues worth raising\n\n---\n\n## metrics.json\n\nOutput from the executor agent. Located at `/outputs/metrics.json`.\n\n```json\n{\n \"tool_calls\": {\n \"Read\": 5,\n \"Write\": 2,\n \"Bash\": 8,\n \"Edit\": 1,\n \"Glob\": 2,\n \"Grep\": 0\n },\n \"total_tool_calls\": 18,\n \"total_steps\": 6,\n \"files_created\": [\"filled_form.pdf\", \"field_values.json\"],\n \"errors_encountered\": 0,\n \"output_chars\": 12450,\n \"transcript_chars\": 3200\n}\n```\n\n**Fields:**\n- `tool_calls`: Count per tool type\n- `total_tool_calls`: Sum of all tool calls\n- `total_steps`: Number of major execution steps\n- `files_created`: List of output files created\n- `errors_encountered`: Number of errors during execution\n- `output_chars`: Total character count of output files\n- `transcript_chars`: Character count of transcript\n\n---\n\n## timing.json\n\nWall clock timing for a run. Located at `/timing.json`.\n\n**How to capture:** When a subagent task completes, the task notification includes `total_tokens` and `duration_ms`. Save these immediately — they are not persisted anywhere else and cannot be recovered after the fact.\n\n```json\n{\n \"total_tokens\": 84852,\n \"duration_ms\": 23332,\n \"total_duration_seconds\": 23.3,\n \"executor_start\": \"2026-01-15T10:30:00Z\",\n \"executor_end\": \"2026-01-15T10:32:45Z\",\n \"executor_duration_seconds\": 165.0,\n \"grader_start\": \"2026-01-15T10:32:46Z\",\n \"grader_end\": \"2026-01-15T10:33:12Z\",\n \"grader_duration_seconds\": 26.0\n}\n```\n\n---\n\n## benchmark.json\n\nOutput from Benchmark mode. Located at `benchmarks//benchmark.json`.\n\n```json\n{\n \"metadata\": {\n \"skill_name\": \"pdf\",\n \"skill_path\": \"/path/to/pdf\",\n \"executor_model\": \"claude-sonnet-4-20250514\",\n \"analyzer_model\": \"most-capable-model\",\n \"timestamp\": \"2026-01-15T10:30:00Z\",\n \"evals_run\": [1, 2, 3],\n \"runs_per_configuration\": 3\n },\n\n \"runs\": [\n {\n \"eval_id\": 1,\n \"eval_name\": \"Ocean\",\n \"configuration\": \"with_skill\",\n \"run_number\": 1,\n \"result\": {\n \"pass_rate\": 0.85,\n \"passed\": 6,\n \"failed\": 1,\n \"total\": 7,\n \"time_seconds\": 42.5,\n \"tokens\": 3800,\n \"tool_calls\": 18,\n \"errors\": 0\n },\n \"expectations\": [\n {\"text\": \"...\", \"passed\": true, \"evidence\": \"...\"}\n ],\n \"notes\": [\n \"Used 2023 data, may be stale\",\n \"Fell back to text overlay for non-fillable fields\"\n ]\n }\n ],\n\n \"run_summary\": {\n \"with_skill\": {\n \"pass_rate\": {\"mean\": 0.85, \"stddev\": 0.05, \"min\": 0.80, \"max\": 0.90},\n \"time_seconds\": {\"mean\": 45.0, \"stddev\": 12.0, \"min\": 32.0, \"max\": 58.0},\n \"tokens\": {\"mean\": 3800, \"stddev\": 400, \"min\": 3200, \"max\": 4100}\n },\n \"without_skill\": {\n \"pass_rate\": {\"mean\": 0.35, \"stddev\": 0.08, \"min\": 0.28, \"max\": 0.45},\n \"time_seconds\": {\"mean\": 32.0, \"stddev\": 8.0, \"min\": 24.0, \"max\": 42.0},\n \"tokens\": {\"mean\": 2100, \"stddev\": 300, \"min\": 1800, \"max\": 2500}\n },\n \"delta\": {\n \"pass_rate\": \"+0.50\",\n \"time_seconds\": \"+13.0\",\n \"tokens\": \"+1700\"\n }\n },\n\n \"notes\": [\n \"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value\",\n \"Eval 3 shows high variance (50% ± 40%) - may be flaky or model-dependent\",\n \"Without-skill runs consistently fail on table extraction expectations\",\n \"Skill adds 13s average execution time but improves pass rate by 50%\"\n ]\n}\n```\n\n**Fields:**\n- `metadata`: Information about the benchmark run\n - `skill_name`: Name of the skill\n - `timestamp`: When the benchmark was run\n - `evals_run`: List of eval names or IDs\n - `runs_per_configuration`: Number of runs per config (e.g. 3)\n- `runs[]`: Individual run results\n - `eval_id`: Numeric eval identifier\n - `eval_name`: Human-readable eval name (used as section header in the viewer)\n - `configuration`: Must be `\"with_skill\"` or `\"without_skill\"` (the viewer uses this exact string for grouping and color coding)\n - `run_number`: Integer run number (1, 2, 3...)\n - `result`: Nested object with `pass_rate`, `passed`, `total`, `time_seconds`, `tokens`, `errors`\n- `run_summary`: Statistical aggregates per configuration\n - `with_skill` / `without_skill`: Each contains `pass_rate`, `time_seconds`, `tokens` objects with `mean` and `stddev` fields\n - `delta`: Difference strings like `\"+0.50\"`, `\"+13.0\"`, `\"+1700\"`\n- `notes`: Freeform observations from the analyzer\n\n**Important:** The viewer reads these field names exactly. Using `config` instead of `configuration`, or putting `pass_rate` at the top level of a run instead of nested under `result`, will cause the viewer to show empty/zero values. Always reference this schema when generating benchmark.json manually.\n\n---\n\n## comparison.json\n\nOutput from blind comparator. Located at `/comparison-N.json`.\n\n```json\n{\n \"winner\": \"A\",\n \"reasoning\": \"Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.\",\n \"rubric\": {\n \"A\": {\n \"content\": {\n \"correctness\": 5,\n \"completeness\": 5,\n \"accuracy\": 4\n },\n \"structure\": {\n \"organization\": 4,\n \"formatting\": 5,\n \"usability\": 4\n },\n \"content_score\": 4.7,\n \"structure_score\": 4.3,\n \"overall_score\": 9.0\n },\n \"B\": {\n \"content\": {\n \"correctness\": 3,\n \"completeness\": 2,\n \"accuracy\": 3\n },\n \"structure\": {\n \"organization\": 3,\n \"formatting\": 2,\n \"usability\": 3\n },\n \"content_score\": 2.7,\n \"structure_score\": 2.7,\n \"overall_score\": 5.4\n }\n },\n \"output_quality\": {\n \"A\": {\n \"score\": 9,\n \"strengths\": [\"Complete solution\", \"Well-formatted\", \"All fields present\"],\n \"weaknesses\": [\"Minor style inconsistency in header\"]\n },\n \"B\": {\n \"score\": 5,\n \"strengths\": [\"Readable output\", \"Correct basic structure\"],\n \"weaknesses\": [\"Missing date field\", \"Formatting inconsistencies\", \"Partial data extraction\"]\n }\n },\n \"expectation_results\": {\n \"A\": {\n \"passed\": 4,\n \"total\": 5,\n \"pass_rate\": 0.80,\n \"details\": [\n {\"text\": \"Output includes name\", \"passed\": true}\n ]\n },\n \"B\": {\n \"passed\": 3,\n \"total\": 5,\n \"pass_rate\": 0.60,\n \"details\": [\n {\"text\": \"Output includes name\", \"passed\": true}\n ]\n }\n }\n}\n```\n\n---\n\n## analysis.json\n\nOutput from post-hoc analyzer. Located at `/analysis.json`.\n\n```json\n{\n \"comparison_summary\": {\n \"winner\": \"A\",\n \"winner_skill\": \"path/to/winner/skill\",\n \"loser_skill\": \"path/to/loser/skill\",\n \"comparator_reasoning\": \"Brief summary of why comparator chose winner\"\n },\n \"winner_strengths\": [\n \"Clear step-by-step instructions for handling multi-page documents\",\n \"Included validation script that caught formatting errors\"\n ],\n \"loser_weaknesses\": [\n \"Vague instruction 'process the document appropriately' led to inconsistent behavior\",\n \"No script for validation, agent had to improvise\"\n ],\n \"instruction_following\": {\n \"winner\": {\n \"score\": 9,\n \"issues\": [\"Minor: skipped optional logging step\"]\n },\n \"loser\": {\n \"score\": 6,\n \"issues\": [\n \"Did not use the skill's formatting template\",\n \"Invented own approach instead of following step 3\"\n ]\n }\n },\n \"improvement_suggestions\": [\n {\n \"priority\": \"high\",\n \"category\": \"instructions\",\n \"suggestion\": \"Replace 'process the document appropriately' with explicit steps\",\n \"expected_impact\": \"Would eliminate ambiguity that caused inconsistent behavior\"\n }\n ],\n \"transcript_insights\": {\n \"winner_execution_pattern\": \"Read skill -> Followed 5-step process -> Used validation script\",\n \"loser_execution_pattern\": \"Read skill -> Unclear on approach -> Tried 3 different methods\"\n }\n}\n```\n","flock-data/skills/using-superpowers/SKILL.md":"---\nname: using-superpowers\ndescription: Use when starting any conversation - establishes how to find and use skills, requiring Skill tool invocation before ANY response including clarifying questions\n---\n\n\nIf you were dispatched as a subagent to execute a specific task, skip this skill.\n\n\n\nIf you think there is even a 1% chance a skill might apply to what you are doing, you ABSOLUTELY MUST invoke the skill.\n\nIF A SKILL APPLIES TO YOUR TASK, YOU DO NOT HAVE A CHOICE. YOU MUST USE IT.\n\nThis is not negotiable. This is not optional. You cannot rationalize your way out of this.\n\n\n## Instruction Priority\n\nSuperpowers skills override default system prompt behavior, but **user instructions always take precedence**:\n\n1. **User's explicit instructions** (CLAUDE.md, GEMINI.md, AGENTS.md, direct requests) — highest priority\n2. **Superpowers skills** — override default system behavior where they conflict\n3. **Default system prompt** — lowest priority\n\nIf CLAUDE.md, GEMINI.md, or AGENTS.md says \"don't use TDD\" and a skill says \"always use TDD,\" follow the user's instructions. The user is in control.\n\n## How to Access Skills\n\n**In Claude Code:** Use the `Skill` tool. When you invoke a skill, its content is loaded and presented to you—follow it directly. Never use the Read tool on skill files.\n\n**In Gemini CLI:** Skills activate via the `activate_skill` tool. Gemini loads skill metadata at session start and activates the full content on demand.\n\n**In other environments:** Check your platform's documentation for how skills are loaded.\n\n## Platform Adaptation\n\nSkills use Claude Code tool names. Non-CC platforms: see `references/codex-tools.md` (Codex) for tool equivalents. Gemini CLI users get the tool mapping loaded automatically via GEMINI.md.\n\n# Using Skills\n\n## The Rule\n\n**Invoke relevant or requested skills BEFORE any response or action.** Even a 1% chance a skill might apply means that you should invoke the skill to check. If an invoked skill turns out to be wrong for the situation, you don't need to use it.\n\n```dot\ndigraph skill_flow {\n \"User message received\" [shape=doublecircle];\n \"About to EnterPlanMode?\" [shape=doublecircle];\n \"Already brainstormed?\" [shape=diamond];\n \"Invoke brainstorming skill\" [shape=box];\n \"Might any skill apply?\" [shape=diamond];\n \"Invoke Skill tool\" [shape=box];\n \"Announce: 'Using [skill] to [purpose]'\" [shape=box];\n \"Has checklist?\" [shape=diamond];\n \"Create TodoWrite todo per item\" [shape=box];\n \"Follow skill exactly\" [shape=box];\n \"Respond (including clarifications)\" [shape=doublecircle];\n\n \"About to EnterPlanMode?\" -> \"Already brainstormed?\";\n \"Already brainstormed?\" -> \"Invoke brainstorming skill\" [label=\"no\"];\n \"Already brainstormed?\" -> \"Might any skill apply?\" [label=\"yes\"];\n \"Invoke brainstorming skill\" -> \"Might any skill apply?\";\n\n \"User message received\" -> \"Might any skill apply?\";\n \"Might any skill apply?\" -> \"Invoke Skill tool\" [label=\"yes, even 1%\"];\n \"Might any skill apply?\" -> \"Respond (including clarifications)\" [label=\"definitely not\"];\n \"Invoke Skill tool\" -> \"Announce: 'Using [skill] to [purpose]'\";\n \"Announce: 'Using [skill] to [purpose]'\" -> \"Has checklist?\";\n \"Has checklist?\" -> \"Create TodoWrite todo per item\" [label=\"yes\"];\n \"Has checklist?\" -> \"Follow skill exactly\" [label=\"no\"];\n \"Create TodoWrite todo per item\" -> \"Follow skill exactly\";\n}\n```\n\n## Red Flags\n\nThese thoughts mean STOP—you're rationalizing:\n\n| Thought | Reality |\n|---------|---------|\n| \"This is just a simple question\" | Questions are tasks. Check for skills. |\n| \"I need more context first\" | Skill check comes BEFORE clarifying questions. |\n| \"Let me explore the codebase first\" | Skills tell you HOW to explore. Check first. |\n| \"I can check git/files quickly\" | Files lack conversation context. Check for skills. |\n| \"Let me gather information first\" | Skills tell you HOW to gather information. |\n| \"This doesn't need a formal skill\" | If a skill exists, use it. |\n| \"I remember this skill\" | Skills evolve. Read current version. |\n| \"This doesn't count as a task\" | Action = task. Check for skills. |\n| \"The skill is overkill\" | Simple things become complex. Use it. |\n| \"I'll just do this one thing first\" | Check BEFORE doing anything. |\n| \"This feels productive\" | Undisciplined action wastes time. Skills prevent this. |\n| \"I know what that means\" | Knowing the concept ≠ using the skill. Invoke it. |\n\n## Skill Priority\n\nWhen multiple skills could apply, use this order:\n\n1. **Process skills first** (brainstorming, debugging) - these determine HOW to approach the task\n2. **Implementation skills second** (frontend-design, mcp-builder) - these guide execution\n\n\"Let's build X\" → brainstorming first, then implementation skills.\n\"Fix this bug\" → debugging first, then domain-specific skills.\n\n## Skill Types\n\n**Rigid** (TDD, debugging): Follow exactly. Don't adapt away discipline.\n\n**Flexible** (patterns): Adapt principles to context.\n\nThe skill itself tells you which.\n\n## User Instructions\n\nInstructions say WHAT, not HOW. \"Add X\" or \"Fix Y\" doesn't mean skip workflows.\n","flock-data/skills/using-superpowers/references/codex-tools.md":"# Codex Tool Mapping\n\nSkills use Claude Code tool names. When you encounter these in a skill, use your platform equivalent:\n\n| Skill references | Codex equivalent |\n|-----------------|------------------|\n| `Task` tool (dispatch subagent) | `spawn_agent` |\n| Multiple `Task` calls (parallel) | Multiple `spawn_agent` calls |\n| Task returns result | `wait` |\n| Task completes automatically | `close_agent` to free slot |\n| `TodoWrite` (task tracking) | `update_plan` |\n| `Skill` tool (invoke a skill) | Skills load natively — just follow the instructions |\n| `Read`, `Write`, `Edit` (files) | Use your native file tools |\n| `Bash` (run commands) | Use your native shell tools |\n\n## Subagent dispatch requires collab\n\nAdd to your Codex config (`~/.codex/config.toml`):\n\n```toml\n[features]\ncollab = true\n```\n\nThis enables `spawn_agent`, `wait`, and `close_agent` for skills like `dispatching-parallel-agents` and `subagent-driven-development`.\n","flock-data/skills/using-superpowers/references/gemini-tools.md":"# Gemini CLI Tool Mapping\n\nSkills use Claude Code tool names. When you encounter these in a skill, use your platform equivalent:\n\n| Skill references | Gemini CLI equivalent |\n|-----------------|----------------------|\n| `Read` (file reading) | `read_file` |\n| `Write` (file creation) | `write_file` |\n| `Edit` (file editing) | `replace` |\n| `Bash` (run commands) | `run_shell_command` |\n| `Grep` (search file content) | `grep_search` |\n| `Glob` (search files by name) | `glob` |\n| `TodoWrite` (task tracking) | `write_todos` |\n| `Skill` tool (invoke a skill) | `activate_skill` |\n| `WebSearch` | `google_web_search` |\n| `WebFetch` | `web_fetch` |\n| `Task` tool (dispatch subagent) | No equivalent — Gemini CLI does not support subagents |\n\n## No subagent support\n\nGemini CLI has no equivalent to Claude Code's `Task` tool. Skills that rely on subagent dispatch (`subagent-driven-development`, `dispatching-parallel-agents`) will fall back to single-session execution via `executing-plans`.\n\n## Additional Gemini CLI tools\n\nThese tools are available in Gemini CLI but have no Claude Code equivalent:\n\n| Tool | Purpose |\n|------|---------|\n| `list_directory` | List files and subdirectories |\n| `save_memory` | Persist facts to GEMINI.md across sessions |\n| `ask_user` | Request structured input from the user |\n| `tracker_create_task` | Rich task management (create, update, list, visualize) |\n| `enter_plan_mode` / `exit_plan_mode` | Switch to read-only research mode before making changes |\n"},"files":{"AGENTS.md":"# AGENTS.md\n\nThis file provides guidance to CodeX when working with code in this repository.\n\n## Project Overview\n\nFlock is a multi-provider AI agent desktop application with tool orchestration, sandbox execution, visual workflow, and browser/computer-use capabilities. It's a Rust workspace with a Tauri-based desktop UI (React + TypeScript + Mantine).\n\n## Build & Development Commands\n\n### Rust Backend\n```bash\n# Build the CLI\ncargo build --release\n\n# Run in single-shot mode\ncargo run -- \"your question\"\n\n# Run interactive REPL\ncargo run\n\n# Run with specific provider/profile\ncargo run -- --profile deepseek \"your question\"\n\n# Run tests (workspace-wide)\ncargo test\n\n# Run a single crate's tests\ncargo test -p flock-core\ncargo test -p flock-agent\ncargo test -p flock-tools\ncargo test -p flock-skills\n\n# Run a specific test\ncargo test -p flock-core test_name\n\n# Lint\ncargo clippy --workspace\n```\n\n### Tauri Desktop UI\n```bash\ncd flock-ui\n\n# Install dependencies\nnpm install\n\n# Development server\nnpm run dev\n# or\nnpm run tauri dev\n\n# Build for production\nnpm run build\n# or\nnpm run tauri build\n\n# Lint frontend\nnpm run lint\n```\n\n## Architecture\n\n### Workspace Crates\n\n| Crate | Purpose |\n|-------|---------|\n| `flock-core` | Core types, config, database, IPC interface, crypto, model factory |\n| `flock-agent` | Agent engine, session management, tool execution, memory, graph orchestration, workflow engine |\n| `flock-tools` | Tool registry, built-in tools (Read/Write/Edit/Bash/Grep/Glob), MCP integration, sandbox tools (CodeExecution, SandboxExec, Browser, ComputerUse) |\n| `flock-skills` | Skill discovery, loading, frontmatter parsing, hooks, permissions, bundled skills |\n| `flock-ui/src-tauri` | Tauri desktop app backend (commands, agent state, workspace management) |\n| `workspace-hack` | cargo-hakari build optimization |\n\n### Key Dependencies\n\n- **LangGraph**: Rust reimplementation ([langgraph-rust](https://github.com/Onelevenvy/langgraph-rust)) for agent graph orchestration\n- **LLM Providers**: OpenAI-compatible (default), Anthropic, AWS Bedrock, Google Vertex\n- **Database**: SQLite via sqlx for sessions, conversations, providers, tools, MCP servers, workflows\n- **Async Runtime**: Tokio\n- **Sandbox**: Daytona cloud container runtime for isolated code execution\n- **Browser Automation**: Playwright (Python, runs inside sandbox via CDP)\n\n### Core Architecture Patterns\n\n**Agent Engine** (`flock-agent/src/engine.rs`):\n- `AgentEngine` orchestrates the LLM interaction loop\n- Uses LangGraph `CompiledStateGraph` for execution flow\n- Manages conversation history via SQLite checkpointer\n- Handles tool approval, context compression, plan mode\n\n**Tool System** (`flock-tools/src/lib.rs`):\n- `Tool` trait defines the interface: `name()`, `description()`, `input_schema()`, `execute()`\n- `ToolRegistry` manages all registered tools\n- Built-in tools: Read, Write, Edit, Bash, Grep, Glob\n- Sandbox tools: CodeExecution, SandboxExec, Browser, ComputerUse, RequestHumanAssistance\n- Extensible via MCP servers and custom tool providers\n\n**Workflow Engine** (`flock-agent/src/workflow_graph/`):\n- Visual workflow builder with ReactFlow canvas\n- 10 node types: start, llm, agent, classifier, ifelse, answer, code, human, plugin, parameter_extractor\n- `build_workflow_graph()` compiles ReactFlow JSON into LangGraph `StateGraph`\n- Streaming execution with human-in-the-loop interrupts\n\n**Sandbox System** (`flock-tools/src/daytona/`):\n- Cloud-based Daytona container runtime for isolated execution\n- VNC desktop provisioning (Xvfb, fluxbox, x11vnc, websockify)\n- Lifecycle management: create, destroy, set-public, snapshot\n- Human takeover via VNC with approval flow\n\n**Skills System** (`flock-skills/`):\n- Markdown files with YAML frontmatter in `.flock/skills/`\n- Supports `$ARGUMENTS` substitution, shell execution, conditional activation\n- Two execution contexts: `inline` (same agent) and `fork` (sub-agent)\n- Hot-reload via file watcher\n\n**Memory System** (`flock-agent/src/memory/`):\n- Four types: `user`, `feedback`, `project`, `reference`\n- Stored as Markdown files with frontmatter in `/flock/projects//memory/`\n- `MEMORY.md` index file auto-loaded into system prompt\n\n**IPC Interface** (`flock-core/src/ipc_interface/`):\n- JSON-based protocol for host ↔ agent communication\n- Events (Agent → Host): `ready`, `stream_start`, `text_delta`, `tool_request`, `tool_result`, etc.\n- Commands (Host → Agent): `message`, `stop`, `tool_approve`, `tool_deny`, `set_mode`, `set_config`\n\n**Configuration** (`flock-core/src/config/settings.rs`):\n- Config priority: CLI args/env > project `flock.toml` > global config\n- Supports profiles with inheritance (`extends`)\n- Provider compatibility settings (`ProviderCompat`) for API differences\n\n### Tauri Desktop UI\n\n**Frontend Stack**:\n- React 18 + TypeScript\n- Mantine UI v8 (components, notifications)\n- Zustand (state management)\n- React Query (server state)\n- react-markdown + react-syntax-highlighter (message rendering)\n- i18next (internationalization)\n- ReactFlow (workflow visual editor)\n\n**State Stores** (`flock-ui/src/store/`):\n- `agentStore`: Agent connection status, messages, pending approvals\n- `uiStore`: Theme, sidebar state, active view, file tree\n- `workspaceStore`: Active workspace/conversation, persisted to localStorage\n- `workflowStore`: ReactFlow nodes/edges, execution state, dirty tracking\n\n**Key Views**:\n- `HomeView`: Welcome screen, assistant selection, workspace picker\n- `WorkspaceView`: Chat panel + file tree + optional preview panel (VNC, sandbox runner)\n- `WorkflowEditor`: Visual workflow builder with ReactFlow canvas, node palette, properties panel, execution panel\n- `AssistantPage`: CRUD for custom assistants\n- `SkillsPage`: Tools, MCP servers, and skills management\n- `SettingsModal`: Model providers, basic settings, sandbox settings, system settings\n\n**Tauri Commands** (`flock-ui/src-tauri/src/commands/`):\n- Agent control: `start_agent`, `stop_agent`, `send_message`, `approve_tool`, `deny_tool`\n- Workspace: `list_workspaces`, `create_workspace`, `delete_workspace`\n- Conversations: `list_conversations`, `create_conversation`, `load_conversation_history`\n- Database: `list_providers`, `upsert_provider`, `list_models`, `upsert_model`\n- MCP: `list_mcp_servers`, `upsert_mcp_server`, `test_mcp_server`\n- Skills: `list_skills`, `get_extra_skill_dirs`, `add_extra_skill_dir`\n- Workflow: `list_workflows`, `get_workflow`, `create_workflow`, `update_workflow`, `delete_workflow`, `run_workflow`, `stop_workflow`\n\n### Data Flow\n\n1. User sends message via UI → Tauri command `send_message`\n2. `AgentState` forwards to `AgentEngine::run()`\n3. Engine builds LangGraph, invokes LLM with tools\n4. LLM response streamed as `ProtocolEvent`s via `TauriProtocolEmitter`\n5. Frontend `useEventStream` hook listens to Tauri events, updates `agentStore`\n6. Tool calls require approval (unless auto-approved) → `ToolApprovalInline` component\n7. Approved tools execute → results fed back to LLM for next turn\n\n## Code Style\n\n- Rust edition 2024\n- Workspace lints: `unused = \"allow\"`, `unused_imports = \"allow\"`\n- Do not treat repository-wide `cargo fmt --check` diffs or warnings such as unused imports/variables as blockers unless they are introduced by the current change or explicitly requested. Focus verification on build/test failures and behavior regressions.\n- Error handling: `anyhow` for applications, `thiserror` for libraries\n- Async: Tokio runtime throughout\n- Frontend: ESLint with TypeScript rules, React hooks plugin\n\n## Global Guidelines & Core Principles\n\nWhen modifying the codebase or interacting with the user, please strictly adhere to the following rules:\n\n1. **Git Commit Format**: Upon completing a task, always output a single-line English `git commit` message beginning with standard prefixes (`feat:`, `fix:`, `chore:`, etc.). Provide the message text clearly, but **DO NOT** execute git commit commands or auto-submit.\n2. **State & Logic Abstraction**: Always separate state management and command logic from UI components. Place custom Hooks in `src/hooks` and Zustand stores in `src/store` rather than defining them inside component files.\n3. **Strict i18n Internationalization**: All newly introduced UI components must leverage `i18next` for user-facing texts (e.g. tooltips, titles, toast notifications, confirm boxes). Do not hardcode Chinese or English strings inside JSX.\n4. **Component Granularity**: Strive for modularity. Always split large layout or page files into focused, localized sub-components (placed in a nested `/components` folder). Avoid keeping all view code inside single massive `index.tsx` files.\n5. **UI Specification Compliance**: Ensure any new styling aligns with the design tokens defined in `docs/resources/ui_specification.md` (e.g. cold gray colors, `--flock-accent` solid blue branding, standard monospace typographies, and hover-card transitions) in both dark and light modes.\n","CLAUDE.md":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nFlock is a multi-provider AI agent desktop application with tool orchestration, sandbox execution, visual workflow, and browser/computer-use capabilities. It's a Rust workspace with a Tauri-based desktop UI (React + TypeScript + Mantine).\n\n## Build & Development Commands\n\n### Rust Backend\n```bash\n# Build the CLI\ncargo build --release\n\n# Run in single-shot mode\ncargo run -- \"your question\"\n\n# Run interactive REPL\ncargo run\n\n# Run with specific provider/profile\ncargo run -- --profile deepseek \"your question\"\n\n# Run tests (workspace-wide)\ncargo test\n\n# Run a single crate's tests\ncargo test -p flock-core\ncargo test -p flock-agent\ncargo test -p flock-tools\ncargo test -p flock-skills\n\n# Run a specific test\ncargo test -p flock-core test_name\n\n# Lint\ncargo clippy --workspace\n```\n\n### Tauri Desktop UI\n```bash\ncd flock-ui\n\n# Install dependencies\nnpm install\n\n# Development server\nnpm run dev\n# or\nnpm run tauri dev\n\n# Build for production\nnpm run build\n# or\nnpm run tauri build\n\n# Lint frontend\nnpm run lint\n```\n\n## Architecture\n\n### Workspace Crates\n\n| Crate | Purpose |\n|-------|---------|\n| `flock-core` | Core types, config, database, IPC interface, crypto, model factory |\n| `flock-agent` | Agent engine, session management, tool execution, memory, graph orchestration, workflow engine |\n| `flock-tools` | Tool registry, built-in tools (Read/Write/Edit/Bash/Grep/Glob), MCP integration, sandbox tools (CodeExecution, SandboxExec, Browser, ComputerUse) |\n| `flock-skills` | Skill discovery, loading, frontmatter parsing, hooks, permissions, bundled skills |\n| `flock-ui/src-tauri` | Tauri desktop app backend (commands, agent state, workspace management) |\n| `workspace-hack` | cargo-hakari build optimization |\n\n### Key Dependencies\n\n- **LangGraph**: Rust reimplementation ([langgraph-rust](https://github.com/Onelevenvy/langgraph-rust)) for agent graph orchestration\n- **LLM Providers**: OpenAI-compatible (default), Anthropic, AWS Bedrock, Google Vertex\n- **Database**: SQLite via sqlx for sessions, conversations, providers, tools, MCP servers, workflows\n- **Async Runtime**: Tokio\n- **Sandbox**: Daytona cloud container runtime for isolated code execution\n- **Browser Automation**: Playwright (Python, runs inside sandbox via CDP)\n\n### Core Architecture Patterns\n\n**Agent Engine** (`flock-agent/src/engine.rs`):\n- `AgentEngine` orchestrates the LLM interaction loop\n- Uses LangGraph `CompiledStateGraph` for execution flow\n- Manages conversation history via SQLite checkpointer\n- Handles tool approval, context compression, plan mode\n\n**Tool System** (`flock-tools/src/lib.rs`):\n- `Tool` trait defines the interface: `name()`, `description()`, `input_schema()`, `execute()`\n- `ToolRegistry` manages all registered tools\n- Built-in tools: Read, Write, Edit, Bash, Grep, Glob\n- Sandbox tools: CodeExecution, SandboxExec, Browser, ComputerUse, RequestHumanAssistance\n- Extensible via MCP servers and custom tool providers\n\n**Workflow Engine** (`flock-agent/src/workflow_graph/`):\n- Visual workflow builder with ReactFlow canvas\n- 10 node types: start, llm, agent, classifier, ifelse, answer, code, human, plugin, parameter_extractor\n- `build_workflow_graph()` compiles ReactFlow JSON into LangGraph `StateGraph`\n- Streaming execution with human-in-the-loop interrupts\n\n**Sandbox System** (`flock-tools/src/daytona/`):\n- Cloud-based Daytona container runtime for isolated execution\n- VNC desktop provisioning (Xvfb, fluxbox, x11vnc, websockify)\n- Lifecycle management: create, destroy, set-public, snapshot\n- Human takeover via VNC with approval flow\n\n**Skills System** (`flock-skills/`):\n- Markdown files with YAML frontmatter in `.flock/skills/`\n- Supports `$ARGUMENTS` substitution, shell execution, conditional activation\n- Two execution contexts: `inline` (same agent) and `fork` (sub-agent)\n- Hot-reload via file watcher\n\n**Memory System** (`flock-agent/src/memory/`):\n- Four types: `user`, `feedback`, `project`, `reference`\n- Stored as Markdown files with frontmatter in `/flock/projects//memory/`\n- `MEMORY.md` index file auto-loaded into system prompt\n\n**IPC Interface** (`flock-core/src/ipc_interface/`):\n- JSON-based protocol for host ↔ agent communication\n- Events (Agent → Host): `ready`, `stream_start`, `text_delta`, `tool_request`, `tool_result`, etc.\n- Commands (Host → Agent): `message`, `stop`, `tool_approve`, `tool_deny`, `set_mode`, `set_config`\n\n**Configuration** (`flock-core/src/config/settings.rs`):\n- Config priority: CLI args/env > project `flock.toml` > global config\n- Supports profiles with inheritance (`extends`)\n- Provider compatibility settings (`ProviderCompat`) for API differences\n\n### Tauri Desktop UI\n\n**Frontend Stack**:\n- React 18 + TypeScript\n- Mantine UI v8 (components, notifications)\n- Zustand (state management)\n- React Query (server state)\n- react-markdown + react-syntax-highlighter (message rendering)\n- i18next (internationalization)\n- ReactFlow (workflow visual editor)\n\n**State Stores** (`flock-ui/src/store/`):\n- `agentStore`: Agent connection status, messages, pending approvals\n- `uiStore`: Theme, sidebar state, active view, file tree\n- `workspaceStore`: Active workspace/conversation, persisted to localStorage\n- `workflowStore`: ReactFlow nodes/edges, execution state, dirty tracking\n\n**Key Views**:\n- `HomeView`: Welcome screen, assistant selection, workspace picker\n- `WorkspaceView`: Chat panel + file tree + optional preview panel (VNC, sandbox runner)\n- `WorkflowEditor`: Visual workflow builder with ReactFlow canvas, node palette, properties panel, execution panel\n- `AssistantPage`: CRUD for custom assistants\n- `SkillsPage`: Tools, MCP servers, and skills management\n- `SettingsModal`: Model providers, basic settings, sandbox settings, system settings\n\n**Tauri Commands** (`flock-ui/src-tauri/src/commands/`):\n- Agent control: `start_agent`, `stop_agent`, `send_message`, `approve_tool`, `deny_tool`\n- Workspace: `list_workspaces`, `create_workspace`, `delete_workspace`\n- Conversations: `list_conversations`, `create_conversation`, `load_conversation_history`\n- Database: `list_providers`, `upsert_provider`, `list_models`, `upsert_model`\n- MCP: `list_mcp_servers`, `upsert_mcp_server`, `test_mcp_server`\n- Skills: `list_skills`, `get_extra_skill_dirs`, `add_extra_skill_dir`\n- Workflow: `list_workflows`, `get_workflow`, `create_workflow`, `update_workflow`, `delete_workflow`, `run_workflow`, `stop_workflow`\n\n### Data Flow\n\n1. User sends message via UI → Tauri command `send_message`\n2. `AgentState` forwards to `AgentEngine::run()`\n3. Engine builds LangGraph, invokes LLM with tools\n4. LLM response streamed as `ProtocolEvent`s via `TauriProtocolEmitter`\n5. Frontend `useEventStream` hook listens to Tauri events, updates `agentStore`\n6. Tool calls require approval (unless auto-approved) → `ToolApprovalInline` component\n7. Approved tools execute → results fed back to LLM for next turn\n\n## Code Style\n\n- Rust edition 2024\n- Workspace lints: `unused = \"allow\"`, `unused_imports = \"allow\"`\n- Do not treat repository-wide `cargo fmt --check` diffs or warnings such as unused imports/variables as blockers unless they are introduced by the current change or explicitly requested. Focus verification on build/test failures and behavior regressions.\n- Error handling: `anyhow` for applications, `thiserror` for libraries\n- Async: Tokio runtime throughout\n- Frontend: ESLint with TypeScript rules, React hooks plugin\n\n\n## Global Guidelines & Core Principles\n\nWhen modifying the codebase or interacting with the user, please strictly adhere to the following rules:\n\n1. **Git Commit Format**: Upon completing a task, always output a single-line English `git commit` message beginning with standard prefixes (`feat:`, `fix:`, `chore:`, etc.). Provide the message text clearly, but **DO NOT** execute git commit commands or auto-submit.\n2. **State & Logic Abstraction**: Always separate state management and command logic from UI components. Place custom Hooks in `src/hooks` and Zustand stores in `src/store` rather than defining them inside component files.\n3. **Strict i18n Internationalization**: All newly introduced UI components must leverage `i18next` for user-facing texts (e.g. tooltips, titles, toast notifications, confirm boxes). Do not hardcode Chinese or English strings inside JSX.\n4. **Component Granularity**: Strive for modularity. Always split large layout or page files into focused, localized sub-components (placed in a nested `/components` folder). Avoid keeping all view code inside single massive `index.tsx` files.\n5. **UI Specification Compliance**: Ensure any new styling aligns with the design tokens defined in `docs/resources/ui_specification.md` (e.g. cold gray colors, `--flock-accent` solid blue branding, standard monospace typographies, and hover-card transitions) in both dark and light modes.\n","flock-data/skills/frontend-design/SKILL.md":"---\nname: frontend-design\ndescription: Create distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, or applications. Generates creative, polished code that avoids generic AI aesthetics.\nlicense: Complete terms in LICENSE.txt\n---\n\nThis skill guides creation of distinctive, production-grade frontend interfaces that avoid generic \"AI slop\" aesthetics. Implement real working code with exceptional attention to aesthetic details and creative choices.\n\nThe user provides frontend requirements: a component, page, application, or interface to build. They may include context about the purpose, audience, or technical constraints.\n\n## Design Thinking\n\nBefore coding, understand the context and commit to a BOLD aesthetic direction:\n- **Purpose**: What problem does this interface solve? Who uses it?\n- **Tone**: Pick an extreme: brutally minimal, maximalist chaos, retro-futuristic, organic/natural, luxury/refined, playful/toy-like, editorial/magazine, brutalist/raw, art deco/geometric, soft/pastel, industrial/utilitarian, etc. There are so many flavors to choose from. Use these for inspiration but design one that is true to the aesthetic direction.\n- **Constraints**: Technical requirements (framework, performance, accessibility).\n- **Differentiation**: What makes this UNFORGETTABLE? What's the one thing someone will remember?\n\n**CRITICAL**: Choose a clear conceptual direction and execute it with precision. Bold maximalism and refined minimalism both work - the key is intentionality, not intensity.\n\nThen implement working code (HTML/CSS/JS, React, Vue, etc.) that is:\n- Production-grade and functional\n- Visually striking and memorable\n- Cohesive with a clear aesthetic point-of-view\n- Meticulously refined in every detail\n\n## Frontend Aesthetics Guidelines\n\nFocus on:\n- **Typography**: Choose fonts that are beautiful, unique, and interesting. Avoid generic fonts like Arial and Inter; opt instead for distinctive choices that elevate the frontend's aesthetics; unexpected, characterful font choices. Pair a distinctive display font with a refined body font.\n- **Color & Theme**: Commit to a cohesive aesthetic. Use CSS variables for consistency. Dominant colors with sharp accents outperform timid, evenly-distributed palettes.\n- **Motion**: Use animations for effects and micro-interactions. Prioritize CSS-only solutions for HTML. Use Motion library for React when available. Focus on high-impact moments: one well-orchestrated page load with staggered reveals (animation-delay) creates more delight than scattered micro-interactions. Use scroll-triggering and hover states that surprise.\n- **Spatial Composition**: Unexpected layouts. Asymmetry. Overlap. Diagonal flow. Grid-breaking elements. Generous negative space OR controlled density.\n- **Backgrounds & Visual Details**: Create atmosphere and depth rather than defaulting to solid colors. Add contextual effects and textures that match the overall aesthetic. Apply creative forms like gradient meshes, noise textures, geometric patterns, layered transparencies, dramatic shadows, decorative borders, custom cursors, and grain overlays.\n\nNEVER use generic AI-generated aesthetics like overused font families (Inter, Roboto, Arial, system fonts), cliched color schemes (particularly purple gradients on white backgrounds), predictable layouts and component patterns, and cookie-cutter design that lacks context-specific character.\n\nInterpret creatively and make unexpected choices that feel genuinely designed for the context. No design should be the same. Vary between light and dark themes, different fonts, different aesthetics. NEVER converge on common choices (Space Grotesk, for example) across generations.\n\n**IMPORTANT**: Match implementation complexity to the aesthetic vision. Maximalist designs need elaborate code with extensive animations and effects. Minimalist or refined designs need restraint, precision, and careful attention to spacing, typography, and subtle details. Elegance comes from executing the vision well.\n\nRemember: Claude is capable of extraordinary creative work. Don't hold back, show what can truly be created when thinking outside the box and committing fully to a distinctive vision.","flock-data/skills/karpathy/SKILL.md":"---\nname: karpathy\ndescription: 'Coding principles for simplicity, clarity, and surgical precision. Use when you want Claude to think before coding, keep solutions minimal, make surgical changes, and define verifiable success criteria. Triggers on: karpathy mode, think before coding, simplicity first, surgical changes.'\n---\n\n# Karpathy Coding Principles\n\nPrinciples for writing clean, minimal, and purposeful code.\n\n---\n\n## 1. Think Before Coding\n\n**Don't assume. Don't hide confusion. Surface tradeoffs.**\n\nBefore implementing:\n\n- State your assumptions explicitly. If uncertain, ask.\n- If multiple interpretations exist, present them - don't pick silently.\n- If a simpler approach exists, say so. Push back when warranted.\n- If something is unclear, stop. Name what's confusing. Ask.\n\n---\n\n## 2. Simplicity First\n\n**Minimum code that solves the problem. Nothing speculative.**\n\n- No features beyond what was asked.\n- No abstractions for single-use code.\n- No \"flexibility\" or \"configurability\" that wasn't requested.\n- No error handling for impossible scenarios.\n- If you write 200 lines and it could be 50, rewrite it.\n\nAsk yourself: \"Would a senior engineer say this is overcomplicated?\" If yes, simplify.\n\n---\n\n## 3. Surgical Changes\n\n**Touch only what you must. Clean up only your own mess.**\n\nWhen editing existing code:\n\n- Don't \"improve\" adjacent code, comments, or formatting.\n- Don't refactor things that aren't broken.\n- Match existing style, even if you'd do it differently.\n- If you notice unrelated dead code, mention it - don't delete it.\n\nWhen your changes create orphans:\n\n- Remove imports/variables/functions that YOUR changes made unused.\n- Don't remove pre-existing dead code unless asked.\n\nThe test: Every changed line should trace directly to the user's request.\n\n---\n\n## 4. Goal-Driven Execution\n\n**Define success criteria. Loop until verified.**\n\nTransform tasks into verifiable goals:\n\n- \"Add validation\" -> \"Write tests for invalid inputs, then make them pass\"\n- \"Fix the bug\" -> \"Write a test that reproduces it, then make it pass\"\n- \"Refactor X\" -> \"Ensure tests pass before and after\"\n\nFor multi-step tasks, state a brief plan:\n\n```\n1. [Step] -> verify: [check]\n2. [Step] -> verify: [check]\n3. [Step] -> verify: [check]\n```\n\nStrong success criteria let you loop independently. Weak criteria (\"make it work\") require constant clarification.\n","flock-data/skills/karpathy/metadata.json":"{\n \"name\": \"karpathy\",\n \"description\": \"Coding principles for simplicity, clarity, and surgical precision. Use when you want Claude to think before coding, keep solutions minimal, make surgical changes, and define verifiable success criteria. Triggers on: karpathy mode, think before coding, simplicity first, surgical changes.\",\n \"category\": \"devops\",\n \"canonical_category\": \"devops\",\n \"repository\": \"majiayu000/claude-skill-registry\",\n \"repository_url\": \"https://github.com/majiayu000/claude-skill-registry\",\n \"author\": \"majiayu000\",\n \"author_avatar\": \"https://github.com/majiayu000.png\",\n \"file_path\": \"skills/other/karpathy/SKILL.md\",\n \"source\": \"github_code_search\",\n \"stars\": 0,\n \"quality_score\": 45,\n \"best_practices_score\": 55,\n \"skill_level\": 2,\n \"skill_level_label\": \"instructions\",\n \"has_scripts\": false,\n \"has_extra_files\": true,\n \"downloads\": 0,\n \"content_hash\": \"ec43cf3b4ae428984a1ad36d4814bf6f95e590ada0c997fab9f1ecb4c89b2886\",\n \"indexed_at\": \"2026-02-25T19:40:17.443Z\",\n \"synced_at\": \"2026-02-28T04:09:42.439Z\",\n \"omni_registry_url\": \"https://omni-skill-registry.omniroute.online/#/skill/27fec53067f2a7ff89a65128ed8a407f2e5f7a042d2891790054748048e9f33b\",\n \"install_command\": \"mkdir -p .claude/skills/karpathy && curl -sL \\\"https://raw.githubusercontent.com/majiayu000/claude-skill-registry/main/skills/other/karpathy/SKILL.md\\\" > .claude/skills/karpathy/SKILL.md\",\n \"raw_url\": \"https://raw.githubusercontent.com/majiayu000/claude-skill-registry/main/skills/other/karpathy/SKILL.md\"\n}\n","flock-data/skills/planning-with-files/SKILL.md":"---\nname: planning-with-files\nversion: \"2.10.0\"\ndescription: Implements Manus-style file-based planning for complex tasks. Creates task_plan.md, findings.md, and progress.md. Use when starting complex multi-step tasks, research projects, or any task requiring >5 tool calls. Now with automatic session recovery after /clear.\nuser-invocable: true\nallowed-tools:\n - Read\n - Write\n - Edit\n - Bash\n - Glob\n - Grep\n - WebFetch\n - WebSearch\nhooks:\n PreToolUse:\n - matcher: \"Write|Edit|Bash|Read|Glob|Grep\"\n hooks:\n - type: command\n command: \"cat task_plan.md 2>/dev/null | head -30 || true\"\n PostToolUse:\n - matcher: \"Write|Edit\"\n hooks:\n - type: command\n command: \"echo '[planning-with-files] File updated. If this completes a phase, update task_plan.md status.'\"\n Stop:\n - hooks:\n - type: command\n command: |\n SCRIPT_DIR=\"${CLAUDE_PLUGIN_ROOT:-$HOME/.claude/plugins/planning-with-files}/scripts\"\n\n IS_WINDOWS=0\n if [ \"${OS-}\" = \"Windows_NT\" ]; then\n IS_WINDOWS=1\n else\n UNAME_S=\"$(uname -s 2>/dev/null || echo '')\"\n case \"$UNAME_S\" in\n CYGWIN*|MINGW*|MSYS*) IS_WINDOWS=1 ;;\n esac\n fi\n\n if [ \"$IS_WINDOWS\" -eq 1 ]; then\n if command -v pwsh >/dev/null 2>&1; then\n pwsh -ExecutionPolicy Bypass -File \"$SCRIPT_DIR/check-complete.ps1\" 2>/dev/null ||\n powershell -ExecutionPolicy Bypass -File \"$SCRIPT_DIR/check-complete.ps1\" 2>/dev/null ||\n sh \"$SCRIPT_DIR/check-complete.sh\"\n else\n powershell -ExecutionPolicy Bypass -File \"$SCRIPT_DIR/check-complete.ps1\" 2>/dev/null ||\n sh \"$SCRIPT_DIR/check-complete.sh\"\n fi\n else\n sh \"$SCRIPT_DIR/check-complete.sh\"\n fi\n---\n\n# Planning with Files\n\nWork like Manus: Use persistent markdown files as your \"working memory on disk.\"\n\n## FIRST: Check for Previous Session (v2.2.0)\n\n**Before starting work**, check for unsynced context from a previous session:\n\n```bash\n# Linux/macOS\n$(command -v python3 || command -v python) ${CLAUDE_PLUGIN_ROOT}/scripts/session-catchup.py \"$(pwd)\"\n```\n\n```powershell\n# Windows PowerShell\n& (Get-Command python -ErrorAction SilentlyContinue).Source \"$env:USERPROFILE\\.claude\\skills\\planning-with-files\\scripts\\session-catchup.py\" (Get-Location)\n```\n\nIf catchup report shows unsynced context:\n1. Run `git diff --stat` to see actual code changes\n2. Read current planning files\n3. Update planning files based on catchup + git diff\n4. Then proceed with task\n\n## Important: Where Files Go\n\n- **Templates** are in `${CLAUDE_PLUGIN_ROOT}/templates/`\n- **Your planning files** go in **your project directory**\n\n| Location | What Goes There |\n|----------|-----------------|\n| Skill directory (`${CLAUDE_PLUGIN_ROOT}/`) | Templates, scripts, reference docs |\n| Your project directory | `task_plan.md`, `findings.md`, `progress.md` |\n\n## Quick Start\n\nBefore ANY complex task:\n\n1. **Create `task_plan.md`** — Use [templates/task_plan.md](templates/task_plan.md) as reference\n2. **Create `findings.md`** — Use [templates/findings.md](templates/findings.md) as reference\n3. **Create `progress.md`** — Use [templates/progress.md](templates/progress.md) as reference\n4. **Re-read plan before decisions** — Refreshes goals in attention window\n5. **Update after each phase** — Mark complete, log errors\n\n> **Note:** Planning files go in your project root, not the skill installation folder.\n\n## The Core Pattern\n\n```\nContext Window = RAM (volatile, limited)\nFilesystem = Disk (persistent, unlimited)\n\n→ Anything important gets written to disk.\n```\n\n## File Purposes\n\n| File | Purpose | When to Update |\n|------|---------|----------------|\n| `task_plan.md` | Phases, progress, decisions | After each phase |\n| `findings.md` | Research, discoveries | After ANY discovery |\n| `progress.md` | Session log, test results | Throughout session |\n\n## Critical Rules\n\n### 1. Create Plan First\nNever start a complex task without `task_plan.md`. Non-negotiable.\n\n### 2. The 2-Action Rule\n> \"After every 2 view/browser/search operations, IMMEDIATELY save key findings to text files.\"\n\nThis prevents visual/multimodal information from being lost.\n\n### 3. Read Before Decide\nBefore major decisions, read the plan file. This keeps goals in your attention window.\n\n### 4. Update After Act\nAfter completing any phase:\n- Mark phase status: `in_progress` → `complete`\n- Log any errors encountered\n- Note files created/modified\n\n### 5. Log ALL Errors\nEvery error goes in the plan file. This builds knowledge and prevents repetition.\n\n```markdown\n## Errors Encountered\n| Error | Attempt | Resolution |\n|-------|---------|------------|\n| FileNotFoundError | 1 | Created default config |\n| API timeout | 2 | Added retry logic |\n```\n\n### 6. Never Repeat Failures\n```\nif action_failed:\n next_action != same_action\n```\nTrack what you tried. Mutate the approach.\n\n## The 3-Strike Error Protocol\n\n```\nATTEMPT 1: Diagnose & Fix\n → Read error carefully\n → Identify root cause\n → Apply targeted fix\n\nATTEMPT 2: Alternative Approach\n → Same error? Try different method\n → Different tool? Different library?\n → NEVER repeat exact same failing action\n\nATTEMPT 3: Broader Rethink\n → Question assumptions\n → Search for solutions\n → Consider updating the plan\n\nAFTER 3 FAILURES: Escalate to User\n → Explain what you tried\n → Share the specific error\n → Ask for guidance\n```\n\n## Read vs Write Decision Matrix\n\n| Situation | Action | Reason |\n|-----------|--------|--------|\n| Just wrote a file | DON'T read | Content still in context |\n| Viewed image/PDF | Write findings NOW | Multimodal → text before lost |\n| Browser returned data | Write to file | Screenshots don't persist |\n| Starting new phase | Read plan/findings | Re-orient if context stale |\n| Error occurred | Read relevant file | Need current state to fix |\n| Resuming after gap | Read all planning files | Recover state |\n\n## The 5-Question Reboot Test\n\nIf you can answer these, your context management is solid:\n\n| Question | Answer Source |\n|----------|---------------|\n| Where am I? | Current phase in task_plan.md |\n| Where am I going? | Remaining phases |\n| What's the goal? | Goal statement in plan |\n| What have I learned? | findings.md |\n| What have I done? | progress.md |\n\n## When to Use This Pattern\n\n**Use for:**\n- Multi-step tasks (3+ steps)\n- Research tasks\n- Building/creating projects\n- Tasks spanning many tool calls\n- Anything requiring organization\n\n**Skip for:**\n- Simple questions\n- Single-file edits\n- Quick lookups\n\n## Templates\n\nCopy these templates to start:\n\n- [templates/task_plan.md](templates/task_plan.md) — Phase tracking\n- [templates/findings.md](templates/findings.md) — Research storage\n- [templates/progress.md](templates/progress.md) — Session logging\n\n## Scripts\n\nHelper scripts for automation:\n\n- `scripts/init-session.sh` — Initialize all planning files\n- `scripts/check-complete.sh` — Verify all phases complete\n- `scripts/session-catchup.py` — Recover context from previous session (v2.2.0)\n\n## Advanced Topics\n\n- **Manus Principles:** See [reference.md](reference.md)\n- **Real Examples:** See [examples.md](examples.md)\n\n## Anti-Patterns\n\n| Don't | Do Instead |\n|-------|------------|\n| Use TodoWrite for persistence | Create task_plan.md file |\n| State goals once and forget | Re-read plan before decisions |\n| Hide errors and retry silently | Log errors to plan file |\n| Stuff everything in context | Store large content in files |\n| Start executing immediately | Create plan file FIRST |\n| Repeat failed actions | Track attempts, mutate approach |\n| Create files in skill directory | Create files in your project |\n","flock-data/skills/planning-with-files/examples.md":"# Examples: Planning with Files in Action\n\n## Example 1: Research Task\n\n**User Request:** \"Research the benefits of morning exercise and write a summary\"\n\n### Loop 1: Create Plan\n```bash\nWrite task_plan.md\n```\n\n```markdown\n# Task Plan: Morning Exercise Benefits Research\n\n## Goal\nCreate a research summary on the benefits of morning exercise.\n\n## Phases\n- [ ] Phase 1: Create this plan ✓\n- [ ] Phase 2: Search and gather sources\n- [ ] Phase 3: Synthesize findings\n- [ ] Phase 4: Deliver summary\n\n## Key Questions\n1. What are the physical health benefits?\n2. What are the mental health benefits?\n3. What scientific studies support this?\n\n## Status\n**Currently in Phase 1** - Creating plan\n```\n\n### Loop 2: Research\n```bash\nRead task_plan.md # Refresh goals\nWebSearch \"morning exercise benefits\"\nWrite notes.md # Store findings\nEdit task_plan.md # Mark Phase 2 complete\n```\n\n### Loop 3: Synthesize\n```bash\nRead task_plan.md # Refresh goals\nRead notes.md # Get findings\nWrite morning_exercise_summary.md\nEdit task_plan.md # Mark Phase 3 complete\n```\n\n### Loop 4: Deliver\n```bash\nRead task_plan.md # Verify complete\nDeliver morning_exercise_summary.md\n```\n\n---\n\n## Example 2: Bug Fix Task\n\n**User Request:** \"Fix the login bug in the authentication module\"\n\n### task_plan.md\n```markdown\n# Task Plan: Fix Login Bug\n\n## Goal\nIdentify and fix the bug preventing successful login.\n\n## Phases\n- [x] Phase 1: Understand the bug report ✓\n- [x] Phase 2: Locate relevant code ✓\n- [ ] Phase 3: Identify root cause (CURRENT)\n- [ ] Phase 4: Implement fix\n- [ ] Phase 5: Test and verify\n\n## Key Questions\n1. What error message appears?\n2. Which file handles authentication?\n3. What changed recently?\n\n## Decisions Made\n- Auth handler is in src/auth/login.ts\n- Error occurs in validateToken() function\n\n## Errors Encountered\n- [Initial] TypeError: Cannot read property 'token' of undefined\n → Root cause: user object not awaited properly\n\n## Status\n**Currently in Phase 3** - Found root cause, preparing fix\n```\n\n---\n\n## Example 3: Feature Development\n\n**User Request:** \"Add a dark mode toggle to the settings page\"\n\n### The 3-File Pattern in Action\n\n**task_plan.md:**\n```markdown\n# Task Plan: Dark Mode Toggle\n\n## Goal\nAdd functional dark mode toggle to settings.\n\n## Phases\n- [x] Phase 1: Research existing theme system ✓\n- [x] Phase 2: Design implementation approach ✓\n- [ ] Phase 3: Implement toggle component (CURRENT)\n- [ ] Phase 4: Add theme switching logic\n- [ ] Phase 5: Test and polish\n\n## Decisions Made\n- Using CSS custom properties for theme\n- Storing preference in localStorage\n- Toggle component in SettingsPage.tsx\n\n## Status\n**Currently in Phase 3** - Building toggle component\n```\n\n**notes.md:**\n```markdown\n# Notes: Dark Mode Implementation\n\n## Existing Theme System\n- Located in: src/styles/theme.ts\n- Uses: CSS custom properties\n- Current themes: light only\n\n## Files to Modify\n1. src/styles/theme.ts - Add dark theme colors\n2. src/components/SettingsPage.tsx - Add toggle\n3. src/hooks/useTheme.ts - Create new hook\n4. src/App.tsx - Wrap with ThemeProvider\n\n## Color Decisions\n- Dark background: #1a1a2e\n- Dark surface: #16213e\n- Dark text: #eaeaea\n```\n\n**dark_mode_implementation.md:** (deliverable)\n```markdown\n# Dark Mode Implementation\n\n## Changes Made\n\n### 1. Added dark theme colors\nFile: src/styles/theme.ts\n...\n\n### 2. Created useTheme hook\nFile: src/hooks/useTheme.ts\n...\n```\n\n---\n\n## Example 4: Error Recovery Pattern\n\nWhen something fails, DON'T hide it:\n\n### Before (Wrong)\n```\nAction: Read config.json\nError: File not found\nAction: Read config.json # Silent retry\nAction: Read config.json # Another retry\n```\n\n### After (Correct)\n```\nAction: Read config.json\nError: File not found\n\n# Update task_plan.md:\n## Errors Encountered\n- config.json not found → Will create default config\n\nAction: Write config.json (default config)\nAction: Read config.json\nSuccess!\n```\n\n---\n\n## The Read-Before-Decide Pattern\n\n**Always read your plan before major decisions:**\n\n```\n[Many tool calls have happened...]\n[Context is getting long...]\n[Original goal might be forgotten...]\n\n→ Read task_plan.md # This brings goals back into attention!\n→ Now make the decision # Goals are fresh in context\n```\n\nThis is why Manus can handle ~50 tool calls without losing track. The plan file acts as a \"goal refresh\" mechanism.\n","flock-data/skills/planning-with-files/reference.md":"# Reference: Manus Context Engineering Principles\n\nThis skill is based on context engineering principles from Manus, the AI agent company acquired by Meta for $2 billion in December 2025.\n\n## The 6 Manus Principles\n\n### Principle 1: Design Around KV-Cache\n\n> \"KV-cache hit rate is THE single most important metric for production AI agents.\"\n\n**Statistics:**\n- ~100:1 input-to-output token ratio\n- Cached tokens: $0.30/MTok vs Uncached: $3/MTok\n- 10x cost difference!\n\n**Implementation:**\n- Keep prompt prefixes STABLE (single-token change invalidates cache)\n- NO timestamps in system prompts\n- Make context APPEND-ONLY with deterministic serialization\n\n### Principle 2: Mask, Don't Remove\n\nDon't dynamically remove tools (breaks KV-cache). Use logit masking instead.\n\n**Best Practice:** Use consistent action prefixes (e.g., `browser_`, `shell_`, `file_`) for easier masking.\n\n### Principle 3: Filesystem as External Memory\n\n> \"Markdown is my 'working memory' on disk.\"\n\n**The Formula:**\n```\nContext Window = RAM (volatile, limited)\nFilesystem = Disk (persistent, unlimited)\n```\n\n**Compression Must Be Restorable:**\n- Keep URLs even if web content is dropped\n- Keep file paths when dropping document contents\n- Never lose the pointer to full data\n\n### Principle 4: Manipulate Attention Through Recitation\n\n> \"Creates and updates todo.md throughout tasks to push global plan into model's recent attention span.\"\n\n**Problem:** After ~50 tool calls, models forget original goals (\"lost in the middle\" effect).\n\n**Solution:** Re-read `task_plan.md` before each decision. Goals appear in the attention window.\n\n```\nStart of context: [Original goal - far away, forgotten]\n...many tool calls...\nEnd of context: [Recently read task_plan.md - gets ATTENTION!]\n```\n\n### Principle 5: Keep the Wrong Stuff In\n\n> \"Leave the wrong turns in the context.\"\n\n**Why:**\n- Failed actions with stack traces let model implicitly update beliefs\n- Reduces mistake repetition\n- Error recovery is \"one of the clearest signals of TRUE agentic behavior\"\n\n### Principle 6: Don't Get Few-Shotted\n\n> \"Uniformity breeds fragility.\"\n\n**Problem:** Repetitive action-observation pairs cause drift and hallucination.\n\n**Solution:** Introduce controlled variation:\n- Vary phrasings slightly\n- Don't copy-paste patterns blindly\n- Recalibrate on repetitive tasks\n\n---\n\n## The 3 Context Engineering Strategies\n\nBased on Lance Martin's analysis of Manus architecture.\n\n### Strategy 1: Context Reduction\n\n**Compaction:**\n```\nTool calls have TWO representations:\n├── FULL: Raw tool content (stored in filesystem)\n└── COMPACT: Reference/file path only\n\nRULES:\n- Apply compaction to STALE (older) tool results\n- Keep RECENT results FULL (to guide next decision)\n```\n\n**Summarization:**\n- Applied when compaction reaches diminishing returns\n- Generated using full tool results\n- Creates standardized summary objects\n\n### Strategy 2: Context Isolation (Multi-Agent)\n\n**Architecture:**\n```\n┌─────────────────────────────────┐\n│ PLANNER AGENT │\n│ └─ Assigns tasks to sub-agents │\n├─────────────────────────────────┤\n│ KNOWLEDGE MANAGER │\n│ └─ Reviews conversations │\n│ └─ Determines filesystem store │\n├─────────────────────────────────┤\n│ EXECUTOR SUB-AGENTS │\n│ └─ Perform assigned tasks │\n│ └─ Have own context windows │\n└─────────────────────────────────┘\n```\n\n**Key Insight:** Manus originally used `todo.md` for task planning but found ~33% of actions were spent updating it. Shifted to dedicated planner agent calling executor sub-agents.\n\n### Strategy 3: Context Offloading\n\n**Tool Design:**\n- Use <20 atomic functions total\n- Store full results in filesystem, not context\n- Use `glob` and `grep` for searching\n- Progressive disclosure: load information only as needed\n\n---\n\n## The Agent Loop\n\nManus operates in a continuous 7-step loop:\n\n```\n┌─────────────────────────────────────────┐\n│ 1. ANALYZE CONTEXT │\n│ - Understand user intent │\n│ - Assess current state │\n│ - Review recent observations │\n├─────────────────────────────────────────┤\n│ 2. THINK │\n│ - Should I update the plan? │\n│ - What's the next logical action? │\n│ - Are there blockers? │\n├─────────────────────────────────────────┤\n│ 3. SELECT TOOL │\n│ - Choose ONE tool │\n│ - Ensure parameters available │\n├─────────────────────────────────────────┤\n│ 4. EXECUTE ACTION │\n│ - Tool runs in sandbox │\n├─────────────────────────────────────────┤\n│ 5. RECEIVE OBSERVATION │\n│ - Result appended to context │\n├─────────────────────────────────────────┤\n│ 6. ITERATE │\n│ - Return to step 1 │\n│ - Continue until complete │\n├─────────────────────────────────────────┤\n│ 7. DELIVER OUTCOME │\n│ - Send results to user │\n│ - Attach all relevant files │\n└─────────────────────────────────────────┘\n```\n\n---\n\n## File Types Manus Creates\n\n| File | Purpose | When Created | When Updated |\n|------|---------|--------------|--------------|\n| `task_plan.md` | Phase tracking, progress | Task start | After completing phases |\n| `findings.md` | Discoveries, decisions | After ANY discovery | After viewing images/PDFs |\n| `progress.md` | Session log, what's done | At breakpoints | Throughout session |\n| Code files | Implementation | Before execution | After errors |\n\n---\n\n## Critical Constraints\n\n- **Single-Action Execution:** ONE tool call per turn. No parallel execution.\n- **Plan is Required:** Agent must ALWAYS know: goal, current phase, remaining phases\n- **Files are Memory:** Context = volatile. Filesystem = persistent.\n- **Never Repeat Failures:** If action failed, next action MUST be different\n- **Communication is a Tool:** Message types: `info` (progress), `ask` (blocking), `result` (terminal)\n\n---\n\n## Manus Statistics\n\n| Metric | Value |\n|--------|-------|\n| Average tool calls per task | ~50 |\n| Input-to-output token ratio | 100:1 |\n| Acquisition price | $2 billion |\n| Time to $100M revenue | 8 months |\n| Framework refactors since launch | 5 times |\n\n---\n\n## Key Quotes\n\n> \"Context window = RAM (volatile, limited). Filesystem = Disk (persistent, unlimited). Anything important gets written to disk.\"\n\n> \"if action_failed: next_action != same_action. Track what you tried. Mutate the approach.\"\n\n> \"Error recovery is one of the clearest signals of TRUE agentic behavior.\"\n\n> \"KV-cache hit rate is the single most important metric for a production-stage AI agent.\"\n\n> \"Leave the wrong turns in the context.\"\n\n---\n\n## Source\n\nBased on Manus's official context engineering documentation:\nhttps://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus\n","flock-data/skills/planning-with-files/templates/findings.md":"# Findings & Decisions\n\n\n## Requirements\n\n\n-\n\n## Research Findings\n\n\n-\n\n## Technical Decisions\n\n\n| Decision | Rationale |\n|----------|-----------|\n| | |\n\n## Issues Encountered\n\n\n| Issue | Resolution |\n|-------|------------|\n| | |\n\n## Resources\n\n\n-\n\n## Visual/Browser Findings\n\n\n\n-\n\n---\n\n*Update this file after every 2 view/browser/search operations*\n*This prevents visual information from being lost*\n","flock-data/skills/planning-with-files/templates/progress.md":"# Progress Log\n\n\n## Session: [DATE]\n\n\n### Phase 1: [Title]\n\n- **Status:** in_progress\n- **Started:** [timestamp]\n\n- Actions taken:\n \n -\n- Files created/modified:\n \n -\n\n### Phase 2: [Title]\n\n- **Status:** pending\n- Actions taken:\n -\n- Files created/modified:\n -\n\n## Test Results\n\n| Test | Input | Expected | Actual | Status |\n|------|-------|----------|--------|--------|\n| | | | | |\n\n## Error Log\n\n\n| Timestamp | Error | Attempt | Resolution |\n|-----------|-------|---------|------------|\n| | | 1 | |\n\n## 5-Question Reboot Check\n\n\n| Question | Answer |\n|----------|--------|\n| Where am I? | Phase X |\n| Where am I going? | Remaining phases |\n| What's the goal? | [goal statement] |\n| What have I learned? | See findings.md |\n| What have I done? | See above |\n\n---\n\n*Update after completing each phase or encountering errors*\n","flock-data/skills/planning-with-files/templates/task_plan.md":"# Task Plan: [Brief Description]\n\n\n## Goal\n\n[One sentence describing the end state]\n\n## Current Phase\n\nPhase 1\n\n## Phases\n\n\n### Phase 1: Requirements & Discovery\n\n- [ ] Understand user intent\n- [ ] Identify constraints and requirements\n- [ ] Document findings in findings.md\n- **Status:** in_progress\n\n\n### Phase 2: Planning & Structure\n\n- [ ] Define technical approach\n- [ ] Create project structure if needed\n- [ ] Document decisions with rationale\n- **Status:** pending\n\n### Phase 3: Implementation\n\n- [ ] Execute the plan step by step\n- [ ] Write code to files before executing\n- [ ] Test incrementally\n- **Status:** pending\n\n### Phase 4: Testing & Verification\n\n- [ ] Verify all requirements met\n- [ ] Document test results in progress.md\n- [ ] Fix any issues found\n- **Status:** pending\n\n### Phase 5: Delivery\n\n- [ ] Review all output files\n- [ ] Ensure deliverables are complete\n- [ ] Deliver to user\n- **Status:** pending\n\n## Key Questions\n\n1. [Question to answer]\n2. [Question to answer]\n\n## Decisions Made\n\n| Decision | Rationale |\n|----------|-----------|\n| | |\n\n## Errors Encountered\n\n| Error | Attempt | Resolution |\n|-------|---------|------------|\n| | 1 | |\n\n## Notes\n\n- Update phase status as you progress: pending → in_progress → complete\n- Re-read this plan before major decisions (attention manipulation)\n- Log ALL errors - they help avoid repetition\n","flock-data/skills/skill-creator/SKILL.md":"---\nname: skill-creator\ndescription: Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.\n---\n\n# Skill Creator\n\nA skill for creating new skills and iteratively improving them.\n\nAt a high level, the process of creating a skill goes like this:\n\n- Decide what you want the skill to do and roughly how it should do it\n- Write a draft of the skill\n- Create a few test prompts and run claude-with-access-to-the-skill on them\n- Help the user evaluate the results both qualitatively and quantitatively\n - While the runs happen in the background, draft some quantitative evals if there aren't any (if there are some, you can either use as is or modify if you feel something needs to change about them). Then explain them to the user (or if they already existed, explain the ones that already exist)\n - Use the `eval-viewer/generate_review.py` script to show the user the results for them to look at, and also let them look at the quantitative metrics\n- Rewrite the skill based on feedback from the user's evaluation of the results (and also if there are any glaring flaws that become apparent from the quantitative benchmarks)\n- Repeat until you're satisfied\n- Expand the test set and try again at larger scale\n\nYour job when using this skill is to figure out where the user is in this process and then jump in and help them progress through these stages. So for instance, maybe they're like \"I want to make a skill for X\". You can help narrow down what they mean, write a draft, write the test cases, figure out how they want to evaluate, run all the prompts, and repeat.\n\nOn the other hand, maybe they already have a draft of the skill. In this case you can go straight to the eval/iterate part of the loop.\n\nOf course, you should always be flexible and if the user is like \"I don't need to run a bunch of evaluations, just vibe with me\", you can do that instead.\n\nThen after the skill is done (but again, the order is flexible), you can also run the skill description improver, which we have a whole separate script for, to optimize the triggering of the skill.\n\nCool? Cool.\n\n## Communicating with the user\n\nThe skill creator is liable to be used by people across a wide range of familiarity with coding jargon. If you haven't heard (and how could you, it's only very recently that it started), there's a trend now where the power of Claude is inspiring plumbers to open up their terminals, parents and grandparents to google \"how to install npm\". On the other hand, the bulk of users are probably fairly computer-literate.\n\nSo please pay attention to context cues to understand how to phrase your communication! In the default case, just to give you some idea:\n\n- \"evaluation\" and \"benchmark\" are borderline, but OK\n- for \"JSON\" and \"assertion\" you want to see serious cues from the user that they know what those things are before using them without explaining them\n\nIt's OK to briefly explain terms if you're in doubt, and feel free to clarify terms with a short definition if you're unsure if the user will get it.\n\n---\n\n## Creating a skill\n\n### Capture Intent\n\nStart by understanding the user's intent. The current conversation might already contain a workflow the user wants to capture (e.g., they say \"turn this into a skill\"). If so, extract answers from the conversation history first — the tools used, the sequence of steps, corrections the user made, input/output formats observed. The user may need to fill the gaps, and should confirm before proceeding to the next step.\n\n1. What should this skill enable Claude to do?\n2. When should this skill trigger? (what user phrases/contexts)\n3. What's the expected output format?\n4. Should we set up test cases to verify the skill works? Skills with objectively verifiable outputs (file transforms, data extraction, code generation, fixed workflow steps) benefit from test cases. Skills with subjective outputs (writing style, art) often don't need them. Suggest the appropriate default based on the skill type, but let the user decide.\n\n### Interview and Research\n\nProactively ask questions about edge cases, input/output formats, example files, success criteria, and dependencies. Wait to write test prompts until you've got this part ironed out.\n\nCheck available MCPs - if useful for research (searching docs, finding similar skills, looking up best practices), research in parallel via subagents if available, otherwise inline. Come prepared with context to reduce burden on the user.\n\n### Write the SKILL.md\n\nBased on the user interview, fill in these components:\n\n- **name**: Skill identifier\n- **description**: When to trigger, what it does. This is the primary triggering mechanism - include both what the skill does AND specific contexts for when to use it. All \"when to use\" info goes here, not in the body. Note: currently Claude has a tendency to \"undertrigger\" skills -- to not use them when they'd be useful. To combat this, please make the skill descriptions a little bit \"pushy\". So for instance, instead of \"How to build a simple fast dashboard to display internal Anthropic data.\", you might write \"How to build a simple fast dashboard to display internal Anthropic data. Make sure to use this skill whenever the user mentions dashboards, data visualization, internal metrics, or wants to display any kind of company data, even if they don't explicitly ask for a 'dashboard.'\"\n- **compatibility**: Required tools, dependencies (optional, rarely needed)\n- **the rest of the skill :)**\n\n### Skill Writing Guide\n\n#### Anatomy of a Skill\n\n```\nskill-name/\n├── SKILL.md (required)\n│ ├── YAML frontmatter (name, description required)\n│ └── Markdown instructions\n└── Bundled Resources (optional)\n ├── scripts/ - Executable code for deterministic/repetitive tasks\n ├── references/ - Docs loaded into context as needed\n └── assets/ - Files used in output (templates, icons, fonts)\n```\n\n#### Progressive Disclosure\n\nSkills use a three-level loading system:\n1. **Metadata** (name + description) - Always in context (~100 words)\n2. **SKILL.md body** - In context whenever skill triggers (<500 lines ideal)\n3. **Bundled resources** - As needed (unlimited, scripts can execute without loading)\n\nThese word counts are approximate and you can feel free to go longer if needed.\n\n**Key patterns:**\n- Keep SKILL.md under 500 lines; if you're approaching this limit, add an additional layer of hierarchy along with clear pointers about where the model using the skill should go next to follow up.\n- Reference files clearly from SKILL.md with guidance on when to read them\n- For large reference files (>300 lines), include a table of contents\n\n**Domain organization**: When a skill supports multiple domains/frameworks, organize by variant:\n```\ncloud-deploy/\n├── SKILL.md (workflow + selection)\n└── references/\n ├── aws.md\n ├── gcp.md\n └── azure.md\n```\nClaude reads only the relevant reference file.\n\n#### Principle of Lack of Surprise\n\nThis goes without saying, but skills must not contain malware, exploit code, or any content that could compromise system security. A skill's contents should not surprise the user in their intent if described. Don't go along with requests to create misleading skills or skills designed to facilitate unauthorized access, data exfiltration, or other malicious activities. Things like a \"roleplay as an XYZ\" are OK though.\n\n#### Writing Patterns\n\nPrefer using the imperative form in instructions.\n\n**Defining output formats** - You can do it like this:\n```markdown\n## Report structure\nALWAYS use this exact template:\n# [Title]\n## Executive summary\n## Key findings\n## Recommendations\n```\n\n**Examples pattern** - It's useful to include examples. You can format them like this (but if \"Input\" and \"Output\" are in the examples you might want to deviate a little):\n```markdown\n## Commit message format\n**Example 1:**\nInput: Added user authentication with JWT tokens\nOutput: feat(auth): implement JWT-based authentication\n```\n\n### Writing Style\n\nTry to explain to the model why things are important in lieu of heavy-handed musty MUSTs. Use theory of mind and try to make the skill general and not super-narrow to specific examples. Start by writing a draft and then look at it with fresh eyes and improve it.\n\n### Test Cases\n\nAfter writing the skill draft, come up with 2-3 realistic test prompts — the kind of thing a real user would actually say. Share them with the user: [you don't have to use this exact language] \"Here are a few test cases I'd like to try. Do these look right, or do you want to add more?\" Then run them.\n\nSave test cases to `evals/evals.json`. Don't write assertions yet — just the prompts. You'll draft assertions in the next step while the runs are in progress.\n\n```json\n{\n \"skill_name\": \"example-skill\",\n \"evals\": [\n {\n \"id\": 1,\n \"prompt\": \"User's task prompt\",\n \"expected_output\": \"Description of expected result\",\n \"files\": []\n }\n ]\n}\n```\n\nSee `references/schemas.md` for the full schema (including the `assertions` field, which you'll add later).\n\n## Running and evaluating test cases\n\nThis section is one continuous sequence — don't stop partway through. Do NOT use `/skill-test` or any other testing skill.\n\nPut results in `-workspace/` as a sibling to the skill directory. Within the workspace, organize results by iteration (`iteration-1/`, `iteration-2/`, etc.) and within that, each test case gets a directory (`eval-0/`, `eval-1/`, etc.). Don't create all of this upfront — just create directories as you go.\n\n### Step 1: Spawn all runs (with-skill AND baseline) in the same turn\n\nFor each test case, spawn two subagents in the same turn — one with the skill, one without. This is important: don't spawn the with-skill runs first and then come back for baselines later. Launch everything at once so it all finishes around the same time.\n\n**With-skill run:**\n\n```\nExecute this task:\n- Skill path: \n- Task: \n- Input files: \n- Save outputs to: /iteration-/eval-/with_skill/outputs/\n- Outputs to save: \n```\n\n**Baseline run** (same prompt, but the baseline depends on context):\n- **Creating a new skill**: no skill at all. Same prompt, no skill path, save to `without_skill/outputs/`.\n- **Improving an existing skill**: the old version. Before editing, snapshot the skill (`cp -r /skill-snapshot/`), then point the baseline subagent at the snapshot. Save to `old_skill/outputs/`.\n\nWrite an `eval_metadata.json` for each test case (assertions can be empty for now). Give each eval a descriptive name based on what it's testing — not just \"eval-0\". Use this name for the directory too. If this iteration uses new or modified eval prompts, create these files for each new eval directory — don't assume they carry over from previous iterations.\n\n```json\n{\n \"eval_id\": 0,\n \"eval_name\": \"descriptive-name-here\",\n \"prompt\": \"The user's task prompt\",\n \"assertions\": []\n}\n```\n\n### Step 2: While runs are in progress, draft assertions\n\nDon't just wait for the runs to finish — you can use this time productively. Draft quantitative assertions for each test case and explain them to the user. If assertions already exist in `evals/evals.json`, review them and explain what they check.\n\nGood assertions are objectively verifiable and have descriptive names — they should read clearly in the benchmark viewer so someone glancing at the results immediately understands what each one checks. Subjective skills (writing style, design quality) are better evaluated qualitatively — don't force assertions onto things that need human judgment.\n\nUpdate the `eval_metadata.json` files and `evals/evals.json` with the assertions once drafted. Also explain to the user what they'll see in the viewer — both the qualitative outputs and the quantitative benchmark.\n\n### Step 3: As runs complete, capture timing data\n\nWhen each subagent task completes, you receive a notification containing `total_tokens` and `duration_ms`. Save this data immediately to `timing.json` in the run directory:\n\n```json\n{\n \"total_tokens\": 84852,\n \"duration_ms\": 23332,\n \"total_duration_seconds\": 23.3\n}\n```\n\nThis is the only opportunity to capture this data — it comes through the task notification and isn't persisted elsewhere. Process each notification as it arrives rather than trying to batch them.\n\n### Step 4: Grade, aggregate, and launch the viewer\n\nOnce all runs are done:\n\n1. **Grade each run** — spawn a grader subagent (or grade inline) that reads `agents/grader.md` and evaluates each assertion against the outputs. Save results to `grading.json` in each run directory. The grading.json expectations array must use the fields `text`, `passed`, and `evidence` (not `name`/`met`/`details` or other variants) — the viewer depends on these exact field names. For assertions that can be checked programmatically, write and run a script rather than eyeballing it — scripts are faster, more reliable, and can be reused across iterations.\n\n2. **Aggregate into benchmark** — run the aggregation script from the skill-creator directory:\n ```bash\n python -m scripts.aggregate_benchmark /iteration-N --skill-name \n ```\n This produces `benchmark.json` and `benchmark.md` with pass_rate, time, and tokens for each configuration, with mean ± stddev and the delta. If generating benchmark.json manually, see `references/schemas.md` for the exact schema the viewer expects.\nPut each with_skill version before its baseline counterpart.\n\n3. **Do an analyst pass** — read the benchmark data and surface patterns the aggregate stats might hide. See `agents/analyzer.md` (the \"Analyzing Benchmark Results\" section) for what to look for — things like assertions that always pass regardless of skill (non-discriminating), high-variance evals (possibly flaky), and time/token tradeoffs.\n\n4. **Launch the viewer** with both qualitative outputs and quantitative data:\n ```bash\n nohup python /eval-viewer/generate_review.py \\\n /iteration-N \\\n --skill-name \"my-skill\" \\\n --benchmark /iteration-N/benchmark.json \\\n > /dev/null 2>&1 &\n VIEWER_PID=$!\n ```\n For iteration 2+, also pass `--previous-workspace /iteration-`.\n\n **Cowork / headless environments:** If `webbrowser.open()` is not available or the environment has no display, use `--static ` to write a standalone HTML file instead of starting a server. Feedback will be downloaded as a `feedback.json` file when the user clicks \"Submit All Reviews\". After download, copy `feedback.json` into the workspace directory for the next iteration to pick up.\n\nNote: please use generate_review.py to create the viewer; there's no need to write custom HTML.\n\n5. **Tell the user** something like: \"I've opened the results in your browser. There are two tabs — 'Outputs' lets you click through each test case and leave feedback, 'Benchmark' shows the quantitative comparison. When you're done, come back here and let me know.\"\n\n### What the user sees in the viewer\n\nThe \"Outputs\" tab shows one test case at a time:\n- **Prompt**: the task that was given\n- **Output**: the files the skill produced, rendered inline where possible\n- **Previous Output** (iteration 2+): collapsed section showing last iteration's output\n- **Formal Grades** (if grading was run): collapsed section showing assertion pass/fail\n- **Feedback**: a textbox that auto-saves as they type\n- **Previous Feedback** (iteration 2+): their comments from last time, shown below the textbox\n\nThe \"Benchmark\" tab shows the stats summary: pass rates, timing, and token usage for each configuration, with per-eval breakdowns and analyst observations.\n\nNavigation is via prev/next buttons or arrow keys. When done, they click \"Submit All Reviews\" which saves all feedback to `feedback.json`.\n\n### Step 5: Read the feedback\n\nWhen the user tells you they're done, read `feedback.json`:\n\n```json\n{\n \"reviews\": [\n {\"run_id\": \"eval-0-with_skill\", \"feedback\": \"the chart is missing axis labels\", \"timestamp\": \"...\"},\n {\"run_id\": \"eval-1-with_skill\", \"feedback\": \"\", \"timestamp\": \"...\"},\n {\"run_id\": \"eval-2-with_skill\", \"feedback\": \"perfect, love this\", \"timestamp\": \"...\"}\n ],\n \"status\": \"complete\"\n}\n```\n\nEmpty feedback means the user thought it was fine. Focus your improvements on the test cases where the user had specific complaints.\n\nKill the viewer server when you're done with it:\n\n```bash\nkill $VIEWER_PID 2>/dev/null\n```\n\n---\n\n## Improving the skill\n\nThis is the heart of the loop. You've run the test cases, the user has reviewed the results, and now you need to make the skill better based on their feedback.\n\n### How to think about improvements\n\n1. **Generalize from the feedback.** The big picture thing that's happening here is that we're trying to create skills that can be used a million times (maybe literally, maybe even more who knows) across many different prompts. Here you and the user are iterating on only a few examples over and over again because it helps move faster. The user knows these examples in and out and it's quick for them to assess new outputs. But if the skill you and the user are codeveloping works only for those examples, it's useless. Rather than put in fiddly overfitty changes, or oppressively constrictive MUSTs, if there's some stubborn issue, you might try branching out and using different metaphors, or recommending different patterns of working. It's relatively cheap to try and maybe you'll land on something great.\n\n2. **Keep the prompt lean.** Remove things that aren't pulling their weight. Make sure to read the transcripts, not just the final outputs — if it looks like the skill is making the model waste a bunch of time doing things that are unproductive, you can try getting rid of the parts of the skill that are making it do that and seeing what happens.\n\n3. **Explain the why.** Try hard to explain the **why** behind everything you're asking the model to do. Today's LLMs are *smart*. They have good theory of mind and when given a good harness can go beyond rote instructions and really make things happen. Even if the feedback from the user is terse or frustrated, try to actually understand the task and why the user is writing what they wrote, and what they actually wrote, and then transmit this understanding into the instructions. If you find yourself writing ALWAYS or NEVER in all caps, or using super rigid structures, that's a yellow flag — if possible, reframe and explain the reasoning so that the model understands why the thing you're asking for is important. That's a more humane, powerful, and effective approach.\n\n4. **Look for repeated work across test cases.** Read the transcripts from the test runs and notice if the subagents all independently wrote similar helper scripts or took the same multi-step approach to something. If all 3 test cases resulted in the subagent writing a `create_docx.py` or a `build_chart.py`, that's a strong signal the skill should bundle that script. Write it once, put it in `scripts/`, and tell the skill to use it. This saves every future invocation from reinventing the wheel.\n\nThis task is pretty important (we are trying to create billions a year in economic value here!) and your thinking time is not the blocker; take your time and really mull things over. I'd suggest writing a draft revision and then looking at it anew and making improvements. Really do your best to get into the head of the user and understand what they want and need.\n\n### The iteration loop\n\nAfter improving the skill:\n\n1. Apply your improvements to the skill\n2. Rerun all test cases into a new `iteration-/` directory, including baseline runs. If you're creating a new skill, the baseline is always `without_skill` (no skill) — that stays the same across iterations. If you're improving an existing skill, use your judgment on what makes sense as the baseline: the original version the user came in with, or the previous iteration.\n3. Launch the reviewer with `--previous-workspace` pointing at the previous iteration\n4. Wait for the user to review and tell you they're done\n5. Read the new feedback, improve again, repeat\n\nKeep going until:\n- The user says they're happy\n- The feedback is all empty (everything looks good)\n- You're not making meaningful progress\n\n---\n\n## Advanced: Blind comparison\n\nFor situations where you want a more rigorous comparison between two versions of a skill (e.g., the user asks \"is the new version actually better?\"), there's a blind comparison system. Read `agents/comparator.md` and `agents/analyzer.md` for the details. The basic idea is: give two outputs to an independent agent without telling it which is which, and let it judge quality. Then analyze why the winner won.\n\nThis is optional, requires subagents, and most users won't need it. The human review loop is usually sufficient.\n\n---\n\n## Description Optimization\n\nThe description field in SKILL.md frontmatter is the primary mechanism that determines whether Claude invokes a skill. After creating or improving a skill, offer to optimize the description for better triggering accuracy.\n\n### Step 1: Generate trigger eval queries\n\nCreate 20 eval queries — a mix of should-trigger and should-not-trigger. Save as JSON:\n\n```json\n[\n {\"query\": \"the user prompt\", \"should_trigger\": true},\n {\"query\": \"another prompt\", \"should_trigger\": false}\n]\n```\n\nThe queries must be realistic and something a Claude Code or Claude.ai user would actually type. Not abstract requests, but requests that are concrete and specific and have a good amount of detail. For instance, file paths, personal context about the user's job or situation, column names and values, company names, URLs. A little bit of backstory. Some might be in lowercase or contain abbreviations or typos or casual speech. Use a mix of different lengths, and focus on edge cases rather than making them clear-cut (the user will get a chance to sign off on them).\n\nBad: `\"Format this data\"`, `\"Extract text from PDF\"`, `\"Create a chart\"`\n\nGood: `\"ok so my boss just sent me this xlsx file (its in my downloads, called something like 'Q4 sales final FINAL v2.xlsx') and she wants me to add a column that shows the profit margin as a percentage. The revenue is in column C and costs are in column D i think\"`\n\nFor the **should-trigger** queries (8-10), think about coverage. You want different phrasings of the same intent — some formal, some casual. Include cases where the user doesn't explicitly name the skill or file type but clearly needs it. Throw in some uncommon use cases and cases where this skill competes with another but should win.\n\nFor the **should-not-trigger** queries (8-10), the most valuable ones are the near-misses — queries that share keywords or concepts with the skill but actually need something different. Think adjacent domains, ambiguous phrasing where a naive keyword match would trigger but shouldn't, and cases where the query touches on something the skill does but in a context where another tool is more appropriate.\n\nThe key thing to avoid: don't make should-not-trigger queries obviously irrelevant. \"Write a fibonacci function\" as a negative test for a PDF skill is too easy — it doesn't test anything. The negative cases should be genuinely tricky.\n\n### Step 2: Review with user\n\nPresent the eval set to the user for review using the HTML template:\n\n1. Read the template from `assets/eval_review.html`\n2. Replace the placeholders:\n - `__EVAL_DATA_PLACEHOLDER__` → the JSON array of eval items (no quotes around it — it's a JS variable assignment)\n - `__SKILL_NAME_PLACEHOLDER__` → the skill's name\n - `__SKILL_DESCRIPTION_PLACEHOLDER__` → the skill's current description\n3. Write to a temp file (e.g., `/tmp/eval_review_.html`) and open it: `open /tmp/eval_review_.html`\n4. The user can edit queries, toggle should-trigger, add/remove entries, then click \"Export Eval Set\"\n5. The file downloads to `~/Downloads/eval_set.json` — check the Downloads folder for the most recent version in case there are multiple (e.g., `eval_set (1).json`)\n\nThis step matters — bad eval queries lead to bad descriptions.\n\n### Step 3: Run the optimization loop\n\nTell the user: \"This will take some time — I'll run the optimization loop in the background and check on it periodically.\"\n\nSave the eval set to the workspace, then run in the background:\n\n```bash\npython -m scripts.run_loop \\\n --eval-set \\\n --skill-path \\\n --model \\\n --max-iterations 5 \\\n --verbose\n```\n\nUse the model ID from your system prompt (the one powering the current session) so the triggering test matches what the user actually experiences.\n\nWhile it runs, periodically tail the output to give the user updates on which iteration it's on and what the scores look like.\n\nThis handles the full optimization loop automatically. It splits the eval set into 60% train and 40% held-out test, evaluates the current description (running each query 3 times to get a reliable trigger rate), then calls Claude to propose improvements based on what failed. It re-evaluates each new description on both train and test, iterating up to 5 times. When it's done, it opens an HTML report in the browser showing the results per iteration and returns JSON with `best_description` — selected by test score rather than train score to avoid overfitting.\n\n### How skill triggering works\n\nUnderstanding the triggering mechanism helps design better eval queries. Skills appear in Claude's `available_skills` list with their name + description, and Claude decides whether to consult a skill based on that description. The important thing to know is that Claude only consults skills for tasks it can't easily handle on its own — simple, one-step queries like \"read this PDF\" may not trigger a skill even if the description matches perfectly, because Claude can handle them directly with basic tools. Complex, multi-step, or specialized queries reliably trigger skills when the description matches.\n\nThis means your eval queries should be substantive enough that Claude would actually benefit from consulting a skill. Simple queries like \"read file X\" are poor test cases — they won't trigger skills regardless of description quality.\n\n### Step 4: Apply the result\n\nTake `best_description` from the JSON output and update the skill's SKILL.md frontmatter. Show the user before/after and report the scores.\n\n---\n\n### Package and Present (only if `present_files` tool is available)\n\nCheck whether you have access to the `present_files` tool. If you don't, skip this step. If you do, package the skill and present the .skill file to the user:\n\n```bash\npython -m scripts.package_skill \n```\n\nAfter packaging, direct the user to the resulting `.skill` file path so they can install it.\n\n---\n\n## Claude.ai-specific instructions\n\nIn Claude.ai, the core workflow is the same (draft → test → review → improve → repeat), but because Claude.ai doesn't have subagents, some mechanics change. Here's what to adapt:\n\n**Running test cases**: No subagents means no parallel execution. For each test case, read the skill's SKILL.md, then follow its instructions to accomplish the test prompt yourself. Do them one at a time. This is less rigorous than independent subagents (you wrote the skill and you're also running it, so you have full context), but it's a useful sanity check — and the human review step compensates. Skip the baseline runs — just use the skill to complete the task as requested.\n\n**Reviewing results**: If you can't open a browser (e.g., Claude.ai's VM has no display, or you're on a remote server), skip the browser reviewer entirely. Instead, present results directly in the conversation. For each test case, show the prompt and the output. If the output is a file the user needs to see (like a .docx or .xlsx), save it to the filesystem and tell them where it is so they can download and inspect it. Ask for feedback inline: \"How does this look? Anything you'd change?\"\n\n**Benchmarking**: Skip the quantitative benchmarking — it relies on baseline comparisons which aren't meaningful without subagents. Focus on qualitative feedback from the user.\n\n**The iteration loop**: Same as before — improve the skill, rerun the test cases, ask for feedback — just without the browser reviewer in the middle. You can still organize results into iteration directories on the filesystem if you have one.\n\n**Description optimization**: This section requires the `claude` CLI tool (specifically `claude -p`) which is only available in Claude Code. Skip it if you're on Claude.ai.\n\n**Blind comparison**: Requires subagents. Skip it.\n\n**Packaging**: The `package_skill.py` script works anywhere with Python and a filesystem. On Claude.ai, you can run it and the user can download the resulting `.skill` file.\n\n**Updating an existing skill**: The user might be asking you to update an existing skill, not create a new one. In this case:\n- **Preserve the original name.** Note the skill's directory name and `name` frontmatter field -- use them unchanged. E.g., if the installed skill is `research-helper`, output `research-helper.skill` (not `research-helper-v2`).\n- **Copy to a writeable location before editing.** The installed skill path may be read-only. Copy to `/tmp/skill-name/`, edit there, and package from the copy.\n- **If packaging manually, stage in `/tmp/` first**, then copy to the output directory -- direct writes may fail due to permissions.\n\n---\n\n## Cowork-Specific Instructions\n\nIf you're in Cowork, the main things to know are:\n\n- You have subagents, so the main workflow (spawn test cases in parallel, run baselines, grade, etc.) all works. (However, if you run into severe problems with timeouts, it's OK to run the test prompts in series rather than parallel.)\n- You don't have a browser or display, so when generating the eval viewer, use `--static ` to write a standalone HTML file instead of starting a server. Then proffer a link that the user can click to open the HTML in their browser.\n- For whatever reason, the Cowork setup seems to disincline Claude from generating the eval viewer after running the tests, so just to reiterate: whether you're in Cowork or in Claude Code, after running tests, you should always generate the eval viewer for the human to look at examples before revising the skill yourself and trying to make corrections, using `generate_review.py` (not writing your own boutique html code). Sorry in advance but I'm gonna go all caps here: GENERATE THE EVAL VIEWER *BEFORE* evaluating inputs yourself. You want to get them in front of the human ASAP!\n- Feedback works differently: since there's no running server, the viewer's \"Submit All Reviews\" button will download `feedback.json` as a file. You can then read it from there (you may have to request access first).\n- Packaging works — `package_skill.py` just needs Python and a filesystem.\n- Description optimization (`run_loop.py` / `run_eval.py`) should work in Cowork just fine since it uses `claude -p` via subprocess, not a browser, but please save it until you've fully finished making the skill and the user agrees it's in good shape.\n- **Updating an existing skill**: The user might be asking you to update an existing skill, not create a new one. Follow the update guidance in the claude.ai section above.\n\n---\n\n## Reference files\n\nThe agents/ directory contains instructions for specialized subagents. Read them when you need to spawn the relevant subagent.\n\n- `agents/grader.md` — How to evaluate assertions against outputs\n- `agents/comparator.md` — How to do blind A/B comparison between two outputs\n- `agents/analyzer.md` — How to analyze why one version beat another\n\nThe references/ directory has additional documentation:\n- `references/schemas.md` — JSON structures for evals.json, grading.json, etc.\n\n---\n\nRepeating one more time the core loop here for emphasis:\n\n- Figure out what the skill is about\n- Draft or edit the skill\n- Run claude-with-access-to-the-skill on test prompts\n- With the user, evaluate the outputs:\n - Create benchmark.json and run `eval-viewer/generate_review.py` to help the user review them\n - Run quantitative evals\n- Repeat until you and the user are satisfied\n- Package the final skill and return it to the user.\n\nPlease add steps to your TodoList, if you have such a thing, to make sure you don't forget. If you're in Cowork, please specifically put \"Create evals JSON and run `eval-viewer/generate_review.py` so human can review test cases\" in your TodoList to make sure it happens.\n\nGood luck!\n","flock-data/skills/skill-creator/agents/analyzer.md":"# Post-hoc Analyzer Agent\n\nAnalyze blind comparison results to understand WHY the winner won and generate improvement suggestions.\n\n## Role\n\nAfter the blind comparator determines a winner, the Post-hoc Analyzer \"unblids\" the results by examining the skills and transcripts. The goal is to extract actionable insights: what made the winner better, and how can the loser be improved?\n\n## Inputs\n\nYou receive these parameters in your prompt:\n\n- **winner**: \"A\" or \"B\" (from blind comparison)\n- **winner_skill_path**: Path to the skill that produced the winning output\n- **winner_transcript_path**: Path to the execution transcript for the winner\n- **loser_skill_path**: Path to the skill that produced the losing output\n- **loser_transcript_path**: Path to the execution transcript for the loser\n- **comparison_result_path**: Path to the blind comparator's output JSON\n- **output_path**: Where to save the analysis results\n\n## Process\n\n### Step 1: Read Comparison Result\n\n1. Read the blind comparator's output at comparison_result_path\n2. Note the winning side (A or B), the reasoning, and any scores\n3. Understand what the comparator valued in the winning output\n\n### Step 2: Read Both Skills\n\n1. Read the winner skill's SKILL.md and key referenced files\n2. Read the loser skill's SKILL.md and key referenced files\n3. Identify structural differences:\n - Instructions clarity and specificity\n - Script/tool usage patterns\n - Example coverage\n - Edge case handling\n\n### Step 3: Read Both Transcripts\n\n1. Read the winner's transcript\n2. Read the loser's transcript\n3. Compare execution patterns:\n - How closely did each follow their skill's instructions?\n - What tools were used differently?\n - Where did the loser diverge from optimal behavior?\n - Did either encounter errors or make recovery attempts?\n\n### Step 4: Analyze Instruction Following\n\nFor each transcript, evaluate:\n- Did the agent follow the skill's explicit instructions?\n- Did the agent use the skill's provided tools/scripts?\n- Were there missed opportunities to leverage skill content?\n- Did the agent add unnecessary steps not in the skill?\n\nScore instruction following 1-10 and note specific issues.\n\n### Step 5: Identify Winner Strengths\n\nDetermine what made the winner better:\n- Clearer instructions that led to better behavior?\n- Better scripts/tools that produced better output?\n- More comprehensive examples that guided edge cases?\n- Better error handling guidance?\n\nBe specific. Quote from skills/transcripts where relevant.\n\n### Step 6: Identify Loser Weaknesses\n\nDetermine what held the loser back:\n- Ambiguous instructions that led to suboptimal choices?\n- Missing tools/scripts that forced workarounds?\n- Gaps in edge case coverage?\n- Poor error handling that caused failures?\n\n### Step 7: Generate Improvement Suggestions\n\nBased on the analysis, produce actionable suggestions for improving the loser skill:\n- Specific instruction changes to make\n- Tools/scripts to add or modify\n- Examples to include\n- Edge cases to address\n\nPrioritize by impact. Focus on changes that would have changed the outcome.\n\n### Step 8: Write Analysis Results\n\nSave structured analysis to `{output_path}`.\n\n## Output Format\n\nWrite a JSON file with this structure:\n\n```json\n{\n \"comparison_summary\": {\n \"winner\": \"A\",\n \"winner_skill\": \"path/to/winner/skill\",\n \"loser_skill\": \"path/to/loser/skill\",\n \"comparator_reasoning\": \"Brief summary of why comparator chose winner\"\n },\n \"winner_strengths\": [\n \"Clear step-by-step instructions for handling multi-page documents\",\n \"Included validation script that caught formatting errors\",\n \"Explicit guidance on fallback behavior when OCR fails\"\n ],\n \"loser_weaknesses\": [\n \"Vague instruction 'process the document appropriately' led to inconsistent behavior\",\n \"No script for validation, agent had to improvise and made errors\",\n \"No guidance on OCR failure, agent gave up instead of trying alternatives\"\n ],\n \"instruction_following\": {\n \"winner\": {\n \"score\": 9,\n \"issues\": [\n \"Minor: skipped optional logging step\"\n ]\n },\n \"loser\": {\n \"score\": 6,\n \"issues\": [\n \"Did not use the skill's formatting template\",\n \"Invented own approach instead of following step 3\",\n \"Missed the 'always validate output' instruction\"\n ]\n }\n },\n \"improvement_suggestions\": [\n {\n \"priority\": \"high\",\n \"category\": \"instructions\",\n \"suggestion\": \"Replace 'process the document appropriately' with explicit steps: 1) Extract text, 2) Identify sections, 3) Format per template\",\n \"expected_impact\": \"Would eliminate ambiguity that caused inconsistent behavior\"\n },\n {\n \"priority\": \"high\",\n \"category\": \"tools\",\n \"suggestion\": \"Add validate_output.py script similar to winner skill's validation approach\",\n \"expected_impact\": \"Would catch formatting errors before final output\"\n },\n {\n \"priority\": \"medium\",\n \"category\": \"error_handling\",\n \"suggestion\": \"Add fallback instructions: 'If OCR fails, try: 1) different resolution, 2) image preprocessing, 3) manual extraction'\",\n \"expected_impact\": \"Would prevent early failure on difficult documents\"\n }\n ],\n \"transcript_insights\": {\n \"winner_execution_pattern\": \"Read skill -> Followed 5-step process -> Used validation script -> Fixed 2 issues -> Produced output\",\n \"loser_execution_pattern\": \"Read skill -> Unclear on approach -> Tried 3 different methods -> No validation -> Output had errors\"\n }\n}\n```\n\n## Guidelines\n\n- **Be specific**: Quote from skills and transcripts, don't just say \"instructions were unclear\"\n- **Be actionable**: Suggestions should be concrete changes, not vague advice\n- **Focus on skill improvements**: The goal is to improve the losing skill, not critique the agent\n- **Prioritize by impact**: Which changes would most likely have changed the outcome?\n- **Consider causation**: Did the skill weakness actually cause the worse output, or is it incidental?\n- **Stay objective**: Analyze what happened, don't editorialize\n- **Think about generalization**: Would this improvement help on other evals too?\n\n## Categories for Suggestions\n\nUse these categories to organize improvement suggestions:\n\n| Category | Description |\n|----------|-------------|\n| `instructions` | Changes to the skill's prose instructions |\n| `tools` | Scripts, templates, or utilities to add/modify |\n| `examples` | Example inputs/outputs to include |\n| `error_handling` | Guidance for handling failures |\n| `structure` | Reorganization of skill content |\n| `references` | External docs or resources to add |\n\n## Priority Levels\n\n- **high**: Would likely change the outcome of this comparison\n- **medium**: Would improve quality but may not change win/loss\n- **low**: Nice to have, marginal improvement\n\n---\n\n# Analyzing Benchmark Results\n\nWhen analyzing benchmark results, the analyzer's purpose is to **surface patterns and anomalies** across multiple runs, not suggest skill improvements.\n\n## Role\n\nReview all benchmark run results and generate freeform notes that help the user understand skill performance. Focus on patterns that wouldn't be visible from aggregate metrics alone.\n\n## Inputs\n\nYou receive these parameters in your prompt:\n\n- **benchmark_data_path**: Path to the in-progress benchmark.json with all run results\n- **skill_path**: Path to the skill being benchmarked\n- **output_path**: Where to save the notes (as JSON array of strings)\n\n## Process\n\n### Step 1: Read Benchmark Data\n\n1. Read the benchmark.json containing all run results\n2. Note the configurations tested (with_skill, without_skill)\n3. Understand the run_summary aggregates already calculated\n\n### Step 2: Analyze Per-Assertion Patterns\n\nFor each expectation across all runs:\n- Does it **always pass** in both configurations? (may not differentiate skill value)\n- Does it **always fail** in both configurations? (may be broken or beyond capability)\n- Does it **always pass with skill but fail without**? (skill clearly adds value here)\n- Does it **always fail with skill but pass without**? (skill may be hurting)\n- Is it **highly variable**? (flaky expectation or non-deterministic behavior)\n\n### Step 3: Analyze Cross-Eval Patterns\n\nLook for patterns across evals:\n- Are certain eval types consistently harder/easier?\n- Do some evals show high variance while others are stable?\n- Are there surprising results that contradict expectations?\n\n### Step 4: Analyze Metrics Patterns\n\nLook at time_seconds, tokens, tool_calls:\n- Does the skill significantly increase execution time?\n- Is there high variance in resource usage?\n- Are there outlier runs that skew the aggregates?\n\n### Step 5: Generate Notes\n\nWrite freeform observations as a list of strings. Each note should:\n- State a specific observation\n- Be grounded in the data (not speculation)\n- Help the user understand something the aggregate metrics don't show\n\nExamples:\n- \"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value\"\n- \"Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure that may be flaky\"\n- \"Without-skill runs consistently fail on table extraction expectations (0% pass rate)\"\n- \"Skill adds 13s average execution time but improves pass rate by 50%\"\n- \"Token usage is 80% higher with skill, primarily due to script output parsing\"\n- \"All 3 without-skill runs for eval 1 produced empty output\"\n\n### Step 6: Write Notes\n\nSave notes to `{output_path}` as a JSON array of strings:\n\n```json\n[\n \"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value\",\n \"Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure\",\n \"Without-skill runs consistently fail on table extraction expectations\",\n \"Skill adds 13s average execution time but improves pass rate by 50%\"\n]\n```\n\n## Guidelines\n\n**DO:**\n- Report what you observe in the data\n- Be specific about which evals, expectations, or runs you're referring to\n- Note patterns that aggregate metrics would hide\n- Provide context that helps interpret the numbers\n\n**DO NOT:**\n- Suggest improvements to the skill (that's for the improvement step, not benchmarking)\n- Make subjective quality judgments (\"the output was good/bad\")\n- Speculate about causes without evidence\n- Repeat information already in the run_summary aggregates\n","flock-data/skills/skill-creator/agents/comparator.md":"# Blind Comparator Agent\n\nCompare two outputs WITHOUT knowing which skill produced them.\n\n## Role\n\nThe Blind Comparator judges which output better accomplishes the eval task. You receive two outputs labeled A and B, but you do NOT know which skill produced which. This prevents bias toward a particular skill or approach.\n\nYour judgment is based purely on output quality and task completion.\n\n## Inputs\n\nYou receive these parameters in your prompt:\n\n- **output_a_path**: Path to the first output file or directory\n- **output_b_path**: Path to the second output file or directory\n- **eval_prompt**: The original task/prompt that was executed\n- **expectations**: List of expectations to check (optional - may be empty)\n\n## Process\n\n### Step 1: Read Both Outputs\n\n1. Examine output A (file or directory)\n2. Examine output B (file or directory)\n3. Note the type, structure, and content of each\n4. If outputs are directories, examine all relevant files inside\n\n### Step 2: Understand the Task\n\n1. Read the eval_prompt carefully\n2. Identify what the task requires:\n - What should be produced?\n - What qualities matter (accuracy, completeness, format)?\n - What would distinguish a good output from a poor one?\n\n### Step 3: Generate Evaluation Rubric\n\nBased on the task, generate a rubric with two dimensions:\n\n**Content Rubric** (what the output contains):\n| Criterion | 1 (Poor) | 3 (Acceptable) | 5 (Excellent) |\n|-----------|----------|----------------|---------------|\n| Correctness | Major errors | Minor errors | Fully correct |\n| Completeness | Missing key elements | Mostly complete | All elements present |\n| Accuracy | Significant inaccuracies | Minor inaccuracies | Accurate throughout |\n\n**Structure Rubric** (how the output is organized):\n| Criterion | 1 (Poor) | 3 (Acceptable) | 5 (Excellent) |\n|-----------|----------|----------------|---------------|\n| Organization | Disorganized | Reasonably organized | Clear, logical structure |\n| Formatting | Inconsistent/broken | Mostly consistent | Professional, polished |\n| Usability | Difficult to use | Usable with effort | Easy to use |\n\nAdapt criteria to the specific task. For example:\n- PDF form → \"Field alignment\", \"Text readability\", \"Data placement\"\n- Document → \"Section structure\", \"Heading hierarchy\", \"Paragraph flow\"\n- Data output → \"Schema correctness\", \"Data types\", \"Completeness\"\n\n### Step 4: Evaluate Each Output Against the Rubric\n\nFor each output (A and B):\n\n1. **Score each criterion** on the rubric (1-5 scale)\n2. **Calculate dimension totals**: Content score, Structure score\n3. **Calculate overall score**: Average of dimension scores, scaled to 1-10\n\n### Step 5: Check Assertions (if provided)\n\nIf expectations are provided:\n\n1. Check each expectation against output A\n2. Check each expectation against output B\n3. Count pass rates for each output\n4. Use expectation scores as secondary evidence (not the primary decision factor)\n\n### Step 6: Determine the Winner\n\nCompare A and B based on (in priority order):\n\n1. **Primary**: Overall rubric score (content + structure)\n2. **Secondary**: Assertion pass rates (if applicable)\n3. **Tiebreaker**: If truly equal, declare a TIE\n\nBe decisive - ties should be rare. One output is usually better, even if marginally.\n\n### Step 7: Write Comparison Results\n\nSave results to a JSON file at the path specified (or `comparison.json` if not specified).\n\n## Output Format\n\nWrite a JSON file with this structure:\n\n```json\n{\n \"winner\": \"A\",\n \"reasoning\": \"Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.\",\n \"rubric\": {\n \"A\": {\n \"content\": {\n \"correctness\": 5,\n \"completeness\": 5,\n \"accuracy\": 4\n },\n \"structure\": {\n \"organization\": 4,\n \"formatting\": 5,\n \"usability\": 4\n },\n \"content_score\": 4.7,\n \"structure_score\": 4.3,\n \"overall_score\": 9.0\n },\n \"B\": {\n \"content\": {\n \"correctness\": 3,\n \"completeness\": 2,\n \"accuracy\": 3\n },\n \"structure\": {\n \"organization\": 3,\n \"formatting\": 2,\n \"usability\": 3\n },\n \"content_score\": 2.7,\n \"structure_score\": 2.7,\n \"overall_score\": 5.4\n }\n },\n \"output_quality\": {\n \"A\": {\n \"score\": 9,\n \"strengths\": [\"Complete solution\", \"Well-formatted\", \"All fields present\"],\n \"weaknesses\": [\"Minor style inconsistency in header\"]\n },\n \"B\": {\n \"score\": 5,\n \"strengths\": [\"Readable output\", \"Correct basic structure\"],\n \"weaknesses\": [\"Missing date field\", \"Formatting inconsistencies\", \"Partial data extraction\"]\n }\n },\n \"expectation_results\": {\n \"A\": {\n \"passed\": 4,\n \"total\": 5,\n \"pass_rate\": 0.80,\n \"details\": [\n {\"text\": \"Output includes name\", \"passed\": true},\n {\"text\": \"Output includes date\", \"passed\": true},\n {\"text\": \"Format is PDF\", \"passed\": true},\n {\"text\": \"Contains signature\", \"passed\": false},\n {\"text\": \"Readable text\", \"passed\": true}\n ]\n },\n \"B\": {\n \"passed\": 3,\n \"total\": 5,\n \"pass_rate\": 0.60,\n \"details\": [\n {\"text\": \"Output includes name\", \"passed\": true},\n {\"text\": \"Output includes date\", \"passed\": false},\n {\"text\": \"Format is PDF\", \"passed\": true},\n {\"text\": \"Contains signature\", \"passed\": false},\n {\"text\": \"Readable text\", \"passed\": true}\n ]\n }\n }\n}\n```\n\nIf no expectations were provided, omit the `expectation_results` field entirely.\n\n## Field Descriptions\n\n- **winner**: \"A\", \"B\", or \"TIE\"\n- **reasoning**: Clear explanation of why the winner was chosen (or why it's a tie)\n- **rubric**: Structured rubric evaluation for each output\n - **content**: Scores for content criteria (correctness, completeness, accuracy)\n - **structure**: Scores for structure criteria (organization, formatting, usability)\n - **content_score**: Average of content criteria (1-5)\n - **structure_score**: Average of structure criteria (1-5)\n - **overall_score**: Combined score scaled to 1-10\n- **output_quality**: Summary quality assessment\n - **score**: 1-10 rating (should match rubric overall_score)\n - **strengths**: List of positive aspects\n - **weaknesses**: List of issues or shortcomings\n- **expectation_results**: (Only if expectations provided)\n - **passed**: Number of expectations that passed\n - **total**: Total number of expectations\n - **pass_rate**: Fraction passed (0.0 to 1.0)\n - **details**: Individual expectation results\n\n## Guidelines\n\n- **Stay blind**: DO NOT try to infer which skill produced which output. Judge purely on output quality.\n- **Be specific**: Cite specific examples when explaining strengths and weaknesses.\n- **Be decisive**: Choose a winner unless outputs are genuinely equivalent.\n- **Output quality first**: Assertion scores are secondary to overall task completion.\n- **Be objective**: Don't favor outputs based on style preferences; focus on correctness and completeness.\n- **Explain your reasoning**: The reasoning field should make it clear why you chose the winner.\n- **Handle edge cases**: If both outputs fail, pick the one that fails less badly. If both are excellent, pick the one that's marginally better.\n","flock-data/skills/skill-creator/agents/grader.md":"# Grader Agent\n\nEvaluate expectations against an execution transcript and outputs.\n\n## Role\n\nThe Grader reviews a transcript and output files, then determines whether each expectation passes or fails. Provide clear evidence for each judgment.\n\nYou have two jobs: grade the outputs, and critique the evals themselves. A passing grade on a weak assertion is worse than useless — it creates false confidence. When you notice an assertion that's trivially satisfied, or an important outcome that no assertion checks, say so.\n\n## Inputs\n\nYou receive these parameters in your prompt:\n\n- **expectations**: List of expectations to evaluate (strings)\n- **transcript_path**: Path to the execution transcript (markdown file)\n- **outputs_dir**: Directory containing output files from execution\n\n## Process\n\n### Step 1: Read the Transcript\n\n1. Read the transcript file completely\n2. Note the eval prompt, execution steps, and final result\n3. Identify any issues or errors documented\n\n### Step 2: Examine Output Files\n\n1. List files in outputs_dir\n2. Read/examine each file relevant to the expectations. If outputs aren't plain text, use the inspection tools provided in your prompt — don't rely solely on what the transcript says the executor produced.\n3. Note contents, structure, and quality\n\n### Step 3: Evaluate Each Assertion\n\nFor each expectation:\n\n1. **Search for evidence** in the transcript and outputs\n2. **Determine verdict**:\n - **PASS**: Clear evidence the expectation is true AND the evidence reflects genuine task completion, not just surface-level compliance\n - **FAIL**: No evidence, or evidence contradicts the expectation, or the evidence is superficial (e.g., correct filename but empty/wrong content)\n3. **Cite the evidence**: Quote the specific text or describe what you found\n\n### Step 4: Extract and Verify Claims\n\nBeyond the predefined expectations, extract implicit claims from the outputs and verify them:\n\n1. **Extract claims** from the transcript and outputs:\n - Factual statements (\"The form has 12 fields\")\n - Process claims (\"Used pypdf to fill the form\")\n - Quality claims (\"All fields were filled correctly\")\n\n2. **Verify each claim**:\n - **Factual claims**: Can be checked against the outputs or external sources\n - **Process claims**: Can be verified from the transcript\n - **Quality claims**: Evaluate whether the claim is justified\n\n3. **Flag unverifiable claims**: Note claims that cannot be verified with available information\n\nThis catches issues that predefined expectations might miss.\n\n### Step 5: Read User Notes\n\nIf `{outputs_dir}/user_notes.md` exists:\n1. Read it and note any uncertainties or issues flagged by the executor\n2. Include relevant concerns in the grading output\n3. These may reveal problems even when expectations pass\n\n### Step 6: Critique the Evals\n\nAfter grading, consider whether the evals themselves could be improved. Only surface suggestions when there's a clear gap.\n\nGood suggestions test meaningful outcomes — assertions that are hard to satisfy without actually doing the work correctly. Think about what makes an assertion *discriminating*: it passes when the skill genuinely succeeds and fails when it doesn't.\n\nSuggestions worth raising:\n- An assertion that passed but would also pass for a clearly wrong output (e.g., checking filename existence but not file content)\n- An important outcome you observed — good or bad — that no assertion covers at all\n- An assertion that can't actually be verified from the available outputs\n\nKeep the bar high. The goal is to flag things the eval author would say \"good catch\" about, not to nitpick every assertion.\n\n### Step 7: Write Grading Results\n\nSave results to `{outputs_dir}/../grading.json` (sibling to outputs_dir).\n\n## Grading Criteria\n\n**PASS when**:\n- The transcript or outputs clearly demonstrate the expectation is true\n- Specific evidence can be cited\n- The evidence reflects genuine substance, not just surface compliance (e.g., a file exists AND contains correct content, not just the right filename)\n\n**FAIL when**:\n- No evidence found for the expectation\n- Evidence contradicts the expectation\n- The expectation cannot be verified from available information\n- The evidence is superficial — the assertion is technically satisfied but the underlying task outcome is wrong or incomplete\n- The output appears to meet the assertion by coincidence rather than by actually doing the work\n\n**When uncertain**: The burden of proof to pass is on the expectation.\n\n### Step 8: Read Executor Metrics and Timing\n\n1. If `{outputs_dir}/metrics.json` exists, read it and include in grading output\n2. If `{outputs_dir}/../timing.json` exists, read it and include timing data\n\n## Output Format\n\nWrite a JSON file with this structure:\n\n```json\n{\n \"expectations\": [\n {\n \"text\": \"The output includes the name 'John Smith'\",\n \"passed\": true,\n \"evidence\": \"Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'\"\n },\n {\n \"text\": \"The spreadsheet has a SUM formula in cell B10\",\n \"passed\": false,\n \"evidence\": \"No spreadsheet was created. The output was a text file.\"\n },\n {\n \"text\": \"The assistant used the skill's OCR script\",\n \"passed\": true,\n \"evidence\": \"Transcript Step 2 shows: 'Tool: Bash - python ocr_script.py image.png'\"\n }\n ],\n \"summary\": {\n \"passed\": 2,\n \"failed\": 1,\n \"total\": 3,\n \"pass_rate\": 0.67\n },\n \"execution_metrics\": {\n \"tool_calls\": {\n \"Read\": 5,\n \"Write\": 2,\n \"Bash\": 8\n },\n \"total_tool_calls\": 15,\n \"total_steps\": 6,\n \"errors_encountered\": 0,\n \"output_chars\": 12450,\n \"transcript_chars\": 3200\n },\n \"timing\": {\n \"executor_duration_seconds\": 165.0,\n \"grader_duration_seconds\": 26.0,\n \"total_duration_seconds\": 191.0\n },\n \"claims\": [\n {\n \"claim\": \"The form has 12 fillable fields\",\n \"type\": \"factual\",\n \"verified\": true,\n \"evidence\": \"Counted 12 fields in field_info.json\"\n },\n {\n \"claim\": \"All required fields were populated\",\n \"type\": \"quality\",\n \"verified\": false,\n \"evidence\": \"Reference section was left blank despite data being available\"\n }\n ],\n \"user_notes_summary\": {\n \"uncertainties\": [\"Used 2023 data, may be stale\"],\n \"needs_review\": [],\n \"workarounds\": [\"Fell back to text overlay for non-fillable fields\"]\n },\n \"eval_feedback\": {\n \"suggestions\": [\n {\n \"assertion\": \"The output includes the name 'John Smith'\",\n \"reason\": \"A hallucinated document that mentions the name would also pass — consider checking it appears as the primary contact with matching phone and email from the input\"\n },\n {\n \"reason\": \"No assertion checks whether the extracted phone numbers match the input — I observed incorrect numbers in the output that went uncaught\"\n }\n ],\n \"overall\": \"Assertions check presence but not correctness. Consider adding content verification.\"\n }\n}\n```\n\n## Field Descriptions\n\n- **expectations**: Array of graded expectations\n - **text**: The original expectation text\n - **passed**: Boolean - true if expectation passes\n - **evidence**: Specific quote or description supporting the verdict\n- **summary**: Aggregate statistics\n - **passed**: Count of passed expectations\n - **failed**: Count of failed expectations\n - **total**: Total expectations evaluated\n - **pass_rate**: Fraction passed (0.0 to 1.0)\n- **execution_metrics**: Copied from executor's metrics.json (if available)\n - **output_chars**: Total character count of output files (proxy for tokens)\n - **transcript_chars**: Character count of transcript\n- **timing**: Wall clock timing from timing.json (if available)\n - **executor_duration_seconds**: Time spent in executor subagent\n - **total_duration_seconds**: Total elapsed time for the run\n- **claims**: Extracted and verified claims from the output\n - **claim**: The statement being verified\n - **type**: \"factual\", \"process\", or \"quality\"\n - **verified**: Boolean - whether the claim holds\n - **evidence**: Supporting or contradicting evidence\n- **user_notes_summary**: Issues flagged by the executor\n - **uncertainties**: Things the executor wasn't sure about\n - **needs_review**: Items requiring human attention\n - **workarounds**: Places where the skill didn't work as expected\n- **eval_feedback**: Improvement suggestions for the evals (only when warranted)\n - **suggestions**: List of concrete suggestions, each with a `reason` and optionally an `assertion` it relates to\n - **overall**: Brief assessment — can be \"No suggestions, evals look solid\" if nothing to flag\n\n## Guidelines\n\n- **Be objective**: Base verdicts on evidence, not assumptions\n- **Be specific**: Quote the exact text that supports your verdict\n- **Be thorough**: Check both transcript and output files\n- **Be consistent**: Apply the same standard to each expectation\n- **Explain failures**: Make it clear why evidence was insufficient\n- **No partial credit**: Each expectation is pass or fail, not partial\n","flock-data/skills/skill-creator/references/schemas.md":"# JSON Schemas\n\nThis document defines the JSON schemas used by skill-creator.\n\n---\n\n## evals.json\n\nDefines the evals for a skill. Located at `evals/evals.json` within the skill directory.\n\n```json\n{\n \"skill_name\": \"example-skill\",\n \"evals\": [\n {\n \"id\": 1,\n \"prompt\": \"User's example prompt\",\n \"expected_output\": \"Description of expected result\",\n \"files\": [\"evals/files/sample1.pdf\"],\n \"expectations\": [\n \"The output includes X\",\n \"The skill used script Y\"\n ]\n }\n ]\n}\n```\n\n**Fields:**\n- `skill_name`: Name matching the skill's frontmatter\n- `evals[].id`: Unique integer identifier\n- `evals[].prompt`: The task to execute\n- `evals[].expected_output`: Human-readable description of success\n- `evals[].files`: Optional list of input file paths (relative to skill root)\n- `evals[].expectations`: List of verifiable statements\n\n---\n\n## history.json\n\nTracks version progression in Improve mode. Located at workspace root.\n\n```json\n{\n \"started_at\": \"2026-01-15T10:30:00Z\",\n \"skill_name\": \"pdf\",\n \"current_best\": \"v2\",\n \"iterations\": [\n {\n \"version\": \"v0\",\n \"parent\": null,\n \"expectation_pass_rate\": 0.65,\n \"grading_result\": \"baseline\",\n \"is_current_best\": false\n },\n {\n \"version\": \"v1\",\n \"parent\": \"v0\",\n \"expectation_pass_rate\": 0.75,\n \"grading_result\": \"won\",\n \"is_current_best\": false\n },\n {\n \"version\": \"v2\",\n \"parent\": \"v1\",\n \"expectation_pass_rate\": 0.85,\n \"grading_result\": \"won\",\n \"is_current_best\": true\n }\n ]\n}\n```\n\n**Fields:**\n- `started_at`: ISO timestamp of when improvement started\n- `skill_name`: Name of the skill being improved\n- `current_best`: Version identifier of the best performer\n- `iterations[].version`: Version identifier (v0, v1, ...)\n- `iterations[].parent`: Parent version this was derived from\n- `iterations[].expectation_pass_rate`: Pass rate from grading\n- `iterations[].grading_result`: \"baseline\", \"won\", \"lost\", or \"tie\"\n- `iterations[].is_current_best`: Whether this is the current best version\n\n---\n\n## grading.json\n\nOutput from the grader agent. Located at `/grading.json`.\n\n```json\n{\n \"expectations\": [\n {\n \"text\": \"The output includes the name 'John Smith'\",\n \"passed\": true,\n \"evidence\": \"Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'\"\n },\n {\n \"text\": \"The spreadsheet has a SUM formula in cell B10\",\n \"passed\": false,\n \"evidence\": \"No spreadsheet was created. The output was a text file.\"\n }\n ],\n \"summary\": {\n \"passed\": 2,\n \"failed\": 1,\n \"total\": 3,\n \"pass_rate\": 0.67\n },\n \"execution_metrics\": {\n \"tool_calls\": {\n \"Read\": 5,\n \"Write\": 2,\n \"Bash\": 8\n },\n \"total_tool_calls\": 15,\n \"total_steps\": 6,\n \"errors_encountered\": 0,\n \"output_chars\": 12450,\n \"transcript_chars\": 3200\n },\n \"timing\": {\n \"executor_duration_seconds\": 165.0,\n \"grader_duration_seconds\": 26.0,\n \"total_duration_seconds\": 191.0\n },\n \"claims\": [\n {\n \"claim\": \"The form has 12 fillable fields\",\n \"type\": \"factual\",\n \"verified\": true,\n \"evidence\": \"Counted 12 fields in field_info.json\"\n }\n ],\n \"user_notes_summary\": {\n \"uncertainties\": [\"Used 2023 data, may be stale\"],\n \"needs_review\": [],\n \"workarounds\": [\"Fell back to text overlay for non-fillable fields\"]\n },\n \"eval_feedback\": {\n \"suggestions\": [\n {\n \"assertion\": \"The output includes the name 'John Smith'\",\n \"reason\": \"A hallucinated document that mentions the name would also pass\"\n }\n ],\n \"overall\": \"Assertions check presence but not correctness.\"\n }\n}\n```\n\n**Fields:**\n- `expectations[]`: Graded expectations with evidence\n- `summary`: Aggregate pass/fail counts\n- `execution_metrics`: Tool usage and output size (from executor's metrics.json)\n- `timing`: Wall clock timing (from timing.json)\n- `claims`: Extracted and verified claims from the output\n- `user_notes_summary`: Issues flagged by the executor\n- `eval_feedback`: (optional) Improvement suggestions for the evals, only present when the grader identifies issues worth raising\n\n---\n\n## metrics.json\n\nOutput from the executor agent. Located at `/outputs/metrics.json`.\n\n```json\n{\n \"tool_calls\": {\n \"Read\": 5,\n \"Write\": 2,\n \"Bash\": 8,\n \"Edit\": 1,\n \"Glob\": 2,\n \"Grep\": 0\n },\n \"total_tool_calls\": 18,\n \"total_steps\": 6,\n \"files_created\": [\"filled_form.pdf\", \"field_values.json\"],\n \"errors_encountered\": 0,\n \"output_chars\": 12450,\n \"transcript_chars\": 3200\n}\n```\n\n**Fields:**\n- `tool_calls`: Count per tool type\n- `total_tool_calls`: Sum of all tool calls\n- `total_steps`: Number of major execution steps\n- `files_created`: List of output files created\n- `errors_encountered`: Number of errors during execution\n- `output_chars`: Total character count of output files\n- `transcript_chars`: Character count of transcript\n\n---\n\n## timing.json\n\nWall clock timing for a run. Located at `/timing.json`.\n\n**How to capture:** When a subagent task completes, the task notification includes `total_tokens` and `duration_ms`. Save these immediately — they are not persisted anywhere else and cannot be recovered after the fact.\n\n```json\n{\n \"total_tokens\": 84852,\n \"duration_ms\": 23332,\n \"total_duration_seconds\": 23.3,\n \"executor_start\": \"2026-01-15T10:30:00Z\",\n \"executor_end\": \"2026-01-15T10:32:45Z\",\n \"executor_duration_seconds\": 165.0,\n \"grader_start\": \"2026-01-15T10:32:46Z\",\n \"grader_end\": \"2026-01-15T10:33:12Z\",\n \"grader_duration_seconds\": 26.0\n}\n```\n\n---\n\n## benchmark.json\n\nOutput from Benchmark mode. Located at `benchmarks//benchmark.json`.\n\n```json\n{\n \"metadata\": {\n \"skill_name\": \"pdf\",\n \"skill_path\": \"/path/to/pdf\",\n \"executor_model\": \"claude-sonnet-4-20250514\",\n \"analyzer_model\": \"most-capable-model\",\n \"timestamp\": \"2026-01-15T10:30:00Z\",\n \"evals_run\": [1, 2, 3],\n \"runs_per_configuration\": 3\n },\n\n \"runs\": [\n {\n \"eval_id\": 1,\n \"eval_name\": \"Ocean\",\n \"configuration\": \"with_skill\",\n \"run_number\": 1,\n \"result\": {\n \"pass_rate\": 0.85,\n \"passed\": 6,\n \"failed\": 1,\n \"total\": 7,\n \"time_seconds\": 42.5,\n \"tokens\": 3800,\n \"tool_calls\": 18,\n \"errors\": 0\n },\n \"expectations\": [\n {\"text\": \"...\", \"passed\": true, \"evidence\": \"...\"}\n ],\n \"notes\": [\n \"Used 2023 data, may be stale\",\n \"Fell back to text overlay for non-fillable fields\"\n ]\n }\n ],\n\n \"run_summary\": {\n \"with_skill\": {\n \"pass_rate\": {\"mean\": 0.85, \"stddev\": 0.05, \"min\": 0.80, \"max\": 0.90},\n \"time_seconds\": {\"mean\": 45.0, \"stddev\": 12.0, \"min\": 32.0, \"max\": 58.0},\n \"tokens\": {\"mean\": 3800, \"stddev\": 400, \"min\": 3200, \"max\": 4100}\n },\n \"without_skill\": {\n \"pass_rate\": {\"mean\": 0.35, \"stddev\": 0.08, \"min\": 0.28, \"max\": 0.45},\n \"time_seconds\": {\"mean\": 32.0, \"stddev\": 8.0, \"min\": 24.0, \"max\": 42.0},\n \"tokens\": {\"mean\": 2100, \"stddev\": 300, \"min\": 1800, \"max\": 2500}\n },\n \"delta\": {\n \"pass_rate\": \"+0.50\",\n \"time_seconds\": \"+13.0\",\n \"tokens\": \"+1700\"\n }\n },\n\n \"notes\": [\n \"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value\",\n \"Eval 3 shows high variance (50% ± 40%) - may be flaky or model-dependent\",\n \"Without-skill runs consistently fail on table extraction expectations\",\n \"Skill adds 13s average execution time but improves pass rate by 50%\"\n ]\n}\n```\n\n**Fields:**\n- `metadata`: Information about the benchmark run\n - `skill_name`: Name of the skill\n - `timestamp`: When the benchmark was run\n - `evals_run`: List of eval names or IDs\n - `runs_per_configuration`: Number of runs per config (e.g. 3)\n- `runs[]`: Individual run results\n - `eval_id`: Numeric eval identifier\n - `eval_name`: Human-readable eval name (used as section header in the viewer)\n - `configuration`: Must be `\"with_skill\"` or `\"without_skill\"` (the viewer uses this exact string for grouping and color coding)\n - `run_number`: Integer run number (1, 2, 3...)\n - `result`: Nested object with `pass_rate`, `passed`, `total`, `time_seconds`, `tokens`, `errors`\n- `run_summary`: Statistical aggregates per configuration\n - `with_skill` / `without_skill`: Each contains `pass_rate`, `time_seconds`, `tokens` objects with `mean` and `stddev` fields\n - `delta`: Difference strings like `\"+0.50\"`, `\"+13.0\"`, `\"+1700\"`\n- `notes`: Freeform observations from the analyzer\n\n**Important:** The viewer reads these field names exactly. Using `config` instead of `configuration`, or putting `pass_rate` at the top level of a run instead of nested under `result`, will cause the viewer to show empty/zero values. Always reference this schema when generating benchmark.json manually.\n\n---\n\n## comparison.json\n\nOutput from blind comparator. Located at `/comparison-N.json`.\n\n```json\n{\n \"winner\": \"A\",\n \"reasoning\": \"Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.\",\n \"rubric\": {\n \"A\": {\n \"content\": {\n \"correctness\": 5,\n \"completeness\": 5,\n \"accuracy\": 4\n },\n \"structure\": {\n \"organization\": 4,\n \"formatting\": 5,\n \"usability\": 4\n },\n \"content_score\": 4.7,\n \"structure_score\": 4.3,\n \"overall_score\": 9.0\n },\n \"B\": {\n \"content\": {\n \"correctness\": 3,\n \"completeness\": 2,\n \"accuracy\": 3\n },\n \"structure\": {\n \"organization\": 3,\n \"formatting\": 2,\n \"usability\": 3\n },\n \"content_score\": 2.7,\n \"structure_score\": 2.7,\n \"overall_score\": 5.4\n }\n },\n \"output_quality\": {\n \"A\": {\n \"score\": 9,\n \"strengths\": [\"Complete solution\", \"Well-formatted\", \"All fields present\"],\n \"weaknesses\": [\"Minor style inconsistency in header\"]\n },\n \"B\": {\n \"score\": 5,\n \"strengths\": [\"Readable output\", \"Correct basic structure\"],\n \"weaknesses\": [\"Missing date field\", \"Formatting inconsistencies\", \"Partial data extraction\"]\n }\n },\n \"expectation_results\": {\n \"A\": {\n \"passed\": 4,\n \"total\": 5,\n \"pass_rate\": 0.80,\n \"details\": [\n {\"text\": \"Output includes name\", \"passed\": true}\n ]\n },\n \"B\": {\n \"passed\": 3,\n \"total\": 5,\n \"pass_rate\": 0.60,\n \"details\": [\n {\"text\": \"Output includes name\", \"passed\": true}\n ]\n }\n }\n}\n```\n\n---\n\n## analysis.json\n\nOutput from post-hoc analyzer. Located at `/analysis.json`.\n\n```json\n{\n \"comparison_summary\": {\n \"winner\": \"A\",\n \"winner_skill\": \"path/to/winner/skill\",\n \"loser_skill\": \"path/to/loser/skill\",\n \"comparator_reasoning\": \"Brief summary of why comparator chose winner\"\n },\n \"winner_strengths\": [\n \"Clear step-by-step instructions for handling multi-page documents\",\n \"Included validation script that caught formatting errors\"\n ],\n \"loser_weaknesses\": [\n \"Vague instruction 'process the document appropriately' led to inconsistent behavior\",\n \"No script for validation, agent had to improvise\"\n ],\n \"instruction_following\": {\n \"winner\": {\n \"score\": 9,\n \"issues\": [\"Minor: skipped optional logging step\"]\n },\n \"loser\": {\n \"score\": 6,\n \"issues\": [\n \"Did not use the skill's formatting template\",\n \"Invented own approach instead of following step 3\"\n ]\n }\n },\n \"improvement_suggestions\": [\n {\n \"priority\": \"high\",\n \"category\": \"instructions\",\n \"suggestion\": \"Replace 'process the document appropriately' with explicit steps\",\n \"expected_impact\": \"Would eliminate ambiguity that caused inconsistent behavior\"\n }\n ],\n \"transcript_insights\": {\n \"winner_execution_pattern\": \"Read skill -> Followed 5-step process -> Used validation script\",\n \"loser_execution_pattern\": \"Read skill -> Unclear on approach -> Tried 3 different methods\"\n }\n}\n```\n","flock-data/skills/using-superpowers/SKILL.md":"---\nname: using-superpowers\ndescription: Use when starting any conversation - establishes how to find and use skills, requiring Skill tool invocation before ANY response including clarifying questions\n---\n\n\nIf you were dispatched as a subagent to execute a specific task, skip this skill.\n\n\n\nIf you think there is even a 1% chance a skill might apply to what you are doing, you ABSOLUTELY MUST invoke the skill.\n\nIF A SKILL APPLIES TO YOUR TASK, YOU DO NOT HAVE A CHOICE. YOU MUST USE IT.\n\nThis is not negotiable. This is not optional. You cannot rationalize your way out of this.\n\n\n## Instruction Priority\n\nSuperpowers skills override default system prompt behavior, but **user instructions always take precedence**:\n\n1. **User's explicit instructions** (CLAUDE.md, GEMINI.md, AGENTS.md, direct requests) — highest priority\n2. **Superpowers skills** — override default system behavior where they conflict\n3. **Default system prompt** — lowest priority\n\nIf CLAUDE.md, GEMINI.md, or AGENTS.md says \"don't use TDD\" and a skill says \"always use TDD,\" follow the user's instructions. The user is in control.\n\n## How to Access Skills\n\n**In Claude Code:** Use the `Skill` tool. When you invoke a skill, its content is loaded and presented to you—follow it directly. Never use the Read tool on skill files.\n\n**In Gemini CLI:** Skills activate via the `activate_skill` tool. Gemini loads skill metadata at session start and activates the full content on demand.\n\n**In other environments:** Check your platform's documentation for how skills are loaded.\n\n## Platform Adaptation\n\nSkills use Claude Code tool names. Non-CC platforms: see `references/codex-tools.md` (Codex) for tool equivalents. Gemini CLI users get the tool mapping loaded automatically via GEMINI.md.\n\n# Using Skills\n\n## The Rule\n\n**Invoke relevant or requested skills BEFORE any response or action.** Even a 1% chance a skill might apply means that you should invoke the skill to check. If an invoked skill turns out to be wrong for the situation, you don't need to use it.\n\n```dot\ndigraph skill_flow {\n \"User message received\" [shape=doublecircle];\n \"About to EnterPlanMode?\" [shape=doublecircle];\n \"Already brainstormed?\" [shape=diamond];\n \"Invoke brainstorming skill\" [shape=box];\n \"Might any skill apply?\" [shape=diamond];\n \"Invoke Skill tool\" [shape=box];\n \"Announce: 'Using [skill] to [purpose]'\" [shape=box];\n \"Has checklist?\" [shape=diamond];\n \"Create TodoWrite todo per item\" [shape=box];\n \"Follow skill exactly\" [shape=box];\n \"Respond (including clarifications)\" [shape=doublecircle];\n\n \"About to EnterPlanMode?\" -> \"Already brainstormed?\";\n \"Already brainstormed?\" -> \"Invoke brainstorming skill\" [label=\"no\"];\n \"Already brainstormed?\" -> \"Might any skill apply?\" [label=\"yes\"];\n \"Invoke brainstorming skill\" -> \"Might any skill apply?\";\n\n \"User message received\" -> \"Might any skill apply?\";\n \"Might any skill apply?\" -> \"Invoke Skill tool\" [label=\"yes, even 1%\"];\n \"Might any skill apply?\" -> \"Respond (including clarifications)\" [label=\"definitely not\"];\n \"Invoke Skill tool\" -> \"Announce: 'Using [skill] to [purpose]'\";\n \"Announce: 'Using [skill] to [purpose]'\" -> \"Has checklist?\";\n \"Has checklist?\" -> \"Create TodoWrite todo per item\" [label=\"yes\"];\n \"Has checklist?\" -> \"Follow skill exactly\" [label=\"no\"];\n \"Create TodoWrite todo per item\" -> \"Follow skill exactly\";\n}\n```\n\n## Red Flags\n\nThese thoughts mean STOP—you're rationalizing:\n\n| Thought | Reality |\n|---------|---------|\n| \"This is just a simple question\" | Questions are tasks. Check for skills. |\n| \"I need more context first\" | Skill check comes BEFORE clarifying questions. |\n| \"Let me explore the codebase first\" | Skills tell you HOW to explore. Check first. |\n| \"I can check git/files quickly\" | Files lack conversation context. Check for skills. |\n| \"Let me gather information first\" | Skills tell you HOW to gather information. |\n| \"This doesn't need a formal skill\" | If a skill exists, use it. |\n| \"I remember this skill\" | Skills evolve. Read current version. |\n| \"This doesn't count as a task\" | Action = task. Check for skills. |\n| \"The skill is overkill\" | Simple things become complex. Use it. |\n| \"I'll just do this one thing first\" | Check BEFORE doing anything. |\n| \"This feels productive\" | Undisciplined action wastes time. Skills prevent this. |\n| \"I know what that means\" | Knowing the concept ≠ using the skill. Invoke it. |\n\n## Skill Priority\n\nWhen multiple skills could apply, use this order:\n\n1. **Process skills first** (brainstorming, debugging) - these determine HOW to approach the task\n2. **Implementation skills second** (frontend-design, mcp-builder) - these guide execution\n\n\"Let's build X\" → brainstorming first, then implementation skills.\n\"Fix this bug\" → debugging first, then domain-specific skills.\n\n## Skill Types\n\n**Rigid** (TDD, debugging): Follow exactly. Don't adapt away discipline.\n\n**Flexible** (patterns): Adapt principles to context.\n\nThe skill itself tells you which.\n\n## User Instructions\n\nInstructions say WHAT, not HOW. \"Add X\" or \"Fix Y\" doesn't mean skip workflows.\n","flock-data/skills/using-superpowers/references/codex-tools.md":"# Codex Tool Mapping\n\nSkills use Claude Code tool names. When you encounter these in a skill, use your platform equivalent:\n\n| Skill references | Codex equivalent |\n|-----------------|------------------|\n| `Task` tool (dispatch subagent) | `spawn_agent` |\n| Multiple `Task` calls (parallel) | Multiple `spawn_agent` calls |\n| Task returns result | `wait` |\n| Task completes automatically | `close_agent` to free slot |\n| `TodoWrite` (task tracking) | `update_plan` |\n| `Skill` tool (invoke a skill) | Skills load natively — just follow the instructions |\n| `Read`, `Write`, `Edit` (files) | Use your native file tools |\n| `Bash` (run commands) | Use your native shell tools |\n\n## Subagent dispatch requires collab\n\nAdd to your Codex config (`~/.codex/config.toml`):\n\n```toml\n[features]\ncollab = true\n```\n\nThis enables `spawn_agent`, `wait`, and `close_agent` for skills like `dispatching-parallel-agents` and `subagent-driven-development`.\n","flock-data/skills/using-superpowers/references/gemini-tools.md":"# Gemini CLI Tool Mapping\n\nSkills use Claude Code tool names. When you encounter these in a skill, use your platform equivalent:\n\n| Skill references | Gemini CLI equivalent |\n|-----------------|----------------------|\n| `Read` (file reading) | `read_file` |\n| `Write` (file creation) | `write_file` |\n| `Edit` (file editing) | `replace` |\n| `Bash` (run commands) | `run_shell_command` |\n| `Grep` (search file content) | `grep_search` |\n| `Glob` (search files by name) | `glob` |\n| `TodoWrite` (task tracking) | `write_todos` |\n| `Skill` tool (invoke a skill) | `activate_skill` |\n| `WebSearch` | `google_web_search` |\n| `WebFetch` | `web_fetch` |\n| `Task` tool (dispatch subagent) | No equivalent — Gemini CLI does not support subagents |\n\n## No subagent support\n\nGemini CLI has no equivalent to Claude Code's `Task` tool. Skills that rely on subagent dispatch (`subagent-driven-development`, `dispatching-parallel-agents`) will fall back to single-session execution via `executing-plans`.\n\n## Additional Gemini CLI tools\n\nThese tools are available in Gemini CLI but have no Claude Code equivalent:\n\n| Tool | Purpose |\n|------|---------|\n| `list_directory` | List files and subdirectories |\n| `save_memory` | Persist facts to GEMINI.md across sessions |\n| `ask_user` | Request structured input from the user |\n| `tracker_create_task` | Rich task management (create, update, list, visualize) |\n| `enter_plan_mode` / `exit_plan_mode` | Switch to read-only research mode before making changes |\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/Onelevenvy/flock/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# AGENTS.md\n\nThis file provides guidance to CodeX when working with code in this repository.\n\n## Project Overview\n\nFlock is a multi-provider AI agent desktop application with tool orchestration, sandbox execution, visual workflow, and browser/computer-use capabilities. It's a Rust workspace with a Tauri-based desktop UI (React + TypeScript + Mantine).\n\n## Build & Development Commands\n\n### Rust Backend\n```bash\n# Build the CLI\ncargo build --release\n\n# Run in single-shot mode\ncargo run -- \"your question\"\n\n# Run interactive REPL\ncargo run\n\n# Run with specific provider/profile\ncargo run -- --profile deepseek \"your question\"\n\n# Run tests (workspace-wide)\ncargo test\n\n# Run a single crate's tests\ncargo test -p flock-core\ncargo test -p flock-agent\ncargo test -p flock-tools\ncargo test -p flock-skills\n\n# Run a specific test\ncargo test -p flock-core test_name\n\n# Lint\ncargo clippy --workspace\n```\n\n### Tauri Desktop UI\n```bash\ncd flock-ui\n\n# Install dependencies\nnpm install\n\n# Development server\nnpm run dev\n# or\nnpm run tauri dev\n\n# Build for production\nnpm run build\n# or\nnpm run tauri build\n\n# Lint frontend\nnpm run lint\n```\n\n## Architecture\n\n### Workspace Crates\n\n| Crate | Purpose |\n|-------|---------|\n| `flock-core` | Core types, config, database, IPC interface, crypto, model factory |\n| `flock-agent` | Agent engine, session management, tool execution, memory, graph orchestration, workflow engine |\n| `flock-tools` | Tool registry, built-in tools (Read/Write/Edit/Bash/Grep/Glob), MCP integration, sandbox tools (CodeExecution, SandboxExec, Browser, ComputerUse) |\n| `flock-skills` | Skill discovery, loading, frontmatter parsing, hooks, permissions, bundled skills |\n| `flock-ui/src-tauri` | Tauri desktop app backend (commands, agent state, workspace management) |\n| `workspace-hack` | cargo-hakari build optimization |\n\n### Key Dependencies\n\n- **LangGraph**: Rust reimplementation ([langgraph-rust](https://github.com/Onelevenvy/langgraph-rust)) for agent graph orchestration\n- **LLM Providers**: OpenAI-compatible (default), Anthropic, AWS Bedrock, Google Vertex\n- **Database**: SQLite via sqlx for sessions, conversations, providers, tools, MCP servers, workflows\n- **Async Runtime**: Tokio\n- **Sandbox**: Daytona cloud container runtime for isolated code execution\n- **Browser Automation**: Playwright (Python, runs inside sandbox via CDP)\n\n### Core Architecture Patterns\n\n**Agent Engine** (`flock-agent/src/engine.rs`):\n- `AgentEngine` orchestrates the LLM interaction loop\n- Uses LangGraph `CompiledStateGraph` for execution flow\n- Manages conversation history via SQLite checkpointer\n- Handles tool approval, context compression, plan mode\n\n**Tool System** (`flock-tools/src/lib.rs`):\n- `Tool` trait defines the interface: `name()`, `description()`, `input_schema()`, `execute()`\n- `ToolRegistry` manages all registered tools\n- Built-in tools: Read, Write, Edit, Bash, Grep, Glob\n- Sandbox tools: CodeExecution, SandboxExec, Browser, ComputerUse, RequestHumanAssistance\n- Extensible via MCP servers and custom tool providers\n\n**Workflow Engine** (`flock-agent/src/workflow_graph/`):\n- Visual workflow builder with ReactFlow canvas\n- 10 node types: start, llm, agent, classifier, ifelse, answer, code, human, plugin, parameter_extractor\n- `build_workflow_graph()` compiles ReactFlow JSON into LangGraph `StateGraph`\n- Streaming execution with human-in-the-loop interrupts\n\n**Sandbox System** (`flock-tools/src/daytona/`):\n- Cloud-based Daytona container runtime for isolated execution\n- VNC desktop provisioning (Xvfb, fluxbox, x11vnc, websockify)\n- Lifecycle management: create, destroy, set-public, snapshot\n- Human takeover via VNC with approval flow\n\n**Skills System** (`flock-skills/`):\n- Markdown files with YAML frontmatter in `.flock/skills/`\n- Supports `$ARGUMENTS` substitution, shell execution, conditional activation\n- Two execution contexts: `inline` (same agent) and `fork` (sub-agent)\n- Hot-reload via file watcher\n\n**Memory System** (`flock-agent/src/memory/`):\n- Four types: `user`, `feedback`, `project`, `reference`\n- Stored as Markdown files with frontmatter in `/flock/projects//memory/`\n- `MEMORY.md` index file auto-loaded into system prompt\n\n**IPC Interface** (`flock-core/src/ipc_interface/`):\n- JSON-based protocol for host ↔ agent communication\n- Events (Agent → Host): `ready`, `stream_start`, `text_delta`, `tool_request`, `tool_result`, etc.\n- Commands (Host → Agent): `message`, `stop`, `tool_approve`, `tool_deny`, `set_mode`, `set_config`\n\n**Configuration** (`flock-core/src/config/settings.rs`):\n- Config priority: CLI args/env > project `flock.toml` > global config\n- Supports profiles with inheritance (`extends`)\n- Provider compatibility settings (`ProviderCompat`) for API differences\n\n### Tauri Desktop UI\n\n**Frontend Stack**:\n- React 18 + TypeScript\n- Mantine UI v8 (components, notifications)\n- Zustand (state management)\n- React Query (server state)\n- react-markdown + react-syntax-highlighter (message rendering)\n- i18next (internationalization)\n- ReactFlow (workflow visual editor)\n\n**State Stores** (`flock-ui/src/store/`):\n- `agentStore`: Agent connection status, messages, pending approvals\n- `uiStore`: Theme, sidebar state, active view, file tree\n- `workspaceStore`: Active workspace/conversation, persisted to localStorage\n- `workflowStore`: ReactFlow nodes/edges, execution state, dirty tracking\n\n**Key Views**:\n- `HomeView`: Welcome screen, assistant selection, workspace picker\n- `WorkspaceView`: Chat panel + file tree + optional preview panel (VNC, sandbox runner)\n- `WorkflowEditor`: Visual workflow builder with ReactFlow canvas, node palette, properties panel, execution panel\n- `AssistantPage`: CRUD for custom assistants\n- `SkillsPage`: Tools, MCP servers, and skills management\n- `SettingsModal`: Model providers, basic settings, sandbox settings, system settings\n\n**Tauri Commands** (`flock-ui/src-tauri/src/commands/`):\n- Agent control: `start_agent`, `stop_agent`, `send_message`, `approve_tool`, `deny_tool`\n- Workspace: `list_workspaces`, `create_workspace`, `delete_workspace`\n- Conversations: `list_conversations`, `create_conversation`, `load_conversation_history`\n- Database: `list_providers`, `upsert_provider`, `list_models`, `upsert_model`\n- MCP: `list_mcp_servers`, `upsert_mcp_server`, `test_mcp_server`\n- Skills: `list_skills`, `get_extra_skill_dirs`, `add_extra_skill_dir`\n- Workflow: `list_workflows`, `get_workflow`, `create_workflow`, `update_workflow`, `delete_workflow`, `run_workflow`, `stop_workflow`\n\n### Data Flow\n\n1. User sends message via UI → Tauri command `send_message`\n2. `AgentState` forwards to `AgentEngine::run()`\n3. Engine builds LangGraph, invokes LLM with tools\n4. LLM response streamed as `ProtocolEvent`s via `TauriProtocolEmitter`\n5. Frontend `useEventStream` hook listens to Tauri events, updates `agentStore`\n6. Tool calls require approval (unless auto-approved) → `ToolApprovalInline` component\n7. Approved tools execute → results fed back to LLM for next turn\n\n## Code Style\n\n- Rust edition 2024\n- Workspace lints: `unused = \"allow\"`, `unused_imports = \"allow\"`\n- Do not treat repository-wide `cargo fmt --check` diffs or warnings such as unused imports/variables as blockers unless they are introduced by the current change or explicitly requested. Focus verification on build/test failures and behavior regressions.\n- Error handling: `anyhow` for applications, `thiserror` for libraries\n- Async: Tokio runtime throughout\n- Frontend: ESLint with TypeScript rules, React hooks plugin\n\n## Global Guidelines & Core Principles\n\nWhen modifying the codebase or interacting with the user, please strictly adhere to the following rules:\n\n1. **Git Commit Format**: Upon completing a task, always output a single-line English `git commit` message beginning with standard prefixes (`feat:`, `fix:`, `chore:`, etc.). Provide the message text clearly, but **DO NOT** execute git commit commands or auto-submit.\n2. **State & Logic Abstraction**: Always separate state management and command logic from UI components. Place custom Hooks in `src/hooks` and Zustand stores in `src/store` rather than defining them inside component files.\n3. **Strict i18n Internationalization**: All newly introduced UI components must leverage `i18next` for user-facing texts (e.g. tooltips, titles, toast notifications, confirm boxes). Do not hardcode Chinese or English strings inside JSX.\n4. **Component Granularity**: Strive for modularity. Always split large layout or page files into focused, localized sub-components (placed in a nested `/components` folder). Avoid keeping all view code inside single massive `index.tsx` files.\n5. **UI Specification Compliance**: Ensure any new styling aligns with the design tokens defined in `docs/resources/ui_specification.md` (e.g. cold gray colors, `--flock-accent` solid blue branding, standard monospace typographies, and hover-card transitions) in both dark and light modes.\n","isInternal":false,"tokens":2122,"sizeBytes":8948},{"name":"CLAUDE.md","path":"CLAUDE.md","rawUrl":"https://raw.githubusercontent.com/Onelevenvy/flock/HEAD/CLAUDE.md","title":"Claude Agent Guidelines & System Prompt","category":"claude-rule","format":"markdown","content":"# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nFlock is a multi-provider AI agent desktop application with tool orchestration, sandbox execution, visual workflow, and browser/computer-use capabilities. It's a Rust workspace with a Tauri-based desktop UI (React + TypeScript + Mantine).\n\n## Build & Development Commands\n\n### Rust Backend\n```bash\n# Build the CLI\ncargo build --release\n\n# Run in single-shot mode\ncargo run -- \"your question\"\n\n# Run interactive REPL\ncargo run\n\n# Run with specific provider/profile\ncargo run -- --profile deepseek \"your question\"\n\n# Run tests (workspace-wide)\ncargo test\n\n# Run a single crate's tests\ncargo test -p flock-core\ncargo test -p flock-agent\ncargo test -p flock-tools\ncargo test -p flock-skills\n\n# Run a specific test\ncargo test -p flock-core test_name\n\n# Lint\ncargo clippy --workspace\n```\n\n### Tauri Desktop UI\n```bash\ncd flock-ui\n\n# Install dependencies\nnpm install\n\n# Development server\nnpm run dev\n# or\nnpm run tauri dev\n\n# Build for production\nnpm run build\n# or\nnpm run tauri build\n\n# Lint frontend\nnpm run lint\n```\n\n## Architecture\n\n### Workspace Crates\n\n| Crate | Purpose |\n|-------|---------|\n| `flock-core` | Core types, config, database, IPC interface, crypto, model factory |\n| `flock-agent` | Agent engine, session management, tool execution, memory, graph orchestration, workflow engine |\n| `flock-tools` | Tool registry, built-in tools (Read/Write/Edit/Bash/Grep/Glob), MCP integration, sandbox tools (CodeExecution, SandboxExec, Browser, ComputerUse) |\n| `flock-skills` | Skill discovery, loading, frontmatter parsing, hooks, permissions, bundled skills |\n| `flock-ui/src-tauri` | Tauri desktop app backend (commands, agent state, workspace management) |\n| `workspace-hack` | cargo-hakari build optimization |\n\n### Key Dependencies\n\n- **LangGraph**: Rust reimplementation ([langgraph-rust](https://github.com/Onelevenvy/langgraph-rust)) for agent graph orchestration\n- **LLM Providers**: OpenAI-compatible (default), Anthropic, AWS Bedrock, Google Vertex\n- **Database**: SQLite via sqlx for sessions, conversations, providers, tools, MCP servers, workflows\n- **Async Runtime**: Tokio\n- **Sandbox**: Daytona cloud container runtime for isolated code execution\n- **Browser Automation**: Playwright (Python, runs inside sandbox via CDP)\n\n### Core Architecture Patterns\n\n**Agent Engine** (`flock-agent/src/engine.rs`):\n- `AgentEngine` orchestrates the LLM interaction loop\n- Uses LangGraph `CompiledStateGraph` for execution flow\n- Manages conversation history via SQLite checkpointer\n- Handles tool approval, context compression, plan mode\n\n**Tool System** (`flock-tools/src/lib.rs`):\n- `Tool` trait defines the interface: `name()`, `description()`, `input_schema()`, `execute()`\n- `ToolRegistry` manages all registered tools\n- Built-in tools: Read, Write, Edit, Bash, Grep, Glob\n- Sandbox tools: CodeExecution, SandboxExec, Browser, ComputerUse, RequestHumanAssistance\n- Extensible via MCP servers and custom tool providers\n\n**Workflow Engine** (`flock-agent/src/workflow_graph/`):\n- Visual workflow builder with ReactFlow canvas\n- 10 node types: start, llm, agent, classifier, ifelse, answer, code, human, plugin, parameter_extractor\n- `build_workflow_graph()` compiles ReactFlow JSON into LangGraph `StateGraph`\n- Streaming execution with human-in-the-loop interrupts\n\n**Sandbox System** (`flock-tools/src/daytona/`):\n- Cloud-based Daytona container runtime for isolated execution\n- VNC desktop provisioning (Xvfb, fluxbox, x11vnc, websockify)\n- Lifecycle management: create, destroy, set-public, snapshot\n- Human takeover via VNC with approval flow\n\n**Skills System** (`flock-skills/`):\n- Markdown files with YAML frontmatter in `.flock/skills/`\n- Supports `$ARGUMENTS` substitution, shell execution, conditional activation\n- Two execution contexts: `inline` (same agent) and `fork` (sub-agent)\n- Hot-reload via file watcher\n\n**Memory System** (`flock-agent/src/memory/`):\n- Four types: `user`, `feedback`, `project`, `reference`\n- Stored as Markdown files with frontmatter in `/flock/projects//memory/`\n- `MEMORY.md` index file auto-loaded into system prompt\n\n**IPC Interface** (`flock-core/src/ipc_interface/`):\n- JSON-based protocol for host ↔ agent communication\n- Events (Agent → Host): `ready`, `stream_start`, `text_delta`, `tool_request`, `tool_result`, etc.\n- Commands (Host → Agent): `message`, `stop`, `tool_approve`, `tool_deny`, `set_mode`, `set_config`\n\n**Configuration** (`flock-core/src/config/settings.rs`):\n- Config priority: CLI args/env > project `flock.toml` > global config\n- Supports profiles with inheritance (`extends`)\n- Provider compatibility settings (`ProviderCompat`) for API differences\n\n### Tauri Desktop UI\n\n**Frontend Stack**:\n- React 18 + TypeScript\n- Mantine UI v8 (components, notifications)\n- Zustand (state management)\n- React Query (server state)\n- react-markdown + react-syntax-highlighter (message rendering)\n- i18next (internationalization)\n- ReactFlow (workflow visual editor)\n\n**State Stores** (`flock-ui/src/store/`):\n- `agentStore`: Agent connection status, messages, pending approvals\n- `uiStore`: Theme, sidebar state, active view, file tree\n- `workspaceStore`: Active workspace/conversation, persisted to localStorage\n- `workflowStore`: ReactFlow nodes/edges, execution state, dirty tracking\n\n**Key Views**:\n- `HomeView`: Welcome screen, assistant selection, workspace picker\n- `WorkspaceView`: Chat panel + file tree + optional preview panel (VNC, sandbox runner)\n- `WorkflowEditor`: Visual workflow builder with ReactFlow canvas, node palette, properties panel, execution panel\n- `AssistantPage`: CRUD for custom assistants\n- `SkillsPage`: Tools, MCP servers, and skills management\n- `SettingsModal`: Model providers, basic settings, sandbox settings, system settings\n\n**Tauri Commands** (`flock-ui/src-tauri/src/commands/`):\n- Agent control: `start_agent`, `stop_agent`, `send_message`, `approve_tool`, `deny_tool`\n- Workspace: `list_workspaces`, `create_workspace`, `delete_workspace`\n- Conversations: `list_conversations`, `create_conversation`, `load_conversation_history`\n- Database: `list_providers`, `upsert_provider`, `list_models`, `upsert_model`\n- MCP: `list_mcp_servers`, `upsert_mcp_server`, `test_mcp_server`\n- Skills: `list_skills`, `get_extra_skill_dirs`, `add_extra_skill_dir`\n- Workflow: `list_workflows`, `get_workflow`, `create_workflow`, `update_workflow`, `delete_workflow`, `run_workflow`, `stop_workflow`\n\n### Data Flow\n\n1. User sends message via UI → Tauri command `send_message`\n2. `AgentState` forwards to `AgentEngine::run()`\n3. Engine builds LangGraph, invokes LLM with tools\n4. LLM response streamed as `ProtocolEvent`s via `TauriProtocolEmitter`\n5. Frontend `useEventStream` hook listens to Tauri events, updates `agentStore`\n6. Tool calls require approval (unless auto-approved) → `ToolApprovalInline` component\n7. Approved tools execute → results fed back to LLM for next turn\n\n## Code Style\n\n- Rust edition 2024\n- Workspace lints: `unused = \"allow\"`, `unused_imports = \"allow\"`\n- Do not treat repository-wide `cargo fmt --check` diffs or warnings such as unused imports/variables as blockers unless they are introduced by the current change or explicitly requested. Focus verification on build/test failures and behavior regressions.\n- Error handling: `anyhow` for applications, `thiserror` for libraries\n- Async: Tokio runtime throughout\n- Frontend: ESLint with TypeScript rules, React hooks plugin\n\n\n## Global Guidelines & Core Principles\n\nWhen modifying the codebase or interacting with the user, please strictly adhere to the following rules:\n\n1. **Git Commit Format**: Upon completing a task, always output a single-line English `git commit` message beginning with standard prefixes (`feat:`, `fix:`, `chore:`, etc.). Provide the message text clearly, but **DO NOT** execute git commit commands or auto-submit.\n2. **State & Logic Abstraction**: Always separate state management and command logic from UI components. Place custom Hooks in `src/hooks` and Zustand stores in `src/store` rather than defining them inside component files.\n3. **Strict i18n Internationalization**: All newly introduced UI components must leverage `i18next` for user-facing texts (e.g. tooltips, titles, toast notifications, confirm boxes). Do not hardcode Chinese or English strings inside JSX.\n4. **Component Granularity**: Strive for modularity. Always split large layout or page files into focused, localized sub-components (placed in a nested `/components` folder). Avoid keeping all view code inside single massive `index.tsx` files.\n5. **UI Specification Compliance**: Ensure any new styling aligns with the design tokens defined in `docs/resources/ui_specification.md` (e.g. cold gray colors, `--flock-accent` solid blue branding, standard monospace typographies, and hover-card transitions) in both dark and light modes.\n","isInternal":false,"tokens":2129,"sizeBytes":8972},{"name":"SKILL.md","path":"flock-data/skills/frontend-design/SKILL.md","rawUrl":"https://raw.githubusercontent.com/Onelevenvy/flock/HEAD/flock-data/skills/frontend-design/SKILL.md","title":"frontend-design","category":"anthropic-skill","format":"markdown","content":"---\nname: frontend-design\ndescription: Create distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, or applications. Generates creative, polished code that avoids generic AI aesthetics.\nlicense: Complete terms in LICENSE.txt\n---\n\nThis skill guides creation of distinctive, production-grade frontend interfaces that avoid generic \"AI slop\" aesthetics. Implement real working code with exceptional attention to aesthetic details and creative choices.\n\nThe user provides frontend requirements: a component, page, application, or interface to build. They may include context about the purpose, audience, or technical constraints.\n\n## Design Thinking\n\nBefore coding, understand the context and commit to a BOLD aesthetic direction:\n- **Purpose**: What problem does this interface solve? Who uses it?\n- **Tone**: Pick an extreme: brutally minimal, maximalist chaos, retro-futuristic, organic/natural, luxury/refined, playful/toy-like, editorial/magazine, brutalist/raw, art deco/geometric, soft/pastel, industrial/utilitarian, etc. There are so many flavors to choose from. Use these for inspiration but design one that is true to the aesthetic direction.\n- **Constraints**: Technical requirements (framework, performance, accessibility).\n- **Differentiation**: What makes this UNFORGETTABLE? What's the one thing someone will remember?\n\n**CRITICAL**: Choose a clear conceptual direction and execute it with precision. Bold maximalism and refined minimalism both work - the key is intentionality, not intensity.\n\nThen implement working code (HTML/CSS/JS, React, Vue, etc.) that is:\n- Production-grade and functional\n- Visually striking and memorable\n- Cohesive with a clear aesthetic point-of-view\n- Meticulously refined in every detail\n\n## Frontend Aesthetics Guidelines\n\nFocus on:\n- **Typography**: Choose fonts that are beautiful, unique, and interesting. Avoid generic fonts like Arial and Inter; opt instead for distinctive choices that elevate the frontend's aesthetics; unexpected, characterful font choices. Pair a distinctive display font with a refined body font.\n- **Color & Theme**: Commit to a cohesive aesthetic. Use CSS variables for consistency. Dominant colors with sharp accents outperform timid, evenly-distributed palettes.\n- **Motion**: Use animations for effects and micro-interactions. Prioritize CSS-only solutions for HTML. Use Motion library for React when available. Focus on high-impact moments: one well-orchestrated page load with staggered reveals (animation-delay) creates more delight than scattered micro-interactions. Use scroll-triggering and hover states that surprise.\n- **Spatial Composition**: Unexpected layouts. Asymmetry. Overlap. Diagonal flow. Grid-breaking elements. Generous negative space OR controlled density.\n- **Backgrounds & Visual Details**: Create atmosphere and depth rather than defaulting to solid colors. Add contextual effects and textures that match the overall aesthetic. Apply creative forms like gradient meshes, noise textures, geometric patterns, layered transparencies, dramatic shadows, decorative borders, custom cursors, and grain overlays.\n\nNEVER use generic AI-generated aesthetics like overused font families (Inter, Roboto, Arial, system fonts), cliched color schemes (particularly purple gradients on white backgrounds), predictable layouts and component patterns, and cookie-cutter design that lacks context-specific character.\n\nInterpret creatively and make unexpected choices that feel genuinely designed for the context. No design should be the same. Vary between light and dark themes, different fonts, different aesthetics. NEVER converge on common choices (Space Grotesk, for example) across generations.\n\n**IMPORTANT**: Match implementation complexity to the aesthetic vision. Maximalist designs need elaborate code with extensive animations and effects. Minimalist or refined designs need restraint, precision, and careful attention to spacing, typography, and subtle details. Elegance comes from executing the vision well.\n\nRemember: Claude is capable of extraordinary creative work. Don't hold back, show what can truly be created when thinking outside the box and committing fully to a distinctive vision.","frontmatter":{"name":"frontend-design","description":"Create distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, or applications. Generates creative, polished code that avoids generic AI aesthetics.","license":"Complete terms in LICENSE.txt"},"isInternal":false,"tokens":823,"sizeBytes":4274},{"name":"metadata.json","path":"flock-data/skills/karpathy/metadata.json","rawUrl":"https://raw.githubusercontent.com/Onelevenvy/flock/HEAD/flock-data/skills/karpathy/metadata.json","title":"karpathy","category":"anthropic-skill","format":"json","content":"{\n \"name\": \"karpathy\",\n \"description\": \"Coding principles for simplicity, clarity, and surgical precision. Use when you want Claude to think before coding, keep solutions minimal, make surgical changes, and define verifiable success criteria. Triggers on: karpathy mode, think before coding, simplicity first, surgical changes.\",\n \"category\": \"devops\",\n \"canonical_category\": \"devops\",\n \"repository\": \"majiayu000/claude-skill-registry\",\n \"repository_url\": \"https://github.com/majiayu000/claude-skill-registry\",\n \"author\": \"majiayu000\",\n \"author_avatar\": \"https://github.com/majiayu000.png\",\n \"file_path\": \"skills/other/karpathy/SKILL.md\",\n \"source\": \"github_code_search\",\n \"stars\": 0,\n \"quality_score\": 45,\n \"best_practices_score\": 55,\n \"skill_level\": 2,\n \"skill_level_label\": \"instructions\",\n \"has_scripts\": false,\n \"has_extra_files\": true,\n \"downloads\": 0,\n \"content_hash\": \"ec43cf3b4ae428984a1ad36d4814bf6f95e590ada0c997fab9f1ecb4c89b2886\",\n \"indexed_at\": \"2026-02-25T19:40:17.443Z\",\n \"synced_at\": \"2026-02-28T04:09:42.439Z\",\n \"omni_registry_url\": \"https://omni-skill-registry.omniroute.online/#/skill/27fec53067f2a7ff89a65128ed8a407f2e5f7a042d2891790054748048e9f33b\",\n \"install_command\": \"mkdir -p .claude/skills/karpathy && curl -sL \\\"https://raw.githubusercontent.com/majiayu000/claude-skill-registry/main/skills/other/karpathy/SKILL.md\\\" > .claude/skills/karpathy/SKILL.md\",\n \"raw_url\": \"https://raw.githubusercontent.com/majiayu000/claude-skill-registry/main/skills/other/karpathy/SKILL.md\"\n}\n","frontmatter":{"name":"karpathy","description":"Coding principles for simplicity, clarity, and surgical precision. Use when you want Claude to think before coding, keep solutions minimal, make surgical changes, and define verifiable success criteria. Triggers on: karpathy mode, think before coding, simplicity first, surgical changes.","category":"devops","canonical_category":"devops","repository":"majiayu000/claude-skill-registry","repository_url":"https://github.com/majiayu000/claude-skill-registry","author":"majiayu000","author_avatar":"https://github.com/majiayu000.png","file_path":"skills/other/karpathy/SKILL.md","source":"github_code_search","stars":0,"quality_score":45,"best_practices_score":55,"skill_level":2,"skill_level_label":"instructions","has_scripts":false,"has_extra_files":true,"downloads":0,"content_hash":"ec43cf3b4ae428984a1ad36d4814bf6f95e590ada0c997fab9f1ecb4c89b2886","indexed_at":"2026-02-25T19:40:17.443Z","synced_at":"2026-02-28T04:09:42.439Z","omni_registry_url":"https://omni-skill-registry.omniroute.online/#/skill/27fec53067f2a7ff89a65128ed8a407f2e5f7a042d2891790054748048e9f33b","install_command":"mkdir -p .claude/skills/karpathy && curl -sL \"https://raw.githubusercontent.com/majiayu000/claude-skill-registry/main/skills/other/karpathy/SKILL.md\" > .claude/skills/karpathy/SKILL.md","raw_url":"https://raw.githubusercontent.com/majiayu000/claude-skill-registry/main/skills/other/karpathy/SKILL.md"},"isInternal":false,"tokens":508,"sizeBytes":1525},{"name":"SKILL.md","path":"flock-data/skills/karpathy/SKILL.md","rawUrl":"https://raw.githubusercontent.com/Onelevenvy/flock/HEAD/flock-data/skills/karpathy/SKILL.md","title":"karpathy","category":"anthropic-skill","format":"markdown","content":"---\nname: karpathy\ndescription: 'Coding principles for simplicity, clarity, and surgical precision. Use when you want Claude to think before coding, keep solutions minimal, make surgical changes, and define verifiable success criteria. Triggers on: karpathy mode, think before coding, simplicity first, surgical changes.'\n---\n\n# Karpathy Coding Principles\n\nPrinciples for writing clean, minimal, and purposeful code.\n\n---\n\n## 1. Think Before Coding\n\n**Don't assume. Don't hide confusion. Surface tradeoffs.**\n\nBefore implementing:\n\n- State your assumptions explicitly. If uncertain, ask.\n- If multiple interpretations exist, present them - don't pick silently.\n- If a simpler approach exists, say so. Push back when warranted.\n- If something is unclear, stop. Name what's confusing. Ask.\n\n---\n\n## 2. Simplicity First\n\n**Minimum code that solves the problem. Nothing speculative.**\n\n- No features beyond what was asked.\n- No abstractions for single-use code.\n- No \"flexibility\" or \"configurability\" that wasn't requested.\n- No error handling for impossible scenarios.\n- If you write 200 lines and it could be 50, rewrite it.\n\nAsk yourself: \"Would a senior engineer say this is overcomplicated?\" If yes, simplify.\n\n---\n\n## 3. Surgical Changes\n\n**Touch only what you must. Clean up only your own mess.**\n\nWhen editing existing code:\n\n- Don't \"improve\" adjacent code, comments, or formatting.\n- Don't refactor things that aren't broken.\n- Match existing style, even if you'd do it differently.\n- If you notice unrelated dead code, mention it - don't delete it.\n\nWhen your changes create orphans:\n\n- Remove imports/variables/functions that YOUR changes made unused.\n- Don't remove pre-existing dead code unless asked.\n\nThe test: Every changed line should trace directly to the user's request.\n\n---\n\n## 4. Goal-Driven Execution\n\n**Define success criteria. Loop until verified.**\n\nTransform tasks into verifiable goals:\n\n- \"Add validation\" -> \"Write tests for invalid inputs, then make them pass\"\n- \"Fix the bug\" -> \"Write a test that reproduces it, then make it pass\"\n- \"Refactor X\" -> \"Ensure tests pass before and after\"\n\nFor multi-step tasks, state a brief plan:\n\n```\n1. [Step] -> verify: [check]\n2. [Step] -> verify: [check]\n3. [Step] -> verify: [check]\n```\n\nStrong success criteria let you loop independently. Weak criteria (\"make it work\") require constant clarification.\n","frontmatter":{"name":"karpathy","description":"Coding principles for simplicity, clarity, and surgical precision. Use when you want Claude to think before coding, keep solutions minimal, make surgical changes, and define verifiable success criteria. Triggers on: karpathy mode, think before coding, simplicity first, surgical changes."},"isInternal":false,"tokens":544,"sizeBytes":2372},{"name":"examples.md","path":"flock-data/skills/planning-with-files/examples.md","rawUrl":"https://raw.githubusercontent.com/Onelevenvy/flock/HEAD/flock-data/skills/planning-with-files/examples.md","title":"Planning-with-files Skill","category":"anthropic-skill","format":"markdown","content":"# Examples: Planning with Files in Action\n\n## Example 1: Research Task\n\n**User Request:** \"Research the benefits of morning exercise and write a summary\"\n\n### Loop 1: Create Plan\n```bash\nWrite task_plan.md\n```\n\n```markdown\n# Task Plan: Morning Exercise Benefits Research\n\n## Goal\nCreate a research summary on the benefits of morning exercise.\n\n## Phases\n- [ ] Phase 1: Create this plan ✓\n- [ ] Phase 2: Search and gather sources\n- [ ] Phase 3: Synthesize findings\n- [ ] Phase 4: Deliver summary\n\n## Key Questions\n1. What are the physical health benefits?\n2. What are the mental health benefits?\n3. What scientific studies support this?\n\n## Status\n**Currently in Phase 1** - Creating plan\n```\n\n### Loop 2: Research\n```bash\nRead task_plan.md # Refresh goals\nWebSearch \"morning exercise benefits\"\nWrite notes.md # Store findings\nEdit task_plan.md # Mark Phase 2 complete\n```\n\n### Loop 3: Synthesize\n```bash\nRead task_plan.md # Refresh goals\nRead notes.md # Get findings\nWrite morning_exercise_summary.md\nEdit task_plan.md # Mark Phase 3 complete\n```\n\n### Loop 4: Deliver\n```bash\nRead task_plan.md # Verify complete\nDeliver morning_exercise_summary.md\n```\n\n---\n\n## Example 2: Bug Fix Task\n\n**User Request:** \"Fix the login bug in the authentication module\"\n\n### task_plan.md\n```markdown\n# Task Plan: Fix Login Bug\n\n## Goal\nIdentify and fix the bug preventing successful login.\n\n## Phases\n- [x] Phase 1: Understand the bug report ✓\n- [x] Phase 2: Locate relevant code ✓\n- [ ] Phase 3: Identify root cause (CURRENT)\n- [ ] Phase 4: Implement fix\n- [ ] Phase 5: Test and verify\n\n## Key Questions\n1. What error message appears?\n2. Which file handles authentication?\n3. What changed recently?\n\n## Decisions Made\n- Auth handler is in src/auth/login.ts\n- Error occurs in validateToken() function\n\n## Errors Encountered\n- [Initial] TypeError: Cannot read property 'token' of undefined\n → Root cause: user object not awaited properly\n\n## Status\n**Currently in Phase 3** - Found root cause, preparing fix\n```\n\n---\n\n## Example 3: Feature Development\n\n**User Request:** \"Add a dark mode toggle to the settings page\"\n\n### The 3-File Pattern in Action\n\n**task_plan.md:**\n```markdown\n# Task Plan: Dark Mode Toggle\n\n## Goal\nAdd functional dark mode toggle to settings.\n\n## Phases\n- [x] Phase 1: Research existing theme system ✓\n- [x] Phase 2: Design implementation approach ✓\n- [ ] Phase 3: Implement toggle component (CURRENT)\n- [ ] Phase 4: Add theme switching logic\n- [ ] Phase 5: Test and polish\n\n## Decisions Made\n- Using CSS custom properties for theme\n- Storing preference in localStorage\n- Toggle component in SettingsPage.tsx\n\n## Status\n**Currently in Phase 3** - Building toggle component\n```\n\n**notes.md:**\n```markdown\n# Notes: Dark Mode Implementation\n\n## Existing Theme System\n- Located in: src/styles/theme.ts\n- Uses: CSS custom properties\n- Current themes: light only\n\n## Files to Modify\n1. src/styles/theme.ts - Add dark theme colors\n2. src/components/SettingsPage.tsx - Add toggle\n3. src/hooks/useTheme.ts - Create new hook\n4. src/App.tsx - Wrap with ThemeProvider\n\n## Color Decisions\n- Dark background: #1a1a2e\n- Dark surface: #16213e\n- Dark text: #eaeaea\n```\n\n**dark_mode_implementation.md:** (deliverable)\n```markdown\n# Dark Mode Implementation\n\n## Changes Made\n\n### 1. Added dark theme colors\nFile: src/styles/theme.ts\n...\n\n### 2. Created useTheme hook\nFile: src/hooks/useTheme.ts\n...\n```\n\n---\n\n## Example 4: Error Recovery Pattern\n\nWhen something fails, DON'T hide it:\n\n### Before (Wrong)\n```\nAction: Read config.json\nError: File not found\nAction: Read config.json # Silent retry\nAction: Read config.json # Another retry\n```\n\n### After (Correct)\n```\nAction: Read config.json\nError: File not found\n\n# Update task_plan.md:\n## Errors Encountered\n- config.json not found → Will create default config\n\nAction: Write config.json (default config)\nAction: Read config.json\nSuccess!\n```\n\n---\n\n## The Read-Before-Decide Pattern\n\n**Always read your plan before major decisions:**\n\n```\n[Many tool calls have happened...]\n[Context is getting long...]\n[Original goal might be forgotten...]\n\n→ Read task_plan.md # This brings goals back into attention!\n→ Now make the decision # Goals are fresh in context\n```\n\nThis is why Manus can handle ~50 tool calls without losing track. The plan file acts as a \"goal refresh\" mechanism.\n","isInternal":false,"tokens":1106,"sizeBytes":4426},{"name":"reference.md","path":"flock-data/skills/planning-with-files/reference.md","rawUrl":"https://raw.githubusercontent.com/Onelevenvy/flock/HEAD/flock-data/skills/planning-with-files/reference.md","title":"Planning-with-files Skill","category":"anthropic-skill","format":"markdown","content":"# Reference: Manus Context Engineering Principles\n\nThis skill is based on context engineering principles from Manus, the AI agent company acquired by Meta for $2 billion in December 2025.\n\n## The 6 Manus Principles\n\n### Principle 1: Design Around KV-Cache\n\n> \"KV-cache hit rate is THE single most important metric for production AI agents.\"\n\n**Statistics:**\n- ~100:1 input-to-output token ratio\n- Cached tokens: $0.30/MTok vs Uncached: $3/MTok\n- 10x cost difference!\n\n**Implementation:**\n- Keep prompt prefixes STABLE (single-token change invalidates cache)\n- NO timestamps in system prompts\n- Make context APPEND-ONLY with deterministic serialization\n\n### Principle 2: Mask, Don't Remove\n\nDon't dynamically remove tools (breaks KV-cache). Use logit masking instead.\n\n**Best Practice:** Use consistent action prefixes (e.g., `browser_`, `shell_`, `file_`) for easier masking.\n\n### Principle 3: Filesystem as External Memory\n\n> \"Markdown is my 'working memory' on disk.\"\n\n**The Formula:**\n```\nContext Window = RAM (volatile, limited)\nFilesystem = Disk (persistent, unlimited)\n```\n\n**Compression Must Be Restorable:**\n- Keep URLs even if web content is dropped\n- Keep file paths when dropping document contents\n- Never lose the pointer to full data\n\n### Principle 4: Manipulate Attention Through Recitation\n\n> \"Creates and updates todo.md throughout tasks to push global plan into model's recent attention span.\"\n\n**Problem:** After ~50 tool calls, models forget original goals (\"lost in the middle\" effect).\n\n**Solution:** Re-read `task_plan.md` before each decision. Goals appear in the attention window.\n\n```\nStart of context: [Original goal - far away, forgotten]\n...many tool calls...\nEnd of context: [Recently read task_plan.md - gets ATTENTION!]\n```\n\n### Principle 5: Keep the Wrong Stuff In\n\n> \"Leave the wrong turns in the context.\"\n\n**Why:**\n- Failed actions with stack traces let model implicitly update beliefs\n- Reduces mistake repetition\n- Error recovery is \"one of the clearest signals of TRUE agentic behavior\"\n\n### Principle 6: Don't Get Few-Shotted\n\n> \"Uniformity breeds fragility.\"\n\n**Problem:** Repetitive action-observation pairs cause drift and hallucination.\n\n**Solution:** Introduce controlled variation:\n- Vary phrasings slightly\n- Don't copy-paste patterns blindly\n- Recalibrate on repetitive tasks\n\n---\n\n## The 3 Context Engineering Strategies\n\nBased on Lance Martin's analysis of Manus architecture.\n\n### Strategy 1: Context Reduction\n\n**Compaction:**\n```\nTool calls have TWO representations:\n├── FULL: Raw tool content (stored in filesystem)\n└── COMPACT: Reference/file path only\n\nRULES:\n- Apply compaction to STALE (older) tool results\n- Keep RECENT results FULL (to guide next decision)\n```\n\n**Summarization:**\n- Applied when compaction reaches diminishing returns\n- Generated using full tool results\n- Creates standardized summary objects\n\n### Strategy 2: Context Isolation (Multi-Agent)\n\n**Architecture:**\n```\n┌─────────────────────────────────┐\n│ PLANNER AGENT │\n│ └─ Assigns tasks to sub-agents │\n├─────────────────────────────────┤\n│ KNOWLEDGE MANAGER │\n│ └─ Reviews conversations │\n│ └─ Determines filesystem store │\n├─────────────────────────────────┤\n│ EXECUTOR SUB-AGENTS │\n│ └─ Perform assigned tasks │\n│ └─ Have own context windows │\n└─────────────────────────────────┘\n```\n\n**Key Insight:** Manus originally used `todo.md` for task planning but found ~33% of actions were spent updating it. Shifted to dedicated planner agent calling executor sub-agents.\n\n### Strategy 3: Context Offloading\n\n**Tool Design:**\n- Use <20 atomic functions total\n- Store full results in filesystem, not context\n- Use `glob` and `grep` for searching\n- Progressive disclosure: load information only as needed\n\n---\n\n## The Agent Loop\n\nManus operates in a continuous 7-step loop:\n\n```\n┌─────────────────────────────────────────┐\n│ 1. ANALYZE CONTEXT │\n│ - Understand user intent │\n│ - Assess current state │\n│ - Review recent observations │\n├─────────────────────────────────────────┤\n│ 2. THINK │\n│ - Should I update the plan? │\n│ - What's the next logical action? │\n│ - Are there blockers? │\n├─────────────────────────────────────────┤\n│ 3. SELECT TOOL │\n│ - Choose ONE tool │\n│ - Ensure parameters available │\n├─────────────────────────────────────────┤\n│ 4. EXECUTE ACTION │\n│ - Tool runs in sandbox │\n├─────────────────────────────────────────┤\n│ 5. RECEIVE OBSERVATION │\n│ - Result appended to context │\n├─────────────────────────────────────────┤\n│ 6. ITERATE │\n│ - Return to step 1 │\n│ - Continue until complete │\n├─────────────────────────────────────────┤\n│ 7. DELIVER OUTCOME │\n│ - Send results to user │\n│ - Attach all relevant files │\n└─────────────────────────────────────────┘\n```\n\n---\n\n## File Types Manus Creates\n\n| File | Purpose | When Created | When Updated |\n|------|---------|--------------|--------------|\n| `task_plan.md` | Phase tracking, progress | Task start | After completing phases |\n| `findings.md` | Discoveries, decisions | After ANY discovery | After viewing images/PDFs |\n| `progress.md` | Session log, what's done | At breakpoints | Throughout session |\n| Code files | Implementation | Before execution | After errors |\n\n---\n\n## Critical Constraints\n\n- **Single-Action Execution:** ONE tool call per turn. No parallel execution.\n- **Plan is Required:** Agent must ALWAYS know: goal, current phase, remaining phases\n- **Files are Memory:** Context = volatile. Filesystem = persistent.\n- **Never Repeat Failures:** If action failed, next action MUST be different\n- **Communication is a Tool:** Message types: `info` (progress), `ask` (blocking), `result` (terminal)\n\n---\n\n## Manus Statistics\n\n| Metric | Value |\n|--------|-------|\n| Average tool calls per task | ~50 |\n| Input-to-output token ratio | 100:1 |\n| Acquisition price | $2 billion |\n| Time to $100M revenue | 8 months |\n| Framework refactors since launch | 5 times |\n\n---\n\n## Key Quotes\n\n> \"Context window = RAM (volatile, limited). Filesystem = Disk (persistent, unlimited). Anything important gets written to disk.\"\n\n> \"if action_failed: next_action != same_action. Track what you tried. Mutate the approach.\"\n\n> \"Error recovery is one of the clearest signals of TRUE agentic behavior.\"\n\n> \"KV-cache hit rate is the single most important metric for a production-stage AI agent.\"\n\n> \"Leave the wrong turns in the context.\"\n\n---\n\n## Source\n\nBased on Manus's official context engineering documentation:\nhttps://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus\n","isInternal":false,"tokens":1627,"sizeBytes":8066},{"name":"SKILL.md","path":"flock-data/skills/planning-with-files/SKILL.md","rawUrl":"https://raw.githubusercontent.com/Onelevenvy/flock/HEAD/flock-data/skills/planning-with-files/SKILL.md","title":"planning-with-files","category":"anthropic-skill","format":"markdown","content":"---\nname: planning-with-files\nversion: \"2.10.0\"\ndescription: Implements Manus-style file-based planning for complex tasks. Creates task_plan.md, findings.md, and progress.md. Use when starting complex multi-step tasks, research projects, or any task requiring >5 tool calls. Now with automatic session recovery after /clear.\nuser-invocable: true\nallowed-tools:\n - Read\n - Write\n - Edit\n - Bash\n - Glob\n - Grep\n - WebFetch\n - WebSearch\nhooks:\n PreToolUse:\n - matcher: \"Write|Edit|Bash|Read|Glob|Grep\"\n hooks:\n - type: command\n command: \"cat task_plan.md 2>/dev/null | head -30 || true\"\n PostToolUse:\n - matcher: \"Write|Edit\"\n hooks:\n - type: command\n command: \"echo '[planning-with-files] File updated. If this completes a phase, update task_plan.md status.'\"\n Stop:\n - hooks:\n - type: command\n command: |\n SCRIPT_DIR=\"${CLAUDE_PLUGIN_ROOT:-$HOME/.claude/plugins/planning-with-files}/scripts\"\n\n IS_WINDOWS=0\n if [ \"${OS-}\" = \"Windows_NT\" ]; then\n IS_WINDOWS=1\n else\n UNAME_S=\"$(uname -s 2>/dev/null || echo '')\"\n case \"$UNAME_S\" in\n CYGWIN*|MINGW*|MSYS*) IS_WINDOWS=1 ;;\n esac\n fi\n\n if [ \"$IS_WINDOWS\" -eq 1 ]; then\n if command -v pwsh >/dev/null 2>&1; then\n pwsh -ExecutionPolicy Bypass -File \"$SCRIPT_DIR/check-complete.ps1\" 2>/dev/null ||\n powershell -ExecutionPolicy Bypass -File \"$SCRIPT_DIR/check-complete.ps1\" 2>/dev/null ||\n sh \"$SCRIPT_DIR/check-complete.sh\"\n else\n powershell -ExecutionPolicy Bypass -File \"$SCRIPT_DIR/check-complete.ps1\" 2>/dev/null ||\n sh \"$SCRIPT_DIR/check-complete.sh\"\n fi\n else\n sh \"$SCRIPT_DIR/check-complete.sh\"\n fi\n---\n\n# Planning with Files\n\nWork like Manus: Use persistent markdown files as your \"working memory on disk.\"\n\n## FIRST: Check for Previous Session (v2.2.0)\n\n**Before starting work**, check for unsynced context from a previous session:\n\n```bash\n# Linux/macOS\n$(command -v python3 || command -v python) ${CLAUDE_PLUGIN_ROOT}/scripts/session-catchup.py \"$(pwd)\"\n```\n\n```powershell\n# Windows PowerShell\n& (Get-Command python -ErrorAction SilentlyContinue).Source \"$env:USERPROFILE\\.claude\\skills\\planning-with-files\\scripts\\session-catchup.py\" (Get-Location)\n```\n\nIf catchup report shows unsynced context:\n1. Run `git diff --stat` to see actual code changes\n2. Read current planning files\n3. Update planning files based on catchup + git diff\n4. Then proceed with task\n\n## Important: Where Files Go\n\n- **Templates** are in `${CLAUDE_PLUGIN_ROOT}/templates/`\n- **Your planning files** go in **your project directory**\n\n| Location | What Goes There |\n|----------|-----------------|\n| Skill directory (`${CLAUDE_PLUGIN_ROOT}/`) | Templates, scripts, reference docs |\n| Your project directory | `task_plan.md`, `findings.md`, `progress.md` |\n\n## Quick Start\n\nBefore ANY complex task:\n\n1. **Create `task_plan.md`** — Use [templates/task_plan.md](templates/task_plan.md) as reference\n2. **Create `findings.md`** — Use [templates/findings.md](templates/findings.md) as reference\n3. **Create `progress.md`** — Use [templates/progress.md](templates/progress.md) as reference\n4. **Re-read plan before decisions** — Refreshes goals in attention window\n5. **Update after each phase** — Mark complete, log errors\n\n> **Note:** Planning files go in your project root, not the skill installation folder.\n\n## The Core Pattern\n\n```\nContext Window = RAM (volatile, limited)\nFilesystem = Disk (persistent, unlimited)\n\n→ Anything important gets written to disk.\n```\n\n## File Purposes\n\n| File | Purpose | When to Update |\n|------|---------|----------------|\n| `task_plan.md` | Phases, progress, decisions | After each phase |\n| `findings.md` | Research, discoveries | After ANY discovery |\n| `progress.md` | Session log, test results | Throughout session |\n\n## Critical Rules\n\n### 1. Create Plan First\nNever start a complex task without `task_plan.md`. Non-negotiable.\n\n### 2. The 2-Action Rule\n> \"After every 2 view/browser/search operations, IMMEDIATELY save key findings to text files.\"\n\nThis prevents visual/multimodal information from being lost.\n\n### 3. Read Before Decide\nBefore major decisions, read the plan file. This keeps goals in your attention window.\n\n### 4. Update After Act\nAfter completing any phase:\n- Mark phase status: `in_progress` → `complete`\n- Log any errors encountered\n- Note files created/modified\n\n### 5. Log ALL Errors\nEvery error goes in the plan file. This builds knowledge and prevents repetition.\n\n```markdown\n## Errors Encountered\n| Error | Attempt | Resolution |\n|-------|---------|------------|\n| FileNotFoundError | 1 | Created default config |\n| API timeout | 2 | Added retry logic |\n```\n\n### 6. Never Repeat Failures\n```\nif action_failed:\n next_action != same_action\n```\nTrack what you tried. Mutate the approach.\n\n## The 3-Strike Error Protocol\n\n```\nATTEMPT 1: Diagnose & Fix\n → Read error carefully\n → Identify root cause\n → Apply targeted fix\n\nATTEMPT 2: Alternative Approach\n → Same error? Try different method\n → Different tool? Different library?\n → NEVER repeat exact same failing action\n\nATTEMPT 3: Broader Rethink\n → Question assumptions\n → Search for solutions\n → Consider updating the plan\n\nAFTER 3 FAILURES: Escalate to User\n → Explain what you tried\n → Share the specific error\n → Ask for guidance\n```\n\n## Read vs Write Decision Matrix\n\n| Situation | Action | Reason |\n|-----------|--------|--------|\n| Just wrote a file | DON'T read | Content still in context |\n| Viewed image/PDF | Write findings NOW | Multimodal → text before lost |\n| Browser returned data | Write to file | Screenshots don't persist |\n| Starting new phase | Read plan/findings | Re-orient if context stale |\n| Error occurred | Read relevant file | Need current state to fix |\n| Resuming after gap | Read all planning files | Recover state |\n\n## The 5-Question Reboot Test\n\nIf you can answer these, your context management is solid:\n\n| Question | Answer Source |\n|----------|---------------|\n| Where am I? | Current phase in task_plan.md |\n| Where am I going? | Remaining phases |\n| What's the goal? | Goal statement in plan |\n| What have I learned? | findings.md |\n| What have I done? | progress.md |\n\n## When to Use This Pattern\n\n**Use for:**\n- Multi-step tasks (3+ steps)\n- Research tasks\n- Building/creating projects\n- Tasks spanning many tool calls\n- Anything requiring organization\n\n**Skip for:**\n- Simple questions\n- Single-file edits\n- Quick lookups\n\n## Templates\n\nCopy these templates to start:\n\n- [templates/task_plan.md](templates/task_plan.md) — Phase tracking\n- [templates/findings.md](templates/findings.md) — Research storage\n- [templates/progress.md](templates/progress.md) — Session logging\n\n## Scripts\n\nHelper scripts for automation:\n\n- `scripts/init-session.sh` — Initialize all planning files\n- `scripts/check-complete.sh` — Verify all phases complete\n- `scripts/session-catchup.py` — Recover context from previous session (v2.2.0)\n\n## Advanced Topics\n\n- **Manus Principles:** See [reference.md](reference.md)\n- **Real Examples:** See [examples.md](examples.md)\n\n## Anti-Patterns\n\n| Don't | Do Instead |\n|-------|------------|\n| Use TodoWrite for persistence | Create task_plan.md file |\n| State goals once and forget | Re-read plan before decisions |\n| Hide errors and retry silently | Log errors to plan file |\n| Stuff everything in context | Store large content in files |\n| Start executing immediately | Create plan file FIRST |\n| Repeat failed actions | Track attempts, mutate approach |\n| Create files in skill directory | Create files in your project |\n","frontmatter":{"name":"planning-with-files","version":"2.10.0","description":"Implements Manus-style file-based planning for complex tasks. Creates task_plan.md, findings.md, and progress.md. Use when starting complex multi-step tasks, research projects, or any task requiring >5 tool calls. Now with automatic session recovery after /clear.","user-invocable":true,"allowed-tools":["Read","Write","Edit","Bash","Glob","Grep","WebFetch","WebSearch"],"hooks":{"PreToolUse":[{"matcher":"Write|Edit|Bash|Read|Glob|Grep","hooks":[{"type":"command","command":"cat task_plan.md 2>/dev/null | head -30 || true"}]}],"PostToolUse":[{"matcher":"Write|Edit","hooks":[{"type":"command","command":"echo '[planning-with-files] File updated. If this completes a phase, update task_plan.md status.'"}]}],"Stop":[{"hooks":[{"type":"command","command":"SCRIPT_DIR=\"${CLAUDE_PLUGIN_ROOT:-$HOME/.claude/plugins/planning-with-files}/scripts\"\n\nIS_WINDOWS=0\nif [ \"${OS-}\" = \"Windows_NT\" ]; then\n IS_WINDOWS=1\nelse\n UNAME_S=\"$(uname -s 2>/dev/null || echo '')\"\n case \"$UNAME_S\" in\n CYGWIN*|MINGW*|MSYS*) IS_WINDOWS=1 ;;\n esac\nfi\n\nif [ \"$IS_WINDOWS\" -eq 1 ]; then\n if command -v pwsh >/dev/null 2>&1; then\n pwsh -ExecutionPolicy Bypass -File \"$SCRIPT_DIR/check-complete.ps1\" 2>/dev/null ||\n powershell -ExecutionPolicy Bypass -File \"$SCRIPT_DIR/check-complete.ps1\" 2>/dev/null ||\n sh \"$SCRIPT_DIR/check-complete.sh\"\n else\n powershell -ExecutionPolicy Bypass -File \"$SCRIPT_DIR/check-complete.ps1\" 2>/dev/null ||\n sh \"$SCRIPT_DIR/check-complete.sh\"\n fi\nelse\n sh \"$SCRIPT_DIR/check-complete.sh\"\nfi\n"}]}]},"allowedTools":["Read","Write","Edit","Bash","Glob","Grep","WebFetch","WebSearch"]},"allowedTools":["Read","Write","Edit","Bash","Glob","Grep","WebFetch","WebSearch"],"isInternal":false,"tokens":1900,"sizeBytes":7853},{"name":"findings.md","path":"flock-data/skills/planning-with-files/templates/findings.md","rawUrl":"https://raw.githubusercontent.com/Onelevenvy/flock/HEAD/flock-data/skills/planning-with-files/templates/findings.md","title":"Templates Skill","category":"anthropic-skill","format":"markdown","content":"# Findings & Decisions\n\n\n## Requirements\n\n\n-\n\n## Research Findings\n\n\n-\n\n## Technical Decisions\n\n\n| Decision | Rationale |\n|----------|-----------|\n| | |\n\n## Issues Encountered\n\n\n| Issue | Resolution |\n|-------|------------|\n| | |\n\n## Resources\n\n\n-\n\n## Visual/Browser Findings\n\n\n\n-\n\n---\n\n*Update this file after every 2 view/browser/search operations*\n*This prevents visual information from being lost*\n","isInternal":false,"tokens":776,"sizeBytes":3561},{"name":"progress.md","path":"flock-data/skills/planning-with-files/templates/progress.md","rawUrl":"https://raw.githubusercontent.com/Onelevenvy/flock/HEAD/flock-data/skills/planning-with-files/templates/progress.md","title":"Templates Skill","category":"anthropic-skill","format":"markdown","content":"# Progress Log\n\n\n## Session: [DATE]\n\n\n### Phase 1: [Title]\n\n- **Status:** in_progress\n- **Started:** [timestamp]\n\n- Actions taken:\n \n -\n- Files created/modified:\n \n -\n\n### Phase 2: [Title]\n\n- **Status:** pending\n- Actions taken:\n -\n- Files created/modified:\n -\n\n## Test Results\n\n| Test | Input | Expected | Actual | Status |\n|------|-------|----------|--------|--------|\n| | | | | |\n\n## Error Log\n\n\n| Timestamp | Error | Attempt | Resolution |\n|-----------|-------|---------|------------|\n| | | 1 | |\n\n## 5-Question Reboot Check\n\n\n| Question | Answer |\n|----------|--------|\n| Where am I? | Phase X |\n| Where am I going? | Remaining phases |\n| What's the goal? | [goal statement] |\n| What have I learned? | See findings.md |\n| What have I done? | See above |\n\n---\n\n*Update after completing each phase or encountering errors*\n","isInternal":false,"tokens":987,"sizeBytes":4001},{"name":"task_plan.md","path":"flock-data/skills/planning-with-files/templates/task_plan.md","rawUrl":"https://raw.githubusercontent.com/Onelevenvy/flock/HEAD/flock-data/skills/planning-with-files/templates/task_plan.md","title":"Templates Skill","category":"anthropic-skill","format":"markdown","content":"# Task Plan: [Brief Description]\n\n\n## Goal\n\n[One sentence describing the end state]\n\n## Current Phase\n\nPhase 1\n\n## Phases\n\n\n### Phase 1: Requirements & Discovery\n\n- [ ] Understand user intent\n- [ ] Identify constraints and requirements\n- [ ] Document findings in findings.md\n- **Status:** in_progress\n\n\n### Phase 2: Planning & Structure\n\n- [ ] Define technical approach\n- [ ] Create project structure if needed\n- [ ] Document decisions with rationale\n- **Status:** pending\n\n### Phase 3: Implementation\n\n- [ ] Execute the plan step by step\n- [ ] Write code to files before executing\n- [ ] Test incrementally\n- **Status:** pending\n\n### Phase 4: Testing & Verification\n\n- [ ] Verify all requirements met\n- [ ] Document test results in progress.md\n- [ ] Fix any issues found\n- **Status:** pending\n\n### Phase 5: Delivery\n\n- [ ] Review all output files\n- [ ] Ensure deliverables are complete\n- [ ] Deliver to user\n- **Status:** pending\n\n## Key Questions\n\n1. [Question to answer]\n2. [Question to answer]\n\n## Decisions Made\n\n| Decision | Rationale |\n|----------|-----------|\n| | |\n\n## Errors Encountered\n\n| Error | Attempt | Resolution |\n|-------|---------|------------|\n| | 1 | |\n\n## Notes\n\n- Update phase status as you progress: pending → in_progress → complete\n- Re-read this plan before major decisions (attention manipulation)\n- Log ALL errors - they help avoid repetition\n","isInternal":false,"tokens":1055,"sizeBytes":4613},{"name":"schemas.md","path":"flock-data/skills/skill-creator/references/schemas.md","rawUrl":"https://raw.githubusercontent.com/Onelevenvy/flock/HEAD/flock-data/skills/skill-creator/references/schemas.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# JSON Schemas\n\nThis document defines the JSON schemas used by skill-creator.\n\n---\n\n## evals.json\n\nDefines the evals for a skill. Located at `evals/evals.json` within the skill directory.\n\n```json\n{\n \"skill_name\": \"example-skill\",\n \"evals\": [\n {\n \"id\": 1,\n \"prompt\": \"User's example prompt\",\n \"expected_output\": \"Description of expected result\",\n \"files\": [\"evals/files/sample1.pdf\"],\n \"expectations\": [\n \"The output includes X\",\n \"The skill used script Y\"\n ]\n }\n ]\n}\n```\n\n**Fields:**\n- `skill_name`: Name matching the skill's frontmatter\n- `evals[].id`: Unique integer identifier\n- `evals[].prompt`: The task to execute\n- `evals[].expected_output`: Human-readable description of success\n- `evals[].files`: Optional list of input file paths (relative to skill root)\n- `evals[].expectations`: List of verifiable statements\n\n---\n\n## history.json\n\nTracks version progression in Improve mode. Located at workspace root.\n\n```json\n{\n \"started_at\": \"2026-01-15T10:30:00Z\",\n \"skill_name\": \"pdf\",\n \"current_best\": \"v2\",\n \"iterations\": [\n {\n \"version\": \"v0\",\n \"parent\": null,\n \"expectation_pass_rate\": 0.65,\n \"grading_result\": \"baseline\",\n \"is_current_best\": false\n },\n {\n \"version\": \"v1\",\n \"parent\": \"v0\",\n \"expectation_pass_rate\": 0.75,\n \"grading_result\": \"won\",\n \"is_current_best\": false\n },\n {\n \"version\": \"v2\",\n \"parent\": \"v1\",\n \"expectation_pass_rate\": 0.85,\n \"grading_result\": \"won\",\n \"is_current_best\": true\n }\n ]\n}\n```\n\n**Fields:**\n- `started_at`: ISO timestamp of when improvement started\n- `skill_name`: Name of the skill being improved\n- `current_best`: Version identifier of the best performer\n- `iterations[].version`: Version identifier (v0, v1, ...)\n- `iterations[].parent`: Parent version this was derived from\n- `iterations[].expectation_pass_rate`: Pass rate from grading\n- `iterations[].grading_result`: \"baseline\", \"won\", \"lost\", or \"tie\"\n- `iterations[].is_current_best`: Whether this is the current best version\n\n---\n\n## grading.json\n\nOutput from the grader agent. Located at `/grading.json`.\n\n```json\n{\n \"expectations\": [\n {\n \"text\": \"The output includes the name 'John Smith'\",\n \"passed\": true,\n \"evidence\": \"Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'\"\n },\n {\n \"text\": \"The spreadsheet has a SUM formula in cell B10\",\n \"passed\": false,\n \"evidence\": \"No spreadsheet was created. The output was a text file.\"\n }\n ],\n \"summary\": {\n \"passed\": 2,\n \"failed\": 1,\n \"total\": 3,\n \"pass_rate\": 0.67\n },\n \"execution_metrics\": {\n \"tool_calls\": {\n \"Read\": 5,\n \"Write\": 2,\n \"Bash\": 8\n },\n \"total_tool_calls\": 15,\n \"total_steps\": 6,\n \"errors_encountered\": 0,\n \"output_chars\": 12450,\n \"transcript_chars\": 3200\n },\n \"timing\": {\n \"executor_duration_seconds\": 165.0,\n \"grader_duration_seconds\": 26.0,\n \"total_duration_seconds\": 191.0\n },\n \"claims\": [\n {\n \"claim\": \"The form has 12 fillable fields\",\n \"type\": \"factual\",\n \"verified\": true,\n \"evidence\": \"Counted 12 fields in field_info.json\"\n }\n ],\n \"user_notes_summary\": {\n \"uncertainties\": [\"Used 2023 data, may be stale\"],\n \"needs_review\": [],\n \"workarounds\": [\"Fell back to text overlay for non-fillable fields\"]\n },\n \"eval_feedback\": {\n \"suggestions\": [\n {\n \"assertion\": \"The output includes the name 'John Smith'\",\n \"reason\": \"A hallucinated document that mentions the name would also pass\"\n }\n ],\n \"overall\": \"Assertions check presence but not correctness.\"\n }\n}\n```\n\n**Fields:**\n- `expectations[]`: Graded expectations with evidence\n- `summary`: Aggregate pass/fail counts\n- `execution_metrics`: Tool usage and output size (from executor's metrics.json)\n- `timing`: Wall clock timing (from timing.json)\n- `claims`: Extracted and verified claims from the output\n- `user_notes_summary`: Issues flagged by the executor\n- `eval_feedback`: (optional) Improvement suggestions for the evals, only present when the grader identifies issues worth raising\n\n---\n\n## metrics.json\n\nOutput from the executor agent. Located at `/outputs/metrics.json`.\n\n```json\n{\n \"tool_calls\": {\n \"Read\": 5,\n \"Write\": 2,\n \"Bash\": 8,\n \"Edit\": 1,\n \"Glob\": 2,\n \"Grep\": 0\n },\n \"total_tool_calls\": 18,\n \"total_steps\": 6,\n \"files_created\": [\"filled_form.pdf\", \"field_values.json\"],\n \"errors_encountered\": 0,\n \"output_chars\": 12450,\n \"transcript_chars\": 3200\n}\n```\n\n**Fields:**\n- `tool_calls`: Count per tool type\n- `total_tool_calls`: Sum of all tool calls\n- `total_steps`: Number of major execution steps\n- `files_created`: List of output files created\n- `errors_encountered`: Number of errors during execution\n- `output_chars`: Total character count of output files\n- `transcript_chars`: Character count of transcript\n\n---\n\n## timing.json\n\nWall clock timing for a run. Located at `/timing.json`.\n\n**How to capture:** When a subagent task completes, the task notification includes `total_tokens` and `duration_ms`. Save these immediately — they are not persisted anywhere else and cannot be recovered after the fact.\n\n```json\n{\n \"total_tokens\": 84852,\n \"duration_ms\": 23332,\n \"total_duration_seconds\": 23.3,\n \"executor_start\": \"2026-01-15T10:30:00Z\",\n \"executor_end\": \"2026-01-15T10:32:45Z\",\n \"executor_duration_seconds\": 165.0,\n \"grader_start\": \"2026-01-15T10:32:46Z\",\n \"grader_end\": \"2026-01-15T10:33:12Z\",\n \"grader_duration_seconds\": 26.0\n}\n```\n\n---\n\n## benchmark.json\n\nOutput from Benchmark mode. Located at `benchmarks//benchmark.json`.\n\n```json\n{\n \"metadata\": {\n \"skill_name\": \"pdf\",\n \"skill_path\": \"/path/to/pdf\",\n \"executor_model\": \"claude-sonnet-4-20250514\",\n \"analyzer_model\": \"most-capable-model\",\n \"timestamp\": \"2026-01-15T10:30:00Z\",\n \"evals_run\": [1, 2, 3],\n \"runs_per_configuration\": 3\n },\n\n \"runs\": [\n {\n \"eval_id\": 1,\n \"eval_name\": \"Ocean\",\n \"configuration\": \"with_skill\",\n \"run_number\": 1,\n \"result\": {\n \"pass_rate\": 0.85,\n \"passed\": 6,\n \"failed\": 1,\n \"total\": 7,\n \"time_seconds\": 42.5,\n \"tokens\": 3800,\n \"tool_calls\": 18,\n \"errors\": 0\n },\n \"expectations\": [\n {\"text\": \"...\", \"passed\": true, \"evidence\": \"...\"}\n ],\n \"notes\": [\n \"Used 2023 data, may be stale\",\n \"Fell back to text overlay for non-fillable fields\"\n ]\n }\n ],\n\n \"run_summary\": {\n \"with_skill\": {\n \"pass_rate\": {\"mean\": 0.85, \"stddev\": 0.05, \"min\": 0.80, \"max\": 0.90},\n \"time_seconds\": {\"mean\": 45.0, \"stddev\": 12.0, \"min\": 32.0, \"max\": 58.0},\n \"tokens\": {\"mean\": 3800, \"stddev\": 400, \"min\": 3200, \"max\": 4100}\n },\n \"without_skill\": {\n \"pass_rate\": {\"mean\": 0.35, \"stddev\": 0.08, \"min\": 0.28, \"max\": 0.45},\n \"time_seconds\": {\"mean\": 32.0, \"stddev\": 8.0, \"min\": 24.0, \"max\": 42.0},\n \"tokens\": {\"mean\": 2100, \"stddev\": 300, \"min\": 1800, \"max\": 2500}\n },\n \"delta\": {\n \"pass_rate\": \"+0.50\",\n \"time_seconds\": \"+13.0\",\n \"tokens\": \"+1700\"\n }\n },\n\n \"notes\": [\n \"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value\",\n \"Eval 3 shows high variance (50% ± 40%) - may be flaky or model-dependent\",\n \"Without-skill runs consistently fail on table extraction expectations\",\n \"Skill adds 13s average execution time but improves pass rate by 50%\"\n ]\n}\n```\n\n**Fields:**\n- `metadata`: Information about the benchmark run\n - `skill_name`: Name of the skill\n - `timestamp`: When the benchmark was run\n - `evals_run`: List of eval names or IDs\n - `runs_per_configuration`: Number of runs per config (e.g. 3)\n- `runs[]`: Individual run results\n - `eval_id`: Numeric eval identifier\n - `eval_name`: Human-readable eval name (used as section header in the viewer)\n - `configuration`: Must be `\"with_skill\"` or `\"without_skill\"` (the viewer uses this exact string for grouping and color coding)\n - `run_number`: Integer run number (1, 2, 3...)\n - `result`: Nested object with `pass_rate`, `passed`, `total`, `time_seconds`, `tokens`, `errors`\n- `run_summary`: Statistical aggregates per configuration\n - `with_skill` / `without_skill`: Each contains `pass_rate`, `time_seconds`, `tokens` objects with `mean` and `stddev` fields\n - `delta`: Difference strings like `\"+0.50\"`, `\"+13.0\"`, `\"+1700\"`\n- `notes`: Freeform observations from the analyzer\n\n**Important:** The viewer reads these field names exactly. Using `config` instead of `configuration`, or putting `pass_rate` at the top level of a run instead of nested under `result`, will cause the viewer to show empty/zero values. Always reference this schema when generating benchmark.json manually.\n\n---\n\n## comparison.json\n\nOutput from blind comparator. Located at `/comparison-N.json`.\n\n```json\n{\n \"winner\": \"A\",\n \"reasoning\": \"Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.\",\n \"rubric\": {\n \"A\": {\n \"content\": {\n \"correctness\": 5,\n \"completeness\": 5,\n \"accuracy\": 4\n },\n \"structure\": {\n \"organization\": 4,\n \"formatting\": 5,\n \"usability\": 4\n },\n \"content_score\": 4.7,\n \"structure_score\": 4.3,\n \"overall_score\": 9.0\n },\n \"B\": {\n \"content\": {\n \"correctness\": 3,\n \"completeness\": 2,\n \"accuracy\": 3\n },\n \"structure\": {\n \"organization\": 3,\n \"formatting\": 2,\n \"usability\": 3\n },\n \"content_score\": 2.7,\n \"structure_score\": 2.7,\n \"overall_score\": 5.4\n }\n },\n \"output_quality\": {\n \"A\": {\n \"score\": 9,\n \"strengths\": [\"Complete solution\", \"Well-formatted\", \"All fields present\"],\n \"weaknesses\": [\"Minor style inconsistency in header\"]\n },\n \"B\": {\n \"score\": 5,\n \"strengths\": [\"Readable output\", \"Correct basic structure\"],\n \"weaknesses\": [\"Missing date field\", \"Formatting inconsistencies\", \"Partial data extraction\"]\n }\n },\n \"expectation_results\": {\n \"A\": {\n \"passed\": 4,\n \"total\": 5,\n \"pass_rate\": 0.80,\n \"details\": [\n {\"text\": \"Output includes name\", \"passed\": true}\n ]\n },\n \"B\": {\n \"passed\": 3,\n \"total\": 5,\n \"pass_rate\": 0.60,\n \"details\": [\n {\"text\": \"Output includes name\", \"passed\": true}\n ]\n }\n }\n}\n```\n\n---\n\n## analysis.json\n\nOutput from post-hoc analyzer. Located at `/analysis.json`.\n\n```json\n{\n \"comparison_summary\": {\n \"winner\": \"A\",\n \"winner_skill\": \"path/to/winner/skill\",\n \"loser_skill\": \"path/to/loser/skill\",\n \"comparator_reasoning\": \"Brief summary of why comparator chose winner\"\n },\n \"winner_strengths\": [\n \"Clear step-by-step instructions for handling multi-page documents\",\n \"Included validation script that caught formatting errors\"\n ],\n \"loser_weaknesses\": [\n \"Vague instruction 'process the document appropriately' led to inconsistent behavior\",\n \"No script for validation, agent had to improvise\"\n ],\n \"instruction_following\": {\n \"winner\": {\n \"score\": 9,\n \"issues\": [\"Minor: skipped optional logging step\"]\n },\n \"loser\": {\n \"score\": 6,\n \"issues\": [\n \"Did not use the skill's formatting template\",\n \"Invented own approach instead of following step 3\"\n ]\n }\n },\n \"improvement_suggestions\": [\n {\n \"priority\": \"high\",\n \"category\": \"instructions\",\n \"suggestion\": \"Replace 'process the document appropriately' with explicit steps\",\n \"expected_impact\": \"Would eliminate ambiguity that caused inconsistent behavior\"\n }\n ],\n \"transcript_insights\": {\n \"winner_execution_pattern\": \"Read skill -> Followed 5-step process -> Used validation script\",\n \"loser_execution_pattern\": \"Read skill -> Unclear on approach -> Tried 3 different methods\"\n }\n}\n```\n","isInternal":false,"tokens":3373,"sizeBytes":12061},{"name":"SKILL.md","path":"flock-data/skills/skill-creator/SKILL.md","rawUrl":"https://raw.githubusercontent.com/Onelevenvy/flock/HEAD/flock-data/skills/skill-creator/SKILL.md","title":"skill-creator","category":"anthropic-skill","format":"markdown","content":"---\nname: skill-creator\ndescription: Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.\n---\n\n# Skill Creator\n\nA skill for creating new skills and iteratively improving them.\n\nAt a high level, the process of creating a skill goes like this:\n\n- Decide what you want the skill to do and roughly how it should do it\n- Write a draft of the skill\n- Create a few test prompts and run claude-with-access-to-the-skill on them\n- Help the user evaluate the results both qualitatively and quantitatively\n - While the runs happen in the background, draft some quantitative evals if there aren't any (if there are some, you can either use as is or modify if you feel something needs to change about them). Then explain them to the user (or if they already existed, explain the ones that already exist)\n - Use the `eval-viewer/generate_review.py` script to show the user the results for them to look at, and also let them look at the quantitative metrics\n- Rewrite the skill based on feedback from the user's evaluation of the results (and also if there are any glaring flaws that become apparent from the quantitative benchmarks)\n- Repeat until you're satisfied\n- Expand the test set and try again at larger scale\n\nYour job when using this skill is to figure out where the user is in this process and then jump in and help them progress through these stages. So for instance, maybe they're like \"I want to make a skill for X\". You can help narrow down what they mean, write a draft, write the test cases, figure out how they want to evaluate, run all the prompts, and repeat.\n\nOn the other hand, maybe they already have a draft of the skill. In this case you can go straight to the eval/iterate part of the loop.\n\nOf course, you should always be flexible and if the user is like \"I don't need to run a bunch of evaluations, just vibe with me\", you can do that instead.\n\nThen after the skill is done (but again, the order is flexible), you can also run the skill description improver, which we have a whole separate script for, to optimize the triggering of the skill.\n\nCool? Cool.\n\n## Communicating with the user\n\nThe skill creator is liable to be used by people across a wide range of familiarity with coding jargon. If you haven't heard (and how could you, it's only very recently that it started), there's a trend now where the power of Claude is inspiring plumbers to open up their terminals, parents and grandparents to google \"how to install npm\". On the other hand, the bulk of users are probably fairly computer-literate.\n\nSo please pay attention to context cues to understand how to phrase your communication! In the default case, just to give you some idea:\n\n- \"evaluation\" and \"benchmark\" are borderline, but OK\n- for \"JSON\" and \"assertion\" you want to see serious cues from the user that they know what those things are before using them without explaining them\n\nIt's OK to briefly explain terms if you're in doubt, and feel free to clarify terms with a short definition if you're unsure if the user will get it.\n\n---\n\n## Creating a skill\n\n### Capture Intent\n\nStart by understanding the user's intent. The current conversation might already contain a workflow the user wants to capture (e.g., they say \"turn this into a skill\"). If so, extract answers from the conversation history first — the tools used, the sequence of steps, corrections the user made, input/output formats observed. The user may need to fill the gaps, and should confirm before proceeding to the next step.\n\n1. What should this skill enable Claude to do?\n2. When should this skill trigger? (what user phrases/contexts)\n3. What's the expected output format?\n4. Should we set up test cases to verify the skill works? Skills with objectively verifiable outputs (file transforms, data extraction, code generation, fixed workflow steps) benefit from test cases. Skills with subjective outputs (writing style, art) often don't need them. Suggest the appropriate default based on the skill type, but let the user decide.\n\n### Interview and Research\n\nProactively ask questions about edge cases, input/output formats, example files, success criteria, and dependencies. Wait to write test prompts until you've got this part ironed out.\n\nCheck available MCPs - if useful for research (searching docs, finding similar skills, looking up best practices), research in parallel via subagents if available, otherwise inline. Come prepared with context to reduce burden on the user.\n\n### Write the SKILL.md\n\nBased on the user interview, fill in these components:\n\n- **name**: Skill identifier\n- **description**: When to trigger, what it does. This is the primary triggering mechanism - include both what the skill does AND specific contexts for when to use it. All \"when to use\" info goes here, not in the body. Note: currently Claude has a tendency to \"undertrigger\" skills -- to not use them when they'd be useful. To combat this, please make the skill descriptions a little bit \"pushy\". So for instance, instead of \"How to build a simple fast dashboard to display internal Anthropic data.\", you might write \"How to build a simple fast dashboard to display internal Anthropic data. Make sure to use this skill whenever the user mentions dashboards, data visualization, internal metrics, or wants to display any kind of company data, even if they don't explicitly ask for a 'dashboard.'\"\n- **compatibility**: Required tools, dependencies (optional, rarely needed)\n- **the rest of the skill :)**\n\n### Skill Writing Guide\n\n#### Anatomy of a Skill\n\n```\nskill-name/\n├── SKILL.md (required)\n│ ├── YAML frontmatter (name, description required)\n│ └── Markdown instructions\n└── Bundled Resources (optional)\n ├── scripts/ - Executable code for deterministic/repetitive tasks\n ├── references/ - Docs loaded into context as needed\n └── assets/ - Files used in output (templates, icons, fonts)\n```\n\n#### Progressive Disclosure\n\nSkills use a three-level loading system:\n1. **Metadata** (name + description) - Always in context (~100 words)\n2. **SKILL.md body** - In context whenever skill triggers (<500 lines ideal)\n3. **Bundled resources** - As needed (unlimited, scripts can execute without loading)\n\nThese word counts are approximate and you can feel free to go longer if needed.\n\n**Key patterns:**\n- Keep SKILL.md under 500 lines; if you're approaching this limit, add an additional layer of hierarchy along with clear pointers about where the model using the skill should go next to follow up.\n- Reference files clearly from SKILL.md with guidance on when to read them\n- For large reference files (>300 lines), include a table of contents\n\n**Domain organization**: When a skill supports multiple domains/frameworks, organize by variant:\n```\ncloud-deploy/\n├── SKILL.md (workflow + selection)\n└── references/\n ├── aws.md\n ├── gcp.md\n └── azure.md\n```\nClaude reads only the relevant reference file.\n\n#### Principle of Lack of Surprise\n\nThis goes without saying, but skills must not contain malware, exploit code, or any content that could compromise system security. A skill's contents should not surprise the user in their intent if described. Don't go along with requests to create misleading skills or skills designed to facilitate unauthorized access, data exfiltration, or other malicious activities. Things like a \"roleplay as an XYZ\" are OK though.\n\n#### Writing Patterns\n\nPrefer using the imperative form in instructions.\n\n**Defining output formats** - You can do it like this:\n```markdown\n## Report structure\nALWAYS use this exact template:\n# [Title]\n## Executive summary\n## Key findings\n## Recommendations\n```\n\n**Examples pattern** - It's useful to include examples. You can format them like this (but if \"Input\" and \"Output\" are in the examples you might want to deviate a little):\n```markdown\n## Commit message format\n**Example 1:**\nInput: Added user authentication with JWT tokens\nOutput: feat(auth): implement JWT-based authentication\n```\n\n### Writing Style\n\nTry to explain to the model why things are important in lieu of heavy-handed musty MUSTs. Use theory of mind and try to make the skill general and not super-narrow to specific examples. Start by writing a draft and then look at it with fresh eyes and improve it.\n\n### Test Cases\n\nAfter writing the skill draft, come up with 2-3 realistic test prompts — the kind of thing a real user would actually say. Share them with the user: [you don't have to use this exact language] \"Here are a few test cases I'd like to try. Do these look right, or do you want to add more?\" Then run them.\n\nSave test cases to `evals/evals.json`. Don't write assertions yet — just the prompts. You'll draft assertions in the next step while the runs are in progress.\n\n```json\n{\n \"skill_name\": \"example-skill\",\n \"evals\": [\n {\n \"id\": 1,\n \"prompt\": \"User's task prompt\",\n \"expected_output\": \"Description of expected result\",\n \"files\": []\n }\n ]\n}\n```\n\nSee `references/schemas.md` for the full schema (including the `assertions` field, which you'll add later).\n\n## Running and evaluating test cases\n\nThis section is one continuous sequence — don't stop partway through. Do NOT use `/skill-test` or any other testing skill.\n\nPut results in `-workspace/` as a sibling to the skill directory. Within the workspace, organize results by iteration (`iteration-1/`, `iteration-2/`, etc.) and within that, each test case gets a directory (`eval-0/`, `eval-1/`, etc.). Don't create all of this upfront — just create directories as you go.\n\n### Step 1: Spawn all runs (with-skill AND baseline) in the same turn\n\nFor each test case, spawn two subagents in the same turn — one with the skill, one without. This is important: don't spawn the with-skill runs first and then come back for baselines later. Launch everything at once so it all finishes around the same time.\n\n**With-skill run:**\n\n```\nExecute this task:\n- Skill path: \n- Task: \n- Input files: \n- Save outputs to: /iteration-/eval-/with_skill/outputs/\n- Outputs to save: \n```\n\n**Baseline run** (same prompt, but the baseline depends on context):\n- **Creating a new skill**: no skill at all. Same prompt, no skill path, save to `without_skill/outputs/`.\n- **Improving an existing skill**: the old version. Before editing, snapshot the skill (`cp -r /skill-snapshot/`), then point the baseline subagent at the snapshot. Save to `old_skill/outputs/`.\n\nWrite an `eval_metadata.json` for each test case (assertions can be empty for now). Give each eval a descriptive name based on what it's testing — not just \"eval-0\". Use this name for the directory too. If this iteration uses new or modified eval prompts, create these files for each new eval directory — don't assume they carry over from previous iterations.\n\n```json\n{\n \"eval_id\": 0,\n \"eval_name\": \"descriptive-name-here\",\n \"prompt\": \"The user's task prompt\",\n \"assertions\": []\n}\n```\n\n### Step 2: While runs are in progress, draft assertions\n\nDon't just wait for the runs to finish — you can use this time productively. Draft quantitative assertions for each test case and explain them to the user. If assertions already exist in `evals/evals.json`, review them and explain what they check.\n\nGood assertions are objectively verifiable and have descriptive names — they should read clearly in the benchmark viewer so someone glancing at the results immediately understands what each one checks. Subjective skills (writing style, design quality) are better evaluated qualitatively — don't force assertions onto things that need human judgment.\n\nUpdate the `eval_metadata.json` files and `evals/evals.json` with the assertions once drafted. Also explain to the user what they'll see in the viewer — both the qualitative outputs and the quantitative benchmark.\n\n### Step 3: As runs complete, capture timing data\n\nWhen each subagent task completes, you receive a notification containing `total_tokens` and `duration_ms`. Save this data immediately to `timing.json` in the run directory:\n\n```json\n{\n \"total_tokens\": 84852,\n \"duration_ms\": 23332,\n \"total_duration_seconds\": 23.3\n}\n```\n\nThis is the only opportunity to capture this data — it comes through the task notification and isn't persisted elsewhere. Process each notification as it arrives rather than trying to batch them.\n\n### Step 4: Grade, aggregate, and launch the viewer\n\nOnce all runs are done:\n\n1. **Grade each run** — spawn a grader subagent (or grade inline) that reads `agents/grader.md` and evaluates each assertion against the outputs. Save results to `grading.json` in each run directory. The grading.json expectations array must use the fields `text`, `passed`, and `evidence` (not `name`/`met`/`details` or other variants) — the viewer depends on these exact field names. For assertions that can be checked programmatically, write and run a script rather than eyeballing it — scripts are faster, more reliable, and can be reused across iterations.\n\n2. **Aggregate into benchmark** — run the aggregation script from the skill-creator directory:\n ```bash\n python -m scripts.aggregate_benchmark /iteration-N --skill-name \n ```\n This produces `benchmark.json` and `benchmark.md` with pass_rate, time, and tokens for each configuration, with mean ± stddev and the delta. If generating benchmark.json manually, see `references/schemas.md` for the exact schema the viewer expects.\nPut each with_skill version before its baseline counterpart.\n\n3. **Do an analyst pass** — read the benchmark data and surface patterns the aggregate stats might hide. See `agents/analyzer.md` (the \"Analyzing Benchmark Results\" section) for what to look for — things like assertions that always pass regardless of skill (non-discriminating), high-variance evals (possibly flaky), and time/token tradeoffs.\n\n4. **Launch the viewer** with both qualitative outputs and quantitative data:\n ```bash\n nohup python /eval-viewer/generate_review.py \\\n /iteration-N \\\n --skill-name \"my-skill\" \\\n --benchmark /iteration-N/benchmark.json \\\n > /dev/null 2>&1 &\n VIEWER_PID=$!\n ```\n For iteration 2+, also pass `--previous-workspace /iteration-`.\n\n **Cowork / headless environments:** If `webbrowser.open()` is not available or the environment has no display, use `--static ` to write a standalone HTML file instead of starting a server. Feedback will be downloaded as a `feedback.json` file when the user clicks \"Submit All Reviews\". After download, copy `feedback.json` into the workspace directory for the next iteration to pick up.\n\nNote: please use generate_review.py to create the viewer; there's no need to write custom HTML.\n\n5. **Tell the user** something like: \"I've opened the results in your browser. There are two tabs — 'Outputs' lets you click through each test case and leave feedback, 'Benchmark' shows the quantitative comparison. When you're done, come back here and let me know.\"\n\n### What the user sees in the viewer\n\nThe \"Outputs\" tab shows one test case at a time:\n- **Prompt**: the task that was given\n- **Output**: the files the skill produced, rendered inline where possible\n- **Previous Output** (iteration 2+): collapsed section showing last iteration's output\n- **Formal Grades** (if grading was run): collapsed section showing assertion pass/fail\n- **Feedback**: a textbox that auto-saves as they type\n- **Previous Feedback** (iteration 2+): their comments from last time, shown below the textbox\n\nThe \"Benchmark\" tab shows the stats summary: pass rates, timing, and token usage for each configuration, with per-eval breakdowns and analyst observations.\n\nNavigation is via prev/next buttons or arrow keys. When done, they click \"Submit All Reviews\" which saves all feedback to `feedback.json`.\n\n### Step 5: Read the feedback\n\nWhen the user tells you they're done, read `feedback.json`:\n\n```json\n{\n \"reviews\": [\n {\"run_id\": \"eval-0-with_skill\", \"feedback\": \"the chart is missing axis labels\", \"timestamp\": \"...\"},\n {\"run_id\": \"eval-1-with_skill\", \"feedback\": \"\", \"timestamp\": \"...\"},\n {\"run_id\": \"eval-2-with_skill\", \"feedback\": \"perfect, love this\", \"timestamp\": \"...\"}\n ],\n \"status\": \"complete\"\n}\n```\n\nEmpty feedback means the user thought it was fine. Focus your improvements on the test cases where the user had specific complaints.\n\nKill the viewer server when you're done with it:\n\n```bash\nkill $VIEWER_PID 2>/dev/null\n```\n\n---\n\n## Improving the skill\n\nThis is the heart of the loop. You've run the test cases, the user has reviewed the results, and now you need to make the skill better based on their feedback.\n\n### How to think about improvements\n\n1. **Generalize from the feedback.** The big picture thing that's happening here is that we're trying to create skills that can be used a million times (maybe literally, maybe even more who knows) across many different prompts. Here you and the user are iterating on only a few examples over and over again because it helps move faster. The user knows these examples in and out and it's quick for them to assess new outputs. But if the skill you and the user are codeveloping works only for those examples, it's useless. Rather than put in fiddly overfitty changes, or oppressively constrictive MUSTs, if there's some stubborn issue, you might try branching out and using different metaphors, or recommending different patterns of working. It's relatively cheap to try and maybe you'll land on something great.\n\n2. **Keep the prompt lean.** Remove things that aren't pulling their weight. Make sure to read the transcripts, not just the final outputs — if it looks like the skill is making the model waste a bunch of time doing things that are unproductive, you can try getting rid of the parts of the skill that are making it do that and seeing what happens.\n\n3. **Explain the why.** Try hard to explain the **why** behind everything you're asking the model to do. Today's LLMs are *smart*. They have good theory of mind and when given a good harness can go beyond rote instructions and really make things happen. Even if the feedback from the user is terse or frustrated, try to actually understand the task and why the user is writing what they wrote, and what they actually wrote, and then transmit this understanding into the instructions. If you find yourself writing ALWAYS or NEVER in all caps, or using super rigid structures, that's a yellow flag — if possible, reframe and explain the reasoning so that the model understands why the thing you're asking for is important. That's a more humane, powerful, and effective approach.\n\n4. **Look for repeated work across test cases.** Read the transcripts from the test runs and notice if the subagents all independently wrote similar helper scripts or took the same multi-step approach to something. If all 3 test cases resulted in the subagent writing a `create_docx.py` or a `build_chart.py`, that's a strong signal the skill should bundle that script. Write it once, put it in `scripts/`, and tell the skill to use it. This saves every future invocation from reinventing the wheel.\n\nThis task is pretty important (we are trying to create billions a year in economic value here!) and your thinking time is not the blocker; take your time and really mull things over. I'd suggest writing a draft revision and then looking at it anew and making improvements. Really do your best to get into the head of the user and understand what they want and need.\n\n### The iteration loop\n\nAfter improving the skill:\n\n1. Apply your improvements to the skill\n2. Rerun all test cases into a new `iteration-/` directory, including baseline runs. If you're creating a new skill, the baseline is always `without_skill` (no skill) — that stays the same across iterations. If you're improving an existing skill, use your judgment on what makes sense as the baseline: the original version the user came in with, or the previous iteration.\n3. Launch the reviewer with `--previous-workspace` pointing at the previous iteration\n4. Wait for the user to review and tell you they're done\n5. Read the new feedback, improve again, repeat\n\nKeep going until:\n- The user says they're happy\n- The feedback is all empty (everything looks good)\n- You're not making meaningful progress\n\n---\n\n## Advanced: Blind comparison\n\nFor situations where you want a more rigorous comparison between two versions of a skill (e.g., the user asks \"is the new version actually better?\"), there's a blind comparison system. Read `agents/comparator.md` and `agents/analyzer.md` for the details. The basic idea is: give two outputs to an independent agent without telling it which is which, and let it judge quality. Then analyze why the winner won.\n\nThis is optional, requires subagents, and most users won't need it. The human review loop is usually sufficient.\n\n---\n\n## Description Optimization\n\nThe description field in SKILL.md frontmatter is the primary mechanism that determines whether Claude invokes a skill. After creating or improving a skill, offer to optimize the description for better triggering accuracy.\n\n### Step 1: Generate trigger eval queries\n\nCreate 20 eval queries — a mix of should-trigger and should-not-trigger. Save as JSON:\n\n```json\n[\n {\"query\": \"the user prompt\", \"should_trigger\": true},\n {\"query\": \"another prompt\", \"should_trigger\": false}\n]\n```\n\nThe queries must be realistic and something a Claude Code or Claude.ai user would actually type. Not abstract requests, but requests that are concrete and specific and have a good amount of detail. For instance, file paths, personal context about the user's job or situation, column names and values, company names, URLs. A little bit of backstory. Some might be in lowercase or contain abbreviations or typos or casual speech. Use a mix of different lengths, and focus on edge cases rather than making them clear-cut (the user will get a chance to sign off on them).\n\nBad: `\"Format this data\"`, `\"Extract text from PDF\"`, `\"Create a chart\"`\n\nGood: `\"ok so my boss just sent me this xlsx file (its in my downloads, called something like 'Q4 sales final FINAL v2.xlsx') and she wants me to add a column that shows the profit margin as a percentage. The revenue is in column C and costs are in column D i think\"`\n\nFor the **should-trigger** queries (8-10), think about coverage. You want different phrasings of the same intent — some formal, some casual. Include cases where the user doesn't explicitly name the skill or file type but clearly needs it. Throw in some uncommon use cases and cases where this skill competes with another but should win.\n\nFor the **should-not-trigger** queries (8-10), the most valuable ones are the near-misses — queries that share keywords or concepts with the skill but actually need something different. Think adjacent domains, ambiguous phrasing where a naive keyword match would trigger but shouldn't, and cases where the query touches on something the skill does but in a context where another tool is more appropriate.\n\nThe key thing to avoid: don't make should-not-trigger queries obviously irrelevant. \"Write a fibonacci function\" as a negative test for a PDF skill is too easy — it doesn't test anything. The negative cases should be genuinely tricky.\n\n### Step 2: Review with user\n\nPresent the eval set to the user for review using the HTML template:\n\n1. Read the template from `assets/eval_review.html`\n2. Replace the placeholders:\n - `__EVAL_DATA_PLACEHOLDER__` → the JSON array of eval items (no quotes around it — it's a JS variable assignment)\n - `__SKILL_NAME_PLACEHOLDER__` → the skill's name\n - `__SKILL_DESCRIPTION_PLACEHOLDER__` → the skill's current description\n3. Write to a temp file (e.g., `/tmp/eval_review_.html`) and open it: `open /tmp/eval_review_.html`\n4. The user can edit queries, toggle should-trigger, add/remove entries, then click \"Export Eval Set\"\n5. The file downloads to `~/Downloads/eval_set.json` — check the Downloads folder for the most recent version in case there are multiple (e.g., `eval_set (1).json`)\n\nThis step matters — bad eval queries lead to bad descriptions.\n\n### Step 3: Run the optimization loop\n\nTell the user: \"This will take some time — I'll run the optimization loop in the background and check on it periodically.\"\n\nSave the eval set to the workspace, then run in the background:\n\n```bash\npython -m scripts.run_loop \\\n --eval-set \\\n --skill-path \\\n --model \\\n --max-iterations 5 \\\n --verbose\n```\n\nUse the model ID from your system prompt (the one powering the current session) so the triggering test matches what the user actually experiences.\n\nWhile it runs, periodically tail the output to give the user updates on which iteration it's on and what the scores look like.\n\nThis handles the full optimization loop automatically. It splits the eval set into 60% train and 40% held-out test, evaluates the current description (running each query 3 times to get a reliable trigger rate), then calls Claude to propose improvements based on what failed. It re-evaluates each new description on both train and test, iterating up to 5 times. When it's done, it opens an HTML report in the browser showing the results per iteration and returns JSON with `best_description` — selected by test score rather than train score to avoid overfitting.\n\n### How skill triggering works\n\nUnderstanding the triggering mechanism helps design better eval queries. Skills appear in Claude's `available_skills` list with their name + description, and Claude decides whether to consult a skill based on that description. The important thing to know is that Claude only consults skills for tasks it can't easily handle on its own — simple, one-step queries like \"read this PDF\" may not trigger a skill even if the description matches perfectly, because Claude can handle them directly with basic tools. Complex, multi-step, or specialized queries reliably trigger skills when the description matches.\n\nThis means your eval queries should be substantive enough that Claude would actually benefit from consulting a skill. Simple queries like \"read file X\" are poor test cases — they won't trigger skills regardless of description quality.\n\n### Step 4: Apply the result\n\nTake `best_description` from the JSON output and update the skill's SKILL.md frontmatter. Show the user before/after and report the scores.\n\n---\n\n### Package and Present (only if `present_files` tool is available)\n\nCheck whether you have access to the `present_files` tool. If you don't, skip this step. If you do, package the skill and present the .skill file to the user:\n\n```bash\npython -m scripts.package_skill \n```\n\nAfter packaging, direct the user to the resulting `.skill` file path so they can install it.\n\n---\n\n## Claude.ai-specific instructions\n\nIn Claude.ai, the core workflow is the same (draft → test → review → improve → repeat), but because Claude.ai doesn't have subagents, some mechanics change. Here's what to adapt:\n\n**Running test cases**: No subagents means no parallel execution. For each test case, read the skill's SKILL.md, then follow its instructions to accomplish the test prompt yourself. Do them one at a time. This is less rigorous than independent subagents (you wrote the skill and you're also running it, so you have full context), but it's a useful sanity check — and the human review step compensates. Skip the baseline runs — just use the skill to complete the task as requested.\n\n**Reviewing results**: If you can't open a browser (e.g., Claude.ai's VM has no display, or you're on a remote server), skip the browser reviewer entirely. Instead, present results directly in the conversation. For each test case, show the prompt and the output. If the output is a file the user needs to see (like a .docx or .xlsx), save it to the filesystem and tell them where it is so they can download and inspect it. Ask for feedback inline: \"How does this look? Anything you'd change?\"\n\n**Benchmarking**: Skip the quantitative benchmarking — it relies on baseline comparisons which aren't meaningful without subagents. Focus on qualitative feedback from the user.\n\n**The iteration loop**: Same as before — improve the skill, rerun the test cases, ask for feedback — just without the browser reviewer in the middle. You can still organize results into iteration directories on the filesystem if you have one.\n\n**Description optimization**: This section requires the `claude` CLI tool (specifically `claude -p`) which is only available in Claude Code. Skip it if you're on Claude.ai.\n\n**Blind comparison**: Requires subagents. Skip it.\n\n**Packaging**: The `package_skill.py` script works anywhere with Python and a filesystem. On Claude.ai, you can run it and the user can download the resulting `.skill` file.\n\n**Updating an existing skill**: The user might be asking you to update an existing skill, not create a new one. In this case:\n- **Preserve the original name.** Note the skill's directory name and `name` frontmatter field -- use them unchanged. E.g., if the installed skill is `research-helper`, output `research-helper.skill` (not `research-helper-v2`).\n- **Copy to a writeable location before editing.** The installed skill path may be read-only. Copy to `/tmp/skill-name/`, edit there, and package from the copy.\n- **If packaging manually, stage in `/tmp/` first**, then copy to the output directory -- direct writes may fail due to permissions.\n\n---\n\n## Cowork-Specific Instructions\n\nIf you're in Cowork, the main things to know are:\n\n- You have subagents, so the main workflow (spawn test cases in parallel, run baselines, grade, etc.) all works. (However, if you run into severe problems with timeouts, it's OK to run the test prompts in series rather than parallel.)\n- You don't have a browser or display, so when generating the eval viewer, use `--static ` to write a standalone HTML file instead of starting a server. Then proffer a link that the user can click to open the HTML in their browser.\n- For whatever reason, the Cowork setup seems to disincline Claude from generating the eval viewer after running the tests, so just to reiterate: whether you're in Cowork or in Claude Code, after running tests, you should always generate the eval viewer for the human to look at examples before revising the skill yourself and trying to make corrections, using `generate_review.py` (not writing your own boutique html code). Sorry in advance but I'm gonna go all caps here: GENERATE THE EVAL VIEWER *BEFORE* evaluating inputs yourself. You want to get them in front of the human ASAP!\n- Feedback works differently: since there's no running server, the viewer's \"Submit All Reviews\" button will download `feedback.json` as a file. You can then read it from there (you may have to request access first).\n- Packaging works — `package_skill.py` just needs Python and a filesystem.\n- Description optimization (`run_loop.py` / `run_eval.py`) should work in Cowork just fine since it uses `claude -p` via subprocess, not a browser, but please save it until you've fully finished making the skill and the user agrees it's in good shape.\n- **Updating an existing skill**: The user might be asking you to update an existing skill, not create a new one. Follow the update guidance in the claude.ai section above.\n\n---\n\n## Reference files\n\nThe agents/ directory contains instructions for specialized subagents. Read them when you need to spawn the relevant subagent.\n\n- `agents/grader.md` — How to evaluate assertions against outputs\n- `agents/comparator.md` — How to do blind A/B comparison between two outputs\n- `agents/analyzer.md` — How to analyze why one version beat another\n\nThe references/ directory has additional documentation:\n- `references/schemas.md` — JSON structures for evals.json, grading.json, etc.\n\n---\n\nRepeating one more time the core loop here for emphasis:\n\n- Figure out what the skill is about\n- Draft or edit the skill\n- Run claude-with-access-to-the-skill on test prompts\n- With the user, evaluate the outputs:\n - Create benchmark.json and run `eval-viewer/generate_review.py` to help the user review them\n - Run quantitative evals\n- Repeat until you and the user are satisfied\n- Package the final skill and return it to the user.\n\nPlease add steps to your TodoList, if you have such a thing, to make sure you don't forget. If you're in Cowork, please specifically put \"Create evals JSON and run `eval-viewer/generate_review.py` so human can review test cases\" in your TodoList to make sure it happens.\n\nGood luck!\n","frontmatter":{"name":"skill-creator","description":"Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy."},"isInternal":false,"tokens":7322,"sizeBytes":33168},{"name":"codex-tools.md","path":"flock-data/skills/using-superpowers/references/codex-tools.md","rawUrl":"https://raw.githubusercontent.com/Onelevenvy/flock/HEAD/flock-data/skills/using-superpowers/references/codex-tools.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Codex Tool Mapping\n\nSkills use Claude Code tool names. When you encounter these in a skill, use your platform equivalent:\n\n| Skill references | Codex equivalent |\n|-----------------|------------------|\n| `Task` tool (dispatch subagent) | `spawn_agent` |\n| Multiple `Task` calls (parallel) | Multiple `spawn_agent` calls |\n| Task returns result | `wait` |\n| Task completes automatically | `close_agent` to free slot |\n| `TodoWrite` (task tracking) | `update_plan` |\n| `Skill` tool (invoke a skill) | Skills load natively — just follow the instructions |\n| `Read`, `Write`, `Edit` (files) | Use your native file tools |\n| `Bash` (run commands) | Use your native shell tools |\n\n## Subagent dispatch requires collab\n\nAdd to your Codex config (`~/.codex/config.toml`):\n\n```toml\n[features]\ncollab = true\n```\n\nThis enables `spawn_agent`, `wait`, and `close_agent` for skills like `dispatching-parallel-agents` and `subagent-driven-development`.\n","isInternal":false,"tokens":238,"sizeBytes":942},{"name":"gemini-tools.md","path":"flock-data/skills/using-superpowers/references/gemini-tools.md","rawUrl":"https://raw.githubusercontent.com/Onelevenvy/flock/HEAD/flock-data/skills/using-superpowers/references/gemini-tools.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Gemini CLI Tool Mapping\n\nSkills use Claude Code tool names. When you encounter these in a skill, use your platform equivalent:\n\n| Skill references | Gemini CLI equivalent |\n|-----------------|----------------------|\n| `Read` (file reading) | `read_file` |\n| `Write` (file creation) | `write_file` |\n| `Edit` (file editing) | `replace` |\n| `Bash` (run commands) | `run_shell_command` |\n| `Grep` (search file content) | `grep_search` |\n| `Glob` (search files by name) | `glob` |\n| `TodoWrite` (task tracking) | `write_todos` |\n| `Skill` tool (invoke a skill) | `activate_skill` |\n| `WebSearch` | `google_web_search` |\n| `WebFetch` | `web_fetch` |\n| `Task` tool (dispatch subagent) | No equivalent — Gemini CLI does not support subagents |\n\n## No subagent support\n\nGemini CLI has no equivalent to Claude Code's `Task` tool. Skills that rely on subagent dispatch (`subagent-driven-development`, `dispatching-parallel-agents`) will fall back to single-session execution via `executing-plans`.\n\n## Additional Gemini CLI tools\n\nThese tools are available in Gemini CLI but have no Claude Code equivalent:\n\n| Tool | Purpose |\n|------|---------|\n| `list_directory` | List files and subdirectories |\n| `save_memory` | Persist facts to GEMINI.md across sessions |\n| `ask_user` | Request structured input from the user |\n| `tracker_create_task` | Rich task management (create, update, list, visualize) |\n| `enter_plan_mode` / `exit_plan_mode` | Switch to read-only research mode before making changes |\n","isInternal":false,"tokens":377,"sizeBytes":1494},{"name":"SKILL.md","path":"flock-data/skills/using-superpowers/SKILL.md","rawUrl":"https://raw.githubusercontent.com/Onelevenvy/flock/HEAD/flock-data/skills/using-superpowers/SKILL.md","title":"using-superpowers","category":"anthropic-skill","format":"markdown","content":"---\nname: using-superpowers\ndescription: Use when starting any conversation - establishes how to find and use skills, requiring Skill tool invocation before ANY response including clarifying questions\n---\n\n\nIf you were dispatched as a subagent to execute a specific task, skip this skill.\n\n\n\nIf you think there is even a 1% chance a skill might apply to what you are doing, you ABSOLUTELY MUST invoke the skill.\n\nIF A SKILL APPLIES TO YOUR TASK, YOU DO NOT HAVE A CHOICE. YOU MUST USE IT.\n\nThis is not negotiable. This is not optional. You cannot rationalize your way out of this.\n\n\n## Instruction Priority\n\nSuperpowers skills override default system prompt behavior, but **user instructions always take precedence**:\n\n1. **User's explicit instructions** (CLAUDE.md, GEMINI.md, AGENTS.md, direct requests) — highest priority\n2. **Superpowers skills** — override default system behavior where they conflict\n3. **Default system prompt** — lowest priority\n\nIf CLAUDE.md, GEMINI.md, or AGENTS.md says \"don't use TDD\" and a skill says \"always use TDD,\" follow the user's instructions. The user is in control.\n\n## How to Access Skills\n\n**In Claude Code:** Use the `Skill` tool. When you invoke a skill, its content is loaded and presented to you—follow it directly. Never use the Read tool on skill files.\n\n**In Gemini CLI:** Skills activate via the `activate_skill` tool. Gemini loads skill metadata at session start and activates the full content on demand.\n\n**In other environments:** Check your platform's documentation for how skills are loaded.\n\n## Platform Adaptation\n\nSkills use Claude Code tool names. Non-CC platforms: see `references/codex-tools.md` (Codex) for tool equivalents. Gemini CLI users get the tool mapping loaded automatically via GEMINI.md.\n\n# Using Skills\n\n## The Rule\n\n**Invoke relevant or requested skills BEFORE any response or action.** Even a 1% chance a skill might apply means that you should invoke the skill to check. If an invoked skill turns out to be wrong for the situation, you don't need to use it.\n\n```dot\ndigraph skill_flow {\n \"User message received\" [shape=doublecircle];\n \"About to EnterPlanMode?\" [shape=doublecircle];\n \"Already brainstormed?\" [shape=diamond];\n \"Invoke brainstorming skill\" [shape=box];\n \"Might any skill apply?\" [shape=diamond];\n \"Invoke Skill tool\" [shape=box];\n \"Announce: 'Using [skill] to [purpose]'\" [shape=box];\n \"Has checklist?\" [shape=diamond];\n \"Create TodoWrite todo per item\" [shape=box];\n \"Follow skill exactly\" [shape=box];\n \"Respond (including clarifications)\" [shape=doublecircle];\n\n \"About to EnterPlanMode?\" -> \"Already brainstormed?\";\n \"Already brainstormed?\" -> \"Invoke brainstorming skill\" [label=\"no\"];\n \"Already brainstormed?\" -> \"Might any skill apply?\" [label=\"yes\"];\n \"Invoke brainstorming skill\" -> \"Might any skill apply?\";\n\n \"User message received\" -> \"Might any skill apply?\";\n \"Might any skill apply?\" -> \"Invoke Skill tool\" [label=\"yes, even 1%\"];\n \"Might any skill apply?\" -> \"Respond (including clarifications)\" [label=\"definitely not\"];\n \"Invoke Skill tool\" -> \"Announce: 'Using [skill] to [purpose]'\";\n \"Announce: 'Using [skill] to [purpose]'\" -> \"Has checklist?\";\n \"Has checklist?\" -> \"Create TodoWrite todo per item\" [label=\"yes\"];\n \"Has checklist?\" -> \"Follow skill exactly\" [label=\"no\"];\n \"Create TodoWrite todo per item\" -> \"Follow skill exactly\";\n}\n```\n\n## Red Flags\n\nThese thoughts mean STOP—you're rationalizing:\n\n| Thought | Reality |\n|---------|---------|\n| \"This is just a simple question\" | Questions are tasks. Check for skills. |\n| \"I need more context first\" | Skill check comes BEFORE clarifying questions. |\n| \"Let me explore the codebase first\" | Skills tell you HOW to explore. Check first. |\n| \"I can check git/files quickly\" | Files lack conversation context. Check for skills. |\n| \"Let me gather information first\" | Skills tell you HOW to gather information. |\n| \"This doesn't need a formal skill\" | If a skill exists, use it. |\n| \"I remember this skill\" | Skills evolve. Read current version. |\n| \"This doesn't count as a task\" | Action = task. Check for skills. |\n| \"The skill is overkill\" | Simple things become complex. Use it. |\n| \"I'll just do this one thing first\" | Check BEFORE doing anything. |\n| \"This feels productive\" | Undisciplined action wastes time. Skills prevent this. |\n| \"I know what that means\" | Knowing the concept ≠ using the skill. Invoke it. |\n\n## Skill Priority\n\nWhen multiple skills could apply, use this order:\n\n1. **Process skills first** (brainstorming, debugging) - these determine HOW to approach the task\n2. **Implementation skills second** (frontend-design, mcp-builder) - these guide execution\n\n\"Let's build X\" → brainstorming first, then implementation skills.\n\"Fix this bug\" → debugging first, then domain-specific skills.\n\n## Skill Types\n\n**Rigid** (TDD, debugging): Follow exactly. Don't adapt away discipline.\n\n**Flexible** (patterns): Adapt principles to context.\n\nThe skill itself tells you which.\n\n## User Instructions\n\nInstructions say WHAT, not HOW. \"Add X\" or \"Fix Y\" doesn't mean skip workflows.\n","frontmatter":{"name":"using-superpowers","description":"Use when starting any conversation - establishes how to find and use skills, requiring Skill tool invocation before ANY response including clarifying questions"},"isInternal":false,"tokens":1262,"sizeBytes":5219},{"name":"analyzer.md","path":"flock-data/skills/skill-creator/agents/analyzer.md","rawUrl":"https://raw.githubusercontent.com/Onelevenvy/flock/HEAD/flock-data/skills/skill-creator/agents/analyzer.md","title":"Subagent: analyzer","category":"subagent-persona","format":"markdown","content":"# Post-hoc Analyzer Agent\n\nAnalyze blind comparison results to understand WHY the winner won and generate improvement suggestions.\n\n## Role\n\nAfter the blind comparator determines a winner, the Post-hoc Analyzer \"unblids\" the results by examining the skills and transcripts. The goal is to extract actionable insights: what made the winner better, and how can the loser be improved?\n\n## Inputs\n\nYou receive these parameters in your prompt:\n\n- **winner**: \"A\" or \"B\" (from blind comparison)\n- **winner_skill_path**: Path to the skill that produced the winning output\n- **winner_transcript_path**: Path to the execution transcript for the winner\n- **loser_skill_path**: Path to the skill that produced the losing output\n- **loser_transcript_path**: Path to the execution transcript for the loser\n- **comparison_result_path**: Path to the blind comparator's output JSON\n- **output_path**: Where to save the analysis results\n\n## Process\n\n### Step 1: Read Comparison Result\n\n1. Read the blind comparator's output at comparison_result_path\n2. Note the winning side (A or B), the reasoning, and any scores\n3. Understand what the comparator valued in the winning output\n\n### Step 2: Read Both Skills\n\n1. Read the winner skill's SKILL.md and key referenced files\n2. Read the loser skill's SKILL.md and key referenced files\n3. Identify structural differences:\n - Instructions clarity and specificity\n - Script/tool usage patterns\n - Example coverage\n - Edge case handling\n\n### Step 3: Read Both Transcripts\n\n1. Read the winner's transcript\n2. Read the loser's transcript\n3. Compare execution patterns:\n - How closely did each follow their skill's instructions?\n - What tools were used differently?\n - Where did the loser diverge from optimal behavior?\n - Did either encounter errors or make recovery attempts?\n\n### Step 4: Analyze Instruction Following\n\nFor each transcript, evaluate:\n- Did the agent follow the skill's explicit instructions?\n- Did the agent use the skill's provided tools/scripts?\n- Were there missed opportunities to leverage skill content?\n- Did the agent add unnecessary steps not in the skill?\n\nScore instruction following 1-10 and note specific issues.\n\n### Step 5: Identify Winner Strengths\n\nDetermine what made the winner better:\n- Clearer instructions that led to better behavior?\n- Better scripts/tools that produced better output?\n- More comprehensive examples that guided edge cases?\n- Better error handling guidance?\n\nBe specific. Quote from skills/transcripts where relevant.\n\n### Step 6: Identify Loser Weaknesses\n\nDetermine what held the loser back:\n- Ambiguous instructions that led to suboptimal choices?\n- Missing tools/scripts that forced workarounds?\n- Gaps in edge case coverage?\n- Poor error handling that caused failures?\n\n### Step 7: Generate Improvement Suggestions\n\nBased on the analysis, produce actionable suggestions for improving the loser skill:\n- Specific instruction changes to make\n- Tools/scripts to add or modify\n- Examples to include\n- Edge cases to address\n\nPrioritize by impact. Focus on changes that would have changed the outcome.\n\n### Step 8: Write Analysis Results\n\nSave structured analysis to `{output_path}`.\n\n## Output Format\n\nWrite a JSON file with this structure:\n\n```json\n{\n \"comparison_summary\": {\n \"winner\": \"A\",\n \"winner_skill\": \"path/to/winner/skill\",\n \"loser_skill\": \"path/to/loser/skill\",\n \"comparator_reasoning\": \"Brief summary of why comparator chose winner\"\n },\n \"winner_strengths\": [\n \"Clear step-by-step instructions for handling multi-page documents\",\n \"Included validation script that caught formatting errors\",\n \"Explicit guidance on fallback behavior when OCR fails\"\n ],\n \"loser_weaknesses\": [\n \"Vague instruction 'process the document appropriately' led to inconsistent behavior\",\n \"No script for validation, agent had to improvise and made errors\",\n \"No guidance on OCR failure, agent gave up instead of trying alternatives\"\n ],\n \"instruction_following\": {\n \"winner\": {\n \"score\": 9,\n \"issues\": [\n \"Minor: skipped optional logging step\"\n ]\n },\n \"loser\": {\n \"score\": 6,\n \"issues\": [\n \"Did not use the skill's formatting template\",\n \"Invented own approach instead of following step 3\",\n \"Missed the 'always validate output' instruction\"\n ]\n }\n },\n \"improvement_suggestions\": [\n {\n \"priority\": \"high\",\n \"category\": \"instructions\",\n \"suggestion\": \"Replace 'process the document appropriately' with explicit steps: 1) Extract text, 2) Identify sections, 3) Format per template\",\n \"expected_impact\": \"Would eliminate ambiguity that caused inconsistent behavior\"\n },\n {\n \"priority\": \"high\",\n \"category\": \"tools\",\n \"suggestion\": \"Add validate_output.py script similar to winner skill's validation approach\",\n \"expected_impact\": \"Would catch formatting errors before final output\"\n },\n {\n \"priority\": \"medium\",\n \"category\": \"error_handling\",\n \"suggestion\": \"Add fallback instructions: 'If OCR fails, try: 1) different resolution, 2) image preprocessing, 3) manual extraction'\",\n \"expected_impact\": \"Would prevent early failure on difficult documents\"\n }\n ],\n \"transcript_insights\": {\n \"winner_execution_pattern\": \"Read skill -> Followed 5-step process -> Used validation script -> Fixed 2 issues -> Produced output\",\n \"loser_execution_pattern\": \"Read skill -> Unclear on approach -> Tried 3 different methods -> No validation -> Output had errors\"\n }\n}\n```\n\n## Guidelines\n\n- **Be specific**: Quote from skills and transcripts, don't just say \"instructions were unclear\"\n- **Be actionable**: Suggestions should be concrete changes, not vague advice\n- **Focus on skill improvements**: The goal is to improve the losing skill, not critique the agent\n- **Prioritize by impact**: Which changes would most likely have changed the outcome?\n- **Consider causation**: Did the skill weakness actually cause the worse output, or is it incidental?\n- **Stay objective**: Analyze what happened, don't editorialize\n- **Think about generalization**: Would this improvement help on other evals too?\n\n## Categories for Suggestions\n\nUse these categories to organize improvement suggestions:\n\n| Category | Description |\n|----------|-------------|\n| `instructions` | Changes to the skill's prose instructions |\n| `tools` | Scripts, templates, or utilities to add/modify |\n| `examples` | Example inputs/outputs to include |\n| `error_handling` | Guidance for handling failures |\n| `structure` | Reorganization of skill content |\n| `references` | External docs or resources to add |\n\n## Priority Levels\n\n- **high**: Would likely change the outcome of this comparison\n- **medium**: Would improve quality but may not change win/loss\n- **low**: Nice to have, marginal improvement\n\n---\n\n# Analyzing Benchmark Results\n\nWhen analyzing benchmark results, the analyzer's purpose is to **surface patterns and anomalies** across multiple runs, not suggest skill improvements.\n\n## Role\n\nReview all benchmark run results and generate freeform notes that help the user understand skill performance. Focus on patterns that wouldn't be visible from aggregate metrics alone.\n\n## Inputs\n\nYou receive these parameters in your prompt:\n\n- **benchmark_data_path**: Path to the in-progress benchmark.json with all run results\n- **skill_path**: Path to the skill being benchmarked\n- **output_path**: Where to save the notes (as JSON array of strings)\n\n## Process\n\n### Step 1: Read Benchmark Data\n\n1. Read the benchmark.json containing all run results\n2. Note the configurations tested (with_skill, without_skill)\n3. Understand the run_summary aggregates already calculated\n\n### Step 2: Analyze Per-Assertion Patterns\n\nFor each expectation across all runs:\n- Does it **always pass** in both configurations? (may not differentiate skill value)\n- Does it **always fail** in both configurations? (may be broken or beyond capability)\n- Does it **always pass with skill but fail without**? (skill clearly adds value here)\n- Does it **always fail with skill but pass without**? (skill may be hurting)\n- Is it **highly variable**? (flaky expectation or non-deterministic behavior)\n\n### Step 3: Analyze Cross-Eval Patterns\n\nLook for patterns across evals:\n- Are certain eval types consistently harder/easier?\n- Do some evals show high variance while others are stable?\n- Are there surprising results that contradict expectations?\n\n### Step 4: Analyze Metrics Patterns\n\nLook at time_seconds, tokens, tool_calls:\n- Does the skill significantly increase execution time?\n- Is there high variance in resource usage?\n- Are there outlier runs that skew the aggregates?\n\n### Step 5: Generate Notes\n\nWrite freeform observations as a list of strings. Each note should:\n- State a specific observation\n- Be grounded in the data (not speculation)\n- Help the user understand something the aggregate metrics don't show\n\nExamples:\n- \"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value\"\n- \"Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure that may be flaky\"\n- \"Without-skill runs consistently fail on table extraction expectations (0% pass rate)\"\n- \"Skill adds 13s average execution time but improves pass rate by 50%\"\n- \"Token usage is 80% higher with skill, primarily due to script output parsing\"\n- \"All 3 without-skill runs for eval 1 produced empty output\"\n\n### Step 6: Write Notes\n\nSave notes to `{output_path}` as a JSON array of strings:\n\n```json\n[\n \"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value\",\n \"Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure\",\n \"Without-skill runs consistently fail on table extraction expectations\",\n \"Skill adds 13s average execution time but improves pass rate by 50%\"\n]\n```\n\n## Guidelines\n\n**DO:**\n- Report what you observe in the data\n- Be specific about which evals, expectations, or runs you're referring to\n- Note patterns that aggregate metrics would hide\n- Provide context that helps interpret the numbers\n\n**DO NOT:**\n- Suggest improvements to the skill (that's for the improvement step, not benchmarking)\n- Make subjective quality judgments (\"the output was good/bad\")\n- Speculate about causes without evidence\n- Repeat information already in the run_summary aggregates\n","isInternal":false,"tokens":2258,"sizeBytes":10376},{"name":"comparator.md","path":"flock-data/skills/skill-creator/agents/comparator.md","rawUrl":"https://raw.githubusercontent.com/Onelevenvy/flock/HEAD/flock-data/skills/skill-creator/agents/comparator.md","title":"Subagent: comparator","category":"subagent-persona","format":"markdown","content":"# Blind Comparator Agent\n\nCompare two outputs WITHOUT knowing which skill produced them.\n\n## Role\n\nThe Blind Comparator judges which output better accomplishes the eval task. You receive two outputs labeled A and B, but you do NOT know which skill produced which. This prevents bias toward a particular skill or approach.\n\nYour judgment is based purely on output quality and task completion.\n\n## Inputs\n\nYou receive these parameters in your prompt:\n\n- **output_a_path**: Path to the first output file or directory\n- **output_b_path**: Path to the second output file or directory\n- **eval_prompt**: The original task/prompt that was executed\n- **expectations**: List of expectations to check (optional - may be empty)\n\n## Process\n\n### Step 1: Read Both Outputs\n\n1. Examine output A (file or directory)\n2. Examine output B (file or directory)\n3. Note the type, structure, and content of each\n4. If outputs are directories, examine all relevant files inside\n\n### Step 2: Understand the Task\n\n1. Read the eval_prompt carefully\n2. Identify what the task requires:\n - What should be produced?\n - What qualities matter (accuracy, completeness, format)?\n - What would distinguish a good output from a poor one?\n\n### Step 3: Generate Evaluation Rubric\n\nBased on the task, generate a rubric with two dimensions:\n\n**Content Rubric** (what the output contains):\n| Criterion | 1 (Poor) | 3 (Acceptable) | 5 (Excellent) |\n|-----------|----------|----------------|---------------|\n| Correctness | Major errors | Minor errors | Fully correct |\n| Completeness | Missing key elements | Mostly complete | All elements present |\n| Accuracy | Significant inaccuracies | Minor inaccuracies | Accurate throughout |\n\n**Structure Rubric** (how the output is organized):\n| Criterion | 1 (Poor) | 3 (Acceptable) | 5 (Excellent) |\n|-----------|----------|----------------|---------------|\n| Organization | Disorganized | Reasonably organized | Clear, logical structure |\n| Formatting | Inconsistent/broken | Mostly consistent | Professional, polished |\n| Usability | Difficult to use | Usable with effort | Easy to use |\n\nAdapt criteria to the specific task. For example:\n- PDF form → \"Field alignment\", \"Text readability\", \"Data placement\"\n- Document → \"Section structure\", \"Heading hierarchy\", \"Paragraph flow\"\n- Data output → \"Schema correctness\", \"Data types\", \"Completeness\"\n\n### Step 4: Evaluate Each Output Against the Rubric\n\nFor each output (A and B):\n\n1. **Score each criterion** on the rubric (1-5 scale)\n2. **Calculate dimension totals**: Content score, Structure score\n3. **Calculate overall score**: Average of dimension scores, scaled to 1-10\n\n### Step 5: Check Assertions (if provided)\n\nIf expectations are provided:\n\n1. Check each expectation against output A\n2. Check each expectation against output B\n3. Count pass rates for each output\n4. Use expectation scores as secondary evidence (not the primary decision factor)\n\n### Step 6: Determine the Winner\n\nCompare A and B based on (in priority order):\n\n1. **Primary**: Overall rubric score (content + structure)\n2. **Secondary**: Assertion pass rates (if applicable)\n3. **Tiebreaker**: If truly equal, declare a TIE\n\nBe decisive - ties should be rare. One output is usually better, even if marginally.\n\n### Step 7: Write Comparison Results\n\nSave results to a JSON file at the path specified (or `comparison.json` if not specified).\n\n## Output Format\n\nWrite a JSON file with this structure:\n\n```json\n{\n \"winner\": \"A\",\n \"reasoning\": \"Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.\",\n \"rubric\": {\n \"A\": {\n \"content\": {\n \"correctness\": 5,\n \"completeness\": 5,\n \"accuracy\": 4\n },\n \"structure\": {\n \"organization\": 4,\n \"formatting\": 5,\n \"usability\": 4\n },\n \"content_score\": 4.7,\n \"structure_score\": 4.3,\n \"overall_score\": 9.0\n },\n \"B\": {\n \"content\": {\n \"correctness\": 3,\n \"completeness\": 2,\n \"accuracy\": 3\n },\n \"structure\": {\n \"organization\": 3,\n \"formatting\": 2,\n \"usability\": 3\n },\n \"content_score\": 2.7,\n \"structure_score\": 2.7,\n \"overall_score\": 5.4\n }\n },\n \"output_quality\": {\n \"A\": {\n \"score\": 9,\n \"strengths\": [\"Complete solution\", \"Well-formatted\", \"All fields present\"],\n \"weaknesses\": [\"Minor style inconsistency in header\"]\n },\n \"B\": {\n \"score\": 5,\n \"strengths\": [\"Readable output\", \"Correct basic structure\"],\n \"weaknesses\": [\"Missing date field\", \"Formatting inconsistencies\", \"Partial data extraction\"]\n }\n },\n \"expectation_results\": {\n \"A\": {\n \"passed\": 4,\n \"total\": 5,\n \"pass_rate\": 0.80,\n \"details\": [\n {\"text\": \"Output includes name\", \"passed\": true},\n {\"text\": \"Output includes date\", \"passed\": true},\n {\"text\": \"Format is PDF\", \"passed\": true},\n {\"text\": \"Contains signature\", \"passed\": false},\n {\"text\": \"Readable text\", \"passed\": true}\n ]\n },\n \"B\": {\n \"passed\": 3,\n \"total\": 5,\n \"pass_rate\": 0.60,\n \"details\": [\n {\"text\": \"Output includes name\", \"passed\": true},\n {\"text\": \"Output includes date\", \"passed\": false},\n {\"text\": \"Format is PDF\", \"passed\": true},\n {\"text\": \"Contains signature\", \"passed\": false},\n {\"text\": \"Readable text\", \"passed\": true}\n ]\n }\n }\n}\n```\n\nIf no expectations were provided, omit the `expectation_results` field entirely.\n\n## Field Descriptions\n\n- **winner**: \"A\", \"B\", or \"TIE\"\n- **reasoning**: Clear explanation of why the winner was chosen (or why it's a tie)\n- **rubric**: Structured rubric evaluation for each output\n - **content**: Scores for content criteria (correctness, completeness, accuracy)\n - **structure**: Scores for structure criteria (organization, formatting, usability)\n - **content_score**: Average of content criteria (1-5)\n - **structure_score**: Average of structure criteria (1-5)\n - **overall_score**: Combined score scaled to 1-10\n- **output_quality**: Summary quality assessment\n - **score**: 1-10 rating (should match rubric overall_score)\n - **strengths**: List of positive aspects\n - **weaknesses**: List of issues or shortcomings\n- **expectation_results**: (Only if expectations provided)\n - **passed**: Number of expectations that passed\n - **total**: Total number of expectations\n - **pass_rate**: Fraction passed (0.0 to 1.0)\n - **details**: Individual expectation results\n\n## Guidelines\n\n- **Stay blind**: DO NOT try to infer which skill produced which output. Judge purely on output quality.\n- **Be specific**: Cite specific examples when explaining strengths and weaknesses.\n- **Be decisive**: Choose a winner unless outputs are genuinely equivalent.\n- **Output quality first**: Assertion scores are secondary to overall task completion.\n- **Be objective**: Don't favor outputs based on style preferences; focus on correctness and completeness.\n- **Explain your reasoning**: The reasoning field should make it clear why you chose the winner.\n- **Handle edge cases**: If both outputs fail, pick the one that fails less badly. If both are excellent, pick the one that's marginally better.\n","isInternal":false,"tokens":1743,"sizeBytes":7287},{"name":"grader.md","path":"flock-data/skills/skill-creator/agents/grader.md","rawUrl":"https://raw.githubusercontent.com/Onelevenvy/flock/HEAD/flock-data/skills/skill-creator/agents/grader.md","title":"Subagent: grader","category":"subagent-persona","format":"markdown","content":"# Grader Agent\n\nEvaluate expectations against an execution transcript and outputs.\n\n## Role\n\nThe Grader reviews a transcript and output files, then determines whether each expectation passes or fails. Provide clear evidence for each judgment.\n\nYou have two jobs: grade the outputs, and critique the evals themselves. A passing grade on a weak assertion is worse than useless — it creates false confidence. When you notice an assertion that's trivially satisfied, or an important outcome that no assertion checks, say so.\n\n## Inputs\n\nYou receive these parameters in your prompt:\n\n- **expectations**: List of expectations to evaluate (strings)\n- **transcript_path**: Path to the execution transcript (markdown file)\n- **outputs_dir**: Directory containing output files from execution\n\n## Process\n\n### Step 1: Read the Transcript\n\n1. Read the transcript file completely\n2. Note the eval prompt, execution steps, and final result\n3. Identify any issues or errors documented\n\n### Step 2: Examine Output Files\n\n1. List files in outputs_dir\n2. Read/examine each file relevant to the expectations. If outputs aren't plain text, use the inspection tools provided in your prompt — don't rely solely on what the transcript says the executor produced.\n3. Note contents, structure, and quality\n\n### Step 3: Evaluate Each Assertion\n\nFor each expectation:\n\n1. **Search for evidence** in the transcript and outputs\n2. **Determine verdict**:\n - **PASS**: Clear evidence the expectation is true AND the evidence reflects genuine task completion, not just surface-level compliance\n - **FAIL**: No evidence, or evidence contradicts the expectation, or the evidence is superficial (e.g., correct filename but empty/wrong content)\n3. **Cite the evidence**: Quote the specific text or describe what you found\n\n### Step 4: Extract and Verify Claims\n\nBeyond the predefined expectations, extract implicit claims from the outputs and verify them:\n\n1. **Extract claims** from the transcript and outputs:\n - Factual statements (\"The form has 12 fields\")\n - Process claims (\"Used pypdf to fill the form\")\n - Quality claims (\"All fields were filled correctly\")\n\n2. **Verify each claim**:\n - **Factual claims**: Can be checked against the outputs or external sources\n - **Process claims**: Can be verified from the transcript\n - **Quality claims**: Evaluate whether the claim is justified\n\n3. **Flag unverifiable claims**: Note claims that cannot be verified with available information\n\nThis catches issues that predefined expectations might miss.\n\n### Step 5: Read User Notes\n\nIf `{outputs_dir}/user_notes.md` exists:\n1. Read it and note any uncertainties or issues flagged by the executor\n2. Include relevant concerns in the grading output\n3. These may reveal problems even when expectations pass\n\n### Step 6: Critique the Evals\n\nAfter grading, consider whether the evals themselves could be improved. Only surface suggestions when there's a clear gap.\n\nGood suggestions test meaningful outcomes — assertions that are hard to satisfy without actually doing the work correctly. Think about what makes an assertion *discriminating*: it passes when the skill genuinely succeeds and fails when it doesn't.\n\nSuggestions worth raising:\n- An assertion that passed but would also pass for a clearly wrong output (e.g., checking filename existence but not file content)\n- An important outcome you observed — good or bad — that no assertion covers at all\n- An assertion that can't actually be verified from the available outputs\n\nKeep the bar high. The goal is to flag things the eval author would say \"good catch\" about, not to nitpick every assertion.\n\n### Step 7: Write Grading Results\n\nSave results to `{outputs_dir}/../grading.json` (sibling to outputs_dir).\n\n## Grading Criteria\n\n**PASS when**:\n- The transcript or outputs clearly demonstrate the expectation is true\n- Specific evidence can be cited\n- The evidence reflects genuine substance, not just surface compliance (e.g., a file exists AND contains correct content, not just the right filename)\n\n**FAIL when**:\n- No evidence found for the expectation\n- Evidence contradicts the expectation\n- The expectation cannot be verified from available information\n- The evidence is superficial — the assertion is technically satisfied but the underlying task outcome is wrong or incomplete\n- The output appears to meet the assertion by coincidence rather than by actually doing the work\n\n**When uncertain**: The burden of proof to pass is on the expectation.\n\n### Step 8: Read Executor Metrics and Timing\n\n1. If `{outputs_dir}/metrics.json` exists, read it and include in grading output\n2. If `{outputs_dir}/../timing.json` exists, read it and include timing data\n\n## Output Format\n\nWrite a JSON file with this structure:\n\n```json\n{\n \"expectations\": [\n {\n \"text\": \"The output includes the name 'John Smith'\",\n \"passed\": true,\n \"evidence\": \"Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'\"\n },\n {\n \"text\": \"The spreadsheet has a SUM formula in cell B10\",\n \"passed\": false,\n \"evidence\": \"No spreadsheet was created. The output was a text file.\"\n },\n {\n \"text\": \"The assistant used the skill's OCR script\",\n \"passed\": true,\n \"evidence\": \"Transcript Step 2 shows: 'Tool: Bash - python ocr_script.py image.png'\"\n }\n ],\n \"summary\": {\n \"passed\": 2,\n \"failed\": 1,\n \"total\": 3,\n \"pass_rate\": 0.67\n },\n \"execution_metrics\": {\n \"tool_calls\": {\n \"Read\": 5,\n \"Write\": 2,\n \"Bash\": 8\n },\n \"total_tool_calls\": 15,\n \"total_steps\": 6,\n \"errors_encountered\": 0,\n \"output_chars\": 12450,\n \"transcript_chars\": 3200\n },\n \"timing\": {\n \"executor_duration_seconds\": 165.0,\n \"grader_duration_seconds\": 26.0,\n \"total_duration_seconds\": 191.0\n },\n \"claims\": [\n {\n \"claim\": \"The form has 12 fillable fields\",\n \"type\": \"factual\",\n \"verified\": true,\n \"evidence\": \"Counted 12 fields in field_info.json\"\n },\n {\n \"claim\": \"All required fields were populated\",\n \"type\": \"quality\",\n \"verified\": false,\n \"evidence\": \"Reference section was left blank despite data being available\"\n }\n ],\n \"user_notes_summary\": {\n \"uncertainties\": [\"Used 2023 data, may be stale\"],\n \"needs_review\": [],\n \"workarounds\": [\"Fell back to text overlay for non-fillable fields\"]\n },\n \"eval_feedback\": {\n \"suggestions\": [\n {\n \"assertion\": \"The output includes the name 'John Smith'\",\n \"reason\": \"A hallucinated document that mentions the name would also pass — consider checking it appears as the primary contact with matching phone and email from the input\"\n },\n {\n \"reason\": \"No assertion checks whether the extracted phone numbers match the input — I observed incorrect numbers in the output that went uncaught\"\n }\n ],\n \"overall\": \"Assertions check presence but not correctness. Consider adding content verification.\"\n }\n}\n```\n\n## Field Descriptions\n\n- **expectations**: Array of graded expectations\n - **text**: The original expectation text\n - **passed**: Boolean - true if expectation passes\n - **evidence**: Specific quote or description supporting the verdict\n- **summary**: Aggregate statistics\n - **passed**: Count of passed expectations\n - **failed**: Count of failed expectations\n - **total**: Total expectations evaluated\n - **pass_rate**: Fraction passed (0.0 to 1.0)\n- **execution_metrics**: Copied from executor's metrics.json (if available)\n - **output_chars**: Total character count of output files (proxy for tokens)\n - **transcript_chars**: Character count of transcript\n- **timing**: Wall clock timing from timing.json (if available)\n - **executor_duration_seconds**: Time spent in executor subagent\n - **total_duration_seconds**: Total elapsed time for the run\n- **claims**: Extracted and verified claims from the output\n - **claim**: The statement being verified\n - **type**: \"factual\", \"process\", or \"quality\"\n - **verified**: Boolean - whether the claim holds\n - **evidence**: Supporting or contradicting evidence\n- **user_notes_summary**: Issues flagged by the executor\n - **uncertainties**: Things the executor wasn't sure about\n - **needs_review**: Items requiring human attention\n - **workarounds**: Places where the skill didn't work as expected\n- **eval_feedback**: Improvement suggestions for the evals (only when warranted)\n - **suggestions**: List of concrete suggestions, each with a `reason` and optionally an `assertion` it relates to\n - **overall**: Brief assessment — can be \"No suggestions, evals look solid\" if nothing to flag\n\n## Guidelines\n\n- **Be objective**: Base verdicts on evidence, not assumptions\n- **Be specific**: Quote the exact text that supports your verdict\n- **Be thorough**: Check both transcript and output files\n- **Be consistent**: Apply the same standard to each expectation\n- **Explain failures**: Make it clear why evidence was insufficient\n- **No partial credit**: Each expectation is pass or fail, not partial\n","isInternal":false,"tokens":2036,"sizeBytes":9049}],"systemPromptSnippet":"\n\n# AGENTS.md\n\nThis file provides guidance to CodeX when working with code in this repository.\n\n## Project Overview\n\nFlock is a multi-provider AI agent desktop application with tool orchestration, sandbox execution, visual workflow, and browser/computer-use capabilities. It's a Rust workspace with a Tauri-based desktop UI (React + TypeScript + Mantine).\n\n## Build & Development Commands\n\n### Rust Backend\n```bash\n# Build the CLI\ncargo build --release\n\n# Run in single-shot mode\ncargo run -- \"your question\"\n\n# Run interactive REPL\ncargo run\n\n# Run with specific provider/profile\ncargo run -- --profile deepseek \"your question\"\n\n# Run tests (workspace-wide)\ncargo test\n\n# Run a single crate's tests\ncargo test -p flock-core\ncargo test -p flock-agent\ncargo test -p flock-tools\ncargo test -p flock-skills\n\n# Run a specific test\ncargo test -p flock-core test_name\n\n# Lint\ncargo clippy --workspace\n```\n\n### Tauri Desktop UI\n```bash\ncd flock-ui\n\n# Install dependencies\nnpm install\n\n# Development server\nnpm run dev\n# or\nnpm run tauri dev\n\n# Build for production\nnpm run build\n# or\nnpm run tauri build\n\n# Lint frontend\nnpm run lint\n```\n\n## Architecture\n\n### Workspace Crates\n\n| Crate | Purpose |\n|-------|---------|\n| `flock-core` | Core types, config, database, IPC interface, crypto, model factory |\n| `flock-agent` | Agent engine, session management, tool execution, memory, graph orchestration, workflow engine |\n| `flock-tools` | Tool registry, built-in tools (Read/Write/Edit/Bash/Grep/Glob), MCP integration, sandbox tools (CodeExecution, SandboxExec, Browser, ComputerUse) |\n| `flock-skills` | Skill discovery, loading, frontmatter parsing, hooks, permissions, bundled skills |\n| `flock-ui/src-tauri` | Tauri desktop app backend (commands, agent state, workspace management) |\n| `workspace-hack` | cargo-hakari build optimization |\n\n### Key Dependencies\n\n- **LangGraph**: Rust reimplementation ([langgraph-rust](https://github.com/Onelevenvy/langgraph-rust)) for agent graph orchestration\n- **LLM Providers**: OpenAI-compatible (default), Anthropic, AWS Bedrock, Google Vertex\n- **Database**: SQLite via sqlx for sessions, conversations, providers, tools, MCP servers, workflows\n- **Async Runtime**: Tokio\n- **Sandbox**: Daytona cloud container runtime for isolated code execution\n- **Browser Automation**: Playwright (Python, runs inside sandbox via CDP)\n\n### Core Architecture Patterns\n\n**Agent Engine** (`flock-agent/src/engine.rs`):\n- `AgentEngine` orchestrates the LLM interaction loop\n- Uses LangGraph `CompiledStateGraph` for execution flow\n- Manages conversation history via SQLite checkpointer\n- Handles tool approval, context compression, plan mode\n\n**Tool System** (`flock-tools/src/lib.rs`):\n- `Tool` trait defines the interface: `name()`, `description()`, `input_schema()`, `execute()`\n- `ToolRegistry` manages all registered tools\n- Built-in tools: Read, Write, Edit, Bash, Grep, Glob\n- Sandbox tools: CodeExecution, SandboxExec, Browser, ComputerUse, RequestHumanAssistance\n- Extensible via MCP servers and custom tool providers\n\n**Workflow Engine** (`flock-agent/src/workflow_graph/`):\n- Visual workflow builder with ReactFlow canvas\n- 10 node types: start, llm, agent, classifier, ifelse, answer, code, human, plugin, parameter_extractor\n- `build_workflow_graph()` compiles ReactFlow JSON into LangGraph `StateGraph`\n- Streaming execution with human-in-the-loop interrupts\n\n**Sandbox System** (`flock-tools/src/daytona/`):\n- Cloud-based Daytona container runtime for isolated execution\n- VNC desktop provisioning (Xvfb, fluxbox, x11vnc, websockify)\n- Lifecycle management: create, destroy, set-public, snapshot\n- Human takeover via VNC with approval flow\n\n**Skills System** (`flock-skills/`):\n- Markdown files with YAML frontmatter in `.flock/skills/`\n- Supports `$ARGUMENTS` substitution, shell execution, conditional activation\n- Two execution contexts: `inline` (same agent) and `fork` (sub-agent)\n- Hot-reload via file watcher\n\n**Memory System** (`flock-agent/src/memory/`):\n- Four types: `user`, `feedback`, `project`, `reference`\n- Stored as Markdown files with frontmatter in `/flock/projects//memory/`\n- `MEMORY.md` index file auto-loaded into system prompt\n\n**IPC Interface** (`flock-core/src/ipc_interface/`):\n- JSON-based protocol for host ↔ agent communication\n- Events (Agent → Host): `ready`, `stream_start`, `text_delta`, `tool_request`, `tool_result`, etc.\n- Commands (Host → Agent): `message`, `stop`, `tool_approve`, `tool_deny`, `set_mode`, `set_config`\n\n**Configuration** (`flock-core/src/config/settings.rs`):\n- Config priority: CLI args/env > project `flock.toml` > global config\n- Supports profiles with inheritance (`extends`)\n- Provider compatibility settings (`ProviderCompat`) for API differences\n\n### Tauri Desktop UI\n\n**Frontend Stack**:\n- React 18 + TypeScript\n- Mantine UI v8 (components, notifications)\n- Zustand (state management)\n- React Query (server state)\n- react-markdown + react-syntax-highlighter (message rendering)\n- i18next (internationalization)\n- ReactFlow (workflow visual editor)\n\n**State Stores** (`flock-ui/src/store/`):\n- `agentStore`: Agent connection status, messages, pending approvals\n- `uiStore`: Theme, sidebar state, active view, file tree\n- `workspaceStore`: Active workspace/conversation, persisted to localStorage\n- `workflowStore`: ReactFlow nodes/edges, execution state, dirty tracking\n\n**Key Views**:\n- `HomeView`: Welcome screen, assistant selection, workspace picker\n- `WorkspaceView`: Chat panel + file tree + optional preview panel (VNC, sandbox runner)\n- `WorkflowEditor`: Visual workflow builder with ReactFlow canvas, node palette, properties panel, execution panel\n- `AssistantPage`: CRUD for custom assistants\n- `SkillsPage`: Tools, MCP servers, and skills management\n- `SettingsModal`: Model providers, basic settings, sandbox settings, system settings\n\n**Tauri Commands** (`flock-ui/src-tauri/src/commands/`):\n- Agent control: `start_agent`, `stop_agent`, `send_message`, `approve_tool`, `deny_tool`\n- Workspace: `list_workspaces`, `create_workspace`, `delete_workspace`\n- Conversations: `list_conversations`, `create_conversation`, `load_conversation_history`\n- Database: `list_providers`, `upsert_provider`, `list_models`, `upsert_model`\n- MCP: `list_mcp_servers`, `upsert_mcp_server`, `test_mcp_server`\n- Skills: `list_skills`, `get_extra_skill_dirs`, `add_extra_skill_dir`\n- Workflow: `list_workflows`, `get_workflow`, `create_workflow`, `update_workflow`, `delete_workflow`, `run_workflow`, `stop_workflow`\n\n### Data Flow\n\n1. User sends message via UI → Tauri command `send_message`\n2. `AgentState` forwards to `AgentEngine::run()`\n3. Engine builds LangGraph, invokes LLM with tools\n4. LLM response streamed as `ProtocolEvent`s via `TauriProtocolEmitter`\n5. Frontend `useEventStream` hook listens to Tauri events, updates `agentStore`\n6. Tool calls require approval (unless auto-approved) → `ToolApprovalInline` component\n7. Approved tools execute → results fed back to LLM for next turn\n\n## Code Style\n\n- Rust edition 2024\n- Workspace lints: `unused = \"allow\"`, `unused_imports = \"allow\"`\n- Do not treat repository-wide `cargo fmt --check` diffs or warnings such as unused imports/variables as blockers unless they are introduced by the current change or explicitly requested. Focus verification on build/test failures and behavior regressions.\n- Error handling: `anyhow` for applications, `thiserror` for libraries\n- Async: Tokio runtime throughout\n- Frontend: ESLint with TypeScript rules, React hooks plugin\n\n## Global Guidelines & Core Principles\n\nWhen modifying the codebase or interacting with the user, please strictly adhere to the following rules:\n\n1. **Git Commit Format**: Upon completing a task, always output a single-line English `git commit` message beginning with standard prefixes (`feat:`, `fix:`, `chore:`, etc.). Provide the message text clearly, but **DO NOT** execute git commit commands or auto-submit.\n2. **State & Logic Abstraction**: Always separate state management and command logic from UI components. Place custom Hooks in `src/hooks` and Zustand stores in `src/store` rather than defining them inside component files.\n3. **Strict i18n Internationalization**: All newly introduced UI components must leverage `i18next` for user-facing texts (e.g. tooltips, titles, toast notifications, confirm boxes). Do not hardcode Chinese or English strings inside JSX.\n4. **Component Granularity**: Strive for modularity. Always split large layout or page files into focused, localized sub-components (placed in a nested `/components` folder). Avoid keeping all view code inside single massive `index.tsx` files.\n5. **UI Specification Compliance**: Ensure any new styling aligns with the design tokens defined in `docs/resources/ui_specification.md` (e.g. cold gray colors, `--flock-accent` solid blue branding, standard monospace typographies, and hover-card transitions) in both dark and light modes.\n\n\n# CLAUDE.md\n\nThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.\n\n## Project Overview\n\nFlock is a multi-provider AI agent desktop application with tool orchestration, sandbox execution, visual workflow, and browser/computer-use capabilities. It's a Rust workspace with a Tauri-based desktop UI (React + TypeScript + Mantine).\n\n## Build & Development Commands\n\n### Rust Backend\n```bash\n# Build the CLI\ncargo build --release\n\n# Run in single-shot mode\ncargo run -- \"your question\"\n\n# Run interactive REPL\ncargo run\n\n# Run with specific provider/profile\ncargo run -- --profile deepseek \"your question\"\n\n# Run tests (workspace-wide)\ncargo test\n\n# Run a single crate's tests\ncargo test -p flock-core\ncargo test -p flock-agent\ncargo test -p flock-tools\ncargo test -p flock-skills\n\n# Run a specific test\ncargo test -p flock-core test_name\n\n# Lint\ncargo clippy --workspace\n```\n\n### Tauri Desktop UI\n```bash\ncd flock-ui\n\n# Install dependencies\nnpm install\n\n# Development server\nnpm run dev\n# or\nnpm run tauri dev\n\n# Build for production\nnpm run build\n# or\nnpm run tauri build\n\n# Lint frontend\nnpm run lint\n```\n\n## Architecture\n\n### Workspace Crates\n\n| Crate | Purpose |\n|-------|---------|\n| `flock-core` | Core types, config, database, IPC interface, crypto, model factory |\n| `flock-agent` | Agent engine, session management, tool execution, memory, graph orchestration, workflow engine |\n| `flock-tools` | Tool registry, built-in tools (Read/Write/Edit/Bash/Grep/Glob), MCP integration, sandbox tools (CodeExecution, SandboxExec, Browser, ComputerUse) |\n| `flock-skills` | Skill discovery, loading, frontmatter parsing, hooks, permissions, bundled skills |\n| `flock-ui/src-tauri` | Tauri desktop app backend (commands, agent state, workspace management) |\n| `workspace-hack` | cargo-hakari build optimization |\n\n### Key Dependencies\n\n- **LangGraph**: Rust reimplementation ([langgraph-rust](https://github.com/Onelevenvy/langgraph-rust)) for agent graph orchestration\n- **LLM Providers**: OpenAI-compatible (default), Anthropic, AWS Bedrock, Google Vertex\n- **Database**: SQLite via sqlx for sessions, conversations, providers, tools, MCP servers, workflows\n- **Async Runtime**: Tokio\n- **Sandbox**: Daytona cloud container runtime for isolated code execution\n- **Browser Automation**: Playwright (Python, runs inside sandbox via CDP)\n\n### Core Architecture Patterns\n\n**Agent Engine** (`flock-agent/src/engine.rs`):\n- `AgentEngine` orchestrates the LLM interaction loop\n- Uses LangGraph `CompiledStateGraph` for execution flow\n- Manages conversation history via SQLite checkpointer\n- Handles tool approval, context compression, plan mode\n\n**Tool System** (`flock-tools/src/lib.rs`):\n- `Tool` trait defines the interface: `name()`, `description()`, `input_schema()`, `execute()`\n- `ToolRegistry` manages all registered tools\n- Built-in tools: Read, Write, Edit, Bash, Grep, Glob\n- Sandbox tools: CodeExecution, SandboxExec, Browser, ComputerUse, RequestHumanAssistance\n- Extensible via MCP servers and custom tool providers\n\n**Workflow Engine** (`flock-agent/src/workflow_graph/`):\n- Visual workflow builder with ReactFlow canvas\n- 10 node types: start, llm, agent, classifier, ifelse, answer, code, human, plugin, parameter_extractor\n- `build_workflow_graph()` compiles ReactFlow JSON into LangGraph `StateGraph`\n- Streaming execution with human-in-the-loop interrupts\n\n**Sandbox System** (`flock-tools/src/daytona/`):\n- Cloud-based Daytona container runtime for isolated execution\n- VNC desktop provisioning (Xvfb, fluxbox, x11vnc, websockify)\n- Lifecycle management: create, destroy, set-public, snapshot\n- Human takeover via VNC with approval flow\n\n**Skills System** (`flock-skills/`):\n- Markdown files with YAML frontmatter in `.flock/skills/`\n- Supports `$ARGUMENTS` substitution, shell execution, conditional activation\n- Two execution contexts: `inline` (same agent) and `fork` (sub-agent)\n- Hot-reload via file watcher\n\n**Memory System** (`flock-agent/src/memory/`):\n- Four types: `user`, `feedback`, `project`, `reference`\n- Stored as Markdown files with frontmatter in `/flock/projects//memory/`\n- `MEMORY.md` index file auto-loaded into system prompt\n\n**IPC Interface** (`flock-core/src/ipc_interface/`):\n- JSON-based protocol for host ↔ agent communication\n- Events (Agent → Host): `ready`, `stream_start`, `text_delta`, `tool_request`, `tool_result`, etc.\n- Commands (Host → Agent): `message`, `stop`, `tool_approve`, `tool_deny`, `set_mode`, `set_config`\n\n**Configuration** (`flock-core/src/config/settings.rs`):\n- Config priority: CLI args/env > project `flock.toml` > global config\n- Supports profiles with inheritance (`extends`)\n- Provider compatibility settings (`ProviderCompat`) for API differences\n\n### Tauri Desktop UI\n\n**Frontend Stack**:\n- React 18 + TypeScript\n- Mantine UI v8 (components, notifications)\n- Zustand (state management)\n- React Query (server state)\n- react-markdown + react-syntax-highlighter (message rendering)\n- i18next (internationalization)\n- ReactFlow (workflow visual editor)\n\n**State Stores** (`flock-ui/src/store/`):\n- `agentStore`: Agent connection status, messages, pending approvals\n- `uiStore`: Theme, sidebar state, active view, file tree\n- `workspaceStore`: Active workspace/conversation, persisted to localStorage\n- `workflowStore`: ReactFlow nodes/edges, execution state, dirty tracking\n\n**Key Views**:\n- `HomeView`: Welcome screen, assistant selection, workspace picker\n- `WorkspaceView`: Chat panel + file tree + optional preview panel (VNC, sandbox runner)\n- `WorkflowEditor`: Visual workflow builder with ReactFlow canvas, node palette, properties panel, execution panel\n- `AssistantPage`: CRUD for custom assistants\n- `SkillsPage`: Tools, MCP servers, and skills management\n- `SettingsModal`: Model providers, basic settings, sandbox settings, system settings\n\n**Tauri Commands** (`flock-ui/src-tauri/src/commands/`):\n- Agent control: `start_agent`, `stop_agent`, `send_message`, `approve_tool`, `deny_tool`\n- Workspace: `list_workspaces`, `create_workspace`, `delete_workspace`\n- Conversations: `list_conversations`, `create_conversation`, `load_conversation_history`\n- Database: `list_providers`, `upsert_provider`, `list_models`, `upsert_model`\n- MCP: `list_mcp_servers`, `upsert_mcp_server`, `test_mcp_server`\n- Skills: `list_skills`, `get_extra_skill_dirs`, `add_extra_skill_dir`\n- Workflow: `list_workflows`, `get_workflow`, `create_workflow`, `update_workflow`, `delete_workflow`, `run_workflow`, `stop_workflow`\n\n### Data Flow\n\n1. User sends message via UI → Tauri command `send_message`\n2. `AgentState` forwards to `AgentEngine::run()`\n3. Engine builds LangGraph, invokes LLM with tools\n4. LLM response streamed as `ProtocolEvent`s via `TauriProtocolEmitter`\n5. Frontend `useEventStream` hook listens to Tauri events, updates `agentStore`\n6. Tool calls require approval (unless auto-approved) → `ToolApprovalInline` component\n7. Approved tools execute → results fed back to LLM for next turn\n\n## Code Style\n\n- Rust edition 2024\n- Workspace lints: `unused = \"allow\"`, `unused_imports = \"allow\"`\n- Do not treat repository-wide `cargo fmt --check` diffs or warnings such as unused imports/variables as blockers unless they are introduced by the current change or explicitly requested. Focus verification on build/test failures and behavior regressions.\n- Error handling: `anyhow` for applications, `thiserror` for libraries\n- Async: Tokio runtime throughout\n- Frontend: ESLint with TypeScript rules, React hooks plugin\n\n\n## Global Guidelines & Core Principles\n\nWhen modifying the codebase or interacting with the user, please strictly adhere to the following rules:\n\n1. **Git Commit Format**: Upon completing a task, always output a single-line English `git commit` message beginning with standard prefixes (`feat:`, `fix:`, `chore:`, etc.). Provide the message text clearly, but **DO NOT** execute git commit commands or auto-submit.\n2. **State & Logic Abstraction**: Always separate state management and command logic from UI components. Place custom Hooks in `src/hooks` and Zustand stores in `src/store` rather than defining them inside component files.\n3. **Strict i18n Internationalization**: All newly introduced UI components must leverage `i18next` for user-facing texts (e.g. tooltips, titles, toast notifications, confirm boxes). Do not hardcode Chinese or English strings inside JSX.\n4. **Component Granularity**: Strive for modularity. Always split large layout or page files into focused, localized sub-components (placed in a nested `/components` folder). Avoid keeping all view code inside single massive `index.tsx` files.\n5. **UI Specification Compliance**: Ensure any new styling aligns with the design tokens defined in `docs/resources/ui_specification.md` (e.g. cold gray colors, `--flock-accent` solid blue branding, standard monospace typographies, and hover-card transitions) in both dark and light modes.\n\n\n---\nname: frontend-design\ndescription: Create distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, or applications. Generates creative, polished code that avoids generic AI aesthetics.\nlicense: Complete terms in LICENSE.txt\n---\n\nThis skill guides creation of distinctive, production-grade frontend interfaces that avoid generic \"AI slop\" aesthetics. Implement real working code with exceptional attention to aesthetic details and creative choices.\n\nThe user provides frontend requirements: a component, page, application, or interface to build. They may include context about the purpose, audience, or technical constraints.\n\n## Design Thinking\n\nBefore coding, understand the context and commit to a BOLD aesthetic direction:\n- **Purpose**: What problem does this interface solve? Who uses it?\n- **Tone**: Pick an extreme: brutally minimal, maximalist chaos, retro-futuristic, organic/natural, luxury/refined, playful/toy-like, editorial/magazine, brutalist/raw, art deco/geometric, soft/pastel, industrial/utilitarian, etc. There are so many flavors to choose from. Use these for inspiration but design one that is true to the aesthetic direction.\n- **Constraints**: Technical requirements (framework, performance, accessibility).\n- **Differentiation**: What makes this UNFORGETTABLE? What's the one thing someone will remember?\n\n**CRITICAL**: Choose a clear conceptual direction and execute it with precision. Bold maximalism and refined minimalism both work - the key is intentionality, not intensity.\n\nThen implement working code (HTML/CSS/JS, React, Vue, etc.) that is:\n- Production-grade and functional\n- Visually striking and memorable\n- Cohesive with a clear aesthetic point-of-view\n- Meticulously refined in every detail\n\n## Frontend Aesthetics Guidelines\n\nFocus on:\n- **Typography**: Choose fonts that are beautiful, unique, and interesting. Avoid generic fonts like Arial and Inter; opt instead for distinctive choices that elevate the frontend's aesthetics; unexpected, characterful font choices. Pair a distinctive display font with a refined body font.\n- **Color & Theme**: Commit to a cohesive aesthetic. Use CSS variables for consistency. Dominant colors with sharp accents outperform timid, evenly-distributed palettes.\n- **Motion**: Use animations for effects and micro-interactions. Prioritize CSS-only solutions for HTML. Use Motion library for React when available. Focus on high-impact moments: one well-orchestrated page load with staggered reveals (animation-delay) creates more delight than scattered micro-interactions. Use scroll-triggering and hover states that surprise.\n- **Spatial Composition**: Unexpected layouts. Asymmetry. Overlap. Diagonal flow. Grid-breaking elements. Generous negative space OR controlled density.\n- **Backgrounds & Visual Details**: Create atmosphere and depth rather than defaulting to solid colors. Add contextual effects and textures that match the overall aesthetic. Apply creative forms like gradient meshes, noise textures, geometric patterns, layered transparencies, dramatic shadows, decorative borders, custom cursors, and grain overlays.\n\nNEVER use generic AI-generated aesthetics like overused font families (Inter, Roboto, Arial, system fonts), cliched color schemes (particularly purple gradients on white backgrounds), predictable layouts and component patterns, and cookie-cutter design that lacks context-specific character.\n\nInterpret creatively and make unexpected choices that feel genuinely designed for the context. No design should be the same. Vary between light and dark themes, different fonts, different aesthetics. NEVER converge on common choices (Space Grotesk, for example) across generations.\n\n**IMPORTANT**: Match implementation complexity to the aesthetic vision. Maximalist designs need elaborate code with extensive animations and effects. Minimalist or refined designs need restraint, precision, and careful attention to spacing, typography, and subtle details. Elegance comes from executing the vision well.\n\nRemember: Claude is capable of extraordinary creative work. Don't hold back, show what can truly be created when thinking outside the box and committing fully to a distinctive vision.\n\n\n{\n \"name\": \"karpathy\",\n \"description\": \"Coding principles for simplicity, clarity, and surgical precision. Use when you want Claude to think before coding, keep solutions minimal, make surgical changes, and define verifiable success criteria. Triggers on: karpathy mode, think before coding, simplicity first, surgical changes.\",\n \"category\": \"devops\",\n \"canonical_category\": \"devops\",\n \"repository\": \"majiayu000/claude-skill-registry\",\n \"repository_url\": \"https://github.com/majiayu000/claude-skill-registry\",\n \"author\": \"majiayu000\",\n \"author_avatar\": \"https://github.com/majiayu000.png\",\n \"file_path\": \"skills/other/karpathy/SKILL.md\",\n \"source\": \"github_code_search\",\n \"stars\": 0,\n \"quality_score\": 45,\n \"best_practices_score\": 55,\n \"skill_level\": 2,\n \"skill_level_label\": \"instructions\",\n \"has_scripts\": false,\n \"has_extra_files\": true,\n \"downloads\": 0,\n \"content_hash\": \"ec43cf3b4ae428984a1ad36d4814bf6f95e590ada0c997fab9f1ecb4c89b2886\",\n \"indexed_at\": \"2026-02-25T19:40:17.443Z\",\n \"synced_at\": \"2026-02-28T04:09:42.439Z\",\n \"omni_registry_url\": \"https://omni-skill-registry.omniroute.online/#/skill/27fec53067f2a7ff89a65128ed8a407f2e5f7a042d2891790054748048e9f33b\",\n \"install_command\": \"mkdir -p .claude/skills/karpathy && curl -sL \\\"https://raw.githubusercontent.com/majiayu000/claude-skill-registry/main/skills/other/karpathy/SKILL.md\\\" > .claude/skills/karpathy/SKILL.md\",\n \"raw_url\": \"https://raw.githubusercontent.com/majiayu000/claude-skill-registry/main/skills/other/karpathy/SKILL.md\"\n}\n\n\n---\nname: karpathy\ndescription: 'Coding principles for simplicity, clarity, and surgical precision. Use when you want Claude to think before coding, keep solutions minimal, make surgical changes, and define verifiable success criteria. Triggers on: karpathy mode, think before coding, simplicity first, surgical changes.'\n---\n\n# Karpathy Coding Principles\n\nPrinciples for writing clean, minimal, and purposeful code.\n\n---\n\n## 1. Think Before Coding\n\n**Don't assume. Don't hide confusion. Surface tradeoffs.**\n\nBefore implementing:\n\n- State your assumptions explicitly. If uncertain, ask.\n- If multiple interpretations exist, present them - don't pick silently.\n- If a simpler approach exists, say so. Push back when warranted.\n- If something is unclear, stop. Name what's confusing. Ask.\n\n---\n\n## 2. Simplicity First\n\n**Minimum code that solves the problem. Nothing speculative.**\n\n- No features beyond what was asked.\n- No abstractions for single-use code.\n- No \"flexibility\" or \"configurability\" that wasn't requested.\n- No error handling for impossible scenarios.\n- If you write 200 lines and it could be 50, rewrite it.\n\nAsk yourself: \"Would a senior engineer say this is overcomplicated?\" If yes, simplify.\n\n---\n\n## 3. Surgical Changes\n\n**Touch only what you must. Clean up only your own mess.**\n\nWhen editing existing code:\n\n- Don't \"improve\" adjacent code, comments, or formatting.\n- Don't refactor things that aren't broken.\n- Match existing style, even if you'd do it differently.\n- If you notice unrelated dead code, mention it - don't delete it.\n\nWhen your changes create orphans:\n\n- Remove imports/variables/functions that YOUR changes made unused.\n- Don't remove pre-existing dead code unless asked.\n\nThe test: Every changed line should trace directly to the user's request.\n\n---\n\n## 4. Goal-Driven Execution\n\n**Define success criteria. Loop until verified.**\n\nTransform tasks into verifiable goals:\n\n- \"Add validation\" -> \"Write tests for invalid inputs, then make them pass\"\n- \"Fix the bug\" -> \"Write a test that reproduces it, then make it pass\"\n- \"Refactor X\" -> \"Ensure tests pass before and after\"\n\nFor multi-step tasks, state a brief plan:\n\n```\n1. [Step] -> verify: [check]\n2. [Step] -> verify: [check]\n3. [Step] -> verify: [check]\n```\n\nStrong success criteria let you loop independently. Weak criteria (\"make it work\") require constant clarification.\n\n\n# Examples: Planning with Files in Action\n\n## Example 1: Research Task\n\n**User Request:** \"Research the benefits of morning exercise and write a summary\"\n\n### Loop 1: Create Plan\n```bash\nWrite task_plan.md\n```\n\n```markdown\n# Task Plan: Morning Exercise Benefits Research\n\n## Goal\nCreate a research summary on the benefits of morning exercise.\n\n## Phases\n- [ ] Phase 1: Create this plan ✓\n- [ ] Phase 2: Search and gather sources\n- [ ] Phase 3: Synthesize findings\n- [ ] Phase 4: Deliver summary\n\n## Key Questions\n1. What are the physical health benefits?\n2. What are the mental health benefits?\n3. What scientific studies support this?\n\n## Status\n**Currently in Phase 1** - Creating plan\n```\n\n### Loop 2: Research\n```bash\nRead task_plan.md # Refresh goals\nWebSearch \"morning exercise benefits\"\nWrite notes.md # Store findings\nEdit task_plan.md # Mark Phase 2 complete\n```\n\n### Loop 3: Synthesize\n```bash\nRead task_plan.md # Refresh goals\nRead notes.md # Get findings\nWrite morning_exercise_summary.md\nEdit task_plan.md # Mark Phase 3 complete\n```\n\n### Loop 4: Deliver\n```bash\nRead task_plan.md # Verify complete\nDeliver morning_exercise_summary.md\n```\n\n---\n\n## Example 2: Bug Fix Task\n\n**User Request:** \"Fix the login bug in the authentication module\"\n\n### task_plan.md\n```markdown\n# Task Plan: Fix Login Bug\n\n## Goal\nIdentify and fix the bug preventing successful login.\n\n## Phases\n- [x] Phase 1: Understand the bug report ✓\n- [x] Phase 2: Locate relevant code ✓\n- [ ] Phase 3: Identify root cause (CURRENT)\n- [ ] Phase 4: Implement fix\n- [ ] Phase 5: Test and verify\n\n## Key Questions\n1. What error message appears?\n2. Which file handles authentication?\n3. What changed recently?\n\n## Decisions Made\n- Auth handler is in src/auth/login.ts\n- Error occurs in validateToken() function\n\n## Errors Encountered\n- [Initial] TypeError: Cannot read property 'token' of undefined\n → Root cause: user object not awaited properly\n\n## Status\n**Currently in Phase 3** - Found root cause, preparing fix\n```\n\n---\n\n## Example 3: Feature Development\n\n**User Request:** \"Add a dark mode toggle to the settings page\"\n\n### The 3-File Pattern in Action\n\n**task_plan.md:**\n```markdown\n# Task Plan: Dark Mode Toggle\n\n## Goal\nAdd functional dark mode toggle to settings.\n\n## Phases\n- [x] Phase 1: Research existing theme system ✓\n- [x] Phase 2: Design implementation approach ✓\n- [ ] Phase 3: Implement toggle component (CURRENT)\n- [ ] Phase 4: Add theme switching logic\n- [ ] Phase 5: Test and polish\n\n## Decisions Made\n- Using CSS custom properties for theme\n- Storing preference in localStorage\n- Toggle component in SettingsPage.tsx\n\n## Status\n**Currently in Phase 3** - Building toggle component\n```\n\n**notes.md:**\n```markdown\n# Notes: Dark Mode Implementation\n\n## Existing Theme System\n- Located in: src/styles/theme.ts\n- Uses: CSS custom properties\n- Current themes: light only\n\n## Files to Modify\n1. src/styles/theme.ts - Add dark theme colors\n2. src/components/SettingsPage.tsx - Add toggle\n3. src/hooks/useTheme.ts - Create new hook\n4. src/App.tsx - Wrap with ThemeProvider\n\n## Color Decisions\n- Dark background: #1a1a2e\n- Dark surface: #16213e\n- Dark text: #eaeaea\n```\n\n**dark_mode_implementation.md:** (deliverable)\n```markdown\n# Dark Mode Implementation\n\n## Changes Made\n\n### 1. Added dark theme colors\nFile: src/styles/theme.ts\n...\n\n### 2. Created useTheme hook\nFile: src/hooks/useTheme.ts\n...\n```\n\n---\n\n## Example 4: Error Recovery Pattern\n\nWhen something fails, DON'T hide it:\n\n### Before (Wrong)\n```\nAction: Read config.json\nError: File not found\nAction: Read config.json # Silent retry\nAction: Read config.json # Another retry\n```\n\n### After (Correct)\n```\nAction: Read config.json\nError: File not found\n\n# Update task_plan.md:\n## Errors Encountered\n- config.json not found → Will create default config\n\nAction: Write config.json (default config)\nAction: Read config.json\nSuccess!\n```\n\n---\n\n## The Read-Before-Decide Pattern\n\n**Always read your plan before major decisions:**\n\n```\n[Many tool calls have happened...]\n[Context is getting long...]\n[Original goal might be forgotten...]\n\n→ Read task_plan.md # This brings goals back into attention!\n→ Now make the decision # Goals are fresh in context\n```\n\nThis is why Manus can handle ~50 tool calls without losing track. The plan file acts as a \"goal refresh\" mechanism.\n\n\n# Reference: Manus Context Engineering Principles\n\nThis skill is based on context engineering principles from Manus, the AI agent company acquired by Meta for $2 billion in December 2025.\n\n## The 6 Manus Principles\n\n### Principle 1: Design Around KV-Cache\n\n> \"KV-cache hit rate is THE single most important metric for production AI agents.\"\n\n**Statistics:**\n- ~100:1 input-to-output token ratio\n- Cached tokens: $0.30/MTok vs Uncached: $3/MTok\n- 10x cost difference!\n\n**Implementation:**\n- Keep prompt prefixes STABLE (single-token change invalidates cache)\n- NO timestamps in system prompts\n- Make context APPEND-ONLY with deterministic serialization\n\n### Principle 2: Mask, Don't Remove\n\nDon't dynamically remove tools (breaks KV-cache). Use logit masking instead.\n\n**Best Practice:** Use consistent action prefixes (e.g., `browser_`, `shell_`, `file_`) for easier masking.\n\n### Principle 3: Filesystem as External Memory\n\n> \"Markdown is my 'working memory' on disk.\"\n\n**The Formula:**\n```\nContext Window = RAM (volatile, limited)\nFilesystem = Disk (persistent, unlimited)\n```\n\n**Compression Must Be Restorable:**\n- Keep URLs even if web content is dropped\n- Keep file paths when dropping document contents\n- Never lose the pointer to full data\n\n### Principle 4: Manipulate Attention Through Recitation\n\n> \"Creates and updates todo.md throughout tasks to push global plan into model's recent attention span.\"\n\n**Problem:** After ~50 tool calls, models forget original goals (\"lost in the middle\" effect).\n\n**Solution:** Re-read `task_plan.md` before each decision. Goals appear in the attention window.\n\n```\nStart of context: [Original goal - far away, forgotten]\n...many tool calls...\nEnd of context: [Recently read task_plan.md - gets ATTENTION!]\n```\n\n### Principle 5: Keep the Wrong Stuff In\n\n> \"Leave the wrong turns in the context.\"\n\n**Why:**\n- Failed actions with stack traces let model implicitly update beliefs\n- Reduces mistake repetition\n- Error recovery is \"one of the clearest signals of TRUE agentic behavior\"\n\n### Principle 6: Don't Get Few-Shotted\n\n> \"Uniformity breeds fragility.\"\n\n**Problem:** Repetitive action-observation pairs cause drift and hallucination.\n\n**Solution:** Introduce controlled variation:\n- Vary phrasings slightly\n- Don't copy-paste patterns blindly\n- Recalibrate on repetitive tasks\n\n---\n\n## The 3 Context Engineering Strategies\n\nBased on Lance Martin's analysis of Manus architecture.\n\n### Strategy 1: Context Reduction\n\n**Compaction:**\n```\nTool calls have TWO representations:\n├── FULL: Raw tool content (stored in filesystem)\n└── COMPACT: Reference/file path only\n\nRULES:\n- Apply compaction to STALE (older) tool results\n- Keep RECENT results FULL (to guide next decision)\n```\n\n**Summarization:**\n- Applied when compaction reaches diminishing returns\n- Generated using full tool results\n- Creates standardized summary objects\n\n### Strategy 2: Context Isolation (Multi-Agent)\n\n**Architecture:**\n```\n┌─────────────────────────────────┐\n│ PLANNER AGENT │\n│ └─ Assigns tasks to sub-agents │\n├─────────────────────────────────┤\n│ KNOWLEDGE MANAGER │\n│ └─ Reviews conversations │\n│ └─ Determines filesystem store │\n├─────────────────────────────────┤\n│ EXECUTOR SUB-AGENTS │\n│ └─ Perform assigned tasks │\n│ └─ Have own context windows │\n└─────────────────────────────────┘\n```\n\n**Key Insight:** Manus originally used `todo.md` for task planning but found ~33% of actions were spent updating it. Shifted to dedicated planner agent calling executor sub-agents.\n\n### Strategy 3: Context Offloading\n\n**Tool Design:**\n- Use <20 atomic functions total\n- Store full results in filesystem, not context\n- Use `glob` and `grep` for searching\n- Progressive disclosure: load information only as needed\n\n---\n\n## The Agent Loop\n\nManus operates in a continuous 7-step loop:\n\n```\n┌─────────────────────────────────────────┐\n│ 1. ANALYZE CONTEXT │\n│ - Understand user intent │\n│ - Assess current state │\n│ - Review recent observations │\n├─────────────────────────────────────────┤\n│ 2. THINK │\n│ - Should I update the plan? │\n│ - What's the next logical action? │\n│ - Are there blockers? │\n├─────────────────────────────────────────┤\n│ 3. SELECT TOOL │\n│ - Choose ONE tool │\n│ - Ensure parameters available │\n├─────────────────────────────────────────┤\n│ 4. EXECUTE ACTION │\n│ - Tool runs in sandbox │\n├─────────────────────────────────────────┤\n│ 5. RECEIVE OBSERVATION │\n│ - Result appended to context │\n├─────────────────────────────────────────┤\n│ 6. ITERATE │\n│ - Return to step 1 │\n│ - Continue until complete │\n├─────────────────────────────────────────┤\n│ 7. DELIVER OUTCOME │\n│ - Send results to user │\n│ - Attach all relevant files │\n└─────────────────────────────────────────┘\n```\n\n---\n\n## File Types Manus Creates\n\n| File | Purpose | When Created | When Updated |\n|------|---------|--------------|--------------|\n| `task_plan.md` | Phase tracking, progress | Task start | After completing phases |\n| `findings.md` | Discoveries, decisions | After ANY discovery | After viewing images/PDFs |\n| `progress.md` | Session log, what's done | At breakpoints | Throughout session |\n| Code files | Implementation | Before execution | After errors |\n\n---\n\n## Critical Constraints\n\n- **Single-Action Execution:** ONE tool call per turn. No parallel execution.\n- **Plan is Required:** Agent must ALWAYS know: goal, current phase, remaining phases\n- **Files are Memory:** Context = volatile. Filesystem = persistent.\n- **Never Repeat Failures:** If action failed, next action MUST be different\n- **Communication is a Tool:** Message types: `info` (progress), `ask` (blocking), `result` (terminal)\n\n---\n\n## Manus Statistics\n\n| Metric | Value |\n|--------|-------|\n| Average tool calls per task | ~50 |\n| Input-to-output token ratio | 100:1 |\n| Acquisition price | $2 billion |\n| Time to $100M revenue | 8 months |\n| Framework refactors since launch | 5 times |\n\n---\n\n## Key Quotes\n\n> \"Context window = RAM (volatile, limited). Filesystem = Disk (persistent, unlimited). Anything important gets written to disk.\"\n\n> \"if action_failed: next_action != same_action. Track what you tried. Mutate the approach.\"\n\n> \"Error recovery is one of the clearest signals of TRUE agentic behavior.\"\n\n> \"KV-cache hit rate is the single most important metric for a production-stage AI agent.\"\n\n> \"Leave the wrong turns in the context.\"\n\n---\n\n## Source\n\nBased on Manus's official context engineering documentation:\nhttps://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus\n\n\n---\nname: planning-with-files\nversion: \"2.10.0\"\ndescription: Implements Manus-style file-based planning for complex tasks. Creates task_plan.md, findings.md, and progress.md. Use when starting complex multi-step tasks, research projects, or any task requiring >5 tool calls. Now with automatic session recovery after /clear.\nuser-invocable: true\nallowed-tools:\n - Read\n - Write\n - Edit\n - Bash\n - Glob\n - Grep\n - WebFetch\n - WebSearch\nhooks:\n PreToolUse:\n - matcher: \"Write|Edit|Bash|Read|Glob|Grep\"\n hooks:\n - type: command\n command: \"cat task_plan.md 2>/dev/null | head -30 || true\"\n PostToolUse:\n - matcher: \"Write|Edit\"\n hooks:\n - type: command\n command: \"echo '[planning-with-files] File updated. If this completes a phase, update task_plan.md status.'\"\n Stop:\n - hooks:\n - type: command\n command: |\n SCRIPT_DIR=\"${CLAUDE_PLUGIN_ROOT:-$HOME/.claude/plugins/planning-with-files}/scripts\"\n\n IS_WINDOWS=0\n if [ \"${OS-}\" = \"Windows_NT\" ]; then\n IS_WINDOWS=1\n else\n UNAME_S=\"$(uname -s 2>/dev/null || echo '')\"\n case \"$UNAME_S\" in\n CYGWIN*|MINGW*|MSYS*) IS_WINDOWS=1 ;;\n esac\n fi\n\n if [ \"$IS_WINDOWS\" -eq 1 ]; then\n if command -v pwsh >/dev/null 2>&1; then\n pwsh -ExecutionPolicy Bypass -File \"$SCRIPT_DIR/check-complete.ps1\" 2>/dev/null ||\n powershell -ExecutionPolicy Bypass -File \"$SCRIPT_DIR/check-complete.ps1\" 2>/dev/null ||\n sh \"$SCRIPT_DIR/check-complete.sh\"\n else\n powershell -ExecutionPolicy Bypass -File \"$SCRIPT_DIR/check-complete.ps1\" 2>/dev/null ||\n sh \"$SCRIPT_DIR/check-complete.sh\"\n fi\n else\n sh \"$SCRIPT_DIR/check-complete.sh\"\n fi\n---\n\n# Planning with Files\n\nWork like Manus: Use persistent markdown files as your \"working memory on disk.\"\n\n## FIRST: Check for Previous Session (v2.2.0)\n\n**Before starting work**, check for unsynced context from a previous session:\n\n```bash\n# Linux/macOS\n$(command -v python3 || command -v python) ${CLAUDE_PLUGIN_ROOT}/scripts/session-catchup.py \"$(pwd)\"\n```\n\n```powershell\n# Windows PowerShell\n& (Get-Command python -ErrorAction SilentlyContinue).Source \"$env:USERPROFILE\\.claude\\skills\\planning-with-files\\scripts\\session-catchup.py\" (Get-Location)\n```\n\nIf catchup report shows unsynced context:\n1. Run `git diff --stat` to see actual code changes\n2. Read current planning files\n3. Update planning files based on catchup + git diff\n4. Then proceed with task\n\n## Important: Where Files Go\n\n- **Templates** are in `${CLAUDE_PLUGIN_ROOT}/templates/`\n- **Your planning files** go in **your project directory**\n\n| Location | What Goes There |\n|----------|-----------------|\n| Skill directory (`${CLAUDE_PLUGIN_ROOT}/`) | Templates, scripts, reference docs |\n| Your project directory | `task_plan.md`, `findings.md`, `progress.md` |\n\n## Quick Start\n\nBefore ANY complex task:\n\n1. **Create `task_plan.md`** — Use [templates/task_plan.md](templates/task_plan.md) as reference\n2. **Create `findings.md`** — Use [templates/findings.md](templates/findings.md) as reference\n3. **Create `progress.md`** — Use [templates/progress.md](templates/progress.md) as reference\n4. **Re-read plan before decisions** — Refreshes goals in attention window\n5. **Update after each phase** — Mark complete, log errors\n\n> **Note:** Planning files go in your project root, not the skill installation folder.\n\n## The Core Pattern\n\n```\nContext Window = RAM (volatile, limited)\nFilesystem = Disk (persistent, unlimited)\n\n→ Anything important gets written to disk.\n```\n\n## File Purposes\n\n| File | Purpose | When to Update |\n|------|---------|----------------|\n| `task_plan.md` | Phases, progress, decisions | After each phase |\n| `findings.md` | Research, discoveries | After ANY discovery |\n| `progress.md` | Session log, test results | Throughout session |\n\n## Critical Rules\n\n### 1. Create Plan First\nNever start a complex task without `task_plan.md`. Non-negotiable.\n\n### 2. The 2-Action Rule\n> \"After every 2 view/browser/search operations, IMMEDIATELY save key findings to text files.\"\n\nThis prevents visual/multimodal information from being lost.\n\n### 3. Read Before Decide\nBefore major decisions, read the plan file. This keeps goals in your attention window.\n\n### 4. Update After Act\nAfter completing any phase:\n- Mark phase status: `in_progress` → `complete`\n- Log any errors encountered\n- Note files created/modified\n\n### 5. Log ALL Errors\nEvery error goes in the plan file. This builds knowledge and prevents repetition.\n\n```markdown\n## Errors Encountered\n| Error | Attempt | Resolution |\n|-------|---------|------------|\n| FileNotFoundError | 1 | Created default config |\n| API timeout | 2 | Added retry logic |\n```\n\n### 6. Never Repeat Failures\n```\nif action_failed:\n next_action != same_action\n```\nTrack what you tried. Mutate the approach.\n\n## The 3-Strike Error Protocol\n\n```\nATTEMPT 1: Diagnose & Fix\n → Read error carefully\n → Identify root cause\n → Apply targeted fix\n\nATTEMPT 2: Alternative Approach\n → Same error? Try different method\n → Different tool? Different library?\n → NEVER repeat exact same failing action\n\nATTEMPT 3: Broader Rethink\n → Question assumptions\n → Search for solutions\n → Consider updating the plan\n\nAFTER 3 FAILURES: Escalate to User\n → Explain what you tried\n → Share the specific error\n → Ask for guidance\n```\n\n## Read vs Write Decision Matrix\n\n| Situation | Action | Reason |\n|-----------|--------|--------|\n| Just wrote a file | DON'T read | Content still in context |\n| Viewed image/PDF | Write findings NOW | Multimodal → text before lost |\n| Browser returned data | Write to file | Screenshots don't persist |\n| Starting new phase | Read plan/findings | Re-orient if context stale |\n| Error occurred | Read relevant file | Need current state to fix |\n| Resuming after gap | Read all planning files | Recover state |\n\n## The 5-Question Reboot Test\n\nIf you can answer these, your context management is solid:\n\n| Question | Answer Source |\n|----------|---------------|\n| Where am I? | Current phase in task_plan.md |\n| Where am I going? | Remaining phases |\n| What's the goal? | Goal statement in plan |\n| What have I learned? | findings.md |\n| What have I done? | progress.md |\n\n## When to Use This Pattern\n\n**Use for:**\n- Multi-step tasks (3+ steps)\n- Research tasks\n- Building/creating projects\n- Tasks spanning many tool calls\n- Anything requiring organization\n\n**Skip for:**\n- Simple questions\n- Single-file edits\n- Quick lookups\n\n## Templates\n\nCopy these templates to start:\n\n- [templates/task_plan.md](templates/task_plan.md) — Phase tracking\n- [templates/findings.md](templates/findings.md) — Research storage\n- [templates/progress.md](templates/progress.md) — Session logging\n\n## Scripts\n\nHelper scripts for automation:\n\n- `scripts/init-session.sh` — Initialize all planning files\n- `scripts/check-complete.sh` — Verify all phases complete\n- `scripts/session-catchup.py` — Recover context from previous session (v2.2.0)\n\n## Advanced Topics\n\n- **Manus Principles:** See [reference.md](reference.md)\n- **Real Examples:** See [examples.md](examples.md)\n\n## Anti-Patterns\n\n| Don't | Do Instead |\n|-------|------------|\n| Use TodoWrite for persistence | Create task_plan.md file |\n| State goals once and forget | Re-read plan before decisions |\n| Hide errors and retry silently | Log errors to plan file |\n| Stuff everything in context | Store large content in files |\n| Start executing immediately | Create plan file FIRST |\n| Repeat failed actions | Track attempts, mutate approach |\n| Create files in skill directory | Create files in your project |\n\n\n# Findings & Decisions\n\n\n## Requirements\n\n\n-\n\n## Research Findings\n\n\n-\n\n## Technical Decisions\n\n\n| Decision | Rationale |\n|----------|-----------|\n| | |\n\n## Issues Encountered\n\n\n| Issue | Resolution |\n|-------|------------|\n| | |\n\n## Resources\n\n\n-\n\n## Visual/Browser Findings\n\n\n\n-\n\n---\n\n*Update this file after every 2 view/browser/search operations*\n*This prevents visual information from being lost*\n\n\n# Progress Log\n\n\n## Session: [DATE]\n\n\n### Phase 1: [Title]\n\n- **Status:** in_progress\n- **Started:** [timestamp]\n\n- Actions taken:\n \n -\n- Files created/modified:\n \n -\n\n### Phase 2: [Title]\n\n- **Status:** pending\n- Actions taken:\n -\n- Files created/modified:\n -\n\n## Test Results\n\n| Test | Input | Expected | Actual | Status |\n|------|-------|----------|--------|--------|\n| | | | | |\n\n## Error Log\n\n\n| Timestamp | Error | Attempt | Resolution |\n|-----------|-------|---------|------------|\n| | | 1 | |\n\n## 5-Question Reboot Check\n\n\n| Question | Answer |\n|----------|--------|\n| Where am I? | Phase X |\n| Where am I going? | Remaining phases |\n| What's the goal? | [goal statement] |\n| What have I learned? | See findings.md |\n| What have I done? | See above |\n\n---\n\n*Update after completing each phase or encountering errors*\n\n\n# Task Plan: [Brief Description]\n\n\n## Goal\n\n[One sentence describing the end state]\n\n## Current Phase\n\nPhase 1\n\n## Phases\n\n\n### Phase 1: Requirements & Discovery\n\n- [ ] Understand user intent\n- [ ] Identify constraints and requirements\n- [ ] Document findings in findings.md\n- **Status:** in_progress\n\n\n### Phase 2: Planning & Structure\n\n- [ ] Define technical approach\n- [ ] Create project structure if needed\n- [ ] Document decisions with rationale\n- **Status:** pending\n\n### Phase 3: Implementation\n\n- [ ] Execute the plan step by step\n- [ ] Write code to files before executing\n- [ ] Test incrementally\n- **Status:** pending\n\n### Phase 4: Testing & Verification\n\n- [ ] Verify all requirements met\n- [ ] Document test results in progress.md\n- [ ] Fix any issues found\n- **Status:** pending\n\n### Phase 5: Delivery\n\n- [ ] Review all output files\n- [ ] Ensure deliverables are complete\n- [ ] Deliver to user\n- **Status:** pending\n\n## Key Questions\n\n1. [Question to answer]\n2. [Question to answer]\n\n## Decisions Made\n\n| Decision | Rationale |\n|----------|-----------|\n| | |\n\n## Errors Encountered\n\n| Error | Attempt | Resolution |\n|-------|---------|------------|\n| | 1 | |\n\n## Notes\n\n- Update phase status as you progress: pending → in_progress → complete\n- Re-read this plan before major decisions (attention manipulation)\n- Log ALL errors - they help avoid repetition\n\n\n# JSON Schemas\n\nThis document defines the JSON schemas used by skill-creator.\n\n---\n\n## evals.json\n\nDefines the evals for a skill. Located at `evals/evals.json` within the skill directory.\n\n```json\n{\n \"skill_name\": \"example-skill\",\n \"evals\": [\n {\n \"id\": 1,\n \"prompt\": \"User's example prompt\",\n \"expected_output\": \"Description of expected result\",\n \"files\": [\"evals/files/sample1.pdf\"],\n \"expectations\": [\n \"The output includes X\",\n \"The skill used script Y\"\n ]\n }\n ]\n}\n```\n\n**Fields:**\n- `skill_name`: Name matching the skill's frontmatter\n- `evals[].id`: Unique integer identifier\n- `evals[].prompt`: The task to execute\n- `evals[].expected_output`: Human-readable description of success\n- `evals[].files`: Optional list of input file paths (relative to skill root)\n- `evals[].expectations`: List of verifiable statements\n\n---\n\n## history.json\n\nTracks version progression in Improve mode. Located at workspace root.\n\n```json\n{\n \"started_at\": \"2026-01-15T10:30:00Z\",\n \"skill_name\": \"pdf\",\n \"current_best\": \"v2\",\n \"iterations\": [\n {\n \"version\": \"v0\",\n \"parent\": null,\n \"expectation_pass_rate\": 0.65,\n \"grading_result\": \"baseline\",\n \"is_current_best\": false\n },\n {\n \"version\": \"v1\",\n \"parent\": \"v0\",\n \"expectation_pass_rate\": 0.75,\n \"grading_result\": \"won\",\n \"is_current_best\": false\n },\n {\n \"version\": \"v2\",\n \"parent\": \"v1\",\n \"expectation_pass_rate\": 0.85,\n \"grading_result\": \"won\",\n \"is_current_best\": true\n }\n ]\n}\n```\n\n**Fields:**\n- `started_at`: ISO timestamp of when improvement started\n- `skill_name`: Name of the skill being improved\n- `current_best`: Version identifier of the best performer\n- `iterations[].version`: Version identifier (v0, v1, ...)\n- `iterations[].parent`: Parent version this was derived from\n- `iterations[].expectation_pass_rate`: Pass rate from grading\n- `iterations[].grading_result`: \"baseline\", \"won\", \"lost\", or \"tie\"\n- `iterations[].is_current_best`: Whether this is the current best version\n\n---\n\n## grading.json\n\nOutput from the grader agent. Located at `/grading.json`.\n\n```json\n{\n \"expectations\": [\n {\n \"text\": \"The output includes the name 'John Smith'\",\n \"passed\": true,\n \"evidence\": \"Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'\"\n },\n {\n \"text\": \"The spreadsheet has a SUM formula in cell B10\",\n \"passed\": false,\n \"evidence\": \"No spreadsheet was created. The output was a text file.\"\n }\n ],\n \"summary\": {\n \"passed\": 2,\n \"failed\": 1,\n \"total\": 3,\n \"pass_rate\": 0.67\n },\n \"execution_metrics\": {\n \"tool_calls\": {\n \"Read\": 5,\n \"Write\": 2,\n \"Bash\": 8\n },\n \"total_tool_calls\": 15,\n \"total_steps\": 6,\n \"errors_encountered\": 0,\n \"output_chars\": 12450,\n \"transcript_chars\": 3200\n },\n \"timing\": {\n \"executor_duration_seconds\": 165.0,\n \"grader_duration_seconds\": 26.0,\n \"total_duration_seconds\": 191.0\n },\n \"claims\": [\n {\n \"claim\": \"The form has 12 fillable fields\",\n \"type\": \"factual\",\n \"verified\": true,\n \"evidence\": \"Counted 12 fields in field_info.json\"\n }\n ],\n \"user_notes_summary\": {\n \"uncertainties\": [\"Used 2023 data, may be stale\"],\n \"needs_review\": [],\n \"workarounds\": [\"Fell back to text overlay for non-fillable fields\"]\n },\n \"eval_feedback\": {\n \"suggestions\": [\n {\n \"assertion\": \"The output includes the name 'John Smith'\",\n \"reason\": \"A hallucinated document that mentions the name would also pass\"\n }\n ],\n \"overall\": \"Assertions check presence but not correctness.\"\n }\n}\n```\n\n**Fields:**\n- `expectations[]`: Graded expectations with evidence\n- `summary`: Aggregate pass/fail counts\n- `execution_metrics`: Tool usage and output size (from executor's metrics.json)\n- `timing`: Wall clock timing (from timing.json)\n- `claims`: Extracted and verified claims from the output\n- `user_notes_summary`: Issues flagged by the executor\n- `eval_feedback`: (optional) Improvement suggestions for the evals, only present when the grader identifies issues worth raising\n\n---\n\n## metrics.json\n\nOutput from the executor agent. Located at `/outputs/metrics.json`.\n\n```json\n{\n \"tool_calls\": {\n \"Read\": 5,\n \"Write\": 2,\n \"Bash\": 8,\n \"Edit\": 1,\n \"Glob\": 2,\n \"Grep\": 0\n },\n \"total_tool_calls\": 18,\n \"total_steps\": 6,\n \"files_created\": [\"filled_form.pdf\", \"field_values.json\"],\n \"errors_encountered\": 0,\n \"output_chars\": 12450,\n \"transcript_chars\": 3200\n}\n```\n\n**Fields:**\n- `tool_calls`: Count per tool type\n- `total_tool_calls`: Sum of all tool calls\n- `total_steps`: Number of major execution steps\n- `files_created`: List of output files created\n- `errors_encountered`: Number of errors during execution\n- `output_chars`: Total character count of output files\n- `transcript_chars`: Character count of transcript\n\n---\n\n## timing.json\n\nWall clock timing for a run. Located at `/timing.json`.\n\n**How to capture:** When a subagent task completes, the task notification includes `total_tokens` and `duration_ms`. Save these immediately — they are not persisted anywhere else and cannot be recovered after the fact.\n\n```json\n{\n \"total_tokens\": 84852,\n \"duration_ms\": 23332,\n \"total_duration_seconds\": 23.3,\n \"executor_start\": \"2026-01-15T10:30:00Z\",\n \"executor_end\": \"2026-01-15T10:32:45Z\",\n \"executor_duration_seconds\": 165.0,\n \"grader_start\": \"2026-01-15T10:32:46Z\",\n \"grader_end\": \"2026-01-15T10:33:12Z\",\n \"grader_duration_seconds\": 26.0\n}\n```\n\n---\n\n## benchmark.json\n\nOutput from Benchmark mode. Located at `benchmarks//benchmark.json`.\n\n```json\n{\n \"metadata\": {\n \"skill_name\": \"pdf\",\n \"skill_path\": \"/path/to/pdf\",\n \"executor_model\": \"claude-sonnet-4-20250514\",\n \"analyzer_model\": \"most-capable-model\",\n \"timestamp\": \"2026-01-15T10:30:00Z\",\n \"evals_run\": [1, 2, 3],\n \"runs_per_configuration\": 3\n },\n\n \"runs\": [\n {\n \"eval_id\": 1,\n \"eval_name\": \"Ocean\",\n \"configuration\": \"with_skill\",\n \"run_number\": 1,\n \"result\": {\n \"pass_rate\": 0.85,\n \"passed\": 6,\n \"failed\": 1,\n \"total\": 7,\n \"time_seconds\": 42.5,\n \"tokens\": 3800,\n \"tool_calls\": 18,\n \"errors\": 0\n },\n \"expectations\": [\n {\"text\": \"...\", \"passed\": true, \"evidence\": \"...\"}\n ],\n \"notes\": [\n \"Used 2023 data, may be stale\",\n \"Fell back to text overlay for non-fillable fields\"\n ]\n }\n ],\n\n \"run_summary\": {\n \"with_skill\": {\n \"pass_rate\": {\"mean\": 0.85, \"stddev\": 0.05, \"min\": 0.80, \"max\": 0.90},\n \"time_seconds\": {\"mean\": 45.0, \"stddev\": 12.0, \"min\": 32.0, \"max\": 58.0},\n \"tokens\": {\"mean\": 3800, \"stddev\": 400, \"min\": 3200, \"max\": 4100}\n },\n \"without_skill\": {\n \"pass_rate\": {\"mean\": 0.35, \"stddev\": 0.08, \"min\": 0.28, \"max\": 0.45},\n \"time_seconds\": {\"mean\": 32.0, \"stddev\": 8.0, \"min\": 24.0, \"max\": 42.0},\n \"tokens\": {\"mean\": 2100, \"stddev\": 300, \"min\": 1800, \"max\": 2500}\n },\n \"delta\": {\n \"pass_rate\": \"+0.50\",\n \"time_seconds\": \"+13.0\",\n \"tokens\": \"+1700\"\n }\n },\n\n \"notes\": [\n \"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value\",\n \"Eval 3 shows high variance (50% ± 40%) - may be flaky or model-dependent\",\n \"Without-skill runs consistently fail on table extraction expectations\",\n \"Skill adds 13s average execution time but improves pass rate by 50%\"\n ]\n}\n```\n\n**Fields:**\n- `metadata`: Information about the benchmark run\n - `skill_name`: Name of the skill\n - `timestamp`: When the benchmark was run\n - `evals_run`: List of eval names or IDs\n - `runs_per_configuration`: Number of runs per config (e.g. 3)\n- `runs[]`: Individual run results\n - `eval_id`: Numeric eval identifier\n - `eval_name`: Human-readable eval name (used as section header in the viewer)\n - `configuration`: Must be `\"with_skill\"` or `\"without_skill\"` (the viewer uses this exact string for grouping and color coding)\n - `run_number`: Integer run number (1, 2, 3...)\n - `result`: Nested object with `pass_rate`, `passed`, `total`, `time_seconds`, `tokens`, `errors`\n- `run_summary`: Statistical aggregates per configuration\n - `with_skill` / `without_skill`: Each contains `pass_rate`, `time_seconds`, `tokens` objects with `mean` and `stddev` fields\n - `delta`: Difference strings like `\"+0.50\"`, `\"+13.0\"`, `\"+1700\"`\n- `notes`: Freeform observations from the analyzer\n\n**Important:** The viewer reads these field names exactly. Using `config` instead of `configuration`, or putting `pass_rate` at the top level of a run instead of nested under `result`, will cause the viewer to show empty/zero values. Always reference this schema when generating benchmark.json manually.\n\n---\n\n## comparison.json\n\nOutput from blind comparator. Located at `/comparison-N.json`.\n\n```json\n{\n \"winner\": \"A\",\n \"reasoning\": \"Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.\",\n \"rubric\": {\n \"A\": {\n \"content\": {\n \"correctness\": 5,\n \"completeness\": 5,\n \"accuracy\": 4\n },\n \"structure\": {\n \"organization\": 4,\n \"formatting\": 5,\n \"usability\": 4\n },\n \"content_score\": 4.7,\n \"structure_score\": 4.3,\n \"overall_score\": 9.0\n },\n \"B\": {\n \"content\": {\n \"correctness\": 3,\n \"completeness\": 2,\n \"accuracy\": 3\n },\n \"structure\": {\n \"organization\": 3,\n \"formatting\": 2,\n \"usability\": 3\n },\n \"content_score\": 2.7,\n \"structure_score\": 2.7,\n \"overall_score\": 5.4\n }\n },\n \"output_quality\": {\n \"A\": {\n \"score\": 9,\n \"strengths\": [\"Complete solution\", \"Well-formatted\", \"All fields present\"],\n \"weaknesses\": [\"Minor style inconsistency in header\"]\n },\n \"B\": {\n \"score\": 5,\n \"strengths\": [\"Readable output\", \"Correct basic structure\"],\n \"weaknesses\": [\"Missing date field\", \"Formatting inconsistencies\", \"Partial data extraction\"]\n }\n },\n \"expectation_results\": {\n \"A\": {\n \"passed\": 4,\n \"total\": 5,\n \"pass_rate\": 0.80,\n \"details\": [\n {\"text\": \"Output includes name\", \"passed\": true}\n ]\n },\n \"B\": {\n \"passed\": 3,\n \"total\": 5,\n \"pass_rate\": 0.60,\n \"details\": [\n {\"text\": \"Output includes name\", \"passed\": true}\n ]\n }\n }\n}\n```\n\n---\n\n## analysis.json\n\nOutput from post-hoc analyzer. Located at `/analysis.json`.\n\n```json\n{\n \"comparison_summary\": {\n \"winner\": \"A\",\n \"winner_skill\": \"path/to/winner/skill\",\n \"loser_skill\": \"path/to/loser/skill\",\n \"comparator_reasoning\": \"Brief summary of why comparator chose winner\"\n },\n \"winner_strengths\": [\n \"Clear step-by-step instructions for handling multi-page documents\",\n \"Included validation script that caught formatting errors\"\n ],\n \"loser_weaknesses\": [\n \"Vague instruction 'process the document appropriately' led to inconsistent behavior\",\n \"No script for validation, agent had to improvise\"\n ],\n \"instruction_following\": {\n \"winner\": {\n \"score\": 9,\n \"issues\": [\"Minor: skipped optional logging step\"]\n },\n \"loser\": {\n \"score\": 6,\n \"issues\": [\n \"Did not use the skill's formatting template\",\n \"Invented own approach instead of following step 3\"\n ]\n }\n },\n \"improvement_suggestions\": [\n {\n \"priority\": \"high\",\n \"category\": \"instructions\",\n \"suggestion\": \"Replace 'process the document appropriately' with explicit steps\",\n \"expected_impact\": \"Would eliminate ambiguity that caused inconsistent behavior\"\n }\n ],\n \"transcript_insights\": {\n \"winner_execution_pattern\": \"Read skill -> Followed 5-step process -> Used validation script\",\n \"loser_execution_pattern\": \"Read skill -> Unclear on approach -> Tried 3 different methods\"\n }\n}\n```\n\n\n---\nname: skill-creator\ndescription: Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.\n---\n\n# Skill Creator\n\nA skill for creating new skills and iteratively improving them.\n\nAt a high level, the process of creating a skill goes like this:\n\n- Decide what you want the skill to do and roughly how it should do it\n- Write a draft of the skill\n- Create a few test prompts and run claude-with-access-to-the-skill on them\n- Help the user evaluate the results both qualitatively and quantitatively\n - While the runs happen in the background, draft some quantitative evals if there aren't any (if there are some, you can either use as is or modify if you feel something needs to change about them). Then explain them to the user (or if they already existed, explain the ones that already exist)\n - Use the `eval-viewer/generate_review.py` script to show the user the results for them to look at, and also let them look at the quantitative metrics\n- Rewrite the skill based on feedback from the user's evaluation of the results (and also if there are any glaring flaws that become apparent from the quantitative benchmarks)\n- Repeat until you're satisfied\n- Expand the test set and try again at larger scale\n\nYour job when using this skill is to figure out where the user is in this process and then jump in and help them progress through these stages. So for instance, maybe they're like \"I want to make a skill for X\". You can help narrow down what they mean, write a draft, write the test cases, figure out how they want to evaluate, run all the prompts, and repeat.\n\nOn the other hand, maybe they already have a draft of the skill. In this case you can go straight to the eval/iterate part of the loop.\n\nOf course, you should always be flexible and if the user is like \"I don't need to run a bunch of evaluations, just vibe with me\", you can do that instead.\n\nThen after the skill is done (but again, the order is flexible), you can also run the skill description improver, which we have a whole separate script for, to optimize the triggering of the skill.\n\nCool? Cool.\n\n## Communicating with the user\n\nThe skill creator is liable to be used by people across a wide range of familiarity with coding jargon. If you haven't heard (and how could you, it's only very recently that it started), there's a trend now where the power of Claude is inspiring plumbers to open up their terminals, parents and grandparents to google \"how to install npm\". On the other hand, the bulk of users are probably fairly computer-literate.\n\nSo please pay attention to context cues to understand how to phrase your communication! In the default case, just to give you some idea:\n\n- \"evaluation\" and \"benchmark\" are borderline, but OK\n- for \"JSON\" and \"assertion\" you want to see serious cues from the user that they know what those things are before using them without explaining them\n\nIt's OK to briefly explain terms if you're in doubt, and feel free to clarify terms with a short definition if you're unsure if the user will get it.\n\n---\n\n## Creating a skill\n\n### Capture Intent\n\nStart by understanding the user's intent. The current conversation might already contain a workflow the user wants to capture (e.g., they say \"turn this into a skill\"). If so, extract answers from the conversation history first — the tools used, the sequence of steps, corrections the user made, input/output formats observed. The user may need to fill the gaps, and should confirm before proceeding to the next step.\n\n1. What should this skill enable Claude to do?\n2. When should this skill trigger? (what user phrases/contexts)\n3. What's the expected output format?\n4. Should we set up test cases to verify the skill works? Skills with objectively verifiable outputs (file transforms, data extraction, code generation, fixed workflow steps) benefit from test cases. Skills with subjective outputs (writing style, art) often don't need them. Suggest the appropriate default based on the skill type, but let the user decide.\n\n### Interview and Research\n\nProactively ask questions about edge cases, input/output formats, example files, success criteria, and dependencies. Wait to write test prompts until you've got this part ironed out.\n\nCheck available MCPs - if useful for research (searching docs, finding similar skills, looking up best practices), research in parallel via subagents if available, otherwise inline. Come prepared with context to reduce burden on the user.\n\n### Write the SKILL.md\n\nBased on the user interview, fill in these components:\n\n- **name**: Skill identifier\n- **description**: When to trigger, what it does. This is the primary triggering mechanism - include both what the skill does AND specific contexts for when to use it. All \"when to use\" info goes here, not in the body. Note: currently Claude has a tendency to \"undertrigger\" skills -- to not use them when they'd be useful. To combat this, please make the skill descriptions a little bit \"pushy\". So for instance, instead of \"How to build a simple fast dashboard to display internal Anthropic data.\", you might write \"How to build a simple fast dashboard to display internal Anthropic data. Make sure to use this skill whenever the user mentions dashboards, data visualization, internal metrics, or wants to display any kind of company data, even if they don't explicitly ask for a 'dashboard.'\"\n- **compatibility**: Required tools, dependencies (optional, rarely needed)\n- **the rest of the skill :)**\n\n### Skill Writing Guide\n\n#### Anatomy of a Skill\n\n```\nskill-name/\n├── SKILL.md (required)\n│ ├── YAML frontmatter (name, description required)\n│ └── Markdown instructions\n└── Bundled Resources (optional)\n ├── scripts/ - Executable code for deterministic/repetitive tasks\n ├── references/ - Docs loaded into context as needed\n └── assets/ - Files used in output (templates, icons, fonts)\n```\n\n#### Progressive Disclosure\n\nSkills use a three-level loading system:\n1. **Metadata** (name + description) - Always in context (~100 words)\n2. **SKILL.md body** - In context whenever skill triggers (<500 lines ideal)\n3. **Bundled resources** - As needed (unlimited, scripts can execute without loading)\n\nThese word counts are approximate and you can feel free to go longer if needed.\n\n**Key patterns:**\n- Keep SKILL.md under 500 lines; if you're approaching this limit, add an additional layer of hierarchy along with clear pointers about where the model using the skill should go next to follow up.\n- Reference files clearly from SKILL.md with guidance on when to read them\n- For large reference files (>300 lines), include a table of contents\n\n**Domain organization**: When a skill supports multiple domains/frameworks, organize by variant:\n```\ncloud-deploy/\n├── SKILL.md (workflow + selection)\n└── references/\n ├── aws.md\n ├── gcp.md\n └── azure.md\n```\nClaude reads only the relevant reference file.\n\n#### Principle of Lack of Surprise\n\nThis goes without saying, but skills must not contain malware, exploit code, or any content that could compromise system security. A skill's contents should not surprise the user in their intent if described. Don't go along with requests to create misleading skills or skills designed to facilitate unauthorized access, data exfiltration, or other malicious activities. Things like a \"roleplay as an XYZ\" are OK though.\n\n#### Writing Patterns\n\nPrefer using the imperative form in instructions.\n\n**Defining output formats** - You can do it like this:\n```markdown\n## Report structure\nALWAYS use this exact template:\n# [Title]\n## Executive summary\n## Key findings\n## Recommendations\n```\n\n**Examples pattern** - It's useful to include examples. You can format them like this (but if \"Input\" and \"Output\" are in the examples you might want to deviate a little):\n```markdown\n## Commit message format\n**Example 1:**\nInput: Added user authentication with JWT tokens\nOutput: feat(auth): implement JWT-based authentication\n```\n\n### Writing Style\n\nTry to explain to the model why things are important in lieu of heavy-handed musty MUSTs. Use theory of mind and try to make the skill general and not super-narrow to specific examples. Start by writing a draft and then look at it with fresh eyes and improve it.\n\n### Test Cases\n\nAfter writing the skill draft, come up with 2-3 realistic test prompts — the kind of thing a real user would actually say. Share them with the user: [you don't have to use this exact language] \"Here are a few test cases I'd like to try. Do these look right, or do you want to add more?\" Then run them.\n\nSave test cases to `evals/evals.json`. Don't write assertions yet — just the prompts. You'll draft assertions in the next step while the runs are in progress.\n\n```json\n{\n \"skill_name\": \"example-skill\",\n \"evals\": [\n {\n \"id\": 1,\n \"prompt\": \"User's task prompt\",\n \"expected_output\": \"Description of expected result\",\n \"files\": []\n }\n ]\n}\n```\n\nSee `references/schemas.md` for the full schema (including the `assertions` field, which you'll add later).\n\n## Running and evaluating test cases\n\nThis section is one continuous sequence — don't stop partway through. Do NOT use `/skill-test` or any other testing skill.\n\nPut results in `-workspace/` as a sibling to the skill directory. Within the workspace, organize results by iteration (`iteration-1/`, `iteration-2/`, etc.) and within that, each test case gets a directory (`eval-0/`, `eval-1/`, etc.). Don't create all of this upfront — just create directories as you go.\n\n### Step 1: Spawn all runs (with-skill AND baseline) in the same turn\n\nFor each test case, spawn two subagents in the same turn — one with the skill, one without. This is important: don't spawn the with-skill runs first and then come back for baselines later. Launch everything at once so it all finishes around the same time.\n\n**With-skill run:**\n\n```\nExecute this task:\n- Skill path: \n- Task: \n- Input files: \n- Save outputs to: /iteration-/eval-/with_skill/outputs/\n- Outputs to save: \n```\n\n**Baseline run** (same prompt, but the baseline depends on context):\n- **Creating a new skill**: no skill at all. Same prompt, no skill path, save to `without_skill/outputs/`.\n- **Improving an existing skill**: the old version. Before editing, snapshot the skill (`cp -r /skill-snapshot/`), then point the baseline subagent at the snapshot. Save to `old_skill/outputs/`.\n\nWrite an `eval_metadata.json` for each test case (assertions can be empty for now). Give each eval a descriptive name based on what it's testing — not just \"eval-0\". Use this name for the directory too. If this iteration uses new or modified eval prompts, create these files for each new eval directory — don't assume they carry over from previous iterations.\n\n```json\n{\n \"eval_id\": 0,\n \"eval_name\": \"descriptive-name-here\",\n \"prompt\": \"The user's task prompt\",\n \"assertions\": []\n}\n```\n\n### Step 2: While runs are in progress, draft assertions\n\nDon't just wait for the runs to finish — you can use this time productively. Draft quantitative assertions for each test case and explain them to the user. If assertions already exist in `evals/evals.json`, review them and explain what they check.\n\nGood assertions are objectively verifiable and have descriptive names — they should read clearly in the benchmark viewer so someone glancing at the results immediately understands what each one checks. Subjective skills (writing style, design quality) are better evaluated qualitatively — don't force assertions onto things that need human judgment.\n\nUpdate the `eval_metadata.json` files and `evals/evals.json` with the assertions once drafted. Also explain to the user what they'll see in the viewer — both the qualitative outputs and the quantitative benchmark.\n\n### Step 3: As runs complete, capture timing data\n\nWhen each subagent task completes, you receive a notification containing `total_tokens` and `duration_ms`. Save this data immediately to `timing.json` in the run directory:\n\n```json\n{\n \"total_tokens\": 84852,\n \"duration_ms\": 23332,\n \"total_duration_seconds\": 23.3\n}\n```\n\nThis is the only opportunity to capture this data — it comes through the task notification and isn't persisted elsewhere. Process each notification as it arrives rather than trying to batch them.\n\n### Step 4: Grade, aggregate, and launch the viewer\n\nOnce all runs are done:\n\n1. **Grade each run** — spawn a grader subagent (or grade inline) that reads `agents/grader.md` and evaluates each assertion against the outputs. Save results to `grading.json` in each run directory. The grading.json expectations array must use the fields `text`, `passed`, and `evidence` (not `name`/`met`/`details` or other variants) — the viewer depends on these exact field names. For assertions that can be checked programmatically, write and run a script rather than eyeballing it — scripts are faster, more reliable, and can be reused across iterations.\n\n2. **Aggregate into benchmark** — run the aggregation script from the skill-creator directory:\n ```bash\n python -m scripts.aggregate_benchmark /iteration-N --skill-name \n ```\n This produces `benchmark.json` and `benchmark.md` with pass_rate, time, and tokens for each configuration, with mean ± stddev and the delta. If generating benchmark.json manually, see `references/schemas.md` for the exact schema the viewer expects.\nPut each with_skill version before its baseline counterpart.\n\n3. **Do an analyst pass** — read the benchmark data and surface patterns the aggregate stats might hide. See `agents/analyzer.md` (the \"Analyzing Benchmark Results\" section) for what to look for — things like assertions that always pass regardless of skill (non-discriminating), high-variance evals (possibly flaky), and time/token tradeoffs.\n\n4. **Launch the viewer** with both qualitative outputs and quantitative data:\n ```bash\n nohup python /eval-viewer/generate_review.py \\\n /iteration-N \\\n --skill-name \"my-skill\" \\\n --benchmark /iteration-N/benchmark.json \\\n > /dev/null 2>&1 &\n VIEWER_PID=$!\n ```\n For iteration 2+, also pass `--previous-workspace /iteration-`.\n\n **Cowork / headless environments:** If `webbrowser.open()` is not available or the environment has no display, use `--static ` to write a standalone HTML file instead of starting a server. Feedback will be downloaded as a `feedback.json` file when the user clicks \"Submit All Reviews\". After download, copy `feedback.json` into the workspace directory for the next iteration to pick up.\n\nNote: please use generate_review.py to create the viewer; there's no need to write custom HTML.\n\n5. **Tell the user** something like: \"I've opened the results in your browser. There are two tabs — 'Outputs' lets you click through each test case and leave feedback, 'Benchmark' shows the quantitative comparison. When you're done, come back here and let me know.\"\n\n### What the user sees in the viewer\n\nThe \"Outputs\" tab shows one test case at a time:\n- **Prompt**: the task that was given\n- **Output**: the files the skill produced, rendered inline where possible\n- **Previous Output** (iteration 2+): collapsed section showing last iteration's output\n- **Formal Grades** (if grading was run): collapsed section showing assertion pass/fail\n- **Feedback**: a textbox that auto-saves as they type\n- **Previous Feedback** (iteration 2+): their comments from last time, shown below the textbox\n\nThe \"Benchmark\" tab shows the stats summary: pass rates, timing, and token usage for each configuration, with per-eval breakdowns and analyst observations.\n\nNavigation is via prev/next buttons or arrow keys. When done, they click \"Submit All Reviews\" which saves all feedback to `feedback.json`.\n\n### Step 5: Read the feedback\n\nWhen the user tells you they're done, read `feedback.json`:\n\n```json\n{\n \"reviews\": [\n {\"run_id\": \"eval-0-with_skill\", \"feedback\": \"the chart is missing axis labels\", \"timestamp\": \"...\"},\n {\"run_id\": \"eval-1-with_skill\", \"feedback\": \"\", \"timestamp\": \"...\"},\n {\"run_id\": \"eval-2-with_skill\", \"feedback\": \"perfect, love this\", \"timestamp\": \"...\"}\n ],\n \"status\": \"complete\"\n}\n```\n\nEmpty feedback means the user thought it was fine. Focus your improvements on the test cases where the user had specific complaints.\n\nKill the viewer server when you're done with it:\n\n```bash\nkill $VIEWER_PID 2>/dev/null\n```\n\n---\n\n## Improving the skill\n\nThis is the heart of the loop. You've run the test cases, the user has reviewed the results, and now you need to make the skill better based on their feedback.\n\n### How to think about improvements\n\n1. **Generalize from the feedback.** The big picture thing that's happening here is that we're trying to create skills that can be used a million times (maybe literally, maybe even more who knows) across many different prompts. Here you and the user are iterating on only a few examples over and over again because it helps move faster. The user knows these examples in and out and it's quick for them to assess new outputs. But if the skill you and the user are codeveloping works only for those examples, it's useless. Rather than put in fiddly overfitty changes, or oppressively constrictive MUSTs, if there's some stubborn issue, you might try branching out and using different metaphors, or recommending different patterns of working. It's relatively cheap to try and maybe you'll land on something great.\n\n2. **Keep the prompt lean.** Remove things that aren't pulling their weight. Make sure to read the transcripts, not just the final outputs — if it looks like the skill is making the model waste a bunch of time doing things that are unproductive, you can try getting rid of the parts of the skill that are making it do that and seeing what happens.\n\n3. **Explain the why.** Try hard to explain the **why** behind everything you're asking the model to do. Today's LLMs are *smart*. They have good theory of mind and when given a good harness can go beyond rote instructions and really make things happen. Even if the feedback from the user is terse or frustrated, try to actually understand the task and why the user is writing what they wrote, and what they actually wrote, and then transmit this understanding into the instructions. If you find yourself writing ALWAYS or NEVER in all caps, or using super rigid structures, that's a yellow flag — if possible, reframe and explain the reasoning so that the model understands why the thing you're asking for is important. That's a more humane, powerful, and effective approach.\n\n4. **Look for repeated work across test cases.** Read the transcripts from the test runs and notice if the subagents all independently wrote similar helper scripts or took the same multi-step approach to something. If all 3 test cases resulted in the subagent writing a `create_docx.py` or a `build_chart.py`, that's a strong signal the skill should bundle that script. Write it once, put it in `scripts/`, and tell the skill to use it. This saves every future invocation from reinventing the wheel.\n\nThis task is pretty important (we are trying to create billions a year in economic value here!) and your thinking time is not the blocker; take your time and really mull things over. I'd suggest writing a draft revision and then looking at it anew and making improvements. Really do your best to get into the head of the user and understand what they want and need.\n\n### The iteration loop\n\nAfter improving the skill:\n\n1. Apply your improvements to the skill\n2. Rerun all test cases into a new `iteration-/` directory, including baseline runs. If you're creating a new skill, the baseline is always `without_skill` (no skill) — that stays the same across iterations. If you're improving an existing skill, use your judgment on what makes sense as the baseline: the original version the user came in with, or the previous iteration.\n3. Launch the reviewer with `--previous-workspace` pointing at the previous iteration\n4. Wait for the user to review and tell you they're done\n5. Read the new feedback, improve again, repeat\n\nKeep going until:\n- The user says they're happy\n- The feedback is all empty (everything looks good)\n- You're not making meaningful progress\n\n---\n\n## Advanced: Blind comparison\n\nFor situations where you want a more rigorous comparison between two versions of a skill (e.g., the user asks \"is the new version actually better?\"), there's a blind comparison system. Read `agents/comparator.md` and `agents/analyzer.md` for the details. The basic idea is: give two outputs to an independent agent without telling it which is which, and let it judge quality. Then analyze why the winner won.\n\nThis is optional, requires subagents, and most users won't need it. The human review loop is usually sufficient.\n\n---\n\n## Description Optimization\n\nThe description field in SKILL.md frontmatter is the primary mechanism that determines whether Claude invokes a skill. After creating or improving a skill, offer to optimize the description for better triggering accuracy.\n\n### Step 1: Generate trigger eval queries\n\nCreate 20 eval queries — a mix of should-trigger and should-not-trigger. Save as JSON:\n\n```json\n[\n {\"query\": \"the user prompt\", \"should_trigger\": true},\n {\"query\": \"another prompt\", \"should_trigger\": false}\n]\n```\n\nThe queries must be realistic and something a Claude Code or Claude.ai user would actually type. Not abstract requests, but requests that are concrete and specific and have a good amount of detail. For instance, file paths, personal context about the user's job or situation, column names and values, company names, URLs. A little bit of backstory. Some might be in lowercase or contain abbreviations or typos or casual speech. Use a mix of different lengths, and focus on edge cases rather than making them clear-cut (the user will get a chance to sign off on them).\n\nBad: `\"Format this data\"`, `\"Extract text from PDF\"`, `\"Create a chart\"`\n\nGood: `\"ok so my boss just sent me this xlsx file (its in my downloads, called something like 'Q4 sales final FINAL v2.xlsx') and she wants me to add a column that shows the profit margin as a percentage. The revenue is in column C and costs are in column D i think\"`\n\nFor the **should-trigger** queries (8-10), think about coverage. You want different phrasings of the same intent — some formal, some casual. Include cases where the user doesn't explicitly name the skill or file type but clearly needs it. Throw in some uncommon use cases and cases where this skill competes with another but should win.\n\nFor the **should-not-trigger** queries (8-10), the most valuable ones are the near-misses — queries that share keywords or concepts with the skill but actually need something different. Think adjacent domains, ambiguous phrasing where a naive keyword match would trigger but shouldn't, and cases where the query touches on something the skill does but in a context where another tool is more appropriate.\n\nThe key thing to avoid: don't make should-not-trigger queries obviously irrelevant. \"Write a fibonacci function\" as a negative test for a PDF skill is too easy — it doesn't test anything. The negative cases should be genuinely tricky.\n\n### Step 2: Review with user\n\nPresent the eval set to the user for review using the HTML template:\n\n1. Read the template from `assets/eval_review.html`\n2. Replace the placeholders:\n - `__EVAL_DATA_PLACEHOLDER__` → the JSON array of eval items (no quotes around it — it's a JS variable assignment)\n - `__SKILL_NAME_PLACEHOLDER__` → the skill's name\n - `__SKILL_DESCRIPTION_PLACEHOLDER__` → the skill's current description\n3. Write to a temp file (e.g., `/tmp/eval_review_.html`) and open it: `open /tmp/eval_review_.html`\n4. The user can edit queries, toggle should-trigger, add/remove entries, then click \"Export Eval Set\"\n5. The file downloads to `~/Downloads/eval_set.json` — check the Downloads folder for the most recent version in case there are multiple (e.g., `eval_set (1).json`)\n\nThis step matters — bad eval queries lead to bad descriptions.\n\n### Step 3: Run the optimization loop\n\nTell the user: \"This will take some time — I'll run the optimization loop in the background and check on it periodically.\"\n\nSave the eval set to the workspace, then run in the background:\n\n```bash\npython -m scripts.run_loop \\\n --eval-set \\\n --skill-path \\\n --model \\\n --max-iterations 5 \\\n --verbose\n```\n\nUse the model ID from your system prompt (the one powering the current session) so the triggering test matches what the user actually experiences.\n\nWhile it runs, periodically tail the output to give the user updates on which iteration it's on and what the scores look like.\n\nThis handles the full optimization loop automatically. It splits the eval set into 60% train and 40% held-out test, evaluates the current description (running each query 3 times to get a reliable trigger rate), then calls Claude to propose improvements based on what failed. It re-evaluates each new description on both train and test, iterating up to 5 times. When it's done, it opens an HTML report in the browser showing the results per iteration and returns JSON with `best_description` — selected by test score rather than train score to avoid overfitting.\n\n### How skill triggering works\n\nUnderstanding the triggering mechanism helps design better eval queries. Skills appear in Claude's `available_skills` list with their name + description, and Claude decides whether to consult a skill based on that description. The important thing to know is that Claude only consults skills for tasks it can't easily handle on its own — simple, one-step queries like \"read this PDF\" may not trigger a skill even if the description matches perfectly, because Claude can handle them directly with basic tools. Complex, multi-step, or specialized queries reliably trigger skills when the description matches.\n\nThis means your eval queries should be substantive enough that Claude would actually benefit from consulting a skill. Simple queries like \"read file X\" are poor test cases — they won't trigger skills regardless of description quality.\n\n### Step 4: Apply the result\n\nTake `best_description` from the JSON output and update the skill's SKILL.md frontmatter. Show the user before/after and report the scores.\n\n---\n\n### Package and Present (only if `present_files` tool is available)\n\nCheck whether you have access to the `present_files` tool. If you don't, skip this step. If you do, package the skill and present the .skill file to the user:\n\n```bash\npython -m scripts.package_skill \n```\n\nAfter packaging, direct the user to the resulting `.skill` file path so they can install it.\n\n---\n\n## Claude.ai-specific instructions\n\nIn Claude.ai, the core workflow is the same (draft → test → review → improve → repeat), but because Claude.ai doesn't have subagents, some mechanics change. Here's what to adapt:\n\n**Running test cases**: No subagents means no parallel execution. For each test case, read the skill's SKILL.md, then follow its instructions to accomplish the test prompt yourself. Do them one at a time. This is less rigorous than independent subagents (you wrote the skill and you're also running it, so you have full context), but it's a useful sanity check — and the human review step compensates. Skip the baseline runs — just use the skill to complete the task as requested.\n\n**Reviewing results**: If you can't open a browser (e.g., Claude.ai's VM has no display, or you're on a remote server), skip the browser reviewer entirely. Instead, present results directly in the conversation. For each test case, show the prompt and the output. If the output is a file the user needs to see (like a .docx or .xlsx), save it to the filesystem and tell them where it is so they can download and inspect it. Ask for feedback inline: \"How does this look? Anything you'd change?\"\n\n**Benchmarking**: Skip the quantitative benchmarking — it relies on baseline comparisons which aren't meaningful without subagents. Focus on qualitative feedback from the user.\n\n**The iteration loop**: Same as before — improve the skill, rerun the test cases, ask for feedback — just without the browser reviewer in the middle. You can still organize results into iteration directories on the filesystem if you have one.\n\n**Description optimization**: This section requires the `claude` CLI tool (specifically `claude -p`) which is only available in Claude Code. Skip it if you're on Claude.ai.\n\n**Blind comparison**: Requires subagents. Skip it.\n\n**Packaging**: The `package_skill.py` script works anywhere with Python and a filesystem. On Claude.ai, you can run it and the user can download the resulting `.skill` file.\n\n**Updating an existing skill**: The user might be asking you to update an existing skill, not create a new one. In this case:\n- **Preserve the original name.** Note the skill's directory name and `name` frontmatter field -- use them unchanged. E.g., if the installed skill is `research-helper`, output `research-helper.skill` (not `research-helper-v2`).\n- **Copy to a writeable location before editing.** The installed skill path may be read-only. Copy to `/tmp/skill-name/`, edit there, and package from the copy.\n- **If packaging manually, stage in `/tmp/` first**, then copy to the output directory -- direct writes may fail due to permissions.\n\n---\n\n## Cowork-Specific Instructions\n\nIf you're in Cowork, the main things to know are:\n\n- You have subagents, so the main workflow (spawn test cases in parallel, run baselines, grade, etc.) all works. (However, if you run into severe problems with timeouts, it's OK to run the test prompts in series rather than parallel.)\n- You don't have a browser or display, so when generating the eval viewer, use `--static ` to write a standalone HTML file instead of starting a server. Then proffer a link that the user can click to open the HTML in their browser.\n- For whatever reason, the Cowork setup seems to disincline Claude from generating the eval viewer after running the tests, so just to reiterate: whether you're in Cowork or in Claude Code, after running tests, you should always generate the eval viewer for the human to look at examples before revising the skill yourself and trying to make corrections, using `generate_review.py` (not writing your own boutique html code). Sorry in advance but I'm gonna go all caps here: GENERATE THE EVAL VIEWER *BEFORE* evaluating inputs yourself. You want to get them in front of the human ASAP!\n- Feedback works differently: since there's no running server, the viewer's \"Submit All Reviews\" button will download `feedback.json` as a file. You can then read it from there (you may have to request access first).\n- Packaging works — `package_skill.py` just needs Python and a filesystem.\n- Description optimization (`run_loop.py` / `run_eval.py`) should work in Cowork just fine since it uses `claude -p` via subprocess, not a browser, but please save it until you've fully finished making the skill and the user agrees it's in good shape.\n- **Updating an existing skill**: The user might be asking you to update an existing skill, not create a new one. Follow the update guidance in the claude.ai section above.\n\n---\n\n## Reference files\n\nThe agents/ directory contains instructions for specialized subagents. Read them when you need to spawn the relevant subagent.\n\n- `agents/grader.md` — How to evaluate assertions against outputs\n- `agents/comparator.md` — How to do blind A/B comparison between two outputs\n- `agents/analyzer.md` — How to analyze why one version beat another\n\nThe references/ directory has additional documentation:\n- `references/schemas.md` — JSON structures for evals.json, grading.json, etc.\n\n---\n\nRepeating one more time the core loop here for emphasis:\n\n- Figure out what the skill is about\n- Draft or edit the skill\n- Run claude-with-access-to-the-skill on test prompts\n- With the user, evaluate the outputs:\n - Create benchmark.json and run `eval-viewer/generate_review.py` to help the user review them\n - Run quantitative evals\n- Repeat until you and the user are satisfied\n- Package the final skill and return it to the user.\n\nPlease add steps to your TodoList, if you have such a thing, to make sure you don't forget. If you're in Cowork, please specifically put \"Create evals JSON and run `eval-viewer/generate_review.py` so human can review test cases\" in your TodoList to make sure it happens.\n\nGood luck!\n\n\n# Codex Tool Mapping\n\nSkills use Claude Code tool names. When you encounter these in a skill, use your platform equivalent:\n\n| Skill references | Codex equivalent |\n|-----------------|------------------|\n| `Task` tool (dispatch subagent) | `spawn_agent` |\n| Multiple `Task` calls (parallel) | Multiple `spawn_agent` calls |\n| Task returns result | `wait` |\n| Task completes automatically | `close_agent` to free slot |\n| `TodoWrite` (task tracking) | `update_plan` |\n| `Skill` tool (invoke a skill) | Skills load natively — just follow the instructions |\n| `Read`, `Write`, `Edit` (files) | Use your native file tools |\n| `Bash` (run commands) | Use your native shell tools |\n\n## Subagent dispatch requires collab\n\nAdd to your Codex config (`~/.codex/config.toml`):\n\n```toml\n[features]\ncollab = true\n```\n\nThis enables `spawn_agent`, `wait`, and `close_agent` for skills like `dispatching-parallel-agents` and `subagent-driven-development`.\n\n\n# Gemini CLI Tool Mapping\n\nSkills use Claude Code tool names. When you encounter these in a skill, use your platform equivalent:\n\n| Skill references | Gemini CLI equivalent |\n|-----------------|----------------------|\n| `Read` (file reading) | `read_file` |\n| `Write` (file creation) | `write_file` |\n| `Edit` (file editing) | `replace` |\n| `Bash` (run commands) | `run_shell_command` |\n| `Grep` (search file content) | `grep_search` |\n| `Glob` (search files by name) | `glob` |\n| `TodoWrite` (task tracking) | `write_todos` |\n| `Skill` tool (invoke a skill) | `activate_skill` |\n| `WebSearch` | `google_web_search` |\n| `WebFetch` | `web_fetch` |\n| `Task` tool (dispatch subagent) | No equivalent — Gemini CLI does not support subagents |\n\n## No subagent support\n\nGemini CLI has no equivalent to Claude Code's `Task` tool. Skills that rely on subagent dispatch (`subagent-driven-development`, `dispatching-parallel-agents`) will fall back to single-session execution via `executing-plans`.\n\n## Additional Gemini CLI tools\n\nThese tools are available in Gemini CLI but have no Claude Code equivalent:\n\n| Tool | Purpose |\n|------|---------|\n| `list_directory` | List files and subdirectories |\n| `save_memory` | Persist facts to GEMINI.md across sessions |\n| `ask_user` | Request structured input from the user |\n| `tracker_create_task` | Rich task management (create, update, list, visualize) |\n| `enter_plan_mode` / `exit_plan_mode` | Switch to read-only research mode before making changes |\n\n\n---\nname: using-superpowers\ndescription: Use when starting any conversation - establishes how to find and use skills, requiring Skill tool invocation before ANY response including clarifying questions\n---\n\n\nIf you were dispatched as a subagent to execute a specific task, skip this skill.\n\n\n\nIf you think there is even a 1% chance a skill might apply to what you are doing, you ABSOLUTELY MUST invoke the skill.\n\nIF A SKILL APPLIES TO YOUR TASK, YOU DO NOT HAVE A CHOICE. YOU MUST USE IT.\n\nThis is not negotiable. This is not optional. You cannot rationalize your way out of this.\n\n\n## Instruction Priority\n\nSuperpowers skills override default system prompt behavior, but **user instructions always take precedence**:\n\n1. **User's explicit instructions** (CLAUDE.md, GEMINI.md, AGENTS.md, direct requests) — highest priority\n2. **Superpowers skills** — override default system behavior where they conflict\n3. **Default system prompt** — lowest priority\n\nIf CLAUDE.md, GEMINI.md, or AGENTS.md says \"don't use TDD\" and a skill says \"always use TDD,\" follow the user's instructions. The user is in control.\n\n## How to Access Skills\n\n**In Claude Code:** Use the `Skill` tool. When you invoke a skill, its content is loaded and presented to you—follow it directly. Never use the Read tool on skill files.\n\n**In Gemini CLI:** Skills activate via the `activate_skill` tool. Gemini loads skill metadata at session start and activates the full content on demand.\n\n**In other environments:** Check your platform's documentation for how skills are loaded.\n\n## Platform Adaptation\n\nSkills use Claude Code tool names. Non-CC platforms: see `references/codex-tools.md` (Codex) for tool equivalents. Gemini CLI users get the tool mapping loaded automatically via GEMINI.md.\n\n# Using Skills\n\n## The Rule\n\n**Invoke relevant or requested skills BEFORE any response or action.** Even a 1% chance a skill might apply means that you should invoke the skill to check. If an invoked skill turns out to be wrong for the situation, you don't need to use it.\n\n```dot\ndigraph skill_flow {\n \"User message received\" [shape=doublecircle];\n \"About to EnterPlanMode?\" [shape=doublecircle];\n \"Already brainstormed?\" [shape=diamond];\n \"Invoke brainstorming skill\" [shape=box];\n \"Might any skill apply?\" [shape=diamond];\n \"Invoke Skill tool\" [shape=box];\n \"Announce: 'Using [skill] to [purpose]'\" [shape=box];\n \"Has checklist?\" [shape=diamond];\n \"Create TodoWrite todo per item\" [shape=box];\n \"Follow skill exactly\" [shape=box];\n \"Respond (including clarifications)\" [shape=doublecircle];\n\n \"About to EnterPlanMode?\" -> \"Already brainstormed?\";\n \"Already brainstormed?\" -> \"Invoke brainstorming skill\" [label=\"no\"];\n \"Already brainstormed?\" -> \"Might any skill apply?\" [label=\"yes\"];\n \"Invoke brainstorming skill\" -> \"Might any skill apply?\";\n\n \"User message received\" -> \"Might any skill apply?\";\n \"Might any skill apply?\" -> \"Invoke Skill tool\" [label=\"yes, even 1%\"];\n \"Might any skill apply?\" -> \"Respond (including clarifications)\" [label=\"definitely not\"];\n \"Invoke Skill tool\" -> \"Announce: 'Using [skill] to [purpose]'\";\n \"Announce: 'Using [skill] to [purpose]'\" -> \"Has checklist?\";\n \"Has checklist?\" -> \"Create TodoWrite todo per item\" [label=\"yes\"];\n \"Has checklist?\" -> \"Follow skill exactly\" [label=\"no\"];\n \"Create TodoWrite todo per item\" -> \"Follow skill exactly\";\n}\n```\n\n## Red Flags\n\nThese thoughts mean STOP—you're rationalizing:\n\n| Thought | Reality |\n|---------|---------|\n| \"This is just a simple question\" | Questions are tasks. Check for skills. |\n| \"I need more context first\" | Skill check comes BEFORE clarifying questions. |\n| \"Let me explore the codebase first\" | Skills tell you HOW to explore. Check first. |\n| \"I can check git/files quickly\" | Files lack conversation context. Check for skills. |\n| \"Let me gather information first\" | Skills tell you HOW to gather information. |\n| \"This doesn't need a formal skill\" | If a skill exists, use it. |\n| \"I remember this skill\" | Skills evolve. Read current version. |\n| \"This doesn't count as a task\" | Action = task. Check for skills. |\n| \"The skill is overkill\" | Simple things become complex. Use it. |\n| \"I'll just do this one thing first\" | Check BEFORE doing anything. |\n| \"This feels productive\" | Undisciplined action wastes time. Skills prevent this. |\n| \"I know what that means\" | Knowing the concept ≠ using the skill. Invoke it. |\n\n## Skill Priority\n\nWhen multiple skills could apply, use this order:\n\n1. **Process skills first** (brainstorming, debugging) - these determine HOW to approach the task\n2. **Implementation skills second** (frontend-design, mcp-builder) - these guide execution\n\n\"Let's build X\" → brainstorming first, then implementation skills.\n\"Fix this bug\" → debugging first, then domain-specific skills.\n\n## Skill Types\n\n**Rigid** (TDD, debugging): Follow exactly. Don't adapt away discipline.\n\n**Flexible** (patterns): Adapt principles to context.\n\nThe skill itself tells you which.\n\n## User Instructions\n\nInstructions say WHAT, not HOW. \"Add X\" or \"Fix Y\" doesn't mean skip workflows.\n\n\n# Post-hoc Analyzer Agent\n\nAnalyze blind comparison results to understand WHY the winner won and generate improvement suggestions.\n\n## Role\n\nAfter the blind comparator determines a winner, the Post-hoc Analyzer \"unblids\" the results by examining the skills and transcripts. The goal is to extract actionable insights: what made the winner better, and how can the loser be improved?\n\n## Inputs\n\nYou receive these parameters in your prompt:\n\n- **winner**: \"A\" or \"B\" (from blind comparison)\n- **winner_skill_path**: Path to the skill that produced the winning output\n- **winner_transcript_path**: Path to the execution transcript for the winner\n- **loser_skill_path**: Path to the skill that produced the losing output\n- **loser_transcript_path**: Path to the execution transcript for the loser\n- **comparison_result_path**: Path to the blind comparator's output JSON\n- **output_path**: Where to save the analysis results\n\n## Process\n\n### Step 1: Read Comparison Result\n\n1. Read the blind comparator's output at comparison_result_path\n2. Note the winning side (A or B), the reasoning, and any scores\n3. Understand what the comparator valued in the winning output\n\n### Step 2: Read Both Skills\n\n1. Read the winner skill's SKILL.md and key referenced files\n2. Read the loser skill's SKILL.md and key referenced files\n3. Identify structural differences:\n - Instructions clarity and specificity\n - Script/tool usage patterns\n - Example coverage\n - Edge case handling\n\n### Step 3: Read Both Transcripts\n\n1. Read the winner's transcript\n2. Read the loser's transcript\n3. Compare execution patterns:\n - How closely did each follow their skill's instructions?\n - What tools were used differently?\n - Where did the loser diverge from optimal behavior?\n - Did either encounter errors or make recovery attempts?\n\n### Step 4: Analyze Instruction Following\n\nFor each transcript, evaluate:\n- Did the agent follow the skill's explicit instructions?\n- Did the agent use the skill's provided tools/scripts?\n- Were there missed opportunities to leverage skill content?\n- Did the agent add unnecessary steps not in the skill?\n\nScore instruction following 1-10 and note specific issues.\n\n### Step 5: Identify Winner Strengths\n\nDetermine what made the winner better:\n- Clearer instructions that led to better behavior?\n- Better scripts/tools that produced better output?\n- More comprehensive examples that guided edge cases?\n- Better error handling guidance?\n\nBe specific. Quote from skills/transcripts where relevant.\n\n### Step 6: Identify Loser Weaknesses\n\nDetermine what held the loser back:\n- Ambiguous instructions that led to suboptimal choices?\n- Missing tools/scripts that forced workarounds?\n- Gaps in edge case coverage?\n- Poor error handling that caused failures?\n\n### Step 7: Generate Improvement Suggestions\n\nBased on the analysis, produce actionable suggestions for improving the loser skill:\n- Specific instruction changes to make\n- Tools/scripts to add or modify\n- Examples to include\n- Edge cases to address\n\nPrioritize by impact. Focus on changes that would have changed the outcome.\n\n### Step 8: Write Analysis Results\n\nSave structured analysis to `{output_path}`.\n\n## Output Format\n\nWrite a JSON file with this structure:\n\n```json\n{\n \"comparison_summary\": {\n \"winner\": \"A\",\n \"winner_skill\": \"path/to/winner/skill\",\n \"loser_skill\": \"path/to/loser/skill\",\n \"comparator_reasoning\": \"Brief summary of why comparator chose winner\"\n },\n \"winner_strengths\": [\n \"Clear step-by-step instructions for handling multi-page documents\",\n \"Included validation script that caught formatting errors\",\n \"Explicit guidance on fallback behavior when OCR fails\"\n ],\n \"loser_weaknesses\": [\n \"Vague instruction 'process the document appropriately' led to inconsistent behavior\",\n \"No script for validation, agent had to improvise and made errors\",\n \"No guidance on OCR failure, agent gave up instead of trying alternatives\"\n ],\n \"instruction_following\": {\n \"winner\": {\n \"score\": 9,\n \"issues\": [\n \"Minor: skipped optional logging step\"\n ]\n },\n \"loser\": {\n \"score\": 6,\n \"issues\": [\n \"Did not use the skill's formatting template\",\n \"Invented own approach instead of following step 3\",\n \"Missed the 'always validate output' instruction\"\n ]\n }\n },\n \"improvement_suggestions\": [\n {\n \"priority\": \"high\",\n \"category\": \"instructions\",\n \"suggestion\": \"Replace 'process the document appropriately' with explicit steps: 1) Extract text, 2) Identify sections, 3) Format per template\",\n \"expected_impact\": \"Would eliminate ambiguity that caused inconsistent behavior\"\n },\n {\n \"priority\": \"high\",\n \"category\": \"tools\",\n \"suggestion\": \"Add validate_output.py script similar to winner skill's validation approach\",\n \"expected_impact\": \"Would catch formatting errors before final output\"\n },\n {\n \"priority\": \"medium\",\n \"category\": \"error_handling\",\n \"suggestion\": \"Add fallback instructions: 'If OCR fails, try: 1) different resolution, 2) image preprocessing, 3) manual extraction'\",\n \"expected_impact\": \"Would prevent early failure on difficult documents\"\n }\n ],\n \"transcript_insights\": {\n \"winner_execution_pattern\": \"Read skill -> Followed 5-step process -> Used validation script -> Fixed 2 issues -> Produced output\",\n \"loser_execution_pattern\": \"Read skill -> Unclear on approach -> Tried 3 different methods -> No validation -> Output had errors\"\n }\n}\n```\n\n## Guidelines\n\n- **Be specific**: Quote from skills and transcripts, don't just say \"instructions were unclear\"\n- **Be actionable**: Suggestions should be concrete changes, not vague advice\n- **Focus on skill improvements**: The goal is to improve the losing skill, not critique the agent\n- **Prioritize by impact**: Which changes would most likely have changed the outcome?\n- **Consider causation**: Did the skill weakness actually cause the worse output, or is it incidental?\n- **Stay objective**: Analyze what happened, don't editorialize\n- **Think about generalization**: Would this improvement help on other evals too?\n\n## Categories for Suggestions\n\nUse these categories to organize improvement suggestions:\n\n| Category | Description |\n|----------|-------------|\n| `instructions` | Changes to the skill's prose instructions |\n| `tools` | Scripts, templates, or utilities to add/modify |\n| `examples` | Example inputs/outputs to include |\n| `error_handling` | Guidance for handling failures |\n| `structure` | Reorganization of skill content |\n| `references` | External docs or resources to add |\n\n## Priority Levels\n\n- **high**: Would likely change the outcome of this comparison\n- **medium**: Would improve quality but may not change win/loss\n- **low**: Nice to have, marginal improvement\n\n---\n\n# Analyzing Benchmark Results\n\nWhen analyzing benchmark results, the analyzer's purpose is to **surface patterns and anomalies** across multiple runs, not suggest skill improvements.\n\n## Role\n\nReview all benchmark run results and generate freeform notes that help the user understand skill performance. Focus on patterns that wouldn't be visible from aggregate metrics alone.\n\n## Inputs\n\nYou receive these parameters in your prompt:\n\n- **benchmark_data_path**: Path to the in-progress benchmark.json with all run results\n- **skill_path**: Path to the skill being benchmarked\n- **output_path**: Where to save the notes (as JSON array of strings)\n\n## Process\n\n### Step 1: Read Benchmark Data\n\n1. Read the benchmark.json containing all run results\n2. Note the configurations tested (with_skill, without_skill)\n3. Understand the run_summary aggregates already calculated\n\n### Step 2: Analyze Per-Assertion Patterns\n\nFor each expectation across all runs:\n- Does it **always pass** in both configurations? (may not differentiate skill value)\n- Does it **always fail** in both configurations? (may be broken or beyond capability)\n- Does it **always pass with skill but fail without**? (skill clearly adds value here)\n- Does it **always fail with skill but pass without**? (skill may be hurting)\n- Is it **highly variable**? (flaky expectation or non-deterministic behavior)\n\n### Step 3: Analyze Cross-Eval Patterns\n\nLook for patterns across evals:\n- Are certain eval types consistently harder/easier?\n- Do some evals show high variance while others are stable?\n- Are there surprising results that contradict expectations?\n\n### Step 4: Analyze Metrics Patterns\n\nLook at time_seconds, tokens, tool_calls:\n- Does the skill significantly increase execution time?\n- Is there high variance in resource usage?\n- Are there outlier runs that skew the aggregates?\n\n### Step 5: Generate Notes\n\nWrite freeform observations as a list of strings. Each note should:\n- State a specific observation\n- Be grounded in the data (not speculation)\n- Help the user understand something the aggregate metrics don't show\n\nExamples:\n- \"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value\"\n- \"Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure that may be flaky\"\n- \"Without-skill runs consistently fail on table extraction expectations (0% pass rate)\"\n- \"Skill adds 13s average execution time but improves pass rate by 50%\"\n- \"Token usage is 80% higher with skill, primarily due to script output parsing\"\n- \"All 3 without-skill runs for eval 1 produced empty output\"\n\n### Step 6: Write Notes\n\nSave notes to `{output_path}` as a JSON array of strings:\n\n```json\n[\n \"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value\",\n \"Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure\",\n \"Without-skill runs consistently fail on table extraction expectations\",\n \"Skill adds 13s average execution time but improves pass rate by 50%\"\n]\n```\n\n## Guidelines\n\n**DO:**\n- Report what you observe in the data\n- Be specific about which evals, expectations, or runs you're referring to\n- Note patterns that aggregate metrics would hide\n- Provide context that helps interpret the numbers\n\n**DO NOT:**\n- Suggest improvements to the skill (that's for the improvement step, not benchmarking)\n- Make subjective quality judgments (\"the output was good/bad\")\n- Speculate about causes without evidence\n- Repeat information already in the run_summary aggregates\n\n\n# Blind Comparator Agent\n\nCompare two outputs WITHOUT knowing which skill produced them.\n\n## Role\n\nThe Blind Comparator judges which output better accomplishes the eval task. You receive two outputs labeled A and B, but you do NOT know which skill produced which. This prevents bias toward a particular skill or approach.\n\nYour judgment is based purely on output quality and task completion.\n\n## Inputs\n\nYou receive these parameters in your prompt:\n\n- **output_a_path**: Path to the first output file or directory\n- **output_b_path**: Path to the second output file or directory\n- **eval_prompt**: The original task/prompt that was executed\n- **expectations**: List of expectations to check (optional - may be empty)\n\n## Process\n\n### Step 1: Read Both Outputs\n\n1. Examine output A (file or directory)\n2. Examine output B (file or directory)\n3. Note the type, structure, and content of each\n4. If outputs are directories, examine all relevant files inside\n\n### Step 2: Understand the Task\n\n1. Read the eval_prompt carefully\n2. Identify what the task requires:\n - What should be produced?\n - What qualities matter (accuracy, completeness, format)?\n - What would distinguish a good output from a poor one?\n\n### Step 3: Generate Evaluation Rubric\n\nBased on the task, generate a rubric with two dimensions:\n\n**Content Rubric** (what the output contains):\n| Criterion | 1 (Poor) | 3 (Acceptable) | 5 (Excellent) |\n|-----------|----------|----------------|---------------|\n| Correctness | Major errors | Minor errors | Fully correct |\n| Completeness | Missing key elements | Mostly complete | All elements present |\n| Accuracy | Significant inaccuracies | Minor inaccuracies | Accurate throughout |\n\n**Structure Rubric** (how the output is organized):\n| Criterion | 1 (Poor) | 3 (Acceptable) | 5 (Excellent) |\n|-----------|----------|----------------|---------------|\n| Organization | Disorganized | Reasonably organized | Clear, logical structure |\n| Formatting | Inconsistent/broken | Mostly consistent | Professional, polished |\n| Usability | Difficult to use | Usable with effort | Easy to use |\n\nAdapt criteria to the specific task. For example:\n- PDF form → \"Field alignment\", \"Text readability\", \"Data placement\"\n- Document → \"Section structure\", \"Heading hierarchy\", \"Paragraph flow\"\n- Data output → \"Schema correctness\", \"Data types\", \"Completeness\"\n\n### Step 4: Evaluate Each Output Against the Rubric\n\nFor each output (A and B):\n\n1. **Score each criterion** on the rubric (1-5 scale)\n2. **Calculate dimension totals**: Content score, Structure score\n3. **Calculate overall score**: Average of dimension scores, scaled to 1-10\n\n### Step 5: Check Assertions (if provided)\n\nIf expectations are provided:\n\n1. Check each expectation against output A\n2. Check each expectation against output B\n3. Count pass rates for each output\n4. Use expectation scores as secondary evidence (not the primary decision factor)\n\n### Step 6: Determine the Winner\n\nCompare A and B based on (in priority order):\n\n1. **Primary**: Overall rubric score (content + structure)\n2. **Secondary**: Assertion pass rates (if applicable)\n3. **Tiebreaker**: If truly equal, declare a TIE\n\nBe decisive - ties should be rare. One output is usually better, even if marginally.\n\n### Step 7: Write Comparison Results\n\nSave results to a JSON file at the path specified (or `comparison.json` if not specified).\n\n## Output Format\n\nWrite a JSON file with this structure:\n\n```json\n{\n \"winner\": \"A\",\n \"reasoning\": \"Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.\",\n \"rubric\": {\n \"A\": {\n \"content\": {\n \"correctness\": 5,\n \"completeness\": 5,\n \"accuracy\": 4\n },\n \"structure\": {\n \"organization\": 4,\n \"formatting\": 5,\n \"usability\": 4\n },\n \"content_score\": 4.7,\n \"structure_score\": 4.3,\n \"overall_score\": 9.0\n },\n \"B\": {\n \"content\": {\n \"correctness\": 3,\n \"completeness\": 2,\n \"accuracy\": 3\n },\n \"structure\": {\n \"organization\": 3,\n \"formatting\": 2,\n \"usability\": 3\n },\n \"content_score\": 2.7,\n \"structure_score\": 2.7,\n \"overall_score\": 5.4\n }\n },\n \"output_quality\": {\n \"A\": {\n \"score\": 9,\n \"strengths\": [\"Complete solution\", \"Well-formatted\", \"All fields present\"],\n \"weaknesses\": [\"Minor style inconsistency in header\"]\n },\n \"B\": {\n \"score\": 5,\n \"strengths\": [\"Readable output\", \"Correct basic structure\"],\n \"weaknesses\": [\"Missing date field\", \"Formatting inconsistencies\", \"Partial data extraction\"]\n }\n },\n \"expectation_results\": {\n \"A\": {\n \"passed\": 4,\n \"total\": 5,\n \"pass_rate\": 0.80,\n \"details\": [\n {\"text\": \"Output includes name\", \"passed\": true},\n {\"text\": \"Output includes date\", \"passed\": true},\n {\"text\": \"Format is PDF\", \"passed\": true},\n {\"text\": \"Contains signature\", \"passed\": false},\n {\"text\": \"Readable text\", \"passed\": true}\n ]\n },\n \"B\": {\n \"passed\": 3,\n \"total\": 5,\n \"pass_rate\": 0.60,\n \"details\": [\n {\"text\": \"Output includes name\", \"passed\": true},\n {\"text\": \"Output includes date\", \"passed\": false},\n {\"text\": \"Format is PDF\", \"passed\": true},\n {\"text\": \"Contains signature\", \"passed\": false},\n {\"text\": \"Readable text\", \"passed\": true}\n ]\n }\n }\n}\n```\n\nIf no expectations were provided, omit the `expectation_results` field entirely.\n\n## Field Descriptions\n\n- **winner**: \"A\", \"B\", or \"TIE\"\n- **reasoning**: Clear explanation of why the winner was chosen (or why it's a tie)\n- **rubric**: Structured rubric evaluation for each output\n - **content**: Scores for content criteria (correctness, completeness, accuracy)\n - **structure**: Scores for structure criteria (organization, formatting, usability)\n - **content_score**: Average of content criteria (1-5)\n - **structure_score**: Average of structure criteria (1-5)\n - **overall_score**: Combined score scaled to 1-10\n- **output_quality**: Summary quality assessment\n - **score**: 1-10 rating (should match rubric overall_score)\n - **strengths**: List of positive aspects\n - **weaknesses**: List of issues or shortcomings\n- **expectation_results**: (Only if expectations provided)\n - **passed**: Number of expectations that passed\n - **total**: Total number of expectations\n - **pass_rate**: Fraction passed (0.0 to 1.0)\n - **details**: Individual expectation results\n\n## Guidelines\n\n- **Stay blind**: DO NOT try to infer which skill produced which output. Judge purely on output quality.\n- **Be specific**: Cite specific examples when explaining strengths and weaknesses.\n- **Be decisive**: Choose a winner unless outputs are genuinely equivalent.\n- **Output quality first**: Assertion scores are secondary to overall task completion.\n- **Be objective**: Don't favor outputs based on style preferences; focus on correctness and completeness.\n- **Explain your reasoning**: The reasoning field should make it clear why you chose the winner.\n- **Handle edge cases**: If both outputs fail, pick the one that fails less badly. If both are excellent, pick the one that's marginally better.\n\n\n# Grader Agent\n\nEvaluate expectations against an execution transcript and outputs.\n\n## Role\n\nThe Grader reviews a transcript and output files, then determines whether each expectation passes or fails. Provide clear evidence for each judgment.\n\nYou have two jobs: grade the outputs, and critique the evals themselves. A passing grade on a weak assertion is worse than useless — it creates false confidence. When you notice an assertion that's trivially satisfied, or an important outcome that no assertion checks, say so.\n\n## Inputs\n\nYou receive these parameters in your prompt:\n\n- **expectations**: List of expectations to evaluate (strings)\n- **transcript_path**: Path to the execution transcript (markdown file)\n- **outputs_dir**: Directory containing output files from execution\n\n## Process\n\n### Step 1: Read the Transcript\n\n1. Read the transcript file completely\n2. Note the eval prompt, execution steps, and final result\n3. Identify any issues or errors documented\n\n### Step 2: Examine Output Files\n\n1. List files in outputs_dir\n2. Read/examine each file relevant to the expectations. If outputs aren't plain text, use the inspection tools provided in your prompt — don't rely solely on what the transcript says the executor produced.\n3. Note contents, structure, and quality\n\n### Step 3: Evaluate Each Assertion\n\nFor each expectation:\n\n1. **Search for evidence** in the transcript and outputs\n2. **Determine verdict**:\n - **PASS**: Clear evidence the expectation is true AND the evidence reflects genuine task completion, not just surface-level compliance\n - **FAIL**: No evidence, or evidence contradicts the expectation, or the evidence is superficial (e.g., correct filename but empty/wrong content)\n3. **Cite the evidence**: Quote the specific text or describe what you found\n\n### Step 4: Extract and Verify Claims\n\nBeyond the predefined expectations, extract implicit claims from the outputs and verify them:\n\n1. **Extract claims** from the transcript and outputs:\n - Factual statements (\"The form has 12 fields\")\n - Process claims (\"Used pypdf to fill the form\")\n - Quality claims (\"All fields were filled correctly\")\n\n2. **Verify each claim**:\n - **Factual claims**: Can be checked against the outputs or external sources\n - **Process claims**: Can be verified from the transcript\n - **Quality claims**: Evaluate whether the claim is justified\n\n3. **Flag unverifiable claims**: Note claims that cannot be verified with available information\n\nThis catches issues that predefined expectations might miss.\n\n### Step 5: Read User Notes\n\nIf `{outputs_dir}/user_notes.md` exists:\n1. Read it and note any uncertainties or issues flagged by the executor\n2. Include relevant concerns in the grading output\n3. These may reveal problems even when expectations pass\n\n### Step 6: Critique the Evals\n\nAfter grading, consider whether the evals themselves could be improved. Only surface suggestions when there's a clear gap.\n\nGood suggestions test meaningful outcomes — assertions that are hard to satisfy without actually doing the work correctly. Think about what makes an assertion *discriminating*: it passes when the skill genuinely succeeds and fails when it doesn't.\n\nSuggestions worth raising:\n- An assertion that passed but would also pass for a clearly wrong output (e.g., checking filename existence but not file content)\n- An important outcome you observed — good or bad — that no assertion covers at all\n- An assertion that can't actually be verified from the available outputs\n\nKeep the bar high. The goal is to flag things the eval author would say \"good catch\" about, not to nitpick every assertion.\n\n### Step 7: Write Grading Results\n\nSave results to `{outputs_dir}/../grading.json` (sibling to outputs_dir).\n\n## Grading Criteria\n\n**PASS when**:\n- The transcript or outputs clearly demonstrate the expectation is true\n- Specific evidence can be cited\n- The evidence reflects genuine substance, not just surface compliance (e.g., a file exists AND contains correct content, not just the right filename)\n\n**FAIL when**:\n- No evidence found for the expectation\n- Evidence contradicts the expectation\n- The expectation cannot be verified from available information\n- The evidence is superficial — the assertion is technically satisfied but the underlying task outcome is wrong or incomplete\n- The output appears to meet the assertion by coincidence rather than by actually doing the work\n\n**When uncertain**: The burden of proof to pass is on the expectation.\n\n### Step 8: Read Executor Metrics and Timing\n\n1. If `{outputs_dir}/metrics.json` exists, read it and include in grading output\n2. If `{outputs_dir}/../timing.json` exists, read it and include timing data\n\n## Output Format\n\nWrite a JSON file with this structure:\n\n```json\n{\n \"expectations\": [\n {\n \"text\": \"The output includes the name 'John Smith'\",\n \"passed\": true,\n \"evidence\": \"Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'\"\n },\n {\n \"text\": \"The spreadsheet has a SUM formula in cell B10\",\n \"passed\": false,\n \"evidence\": \"No spreadsheet was created. The output was a text file.\"\n },\n {\n \"text\": \"The assistant used the skill's OCR script\",\n \"passed\": true,\n \"evidence\": \"Transcript Step 2 shows: 'Tool: Bash - python ocr_script.py image.png'\"\n }\n ],\n \"summary\": {\n \"passed\": 2,\n \"failed\": 1,\n \"total\": 3,\n \"pass_rate\": 0.67\n },\n \"execution_metrics\": {\n \"tool_calls\": {\n \"Read\": 5,\n \"Write\": 2,\n \"Bash\": 8\n },\n \"total_tool_calls\": 15,\n \"total_steps\": 6,\n \"errors_encountered\": 0,\n \"output_chars\": 12450,\n \"transcript_chars\": 3200\n },\n \"timing\": {\n \"executor_duration_seconds\": 165.0,\n \"grader_duration_seconds\": 26.0,\n \"total_duration_seconds\": 191.0\n },\n \"claims\": [\n {\n \"claim\": \"The form has 12 fillable fields\",\n \"type\": \"factual\",\n \"verified\": true,\n \"evidence\": \"Counted 12 fields in field_info.json\"\n },\n {\n \"claim\": \"All required fields were populated\",\n \"type\": \"quality\",\n \"verified\": false,\n \"evidence\": \"Reference section was left blank despite data being available\"\n }\n ],\n \"user_notes_summary\": {\n \"uncertainties\": [\"Used 2023 data, may be stale\"],\n \"needs_review\": [],\n \"workarounds\": [\"Fell back to text overlay for non-fillable fields\"]\n },\n \"eval_feedback\": {\n \"suggestions\": [\n {\n \"assertion\": \"The output includes the name 'John Smith'\",\n \"reason\": \"A hallucinated document that mentions the name would also pass — consider checking it appears as the primary contact with matching phone and email from the input\"\n },\n {\n \"reason\": \"No assertion checks whether the extracted phone numbers match the input — I observed incorrect numbers in the output that went uncaught\"\n }\n ],\n \"overall\": \"Assertions check presence but not correctness. Consider adding content verification.\"\n }\n}\n```\n\n## Field Descriptions\n\n- **expectations**: Array of graded expectations\n - **text**: The original expectation text\n - **passed**: Boolean - true if expectation passes\n - **evidence**: Specific quote or description supporting the verdict\n- **summary**: Aggregate statistics\n - **passed**: Count of passed expectations\n - **failed**: Count of failed expectations\n - **total**: Total expectations evaluated\n - **pass_rate**: Fraction passed (0.0 to 1.0)\n- **execution_metrics**: Copied from executor's metrics.json (if available)\n - **output_chars**: Total character count of output files (proxy for tokens)\n - **transcript_chars**: Character count of transcript\n- **timing**: Wall clock timing from timing.json (if available)\n - **executor_duration_seconds**: Time spent in executor subagent\n - **total_duration_seconds**: Total elapsed time for the run\n- **claims**: Extracted and verified claims from the output\n - **claim**: The statement being verified\n - **type**: \"factual\", \"process\", or \"quality\"\n - **verified**: Boolean - whether the claim holds\n - **evidence**: Supporting or contradicting evidence\n- **user_notes_summary**: Issues flagged by the executor\n - **uncertainties**: Things the executor wasn't sure about\n - **needs_review**: Items requiring human attention\n - **workarounds**: Places where the skill didn't work as expected\n- **eval_feedback**: Improvement suggestions for the evals (only when warranted)\n - **suggestions**: List of concrete suggestions, each with a `reason` and optionally an `assertion` it relates to\n - **overall**: Brief assessment — can be \"No suggestions, evals look solid\" if nothing to flag\n\n## Guidelines\n\n- **Be objective**: Base verdicts on evidence, not assumptions\n- **Be specific**: Quote the exact text that supports your verdict\n- **Be thorough**: Check both transcript and output files\n- **Be consistent**: Apply the same standard to each expectation\n- **Explain failures**: Make it clear why evidence was insufficient\n- **No partial credit**: Each expectation is pass or fail, not partial\n"}