{"owner":"openai","repo":"openai-cookbook","hasSkills":true,"totalSkillsCount":27,"totalTokensCount":14467,"categories":["anthropic-skill","root-instruction","marketplace","plugin-manifest"],"hasMcp":true,"mcpConfig":{"mcpServers":{"openai-cookbook":{"command":"npx","args":["-y","@modelcontextprotocol/server-openai-cookbook"]}}},"found":["examples/evals/realtime_evals/README.md",".codex/skills/docs-editor/SKILL.md","examples/evals/imagegen_evals/README.md","examples/agents_api/apps/document_review/skills/expense-review-policy/SKILL.md","AGENTS.md","examples/evals/realtime_evals/skills/bootstrap-realtime-eval/SKILL.md",".codex/skills/docs-editor/resources/style-guide.md","examples/evals/imagegen_evals/editing_harness/README.md","examples/evals/realtime_evals/run_harness/README.md","examples/evals/realtime_evals/results_viewer/README.md","examples/evals/realtime_evals/crawl_harness/README.md","examples/evals/imagegen_evals/generation_harness/README.md","examples/evals/realtime_evals/shared/tools.json","examples/evals/realtime_evals/walk_harness/README.md","examples/evals/realtime_evals/run_harness/data/sim_cs_address_update.json","examples/evals/realtime_evals/run_harness/data/sim_cs_order_status.json","examples/evals/realtime_evals/run_harness/data/sim_cs_refund.json","examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/definitions.json","examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/restaurant_facts.json","examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/definitions.json","examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/restaurant_facts.json","examples/evals/realtime_evals/skills/bootstrap-realtime-eval/agents/openai.yaml","examples/evals/realtime_evals/skills/bootstrap-realtime-eval/references/harness-selection.md","examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/.agents/plugins/marketplace.json","examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.mcp.json","examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/README.md","examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.codex-plugin/plugin.json"],"skills":{"examples/evals/realtime_evals/README.md":"# Realtime Evals\n\nThis folder contains three evaluation harnesses for the Realtime API that increase in complexity:\n**crawl → walk → run**.\n\n- **crawl**: synthetic single-turn (TTS → stream → grade)\n- **walk**: deterministic replay of saved phone-audio (more realistic, still comparable)\n- **run**: model-simulated multi-turn conversations (finds multi-turn/tooling failures)\n\nDepending on your realtime eval maturity, point Codex (or your preferred coding assistant) to the folder that fits your use case so it can adapt the harness and bootstrap sample data; you can also have it run the included tests to confirm everything works.\n\n## Quickstart\n\nPython 3.12+ required.\n\n```bash\nmake install\nsource .venv/bin/activate\nexport OPENAI_API_KEY=\"your_api_key\"\n```\n\n`make install` creates the local `.venv` and installs both runtime and dev dependencies. It uses `uv` when available and otherwise falls back to `python -m venv` plus `pip install -r requirements.txt -r requirements-dev.txt`.\n\nRun a first command per harness. If uv is not installed, replace `uv run` with `python` and run these scripts with your `.venv` activated:\n\n- Crawl: `uv run python crawl_harness/run_realtime_evals.py`\n- Walk: install ffmpeg (`brew install ffmpeg`), then:\n  - `uv run python walk_harness/generate_audio.py`\n  - `uv run python walk_harness/run_realtime_evals.py`\n- Run: `uv run python run_harness/run_realtime_evals.py --max-examples 1`\n\n## Dev commands\nUse the root `Makefile` for common checks. Run `make install` first to create `.venv`. These targets work with or without `uv`: when `uv` is installed they run through `uv run`, and otherwise they use the matching tool binaries from the local `.venv`.\n\n- `make install`\n- `make streamlit`\n- `make format`\n- `make lint`\n- `make lint-fix`\n- `make typecheck`\n- `make test`\n\n## [Crawl (synthetic single-turn)](./crawl_harness)\n\n**Best for:** Fast iteration and controlled comparisons.\n\n- Uses text prompts from a CSV and synthesizes TTS audio per row.\n- Streams fixed-size audio chunks into a realtime session.\n- Captures the first assistant turn and grades tool-call correctness.\n- Writes results, summary metrics, audio, and event logs per datapoint.\n\nRun:\n\n```\nuv run python crawl_harness/run_realtime_evals.py\n```\n\n## [Walk (saved audio replay)](./walk_harness)\n\n**Best for:** Realism with reproducibility.\n\n- Replays saved, phone-compressed audio (G.711 mu-law WAV at 8 kHz).\n- Keeps chunking and turn boundaries fixed (VAD off).\n- Captures responses, tool calls, latencies, and event logs per datapoint.\n- Output audio is saved as PCM16 WAV for easy listening.\n\nRun:\n\n```\nuv run python walk_harness/generate_audio.py\nuv run python walk_harness/run_realtime_evals.py\n```\n\n## [Run (multi-turn simulation)](./run_harness)\n\n**Best for:** Discovering multi-turn failures and tool behavior.\n\n- Uses a simulator model to generate user turns and an assistant model under test.\n- Streams simulator audio into the assistant with fixed chunking.\n- Mocks tool outputs deterministically and grades with LLM-as-judge.\n- Writes results, summaries, transcripts, and event logs per simulation.\n\nRun:\n\n```\nuv run python run_harness/run_realtime_evals.py\n```\n\n## Results layout\n\nEach harness writes into its own `results/` folder:\n\n- `results/<run_id>/results.csv`: per-example outputs and grades\n- `results/<run_id>/summary.json`: aggregate metrics\n- `results/<run_id>/plots/*.png`: warm-editorial summary charts for scores, latency, tokens, and run shape\n- `results/<run_id>/events/*.jsonl`: full realtime event stream per datapoint\n\nThe shared typed representation of these artifacts for the crawl, walk, and run harnesses lives in `shared/result_types.py`.\n\nTo render charts for an existing run after the fact:\n\n```bash\nuv run python plot_eval_results.py --run-dir run_harness/results/<run_id>\n```\n\n## Results Viewer\n\nUse the Streamlit results viewer to browse saved runs from `crawl_harness`, `walk_harness`, and `run_harness` without opening the raw artifacts by hand.\n\n- `Comparison View`: select a harness, choose one or more saved runs, and compare summary metrics, scores, latency, and token usage across runs.\n- `Run Viewer`: inspect one saved run in detail. Crawl and walk runs show row-level audio artifacts and event logs; run-harness runs use a Simulation Viewer with transcripts, event logs, and turn audio.\n\nRun it from this directory with either:\n\n```bash\nmake streamlit\n```\n\nor:\n\n```bash\ncd results_viewer\nuv run streamlit run app.py\n```\n\nThen open the local Streamlit URL, usually `http://localhost:8501`.\n\n## Common CLI flags\n\nAll harnesses share a core set of flags so you can switch between them easily:\n\n- `--data-csv`, `--results-dir`, `--run-name`\n- `--model` (assistant under test), `--system-prompt-file`, `--tools-file`\n- `--chunk-ms`, `--sample-rate-hz`, `--real-time`\n- `--max-examples` (quick smoke tests)\n- `--skip-plots` (skip post-run PNG chart generation)\n\nRun harness adds multi-turn and simulator-specific flags (see its README).\n",".codex/skills/docs-editor/SKILL.md":"---\nname: docs-editor\ndescription: Review and edit OpenAI Cookbook notebooks, Markdown, and MDX for technical accuracy, clarity, grammar, consistency, runnable examples, and repository publication requirements. Use for editorial reviews, pre-merge documentation checks, or notebook Markdown-cell sweeps in openai-cookbook.\n---\n\n# OpenAI Cookbook docs editor\n\n## Objective\n\nReview Cookbook content conservatively. Improve correctness and readability without changing the author's intent, inventing facts, or creating unrelated churn.\n\n## Workflow\n\n### 1. Define the scope\n\n- Prefer explicit paths or the files changed in the pull request.\n- For pull requests, review changed `.md` and `.mdx` files plus Markdown cells in changed `.ipynb` files.\n- Review `registry.yaml` and `authors.yaml` when content is added, moved, removed, or attributed to a new author.\n- Stay within the requested content. Do not turn a focused review into a repository-wide rewrite.\n\n### 2. Protect notebook integrity\n\n- When the request is editorial, edit only notebook Markdown cells.\n- Preserve code cells, outputs, execution counts, attachments, cell order, cell IDs, and notebook and cell metadata.\n- Avoid reserializing an entire notebook for a small prose change. Inspect the diff for unintended JSON churn.\n- Do not execute notebook code unless the user requests it or code changes are in scope. Markdown-only edits require structural validation, not calls to external services.\n- Never add hard-coded secrets. Document required environment variables such as `OPENAI_API_KEY`.\n\n### 3. Review the content\n\nLoad `resources/style-guide.md` and check:\n\n- Technical claims, API names, model names, identifiers, commands, paths, and URLs.\n- Agreement between prose, code, sample inputs, and expected outputs.\n- Prerequisites, dependencies, environment variables, and steps needed to run an example.\n- File and notebook names, headings, lists, numbering, cross-references, and links.\n- Grammar, spelling, punctuation, terminology, capitalization, and concise wording.\n- Publication metadata in `registry.yaml` and author metadata in `authors.yaml` when applicable.\n\nVerify uncertain technical claims from an authoritative source or flag them for review. Do not guess.\n\n### 4. Prioritize findings\n\nUse the repository's review bar:\n\n- **P0**: Broken notebook JSON; hard-coded secrets; materially false or unsafe guidance; or any repository-defined P0 issue, including naming violations, typos, broken links, inconsistent formatting, placeholder identifiers, undocumented environment variables, or out-of-sync publication metadata.\n- **P1**: Misleading instructions, prose that contradicts code or output, missing prerequisites, or a required link that does not work.\n- **P2**: Clear improvements to grammar, consistency, organization, or readability that do not affect correctness.\n- **P3**: Optional polish with little effect on comprehension.\n\n### 5. Make safe edits\n\n- Fix obvious typos, grammar, punctuation, broken local references, and clear prose/code mismatches.\n- Keep changes minimal, local, and reversible.\n- Preserve exact literals, code identifiers, commands, values, counts, conditions, modality, negation, and uncertainty markers.\n- Tighten prose only when meaning stays unchanged. Prefer concrete nouns, strong verbs, short introductions, and direct instructions.\n- Flag any change that depends on product knowledge or could alter meaning instead of guessing.\n- Do not make file-wide formatting or stylistic changes solely for uniformity.\n\n### 6. Validate the result\n\nFor any changed notebook, run the repository validator:\n\n```bash\nenv -u VIRTUAL_ENV uv run --with nbformat python .github/scripts/check_notebooks.py\n```\n\nThen:\n\n- Run `git diff --check`.\n- Inspect the diff and confirm that notebook changes are limited to intended Markdown cells.\n- Run an available YAML linter if `registry.yaml` or `authors.yaml` changed.\n- Run the relevant example or test only when executable content changed and its dependencies are available.\n\n### 7. Report the review\n\nProvide:\n\n- A short summary of the scope and edits.\n- Auto-fixed items.\n- Remaining findings grouped by priority with file and cell or line references.\n- The validation commands run and their results.\n\nIf no issues remain, say so explicitly.\n","examples/evals/imagegen_evals/README.md":"# Image Generation + Editing Evals\n\nThis folder contains a lightweight vision eval harness plus example runners for\nimage generation and image editing. The code mirrors the structure of\n`examples/evals/realtime_evals/` so you can adapt it quickly.\n\n**Directory layout**\n\n- `vision_harness/`: minimal shared library (types, runners, graders, evaluate loop)\n- `generation_harness/`: text-to-image evals (UI mockups + marketing flyer)\n- `editing_harness/`: image-edit evals (virtual try-on + logo edit)\n- `shared/`: reporting and optional rendering helpers\n\n## Quickstart\n\nPython 3.9+ required.\n\n```bash\npip install -r requirements.txt\nexport OPENAI_API_KEY=\"your_api_key\"\n```\n\nRun a harness:\n\n- Generation: `python generation_harness/run_imagegen_evals.py`\n- Editing: `python editing_harness/run_imagegen_evals.py`\n\n## What the harness does\n\n1. Builds a small set of `TestCase` objects (prompt + criteria).\n2. Runs the image model for each case.\n3. Grades each output with an LLM judge using a strict JSON schema.\n4. Writes results and artifacts to `results/<run_id>/`.\n\nThe harness is intentionally small so you can copy/paste parts into your own\nproduction eval setup.\n\n## Example cases\n\nGeneration cases:\n- `ui_checkout_mockup`: mobile checkout screen with strict text + layout rules\n- `coffee_flyer_generation`: marketing flyer with exact copy constraints\n\nEditing cases:\n- `vto_jacket_tryon`: virtual try-on with reference person + garment\n- `logo_year_edit`: precision logo text edit\n\n## Required assets (editing harness)\n\nThe editing harness expects these files in `images/`:\n\n- `images/base_woman.png`\n- `images/jacket.png`\n- `images/logo_generation_1.png`\n\n## Results layout\n\nEach harness writes into its own `results/` folder:\n\n- `results/<run_id>/results.json`: per-example outputs and grades\n- `results/<run_id>/results.csv`: tabular results\n- `results/<run_id>/summary.json`: aggregated metrics\n- `results/<run_id>/artifacts/*.png`: generated or edited images\n\nTip: keep `results/` out of git (delete after runs or add to `.gitignore`).\n\n## Common CLI flags\n\nBoth harnesses accept a shared core set of flags:\n\n- `--results-dir`, `--run-name`\n- `--model` (image model under test), `--judge-model` (LLM judge)\n- `--num-images` (images per case)\n- `--cases` (comma-separated list of case ids)\n\nGeneration harness adds `--image-size` and `--run-ocr` (text extraction check).\n\n## Suggested workflows\n\n- Quick smoke test:\n  - `python generation_harness/run_imagegen_evals.py --cases ui_checkout_mockup`\n  - `python editing_harness/run_imagegen_evals.py --cases vto_jacket_tryon`\n- Compare model variants: run once per model name and compare `summary.json`.\n- Use `--run-ocr` on the flyer case to validate exact text rendering.\n\n## Adapting for your use case\n\n- Replace prompts/criteria in the harness scripts.\n- Update JSON schemas to match your rubric (and parse them into `Score` values).\n- Add new cases by creating new `TestCase` objects and graders.\n- Swap the output store to change where artifacts are written.\n\n## Notes\n\n- This harness uses the OpenAI Images API and a judge model. Costs scale with\n  the number of images and judge calls.\n- Judge outputs are best treated as signals, not ground truth. Keep a small\n  human review loop for high-stakes decisions.\n","examples/agents_api/apps/document_review/skills/expense-review-policy/SKILL.md":"---\nname: expense-review-policy\ndescription: Review invoices and contracts against accounts-payable policy before human approval.\n---\n\n# Expense review policy\n\nPolicy ID: `AP-104`.\n\nApply this policy to invoices, expense receipts, and service contracts.\n\n## Invoices and expenses\n\n- Read the entire invoice, including the vendor and every line item.\n- Extract every line item's quantity and unit price before calculating. Check the extracted list against every source line; an empty list is an error, not a zero subtotal.\n- Use Python's `decimal.Decimal` to multiply quantities by unit prices and sum the line totals. Compare this subtotal with the printed subtotal, then add shipping to calculate the total.\n- Strip currency separators, compare the result with `Total amount due`, and report the exact difference (stated total minus calculated total).\n- Require a purchase order or another documented approval.\n- Flag missing receipts, unsupported charges, and changed payment instructions.\n- Escalate incorrect totals and unverified changes to bank details.\n\n## Contracts\n\n- Read every clause and report all matching risks, even when one already requires escalation.\n- Flag automatic renewals and restrictive cancellation windows.\n- Flag unilateral price increases and missing liability limits.\n- Escalate customer-data sharing or subcontractor access without approval.\n- Identify missing confidentiality, security, or termination terms.\n\n## Review decision\n\n- `needs_info`: Required documentation or supporting information is missing.\n- `escalated`: Fraud indicators, financial discrepancies, or risky terms need review.\n- `ready_for_approval`: No policy violations remain, but a human must still approve.\n\nWrite `/workspace/output/<document-stem>.json` with the following fields:\n\n```json\n{\n  \"document\": \"invoice.txt\",\n  \"document_type\": \"invoice\",\n  \"policy_id\": \"AP-104\",\n  \"decision\": \"escalated\",\n  \"vendor\": \"Cedar Office Supply\",\n  \"amount\": 6420,\n  \"issues\": [\"The claimed total exceeds the calculated total.\"],\n  \"recommendation\": \"Escalate for human review.\",\n  \"calculation\": {\n    \"line_items\": [{\"quantity\": 2, \"unit_price\": 100}],\n    \"shipping\": 20,\n    \"calculated_total\": 220,\n    \"difference\": 6200\n  }\n}\n```\n\nEvery `issues` entry must be a plain-English string, not a nested object.\nUse the actual document values, not the illustrative numbers above. Include every\ninvoice line item in `calculation`; `amount` is the stated total. For contracts,\nset `document_type` to `contract` and `calculation` to `null`.\n\nOnly the coordinator writes `summary.json`. A specialist writes its assigned report.\n\nNever approve payments, sign contracts, or take external actions.\n","AGENTS.md":"# Repository Guidelines\n\n## Project Structure & Module Organization\n\nThe cookbook is organized around runnable examples and reference articles for OpenAI APIs. Place notebooks and Python scripts under `examples/<topic>/`, grouping related assets inside topic subfolders (for example, `examples/agents_sdk/`). Narrative guides and long-form docs live in `articles/`, and shared diagrams or screenshots belong in `images/`. Update `registry.yaml` whenever you add content so it appears on cookbook.openai.com, and add new author metadata in `authors.yaml` if you want custom attribution. Keep large datasets outside the repo; instead, document how to fetch them in the notebook.\n\n## Build, Test, and Development Commands\n\nUse a virtual environment to isolate dependencies:\n\n- `python -m venv .venv && source .venv/bin/activate`\n- `pip install -r examples/<topic>/requirements.txt` (each sample lists only what it needs)\n- `jupyter lab` or `jupyter notebook` to develop interactively\n- `python .github/scripts/check_notebooks.py` to validate notebook structure before pushing\n\n## Coding Style & Naming Conventions\n\nWrite Python to PEP 8 with four-space indentation, descriptive variable names, and concise docstrings that explain API usage choices. Name new notebooks with lowercase, dash-or-underscore-separated phrases that match their directory—for example `examples/gpt-5/prompt-optimization-cookbook.ipynb`. Keep markdown cells focused and prefer numbered steps for multi-part workflows. Store secrets in environment variables such as `OPENAI_API_KEY`; never hard-code keys inside notebooks.\n\n## Testing Guidelines\n\nExecute notebooks top-to-bottom after installing dependencies and clear lingering execution counts before committing. For Python modules or utilities, include self-check cells or lightweight `pytest` snippets and show how to run them (for example, `pytest examples/object_oriented_agentic_approach/tests`). When contributions depend on external services, mock responses or gate the cells behind clearly labeled opt-in flags.\n\n## Commit & Pull Request Guidelines\n\nUse concise, imperative commit messages that describe the change scope (e.g., \"Add agent portfolio collaboration demo\"). Every PR should provide a summary, motivation, and self-review, and must tick the registry and authors checklist from `.github/pull_request_template.md`. Link issues when applicable and attach screenshots or output snippets for UI-heavy content. Confirm CI notebook validation passes locally before requesting review.\n\nBefore a pull request is merged, run the repository's `docs-editor` skill on every changed article, Markdown or MDX file, and notebook Markdown cell. For editorial notebook changes, preserve code cells, outputs, execution counts, attachments, cell order, and metadata. Fix safe issues, rerun the relevant checks, and resolve or explicitly document all remaining P0 and P1 findings in the pull request.\n\n## Metadata & Publication Workflow\n\nNew or relocated content must have an entry in `registry.yaml` with an accurate path, date, and tag set so the static site generator includes it. When collaborating, coordinate author slugs in `authors.yaml` to avoid duplicates, and run `python -m yaml lint registry.yaml` (or your preferred YAML linter) to catch syntax errors before submitting.\n\n## Review Guidelines\n\nThese are considered priority 0 issues for this repo, in addition to the normal priority for possible issues.\n- Verify file, function, and notebook names follow the repo's naming conventions and clearly describe their purpose.\n- Scan prose and markdown for typos, broken links, and inconsistent formatting before approving.\n- Check that code identifiers remain descriptive (no leftover placeholder names) and that repeated values are factored into constants when practical.\n- Ensure notebooks or scripts document any required environment variables instead of hard-coding secrets or keys.\n- Confirm metadata files (`registry.yaml`, `authors.yaml`) stay in sync with new or relocated content.\n\n## Recent Learnings\n\n- **`uv run` can inherit the wrong virtualenv in this repo** -> Clear `VIRTUAL_ENV` (for example `env -u VIRTUAL_ENV uv run ...`) when an unrelated environment is active -> Avoids misleading mismatch warnings and makes it clear the repo's `.venv` is the interpreter actually running the harnesses.\n- **Realtime eval shared imports can resolve the wrong module under pytest** -> Add `shared/__init__.py` and ensure tests prepend `examples/evals/realtime_evals` to `sys.path` before importing `shared.*` -> Prevents collection failures caused by unrelated installed packages named `shared`.\n- **Run-level grades can be overweighted by long simulations** -> Store turn-level grades on the matching turn and trace-level grades on one row per simulation instead of copying them onto every row -> Keeps `results.csv` row semantics intact and prevents summary means from favoring longer conversations.\n- **Synthetic-audio scaffold requests can pick the wrong harness** -> Default unspecified synthetic-audio evals to `crawl` text-to-TTS and reserve `walk` for replay-specific audio traits like noise, telephony artifacts, or speaker characteristics -> Keeps new realtime evals on the simplest harness unless audio realism is itself under test.\n- **Task-specific single-turn grading can outgrow the shared crawl schema** -> Keep the shared crawl harness for realtime execution, then add eval-local wrapper scripts that post-grade domain-specific quality and overwrite `results.csv` while preserving `results_base.csv` -> Avoids forking the harness when a use case needs richer grading than tool-call correctness.\n- **Synthetic learner audio can sound like eval scaffolding** -> Write `user_text` as a realistic in-app learner request and keep evaluation rules in metadata plus the system prompt -> Produces audio inputs that match the product surface instead of teaching the model the grading rubric through the spoken prompt.\n","examples/evals/realtime_evals/skills/bootstrap-realtime-eval/SKILL.md":"---\nname: bootstrap-realtime-eval\ndescription: Bootstrap a new realtime eval folder inside this cookbook repo by choosing the right harness from examples/evals/realtime_evals, scaffolding prompt/tools/data files, generating a useful README, and validating it with smoke, full eval, and test runs. Use when a user wants to start a new crawl, walk, or run realtime eval in this repository.\n---\n\n# Bootstrap Realtime Eval\n\nUse this skill when the user wants a new realtime eval scaffold under `examples/evals/realtime_evals/`.\n\nThis skill is repo-specific. Do not copy harness code into the generated folder. The generated eval should point at the shared harnesses already in:\n\n- `examples/evals/realtime_evals/crawl_harness`\n- `examples/evals/realtime_evals/walk_harness`\n- `examples/evals/realtime_evals/run_harness`\n\n## Inputs To Collect\n\nAlways ask the user for the minimum set needed to choose and scaffold the eval before you create files, run the scaffold script, or author starter data. Do not skip this just because you can infer a default.\n\nAsk for:\n\n- Eval name\n- Goal or scenario\n- Harness choice, or enough context to recommend one\n- System prompt path or inline text\n- Tools JSON path or tool descriptions\n- Data path or source materials\n- Desired graders\n\nIf the user does not know which harness they want, explain the options briefly and recommend one. See `references/harness-selection.md`.\n\nWhen the user asks for synthetic audio but does not specify a harness, default to `crawl` text-to-TTS unless they need the generated audio to carry particular noise, telephony artifacts, speaker characteristics, or other replay-specific properties. Use `walk` for those cases.\n\nKeep the questions concise and grouped into one short batch whenever possible.\n\nIf the user only provides `user_text` or a short task description, still ask the questions above first. If they answer only partially, then infer the remaining low-risk details, call out the assumptions, and make the scaffold easy to revise later.\n\n## Workflow\n\n1. Ask the user for the required inputs first.\n   - Do this before making files or selecting a final harness.\n   - If the user already supplied some of the inputs, ask only for the missing ones.\n   - If you recommend a harness, wait for the user response before scaffolding.\n\n2. Pick the harness.\n   - `crawl`: single-turn text-to-TTS.\n   - `walk`: replay saved audio or generate audio from text rows.\n   - `run`: multi-turn simulation with tool mocks and judge criteria.\n   - If the user wants synthetic audio but does not care about replay-specific audio characteristics, prefer `crawl` over `walk`.\n\n3. Normalize the inputs.\n   - If the user gives inline prompt or tool content, write it into the generated folder.\n   - If the user gives CSV data, inspect it with pandas before wiring it in.\n   - If the data is not yet harness-ready, scaffold the files and then have you author the starter dataset directly in those files.\n   - If the user only gives `user_text`, infer `example_id` values and leave optional grading fields blank unless you have enough signal to fill them.\n   - Ground the starter data in the use case, prompts, tools, and any other material the user provided.\n   - Make the starter data realistic. Put yourself in the shoes of the end user in that use case and craft datapoints that a real user would plausibly say or do.\n\n4. Be proactive when data is missing.\n   - `crawl`: you should author 3 starter rows covering one happy path and a couple of nearby variants or edge cases.\n   - `walk`: you should author 3 source CSV rows and prepare the audio-generation step so the user can create audio immediately.\n   - `run`: you should author 2 starter simulations, not 1.\n   - Do not rely on deterministic script-generated samples for use-case-specific starter data.\n   - Show the generated starter samples to the user, ask for a quick greenlight or correction, then expand only after feedback when the task calls for more coverage.\n\n5. Run the scaffold script:\n\n```bash\npython examples/evals/realtime_evals/skills/bootstrap-realtime-eval/scripts/bootstrap_realtime_eval.py --name \"<eval_name>\" --harness \"<crawl|walk|run>\"\n```\n\nAdd flags for prompt, tools, data, graders, and run-specific fields as needed. Read the script help if you need the exact flag names.\n\n6. Review the generated folder.\n   - Confirm the README is accurate for the selected harness.\n   - Confirm the system prompt, tools, and data files point at the generated folder.\n   - If the user provided real data, make sure it is wired in instead of leaving starter placeholders.\n   - If you authored inferred starter data, show the user the generated rows or simulations before scaling up.\n\n7. Enrich the scaffold.\n   - For `crawl` and `walk`, make the CSV realistic and ensure the expected tool columns are present.\n   - For `walk`, if the dataset lacks `audio_path`, use the shared `walk_harness/generate_audio.py` flow described in the README.\n   - For `run`, make sure `simulations.csv` and the starter `sim_*.json` file reflect the user’s scenario, tool mocks, and graders.\n\n8. Validate before returning.\n   - Run a smoke eval for the generated folder.\n   - If the smoke eval succeeds, automatically run the full eval for the generated folder in the same turn.\n   - If the smoke eval fails, stop and fix or report the blocker before attempting the full eval.\n   - Run `pytest examples/evals/realtime_evals/tests -q`.\n   - Inspect the generated README and at least one generated data file.\n\n## Hard Rules\n\n- Keep the generated folder inside `examples/evals/realtime_evals/`.\n- Ask the user for the missing setup information before scaffolding. Do not jump straight to building a default eval when the request is to create a new eval.\n- Do not duplicate the main harness scripts into the generated folder.\n- Do not write generic placeholder starter data when the use case provides enough context to do better.\n- Use the prompt, tools, and scenario details to make the initial datapoints feel like a real user interaction.\n- The README must include:\n  - why this harness was chosen\n  - which files to edit first\n  - smoke and full run commands\n  - data contract notes\n  - troubleshooting notes\n- Prefer assistant-specific flags for run harness commands:\n  - `--assistant-system-prompt-file`\n  - `--assistant-tools-file`\n\n## Completion Criteria\n\nTreat the task as complete only when:\n\n- the folder exists under `examples/evals/realtime_evals/<name>_realtime_eval/`\n- `README.md`, `system_prompt.txt`, and `tools.json` exist\n- harness-specific starter data exists, authored by you when the user did not provide it\n- the smoke command has been run or is blocked for a clear reason\n- if the smoke command succeeded, the full eval command has also been run or is blocked for a clear reason\n- `pytest examples/evals/realtime_evals/tests -q` has been run\n\n## Learnings\n\nWhen this skill uncovers a reusable workflow or harness constraint that should guide future bootstrap work, add a short note here.\n\nKeep learnings concise and action-oriented:\n\n- **Problem -> Fix -> Why**\n\nOnly add items that are likely to help future realtime-eval scaffolding in this repo. Remove stale items when they no longer apply.\n\n- **`gpt-realtime` temperature is unsupported** -> Do not add a `temperature` field or CLI flag when scaffolding `gpt-realtime` evals -> Avoids invalid config and keeps runs aligned with the realtime harness constraints.\n- **Starter samples should come from the model, not the scaffold script** -> Use the script to create the folder structure and template files, then have you author the initial rows or simulations from the user’s use case -> Produces more relevant starter data and makes iteration easier with the user.\n- **Synthetic audio can be overfit to the wrong harness** -> Default unspecified synthetic-audio requests to `crawl` text-to-TTS and reserve `walk` for replay-specific audio characteristics like noise or telephony artifacts -> Keeps the bootstrap path simpler unless audio realism is the actual target.\n- **Smoke-only validation can leave scaffolds half-proven** -> After a successful smoke run, automatically run the full eval before declaring the bootstrap complete -> Catches dataset-wide or late-row failures that a one-example smoke test misses.\n",".codex/skills/docs-editor/resources/style-guide.md":"# OpenAI Cookbook editorial checklist\n\nUse this checklist for articles and notebook Markdown cells in `openai-cookbook`. Apply only the rules relevant to the content under review.\n\n## Audience and purpose\n\n- Write for developers who want to understand and run an OpenAI API example.\n- State the task or outcome early. Keep background material only when it helps the reader use the example correctly.\n- Assume general programming knowledge, but explain Cookbook-specific setup, choices, and tradeoffs.\n- Avoid promotional language, vague praise, and claims that the example does not demonstrate.\n\n## Technical accuracy\n\n- Match API, model, method, parameter, field, file, and environment-variable names exactly.\n- Keep prose consistent with the code, sample inputs, outputs, diagrams, and nearby cells.\n- Check counts, ordered steps, cross-references, defaults, constraints, and stated results.\n- Do not infer a correction when the intended fact is unclear. Preserve the source and flag it for review.\n- Use authoritative sources to verify uncertain product behavior.\n- Do not imply that an optional step is required or that a required step is optional.\n\n## Runnable examples\n\n- List prerequisites and dependencies that readers need before running the example.\n- Document required environment variables. Never hard-code API keys, tokens, credentials, or other secrets.\n- Keep setup instructions in the same order that readers perform them.\n- Introduce code with the goal or reason for the step; do not narrate every line.\n- Use realistic inputs and clearly label placeholders.\n- Show expected output only when it helps readers verify success, and keep it consistent with the code.\n- Keep external-service calls and other opt-in behavior clearly labeled.\n\n## Notebook integrity\n\n- Edit Markdown cells only during an editorial pass.\n- Preserve code cells, outputs, execution counts, attachments, cell IDs, cell order, and metadata.\n- Keep notebook JSON valid and avoid whole-file reserialization for a small prose edit.\n- Keep each Markdown cell focused. Split or merge cells only when the structure is genuinely confusing and the request permits it.\n- Use Markdown headings in a logical hierarchy without skipping levels unnecessarily.\n\n## Prose\n\n- Lead with the point. Prefer concrete nouns and strong verbs.\n- Use direct instructions and address the reader as “you” when helpful.\n- Prefer active voice, but keep passive voice when the actor is unknown or unimportant.\n- Remove throat-clearing, repeated conclusions, filler, and ornamental transitions.\n- Keep paragraphs short enough to scan and use lists for genuinely parallel items.\n- Use numbered lists for ordered procedures and bullets for unordered groups.\n- Keep sibling headings and list items parallel.\n- Use sentence-style capitalization for headings and do not end headings with periods.\n- Preserve useful warmth and explanation; do not flatten the content into terse fragments.\n\n## Terminology and formatting\n\n- Use exact product terminology and consistent capitalization.\n- Format commands, options, environment variables, paths, filenames, code identifiers, and literal values with backticks.\n- Match the capitalization and spelling used by the language or API.\n- Expand uncommon abbreviations on first use unless the surrounding audience clearly knows them.\n- Use inclusive, precise language and neutral role terms.\n- Do not replace a technically precise term merely to vary wording.\n\n## Links and assets\n\n- Prefer descriptive link text over “here” or a pasted URL.\n- Verify local links, anchors, image paths, and referenced notebook or article paths.\n- Use stable, authoritative external links when available.\n- Give images meaningful alt text unless they are purely decorative.\n- Keep diagrams, screenshots, captions, and prose consistent.\n\n## Publication metadata\n\n- Add or update `registry.yaml` for new, moved, renamed, or removed published content.\n- Confirm registry paths, dates, tags, and titles match the content.\n- Add or update `authors.yaml` when introducing or changing author attribution.\n- Check that filenames follow the repository's naming conventions and clearly describe the content.\n\n## Final review\n\n- Read each edited passage in context, not only as an isolated diff hunk.\n- Confirm no factual meaning, literal value, condition, negation, or uncertainty changed unintentionally.\n- Confirm all repository-defined P0 issues are fixed.\n- Run the notebook structural validator for changed notebooks and inspect the final diff for unintended changes.\n","examples/evals/imagegen_evals/editing_harness/README.md":"# Editing Harness\n\n**Best for:** high-precision checks where edits must be exact and non-target\ncontent should remain unchanged.\n\nThis harness runs two example cases:\n\n- `vto_jacket_tryon`: virtual try-on using reference person + garment\n- `logo_year_edit`: precision logo text edit\n\n## Run\n\n```bash\npython editing_harness/run_imagegen_evals.py\n```\n\nRun a single case (use `--cases`):\n\n```bash\npython editing_harness/run_imagegen_evals.py --cases vto_jacket_tryon\n```\n\nRun multiple cases (comma-separated, no spaces):\n\n```bash\npython editing_harness/run_imagegen_evals.py --cases vto_jacket_tryon,logo_year_edit\n```\n\n## Flags (when to use them)\n\n- `--cases`: limit runs to specific case ids. Valid values:\n  - `vto_jacket_tryon`\n  - `logo_year_edit`\n- `--model`: image model under test (defaults to `gpt-image-1.5`).\n- `--judge-model`: LLM used to grade outputs (defaults to `gpt-5.2`).\n- `--num-images`: number of edited images to generate per case (defaults to 1).\n\n## Inputs\n\nThis harness expects the following reference images in `images/`:\n\n- `images/base_woman.png`\n- `images/jacket.png`\n- `images/logo_generation_1.png`\n\n## Outputs\n\nResults are written under `editing_harness/results/<run_id>/`:\n\n- `results.json` and `results.csv` with per-example scores\n- `summary.json` with aggregated metrics\n- `artifacts/` for edited images\n\n## Adapting the harness\n\n- Replace the input images and edit prompts in `run_imagegen_evals.py`.\n- Update the schemas and verdict rules to fit your workflow.\n","examples/evals/realtime_evals/run_harness/README.md":"# Run Harness\n\nModel-simulated multi-turn eval harness for the Realtime API. This harness uses\none realtime model as the user simulator (audio-only) and another realtime model\nas the assistant under test. It replays fixed chunking (VAD off), mocks tools\nfrom the simulation JSON, and records full traces.\n\n## What it does\n\n- Loads `run_harness/data/simulations.csv` with pandas.\n- Reads per-simulation JSON files defining scenario, simulator identity, tool mocks,\n  and LLM-as-judge grading criteria.\n- Generates user audio turns via a realtime simulator model (audio-only).\n- Streams user audio to the assistant in fixed-size chunks and commits manually.\n- Captures assistant audio/text, tool calls, tool outputs, and latencies.\n- Grades turn-level and trace-level criteria with an LLM-as-judge.\n- Writes `results.csv`, `summary.json`, and full trace logs under `run_harness/results/`.\n- Renders styled PNG plots under `run_harness/results/<run_id>/plots/` by default.\n\n## Files\n\n- `run_harness/run_realtime_evals.py`: Run harness script.\n- `run_harness/data/simulations.csv`: Index of simulation files.\n- `run_harness/data/sim_*.json`: Simulation definitions (this repo currently ships 3 examples).\n- `run_harness/results/<run_id>/events/*.jsonl`: Full event trace per simulation.\n- `run_harness/results/<run_id>/conversations/*.txt`: Human-readable transcript with tool calls.\n\n## Simulation definitions (high level)\n\nEach `sim_*.json` file defines:\n\n- **the scenario** (what the user is trying to do)\n- **tool mocks** (what each tool should return when called)\n- **judge rubric** (what should be graded per turn / overall)\n\nThis lets you run repeatable multi-turn evals without needing live backend integrations.\n\n## How to run\n\nFrom repo root:\n\n```bash\npython run_harness/run_realtime_evals.py --max-examples 1\n```\n\nCommon options:\n\n- `--data-csv`: Simulation index CSV.\n- `--model`: Alias for the assistant model under test.\n- `--assistant-model`: Realtime model under test (overrides `--model`).\n- `--simulator-model`: Realtime model used as the user.\n- `--system-prompt-file`: Alias for assistant system prompt file.\n- `--tools-file`: Alias for assistant tools file.\n- `--assistant-system-prompt-file`: Override assistant system prompt file (more specific).\n- `--assistant-tools-file`: Override tools file for the assistant (more specific).\n- `--simulator-system-prompt`: Override simulator system prompt (string).\n- `--chunk-ms`, `--sample-rate-hz`, `--real-time`.\n- `--max-turns`, `--max-examples`.\n- `--skip-plots`.\n\nNotes:\n\n- Temperature and max output tokens are intentionally not supported.\n- The simulator emits audio only. Transcripts are captured from audio deltas if available.\n- Tool outputs are mocked deterministically from each simulation JSON.\n- `results.csv` includes an `event_log_path` column pointing to the JSONL trace.\n","examples/evals/realtime_evals/results_viewer/README.md":"# Results Viewer\n\nThis directory contains a [Streamlit](https://streamlit.io/) app for exploring\nsaved realtime eval runs from the crawl, walk, and run harnesses, plus\nbootstrap-generated eval folders created under `examples/evals/realtime_evals/`.\n\nThe app auto-discovers run directories under:\n\n- `crawl_harness/results/`\n- `walk_harness/results/`\n- `run_harness/results/`\n- `*_realtime_eval/results/` when the folder includes `bootstrap_manifest.json`\n\n## What It Shows\n\n- **Comparison View**: compare summary metrics across one or more saved runs\n- **Run Viewer**: inspect a single saved run, including:\n  - `results.csv` rows\n  - crawl/walk input and output audio artifacts\n  - crawl/walk per-example event logs\n  - run-harness simulation transcripts, event logs, and turn audio\n\nNote: `run_harness` runs are inspected via the app's **Simulation Viewer**,\nwhich has a different UI and a different artifact set than the crawl/walk\nviewer.\n\n## Run Locally\n\nFrom `examples/evals/realtime_evals/`:\n\n```bash\nuv venv .venv\nsource .venv/bin/activate\nuv sync --group dev\ncd results_viewer\nuv run streamlit run app.py\n```\n\nThen open the local URL that Streamlit prints, usually\n`http://localhost:8501`.\n\nIf you are using the pip-based install path instead of `uv`, install the dev\ndependencies first so `streamlit` is available:\n\n```bash\npip install -r requirements.txt -r requirements-dev.txt\ncd results_viewer\nstreamlit run app.py\n```\n\n## Expected Data Layout\n\nThe viewer expects each saved run directory to contain:\n\n- `summary.json` for aggregate metrics\n- `results.csv` for per-example results\n\nFor crawl and walk runs, the app can also display:\n\n- `audio/<example_id>/input.wav`\n- `audio/<example_id>/output.wav`\n- `events/<example_id>.jsonl`\n\nThe app discovers runs recursively, so nested result directories are fine as\nlong as those files are present.\n","examples/evals/realtime_evals/crawl_harness/README.md":"# Crawl Harness\n\nDeterministic single-turn replay for Realtime evals. This harness feeds fixed audio\ninputs to a Realtime session so you can compare runs with minimal variance.\nIt is best for quick iteration and controlled comparisons.\n\n## What it does\n\n- Loads a CSV of eval prompts from `crawl_harness/data/`.\n- Uses TTS to synthesize a fixed input audio file per row.\n- Streams audio into the Realtime API in fixed-size chunks.\n- Captures the first assistant turn (text/audio/tool calls).\n- Grades tool-call correctness and tool-call-arg correctness only.\n- Writes artifacts into `crawl_harness/results/<run_id>/`.\n\n## Inputs\n\nCSV columns (required):\n\n- `example_id`: Unique ID for the datapoint.\n- `user_text`: Text prompt to synthesize into audio.\n- `gt_tool_call`: Expected tool name or empty if no tool call is expected.\n- `gt_tool_call_arg`: Expected tool arguments as JSON string or empty.\n\n## Outputs\n\nEach run writes:\n\n- `results.csv`: Per-example outputs and grades.\n- `summary.json`: Aggregate metrics (latency + correctness).\n- `plots/*.png`: Styled score, latency, token, and status charts.\n- `audio/<example_id>/input.wav`: The TTS input audio.\n- `audio/<example_id>/output.wav`: The model output audio (if any).\n- `events/<example_id>.jsonl`: Realtime event stream for the datapoint.\n\n## How it works\n\n1. TTS generates PCM audio for each row.\n2. PCM is wrapped into a WAV file for easy playback.\n3. Audio is streamed into a Realtime session with fixed chunk sizes.\n4. The harness listens for `response.done` and records tool calls + output.\n5. Grades are computed:\n   - `tool_call_correctness`: correct tool chosen (or no tool call when none expected).\n   - `tool_call_arg_correctness`: expected args present in the tool call.\n\n## Run\n\nFrom repo root:\n\n```bash\npython crawl_harness/run_realtime_evals.py\n```\n\nCommon options:\n\n- `--data-csv`: Path to a CSV file.\n- `--results-dir`: Output folder (defaults to `crawl_harness/results`).\n- `--run-name`: Optional run name (defaults to timestamp).\n- `--model`: Realtime model name.\n- `--tts-model`: TTS model name.\n- `--system-prompt-file`: System prompt file path.\n- `--tools-file`: Tools JSON file path.\n- `--voice`: Output voice name.\n- `--chunk-ms`: Audio chunk size in ms.\n- `--sample-rate-hz`: Input sample rate in Hz.\n- `--input-audio-format`: Input audio format (default `pcm16`).\n- `--output-audio-format`: Output audio format (default `pcm16`).\n- `--real-time`: Stream audio in real-time cadence.\n- `--max-examples`: Limit number of examples for quick checks.\n- `--skip-plots`: Skip post-run PNG chart generation.\n\n## Adapt\n\n- Add or replace rows in `crawl_harness/data/customer_service_synthetic.csv`.\n- Change tools or system prompt in `shared/`.\n- Update grading logic in `run_realtime_evals.py`.\n\n## Notes\n\n- The harness only evaluates the first assistant turn and does not provide tool outputs.\n- `results.csv` includes an `event_log_path` column pointing to the JSONL trace.\n- If you expect tools to be called reliably, adjust the system prompt or tool choice.\n","examples/evals/imagegen_evals/generation_harness/README.md":"# Generation Harness\n\n**Best for:** fast iteration on prompt-following, layout, and text-rendering\nquality for text-to-image workflows.\n\nThis harness runs two example cases:\n\n- `ui_checkout_mockup`: mobile checkout UI mockup\n- `coffee_flyer_generation`: marketing flyer with strict copy constraints\n\n## Run\n\n```bash\npython generation_harness/run_imagegen_evals.py\n```\n\nQuick smoke test for a single case (use `--cases`):\n\n```bash\npython generation_harness/run_imagegen_evals.py --cases ui_checkout_mockup\n```\n\nRun multiple cases (comma-separated, no spaces):\n\n```bash\npython generation_harness/run_imagegen_evals.py --cases ui_checkout_mockup,coffee_flyer_generation\n```\n\nOptional OCR-style text check for the coffee flyer:\n\n```bash\npython generation_harness/run_imagegen_evals.py --run-ocr\n```\n\n## Flags (when to use them)\n\n- `--cases`: limit runs to specific case ids. Valid values:\n  - `ui_checkout_mockup`\n  - `coffee_flyer_generation`\n- `--model`: image model under test (defaults to `gpt-image-1.5`).\n- `--judge-model`: LLM used to grade outputs (defaults to `gpt-5.2`).\n- `--num-images`: number of images to generate per case (defaults to 1).\n- `--image-size`: size for generated images (defaults to `1024x1024`).\n- `--run-ocr`: only relevant for `coffee_flyer_generation` (checks exact text).\n\n## Outputs\n\nResults are written under `generation_harness/results/<run_id>/`:\n\n- `results.json` and `results.csv` with per-example scores\n- `summary.json` with aggregated metrics\n- `artifacts/` for generated images\n\n## Adapting the harness\n\n- Update the prompts/criteria in `run_imagegen_evals.py`.\n- Tune JSON schemas + parsers to match your evaluation rubric.\n- Add more cases by creating new `TestCase` entries and graders.\n","examples/evals/realtime_evals/shared/tools.json":"# Shared Skill\n\nPath: `examples/evals/realtime_evals/shared/tools.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/shared/tools.json)","examples/evals/realtime_evals/walk_harness/README.md":"# Walk_harness Skill\n\nPath: `examples/evals/realtime_evals/walk_harness/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/walk_harness/README.md)","examples/evals/realtime_evals/run_harness/data/sim_cs_address_update.json":"# Data Skill\n\nPath: `examples/evals/realtime_evals/run_harness/data/sim_cs_address_update.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/run_harness/data/sim_cs_address_update.json)","examples/evals/realtime_evals/run_harness/data/sim_cs_order_status.json":"# Data Skill\n\nPath: `examples/evals/realtime_evals/run_harness/data/sim_cs_order_status.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/run_harness/data/sim_cs_order_status.json)","examples/evals/realtime_evals/run_harness/data/sim_cs_refund.json":"# Data Skill\n\nPath: `examples/evals/realtime_evals/run_harness/data/sim_cs_refund.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/run_harness/data/sim_cs_refund.json)","examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/definitions.json":"# Tools Skill\n\nPath: `examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/definitions.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/definitions.json)","examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/restaurant_facts.json":"# Tools Skill\n\nPath: `examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/restaurant_facts.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/restaurant_facts.json)","examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/definitions.json":"# Tools Skill\n\nPath: `examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/definitions.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/definitions.json)","examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/restaurant_facts.json":"# Tools Skill\n\nPath: `examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/restaurant_facts.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/restaurant_facts.json)","examples/evals/realtime_evals/skills/bootstrap-realtime-eval/agents/openai.yaml":"# Agents Skill\n\nPath: `examples/evals/realtime_evals/skills/bootstrap-realtime-eval/agents/openai.yaml`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/skills/bootstrap-realtime-eval/agents/openai.yaml)","examples/evals/realtime_evals/skills/bootstrap-realtime-eval/references/harness-selection.md":"# References Skill\n\nPath: `examples/evals/realtime_evals/skills/bootstrap-realtime-eval/references/harness-selection.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/skills/bootstrap-realtime-eval/references/harness-selection.md)","examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/.agents/plugins/marketplace.json":"# Claude Plugin Marketplace Catalog\n\nPath: `examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/.agents/plugins/marketplace.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/.agents/plugins/marketplace.json)","examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.mcp.json":"# Bls Skill\n\nPath: `examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.mcp.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.mcp.json)","examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/README.md":"# bls Documentation\n\nPath: `examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/README.md)","examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.codex-plugin/plugin.json":"# codex-plugin Plugin Manifest\n\nPath: `examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.codex-plugin/plugin.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.codex-plugin/plugin.json)"},"files":{"examples/evals/realtime_evals/README.md":"# Realtime Evals\n\nThis folder contains three evaluation harnesses for the Realtime API that increase in complexity:\n**crawl → walk → run**.\n\n- **crawl**: synthetic single-turn (TTS → stream → grade)\n- **walk**: deterministic replay of saved phone-audio (more realistic, still comparable)\n- **run**: model-simulated multi-turn conversations (finds multi-turn/tooling failures)\n\nDepending on your realtime eval maturity, point Codex (or your preferred coding assistant) to the folder that fits your use case so it can adapt the harness and bootstrap sample data; you can also have it run the included tests to confirm everything works.\n\n## Quickstart\n\nPython 3.12+ required.\n\n```bash\nmake install\nsource .venv/bin/activate\nexport OPENAI_API_KEY=\"your_api_key\"\n```\n\n`make install` creates the local `.venv` and installs both runtime and dev dependencies. It uses `uv` when available and otherwise falls back to `python -m venv` plus `pip install -r requirements.txt -r requirements-dev.txt`.\n\nRun a first command per harness. If uv is not installed, replace `uv run` with `python` and run these scripts with your `.venv` activated:\n\n- Crawl: `uv run python crawl_harness/run_realtime_evals.py`\n- Walk: install ffmpeg (`brew install ffmpeg`), then:\n  - `uv run python walk_harness/generate_audio.py`\n  - `uv run python walk_harness/run_realtime_evals.py`\n- Run: `uv run python run_harness/run_realtime_evals.py --max-examples 1`\n\n## Dev commands\nUse the root `Makefile` for common checks. Run `make install` first to create `.venv`. These targets work with or without `uv`: when `uv` is installed they run through `uv run`, and otherwise they use the matching tool binaries from the local `.venv`.\n\n- `make install`\n- `make streamlit`\n- `make format`\n- `make lint`\n- `make lint-fix`\n- `make typecheck`\n- `make test`\n\n## [Crawl (synthetic single-turn)](./crawl_harness)\n\n**Best for:** Fast iteration and controlled comparisons.\n\n- Uses text prompts from a CSV and synthesizes TTS audio per row.\n- Streams fixed-size audio chunks into a realtime session.\n- Captures the first assistant turn and grades tool-call correctness.\n- Writes results, summary metrics, audio, and event logs per datapoint.\n\nRun:\n\n```\nuv run python crawl_harness/run_realtime_evals.py\n```\n\n## [Walk (saved audio replay)](./walk_harness)\n\n**Best for:** Realism with reproducibility.\n\n- Replays saved, phone-compressed audio (G.711 mu-law WAV at 8 kHz).\n- Keeps chunking and turn boundaries fixed (VAD off).\n- Captures responses, tool calls, latencies, and event logs per datapoint.\n- Output audio is saved as PCM16 WAV for easy listening.\n\nRun:\n\n```\nuv run python walk_harness/generate_audio.py\nuv run python walk_harness/run_realtime_evals.py\n```\n\n## [Run (multi-turn simulation)](./run_harness)\n\n**Best for:** Discovering multi-turn failures and tool behavior.\n\n- Uses a simulator model to generate user turns and an assistant model under test.\n- Streams simulator audio into the assistant with fixed chunking.\n- Mocks tool outputs deterministically and grades with LLM-as-judge.\n- Writes results, summaries, transcripts, and event logs per simulation.\n\nRun:\n\n```\nuv run python run_harness/run_realtime_evals.py\n```\n\n## Results layout\n\nEach harness writes into its own `results/` folder:\n\n- `results/<run_id>/results.csv`: per-example outputs and grades\n- `results/<run_id>/summary.json`: aggregate metrics\n- `results/<run_id>/plots/*.png`: warm-editorial summary charts for scores, latency, tokens, and run shape\n- `results/<run_id>/events/*.jsonl`: full realtime event stream per datapoint\n\nThe shared typed representation of these artifacts for the crawl, walk, and run harnesses lives in `shared/result_types.py`.\n\nTo render charts for an existing run after the fact:\n\n```bash\nuv run python plot_eval_results.py --run-dir run_harness/results/<run_id>\n```\n\n## Results Viewer\n\nUse the Streamlit results viewer to browse saved runs from `crawl_harness`, `walk_harness`, and `run_harness` without opening the raw artifacts by hand.\n\n- `Comparison View`: select a harness, choose one or more saved runs, and compare summary metrics, scores, latency, and token usage across runs.\n- `Run Viewer`: inspect one saved run in detail. Crawl and walk runs show row-level audio artifacts and event logs; run-harness runs use a Simulation Viewer with transcripts, event logs, and turn audio.\n\nRun it from this directory with either:\n\n```bash\nmake streamlit\n```\n\nor:\n\n```bash\ncd results_viewer\nuv run streamlit run app.py\n```\n\nThen open the local Streamlit URL, usually `http://localhost:8501`.\n\n## Common CLI flags\n\nAll harnesses share a core set of flags so you can switch between them easily:\n\n- `--data-csv`, `--results-dir`, `--run-name`\n- `--model` (assistant under test), `--system-prompt-file`, `--tools-file`\n- `--chunk-ms`, `--sample-rate-hz`, `--real-time`\n- `--max-examples` (quick smoke tests)\n- `--skip-plots` (skip post-run PNG chart generation)\n\nRun harness adds multi-turn and simulator-specific flags (see its README).\n",".codex/skills/docs-editor/SKILL.md":"---\nname: docs-editor\ndescription: Review and edit OpenAI Cookbook notebooks, Markdown, and MDX for technical accuracy, clarity, grammar, consistency, runnable examples, and repository publication requirements. Use for editorial reviews, pre-merge documentation checks, or notebook Markdown-cell sweeps in openai-cookbook.\n---\n\n# OpenAI Cookbook docs editor\n\n## Objective\n\nReview Cookbook content conservatively. Improve correctness and readability without changing the author's intent, inventing facts, or creating unrelated churn.\n\n## Workflow\n\n### 1. Define the scope\n\n- Prefer explicit paths or the files changed in the pull request.\n- For pull requests, review changed `.md` and `.mdx` files plus Markdown cells in changed `.ipynb` files.\n- Review `registry.yaml` and `authors.yaml` when content is added, moved, removed, or attributed to a new author.\n- Stay within the requested content. Do not turn a focused review into a repository-wide rewrite.\n\n### 2. Protect notebook integrity\n\n- When the request is editorial, edit only notebook Markdown cells.\n- Preserve code cells, outputs, execution counts, attachments, cell order, cell IDs, and notebook and cell metadata.\n- Avoid reserializing an entire notebook for a small prose change. Inspect the diff for unintended JSON churn.\n- Do not execute notebook code unless the user requests it or code changes are in scope. Markdown-only edits require structural validation, not calls to external services.\n- Never add hard-coded secrets. Document required environment variables such as `OPENAI_API_KEY`.\n\n### 3. Review the content\n\nLoad `resources/style-guide.md` and check:\n\n- Technical claims, API names, model names, identifiers, commands, paths, and URLs.\n- Agreement between prose, code, sample inputs, and expected outputs.\n- Prerequisites, dependencies, environment variables, and steps needed to run an example.\n- File and notebook names, headings, lists, numbering, cross-references, and links.\n- Grammar, spelling, punctuation, terminology, capitalization, and concise wording.\n- Publication metadata in `registry.yaml` and author metadata in `authors.yaml` when applicable.\n\nVerify uncertain technical claims from an authoritative source or flag them for review. Do not guess.\n\n### 4. Prioritize findings\n\nUse the repository's review bar:\n\n- **P0**: Broken notebook JSON; hard-coded secrets; materially false or unsafe guidance; or any repository-defined P0 issue, including naming violations, typos, broken links, inconsistent formatting, placeholder identifiers, undocumented environment variables, or out-of-sync publication metadata.\n- **P1**: Misleading instructions, prose that contradicts code or output, missing prerequisites, or a required link that does not work.\n- **P2**: Clear improvements to grammar, consistency, organization, or readability that do not affect correctness.\n- **P3**: Optional polish with little effect on comprehension.\n\n### 5. Make safe edits\n\n- Fix obvious typos, grammar, punctuation, broken local references, and clear prose/code mismatches.\n- Keep changes minimal, local, and reversible.\n- Preserve exact literals, code identifiers, commands, values, counts, conditions, modality, negation, and uncertainty markers.\n- Tighten prose only when meaning stays unchanged. Prefer concrete nouns, strong verbs, short introductions, and direct instructions.\n- Flag any change that depends on product knowledge or could alter meaning instead of guessing.\n- Do not make file-wide formatting or stylistic changes solely for uniformity.\n\n### 6. Validate the result\n\nFor any changed notebook, run the repository validator:\n\n```bash\nenv -u VIRTUAL_ENV uv run --with nbformat python .github/scripts/check_notebooks.py\n```\n\nThen:\n\n- Run `git diff --check`.\n- Inspect the diff and confirm that notebook changes are limited to intended Markdown cells.\n- Run an available YAML linter if `registry.yaml` or `authors.yaml` changed.\n- Run the relevant example or test only when executable content changed and its dependencies are available.\n\n### 7. Report the review\n\nProvide:\n\n- A short summary of the scope and edits.\n- Auto-fixed items.\n- Remaining findings grouped by priority with file and cell or line references.\n- The validation commands run and their results.\n\nIf no issues remain, say so explicitly.\n","examples/evals/imagegen_evals/README.md":"# Image Generation + Editing Evals\n\nThis folder contains a lightweight vision eval harness plus example runners for\nimage generation and image editing. The code mirrors the structure of\n`examples/evals/realtime_evals/` so you can adapt it quickly.\n\n**Directory layout**\n\n- `vision_harness/`: minimal shared library (types, runners, graders, evaluate loop)\n- `generation_harness/`: text-to-image evals (UI mockups + marketing flyer)\n- `editing_harness/`: image-edit evals (virtual try-on + logo edit)\n- `shared/`: reporting and optional rendering helpers\n\n## Quickstart\n\nPython 3.9+ required.\n\n```bash\npip install -r requirements.txt\nexport OPENAI_API_KEY=\"your_api_key\"\n```\n\nRun a harness:\n\n- Generation: `python generation_harness/run_imagegen_evals.py`\n- Editing: `python editing_harness/run_imagegen_evals.py`\n\n## What the harness does\n\n1. Builds a small set of `TestCase` objects (prompt + criteria).\n2. Runs the image model for each case.\n3. Grades each output with an LLM judge using a strict JSON schema.\n4. Writes results and artifacts to `results/<run_id>/`.\n\nThe harness is intentionally small so you can copy/paste parts into your own\nproduction eval setup.\n\n## Example cases\n\nGeneration cases:\n- `ui_checkout_mockup`: mobile checkout screen with strict text + layout rules\n- `coffee_flyer_generation`: marketing flyer with exact copy constraints\n\nEditing cases:\n- `vto_jacket_tryon`: virtual try-on with reference person + garment\n- `logo_year_edit`: precision logo text edit\n\n## Required assets (editing harness)\n\nThe editing harness expects these files in `images/`:\n\n- `images/base_woman.png`\n- `images/jacket.png`\n- `images/logo_generation_1.png`\n\n## Results layout\n\nEach harness writes into its own `results/` folder:\n\n- `results/<run_id>/results.json`: per-example outputs and grades\n- `results/<run_id>/results.csv`: tabular results\n- `results/<run_id>/summary.json`: aggregated metrics\n- `results/<run_id>/artifacts/*.png`: generated or edited images\n\nTip: keep `results/` out of git (delete after runs or add to `.gitignore`).\n\n## Common CLI flags\n\nBoth harnesses accept a shared core set of flags:\n\n- `--results-dir`, `--run-name`\n- `--model` (image model under test), `--judge-model` (LLM judge)\n- `--num-images` (images per case)\n- `--cases` (comma-separated list of case ids)\n\nGeneration harness adds `--image-size` and `--run-ocr` (text extraction check).\n\n## Suggested workflows\n\n- Quick smoke test:\n  - `python generation_harness/run_imagegen_evals.py --cases ui_checkout_mockup`\n  - `python editing_harness/run_imagegen_evals.py --cases vto_jacket_tryon`\n- Compare model variants: run once per model name and compare `summary.json`.\n- Use `--run-ocr` on the flyer case to validate exact text rendering.\n\n## Adapting for your use case\n\n- Replace prompts/criteria in the harness scripts.\n- Update JSON schemas to match your rubric (and parse them into `Score` values).\n- Add new cases by creating new `TestCase` objects and graders.\n- Swap the output store to change where artifacts are written.\n\n## Notes\n\n- This harness uses the OpenAI Images API and a judge model. Costs scale with\n  the number of images and judge calls.\n- Judge outputs are best treated as signals, not ground truth. Keep a small\n  human review loop for high-stakes decisions.\n","examples/agents_api/apps/document_review/skills/expense-review-policy/SKILL.md":"---\nname: expense-review-policy\ndescription: Review invoices and contracts against accounts-payable policy before human approval.\n---\n\n# Expense review policy\n\nPolicy ID: `AP-104`.\n\nApply this policy to invoices, expense receipts, and service contracts.\n\n## Invoices and expenses\n\n- Read the entire invoice, including the vendor and every line item.\n- Extract every line item's quantity and unit price before calculating. Check the extracted list against every source line; an empty list is an error, not a zero subtotal.\n- Use Python's `decimal.Decimal` to multiply quantities by unit prices and sum the line totals. Compare this subtotal with the printed subtotal, then add shipping to calculate the total.\n- Strip currency separators, compare the result with `Total amount due`, and report the exact difference (stated total minus calculated total).\n- Require a purchase order or another documented approval.\n- Flag missing receipts, unsupported charges, and changed payment instructions.\n- Escalate incorrect totals and unverified changes to bank details.\n\n## Contracts\n\n- Read every clause and report all matching risks, even when one already requires escalation.\n- Flag automatic renewals and restrictive cancellation windows.\n- Flag unilateral price increases and missing liability limits.\n- Escalate customer-data sharing or subcontractor access without approval.\n- Identify missing confidentiality, security, or termination terms.\n\n## Review decision\n\n- `needs_info`: Required documentation or supporting information is missing.\n- `escalated`: Fraud indicators, financial discrepancies, or risky terms need review.\n- `ready_for_approval`: No policy violations remain, but a human must still approve.\n\nWrite `/workspace/output/<document-stem>.json` with the following fields:\n\n```json\n{\n  \"document\": \"invoice.txt\",\n  \"document_type\": \"invoice\",\n  \"policy_id\": \"AP-104\",\n  \"decision\": \"escalated\",\n  \"vendor\": \"Cedar Office Supply\",\n  \"amount\": 6420,\n  \"issues\": [\"The claimed total exceeds the calculated total.\"],\n  \"recommendation\": \"Escalate for human review.\",\n  \"calculation\": {\n    \"line_items\": [{\"quantity\": 2, \"unit_price\": 100}],\n    \"shipping\": 20,\n    \"calculated_total\": 220,\n    \"difference\": 6200\n  }\n}\n```\n\nEvery `issues` entry must be a plain-English string, not a nested object.\nUse the actual document values, not the illustrative numbers above. Include every\ninvoice line item in `calculation`; `amount` is the stated total. For contracts,\nset `document_type` to `contract` and `calculation` to `null`.\n\nOnly the coordinator writes `summary.json`. A specialist writes its assigned report.\n\nNever approve payments, sign contracts, or take external actions.\n","AGENTS.md":"# Repository Guidelines\n\n## Project Structure & Module Organization\n\nThe cookbook is organized around runnable examples and reference articles for OpenAI APIs. Place notebooks and Python scripts under `examples/<topic>/`, grouping related assets inside topic subfolders (for example, `examples/agents_sdk/`). Narrative guides and long-form docs live in `articles/`, and shared diagrams or screenshots belong in `images/`. Update `registry.yaml` whenever you add content so it appears on cookbook.openai.com, and add new author metadata in `authors.yaml` if you want custom attribution. Keep large datasets outside the repo; instead, document how to fetch them in the notebook.\n\n## Build, Test, and Development Commands\n\nUse a virtual environment to isolate dependencies:\n\n- `python -m venv .venv && source .venv/bin/activate`\n- `pip install -r examples/<topic>/requirements.txt` (each sample lists only what it needs)\n- `jupyter lab` or `jupyter notebook` to develop interactively\n- `python .github/scripts/check_notebooks.py` to validate notebook structure before pushing\n\n## Coding Style & Naming Conventions\n\nWrite Python to PEP 8 with four-space indentation, descriptive variable names, and concise docstrings that explain API usage choices. Name new notebooks with lowercase, dash-or-underscore-separated phrases that match their directory—for example `examples/gpt-5/prompt-optimization-cookbook.ipynb`. Keep markdown cells focused and prefer numbered steps for multi-part workflows. Store secrets in environment variables such as `OPENAI_API_KEY`; never hard-code keys inside notebooks.\n\n## Testing Guidelines\n\nExecute notebooks top-to-bottom after installing dependencies and clear lingering execution counts before committing. For Python modules or utilities, include self-check cells or lightweight `pytest` snippets and show how to run them (for example, `pytest examples/object_oriented_agentic_approach/tests`). When contributions depend on external services, mock responses or gate the cells behind clearly labeled opt-in flags.\n\n## Commit & Pull Request Guidelines\n\nUse concise, imperative commit messages that describe the change scope (e.g., \"Add agent portfolio collaboration demo\"). Every PR should provide a summary, motivation, and self-review, and must tick the registry and authors checklist from `.github/pull_request_template.md`. Link issues when applicable and attach screenshots or output snippets for UI-heavy content. Confirm CI notebook validation passes locally before requesting review.\n\nBefore a pull request is merged, run the repository's `docs-editor` skill on every changed article, Markdown or MDX file, and notebook Markdown cell. For editorial notebook changes, preserve code cells, outputs, execution counts, attachments, cell order, and metadata. Fix safe issues, rerun the relevant checks, and resolve or explicitly document all remaining P0 and P1 findings in the pull request.\n\n## Metadata & Publication Workflow\n\nNew or relocated content must have an entry in `registry.yaml` with an accurate path, date, and tag set so the static site generator includes it. When collaborating, coordinate author slugs in `authors.yaml` to avoid duplicates, and run `python -m yaml lint registry.yaml` (or your preferred YAML linter) to catch syntax errors before submitting.\n\n## Review Guidelines\n\nThese are considered priority 0 issues for this repo, in addition to the normal priority for possible issues.\n- Verify file, function, and notebook names follow the repo's naming conventions and clearly describe their purpose.\n- Scan prose and markdown for typos, broken links, and inconsistent formatting before approving.\n- Check that code identifiers remain descriptive (no leftover placeholder names) and that repeated values are factored into constants when practical.\n- Ensure notebooks or scripts document any required environment variables instead of hard-coding secrets or keys.\n- Confirm metadata files (`registry.yaml`, `authors.yaml`) stay in sync with new or relocated content.\n\n## Recent Learnings\n\n- **`uv run` can inherit the wrong virtualenv in this repo** -> Clear `VIRTUAL_ENV` (for example `env -u VIRTUAL_ENV uv run ...`) when an unrelated environment is active -> Avoids misleading mismatch warnings and makes it clear the repo's `.venv` is the interpreter actually running the harnesses.\n- **Realtime eval shared imports can resolve the wrong module under pytest** -> Add `shared/__init__.py` and ensure tests prepend `examples/evals/realtime_evals` to `sys.path` before importing `shared.*` -> Prevents collection failures caused by unrelated installed packages named `shared`.\n- **Run-level grades can be overweighted by long simulations** -> Store turn-level grades on the matching turn and trace-level grades on one row per simulation instead of copying them onto every row -> Keeps `results.csv` row semantics intact and prevents summary means from favoring longer conversations.\n- **Synthetic-audio scaffold requests can pick the wrong harness** -> Default unspecified synthetic-audio evals to `crawl` text-to-TTS and reserve `walk` for replay-specific audio traits like noise, telephony artifacts, or speaker characteristics -> Keeps new realtime evals on the simplest harness unless audio realism is itself under test.\n- **Task-specific single-turn grading can outgrow the shared crawl schema** -> Keep the shared crawl harness for realtime execution, then add eval-local wrapper scripts that post-grade domain-specific quality and overwrite `results.csv` while preserving `results_base.csv` -> Avoids forking the harness when a use case needs richer grading than tool-call correctness.\n- **Synthetic learner audio can sound like eval scaffolding** -> Write `user_text` as a realistic in-app learner request and keep evaluation rules in metadata plus the system prompt -> Produces audio inputs that match the product surface instead of teaching the model the grading rubric through the spoken prompt.\n","examples/evals/realtime_evals/skills/bootstrap-realtime-eval/SKILL.md":"---\nname: bootstrap-realtime-eval\ndescription: Bootstrap a new realtime eval folder inside this cookbook repo by choosing the right harness from examples/evals/realtime_evals, scaffolding prompt/tools/data files, generating a useful README, and validating it with smoke, full eval, and test runs. Use when a user wants to start a new crawl, walk, or run realtime eval in this repository.\n---\n\n# Bootstrap Realtime Eval\n\nUse this skill when the user wants a new realtime eval scaffold under `examples/evals/realtime_evals/`.\n\nThis skill is repo-specific. Do not copy harness code into the generated folder. The generated eval should point at the shared harnesses already in:\n\n- `examples/evals/realtime_evals/crawl_harness`\n- `examples/evals/realtime_evals/walk_harness`\n- `examples/evals/realtime_evals/run_harness`\n\n## Inputs To Collect\n\nAlways ask the user for the minimum set needed to choose and scaffold the eval before you create files, run the scaffold script, or author starter data. Do not skip this just because you can infer a default.\n\nAsk for:\n\n- Eval name\n- Goal or scenario\n- Harness choice, or enough context to recommend one\n- System prompt path or inline text\n- Tools JSON path or tool descriptions\n- Data path or source materials\n- Desired graders\n\nIf the user does not know which harness they want, explain the options briefly and recommend one. See `references/harness-selection.md`.\n\nWhen the user asks for synthetic audio but does not specify a harness, default to `crawl` text-to-TTS unless they need the generated audio to carry particular noise, telephony artifacts, speaker characteristics, or other replay-specific properties. Use `walk` for those cases.\n\nKeep the questions concise and grouped into one short batch whenever possible.\n\nIf the user only provides `user_text` or a short task description, still ask the questions above first. If they answer only partially, then infer the remaining low-risk details, call out the assumptions, and make the scaffold easy to revise later.\n\n## Workflow\n\n1. Ask the user for the required inputs first.\n   - Do this before making files or selecting a final harness.\n   - If the user already supplied some of the inputs, ask only for the missing ones.\n   - If you recommend a harness, wait for the user response before scaffolding.\n\n2. Pick the harness.\n   - `crawl`: single-turn text-to-TTS.\n   - `walk`: replay saved audio or generate audio from text rows.\n   - `run`: multi-turn simulation with tool mocks and judge criteria.\n   - If the user wants synthetic audio but does not care about replay-specific audio characteristics, prefer `crawl` over `walk`.\n\n3. Normalize the inputs.\n   - If the user gives inline prompt or tool content, write it into the generated folder.\n   - If the user gives CSV data, inspect it with pandas before wiring it in.\n   - If the data is not yet harness-ready, scaffold the files and then have you author the starter dataset directly in those files.\n   - If the user only gives `user_text`, infer `example_id` values and leave optional grading fields blank unless you have enough signal to fill them.\n   - Ground the starter data in the use case, prompts, tools, and any other material the user provided.\n   - Make the starter data realistic. Put yourself in the shoes of the end user in that use case and craft datapoints that a real user would plausibly say or do.\n\n4. Be proactive when data is missing.\n   - `crawl`: you should author 3 starter rows covering one happy path and a couple of nearby variants or edge cases.\n   - `walk`: you should author 3 source CSV rows and prepare the audio-generation step so the user can create audio immediately.\n   - `run`: you should author 2 starter simulations, not 1.\n   - Do not rely on deterministic script-generated samples for use-case-specific starter data.\n   - Show the generated starter samples to the user, ask for a quick greenlight or correction, then expand only after feedback when the task calls for more coverage.\n\n5. Run the scaffold script:\n\n```bash\npython examples/evals/realtime_evals/skills/bootstrap-realtime-eval/scripts/bootstrap_realtime_eval.py --name \"<eval_name>\" --harness \"<crawl|walk|run>\"\n```\n\nAdd flags for prompt, tools, data, graders, and run-specific fields as needed. Read the script help if you need the exact flag names.\n\n6. Review the generated folder.\n   - Confirm the README is accurate for the selected harness.\n   - Confirm the system prompt, tools, and data files point at the generated folder.\n   - If the user provided real data, make sure it is wired in instead of leaving starter placeholders.\n   - If you authored inferred starter data, show the user the generated rows or simulations before scaling up.\n\n7. Enrich the scaffold.\n   - For `crawl` and `walk`, make the CSV realistic and ensure the expected tool columns are present.\n   - For `walk`, if the dataset lacks `audio_path`, use the shared `walk_harness/generate_audio.py` flow described in the README.\n   - For `run`, make sure `simulations.csv` and the starter `sim_*.json` file reflect the user’s scenario, tool mocks, and graders.\n\n8. Validate before returning.\n   - Run a smoke eval for the generated folder.\n   - If the smoke eval succeeds, automatically run the full eval for the generated folder in the same turn.\n   - If the smoke eval fails, stop and fix or report the blocker before attempting the full eval.\n   - Run `pytest examples/evals/realtime_evals/tests -q`.\n   - Inspect the generated README and at least one generated data file.\n\n## Hard Rules\n\n- Keep the generated folder inside `examples/evals/realtime_evals/`.\n- Ask the user for the missing setup information before scaffolding. Do not jump straight to building a default eval when the request is to create a new eval.\n- Do not duplicate the main harness scripts into the generated folder.\n- Do not write generic placeholder starter data when the use case provides enough context to do better.\n- Use the prompt, tools, and scenario details to make the initial datapoints feel like a real user interaction.\n- The README must include:\n  - why this harness was chosen\n  - which files to edit first\n  - smoke and full run commands\n  - data contract notes\n  - troubleshooting notes\n- Prefer assistant-specific flags for run harness commands:\n  - `--assistant-system-prompt-file`\n  - `--assistant-tools-file`\n\n## Completion Criteria\n\nTreat the task as complete only when:\n\n- the folder exists under `examples/evals/realtime_evals/<name>_realtime_eval/`\n- `README.md`, `system_prompt.txt`, and `tools.json` exist\n- harness-specific starter data exists, authored by you when the user did not provide it\n- the smoke command has been run or is blocked for a clear reason\n- if the smoke command succeeded, the full eval command has also been run or is blocked for a clear reason\n- `pytest examples/evals/realtime_evals/tests -q` has been run\n\n## Learnings\n\nWhen this skill uncovers a reusable workflow or harness constraint that should guide future bootstrap work, add a short note here.\n\nKeep learnings concise and action-oriented:\n\n- **Problem -> Fix -> Why**\n\nOnly add items that are likely to help future realtime-eval scaffolding in this repo. Remove stale items when they no longer apply.\n\n- **`gpt-realtime` temperature is unsupported** -> Do not add a `temperature` field or CLI flag when scaffolding `gpt-realtime` evals -> Avoids invalid config and keeps runs aligned with the realtime harness constraints.\n- **Starter samples should come from the model, not the scaffold script** -> Use the script to create the folder structure and template files, then have you author the initial rows or simulations from the user’s use case -> Produces more relevant starter data and makes iteration easier with the user.\n- **Synthetic audio can be overfit to the wrong harness** -> Default unspecified synthetic-audio requests to `crawl` text-to-TTS and reserve `walk` for replay-specific audio characteristics like noise or telephony artifacts -> Keeps the bootstrap path simpler unless audio realism is the actual target.\n- **Smoke-only validation can leave scaffolds half-proven** -> After a successful smoke run, automatically run the full eval before declaring the bootstrap complete -> Catches dataset-wide or late-row failures that a one-example smoke test misses.\n",".codex/skills/docs-editor/resources/style-guide.md":"# OpenAI Cookbook editorial checklist\n\nUse this checklist for articles and notebook Markdown cells in `openai-cookbook`. Apply only the rules relevant to the content under review.\n\n## Audience and purpose\n\n- Write for developers who want to understand and run an OpenAI API example.\n- State the task or outcome early. Keep background material only when it helps the reader use the example correctly.\n- Assume general programming knowledge, but explain Cookbook-specific setup, choices, and tradeoffs.\n- Avoid promotional language, vague praise, and claims that the example does not demonstrate.\n\n## Technical accuracy\n\n- Match API, model, method, parameter, field, file, and environment-variable names exactly.\n- Keep prose consistent with the code, sample inputs, outputs, diagrams, and nearby cells.\n- Check counts, ordered steps, cross-references, defaults, constraints, and stated results.\n- Do not infer a correction when the intended fact is unclear. Preserve the source and flag it for review.\n- Use authoritative sources to verify uncertain product behavior.\n- Do not imply that an optional step is required or that a required step is optional.\n\n## Runnable examples\n\n- List prerequisites and dependencies that readers need before running the example.\n- Document required environment variables. Never hard-code API keys, tokens, credentials, or other secrets.\n- Keep setup instructions in the same order that readers perform them.\n- Introduce code with the goal or reason for the step; do not narrate every line.\n- Use realistic inputs and clearly label placeholders.\n- Show expected output only when it helps readers verify success, and keep it consistent with the code.\n- Keep external-service calls and other opt-in behavior clearly labeled.\n\n## Notebook integrity\n\n- Edit Markdown cells only during an editorial pass.\n- Preserve code cells, outputs, execution counts, attachments, cell IDs, cell order, and metadata.\n- Keep notebook JSON valid and avoid whole-file reserialization for a small prose edit.\n- Keep each Markdown cell focused. Split or merge cells only when the structure is genuinely confusing and the request permits it.\n- Use Markdown headings in a logical hierarchy without skipping levels unnecessarily.\n\n## Prose\n\n- Lead with the point. Prefer concrete nouns and strong verbs.\n- Use direct instructions and address the reader as “you” when helpful.\n- Prefer active voice, but keep passive voice when the actor is unknown or unimportant.\n- Remove throat-clearing, repeated conclusions, filler, and ornamental transitions.\n- Keep paragraphs short enough to scan and use lists for genuinely parallel items.\n- Use numbered lists for ordered procedures and bullets for unordered groups.\n- Keep sibling headings and list items parallel.\n- Use sentence-style capitalization for headings and do not end headings with periods.\n- Preserve useful warmth and explanation; do not flatten the content into terse fragments.\n\n## Terminology and formatting\n\n- Use exact product terminology and consistent capitalization.\n- Format commands, options, environment variables, paths, filenames, code identifiers, and literal values with backticks.\n- Match the capitalization and spelling used by the language or API.\n- Expand uncommon abbreviations on first use unless the surrounding audience clearly knows them.\n- Use inclusive, precise language and neutral role terms.\n- Do not replace a technically precise term merely to vary wording.\n\n## Links and assets\n\n- Prefer descriptive link text over “here” or a pasted URL.\n- Verify local links, anchors, image paths, and referenced notebook or article paths.\n- Use stable, authoritative external links when available.\n- Give images meaningful alt text unless they are purely decorative.\n- Keep diagrams, screenshots, captions, and prose consistent.\n\n## Publication metadata\n\n- Add or update `registry.yaml` for new, moved, renamed, or removed published content.\n- Confirm registry paths, dates, tags, and titles match the content.\n- Add or update `authors.yaml` when introducing or changing author attribution.\n- Check that filenames follow the repository's naming conventions and clearly describe the content.\n\n## Final review\n\n- Read each edited passage in context, not only as an isolated diff hunk.\n- Confirm no factual meaning, literal value, condition, negation, or uncertainty changed unintentionally.\n- Confirm all repository-defined P0 issues are fixed.\n- Run the notebook structural validator for changed notebooks and inspect the final diff for unintended changes.\n","examples/evals/imagegen_evals/editing_harness/README.md":"# Editing Harness\n\n**Best for:** high-precision checks where edits must be exact and non-target\ncontent should remain unchanged.\n\nThis harness runs two example cases:\n\n- `vto_jacket_tryon`: virtual try-on using reference person + garment\n- `logo_year_edit`: precision logo text edit\n\n## Run\n\n```bash\npython editing_harness/run_imagegen_evals.py\n```\n\nRun a single case (use `--cases`):\n\n```bash\npython editing_harness/run_imagegen_evals.py --cases vto_jacket_tryon\n```\n\nRun multiple cases (comma-separated, no spaces):\n\n```bash\npython editing_harness/run_imagegen_evals.py --cases vto_jacket_tryon,logo_year_edit\n```\n\n## Flags (when to use them)\n\n- `--cases`: limit runs to specific case ids. Valid values:\n  - `vto_jacket_tryon`\n  - `logo_year_edit`\n- `--model`: image model under test (defaults to `gpt-image-1.5`).\n- `--judge-model`: LLM used to grade outputs (defaults to `gpt-5.2`).\n- `--num-images`: number of edited images to generate per case (defaults to 1).\n\n## Inputs\n\nThis harness expects the following reference images in `images/`:\n\n- `images/base_woman.png`\n- `images/jacket.png`\n- `images/logo_generation_1.png`\n\n## Outputs\n\nResults are written under `editing_harness/results/<run_id>/`:\n\n- `results.json` and `results.csv` with per-example scores\n- `summary.json` with aggregated metrics\n- `artifacts/` for edited images\n\n## Adapting the harness\n\n- Replace the input images and edit prompts in `run_imagegen_evals.py`.\n- Update the schemas and verdict rules to fit your workflow.\n","examples/evals/realtime_evals/run_harness/README.md":"# Run Harness\n\nModel-simulated multi-turn eval harness for the Realtime API. This harness uses\none realtime model as the user simulator (audio-only) and another realtime model\nas the assistant under test. It replays fixed chunking (VAD off), mocks tools\nfrom the simulation JSON, and records full traces.\n\n## What it does\n\n- Loads `run_harness/data/simulations.csv` with pandas.\n- Reads per-simulation JSON files defining scenario, simulator identity, tool mocks,\n  and LLM-as-judge grading criteria.\n- Generates user audio turns via a realtime simulator model (audio-only).\n- Streams user audio to the assistant in fixed-size chunks and commits manually.\n- Captures assistant audio/text, tool calls, tool outputs, and latencies.\n- Grades turn-level and trace-level criteria with an LLM-as-judge.\n- Writes `results.csv`, `summary.json`, and full trace logs under `run_harness/results/`.\n- Renders styled PNG plots under `run_harness/results/<run_id>/plots/` by default.\n\n## Files\n\n- `run_harness/run_realtime_evals.py`: Run harness script.\n- `run_harness/data/simulations.csv`: Index of simulation files.\n- `run_harness/data/sim_*.json`: Simulation definitions (this repo currently ships 3 examples).\n- `run_harness/results/<run_id>/events/*.jsonl`: Full event trace per simulation.\n- `run_harness/results/<run_id>/conversations/*.txt`: Human-readable transcript with tool calls.\n\n## Simulation definitions (high level)\n\nEach `sim_*.json` file defines:\n\n- **the scenario** (what the user is trying to do)\n- **tool mocks** (what each tool should return when called)\n- **judge rubric** (what should be graded per turn / overall)\n\nThis lets you run repeatable multi-turn evals without needing live backend integrations.\n\n## How to run\n\nFrom repo root:\n\n```bash\npython run_harness/run_realtime_evals.py --max-examples 1\n```\n\nCommon options:\n\n- `--data-csv`: Simulation index CSV.\n- `--model`: Alias for the assistant model under test.\n- `--assistant-model`: Realtime model under test (overrides `--model`).\n- `--simulator-model`: Realtime model used as the user.\n- `--system-prompt-file`: Alias for assistant system prompt file.\n- `--tools-file`: Alias for assistant tools file.\n- `--assistant-system-prompt-file`: Override assistant system prompt file (more specific).\n- `--assistant-tools-file`: Override tools file for the assistant (more specific).\n- `--simulator-system-prompt`: Override simulator system prompt (string).\n- `--chunk-ms`, `--sample-rate-hz`, `--real-time`.\n- `--max-turns`, `--max-examples`.\n- `--skip-plots`.\n\nNotes:\n\n- Temperature and max output tokens are intentionally not supported.\n- The simulator emits audio only. Transcripts are captured from audio deltas if available.\n- Tool outputs are mocked deterministically from each simulation JSON.\n- `results.csv` includes an `event_log_path` column pointing to the JSONL trace.\n","examples/evals/realtime_evals/results_viewer/README.md":"# Results Viewer\n\nThis directory contains a [Streamlit](https://streamlit.io/) app for exploring\nsaved realtime eval runs from the crawl, walk, and run harnesses, plus\nbootstrap-generated eval folders created under `examples/evals/realtime_evals/`.\n\nThe app auto-discovers run directories under:\n\n- `crawl_harness/results/`\n- `walk_harness/results/`\n- `run_harness/results/`\n- `*_realtime_eval/results/` when the folder includes `bootstrap_manifest.json`\n\n## What It Shows\n\n- **Comparison View**: compare summary metrics across one or more saved runs\n- **Run Viewer**: inspect a single saved run, including:\n  - `results.csv` rows\n  - crawl/walk input and output audio artifacts\n  - crawl/walk per-example event logs\n  - run-harness simulation transcripts, event logs, and turn audio\n\nNote: `run_harness` runs are inspected via the app's **Simulation Viewer**,\nwhich has a different UI and a different artifact set than the crawl/walk\nviewer.\n\n## Run Locally\n\nFrom `examples/evals/realtime_evals/`:\n\n```bash\nuv venv .venv\nsource .venv/bin/activate\nuv sync --group dev\ncd results_viewer\nuv run streamlit run app.py\n```\n\nThen open the local URL that Streamlit prints, usually\n`http://localhost:8501`.\n\nIf you are using the pip-based install path instead of `uv`, install the dev\ndependencies first so `streamlit` is available:\n\n```bash\npip install -r requirements.txt -r requirements-dev.txt\ncd results_viewer\nstreamlit run app.py\n```\n\n## Expected Data Layout\n\nThe viewer expects each saved run directory to contain:\n\n- `summary.json` for aggregate metrics\n- `results.csv` for per-example results\n\nFor crawl and walk runs, the app can also display:\n\n- `audio/<example_id>/input.wav`\n- `audio/<example_id>/output.wav`\n- `events/<example_id>.jsonl`\n\nThe app discovers runs recursively, so nested result directories are fine as\nlong as those files are present.\n","examples/evals/realtime_evals/crawl_harness/README.md":"# Crawl Harness\n\nDeterministic single-turn replay for Realtime evals. This harness feeds fixed audio\ninputs to a Realtime session so you can compare runs with minimal variance.\nIt is best for quick iteration and controlled comparisons.\n\n## What it does\n\n- Loads a CSV of eval prompts from `crawl_harness/data/`.\n- Uses TTS to synthesize a fixed input audio file per row.\n- Streams audio into the Realtime API in fixed-size chunks.\n- Captures the first assistant turn (text/audio/tool calls).\n- Grades tool-call correctness and tool-call-arg correctness only.\n- Writes artifacts into `crawl_harness/results/<run_id>/`.\n\n## Inputs\n\nCSV columns (required):\n\n- `example_id`: Unique ID for the datapoint.\n- `user_text`: Text prompt to synthesize into audio.\n- `gt_tool_call`: Expected tool name or empty if no tool call is expected.\n- `gt_tool_call_arg`: Expected tool arguments as JSON string or empty.\n\n## Outputs\n\nEach run writes:\n\n- `results.csv`: Per-example outputs and grades.\n- `summary.json`: Aggregate metrics (latency + correctness).\n- `plots/*.png`: Styled score, latency, token, and status charts.\n- `audio/<example_id>/input.wav`: The TTS input audio.\n- `audio/<example_id>/output.wav`: The model output audio (if any).\n- `events/<example_id>.jsonl`: Realtime event stream for the datapoint.\n\n## How it works\n\n1. TTS generates PCM audio for each row.\n2. PCM is wrapped into a WAV file for easy playback.\n3. Audio is streamed into a Realtime session with fixed chunk sizes.\n4. The harness listens for `response.done` and records tool calls + output.\n5. Grades are computed:\n   - `tool_call_correctness`: correct tool chosen (or no tool call when none expected).\n   - `tool_call_arg_correctness`: expected args present in the tool call.\n\n## Run\n\nFrom repo root:\n\n```bash\npython crawl_harness/run_realtime_evals.py\n```\n\nCommon options:\n\n- `--data-csv`: Path to a CSV file.\n- `--results-dir`: Output folder (defaults to `crawl_harness/results`).\n- `--run-name`: Optional run name (defaults to timestamp).\n- `--model`: Realtime model name.\n- `--tts-model`: TTS model name.\n- `--system-prompt-file`: System prompt file path.\n- `--tools-file`: Tools JSON file path.\n- `--voice`: Output voice name.\n- `--chunk-ms`: Audio chunk size in ms.\n- `--sample-rate-hz`: Input sample rate in Hz.\n- `--input-audio-format`: Input audio format (default `pcm16`).\n- `--output-audio-format`: Output audio format (default `pcm16`).\n- `--real-time`: Stream audio in real-time cadence.\n- `--max-examples`: Limit number of examples for quick checks.\n- `--skip-plots`: Skip post-run PNG chart generation.\n\n## Adapt\n\n- Add or replace rows in `crawl_harness/data/customer_service_synthetic.csv`.\n- Change tools or system prompt in `shared/`.\n- Update grading logic in `run_realtime_evals.py`.\n\n## Notes\n\n- The harness only evaluates the first assistant turn and does not provide tool outputs.\n- `results.csv` includes an `event_log_path` column pointing to the JSONL trace.\n- If you expect tools to be called reliably, adjust the system prompt or tool choice.\n","examples/evals/imagegen_evals/generation_harness/README.md":"# Generation Harness\n\n**Best for:** fast iteration on prompt-following, layout, and text-rendering\nquality for text-to-image workflows.\n\nThis harness runs two example cases:\n\n- `ui_checkout_mockup`: mobile checkout UI mockup\n- `coffee_flyer_generation`: marketing flyer with strict copy constraints\n\n## Run\n\n```bash\npython generation_harness/run_imagegen_evals.py\n```\n\nQuick smoke test for a single case (use `--cases`):\n\n```bash\npython generation_harness/run_imagegen_evals.py --cases ui_checkout_mockup\n```\n\nRun multiple cases (comma-separated, no spaces):\n\n```bash\npython generation_harness/run_imagegen_evals.py --cases ui_checkout_mockup,coffee_flyer_generation\n```\n\nOptional OCR-style text check for the coffee flyer:\n\n```bash\npython generation_harness/run_imagegen_evals.py --run-ocr\n```\n\n## Flags (when to use them)\n\n- `--cases`: limit runs to specific case ids. Valid values:\n  - `ui_checkout_mockup`\n  - `coffee_flyer_generation`\n- `--model`: image model under test (defaults to `gpt-image-1.5`).\n- `--judge-model`: LLM used to grade outputs (defaults to `gpt-5.2`).\n- `--num-images`: number of images to generate per case (defaults to 1).\n- `--image-size`: size for generated images (defaults to `1024x1024`).\n- `--run-ocr`: only relevant for `coffee_flyer_generation` (checks exact text).\n\n## Outputs\n\nResults are written under `generation_harness/results/<run_id>/`:\n\n- `results.json` and `results.csv` with per-example scores\n- `summary.json` with aggregated metrics\n- `artifacts/` for generated images\n\n## Adapting the harness\n\n- Update the prompts/criteria in `run_imagegen_evals.py`.\n- Tune JSON schemas + parsers to match your evaluation rubric.\n- Add more cases by creating new `TestCase` entries and graders.\n","examples/evals/realtime_evals/shared/tools.json":"# Shared Skill\n\nPath: `examples/evals/realtime_evals/shared/tools.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/shared/tools.json)","examples/evals/realtime_evals/walk_harness/README.md":"# Walk_harness Skill\n\nPath: `examples/evals/realtime_evals/walk_harness/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/walk_harness/README.md)","examples/evals/realtime_evals/run_harness/data/sim_cs_address_update.json":"# Data Skill\n\nPath: `examples/evals/realtime_evals/run_harness/data/sim_cs_address_update.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/run_harness/data/sim_cs_address_update.json)","examples/evals/realtime_evals/run_harness/data/sim_cs_order_status.json":"# Data Skill\n\nPath: `examples/evals/realtime_evals/run_harness/data/sim_cs_order_status.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/run_harness/data/sim_cs_order_status.json)","examples/evals/realtime_evals/run_harness/data/sim_cs_refund.json":"# Data Skill\n\nPath: `examples/evals/realtime_evals/run_harness/data/sim_cs_refund.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/run_harness/data/sim_cs_refund.json)","examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/definitions.json":"# Tools Skill\n\nPath: `examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/definitions.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/definitions.json)","examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/restaurant_facts.json":"# Tools Skill\n\nPath: `examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/restaurant_facts.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/restaurant_facts.json)","examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/definitions.json":"# Tools Skill\n\nPath: `examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/definitions.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/definitions.json)","examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/restaurant_facts.json":"# Tools Skill\n\nPath: `examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/restaurant_facts.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/restaurant_facts.json)","examples/evals/realtime_evals/skills/bootstrap-realtime-eval/agents/openai.yaml":"# Agents Skill\n\nPath: `examples/evals/realtime_evals/skills/bootstrap-realtime-eval/agents/openai.yaml`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/skills/bootstrap-realtime-eval/agents/openai.yaml)","examples/evals/realtime_evals/skills/bootstrap-realtime-eval/references/harness-selection.md":"# References Skill\n\nPath: `examples/evals/realtime_evals/skills/bootstrap-realtime-eval/references/harness-selection.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/skills/bootstrap-realtime-eval/references/harness-selection.md)","examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/.agents/plugins/marketplace.json":"# Claude Plugin Marketplace Catalog\n\nPath: `examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/.agents/plugins/marketplace.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/.agents/plugins/marketplace.json)","examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.mcp.json":"# Bls Skill\n\nPath: `examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.mcp.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.mcp.json)","examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/README.md":"# bls Documentation\n\nPath: `examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/README.md)","examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.codex-plugin/plugin.json":"# codex-plugin Plugin Manifest\n\nPath: `examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.codex-plugin/plugin.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.codex-plugin/plugin.json)"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# Repository Guidelines\n\n## Project Structure & Module Organization\n\nThe cookbook is organized around runnable examples and reference articles for OpenAI APIs. Place notebooks and Python scripts under `examples/<topic>/`, grouping related assets inside topic subfolders (for example, `examples/agents_sdk/`). Narrative guides and long-form docs live in `articles/`, and shared diagrams or screenshots belong in `images/`. Update `registry.yaml` whenever you add content so it appears on cookbook.openai.com, and add new author metadata in `authors.yaml` if you want custom attribution. Keep large datasets outside the repo; instead, document how to fetch them in the notebook.\n\n## Build, Test, and Development Commands\n\nUse a virtual environment to isolate dependencies:\n\n- `python -m venv .venv && source .venv/bin/activate`\n- `pip install -r examples/<topic>/requirements.txt` (each sample lists only what it needs)\n- `jupyter lab` or `jupyter notebook` to develop interactively\n- `python .github/scripts/check_notebooks.py` to validate notebook structure before pushing\n\n## Coding Style & Naming Conventions\n\nWrite Python to PEP 8 with four-space indentation, descriptive variable names, and concise docstrings that explain API usage choices. Name new notebooks with lowercase, dash-or-underscore-separated phrases that match their directory—for example `examples/gpt-5/prompt-optimization-cookbook.ipynb`. Keep markdown cells focused and prefer numbered steps for multi-part workflows. Store secrets in environment variables such as `OPENAI_API_KEY`; never hard-code keys inside notebooks.\n\n## Testing Guidelines\n\nExecute notebooks top-to-bottom after installing dependencies and clear lingering execution counts before committing. For Python modules or utilities, include self-check cells or lightweight `pytest` snippets and show how to run them (for example, `pytest examples/object_oriented_agentic_approach/tests`). When contributions depend on external services, mock responses or gate the cells behind clearly labeled opt-in flags.\n\n## Commit & Pull Request Guidelines\n\nUse concise, imperative commit messages that describe the change scope (e.g., \"Add agent portfolio collaboration demo\"). Every PR should provide a summary, motivation, and self-review, and must tick the registry and authors checklist from `.github/pull_request_template.md`. Link issues when applicable and attach screenshots or output snippets for UI-heavy content. Confirm CI notebook validation passes locally before requesting review.\n\nBefore a pull request is merged, run the repository's `docs-editor` skill on every changed article, Markdown or MDX file, and notebook Markdown cell. For editorial notebook changes, preserve code cells, outputs, execution counts, attachments, cell order, and metadata. Fix safe issues, rerun the relevant checks, and resolve or explicitly document all remaining P0 and P1 findings in the pull request.\n\n## Metadata & Publication Workflow\n\nNew or relocated content must have an entry in `registry.yaml` with an accurate path, date, and tag set so the static site generator includes it. When collaborating, coordinate author slugs in `authors.yaml` to avoid duplicates, and run `python -m yaml lint registry.yaml` (or your preferred YAML linter) to catch syntax errors before submitting.\n\n## Review Guidelines\n\nThese are considered priority 0 issues for this repo, in addition to the normal priority for possible issues.\n- Verify file, function, and notebook names follow the repo's naming conventions and clearly describe their purpose.\n- Scan prose and markdown for typos, broken links, and inconsistent formatting before approving.\n- Check that code identifiers remain descriptive (no leftover placeholder names) and that repeated values are factored into constants when practical.\n- Ensure notebooks or scripts document any required environment variables instead of hard-coding secrets or keys.\n- Confirm metadata files (`registry.yaml`, `authors.yaml`) stay in sync with new or relocated content.\n\n## Recent Learnings\n\n- **`uv run` can inherit the wrong virtualenv in this repo** -> Clear `VIRTUAL_ENV` (for example `env -u VIRTUAL_ENV uv run ...`) when an unrelated environment is active -> Avoids misleading mismatch warnings and makes it clear the repo's `.venv` is the interpreter actually running the harnesses.\n- **Realtime eval shared imports can resolve the wrong module under pytest** -> Add `shared/__init__.py` and ensure tests prepend `examples/evals/realtime_evals` to `sys.path` before importing `shared.*` -> Prevents collection failures caused by unrelated installed packages named `shared`.\n- **Run-level grades can be overweighted by long simulations** -> Store turn-level grades on the matching turn and trace-level grades on one row per simulation instead of copying them onto every row -> Keeps `results.csv` row semantics intact and prevents summary means from favoring longer conversations.\n- **Synthetic-audio scaffold requests can pick the wrong harness** -> Default unspecified synthetic-audio evals to `crawl` text-to-TTS and reserve `walk` for replay-specific audio traits like noise, telephony artifacts, or speaker characteristics -> Keeps new realtime evals on the simplest harness unless audio realism is itself under test.\n- **Task-specific single-turn grading can outgrow the shared crawl schema** -> Keep the shared crawl harness for realtime execution, then add eval-local wrapper scripts that post-grade domain-specific quality and overwrite `results.csv` while preserving `results_base.csv` -> Avoids forking the harness when a use case needs richer grading than tool-call correctness.\n- **Synthetic learner audio can sound like eval scaffolding** -> Write `user_text` as a realistic in-app learner request and keep evaluation rules in metadata plus the system prompt -> Produces audio inputs that match the product surface instead of teaching the model the grading rubric through the spoken prompt.\n","isInternal":false,"tokens":1170,"sizeBytes":5964},{"name":"style-guide.md","path":".codex/skills/docs-editor/resources/style-guide.md","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/.codex/skills/docs-editor/resources/style-guide.md","title":"Resources Skill","category":"anthropic-skill","format":"markdown","content":"# OpenAI Cookbook editorial checklist\n\nUse this checklist for articles and notebook Markdown cells in `openai-cookbook`. Apply only the rules relevant to the content under review.\n\n## Audience and purpose\n\n- Write for developers who want to understand and run an OpenAI API example.\n- State the task or outcome early. Keep background material only when it helps the reader use the example correctly.\n- Assume general programming knowledge, but explain Cookbook-specific setup, choices, and tradeoffs.\n- Avoid promotional language, vague praise, and claims that the example does not demonstrate.\n\n## Technical accuracy\n\n- Match API, model, method, parameter, field, file, and environment-variable names exactly.\n- Keep prose consistent with the code, sample inputs, outputs, diagrams, and nearby cells.\n- Check counts, ordered steps, cross-references, defaults, constraints, and stated results.\n- Do not infer a correction when the intended fact is unclear. Preserve the source and flag it for review.\n- Use authoritative sources to verify uncertain product behavior.\n- Do not imply that an optional step is required or that a required step is optional.\n\n## Runnable examples\n\n- List prerequisites and dependencies that readers need before running the example.\n- Document required environment variables. Never hard-code API keys, tokens, credentials, or other secrets.\n- Keep setup instructions in the same order that readers perform them.\n- Introduce code with the goal or reason for the step; do not narrate every line.\n- Use realistic inputs and clearly label placeholders.\n- Show expected output only when it helps readers verify success, and keep it consistent with the code.\n- Keep external-service calls and other opt-in behavior clearly labeled.\n\n## Notebook integrity\n\n- Edit Markdown cells only during an editorial pass.\n- Preserve code cells, outputs, execution counts, attachments, cell IDs, cell order, and metadata.\n- Keep notebook JSON valid and avoid whole-file reserialization for a small prose edit.\n- Keep each Markdown cell focused. Split or merge cells only when the structure is genuinely confusing and the request permits it.\n- Use Markdown headings in a logical hierarchy without skipping levels unnecessarily.\n\n## Prose\n\n- Lead with the point. Prefer concrete nouns and strong verbs.\n- Use direct instructions and address the reader as “you” when helpful.\n- Prefer active voice, but keep passive voice when the actor is unknown or unimportant.\n- Remove throat-clearing, repeated conclusions, filler, and ornamental transitions.\n- Keep paragraphs short enough to scan and use lists for genuinely parallel items.\n- Use numbered lists for ordered procedures and bullets for unordered groups.\n- Keep sibling headings and list items parallel.\n- Use sentence-style capitalization for headings and do not end headings with periods.\n- Preserve useful warmth and explanation; do not flatten the content into terse fragments.\n\n## Terminology and formatting\n\n- Use exact product terminology and consistent capitalization.\n- Format commands, options, environment variables, paths, filenames, code identifiers, and literal values with backticks.\n- Match the capitalization and spelling used by the language or API.\n- Expand uncommon abbreviations on first use unless the surrounding audience clearly knows them.\n- Use inclusive, precise language and neutral role terms.\n- Do not replace a technically precise term merely to vary wording.\n\n## Links and assets\n\n- Prefer descriptive link text over “here” or a pasted URL.\n- Verify local links, anchors, image paths, and referenced notebook or article paths.\n- Use stable, authoritative external links when available.\n- Give images meaningful alt text unless they are purely decorative.\n- Keep diagrams, screenshots, captions, and prose consistent.\n\n## Publication metadata\n\n- Add or update `registry.yaml` for new, moved, renamed, or removed published content.\n- Confirm registry paths, dates, tags, and titles match the content.\n- Add or update `authors.yaml` when introducing or changing author attribution.\n- Check that filenames follow the repository's naming conventions and clearly describe the content.\n\n## Final review\n\n- Read each edited passage in context, not only as an isolated diff hunk.\n- Confirm no factual meaning, literal value, condition, negation, or uncertainty changed unintentionally.\n- Confirm all repository-defined P0 issues are fixed.\n- Run the notebook structural validator for changed notebooks and inspect the final diff for unintended changes.\n","isInternal":false,"tokens":865,"sizeBytes":4542},{"name":"SKILL.md","path":".codex/skills/docs-editor/SKILL.md","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/.codex/skills/docs-editor/SKILL.md","title":"docs-editor","category":"anthropic-skill","format":"markdown","content":"---\nname: docs-editor\ndescription: Review and edit OpenAI Cookbook notebooks, Markdown, and MDX for technical accuracy, clarity, grammar, consistency, runnable examples, and repository publication requirements. Use for editorial reviews, pre-merge documentation checks, or notebook Markdown-cell sweeps in openai-cookbook.\n---\n\n# OpenAI Cookbook docs editor\n\n## Objective\n\nReview Cookbook content conservatively. Improve correctness and readability without changing the author's intent, inventing facts, or creating unrelated churn.\n\n## Workflow\n\n### 1. Define the scope\n\n- Prefer explicit paths or the files changed in the pull request.\n- For pull requests, review changed `.md` and `.mdx` files plus Markdown cells in changed `.ipynb` files.\n- Review `registry.yaml` and `authors.yaml` when content is added, moved, removed, or attributed to a new author.\n- Stay within the requested content. Do not turn a focused review into a repository-wide rewrite.\n\n### 2. Protect notebook integrity\n\n- When the request is editorial, edit only notebook Markdown cells.\n- Preserve code cells, outputs, execution counts, attachments, cell order, cell IDs, and notebook and cell metadata.\n- Avoid reserializing an entire notebook for a small prose change. Inspect the diff for unintended JSON churn.\n- Do not execute notebook code unless the user requests it or code changes are in scope. Markdown-only edits require structural validation, not calls to external services.\n- Never add hard-coded secrets. Document required environment variables such as `OPENAI_API_KEY`.\n\n### 3. Review the content\n\nLoad `resources/style-guide.md` and check:\n\n- Technical claims, API names, model names, identifiers, commands, paths, and URLs.\n- Agreement between prose, code, sample inputs, and expected outputs.\n- Prerequisites, dependencies, environment variables, and steps needed to run an example.\n- File and notebook names, headings, lists, numbering, cross-references, and links.\n- Grammar, spelling, punctuation, terminology, capitalization, and concise wording.\n- Publication metadata in `registry.yaml` and author metadata in `authors.yaml` when applicable.\n\nVerify uncertain technical claims from an authoritative source or flag them for review. Do not guess.\n\n### 4. Prioritize findings\n\nUse the repository's review bar:\n\n- **P0**: Broken notebook JSON; hard-coded secrets; materially false or unsafe guidance; or any repository-defined P0 issue, including naming violations, typos, broken links, inconsistent formatting, placeholder identifiers, undocumented environment variables, or out-of-sync publication metadata.\n- **P1**: Misleading instructions, prose that contradicts code or output, missing prerequisites, or a required link that does not work.\n- **P2**: Clear improvements to grammar, consistency, organization, or readability that do not affect correctness.\n- **P3**: Optional polish with little effect on comprehension.\n\n### 5. Make safe edits\n\n- Fix obvious typos, grammar, punctuation, broken local references, and clear prose/code mismatches.\n- Keep changes minimal, local, and reversible.\n- Preserve exact literals, code identifiers, commands, values, counts, conditions, modality, negation, and uncertainty markers.\n- Tighten prose only when meaning stays unchanged. Prefer concrete nouns, strong verbs, short introductions, and direct instructions.\n- Flag any change that depends on product knowledge or could alter meaning instead of guessing.\n- Do not make file-wide formatting or stylistic changes solely for uniformity.\n\n### 6. Validate the result\n\nFor any changed notebook, run the repository validator:\n\n```bash\nenv -u VIRTUAL_ENV uv run --with nbformat python .github/scripts/check_notebooks.py\n```\n\nThen:\n\n- Run `git diff --check`.\n- Inspect the diff and confirm that notebook changes are limited to intended Markdown cells.\n- Run an available YAML linter if `registry.yaml` or `authors.yaml` changed.\n- Run the relevant example or test only when executable content changed and its dependencies are available.\n\n### 7. Report the review\n\nProvide:\n\n- A short summary of the scope and edits.\n- Auto-fixed items.\n- Remaining findings grouped by priority with file and cell or line references.\n- The validation commands run and their results.\n\nIf no issues remain, say so explicitly.\n","frontmatter":{"name":"docs-editor","description":"Review and edit OpenAI Cookbook notebooks, Markdown, and MDX for technical accuracy, clarity, grammar, consistency, runnable examples, and repository publication requirements. Use for editorial reviews, pre-merge documentation checks, or notebook Markdown-cell sweeps in openai-cookbook."},"isInternal":false,"tokens":876,"sizeBytes":4290},{"name":"SKILL.md","path":"examples/agents_api/apps/document_review/skills/expense-review-policy/SKILL.md","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/agents_api/apps/document_review/skills/expense-review-policy/SKILL.md","title":"expense-review-policy","category":"anthropic-skill","format":"markdown","content":"---\nname: expense-review-policy\ndescription: Review invoices and contracts against accounts-payable policy before human approval.\n---\n\n# Expense review policy\n\nPolicy ID: `AP-104`.\n\nApply this policy to invoices, expense receipts, and service contracts.\n\n## Invoices and expenses\n\n- Read the entire invoice, including the vendor and every line item.\n- Extract every line item's quantity and unit price before calculating. Check the extracted list against every source line; an empty list is an error, not a zero subtotal.\n- Use Python's `decimal.Decimal` to multiply quantities by unit prices and sum the line totals. Compare this subtotal with the printed subtotal, then add shipping to calculate the total.\n- Strip currency separators, compare the result with `Total amount due`, and report the exact difference (stated total minus calculated total).\n- Require a purchase order or another documented approval.\n- Flag missing receipts, unsupported charges, and changed payment instructions.\n- Escalate incorrect totals and unverified changes to bank details.\n\n## Contracts\n\n- Read every clause and report all matching risks, even when one already requires escalation.\n- Flag automatic renewals and restrictive cancellation windows.\n- Flag unilateral price increases and missing liability limits.\n- Escalate customer-data sharing or subcontractor access without approval.\n- Identify missing confidentiality, security, or termination terms.\n\n## Review decision\n\n- `needs_info`: Required documentation or supporting information is missing.\n- `escalated`: Fraud indicators, financial discrepancies, or risky terms need review.\n- `ready_for_approval`: No policy violations remain, but a human must still approve.\n\nWrite `/workspace/output/<document-stem>.json` with the following fields:\n\n```json\n{\n  \"document\": \"invoice.txt\",\n  \"document_type\": \"invoice\",\n  \"policy_id\": \"AP-104\",\n  \"decision\": \"escalated\",\n  \"vendor\": \"Cedar Office Supply\",\n  \"amount\": 6420,\n  \"issues\": [\"The claimed total exceeds the calculated total.\"],\n  \"recommendation\": \"Escalate for human review.\",\n  \"calculation\": {\n    \"line_items\": [{\"quantity\": 2, \"unit_price\": 100}],\n    \"shipping\": 20,\n    \"calculated_total\": 220,\n    \"difference\": 6200\n  }\n}\n```\n\nEvery `issues` entry must be a plain-English string, not a nested object.\nUse the actual document values, not the illustrative numbers above. Include every\ninvoice line item in `calculation`; `amount` is the stated total. For contracts,\nset `document_type` to `contract` and `calculation` to `null`.\n\nOnly the coordinator writes `summary.json`. A specialist writes its assigned report.\n\nNever approve payments, sign contracts, or take external actions.\n","frontmatter":{"name":"expense-review-policy","description":"Review invoices and contracts against accounts-payable policy before human approval."},"isInternal":false,"tokens":570,"sizeBytes":2685},{"name":"definitions.json","path":"examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/definitions.json","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/definitions.json","title":"Tools Skill","category":"anthropic-skill","format":"json","content":"# Tools Skill\n\nPath: `examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/definitions.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/definitions.json)","tokens":300,"sizeBytes":287},{"name":"restaurant_facts.json","path":"examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/restaurant_facts.json","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/restaurant_facts.json","title":"Tools Skill","category":"anthropic-skill","format":"json","content":"# Tools Skill\n\nPath: `examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/restaurant_facts.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/restaurant_facts.json)","tokens":300,"sizeBytes":297},{"name":"definitions.json","path":"examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/definitions.json","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/definitions.json","title":"Tools Skill","category":"anthropic-skill","format":"json","content":"# Tools Skill\n\nPath: `examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/definitions.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/definitions.json)","tokens":300,"sizeBytes":293},{"name":"restaurant_facts.json","path":"examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/restaurant_facts.json","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/restaurant_facts.json","title":"Tools Skill","category":"anthropic-skill","format":"json","content":"# Tools Skill\n\nPath: `examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/restaurant_facts.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/restaurant_facts.json)","tokens":300,"sizeBytes":303},{"name":"README.md","path":"examples/evals/imagegen_evals/editing_harness/README.md","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/imagegen_evals/editing_harness/README.md","title":"Editing_harness Skill","category":"anthropic-skill","format":"markdown","content":"# Editing Harness\n\n**Best for:** high-precision checks where edits must be exact and non-target\ncontent should remain unchanged.\n\nThis harness runs two example cases:\n\n- `vto_jacket_tryon`: virtual try-on using reference person + garment\n- `logo_year_edit`: precision logo text edit\n\n## Run\n\n```bash\npython editing_harness/run_imagegen_evals.py\n```\n\nRun a single case (use `--cases`):\n\n```bash\npython editing_harness/run_imagegen_evals.py --cases vto_jacket_tryon\n```\n\nRun multiple cases (comma-separated, no spaces):\n\n```bash\npython editing_harness/run_imagegen_evals.py --cases vto_jacket_tryon,logo_year_edit\n```\n\n## Flags (when to use them)\n\n- `--cases`: limit runs to specific case ids. Valid values:\n  - `vto_jacket_tryon`\n  - `logo_year_edit`\n- `--model`: image model under test (defaults to `gpt-image-1.5`).\n- `--judge-model`: LLM used to grade outputs (defaults to `gpt-5.2`).\n- `--num-images`: number of edited images to generate per case (defaults to 1).\n\n## Inputs\n\nThis harness expects the following reference images in `images/`:\n\n- `images/base_woman.png`\n- `images/jacket.png`\n- `images/logo_generation_1.png`\n\n## Outputs\n\nResults are written under `editing_harness/results/<run_id>/`:\n\n- `results.json` and `results.csv` with per-example scores\n- `summary.json` with aggregated metrics\n- `artifacts/` for edited images\n\n## Adapting the harness\n\n- Replace the input images and edit prompts in `run_imagegen_evals.py`.\n- Update the schemas and verdict rules to fit your workflow.\n","isInternal":false,"tokens":388,"sizeBytes":1496},{"name":"README.md","path":"examples/evals/imagegen_evals/generation_harness/README.md","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/imagegen_evals/generation_harness/README.md","title":"Generation_harness Skill","category":"anthropic-skill","format":"markdown","content":"# Generation Harness\n\n**Best for:** fast iteration on prompt-following, layout, and text-rendering\nquality for text-to-image workflows.\n\nThis harness runs two example cases:\n\n- `ui_checkout_mockup`: mobile checkout UI mockup\n- `coffee_flyer_generation`: marketing flyer with strict copy constraints\n\n## Run\n\n```bash\npython generation_harness/run_imagegen_evals.py\n```\n\nQuick smoke test for a single case (use `--cases`):\n\n```bash\npython generation_harness/run_imagegen_evals.py --cases ui_checkout_mockup\n```\n\nRun multiple cases (comma-separated, no spaces):\n\n```bash\npython generation_harness/run_imagegen_evals.py --cases ui_checkout_mockup,coffee_flyer_generation\n```\n\nOptional OCR-style text check for the coffee flyer:\n\n```bash\npython generation_harness/run_imagegen_evals.py --run-ocr\n```\n\n## Flags (when to use them)\n\n- `--cases`: limit runs to specific case ids. Valid values:\n  - `ui_checkout_mockup`\n  - `coffee_flyer_generation`\n- `--model`: image model under test (defaults to `gpt-image-1.5`).\n- `--judge-model`: LLM used to grade outputs (defaults to `gpt-5.2`).\n- `--num-images`: number of images to generate per case (defaults to 1).\n- `--image-size`: size for generated images (defaults to `1024x1024`).\n- `--run-ocr`: only relevant for `coffee_flyer_generation` (checks exact text).\n\n## Outputs\n\nResults are written under `generation_harness/results/<run_id>/`:\n\n- `results.json` and `results.csv` with per-example scores\n- `summary.json` with aggregated metrics\n- `artifacts/` for generated images\n\n## Adapting the harness\n\n- Update the prompts/criteria in `run_imagegen_evals.py`.\n- Tune JSON schemas + parsers to match your evaluation rubric.\n- Add more cases by creating new `TestCase` entries and graders.\n","isInternal":false,"tokens":437,"sizeBytes":1729},{"name":"README.md","path":"examples/evals/imagegen_evals/README.md","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/imagegen_evals/README.md","title":"Imagegen_evals Skill","category":"anthropic-skill","format":"markdown","content":"# Image Generation + Editing Evals\n\nThis folder contains a lightweight vision eval harness plus example runners for\nimage generation and image editing. The code mirrors the structure of\n`examples/evals/realtime_evals/` so you can adapt it quickly.\n\n**Directory layout**\n\n- `vision_harness/`: minimal shared library (types, runners, graders, evaluate loop)\n- `generation_harness/`: text-to-image evals (UI mockups + marketing flyer)\n- `editing_harness/`: image-edit evals (virtual try-on + logo edit)\n- `shared/`: reporting and optional rendering helpers\n\n## Quickstart\n\nPython 3.9+ required.\n\n```bash\npip install -r requirements.txt\nexport OPENAI_API_KEY=\"your_api_key\"\n```\n\nRun a harness:\n\n- Generation: `python generation_harness/run_imagegen_evals.py`\n- Editing: `python editing_harness/run_imagegen_evals.py`\n\n## What the harness does\n\n1. Builds a small set of `TestCase` objects (prompt + criteria).\n2. Runs the image model for each case.\n3. Grades each output with an LLM judge using a strict JSON schema.\n4. Writes results and artifacts to `results/<run_id>/`.\n\nThe harness is intentionally small so you can copy/paste parts into your own\nproduction eval setup.\n\n## Example cases\n\nGeneration cases:\n- `ui_checkout_mockup`: mobile checkout screen with strict text + layout rules\n- `coffee_flyer_generation`: marketing flyer with exact copy constraints\n\nEditing cases:\n- `vto_jacket_tryon`: virtual try-on with reference person + garment\n- `logo_year_edit`: precision logo text edit\n\n## Required assets (editing harness)\n\nThe editing harness expects these files in `images/`:\n\n- `images/base_woman.png`\n- `images/jacket.png`\n- `images/logo_generation_1.png`\n\n## Results layout\n\nEach harness writes into its own `results/` folder:\n\n- `results/<run_id>/results.json`: per-example outputs and grades\n- `results/<run_id>/results.csv`: tabular results\n- `results/<run_id>/summary.json`: aggregated metrics\n- `results/<run_id>/artifacts/*.png`: generated or edited images\n\nTip: keep `results/` out of git (delete after runs or add to `.gitignore`).\n\n## Common CLI flags\n\nBoth harnesses accept a shared core set of flags:\n\n- `--results-dir`, `--run-name`\n- `--model` (image model under test), `--judge-model` (LLM judge)\n- `--num-images` (images per case)\n- `--cases` (comma-separated list of case ids)\n\nGeneration harness adds `--image-size` and `--run-ocr` (text extraction check).\n\n## Suggested workflows\n\n- Quick smoke test:\n  - `python generation_harness/run_imagegen_evals.py --cases ui_checkout_mockup`\n  - `python editing_harness/run_imagegen_evals.py --cases vto_jacket_tryon`\n- Compare model variants: run once per model name and compare `summary.json`.\n- Use `--run-ocr` on the flyer case to validate exact text rendering.\n\n## Adapting for your use case\n\n- Replace prompts/criteria in the harness scripts.\n- Update JSON schemas to match your rubric (and parse them into `Score` values).\n- Add new cases by creating new `TestCase` objects and graders.\n- Swap the output store to change where artifacts are written.\n\n## Notes\n\n- This harness uses the OpenAI Images API and a judge model. Costs scale with\n  the number of images and judge calls.\n- Judge outputs are best treated as signals, not ground truth. Keep a small\n  human review loop for high-stakes decisions.\n","isInternal":false,"tokens":790,"sizeBytes":3275},{"name":"README.md","path":"examples/evals/realtime_evals/crawl_harness/README.md","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/crawl_harness/README.md","title":"Crawl_harness Skill","category":"anthropic-skill","format":"markdown","content":"# Crawl Harness\n\nDeterministic single-turn replay for Realtime evals. This harness feeds fixed audio\ninputs to a Realtime session so you can compare runs with minimal variance.\nIt is best for quick iteration and controlled comparisons.\n\n## What it does\n\n- Loads a CSV of eval prompts from `crawl_harness/data/`.\n- Uses TTS to synthesize a fixed input audio file per row.\n- Streams audio into the Realtime API in fixed-size chunks.\n- Captures the first assistant turn (text/audio/tool calls).\n- Grades tool-call correctness and tool-call-arg correctness only.\n- Writes artifacts into `crawl_harness/results/<run_id>/`.\n\n## Inputs\n\nCSV columns (required):\n\n- `example_id`: Unique ID for the datapoint.\n- `user_text`: Text prompt to synthesize into audio.\n- `gt_tool_call`: Expected tool name or empty if no tool call is expected.\n- `gt_tool_call_arg`: Expected tool arguments as JSON string or empty.\n\n## Outputs\n\nEach run writes:\n\n- `results.csv`: Per-example outputs and grades.\n- `summary.json`: Aggregate metrics (latency + correctness).\n- `plots/*.png`: Styled score, latency, token, and status charts.\n- `audio/<example_id>/input.wav`: The TTS input audio.\n- `audio/<example_id>/output.wav`: The model output audio (if any).\n- `events/<example_id>.jsonl`: Realtime event stream for the datapoint.\n\n## How it works\n\n1. TTS generates PCM audio for each row.\n2. PCM is wrapped into a WAV file for easy playback.\n3. Audio is streamed into a Realtime session with fixed chunk sizes.\n4. The harness listens for `response.done` and records tool calls + output.\n5. Grades are computed:\n   - `tool_call_correctness`: correct tool chosen (or no tool call when none expected).\n   - `tool_call_arg_correctness`: expected args present in the tool call.\n\n## Run\n\nFrom repo root:\n\n```bash\npython crawl_harness/run_realtime_evals.py\n```\n\nCommon options:\n\n- `--data-csv`: Path to a CSV file.\n- `--results-dir`: Output folder (defaults to `crawl_harness/results`).\n- `--run-name`: Optional run name (defaults to timestamp).\n- `--model`: Realtime model name.\n- `--tts-model`: TTS model name.\n- `--system-prompt-file`: System prompt file path.\n- `--tools-file`: Tools JSON file path.\n- `--voice`: Output voice name.\n- `--chunk-ms`: Audio chunk size in ms.\n- `--sample-rate-hz`: Input sample rate in Hz.\n- `--input-audio-format`: Input audio format (default `pcm16`).\n- `--output-audio-format`: Output audio format (default `pcm16`).\n- `--real-time`: Stream audio in real-time cadence.\n- `--max-examples`: Limit number of examples for quick checks.\n- `--skip-plots`: Skip post-run PNG chart generation.\n\n## Adapt\n\n- Add or replace rows in `crawl_harness/data/customer_service_synthetic.csv`.\n- Change tools or system prompt in `shared/`.\n- Update grading logic in `run_realtime_evals.py`.\n\n## Notes\n\n- The harness only evaluates the first assistant turn and does not provide tool outputs.\n- `results.csv` includes an `event_log_path` column pointing to the JSONL trace.\n- If you expect tools to be called reliably, adjust the system prompt or tool choice.\n","isInternal":false,"tokens":739,"sizeBytes":3039},{"name":"README.md","path":"examples/evals/realtime_evals/README.md","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/README.md","title":"Realtime_evals Skill","category":"anthropic-skill","format":"markdown","content":"# Realtime Evals\n\nThis folder contains three evaluation harnesses for the Realtime API that increase in complexity:\n**crawl → walk → run**.\n\n- **crawl**: synthetic single-turn (TTS → stream → grade)\n- **walk**: deterministic replay of saved phone-audio (more realistic, still comparable)\n- **run**: model-simulated multi-turn conversations (finds multi-turn/tooling failures)\n\nDepending on your realtime eval maturity, point Codex (or your preferred coding assistant) to the folder that fits your use case so it can adapt the harness and bootstrap sample data; you can also have it run the included tests to confirm everything works.\n\n## Quickstart\n\nPython 3.12+ required.\n\n```bash\nmake install\nsource .venv/bin/activate\nexport OPENAI_API_KEY=\"your_api_key\"\n```\n\n`make install` creates the local `.venv` and installs both runtime and dev dependencies. It uses `uv` when available and otherwise falls back to `python -m venv` plus `pip install -r requirements.txt -r requirements-dev.txt`.\n\nRun a first command per harness. If uv is not installed, replace `uv run` with `python` and run these scripts with your `.venv` activated:\n\n- Crawl: `uv run python crawl_harness/run_realtime_evals.py`\n- Walk: install ffmpeg (`brew install ffmpeg`), then:\n  - `uv run python walk_harness/generate_audio.py`\n  - `uv run python walk_harness/run_realtime_evals.py`\n- Run: `uv run python run_harness/run_realtime_evals.py --max-examples 1`\n\n## Dev commands\nUse the root `Makefile` for common checks. Run `make install` first to create `.venv`. These targets work with or without `uv`: when `uv` is installed they run through `uv run`, and otherwise they use the matching tool binaries from the local `.venv`.\n\n- `make install`\n- `make streamlit`\n- `make format`\n- `make lint`\n- `make lint-fix`\n- `make typecheck`\n- `make test`\n\n## [Crawl (synthetic single-turn)](./crawl_harness)\n\n**Best for:** Fast iteration and controlled comparisons.\n\n- Uses text prompts from a CSV and synthesizes TTS audio per row.\n- Streams fixed-size audio chunks into a realtime session.\n- Captures the first assistant turn and grades tool-call correctness.\n- Writes results, summary metrics, audio, and event logs per datapoint.\n\nRun:\n\n```\nuv run python crawl_harness/run_realtime_evals.py\n```\n\n## [Walk (saved audio replay)](./walk_harness)\n\n**Best for:** Realism with reproducibility.\n\n- Replays saved, phone-compressed audio (G.711 mu-law WAV at 8 kHz).\n- Keeps chunking and turn boundaries fixed (VAD off).\n- Captures responses, tool calls, latencies, and event logs per datapoint.\n- Output audio is saved as PCM16 WAV for easy listening.\n\nRun:\n\n```\nuv run python walk_harness/generate_audio.py\nuv run python walk_harness/run_realtime_evals.py\n```\n\n## [Run (multi-turn simulation)](./run_harness)\n\n**Best for:** Discovering multi-turn failures and tool behavior.\n\n- Uses a simulator model to generate user turns and an assistant model under test.\n- Streams simulator audio into the assistant with fixed chunking.\n- Mocks tool outputs deterministically and grades with LLM-as-judge.\n- Writes results, summaries, transcripts, and event logs per simulation.\n\nRun:\n\n```\nuv run python run_harness/run_realtime_evals.py\n```\n\n## Results layout\n\nEach harness writes into its own `results/` folder:\n\n- `results/<run_id>/results.csv`: per-example outputs and grades\n- `results/<run_id>/summary.json`: aggregate metrics\n- `results/<run_id>/plots/*.png`: warm-editorial summary charts for scores, latency, tokens, and run shape\n- `results/<run_id>/events/*.jsonl`: full realtime event stream per datapoint\n\nThe shared typed representation of these artifacts for the crawl, walk, and run harnesses lives in `shared/result_types.py`.\n\nTo render charts for an existing run after the fact:\n\n```bash\nuv run python plot_eval_results.py --run-dir run_harness/results/<run_id>\n```\n\n## Results Viewer\n\nUse the Streamlit results viewer to browse saved runs from `crawl_harness`, `walk_harness`, and `run_harness` without opening the raw artifacts by hand.\n\n- `Comparison View`: select a harness, choose one or more saved runs, and compare summary metrics, scores, latency, and token usage across runs.\n- `Run Viewer`: inspect one saved run in detail. Crawl and walk runs show row-level audio artifacts and event logs; run-harness runs use a Simulation Viewer with transcripts, event logs, and turn audio.\n\nRun it from this directory with either:\n\n```bash\nmake streamlit\n```\n\nor:\n\n```bash\ncd results_viewer\nuv run streamlit run app.py\n```\n\nThen open the local Streamlit URL, usually `http://localhost:8501`.\n\n## Common CLI flags\n\nAll harnesses share a core set of flags so you can switch between them easily:\n\n- `--data-csv`, `--results-dir`, `--run-name`\n- `--model` (assistant under test), `--system-prompt-file`, `--tools-file`\n- `--chunk-ms`, `--sample-rate-hz`, `--real-time`\n- `--max-examples` (quick smoke tests)\n- `--skip-plots` (skip post-run PNG chart generation)\n\nRun harness adds multi-turn and simulator-specific flags (see its README).\n","isInternal":false,"tokens":1207,"sizeBytes":5001},{"name":"README.md","path":"examples/evals/realtime_evals/results_viewer/README.md","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/results_viewer/README.md","title":"Results_viewer Skill","category":"anthropic-skill","format":"markdown","content":"# Results Viewer\n\nThis directory contains a [Streamlit](https://streamlit.io/) app for exploring\nsaved realtime eval runs from the crawl, walk, and run harnesses, plus\nbootstrap-generated eval folders created under `examples/evals/realtime_evals/`.\n\nThe app auto-discovers run directories under:\n\n- `crawl_harness/results/`\n- `walk_harness/results/`\n- `run_harness/results/`\n- `*_realtime_eval/results/` when the folder includes `bootstrap_manifest.json`\n\n## What It Shows\n\n- **Comparison View**: compare summary metrics across one or more saved runs\n- **Run Viewer**: inspect a single saved run, including:\n  - `results.csv` rows\n  - crawl/walk input and output audio artifacts\n  - crawl/walk per-example event logs\n  - run-harness simulation transcripts, event logs, and turn audio\n\nNote: `run_harness` runs are inspected via the app's **Simulation Viewer**,\nwhich has a different UI and a different artifact set than the crawl/walk\nviewer.\n\n## Run Locally\n\nFrom `examples/evals/realtime_evals/`:\n\n```bash\nuv venv .venv\nsource .venv/bin/activate\nuv sync --group dev\ncd results_viewer\nuv run streamlit run app.py\n```\n\nThen open the local URL that Streamlit prints, usually\n`http://localhost:8501`.\n\nIf you are using the pip-based install path instead of `uv`, install the dev\ndependencies first so `streamlit` is available:\n\n```bash\npip install -r requirements.txt -r requirements-dev.txt\ncd results_viewer\nstreamlit run app.py\n```\n\n## Expected Data Layout\n\nThe viewer expects each saved run directory to contain:\n\n- `summary.json` for aggregate metrics\n- `results.csv` for per-example results\n\nFor crawl and walk runs, the app can also display:\n\n- `audio/<example_id>/input.wav`\n- `audio/<example_id>/output.wav`\n- `events/<example_id>.jsonl`\n\nThe app discovers runs recursively, so nested result directories are fine as\nlong as those files are present.\n","isInternal":false,"tokens":451,"sizeBytes":1856},{"name":"sim_cs_address_update.json","path":"examples/evals/realtime_evals/run_harness/data/sim_cs_address_update.json","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/run_harness/data/sim_cs_address_update.json","title":"Data Skill","category":"anthropic-skill","format":"json","content":"# Data Skill\n\nPath: `examples/evals/realtime_evals/run_harness/data/sim_cs_address_update.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/run_harness/data/sim_cs_address_update.json)","tokens":300,"sizeBytes":262},{"name":"sim_cs_order_status.json","path":"examples/evals/realtime_evals/run_harness/data/sim_cs_order_status.json","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/run_harness/data/sim_cs_order_status.json","title":"Data Skill","category":"anthropic-skill","format":"json","content":"# Data Skill\n\nPath: `examples/evals/realtime_evals/run_harness/data/sim_cs_order_status.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/run_harness/data/sim_cs_order_status.json)","tokens":300,"sizeBytes":258},{"name":"sim_cs_refund.json","path":"examples/evals/realtime_evals/run_harness/data/sim_cs_refund.json","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/run_harness/data/sim_cs_refund.json","title":"Data Skill","category":"anthropic-skill","format":"json","content":"# Data Skill\n\nPath: `examples/evals/realtime_evals/run_harness/data/sim_cs_refund.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/run_harness/data/sim_cs_refund.json)","tokens":300,"sizeBytes":246},{"name":"README.md","path":"examples/evals/realtime_evals/run_harness/README.md","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/run_harness/README.md","title":"Run_harness Skill","category":"anthropic-skill","format":"markdown","content":"# Run Harness\n\nModel-simulated multi-turn eval harness for the Realtime API. This harness uses\none realtime model as the user simulator (audio-only) and another realtime model\nas the assistant under test. It replays fixed chunking (VAD off), mocks tools\nfrom the simulation JSON, and records full traces.\n\n## What it does\n\n- Loads `run_harness/data/simulations.csv` with pandas.\n- Reads per-simulation JSON files defining scenario, simulator identity, tool mocks,\n  and LLM-as-judge grading criteria.\n- Generates user audio turns via a realtime simulator model (audio-only).\n- Streams user audio to the assistant in fixed-size chunks and commits manually.\n- Captures assistant audio/text, tool calls, tool outputs, and latencies.\n- Grades turn-level and trace-level criteria with an LLM-as-judge.\n- Writes `results.csv`, `summary.json`, and full trace logs under `run_harness/results/`.\n- Renders styled PNG plots under `run_harness/results/<run_id>/plots/` by default.\n\n## Files\n\n- `run_harness/run_realtime_evals.py`: Run harness script.\n- `run_harness/data/simulations.csv`: Index of simulation files.\n- `run_harness/data/sim_*.json`: Simulation definitions (this repo currently ships 3 examples).\n- `run_harness/results/<run_id>/events/*.jsonl`: Full event trace per simulation.\n- `run_harness/results/<run_id>/conversations/*.txt`: Human-readable transcript with tool calls.\n\n## Simulation definitions (high level)\n\nEach `sim_*.json` file defines:\n\n- **the scenario** (what the user is trying to do)\n- **tool mocks** (what each tool should return when called)\n- **judge rubric** (what should be graded per turn / overall)\n\nThis lets you run repeatable multi-turn evals without needing live backend integrations.\n\n## How to run\n\nFrom repo root:\n\n```bash\npython run_harness/run_realtime_evals.py --max-examples 1\n```\n\nCommon options:\n\n- `--data-csv`: Simulation index CSV.\n- `--model`: Alias for the assistant model under test.\n- `--assistant-model`: Realtime model under test (overrides `--model`).\n- `--simulator-model`: Realtime model used as the user.\n- `--system-prompt-file`: Alias for assistant system prompt file.\n- `--tools-file`: Alias for assistant tools file.\n- `--assistant-system-prompt-file`: Override assistant system prompt file (more specific).\n- `--assistant-tools-file`: Override tools file for the assistant (more specific).\n- `--simulator-system-prompt`: Override simulator system prompt (string).\n- `--chunk-ms`, `--sample-rate-hz`, `--real-time`.\n- `--max-turns`, `--max-examples`.\n- `--skip-plots`.\n\nNotes:\n\n- Temperature and max output tokens are intentionally not supported.\n- The simulator emits audio only. Transcripts are captured from audio deltas if available.\n- Tool outputs are mocked deterministically from each simulation JSON.\n- `results.csv` includes an `event_log_path` column pointing to the JSONL trace.\n","isInternal":false,"tokens":667,"sizeBytes":2848},{"name":"tools.json","path":"examples/evals/realtime_evals/shared/tools.json","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/shared/tools.json","title":"Shared Skill","category":"anthropic-skill","format":"json","content":"# Shared Skill\n\nPath: `examples/evals/realtime_evals/shared/tools.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/shared/tools.json)","tokens":300,"sizeBytes":212},{"name":"openai.yaml","path":"examples/evals/realtime_evals/skills/bootstrap-realtime-eval/agents/openai.yaml","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/skills/bootstrap-realtime-eval/agents/openai.yaml","title":"Agents Skill","category":"anthropic-skill","format":"yaml","content":"# Agents Skill\n\nPath: `examples/evals/realtime_evals/skills/bootstrap-realtime-eval/agents/openai.yaml`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/skills/bootstrap-realtime-eval/agents/openai.yaml)","tokens":300,"sizeBytes":276},{"name":"harness-selection.md","path":"examples/evals/realtime_evals/skills/bootstrap-realtime-eval/references/harness-selection.md","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/skills/bootstrap-realtime-eval/references/harness-selection.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `examples/evals/realtime_evals/skills/bootstrap-realtime-eval/references/harness-selection.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/skills/bootstrap-realtime-eval/references/harness-selection.md)","tokens":300,"sizeBytes":306},{"name":"SKILL.md","path":"examples/evals/realtime_evals/skills/bootstrap-realtime-eval/SKILL.md","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/skills/bootstrap-realtime-eval/SKILL.md","title":"bootstrap-realtime-eval","category":"anthropic-skill","format":"markdown","content":"---\nname: bootstrap-realtime-eval\ndescription: Bootstrap a new realtime eval folder inside this cookbook repo by choosing the right harness from examples/evals/realtime_evals, scaffolding prompt/tools/data files, generating a useful README, and validating it with smoke, full eval, and test runs. Use when a user wants to start a new crawl, walk, or run realtime eval in this repository.\n---\n\n# Bootstrap Realtime Eval\n\nUse this skill when the user wants a new realtime eval scaffold under `examples/evals/realtime_evals/`.\n\nThis skill is repo-specific. Do not copy harness code into the generated folder. The generated eval should point at the shared harnesses already in:\n\n- `examples/evals/realtime_evals/crawl_harness`\n- `examples/evals/realtime_evals/walk_harness`\n- `examples/evals/realtime_evals/run_harness`\n\n## Inputs To Collect\n\nAlways ask the user for the minimum set needed to choose and scaffold the eval before you create files, run the scaffold script, or author starter data. Do not skip this just because you can infer a default.\n\nAsk for:\n\n- Eval name\n- Goal or scenario\n- Harness choice, or enough context to recommend one\n- System prompt path or inline text\n- Tools JSON path or tool descriptions\n- Data path or source materials\n- Desired graders\n\nIf the user does not know which harness they want, explain the options briefly and recommend one. See `references/harness-selection.md`.\n\nWhen the user asks for synthetic audio but does not specify a harness, default to `crawl` text-to-TTS unless they need the generated audio to carry particular noise, telephony artifacts, speaker characteristics, or other replay-specific properties. Use `walk` for those cases.\n\nKeep the questions concise and grouped into one short batch whenever possible.\n\nIf the user only provides `user_text` or a short task description, still ask the questions above first. If they answer only partially, then infer the remaining low-risk details, call out the assumptions, and make the scaffold easy to revise later.\n\n## Workflow\n\n1. Ask the user for the required inputs first.\n   - Do this before making files or selecting a final harness.\n   - If the user already supplied some of the inputs, ask only for the missing ones.\n   - If you recommend a harness, wait for the user response before scaffolding.\n\n2. Pick the harness.\n   - `crawl`: single-turn text-to-TTS.\n   - `walk`: replay saved audio or generate audio from text rows.\n   - `run`: multi-turn simulation with tool mocks and judge criteria.\n   - If the user wants synthetic audio but does not care about replay-specific audio characteristics, prefer `crawl` over `walk`.\n\n3. Normalize the inputs.\n   - If the user gives inline prompt or tool content, write it into the generated folder.\n   - If the user gives CSV data, inspect it with pandas before wiring it in.\n   - If the data is not yet harness-ready, scaffold the files and then have you author the starter dataset directly in those files.\n   - If the user only gives `user_text`, infer `example_id` values and leave optional grading fields blank unless you have enough signal to fill them.\n   - Ground the starter data in the use case, prompts, tools, and any other material the user provided.\n   - Make the starter data realistic. Put yourself in the shoes of the end user in that use case and craft datapoints that a real user would plausibly say or do.\n\n4. Be proactive when data is missing.\n   - `crawl`: you should author 3 starter rows covering one happy path and a couple of nearby variants or edge cases.\n   - `walk`: you should author 3 source CSV rows and prepare the audio-generation step so the user can create audio immediately.\n   - `run`: you should author 2 starter simulations, not 1.\n   - Do not rely on deterministic script-generated samples for use-case-specific starter data.\n   - Show the generated starter samples to the user, ask for a quick greenlight or correction, then expand only after feedback when the task calls for more coverage.\n\n5. Run the scaffold script:\n\n```bash\npython examples/evals/realtime_evals/skills/bootstrap-realtime-eval/scripts/bootstrap_realtime_eval.py --name \"<eval_name>\" --harness \"<crawl|walk|run>\"\n```\n\nAdd flags for prompt, tools, data, graders, and run-specific fields as needed. Read the script help if you need the exact flag names.\n\n6. Review the generated folder.\n   - Confirm the README is accurate for the selected harness.\n   - Confirm the system prompt, tools, and data files point at the generated folder.\n   - If the user provided real data, make sure it is wired in instead of leaving starter placeholders.\n   - If you authored inferred starter data, show the user the generated rows or simulations before scaling up.\n\n7. Enrich the scaffold.\n   - For `crawl` and `walk`, make the CSV realistic and ensure the expected tool columns are present.\n   - For `walk`, if the dataset lacks `audio_path`, use the shared `walk_harness/generate_audio.py` flow described in the README.\n   - For `run`, make sure `simulations.csv` and the starter `sim_*.json` file reflect the user’s scenario, tool mocks, and graders.\n\n8. Validate before returning.\n   - Run a smoke eval for the generated folder.\n   - If the smoke eval succeeds, automatically run the full eval for the generated folder in the same turn.\n   - If the smoke eval fails, stop and fix or report the blocker before attempting the full eval.\n   - Run `pytest examples/evals/realtime_evals/tests -q`.\n   - Inspect the generated README and at least one generated data file.\n\n## Hard Rules\n\n- Keep the generated folder inside `examples/evals/realtime_evals/`.\n- Ask the user for the missing setup information before scaffolding. Do not jump straight to building a default eval when the request is to create a new eval.\n- Do not duplicate the main harness scripts into the generated folder.\n- Do not write generic placeholder starter data when the use case provides enough context to do better.\n- Use the prompt, tools, and scenario details to make the initial datapoints feel like a real user interaction.\n- The README must include:\n  - why this harness was chosen\n  - which files to edit first\n  - smoke and full run commands\n  - data contract notes\n  - troubleshooting notes\n- Prefer assistant-specific flags for run harness commands:\n  - `--assistant-system-prompt-file`\n  - `--assistant-tools-file`\n\n## Completion Criteria\n\nTreat the task as complete only when:\n\n- the folder exists under `examples/evals/realtime_evals/<name>_realtime_eval/`\n- `README.md`, `system_prompt.txt`, and `tools.json` exist\n- harness-specific starter data exists, authored by you when the user did not provide it\n- the smoke command has been run or is blocked for a clear reason\n- if the smoke command succeeded, the full eval command has also been run or is blocked for a clear reason\n- `pytest examples/evals/realtime_evals/tests -q` has been run\n\n## Learnings\n\nWhen this skill uncovers a reusable workflow or harness constraint that should guide future bootstrap work, add a short note here.\n\nKeep learnings concise and action-oriented:\n\n- **Problem -> Fix -> Why**\n\nOnly add items that are likely to help future realtime-eval scaffolding in this repo. Remove stale items when they no longer apply.\n\n- **`gpt-realtime` temperature is unsupported** -> Do not add a `temperature` field or CLI flag when scaffolding `gpt-realtime` evals -> Avoids invalid config and keeps runs aligned with the realtime harness constraints.\n- **Starter samples should come from the model, not the scaffold script** -> Use the script to create the folder structure and template files, then have you author the initial rows or simulations from the user’s use case -> Produces more relevant starter data and makes iteration easier with the user.\n- **Synthetic audio can be overfit to the wrong harness** -> Default unspecified synthetic-audio requests to `crawl` text-to-TTS and reserve `walk` for replay-specific audio characteristics like noise or telephony artifacts -> Keeps the bootstrap path simpler unless audio realism is the actual target.\n- **Smoke-only validation can leave scaffolds half-proven** -> After a successful smoke run, automatically run the full eval before declaring the bootstrap complete -> Catches dataset-wide or late-row failures that a one-example smoke test misses.\n","frontmatter":{"name":"bootstrap-realtime-eval","description":"Bootstrap a new realtime eval folder inside this cookbook repo by choosing the right harness from examples/evals/realtime_evals, scaffolding prompt/tools/data files, generating a useful README, and validating it with smoke, full eval, and test runs. Use when a user wants to start a new crawl, walk, or run realtime eval in this repository."},"isInternal":false,"tokens":1807,"sizeBytes":8293},{"name":"README.md","path":"examples/evals/realtime_evals/walk_harness/README.md","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/walk_harness/README.md","title":"Walk_harness Skill","category":"anthropic-skill","format":"markdown","content":"# Walk_harness Skill\n\nPath: `examples/evals/realtime_evals/walk_harness/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/walk_harness/README.md)","tokens":300,"sizeBytes":228},{"name":".mcp.json","path":"examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.mcp.json","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.mcp.json","title":"Bls Skill","category":"anthropic-skill","format":"json","content":"# Bls Skill\n\nPath: `examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.mcp.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.mcp.json)","tokens":300,"sizeBytes":331},{"name":"marketplace.json","path":"examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/.agents/plugins/marketplace.json","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/.agents/plugins/marketplace.json","title":"Claude Plugin Marketplace Catalog","category":"marketplace","format":"json","content":"# Claude Plugin Marketplace Catalog\n\nPath: `examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/.agents/plugins/marketplace.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/.agents/plugins/marketplace.json)","tokens":300,"sizeBytes":377},{"name":"plugin.json","path":"examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.codex-plugin/plugin.json","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.codex-plugin/plugin.json","title":"codex-plugin Plugin Manifest","category":"plugin-manifest","format":"json","content":"# codex-plugin Plugin Manifest\n\nPath: `examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.codex-plugin/plugin.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.codex-plugin/plugin.json)","tokens":300,"sizeBytes":382},{"name":"README.md","path":"examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/README.md","rawUrl":"https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/README.md","title":"bls Documentation","category":"plugin-manifest","format":"markdown","content":"# bls Documentation\n\nPath: `examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/README.md)","tokens":300,"sizeBytes":339}],"systemPromptSnippet":"<yakaai_skills repo=\"openai/openai-cookbook\">\n<!-- File: AGENTS.md (Tokens: ~1170 | Category: root-instruction) -->\n# Repository Guidelines\n\n## Project Structure & Module Organization\n\nThe cookbook is organized around runnable examples and reference articles for OpenAI APIs. Place notebooks and Python scripts under `examples/<topic>/`, grouping related assets inside topic subfolders (for example, `examples/agents_sdk/`). Narrative guides and long-form docs live in `articles/`, and shared diagrams or screenshots belong in `images/`. Update `registry.yaml` whenever you add content so it appears on cookbook.openai.com, and add new author metadata in `authors.yaml` if you want custom attribution. Keep large datasets outside the repo; instead, document how to fetch them in the notebook.\n\n## Build, Test, and Development Commands\n\nUse a virtual environment to isolate dependencies:\n\n- `python -m venv .venv && source .venv/bin/activate`\n- `pip install -r examples/<topic>/requirements.txt` (each sample lists only what it needs)\n- `jupyter lab` or `jupyter notebook` to develop interactively\n- `python .github/scripts/check_notebooks.py` to validate notebook structure before pushing\n\n## Coding Style & Naming Conventions\n\nWrite Python to PEP 8 with four-space indentation, descriptive variable names, and concise docstrings that explain API usage choices. Name new notebooks with lowercase, dash-or-underscore-separated phrases that match their directory—for example `examples/gpt-5/prompt-optimization-cookbook.ipynb`. Keep markdown cells focused and prefer numbered steps for multi-part workflows. Store secrets in environment variables such as `OPENAI_API_KEY`; never hard-code keys inside notebooks.\n\n## Testing Guidelines\n\nExecute notebooks top-to-bottom after installing dependencies and clear lingering execution counts before committing. For Python modules or utilities, include self-check cells or lightweight `pytest` snippets and show how to run them (for example, `pytest examples/object_oriented_agentic_approach/tests`). When contributions depend on external services, mock responses or gate the cells behind clearly labeled opt-in flags.\n\n## Commit & Pull Request Guidelines\n\nUse concise, imperative commit messages that describe the change scope (e.g., \"Add agent portfolio collaboration demo\"). Every PR should provide a summary, motivation, and self-review, and must tick the registry and authors checklist from `.github/pull_request_template.md`. Link issues when applicable and attach screenshots or output snippets for UI-heavy content. Confirm CI notebook validation passes locally before requesting review.\n\nBefore a pull request is merged, run the repository's `docs-editor` skill on every changed article, Markdown or MDX file, and notebook Markdown cell. For editorial notebook changes, preserve code cells, outputs, execution counts, attachments, cell order, and metadata. Fix safe issues, rerun the relevant checks, and resolve or explicitly document all remaining P0 and P1 findings in the pull request.\n\n## Metadata & Publication Workflow\n\nNew or relocated content must have an entry in `registry.yaml` with an accurate path, date, and tag set so the static site generator includes it. When collaborating, coordinate author slugs in `authors.yaml` to avoid duplicates, and run `python -m yaml lint registry.yaml` (or your preferred YAML linter) to catch syntax errors before submitting.\n\n## Review Guidelines\n\nThese are considered priority 0 issues for this repo, in addition to the normal priority for possible issues.\n- Verify file, function, and notebook names follow the repo's naming conventions and clearly describe their purpose.\n- Scan prose and markdown for typos, broken links, and inconsistent formatting before approving.\n- Check that code identifiers remain descriptive (no leftover placeholder names) and that repeated values are factored into constants when practical.\n- Ensure notebooks or scripts document any required environment variables instead of hard-coding secrets or keys.\n- Confirm metadata files (`registry.yaml`, `authors.yaml`) stay in sync with new or relocated content.\n\n## Recent Learnings\n\n- **`uv run` can inherit the wrong virtualenv in this repo** -> Clear `VIRTUAL_ENV` (for example `env -u VIRTUAL_ENV uv run ...`) when an unrelated environment is active -> Avoids misleading mismatch warnings and makes it clear the repo's `.venv` is the interpreter actually running the harnesses.\n- **Realtime eval shared imports can resolve the wrong module under pytest** -> Add `shared/__init__.py` and ensure tests prepend `examples/evals/realtime_evals` to `sys.path` before importing `shared.*` -> Prevents collection failures caused by unrelated installed packages named `shared`.\n- **Run-level grades can be overweighted by long simulations** -> Store turn-level grades on the matching turn and trace-level grades on one row per simulation instead of copying them onto every row -> Keeps `results.csv` row semantics intact and prevents summary means from favoring longer conversations.\n- **Synthetic-audio scaffold requests can pick the wrong harness** -> Default unspecified synthetic-audio evals to `crawl` text-to-TTS and reserve `walk` for replay-specific audio traits like noise, telephony artifacts, or speaker characteristics -> Keeps new realtime evals on the simplest harness unless audio realism is itself under test.\n- **Task-specific single-turn grading can outgrow the shared crawl schema** -> Keep the shared crawl harness for realtime execution, then add eval-local wrapper scripts that post-grade domain-specific quality and overwrite `results.csv` while preserving `results_base.csv` -> Avoids forking the harness when a use case needs richer grading than tool-call correctness.\n- **Synthetic learner audio can sound like eval scaffolding** -> Write `user_text` as a realistic in-app learner request and keep evaluation rules in metadata plus the system prompt -> Produces audio inputs that match the product surface instead of teaching the model the grading rubric through the spoken prompt.\n\n<!-- File: .codex/skills/docs-editor/resources/style-guide.md (Tokens: ~865 | Category: anthropic-skill) -->\n# OpenAI Cookbook editorial checklist\n\nUse this checklist for articles and notebook Markdown cells in `openai-cookbook`. Apply only the rules relevant to the content under review.\n\n## Audience and purpose\n\n- Write for developers who want to understand and run an OpenAI API example.\n- State the task or outcome early. Keep background material only when it helps the reader use the example correctly.\n- Assume general programming knowledge, but explain Cookbook-specific setup, choices, and tradeoffs.\n- Avoid promotional language, vague praise, and claims that the example does not demonstrate.\n\n## Technical accuracy\n\n- Match API, model, method, parameter, field, file, and environment-variable names exactly.\n- Keep prose consistent with the code, sample inputs, outputs, diagrams, and nearby cells.\n- Check counts, ordered steps, cross-references, defaults, constraints, and stated results.\n- Do not infer a correction when the intended fact is unclear. Preserve the source and flag it for review.\n- Use authoritative sources to verify uncertain product behavior.\n- Do not imply that an optional step is required or that a required step is optional.\n\n## Runnable examples\n\n- List prerequisites and dependencies that readers need before running the example.\n- Document required environment variables. Never hard-code API keys, tokens, credentials, or other secrets.\n- Keep setup instructions in the same order that readers perform them.\n- Introduce code with the goal or reason for the step; do not narrate every line.\n- Use realistic inputs and clearly label placeholders.\n- Show expected output only when it helps readers verify success, and keep it consistent with the code.\n- Keep external-service calls and other opt-in behavior clearly labeled.\n\n## Notebook integrity\n\n- Edit Markdown cells only during an editorial pass.\n- Preserve code cells, outputs, execution counts, attachments, cell IDs, cell order, and metadata.\n- Keep notebook JSON valid and avoid whole-file reserialization for a small prose edit.\n- Keep each Markdown cell focused. Split or merge cells only when the structure is genuinely confusing and the request permits it.\n- Use Markdown headings in a logical hierarchy without skipping levels unnecessarily.\n\n## Prose\n\n- Lead with the point. Prefer concrete nouns and strong verbs.\n- Use direct instructions and address the reader as “you” when helpful.\n- Prefer active voice, but keep passive voice when the actor is unknown or unimportant.\n- Remove throat-clearing, repeated conclusions, filler, and ornamental transitions.\n- Keep paragraphs short enough to scan and use lists for genuinely parallel items.\n- Use numbered lists for ordered procedures and bullets for unordered groups.\n- Keep sibling headings and list items parallel.\n- Use sentence-style capitalization for headings and do not end headings with periods.\n- Preserve useful warmth and explanation; do not flatten the content into terse fragments.\n\n## Terminology and formatting\n\n- Use exact product terminology and consistent capitalization.\n- Format commands, options, environment variables, paths, filenames, code identifiers, and literal values with backticks.\n- Match the capitalization and spelling used by the language or API.\n- Expand uncommon abbreviations on first use unless the surrounding audience clearly knows them.\n- Use inclusive, precise language and neutral role terms.\n- Do not replace a technically precise term merely to vary wording.\n\n## Links and assets\n\n- Prefer descriptive link text over “here” or a pasted URL.\n- Verify local links, anchors, image paths, and referenced notebook or article paths.\n- Use stable, authoritative external links when available.\n- Give images meaningful alt text unless they are purely decorative.\n- Keep diagrams, screenshots, captions, and prose consistent.\n\n## Publication metadata\n\n- Add or update `registry.yaml` for new, moved, renamed, or removed published content.\n- Confirm registry paths, dates, tags, and titles match the content.\n- Add or update `authors.yaml` when introducing or changing author attribution.\n- Check that filenames follow the repository's naming conventions and clearly describe the content.\n\n## Final review\n\n- Read each edited passage in context, not only as an isolated diff hunk.\n- Confirm no factual meaning, literal value, condition, negation, or uncertainty changed unintentionally.\n- Confirm all repository-defined P0 issues are fixed.\n- Run the notebook structural validator for changed notebooks and inspect the final diff for unintended changes.\n\n<!-- File: .codex/skills/docs-editor/SKILL.md (Tokens: ~876 | Category: anthropic-skill) -->\n---\nname: docs-editor\ndescription: Review and edit OpenAI Cookbook notebooks, Markdown, and MDX for technical accuracy, clarity, grammar, consistency, runnable examples, and repository publication requirements. Use for editorial reviews, pre-merge documentation checks, or notebook Markdown-cell sweeps in openai-cookbook.\n---\n\n# OpenAI Cookbook docs editor\n\n## Objective\n\nReview Cookbook content conservatively. Improve correctness and readability without changing the author's intent, inventing facts, or creating unrelated churn.\n\n## Workflow\n\n### 1. Define the scope\n\n- Prefer explicit paths or the files changed in the pull request.\n- For pull requests, review changed `.md` and `.mdx` files plus Markdown cells in changed `.ipynb` files.\n- Review `registry.yaml` and `authors.yaml` when content is added, moved, removed, or attributed to a new author.\n- Stay within the requested content. Do not turn a focused review into a repository-wide rewrite.\n\n### 2. Protect notebook integrity\n\n- When the request is editorial, edit only notebook Markdown cells.\n- Preserve code cells, outputs, execution counts, attachments, cell order, cell IDs, and notebook and cell metadata.\n- Avoid reserializing an entire notebook for a small prose change. Inspect the diff for unintended JSON churn.\n- Do not execute notebook code unless the user requests it or code changes are in scope. Markdown-only edits require structural validation, not calls to external services.\n- Never add hard-coded secrets. Document required environment variables such as `OPENAI_API_KEY`.\n\n### 3. Review the content\n\nLoad `resources/style-guide.md` and check:\n\n- Technical claims, API names, model names, identifiers, commands, paths, and URLs.\n- Agreement between prose, code, sample inputs, and expected outputs.\n- Prerequisites, dependencies, environment variables, and steps needed to run an example.\n- File and notebook names, headings, lists, numbering, cross-references, and links.\n- Grammar, spelling, punctuation, terminology, capitalization, and concise wording.\n- Publication metadata in `registry.yaml` and author metadata in `authors.yaml` when applicable.\n\nVerify uncertain technical claims from an authoritative source or flag them for review. Do not guess.\n\n### 4. Prioritize findings\n\nUse the repository's review bar:\n\n- **P0**: Broken notebook JSON; hard-coded secrets; materially false or unsafe guidance; or any repository-defined P0 issue, including naming violations, typos, broken links, inconsistent formatting, placeholder identifiers, undocumented environment variables, or out-of-sync publication metadata.\n- **P1**: Misleading instructions, prose that contradicts code or output, missing prerequisites, or a required link that does not work.\n- **P2**: Clear improvements to grammar, consistency, organization, or readability that do not affect correctness.\n- **P3**: Optional polish with little effect on comprehension.\n\n### 5. Make safe edits\n\n- Fix obvious typos, grammar, punctuation, broken local references, and clear prose/code mismatches.\n- Keep changes minimal, local, and reversible.\n- Preserve exact literals, code identifiers, commands, values, counts, conditions, modality, negation, and uncertainty markers.\n- Tighten prose only when meaning stays unchanged. Prefer concrete nouns, strong verbs, short introductions, and direct instructions.\n- Flag any change that depends on product knowledge or could alter meaning instead of guessing.\n- Do not make file-wide formatting or stylistic changes solely for uniformity.\n\n### 6. Validate the result\n\nFor any changed notebook, run the repository validator:\n\n```bash\nenv -u VIRTUAL_ENV uv run --with nbformat python .github/scripts/check_notebooks.py\n```\n\nThen:\n\n- Run `git diff --check`.\n- Inspect the diff and confirm that notebook changes are limited to intended Markdown cells.\n- Run an available YAML linter if `registry.yaml` or `authors.yaml` changed.\n- Run the relevant example or test only when executable content changed and its dependencies are available.\n\n### 7. Report the review\n\nProvide:\n\n- A short summary of the scope and edits.\n- Auto-fixed items.\n- Remaining findings grouped by priority with file and cell or line references.\n- The validation commands run and their results.\n\nIf no issues remain, say so explicitly.\n\n<!-- File: examples/agents_api/apps/document_review/skills/expense-review-policy/SKILL.md (Tokens: ~570 | Category: anthropic-skill) -->\n---\nname: expense-review-policy\ndescription: Review invoices and contracts against accounts-payable policy before human approval.\n---\n\n# Expense review policy\n\nPolicy ID: `AP-104`.\n\nApply this policy to invoices, expense receipts, and service contracts.\n\n## Invoices and expenses\n\n- Read the entire invoice, including the vendor and every line item.\n- Extract every line item's quantity and unit price before calculating. Check the extracted list against every source line; an empty list is an error, not a zero subtotal.\n- Use Python's `decimal.Decimal` to multiply quantities by unit prices and sum the line totals. Compare this subtotal with the printed subtotal, then add shipping to calculate the total.\n- Strip currency separators, compare the result with `Total amount due`, and report the exact difference (stated total minus calculated total).\n- Require a purchase order or another documented approval.\n- Flag missing receipts, unsupported charges, and changed payment instructions.\n- Escalate incorrect totals and unverified changes to bank details.\n\n## Contracts\n\n- Read every clause and report all matching risks, even when one already requires escalation.\n- Flag automatic renewals and restrictive cancellation windows.\n- Flag unilateral price increases and missing liability limits.\n- Escalate customer-data sharing or subcontractor access without approval.\n- Identify missing confidentiality, security, or termination terms.\n\n## Review decision\n\n- `needs_info`: Required documentation or supporting information is missing.\n- `escalated`: Fraud indicators, financial discrepancies, or risky terms need review.\n- `ready_for_approval`: No policy violations remain, but a human must still approve.\n\nWrite `/workspace/output/<document-stem>.json` with the following fields:\n\n```json\n{\n  \"document\": \"invoice.txt\",\n  \"document_type\": \"invoice\",\n  \"policy_id\": \"AP-104\",\n  \"decision\": \"escalated\",\n  \"vendor\": \"Cedar Office Supply\",\n  \"amount\": 6420,\n  \"issues\": [\"The claimed total exceeds the calculated total.\"],\n  \"recommendation\": \"Escalate for human review.\",\n  \"calculation\": {\n    \"line_items\": [{\"quantity\": 2, \"unit_price\": 100}],\n    \"shipping\": 20,\n    \"calculated_total\": 220,\n    \"difference\": 6200\n  }\n}\n```\n\nEvery `issues` entry must be a plain-English string, not a nested object.\nUse the actual document values, not the illustrative numbers above. Include every\ninvoice line item in `calculation`; `amount` is the stated total. For contracts,\nset `document_type` to `contract` and `calculation` to `null`.\n\nOnly the coordinator writes `summary.json`. A specialist writes its assigned report.\n\nNever approve payments, sign contracts, or take external actions.\n\n<!-- File: examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/definitions.json (Tokens: ~300 | Category: anthropic-skill) -->\n# Tools Skill\n\nPath: `examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/definitions.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/definitions.json)\n\n<!-- File: examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/restaurant_facts.json (Tokens: ~300 | Category: anthropic-skill) -->\n# Tools Skill\n\nPath: `examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/restaurant_facts.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/audio/duplex_voice_agent_evaluation/assistants/client/tools/restaurant_facts.json)\n\n<!-- File: examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/definitions.json (Tokens: ~300 | Category: anthropic-skill) -->\n# Tools Skill\n\nPath: `examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/definitions.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/definitions.json)\n\n<!-- File: examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/restaurant_facts.json (Tokens: ~300 | Category: anthropic-skill) -->\n# Tools Skill\n\nPath: `examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/restaurant_facts.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/audio/duplex_voice_agent_evaluation/assistants/responses/tools/restaurant_facts.json)\n\n<!-- File: examples/evals/imagegen_evals/editing_harness/README.md (Tokens: ~388 | Category: anthropic-skill) -->\n# Editing Harness\n\n**Best for:** high-precision checks where edits must be exact and non-target\ncontent should remain unchanged.\n\nThis harness runs two example cases:\n\n- `vto_jacket_tryon`: virtual try-on using reference person + garment\n- `logo_year_edit`: precision logo text edit\n\n## Run\n\n```bash\npython editing_harness/run_imagegen_evals.py\n```\n\nRun a single case (use `--cases`):\n\n```bash\npython editing_harness/run_imagegen_evals.py --cases vto_jacket_tryon\n```\n\nRun multiple cases (comma-separated, no spaces):\n\n```bash\npython editing_harness/run_imagegen_evals.py --cases vto_jacket_tryon,logo_year_edit\n```\n\n## Flags (when to use them)\n\n- `--cases`: limit runs to specific case ids. Valid values:\n  - `vto_jacket_tryon`\n  - `logo_year_edit`\n- `--model`: image model under test (defaults to `gpt-image-1.5`).\n- `--judge-model`: LLM used to grade outputs (defaults to `gpt-5.2`).\n- `--num-images`: number of edited images to generate per case (defaults to 1).\n\n## Inputs\n\nThis harness expects the following reference images in `images/`:\n\n- `images/base_woman.png`\n- `images/jacket.png`\n- `images/logo_generation_1.png`\n\n## Outputs\n\nResults are written under `editing_harness/results/<run_id>/`:\n\n- `results.json` and `results.csv` with per-example scores\n- `summary.json` with aggregated metrics\n- `artifacts/` for edited images\n\n## Adapting the harness\n\n- Replace the input images and edit prompts in `run_imagegen_evals.py`.\n- Update the schemas and verdict rules to fit your workflow.\n\n<!-- File: examples/evals/imagegen_evals/generation_harness/README.md (Tokens: ~437 | Category: anthropic-skill) -->\n# Generation Harness\n\n**Best for:** fast iteration on prompt-following, layout, and text-rendering\nquality for text-to-image workflows.\n\nThis harness runs two example cases:\n\n- `ui_checkout_mockup`: mobile checkout UI mockup\n- `coffee_flyer_generation`: marketing flyer with strict copy constraints\n\n## Run\n\n```bash\npython generation_harness/run_imagegen_evals.py\n```\n\nQuick smoke test for a single case (use `--cases`):\n\n```bash\npython generation_harness/run_imagegen_evals.py --cases ui_checkout_mockup\n```\n\nRun multiple cases (comma-separated, no spaces):\n\n```bash\npython generation_harness/run_imagegen_evals.py --cases ui_checkout_mockup,coffee_flyer_generation\n```\n\nOptional OCR-style text check for the coffee flyer:\n\n```bash\npython generation_harness/run_imagegen_evals.py --run-ocr\n```\n\n## Flags (when to use them)\n\n- `--cases`: limit runs to specific case ids. Valid values:\n  - `ui_checkout_mockup`\n  - `coffee_flyer_generation`\n- `--model`: image model under test (defaults to `gpt-image-1.5`).\n- `--judge-model`: LLM used to grade outputs (defaults to `gpt-5.2`).\n- `--num-images`: number of images to generate per case (defaults to 1).\n- `--image-size`: size for generated images (defaults to `1024x1024`).\n- `--run-ocr`: only relevant for `coffee_flyer_generation` (checks exact text).\n\n## Outputs\n\nResults are written under `generation_harness/results/<run_id>/`:\n\n- `results.json` and `results.csv` with per-example scores\n- `summary.json` with aggregated metrics\n- `artifacts/` for generated images\n\n## Adapting the harness\n\n- Update the prompts/criteria in `run_imagegen_evals.py`.\n- Tune JSON schemas + parsers to match your evaluation rubric.\n- Add more cases by creating new `TestCase` entries and graders.\n\n<!-- File: examples/evals/imagegen_evals/README.md (Tokens: ~790 | Category: anthropic-skill) -->\n# Image Generation + Editing Evals\n\nThis folder contains a lightweight vision eval harness plus example runners for\nimage generation and image editing. The code mirrors the structure of\n`examples/evals/realtime_evals/` so you can adapt it quickly.\n\n**Directory layout**\n\n- `vision_harness/`: minimal shared library (types, runners, graders, evaluate loop)\n- `generation_harness/`: text-to-image evals (UI mockups + marketing flyer)\n- `editing_harness/`: image-edit evals (virtual try-on + logo edit)\n- `shared/`: reporting and optional rendering helpers\n\n## Quickstart\n\nPython 3.9+ required.\n\n```bash\npip install -r requirements.txt\nexport OPENAI_API_KEY=\"your_api_key\"\n```\n\nRun a harness:\n\n- Generation: `python generation_harness/run_imagegen_evals.py`\n- Editing: `python editing_harness/run_imagegen_evals.py`\n\n## What the harness does\n\n1. Builds a small set of `TestCase` objects (prompt + criteria).\n2. Runs the image model for each case.\n3. Grades each output with an LLM judge using a strict JSON schema.\n4. Writes results and artifacts to `results/<run_id>/`.\n\nThe harness is intentionally small so you can copy/paste parts into your own\nproduction eval setup.\n\n## Example cases\n\nGeneration cases:\n- `ui_checkout_mockup`: mobile checkout screen with strict text + layout rules\n- `coffee_flyer_generation`: marketing flyer with exact copy constraints\n\nEditing cases:\n- `vto_jacket_tryon`: virtual try-on with reference person + garment\n- `logo_year_edit`: precision logo text edit\n\n## Required assets (editing harness)\n\nThe editing harness expects these files in `images/`:\n\n- `images/base_woman.png`\n- `images/jacket.png`\n- `images/logo_generation_1.png`\n\n## Results layout\n\nEach harness writes into its own `results/` folder:\n\n- `results/<run_id>/results.json`: per-example outputs and grades\n- `results/<run_id>/results.csv`: tabular results\n- `results/<run_id>/summary.json`: aggregated metrics\n- `results/<run_id>/artifacts/*.png`: generated or edited images\n\nTip: keep `results/` out of git (delete after runs or add to `.gitignore`).\n\n## Common CLI flags\n\nBoth harnesses accept a shared core set of flags:\n\n- `--results-dir`, `--run-name`\n- `--model` (image model under test), `--judge-model` (LLM judge)\n- `--num-images` (images per case)\n- `--cases` (comma-separated list of case ids)\n\nGeneration harness adds `--image-size` and `--run-ocr` (text extraction check).\n\n## Suggested workflows\n\n- Quick smoke test:\n  - `python generation_harness/run_imagegen_evals.py --cases ui_checkout_mockup`\n  - `python editing_harness/run_imagegen_evals.py --cases vto_jacket_tryon`\n- Compare model variants: run once per model name and compare `summary.json`.\n- Use `--run-ocr` on the flyer case to validate exact text rendering.\n\n## Adapting for your use case\n\n- Replace prompts/criteria in the harness scripts.\n- Update JSON schemas to match your rubric (and parse them into `Score` values).\n- Add new cases by creating new `TestCase` objects and graders.\n- Swap the output store to change where artifacts are written.\n\n## Notes\n\n- This harness uses the OpenAI Images API and a judge model. Costs scale with\n  the number of images and judge calls.\n- Judge outputs are best treated as signals, not ground truth. Keep a small\n  human review loop for high-stakes decisions.\n\n<!-- File: examples/evals/realtime_evals/crawl_harness/README.md (Tokens: ~739 | Category: anthropic-skill) -->\n# Crawl Harness\n\nDeterministic single-turn replay for Realtime evals. This harness feeds fixed audio\ninputs to a Realtime session so you can compare runs with minimal variance.\nIt is best for quick iteration and controlled comparisons.\n\n## What it does\n\n- Loads a CSV of eval prompts from `crawl_harness/data/`.\n- Uses TTS to synthesize a fixed input audio file per row.\n- Streams audio into the Realtime API in fixed-size chunks.\n- Captures the first assistant turn (text/audio/tool calls).\n- Grades tool-call correctness and tool-call-arg correctness only.\n- Writes artifacts into `crawl_harness/results/<run_id>/`.\n\n## Inputs\n\nCSV columns (required):\n\n- `example_id`: Unique ID for the datapoint.\n- `user_text`: Text prompt to synthesize into audio.\n- `gt_tool_call`: Expected tool name or empty if no tool call is expected.\n- `gt_tool_call_arg`: Expected tool arguments as JSON string or empty.\n\n## Outputs\n\nEach run writes:\n\n- `results.csv`: Per-example outputs and grades.\n- `summary.json`: Aggregate metrics (latency + correctness).\n- `plots/*.png`: Styled score, latency, token, and status charts.\n- `audio/<example_id>/input.wav`: The TTS input audio.\n- `audio/<example_id>/output.wav`: The model output audio (if any).\n- `events/<example_id>.jsonl`: Realtime event stream for the datapoint.\n\n## How it works\n\n1. TTS generates PCM audio for each row.\n2. PCM is wrapped into a WAV file for easy playback.\n3. Audio is streamed into a Realtime session with fixed chunk sizes.\n4. The harness listens for `response.done` and records tool calls + output.\n5. Grades are computed:\n   - `tool_call_correctness`: correct tool chosen (or no tool call when none expected).\n   - `tool_call_arg_correctness`: expected args present in the tool call.\n\n## Run\n\nFrom repo root:\n\n```bash\npython crawl_harness/run_realtime_evals.py\n```\n\nCommon options:\n\n- `--data-csv`: Path to a CSV file.\n- `--results-dir`: Output folder (defaults to `crawl_harness/results`).\n- `--run-name`: Optional run name (defaults to timestamp).\n- `--model`: Realtime model name.\n- `--tts-model`: TTS model name.\n- `--system-prompt-file`: System prompt file path.\n- `--tools-file`: Tools JSON file path.\n- `--voice`: Output voice name.\n- `--chunk-ms`: Audio chunk size in ms.\n- `--sample-rate-hz`: Input sample rate in Hz.\n- `--input-audio-format`: Input audio format (default `pcm16`).\n- `--output-audio-format`: Output audio format (default `pcm16`).\n- `--real-time`: Stream audio in real-time cadence.\n- `--max-examples`: Limit number of examples for quick checks.\n- `--skip-plots`: Skip post-run PNG chart generation.\n\n## Adapt\n\n- Add or replace rows in `crawl_harness/data/customer_service_synthetic.csv`.\n- Change tools or system prompt in `shared/`.\n- Update grading logic in `run_realtime_evals.py`.\n\n## Notes\n\n- The harness only evaluates the first assistant turn and does not provide tool outputs.\n- `results.csv` includes an `event_log_path` column pointing to the JSONL trace.\n- If you expect tools to be called reliably, adjust the system prompt or tool choice.\n\n<!-- File: examples/evals/realtime_evals/README.md (Tokens: ~1207 | Category: anthropic-skill) -->\n# Realtime Evals\n\nThis folder contains three evaluation harnesses for the Realtime API that increase in complexity:\n**crawl → walk → run**.\n\n- **crawl**: synthetic single-turn (TTS → stream → grade)\n- **walk**: deterministic replay of saved phone-audio (more realistic, still comparable)\n- **run**: model-simulated multi-turn conversations (finds multi-turn/tooling failures)\n\nDepending on your realtime eval maturity, point Codex (or your preferred coding assistant) to the folder that fits your use case so it can adapt the harness and bootstrap sample data; you can also have it run the included tests to confirm everything works.\n\n## Quickstart\n\nPython 3.12+ required.\n\n```bash\nmake install\nsource .venv/bin/activate\nexport OPENAI_API_KEY=\"your_api_key\"\n```\n\n`make install` creates the local `.venv` and installs both runtime and dev dependencies. It uses `uv` when available and otherwise falls back to `python -m venv` plus `pip install -r requirements.txt -r requirements-dev.txt`.\n\nRun a first command per harness. If uv is not installed, replace `uv run` with `python` and run these scripts with your `.venv` activated:\n\n- Crawl: `uv run python crawl_harness/run_realtime_evals.py`\n- Walk: install ffmpeg (`brew install ffmpeg`), then:\n  - `uv run python walk_harness/generate_audio.py`\n  - `uv run python walk_harness/run_realtime_evals.py`\n- Run: `uv run python run_harness/run_realtime_evals.py --max-examples 1`\n\n## Dev commands\nUse the root `Makefile` for common checks. Run `make install` first to create `.venv`. These targets work with or without `uv`: when `uv` is installed they run through `uv run`, and otherwise they use the matching tool binaries from the local `.venv`.\n\n- `make install`\n- `make streamlit`\n- `make format`\n- `make lint`\n- `make lint-fix`\n- `make typecheck`\n- `make test`\n\n## [Crawl (synthetic single-turn)](./crawl_harness)\n\n**Best for:** Fast iteration and controlled comparisons.\n\n- Uses text prompts from a CSV and synthesizes TTS audio per row.\n- Streams fixed-size audio chunks into a realtime session.\n- Captures the first assistant turn and grades tool-call correctness.\n- Writes results, summary metrics, audio, and event logs per datapoint.\n\nRun:\n\n```\nuv run python crawl_harness/run_realtime_evals.py\n```\n\n## [Walk (saved audio replay)](./walk_harness)\n\n**Best for:** Realism with reproducibility.\n\n- Replays saved, phone-compressed audio (G.711 mu-law WAV at 8 kHz).\n- Keeps chunking and turn boundaries fixed (VAD off).\n- Captures responses, tool calls, latencies, and event logs per datapoint.\n- Output audio is saved as PCM16 WAV for easy listening.\n\nRun:\n\n```\nuv run python walk_harness/generate_audio.py\nuv run python walk_harness/run_realtime_evals.py\n```\n\n## [Run (multi-turn simulation)](./run_harness)\n\n**Best for:** Discovering multi-turn failures and tool behavior.\n\n- Uses a simulator model to generate user turns and an assistant model under test.\n- Streams simulator audio into the assistant with fixed chunking.\n- Mocks tool outputs deterministically and grades with LLM-as-judge.\n- Writes results, summaries, transcripts, and event logs per simulation.\n\nRun:\n\n```\nuv run python run_harness/run_realtime_evals.py\n```\n\n## Results layout\n\nEach harness writes into its own `results/` folder:\n\n- `results/<run_id>/results.csv`: per-example outputs and grades\n- `results/<run_id>/summary.json`: aggregate metrics\n- `results/<run_id>/plots/*.png`: warm-editorial summary charts for scores, latency, tokens, and run shape\n- `results/<run_id>/events/*.jsonl`: full realtime event stream per datapoint\n\nThe shared typed representation of these artifacts for the crawl, walk, and run harnesses lives in `shared/result_types.py`.\n\nTo render charts for an existing run after the fact:\n\n```bash\nuv run python plot_eval_results.py --run-dir run_harness/results/<run_id>\n```\n\n## Results Viewer\n\nUse the Streamlit results viewer to browse saved runs from `crawl_harness`, `walk_harness`, and `run_harness` without opening the raw artifacts by hand.\n\n- `Comparison View`: select a harness, choose one or more saved runs, and compare summary metrics, scores, latency, and token usage across runs.\n- `Run Viewer`: inspect one saved run in detail. Crawl and walk runs show row-level audio artifacts and event logs; run-harness runs use a Simulation Viewer with transcripts, event logs, and turn audio.\n\nRun it from this directory with either:\n\n```bash\nmake streamlit\n```\n\nor:\n\n```bash\ncd results_viewer\nuv run streamlit run app.py\n```\n\nThen open the local Streamlit URL, usually `http://localhost:8501`.\n\n## Common CLI flags\n\nAll harnesses share a core set of flags so you can switch between them easily:\n\n- `--data-csv`, `--results-dir`, `--run-name`\n- `--model` (assistant under test), `--system-prompt-file`, `--tools-file`\n- `--chunk-ms`, `--sample-rate-hz`, `--real-time`\n- `--max-examples` (quick smoke tests)\n- `--skip-plots` (skip post-run PNG chart generation)\n\nRun harness adds multi-turn and simulator-specific flags (see its README).\n\n<!-- File: examples/evals/realtime_evals/results_viewer/README.md (Tokens: ~451 | Category: anthropic-skill) -->\n# Results Viewer\n\nThis directory contains a [Streamlit](https://streamlit.io/) app for exploring\nsaved realtime eval runs from the crawl, walk, and run harnesses, plus\nbootstrap-generated eval folders created under `examples/evals/realtime_evals/`.\n\nThe app auto-discovers run directories under:\n\n- `crawl_harness/results/`\n- `walk_harness/results/`\n- `run_harness/results/`\n- `*_realtime_eval/results/` when the folder includes `bootstrap_manifest.json`\n\n## What It Shows\n\n- **Comparison View**: compare summary metrics across one or more saved runs\n- **Run Viewer**: inspect a single saved run, including:\n  - `results.csv` rows\n  - crawl/walk input and output audio artifacts\n  - crawl/walk per-example event logs\n  - run-harness simulation transcripts, event logs, and turn audio\n\nNote: `run_harness` runs are inspected via the app's **Simulation Viewer**,\nwhich has a different UI and a different artifact set than the crawl/walk\nviewer.\n\n## Run Locally\n\nFrom `examples/evals/realtime_evals/`:\n\n```bash\nuv venv .venv\nsource .venv/bin/activate\nuv sync --group dev\ncd results_viewer\nuv run streamlit run app.py\n```\n\nThen open the local URL that Streamlit prints, usually\n`http://localhost:8501`.\n\nIf you are using the pip-based install path instead of `uv`, install the dev\ndependencies first so `streamlit` is available:\n\n```bash\npip install -r requirements.txt -r requirements-dev.txt\ncd results_viewer\nstreamlit run app.py\n```\n\n## Expected Data Layout\n\nThe viewer expects each saved run directory to contain:\n\n- `summary.json` for aggregate metrics\n- `results.csv` for per-example results\n\nFor crawl and walk runs, the app can also display:\n\n- `audio/<example_id>/input.wav`\n- `audio/<example_id>/output.wav`\n- `events/<example_id>.jsonl`\n\nThe app discovers runs recursively, so nested result directories are fine as\nlong as those files are present.\n\n<!-- File: examples/evals/realtime_evals/run_harness/data/sim_cs_address_update.json (Tokens: ~300 | Category: anthropic-skill) -->\n# Data Skill\n\nPath: `examples/evals/realtime_evals/run_harness/data/sim_cs_address_update.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/run_harness/data/sim_cs_address_update.json)\n\n<!-- File: examples/evals/realtime_evals/run_harness/data/sim_cs_order_status.json (Tokens: ~300 | Category: anthropic-skill) -->\n# Data Skill\n\nPath: `examples/evals/realtime_evals/run_harness/data/sim_cs_order_status.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/run_harness/data/sim_cs_order_status.json)\n\n<!-- File: examples/evals/realtime_evals/run_harness/data/sim_cs_refund.json (Tokens: ~300 | Category: anthropic-skill) -->\n# Data Skill\n\nPath: `examples/evals/realtime_evals/run_harness/data/sim_cs_refund.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/run_harness/data/sim_cs_refund.json)\n\n<!-- File: examples/evals/realtime_evals/run_harness/README.md (Tokens: ~667 | Category: anthropic-skill) -->\n# Run Harness\n\nModel-simulated multi-turn eval harness for the Realtime API. This harness uses\none realtime model as the user simulator (audio-only) and another realtime model\nas the assistant under test. It replays fixed chunking (VAD off), mocks tools\nfrom the simulation JSON, and records full traces.\n\n## What it does\n\n- Loads `run_harness/data/simulations.csv` with pandas.\n- Reads per-simulation JSON files defining scenario, simulator identity, tool mocks,\n  and LLM-as-judge grading criteria.\n- Generates user audio turns via a realtime simulator model (audio-only).\n- Streams user audio to the assistant in fixed-size chunks and commits manually.\n- Captures assistant audio/text, tool calls, tool outputs, and latencies.\n- Grades turn-level and trace-level criteria with an LLM-as-judge.\n- Writes `results.csv`, `summary.json`, and full trace logs under `run_harness/results/`.\n- Renders styled PNG plots under `run_harness/results/<run_id>/plots/` by default.\n\n## Files\n\n- `run_harness/run_realtime_evals.py`: Run harness script.\n- `run_harness/data/simulations.csv`: Index of simulation files.\n- `run_harness/data/sim_*.json`: Simulation definitions (this repo currently ships 3 examples).\n- `run_harness/results/<run_id>/events/*.jsonl`: Full event trace per simulation.\n- `run_harness/results/<run_id>/conversations/*.txt`: Human-readable transcript with tool calls.\n\n## Simulation definitions (high level)\n\nEach `sim_*.json` file defines:\n\n- **the scenario** (what the user is trying to do)\n- **tool mocks** (what each tool should return when called)\n- **judge rubric** (what should be graded per turn / overall)\n\nThis lets you run repeatable multi-turn evals without needing live backend integrations.\n\n## How to run\n\nFrom repo root:\n\n```bash\npython run_harness/run_realtime_evals.py --max-examples 1\n```\n\nCommon options:\n\n- `--data-csv`: Simulation index CSV.\n- `--model`: Alias for the assistant model under test.\n- `--assistant-model`: Realtime model under test (overrides `--model`).\n- `--simulator-model`: Realtime model used as the user.\n- `--system-prompt-file`: Alias for assistant system prompt file.\n- `--tools-file`: Alias for assistant tools file.\n- `--assistant-system-prompt-file`: Override assistant system prompt file (more specific).\n- `--assistant-tools-file`: Override tools file for the assistant (more specific).\n- `--simulator-system-prompt`: Override simulator system prompt (string).\n- `--chunk-ms`, `--sample-rate-hz`, `--real-time`.\n- `--max-turns`, `--max-examples`.\n- `--skip-plots`.\n\nNotes:\n\n- Temperature and max output tokens are intentionally not supported.\n- The simulator emits audio only. Transcripts are captured from audio deltas if available.\n- Tool outputs are mocked deterministically from each simulation JSON.\n- `results.csv` includes an `event_log_path` column pointing to the JSONL trace.\n\n<!-- File: examples/evals/realtime_evals/shared/tools.json (Tokens: ~300 | Category: anthropic-skill) -->\n# Shared Skill\n\nPath: `examples/evals/realtime_evals/shared/tools.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/shared/tools.json)\n\n<!-- File: examples/evals/realtime_evals/skills/bootstrap-realtime-eval/agents/openai.yaml (Tokens: ~300 | Category: anthropic-skill) -->\n# Agents Skill\n\nPath: `examples/evals/realtime_evals/skills/bootstrap-realtime-eval/agents/openai.yaml`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/skills/bootstrap-realtime-eval/agents/openai.yaml)\n\n<!-- File: examples/evals/realtime_evals/skills/bootstrap-realtime-eval/references/harness-selection.md (Tokens: ~300 | Category: anthropic-skill) -->\n# References Skill\n\nPath: `examples/evals/realtime_evals/skills/bootstrap-realtime-eval/references/harness-selection.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/skills/bootstrap-realtime-eval/references/harness-selection.md)\n\n<!-- File: examples/evals/realtime_evals/skills/bootstrap-realtime-eval/SKILL.md (Tokens: ~1807 | Category: anthropic-skill) -->\n---\nname: bootstrap-realtime-eval\ndescription: Bootstrap a new realtime eval folder inside this cookbook repo by choosing the right harness from examples/evals/realtime_evals, scaffolding prompt/tools/data files, generating a useful README, and validating it with smoke, full eval, and test runs. Use when a user wants to start a new crawl, walk, or run realtime eval in this repository.\n---\n\n# Bootstrap Realtime Eval\n\nUse this skill when the user wants a new realtime eval scaffold under `examples/evals/realtime_evals/`.\n\nThis skill is repo-specific. Do not copy harness code into the generated folder. The generated eval should point at the shared harnesses already in:\n\n- `examples/evals/realtime_evals/crawl_harness`\n- `examples/evals/realtime_evals/walk_harness`\n- `examples/evals/realtime_evals/run_harness`\n\n## Inputs To Collect\n\nAlways ask the user for the minimum set needed to choose and scaffold the eval before you create files, run the scaffold script, or author starter data. Do not skip this just because you can infer a default.\n\nAsk for:\n\n- Eval name\n- Goal or scenario\n- Harness choice, or enough context to recommend one\n- System prompt path or inline text\n- Tools JSON path or tool descriptions\n- Data path or source materials\n- Desired graders\n\nIf the user does not know which harness they want, explain the options briefly and recommend one. See `references/harness-selection.md`.\n\nWhen the user asks for synthetic audio but does not specify a harness, default to `crawl` text-to-TTS unless they need the generated audio to carry particular noise, telephony artifacts, speaker characteristics, or other replay-specific properties. Use `walk` for those cases.\n\nKeep the questions concise and grouped into one short batch whenever possible.\n\nIf the user only provides `user_text` or a short task description, still ask the questions above first. If they answer only partially, then infer the remaining low-risk details, call out the assumptions, and make the scaffold easy to revise later.\n\n## Workflow\n\n1. Ask the user for the required inputs first.\n   - Do this before making files or selecting a final harness.\n   - If the user already supplied some of the inputs, ask only for the missing ones.\n   - If you recommend a harness, wait for the user response before scaffolding.\n\n2. Pick the harness.\n   - `crawl`: single-turn text-to-TTS.\n   - `walk`: replay saved audio or generate audio from text rows.\n   - `run`: multi-turn simulation with tool mocks and judge criteria.\n   - If the user wants synthetic audio but does not care about replay-specific audio characteristics, prefer `crawl` over `walk`.\n\n3. Normalize the inputs.\n   - If the user gives inline prompt or tool content, write it into the generated folder.\n   - If the user gives CSV data, inspect it with pandas before wiring it in.\n   - If the data is not yet harness-ready, scaffold the files and then have you author the starter dataset directly in those files.\n   - If the user only gives `user_text`, infer `example_id` values and leave optional grading fields blank unless you have enough signal to fill them.\n   - Ground the starter data in the use case, prompts, tools, and any other material the user provided.\n   - Make the starter data realistic. Put yourself in the shoes of the end user in that use case and craft datapoints that a real user would plausibly say or do.\n\n4. Be proactive when data is missing.\n   - `crawl`: you should author 3 starter rows covering one happy path and a couple of nearby variants or edge cases.\n   - `walk`: you should author 3 source CSV rows and prepare the audio-generation step so the user can create audio immediately.\n   - `run`: you should author 2 starter simulations, not 1.\n   - Do not rely on deterministic script-generated samples for use-case-specific starter data.\n   - Show the generated starter samples to the user, ask for a quick greenlight or correction, then expand only after feedback when the task calls for more coverage.\n\n5. Run the scaffold script:\n\n```bash\npython examples/evals/realtime_evals/skills/bootstrap-realtime-eval/scripts/bootstrap_realtime_eval.py --name \"<eval_name>\" --harness \"<crawl|walk|run>\"\n```\n\nAdd flags for prompt, tools, data, graders, and run-specific fields as needed. Read the script help if you need the exact flag names.\n\n6. Review the generated folder.\n   - Confirm the README is accurate for the selected harness.\n   - Confirm the system prompt, tools, and data files point at the generated folder.\n   - If the user provided real data, make sure it is wired in instead of leaving starter placeholders.\n   - If you authored inferred starter data, show the user the generated rows or simulations before scaling up.\n\n7. Enrich the scaffold.\n   - For `crawl` and `walk`, make the CSV realistic and ensure the expected tool columns are present.\n   - For `walk`, if the dataset lacks `audio_path`, use the shared `walk_harness/generate_audio.py` flow described in the README.\n   - For `run`, make sure `simulations.csv` and the starter `sim_*.json` file reflect the user’s scenario, tool mocks, and graders.\n\n8. Validate before returning.\n   - Run a smoke eval for the generated folder.\n   - If the smoke eval succeeds, automatically run the full eval for the generated folder in the same turn.\n   - If the smoke eval fails, stop and fix or report the blocker before attempting the full eval.\n   - Run `pytest examples/evals/realtime_evals/tests -q`.\n   - Inspect the generated README and at least one generated data file.\n\n## Hard Rules\n\n- Keep the generated folder inside `examples/evals/realtime_evals/`.\n- Ask the user for the missing setup information before scaffolding. Do not jump straight to building a default eval when the request is to create a new eval.\n- Do not duplicate the main harness scripts into the generated folder.\n- Do not write generic placeholder starter data when the use case provides enough context to do better.\n- Use the prompt, tools, and scenario details to make the initial datapoints feel like a real user interaction.\n- The README must include:\n  - why this harness was chosen\n  - which files to edit first\n  - smoke and full run commands\n  - data contract notes\n  - troubleshooting notes\n- Prefer assistant-specific flags for run harness commands:\n  - `--assistant-system-prompt-file`\n  - `--assistant-tools-file`\n\n## Completion Criteria\n\nTreat the task as complete only when:\n\n- the folder exists under `examples/evals/realtime_evals/<name>_realtime_eval/`\n- `README.md`, `system_prompt.txt`, and `tools.json` exist\n- harness-specific starter data exists, authored by you when the user did not provide it\n- the smoke command has been run or is blocked for a clear reason\n- if the smoke command succeeded, the full eval command has also been run or is blocked for a clear reason\n- `pytest examples/evals/realtime_evals/tests -q` has been run\n\n## Learnings\n\nWhen this skill uncovers a reusable workflow or harness constraint that should guide future bootstrap work, add a short note here.\n\nKeep learnings concise and action-oriented:\n\n- **Problem -> Fix -> Why**\n\nOnly add items that are likely to help future realtime-eval scaffolding in this repo. Remove stale items when they no longer apply.\n\n- **`gpt-realtime` temperature is unsupported** -> Do not add a `temperature` field or CLI flag when scaffolding `gpt-realtime` evals -> Avoids invalid config and keeps runs aligned with the realtime harness constraints.\n- **Starter samples should come from the model, not the scaffold script** -> Use the script to create the folder structure and template files, then have you author the initial rows or simulations from the user’s use case -> Produces more relevant starter data and makes iteration easier with the user.\n- **Synthetic audio can be overfit to the wrong harness** -> Default unspecified synthetic-audio requests to `crawl` text-to-TTS and reserve `walk` for replay-specific audio characteristics like noise or telephony artifacts -> Keeps the bootstrap path simpler unless audio realism is the actual target.\n- **Smoke-only validation can leave scaffolds half-proven** -> After a successful smoke run, automatically run the full eval before declaring the bootstrap complete -> Catches dataset-wide or late-row failures that a one-example smoke test misses.\n\n<!-- File: examples/evals/realtime_evals/walk_harness/README.md (Tokens: ~300 | Category: anthropic-skill) -->\n# Walk_harness Skill\n\nPath: `examples/evals/realtime_evals/walk_harness/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/evals/realtime_evals/walk_harness/README.md)\n\n<!-- File: examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.mcp.json (Tokens: ~300 | Category: anthropic-skill) -->\n# Bls Skill\n\nPath: `examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.mcp.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.mcp.json)\n\n<!-- File: examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/.agents/plugins/marketplace.json (Tokens: ~300 | Category: marketplace) -->\n# Claude Plugin Marketplace Catalog\n\nPath: `examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/.agents/plugins/marketplace.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/.agents/plugins/marketplace.json)\n\n<!-- File: examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.codex-plugin/plugin.json (Tokens: ~300 | Category: plugin-manifest) -->\n# codex-plugin Plugin Manifest\n\nPath: `examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.codex-plugin/plugin.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/.codex-plugin/plugin.json)\n\n<!-- File: examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/README.md (Tokens: ~300 | Category: plugin-manifest) -->\n# bls Documentation\n\nPath: `examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/openai/openai-cookbook/HEAD/examples/partners/harness_aware_plugin_evals/.codex-home/marketplaces/cookbook-plugins/plugins/bls/README.md)\n</yakaai_skills>"}